Output:
Chunk ID: 9d8955
Wall time: 1.7806 seconds
Process exited with code 1
Original token count: 3263
Output:
(EngineCore_DP0 pid=3705058) INFO 02-17 00:22:54 [core.py:96] Initializing a V1 LLM engine (v0.15.1) with config: model='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', speculative_config=None, tokenizer='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [4096], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 256, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': True}, 'local_cache_dir': None, 'static_all_moe_layers': []}
(EngineCore_DP0 pid=3705058) INFO 02-17 00:22:56 [parallel_state.py:1212] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://216.81.248.184:34285 backend=nccl
(EngineCore_DP0 pid=3705058) INFO 02-17 00:22:56 [parallel_state.py:1423] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
(EngineCore_DP0 pid=3705058) ERROR 02-17 00:22:57 [core.py:946] EngineCore failed to start.
(EngineCore_DP0 pid=3705058) ERROR 02-17 00:22:57 [core.py:946] Traceback (most recent call last):
(EngineCore_DP0 pid=3705058) ERROR 02-17 00:22:57 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 937, in run_engine_core
(EngineCore_DP0 pid=3705058) ERROR 02-17 00:22:57 [core.py:946]     engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore_DP0 pid=3705058) ERROR 02-17 00:22:57 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 691, in __init__
(EngineCore_DP0 pid=3705058) ERROR 02-17 00:22:57 [core.py:946]     super().__init__(
(EngineCore_DP0 pid=3705058) ERROR 02-17 00:22:57 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 105, in __init__
(EngineCore_DP0 pid=3705058) ERROR 02-17 00:22:57 [core.py:946]     self.model_executor = executor_class(vllm_config)
(EngineCore_DP0 pid=3705058) ERROR 02-17 00:22:57 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/executor/abstract.py", line 101, in __init__
(EngineCore_DP0 pid=3705058) ERROR 02-17 00:22:57 [core.py:946]     self._init_executor()
(EngineCore_DP0 pid=3705058) ERROR 02-17 00:22:57 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/executor/uniproc_executor.py", line 47, in _init_executor
(EngineCore_DP0 pid=3705058) ERROR 02-17 00:22:57 [core.py:946]     self.driver_worker.init_device()
(EngineCore_DP0 pid=3705058) ERROR 02-17 00:22:57 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/worker/worker_base.py", line 326, in init_device
(EngineCore_DP0 pid=3705058) ERROR 02-17 00:22:57 [core.py:946]     self.worker.init_device()  # type: ignore
(EngineCore_DP0 pid=3705058) ERROR 02-17 00:22:57 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/worker/gpu_worker.py", line 235, in init_device
(EngineCore_DP0 pid=3705058) ERROR 02-17 00:22:57 [core.py:946]     self.requested_memory = request_memory(init_snapshot, self.cache_config)
(EngineCore_DP0 pid=3705058) ERROR 02-17 00:22:57 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/worker/utils.py", line 260, in request_memory
(EngineCore_DP0 pid=3705058) ERROR 02-17 00:22:57 [core.py:946]     raise ValueError(
(EngineCore_DP0 pid=3705058) ERROR 02-17 00:22:57 [core.py:946] ValueError: Free memory on device cuda:0 (14.37/79.25 GiB) on startup is less than desired GPU memory utilization (0.25, 19.81 GiB). Decrease GPU memory utilization or reduce GPU memory used by other processes.
(EngineCore_DP0 pid=3705058) Process EngineCore_DP0:
(EngineCore_DP0 pid=3705058) Traceback (most recent call last):
(EngineCore_DP0 pid=3705058)   File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore_DP0 pid=3705058)     self.run()
(EngineCore_DP0 pid=3705058)   File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
(EngineCore_DP0 pid=3705058)     self._target(*self._args, **self._kwargs)
(EngineCore_DP0 pid=3705058)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 950, in run_engine_core
(EngineCore_DP0 pid=3705058)     raise e
(EngineCore_DP0 pid=3705058)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 937, in run_engine_core
(EngineCore_DP0 pid=3705058)     engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore_DP0 pid=3705058)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 691, in __init__
(EngineCore_DP0 pid=3705058)     super().__init__(
(EngineCore_DP0 pid=3705058)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 105, in __init__
(EngineCore_DP0 pid=3705058)     self.model_executor = executor_class(vllm_config)
(EngineCore_DP0 pid=3705058)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/executor/abstract.py", line 101, in __init__
(EngineCore_DP0 pid=3705058)     self._init_executor()
(EngineCore_DP0 pid=3705058)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/executor/uniproc_executor.py", line 47, in _init_executor
(EngineCore_DP0 pid=3705058)     self.driver_worker.init_device()
(EngineCore_DP0 pid=3705058)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/worker/worker_base.py", line 326, in init_device
(EngineCore_DP0 pid=3705058)     self.worker.init_device()  # type: ignore
(EngineCore_DP0 pid=3705058)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/worker/gpu_worker.py", line 235, in init_device
(EngineCore_DP0 pid=3705058)     self.requested_memory = request_memory(init_snapshot, self.cache_config)
(EngineCore_DP0 pid=3705058)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/worker/utils.py", line 260, in request_memory
(EngineCore_DP0 pid=3705058)     raise ValueError(
(EngineCore_DP0 pid=3705058) ValueError: Free memory on device cuda:0 (14.37/79.25 GiB) on startup is less than desired GPU memory utilization (0.25, 19.81 GiB). Decrease GPU memory utilization or reduce GPU memory used by other processes.
2026-02-17 00:22:57 | ERROR    | veena3modal.services.tts_runtime | ❌ Failed to initialize TTS runtime: Engine core initialization failed. See root cause above. Failed core proc(s): {}
Traceback (most recent call last):
  File "/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py", line 167, in initialize_runtime
    model = SparkTTSModel(
  File "/home/ubuntu/veenaModal/veena3modal/core/model_loader.py", line 128, in __init__
    self.engine = AsyncLLMEngine.from_engine_args(engine_args)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/async_llm.py", line 257, in from_engine_args
    return cls(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/async_llm.py", line 155, in __init__
    self.engine_core = EngineCoreClient.make_async_mp_client(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 122, in make_async_mp_client
    return AsyncMPClient(*client_args)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 819, in __init__
    super().__init__(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 479, in __init__
    with launch_core_engines(vllm_config, executor_class, log_stats) as (
  File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
    next(self.gen)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/utils.py", line 933, in launch_core_engines
    wait_for_engine_startup(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/utils.py", line 992, in wait_for_engine_startup
    raise RuntimeError(
RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
Traceback (most recent call last):
  File "/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py", line 167, in initialize_runtime
    model = SparkTTSModel(
  File "/home/ubuntu/veenaModal/veena3modal/core/model_loader.py", line 128, in __init__
    self.engine = AsyncLLMEngine.from_engine_args(engine_args)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/async_llm.py", line 257, in from_engine_args
    return cls(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/async_llm.py", line 155, in __init__
    self.engine_core = EngineCoreClient.make_async_mp_client(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 122, in make_async_mp_client
    return AsyncMPClient(*client_args)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 819, in __init__
    super().__init__(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 479, in __init__
    with launch_core_engines(vllm_config, executor_class, log_stats) as (
  File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
    next(self.gen)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/utils.py", line 933, in launch_core_engines
    wait_for_engine_startup(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/utils.py", line 992, in wait_for_engine_startup
    raise RuntimeError(
RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/ubuntu/veenaModal/veena3modal/local_server.py", line 465, in <module>
    main()
  File "/home/ubuntu/veenaModal/veena3modal/local_server.py", line 426, in main
    initialize_local_runtime(
  File "/home/ubuntu/veenaModal/veena3modal/local_server.py", line 224, in initialize_local_runtime
    runtime = initialize_runtime(
  File "/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py", line 272, in initialize_runtime
    raise RuntimeError(f"TTS runtime initialization failed: {e}") from e
RuntimeError: TTS runtime initialization failed: Engine core initialization failed. See root cause above. Failed core proc(s): {}


# Collaboration Mode: Default

You are now in Default mode. Any previous instructions for other modes (e.g. Plan mode) are no longer active.

Your active mode changes only when new developer instructions with a different `<collaboration_mode>...</collaboration_mode>` change it; user requests or tool descriptions do not change mode by themselves. Known mode names are Default and Plan.

## request_user_input availability

The `request_user_input` tool is unavailable in Default mode. If you call it while in Default mode, it will return an error.

If a decision is necessary and cannot be discovered from local context, ask the user directly. However, in Default mode you should strongly prefer executing the user's request rather than stopping to ask questions.

Summary: auto

## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) When `SKILL.md` references relative paths (e.g., `scripts/foo.py`), resolve them relative to the skill directory listed above first, and only consider other paths if needed.
  3) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  4) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  5) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.


pgrep -af "veena3modal.local_server|python -m veena3modal.local_server" | head


Output:
Chunk ID: 47b2a6
Wall time: 0.0514 seconds
Process exited with code 0
Original token count: 26
Output:
3705120 /bin/bash -lc pgrep -af "veena3modal.local_server|python -m veena3modal.local_server" | head


Summary: auto


I hit a startup failure: vLLM wanted ~19.8GB at `gpu_memory_utilization=0.25` but only ~14.4GB is free. I’ll restart with `--gpu-memory 0.15` (fits within available memory) and proceed from there.


cd /home/ubuntu/veenaModal && VEENA3_TIMING_LOGS=true python3 -m veena3modal.local_server --skip-download --model-path /home/ubuntu/veenaModal/models/spark_tts_4speaker --port 8000 --gpu-memory 0.15 --num-engines 1 --log-level INFO


Output:
Chunk ID: b26763
Wall time: 1.0011 seconds
Process running with session ID 32618
Original token count: 454
Output:
2026-02-17 00:23:01 | INFO     | veena3.local | Auth bypass enabled (local mode). Use --auth to require API keys.
2026-02-17 00:23:01 | INFO     | veena3.local | Resolved model paths:
2026-02-17 00:23:01 | INFO     | veena3.local |   LLM (vLLM):    /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
2026-02-17 00:23:01 | INFO     | veena3.local |   BiCodec:       /home/ubuntu/veenaModal/models/spark_tts_4speaker
2026-02-17 00:23:01 | INFO     | veena3.local | ============================================================
2026-02-17 00:23:01 | INFO     | veena3.local | Initializing Veena3 TTS Runtime (local)
2026-02-17 00:23:01 | INFO     | veena3.local |   Model:       /home/ubuntu/veenaModal/models/spark_tts_4speaker
2026-02-17 00:23:01 | INFO     | veena3.local |   LLM path:    /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
2026-02-17 00:23:01 | INFO     | veena3.local |   BiCodec:     /home/ubuntu/veenaModal/models/spark_tts_4speaker
2026-02-17 00:23:01 | INFO     | veena3.local |   Device:      cuda
2026-02-17 00:23:01 | INFO     | veena3.local |   GPU Memory:  15% (total)
2026-02-17 00:23:01 | INFO     | veena3.local |   Engines:     1
2026-02-17 00:23:01 | INFO     | veena3.local |   SR:          disabled
2026-02-17 00:23:01 | INFO     | veena3.local | ============================================================
2026-02-17 00:23:01 | INFO     | veena3modal.services.tts_runtime | Initializing TTS runtime...
2026-02-17 00:23:01 | INFO     | veena3modal.services.tts_runtime |   Model path: /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
2026-02-17 00:23:01 | INFO     | veena3modal.services.tts_runtime |   BiCodec path: /home/ubuntu/veenaModal/models/spark_tts_4speaker
2026-02-17 00:23:01 | INFO     | veena3modal.services.tts_runtime |   SR enabled: False, path: None


Output:
Chunk ID: 4a6a76
Wall time: 5.0007 seconds
Process running with session ID 32618
Original token count: 509
Output:
2026-02-17 00:23:07 | INFO     | veena3modal.services.tts_runtime | Loading SparkTTS model with vLLM...
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
🚀 Initializing Spark TTS Model
📁 Model path: /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
🗣  Speakers: 12 predefined (lipakshi, vardan, reet, Nandini, krishna, anika, adarsh, Nilay, Aarvi, Asha, Bittu, Mira)
🎭 Emotions: 10 tags in [bracket] format

📝 Loading tokenizer from model...
✅ Tokenizer loaded: 165168 tokens

🔍 Validating 12 speakers for Spark TTS...
  lipakshi → <|speaker_0|>: 7 tokens
  vardan → <|speaker_1|>: 7 tokens
  reet → <|speaker_2|>: 7 tokens
  Nandini → <|speaker_3|>: 7 tokens
  krishna → <|speaker_4|>: 7 tokens
  anika → <|speaker_5|>: 7 tokens
  adarsh → <|speaker_6|>: 7 tokens
  Nilay → <|speaker_7|>: 7 tokens
  Aarvi → <|speaker_8|>: 7 tokens
  Asha → <|speaker_9|>: 7 tokens
  Bittu → <|speaker_10|>: 8 tokens
  Mira → <|speaker_11|>: 8 tokens
✅ All 12 speakers validated with mapping

🔍 Validating 10 emotion tags for Spark TTS...
✅ All 10 emotion tags validated

📋 Sample emotion tags:
  [angry]: 1 tokens
  [curious]: 1 tokens
  [excited]: 1 tokens
  [giggle]: 1 tokens
  [laughs harder]: 1 tokens
  ...

🔧 Initializing vLLM engine with optimizations...
  - dtype: bfloat16
  - max_model_len: 4096
  - gpu_memory_utilization: 0.15
  - enable_prefix_caching: True
  - enforce_eager (disable CUDA graphs): False
INFO 02-17 00:23:08 [model.py:541] Resolved architecture: Qwen2ForCausalLM
INFO 02-17 00:23:08 [model.py:1561] Using max model len 4096
INFO 02-17 00:23:08 [scheduler.py:226] Chunked prefill is enabled with max_num_batched_tokens=4096.
INFO 02-17 00:23:08 [vllm.py:624] Asynchronous scheduling is enabled.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.


Output:
Chunk ID: ad82cd
Wall time: 2.7261 seconds
Process exited with code 1
Original token count: 4955
Output:
(EngineCore_DP0 pid=3705252) INFO 02-17 00:23:09 [core.py:96] Initializing a V1 LLM engine (v0.15.1) with config: model='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', speculative_config=None, tokenizer='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [4096], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 256, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': True}, 'local_cache_dir': None, 'static_all_moe_layers': []}
(EngineCore_DP0 pid=3705252) INFO 02-17 00:23:11 [parallel_state.py:1212] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://216.81.248.184:57607 backend=nccl
(EngineCore_DP0 pid=3705252) INFO 02-17 00:23:11 [parallel_state.py:1423] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
(EngineCore_DP0 pid=3705252) ERROR 02-17 00:23:11 [core.py:946] EngineCore failed to start.
(EngineCore_DP0 pid=3705252) ERROR 02-17 00:23:11 [core.py:946] Traceback (most recent call last):
(EngineCore_DP0 pid=3705252) ERROR 02-17 00:23:11 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 937, in run_engine_core
(EngineCore_DP0 pid=3705252) ERROR 02-17 00:23:11 [core.py:946]     engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore_DP0 pid=3705252) ERROR 02-17 00:23:11 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 691, in __init__
(EngineCore_DP0 pid=3705252) ERROR 02-17 00:23:11 [core.py:946]     super().__init__(
(EngineCore_DP0 pid=3705252) ERROR 02-17 00:23:11 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 105, in __init__
(EngineCore_DP0 pid=3705252) ERROR 02-17 00:23:11 [core.py:946]     self.model_executor = executor_class(vllm_config)
(EngineCore_DP0 pid=3705252) ERROR 02-17 00:23:11 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/executor/abstract.py", line 101, in __init__
(EngineCore_DP0 pid=3705252) ERROR 02-17 00:23:11 [core.py:946]     self._init_executor()
(EngineCore_DP0 pid=3705252) ERROR 02-17 00:23:11 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/executor/uniproc_executor.py", line 47, in _init_executor
(EngineCore_DP0 pid=3705252) ERROR 02-17 00:23:11 [core.py:946]     self.driver_worker.init_device()
(EngineCore_DP0 pid=3705252) ERROR 02-17 00:23:11 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/worker/worker_base.py", line 326, in init_device
(EngineCore_DP0 pid=3705252) ERROR 02-17 00:23:11 [core.py:946]     self.worker.init_device()  # type: ignore
(EngineCore_DP0 pid=3705252) ERROR 02-17 00:23:11 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/worker/gpu_worker.py", line 258, in init_device
(EngineCore_DP0 pid=3705252) ERROR 02-17 00:23:11 [core.py:946]     from vllm.v1.worker.gpu_model_runner import (
(EngineCore_DP0 pid=3705252) ERROR 02-17 00:23:11 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/worker/gpu_model_runner.py", line 153, in <module>
(EngineCore_DP0 pid=3705252) ERROR 02-17 00:23:11 [core.py:946]     from vllm.v1.spec_decode.ngram_proposer import NgramProposer
(EngineCore_DP0 pid=3705252) ERROR 02-17 00:23:11 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/spec_decode/ngram_proposer.py", line 170, in <module>
(EngineCore_DP0 pid=3705252) ERROR 02-17 00:23:11 [core.py:946]     def batch_propose_numba(
(EngineCore_DP0 pid=3705252) ERROR 02-17 00:23:11 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/decorators.py", line 221, in wrapper
(EngineCore_DP0 pid=3705252) ERROR 02-17 00:23:11 [core.py:946]     disp = dispatcher(py_func=func, locals=locals,
(EngineCore_DP0 pid=3705252) ERROR 02-17 00:23:11 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/dispatcher.py", line 772, in __init__
(EngineCore_DP0 pid=3705252) ERROR 02-17 00:23:11 [core.py:946]     self.targetctx = self.targetdescr.target_context
(EngineCore_DP0 pid=3705252) ERROR 02-17 00:23:11 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/registry.py", line 30, in target_context
(EngineCore_DP0 pid=3705252) ERROR 02-17 00:23:11 [core.py:946]     return self._toplevel_target_context
(EngineCore_DP0 pid=3705252) ERROR 02-17 00:23:11 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/utils.py", line 741, in __get__
(EngineCore_DP0 pid=3705252) ERROR 02-17 00:23:11 [core.py:946]     return super().__get__(*args, **kwargs)
(EngineCore_DP0 pid=3705252) ERROR 02-17 00:23:11 [core.py:946]   File "/usr/lib/python3.10/functools.py", line 981, in __get__
(EngineCore_DP0 pid=3705252) ERROR 02-17 00:23:11 [core.py:946]     val = self.func(instance)
(EngineCore_DP0 pid=3705252) ERROR 02-17 00:23:11 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/registry.py", line 18, in _toplevel_target_context
(EngineCore_DP0 pid=3705252) ERROR 02-17 00:23:11 [core.py:946]     return cpu.CPUContext(self.typing_context, self._target_name)
(EngineCore_DP0 pid=3705252) ERROR 02-17 00:23:11 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/cpu.py", line 42, in __init__
(EngineCore_DP0 pid=3705252) ERROR 02-17 00:23:11 [core.py:946]     super().__init__(typingctx, target)
(EngineCore_DP0 pid=3705252) ERROR 02-17 00:23:11 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/base.py", line 254, in __init__
(EngineCore_DP0 pid=3705252) ERROR 02-17 00:23:11 [core.py:946]     self.init()
(EngineCore_DP0 pid=3705252) ERROR 02-17 00:23:11 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/compiler_lock.py", line 35, in _acquire_compile_lock
(EngineCore_DP0 pid=3705252) ERROR 02-17 00:23:11 [core.py:946]     return func(*args, **kwargs)
(EngineCore_DP0 pid=3705252) ERROR 02-17 00:23:11 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/cpu.py", line 51, in init
(EngineCore_DP0 pid=3705252) ERROR 02-17 00:23:11 [core.py:946]     self._internal_codegen = codegen.JITCPUCodegen("numba.exec")
(EngineCore_DP0 pid=3705252) ERROR 02-17 00:23:11 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/codegen.py", line 1172, in __init__
(EngineCore_DP0 pid=3705252) ERROR 02-17 00:23:11 [core.py:946]     initialize_llvm()
(EngineCore_DP0 pid=3705252) ERROR 02-17 00:23:11 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/codegen.py", line 1434, in initialize_llvm
(EngineCore_DP0 pid=3705252) ERROR 02-17 00:23:11 [core.py:946]     ll.initialize()
(EngineCore_DP0 pid=3705252) ERROR 02-17 00:23:11 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/llvmlite/binding/initfini.py", line 17, in initialize
(EngineCore_DP0 pid=3705252) ERROR 02-17 00:23:11 [core.py:946]     raise RuntimeError(
(EngineCore_DP0 pid=3705252) ERROR 02-17 00:23:11 [core.py:946] RuntimeError: llvmlite.binding.initialize() is deprecated and will be removed. LLVM initialization is now handled automatically. Please remove calls to this function from your code and check for other behavioral changes that may have occurred due to LLVM updates.
(EngineCore_DP0 pid=3705252) Process EngineCore_DP0:
(EngineCore_DP0 pid=3705252) Traceback (most recent call last):
(EngineCore_DP0 pid=3705252)   File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore_DP0 pid=3705252)     self.run()
(EngineCore_DP0 pid=3705252)   File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
(EngineCore_DP0 pid=3705252)     self._target(*self._args, **self._kwargs)
(EngineCore_DP0 pid=3705252)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 950, in run_engine_core
(EngineCore_DP0 pid=3705252)     raise e
(EngineCore_DP0 pid=3705252)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 937, in run_engine_core
(EngineCore_DP0 pid=3705252)     engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore_DP0 pid=3705252)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 691, in __init__
(EngineCore_DP0 pid=3705252)     super().__init__(
(EngineCore_DP0 pid=3705252)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 105, in __init__
(EngineCore_DP0 pid=3705252)     self.model_executor = executor_class(vllm_config)
(EngineCore_DP0 pid=3705252)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/executor/abstract.py", line 101, in __init__
(EngineCore_DP0 pid=3705252)     self._init_executor()
(EngineCore_DP0 pid=3705252)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/executor/uniproc_executor.py", line 47, in _init_executor
(EngineCore_DP0 pid=3705252)     self.driver_worker.init_device()
(EngineCore_DP0 pid=3705252)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/worker/worker_base.py", line 326, in init_device
(EngineCore_DP0 pid=3705252)     self.worker.init_device()  # type: ignore
(EngineCore_DP0 pid=3705252)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/worker/gpu_worker.py", line 258, in init_device
(EngineCore_DP0 pid=3705252)     from vllm.v1.worker.gpu_model_runner import (
(EngineCore_DP0 pid=3705252)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/worker/gpu_model_runner.py", line 153, in <module>
(EngineCore_DP0 pid=3705252)     from vllm.v1.spec_decode.ngram_proposer import NgramProposer
(EngineCore_DP0 pid=3705252)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/spec_decode/ngram_proposer.py", line 170, in <module>
(EngineCore_DP0 pid=3705252)     def batch_propose_numba(
(EngineCore_DP0 pid=3705252)   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/decorators.py", line 221, in wrapper
(EngineCore_DP0 pid=3705252)     disp = dispatcher(py_func=func, locals=locals,
(EngineCore_DP0 pid=3705252)   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/dispatcher.py", line 772, in __init__
(EngineCore_DP0 pid=3705252)     self.targetctx = self.targetdescr.target_context
(EngineCore_DP0 pid=3705252)   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/registry.py", line 30, in target_context
(EngineCore_DP0 pid=3705252)     return self._toplevel_target_context
(EngineCore_DP0 pid=3705252)   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/utils.py", line 741, in __get__
(EngineCore_DP0 pid=3705252)     return super().__get__(*args, **kwargs)
(EngineCore_DP0 pid=3705252)   File "/usr/lib/python3.10/functools.py", line 981, in __get__
(EngineCore_DP0 pid=3705252)     val = self.func(instance)
(EngineCore_DP0 pid=3705252)   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/registry.py", line 18, in _toplevel_target_context
(EngineCore_DP0 pid=3705252)     return cpu.CPUContext(self.typing_context, self._target_name)
(EngineCore_DP0 pid=3705252)   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/cpu.py", line 42, in __init__
(EngineCore_DP0 pid=3705252)     super().__init__(typingctx, target)
(EngineCore_DP0 pid=3705252)   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/base.py", line 254, in __init__
(EngineCore_DP0 pid=3705252)     self.init()
(EngineCore_DP0 pid=3705252)   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/compiler_lock.py", line 35, in _acquire_compile_lock
(EngineCore_DP0 pid=3705252)     return func(*args, **kwargs)
(EngineCore_DP0 pid=3705252)   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/cpu.py", line 51, in init
(EngineCore_DP0 pid=3705252)     self._internal_codegen = codegen.JITCPUCodegen("numba.exec")
(EngineCore_DP0 pid=3705252)   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/codegen.py", line 1172, in __init__
(EngineCore_DP0 pid=3705252)     initialize_llvm()
(EngineCore_DP0 pid=3705252)   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/codegen.py", line 1434, in initialize_llvm
(EngineCore_DP0 pid=3705252)     ll.initialize()
(EngineCore_DP0 pid=3705252)   File "/home/ubuntu/.local/lib/python3.10/site-packages/llvmlite/binding/initfini.py", line 17, in initialize
(EngineCore_DP0 pid=3705252)     raise RuntimeError(
(EngineCore_DP0 pid=3705252) RuntimeError: llvmlite.binding.initialize() is deprecated and will be removed. LLVM initialization is now handled automatically. Please remove calls to this function from your code and check for other behavioral changes that may have occurred due to LLVM updates.
2026-02-17 00:23:12 | ERROR    | veena3modal.services.tts_runtime | ❌ Failed to initialize TTS runtime: Engine core initialization failed. See root cause above. Failed core proc(s): {}
Traceback (most recent call last):
  File "/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py", line 167, in initialize_runtime
    model = SparkTTSModel(
  File "/home/ubuntu/veenaModal/veena3modal/core/model_loader.py", line 128, in __init__
    self.engine = AsyncLLMEngine.from_engine_args(engine_args)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/async_llm.py", line 257, in from_engine_args
    return cls(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/async_llm.py", line 155, in __init__
    self.engine_core = EngineCoreClient.make_async_mp_client(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 122, in make_async_mp_client
    return AsyncMPClient(*client_args)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 819, in __init__
    super().__init__(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 479, in __init__
    with launch_core_engines(vllm_config, executor_class, log_stats) as (
  File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
    next(self.gen)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/utils.py", line 933, in launch_core_engines
    wait_for_engine_startup(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/utils.py", line 992, in wait_for_engine_startup
    raise RuntimeError(
RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
Traceback (most recent call last):
  File "/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py", line 167, in initialize_runtime
    model = SparkTTSModel(
  File "/home/ubuntu/veenaModal/veena3modal/core/model_loader.py", line 128, in __init__
    self.engine = AsyncLLMEngine.from_engine_args(engine_args)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/async_llm.py", line 257, in from_engine_args
    return cls(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/async_llm.py", line 155, in __init__
    self.engine_core = EngineCoreClient.make_async_mp_client(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 122, in make_async_mp_client
    return AsyncMPClient(*client_args)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 819, in __init__
    super().__init__(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 479, in __init__
    with launch_core_engines(vllm_config, executor_class, log_stats) as (
  File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
    next(self.gen)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/utils.py", line 933, in launch_core_engines
    wait_for_engine_startup(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/utils.py", line 992, in wait_for_engine_startup
    raise RuntimeError(
RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/ubuntu/veenaModal/veena3modal/local_server.py", line 465, in <module>
    main()
  File "/home/ubuntu/veenaModal/veena3modal/local_server.py", line 426, in main
    initialize_local_runtime(
  File "/home/ubuntu/veenaModal/veena3modal/local_server.py", line 224, in initialize_local_runtime
    runtime = initialize_runtime(
  File "/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py", line 272, in initialize_runtime
    raise RuntimeError(f"TTS runtime initialization failed: {e}") from e
RuntimeError: TTS runtime initialization failed: Engine core initialization failed. See root cause above. Failed core proc(s): {}


sed -n '1,280p' veena3modal/core/model_loader.py


Output:
Chunk ID: ccfae3
Wall time: 0.0506 seconds
Process exited with code 0
Original token count: 1800
Output:
"""
Spark TTS Model Loader

Loads Spark TTS model with vLLM engine and validates speakers/emotions.
Migrated from Veena3/Orpheus to Spark TTS architecture.
"""

import os
import torch
from typing import Optional, Dict, Any
from transformers import AutoTokenizer
from vllm import AsyncLLMEngine, AsyncEngineArgs, SamplingParams

from veena3modal.core.constants import (
    INDIC_EMOTION_TAGS,
    INDIC_SPEAKERS,
    SPEAKER_MAP,
    DEFAULT_MODEL_PATH,
    DEFAULT_MAX_MODEL_LEN,
    VLLM_CONFIG,
)

class SparkTTSModel:
    """
    Spark TTS Model with vLLM inference engine and BiCodec audio tokenizer.

    Model: BayAreaBoys/spark_tts_4speaker
    Architecture: Qwen2ForCausalLM with BiCodec audio tokenization
    Speakers: 12 predefined (mapped to speaker_0-11 as per training)
    Emotions: 10 tags in [bracket] format

    CRITICAL REQUIREMENTS:
    - Load tokenizer with special Spark TTS tokens
    - Configure vLLM with optimizations (prefix caching, CUDA graphs)
    - Support speaker mapping (lipakshi → speaker_0, etc.)
    """

    def __init__(
        self,
        model_path: str = None,
        model_type: str = "indic_speakers",  # Default to indic_speakers for backward compatibility
        hf_token: str = None,
        **engine_kwargs
    ):
        """
        Initialize Spark TTS model with vLLM.

        Args:
            model_path: Path to model directory (local or HF). If None, uses env var or default.
            model_type: Model type (kept for compatibility, always uses indic_speakers)
            hf_token: HuggingFace token for private models
            **engine_kwargs: Additional vLLM engine arguments (override defaults from VLLM_CONFIG)
        """
        self.model_type = "indic_speakers"  # Spark TTS only supports speaker-based

        # Use provided path, or check environment variable, or use default
        if model_path is None:
            model_path = os.environ.get(
                'SPARK_TTS_MODEL_PATH',
                DEFAULT_MODEL_PATH
            )

        self.model_path = model_path
        self.hf_token = hf_token

        print(f"🚀 Initializing Spark TTS Model")
        print(f"📁 Model path: {model_path}")
        print("🗣  Speakers: 12 predefined "
              "(lipakshi, vardan, reet, Nandini, krishna, anika, adarsh, "
              "Nilay, Aarvi, Asha, Bittu, Mira)")
        print(f"🎭 Emotions: 10 tags in [bracket] format")

        # Load tokenizer
        print(f"\n📝 Loading tokenizer from model...")
        tokenizer_kwargs = {
            'trust_remote_code': True,
        }

        # Add HF token for private repos
        if hf_token:
            tokenizer_kwargs['token'] = hf_token

        # Add emotion tags as special tokens
        special_tokens = {
            'additional_special_tokens': INDIC_EMOTION_TAGS
        }

        self.tokenizer = AutoTokenizer.from_pretrained(
            model_path,
            **tokenizer_kwargs
        )

        # Add special tokens
        self.tokenizer.add_special_tokens(special_tokens)

        print(f"✅ Tokenizer loaded: {len(self.tokenizer)} tokens")

        # Validate speakers and emotions
        self._validate_speakers()
        self._validate_emotion_tags()

        # Initialize vLLM engine with Spark TTS optimizations
        print(f"\n🔧 Initializing vLLM engine with optimizations...")

        # Start with default VLLM_CONFIG and override with kwargs
        engine_kwargs_dict = VLLM_CONFIG.copy()
        engine_kwargs_dict.update({
            'model': model_path,
            'tokenizer': model_path,
        })

        # Add HF token for private models
        if hf_token:
            os.environ['HUGGING_FACE_HUB_TOKEN'] = hf_token

        # Override with any user-provided kwargs
        engine_kwargs_dict.update(engine_kwargs)

        # Print configuration
        print(f"  - dtype: {engine_kwargs_dict['dtype']}")
        print(f"  - max_model_len: {engine_kwargs_dict['max_model_len']}")
        print(f"  - gpu_memory_utilization: {engine_kwargs_dict['gpu_memory_utilization']}")
        print(f"  - enable_prefix_caching: {engine_kwargs_dict.get('enable_prefix_caching', False)}")
        print(f"  - enforce_eager (disable CUDA graphs): {engine_kwargs_dict.get('enforce_eager', True)}")

        engine_args = AsyncEngineArgs(**engine_kwargs_dict)
        self.engine = AsyncLLMEngine.from_engine_args(engine_args)

        print(f"✅ vLLM engine initialized successfully!")
        print(f"\n🎉 Spark TTS Model ready for inference!\n")

    def _validate_emotion_tags(self):
        """
        Validate that emotion tags are properly configured for Spark TTS.

        For Spark TTS, emotion tags use [bracket] format and should be
        added as special tokens to the tokenizer.
        """
        print(f"\n🔍 Validating {len(INDIC_EMOTION_TAGS)} emotion tags for Spark TTS...")

        # Check if emotion tags can be tokenized
        for tag in INDIC_EMOTION_TAGS:
            token_ids = self.tokenizer.encode(tag, add_special_tokens=False)
            # Spark TTS emotion tags may be multiple tokens, that's okay
            if not token_ids:
                print(f"⚠  Warning: Emotion tag {tag} produced no tokens")

        print(f"✅ All {len(INDIC_EMOTION_TAGS)} emotion tags validated")

        # Print sample emotion tags
        print(f"\n📋 Sample emotion tags:")
        for i, tag in enumerate(INDIC_EMOTION_TAGS[:5]):
            token_ids = self.tokenizer.encode(tag, add_special_tokens=False)
            print(f"  {tag}: {len(token_ids)} tokens")
        if len(INDIC_EMOTION_TAGS) > 5:
            print(f"  ...")

    def _validate_speakers(self):
        """
        Validate that all speakers are properly mapped for Spark TTS.

        Spark TTS uses speaker tokens like <|speaker_0|> to <|speaker_7|>.
        User-facing names (lipakshi, vardan, etc.) are mapped to these IDs.
        """
        print(f"\n🔍 Validating {len(INDIC_SPEAKERS)} speakers for Spark TTS...")

        # Validate speaker mapping
        for speaker_name in INDIC_SPEAKERS:
            if speaker_name not in SPEAKER_MAP:
                print(f"⚠  Warning: Speaker {speaker_name} not in SPEAKER_MAP")
                continue

            speaker_id = SPEAKER_MAP[speaker_name]
            speaker_token = f'<|speaker_{speaker_id}|>'
            token_ids = self.tokenizer.encode(speaker_token, add_special_tokens=False)
            print(f"  {speaker_name} → {speaker_token}: {len(token_ids)} tokens")

        print(f"✅ All {len(INDIC_SPEAKERS)} speakers validated with mapping")

    def get_model_type(self) -> str:
        """Get the model type (always 'indic_speakers' for Spark TTS)."""
        return self.model_type

    def get_tokenizer(self):
        """Get the tokenizer."""
        return self.tokenizer

    def get_engine(self):
        """Get the vLLM engine."""
        return self.engine


sed -n '1,260p' veena3modal/core/constants.py


Output:
Chunk ID: b34af9
Wall time: 0.0512 seconds
Process exited with code 0
Original token count: 1489
Output:
"""
Spark TTS Constants

Token IDs, special tokens, speakers, and emotions for Spark TTS model.
Migrated from Veena3/Orpheus to Spark TTS (BiCodec) architecture.
"""

import os

# Stop tokens for generation
# Spark TTS uses <|im_end|> as stop token
TRAINING_STOP_TOKEN_IDS = ["<|im_end|>"]  # String-based stop token for Spark TTS

# Spark TTS Speaker System - 12 Speakers from Training
# Maps user-facing speaker names to internal speaker IDs (speaker_0 to speaker_11)
# Model: BayAreaBoys/spark_tts_4speaker (HuggingFace) has 12 speaker tokens
SPEAKER_MAP = {
    "lipakshi": 0,    # speaker_0
    "vardan": 1,      # speaker_1
    "reet": 2,        # speaker_2
    "Nandini": 3,     # speaker_3
    "krishna": 4,     # speaker_4
    "anika": 5,       # speaker_5
    "adarsh": 6,      # speaker_6
    "Nilay": 7,       # speaker_7
    "Aarvi": 8,       # speaker_8
    "Asha": 9,        # speaker_9
    "Bittu": 10,      # speaker_10
    "Mira": 11,       # speaker_11
}

# Friendly speaker name mappings (user-facing → internal)
# Users can use these friendly names instead of internal names
FRIENDLY_SPEAKER_MAP = {
    "Mitra": "lipakshi",
    "Aaranya": "reet",
    "Taru": "Nandini",
    "Neer": "Nilay",
    "Dhruva": "vardan",
    "Ira": "anika",
    "Veda": "adarsh",
    "Aria": "krishna",
    # Direct mappings for new speakers
    "Aarvi": "Aarvi",
    "Asha": "Asha",
    "Bittu": "Bittu",
    "Mira": "Mira",
}

# Valid speaker names (for API validation)
# Include both internal names and friendly names
INDIC_SPEAKERS = list(SPEAKER_MAP.keys())
ALL_SPEAKER_NAMES = INDIC_SPEAKERS + list(FRIENDLY_SPEAKER_MAP.keys())

def resolve_speaker_name(name: str) -> str:
    """
    Resolve friendly speaker name to internal name.

    Args:
        name: Speaker name (friendly or internal)

    Returns:
        Internal speaker name

    Examples:
        resolve_speaker_name("Mitra") -> "lipakshi"
        resolve_speaker_name("lipakshi") -> "lipakshi"
    """
    # Check if it's a friendly name
    if name in FRIENDLY_SPEAKER_MAP:
        return FRIENDLY_SPEAKER_MAP[name]

    # Check if it's already an internal name
    if name in INDIC_SPEAKERS:
        return name

    # Invalid speaker name
    raise ValueError(f"Invalid speaker name: {name}. Valid names: {', '.join(ALL_SPEAKER_NAMES)}")

# Spark TTS Emotion Tags (bracket format: [emotion])
# NOTE: API users must now use [emotion] instead of <emotion>
INDIC_EMOTION_TAGS = [
    "[angry]",
    "[curious]",
    "[excited]",
    "[giggle]",
    "[laughs harder]",
    "[laughs]",
    "[screams]",
    "[sighs]",
    "[sings]",
    "[whispers]"
]

# Legacy emotion tags for backward compatibility mapping
# Maps old <emotion> format to new [emotion] format
LEGACY_EMOTION_MAP = {
    "<angry>": "[angry]",
    "<curious>": "[curious]",
    "<excited>": "[excited]",
    "<giggle>": "[giggle]",
    "<laugh_harder>": "[laughs harder]",
    "<laugh>": "[laughs]",
    "<scream>": "[screams]",
    "<sigh>": "[sighs]",
    "<sing>": "[sings]",
    "<whisper>": "[whispers]"
}

# All emotion tags (for validation)
ALL_EMOTION_TAGS = INDIC_EMOTION_TAGS

# Model configuration
DEFAULT_MODEL_PATH = "BayAreaBoys/spark_tts_4speaker"
DEFAULT_MAX_MODEL_LEN = 4096  # Lower than old model (was 8192)

# BiCodec Audio Tokenizer
# Location: resolved from env var MODEL_PATH or default
BICODEC_TOKENIZER_PATH = os.environ.get('MODEL_PATH', os.environ.get('SPARK_TTS_MODEL_PATH', '/models/spark_tts_4speaker'))

# Audio configuration
AUDIO_SAMPLE_RATE = 16000  # Changed from 24kHz to 16kHz for BiCodec
AUDIO_CHANNELS = 1
AUDIO_BITS_PER_SAMPLE = 16

# Generation defaults for Spark TTS
DEFAULT_TEMPERATURE = 0.8  # Higher than old model (was 0.4)
DEFAULT_TOP_K = 50
DEFAULT_TOP_P = 1.0
DEFAULT_MAX_TOKENS = 4096  # Increased to handle complex multilingual generation
DEFAULT_MIN_TOKENS = 28  # Minimum tokens for generation (legacy compatibility)
DEFAULT_REPETITION_PENALTY = 1.0  # Repetition penalty (legacy compatibility)
DEFAULT_SEED = None  # None = random, set integer for reproducibility

# vLLM Configuration optimizations for Spark TTS
# NOTE: gpu_memory_utilization adjusted based on available memory
# Production: 0.85, Limited memory: 0.3-0.5
#
# OPTIMIZATION Dec 2025:
# - enable_chunked_prefill: Prevents long prompts from blocking concurrent streams
# - async_scheduling: DISABLED - caused EngineCore issues in production
# OPTIMIZATION Feb 2026:
# Reduced gpu_memory_utilization from 0.85 to 0.25
# Model is only 0.5B (~1.3GB weights). At 0.85, vLLM pre-allocated 65GB for KV cache
# (enough for 1,399 concurrent seqs) when peak actual need is ~100 concurrent.
# At 0.25: ~18GB KV cache, supports ~380 concurrent seqs (4x peak), frees ~48GB VRAM.
# This enables fitting on L4 (24GB) or even T4 (16GB) GPUs.
VLLM_CONFIG = {
    "dtype": "bfloat16",
    "gpu_memory_utilization": 0.25,  # 0.5B model needs <2GB; 0.25 gives ~380 concurrent seqs
    "max_model_len": 4096,
    "max_num_batched_tokens": 4096,
    "enable_prefix_caching": True,  # Cache common prompts for faster TTFB
    "enable_chunked_prefill": True,  # Chunk long prefills to avoid blocking concurrent streams
    "enforce_eager": False,  # Enable CUDA graphs for low latency
    "disable_log_stats": False,
    "trust_remote_code": True,
    "tensor_parallel_size": 1,
}

# Streaming configuration (may need adjustment for BiCodec)
STREAM_BUFFER_SIZE = 50  # Buffer size for streaming (BiCodec-specific, TBD)
BICODEC_BATCH_SIZE = 64
BICODEC_BATCH_TIMEOUT_MS = 15

# Backward compatibility: Old SNAC constants for files not yet migrated
# These are NOT used by Spark TTS but needed by legacy streaming code
# TODO: Remove once streaming_pipeline.py is fully migrated to BiCodec
CODE_START_TOKEN_ID = 128257  # Legacy SNAC constant
CODE_END_TOKEN_ID = 128258    # Legacy SNAC constant
SNAC_MIN_ID = 128266          # Legacy SNAC constant
SNAC_MAX_ID = 156937          # Legacy SNAC constant


Output: write_stdin failed: Unknown process id 32618


I found a deeper vLLM issue on startup: `llvmlite` in this environment crashes during vLLM V1 spec-decode import. I’ll try forcing the V1 engine off (env fallback path) and then resume benchmarking if the local server boots cleanly.


cd /home/ubuntu/veenaModal && VLLM_USE_V1=0 VEENA3_TIMING_LOGS=true python3 -m veena3modal.local_server --skip-download --model-path /home/ubuntu/veenaModal/models/spark_tts_4speaker --port 8000 --gpu-memory 0.15 --num-engines 1 --log-level INFO


Output:
Chunk ID: 83a811
Wall time: 1.0034 seconds
Process running with session ID 40326
Original token count: 454
Output:
2026-02-17 00:23:19 | INFO     | veena3.local | Auth bypass enabled (local mode). Use --auth to require API keys.
2026-02-17 00:23:19 | INFO     | veena3.local | Resolved model paths:
2026-02-17 00:23:19 | INFO     | veena3.local |   LLM (vLLM):    /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
2026-02-17 00:23:19 | INFO     | veena3.local |   BiCodec:       /home/ubuntu/veenaModal/models/spark_tts_4speaker
2026-02-17 00:23:19 | INFO     | veena3.local | ============================================================
2026-02-17 00:23:19 | INFO     | veena3.local | Initializing Veena3 TTS Runtime (local)
2026-02-17 00:23:19 | INFO     | veena3.local |   Model:       /home/ubuntu/veenaModal/models/spark_tts_4speaker
2026-02-17 00:23:19 | INFO     | veena3.local |   LLM path:    /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
2026-02-17 00:23:19 | INFO     | veena3.local |   BiCodec:     /home/ubuntu/veenaModal/models/spark_tts_4speaker
2026-02-17 00:23:19 | INFO     | veena3.local |   Device:      cuda
2026-02-17 00:23:19 | INFO     | veena3.local |   GPU Memory:  15% (total)
2026-02-17 00:23:19 | INFO     | veena3.local |   Engines:     1
2026-02-17 00:23:19 | INFO     | veena3.local |   SR:          disabled
2026-02-17 00:23:19 | INFO     | veena3.local | ============================================================
2026-02-17 00:23:19 | INFO     | veena3modal.services.tts_runtime | Initializing TTS runtime...
2026-02-17 00:23:19 | INFO     | veena3modal.services.tts_runtime |   Model path: /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
2026-02-17 00:23:19 | INFO     | veena3modal.services.tts_runtime |   BiCodec path: /home/ubuntu/veenaModal/models/spark_tts_4speaker
2026-02-17 00:23:19 | INFO     | veena3modal.services.tts_runtime |   SR enabled: False, path: None


Output:
Chunk ID: 9567ec
Wall time: 5.0011 seconds
Process running with session ID 40326
Original token count: 509
Output:
2026-02-17 00:23:25 | INFO     | veena3modal.services.tts_runtime | Loading SparkTTS model with vLLM...
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
🚀 Initializing Spark TTS Model
📁 Model path: /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
🗣  Speakers: 12 predefined (lipakshi, vardan, reet, Nandini, krishna, anika, adarsh, Nilay, Aarvi, Asha, Bittu, Mira)
🎭 Emotions: 10 tags in [bracket] format

📝 Loading tokenizer from model...
✅ Tokenizer loaded: 165168 tokens

🔍 Validating 12 speakers for Spark TTS...
  lipakshi → <|speaker_0|>: 7 tokens
  vardan → <|speaker_1|>: 7 tokens
  reet → <|speaker_2|>: 7 tokens
  Nandini → <|speaker_3|>: 7 tokens
  krishna → <|speaker_4|>: 7 tokens
  anika → <|speaker_5|>: 7 tokens
  adarsh → <|speaker_6|>: 7 tokens
  Nilay → <|speaker_7|>: 7 tokens
  Aarvi → <|speaker_8|>: 7 tokens
  Asha → <|speaker_9|>: 7 tokens
  Bittu → <|speaker_10|>: 8 tokens
  Mira → <|speaker_11|>: 8 tokens
✅ All 12 speakers validated with mapping

🔍 Validating 10 emotion tags for Spark TTS...
✅ All 10 emotion tags validated

📋 Sample emotion tags:
  [angry]: 1 tokens
  [curious]: 1 tokens
  [excited]: 1 tokens
  [giggle]: 1 tokens
  [laughs harder]: 1 tokens
  ...

🔧 Initializing vLLM engine with optimizations...
  - dtype: bfloat16
  - max_model_len: 4096
  - gpu_memory_utilization: 0.15
  - enable_prefix_caching: True
  - enforce_eager (disable CUDA graphs): False
INFO 02-17 00:23:26 [model.py:541] Resolved architecture: Qwen2ForCausalLM
INFO 02-17 00:23:26 [model.py:1561] Using max model len 4096
INFO 02-17 00:23:26 [scheduler.py:226] Chunked prefill is enabled with max_num_batched_tokens=4096.
INFO 02-17 00:23:26 [vllm.py:624] Asynchronous scheduling is enabled.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.


Output:
Chunk ID: 521be1
Wall time: 4.1069 seconds
Process exited with code 1
Original token count: 4955
Output:
(EngineCore_DP0 pid=3705469) INFO 02-17 00:23:27 [core.py:96] Initializing a V1 LLM engine (v0.15.1) with config: model='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', speculative_config=None, tokenizer='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [4096], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 256, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': True}, 'local_cache_dir': None, 'static_all_moe_layers': []}
(EngineCore_DP0 pid=3705469) INFO 02-17 00:23:29 [parallel_state.py:1212] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://216.81.248.184:46985 backend=nccl
(EngineCore_DP0 pid=3705469) INFO 02-17 00:23:29 [parallel_state.py:1423] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
(EngineCore_DP0 pid=3705469) ERROR 02-17 00:23:30 [core.py:946] EngineCore failed to start.
(EngineCore_DP0 pid=3705469) ERROR 02-17 00:23:30 [core.py:946] Traceback (most recent call last):
(EngineCore_DP0 pid=3705469) ERROR 02-17 00:23:30 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 937, in run_engine_core
(EngineCore_DP0 pid=3705469) ERROR 02-17 00:23:30 [core.py:946]     engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore_DP0 pid=3705469) ERROR 02-17 00:23:30 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 691, in __init__
(EngineCore_DP0 pid=3705469) ERROR 02-17 00:23:30 [core.py:946]     super().__init__(
(EngineCore_DP0 pid=3705469) ERROR 02-17 00:23:30 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 105, in __init__
(EngineCore_DP0 pid=3705469) ERROR 02-17 00:23:30 [core.py:946]     self.model_executor = executor_class(vllm_config)
(EngineCore_DP0 pid=3705469) ERROR 02-17 00:23:30 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/executor/abstract.py", line 101, in __init__
(EngineCore_DP0 pid=3705469) ERROR 02-17 00:23:30 [core.py:946]     self._init_executor()
(EngineCore_DP0 pid=3705469) ERROR 02-17 00:23:30 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/executor/uniproc_executor.py", line 47, in _init_executor
(EngineCore_DP0 pid=3705469) ERROR 02-17 00:23:30 [core.py:946]     self.driver_worker.init_device()
(EngineCore_DP0 pid=3705469) ERROR 02-17 00:23:30 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/worker/worker_base.py", line 326, in init_device
(EngineCore_DP0 pid=3705469) ERROR 02-17 00:23:30 [core.py:946]     self.worker.init_device()  # type: ignore
(EngineCore_DP0 pid=3705469) ERROR 02-17 00:23:30 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/worker/gpu_worker.py", line 258, in init_device
(EngineCore_DP0 pid=3705469) ERROR 02-17 00:23:30 [core.py:946]     from vllm.v1.worker.gpu_model_runner import (
(EngineCore_DP0 pid=3705469) ERROR 02-17 00:23:30 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/worker/gpu_model_runner.py", line 153, in <module>
(EngineCore_DP0 pid=3705469) ERROR 02-17 00:23:30 [core.py:946]     from vllm.v1.spec_decode.ngram_proposer import NgramProposer
(EngineCore_DP0 pid=3705469) ERROR 02-17 00:23:30 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/spec_decode/ngram_proposer.py", line 170, in <module>
(EngineCore_DP0 pid=3705469) ERROR 02-17 00:23:30 [core.py:946]     def batch_propose_numba(
(EngineCore_DP0 pid=3705469) ERROR 02-17 00:23:30 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/decorators.py", line 221, in wrapper
(EngineCore_DP0 pid=3705469) ERROR 02-17 00:23:30 [core.py:946]     disp = dispatcher(py_func=func, locals=locals,
(EngineCore_DP0 pid=3705469) ERROR 02-17 00:23:30 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/dispatcher.py", line 772, in __init__
(EngineCore_DP0 pid=3705469) ERROR 02-17 00:23:30 [core.py:946]     self.targetctx = self.targetdescr.target_context
(EngineCore_DP0 pid=3705469) ERROR 02-17 00:23:30 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/registry.py", line 30, in target_context
(EngineCore_DP0 pid=3705469) ERROR 02-17 00:23:30 [core.py:946]     return self._toplevel_target_context
(EngineCore_DP0 pid=3705469) ERROR 02-17 00:23:30 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/utils.py", line 741, in __get__
(EngineCore_DP0 pid=3705469) ERROR 02-17 00:23:30 [core.py:946]     return super().__get__(*args, **kwargs)
(EngineCore_DP0 pid=3705469) ERROR 02-17 00:23:30 [core.py:946]   File "/usr/lib/python3.10/functools.py", line 981, in __get__
(EngineCore_DP0 pid=3705469) ERROR 02-17 00:23:30 [core.py:946]     val = self.func(instance)
(EngineCore_DP0 pid=3705469) ERROR 02-17 00:23:30 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/registry.py", line 18, in _toplevel_target_context
(EngineCore_DP0 pid=3705469) ERROR 02-17 00:23:30 [core.py:946]     return cpu.CPUContext(self.typing_context, self._target_name)
(EngineCore_DP0 pid=3705469) ERROR 02-17 00:23:30 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/cpu.py", line 42, in __init__
(EngineCore_DP0 pid=3705469) ERROR 02-17 00:23:30 [core.py:946]     super().__init__(typingctx, target)
(EngineCore_DP0 pid=3705469) ERROR 02-17 00:23:30 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/base.py", line 254, in __init__
(EngineCore_DP0 pid=3705469) ERROR 02-17 00:23:30 [core.py:946]     self.init()
(EngineCore_DP0 pid=3705469) ERROR 02-17 00:23:30 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/compiler_lock.py", line 35, in _acquire_compile_lock
(EngineCore_DP0 pid=3705469) ERROR 02-17 00:23:30 [core.py:946]     return func(*args, **kwargs)
(EngineCore_DP0 pid=3705469) ERROR 02-17 00:23:30 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/cpu.py", line 51, in init
(EngineCore_DP0 pid=3705469) ERROR 02-17 00:23:30 [core.py:946]     self._internal_codegen = codegen.JITCPUCodegen("numba.exec")
(EngineCore_DP0 pid=3705469) ERROR 02-17 00:23:30 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/codegen.py", line 1172, in __init__
(EngineCore_DP0 pid=3705469) ERROR 02-17 00:23:30 [core.py:946]     initialize_llvm()
(EngineCore_DP0 pid=3705469) ERROR 02-17 00:23:30 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/codegen.py", line 1434, in initialize_llvm
(EngineCore_DP0 pid=3705469) ERROR 02-17 00:23:30 [core.py:946]     ll.initialize()
(EngineCore_DP0 pid=3705469) ERROR 02-17 00:23:30 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/llvmlite/binding/initfini.py", line 17, in initialize
(EngineCore_DP0 pid=3705469) ERROR 02-17 00:23:30 [core.py:946]     raise RuntimeError(
(EngineCore_DP0 pid=3705469) ERROR 02-17 00:23:30 [core.py:946] RuntimeError: llvmlite.binding.initialize() is deprecated and will be removed. LLVM initialization is now handled automatically. Please remove calls to this function from your code and check for other behavioral changes that may have occurred due to LLVM updates.
(EngineCore_DP0 pid=3705469) Process EngineCore_DP0:
(EngineCore_DP0 pid=3705469) Traceback (most recent call last):
(EngineCore_DP0 pid=3705469)   File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore_DP0 pid=3705469)     self.run()
(EngineCore_DP0 pid=3705469)   File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
(EngineCore_DP0 pid=3705469)     self._target(*self._args, **self._kwargs)
(EngineCore_DP0 pid=3705469)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 950, in run_engine_core
(EngineCore_DP0 pid=3705469)     raise e
(EngineCore_DP0 pid=3705469)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 937, in run_engine_core
(EngineCore_DP0 pid=3705469)     engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore_DP0 pid=3705469)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 691, in __init__
(EngineCore_DP0 pid=3705469)     super().__init__(
(EngineCore_DP0 pid=3705469)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 105, in __init__
(EngineCore_DP0 pid=3705469)     self.model_executor = executor_class(vllm_config)
(EngineCore_DP0 pid=3705469)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/executor/abstract.py", line 101, in __init__
(EngineCore_DP0 pid=3705469)     self._init_executor()
(EngineCore_DP0 pid=3705469)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/executor/uniproc_executor.py", line 47, in _init_executor
(EngineCore_DP0 pid=3705469)     self.driver_worker.init_device()
(EngineCore_DP0 pid=3705469)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/worker/worker_base.py", line 326, in init_device
(EngineCore_DP0 pid=3705469)     self.worker.init_device()  # type: ignore
(EngineCore_DP0 pid=3705469)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/worker/gpu_worker.py", line 258, in init_device
(EngineCore_DP0 pid=3705469)     from vllm.v1.worker.gpu_model_runner import (
(EngineCore_DP0 pid=3705469)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/worker/gpu_model_runner.py", line 153, in <module>
(EngineCore_DP0 pid=3705469)     from vllm.v1.spec_decode.ngram_proposer import NgramProposer
(EngineCore_DP0 pid=3705469)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/spec_decode/ngram_proposer.py", line 170, in <module>
(EngineCore_DP0 pid=3705469)     def batch_propose_numba(
(EngineCore_DP0 pid=3705469)   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/decorators.py", line 221, in wrapper
(EngineCore_DP0 pid=3705469)     disp = dispatcher(py_func=func, locals=locals,
(EngineCore_DP0 pid=3705469)   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/dispatcher.py", line 772, in __init__
(EngineCore_DP0 pid=3705469)     self.targetctx = self.targetdescr.target_context
(EngineCore_DP0 pid=3705469)   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/registry.py", line 30, in target_context
(EngineCore_DP0 pid=3705469)     return self._toplevel_target_context
(EngineCore_DP0 pid=3705469)   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/utils.py", line 741, in __get__
(EngineCore_DP0 pid=3705469)     return super().__get__(*args, **kwargs)
(EngineCore_DP0 pid=3705469)   File "/usr/lib/python3.10/functools.py", line 981, in __get__
(EngineCore_DP0 pid=3705469)     val = self.func(instance)
(EngineCore_DP0 pid=3705469)   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/registry.py", line 18, in _toplevel_target_context
(EngineCore_DP0 pid=3705469)     return cpu.CPUContext(self.typing_context, self._target_name)
(EngineCore_DP0 pid=3705469)   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/cpu.py", line 42, in __init__
(EngineCore_DP0 pid=3705469)     super().__init__(typingctx, target)
(EngineCore_DP0 pid=3705469)   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/base.py", line 254, in __init__
(EngineCore_DP0 pid=3705469)     self.init()
(EngineCore_DP0 pid=3705469)   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/compiler_lock.py", line 35, in _acquire_compile_lock
(EngineCore_DP0 pid=3705469)     return func(*args, **kwargs)
(EngineCore_DP0 pid=3705469)   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/cpu.py", line 51, in init
(EngineCore_DP0 pid=3705469)     self._internal_codegen = codegen.JITCPUCodegen("numba.exec")
(EngineCore_DP0 pid=3705469)   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/codegen.py", line 1172, in __init__
(EngineCore_DP0 pid=3705469)     initialize_llvm()
(EngineCore_DP0 pid=3705469)   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/codegen.py", line 1434, in initialize_llvm
(EngineCore_DP0 pid=3705469)     ll.initialize()
(EngineCore_DP0 pid=3705469)   File "/home/ubuntu/.local/lib/python3.10/site-packages/llvmlite/binding/initfini.py", line 17, in initialize
(EngineCore_DP0 pid=3705469)     raise RuntimeError(
(EngineCore_DP0 pid=3705469) RuntimeError: llvmlite.binding.initialize() is deprecated and will be removed. LLVM initialization is now handled automatically. Please remove calls to this function from your code and check for other behavioral changes that may have occurred due to LLVM updates.
2026-02-17 00:23:30 | ERROR    | veena3modal.services.tts_runtime | ❌ Failed to initialize TTS runtime: Engine core initialization failed. See root cause above. Failed core proc(s): {}
Traceback (most recent call last):
  File "/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py", line 167, in initialize_runtime
    model = SparkTTSModel(
  File "/home/ubuntu/veenaModal/veena3modal/core/model_loader.py", line 128, in __init__
    self.engine = AsyncLLMEngine.from_engine_args(engine_args)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/async_llm.py", line 257, in from_engine_args
    return cls(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/async_llm.py", line 155, in __init__
    self.engine_core = EngineCoreClient.make_async_mp_client(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 122, in make_async_mp_client
    return AsyncMPClient(*client_args)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 819, in __init__
    super().__init__(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 479, in __init__
    with launch_core_engines(vllm_config, executor_class, log_stats) as (
  File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
    next(self.gen)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/utils.py", line 933, in launch_core_engines
    wait_for_engine_startup(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/utils.py", line 992, in wait_for_engine_startup
    raise RuntimeError(
RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
Traceback (most recent call last):
  File "/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py", line 167, in initialize_runtime
    model = SparkTTSModel(
  File "/home/ubuntu/veenaModal/veena3modal/core/model_loader.py", line 128, in __init__
    self.engine = AsyncLLMEngine.from_engine_args(engine_args)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/async_llm.py", line 257, in from_engine_args
    return cls(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/async_llm.py", line 155, in __init__
    self.engine_core = EngineCoreClient.make_async_mp_client(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 122, in make_async_mp_client
    return AsyncMPClient(*client_args)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 819, in __init__
    super().__init__(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 479, in __init__
    with launch_core_engines(vllm_config, executor_class, log_stats) as (
  File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
    next(self.gen)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/utils.py", line 933, in launch_core_engines
    wait_for_engine_startup(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/utils.py", line 992, in wait_for_engine_startup
    raise RuntimeError(
RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/ubuntu/veenaModal/veena3modal/local_server.py", line 465, in <module>
    main()
  File "/home/ubuntu/veenaModal/veena3modal/local_server.py", line 426, in main
    initialize_local_runtime(
  File "/home/ubuntu/veenaModal/veena3modal/local_server.py", line 224, in initialize_local_runtime
    runtime = initialize_runtime(
  File "/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py", line 272, in initialize_runtime
    raise RuntimeError(f"TTS runtime initialization failed: {e}") from e
RuntimeError: TTS runtime initialization failed: Engine core initialization failed. See root cause above. Failed core proc(s): {}


python3 - <<'PY'
from vllm import AsyncEngineArgs
import inspect
sig=inspect.signature(AsyncEngineArgs.__init__)
print(sig)
PY


Output:
Chunk ID: 954661
Wall time: 5.1711 seconds
Process exited with code 0
Original token count: 1871
Output:
(self, model: str = 'Qwen/Qwen3-0.6B', enable_return_routed_experts: bool = False, model_weights: str = '', served_model_name: str | list[str] | None = None, tokenizer: str | None = None, hf_config_path: str | None = None, runner: Literal['auto', 'generate', 'pooling', 'draft'] = 'auto', convert: Literal['auto', 'none', 'embed', 'classify', 'reward', 'mm_encoder_only'] = 'auto', skip_tokenizer_init: bool = False, enable_prompt_embeds: bool = False, tokenizer_mode: Union[Literal['auto', 'hf', 'slow', 'mistral', 'deepseek_v32'], str] = 'auto', trust_remote_code: bool = False, allowed_local_media_path: str = '', allowed_media_domains: list[str] | None = None, download_dir: str | None = None, safetensors_load_strategy: str = 'lazy', load_format: Union[str, Any] = 'auto', config_format: str = 'auto', dtype: Literal['auto', 'half', 'float16', 'bfloat16', 'float', 'float32'] = 'auto', kv_cache_dtype: Literal['auto', 'bfloat16', 'fp8', 'fp8_e4m3', 'fp8_e5m2', 'fp8_inc', 'fp8_ds_mla'] = 'auto', seed: int = 0, max_model_len: int | None = None, cudagraph_capture_sizes: list[int] | None = None, max_cudagraph_capture_size: int | None = None, distributed_executor_backend: Union[str, Literal['ray', 'mp', 'uni', 'external_launcher'], type[Any], NoneType] = None, pipeline_parallel_size: int = 1, master_addr: str = '127.0.0.1', master_port: int = 29501, nnodes: int = 1, node_rank: int = 0, tensor_parallel_size: int = 1, prefill_context_parallel_size: int = 1, decode_context_parallel_size: int = 1, dcp_kv_cache_interleave_size: int = 1, cp_kv_cache_interleave_size: int = 1, data_parallel_size: int = 1, data_parallel_rank: int | None = None, data_parallel_start_rank: int | None = None, data_parallel_size_local: int | None = None, data_parallel_address: str | None = None, data_parallel_rpc_port: int | None = None, data_parallel_hybrid_lb: bool = False, data_parallel_external_lb: bool = False, data_parallel_backend: str = 'mp', enable_expert_parallel: bool = False, all2all_backend: str = 'allgather_reducescatter', enable_dbo: bool = False, ubatch_size: int = 0, dbo_decode_token_threshold: int = 32, dbo_prefill_token_threshold: int = 512, disable_nccl_for_dp_synchronization: bool | None = None, eplb_config: vllm.config.parallel.EPLBConfig = <factory>, enable_eplb: bool = False, expert_placement_strategy: Literal['linear', 'round_robin'] = 'linear', _api_process_count: int = 1, _api_process_rank: int = 0, max_parallel_loading_workers: int | None = None, block_size: Optional[Literal[1, 8, 16, 32, 64, 128, 256]] = None, enable_prefix_caching: bool | None = None, prefix_caching_hash_algo: Literal['sha256', 'sha256_cbor', 'xxhash', 'xxhash_cbor'] = 'sha256', disable_sliding_window: bool = False, disable_cascade_attn: bool = False, swap_space: float = 4, cpu_offload_gb: float = 0, gpu_memory_utilization: float = 0.9, kv_cache_memory_bytes: int | None = None, max_num_batched_tokens: int | None = None, max_num_partial_prefills: int = 1, max_long_partial_prefills: int = 1, long_prefill_token_threshold: int = 0, max_num_seqs: int | None = None, max_logprobs: int = 20, logprobs_mode: Literal['raw_logits', 'raw_logprobs', 'processed_logits', 'processed_logprobs'] = 'raw_logprobs', disable_log_stats: bool = False, aggregate_engine_logging: bool = False, revision: str | None = None, code_revision: str | None = None, hf_token: bool | str | None = None, hf_overrides: dict[str, typing.Any] | collections.abc.Callable[[typing.Any], typing.Any] = <factory>, tokenizer_revision: str | None = None, quantization: Optional[Any] = None, allow_deprecated_quantization: bool = False, enforce_eager: bool = False, disable_custom_all_reduce: bool = False, limit_mm_per_prompt: dict[str, int | dict[str, int]] = <factory>, enable_mm_embeds: bool = False, interleave_mm_strings: bool = False, media_io_kwargs: dict[str, dict[str, typing.Any]] = <factory>, mm_processor_kwargs: dict[str, typing.Any] | None = None, mm_processor_cache_gb: float = 4, mm_processor_cache_type: Optional[Literal['shm', 'lru']] = 'lru', mm_shm_cache_max_object_size_mb: int = 128, mm_encoder_only: bool = False, mm_encoder_tp_mode: Literal['weights', 'data'] = 'weights', mm_encoder_attn_backend: vllm.v1.attention.backends.registry.AttentionBackendEnum | str | None = None, io_processor_plugin: str | None = None, skip_mm_profiling: bool = False, video_pruning_rate: float = None, enable_lora: bool = False, max_loras: int = 1, max_lora_rank: int = 16, default_mm_loras: dict[str, str] | None = None, fully_sharded_loras: bool = False, max_cpu_loras: int | None = None, lora_dtype: str | torch.dtype | None = 'auto', enable_tower_connector_lora: bool = False, ray_workers_use_nsight: bool = False, num_gpu_blocks_override: int | None = None, model_loader_extra_config: dict = <factory>, ignore_patterns: str | list[str] = <factory>, enable_chunked_prefill: bool | None = None, disable_chunked_mm_input: bool = False, disable_hybrid_kv_cache_manager: bool | None = None, structured_outputs_config: vllm.config.structured_outputs.StructuredOutputsConfig = <factory>, reasoning_parser: str = '', reasoning_parser_plugin: str | None = None, logits_processor_pattern: str | None = None, speculative_config: dict[str, typing.Any] | None = None, show_hidden_metrics_for_version: str | None = None, otlp_traces_endpoint: str | None = None, collect_detailed_traces: list[typing.Literal['model', 'worker', 'all']] | None = None, kv_cache_metrics: bool = False, kv_cache_metrics_sample: float = 0.01, cudagraph_metrics: bool = False, enable_layerwise_nvtx_tracing: bool = False, enable_mfu_metrics: bool = False, enable_logging_iteration_details: bool = False, enable_mm_processor_stats: bool = False, scheduling_policy: Literal['fcfs', 'priority'] = 'fcfs', scheduler_cls: str | type[object] | None = None, pooler_config: vllm.config.pooler.PoolerConfig | None = None, compilation_config: vllm.config.compilation.CompilationConfig = <factory>, attention_config: vllm.config.attention.AttentionConfig = <factory>, worker_cls: str = 'auto', worker_extension_cls: str = '', profiler_config: vllm.config.profiler.ProfilerConfig = <factory>, kv_transfer_config: vllm.config.kv_transfer.KVTransferConfig | None = None, kv_events_config: vllm.config.kv_events.KVEventsConfig | None = None, ec_transfer_config: vllm.config.ec_transfer.ECTransferConfig | None = None, generation_config: str = 'auto', enable_sleep_mode: bool = False, override_generation_config: dict[str, typing.Any] = <factory>, model_impl: str = 'auto', override_attention_dtype: str = None, attention_backend: vllm.v1.attention.backends.registry.AttentionBackendEnum | None = None, calculate_kv_scales: bool = False, mamba_cache_dtype: Literal['auto', 'float32', 'float16'] = 'auto', mamba_ssm_cache_dtype: Literal['auto', 'float32', 'float16'] = 'auto', mamba_block_size: int | None = None, mamba_cache_mode: Literal['all', 'align', 'none'] = 'none', additional_config: dict[str, typing.Any] = <factory>, use_tqdm_on_load: bool = True, pt_load_map_location: str = 'cpu', logits_processors: list[str | type[vllm.v1.sample.logits_processor.interface.LogitsProcessor]] | None = None, async_scheduling: bool | None = None, stream_interval: int = 1, kv_sharing_fast_prefill: bool = False, optimization_level: vllm.config.vllm.OptimizationLevel = <OptimizationLevel.O2: 2>, kv_offloading_size: float | None = None, kv_offloading_backend: Literal['native', 'lmcache'] = 'native', tokens_only: bool = False, enable_log_requests: bool = False) -> None


python3 - <<'PY'
import numba, llvmlite
print('numba', numba.__version__)
print('llvmlite', llvmlite.__version__)
PY


Output:
Chunk ID: 4a547e
Wall time: 0.3082 seconds
Process exited with code 0
Original token count: 8
Output:
numba 0.61.2
llvmlite 0.46.0


rg -n "llvmlite|ngram_proposer|spec_decode|disable_spec" /home/ubuntu/.local/lib/python3.10/site-packages/vllm -g '*.py' | head -n 80


Output:
Chunk ID: 05c894
Wall time: 0.0511 seconds
Process exited with code 0
Original token count: 2952
Output:
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/config/compilation.py:1093:    def adjust_cudagraph_sizes_for_spec_decode(
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/qwen3_next.py:560:                    : attn_metadata.num_spec_decodes
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/qwen3_next.py:635:                cu_seqlens=spec_query_start_loc[: attn_metadata.num_spec_decodes + 1],
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py:146:        self.use_spec_decode = vllm_config.speculative_config is not None
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py:404:        if not self.async_scheduling and self.use_spec_decode and model_executed:
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py:507:            if self.use_spec_decode:
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/spec_decode/draft_model.py:17:from vllm.v1.spec_decode.eagle import PADDING_SLOT_ID, SpecDecodeBaseProposer
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/structured_output/utils.py:73:    spec_tokens = scheduler_output.scheduled_spec_decode_tokens
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/benchmarks/serve.py:99:async def fetch_spec_decode_metrics(
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/benchmarks/serve.py:117:            found_spec_decode = False
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/benchmarks/serve.py:124:                if line.startswith("vllm:spec_decode"):
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/benchmarks/serve.py:125:                    found_spec_decode = True
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/benchmarks/serve.py:146:            if not found_spec_decode:
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/benchmarks/serve.py:758:    spec_decode_metrics_before = await fetch_spec_decode_metrics(base_url, session)
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/benchmarks/serve.py:843:    spec_decode_metrics_after = await fetch_spec_decode_metrics(base_url, session)
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/benchmarks/serve.py:844:    spec_decode_stats: dict[str, Any] | None = None
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/benchmarks/serve.py:845:    if spec_decode_metrics_before is not None and spec_decode_metrics_after is not None:
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/benchmarks/serve.py:847:            spec_decode_metrics_after.num_drafts - spec_decode_metrics_before.num_drafts
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/benchmarks/serve.py:850:            spec_decode_metrics_after.num_draft_tokens
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/benchmarks/serve.py:851:            - spec_decode_metrics_before.num_draft_tokens
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/benchmarks/serve.py:854:            spec_decode_metrics_after.num_accepted_tokens
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/benchmarks/serve.py:855:            - spec_decode_metrics_before.num_accepted_tokens
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/benchmarks/serve.py:860:                set(spec_decode_metrics_before.accepted_per_pos.keys())
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/benchmarks/serve.py:861:                | set(spec_decode_metrics_after.accepted_per_pos.keys())
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/benchmarks/serve.py:864:                before_val = spec_decode_metrics_before.accepted_per_pos.get(pos, 0)
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/benchmarks/serve.py:865:                after_val = spec_decode_metrics_after.accepted_per_pos.get(
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/benchmarks/serve.py:876:            spec_decode_stats = {
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/benchmarks/serve.py:981:    if spec_decode_stats is not None:
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/benchmarks/serve.py:982:        result["spec_decode_acceptance_rate"] = spec_decode_stats["acceptance_rate"]
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/benchmarks/serve.py:983:        result["spec_decode_acceptance_length"] = spec_decode_stats["acceptance_length"]
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/benchmarks/serve.py:984:        result["spec_decode_num_drafts"] = int(spec_decode_stats["num_drafts"])
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/benchmarks/serve.py:985:        result["spec_decode_draft_tokens"] = int(spec_decode_stats["draft_tokens"])
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/benchmarks/serve.py:986:        result["spec_decode_accepted_tokens"] = int(
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/benchmarks/serve.py:987:            spec_decode_stats["accepted_tokens"]
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/benchmarks/serve.py:989:        result["spec_decode_per_position_acceptance_rates"] = spec_decode_stats.get(
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/benchmarks/serve.py:1038:    if spec_decode_stats is not None:
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/benchmarks/serve.py:1042:                "Acceptance rate (%):", spec_decode_stats["acceptance_rate"]
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/benchmarks/serve.py:1047:                "Acceptance length:", spec_decode_stats["acceptance_length"]
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/benchmarks/serve.py:1050:        print("{:<40} {:<10}".format("Drafts:", int(spec_decode_stats["num_drafts"])))
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/benchmarks/serve.py:1053:                "Draft tokens:", int(spec_decode_stats["draft_tokens"])
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/benchmarks/serve.py:1058:                "Accepted tokens:", int(spec_decode_stats["accepted_tokens"])
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/benchmarks/serve.py:1061:        per_pos = spec_decode_stats.get("per_position_acceptance_rates", [])
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/layers/attention/mla_attention.py:713:        supports_spec_decode = self.query_len_support != QueryLenSupport.SINGLE_ONLY
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/layers/attention/mla_attention.py:715:            self.reorder_batch_threshold, supports_spec_decode, supports_dcp_with_varlen
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/sample/rejection_sampler.py:18:from vllm.v1.spec_decode.metadata import SpecDecodeMetadata
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/metrics/stats.py:12:from vllm.v1.spec_decode.metrics import SpecDecodingStats
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/core/sched/async_scheduler.py:17:        spec_decode_tokens = scheduler_output.scheduled_spec_decode_tokens
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/core/sched/async_scheduler.py:24:            cur_num_spec_tokens = len(spec_decode_tokens.get(req_id, ()))
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/metrics/loggers.py:30:from vllm.v1.spec_decode.metrics import SpecDecodingLogging, SpecDecodingProm
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/metrics/ray_wrappers.py:7:from vllm.v1.spec_decode.metrics import SpecDecodingProm
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/metrics/reader.py:36:    specific metric, vllm:spec_decode_num_accepted_tokens_per_pos.
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/metrics/reader.py:98:            if metric.name == "vllm:spec_decode_num_accepted_tokens_per_pos":
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/core/sched/scheduler.py:56:from vllm.v1.spec_decode.metrics import SpecDecodingStats
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/core/sched/scheduler.py:337:        scheduled_spec_decode_tokens: dict[str, list[int]] = {}
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/core/sched/scheduler.py:454:                            scheduled_spec_decode_tokens.pop(
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/core/sched/scheduler.py:500:                    scheduled_spec_decode_tokens[request.request_id] = (
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/core/sched/scheduler.py:846:                scheduled_spec_decode_tokens,
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/core/sched/scheduler.py:859:            scheduled_spec_decode_tokens=scheduled_spec_decode_tokens,
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/core/sched/scheduler.py:991:        spec_decode_tokens: dict[str, list[int]],
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/core/sched/scheduler.py:1013:                    spec_decode_tokens.get(req_id, ())
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/core/sched/scheduler.py:1221:            scheduler_output.scheduled_spec_decode_tokens,
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/core/sched/scheduler.py:1285:                scheduler_output.scheduled_spec_decode_tokens.get(req_id)
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/core/sched/scheduler.py:1580:        sched_spec_tokens = scheduler_output.scheduled_spec_decode_tokens
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/spec_decode/metrics.py:125:      rate(vllm:spec_decode_num_accepted_tokens_total[$interval]) /
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/spec_decode/metrics.py:126:      rate(vllm:spec_decode_num_draft_tokens_total[$interval])
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/spec_decode/metrics.py:132:      rate(vllm:spec_decode_num_accepted_tokens_total[$interval]) /
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/spec_decode/metrics.py:133:      rate(vllm:spec_decode_num_drafts[$interval]))
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/spec_decode/metrics.py:137:      vllm:spec_decode_num_accepted_tokens_per_pos[$interval] /
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/spec_decode/metrics.py:138:      vllm:spec_decode_num_drafts[$interval]
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/spec_decode/metrics.py:154:            name="vllm:spec_decode_num_drafts",
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/spec_decode/metrics.py:158:        self.counter_spec_decode_num_drafts = make_per_engine(
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/spec_decode/metrics.py:163:            name="vllm:spec_decode_num_draft_tokens",
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/spec_decode/metrics.py:167:        self.counter_spec_decode_num_draft_tokens = make_per_engine(
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/spec_decode/metrics.py:172:            name="vllm:spec_decode_num_accepted_tokens",
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/spec_decode/metrics.py:176:        self.counter_spec_decode_num_accepted_tokens = make_per_engine(
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/spec_decode/metrics.py:188:            name="vllm:spec_decode_num_accepted_tokens_per_pos",
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/spec_decode/metrics.py:192:        self.counter_spec_decode_num_accepted_tokens_per_pos: dict[
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/spec_decode/metrics.py:202:        self.counter_spec_decode_num_drafts[engine_idx].inc(
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/spec_decode/metrics.py:205:        self.counter_spec_decode_num_draft_tokens[engine_idx].inc(
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/spec_decode/metrics.py:208:        self.counter_spec_decode_num_accepted_tokens[engine_idx].inc(


NUMBA_DISABLE_JIT=1 python3 - <<'PY'
from vllm.v1.spec_decode.ngram_proposer import NgramProposer
print('import ok')
PY


Output:
Chunk ID: 71478d
Wall time: 5.1898 seconds
Process exited with code 0
Original token count: 3
Output:
import ok


I found a workable startup workaround: setting `NUMBA_DISABLE_JIT=1` avoids the `llvmlite` crash when `vLLM` imports N-gram proposer code. I’ll launch the benchmark server with that flag and see if we can complete generation timing captures.


pkill -f 'python3 -m veena3modal.local_server' || true && cd /home/ubuntu/veenaModal && VLLM_USE_V1=0 NUMBA_DISABLE_JIT=1 VEENA3_TIMING_LOGS=true python3 -m veena3modal.local_server --skip-download --model-path /home/ubuntu/veenaModal/models/spark_tts_4speaker --port 8000 --gpu-memory 0.15 --num-engines 1 --log-level INFO