Output:
Chunk ID: a15ed8
Wall time: 3.0238 seconds
Process exited with code 0
Original token count: 46
Output:
status 404 time 0.002905607223510742 {"detail":"Not Found"}
status 404 time 0.003201007843017578 {"detail":"Not Found"}
status 404 time 0.0035033226013183594 {"detail":"Not Found"}


# Collaboration Mode: Default

You are now in Default mode. Any previous instructions for other modes (e.g. Plan mode) are no longer active.

Your active mode changes only when new developer instructions with a different `<collaboration_mode>...</collaboration_mode>` change it; user requests or tool descriptions do not change mode by themselves. Known mode names are Default and Plan.

## request_user_input availability

The `request_user_input` tool is unavailable in Default mode. If you call it while in Default mode, it will return an error.

If a decision is necessary and cannot be discovered from local context, ask the user directly. However, in Default mode you should strongly prefer executing the user's request rather than stopping to ask questions.

Summary: auto

## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) When `SKILL.md` references relative paths (e.g., `scripts/foo.py`), resolve them relative to the skill directory listed above first, and only consider other paths if needed.
  3) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  4) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  5) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.


python3 - <<'PY'
import requests, json, time
url='http://localhost:8000/v1/tts/generate'
payload={"text":"Hello","speaker":"lipakshi","language":"English","model":"default","output_sample_rate":"16khz","max_tokens":64,"chunking":false,"stream":false}
try:
    t=time.time();
    r=requests.post(url,json=payload,timeout=20)
    print('status',r.status_code,'time',time.time()-t)
    print('headers', {k:v for k,v in r.headers.items() if k.lower().startswith('x-') and 'llm' in k.lower() or 'perf' in k.lower()})
    print('len',len(r.content))
except Exception as e:
    print('err',type(e).__name__,e)
PY


Output:
Chunk ID: 98f5a5
Wall time: 0.0509 seconds
Process exited with code 1
Original token count: 34
Output:
Traceback (most recent call last):
  File "<stdin>", line 3, in <module>
NameError: name 'false' is not defined. Did you mean: 'False'?


Summary: auto


python3 - <<'PY'
import requests, json, time
url='http://localhost:8000/v1/tts/generate'
payload={"text":"Hello","speaker":"lipakshi","language":"English","model":"default","output_sample_rate":"16khz","max_tokens":64,"chunking":False,"stream":False}
try:
    t=time.time();
    r=requests.post(url,json=payload,timeout=20)
    print('status',r.status_code,'time',time.time()-t)
    print('headers', {k:v for k,v in r.headers.items() if k.lower().startswith('x-') and ('llm' in k.lower() or 'perf' in k.lower() or 'timing' in k.lower())})
    print('len',len(r.content))
except Exception as e:
    print('err',type(e).__name__,e)
PY


Output:
Chunk ID: a4447e
Wall time: 0.0511 seconds
Process exited with code 0
Original token count: 14
Output:
status 400 time 0.003292083740234375
headers {}
len 350


python3 - <<'PY'
import requests, json, time
url='http://localhost:8000/v1/tts/generate'
payload={"text":"Hello","speaker":"lipakshi","language":"English","model":"default","output_sample_rate":"16khz","max_tokens":64,"chunking":False,"stream":False}
r=requests.post(url,json=payload,timeout=20)
print(r.status_code)
print(r.text)
PY


Output:
Chunk ID: 1a92e5
Wall time: 0.0513 seconds
Process exited with code 0
Original token count: 89
Output:
400
{"error":{"code":"VALIDATION_ERROR","message":"1 validation error for TTSGenerateRequest\nmax_tokens\n  Input should be greater than or equal to 128 [type=greater_than_equal, input_value=64, input_type=int]\n    For further information visit https://errors.pydantic.dev/2.12/v/greater_than_equal","request_id":"72770ec0-0170-4f2b-acd0-57f073f40bc9"}}


python3 - <<'PY'
import requests, time
url='http://localhost:8000/v1/tts/generate'
payload={"text":"Hello","speaker":"lipakshi","language":"English","output_sample_rate":"16khz","max_tokens":128,"chunking":False,"stream":False}
t=time.time()
try:
    r=requests.post(url,json=payload,timeout=20)
    print('status',r.status_code,'time',time.time()-t)
    print('len',len(r.content))
    print('headers keys',[k for k in r.headers if k.lower().startswith('x-')][:20])
    print('x-perf',r.headers.get('x-perf-details'))
    print('x-llm-total',r.headers.get('x-llm-total-ms'))
    print('sample',r.text[:200])
except Exception as e:
    print('err',type(e).__name__,e)
PY


Output:
Chunk ID: e920d2
Wall time: 10.0018 seconds
Process exited with code 0
Original token count: 62
Output:
status 500 time 10.005747318267822
len 137
headers keys ['x-request-id']
x-perf None
x-llm-total None
sample {"error":{"code":"INTERNAL_ERROR","message":"LLM batch wait timed out after 10.00s","request_id":"0a2684b5-7bad-43e4-988d-cc57e15f3710"}}


I see the non-stream baseline is currently failing with a 10s batch wait timeout (`LLM batch wait timed out after 10.00s`). I’m going to restart the local server with higher timeout settings so we get full generation traces instead of hard failures, then rerun the same benchmarks.


pkill -f 'python3 -m veena3modal.local_server' && sleep 1 && pgrep -f 'python3 -m veena3modal.local_server' || true


Output:
Chunk ID: 501277
Wall time: 0.0509 seconds
Process exited with code -1
Original token count: 0
Output:


cd /home/ubuntu/veenaModal && VEENA3_TIMING_LOGS=true VEENA3_REQUEST_TIMEOUT_SECONDS=300 VEENA3_PIPELINE_BATCH_TIMEOUT_SECONDS=120 NUMBA_DISABLE_JIT=1 python3 -m veena3modal.local_server --skip-download --model-path /home/ubuntu/veenaModal/models/spark_tts_4speaker --port 8000 --gpu-memory 0.15 --num-engines 1 --log-level INFO


Output:
Chunk ID: 47a462
Wall time: 30.0013 seconds
Process running with session ID 37173
Original token count: 3043
Output:
2026-02-17 00:36:31 | INFO     | veena3.local | Auth bypass enabled (local mode). Use --auth to require API keys.
2026-02-17 00:36:31 | INFO     | veena3.local | Resolved model paths:
2026-02-17 00:36:31 | INFO     | veena3.local |   LLM (vLLM):    /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
2026-02-17 00:36:31 | INFO     | veena3.local |   BiCodec:       /home/ubuntu/veenaModal/models/spark_tts_4speaker
2026-02-17 00:36:31 | INFO     | veena3.local | ============================================================
2026-02-17 00:36:31 | INFO     | veena3.local | Initializing Veena3 TTS Runtime (local)
2026-02-17 00:36:31 | INFO     | veena3.local |   Model:       /home/ubuntu/veenaModal/models/spark_tts_4speaker
2026-02-17 00:36:31 | INFO     | veena3.local |   LLM path:    /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
2026-02-17 00:36:31 | INFO     | veena3.local |   BiCodec:     /home/ubuntu/veenaModal/models/spark_tts_4speaker
2026-02-17 00:36:31 | INFO     | veena3.local |   Device:      cuda
2026-02-17 00:36:31 | INFO     | veena3.local |   GPU Memory:  15% (total)
2026-02-17 00:36:31 | INFO     | veena3.local |   Engines:     1
2026-02-17 00:36:31 | INFO     | veena3.local |   SR:          disabled
2026-02-17 00:36:31 | INFO     | veena3.local | ============================================================
2026-02-17 00:36:31 | INFO     | veena3modal.services.tts_runtime | Initializing TTS runtime...
2026-02-17 00:36:31 | INFO     | veena3modal.services.tts_runtime |   Model path: /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
2026-02-17 00:36:31 | INFO     | veena3modal.services.tts_runtime |   BiCodec path: /home/ubuntu/veenaModal/models/spark_tts_4speaker
2026-02-17 00:36:31 | INFO     | veena3modal.services.tts_runtime |   SR enabled: False, path: None
2026-02-17 00:36:36 | INFO     | veena3modal.services.tts_runtime | Loading SparkTTS model with vLLM...
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
🚀 Initializing Spark TTS Model
📁 Model path: /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
🗣  Speakers: 12 predefined (lipakshi, vardan, reet, Nandini, krishna, anika, adarsh, Nilay, Aarvi, Asha, Bittu, Mira)
🎭 Emotions: 10 tags in [bracket] format

📝 Loading tokenizer from model...
✅ Tokenizer loaded: 165168 tokens

🔍 Validating 12 speakers for Spark TTS...
  lipakshi → <|speaker_0|>: 7 tokens
  vardan → <|speaker_1|>: 7 tokens
  reet → <|speaker_2|>: 7 tokens
  Nandini → <|speaker_3|>: 7 tokens
  krishna → <|speaker_4|>: 7 tokens
  anika → <|speaker_5|>: 7 tokens
  adarsh → <|speaker_6|>: 7 tokens
  Nilay → <|speaker_7|>: 7 tokens
  Aarvi → <|speaker_8|>: 7 tokens
  Asha → <|speaker_9|>: 7 tokens
  Bittu → <|speaker_10|>: 8 tokens
  Mira → <|speaker_11|>: 8 tokens
✅ All 12 speakers validated with mapping

🔍 Validating 10 emotion tags for Spark TTS...
✅ All 10 emotion tags validated

📋 Sample emotion tags:
  [angry]: 1 tokens
  [curious]: 1 tokens
  [excited]: 1 tokens
  [giggle]: 1 tokens
  [laughs harder]: 1 tokens
  ...

🔧 Initializing vLLM engine with optimizations...
  - dtype: bfloat16
  - max_model_len: 4096
  - gpu_memory_utilization: 0.15
  - enable_prefix_caching: True
  - enforce_eager (disable CUDA graphs): False
INFO 02-17 00:36:37 [model.py:541] Resolved architecture: Qwen2ForCausalLM
INFO 02-17 00:36:37 [model.py:1561] Using max model len 4096
INFO 02-17 00:36:37 [scheduler.py:226] Chunked prefill is enabled with max_num_batched_tokens=4096.
INFO 02-17 00:36:37 [vllm.py:624] Asynchronous scheduling is enabled.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(EngineCore_DP0 pid=3713837) INFO 02-17 00:36:38 [core.py:96] Initializing a V1 LLM engine (v0.15.1) with config: model='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', speculative_config=None, tokenizer='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [4096], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 256, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': True}, 'local_cache_dir': None, 'static_all_moe_layers': []}
(EngineCore_DP0 pid=3713837) INFO 02-17 00:36:41 [parallel_state.py:1212] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://216.81.248.184:55449 backend=nccl
(EngineCore_DP0 pid=3713837) INFO 02-17 00:36:41 [parallel_state.py:1423] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
(EngineCore_DP0 pid=3713837) INFO 02-17 00:36:41 [gpu_model_runner.py:4033] Starting to load model /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM...
(EngineCore_DP0 pid=3713837) INFO 02-17 00:36:42 [cuda.py:364] Using FLASH_ATTN attention backend out of potential backends: ('FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION')
(EngineCore_DP0 pid=3713837) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
(EngineCore_DP0 pid=3713837) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
(EngineCore_DP0 pid=3713837)
Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
(EngineCore_DP0 pid=3713837)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:01<00:00,  1.29s/it]
(EngineCore_DP0 pid=3713837)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:01<00:00,  1.29s/it]
(EngineCore_DP0 pid=3713837)
(EngineCore_DP0 pid=3713837) INFO 02-17 00:36:44 [default_loader.py:291] Loading weights took 1.37 seconds
(EngineCore_DP0 pid=3713837) INFO 02-17 00:36:44 [gpu_model_runner.py:4130] Model loading took 0.95 GiB memory and 1.900412 seconds
(EngineCore_DP0 pid=3713837) INFO 02-17 00:36:49 [backends.py:812] Using cache directory: /home/ubuntu/.cache/vllm/torch_compile_cache/f3ff13511c/rank_0_0/backbone for vLLM's torch.compile
(EngineCore_DP0 pid=3713837) INFO 02-17 00:36:49 [backends.py:872] Dynamo bytecode transform time: 4.21 s
(EngineCore_DP0 pid=3713837) INFO 02-17 00:36:52 [backends.py:267] Directly load the compiled graph(s) for compile range (1, 4096) from the cache, took 0.676 s
(EngineCore_DP0 pid=3713837) INFO 02-17 00:36:52 [monitor.py:34] torch.compile takes 4.88 s in total
(EngineCore_DP0 pid=3713837) INFO 02-17 00:36:53 [gpu_worker.py:356] Available KV cache memory: 10.13 GiB
(EngineCore_DP0 pid=3713837) INFO 02-17 00:36:53 [kv_cache_utils.py:1307] GPU KV cache size: 885,184 tokens
(EngineCore_DP0 pid=3713837) INFO 02-17 00:36:53 [kv_cache_utils.py:1312] Maximum concurrency for 4,096 tokens per request: 216.11x
(EngineCore_DP0 pid=3713837)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   0%|          | 0/35 [00:00<?, ?it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  11%|█▏        | 4/35 [00:00<00:00, 32.62it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  23%|██▎       | 8/35 [00:00<00:00, 35.24it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  34%|███▍      | 12/35 [00:00<00:00, 36.72it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  46%|████▌     | 16/35 [00:00<00:00, 36.87it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  57%|█████▋    | 20/35 [00:00<00:00, 37.03it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  69%|██████▊   | 24/35 [00:00<00:00, 37.03it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  80%|████████  | 28/35 [00:00<00:00, 36.91it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  91%|█████████▏| 32/35 [00:00<00:00, 35.20it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 35/35 [00:00<00:00, 35.24it/s]
(EngineCore_DP0 pid=3713837)
Capturing CUDA graphs (decode, FULL):   0%|          | 0/19 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL):  16%|█▌        | 3/19 [00:00<00:00, 29.48it/s]
Capturing CUDA graphs (decode, FULL):  37%|███▋      | 7/19 [00:00<00:00, 33.73it/s]
Capturing CUDA graphs (decode, FULL):  58%|█████▊    | 11/19 [00:00<00:00, 34.73it/s]
Capturing CUDA graphs (decode, FULL):  79%|███████▉  | 15/19 [00:00<00:00, 35.38it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 36.03it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 35.14it/s]
(EngineCore_DP0 pid=3713837) INFO 02-17 00:36:55 [gpu_model_runner.py:5063] Graph capturing finished in 2 secs, took 0.28 GiB
(EngineCore_DP0 pid=3713837) INFO 02-17 00:36:55 [core.py:272] init engine (profile, create kv cache, warmup model) took 10.79 seconds
2026-02-17 00:36:57 | INFO     | veena3modal.services.tts_runtime | Initializing prompt builder...
2026-02-17 00:36:57 | INFO     | veena3modal.services.tts_runtime | Initializing BiCodec decoder...
/home/ubuntu/.local/lib/python3.10/site-packages/torch/nn/utils/weight_norm.py:144: FutureWarning: `torch.nn.utils.weight_norm` is deprecated in favor of `torch.nn.utils.parametrizations.weight_norm`.
  WeightNorm.apply(module, name, dim)


Output:
Chunk ID: 748f90
Wall time: 30.0002 seconds
Process running with session ID 37173
Original token count: 1013
Output:
2026-02-17 00:37:04 | INFO     | veena3modal.services.tts_runtime | Initializing TTS pipeline...
2026-02-17 00:37:04 | INFO     | veena3modal.services.tts_runtime | Initializing streaming pipeline...
2026-02-17 00:37:04 | INFO     | veena3modal.services.tts_runtime | ✅ TTS runtime initialized in 33210ms
2026-02-17 00:37:04 | INFO     | veena3modal.services.tts_runtime |    Model version: LLM
2026-02-17 00:37:04 | INFO     | veena3modal.services.tts_runtime | Pre-computing global tokens for all speakers...
2026-02-17 00:37:05 | INFO     | veena3modal.services.tts_runtime |   Cached globals for lipakshi
2026-02-17 00:37:05 | INFO     | veena3modal.services.tts_runtime |   Cached globals for vardan
2026-02-17 00:37:06 | INFO     | veena3modal.services.tts_runtime |   Cached globals for reet
2026-02-17 00:37:06 | INFO     | veena3modal.services.tts_runtime |   Cached globals for Nandini
2026-02-17 00:37:06 | INFO     | veena3modal.services.tts_runtime |   Cached globals for krishna
2026-02-17 00:37:07 | INFO     | veena3modal.services.tts_runtime |   Cached globals for anika
2026-02-17 00:37:07 | INFO     | veena3modal.services.tts_runtime |   Cached globals for adarsh
2026-02-17 00:37:08 | INFO     | veena3modal.services.tts_runtime |   Cached globals for Nilay
2026-02-17 00:37:08 | INFO     | veena3modal.services.tts_runtime |   Cached globals for Aarvi
2026-02-17 00:37:08 | INFO     | veena3modal.services.tts_runtime |   Cached globals for Asha
2026-02-17 00:37:09 | INFO     | veena3modal.services.tts_runtime |   Cached globals for Bittu
2026-02-17 00:37:09 | INFO     | veena3modal.services.tts_runtime |   Cached globals for Mira
2026-02-17 00:37:09 | INFO     | veena3modal.services.tts_runtime | Speaker globals cached: 12/12 speakers in 4986ms
2026-02-17 00:37:09 | INFO     | veena3.local | Runtime initialized in 38.2s
2026-02-17 00:37:09 | INFO     | veena3.local |   Model version: LLM
2026-02-17 00:37:09 | INFO     | veena3.local |   Streaming: available
2026-02-17 00:37:09 | INFO     | veena3.local |   Super-resolution: disabled
2026-02-17 00:37:09 | INFO     | veena3.local |
2026-02-17 00:37:09 | INFO     | veena3.local | ============================================================
2026-02-17 00:37:09 | INFO     | veena3.local | Veena3 TTS Local Server starting
2026-02-17 00:37:09 | INFO     | veena3.local |   URL: http://0.0.0.0:8000
2026-02-17 00:37:09 | INFO     | veena3.local |   Health: http://0.0.0.0:8000/v1/tts/health
2026-02-17 00:37:09 | INFO     | veena3.local |   Generate: POST http://0.0.0.0:8000/v1/tts/generate
2026-02-17 00:37:09 | INFO     | veena3.local |   Metrics: http://0.0.0.0:8000/v1/tts/metrics
2026-02-17 00:37:09 | INFO     | veena3.local |   Docs: http://0.0.0.0:8000/docs
2026-02-17 00:37:09 | INFO     | veena3.local | ============================================================
✅ vLLM engine initialized successfully!

🎉 Spark TTS Model ready for inference!

🎵 Loading BiCodec audio tokenizer from /home/ubuntu/veenaModal/models/spark_tts_4speaker...
   Device: cuda
Missing tensor: mel_transformer.spectrogram.window
Missing tensor: mel_transformer.mel_scale.fb
✅ BiCodec decoder initialized (sample rate: 16000Hz)
🚀 SparkTTSPipeline initialized
🌊 Veena3SlidingWindowPipeline initialized (sliding window: 28 tokens)
{"timestamp": "2026-02-17T00:37:09.766740+00:00", "level": "INFO", "logger": "veena3modal.api.websocket_handler", "message": "WebSocket route /v1/tts/ws registered"}
{"timestamp": "2026-02-17T00:37:09.766887+00:00", "level": "INFO", "logger": "veena3modal.api.fastapi_app", "message": "WebSocket support enabled at /v1/tts/ws"}
2026-02-17 00:37:09 | INFO     | veena3modal.api.websocket_handler | WebSocket route /v1/tts/ws registered
2026-02-17 00:37:09 | INFO     | veena3modal.api.fastapi_app | WebSocket support enabled at /v1/tts/ws
INFO:     Started server process [3713724]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)


Local server is now fully initialized and responsive. I’ll run a single direct request and then the scripted benchmark levels to capture the request-level timing fields plus GPU snapshots and concurrency stats.


cd /home/ubuntu/veenaModal && python3 - <<'PY'
import requests, time
url='http://localhost:8000/v1/tts/generate'
payload={"text":"Hello","speaker":"lipakshi","language":"English","output_sample_rate":"16khz","max_tokens":128,"chunking":False,"stream":False}
t=time.time()
r=requests.post(url,json=payload,timeout=120)
print('status',r.status_code,'time',time.time()-t)
print('request_id',r.headers.get('x-request-id'))
for k in sorted([x for x in r.headers if x.lower().startswith('x-') and ('llm' in x.lower() or 'perf' in x.lower() or 'timing' in x.lower()]):
    print(f"{k}: {r.headers[k]}")
print('body len',len(r.content))
print(r.text[:200])
PY


Output:
Chunk ID: e943db
Wall time: 0.0512 seconds
Process exited with code 1
Original token count: 99
Output:
  File "<stdin>", line 8
    for k in sorted([x for x in r.headers if x.lower().startswith('x-') and ('llm' in x.lower() or 'perf' in x.lower() or 'timing' in x.lower()]):
                                                                                ^
SyntaxError: closing parenthesis ']' does not match opening parenthesis '('


cd /home/ubuntu/veenaModal && python3 - <<'PY'
import requests, time
url='http://localhost:8000/v1/tts/generate'
payload={"text":"Hello","speaker":"lipakshi","language":"English","output_sample_rate":"16khz","max_tokens":128,"chunking":False,"stream":False}
t=time.time()
r=requests.post(url,json=payload,timeout=120)
print('status',r.status_code,'time',time.time()-t)
print('request_id',r.headers.get('x-request-id'))
for k in sorted([x for x in r.headers if x.lower().startswith('x-') and (('llm' in x.lower()) or ('perf' in x.lower()) or ('timing' in x.lower()))]):
    print(f"{k}: {r.headers[k]}")
print('body len',len(r.content))
print(r.text[:120])
PY


Output:
Chunk ID: 1fe840
Wall time: 30.0009 seconds
Process running with session ID 97826
Original token count: 0
Output:


Output:
Chunk ID: e32537
Wall time: 30.0011 seconds
Process running with session ID 97826
Original token count: 0
Output:


Output:
Chunk ID: 533e71
Wall time: 30.0009 seconds
Process running with session ID 97826
Original token count: 0
Output:


The request is taking much longer than 30s with the current parameters, so I’m checking server logs for the per-request stage timings and any timeout/bottleneck signals.


Output:
Chunk ID: 61b69b
Wall time: 30.0009 seconds
Process running with session ID 37173
Original token count: 1857
Output:
{"timestamp": "2026-02-17T00:37:36.236586+00:00", "level": "INFO", "logger": "veena3modal.tts", "message": "request_received", "request_id": "a580df4f-badd-4c15-9fdc-b9fe1b045e2d", "event": "request_received", "text_length": 5, "speaker": "lipakshi", "stream": false, "format": "wav"}
2026-02-17 00:37:36 | INFO     | veena3modal.tts | request_received
2026-02-17 00:37:36 | INFO     | veena3modal.services.tts_runtime | tts_runtime_nonstream_start
2026-02-17 00:37:36 | INFO     | veena3modal.core.pipeline | tts_llm_generate_start
2026-02-17 00:39:36 | ERROR    | veena3modal.core.pipeline | tts_llm_batch_timeout
2026-02-17 00:39:36 | ERROR    | veena3modal.services.tts_runtime | Speech generation failed: LLM batch wait timed out after 120.00s
2026-02-17 00:39:36 | ERROR    | veena3modal.services.tts_runtime | tts_runtime_nonstream_error
Traceback (most recent call last):
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/async_llm.py", line 572, in generate
    out = q.get_nowait() or await q.get()
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/output_processor.py", line 76, in get
    await self.ready.wait()
  File "/usr/lib/python3.10/asyncio/locks.py", line 214, in wait
    await fut
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.10/asyncio/tasks.py", line 456, in wait_for
    return fut.result()
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ubuntu/veenaModal/veena3modal/core/pipeline.py", line 172, in generate_speech
    request_output = await asyncio.wait_for(
  File "/usr/lib/python3.10/asyncio/tasks.py", line 458, in wait_for
    raise exceptions.TimeoutError() from exc
asyncio.exceptions.TimeoutError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py", line 472, in generate_speech
    audio_bytes, pipeline_timings = await asyncio.wait_for(
  File "/usr/lib/python3.10/asyncio/tasks.py", line 445, in wait_for
    return fut.result()
  File "/home/ubuntu/veenaModal/veena3modal/core/pipeline.py", line 401, in generate_speech_indic
    return await self.generate_speech(
  File "/home/ubuntu/veenaModal/veena3modal/core/pipeline.py", line 189, in generate_speech
    raise TimeoutError(
TimeoutError: LLM batch wait timed out after 120.00s
2026-02-17 00:39:36 | INFO     | veena3modal.services.tts_runtime | tts_runtime_nonstream_complete
2026-02-17 00:39:36 | INFO     | veena3modal.services.tts_runtime | tts_runtime_nonstream_inflight
{"timestamp": "2026-02-17T00:39:36.239130+00:00", "level": "ERROR", "logger": "veena3modal.api.fastapi_app", "message": "TTS generation error: LLM batch wait timed out after 120.00s", "request_id": "a580df4f-badd-4c15-9fdc-b9fe1b045e2d", "exception": "Traceback (most recent call last):\n  File \"/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/async_llm.py\", line 572, in generate\n    out = q.get_nowait() or await q.get()\n  File \"/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/output_processor.py\", line 76, in get\n    await self.ready.wait()\n  File \"/usr/lib/python3.10/asyncio/locks.py\", line 214, in wait\n    await fut\nasyncio.exceptions.CancelledError\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/usr/lib/python3.10/asyncio/tasks.py\", line 456, in wait_for\n    return fut.result()\nasyncio.exceptions.CancelledError\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n  File \"/home/ubuntu/veenaModal/veena3modal/core/pipeline.py\", line 172, in generate_speech\n    request_output = await asyncio.wait_for(\n  File \"/usr/lib/python3.10/asyncio/tasks.py\", line 458, in wait_for\n    raise exceptions.TimeoutError() from exc\nasyncio.exceptions.TimeoutError\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n  File \"/home/ubuntu/veenaModal/veena3modal/api/fastapi_app.py\", line 460, in tts_generate\n    audio_bytes, metrics = await tts_runtime.generate_speech(\n  File \"/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py\", line 472, in generate_speech\n    audio_bytes, pipeline_timings = await asyncio.wait_for(\n  File \"/usr/lib/python3.10/asyncio/tasks.py\", line 445, in wait_for\n    return fut.result()\n  File \"/home/ubuntu/veenaModal/veena3modal/core/pipeline.py\", line 401, in generate_speech_indic\n    return await self.generate_speech(\n  File \"/home/ubuntu/veenaModal/veena3modal/core/pipeline.py\", line 189, in generate_speech\n    raise TimeoutError(\nTimeoutError: LLM batch wait timed out after 120.00s\n"}
2026-02-17 00:39:36 | ERROR    | veena3modal.api.fastapi_app | TTS generation error: LLM batch wait timed out after 120.00s
Traceback (most recent call last):
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/async_llm.py", line 572, in generate
    out = q.get_nowait() or await q.get()
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/output_processor.py", line 76, in get
    await self.ready.wait()
  File "/usr/lib/python3.10/asyncio/locks.py", line 214, in wait
    await fut
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.10/asyncio/tasks.py", line 456, in wait_for
    return fut.result()
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ubuntu/veenaModal/veena3modal/core/pipeline.py", line 172, in generate_speech
    request_output = await asyncio.wait_for(
  File "/usr/lib/python3.10/asyncio/tasks.py", line 458, in wait_for
    raise exceptions.TimeoutError() from exc
asyncio.exceptions.TimeoutError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ubuntu/veenaModal/veena3modal/api/fastapi_app.py", line 460, in tts_generate
    audio_bytes, metrics = await tts_runtime.generate_speech(
  File "/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py", line 472, in generate_speech
    audio_bytes, pipeline_timings = await asyncio.wait_for(
  File "/usr/lib/python3.10/asyncio/tasks.py", line 445, in wait_for
    return fut.result()
  File "/home/ubuntu/veenaModal/veena3modal/core/pipeline.py", line 401, in generate_speech_indic
    return await self.generate_speech(
  File "/home/ubuntu/veenaModal/veena3modal/core/pipeline.py", line 189, in generate_speech
    raise TimeoutError(
TimeoutError: LLM batch wait timed out after 120.00s
{"timestamp": "2026-02-17T00:39:36.239633+00:00", "level": "ERROR", "logger": "veena3modal.tts", "message": "request_failed", "request_id": "a580df4f-badd-4c15-9fdc-b9fe1b045e2d", "event": "request_failed", "status_code": 500, "error_code": "INTERNAL_ERROR", "error_message": "LLM batch wait timed out after 120.00s"}
2026-02-17 00:39:36 | ERROR    | veena3modal.tts | request_failed
INFO:     127.0.0.1:33000 - "POST /v1/tts/generate HTTP/1.1" 500 Internal Server Error


cd /home/ubuntu/veenaModal && python3 - <<'PY'
import requests, time
url='http://localhost:8000/v1/tts/generate'
payload={"text":"Hello","speaker":"lipakshi","max_tokens":128,"stream":True,"chunking":False}
t=time.time()
try:
    r=requests.post(url,json=payload,timeout=120,stream=True)
    print('status',r.status_code,'time',time.time()-t)
    print('headers',r.headers.get('x-request-id'))
    print('body bytes',len(r.content))
    print(r.text[:200])
except Exception as e:
    print('err',type(e).__name__,e)
PY


Output:
Chunk ID: 9eead2
Wall time: 30.0013 seconds
Process running with session ID 82727
Original token count: 0
Output:


Output:
Chunk ID: 579628
Wall time: 30.0009 seconds
Process running with session ID 82727
Original token count: 0
Output:


Output:
Chunk ID: 6bd521
Wall time: 30.0008 seconds
Process running with session ID 37173
Original token count: 156
Output:
{"timestamp": "2026-02-17T00:39:41.830807+00:00", "level": "INFO", "logger": "veena3modal.tts", "message": "request_received", "request_id": "db132c51-9eab-4f64-98a3-cc849389966e", "event": "request_received", "text_length": 5, "speaker": "lipakshi", "stream": true, "format": "wav"}
2026-02-17 00:39:41 | INFO     | veena3modal.tts | request_received
INFO:     127.0.0.1:55994 - "POST /v1/tts/generate HTTP/1.1" 200 OK
2026-02-17 00:39:41 | INFO     | veena3modal.services.tts_runtime | tts_runtime_stream_start
2026-02-17 00:39:41 | INFO     | veena3modal.processing.long_text_processor | LongTextProcessor initialized


Output:
Chunk ID: 099365
Wall time: 26.8225 seconds
Process exited with code 0
Original token count: 42
Output:
status 200 time 0.009625911712646484
headers db132c51-9eab-4f64-98a3-cc849389966e
err ConnectionError HTTPConnectionPool(host='localhost', port=8000): Read timed out.


Output:
Chunk ID: 4fb8ae
Wall time: 30.0010 seconds
Process running with session ID 37173
Original token count: 23
Output:
2026-02-17 00:41:41 | INFO     | veena3modal.services.tts_runtime | tts_runtime_stream_end


python3 - <<'PY'
import requests, json
u='http://localhost:8000/openapi.json'
obj=requests.get(u,timeout=10).json()
op=obj['paths']['/v1/tts/generate']['post']
print(op['summary'])
req=op['requestBody']['content']['application/json']['schema']
print(json.dumps(req,indent=2)[:2000])
PY


Output:
Chunk ID: 33d8ed
Wall time: 0.0508 seconds
Process exited with code 1
Original token count: 28
Output:
Tts Generate
Traceback (most recent call last):
  File "<stdin>", line 6, in <module>
KeyError: 'requestBody'


python3 - <<'PY'
import requests, json
obj=requests.get('http://localhost:8000/openapi.json',timeout=10).json()
pp=obj['paths']['/v1/tts/generate']
print(pp.keys())
print(pp['post'].keys())
print(pp['post'].get('requestBody'))
print(pp['post'].get('parameters'))
print(pp['post'].get('description',''))
PY


Output:
Chunk ID: af96bc
Wall time: 0.0506 seconds
Process exited with code 0
Original token count: 59
Output:
dict_keys(['post'])
dict_keys(['summary', 'description', 'operationId', 'responses'])
None
None
Generate speech from text.

Supports streaming (M4) and non-streaming (M3) modes.
Currently implements non-streaming WAV generation only.


I validated the API is alive. The key problem now is request execution: non-stream calls still don’t complete and stream mode appears to close before payload emission. I’m running an even narrower payload probe to check whether the route schema is causing prompt/path mismatches.


cd /home/ubuntu/veenaModal && python3 - <<'PY'
import requests, time
url='http://localhost:8000/v1/tts/generate'
payload={"text":"Hello","speaker":"lipakshi"}
t=time.time()
try:
    r=requests.post(url,json=payload,timeout=120)
    print('status',r.status_code,'time',time.time()-t)
    print(r.text[:200])
except Exception as e:
    print('err',type(e).__name__,e)
PY


Output:
Chunk ID: eb0949
Wall time: 30.0003 seconds
Process running with session ID 63572
Original token count: 0
Output:


Output:
Chunk ID: ab3a52
Wall time: 30.0010 seconds
Process running with session ID 63572
Original token count: 0
Output:


Output:
Chunk ID: 781c23
Wall time: 30.0014 seconds
Process running with session ID 37173
Original token count: 192
Output:
INFO:     127.0.0.1:57060 - "GET /openapi.json HTTP/1.1" 200 OK
INFO:     127.0.0.1:57064 - "GET /openapi.json HTTP/1.1" 200 OK
{"timestamp": "2026-02-17T00:42:20.083868+00:00", "level": "INFO", "logger": "veena3modal.tts", "message": "request_received", "request_id": "7ea11c03-be2e-4e17-bc4c-f4cfb79ec90a", "event": "request_received", "text_length": 5, "speaker": "lipakshi", "stream": false, "format": "wav"}
2026-02-17 00:42:20 | INFO     | veena3modal.tts | request_received
2026-02-17 00:42:20 | INFO     | veena3modal.services.tts_runtime | tts_runtime_chunking_start
2026-02-17 00:42:20 | INFO     | veena3modal.processing.long_text_processor | LongTextProcessor initialized
2026-02-17 00:42:20 | INFO     | veena3modal.core.pipeline | tts_llm_generate_start


Output:
Chunk ID: ef6526
Wall time: 30.0011 seconds
Process running with session ID 37173
Original token count: 1230
Output:
2026-02-17 00:44:20 | ERROR    | veena3modal.core.pipeline | tts_llm_batch_timeout
2026-02-17 00:44:20 | ERROR    | veena3modal.services.tts_runtime | Chunked speech generation failed: LLM batch wait timed out after 120.00s
{"timestamp": "2026-02-17T00:44:20.085494+00:00", "level": "ERROR", "logger": "veena3modal.api.fastapi_app", "message": "TTS generation error: LLM batch wait timed out after 120.00s", "request_id": "7ea11c03-be2e-4e17-bc4c-f4cfb79ec90a", "exception": "Traceback (most recent call last):\n  File \"/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/async_llm.py\", line 572, in generate\n    out = q.get_nowait() or await q.get()\n  File \"/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/output_processor.py\", line 76, in get\n    await self.ready.wait()\n  File \"/usr/lib/python3.10/asyncio/locks.py\", line 214, in wait\n    await fut\nasyncio.exceptions.CancelledError\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/usr/lib/python3.10/asyncio/tasks.py\", line 456, in wait_for\n    return fut.result()\nasyncio.exceptions.CancelledError\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n  File \"/home/ubuntu/veenaModal/veena3modal/core/pipeline.py\", line 172, in generate_speech\n    request_output = await asyncio.wait_for(\n  File \"/usr/lib/python3.10/asyncio/tasks.py\", line 458, in wait_for\n    raise exceptions.TimeoutError() from exc\nasyncio.exceptions.TimeoutError\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n  File \"/home/ubuntu/veenaModal/veena3modal/api/fastapi_app.py\", line 446, in tts_generate\n    audio_bytes, metrics = await tts_runtime.generate_speech_chunked(\n  File \"/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py\", line 770, in generate_speech_chunked\n    audio_bytes, pipeline_timings = await asyncio.wait_for(\n  File \"/usr/lib/python3.10/asyncio/tasks.py\", line 445, in wait_for\n    return fut.result()\n  File \"/home/ubuntu/veenaModal/veena3modal/core/pipeline.py\", line 401, in generate_speech_indic\n    return await self.generate_speech(\n  File \"/home/ubuntu/veenaModal/veena3modal/core/pipeline.py\", line 189, in generate_speech\n    raise TimeoutError(\nTimeoutError: LLM batch wait timed out after 120.00s\n"}
2026-02-17 00:44:20 | ERROR    | veena3modal.api.fastapi_app | TTS generation error: LLM batch wait timed out after 120.00s
Traceback (most recent call last):
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/async_llm.py", line 572, in generate
    out = q.get_nowait() or await q.get()
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/output_processor.py", line 76, in get
    await self.ready.wait()
  File "/usr/lib/python3.10/asyncio/locks.py", line 214, in wait
    await fut
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.10/asyncio/tasks.py", line 456, in wait_for
    return fut.result()
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ubuntu/veenaModal/veena3modal/core/pipeline.py", line 172, in generate_speech
    request_output = await asyncio.wait_for(
  File "/usr/lib/python3.10/asyncio/tasks.py", line 458, in wait_for
    raise exceptions.TimeoutError() from exc
asyncio.exceptions.TimeoutError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ubuntu/veenaModal/veena3modal/api/fastapi_app.py", line 446, in tts_generate
    audio_bytes, metrics = await tts_runtime.generate_speech_chunked(
  File "/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py", line 770, in generate_speech_chunked
    audio_bytes, pipeline_timings = await asyncio.wait_for(
  File "/usr/lib/python3.10/asyncio/tasks.py", line 445, in wait_for
    return fut.result()
  File "/home/ubuntu/veenaModal/veena3modal/core/pipeline.py", line 401, in generate_speech_indic
    return await self.generate_speech(
  File "/home/ubuntu/veenaModal/veena3modal/core/pipeline.py", line 189, in generate_speech
    raise TimeoutError(
TimeoutError: LLM batch wait timed out after 120.00s
{"timestamp": "2026-02-17T00:44:20.086203+00:00", "level": "ERROR", "logger": "veena3modal.tts", "message": "request_failed", "request_id": "7ea11c03-be2e-4e17-bc4c-f4cfb79ec90a", "event": "request_failed", "status_code": 500, "error_code": "INTERNAL_ERROR", "error_message": "LLM batch wait timed out after 120.00s"}
2026-02-17 00:44:20 | ERROR    | veena3modal.tts | request_failed
INFO:     127.0.0.1:57068 - "POST /v1/tts/generate HTTP/1.1" 500 Internal Server Error


I’m confirming whether the stall is API-path specific or model-engine specific. I’ll run an in-process `tts_runtime.generate_speech` call with the same payload right now to isolate whether generation itself is producing outputs.


cd /home/ubuntu/veenaModal && python3 - <<'PY'
import asyncio, os, time
from veena3modal.services import tts_runtime
from veena3modal.services.tts_runtime import initialize_runtime

async def main():
    initialize_runtime(
        model_path='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM',
        bicodec_path='/home/ubuntu/veenaModal/models/spark_tts_4speaker',
        device='cuda',
        gpu_memory_utilization=0.15,
        num_engines=1,
    )
    payload={
        'text':'Hello',
        'speaker':'lipakshi',
        'output_sample_rate':'16khz',
        'max_tokens':128,
        'chunking':False,
        'stream':False,
    }
    t=time.time()
    try:
        audio_bytes, metrics = await asyncio.wait_for(
            tts_runtime.generate_speech(**payload),
            timeout=180,
        )
        print('ok',len(audio_bytes),metrics.keys())
    except Exception as e:
        print(type(e).__name__,e)
    print('elapsed',time.time()-t)

asyncio.run(main())
PY


Output:
Chunk ID: 7adb22
Wall time: 11.8885 seconds
Process exited with code 1
Original token count: 5390
Output:
[2026-02-17 00:44:30] INFO tts_runtime.py:179: Loading SparkTTS model with vLLM...
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
🚀 Initializing Spark TTS Model
📁 Model path: /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
🗣  Speakers: 12 predefined (lipakshi, vardan, reet, Nandini, krishna, anika, adarsh, Nilay, Aarvi, Asha, Bittu, Mira)
🎭 Emotions: 10 tags in [bracket] format

📝 Loading tokenizer from model...
✅ Tokenizer loaded: 165168 tokens

🔍 Validating 12 speakers for Spark TTS...
  lipakshi → <|speaker_0|>: 7 tokens
  vardan → <|speaker_1|>: 7 tokens
  reet → <|speaker_2|>: 7 tokens
  Nandini → <|speaker_3|>: 7 tokens
  krishna → <|speaker_4|>: 7 tokens
  anika → <|speaker_5|>: 7 tokens
  adarsh → <|speaker_6|>: 7 tokens
  Nilay → <|speaker_7|>: 7 tokens
  Aarvi → <|speaker_8|>: 7 tokens
  Asha → <|speaker_9|>: 7 tokens
  Bittu → <|speaker_10|>: 8 tokens
  Mira → <|speaker_11|>: 8 tokens
✅ All 12 speakers validated with mapping

🔍 Validating 10 emotion tags for Spark TTS...
✅ All 10 emotion tags validated

📋 Sample emotion tags:
  [angry]: 1 tokens
  [curious]: 1 tokens
  [excited]: 1 tokens
  [giggle]: 1 tokens
  [laughs harder]: 1 tokens
  ...

🔧 Initializing vLLM engine with optimizations...
  - dtype: bfloat16
  - max_model_len: 4096
  - gpu_memory_utilization: 0.15
  - enable_prefix_caching: True
  - enforce_eager (disable CUDA graphs): False
INFO 02-17 00:44:31 [model.py:541] Resolved architecture: Qwen2ForCausalLM
INFO 02-17 00:44:31 [model.py:1561] Using max model len 4096
INFO 02-17 00:44:31 [scheduler.py:226] Chunked prefill is enabled with max_num_batched_tokens=4096.
INFO 02-17 00:44:31 [vllm.py:624] Asynchronous scheduling is enabled.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(EngineCore_DP0 pid=3716665) INFO 02-17 00:44:32 [core.py:96] Initializing a V1 LLM engine (v0.15.1) with config: model='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', speculative_config=None, tokenizer='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [4096], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 256, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': True}, 'local_cache_dir': None, 'static_all_moe_layers': []}
(EngineCore_DP0 pid=3716665) INFO 02-17 00:44:35 [parallel_state.py:1212] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://216.81.248.184:59743 backend=nccl
(EngineCore_DP0 pid=3716665) INFO 02-17 00:44:35 [parallel_state.py:1423] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
(EngineCore_DP0 pid=3716665) ERROR 02-17 00:44:35 [core.py:946] EngineCore failed to start.
(EngineCore_DP0 pid=3716665) ERROR 02-17 00:44:35 [core.py:946] Traceback (most recent call last):
(EngineCore_DP0 pid=3716665) ERROR 02-17 00:44:35 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 937, in run_engine_core
(EngineCore_DP0 pid=3716665) ERROR 02-17 00:44:35 [core.py:946]     engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore_DP0 pid=3716665) ERROR 02-17 00:44:35 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 691, in __init__
(EngineCore_DP0 pid=3716665) ERROR 02-17 00:44:35 [core.py:946]     super().__init__(
(EngineCore_DP0 pid=3716665) ERROR 02-17 00:44:35 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 105, in __init__
(EngineCore_DP0 pid=3716665) ERROR 02-17 00:44:35 [core.py:946]     self.model_executor = executor_class(vllm_config)
(EngineCore_DP0 pid=3716665) ERROR 02-17 00:44:35 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/executor/abstract.py", line 101, in __init__
(EngineCore_DP0 pid=3716665) ERROR 02-17 00:44:35 [core.py:946]     self._init_executor()
(EngineCore_DP0 pid=3716665) ERROR 02-17 00:44:35 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/executor/uniproc_executor.py", line 47, in _init_executor
(EngineCore_DP0 pid=3716665) ERROR 02-17 00:44:35 [core.py:946]     self.driver_worker.init_device()
(EngineCore_DP0 pid=3716665) ERROR 02-17 00:44:35 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/worker/worker_base.py", line 326, in init_device
(EngineCore_DP0 pid=3716665) ERROR 02-17 00:44:35 [core.py:946]     self.worker.init_device()  # type: ignore
(EngineCore_DP0 pid=3716665) ERROR 02-17 00:44:35 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/worker/gpu_worker.py", line 258, in init_device
(EngineCore_DP0 pid=3716665) ERROR 02-17 00:44:35 [core.py:946]     from vllm.v1.worker.gpu_model_runner import (
(EngineCore_DP0 pid=3716665) ERROR 02-17 00:44:35 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/worker/gpu_model_runner.py", line 153, in <module>
(EngineCore_DP0 pid=3716665) ERROR 02-17 00:44:35 [core.py:946]     from vllm.v1.spec_decode.ngram_proposer import NgramProposer
(EngineCore_DP0 pid=3716665) ERROR 02-17 00:44:35 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/spec_decode/ngram_proposer.py", line 170, in <module>
(EngineCore_DP0 pid=3716665) ERROR 02-17 00:44:35 [core.py:946]     def batch_propose_numba(
(EngineCore_DP0 pid=3716665) ERROR 02-17 00:44:35 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/decorators.py", line 221, in wrapper
(EngineCore_DP0 pid=3716665) ERROR 02-17 00:44:35 [core.py:946]     disp = dispatcher(py_func=func, locals=locals,
(EngineCore_DP0 pid=3716665) ERROR 02-17 00:44:35 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/dispatcher.py", line 772, in __init__
(EngineCore_DP0 pid=3716665) ERROR 02-17 00:44:35 [core.py:946]     self.targetctx = self.targetdescr.target_context
(EngineCore_DP0 pid=3716665) ERROR 02-17 00:44:35 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/registry.py", line 30, in target_context
(EngineCore_DP0 pid=3716665) ERROR 02-17 00:44:35 [core.py:946]     return self._toplevel_target_context
(EngineCore_DP0 pid=3716665) ERROR 02-17 00:44:35 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/utils.py", line 741, in __get__
(EngineCore_DP0 pid=3716665) ERROR 02-17 00:44:35 [core.py:946]     return super().__get__(*args, **kwargs)
(EngineCore_DP0 pid=3716665) ERROR 02-17 00:44:35 [core.py:946]   File "/usr/lib/python3.10/functools.py", line 981, in __get__
(EngineCore_DP0 pid=3716665) ERROR 02-17 00:44:35 [core.py:946]     val = self.func(instance)
(EngineCore_DP0 pid=3716665) ERROR 02-17 00:44:35 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/registry.py", line 18, in _toplevel_target_context
(EngineCore_DP0 pid=3716665) ERROR 02-17 00:44:35 [core.py:946]     return cpu.CPUContext(self.typing_context, self._target_name)
(EngineCore_DP0 pid=3716665) ERROR 02-17 00:44:35 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/cpu.py", line 42, in __init__
(EngineCore_DP0 pid=3716665) ERROR 02-17 00:44:35 [core.py:946]     super().__init__(typingctx, target)
(EngineCore_DP0 pid=3716665) ERROR 02-17 00:44:35 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/base.py", line 254, in __init__
(EngineCore_DP0 pid=3716665) ERROR 02-17 00:44:35 [core.py:946]     self.init()
(EngineCore_DP0 pid=3716665) ERROR 02-17 00:44:35 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/compiler_lock.py", line 35, in _acquire_compile_lock
(EngineCore_DP0 pid=3716665) ERROR 02-17 00:44:35 [core.py:946]     return func(*args, **kwargs)
(EngineCore_DP0 pid=3716665) ERROR 02-17 00:44:35 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/cpu.py", line 51, in init
(EngineCore_DP0 pid=3716665) ERROR 02-17 00:44:35 [core.py:946]     self._internal_codegen = codegen.JITCPUCodegen("numba.exec")
(EngineCore_DP0 pid=3716665) ERROR 02-17 00:44:35 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/codegen.py", line 1172, in __init__
(EngineCore_DP0 pid=3716665) ERROR 02-17 00:44:35 [core.py:946]     initialize_llvm()
(EngineCore_DP0 pid=3716665) ERROR 02-17 00:44:35 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/codegen.py", line 1434, in initialize_llvm
(EngineCore_DP0 pid=3716665) ERROR 02-17 00:44:35 [core.py:946]     ll.initialize()
(EngineCore_DP0 pid=3716665) ERROR 02-17 00:44:35 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/llvmlite/binding/initfini.py", line 17, in initialize
(EngineCore_DP0 pid=3716665) ERROR 02-17 00:44:35 [core.py:946]     raise RuntimeError(
(EngineCore_DP0 pid=3716665) ERROR 02-17 00:44:35 [core.py:946] RuntimeError: llvmlite.binding.initialize() is deprecated and will be removed. LLVM initialization is now handled automatically. Please remove calls to this function from your code and check for other behavioral changes that may have occurred due to LLVM updates.
(EngineCore_DP0 pid=3716665) Process EngineCore_DP0:
(EngineCore_DP0 pid=3716665) Traceback (most recent call last):
(EngineCore_DP0 pid=3716665)   File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore_DP0 pid=3716665)     self.run()
(EngineCore_DP0 pid=3716665)   File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
(EngineCore_DP0 pid=3716665)     self._target(*self._args, **self._kwargs)
(EngineCore_DP0 pid=3716665)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 950, in run_engine_core
(EngineCore_DP0 pid=3716665)     raise e
(EngineCore_DP0 pid=3716665)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 937, in run_engine_core
(EngineCore_DP0 pid=3716665)     engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore_DP0 pid=3716665)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 691, in __init__
(EngineCore_DP0 pid=3716665)     super().__init__(
(EngineCore_DP0 pid=3716665)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 105, in __init__
(EngineCore_DP0 pid=3716665)     self.model_executor = executor_class(vllm_config)
(EngineCore_DP0 pid=3716665)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/executor/abstract.py", line 101, in __init__
(EngineCore_DP0 pid=3716665)     self._init_executor()
(EngineCore_DP0 pid=3716665)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/executor/uniproc_executor.py", line 47, in _init_executor
(EngineCore_DP0 pid=3716665)     self.driver_worker.init_device()
(EngineCore_DP0 pid=3716665)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/worker/worker_base.py", line 326, in init_device
(EngineCore_DP0 pid=3716665)     self.worker.init_device()  # type: ignore
(EngineCore_DP0 pid=3716665)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/worker/gpu_worker.py", line 258, in init_device
(EngineCore_DP0 pid=3716665)     from vllm.v1.worker.gpu_model_runner import (
(EngineCore_DP0 pid=3716665)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/worker/gpu_model_runner.py", line 153, in <module>
(EngineCore_DP0 pid=3716665)     from vllm.v1.spec_decode.ngram_proposer import NgramProposer
(EngineCore_DP0 pid=3716665)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/spec_decode/ngram_proposer.py", line 170, in <module>
(EngineCore_DP0 pid=3716665)     def batch_propose_numba(
(EngineCore_DP0 pid=3716665)   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/decorators.py", line 221, in wrapper
(EngineCore_DP0 pid=3716665)     disp = dispatcher(py_func=func, locals=locals,
(EngineCore_DP0 pid=3716665)   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/dispatcher.py", line 772, in __init__
(EngineCore_DP0 pid=3716665)     self.targetctx = self.targetdescr.target_context
(EngineCore_DP0 pid=3716665)   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/registry.py", line 30, in target_context
(EngineCore_DP0 pid=3716665)     return self._toplevel_target_context
(EngineCore_DP0 pid=3716665)   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/utils.py", line 741, in __get__
(EngineCore_DP0 pid=3716665)     return super().__get__(*args, **kwargs)
(EngineCore_DP0 pid=3716665)   File "/usr/lib/python3.10/functools.py", line 981, in __get__
(EngineCore_DP0 pid=3716665)     val = self.func(instance)
(EngineCore_DP0 pid=3716665)   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/registry.py", line 18, in _toplevel_target_context
(EngineCore_DP0 pid=3716665)     return cpu.CPUContext(self.typing_context, self._target_name)
(EngineCore_DP0 pid=3716665)   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/cpu.py", line 42, in __init__
(EngineCore_DP0 pid=3716665)     super().__init__(typingctx, target)
(EngineCore_DP0 pid=3716665)   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/base.py", line 254, in __init__
(EngineCore_DP0 pid=3716665)     self.init()
(EngineCore_DP0 pid=3716665)   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/compiler_lock.py", line 35, in _acquire_compile_lock
(EngineCore_DP0 pid=3716665)     return func(*args, **kwargs)
(EngineCore_DP0 pid=3716665)   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/cpu.py", line 51, in init
(EngineCore_DP0 pid=3716665)     self._internal_codegen = codegen.JITCPUCodegen("numba.exec")
(EngineCore_DP0 pid=3716665)   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/codegen.py", line 1172, in __init__
(EngineCore_DP0 pid=3716665)     initialize_llvm()
(EngineCore_DP0 pid=3716665)   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/codegen.py", line 1434, in initialize_llvm
(EngineCore_DP0 pid=3716665)     ll.initialize()
(EngineCore_DP0 pid=3716665)   File "/home/ubuntu/.local/lib/python3.10/site-packages/llvmlite/binding/initfini.py", line 17, in initialize
(EngineCore_DP0 pid=3716665)     raise RuntimeError(
(EngineCore_DP0 pid=3716665) RuntimeError: llvmlite.binding.initialize() is deprecated and will be removed. LLVM initialization is now handled automatically. Please remove calls to this function from your code and check for other behavioral changes that may have occurred due to LLVM updates.
[2026-02-17 00:44:35] ERROR tts_runtime.py:282: ❌ Failed to initialize TTS runtime: Engine core initialization failed. See root cause above. Failed core proc(s): {}
Traceback (most recent call last):
  File "/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py", line 180, in initialize_runtime
    model = SparkTTSModel(
  File "/home/ubuntu/veenaModal/veena3modal/core/model_loader.py", line 128, in __init__
    self.engine = AsyncLLMEngine.from_engine_args(engine_args)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/async_llm.py", line 257, in from_engine_args
    return cls(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/async_llm.py", line 155, in __init__
    self.engine_core = EngineCoreClient.make_async_mp_client(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 122, in make_async_mp_client
    return AsyncMPClient(*client_args)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 819, in __init__
    super().__init__(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 479, in __init__
    with launch_core_engines(vllm_config, executor_class, log_stats) as (
  File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
    next(self.gen)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/utils.py", line 933, in launch_core_engines
    wait_for_engine_startup(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/utils.py", line 992, in wait_for_engine_startup
    raise RuntimeError(
RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
Traceback (most recent call last):
  File "/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py", line 180, in initialize_runtime
    model = SparkTTSModel(
  File "/home/ubuntu/veenaModal/veena3modal/core/model_loader.py", line 128, in __init__
    self.engine = AsyncLLMEngine.from_engine_args(engine_args)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/async_llm.py", line 257, in from_engine_args
    return cls(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/async_llm.py", line 155, in __init__
    self.engine_core = EngineCoreClient.make_async_mp_client(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 122, in make_async_mp_client
    return AsyncMPClient(*client_args)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 819, in __init__
    super().__init__(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 479, in __init__
    with launch_core_engines(vllm_config, executor_class, log_stats) as (
  File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
    next(self.gen)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/utils.py", line 933, in launch_core_engines
    wait_for_engine_startup(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/utils.py", line 992, in wait_for_engine_startup
    raise RuntimeError(
RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 32, in <module>
  File "/usr/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "<stdin>", line 6, in main
  File "/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py", line 285, in initialize_runtime
    raise RuntimeError(f"TTS runtime initialization failed: {e}") from e
RuntimeError: TTS runtime initialization failed: Engine core initialization failed. See root cause above. Failed core proc(s): {}


Summary: auto


cd /home/ubuntu/veenaModal && NUMBA_DISABLE_JIT=1 python3 - <<'PY'
import asyncio, os, time
from veena3modal.services import tts_runtime
from veena3modal.services.tts_runtime import initialize_runtime

async def main():
    initialize_runtime(
        model_path='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM',
        bicodec_path='/home/ubuntu/veenaModal/models/spark_tts_4speaker',
        device='cuda',
        gpu_memory_utilization=0.15,
        num_engines=1,
    )
    t=time.time()
    try:
        audio_bytes, metrics = await asyncio.wait_for(
            tts_runtime.generate_speech(text='Hello', speaker='lipakshi', output_sample_rate='16khz', max_tokens=128, chunking=False, stream=False),
            timeout=180,
        )
        print('ok', len(audio_bytes), metrics)
    except Exception as e:
        print(type(e).__name__,e)
    print('elapsed',time.time()-t)

asyncio.run(main())
PY


Output:
Chunk ID: 997e02
Wall time: 30.0016 seconds
Process running with session ID 90645
Original token count: 2574
Output:
[2026-02-17 00:44:44] INFO tts_runtime.py:179: Loading SparkTTS model with vLLM...
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
🚀 Initializing Spark TTS Model
📁 Model path: /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
🗣  Speakers: 12 predefined (lipakshi, vardan, reet, Nandini, krishna, anika, adarsh, Nilay, Aarvi, Asha, Bittu, Mira)
🎭 Emotions: 10 tags in [bracket] format

📝 Loading tokenizer from model...
✅ Tokenizer loaded: 165168 tokens

🔍 Validating 12 speakers for Spark TTS...
  lipakshi → <|speaker_0|>: 7 tokens
  vardan → <|speaker_1|>: 7 tokens
  reet → <|speaker_2|>: 7 tokens
  Nandini → <|speaker_3|>: 7 tokens
  krishna → <|speaker_4|>: 7 tokens
  anika → <|speaker_5|>: 7 tokens
  adarsh → <|speaker_6|>: 7 tokens
  Nilay → <|speaker_7|>: 7 tokens
  Aarvi → <|speaker_8|>: 7 tokens
  Asha → <|speaker_9|>: 7 tokens
  Bittu → <|speaker_10|>: 8 tokens
  Mira → <|speaker_11|>: 8 tokens
✅ All 12 speakers validated with mapping

🔍 Validating 10 emotion tags for Spark TTS...
✅ All 10 emotion tags validated

📋 Sample emotion tags:
  [angry]: 1 tokens
  [curious]: 1 tokens
  [excited]: 1 tokens
  [giggle]: 1 tokens
  [laughs harder]: 1 tokens
  ...

🔧 Initializing vLLM engine with optimizations...
  - dtype: bfloat16
  - max_model_len: 4096
  - gpu_memory_utilization: 0.15
  - enable_prefix_caching: True
  - enforce_eager (disable CUDA graphs): False
INFO 02-17 00:44:45 [model.py:541] Resolved architecture: Qwen2ForCausalLM
INFO 02-17 00:44:45 [model.py:1561] Using max model len 4096
INFO 02-17 00:44:45 [scheduler.py:226] Chunked prefill is enabled with max_num_batched_tokens=4096.
INFO 02-17 00:44:45 [vllm.py:624] Asynchronous scheduling is enabled.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(EngineCore_DP0 pid=3716918) INFO 02-17 00:44:46 [core.py:96] Initializing a V1 LLM engine (v0.15.1) with config: model='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', speculative_config=None, tokenizer='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [4096], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 256, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': True}, 'local_cache_dir': None, 'static_all_moe_layers': []}
(EngineCore_DP0 pid=3716918) INFO 02-17 00:44:48 [parallel_state.py:1212] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://216.81.248.184:59239 backend=nccl
(EngineCore_DP0 pid=3716918) INFO 02-17 00:44:48 [parallel_state.py:1423] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
(EngineCore_DP0 pid=3716918) INFO 02-17 00:44:49 [gpu_model_runner.py:4033] Starting to load model /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM...
(EngineCore_DP0 pid=3716918) INFO 02-17 00:44:49 [cuda.py:364] Using FLASH_ATTN attention backend out of potential backends: ('FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION')
(EngineCore_DP0 pid=3716918) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
(EngineCore_DP0 pid=3716918) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
(EngineCore_DP0 pid=3716918)
Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
(EngineCore_DP0 pid=3716918)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:01<00:00,  1.40s/it]
(EngineCore_DP0 pid=3716918)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:01<00:00,  1.40s/it]
(EngineCore_DP0 pid=3716918)
(EngineCore_DP0 pid=3716918) INFO 02-17 00:44:51 [default_loader.py:291] Loading weights took 1.47 seconds
(EngineCore_DP0 pid=3716918) INFO 02-17 00:44:52 [gpu_model_runner.py:4130] Model loading took 0.95 GiB memory and 2.023603 seconds
(EngineCore_DP0 pid=3716918) INFO 02-17 00:44:56 [backends.py:812] Using cache directory: /home/ubuntu/.cache/vllm/torch_compile_cache/f3ff13511c/rank_0_0/backbone for vLLM's torch.compile
(EngineCore_DP0 pid=3716918) INFO 02-17 00:44:56 [backends.py:872] Dynamo bytecode transform time: 4.18 s
(EngineCore_DP0 pid=3716918) INFO 02-17 00:44:59 [backends.py:267] Directly load the compiled graph(s) for compile range (1, 4096) from the cache, took 0.651 s
(EngineCore_DP0 pid=3716918) INFO 02-17 00:44:59 [monitor.py:34] torch.compile takes 4.83 s in total
(EngineCore_DP0 pid=3716918) INFO 02-17 00:45:00 [gpu_worker.py:356] Available KV cache memory: 10.13 GiB
(EngineCore_DP0 pid=3716918) INFO 02-17 00:45:00 [kv_cache_utils.py:1307] GPU KV cache size: 885,184 tokens
(EngineCore_DP0 pid=3716918) INFO 02-17 00:45:00 [kv_cache_utils.py:1312] Maximum concurrency for 4,096 tokens per request: 216.11x
(EngineCore_DP0 pid=3716918)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   0%|          | 0/35 [00:00<?, ?it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  11%|█▏        | 4/35 [00:00<00:00, 32.79it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  23%|██▎       | 8/35 [00:00<00:00, 35.16it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  34%|███▍      | 12/35 [00:00<00:00, 36.06it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  46%|████▌     | 16/35 [00:00<00:00, 36.46it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  57%|█████▋    | 20/35 [00:00<00:00, 36.76it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  69%|██████▊   | 24/35 [00:00<00:00, 36.63it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  80%|████████  | 28/35 [00:00<00:00, 36.41it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  91%|█████████▏| 32/35 [00:00<00:00, 34.87it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 35/35 [00:01<00:00, 34.87it/s]
(EngineCore_DP0 pid=3716918)
Capturing CUDA graphs (decode, FULL):   0%|          | 0/19 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL):  16%|█▌        | 3/19 [00:00<00:00, 28.85it/s]
Capturing CUDA graphs (decode, FULL):  37%|███▋      | 7/19 [00:00<00:00, 32.90it/s]
Capturing CUDA graphs (decode, FULL):  58%|█████▊    | 11/19 [00:00<00:00, 34.26it/s]
Capturing CUDA graphs (decode, FULL):  79%|███████▉  | 15/19 [00:00<00:00, 34.87it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 35.57it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 34.62it/s]
(EngineCore_DP0 pid=3716918) INFO 02-17 00:45:03 [gpu_model_runner.py:5063] Graph capturing finished in 2 secs, took 0.28 GiB
(EngineCore_DP0 pid=3716918) INFO 02-17 00:45:03 [core.py:272] init engine (profile, create kv cache, warmup model) took 10.88 seconds
[2026-02-17 00:45:04] INFO tts_runtime.py:187: Initializing prompt builder...
[2026-02-17 00:45:04] INFO tts_runtime.py:194: Initializing BiCodec decoder...
/home/ubuntu/.local/lib/python3.10/site-packages/torch/nn/utils/weight_norm.py:144: FutureWarning: `torch.nn.utils.weight_norm` is deprecated in favor of `torch.nn.utils.parametrizations.weight_norm`.
  WeightNorm.apply(module, name, dim)


Output:
Chunk ID: da112f
Wall time: 30.0010 seconds
Process running with session ID 90645
Original token count: 102
Output:
[2026-02-17 00:45:10] INFO tts_runtime.py:201: Initializing TTS pipeline...
[2026-02-17 00:45:10] INFO tts_runtime.py:212: Initializing streaming pipeline...
[2026-02-17 00:45:10] INFO tts_runtime.py:262: ✅ TTS runtime initialized in 31815ms
[2026-02-17 00:45:10] INFO tts_runtime.py:263:    Model version: LLM
[2026-02-17 00:45:10] INFO tts_runtime.py:308: Pre-computing global tokens for all speakers...


Output:
Chunk ID: aacd91
Wall time: 30.0007 seconds
Process running with session ID 90645
Original token count: 0
Output:


Output:
Chunk ID: ab97ea
Wall time: 30.0010 seconds
Process running with session ID 90645
Original token count: 0
Output:


Output:
Chunk ID: 0526df
Wall time: 30.0010 seconds
Process running with session ID 90645
Original token count: 0
Output:


Output:
Chunk ID: 4a1fcd
Wall time: 30.0011 seconds
Process running with session ID 90645
Original token count: 0
Output:


Output:
Chunk ID: 605382
Wall time: 30.0018 seconds
Process running with session ID 90645
Original token count: 0
Output:


cd /home/ubuntu/veenaModal && NUMBA_DISABLE_JIT=1 python3 - <<'PY'
import asyncio
import os, time
from vllm import AsyncEngineArgs, SamplingParams
from vllm.v1.engine.async_llm import AsyncLLMEngine

async def main():
    args=AsyncEngineArgs(
        model='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM',
        tokenizer='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM',
        dtype='bfloat16',
        max_model_len=4096,
        gpu_memory_utilization=0.15,
        tensor_parallel_size=1,
        pipeline_parallel_size=1,
        data_parallel_size=1,
        trust_remote_code=True,
        enforce_eager=False,
        enable_prefix_caching=True,
        enable_chunked_prefill=True,
    )
    engine=AsyncLLMEngine.from_engine_args(args)
    sp=SamplingParams(temperature=0.8, top_p=0.95, top_k=50, max_tokens=128, repetition_penalty=1.05)
    prompt='<|speaker_0|> [neutral] Hello'
    t=time.time()
    out = await asyncio.wait_for(engine.generate(prompt, sp, request_id='t'), timeout=180)
    print('generated',type(out))
    # In v1, out should be RequestOutput iterator maybe
    # consume
    async for chunk in out:
        print('chunk',chunk)
    print('elapsed',time.time()-t)
    await engine.shutdown()

asyncio.run(main())
PY


Output:
Chunk ID: cbdba0
Wall time: 6.6849 seconds
Process exited with code 1
Original token count: 59
Output:
Traceback (most recent call last):
  File "<stdin>", line 4, in <module>
ImportError: cannot import name 'AsyncLLMEngine' from 'vllm.v1.engine.async_llm' (/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/async_llm.py)


cd /home/ubuntu/veenaModal && NUMBA_DISABLE_JIT=1 python3 - <<'PY'
import asyncio, time
from vllm import AsyncLLMEngine, AsyncEngineArgs, SamplingParams

async def main():
    engine=AsyncLLMEngine.from_engine_args(
        AsyncEngineArgs(model='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM',
                       tokenizer='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM',
                       dtype='bfloat16',
                       max_model_len=4096,
                       gpu_memory_utilization=0.15,
                       tensor_parallel_size=1,
                       pipeline_parallel_size=1,
                       data_parallel_size=1,
                       trust_remote_code=True,
                       enforce_eager=False,
                       enable_prefix_caching=True,
                       enable_chunked_prefill=True)
    )
    sp=SamplingParams(temperature=0.8, top_p=1.0, top_k=50, max_tokens=128, repetition_penalty=1.05)
    t=time.time()
    try:
      out = await asyncio.wait_for(engine.generate(prompt='<|speaker_0|> [neutral] Hello', sampling_params=sp, request_id='x'), timeout=180)
      print('gen call returned',type(out))
      async for x in out:
        print('chunk',x)
    except Exception as e:
      print(type(e).__name__,e)
    print('elapsed',time.time()-t)
    await engine.aclose()

asyncio.run(main())
PY


Output:
Chunk ID: e1dfdb
Wall time: 29.8211 seconds
Process exited with code 1
Original token count: 2298
Output:
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
INFO 02-17 00:48:31 [model.py:541] Resolved architecture: Qwen2ForCausalLM
INFO 02-17 00:48:31 [model.py:1561] Using max model len 4096
INFO 02-17 00:48:31 [scheduler.py:226] Chunked prefill is enabled with max_num_batched_tokens=2048.
INFO 02-17 00:48:31 [vllm.py:624] Asynchronous scheduling is enabled.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(EngineCore_DP0 pid=3718434) INFO 02-17 00:48:32 [core.py:96] Initializing a V1 LLM engine (v0.15.1) with config: model='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', speculative_config=None, tokenizer='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 256, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': True}, 'local_cache_dir': None, 'static_all_moe_layers': []}
(EngineCore_DP0 pid=3718434) INFO 02-17 00:48:34 [parallel_state.py:1212] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://216.81.248.184:50147 backend=nccl
(EngineCore_DP0 pid=3718434) INFO 02-17 00:48:34 [parallel_state.py:1423] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
(EngineCore_DP0 pid=3718434) INFO 02-17 00:48:35 [gpu_model_runner.py:4033] Starting to load model /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM...
(EngineCore_DP0 pid=3718434) INFO 02-17 00:48:36 [cuda.py:364] Using FLASH_ATTN attention backend out of potential backends: ('FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION')
(EngineCore_DP0 pid=3718434) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
(EngineCore_DP0 pid=3718434) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
(EngineCore_DP0 pid=3718434)
Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
(EngineCore_DP0 pid=3718434)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.11it/s]
(EngineCore_DP0 pid=3718434)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.11it/s]
(EngineCore_DP0 pid=3718434)
(EngineCore_DP0 pid=3718434) INFO 02-17 00:48:37 [default_loader.py:291] Loading weights took 0.97 seconds
(EngineCore_DP0 pid=3718434) INFO 02-17 00:48:37 [gpu_model_runner.py:4130] Model loading took 0.95 GiB memory and 1.483942 seconds
(EngineCore_DP0 pid=3718434) INFO 02-17 00:48:42 [backends.py:812] Using cache directory: /home/ubuntu/.cache/vllm/torch_compile_cache/5591bfed59/rank_0_0/backbone for vLLM's torch.compile
(EngineCore_DP0 pid=3718434) INFO 02-17 00:48:42 [backends.py:872] Dynamo bytecode transform time: 4.15 s
(EngineCore_DP0 pid=3718434) INFO 02-17 00:48:46 [backends.py:302] Cache the graph of compile range (1, 2048) for later use
(EngineCore_DP0 pid=3718434) INFO 02-17 00:48:49 [backends.py:319] Compiling a graph for compile range (1, 2048) takes 5.11 s
(EngineCore_DP0 pid=3718434) INFO 02-17 00:48:49 [monitor.py:34] torch.compile takes 9.26 s in total
(EngineCore_DP0 pid=3718434) INFO 02-17 00:48:50 [gpu_worker.py:356] Available KV cache memory: 10.13 GiB
(EngineCore_DP0 pid=3718434) INFO 02-17 00:48:50 [kv_cache_utils.py:1307] GPU KV cache size: 885,584 tokens
(EngineCore_DP0 pid=3718434) INFO 02-17 00:48:50 [kv_cache_utils.py:1312] Maximum concurrency for 4,096 tokens per request: 216.21x
(EngineCore_DP0 pid=3718434)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   0%|          | 0/35 [00:00<?, ?it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  11%|█▏        | 4/35 [00:00<00:00, 31.83it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  23%|██▎       | 8/35 [00:00<00:00, 34.78it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  34%|███▍      | 12/35 [00:00<00:00, 35.99it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  46%|████▌     | 16/35 [00:00<00:00, 36.32it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  57%|█████▋    | 20/35 [00:00<00:00, 36.45it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  69%|██████▊   | 24/35 [00:00<00:00, 35.87it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  80%|████████  | 28/35 [00:00<00:00, 34.87it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  91%|█████████▏| 32/35 [00:00<00:00, 32.50it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 35/35 [00:01<00:00, 33.51it/s]
(EngineCore_DP0 pid=3718434)
Capturing CUDA graphs (decode, FULL):   0%|          | 0/19 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL):  16%|█▌        | 3/19 [00:00<00:00, 29.12it/s]
Capturing CUDA graphs (decode, FULL):  37%|███▋      | 7/19 [00:00<00:00, 33.37it/s]
Capturing CUDA graphs (decode, FULL):  58%|█████▊    | 11/19 [00:00<00:00, 34.94it/s]
Capturing CUDA graphs (decode, FULL):  79%|███████▉  | 15/19 [00:00<00:00, 35.52it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 36.11it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 35.17it/s]
(EngineCore_DP0 pid=3718434) INFO 02-17 00:48:53 [gpu_model_runner.py:5063] Graph capturing finished in 3 secs, took 0.28 GiB
(EngineCore_DP0 pid=3718434) INFO 02-17 00:48:53 [core.py:272] init engine (profile, create kv cache, warmup model) took 15.52 seconds
TypeError An asyncio.Future, a coroutine or an awaitable is required
elapsed 0.0001506805419921875
Traceback (most recent call last):
  File "<stdin>", line 31, in <module>
  File "/usr/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "<stdin>", line 29, in main
AttributeError: 'AsyncLLM' object has no attribute 'aclose'
ERROR 02-17 00:48:55 [core_client.py:605] Engine core proc EngineCore_DP0 died unexpectedly, shutting down client.


python3 - <<'PY'
from vllm import AsyncLLMEngine
import inspect
print(inspect.signature(AsyncLLMEngine.generate))
print(inspect.signature(AsyncLLMEngine.__init__))
PY


Output:
Chunk ID: 48e975
Wall time: 6.5511 seconds
Process exited with code 0
Original token count: 357
Output:
(self, prompt: vllm.v1.engine.EngineCoreRequest | str | vllm.inputs.data.TextPrompt | vllm.inputs.data.TokensPrompt | vllm.inputs.data.EmbedsPrompt | vllm.inputs.data.ExplicitEncoderDecoderPrompt | collections.abc.AsyncGenerator[vllm.v1.engine.async_llm.StreamingInput, None], sampling_params: vllm.sampling_params.SamplingParams, request_id: str, *, prompt_text: str | None = None, lora_request: vllm.lora.request.LoRARequest | None = None, tokenization_kwargs: dict[str, typing.Any] | None = None, trace_headers: collections.abc.Mapping[str, str] | None = None, priority: int = 0, data_parallel_rank: int | None = None) -> collections.abc.AsyncGenerator[vllm.outputs.RequestOutput, None]
(self, vllm_config: vllm.config.vllm.VllmConfig, executor_class: type[vllm.v1.executor.abstract.Executor], log_stats: bool, usage_context: vllm.usage.usage_lib.UsageContext = <UsageContext.ENGINE_CONTEXT: 'ENGINE_CONTEXT'>, mm_registry: vllm.multimodal.registry.MultiModalRegistry = <vllm.multimodal.registry.MultiModalRegistry object at 0x71af9caaf4f0>, use_cached_outputs: bool = False, log_requests: bool = True, start_engine_loop: bool = True, stat_loggers: list[type['AggregateStatLoggerBase'] | collections.abc.Callable[[vllm.config.vllm.VllmConfig, int], 'StatLoggerBase']] | None = None, aggregate_engine_logging: bool = False, client_addresses: dict[str, str] | None = None, client_count: int = 1, client_index: int = 0) -> None


cd /home/ubuntu/veenaModal && NUMBA_DISABLE_JIT=1 python3 - <<'PY'
import asyncio, time
from vllm import AsyncLLMEngine, AsyncEngineArgs, SamplingParams

async def main():
    engine=AsyncLLMEngine.from_engine_args(AsyncEngineArgs(model='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM',
                       tokenizer='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM',
                       dtype='bfloat16',
                       max_model_len=4096,
                       gpu_memory_utilization=0.15,
                       tensor_parallel_size=1,
                       pipeline_parallel_size=1,
                       data_parallel_size=1,
                       trust_remote_code=True,
                       enforce_eager=False,
                       enable_prefix_caching=True,
                       enable_chunked_prefill=True))
    sp=SamplingParams(temperature=0.8, top_p=1.0, top_k=50, max_tokens=128, repetition_penalty=1.05)
    t=time.time()
    gen=engine.generate(prompt='<|speaker_0|> [neutral] Hello', sampling_params=sp, request_id='x')
    out=[]
    try:
        async for chunk in gen:
            print('chunk',chunk.request_id, chunk.finished, len(chunk.outputs[0].token_ids) if chunk.outputs else None)
            out.append(chunk)
            if len(out)>=3:
                break
    except Exception as e:
        print('err',type(e).__name__,e)
    print('elapsed',time.time()-t,'chunks',len(out))
    await engine.abort('x')
    # not closing maybe

asyncio.run(main())
PY