Input:
*** Begin Patch
*** Update File: /home/ubuntu/veenaModal/scripts/stress_test_runtime_detailed.py
@@
         "config": {
             "levels": args.levels,
             "text": args.text,
             "max_tokens": args.max_tokens,
             "timeout_s": args.timeout,
             "chunking": args.chunking,
             "requests_multiplier": args.requests_multiplier,
+            "gpu_memory": args.gpu_memory,
+            "num_engines": args.num_engines,
+            "max_num_batched_tokens": args.max_num_batched_tokens,
+            "max_num_seqs": args.max_num_seqs,
             "timestamp": int(time.time()),
         },
*** End Patch


Output: {"output":"Success. Updated the following files:\nM /home/ubuntu/veenaModal/scripts/stress_test_runtime_detailed.py\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


# Collaboration Mode: Default

You are now in Default mode. Any previous instructions for other modes (e.g. Plan mode) are no longer active.

Your active mode changes only when new developer instructions with a different `<collaboration_mode>...</collaboration_mode>` change it; user requests or tool descriptions do not change mode by themselves. Known mode names are Default and Plan.

## request_user_input availability

The `request_user_input` tool is unavailable in Default mode. If you call it while in Default mode, it will return an error.

If a decision is necessary and cannot be discovered from local context, ask the user directly. However, in Default mode you should strongly prefer executing the user's request rather than stopping to ask questions.

Summary: auto

## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) When `SKILL.md` references relative paths (e.g., `scripts/foo.py`), resolve them relative to the skill directory listed above first, and only consider other paths if needed.
  3) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  4) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  5) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.


**Checking type imports and engine references**


I’ve implemented a substantive scheduler-path change: multi-engine dispatch is now load-aware (least-inflight) instead of strict round-robin, and I added runtime/local-server knobs for vLLM scheduler limits (`max_num_batched_tokens`, `max_num_seqs`) so we can sweep for the best high-concurrency setting. I’m validating build and then running A/B benchmarks across engine counts.


Output:
Chunk ID: 717ef0
Wall time: 0.0511 seconds
Process exited with code 0
Original token count: 0
Output:


Output:
Chunk ID: 740dd4
Wall time: 0.0511 seconds
Process exited with code 0
Original token count: 0
Output:


Summary: auto


**Running benchmark baseline tests**


python3 /home/ubuntu/veenaModal/scripts/stress_test_runtime_detailed.py --levels 32,64,96 --text short --max-tokens 128 --timeout 180 --requests-multiplier 2 --min-requests 8 --gpu-memory 0.25 --num-engines 1 --output /tmp/runtime_tuned_e1_g25.json


Output:
Chunk ID: 0a314c
Wall time: 1.0011 seconds
Process running with session ID 61231
Original token count: 0
Output:


**Planning periodic update polling**


Baseline run (`1 engine`, `gpu_memory=0.25`) is running now; once it finishes I’ll execute `2-engine` and `3-engine` sweeps with the same workload so we can pick the best throughput/latency point empirically.


Output:
Chunk ID: cd2317
Wall time: 5.0006 seconds
Process running with session ID 61231
Original token count: 1714
Output:
[2026-02-18 00:04:51] INFO tts_runtime.py:167: Loading SparkTTS model with vLLM...
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
🚀 Initializing Spark TTS Model
📁 Model path: /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
🗣  Speakers: 12 predefined (lipakshi, vardan, reet, Nandini, krishna, anika, adarsh, Nilay, Aarvi, Asha, Bittu, Mira)
🎭 Emotions: 10 tags in [bracket] format

📝 Loading tokenizer from model...
✅ Tokenizer loaded: 165168 tokens

🔍 Validating 12 speakers for Spark TTS...
  lipakshi → <|speaker_0|>: 7 tokens
  vardan → <|speaker_1|>: 7 tokens
  reet → <|speaker_2|>: 7 tokens
  Nandini → <|speaker_3|>: 7 tokens
  krishna → <|speaker_4|>: 7 tokens
  anika → <|speaker_5|>: 7 tokens
  adarsh → <|speaker_6|>: 7 tokens
  Nilay → <|speaker_7|>: 7 tokens
  Aarvi → <|speaker_8|>: 7 tokens
  Asha → <|speaker_9|>: 7 tokens
  Bittu → <|speaker_10|>: 8 tokens
  Mira → <|speaker_11|>: 8 tokens
✅ All 12 speakers validated with mapping

🔍 Validating 10 emotion tags for Spark TTS...
✅ All 10 emotion tags validated

📋 Sample emotion tags:
  [angry]: 1 tokens
  [curious]: 1 tokens
  [excited]: 1 tokens
  [giggle]: 1 tokens
  [laughs harder]: 1 tokens
  ...

🔧 Initializing vLLM engine with optimizations...
  - dtype: bfloat16
  - max_model_len: 4096
  - gpu_memory_utilization: 0.25
  - enable_prefix_caching: True
  - enforce_eager (disable CUDA graphs): False
INFO 02-18 00:04:52 [model.py:541] Resolved architecture: Qwen2ForCausalLM
INFO 02-18 00:04:52 [model.py:1561] Using max model len 4096
INFO 02-18 00:04:52 [scheduler.py:226] Chunked prefill is enabled with max_num_batched_tokens=4096.
INFO 02-18 00:04:52 [vllm.py:624] Asynchronous scheduling is enabled.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(EngineCore_DP0 pid=4090835) INFO 02-18 00:04:53 [core.py:96] Initializing a V1 LLM engine (v0.15.1) with config: model='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', speculative_config=None, tokenizer='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [4096], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 256, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': True}, 'local_cache_dir': None, 'static_all_moe_layers': []}
(EngineCore_DP0 pid=4090835) INFO 02-18 00:04:56 [parallel_state.py:1212] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://216.81.248.184:49787 backend=nccl
(EngineCore_DP0 pid=4090835) INFO 02-18 00:04:56 [parallel_state.py:1423] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
(EngineCore_DP0 pid=4090835) INFO 02-18 00:04:56 [gpu_model_runner.py:4033] Starting to load model /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM...
(EngineCore_DP0 pid=4090835) INFO 02-18 00:04:57 [cuda.py:364] Using FLASH_ATTN attention backend out of potential backends: ('FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION')
(EngineCore_DP0 pid=4090835) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
(EngineCore_DP0 pid=4090835) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
(EngineCore_DP0 pid=4090835)
Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
(EngineCore_DP0 pid=4090835)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  2.00it/s]
(EngineCore_DP0 pid=4090835)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  2.00it/s]
(EngineCore_DP0 pid=4090835)
(EngineCore_DP0 pid=4090835) INFO 02-18 00:04:58 [default_loader.py:291] Loading weights took 0.57 seconds
(EngineCore_DP0 pid=4090835) INFO 02-18 00:04:58 [gpu_model_runner.py:4130] Model loading took 0.95 GiB memory and 1.125408 seconds


Output:
Chunk ID: 7bf796
Wall time: 5.0002 seconds
Process running with session ID 61231
Original token count: 696
Output:
(EngineCore_DP0 pid=4090835) INFO 02-18 00:05:03 [backends.py:812] Using cache directory: /home/ubuntu/.cache/vllm/torch_compile_cache/f3ff13511c/rank_0_0/backbone for vLLM's torch.compile
(EngineCore_DP0 pid=4090835) INFO 02-18 00:05:03 [backends.py:872] Dynamo bytecode transform time: 4.27 s
(EngineCore_DP0 pid=4090835) INFO 02-18 00:05:06 [backends.py:267] Directly load the compiled graph(s) for compile range (1, 4096) from the cache, took 0.702 s
(EngineCore_DP0 pid=4090835) INFO 02-18 00:05:06 [monitor.py:34] torch.compile takes 4.97 s in total
(EngineCore_DP0 pid=4090835) INFO 02-18 00:05:07 [gpu_worker.py:356] Available KV cache memory: 18.06 GiB
(EngineCore_DP0 pid=4090835) INFO 02-18 00:05:07 [kv_cache_utils.py:1307] GPU KV cache size: 1,577,712 tokens
(EngineCore_DP0 pid=4090835) INFO 02-18 00:05:07 [kv_cache_utils.py:1312] Maximum concurrency for 4,096 tokens per request: 385.18x
(EngineCore_DP0 pid=4090835)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   0%|          | 0/35 [00:00<?, ?it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  11%|█▏        | 4/35 [00:00<00:00, 32.41it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  23%|██▎       | 8/35 [00:00<00:00, 35.18it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  34%|███▍      | 12/35 [00:00<00:00, 36.24it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  46%|████▌     | 16/35 [00:00<00:00, 36.53it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  57%|█████▋    | 20/35 [00:00<00:00, 36.48it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  69%|██████▊   | 24/35 [00:00<00:00, 36.38it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  80%|████████  | 28/35 [00:00<00:00, 36.11it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  91%|█████████▏| 32/35 [00:00<00:00, 34.53it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 35/35 [00:01<00:00, 34.62it/s]
(EngineCore_DP0 pid=4090835)
Capturing CUDA graphs (decode, FULL):   0%|          | 0/19 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL):  16%|█▌        | 3/19 [00:00<00:00, 28.84it/s]
Capturing CUDA graphs (decode, FULL):  37%|███▋      | 7/19 [00:00<00:00, 33.05it/s]
Capturing CUDA graphs (decode, FULL):  58%|█████▊    | 11/19 [00:00<00:00, 34.42it/s]
Capturing CUDA graphs (decode, FULL):  79%|███████▉  | 15/19 [00:00<00:00, 35.23it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 35.84it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 34.86it/s]


Output:
Chunk ID: 45e56b
Wall time: 5.0011 seconds
Process running with session ID 61231
Original token count: 291
Output:
(EngineCore_DP0 pid=4090835) INFO 02-18 00:05:10 [gpu_model_runner.py:5063] Graph capturing finished in 2 secs, took 0.28 GiB
(EngineCore_DP0 pid=4090835) INFO 02-18 00:05:10 [core.py:272] init engine (profile, create kv cache, warmup model) took 11.14 seconds
[2026-02-18 00:05:11] INFO tts_runtime.py:176: Initializing prompt builder...
[2026-02-18 00:05:11] INFO tts_runtime.py:183: Initializing BiCodec decoder...
/home/ubuntu/.local/lib/python3.10/site-packages/torch/nn/utils/weight_norm.py:144: FutureWarning: `torch.nn.utils.weight_norm` is deprecated in favor of `torch.nn.utils.parametrizations.weight_norm`.
  WeightNorm.apply(module, name, dim)
[2026-02-18 00:05:17] INFO tts_runtime.py:190: Initializing TTS pipeline...
[2026-02-18 00:05:17] INFO tts_runtime.py:201: Initializing streaming pipeline...
[2026-02-18 00:05:18] INFO long_text_processor.py:112: LongTextProcessor initialized
[2026-02-18 00:05:18] INFO tts_runtime.py:258: ✅ TTS runtime initialized in 31525ms
[2026-02-18 00:05:18] INFO tts_runtime.py:259:    Model version: LLM
[2026-02-18 00:05:18] INFO tts_runtime.py:270: Speaker globals cache startup precompute skipped (disabled)


Output:
Chunk ID: 8d5898
Wall time: 5.0014 seconds
Process running with session ID 61231
Original token count: 0
Output:


Output:
Chunk ID: 54ad83
Wall time: 0.0517 seconds
Process exited with code 0
Original token count: 1124
Output:
✅ vLLM engine initialized successfully!

🎉 Spark TTS Model ready for inference!

🎵 Loading BiCodec audio tokenizer from /home/ubuntu/veenaModal/models/spark_tts_4speaker...
   Device: cuda
Missing tensor: mel_transformer.spectrogram.window
Missing tensor: mel_transformer.mel_scale.fb
✅ BiCodec decoder initialized (sample rate: 16000Hz)
🚀 SparkTTSPipeline initialized
🌊 Veena3SlidingWindowPipeline initialized (sliding window: 28 tokens)

=== concurrency=32 requests=64 ===
success=64/64 (100%)
wall=2.01s throughput=31.76 req/s
latency_ms avg=982 p50=967 p95=1072 max=1072
generation_ms: avg=818.55 p50=809.00 p95=994.00 max=1043.00
timeline_llm_first_batch_ms: avg=32.85 p50=27.44 p95=34.62 max=399.17
timeline_llm_done_ms: avg=807.51 p50=797.69 p95=982.29 max=1030.95
timeline_parse_done_ms: avg=807.57 p50=797.76 p95=982.35 max=1031.01
timeline_bicodec_done_ms: avg=819.07 p50=809.34 p95=994.77 max=1043.82
timeline_total_ms: avg=819.09 p50=809.36 p95=994.78 max=1043.85
llm_batch_wall_ms_total: avg=807.35 p50=797.55 p95=982.16 max=1030.81
llm_decode_wall_ms_total: avg=774.64 p50=775.12 p95=952.29 max=1002.66
llm_parse_ms: avg=0.06 p50=0.06 p95=0.07 max=0.08
llm_time_per_token_ms: avg=6.31 p50=6.23 p95=7.67 max=8.05
bicodec_decode_wall_ms: avg=11.44 p50=12.27 p95=12.76 max=14.41
bicodec_decode_gpu_ms: avg=11.37 p50=12.20 p95=12.67 max=14.34
latency_over_timeline_ms: avg=163.18 p50=160.87 p95=332.82 max=371.76
latency_over_generation_ms: avg=163.72 p50=161.24 p95=333.68 max=372.67
timeline_share: llm=98.6% bicodec=1.4% other=0.0%
timeline_sample_ms: first_batch=35.6 llm_done=670.4 parse_done=670.4 bicodec_done=684.9 total=684.9
gpu: util_avg=62% util_max=67% mem_avg=23601MB mem_max=23601MB

=== concurrency=64 requests=128 ===
success=128/128 (100%)
wall=3.08s throughput=41.53 req/s
latency_ms avg=1506 p50=1424 p95=1650 max=1650
generation_ms: avg=1147.16 p50=1133.00 p95=1531.00 max=1620.00
timeline_llm_first_batch_ms: avg=51.84 p50=37.00 p95=56.19 max=872.89
timeline_llm_done_ms: avg=1135.73 p50=1123.31 p95=1517.06 max=1607.44
timeline_parse_done_ms: avg=1135.79 p50=1123.37 p95=1517.14 max=1607.50
timeline_bicodec_done_ms: avg=1147.58 p50=1133.33 p95=1531.49 max=1620.05
timeline_total_ms: avg=1147.59 p50=1133.34 p95=1531.51 max=1620.07
llm_batch_wall_ms_total: avg=1135.58 p50=1123.16 p95=1516.91 max=1607.29
llm_decode_wall_ms_total: avg=1083.87 p50=1085.71 p95=1491.36 max=1578.49
llm_parse_ms: avg=0.06 p50=0.06 p95=0.07 max=0.07
llm_time_per_token_ms: avg=8.87 p50=8.77 p95=11.85 max=12.56
bicodec_decode_wall_ms: avg=11.74 p50=12.19 p95=14.40 max=17.64
bicodec_decode_gpu_ms: avg=11.66 p50=12.05 p95=14.30 max=17.57
latency_over_timeline_ms: avg=358.66 p50=349.06 p95=726.49 max=820.51
latency_over_generation_ms: avg=359.09 p50=349.13 p95=727.14 max=821.14
timeline_share: llm=99.0% bicodec=1.0% other=0.0%
timeline_sample_ms: first_batch=56.6 llm_done=815.3 parse_done=815.3 bicodec_done=825.4 total=825.4
gpu: util_avg=75% util_max=100% mem_avg=23601MB mem_max=23601MB

=== concurrency=96 requests=192 ===
success=192/192 (100%)
wall=4.05s throughput=47.47 req/s
latency_ms avg=1910 p50=1898 p95=2138 max=2138
generation_ms: avg=1473.40 p50=1462.00 p95=1995.00 max=2093.00
timeline_llm_first_batch_ms: avg=63.96 p50=49.93 p95=72.52 max=1223.84
timeline_llm_done_ms: avg=1462.60 p50=1451.94 p95=1985.88 max=2082.92
timeline_parse_done_ms: avg=1462.66 p50=1452.00 p95=1985.95 max=2082.98
timeline_bicodec_done_ms: avg=1473.86 p50=1462.00 p95=1995.85 max=2092.96
timeline_total_ms: avg=1473.88 p50=1462.02 p95=1995.86 max=2092.98
llm_batch_wall_ms_total: avg=1462.45 p50=1451.80 p95=1985.73 max=2082.78
llm_decode_wall_ms_total: avg=1413.35 p50=1404.64 p95=1942.50 max=2036.26
llm_parse_ms: avg=0.06 p50=0.06 p95=0.07 max=0.07
llm_time_per_token_ms: avg=11.43 p50=11.34 p95=15.51 max=16.27
bicodec_decode_wall_ms: avg=11.16 p50=10.08 p95=13.99 max=16.04
bicodec_decode_gpu_ms: avg=11.08 p50=10.02 p95=13.91 max=15.97
latency_over_timeline_ms: avg=436.33 p50=386.91 p95=1013.03 max=1142.25
latency_over_generation_ms: avg=436.81 p50=387.73 p95=1013.34 max=1142.33
timeline_share: llm=99.2% bicodec=0.8% other=0.0%
timeline_sample_ms: first_batch=64.7 llm_done=1180.1 parse_done=1180.1 bicodec_done=1190.2 total=1190.2
gpu: util_avg=59% util_max=100% mem_avg=23603MB mem_max=23603MB

results saved: /tmp/runtime_tuned_e1_g25.json
ERROR 02-18 00:05:33 [core_client.py:605] Engine core proc EngineCore_DP0 died unexpectedly, shutting down client.


**Planning GPU memory configurations**


python3 /home/ubuntu/veenaModal/scripts/stress_test_runtime_detailed.py --levels 32,64,96 --text short --max-tokens 128 --timeout 180 --requests-multiplier 2 --min-requests 8 --gpu-memory 0.50 --num-engines 2 --output /tmp/runtime_tuned_e2_g50.json


Output:
Chunk ID: a095db
Wall time: 0.0502 seconds
Process exited with code 0
Original token count: 0
Output:


Output:
Chunk ID: a1d2f1
Wall time: 1.0011 seconds
Process running with session ID 18284
Original token count: 0
Output:


Now running the primary optimization candidate: `2 engines` with `0.50` total GPU memory (0.25 each). This should tell us if multi-engine dispatch removes scheduler bottlenecks and improves high-concurrency throughput.


Output:
Chunk ID: 083023
Wall time: 5.0015 seconds
Process running with session ID 18284
Original token count: 1636
Output:
[2026-02-18 00:05:53] INFO tts_runtime.py:158: Loading 2 vLLM engines (0.25 GPU mem each)...
[2026-02-18 00:05:53] INFO multi_engine.py:157: Creating multi-engine model: 2 engines x 25% GPU each
[2026-02-18 00:05:53] INFO multi_engine.py:161:   Initializing engine 1/2...
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
🚀 Initializing Spark TTS Model
📁 Model path: /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
🗣  Speakers: 12 predefined (lipakshi, vardan, reet, Nandini, krishna, anika, adarsh, Nilay, Aarvi, Asha, Bittu, Mira)
🎭 Emotions: 10 tags in [bracket] format

📝 Loading tokenizer from model...
✅ Tokenizer loaded: 165168 tokens

🔍 Validating 12 speakers for Spark TTS...
  lipakshi → <|speaker_0|>: 7 tokens
  vardan → <|speaker_1|>: 7 tokens
  reet → <|speaker_2|>: 7 tokens
  Nandini → <|speaker_3|>: 7 tokens
  krishna → <|speaker_4|>: 7 tokens
  anika → <|speaker_5|>: 7 tokens
  adarsh → <|speaker_6|>: 7 tokens
  Nilay → <|speaker_7|>: 7 tokens
  Aarvi → <|speaker_8|>: 7 tokens
  Asha → <|speaker_9|>: 7 tokens
  Bittu → <|speaker_10|>: 8 tokens
  Mira → <|speaker_11|>: 8 tokens
✅ All 12 speakers validated with mapping

🔍 Validating 10 emotion tags for Spark TTS...
✅ All 10 emotion tags validated

📋 Sample emotion tags:
  [angry]: 1 tokens
  [curious]: 1 tokens
  [excited]: 1 tokens
  [giggle]: 1 tokens
  [laughs harder]: 1 tokens
  ...

🔧 Initializing vLLM engine with optimizations...
  - dtype: bfloat16
  - max_model_len: 4096
  - gpu_memory_utilization: 0.25
  - enable_prefix_caching: True
  - enforce_eager (disable CUDA graphs): False
INFO 02-18 00:05:54 [model.py:541] Resolved architecture: Qwen2ForCausalLM
INFO 02-18 00:05:54 [model.py:1561] Using max model len 4096
INFO 02-18 00:05:54 [scheduler.py:226] Chunked prefill is enabled with max_num_batched_tokens=4096.
INFO 02-18 00:05:54 [vllm.py:624] Asynchronous scheduling is enabled.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(EngineCore_DP0 pid=4091244) INFO 02-18 00:05:55 [core.py:96] Initializing a V1 LLM engine (v0.15.1) with config: model='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', speculative_config=None, tokenizer='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [4096], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 256, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': True}, 'local_cache_dir': None, 'static_all_moe_layers': []}
(EngineCore_DP0 pid=4091244) INFO 02-18 00:05:57 [parallel_state.py:1212] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://216.81.248.184:50415 backend=nccl
(EngineCore_DP0 pid=4091244) INFO 02-18 00:05:57 [parallel_state.py:1423] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
(EngineCore_DP0 pid=4091244) INFO 02-18 00:05:58 [gpu_model_runner.py:4033] Starting to load model /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM...
(EngineCore_DP0 pid=4091244) INFO 02-18 00:05:59 [cuda.py:364] Using FLASH_ATTN attention backend out of potential backends: ('FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION')
(EngineCore_DP0 pid=4091244) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
(EngineCore_DP0 pid=4091244) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
(EngineCore_DP0 pid=4091244)
Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]


Output:
Chunk ID: 907949
Wall time: 5.0008 seconds
Process running with session ID 18284
Original token count: 494
Output:
(EngineCore_DP0 pid=4091244)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.28it/s]
(EngineCore_DP0 pid=4091244)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.27it/s]
(EngineCore_DP0 pid=4091244)
(EngineCore_DP0 pid=4091244) INFO 02-18 00:05:59 [default_loader.py:291] Loading weights took 0.86 seconds
(EngineCore_DP0 pid=4091244) INFO 02-18 00:06:00 [gpu_model_runner.py:4130] Model loading took 0.95 GiB memory and 1.407917 seconds
(EngineCore_DP0 pid=4091244) INFO 02-18 00:06:05 [backends.py:812] Using cache directory: /home/ubuntu/.cache/vllm/torch_compile_cache/f3ff13511c/rank_0_0/backbone for vLLM's torch.compile
(EngineCore_DP0 pid=4091244) INFO 02-18 00:06:05 [backends.py:872] Dynamo bytecode transform time: 4.19 s
(EngineCore_DP0 pid=4091244) INFO 02-18 00:06:08 [backends.py:267] Directly load the compiled graph(s) for compile range (1, 4096) from the cache, took 0.665 s
(EngineCore_DP0 pid=4091244) INFO 02-18 00:06:08 [monitor.py:34] torch.compile takes 4.86 s in total
(EngineCore_DP0 pid=4091244) INFO 02-18 00:06:09 [gpu_worker.py:356] Available KV cache memory: 18.06 GiB
(EngineCore_DP0 pid=4091244) INFO 02-18 00:06:09 [kv_cache_utils.py:1307] GPU KV cache size: 1,577,712 tokens
(EngineCore_DP0 pid=4091244) INFO 02-18 00:06:09 [kv_cache_utils.py:1312] Maximum concurrency for 4,096 tokens per request: 385.18x
(EngineCore_DP0 pid=4091244)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   0%|          | 0/35 [00:00<?, ?it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  11%|█▏        | 4/35 [00:00<00:00, 32.93it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  23%|██▎       | 8/35 [00:00<00:00, 35.11it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  34%|███▍      | 12/35 [00:00<00:00, 36.39it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  46%|████▌     | 16/35 [00:00<00:00, 36.98it/s]


Output:
Chunk ID: 93e764
Wall time: 5.0013 seconds
Process running with session ID 18284
Original token count: 2125
Output:

Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  57%|█████▋    | 20/35 [00:00<00:00, 37.26it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  69%|██████▊   | 24/35 [00:00<00:00, 37.00it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  80%|████████  | 28/35 [00:00<00:00, 36.84it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  91%|█████████▏| 32/35 [00:00<00:00, 35.40it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 35/35 [00:00<00:00, 35.34it/s]
(EngineCore_DP0 pid=4091244)
Capturing CUDA graphs (decode, FULL):   0%|          | 0/19 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL):  16%|█▌        | 3/19 [00:00<00:00, 29.30it/s]
Capturing CUDA graphs (decode, FULL):  37%|███▋      | 7/19 [00:00<00:00, 33.39it/s]
Capturing CUDA graphs (decode, FULL):  58%|█████▊    | 11/19 [00:00<00:00, 34.88it/s]
Capturing CUDA graphs (decode, FULL):  79%|███████▉  | 15/19 [00:00<00:00, 35.35it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 36.00it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 35.10it/s]
(EngineCore_DP0 pid=4091244) INFO 02-18 00:06:11 [gpu_model_runner.py:5063] Graph capturing finished in 2 secs, took 0.28 GiB
(EngineCore_DP0 pid=4091244) INFO 02-18 00:06:11 [core.py:272] init engine (profile, create kv cache, warmup model) took 10.94 seconds
[2026-02-18 00:06:13] INFO multi_engine.py:169:   Engine 1 ready
[2026-02-18 00:06:13] INFO multi_engine.py:161:   Initializing engine 2/2...
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
✅ vLLM engine initialized successfully!

🎉 Spark TTS Model ready for inference!

🚀 Initializing Spark TTS Model
📁 Model path: /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
🗣  Speakers: 12 predefined (lipakshi, vardan, reet, Nandini, krishna, anika, adarsh, Nilay, Aarvi, Asha, Bittu, Mira)
🎭 Emotions: 10 tags in [bracket] format

📝 Loading tokenizer from model...
✅ Tokenizer loaded: 165168 tokens

🔍 Validating 12 speakers for Spark TTS...
  lipakshi → <|speaker_0|>: 7 tokens
  vardan → <|speaker_1|>: 7 tokens
  reet → <|speaker_2|>: 7 tokens
  Nandini → <|speaker_3|>: 7 tokens
  krishna → <|speaker_4|>: 7 tokens
  anika → <|speaker_5|>: 7 tokens
  adarsh → <|speaker_6|>: 7 tokens
  Nilay → <|speaker_7|>: 7 tokens
  Aarvi → <|speaker_8|>: 7 tokens
  Asha → <|speaker_9|>: 7 tokens
  Bittu → <|speaker_10|>: 8 tokens
  Mira → <|speaker_11|>: 8 tokens
✅ All 12 speakers validated with mapping

🔍 Validating 10 emotion tags for Spark TTS...
✅ All 10 emotion tags validated

📋 Sample emotion tags:
  [angry]: 1 tokens
  [curious]: 1 tokens
  [excited]: 1 tokens
  [giggle]: 1 tokens
  [laughs harder]: 1 tokens
  ...

🔧 Initializing vLLM engine with optimizations...
  - dtype: bfloat16
  - max_model_len: 4096
  - gpu_memory_utilization: 0.25
  - enable_prefix_caching: True
  - enforce_eager (disable CUDA graphs): False
INFO 02-18 00:06:14 [model.py:541] Resolved architecture: Qwen2ForCausalLM
INFO 02-18 00:06:14 [model.py:1561] Using max model len 4096
INFO 02-18 00:06:14 [scheduler.py:226] Chunked prefill is enabled with max_num_batched_tokens=4096.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(EngineCore_DP0 pid=4091359) INFO 02-18 00:06:14 [core.py:96] Initializing a V1 LLM engine (v0.15.1) with config: model='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', speculative_config=None, tokenizer='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [4096], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 256, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': True}, 'local_cache_dir': None, 'static_all_moe_layers': []}
(EngineCore_DP0 pid=4091359) INFO 02-18 00:06:16 [parallel_state.py:1212] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://216.81.248.184:56707 backend=nccl
(EngineCore_DP0 pid=4091359) INFO 02-18 00:06:16 [parallel_state.py:1423] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
(EngineCore_DP0 pid=4091359) INFO 02-18 00:06:17 [gpu_model_runner.py:4033] Starting to load model /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM...
(EngineCore_DP0 pid=4091359) INFO 02-18 00:06:17 [cuda.py:364] Using FLASH_ATTN attention backend out of potential backends: ('FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION')
(EngineCore_DP0 pid=4091359) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
(EngineCore_DP0 pid=4091359) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
(EngineCore_DP0 pid=4091359)
Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
(EngineCore_DP0 pid=4091359)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:01<00:00,  1.34s/it]
(EngineCore_DP0 pid=4091359)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:01<00:00,  1.34s/it]
(EngineCore_DP0 pid=4091359)
(EngineCore_DP0 pid=4091359) INFO 02-18 00:06:19 [default_loader.py:291] Loading weights took 1.42 seconds
(EngineCore_DP0 pid=4091359) INFO 02-18 00:06:20 [gpu_model_runner.py:4130] Model loading took 0.95 GiB memory and 1.950680 seconds


Output:
Chunk ID: 22ed1e
Wall time: 5.0008 seconds
Process running with session ID 18284
Original token count: 903
Output:
(EngineCore_DP0 pid=4091359) INFO 02-18 00:06:24 [backends.py:812] Using cache directory: /home/ubuntu/.cache/vllm/torch_compile_cache/f3ff13511c/rank_0_0/backbone for vLLM's torch.compile
(EngineCore_DP0 pid=4091359) INFO 02-18 00:06:24 [backends.py:872] Dynamo bytecode transform time: 4.22 s
(EngineCore_DP0 pid=4091359) INFO 02-18 00:06:27 [backends.py:267] Directly load the compiled graph(s) for compile range (1, 4096) from the cache, took 0.635 s
(EngineCore_DP0 pid=4091359) INFO 02-18 00:06:27 [monitor.py:34] torch.compile takes 4.86 s in total
(EngineCore_DP0 pid=4091359) INFO 02-18 00:06:28 [gpu_worker.py:356] Available KV cache memory: 18.06 GiB
(EngineCore_DP0 pid=4091359) INFO 02-18 00:06:28 [kv_cache_utils.py:1307] GPU KV cache size: 1,577,712 tokens
(EngineCore_DP0 pid=4091359) INFO 02-18 00:06:28 [kv_cache_utils.py:1312] Maximum concurrency for 4,096 tokens per request: 385.18x
(EngineCore_DP0 pid=4091359)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   0%|          | 0/35 [00:00<?, ?it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  11%|█▏        | 4/35 [00:00<00:00, 33.64it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  23%|██▎       | 8/35 [00:00<00:00, 35.70it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  34%|███▍      | 12/35 [00:00<00:00, 36.42it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  46%|████▌     | 16/35 [00:00<00:00, 36.67it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  57%|█████▋    | 20/35 [00:00<00:00, 36.94it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  69%|██████▊   | 24/35 [00:00<00:00, 37.07it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  80%|████████  | 28/35 [00:00<00:00, 37.11it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  91%|█████████▏| 32/35 [00:00<00:00, 35.41it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 35/35 [00:00<00:00, 35.36it/s]
(EngineCore_DP0 pid=4091359)
Capturing CUDA graphs (decode, FULL):   0%|          | 0/19 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL):  16%|█▌        | 3/19 [00:00<00:00, 29.88it/s]
Capturing CUDA graphs (decode, FULL):  37%|███▋      | 7/19 [00:00<00:00, 33.04it/s]
Capturing CUDA graphs (decode, FULL):  58%|█████▊    | 11/19 [00:00<00:00, 34.85it/s]
Capturing CUDA graphs (decode, FULL):  79%|███████▉  | 15/19 [00:00<00:00, 35.58it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 36.10it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 35.18it/s]
(EngineCore_DP0 pid=4091359) INFO 02-18 00:06:30 [gpu_model_runner.py:5063] Graph capturing finished in 2 secs, took 0.28 GiB
(EngineCore_DP0 pid=4091359) INFO 02-18 00:06:30 [core.py:272] init engine (profile, create kv cache, warmup model) took 10.76 seconds
[2026-02-18 00:06:32] INFO multi_engine.py:169:   Engine 2 ready
[2026-02-18 00:06:32] INFO multi_engine.py:172: Multi-engine model ready: 2 engines, round-robin dispatch
[2026-02-18 00:06:32] INFO tts_runtime.py:176: Initializing prompt builder...
[2026-02-18 00:06:32] INFO tts_runtime.py:183: Initializing BiCodec decoder...
/home/ubuntu/.local/lib/python3.10/site-packages/torch/nn/utils/weight_norm.py:144: FutureWarning: `torch.nn.utils.weight_norm` is deprecated in favor of `torch.nn.utils.parametrizations.weight_norm`.
  WeightNorm.apply(module, name, dim)


**Waiting benchmark completion**


Output:
Chunk ID: 6ff9d6
Wall time: 5.0005 seconds
Process running with session ID 18284
Original token count: 129
Output:
[2026-02-18 00:06:38] INFO tts_runtime.py:190: Initializing TTS pipeline...
[2026-02-18 00:06:38] INFO tts_runtime.py:201: Initializing streaming pipeline...
[2026-02-18 00:06:38] INFO long_text_processor.py:112: LongTextProcessor initialized
[2026-02-18 00:06:38] INFO tts_runtime.py:258: ✅ TTS runtime initialized in 50351ms
[2026-02-18 00:06:38] INFO tts_runtime.py:259:    Model version: LLM
[2026-02-18 00:06:38] INFO tts_runtime.py:268: Speaker globals cache startup precompute skipped (multi-engine mode)


Output:
Chunk ID: 14b876
Wall time: 5.0007 seconds
Process running with session ID 18284
Original token count: 1158
Output:
✅ vLLM engine initialized successfully!

🎉 Spark TTS Model ready for inference!

🎵 Loading BiCodec audio tokenizer from /home/ubuntu/veenaModal/models/spark_tts_4speaker...
   Device: cuda
Missing tensor: mel_transformer.spectrogram.window
Missing tensor: mel_transformer.mel_scale.fb
✅ BiCodec decoder initialized (sample rate: 16000Hz)
🚀 SparkTTSPipeline initialized
🌊 Veena3SlidingWindowPipeline initialized (sliding window: 28 tokens)

=== concurrency=32 requests=64 ===
success=64/64 (100%)
wall=2.38s throughput=26.93 req/s
latency_ms avg=1134 p50=1095 p95=1323 max=1323
generation_ms: avg=1037.75 p50=1006.00 p95=1255.00 max=1305.00
timeline_llm_first_batch_ms: avg=82.97 p50=30.78 p95=224.09 max=420.54
timeline_llm_done_ms: avg=1025.69 p50=994.93 p95=1241.48 max=1291.96
timeline_parse_done_ms: avg=1025.75 p50=994.99 p95=1241.54 max=1292.02
timeline_bicodec_done_ms: avg=1038.20 p50=1006.83 p95=1255.06 max=1305.49
timeline_total_ms: avg=1038.22 p50=1006.84 p95=1255.08 max=1305.51
llm_batch_wall_ms_total: avg=1025.50 p50=994.78 p95=1241.32 max=1291.82
llm_decode_wall_ms_total: avg=942.70 p50=937.40 p95=1197.10 max=1251.12
llm_parse_ms: avg=0.06 p50=0.06 p95=0.07 max=0.11
llm_time_per_token_ms: avg=8.01 p50=7.77 p95=9.70 max=10.09
bicodec_decode_wall_ms: avg=12.40 p50=13.06 p95=14.41 max=15.71
bicodec_decode_gpu_ms: avg=12.32 p50=13.00 p95=14.17 max=15.44
latency_over_timeline_ms: avg=96.27 p50=92.68 p95=179.64 max=199.13
latency_over_generation_ms: avg=96.74 p50=93.07 p95=180.31 max=199.18
timeline_share: llm=98.8% bicodec=1.2% other=0.0%
timeline_sample_ms: first_batch=226.0 llm_done=888.2 parse_done=888.3 bicodec_done=902.8 total=902.8
gpu: util_avg=98% util_max=100% mem_avg=44793MB mem_max=44793MB

=== concurrency=64 requests=128 ===
success=128/128 (100%)
wall=3.64s throughput=35.19 req/s
latency_ms avg=1689 p50=1599 p95=2031 max=2031
generation_ms: avg=1469.31 p50=1462.00 p95=1867.00 max=1986.00
timeline_llm_first_batch_ms: avg=134.62 p50=34.44 p95=374.78 max=574.26
timeline_llm_done_ms: avg=1456.17 p50=1449.19 p95=1848.87 max=1969.00
timeline_parse_done_ms: avg=1456.23 p50=1449.25 p95=1848.93 max=1969.06
timeline_bicodec_done_ms: avg=1469.78 p50=1462.36 p95=1867.58 max=1986.64
timeline_total_ms: avg=1469.79 p50=1462.41 p95=1867.60 max=1986.66
llm_batch_wall_ms_total: avg=1456.00 p50=1449.03 p95=1848.72 max=1968.85
llm_decode_wall_ms_total: avg=1321.53 p50=1312.41 p95=1818.77 max=1936.29
llm_parse_ms: avg=0.06 p50=0.06 p95=0.07 max=0.10
llm_time_per_token_ms: avg=11.38 p50=11.32 p95=14.44 max=15.38
bicodec_decode_wall_ms: avg=13.50 p50=12.47 p95=18.89 max=21.00
bicodec_decode_gpu_ms: avg=13.42 p50=12.40 p95=18.82 max=20.92
latency_over_timeline_ms: avg=219.63 p50=202.45 p95=434.56 max=480.96
latency_over_generation_ms: avg=220.11 p50=202.77 p95=435.08 max=481.69
timeline_share: llm=99.1% bicodec=0.9% other=0.0%
timeline_sample_ms: first_batch=27.5 llm_done=1512.0 parse_done=1512.1 bicodec_done=1522.2 total=1522.2
gpu: util_avg=87% util_max=100% mem_avg=44793MB mem_max=44793MB

=== concurrency=96 requests=192 ===
success=192/192 (100%)
wall=4.52s throughput=42.45 req/s
latency_ms avg=2039 p50=2020 p95=2470 max=2470
generation_ms: avg=1671.74 p50=1635.00 p95=2270.00 max=2395.00
timeline_llm_first_batch_ms: avg=91.99 p50=34.93 p95=526.66 max=759.32
timeline_llm_done_ms: avg=1659.74 p50=1622.59 p95=2259.24 max=2381.27
timeline_parse_done_ms: avg=1659.80 p50=1622.65 p95=2259.31 max=2381.33
timeline_bicodec_done_ms: avg=1672.16 p50=1635.57 p95=2270.75 max=2395.21
timeline_total_ms: avg=1672.18 p50=1635.59 p95=2270.78 max=2395.23
llm_batch_wall_ms_total: avg=1659.57 p50=1622.45 p95=2259.09 max=2381.13
llm_decode_wall_ms_total: avg=1567.73 p50=1591.86 p95=2221.29 max=2351.95
llm_parse_ms: avg=0.06 p50=0.05 p95=0.07 max=0.11
llm_time_per_token_ms: avg=12.97 p50=12.68 p95=17.65 max=18.60
bicodec_decode_wall_ms: avg=12.31 p50=12.31 p95=17.26 max=21.67
bicodec_decode_gpu_ms: avg=12.24 p50=12.24 p95=17.17 max=21.38
latency_over_timeline_ms: avg=366.44 p50=292.41 p95=859.56 max=981.92
latency_over_generation_ms: avg=366.88 p50=292.80 p95=859.63 max=982.73
timeline_share: llm=99.2% bicodec=0.7% other=0.0%
timeline_sample_ms: first_batch=27.1 llm_done=1650.1 parse_done=1650.1 bicodec_done=1663.1 total=1663.1
gpu: util_avg=91% util_max=100% mem_avg=44793MB mem_max=44793MB

results saved: /tmp/runtime_tuned_e2_g50.json
ERROR 02-18 00:06:55 [core_client.py:605] Engine core proc EngineCore_DP0 died unexpectedly, shutting down client.
ERROR 02-18 00:06:55 [core_client.py:605] Engine core proc EngineCore_DP0 died unexpectedly, shutting down client.


**Planning tuning for scheduling overhead**


Input:
*** Begin Patch
*** Update File: /home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py
@@
     num_engines: int = 1,
     max_num_batched_tokens: Optional[int] = None,
     max_num_seqs: Optional[int] = None,
+    enable_chunked_prefill: Optional[bool] = None,
+    enable_prefix_caching: Optional[bool] = None,
+    disable_log_stats: Optional[bool] = None,
+    enforce_eager: Optional[bool] = None,
     precompute_speaker_globals: Optional[bool] = None,
 ) -> TTSRuntime:
@@
         num_engines: Number of vLLM engine instances (default: 1, set to 2-3 for Tier 3 optimization)
         max_num_batched_tokens: Optional vLLM scheduler cap override.
         max_num_seqs: Optional vLLM concurrent sequence cap override.
+        enable_chunked_prefill: Optional vLLM chunked prefill toggle.
+        enable_prefix_caching: Optional vLLM prefix caching toggle.
+        disable_log_stats: Optional vLLM internal stats log toggle.
+        enforce_eager: Optional vLLM eager-mode toggle (disables CUDA graphs when true).
         precompute_speaker_globals: Whether to warm speaker global tokens at startup.
             If None, reads PRECOMPUTE_SPEAKER_GLOBALS env var (default: false).
@@
         if isinstance(max_num_batched_tokens, int) and max_num_batched_tokens > 0:
             engine_kwargs["max_num_batched_tokens"] = max_num_batched_tokens
         if isinstance(max_num_seqs, int) and max_num_seqs > 0:
             engine_kwargs["max_num_seqs"] = max_num_seqs
+        if enable_chunked_prefill is not None:
+            engine_kwargs["enable_chunked_prefill"] = bool(enable_chunked_prefill)
+        if enable_prefix_caching is not None:
+            engine_kwargs["enable_prefix_caching"] = bool(enable_prefix_caching)
+        if disable_log_stats is not None:
+            engine_kwargs["disable_log_stats"] = bool(disable_log_stats)
+        if enforce_eager is not None:
+            engine_kwargs["enforce_eager"] = bool(enforce_eager)
*** End Patch


Output: {"output":"Success. Updated the following files:\nM /home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


**Patching local server arguments**


Input:
*** Begin Patch
*** Update File: /home/ubuntu/veenaModal/veena3modal/local_server.py
@@
     num_engines: int = 1,
     max_num_batched_tokens: int | None = None,
     max_num_seqs: int | None = None,
+    enable_chunked_prefill: bool | None = None,
+    enable_prefix_caching: bool | None = None,
+    disable_log_stats: bool | None = None,
+    enforce_eager: bool | None = None,
     precompute_speaker_globals: bool = False,
 ):
@@
     if max_num_batched_tokens is not None:
         logger.info(f"  Max batched tokens: {max_num_batched_tokens}")
     if max_num_seqs is not None:
         logger.info(f"  Max seqs:    {max_num_seqs}")
+    if enable_chunked_prefill is not None:
+        logger.info(f"  Chunked prefill: {'enabled' if enable_chunked_prefill else 'disabled'}")
+    if enable_prefix_caching is not None:
+        logger.info(f"  Prefix caching: {'enabled' if enable_prefix_caching else 'disabled'}")
+    if disable_log_stats is not None:
+        logger.info(f"  Engine stats logs: {'disabled' if disable_log_stats else 'enabled'}")
+    if enforce_eager is not None:
+        logger.info(f"  Enforce eager: {'enabled' if enforce_eager else 'disabled'}")
@@
         num_engines=num_engines,
         max_num_batched_tokens=max_num_batched_tokens,
         max_num_seqs=max_num_seqs,
+        enable_chunked_prefill=enable_chunked_prefill,
+        enable_prefix_caching=enable_prefix_caching,
+        disable_log_stats=disable_log_stats,
+        enforce_eager=enforce_eager,
         precompute_speaker_globals=precompute_speaker_globals,
     )
@@
     gpu_group.add_argument(
         "--max-num-seqs",
         type=int,
         default=None,
         help="Override vLLM max_num_seqs scheduler limit (default: vLLM auto)",
     )
+    gpu_group.add_argument(
+        "--enable-chunked-prefill",
+        action="store_true",
+        default=None,
+        help="Force-enable vLLM chunked prefill",
+    )
+    gpu_group.add_argument(
+        "--disable-chunked-prefill",
+        action="store_true",
+        default=False,
+        help="Force-disable vLLM chunked prefill",
+    )
+    gpu_group.add_argument(
+        "--disable-prefix-caching",
+        action="store_true",
+        default=False,
+        help="Disable vLLM prefix caching",
+    )
+    gpu_group.add_argument(
+        "--disable-engine-stats-logs",
+        action="store_true",
+        default=False,
+        help="Disable vLLM periodic stats logging",
+    )
+    gpu_group.add_argument(
+        "--enforce-eager",
+        action="store_true",
+        default=False,
+        help="Enable eager mode (disables CUDA graphs)",
+    )
@@
     # Initialize TTS runtime (this is the expensive step - loads model onto GPU)
+    enable_chunked_prefill = args.enable_chunked_prefill
+    if args.disable_chunked_prefill:
+        enable_chunked_prefill = False
     initialize_local_runtime(
         model_path=model_path,
         device=args.device,
         gpu_memory_utilization=args.gpu_memory,
         enable_sr=args.enable_sr,
         sr_checkpoint_dir=args.sr_path,
         hf_token=args.hf_token,
         num_engines=args.num_engines,
         max_num_batched_tokens=args.max_num_batched_tokens,
         max_num_seqs=args.max_num_seqs,
+        enable_chunked_prefill=enable_chunked_prefill,
+        enable_prefix_caching=(False if args.disable_prefix_caching else None),
+        disable_log_stats=(True if args.disable_engine_stats_logs else None),
+        enforce_eager=(True if args.enforce_eager else None),
         precompute_speaker_globals=args.precompute_globals,
     )
*** End Patch


Output: {"output":"Success. Updated the following files:\nM /home/ubuntu/veenaModal/veena3modal/local_server.py\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


**Adjusting argparse defaults and patching toggles**


Input:
*** Begin Patch
*** Update File: /home/ubuntu/veenaModal/scripts/stress_test_runtime_detailed.py
@@
 def init_runtime(
     model_path: str,
     gpu_mem: float,
     num_engines: int,
     max_num_batched_tokens: Optional[int],
     max_num_seqs: Optional[int],
+    enable_chunked_prefill: Optional[bool],
+    enable_prefix_caching: Optional[bool],
+    disable_log_stats: Optional[bool],
+    enforce_eager: Optional[bool],
 ) -> None:
@@
             num_engines=num_engines,
             max_num_batched_tokens=max_num_batched_tokens,
             max_num_seqs=max_num_seqs,
+            enable_chunked_prefill=enable_chunked_prefill,
+            enable_prefix_caching=enable_prefix_caching,
+            disable_log_stats=disable_log_stats,
+            enforce_eager=enforce_eager,
             precompute_speaker_globals=False,
         )
@@
             "gpu_memory": args.gpu_memory,
             "num_engines": args.num_engines,
             "max_num_batched_tokens": args.max_num_batched_tokens,
             "max_num_seqs": args.max_num_seqs,
+            "enable_chunked_prefill": args.enable_chunked_prefill,
+            "disable_chunked_prefill": args.disable_chunked_prefill,
+            "disable_prefix_caching": args.disable_prefix_caching,
+            "disable_engine_stats_logs": args.disable_engine_stats_logs,
+            "enforce_eager": args.enforce_eager,
             "timestamp": int(time.time()),
         },
@@
     parser.add_argument("--gpu-memory", type=float, default=0.25)
     parser.add_argument("--num-engines", type=int, default=1, help="Number of vLLM engines")
     parser.add_argument("--max-num-batched-tokens", type=int, default=None)
     parser.add_argument("--max-num-seqs", type=int, default=None)
+    parser.add_argument("--enable-chunked-prefill", action="store_true", default=None)
+    parser.add_argument("--disable-chunked-prefill", action="store_true", default=False)
+    parser.add_argument("--disable-prefix-caching", action="store_true", default=False)
+    parser.add_argument("--disable-engine-stats-logs", action="store_true", default=False)
+    parser.add_argument("--enforce-eager", action="store_true", default=False)
     parser.add_argument("--model-path", type=str, default="")
     parser.add_argument("--output", default="stress_runtime_detailed.json")
     args = parser.parse_args()
@@
+    enable_chunked_prefill = args.enable_chunked_prefill
+    if args.disable_chunked_prefill:
+        enable_chunked_prefill = False
     init_runtime(
         model_path=model_path,
         gpu_mem=args.gpu_memory,
         num_engines=args.num_engines,
         max_num_batched_tokens=args.max_num_batched_tokens,
         max_num_seqs=args.max_num_seqs,
+        enable_chunked_prefill=enable_chunked_prefill,
+        enable_prefix_caching=(False if args.disable_prefix_caching else None),
+        disable_log_stats=(True if args.disable_engine_stats_logs else None),
+        enforce_eager=(True if args.enforce_eager else None),
     )
*** End Patch


Output: {"output":"Success. Updated the following files:\nM /home/ubuntu/veenaModal/scripts/stress_test_runtime_detailed.py\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


**Verifying session completion**


`2-engine` run completed and is clearly worse than single-engine for this workload, so I’m pivoting to scheduler-parameter tuning on single-engine (chunked prefill/prefix caching/log-stats/eager mode) to target decode+queue efficiency directly. I’ve already added the runtime flags and I’m about to run the tuning sweep.


Output:
Chunk ID: bd9ada
Wall time: 0.0511 seconds
Process exited with code 0
Original token count: 0
Output:


**Running bytecode compilation**


Output:
Chunk ID: bafb57
Wall time: 0.0511 seconds
Process exited with code 0
Original token count: 0
Output:


**Checking GPU usage before tuning**


Output:
Chunk ID: f7566a
Wall time: 0.0504 seconds
Process exited with code 0
Original token count: 0
Output:


**Selecting test configurations**


python3 /home/ubuntu/veenaModal/scripts/stress_test_runtime_detailed.py --levels 32,64,96 --text short --max-tokens 128 --timeout 180 --requests-multiplier 2 --min-requests 8 --gpu-memory 0.25 --num-engines 1 --disable-chunked-prefill --disable-engine-stats-logs --output /tmp/runtime_tuned_e1_nochunk_nostats.json


Output:
Chunk ID: 961044
Wall time: 1.0011 seconds
Process running with session ID 74090
Original token count: 0
Output:


Summary: auto


Output:
Chunk ID: c5acba
Wall time: 5.0023 seconds
Process running with session ID 74090
Original token count: 1386
Output:
[2026-02-18 00:08:13] INFO tts_runtime.py:183: Loading SparkTTS model with vLLM...
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
🚀 Initializing Spark TTS Model
📁 Model path: /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
🗣  Speakers: 12 predefined (lipakshi, vardan, reet, Nandini, krishna, anika, adarsh, Nilay, Aarvi, Asha, Bittu, Mira)
🎭 Emotions: 10 tags in [bracket] format

📝 Loading tokenizer from model...
✅ Tokenizer loaded: 165168 tokens

🔍 Validating 12 speakers for Spark TTS...
  lipakshi → <|speaker_0|>: 7 tokens
  vardan → <|speaker_1|>: 7 tokens
  reet → <|speaker_2|>: 7 tokens
  Nandini → <|speaker_3|>: 7 tokens
  krishna → <|speaker_4|>: 7 tokens
  anika → <|speaker_5|>: 7 tokens
  adarsh → <|speaker_6|>: 7 tokens
  Nilay → <|speaker_7|>: 7 tokens
  Aarvi → <|speaker_8|>: 7 tokens
  Asha → <|speaker_9|>: 7 tokens
  Bittu → <|speaker_10|>: 8 tokens
  Mira → <|speaker_11|>: 8 tokens
✅ All 12 speakers validated with mapping

🔍 Validating 10 emotion tags for Spark TTS...
✅ All 10 emotion tags validated

📋 Sample emotion tags:
  [angry]: 1 tokens
  [curious]: 1 tokens
  [excited]: 1 tokens
  [giggle]: 1 tokens
  [laughs harder]: 1 tokens
  ...

🔧 Initializing vLLM engine with optimizations...
  - dtype: bfloat16
  - max_model_len: 4096
  - gpu_memory_utilization: 0.25
  - enable_prefix_caching: True
  - enforce_eager (disable CUDA graphs): False
INFO 02-18 00:08:14 [model.py:541] Resolved architecture: Qwen2ForCausalLM
INFO 02-18 00:08:14 [model.py:1561] Using max model len 4096
WARNING 02-18 00:08:14 [arg_utils.py:1909] This model does not officially support disabling chunked prefill. Disabling this manually may cause the engine to crash or produce incorrect outputs.
INFO 02-18 00:08:14 [vllm.py:624] Asynchronous scheduling is enabled.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(EngineCore_DP0 pid=4091861) INFO 02-18 00:08:15 [core.py:96] Initializing a V1 LLM engine (v0.15.1) with config: model='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', speculative_config=None, tokenizer='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM, enable_prefix_caching=True, enable_chunked_prefill=False, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [4096], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 256, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': True}, 'local_cache_dir': None, 'static_all_moe_layers': []}
(EngineCore_DP0 pid=4091861) INFO 02-18 00:08:17 [parallel_state.py:1212] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://216.81.248.184:48271 backend=nccl
(EngineCore_DP0 pid=4091861) INFO 02-18 00:08:17 [parallel_state.py:1423] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A


**Noting chunked prefill risk**


Output:
Chunk ID: c32c2d
Wall time: 5.0013 seconds
Process running with session ID 74090
Original token count: 969
Output:
(EngineCore_DP0 pid=4091861) INFO 02-18 00:08:18 [gpu_model_runner.py:4033] Starting to load model /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM...
(EngineCore_DP0 pid=4091861) INFO 02-18 00:08:18 [cuda.py:364] Using FLASH_ATTN attention backend out of potential backends: ('FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION')
(EngineCore_DP0 pid=4091861) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
(EngineCore_DP0 pid=4091861) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
(EngineCore_DP0 pid=4091861)
Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
(EngineCore_DP0 pid=4091861)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.89it/s]
(EngineCore_DP0 pid=4091861)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.89it/s]
(EngineCore_DP0 pid=4091861)
(EngineCore_DP0 pid=4091861) INFO 02-18 00:08:19 [default_loader.py:291] Loading weights took 0.61 seconds
(EngineCore_DP0 pid=4091861) INFO 02-18 00:08:20 [gpu_model_runner.py:4130] Model loading took 0.95 GiB memory and 1.149440 seconds
(EngineCore_DP0 pid=4091861) INFO 02-18 00:08:24 [backends.py:812] Using cache directory: /home/ubuntu/.cache/vllm/torch_compile_cache/f3ff13511c/rank_0_0/backbone for vLLM's torch.compile
(EngineCore_DP0 pid=4091861) INFO 02-18 00:08:24 [backends.py:872] Dynamo bytecode transform time: 4.21 s
(EngineCore_DP0 pid=4091861) INFO 02-18 00:08:27 [backends.py:267] Directly load the compiled graph(s) for compile range (1, 4096) from the cache, took 0.631 s
(EngineCore_DP0 pid=4091861) INFO 02-18 00:08:27 [monitor.py:34] torch.compile takes 4.85 s in total
(EngineCore_DP0 pid=4091861) INFO 02-18 00:08:28 [gpu_worker.py:356] Available KV cache memory: 18.06 GiB
(EngineCore_DP0 pid=4091861) INFO 02-18 00:08:28 [kv_cache_utils.py:1307] GPU KV cache size: 1,577,712 tokens
(EngineCore_DP0 pid=4091861) INFO 02-18 00:08:28 [kv_cache_utils.py:1312] Maximum concurrency for 4,096 tokens per request: 385.18x
(EngineCore_DP0 pid=4091861)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   0%|          | 0/35 [00:00<?, ?it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  11%|█▏        | 4/35 [00:00<00:00, 32.17it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  23%|██▎       | 8/35 [00:00<00:00, 35.26it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  34%|███▍      | 12/35 [00:00<00:00, 36.81it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  46%|████▌     | 16/35 [00:00<00:00, 37.11it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  57%|█████▋    | 20/35 [00:00<00:00, 37.23it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  69%|██████▊   | 24/35 [00:00<00:00, 37.36it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  80%|████████  | 28/35 [00:00<00:00, 37.25it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  91%|█████████▏| 32/35 [00:00<00:00, 35.65it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 35/35 [00:00<00:00, 35.49it/s]
(EngineCore_DP0 pid=4091861)
Capturing CUDA graphs (decode, FULL):   0%|          | 0/19 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL):  16%|█▌        | 3/19 [00:00<00:00, 29.27it/s]
Capturing CUDA graphs (decode, FULL):  37%|███▋      | 7/19 [00:00<00:00, 33.87it/s]
Capturing CUDA graphs (decode, FULL):  58%|█████▊    | 11/19 [00:00<00:00, 35.29it/s]


Output:
Chunk ID: 81ff83
Wall time: 5.0009 seconds
Process running with session ID 74090
Original token count: 370
Output:

Capturing CUDA graphs (decode, FULL):  79%|███████▉  | 15/19 [00:00<00:00, 35.71it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 36.17it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 35.34it/s]
(EngineCore_DP0 pid=4091861) INFO 02-18 00:08:31 [gpu_model_runner.py:5063] Graph capturing finished in 2 secs, took 0.28 GiB
(EngineCore_DP0 pid=4091861) INFO 02-18 00:08:31 [core.py:272] init engine (profile, create kv cache, warmup model) took 10.75 seconds
[2026-02-18 00:08:32] INFO tts_runtime.py:192: Initializing prompt builder...
[2026-02-18 00:08:32] INFO tts_runtime.py:199: Initializing BiCodec decoder...
/home/ubuntu/.local/lib/python3.10/site-packages/torch/nn/utils/weight_norm.py:144: FutureWarning: `torch.nn.utils.weight_norm` is deprecated in favor of `torch.nn.utils.parametrizations.weight_norm`.
  WeightNorm.apply(module, name, dim)
[2026-02-18 00:08:40] INFO tts_runtime.py:206: Initializing TTS pipeline...
[2026-02-18 00:08:40] INFO tts_runtime.py:217: Initializing streaming pipeline...
[2026-02-18 00:08:40] INFO long_text_processor.py:112: LongTextProcessor initialized
[2026-02-18 00:08:40] INFO tts_runtime.py:274: ✅ TTS runtime initialized in 32733ms
[2026-02-18 00:08:40] INFO tts_runtime.py:275:    Model version: LLM
[2026-02-18 00:08:40] INFO tts_runtime.py:286: Speaker globals cache startup precompute skipped (disabled)


Output:
Chunk ID: b97449
Wall time: 5.0010 seconds
Process running with session ID 74090
Original token count: 0
Output:


Output:
Chunk ID: 1d5877
Wall time: 3.3721 seconds
Process exited with code 0
Original token count: 1126
Output:
✅ vLLM engine initialized successfully!

🎉 Spark TTS Model ready for inference!

🎵 Loading BiCodec audio tokenizer from /home/ubuntu/veenaModal/models/spark_tts_4speaker...
   Device: cuda
Missing tensor: mel_transformer.spectrogram.window
Missing tensor: mel_transformer.mel_scale.fb
✅ BiCodec decoder initialized (sample rate: 16000Hz)
🚀 SparkTTSPipeline initialized
🌊 Veena3SlidingWindowPipeline initialized (sliding window: 28 tokens)

=== concurrency=32 requests=64 ===
success=64/64 (100%)
wall=2.01s throughput=31.85 req/s
latency_ms avg=956 p50=951 p95=1056 max=1056
generation_ms: avg=810.73 p50=806.00 p95=989.00 max=1038.00
timeline_llm_first_batch_ms: avg=31.33 p50=24.78 p95=30.09 max=430.63
timeline_llm_done_ms: avg=799.32 p50=790.46 p95=976.47 max=1025.97
timeline_parse_done_ms: avg=799.39 p50=790.52 p95=976.53 max=1026.03
timeline_bicodec_done_ms: avg=811.17 p50=806.09 p95=989.40 max=1038.44
timeline_total_ms: avg=811.19 p50=806.13 p95=989.42 max=1038.45
llm_batch_wall_ms_total: avg=799.16 p50=790.31 p95=976.29 max=1025.82
llm_decode_wall_ms_total: avg=767.98 p50=764.63 p95=956.71 max=1003.13
llm_parse_ms: avg=0.06 p50=0.06 p95=0.08 max=0.11
llm_time_per_token_ms: avg=6.24 p50=6.17 p95=7.63 max=8.01
bicodec_decode_wall_ms: avg=11.72 p50=12.27 p95=14.45 max=16.47
bicodec_decode_gpu_ms: avg=11.64 p50=12.18 p95=14.38 max=16.27
latency_over_timeline_ms: avg=145.20 p50=114.37 p95=352.00 max=402.50
latency_over_generation_ms: avg=145.66 p50=114.70 p95=352.52 max=402.57
timeline_share: llm=98.5% bicodec=1.4% other=0.0%
timeline_sample_ms: first_batch=28.1 llm_done=756.2 parse_done=756.2 bicodec_done=766.2 total=766.3
gpu: util_avg=61% util_max=68% mem_avg=23601MB mem_max=23601MB

=== concurrency=64 requests=128 ===
success=128/128 (100%)
wall=3.05s throughput=41.91 req/s
latency_ms avg=1508 p50=1423 p95=1624 max=1624
generation_ms: avg=1142.04 p50=1132.00 p95=1511.00 max=1594.00
timeline_llm_first_batch_ms: avg=38.64 p50=32.07 p95=39.46 max=861.54
timeline_llm_done_ms: avg=1130.79 p50=1120.70 p95=1498.71 max=1583.87
timeline_parse_done_ms: avg=1130.86 p50=1120.76 p95=1498.80 max=1583.93
timeline_bicodec_done_ms: avg=1142.46 p50=1132.91 p95=1511.72 max=1594.17
timeline_total_ms: avg=1142.48 p50=1132.93 p95=1511.74 max=1594.19
llm_batch_wall_ms_total: avg=1130.65 p50=1120.57 p95=1498.57 max=1583.75
llm_decode_wall_ms_total: avg=1100.74 p50=1093.12 p95=1475.91 max=1557.49
llm_parse_ms: avg=0.06 p50=0.06 p95=0.07 max=0.10
llm_time_per_token_ms: avg=8.83 p50=8.75 p95=11.71 max=12.37
bicodec_decode_wall_ms: avg=11.55 p50=11.14 p95=14.55 max=17.84
bicodec_decode_gpu_ms: avg=11.47 p50=11.05 p95=14.47 max=17.75
latency_over_timeline_ms: avg=365.50 p50=358.16 p95=721.54 max=815.94
latency_over_generation_ms: avg=365.95 p50=358.33 p95=722.38 max=816.37
timeline_share: llm=99.0% bicodec=1.0% other=0.0%
timeline_sample_ms: first_batch=41.0 llm_done=792.9 parse_done=793.0 bicodec_done=803.0 total=803.0
gpu: util_avg=75% util_max=99% mem_avg=23601MB mem_max=23601MB

=== concurrency=96 requests=192 ===
success=192/192 (100%)
wall=4.01s throughput=47.85 req/s
latency_ms avg=1897 p50=1881 p95=2126 max=2126
generation_ms: avg=1456.53 p50=1444.00 p95=1969.00 max=2066.00
timeline_llm_first_batch_ms: avg=42.27 p50=33.81 p95=59.79 max=1142.45
timeline_llm_done_ms: avg=1445.78 p50=1434.50 p95=1959.47 max=2056.32
timeline_parse_done_ms: avg=1445.84 p50=1434.56 p95=1959.53 max=2056.38
timeline_bicodec_done_ms: avg=1456.96 p50=1444.89 p95=1969.54 max=2066.45
timeline_total_ms: avg=1456.98 p50=1444.91 p95=1969.56 max=2066.47
llm_batch_wall_ms_total: avg=1445.63 p50=1434.37 p95=1959.33 max=2056.19
llm_decode_wall_ms_total: avg=1403.50 p50=1403.75 p95=1931.10 max=2024.63
llm_parse_ms: avg=0.06 p50=0.06 p95=0.07 max=0.10
llm_time_per_token_ms: avg=11.29 p50=11.21 p95=15.31 max=16.06
bicodec_decode_wall_ms: avg=11.06 p50=10.13 p95=13.65 max=18.54
bicodec_decode_gpu_ms: avg=10.99 p50=10.05 p95=13.56 max=18.29
latency_over_timeline_ms: avg=440.00 p50=386.93 p95=955.98 max=1056.19
latency_over_generation_ms: avg=440.46 p50=387.68 p95=956.02 max=1057.06
timeline_share: llm=99.2% bicodec=0.8% other=0.0%
timeline_sample_ms: first_batch=31.5 llm_done=1156.8 parse_done=1156.8 bicodec_done=1169.1 total=1169.2
gpu: util_avg=60% util_max=99% mem_avg=23601MB mem_max=23601MB

results saved: /tmp/runtime_tuned_e1_nochunk_nostats.json
ERROR 02-18 00:08:55 [core_client.py:605] Engine core proc EngineCore_DP0 died unexpectedly, shutting down client.


**Testing safer engine stats config**


python3 /home/ubuntu/veenaModal/scripts/stress_test_runtime_detailed.py --levels 32,64,96 --text short --max-tokens 128 --timeout 180 --requests-multiplier 2 --min-requests 8 --gpu-memory 0.25 --num-engines 1 --disable-engine-stats-logs --output /tmp/runtime_tuned_e1_nostats.json


Output:
Chunk ID: e55cd5
Wall time: 1.0012 seconds
Process running with session ID 49525
Original token count: 0
Output:


**Planning polling mechanism**


I ran a no-risk variant next (`disable engine stats logs` only, with chunked prefill left on) to see if we can keep gains without relying on an unsupported vLLM setting. I’ll compare all three immediately after it finishes.


Output:
Chunk ID: 22b76d
Wall time: 5.0017 seconds
Process running with session ID 49525
Original token count: 1401
Output:
[2026-02-18 00:09:08] INFO tts_runtime.py:183: Loading SparkTTS model with vLLM...
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
🚀 Initializing Spark TTS Model
📁 Model path: /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
🗣  Speakers: 12 predefined (lipakshi, vardan, reet, Nandini, krishna, anika, adarsh, Nilay, Aarvi, Asha, Bittu, Mira)
🎭 Emotions: 10 tags in [bracket] format

📝 Loading tokenizer from model...
✅ Tokenizer loaded: 165168 tokens

🔍 Validating 12 speakers for Spark TTS...
  lipakshi → <|speaker_0|>: 7 tokens
  vardan → <|speaker_1|>: 7 tokens
  reet → <|speaker_2|>: 7 tokens
  Nandini → <|speaker_3|>: 7 tokens
  krishna → <|speaker_4|>: 7 tokens
  anika → <|speaker_5|>: 7 tokens
  adarsh → <|speaker_6|>: 7 tokens
  Nilay → <|speaker_7|>: 7 tokens
  Aarvi → <|speaker_8|>: 7 tokens
  Asha → <|speaker_9|>: 7 tokens
  Bittu → <|speaker_10|>: 8 tokens
  Mira → <|speaker_11|>: 8 tokens
✅ All 12 speakers validated with mapping

🔍 Validating 10 emotion tags for Spark TTS...
✅ All 10 emotion tags validated

📋 Sample emotion tags:
  [angry]: 1 tokens
  [curious]: 1 tokens
  [excited]: 1 tokens
  [giggle]: 1 tokens
  [laughs harder]: 1 tokens
  ...

🔧 Initializing vLLM engine with optimizations...
  - dtype: bfloat16
  - max_model_len: 4096
  - gpu_memory_utilization: 0.25
  - enable_prefix_caching: True
  - enforce_eager (disable CUDA graphs): False
INFO 02-18 00:09:08 [model.py:541] Resolved architecture: Qwen2ForCausalLM
INFO 02-18 00:09:08 [model.py:1561] Using max model len 4096
INFO 02-18 00:09:09 [scheduler.py:226] Chunked prefill is enabled with max_num_batched_tokens=4096.
INFO 02-18 00:09:09 [vllm.py:624] Asynchronous scheduling is enabled.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(EngineCore_DP0 pid=4092228) INFO 02-18 00:09:09 [core.py:96] Initializing a V1 LLM engine (v0.15.1) with config: model='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', speculative_config=None, tokenizer='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [4096], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 256, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': True}, 'local_cache_dir': None, 'static_all_moe_layers': []}
(EngineCore_DP0 pid=4092228) INFO 02-18 00:09:12 [parallel_state.py:1212] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://216.81.248.184:34321 backend=nccl
(EngineCore_DP0 pid=4092228) INFO 02-18 00:09:12 [parallel_state.py:1423] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
(EngineCore_DP0 pid=4092228) INFO 02-18 00:09:12 [gpu_model_runner.py:4033] Starting to load model /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM...


Output:
Chunk ID: 0b7525
Wall time: 5.0018 seconds
Process running with session ID 49525
Original token count: 452
Output:
(EngineCore_DP0 pid=4092228) INFO 02-18 00:09:13 [cuda.py:364] Using FLASH_ATTN attention backend out of potential backends: ('FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION')
(EngineCore_DP0 pid=4092228) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
(EngineCore_DP0 pid=4092228) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
(EngineCore_DP0 pid=4092228)
Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
(EngineCore_DP0 pid=4092228)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.86it/s]
(EngineCore_DP0 pid=4092228)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.86it/s]
(EngineCore_DP0 pid=4092228)
(EngineCore_DP0 pid=4092228) INFO 02-18 00:09:14 [default_loader.py:291] Loading weights took 0.62 seconds
(EngineCore_DP0 pid=4092228) INFO 02-18 00:09:15 [gpu_model_runner.py:4130] Model loading took 0.95 GiB memory and 1.154359 seconds
(EngineCore_DP0 pid=4092228) INFO 02-18 00:09:19 [backends.py:812] Using cache directory: /home/ubuntu/.cache/vllm/torch_compile_cache/f3ff13511c/rank_0_0/backbone for vLLM's torch.compile
(EngineCore_DP0 pid=4092228) INFO 02-18 00:09:19 [backends.py:872] Dynamo bytecode transform time: 4.22 s
(EngineCore_DP0 pid=4092228) INFO 02-18 00:09:22 [backends.py:267] Directly load the compiled graph(s) for compile range (1, 4096) from the cache, took 0.648 s
(EngineCore_DP0 pid=4092228) INFO 02-18 00:09:22 [monitor.py:34] torch.compile takes 4.87 s in total


Output:
Chunk ID: 001107
Wall time: 5.0007 seconds
Process running with session ID 49525
Original token count: 848
Output:
(EngineCore_DP0 pid=4092228) INFO 02-18 00:09:23 [gpu_worker.py:356] Available KV cache memory: 18.06 GiB
(EngineCore_DP0 pid=4092228) INFO 02-18 00:09:23 [kv_cache_utils.py:1307] GPU KV cache size: 1,577,712 tokens
(EngineCore_DP0 pid=4092228) INFO 02-18 00:09:23 [kv_cache_utils.py:1312] Maximum concurrency for 4,096 tokens per request: 385.18x
(EngineCore_DP0 pid=4092228)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   0%|          | 0/35 [00:00<?, ?it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  11%|█▏        | 4/35 [00:00<00:00, 32.45it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  23%|██▎       | 8/35 [00:00<00:00, 33.88it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  34%|███▍      | 12/35 [00:00<00:00, 34.86it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  46%|████▌     | 16/35 [00:00<00:00, 35.99it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  57%|█████▋    | 20/35 [00:00<00:00, 35.66it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  69%|██████▊   | 24/35 [00:00<00:00, 35.27it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  80%|████████  | 28/35 [00:00<00:00, 35.42it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  91%|█████████▏| 32/35 [00:00<00:00, 33.73it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 35/35 [00:01<00:00, 33.95it/s]
(EngineCore_DP0 pid=4092228)
Capturing CUDA graphs (decode, FULL):   0%|          | 0/19 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL):  16%|█▌        | 3/19 [00:00<00:00, 29.00it/s]
Capturing CUDA graphs (decode, FULL):  37%|███▋      | 7/19 [00:00<00:00, 33.32it/s]
Capturing CUDA graphs (decode, FULL):  58%|█████▊    | 11/19 [00:00<00:00, 34.94it/s]
Capturing CUDA graphs (decode, FULL):  79%|███████▉  | 15/19 [00:00<00:00, 35.14it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 35.68it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 34.86it/s]
(EngineCore_DP0 pid=4092228) INFO 02-18 00:09:26 [gpu_model_runner.py:5063] Graph capturing finished in 2 secs, took 0.28 GiB
(EngineCore_DP0 pid=4092228) INFO 02-18 00:09:26 [core.py:272] init engine (profile, create kv cache, warmup model) took 10.96 seconds
[2026-02-18 00:09:27] INFO tts_runtime.py:192: Initializing prompt builder...
[2026-02-18 00:09:27] INFO tts_runtime.py:199: Initializing BiCodec decoder...
/home/ubuntu/.local/lib/python3.10/site-packages/torch/nn/utils/weight_norm.py:144: FutureWarning: `torch.nn.utils.weight_norm` is deprecated in favor of `torch.nn.utils.parametrizations.weight_norm`.
  WeightNorm.apply(module, name, dim)
[2026-02-18 00:09:32] INFO tts_runtime.py:206: Initializing TTS pipeline...
[2026-02-18 00:09:32] INFO tts_runtime.py:217: Initializing streaming pipeline...
[2026-02-18 00:09:32] INFO long_text_processor.py:112: LongTextProcessor initialized
[2026-02-18 00:09:32] INFO tts_runtime.py:274: ✅ TTS runtime initialized in 29942ms
[2026-02-18 00:09:32] INFO tts_runtime.py:275:    Model version: LLM
[2026-02-18 00:09:32] INFO tts_runtime.py:286: Speaker globals cache startup precompute skipped (disabled)


Output:
Chunk ID: f921c8
Wall time: 5.0009 seconds
Process running with session ID 49525
Original token count: 0
Output:


Output:
Chunk ID: 8149c2
Wall time: 0.0515 seconds
Process exited with code 0
Original token count: 1124
Output:
✅ vLLM engine initialized successfully!

🎉 Spark TTS Model ready for inference!

🎵 Loading BiCodec audio tokenizer from /home/ubuntu/veenaModal/models/spark_tts_4speaker...
   Device: cuda
Missing tensor: mel_transformer.spectrogram.window
Missing tensor: mel_transformer.mel_scale.fb
✅ BiCodec decoder initialized (sample rate: 16000Hz)
🚀 SparkTTSPipeline initialized
🌊 Veena3SlidingWindowPipeline initialized (sliding window: 28 tokens)

=== concurrency=32 requests=64 ===
success=64/64 (100%)
wall=2.01s throughput=31.91 req/s
latency_ms avg=970 p50=936 p95=1067 max=1067
generation_ms: avg=811.75 p50=801.00 p95=1000.00 max=1051.00
timeline_llm_first_batch_ms: avg=36.20 p50=23.19 p95=30.27 max=431.78
timeline_llm_done_ms: avg=800.16 p50=785.83 p95=986.13 max=1038.34
timeline_parse_done_ms: avg=800.23 p50=785.89 p95=986.19 max=1038.40
timeline_bicodec_done_ms: avg=812.21 p50=801.73 p95=1000.49 max=1051.42
timeline_total_ms: avg=812.23 p50=801.74 p95=1000.50 max=1051.43
llm_batch_wall_ms_total: avg=800.01 p50=785.70 p95=986.00 max=1038.19
llm_decode_wall_ms_total: avg=763.95 p50=762.17 p95=965.96 max=1014.88
llm_parse_ms: avg=0.06 p50=0.06 p95=0.07 max=0.09
llm_time_per_token_ms: avg=6.25 p50=6.14 p95=7.70 max=8.11
bicodec_decode_wall_ms: avg=11.93 p50=12.59 p95=14.31 max=15.78
bicodec_decode_gpu_ms: avg=11.86 p50=12.52 p95=14.23 max=15.71
latency_over_timeline_ms: avg=158.02 p50=148.77 p95=347.00 max=401.31
latency_over_generation_ms: avg=158.50 p50=148.79 p95=347.82 max=401.82
timeline_share: llm=98.5% bicodec=1.5% other=0.0%
timeline_sample_ms: first_batch=25.6 llm_done=676.6 parse_done=676.7 bicodec_done=686.7 total=686.7
gpu: util_avg=67% util_max=73% mem_avg=23601MB mem_max=23601MB

=== concurrency=64 requests=128 ===
success=128/128 (100%)
wall=3.10s throughput=41.30 req/s
latency_ms avg=1520 p50=1419 p95=1673 max=1673
generation_ms: avg=1152.91 p50=1136.00 p95=1551.00 max=1642.00
timeline_llm_first_batch_ms: avg=47.35 p50=32.17 p95=51.07 max=899.01
timeline_llm_done_ms: avg=1141.37 p50=1121.43 p95=1538.84 max=1629.32
timeline_parse_done_ms: avg=1141.44 p50=1121.49 p95=1538.91 max=1629.38
timeline_bicodec_done_ms: avg=1153.31 p50=1136.79 p95=1551.31 max=1642.54
timeline_total_ms: avg=1153.32 p50=1136.80 p95=1551.32 max=1642.56
llm_batch_wall_ms_total: avg=1141.22 p50=1121.30 p95=1538.71 max=1629.19
llm_decode_wall_ms_total: avg=1111.37 p50=1094.27 p95=1513.28 max=1606.70
llm_parse_ms: avg=0.06 p50=0.06 p95=0.07 max=0.11
llm_time_per_token_ms: avg=8.92 p50=8.76 p95=12.02 max=12.73
bicodec_decode_wall_ms: avg=11.81 p50=11.60 p95=14.43 max=19.11
bicodec_decode_gpu_ms: avg=11.74 p50=11.53 p95=14.36 max=19.00
latency_over_timeline_ms: avg=366.86 p50=356.16 p95=748.72 max=839.37
latency_over_generation_ms: avg=367.28 p50=356.41 p95=749.18 max=840.13
timeline_share: llm=99.0% bicodec=1.0% other=0.0%
timeline_sample_ms: first_batch=52.0 llm_done=807.3 parse_done=807.4 bicodec_done=817.4 total=817.5
gpu: util_avg=63% util_max=99% mem_avg=23601MB mem_max=23601MB

=== concurrency=96 requests=192 ===
success=192/192 (100%)
wall=3.92s throughput=48.95 req/s
latency_ms avg=1795 p50=1634 p95=2268 max=2268
generation_ms: avg=1492.87 p50=1446.00 p95=2087.00 max=2220.00
timeline_llm_first_batch_ms: avg=211.48 p50=36.57 p95=670.10 max=1313.50
timeline_llm_done_ms: avg=1480.57 p50=1436.46 p95=2075.16 max=2205.65
timeline_parse_done_ms: avg=1480.64 p50=1436.52 p95=2075.22 max=2205.72
timeline_bicodec_done_ms: avg=1493.30 p50=1446.54 p95=2087.12 max=2220.32
timeline_total_ms: avg=1493.32 p50=1446.55 p95=2087.14 max=2220.35
llm_batch_wall_ms_total: avg=1480.43 p50=1436.32 p95=2075.04 max=2205.51
llm_decode_wall_ms_total: avg=1269.08 p50=1253.18 p95=2042.28 max=2176.90
llm_parse_ms: avg=0.06 p50=0.06 p95=0.08 max=0.11
llm_time_per_token_ms: avg=11.57 p50=11.22 p95=16.21 max=17.23
bicodec_decode_wall_ms: avg=12.62 p50=12.46 p95=15.58 max=17.99
bicodec_decode_gpu_ms: avg=12.54 p50=12.39 p95=15.52 max=17.76
latency_over_timeline_ms: avg=301.55 p50=292.65 p95=598.47 max=673.04
latency_over_generation_ms: avg=302.00 p50=293.11 p95=598.62 max=673.60
timeline_share: llm=99.1% bicodec=0.8% other=0.0%
timeline_sample_ms: first_batch=38.1 llm_done=1047.0 parse_done=1047.1 bicodec_done=1059.4 total=1059.4
gpu: util_avg=70% util_max=98% mem_avg=23601MB mem_max=23601MB

results saved: /tmp/runtime_tuned_e1_nostats.json
ERROR 02-18 00:09:47 [core_client.py:605] Engine core proc EngineCore_DP0 died unexpectedly, shutting down client.


**Confirming throughput improvements**


The `no-stats` single-engine run looks promising (best throughput so far), but I want a clean apples-to-apples confirmation. I’m running a focused `96c` A/B (baseline vs `disable_engine_stats_logs`) with a larger sample size before locking this in as the recommended production tuning.


nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv,noheader && python3 /home/ubuntu/veenaModal/scripts/stress_test_runtime_detailed.py --levels 96 --text short --max-tokens 128 --timeout 180 --requests-multiplier 4 --min-requests 8 --gpu-memory 0.25 --num-engines 1 --output /tmp/runtime_c96_baseline_confirm.json


Output:
Chunk ID: 2d3d3e
Wall time: 1.0012 seconds
Process running with session ID 31949
Original token count: 0
Output:


Output:
Chunk ID: 0c4572
Wall time: 5.0006 seconds
Process running with session ID 31949
Original token count: 1589
Output:
[2026-02-18 00:10:09] INFO tts_runtime.py:183: Loading SparkTTS model with vLLM...
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
🚀 Initializing Spark TTS Model
📁 Model path: /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
🗣  Speakers: 12 predefined (lipakshi, vardan, reet, Nandini, krishna, anika, adarsh, Nilay, Aarvi, Asha, Bittu, Mira)
🎭 Emotions: 10 tags in [bracket] format

📝 Loading tokenizer from model...
✅ Tokenizer loaded: 165168 tokens

🔍 Validating 12 speakers for Spark TTS...
  lipakshi → <|speaker_0|>: 7 tokens
  vardan → <|speaker_1|>: 7 tokens
  reet → <|speaker_2|>: 7 tokens
  Nandini → <|speaker_3|>: 7 tokens
  krishna → <|speaker_4|>: 7 tokens
  anika → <|speaker_5|>: 7 tokens
  adarsh → <|speaker_6|>: 7 tokens
  Nilay → <|speaker_7|>: 7 tokens
  Aarvi → <|speaker_8|>: 7 tokens
  Asha → <|speaker_9|>: 7 tokens
  Bittu → <|speaker_10|>: 8 tokens
  Mira → <|speaker_11|>: 8 tokens
✅ All 12 speakers validated with mapping

🔍 Validating 10 emotion tags for Spark TTS...
✅ All 10 emotion tags validated

📋 Sample emotion tags:
  [angry]: 1 tokens
  [curious]: 1 tokens
  [excited]: 1 tokens
  [giggle]: 1 tokens
  [laughs harder]: 1 tokens
  ...

🔧 Initializing vLLM engine with optimizations...
  - dtype: bfloat16
  - max_model_len: 4096
  - gpu_memory_utilization: 0.25
  - enable_prefix_caching: True
  - enforce_eager (disable CUDA graphs): False
INFO 02-18 00:10:10 [model.py:541] Resolved architecture: Qwen2ForCausalLM
INFO 02-18 00:10:10 [model.py:1561] Using max model len 4096
INFO 02-18 00:10:10 [scheduler.py:226] Chunked prefill is enabled with max_num_batched_tokens=4096.
INFO 02-18 00:10:10 [vllm.py:624] Asynchronous scheduling is enabled.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(EngineCore_DP0 pid=4092599) INFO 02-18 00:10:11 [core.py:96] Initializing a V1 LLM engine (v0.15.1) with config: model='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', speculative_config=None, tokenizer='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [4096], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 256, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': True}, 'local_cache_dir': None, 'static_all_moe_layers': []}
(EngineCore_DP0 pid=4092599) INFO 02-18 00:10:13 [parallel_state.py:1212] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://216.81.248.184:44749 backend=nccl
(EngineCore_DP0 pid=4092599) INFO 02-18 00:10:13 [parallel_state.py:1423] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
(EngineCore_DP0 pid=4092599) INFO 02-18 00:10:13 [gpu_model_runner.py:4033] Starting to load model /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM...
(EngineCore_DP0 pid=4092599) INFO 02-18 00:10:14 [cuda.py:364] Using FLASH_ATTN attention backend out of potential backends: ('FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION')
(EngineCore_DP0 pid=4092599) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
(EngineCore_DP0 pid=4092599) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
(EngineCore_DP0 pid=4092599)
Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]


Output:
Chunk ID: 604234
Wall time: 5.0017 seconds
Process running with session ID 31949
Original token count: 382
Output:
(EngineCore_DP0 pid=4092599)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:01<00:00,  1.16s/it]
(EngineCore_DP0 pid=4092599)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:01<00:00,  1.16s/it]
(EngineCore_DP0 pid=4092599)
(EngineCore_DP0 pid=4092599) INFO 02-18 00:10:15 [default_loader.py:291] Loading weights took 1.25 seconds
(EngineCore_DP0 pid=4092599) INFO 02-18 00:10:16 [gpu_model_runner.py:4130] Model loading took 0.95 GiB memory and 1.795528 seconds
(EngineCore_DP0 pid=4092599) INFO 02-18 00:10:21 [backends.py:812] Using cache directory: /home/ubuntu/.cache/vllm/torch_compile_cache/f3ff13511c/rank_0_0/backbone for vLLM's torch.compile
(EngineCore_DP0 pid=4092599) INFO 02-18 00:10:21 [backends.py:872] Dynamo bytecode transform time: 4.14 s
(EngineCore_DP0 pid=4092599) INFO 02-18 00:10:24 [backends.py:267] Directly load the compiled graph(s) for compile range (1, 4096) from the cache, took 0.661 s
(EngineCore_DP0 pid=4092599) INFO 02-18 00:10:24 [monitor.py:34] torch.compile takes 4.80 s in total
(EngineCore_DP0 pid=4092599) INFO 02-18 00:10:25 [gpu_worker.py:356] Available KV cache memory: 18.06 GiB
(EngineCore_DP0 pid=4092599) INFO 02-18 00:10:25 [kv_cache_utils.py:1307] GPU KV cache size: 1,577,712 tokens
(EngineCore_DP0 pid=4092599) INFO 02-18 00:10:25 [kv_cache_utils.py:1312] Maximum concurrency for 4,096 tokens per request: 385.18x
(EngineCore_DP0 pid=4092599)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   0%|          | 0/35 [00:00<?, ?it/s]