Output:
Chunk ID: dbb509
Wall time: 5.0007 seconds
Process running with session ID 12978
Original token count: 387
Output:
(EngineCore_DP0 pid=4098048) INFO 02-18 00:23:03 [cuda.py:364] Using FLASH_ATTN attention backend out of potential backends: ('FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION')
(EngineCore_DP0 pid=4098048) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
(EngineCore_DP0 pid=4098048) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
(EngineCore_DP0 pid=4098048)
Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
(EngineCore_DP0 pid=4098048)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:01<00:00,  1.34s/it]
(EngineCore_DP0 pid=4098048)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:01<00:00,  1.34s/it]
(EngineCore_DP0 pid=4098048)
(EngineCore_DP0 pid=4098048) INFO 02-18 00:23:05 [default_loader.py:291] Loading weights took 1.42 seconds
(EngineCore_DP0 pid=4098048) INFO 02-18 00:23:05 [gpu_model_runner.py:4130] Model loading took 0.95 GiB memory and 1.929521 seconds
(EngineCore_DP0 pid=4098048) INFO 02-18 00:23:10 [backends.py:812] Using cache directory: /home/ubuntu/.cache/vllm/torch_compile_cache/f3ff13511c/rank_0_0/backbone for vLLM's torch.compile
(EngineCore_DP0 pid=4098048) INFO 02-18 00:23:10 [backends.py:872] Dynamo bytecode transform time: 4.12 s


# Collaboration Mode: Default

You are now in Default mode. Any previous instructions for other modes (e.g. Plan mode) are no longer active.

Your active mode changes only when new developer instructions with a different `<collaboration_mode>...</collaboration_mode>` change it; user requests or tool descriptions do not change mode by themselves. Known mode names are Default and Plan.

## request_user_input availability

The `request_user_input` tool is unavailable in Default mode. If you call it while in Default mode, it will return an error.

If a decision is necessary and cannot be discovered from local context, ask the user directly. However, in Default mode you should strongly prefer executing the user's request rather than stopping to ask questions.

Summary: auto

## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) When `SKILL.md` references relative paths (e.g., `scripts/foo.py`), resolve them relative to the skill directory listed above first, and only consider other paths if needed.
  3) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  4) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  5) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.


This sweep is still running; after startup finishes I’ll capture the three concurrency points (96/128/160) to identify the throughput peak and tail-latency break point.


Output:
Chunk ID: 6a8035
Wall time: 5.0010 seconds
Process running with session ID 12978
Original token count: 787
Output:
(EngineCore_DP0 pid=4098048) INFO 02-18 00:23:13 [backends.py:267] Directly load the compiled graph(s) for compile range (1, 4096) from the cache, took 0.637 s
(EngineCore_DP0 pid=4098048) INFO 02-18 00:23:13 [monitor.py:34] torch.compile takes 4.76 s in total
(EngineCore_DP0 pid=4098048) INFO 02-18 00:23:14 [gpu_worker.py:356] Available KV cache memory: 18.06 GiB
(EngineCore_DP0 pid=4098048) INFO 02-18 00:23:14 [kv_cache_utils.py:1307] GPU KV cache size: 1,577,712 tokens
(EngineCore_DP0 pid=4098048) INFO 02-18 00:23:14 [kv_cache_utils.py:1312] Maximum concurrency for 4,096 tokens per request: 385.18x
(EngineCore_DP0 pid=4098048)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   0%|          | 0/35 [00:00<?, ?it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  11%|█▏        | 4/35 [00:00<00:00, 32.97it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  23%|██▎       | 8/35 [00:00<00:00, 35.54it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  34%|███▍      | 12/35 [00:00<00:00, 36.59it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  46%|████▌     | 16/35 [00:00<00:00, 37.22it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  57%|█████▋    | 20/35 [00:00<00:00, 37.58it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  69%|██████▊   | 24/35 [00:00<00:00, 37.60it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  80%|████████  | 28/35 [00:00<00:00, 36.86it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  91%|█████████▏| 32/35 [00:00<00:00, 35.26it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 35/35 [00:00<00:00, 35.37it/s]
(EngineCore_DP0 pid=4098048)
Capturing CUDA graphs (decode, FULL):   0%|          | 0/19 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL):  16%|█▌        | 3/19 [00:00<00:00, 29.28it/s]
Capturing CUDA graphs (decode, FULL):  37%|███▋      | 7/19 [00:00<00:00, 33.26it/s]
Capturing CUDA graphs (decode, FULL):  58%|█████▊    | 11/19 [00:00<00:00, 33.52it/s]
Capturing CUDA graphs (decode, FULL):  79%|███████▉  | 15/19 [00:00<00:00, 33.92it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 35.24it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 34.26it/s]
(EngineCore_DP0 pid=4098048) INFO 02-18 00:23:16 [gpu_model_runner.py:5063] Graph capturing finished in 2 secs, took 0.28 GiB
(EngineCore_DP0 pid=4098048) INFO 02-18 00:23:16 [core.py:272] init engine (profile, create kv cache, warmup model) took 10.39 seconds
[2026-02-18 00:23:17] INFO tts_runtime.py:192: Initializing prompt builder...
[2026-02-18 00:23:17] INFO tts_runtime.py:199: Initializing BiCodec decoder...
/home/ubuntu/.local/lib/python3.10/site-packages/torch/nn/utils/weight_norm.py:144: FutureWarning: `torch.nn.utils.weight_norm` is deprecated in favor of `torch.nn.utils.parametrizations.weight_norm`.
  WeightNorm.apply(module, name, dim)


Summary: auto


Output:
Chunk ID: 95db0b
Wall time: 5.0009 seconds
Process running with session ID 12978
Original token count: 127
Output:
[2026-02-18 00:23:25] INFO tts_runtime.py:206: Initializing TTS pipeline...
[2026-02-18 00:23:25] INFO tts_runtime.py:217: Initializing streaming pipeline...
[2026-02-18 00:23:26] INFO long_text_processor.py:112: LongTextProcessor initialized
[2026-02-18 00:23:26] INFO tts_runtime.py:274: ✅ TTS runtime initialized in 32718ms
[2026-02-18 00:23:26] INFO tts_runtime.py:275:    Model version: LLM
[2026-02-18 00:23:26] INFO tts_runtime.py:286: Speaker globals cache startup precompute skipped (disabled)


**Planning benchmark polling**


Initialization is done; the actual concurrency sweep is running now. I’ll report the throughput curve as soon as all three levels complete.


Output:
Chunk ID: 748411
Wall time: 5.0008 seconds
Process running with session ID 12978
Original token count: 0
Output:


Output:
Chunk ID: da2f01
Wall time: 5.0011 seconds
Process running with session ID 12978
Original token count: 0
Output:


**Waiting for output evidence**


Output:
Chunk ID: 227812
Wall time: 3.7778 seconds
Process exited with code 0
Original token count: 1085
Output:
✅ vLLM engine initialized successfully!

🎉 Spark TTS Model ready for inference!

🎵 Loading BiCodec audio tokenizer from /home/ubuntu/veenaModal/models/spark_tts_4speaker...
   Device: cuda
Missing tensor: mel_transformer.spectrogram.window
Missing tensor: mel_transformer.mel_scale.fb
✅ BiCodec decoder initialized (sample rate: 16000Hz)
🚀 SparkTTSPipeline initialized
🌊 Veena3SlidingWindowPipeline initialized (sliding window: 28 tokens)

=== concurrency=96 requests=288 ===
success=288/288 (100%)
wall=5.80s throughput=49.64 req/s
latency_ms avg=1786 p50=1873 p95=2167 max=2167
generation_ms: avg=1459.40 p50=1414.00 p95=1949.00 max=2117.00
timeline_llm_first_batch_ms: avg=1447.55 p50=1399.56 p95=1936.94 max=2103.49
timeline_llm_done_ms: avg=1447.56 p50=1399.57 p95=1936.95 max=2103.50
timeline_parse_done_ms: avg=1447.63 p50=1399.63 p95=1937.01 max=2103.56
timeline_bicodec_done_ms: avg=1459.87 p50=1414.37 p95=1949.91 max=2117.73
timeline_total_ms: avg=1459.89 p50=1414.39 p95=1949.93 max=2117.75
llm_batch_wall_ms_total: avg=1447.42 p50=1399.43 p95=1936.82 max=2103.37
llm_parse_ms: avg=0.06 p50=0.06 p95=0.07 max=0.08
llm_time_per_token_ms: avg=11.31 p50=10.93 p95=15.13 max=16.43
bicodec_decode_wall_ms: avg=12.22 p50=12.40 p95=14.93 max=17.27
bicodec_decode_gpu_ms: avg=0.00 p50=0.00 p95=0.00 max=0.00
latency_over_timeline_ms: avg=325.83 p50=263.89 p95=788.25 max=911.44
latency_over_generation_ms: avg=326.32 p50=264.16 p95=788.64 max=911.64
timeline_share: llm=99.1% bicodec=0.8% other=0.0%
timeline_sample_ms: first_batch=1639.4 llm_done=1639.4 parse_done=1639.4 bicodec_done=1651.5 total=1651.5
gpu: util_avg=78% util_max=99% mem_avg=23601MB mem_max=23601MB

=== concurrency=128 requests=384 ===
success=384/384 (100%)
wall=7.41s throughput=51.81 req/s
latency_ms avg=2259 p50=2381 p95=2781 max=3542
generation_ms: avg=1771.72 p50=1707.00 p95=2520.00 max=2710.00
timeline_llm_first_batch_ms: avg=1760.20 p50=1695.65 p95=2505.50 max=2696.18
timeline_llm_done_ms: avg=1760.21 p50=1695.66 p95=2505.51 max=2696.19
timeline_parse_done_ms: avg=1760.27 p50=1695.72 p95=2505.57 max=2696.24
timeline_bicodec_done_ms: avg=1772.17 p50=1707.76 p95=2520.76 max=2710.79
timeline_total_ms: avg=1772.19 p50=1707.77 p95=2520.78 max=2710.81
llm_batch_wall_ms_total: avg=1760.07 p50=1695.53 p95=2505.38 max=2696.05
llm_parse_ms: avg=0.06 p50=0.05 p95=0.07 max=0.15
llm_time_per_token_ms: avg=13.75 p50=13.25 p95=19.57 max=21.06
bicodec_decode_wall_ms: avg=11.87 p50=12.12 p95=14.77 max=16.79
bicodec_decode_gpu_ms: avg=0.00 p50=0.00 p95=0.00 max=0.00
latency_over_timeline_ms: avg=486.33 p50=402.69 p95=1154.47 max=1874.62
latency_over_generation_ms: avg=486.80 p50=403.08 p95=1155.06 max=1874.99
timeline_share: llm=99.3% bicodec=0.7% other=0.0%
timeline_sample_ms: first_batch=1239.6 llm_done=1239.6 parse_done=1239.6 bicodec_done=1252.2 total=1252.2
gpu: util_avg=78% util_max=99% mem_avg=23604MB mem_max=23605MB

=== concurrency=160 requests=480 ===
success=480/480 (100%)
wall=9.05s throughput=53.05 req/s
latency_ms avg=2649 p50=2780 p95=3460 max=3460
generation_ms: avg=2000.92 p50=1894.00 p95=3016.00 max=3192.00
timeline_llm_first_batch_ms: avg=1988.98 p50=1881.22 p95=3005.47 max=3182.29
timeline_llm_done_ms: avg=1988.99 p50=1881.23 p95=3005.48 max=3182.30
timeline_parse_done_ms: avg=1989.05 p50=1881.29 p95=3005.55 max=3182.35
timeline_bicodec_done_ms: avg=2001.36 p50=1894.60 p95=3016.09 max=3192.10
timeline_total_ms: avg=2001.38 p50=1894.61 p95=3016.12 max=3192.12
llm_batch_wall_ms_total: avg=1988.85 p50=1881.10 p95=3005.32 max=3182.17
llm_parse_ms: avg=0.06 p50=0.06 p95=0.07 max=0.16
llm_time_per_token_ms: avg=15.54 p50=14.70 p95=23.48 max=24.86
bicodec_decode_wall_ms: avg=12.29 p50=12.14 p95=15.25 max=18.85
bicodec_decode_gpu_ms: avg=0.00 p50=0.00 p95=0.00 max=0.00
latency_over_timeline_ms: avg=647.22 p50=565.45 p95=1482.03 max=1674.83
latency_over_generation_ms: avg=647.69 p50=565.66 p95=1482.76 max=1675.74
timeline_share: llm=99.4% bicodec=0.6% other=0.0%
timeline_sample_ms: first_batch=2704.4 llm_done=2704.4 parse_done=2704.4 bicodec_done=2714.4 total=2714.4
gpu: util_avg=79% util_max=99% mem_avg=23605MB mem_max=23605MB

results saved: /tmp/runtime_levels_96_160_finalonly.json
ERROR 02-18 00:23:54 [core_client.py:605] Engine core proc EngineCore_DP0 died unexpectedly, shutting down client.


**Planning higher concurrency tests**


The sweep shows throughput still climbing at 160 concurrency, so I’m extending to 192/224/256 to locate the peak throughput region and saturation behavior.


cd /home/ubuntu/veenaModal && python3 scripts/stress_test_runtime_detailed.py --levels 192,224,256 --text short --max-tokens 128 --timeout 180 --requests-multiplier 2 --min-requests 8 --gpu-memory 0.25 --num-engines 1 --disable-engine-stats-logs --non-stream-final-only --output /tmp/runtime_levels_192_256_finalonly.json


Output:
Chunk ID: a108bc
Wall time: 1.0012 seconds
Process running with session ID 47309
Original token count: 0
Output:


**Scheduling regular polling**


Output:
Chunk ID: ae45e0
Wall time: 5.0011 seconds
Process running with session ID 47309
Original token count: 1589
Output:
[2026-02-18 00:24:10] INFO tts_runtime.py:183: Loading SparkTTS model with vLLM...
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
🚀 Initializing Spark TTS Model
📁 Model path: /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
🗣  Speakers: 12 predefined (lipakshi, vardan, reet, Nandini, krishna, anika, adarsh, Nilay, Aarvi, Asha, Bittu, Mira)
🎭 Emotions: 10 tags in [bracket] format

📝 Loading tokenizer from model...
✅ Tokenizer loaded: 165168 tokens

🔍 Validating 12 speakers for Spark TTS...
  lipakshi → <|speaker_0|>: 7 tokens
  vardan → <|speaker_1|>: 7 tokens
  reet → <|speaker_2|>: 7 tokens
  Nandini → <|speaker_3|>: 7 tokens
  krishna → <|speaker_4|>: 7 tokens
  anika → <|speaker_5|>: 7 tokens
  adarsh → <|speaker_6|>: 7 tokens
  Nilay → <|speaker_7|>: 7 tokens
  Aarvi → <|speaker_8|>: 7 tokens
  Asha → <|speaker_9|>: 7 tokens
  Bittu → <|speaker_10|>: 8 tokens
  Mira → <|speaker_11|>: 8 tokens
✅ All 12 speakers validated with mapping

🔍 Validating 10 emotion tags for Spark TTS...
✅ All 10 emotion tags validated

📋 Sample emotion tags:
  [angry]: 1 tokens
  [curious]: 1 tokens
  [excited]: 1 tokens
  [giggle]: 1 tokens
  [laughs harder]: 1 tokens
  ...

🔧 Initializing vLLM engine with optimizations...
  - dtype: bfloat16
  - max_model_len: 4096
  - gpu_memory_utilization: 0.25
  - enable_prefix_caching: True
  - enforce_eager (disable CUDA graphs): False
INFO 02-18 00:24:10 [model.py:541] Resolved architecture: Qwen2ForCausalLM
INFO 02-18 00:24:10 [model.py:1561] Using max model len 4096
INFO 02-18 00:24:11 [scheduler.py:226] Chunked prefill is enabled with max_num_batched_tokens=4096.
INFO 02-18 00:24:11 [vllm.py:624] Asynchronous scheduling is enabled.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(EngineCore_DP0 pid=4098980) INFO 02-18 00:24:11 [core.py:96] Initializing a V1 LLM engine (v0.15.1) with config: model='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', speculative_config=None, tokenizer='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [4096], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 256, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': True}, 'local_cache_dir': None, 'static_all_moe_layers': []}
(EngineCore_DP0 pid=4098980) INFO 02-18 00:24:14 [parallel_state.py:1212] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://216.81.248.184:48465 backend=nccl
(EngineCore_DP0 pid=4098980) INFO 02-18 00:24:14 [parallel_state.py:1423] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
(EngineCore_DP0 pid=4098980) INFO 02-18 00:24:14 [gpu_model_runner.py:4033] Starting to load model /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM...
(EngineCore_DP0 pid=4098980) INFO 02-18 00:24:15 [cuda.py:364] Using FLASH_ATTN attention backend out of potential backends: ('FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION')
(EngineCore_DP0 pid=4098980) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
(EngineCore_DP0 pid=4098980) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
(EngineCore_DP0 pid=4098980)
Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]


Output:
Chunk ID: 0fe8b6
Wall time: 5.0006 seconds
Process running with session ID 47309
Original token count: 199
Output:
(EngineCore_DP0 pid=4098980)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.26it/s]
(EngineCore_DP0 pid=4098980)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.26it/s]
(EngineCore_DP0 pid=4098980)
(EngineCore_DP0 pid=4098980) INFO 02-18 00:24:16 [default_loader.py:291] Loading weights took 0.87 seconds
(EngineCore_DP0 pid=4098980) INFO 02-18 00:24:17 [gpu_model_runner.py:4130] Model loading took 0.95 GiB memory and 1.387151 seconds
(EngineCore_DP0 pid=4098980) INFO 02-18 00:24:21 [backends.py:812] Using cache directory: /home/ubuntu/.cache/vllm/torch_compile_cache/f3ff13511c/rank_0_0/backbone for vLLM's torch.compile
(EngineCore_DP0 pid=4098980) INFO 02-18 00:24:21 [backends.py:872] Dynamo bytecode transform time: 4.17 s


**Planning periodic updates**


The extended sweep is still active; once this finishes I’ll pinpoint the best concurrency operating point and where token-time starts degrading too sharply.


Output:
Chunk ID: fc56f5
Wall time: 5.0011 seconds
Process running with session ID 47309
Original token count: 787
Output:
(EngineCore_DP0 pid=4098980) INFO 02-18 00:24:24 [backends.py:267] Directly load the compiled graph(s) for compile range (1, 4096) from the cache, took 0.652 s
(EngineCore_DP0 pid=4098980) INFO 02-18 00:24:24 [monitor.py:34] torch.compile takes 4.82 s in total
(EngineCore_DP0 pid=4098980) INFO 02-18 00:24:25 [gpu_worker.py:356] Available KV cache memory: 18.06 GiB
(EngineCore_DP0 pid=4098980) INFO 02-18 00:24:25 [kv_cache_utils.py:1307] GPU KV cache size: 1,577,712 tokens
(EngineCore_DP0 pid=4098980) INFO 02-18 00:24:25 [kv_cache_utils.py:1312] Maximum concurrency for 4,096 tokens per request: 385.18x
(EngineCore_DP0 pid=4098980)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   0%|          | 0/35 [00:00<?, ?it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  11%|█▏        | 4/35 [00:00<00:00, 32.71it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  23%|██▎       | 8/35 [00:00<00:00, 35.57it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  34%|███▍      | 12/35 [00:00<00:00, 36.51it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  46%|████▌     | 16/35 [00:00<00:00, 36.84it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  57%|█████▋    | 20/35 [00:00<00:00, 37.26it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  69%|██████▊   | 24/35 [00:00<00:00, 37.52it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  80%|████████  | 28/35 [00:00<00:00, 37.32it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  91%|█████████▏| 32/35 [00:00<00:00, 35.56it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 35/35 [00:00<00:00, 35.50it/s]
(EngineCore_DP0 pid=4098980)
Capturing CUDA graphs (decode, FULL):   0%|          | 0/19 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL):  16%|█▌        | 3/19 [00:00<00:00, 29.80it/s]
Capturing CUDA graphs (decode, FULL):  37%|███▋      | 7/19 [00:00<00:00, 34.07it/s]
Capturing CUDA graphs (decode, FULL):  58%|█████▊    | 11/19 [00:00<00:00, 35.36it/s]
Capturing CUDA graphs (decode, FULL):  79%|███████▉  | 15/19 [00:00<00:00, 35.67it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 36.21it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 35.42it/s]
(EngineCore_DP0 pid=4098980) INFO 02-18 00:24:27 [gpu_model_runner.py:5063] Graph capturing finished in 2 secs, took 0.28 GiB
(EngineCore_DP0 pid=4098980) INFO 02-18 00:24:27 [core.py:272] init engine (profile, create kv cache, warmup model) took 10.71 seconds
[2026-02-18 00:24:29] INFO tts_runtime.py:192: Initializing prompt builder...
[2026-02-18 00:24:29] INFO tts_runtime.py:199: Initializing BiCodec decoder...
/home/ubuntu/.local/lib/python3.10/site-packages/torch/nn/utils/weight_norm.py:144: FutureWarning: `torch.nn.utils.weight_norm` is deprecated in favor of `torch.nn.utils.parametrizations.weight_norm`.
  WeightNorm.apply(module, name, dim)


Output:
Chunk ID: 61e9d7
Wall time: 5.0006 seconds
Process running with session ID 47309
Original token count: 127
Output:
[2026-02-18 00:24:34] INFO tts_runtime.py:206: Initializing TTS pipeline...
[2026-02-18 00:24:34] INFO tts_runtime.py:217: Initializing streaming pipeline...
[2026-02-18 00:24:34] INFO long_text_processor.py:112: LongTextProcessor initialized
[2026-02-18 00:24:34] INFO tts_runtime.py:274: ✅ TTS runtime initialized in 29808ms
[2026-02-18 00:24:34] INFO tts_runtime.py:275:    Model version: LLM
[2026-02-18 00:24:34] INFO tts_runtime.py:286: Speaker globals cache startup precompute skipped (disabled)


Output:
Chunk ID: 2249b3
Wall time: 5.0013 seconds
Process running with session ID 47309
Original token count: 0
Output:


Output:
Chunk ID: 984917
Wall time: 5.0008 seconds
Process running with session ID 47309
Original token count: 0
Output:


Output:
Chunk ID: ab352b
Wall time: 2.0734 seconds
Process exited with code 0
Original token count: 1087
Output:
✅ vLLM engine initialized successfully!

🎉 Spark TTS Model ready for inference!

🎵 Loading BiCodec audio tokenizer from /home/ubuntu/veenaModal/models/spark_tts_4speaker...
   Device: cuda
Missing tensor: mel_transformer.spectrogram.window
Missing tensor: mel_transformer.mel_scale.fb
✅ BiCodec decoder initialized (sample rate: 16000Hz)
🚀 SparkTTSPipeline initialized
🌊 Veena3SlidingWindowPipeline initialized (sliding window: 28 tokens)

=== concurrency=192 requests=384 ===
success=384/384 (100%)
wall=6.98s throughput=54.98 req/s
latency_ms avg=3059 p50=2902 p95=3891 max=3891
generation_ms: avg=2290.92 p50=2306.00 p95=3321.00 max=3516.00
timeline_llm_first_batch_ms: avg=2279.31 p50=2296.79 p95=3311.50 max=3506.41
timeline_llm_done_ms: avg=2279.32 p50=2296.79 p95=3311.51 max=3506.42
timeline_parse_done_ms: avg=2279.38 p50=2296.85 p95=3311.56 max=3506.48
timeline_bicodec_done_ms: avg=2291.36 p50=2306.71 p95=3321.47 max=3516.42
timeline_total_ms: avg=2291.38 p50=2306.72 p95=3321.49 max=3516.44
llm_batch_wall_ms_total: avg=2279.18 p50=2296.67 p95=3311.38 max=3506.10
llm_parse_ms: avg=0.06 p50=0.05 p95=0.07 max=0.11
llm_time_per_token_ms: avg=17.81 p50=17.94 p95=25.87 max=27.39
bicodec_decode_wall_ms: avg=11.95 p50=11.34 p95=15.68 max=18.28
bicodec_decode_gpu_ms: avg=0.00 p50=0.00 p95=0.00 max=0.00
latency_over_timeline_ms: avg=767.70 p50=665.82 p95=1892.18 max=2153.90
latency_over_generation_ms: avg=768.16 p50=666.12 p95=1892.61 max=2154.71
timeline_share: llm=99.5% bicodec=0.5% other=0.0%
timeline_sample_ms: first_batch=2745.4 llm_done=2745.4 parse_done=2745.5 bicodec_done=2755.4 total=2755.4
gpu: util_avg=75% util_max=100% mem_avg=23605MB mem_max=23605MB

=== concurrency=224 requests=448 ===
success=448/448 (100%)
wall=7.64s throughput=58.67 req/s
latency_ms avg=3225 p50=3197 p95=4285 max=4285
generation_ms: avg=2368.04 p50=2298.00 p95=3599.00 max=3823.00
timeline_llm_first_batch_ms: avg=2355.51 p50=2280.59 p95=3589.76 max=3813.50
timeline_llm_done_ms: avg=2355.52 p50=2280.60 p95=3589.77 max=3813.51
timeline_parse_done_ms: avg=2355.58 p50=2280.66 p95=3589.82 max=3813.57
timeline_bicodec_done_ms: avg=2368.50 p50=2297.99 p95=3599.83 max=3823.38
timeline_total_ms: avg=2368.52 p50=2298.01 p95=3599.85 max=3823.40
llm_batch_wall_ms_total: avg=2355.37 p50=2280.46 p95=3589.62 max=3813.38
llm_parse_ms: avg=0.06 p50=0.06 p95=0.07 max=0.10
llm_time_per_token_ms: avg=18.40 p50=17.82 p95=28.04 max=29.79
bicodec_decode_wall_ms: avg=12.90 p50=13.15 p95=16.10 max=19.14
bicodec_decode_gpu_ms: avg=0.00 p50=0.00 p95=0.00 max=0.00
latency_over_timeline_ms: avg=856.00 p50=867.67 p95=1532.90 max=1857.63
latency_over_generation_ms: avg=856.47 p50=868.35 p95=1533.35 max=1858.43
timeline_share: llm=99.4% bicodec=0.5% other=0.0%
timeline_sample_ms: first_batch=2842.3 llm_done=2842.3 parse_done=2842.4 bicodec_done=2860.4 total=2860.5
gpu: util_avg=84% util_max=99% mem_avg=23605MB mem_max=23605MB

=== concurrency=256 requests=512 ===
success=512/512 (100%)
wall=8.62s throughput=59.43 req/s
latency_ms avg=3732 p50=3984 p95=4610 max=4610
generation_ms: avg=2752.71 p50=2741.00 p95=4189.00 max=4446.00
timeline_llm_first_batch_ms: avg=2740.51 p50=2731.65 p95=4179.28 max=4435.84
timeline_llm_done_ms: avg=2740.52 p50=2731.66 p95=4179.28 max=4435.84
timeline_parse_done_ms: avg=2740.58 p50=2731.71 p95=4179.34 max=4435.89
timeline_bicodec_done_ms: avg=2753.13 p50=2741.76 p95=4189.14 max=4446.00
timeline_total_ms: avg=2753.15 p50=2741.77 p95=4189.17 max=4446.02
llm_batch_wall_ms_total: avg=2740.38 p50=2731.52 p95=4179.15 max=4435.72
llm_parse_ms: avg=0.05 p50=0.05 p95=0.07 max=0.10
llm_time_per_token_ms: avg=21.41 p50=21.34 p95=32.65 max=34.65
bicodec_decode_wall_ms: avg=12.54 p50=12.85 p95=16.11 max=19.52
bicodec_decode_gpu_ms: avg=0.00 p50=0.00 p95=0.00 max=0.00
latency_over_timeline_ms: avg=979.33 p50=868.17 p95=2165.23 max=2529.78
latency_over_generation_ms: avg=979.77 p50=868.89 p95=2165.24 max=2530.25
timeline_share: llm=99.5% bicodec=0.5% other=0.0%
timeline_sample_ms: first_batch=3234.3 llm_done=3234.3 parse_done=3234.4 bicodec_done=3244.3 total=3244.3
gpu: util_avg=79% util_max=97% mem_avg=23605MB mem_max=23605MB

results saved: /tmp/runtime_levels_192_256_finalonly.json
ERROR 02-18 00:25:03 [core_client.py:605] Engine core proc EngineCore_DP0 died unexpectedly, shutting down client.


**Planning extended throughput tests**


Throughput is still rising at 256, so I’m doing one more sweep at 288/320 to find where it plateaus and to quantify the latency tradeoff at peak load.


cd /home/ubuntu/veenaModal && python3 scripts/stress_test_runtime_detailed.py --levels 288,320 --text short --max-tokens 128 --timeout 180 --requests-multiplier 2 --min-requests 8 --gpu-memory 0.25 --num-engines 1 --disable-engine-stats-logs --non-stream-final-only --output /tmp/runtime_levels_288_320_finalonly.json


Output:
Chunk ID: 271e42
Wall time: 1.0005 seconds
Process running with session ID 71214
Original token count: 0
Output:


Output:
Chunk ID: 357901
Wall time: 5.0009 seconds
Process running with session ID 71214
Original token count: 1277
Output:
[2026-02-18 00:25:19] INFO tts_runtime.py:183: Loading SparkTTS model with vLLM...
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
🚀 Initializing Spark TTS Model
📁 Model path: /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
🗣  Speakers: 12 predefined (lipakshi, vardan, reet, Nandini, krishna, anika, adarsh, Nilay, Aarvi, Asha, Bittu, Mira)
🎭 Emotions: 10 tags in [bracket] format

📝 Loading tokenizer from model...
✅ Tokenizer loaded: 165168 tokens

🔍 Validating 12 speakers for Spark TTS...
  lipakshi → <|speaker_0|>: 7 tokens
  vardan → <|speaker_1|>: 7 tokens
  reet → <|speaker_2|>: 7 tokens
  Nandini → <|speaker_3|>: 7 tokens
  krishna → <|speaker_4|>: 7 tokens
  anika → <|speaker_5|>: 7 tokens
  adarsh → <|speaker_6|>: 7 tokens
  Nilay → <|speaker_7|>: 7 tokens
  Aarvi → <|speaker_8|>: 7 tokens
  Asha → <|speaker_9|>: 7 tokens
  Bittu → <|speaker_10|>: 8 tokens
  Mira → <|speaker_11|>: 8 tokens
✅ All 12 speakers validated with mapping

🔍 Validating 10 emotion tags for Spark TTS...
✅ All 10 emotion tags validated

📋 Sample emotion tags:
  [angry]: 1 tokens
  [curious]: 1 tokens
  [excited]: 1 tokens
  [giggle]: 1 tokens
  [laughs harder]: 1 tokens
  ...

🔧 Initializing vLLM engine with optimizations...
  - dtype: bfloat16
  - max_model_len: 4096
  - gpu_memory_utilization: 0.25
  - enable_prefix_caching: True
  - enforce_eager (disable CUDA graphs): False
INFO 02-18 00:25:19 [model.py:541] Resolved architecture: Qwen2ForCausalLM
INFO 02-18 00:25:19 [model.py:1561] Using max model len 4096
INFO 02-18 00:25:20 [scheduler.py:226] Chunked prefill is enabled with max_num_batched_tokens=4096.
INFO 02-18 00:25:20 [vllm.py:624] Asynchronous scheduling is enabled.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(EngineCore_DP0 pid=4099802) INFO 02-18 00:25:20 [core.py:96] Initializing a V1 LLM engine (v0.15.1) with config: model='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', speculative_config=None, tokenizer='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [4096], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 256, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': True}, 'local_cache_dir': None, 'static_all_moe_layers': []}


Output:
Chunk ID: 9e46f8
Wall time: 5.0010 seconds
Process running with session ID 71214
Original token count: 663
Output:
(EngineCore_DP0 pid=4099802) INFO 02-18 00:25:23 [parallel_state.py:1212] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://216.81.248.184:44403 backend=nccl
(EngineCore_DP0 pid=4099802) INFO 02-18 00:25:23 [parallel_state.py:1423] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
(EngineCore_DP0 pid=4099802) INFO 02-18 00:25:23 [gpu_model_runner.py:4033] Starting to load model /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM...
(EngineCore_DP0 pid=4099802) INFO 02-18 00:25:24 [cuda.py:364] Using FLASH_ATTN attention backend out of potential backends: ('FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION')
(EngineCore_DP0 pid=4099802) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
(EngineCore_DP0 pid=4099802) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
(EngineCore_DP0 pid=4099802)
Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
(EngineCore_DP0 pid=4099802)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.36it/s]
(EngineCore_DP0 pid=4099802)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.36it/s]
(EngineCore_DP0 pid=4099802)
(EngineCore_DP0 pid=4099802) INFO 02-18 00:25:25 [default_loader.py:291] Loading weights took 0.81 seconds
(EngineCore_DP0 pid=4099802) INFO 02-18 00:25:26 [gpu_model_runner.py:4130] Model loading took 0.95 GiB memory and 1.338979 seconds
(EngineCore_DP0 pid=4099802) INFO 02-18 00:25:30 [backends.py:812] Using cache directory: /home/ubuntu/.cache/vllm/torch_compile_cache/f3ff13511c/rank_0_0/backbone for vLLM's torch.compile
(EngineCore_DP0 pid=4099802) INFO 02-18 00:25:30 [backends.py:872] Dynamo bytecode transform time: 4.26 s
(EngineCore_DP0 pid=4099802) INFO 02-18 00:25:33 [backends.py:267] Directly load the compiled graph(s) for compile range (1, 4096) from the cache, took 0.665 s
(EngineCore_DP0 pid=4099802) INFO 02-18 00:25:33 [monitor.py:34] torch.compile takes 4.93 s in total
(EngineCore_DP0 pid=4099802) INFO 02-18 00:25:34 [gpu_worker.py:356] Available KV cache memory: 18.06 GiB
(EngineCore_DP0 pid=4099802) INFO 02-18 00:25:34 [kv_cache_utils.py:1307] GPU KV cache size: 1,577,712 tokens
(EngineCore_DP0 pid=4099802) INFO 02-18 00:25:34 [kv_cache_utils.py:1312] Maximum concurrency for 4,096 tokens per request: 385.18x


**Updating commentary on running state**


This final sweep is still in progress; once complete, we’ll have the full throughput/latency frontier and can choose the practical concurrency target.


Output:
Chunk ID: 3c01a5
Wall time: 5.0019 seconds
Process running with session ID 71214
Original token count: 635
Output:
(EngineCore_DP0 pid=4099802)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   0%|          | 0/35 [00:00<?, ?it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  11%|█▏        | 4/35 [00:00<00:00, 32.90it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  23%|██▎       | 8/35 [00:00<00:00, 35.20it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  34%|███▍      | 12/35 [00:00<00:00, 36.13it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  46%|████▌     | 16/35 [00:00<00:00, 36.35it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  57%|█████▋    | 20/35 [00:00<00:00, 36.57it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  69%|██████▊   | 24/35 [00:00<00:00, 36.67it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  80%|████████  | 28/35 [00:00<00:00, 36.51it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  91%|█████████▏| 32/35 [00:00<00:00, 34.80it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 35/35 [00:01<00:00, 34.82it/s]
(EngineCore_DP0 pid=4099802)
Capturing CUDA graphs (decode, FULL):   0%|          | 0/19 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL):  16%|█▌        | 3/19 [00:00<00:00, 28.77it/s]
Capturing CUDA graphs (decode, FULL):  37%|███▋      | 7/19 [00:00<00:00, 33.26it/s]
Capturing CUDA graphs (decode, FULL):  58%|█████▊    | 11/19 [00:00<00:00, 34.86it/s]
Capturing CUDA graphs (decode, FULL):  79%|███████▉  | 15/19 [00:00<00:00, 35.33it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 35.95it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 35.01it/s]
(EngineCore_DP0 pid=4099802) INFO 02-18 00:25:37 [gpu_model_runner.py:5063] Graph capturing finished in 2 secs, took 0.28 GiB
(EngineCore_DP0 pid=4099802) INFO 02-18 00:25:37 [core.py:272] init engine (profile, create kv cache, warmup model) took 10.94 seconds
[2026-02-18 00:25:38] INFO tts_runtime.py:192: Initializing prompt builder...
[2026-02-18 00:25:38] INFO tts_runtime.py:199: Initializing BiCodec decoder...
/home/ubuntu/.local/lib/python3.10/site-packages/torch/nn/utils/weight_norm.py:144: FutureWarning: `torch.nn.utils.weight_norm` is deprecated in favor of `torch.nn.utils.parametrizations.weight_norm`.
  WeightNorm.apply(module, name, dim)


Output:
Chunk ID: ed3242
Wall time: 5.0006 seconds
Process running with session ID 71214
Original token count: 127
Output:
[2026-02-18 00:25:45] INFO tts_runtime.py:206: Initializing TTS pipeline...
[2026-02-18 00:25:45] INFO tts_runtime.py:217: Initializing streaming pipeline...
[2026-02-18 00:25:45] INFO long_text_processor.py:112: LongTextProcessor initialized
[2026-02-18 00:25:45] INFO tts_runtime.py:274: ✅ TTS runtime initialized in 31797ms
[2026-02-18 00:25:45] INFO tts_runtime.py:275:    Model version: LLM
[2026-02-18 00:25:45] INFO tts_runtime.py:286: Speaker globals cache startup precompute skipped (disabled)


Output:
Chunk ID: e280c6
Wall time: 5.0012 seconds
Process running with session ID 71214
Original token count: 0
Output:


Output:
Chunk ID: b63a4e
Wall time: 5.0018 seconds
Process running with session ID 71214
Original token count: 0
Output:


Output:
Chunk ID: 54e367
Wall time: 0.0509 seconds
Process exited with code 0
Original token count: 779
Output:
✅ vLLM engine initialized successfully!

🎉 Spark TTS Model ready for inference!

🎵 Loading BiCodec audio tokenizer from /home/ubuntu/veenaModal/models/spark_tts_4speaker...
   Device: cuda
Missing tensor: mel_transformer.spectrogram.window
Missing tensor: mel_transformer.mel_scale.fb
✅ BiCodec decoder initialized (sample rate: 16000Hz)
🚀 SparkTTSPipeline initialized
🌊 Veena3SlidingWindowPipeline initialized (sliding window: 28 tokens)

=== concurrency=288 requests=576 ===
success=576/576 (100%)
wall=9.64s throughput=59.73 req/s
latency_ms avg=4071 p50=4260 p95=5910 max=5910
generation_ms: avg=2918.75 p50=2829.00 p95=4979.00 max=5395.00
timeline_llm_first_batch_ms: avg=2905.96 p50=2817.61 p95=4964.53 max=5378.57
timeline_llm_done_ms: avg=2905.97 p50=2817.62 p95=4964.54 max=5378.58
timeline_parse_done_ms: avg=2906.03 p50=2817.67 p95=4964.61 max=5378.64
timeline_bicodec_done_ms: avg=2919.22 p50=2829.79 p95=4979.90 max=5395.66
timeline_total_ms: avg=2919.24 p50=2829.81 p95=4979.92 max=5395.68
llm_batch_wall_ms_total: avg=2905.32 p50=2817.47 p95=4964.41 max=5378.45
llm_parse_ms: avg=0.06 p50=0.06 p95=0.07 max=0.09
llm_time_per_token_ms: avg=22.70 p50=22.01 p95=38.78 max=42.02
bicodec_decode_wall_ms: avg=13.17 p50=13.75 p95=16.70 max=19.86
bicodec_decode_gpu_ms: avg=0.00 p50=0.00 p95=0.00 max=0.00
latency_over_timeline_ms: avg=1151.72 p50=1075.30 p95=2503.82 max=2950.51
latency_over_generation_ms: avg=1152.20 p50=1075.54 p95=2503.99 max=2951.14
timeline_share: llm=99.5% bicodec=0.5% other=0.0%
timeline_sample_ms: first_batch=2901.5 llm_done=2901.5 parse_done=2901.6 bicodec_done=2911.4 total=2911.5
gpu: util_avg=83% util_max=98% mem_avg=23605MB mem_max=23605MB

=== concurrency=320 requests=640 ===
success=640/640 (100%)
wall=10.34s throughput=61.91 req/s
latency_ms avg=4329 p50=4605 p95=6604 max=6604
generation_ms: avg=3089.47 p50=2717.00 p95=5328.00 max=5763.00
timeline_llm_first_batch_ms: avg=3076.62 p50=2703.51 p95=5313.65 max=5749.64
timeline_llm_done_ms: avg=3076.63 p50=2703.52 p95=5313.66 max=5749.65
timeline_parse_done_ms: avg=3076.68 p50=2703.58 p95=5313.71 max=5749.71
timeline_bicodec_done_ms: avg=3089.91 p50=2717.37 p95=5328.41 max=5763.19
timeline_total_ms: avg=3089.93 p50=2717.39 p95=5328.42 max=5763.20
llm_batch_wall_ms_total: avg=3076.49 p50=2703.39 p95=5313.53 max=5749.52
llm_parse_ms: avg=0.05 p50=0.05 p95=0.07 max=0.26
llm_time_per_token_ms: avg=24.04 p50=21.12 p95=41.51 max=44.92
bicodec_decode_wall_ms: avg=13.21 p50=13.77 p95=16.61 max=22.52
bicodec_decode_gpu_ms: avg=0.00 p50=0.00 p95=0.00 max=0.00
latency_over_timeline_ms: avg=1238.75 p50=1262.88 p95=2449.70 max=2805.45
latency_over_generation_ms: avg=1239.21 p50=1263.48 p95=2449.79 max=2805.95
timeline_share: llm=99.6% bicodec=0.4% other=0.0%
timeline_sample_ms: first_batch=4382.7 llm_done=4382.7 parse_done=4382.8 bicodec_done=4392.7 total=4392.7
gpu: util_avg=79% util_max=98% mem_avg=23605MB mem_max=23605MB

results saved: /tmp/runtime_levels_288_320_finalonly.json
ERROR 02-18 00:26:09 [core_client.py:605] Engine core proc EngineCore_DP0 died unexpectedly, shutting down client.


**Testing peak throughput limits**


Throughput is still increasing at 320, so I’m running one final edge sweep near KV concurrency limits (352/384) to find the practical maximum on this GPU setup.


cd /home/ubuntu/veenaModal && python3 scripts/stress_test_runtime_detailed.py --levels 352,384 --text short --max-tokens 128 --timeout 240 --requests-multiplier 1 --min-requests 8 --gpu-memory 0.25 --num-engines 1 --disable-engine-stats-logs --non-stream-final-only --output /tmp/runtime_levels_352_384_finalonly.json


Output:
Chunk ID: 9440e8
Wall time: 1.0011 seconds
Process running with session ID 23052
Original token count: 0
Output:


Summary: auto


Output:
Chunk ID: 0fd322
Wall time: 5.0007 seconds
Process running with session ID 23052
Original token count: 1681
Output:
[2026-02-18 00:26:26] INFO tts_runtime.py:183: Loading SparkTTS model with vLLM...
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
🚀 Initializing Spark TTS Model
📁 Model path: /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
🗣  Speakers: 12 predefined (lipakshi, vardan, reet, Nandini, krishna, anika, adarsh, Nilay, Aarvi, Asha, Bittu, Mira)
🎭 Emotions: 10 tags in [bracket] format

📝 Loading tokenizer from model...
✅ Tokenizer loaded: 165168 tokens

🔍 Validating 12 speakers for Spark TTS...
  lipakshi → <|speaker_0|>: 7 tokens
  vardan → <|speaker_1|>: 7 tokens
  reet → <|speaker_2|>: 7 tokens
  Nandini → <|speaker_3|>: 7 tokens
  krishna → <|speaker_4|>: 7 tokens
  anika → <|speaker_5|>: 7 tokens
  adarsh → <|speaker_6|>: 7 tokens
  Nilay → <|speaker_7|>: 7 tokens
  Aarvi → <|speaker_8|>: 7 tokens
  Asha → <|speaker_9|>: 7 tokens
  Bittu → <|speaker_10|>: 8 tokens
  Mira → <|speaker_11|>: 8 tokens
✅ All 12 speakers validated with mapping

🔍 Validating 10 emotion tags for Spark TTS...
✅ All 10 emotion tags validated

📋 Sample emotion tags:
  [angry]: 1 tokens
  [curious]: 1 tokens
  [excited]: 1 tokens
  [giggle]: 1 tokens
  [laughs harder]: 1 tokens
  ...

🔧 Initializing vLLM engine with optimizations...
  - dtype: bfloat16
  - max_model_len: 4096
  - gpu_memory_utilization: 0.25
  - enable_prefix_caching: True
  - enforce_eager (disable CUDA graphs): False
INFO 02-18 00:26:27 [model.py:541] Resolved architecture: Qwen2ForCausalLM
INFO 02-18 00:26:27 [model.py:1561] Using max model len 4096
INFO 02-18 00:26:27 [scheduler.py:226] Chunked prefill is enabled with max_num_batched_tokens=4096.
INFO 02-18 00:26:27 [vllm.py:624] Asynchronous scheduling is enabled.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(EngineCore_DP0 pid=4100396) INFO 02-18 00:26:28 [core.py:96] Initializing a V1 LLM engine (v0.15.1) with config: model='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', speculative_config=None, tokenizer='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [4096], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 256, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': True}, 'local_cache_dir': None, 'static_all_moe_layers': []}
(EngineCore_DP0 pid=4100396) INFO 02-18 00:26:30 [parallel_state.py:1212] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://216.81.248.184:36531 backend=nccl
(EngineCore_DP0 pid=4100396) INFO 02-18 00:26:30 [parallel_state.py:1423] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
(EngineCore_DP0 pid=4100396) INFO 02-18 00:26:31 [gpu_model_runner.py:4033] Starting to load model /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM...
(EngineCore_DP0 pid=4100396) INFO 02-18 00:26:31 [cuda.py:364] Using FLASH_ATTN attention backend out of potential backends: ('FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION')
(EngineCore_DP0 pid=4100396) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
(EngineCore_DP0 pid=4100396) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
(EngineCore_DP0 pid=4100396)
Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
(EngineCore_DP0 pid=4100396)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.99it/s]
(EngineCore_DP0 pid=4100396)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.98it/s]
(EngineCore_DP0 pid=4100396)
(EngineCore_DP0 pid=4100396) INFO 02-18 00:26:32 [default_loader.py:291] Loading weights took 0.58 seconds


Output:
Chunk ID: d596e6
Wall time: 5.0015 seconds
Process running with session ID 23052
Original token count: 172
Output:
(EngineCore_DP0 pid=4100396) INFO 02-18 00:26:33 [gpu_model_runner.py:4130] Model loading took 0.95 GiB memory and 1.118116 seconds
(EngineCore_DP0 pid=4100396) INFO 02-18 00:26:37 [backends.py:812] Using cache directory: /home/ubuntu/.cache/vllm/torch_compile_cache/f3ff13511c/rank_0_0/backbone for vLLM's torch.compile
(EngineCore_DP0 pid=4100396) INFO 02-18 00:26:37 [backends.py:872] Dynamo bytecode transform time: 4.21 s
(EngineCore_DP0 pid=4100396) INFO 02-18 00:26:40 [backends.py:267] Directly load the compiled graph(s) for compile range (1, 4096) from the cache, took 0.667 s
(EngineCore_DP0 pid=4100396) INFO 02-18 00:26:40 [monitor.py:34] torch.compile takes 4.88 s in total


This near-limit run is still active; after it completes we’ll know if throughput keeps climbing beyond 320 or if we’re at/near the ceiling.


Output:
Chunk ID: 642717
Wall time: 5.0012 seconds
Process running with session ID 23052
Original token count: 751
Output:
(EngineCore_DP0 pid=4100396) INFO 02-18 00:26:41 [gpu_worker.py:356] Available KV cache memory: 18.06 GiB
(EngineCore_DP0 pid=4100396) INFO 02-18 00:26:41 [kv_cache_utils.py:1307] GPU KV cache size: 1,577,712 tokens
(EngineCore_DP0 pid=4100396) INFO 02-18 00:26:41 [kv_cache_utils.py:1312] Maximum concurrency for 4,096 tokens per request: 385.18x
(EngineCore_DP0 pid=4100396)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   0%|          | 0/35 [00:00<?, ?it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   9%|▊         | 3/35 [00:00<00:01, 29.42it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  20%|██        | 7/35 [00:00<00:00, 33.62it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  31%|███▏      | 11/35 [00:00<00:00, 35.21it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  43%|████▎     | 15/35 [00:00<00:00, 34.77it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  54%|█████▍    | 19/35 [00:00<00:00, 35.01it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  66%|██████▌   | 23/35 [00:00<00:00, 35.64it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  77%|███████▋  | 27/35 [00:00<00:00, 35.16it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  89%|████████▊ | 31/35 [00:00<00:00, 32.91it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 35/35 [00:01<00:00, 31.75it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 35/35 [00:01<00:00, 33.36it/s]
(EngineCore_DP0 pid=4100396)
Capturing CUDA graphs (decode, FULL):   0%|          | 0/19 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL):  16%|█▌        | 3/19 [00:00<00:00, 28.74it/s]
Capturing CUDA graphs (decode, FULL):  37%|███▋      | 7/19 [00:00<00:00, 32.32it/s]
Capturing CUDA graphs (decode, FULL):  58%|█████▊    | 11/19 [00:00<00:00, 33.25it/s]
Capturing CUDA graphs (decode, FULL):  79%|███████▉  | 15/19 [00:00<00:00, 32.85it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 33.88it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 33.18it/s]
(EngineCore_DP0 pid=4100396) INFO 02-18 00:26:44 [gpu_model_runner.py:5063] Graph capturing finished in 3 secs, took 0.28 GiB
(EngineCore_DP0 pid=4100396) INFO 02-18 00:26:44 [core.py:272] init engine (profile, create kv cache, warmup model) took 11.10 seconds
[2026-02-18 00:26:45] INFO tts_runtime.py:192: Initializing prompt builder...
[2026-02-18 00:26:45] INFO tts_runtime.py:199: Initializing BiCodec decoder...
/home/ubuntu/.local/lib/python3.10/site-packages/torch/nn/utils/weight_norm.py:144: FutureWarning: `torch.nn.utils.weight_norm` is deprecated in favor of `torch.nn.utils.parametrizations.weight_norm`.
  WeightNorm.apply(module, name, dim)


Output:
Chunk ID: 64c654
Wall time: 5.0012 seconds
Process running with session ID 23052
Original token count: 127
Output:
[2026-02-18 00:26:51] INFO tts_runtime.py:206: Initializing TTS pipeline...
[2026-02-18 00:26:51] INFO tts_runtime.py:217: Initializing streaming pipeline...
[2026-02-18 00:26:51] INFO long_text_processor.py:112: LongTextProcessor initialized
[2026-02-18 00:26:51] INFO tts_runtime.py:274: ✅ TTS runtime initialized in 30191ms
[2026-02-18 00:26:51] INFO tts_runtime.py:275:    Model version: LLM
[2026-02-18 00:26:51] INFO tts_runtime.py:286: Speaker globals cache startup precompute skipped (disabled)


Output:
Chunk ID: 04b180
Wall time: 5.0001 seconds
Process running with session ID 23052
Original token count: 0
Output:


Output:
Chunk ID: 7dc6f4
Wall time: 0.0506 seconds
Process exited with code 0
Original token count: 778
Output:
✅ vLLM engine initialized successfully!

🎉 Spark TTS Model ready for inference!

🎵 Loading BiCodec audio tokenizer from /home/ubuntu/veenaModal/models/spark_tts_4speaker...
   Device: cuda
Missing tensor: mel_transformer.spectrogram.window
Missing tensor: mel_transformer.mel_scale.fb
✅ BiCodec decoder initialized (sample rate: 16000Hz)
🚀 SparkTTSPipeline initialized
🌊 Veena3SlidingWindowPipeline initialized (sliding window: 28 tokens)

=== concurrency=352 requests=352 ===
success=352/352 (100%)
wall=5.82s throughput=60.51 req/s
latency_ms avg=4339 p50=4781 p95=5811 max=5811
generation_ms: avg=3625.67 p50=3820.00 p95=5481.00 max=5657.00
timeline_llm_first_batch_ms: avg=3613.37 p50=3805.21 p95=5471.15 max=5647.34
timeline_llm_done_ms: avg=3613.38 p50=3805.21 p95=5471.16 max=5647.35
timeline_parse_done_ms: avg=3613.44 p50=3805.27 p95=5471.21 max=5647.58
timeline_bicodec_done_ms: avg=3626.13 p50=3820.32 p95=5481.24 max=5657.75
timeline_total_ms: avg=3626.15 p50=3820.34 p95=5481.26 max=5657.76
llm_batch_wall_ms_total: avg=3613.24 p50=3805.07 p95=5471.03 max=5647.20
llm_parse_ms: avg=0.06 p50=0.05 p95=0.07 max=0.23
llm_time_per_token_ms: avg=28.23 p50=29.73 p95=42.74 max=44.12
bicodec_decode_wall_ms: avg=12.67 p50=13.04 p95=16.71 max=19.27
bicodec_decode_gpu_ms: avg=0.00 p50=0.00 p95=0.00 max=0.00
latency_over_timeline_ms: avg=713.04 p50=661.07 p95=1476.95 max=1627.01
latency_over_generation_ms: avg=713.52 p50=661.58 p95=1477.10 max=1627.30
timeline_share: llm=99.6% bicodec=0.3% other=0.0%
timeline_sample_ms: first_batch=4184.8 llm_done=4184.8 parse_done=4184.9 bicodec_done=4194.8 total=4194.9
gpu: util_avg=75% util_max=97% mem_avg=23603MB mem_max=23603MB

=== concurrency=384 requests=384 ===
success=384/384 (100%)
wall=6.53s throughput=58.80 req/s
latency_ms avg=4838 p50=5145 p95=6523 max=6523
generation_ms: avg=3789.67 p50=4005.00 p95=5840.00 max=6035.00
timeline_llm_first_batch_ms: avg=3777.18 p50=3992.00 p95=5830.90 max=6025.11
timeline_llm_done_ms: avg=3777.19 p50=3992.01 p95=5830.90 max=6025.12
timeline_parse_done_ms: avg=3777.25 p50=3992.08 p95=5830.96 max=6025.18
timeline_bicodec_done_ms: avg=3790.11 p50=4005.94 p95=5840.95 max=6035.33
timeline_total_ms: avg=3790.12 p50=4005.96 p95=5840.96 max=6035.34
llm_batch_wall_ms_total: avg=3777.06 p50=3991.87 p95=5830.77 max=6024.99
llm_parse_ms: avg=0.06 p50=0.05 p95=0.07 max=0.25
llm_time_per_token_ms: avg=29.51 p50=31.19 p95=45.55 max=47.07
bicodec_decode_wall_ms: avg=12.83 p50=13.45 p95=16.84 max=18.59
bicodec_decode_gpu_ms: avg=0.00 p50=0.00 p95=0.00 max=0.00
latency_over_timeline_ms: avg=1048.27 p50=1019.36 p95=1772.02 max=2076.88
latency_over_generation_ms: avg=1048.73 p50=1019.38 p95=1772.64 max=2077.76
timeline_share: llm=99.7% bicodec=0.3% other=0.0%
timeline_sample_ms: first_batch=4005.6 llm_done=4005.6 parse_done=4005.7 bicodec_done=4021.5 total=4021.5
gpu: util_avg=78% util_max=97% mem_avg=23603MB mem_max=23603MB

results saved: /tmp/runtime_levels_352_384_finalonly.json
ERROR 02-18 00:27:07 [core_client.py:605] Engine core proc EngineCore_DP0 died unexpectedly, shutting down client.


**Testing throughput peak at 336**


The edge sweep shows throughput falling after 320, but I’m doing one quick confirmation at 336 concurrency to lock the optimal operating point before summarizing.


cd /home/ubuntu/veenaModal && python3 scripts/stress_test_runtime_detailed.py --levels 336 --text short --max-tokens 128 --timeout 240 --requests-multiplier 1 --min-requests 8 --gpu-memory 0.25 --num-engines 1 --disable-engine-stats-logs --non-stream-final-only --output /tmp/runtime_c336_finalonly.json


Output:
Chunk ID: 477343
Wall time: 1.0003 seconds
Process running with session ID 41088
Original token count: 0
Output:


Output:
Chunk ID: e595fb
Wall time: 5.0015 seconds
Process running with session ID 41088
Original token count: 1277
Output:
[2026-02-18 00:27:22] INFO tts_runtime.py:183: Loading SparkTTS model with vLLM...
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
🚀 Initializing Spark TTS Model
📁 Model path: /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
🗣  Speakers: 12 predefined (lipakshi, vardan, reet, Nandini, krishna, anika, adarsh, Nilay, Aarvi, Asha, Bittu, Mira)
🎭 Emotions: 10 tags in [bracket] format

📝 Loading tokenizer from model...
✅ Tokenizer loaded: 165168 tokens

🔍 Validating 12 speakers for Spark TTS...
  lipakshi → <|speaker_0|>: 7 tokens
  vardan → <|speaker_1|>: 7 tokens
  reet → <|speaker_2|>: 7 tokens
  Nandini → <|speaker_3|>: 7 tokens
  krishna → <|speaker_4|>: 7 tokens
  anika → <|speaker_5|>: 7 tokens
  adarsh → <|speaker_6|>: 7 tokens
  Nilay → <|speaker_7|>: 7 tokens
  Aarvi → <|speaker_8|>: 7 tokens
  Asha → <|speaker_9|>: 7 tokens
  Bittu → <|speaker_10|>: 8 tokens
  Mira → <|speaker_11|>: 8 tokens
✅ All 12 speakers validated with mapping

🔍 Validating 10 emotion tags for Spark TTS...
✅ All 10 emotion tags validated

📋 Sample emotion tags:
  [angry]: 1 tokens
  [curious]: 1 tokens
  [excited]: 1 tokens
  [giggle]: 1 tokens
  [laughs harder]: 1 tokens
  ...

🔧 Initializing vLLM engine with optimizations...
  - dtype: bfloat16
  - max_model_len: 4096
  - gpu_memory_utilization: 0.25
  - enable_prefix_caching: True
  - enforce_eager (disable CUDA graphs): False
INFO 02-18 00:27:23 [model.py:541] Resolved architecture: Qwen2ForCausalLM
INFO 02-18 00:27:23 [model.py:1561] Using max model len 4096
INFO 02-18 00:27:23 [scheduler.py:226] Chunked prefill is enabled with max_num_batched_tokens=4096.
INFO 02-18 00:27:23 [vllm.py:624] Asynchronous scheduling is enabled.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(EngineCore_DP0 pid=4100800) INFO 02-18 00:27:24 [core.py:96] Initializing a V1 LLM engine (v0.15.1) with config: model='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', speculative_config=None, tokenizer='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [4096], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 256, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': True}, 'local_cache_dir': None, 'static_all_moe_layers': []}


Output:
Chunk ID: 624121
Wall time: 5.0008 seconds
Process running with session ID 41088
Original token count: 511
Output:
(EngineCore_DP0 pid=4100800) INFO 02-18 00:27:26 [parallel_state.py:1212] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://216.81.248.184:53061 backend=nccl
(EngineCore_DP0 pid=4100800) INFO 02-18 00:27:26 [parallel_state.py:1423] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
(EngineCore_DP0 pid=4100800) INFO 02-18 00:27:26 [gpu_model_runner.py:4033] Starting to load model /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM...
(EngineCore_DP0 pid=4100800) INFO 02-18 00:27:27 [cuda.py:364] Using FLASH_ATTN attention backend out of potential backends: ('FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION')
(EngineCore_DP0 pid=4100800) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
(EngineCore_DP0 pid=4100800) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
(EngineCore_DP0 pid=4100800)
Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
(EngineCore_DP0 pid=4100800)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:01<00:00,  1.28s/it]
(EngineCore_DP0 pid=4100800)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:01<00:00,  1.28s/it]
(EngineCore_DP0 pid=4100800)
(EngineCore_DP0 pid=4100800) INFO 02-18 00:27:29 [default_loader.py:291] Loading weights took 1.34 seconds
(EngineCore_DP0 pid=4100800) INFO 02-18 00:27:29 [gpu_model_runner.py:4130] Model loading took 0.95 GiB memory and 1.880681 seconds
(EngineCore_DP0 pid=4100800) INFO 02-18 00:27:34 [backends.py:812] Using cache directory: /home/ubuntu/.cache/vllm/torch_compile_cache/f3ff13511c/rank_0_0/backbone for vLLM's torch.compile
(EngineCore_DP0 pid=4100800) INFO 02-18 00:27:34 [backends.py:872] Dynamo bytecode transform time: 4.22 s


I’m finishing the 336 confirmation run now; this will determine whether 320 is truly the throughput sweet spot or if there’s a slightly better point nearby.


Output:
Chunk ID: 7622c5
Wall time: 5.0006 seconds
Process running with session ID 41088
Original token count: 787
Output:
(EngineCore_DP0 pid=4100800) INFO 02-18 00:27:37 [backends.py:267] Directly load the compiled graph(s) for compile range (1, 4096) from the cache, took 0.650 s
(EngineCore_DP0 pid=4100800) INFO 02-18 00:27:37 [monitor.py:34] torch.compile takes 4.87 s in total
(EngineCore_DP0 pid=4100800) INFO 02-18 00:27:38 [gpu_worker.py:356] Available KV cache memory: 18.06 GiB
(EngineCore_DP0 pid=4100800) INFO 02-18 00:27:38 [kv_cache_utils.py:1307] GPU KV cache size: 1,577,712 tokens
(EngineCore_DP0 pid=4100800) INFO 02-18 00:27:38 [kv_cache_utils.py:1312] Maximum concurrency for 4,096 tokens per request: 385.18x
(EngineCore_DP0 pid=4100800)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   0%|          | 0/35 [00:00<?, ?it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  11%|█▏        | 4/35 [00:00<00:00, 32.81it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  23%|██▎       | 8/35 [00:00<00:00, 35.09it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  34%|███▍      | 12/35 [00:00<00:00, 36.11it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  46%|████▌     | 16/35 [00:00<00:00, 36.43it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  57%|█████▋    | 20/35 [00:00<00:00, 36.30it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  69%|██████▊   | 24/35 [00:00<00:00, 36.33it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  80%|████████  | 28/35 [00:00<00:00, 36.26it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  91%|█████████▏| 32/35 [00:00<00:00, 34.71it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 35/35 [00:01<00:00, 34.75it/s]
(EngineCore_DP0 pid=4100800)
Capturing CUDA graphs (decode, FULL):   0%|          | 0/19 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL):  16%|█▌        | 3/19 [00:00<00:00, 27.42it/s]
Capturing CUDA graphs (decode, FULL):  37%|███▋      | 7/19 [00:00<00:00, 32.40it/s]
Capturing CUDA graphs (decode, FULL):  58%|█████▊    | 11/19 [00:00<00:00, 34.24it/s]
Capturing CUDA graphs (decode, FULL):  79%|███████▉  | 15/19 [00:00<00:00, 34.86it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 35.39it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 34.36it/s]
(EngineCore_DP0 pid=4100800) INFO 02-18 00:27:40 [gpu_model_runner.py:5063] Graph capturing finished in 2 secs, took 0.28 GiB
(EngineCore_DP0 pid=4100800) INFO 02-18 00:27:40 [core.py:272] init engine (profile, create kv cache, warmup model) took 10.94 seconds
[2026-02-18 00:27:42] INFO tts_runtime.py:192: Initializing prompt builder...
[2026-02-18 00:27:42] INFO tts_runtime.py:199: Initializing BiCodec decoder...
/home/ubuntu/.local/lib/python3.10/site-packages/torch/nn/utils/weight_norm.py:144: FutureWarning: `torch.nn.utils.weight_norm` is deprecated in favor of `torch.nn.utils.parametrizations.weight_norm`.
  WeightNorm.apply(module, name, dim)


Output:
Chunk ID: 78e1a9
Wall time: 5.0018 seconds
Process running with session ID 41088
Original token count: 127
Output:
[2026-02-18 00:27:48] INFO tts_runtime.py:206: Initializing TTS pipeline...
[2026-02-18 00:27:48] INFO tts_runtime.py:217: Initializing streaming pipeline...
[2026-02-18 00:27:48] INFO long_text_processor.py:112: LongTextProcessor initialized
[2026-02-18 00:27:48] INFO tts_runtime.py:274: ✅ TTS runtime initialized in 31661ms
[2026-02-18 00:27:48] INFO tts_runtime.py:275:    Model version: LLM
[2026-02-18 00:27:48] INFO tts_runtime.py:286: Speaker globals cache startup precompute skipped (disabled)


Output:
Chunk ID: 85ffaa
Wall time: 0.0514 seconds
Process exited with code 0
Original token count: 465
Output:
✅ vLLM engine initialized successfully!

🎉 Spark TTS Model ready for inference!

🎵 Loading BiCodec audio tokenizer from /home/ubuntu/veenaModal/models/spark_tts_4speaker...
   Device: cuda
Missing tensor: mel_transformer.spectrogram.window
Missing tensor: mel_transformer.mel_scale.fb
✅ BiCodec decoder initialized (sample rate: 16000Hz)
🚀 SparkTTSPipeline initialized
🌊 Veena3SlidingWindowPipeline initialized (sliding window: 28 tokens)

=== concurrency=336 requests=336 ===
success=336/336 (100%)
wall=5.63s throughput=59.69 req/s
latency_ms avg=4205 p50=4762 p95=5623 max=5623
generation_ms: avg=3527.55 p50=3704.00 p95=5310.00 max=5478.00
timeline_llm_first_batch_ms: avg=3515.22 p50=3688.94 p95=5300.53 max=5468.27
timeline_llm_done_ms: avg=3515.23 p50=3688.95 p95=5300.54 max=5468.28
timeline_parse_done_ms: avg=3515.29 p50=3689.01 p95=5300.59 max=5468.34
timeline_bicodec_done_ms: avg=3528.03 p50=3704.06 p95=5310.58 max=5478.41
timeline_total_ms: avg=3528.05 p50=3704.08 p95=5310.59 max=5478.43
llm_batch_wall_ms_total: avg=3515.09 p50=3688.82 p95=5300.41 max=5468.15
llm_parse_ms: avg=0.06 p50=0.06 p95=0.07 max=0.26
llm_time_per_token_ms: avg=27.46 p50=28.82 p95=41.41 max=42.72
bicodec_decode_wall_ms: avg=12.72 p50=12.73 p95=16.47 max=18.76
bicodec_decode_gpu_ms: avg=0.00 p50=0.00 p95=0.00 max=0.00
latency_over_timeline_ms: avg=677.38 p50=615.56 p95=1437.37 max=1603.04
latency_over_generation_ms: avg=677.88 p50=616.22 p95=1437.90 max=1603.93
timeline_share: llm=99.6% bicodec=0.4% other=0.0%
timeline_sample_ms: first_batch=4239.2 llm_done=4239.2 parse_done=4239.2 bicodec_done=4249.6 total=4249.7
gpu: util_avg=72% util_max=97% mem_avg=23605MB mem_max=23605MB

results saved: /tmp/runtime_c336_finalonly.json
ERROR 02-18 00:27:56 [core_client.py:605] Engine core proc EngineCore_DP0 died unexpectedly, shutting down client.


**Preparing to combine results and diffs**


cd /home/ubuntu/veenaModal && git diff -- veena3modal/core/pipeline.py | sed -n '1,260p'


cd /home/ubuntu/veenaModal && git diff -- veena3modal/core/bicodec_decoder.py | sed -n '1,220p'


cd /home/ubuntu/veenaModal && git diff -- scripts/stress_test_runtime_detailed.py | sed -n '1,260p'


Output:
Chunk ID: ea4ae4
Wall time: 0.0513 seconds
Process exited with code 0
Original token count: 2505
Output:
diff --git a/veena3modal/core/pipeline.py b/veena3modal/core/pipeline.py
index 9e44932..d0bfb1b 100644
--- a/veena3modal/core/pipeline.py
+++ b/veena3modal/core/pipeline.py
@@ -5,14 +5,50 @@ End-to-end pipeline for TTS generation with BiCodec audio tokenizer.
 Migrated from Veena3/Orpheus (SNAC) to Spark TTS (BiCodec).
 """

-import asyncio
 import logging
+import os
 import re
+import time
+import uuid
+from dataclasses import asdict, is_dataclass
 from typing import Optional, List, Dict, Any, Tuple
 from vllm import SamplingParams
+from vllm.sampling_params import RequestOutputKind
 import torch

 logger = logging.getLogger(__name__)
+_PERF_STAGE_LOG_ENABLED = os.environ.get("VEENA3_PERF_STAGE_LOG", "").strip().lower() in {
+    "1",
+    "true",
+    "yes",
+    "on",
+}
+_PERF_GPU_TIMING_ENABLED = os.environ.get("VEENA3_PERF_GPU_TIMING", "").strip().lower() in {
+    "1",
+    "true",
+    "yes",
+    "on",
+}
+_ASYNC_BICODEC_DECODE_ENABLED = os.environ.get("VEENA3_ASYNC_BICODEC_DECODE", "").strip().lower() in {
+    "1",
+    "true",
+    "yes",
+    "on",
+}
+_NON_STREAM_FINAL_ONLY_ENABLED = os.environ.get("VEENA3_NON_STREAM_FINAL_ONLY", "").strip().lower() in {
+    "1",
+    "true",
+    "yes",
+    "on",
+}
+
+
+def _log_perf_stage(event: str, extra: Dict[str, Any]) -> None:
+    """Emit hot-path perf logs only when explicitly enabled or DEBUG logging is active."""
+    if _PERF_STAGE_LOG_ENABLED:
+        logger.info(event, extra=extra)
+    elif logger.isEnabledFor(logging.DEBUG):
+        logger.debug(event, extra=extra)

 from veena3modal.core.constants import (
     TRAINING_STOP_TOKEN_IDS,
@@ -51,8 +87,88 @@ class SparkTTSPipeline:
         self.bicodec_decoder = bicodec_decoder

         print(f"🚀 SparkTTSPipeline initialized")
-
-    async def generate_speech(
+
+    @staticmethod
+    def _p50(values: List[float]) -> float:
+        """Compute a stable p50 for small lists."""
+        if not values:
+            return 0.0
+        ordered = sorted(values)
+        return ordered[(len(ordered) - 1) // 2]
+
+    @staticmethod
+    def _to_number(value: Any) -> Optional[float]:
+        """Best-effort conversion of scalar values to float."""
+        if value is None:
+            return None
+        if isinstance(value, (int, float)):
+            return float(value)
+        try:
+            return float(value)
+        except (TypeError, ValueError):
+            return None
+
+    def _extract_request_metrics(self, raw_metrics: Any) -> Dict[str, float]:
+        """
+        Convert vLLM request_output.metrics into a numeric dictionary.
+
+        vLLM may return a dataclass-like object or a plain dict depending on version.
+        """
+        if raw_metrics is None:
+            return {}
+
+        numeric: Dict[str, float] = {}
+        items: List[Tuple[str, Any]] = []
+
+        if isinstance(raw_metrics, dict):
+            items.extend(raw_metrics.items())
+        else:
+            # Modern vLLM versions expose RequestStateStats dataclasses with
+            # fields like queued_ts/scheduled_ts/first_token_ts/last_token_ts.
+            if is_dataclass(raw_metrics):
+                try:
+                    items.extend(asdict(raw_metrics).items())
+                except Exception:
+                    pass
+            if hasattr(raw_metrics, "__dict__"):
+                try:
+                    items.extend(dict(raw_metrics.__dict__).items())
+                except Exception:
+                    pass
+
+            # Fallback: scan public scalar attributes.
+            for name in dir(raw_metrics):
+                if name.startswith("_"):
+                    continue
+                try:
+                    value = getattr(raw_metrics, name)
+                except Exception:
+                    continue
+                if callable(value):
+                    continue
+                items.append((name, value))
+
+        seen: set[str] = set()
+        for key, value in items:
+            key_s = str(key)
+            if key_s in seen:
+                continue
+            seen.add(key_s)
+            parsed = self._to_number(value)
+            if parsed is not None:
+                numeric[key_s] = parsed
+        return numeric
+
+    def _apply_stats(self, perf: Dict[str, Any], prefix: str, values: List[float]) -> None:
+        """Populate total/min/max/p50 stats for a list of ms samples."""
+        if not values:
+            return
+        perf[f"{prefix}_total"] = float(sum(values))
+        perf[f"{prefix}_min"] = float(min(values))
+        perf[f"{prefix}_max"] = float(max(values))
+        perf[f"{prefix}_p50"] = float(self._p50(values))
+
+    async def generate_speech_profiled(
         self,
         speaker: str,
         text: str,
@@ -62,106 +178,449 @@ class SparkTTSPipeline:
         max_tokens: int = DEFAULT_MAX_TOKENS,
         repetition_penalty: float = 1.05,
         seed: Optional[int] = None,
-    ) -> Optional[bytes]:
+    ) -> Tuple[Optional[bytes], Dict[str, Any]]:
         """
-        Generate speech audio (non-streaming) using Spark TTS.
-
-        NOTE: This method signature changed from description-based to speaker-based
-        to align with Spark TTS architecture.
-
-        Args:
-            speaker: Speaker name (one of 12 predefined speakers)
-            text: Text to synthesize (with optional [emotion] tags)
-            temperature: Sampling temperature (default: 0.8 for Spark TTS)
-            top_k: Top-k sampling (default: 50)
-            top_p: Nucleus sampling (default: 1.0)
-            max_tokens: Max BiCodec tokens to generate (default: 2048)
-            seed: Random seed for reproducibility
-
+        Generate speech with detailed per-request timing metrics.
+
         Returns:
-            Audio bytes (int16 PCM WAV, 16kHz mono) or None if failed
+            (wav_bytes, perf_dict)
         """
-        # Build prompt using Spark TTS format
+        from veena3modal.audio.utils import add_wav_header
+
+        perf: Dict[str, Any] = {
+            "llm_prompt_token_total": 0,
+            "llm_token_total": 0,
+            "semantic_token_total": 0,
+            "global_token_total": 0,
+            "llm_batch_count": 0,
+            "llm_parse_ms": 0.0,
+            "llm_decode_calls": 0,
+            "bicodec_decode_wall_ms": 0.0,
+            "bicodec_decode_gpu_ms": 0.0,
+            "bicodec_decode_cpu_ms": 0.0,
+            "wav_pack_ms": 0.0,
+        }
+
+        t_request_start = time.perf_counter()
+        timeline_marks: List[Dict[str, float]] = [{"stage": "request_start", "t_ms": 0.0, "dt_ms": 0.0}]
+        last_mark_ms = 0.0
+        perf["timeline_markers"] = timeline_marks
+        perf["timeline_request_start_ms"] = 0.0
+
+        def _mark(stage: str) -> float:
+            nonlocal last_mark_ms
+            now_ms = (time.perf_counter() - t_request_start) * 1000
+            dt_ms = now_ms - last_mark_ms
+            last_mark_ms = now_ms
+            timeline_marks.append(
+                {
+                    "stage": stage,
+                    "t_ms": float(now_ms),
+                    "dt_ms": float(dt_ms),
+                }
+            )
+            perf[f"timeline_{stage}_ms"] = float(now_ms)
+            perf[f"timeline_{stage}_delta_ms"] = float(dt_ms)
+            return float(now_ms)
+
+        def _finalize_timeline() -> None:
+            if not timeline_marks or timeline_marks[-1].get("stage") != "request_done":
+                _mark("request_done")
+
+            # Keep a compact, ordered timeline payload for downstream benchmarking.
+            perf["timeline_markers"] = list(timeline_marks)
+            perf["timeline_total_ms"] = float(perf.get("timeline_request_done_ms", 0.0))
+
+            first_batch_ms = perf.get("timeline_llm_first_batch_ms")
+            llm_done_ms = perf.get("timeline_llm_done_ms")
+            parse_done_ms = perf.get("timeline_parse_done_ms")
+            bicodec_done_ms = perf.get("timeline_bicodec_done_ms")
+            request_done_ms = perf.get("timeline_request_done_ms")
+
+            if isinstance(first_batch_ms, (int, float)):
+                perf["timeline_to_first_batch_ms"] = float(first_batch_ms)
+            if isinstance(first_batch_ms, (int, float)) and isinstance(llm_done_ms, (int, float)) and llm_done_ms >= first_batch_ms:
+                perf["timeline_first_batch_to_llm_done_ms"] = float(llm_done_ms - first_batch_ms)
+            if isinstance(llm_done_ms, (int, float)) and isinstance(request_done_ms, (int, float)) and request_done_ms >= llm_done_ms:
+                perf["timeline_post_llm_ms"] = float(request_done_ms - llm_done_ms)
+            if isinstance(parse_done_ms, (int, float)) and isinstance(bicodec_done_ms, (int, float)) and bicodec_done_ms >= parse_done_ms:
+                perf["timeline_parse_to_bicodec_done_ms"] = float(bicodec_done_ms - parse_done_ms)
+            if isinstance(bicodec_done_ms, (int, float)) and isinstance(request_done_ms, (int, float)) and request_done_ms >= bicodec_done_ms:
+                perf["timeline_bicodec_to_request_done_ms"] = float(request_done_ms - bicodec_done_ms)
+
+        # Stage 1: prompt build
+        t0 = time.perf_counter()
         prompt = self.prompt_builder.build_prefix(speaker, text)
-
-        # Configure sampling for Spark TTS
+        perf["prompt_build_ms"] = (time.perf_counter() - t0) * 1000
+        _mark("prompt_ready")
+
+        # Prompt token count (for TPS normalization)
+        try:
+            perf["llm_prompt_token_total"] = len(
+                self.model.tokenizer.encode(prompt, add_special_tokens=False)
+            )
+        except Exception:
+            perf["llm_prompt_token_total"] = 0
+
+        # Stage 2: sampling params
+        t0 = time.perf_counter()
         sampling_params = SamplingParams(
             temperature=temperature,
             top_k=top_k,
             top_p=top_p,


Output:
Chunk ID: b8218f
Wall time: 0.0508 seconds
Process exited with code 0
Original token count: 1049
Output:
diff --git a/veena3modal/core/bicodec_decoder.py b/veena3modal/core/bicodec_decoder.py
index 820609a..eac389e 100644
--- a/veena3modal/core/bicodec_decoder.py
+++ b/veena3modal/core/bicodec_decoder.py
@@ -8,6 +8,7 @@ Replaces SNAC decoder for Spark TTS model.
 import torch
 import numpy as np
 import sys
+import asyncio
 from pathlib import Path
 from typing import List, Optional, Tuple

@@ -67,6 +68,16 @@ class BiCodecDecoder:
         # Initialize BiCodec tokenizer
         self.audio_tokenizer = BiCodecTokenizer(model_path, device=str(self.device))

+        # OPTIMIZATION: Ensure model is on device once at init, not per-decode call
+        self.audio_tokenizer.device = self.device
+        self.audio_tokenizer.model.to(self.device)
+
+        # NOTE: torch.compile disabled for BiCodec -- dynamic semantic sequence lengths
+        # cause repeated recompilation (~120s each), worse than eager mode.
+        # torch.compile works best with static shapes. If needed, pre-warm with
+        # representative lengths or use mode="default" instead of "reduce-overhead".
+        # The multi-engine approach (Tier 3) provides more impactful gains.
+
         print(f"✅ BiCodec decoder initialized (sample rate: {AUDIO_SAMPLE_RATE}Hz)")
         if enable_batching:
             print(f"   ℹ  Batching enabled (for streaming support)")
@@ -115,10 +126,7 @@ class BiCodecDecoder:
             pred_semantic = torch.tensor(semantic_ids).long().unsqueeze(0).to(self.device)
             pred_global = torch.tensor(global_ids).long().unsqueeze(0).to(self.device)

-            # Ensure audio tokenizer is on correct device
-            self.audio_tokenizer.device = self.device
-            self.audio_tokenizer.model.to(self.device)
-
+            # Device migration done once in __init__, not per-call
             # Decode through BiCodec
             # BiCodecTokenizer.detokenize() expects:
             #   - global_tokens: (batch, global_dim) -> will unsqueeze(1) internally
@@ -163,6 +171,25 @@ class BiCodecDecoder:
         audio_int16 = (audio * 32767).astype(np.int16)

         return audio_int16.tobytes()
+
+    async def decode_to_bytes_async(
+        self,
+        semantic_ids: List[int],
+        global_ids: List[int],
+    ) -> Optional[bytes]:
+        """
+        Async wrapper for decode_to_bytes.
+
+        Runs decode in the default executor so the event loop stays responsive
+        under high request concurrency.
+        """
+        loop = asyncio.get_running_loop()
+        return await loop.run_in_executor(
+            None,
+            self.decode_to_bytes,
+            semantic_ids,
+            global_ids,
+        )

     def validate_tokens(self, semantic_ids: List[int], global_ids: List[int]) -> bool:
         """
@@ -289,7 +316,11 @@ class BiCodecDecoder:
         use_sliding_window: bool = False,
     ) -> Optional[bytes]:
         """
-        Async wrapper for streaming decode (for compatibility with streaming pipeline).
+        Async wrapper for streaming decode - runs in executor to avoid blocking event loop.
+
+        OPTIMIZATION: Previously ran synchronously, blocking all concurrent coroutines
+        during GPU decode (~20-50ms). Now uses run_in_executor to unblock the event loop
+        so other streams can progress while this decode runs on GPU.

         Args:
             semantic_ids: List of semantic token IDs
@@ -300,12 +331,14 @@ class BiCodecDecoder:
         Returns:
             Audio bytes (int16 PCM) or None
         """
-        # BiCodec decode is fast enough to run synchronously
-        return self.decode_streaming(
-            semantic_ids=semantic_ids,
-            global_ids=global_ids,
-            use_sliding_window=use_sliding_window,
-            trim_warmup=trim_warmup
+        import asyncio
+        loop = asyncio.get_event_loop()
+        return await loop.run_in_executor(
+            None,  # Default ThreadPoolExecutor
+            self.decode_streaming,
+            semantic_ids,
+            global_ids,
+            use_sliding_window,
+            trim_warmup,
         )

-


Output:
Chunk ID: 594d34
Wall time: 0.0508 seconds
Process exited with code 0
Original token count: 0
Output:


**Planning benchmark aggregation**


python3 - <<'PY'
import json,glob,os
files=[
'/tmp/runtime_c96_after_patch_stats_on_v2.json',
'/tmp/runtime_c96_after_patch_nostats.json',
'/tmp/runtime_c96_after_patch_nostats_finalonly.json',
'/tmp/runtime_levels_96_160_finalonly.json',
'/tmp/runtime_levels_192_256_finalonly.json',
'/tmp/runtime_levels_288_320_finalonly.json',
'/tmp/runtime_levels_352_384_finalonly.json',
'/tmp/runtime_c336_finalonly.json',
]
rows=[]
for p in files:
    if not os.path.exists(p):
        continue
    with open(p) as f:d=json.load(f)
    cfg=d.get('config',{})
    for lvl in d['levels']:
        c=lvl['concurrency']
        rows.append({
            'file':os.path.basename(p),
            'c':c,
            'thr':lvl['throughput_rps'],
            'lat':lvl['latency_ms']['avg'],
            'gen':lvl['timing']['generation_ms']['avg'] if lvl['timing'].get('generation_ms') else None,
            'tok':lvl['timing']['llm_time_per_token_ms']['avg'] if lvl['timing'].get('llm_time_per_token_ms') else None,
            'over':lvl['timing']['latency_over_generation_ms']['avg'] if lvl['timing'].get('latency_over_generation_ms') else None,
            'final_only':cfg.get('non_stream_final_only',False),
            'nostats':cfg.get('disable_engine_stats_logs',False),
        })
rows=sorted(rows,key=lambda r:(r['c'],r['thr']))
print('total rows',len(rows))
for r in rows:
    print(f"c={r['c']:>3} thr={r['thr']:.2f} lat={r['lat']:.0f} gen={r['gen']:.0f if r['gen'] else 0} tpt={r['tok']:.2f if r['tok'] else 0} over={r['over']:.0f if r['over'] else 0} final={r['final_only']} nostats={r['nostats']} file={r['file']}")
print('\nTop throughput:')
for r in sorted(rows,key=lambda x:x['thr'],reverse=True)[:10]:
    print(f"c={r['c']} thr={r['thr']:.2f} lat={r['lat']:.0f} final={r['final_only']} nostats={r['nostats']} file={r['file']}")
PY


Output:
Chunk ID: 2868ed
Wall time: 0.2620 seconds
Process exited with code 1
Original token count: 32
Output:
total rows 14
Traceback (most recent call last):
  File "<stdin>", line 34, in <module>
ValueError: Invalid format specifier


python3 - <<'PY'
import json,os
files=[
'/tmp/runtime_c96_after_patch_stats_on_v2.json',
'/tmp/runtime_c96_after_patch_nostats.json',
'/tmp/runtime_c96_after_patch_nostats_finalonly.json',
'/tmp/runtime_levels_96_160_finalonly.json',
'/tmp/runtime_levels_192_256_finalonly.json',
'/tmp/runtime_levels_288_320_finalonly.json',
'/tmp/runtime_levels_352_384_finalonly.json',
'/tmp/runtime_c336_finalonly.json',
]
rows=[]
for p in files:
    if not os.path.exists(p):
        continue
    with open(p) as f:d=json.load(f)
    cfg=d.get('config',{})
    for lvl in d['levels']:
        def get(path,default=None):
            obj=lvl
            for key in path:
                if not isinstance(obj,dict) or key not in obj:
                    return default
                obj=obj[key]
            return obj
        rows.append({
            'file':os.path.basename(p),
            'c':lvl['concurrency'],
            'thr':lvl['throughput_rps'],
            'lat':get(['latency_ms','avg']),
            'gen':get(['timing','generation_ms','avg']),
            'tok':get(['timing','llm_time_per_token_ms','avg']),
            'over':get(['timing','latency_over_generation_ms','avg']),
            'final_only':bool(cfg.get('non_stream_final_only',False)),
            'nostats':bool(cfg.get('disable_engine_stats_logs',False)),
        })

def f(x,p=2):
    return 'na' if x is None else f"{x:.{p}f}"

rows=sorted(rows,key=lambda r:(r['c'],r['thr']))
print('total rows',len(rows))
for r in rows:
    print(f"c={r['c']:>3} thr={f(r['thr'])} lat={f(r['lat'],0)} gen={f(r['gen'],0)} tpt={f(r['tok'])} over={f(r['over'],0)} final={r['final_only']} nostats={r['nostats']} file={r['file']}")
print('\nTop throughput:')
for r in sorted(rows,key=lambda x:x['thr'],reverse=True)[:10]:
    print(f"c={r['c']} thr={f(r['thr'])} lat={f(r['lat'],0)} final={r['final_only']} nostats={r['nostats']} file={r['file']}")
PY


Output:
Chunk ID: 35fccc
Wall time: 0.2114 seconds
Process exited with code 0
Original token count: 656
Output:
total rows 14
c= 96 thr=46.03 lat=2047 gen=1510 tpt=11.70 over=537 final=False nostats=False file=runtime_c96_after_patch_stats_on_v2.json
c= 96 thr=47.15 lat=1925 gen=1469 tpt=11.39 over=456 final=False nostats=True file=runtime_c96_after_patch_nostats.json
c= 96 thr=47.74 lat=1844 gen=1453 tpt=11.26 over=391 final=True nostats=True file=runtime_c96_after_patch_nostats_finalonly.json
c= 96 thr=49.64 lat=1786 gen=1459 tpt=11.31 over=326 final=True nostats=True file=runtime_levels_96_160_finalonly.json
c=128 thr=51.81 lat=2259 gen=1772 tpt=13.75 over=487 final=True nostats=True file=runtime_levels_96_160_finalonly.json
c=160 thr=53.05 lat=2649 gen=2001 tpt=15.54 over=648 final=True nostats=True file=runtime_levels_96_160_finalonly.json
c=192 thr=54.98 lat=3059 gen=2291 tpt=17.81 over=768 final=True nostats=True file=runtime_levels_192_256_finalonly.json
c=224 thr=58.67 lat=3225 gen=2368 tpt=18.40 over=856 final=True nostats=True file=runtime_levels_192_256_finalonly.json
c=256 thr=59.43 lat=3732 gen=2753 tpt=21.41 over=980 final=True nostats=True file=runtime_levels_192_256_finalonly.json
c=288 thr=59.73 lat=4071 gen=2919 tpt=22.70 over=1152 final=True nostats=True file=runtime_levels_288_320_finalonly.json
c=320 thr=61.91 lat=4329 gen=3089 tpt=24.04 over=1239 final=True nostats=True file=runtime_levels_288_320_finalonly.json
c=336 thr=59.69 lat=4205 gen=3528 tpt=27.46 over=678 final=True nostats=True file=runtime_c336_finalonly.json
c=352 thr=60.51 lat=4339 gen=3626 tpt=28.23 over=714 final=True nostats=True file=runtime_levels_352_384_finalonly.json
c=384 thr=58.80 lat=4838 gen=3790 tpt=29.51 over=1049 final=True nostats=True file=runtime_levels_352_384_finalonly.json

Top throughput:
c=320 thr=61.91 lat=4329 final=True nostats=True file=runtime_levels_288_320_finalonly.json
c=352 thr=60.51 lat=4339 final=True nostats=True file=runtime_levels_352_384_finalonly.json
c=288 thr=59.73 lat=4071 final=True nostats=True file=runtime_levels_288_320_finalonly.json
c=336 thr=59.69 lat=4205 final=True nostats=True file=runtime_c336_finalonly.json
c=256 thr=59.43 lat=3732 final=True nostats=True file=runtime_levels_192_256_finalonly.json
c=384 thr=58.80 lat=4838 final=True nostats=True file=runtime_levels_352_384_finalonly.json
c=224 thr=58.67 lat=3225 final=True nostats=True file=runtime_levels_192_256_finalonly.json
c=192 thr=54.98 lat=3059 final=True nostats=True file=runtime_levels_192_256_finalonly.json
c=160 thr=53.05 lat=2649 final=True nostats=True file=runtime_levels_96_160_finalonly.json
c=128 thr=51.81 lat=2259 final=True nostats=True file=runtime_levels_96_160_finalonly.json


**Planning detailed single-request benchmark**


I’m running a dedicated single-request profile now (`concurrency=1`) with full detailed timing so we have a clean per-request baseline (TTFB, per-token time, batch timing, decode split).


cd /home/ubuntu/veenaModal && python3 scripts/stress_test_runtime_detailed.py --levels 1 --text short --max-tokens 128 --timeout 120 --requests-multiplier 1 --min-requests 1 --gpu-memory 0.25 --num-engines 1 --output /tmp/runtime_c1_detailed_statson.json


Output:
Chunk ID: 86d7d0
Wall time: 1.0010 seconds
Process running with session ID 20193
Original token count: 0
Output: