Input:
*** Begin Patch
*** Update File: /home/ubuntu/veenaModal/scripts/stress_test_streaming_runtime.py
@@
         "config": {
             "levels": args.levels,
             "text": args.text,
             "max_tokens": args.max_tokens,
             "timeout_s": args.timeout,
             "requests_multiplier": args.requests_multiplier,
             "min_requests": args.min_requests,
             "gpu_memory": args.gpu_memory,
             "stream_output_kind": args.stream_output_kind,
             "chunking": args.chunking,
+            "num_engines": args.num_engines,
+            "max_num_batched_tokens": args.max_num_batched_tokens,
+            "max_num_seqs": args.max_num_seqs,
+            "enable_chunked_prefill": args.enable_chunked_prefill,
+            "disable_chunked_prefill": args.disable_chunked_prefill,
+            "disable_prefix_caching": args.disable_prefix_caching,
+            "disable_engine_stats_logs": args.disable_engine_stats_logs,
+            "enforce_eager": args.enforce_eager,
+            "stream_decode_interval": args.stream_decode_interval,
+            "stream_window_size": args.stream_window_size,
+            "stream_min_semantic_first": args.stream_min_semantic_first,
+            "stream_crossfade_ms": args.stream_crossfade_ms,
+            "disable_windowed_decode": args.disable_windowed_decode,
             "timestamp": int(time.time()),
         },
         "levels": [],
     }
@@
                 "inter_chunk_ms": lvl.inter_chunk_stats(),
                 "tail_emit_speed_x": lvl.emit_speed_stats(),
                 "gpu": lvl.gpu_summary,
+                "timing": {
+                    "timeline_to_first_batch_ms": lvl.timing_stats("timeline_to_first_batch_ms"),
+                    "timeline_to_first_chunk_emitted_ms": lvl.timing_stats("timeline_to_first_chunk_emitted_ms"),
+                    "timeline_total_ms": lvl.timing_stats("timeline_total_ms"),
+                    "llm_time_in_queue_ms": lvl.timing_stats("llm_time_in_queue_ms"),
+                    "llm_scheduler_ms": lvl.timing_stats("llm_scheduler_ms"),
+                    "llm_first_token_ms": lvl.timing_stats("llm_first_token_ms"),
+                    "llm_request_lifecycle_ms": lvl.timing_stats("llm_request_lifecycle_ms"),
+                    "llm_generation_wall_ms": lvl.timing_stats("llm_generation_wall_ms"),
+                    "llm_time_per_token_ms": lvl.timing_stats("llm_time_per_token_ms"),
+                    "llm_batch_wall_ms_total": lvl.timing_stats("llm_batch_wall_ms_total"),
+                    "llm_decode_wall_ms_total": lvl.timing_stats("llm_decode_wall_ms_total"),
+                    "tokens_per_batch_p50": lvl.timing_stats("tokens_per_batch_p50"),
+                    "bicodec_decode_calls": lvl.timing_stats("bicodec_decode_calls"),
+                    "bicodec_decode_wall_ms_total": lvl.timing_stats("bicodec_decode_wall_ms_total"),
+                    "bicodec_decode_interval_ms_p50": lvl.timing_stats("bicodec_decode_interval_ms_p50"),
+                    "latency_over_timeline_ms": lvl.latency_over_metric_stats("timeline_total_ms"),
+                },
                 "requests": [
                     {
                         "success": r.success,
*** End Patch


Output: {"output":"Success. Updated the following files:\nM /home/ubuntu/veenaModal/scripts/stress_test_streaming_runtime.py\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


# Collaboration Mode: Default

You are now in Default mode. Any previous instructions for other modes (e.g. Plan mode) are no longer active.

Your active mode changes only when new developer instructions with a different `<collaboration_mode>...</collaboration_mode>` change it; user requests or tool descriptions do not change mode by themselves. Known mode names are Default and Plan.

## request_user_input availability

The `request_user_input` tool is unavailable in Default mode. If you call it while in Default mode, it will return an error.

If a decision is necessary and cannot be discovered from local context, ask the user directly. However, in Default mode you should strongly prefer executing the user's request rather than stopping to ask questions.

Summary: auto

## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) When `SKILL.md` references relative paths (e.g., `scripts/foo.py`), resolve them relative to the skill directory listed above first, and only consider other paths if needed.
  3) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  4) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  5) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.


**Adding runtime arguments and env vars**


Input:
*** Begin Patch
*** Update File: /home/ubuntu/veenaModal/scripts/stress_test_streaming_runtime.py
@@
 def main() -> None:
     parser = argparse.ArgumentParser(description="Direct runtime streaming stress benchmark")
     parser.add_argument("--levels", default="16,32,64", help="Comma-separated concurrency levels")
     parser.add_argument("--text", choices=["short", "medium", "long"], default="short")
     parser.add_argument("--max-tokens", type=int, default=512, help="max_tokens sent to streaming generation")
@@
     parser.add_argument("--min-requests", type=int, default=8)
     parser.add_argument("--chunking", action="store_true", default=False, help="Enable long-text chunking path")
     parser.add_argument("--gpu-memory", type=float, default=0.25)
     parser.add_argument("--stream-output-kind", choices=["delta", "cumulative"], default="delta")
+    parser.add_argument("--num-engines", type=int, default=1, help="Number of vLLM engines")
+    parser.add_argument("--max-num-batched-tokens", type=int, default=None)
+    parser.add_argument("--max-num-seqs", type=int, default=None)
+    parser.add_argument("--enable-chunked-prefill", action="store_true", default=None)
+    parser.add_argument("--disable-chunked-prefill", action="store_true", default=False)
+    parser.add_argument("--disable-prefix-caching", action="store_true", default=False)
+    parser.add_argument("--disable-engine-stats-logs", action="store_true", default=False)
+    parser.add_argument("--enforce-eager", action="store_true", default=False)
+    parser.add_argument("--stream-decode-interval", type=int, default=48)
+    parser.add_argument("--stream-window-size", type=int, default=128)
+    parser.add_argument("--stream-min-semantic-first", type=int, default=10)
+    parser.add_argument("--stream-crossfade-ms", type=int, default=50)
+    parser.add_argument("--disable-windowed-decode", action="store_true", default=False)
     parser.add_argument("--model-path", type=str, default="")
     parser.add_argument("--output", default="stress_streaming_runtime.json")
     args = parser.parse_args()
@@
     from veena3modal.local_server import DEFAULT_LOCAL_MODEL_DIR

     model_path = args.model_path or DEFAULT_LOCAL_MODEL_DIR
-    init_runtime(model_path=model_path, gpu_mem=args.gpu_memory, stream_output_kind=args.stream_output_kind)
+
+    os.environ["VEENA3_STREAM_OUTPUT_KIND"] = args.stream_output_kind
+    os.environ["VEENA3_STREAM_DECODE_INTERVAL"] = str(args.stream_decode_interval)
+    os.environ["VEENA3_STREAM_WINDOW_SIZE"] = str(args.stream_window_size)
+    os.environ["VEENA3_STREAM_MIN_SEMANTIC_FIRST"] = str(args.stream_min_semantic_first)
+    os.environ["VEENA3_STREAM_CROSSFADE_MS"] = str(args.stream_crossfade_ms)
+    os.environ["VEENA3_STREAM_WINDOWED_DECODE"] = "0" if args.disable_windowed_decode else "1"
+
+    enable_chunked_prefill = args.enable_chunked_prefill
+    if args.disable_chunked_prefill:
+        enable_chunked_prefill = False
+
+    init_runtime(
+        model_path=model_path,
+        gpu_mem=args.gpu_memory,
+        stream_output_kind=args.stream_output_kind,
+        num_engines=args.num_engines,
+        max_num_batched_tokens=args.max_num_batched_tokens,
+        max_num_seqs=args.max_num_seqs,
+        enable_chunked_prefill=enable_chunked_prefill,
+        enable_prefix_caching=(False if args.disable_prefix_caching else None),
+        disable_log_stats=(True if args.disable_engine_stats_logs else None),
+        enforce_eager=(True if args.enforce_eager else None),
+    )
     asyncio.run(main_async(args))
*** End Patch


Output: {"output":"Success. Updated the following files:\nM /home/ubuntu/veenaModal/scripts/stress_test_streaming_runtime.py\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


Summary: auto


**Fixing final metrics propagation**


Input:
*** Begin Patch
*** Update File: /home/ubuntu/veenaModal/scripts/stress_test_streaming_runtime.py
@@
         async for audio_chunk, metrics in gen:
             now = time.perf_counter()
             if first_chunk_t is None:
                 first_chunk_t = now
             if prev_chunk_t is not None:
                 inter_chunk_ms.append((now - prev_chunk_t) * 1000.0)
             prev_chunk_t = now
             chunks += 1
             total_bytes += len(audio_chunk)
-            final_metrics = dict(metrics or {})
+            if isinstance(metrics, dict):
+                # Keep reference; runtime may mutate this dict after the final chunk
+                # when internal stream generation completes.
+                final_metrics = metrics
@@
             tail_emit_speed_x=tail_emit_speed_x,
-            final_metrics=final_metrics,
+            final_metrics=dict(final_metrics),
         )
@@
             error="timeout",
             inter_chunk_ms=inter_chunk_ms,
-            final_metrics=final_metrics,
+            final_metrics=dict(final_metrics),
         )
@@
             error=str(exc)[:200],
             inter_chunk_ms=inter_chunk_ms,
-            final_metrics=final_metrics,
+            final_metrics=dict(final_metrics),
         )
*** End Patch


Output: {"output":"Success. Updated the following files:\nM /home/ubuntu/veenaModal/scripts/stress_test_streaming_runtime.py\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


**Confirming metrics dict handling**


Instrumentation and stress-harness changes are in. I’m running a compile check now, then I’ll execute a tuned-vs-baseline streaming benchmark matrix and report where queue/scheduler/decode time is actually going at high concurrency.


cd /home/ubuntu/veenaModal && python3 -m py_compile veena3modal/core/streaming_pipeline.py veena3modal/services/tts_runtime.py scripts/stress_test_streaming_runtime.py


Output:
Chunk ID: 7a9f3d
Wall time: 0.0513 seconds
Process exited with code 0
Original token count: 0
Output:


**Running initial benchmark smoke test**


cd /home/ubuntu/veenaModal && python3 scripts/stress_test_streaming_runtime.py --levels 8 --text short --max-tokens 512 --timeout 120 --requests-multiplier 1 --min-requests 8 --gpu-memory 0.25 --stream-output-kind delta --output /tmp/stream_metrics_smoke.json


Output:
Chunk ID: 7d0a70
Wall time: 1.0011 seconds
Process running with session ID 25613
Original token count: 0
Output:


Output:
Chunk ID: 1092c7
Wall time: 5.0006 seconds
Process running with session ID 25613
Original token count: 1362
Output:
[2026-02-18 01:07:20] INFO tts_runtime.py:183: Loading SparkTTS model with vLLM...
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
🚀 Initializing Spark TTS Model
📁 Model path: /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
🗣  Speakers: 12 predefined (lipakshi, vardan, reet, Nandini, krishna, anika, adarsh, Nilay, Aarvi, Asha, Bittu, Mira)
🎭 Emotions: 10 tags in [bracket] format

📝 Loading tokenizer from model...
✅ Tokenizer loaded: 165168 tokens

🔍 Validating 12 speakers for Spark TTS...
  lipakshi → <|speaker_0|>: 7 tokens
  vardan → <|speaker_1|>: 7 tokens
  reet → <|speaker_2|>: 7 tokens
  Nandini → <|speaker_3|>: 7 tokens
  krishna → <|speaker_4|>: 7 tokens
  anika → <|speaker_5|>: 7 tokens
  adarsh → <|speaker_6|>: 7 tokens
  Nilay → <|speaker_7|>: 7 tokens
  Aarvi → <|speaker_8|>: 7 tokens
  Asha → <|speaker_9|>: 7 tokens
  Bittu → <|speaker_10|>: 8 tokens
  Mira → <|speaker_11|>: 8 tokens
✅ All 12 speakers validated with mapping

🔍 Validating 10 emotion tags for Spark TTS...
✅ All 10 emotion tags validated

📋 Sample emotion tags:
  [angry]: 1 tokens
  [curious]: 1 tokens
  [excited]: 1 tokens
  [giggle]: 1 tokens
  [laughs harder]: 1 tokens
  ...

🔧 Initializing vLLM engine with optimizations...
  - dtype: bfloat16
  - max_model_len: 4096
  - gpu_memory_utilization: 0.25
  - enable_prefix_caching: True
  - enforce_eager (disable CUDA graphs): False
INFO 02-18 01:07:21 [model.py:541] Resolved architecture: Qwen2ForCausalLM
INFO 02-18 01:07:21 [model.py:1561] Using max model len 4096
INFO 02-18 01:07:21 [scheduler.py:226] Chunked prefill is enabled with max_num_batched_tokens=4096.
INFO 02-18 01:07:21 [vllm.py:624] Asynchronous scheduling is enabled.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(EngineCore_DP0 pid=4119992) INFO 02-18 01:07:22 [core.py:96] Initializing a V1 LLM engine (v0.15.1) with config: model='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', speculative_config=None, tokenizer='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [4096], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 256, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': True}, 'local_cache_dir': None, 'static_all_moe_layers': []}
(EngineCore_DP0 pid=4119992) INFO 02-18 01:07:24 [parallel_state.py:1212] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://216.81.248.184:46031 backend=nccl
(EngineCore_DP0 pid=4119992) INFO 02-18 01:07:24 [parallel_state.py:1423] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A


Output:
Chunk ID: 179aed
Wall time: 5.0008 seconds
Process running with session ID 25613
Original token count: 426
Output:
(EngineCore_DP0 pid=4119992) INFO 02-18 01:07:25 [gpu_model_runner.py:4033] Starting to load model /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM...
(EngineCore_DP0 pid=4119992) INFO 02-18 01:07:25 [cuda.py:364] Using FLASH_ATTN attention backend out of potential backends: ('FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION')
(EngineCore_DP0 pid=4119992) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
(EngineCore_DP0 pid=4119992) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
(EngineCore_DP0 pid=4119992)
Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
(EngineCore_DP0 pid=4119992)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:01<00:00,  1.36s/it]
(EngineCore_DP0 pid=4119992)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:01<00:00,  1.36s/it]
(EngineCore_DP0 pid=4119992)
(EngineCore_DP0 pid=4119992) INFO 02-18 01:07:27 [default_loader.py:291] Loading weights took 1.42 seconds
(EngineCore_DP0 pid=4119992) INFO 02-18 01:07:28 [gpu_model_runner.py:4130] Model loading took 0.95 GiB memory and 1.930260 seconds
(EngineCore_DP0 pid=4119992) INFO 02-18 01:07:33 [backends.py:812] Using cache directory: /home/ubuntu/.cache/vllm/torch_compile_cache/f3ff13511c/rank_0_0/backbone for vLLM's torch.compile
(EngineCore_DP0 pid=4119992) INFO 02-18 01:07:33 [backends.py:872] Dynamo bytecode transform time: 4.19 s


Compile succeeded and the updated streaming run is executing; after this finishes I’ll validate the new per-request queue/scheduler/decode fields are present, then run baseline vs tuned vLLM configs for direct throughput comparison.


Output:
Chunk ID: bc5624
Wall time: 5.0023 seconds
Process running with session ID 25613
Original token count: 787
Output:
(EngineCore_DP0 pid=4119992) INFO 02-18 01:07:35 [backends.py:267] Directly load the compiled graph(s) for compile range (1, 4096) from the cache, took 0.632 s
(EngineCore_DP0 pid=4119992) INFO 02-18 01:07:35 [monitor.py:34] torch.compile takes 4.82 s in total
(EngineCore_DP0 pid=4119992) INFO 02-18 01:07:36 [gpu_worker.py:356] Available KV cache memory: 18.06 GiB
(EngineCore_DP0 pid=4119992) INFO 02-18 01:07:36 [kv_cache_utils.py:1307] GPU KV cache size: 1,577,712 tokens
(EngineCore_DP0 pid=4119992) INFO 02-18 01:07:36 [kv_cache_utils.py:1312] Maximum concurrency for 4,096 tokens per request: 385.18x
(EngineCore_DP0 pid=4119992)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   0%|          | 0/35 [00:00<?, ?it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  11%|█▏        | 4/35 [00:00<00:00, 32.91it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  23%|██▎       | 8/35 [00:00<00:00, 35.20it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  34%|███▍      | 12/35 [00:00<00:00, 36.40it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  46%|████▌     | 16/35 [00:00<00:00, 36.81it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  57%|█████▋    | 20/35 [00:00<00:00, 37.16it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  69%|██████▊   | 24/35 [00:00<00:00, 37.47it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  80%|████████  | 28/35 [00:00<00:00, 37.04it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  91%|█████████▏| 32/35 [00:00<00:00, 35.58it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 35/35 [00:00<00:00, 35.46it/s]
(EngineCore_DP0 pid=4119992)
Capturing CUDA graphs (decode, FULL):   0%|          | 0/19 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL):  16%|█▌        | 3/19 [00:00<00:00, 29.54it/s]
Capturing CUDA graphs (decode, FULL):  37%|███▋      | 7/19 [00:00<00:00, 34.04it/s]
Capturing CUDA graphs (decode, FULL):  58%|█████▊    | 11/19 [00:00<00:00, 34.72it/s]
Capturing CUDA graphs (decode, FULL):  79%|███████▉  | 15/19 [00:00<00:00, 35.62it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 36.37it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 35.40it/s]
(EngineCore_DP0 pid=4119992) INFO 02-18 01:07:38 [gpu_model_runner.py:5063] Graph capturing finished in 2 secs, took 0.28 GiB
(EngineCore_DP0 pid=4119992) INFO 02-18 01:07:38 [core.py:272] init engine (profile, create kv cache, warmup model) took 10.50 seconds
[2026-02-18 01:07:40] INFO tts_runtime.py:192: Initializing prompt builder...
[2026-02-18 01:07:40] INFO tts_runtime.py:199: Initializing BiCodec decoder...
/home/ubuntu/.local/lib/python3.10/site-packages/torch/nn/utils/weight_norm.py:144: FutureWarning: `torch.nn.utils.weight_norm` is deprecated in favor of `torch.nn.utils.parametrizations.weight_norm`.
  WeightNorm.apply(module, name, dim)


Output:
Chunk ID: dcaf40
Wall time: 5.0014 seconds
Process running with session ID 25613
Original token count: 663
Output:
[2026-02-18 01:07:48] INFO tts_runtime.py:206: Initializing TTS pipeline...
[2026-02-18 01:07:48] INFO tts_runtime.py:217: Initializing streaming pipeline...
[2026-02-18 01:07:48] INFO long_text_processor.py:112: LongTextProcessor initialized
[2026-02-18 01:07:48] INFO tts_runtime.py:274: ✅ TTS runtime initialized in 32558ms
[2026-02-18 01:07:48] INFO tts_runtime.py:275:    Model version: LLM
[2026-02-18 01:07:48] INFO tts_runtime.py:286: Speaker globals cache startup precompute skipped (disabled)
✅ vLLM engine initialized successfully!

🎉 Spark TTS Model ready for inference!

🎵 Loading BiCodec audio tokenizer from /home/ubuntu/veenaModal/models/spark_tts_4speaker...
   Device: cuda
Missing tensor: mel_transformer.spectrogram.window
Missing tensor: mel_transformer.mel_scale.fb
✅ BiCodec decoder initialized (sample rate: 16000Hz)
🚀 SparkTTSPipeline initialized
🌊 Veena3SlidingWindowPipeline initialized (sliding window: 28 tokens)

=== streaming concurrency=8 requests=8 ===
success=8/8 (100%)
wall=1.24s throughput=6.44 req/s
latency_ms avg=1137 p50=1113 p95=1184 max=1243
ttfb_ms avg=479 p50=482 p95=491 max=493
chunks_per_req avg=5.1 p50=5.0 p95=5.0 max=6.0
inter_chunk_ms avg=157.1 p50=227.0 p95=276.4 max=316.4
tail_emit_speed_x avg=5.64 p50=5.51 p95=6.00 max=6.08
timeline_to_first_batch_ms: avg=24.21 p50=24.37 p95=25.52 max=26.02
timeline_to_first_chunk_emitted_ms: avg=477.17 p50=480.58 p95=488.60 max=490.10
timeline_total_ms: avg=1125.10 p50=1109.66 p95=1181.44 max=1241.35
llm_time_in_queue_ms: avg=0.02 p50=0.02 p95=0.04 max=0.05
llm_scheduler_ms: avg=15.84 p50=14.70 p95=19.24 max=19.24
llm_first_token_ms: avg=23.74 p50=23.85 p95=25.09 max=25.62
llm_request_lifecycle_ms: avg=1021.13 p50=1010.90 p95=1091.63 max=1205.18
llm_generation_wall_ms: avg=1101.94 p50=1075.40 p95=1181.39 max=1206.93
llm_time_per_token_ms: avg=4.94 p50=4.87 p95=5.19 max=5.31
llm_batch_wall_ms_total: avg=1077.09 p50=1075.35 p95=1114.99 max=1206.90
llm_decode_wall_ms_total: avg=1052.92 p50=1050.63 p95=1092.76 max=1185.39
tokens_per_batch_p50: avg=1.00 p50=1.00 p95=1.00 max=1.00
bicodec_decode_calls: avg=4.12 p50=4.00 p95=4.00 max=5.00
bicodec_decode_wall_ms_total: avg=554.72 p50=557.66 p95=578.39 max=646.89
bicodec_decode_interval_ms_p50: avg=225.52 p50=232.04 p95=234.49 max=239.65
latency_over_timeline_ms: avg=11.79 p50=3.24 p95=36.49 max=39.50
timeline_share: llm=97.9% bicodec=49.3% other=0.0%
gpu: util_avg=66% util_max=94% mem_avg=23941MB mem_max=24205MB

results saved: /tmp/stream_metrics_smoke.json
ERROR 02-18 01:07:51 [core_client.py:605] Engine core proc EngineCore_DP0 died unexpectedly, shutting down client.


**Noticing overlapping timeline metrics**


Output:
Chunk ID: 83fc99
Wall time: 0.0516 seconds
Process exited with code 0
Original token count: 0
Output:


**Running baseline with detailed metrics**


cd /home/ubuntu/veenaModal && python3 scripts/stress_test_streaming_runtime.py --levels 64 --text short --max-tokens 1024 --timeout 240 --requests-multiplier 1 --min-requests 8 --chunking --no-such-option


Output:
Chunk ID: 48f30b
Wall time: 0.0511 seconds
Process exited with code 2
Original token count: 440
Output:
usage: stress_test_streaming_runtime.py [-h] [--levels LEVELS]
                                        [--text {short,medium,long}]
                                        [--max-tokens MAX_TOKENS]
                                        [--timeout TIMEOUT]
                                        [--requests-multiplier REQUESTS_MULTIPLIER]
                                        [--min-requests MIN_REQUESTS]
                                        [--chunking] [--gpu-memory GPU_MEMORY]
                                        [--stream-output-kind {delta,cumulative}]
                                        [--num-engines NUM_ENGINES]
                                        [--max-num-batched-tokens MAX_NUM_BATCHED_TOKENS]
                                        [--max-num-seqs MAX_NUM_SEQS]
                                        [--enable-chunked-prefill]
                                        [--disable-chunked-prefill]
                                        [--disable-prefix-caching]
                                        [--disable-engine-stats-logs]
                                        [--enforce-eager]
                                        [--stream-decode-interval STREAM_DECODE_INTERVAL]
                                        [--stream-window-size STREAM_WINDOW_SIZE]
                                        [--stream-min-semantic-first STREAM_MIN_SEMANTIC_FIRST]
                                        [--stream-crossfade-ms STREAM_CROSSFADE_MS]
                                        [--disable-windowed-decode]
                                        [--model-path MODEL_PATH]
                                        [--output OUTPUT]
stress_test_streaming_runtime.py: error: unrecognized arguments: --no-such-option


**Correcting command usage**


The harness is working and producing the detailed timing fields. I hit an argument typo on the first high-concurrency run; rerunning now with the correct baseline and then a tuned vLLM config for direct A/B.


cd /home/ubuntu/veenaModal && python3 scripts/stress_test_streaming_runtime.py --levels 64 --text short --max-tokens 1024 --timeout 240 --requests-multiplier 1 --min-requests 8 --gpu-memory 0.25 --stream-output-kind delta --output /tmp/stream_vllm_baseline_detailed.json


Output:
Chunk ID: 75114b
Wall time: 1.0012 seconds
Process running with session ID 74544
Original token count: 0
Output:


Output:
Chunk ID: 5325e0
Wall time: 5.0013 seconds
Process running with session ID 74544
Original token count: 1277
Output:
[2026-02-18 01:08:11] INFO tts_runtime.py:183: Loading SparkTTS model with vLLM...
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
🚀 Initializing Spark TTS Model
📁 Model path: /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
🗣  Speakers: 12 predefined (lipakshi, vardan, reet, Nandini, krishna, anika, adarsh, Nilay, Aarvi, Asha, Bittu, Mira)
🎭 Emotions: 10 tags in [bracket] format

📝 Loading tokenizer from model...
✅ Tokenizer loaded: 165168 tokens

🔍 Validating 12 speakers for Spark TTS...
  lipakshi → <|speaker_0|>: 7 tokens
  vardan → <|speaker_1|>: 7 tokens
  reet → <|speaker_2|>: 7 tokens
  Nandini → <|speaker_3|>: 7 tokens
  krishna → <|speaker_4|>: 7 tokens
  anika → <|speaker_5|>: 7 tokens
  adarsh → <|speaker_6|>: 7 tokens
  Nilay → <|speaker_7|>: 7 tokens
  Aarvi → <|speaker_8|>: 7 tokens
  Asha → <|speaker_9|>: 7 tokens
  Bittu → <|speaker_10|>: 8 tokens
  Mira → <|speaker_11|>: 8 tokens
✅ All 12 speakers validated with mapping

🔍 Validating 10 emotion tags for Spark TTS...
✅ All 10 emotion tags validated

📋 Sample emotion tags:
  [angry]: 1 tokens
  [curious]: 1 tokens
  [excited]: 1 tokens
  [giggle]: 1 tokens
  [laughs harder]: 1 tokens
  ...

🔧 Initializing vLLM engine with optimizations...
  - dtype: bfloat16
  - max_model_len: 4096
  - gpu_memory_utilization: 0.25
  - enable_prefix_caching: True
  - enforce_eager (disable CUDA graphs): False
INFO 02-18 01:08:12 [model.py:541] Resolved architecture: Qwen2ForCausalLM
INFO 02-18 01:08:12 [model.py:1561] Using max model len 4096
INFO 02-18 01:08:12 [scheduler.py:226] Chunked prefill is enabled with max_num_batched_tokens=4096.
INFO 02-18 01:08:12 [vllm.py:624] Asynchronous scheduling is enabled.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(EngineCore_DP0 pid=4120694) INFO 02-18 01:08:13 [core.py:96] Initializing a V1 LLM engine (v0.15.1) with config: model='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', speculative_config=None, tokenizer='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [4096], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 256, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': True}, 'local_cache_dir': None, 'static_all_moe_layers': []}


**Polling for completion**


Output:
Chunk ID: 4d50ca
Wall time: 5.0011 seconds
Process running with session ID 74544
Original token count: 437
Output:
(EngineCore_DP0 pid=4120694) INFO 02-18 01:08:15 [parallel_state.py:1212] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://216.81.248.184:44121 backend=nccl
(EngineCore_DP0 pid=4120694) INFO 02-18 01:08:15 [parallel_state.py:1423] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
(EngineCore_DP0 pid=4120694) INFO 02-18 01:08:15 [gpu_model_runner.py:4033] Starting to load model /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM...
(EngineCore_DP0 pid=4120694) INFO 02-18 01:08:16 [cuda.py:364] Using FLASH_ATTN attention backend out of potential backends: ('FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION')
(EngineCore_DP0 pid=4120694) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
(EngineCore_DP0 pid=4120694) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
(EngineCore_DP0 pid=4120694)
Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
(EngineCore_DP0 pid=4120694)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.46it/s]
(EngineCore_DP0 pid=4120694)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.46it/s]
(EngineCore_DP0 pid=4120694)
(EngineCore_DP0 pid=4120694) INFO 02-18 01:08:17 [default_loader.py:291] Loading weights took 0.75 seconds
(EngineCore_DP0 pid=4120694) INFO 02-18 01:08:18 [gpu_model_runner.py:4130] Model loading took 0.95 GiB memory and 1.255425 seconds


Output:
Chunk ID: 9a5fcb
Wall time: 5.0011 seconds
Process running with session ID 74544
Original token count: 762
Output:
(EngineCore_DP0 pid=4120694) INFO 02-18 01:08:23 [backends.py:812] Using cache directory: /home/ubuntu/.cache/vllm/torch_compile_cache/f3ff13511c/rank_0_0/backbone for vLLM's torch.compile
(EngineCore_DP0 pid=4120694) INFO 02-18 01:08:23 [backends.py:872] Dynamo bytecode transform time: 4.21 s
(EngineCore_DP0 pid=4120694) INFO 02-18 01:08:26 [backends.py:267] Directly load the compiled graph(s) for compile range (1, 4096) from the cache, took 0.658 s
(EngineCore_DP0 pid=4120694) INFO 02-18 01:08:26 [monitor.py:34] torch.compile takes 4.87 s in total
(EngineCore_DP0 pid=4120694) INFO 02-18 01:08:27 [gpu_worker.py:356] Available KV cache memory: 18.06 GiB
(EngineCore_DP0 pid=4120694) INFO 02-18 01:08:27 [kv_cache_utils.py:1307] GPU KV cache size: 1,577,712 tokens
(EngineCore_DP0 pid=4120694) INFO 02-18 01:08:27 [kv_cache_utils.py:1312] Maximum concurrency for 4,096 tokens per request: 385.18x
(EngineCore_DP0 pid=4120694)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   0%|          | 0/35 [00:00<?, ?it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  11%|█▏        | 4/35 [00:00<00:00, 32.31it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  23%|██▎       | 8/35 [00:00<00:00, 34.73it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  34%|███▍      | 12/35 [00:00<00:00, 35.61it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  46%|████▌     | 16/35 [00:00<00:00, 35.76it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  57%|█████▋    | 20/35 [00:00<00:00, 35.35it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  69%|██████▊   | 24/35 [00:00<00:00, 35.31it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  80%|████████  | 28/35 [00:00<00:00, 35.08it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  91%|█████████▏| 32/35 [00:00<00:00, 33.72it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 35/35 [00:01<00:00, 33.95it/s]
(EngineCore_DP0 pid=4120694)
Capturing CUDA graphs (decode, FULL):   0%|          | 0/19 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL):  16%|█▌        | 3/19 [00:00<00:00, 28.80it/s]
Capturing CUDA graphs (decode, FULL):  37%|███▋      | 7/19 [00:00<00:00, 33.18it/s]
Capturing CUDA graphs (decode, FULL):  58%|█████▊    | 11/19 [00:00<00:00, 34.32it/s]
Capturing CUDA graphs (decode, FULL):  79%|███████▉  | 15/19 [00:00<00:00, 34.76it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 34.58it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 34.10it/s]
(EngineCore_DP0 pid=4120694) INFO 02-18 01:08:29 [gpu_model_runner.py:5063] Graph capturing finished in 3 secs, took 0.28 GiB
(EngineCore_DP0 pid=4120694) INFO 02-18 01:08:29 [core.py:272] init engine (profile, create kv cache, warmup model) took 11.34 seconds


Output:
Chunk ID: 3172d4
Wall time: 5.0014 seconds
Process running with session ID 74544
Original token count: 99
Output:
[2026-02-18 01:08:31] INFO tts_runtime.py:192: Initializing prompt builder...
[2026-02-18 01:08:31] INFO tts_runtime.py:199: Initializing BiCodec decoder...
/home/ubuntu/.local/lib/python3.10/site-packages/torch/nn/utils/weight_norm.py:144: FutureWarning: `torch.nn.utils.weight_norm` is deprecated in favor of `torch.nn.utils.parametrizations.weight_norm`.
  WeightNorm.apply(module, name, dim)


Output:
Chunk ID: 521727
Wall time: 5.0013 seconds
Process running with session ID 74544
Original token count: 673
Output:
[2026-02-18 01:08:40] INFO tts_runtime.py:206: Initializing TTS pipeline...
[2026-02-18 01:08:40] INFO tts_runtime.py:217: Initializing streaming pipeline...
[2026-02-18 01:08:40] INFO long_text_processor.py:112: LongTextProcessor initialized
[2026-02-18 01:08:40] INFO tts_runtime.py:274: ✅ TTS runtime initialized in 34487ms
[2026-02-18 01:08:40] INFO tts_runtime.py:275:    Model version: LLM
[2026-02-18 01:08:40] INFO tts_runtime.py:286: Speaker globals cache startup precompute skipped (disabled)
✅ vLLM engine initialized successfully!

🎉 Spark TTS Model ready for inference!

🎵 Loading BiCodec audio tokenizer from /home/ubuntu/veenaModal/models/spark_tts_4speaker...
   Device: cuda
Missing tensor: mel_transformer.spectrogram.window
Missing tensor: mel_transformer.mel_scale.fb
✅ BiCodec decoder initialized (sample rate: 16000Hz)
🚀 SparkTTSPipeline initialized
🌊 Veena3SlidingWindowPipeline initialized (sliding window: 28 tokens)

=== streaming concurrency=64 requests=64 ===
success=64/64 (100%)
wall=3.70s throughput=17.31 req/s
latency_ms avg=3690 p50=3695 p95=3696 max=3696
ttfb_ms avg=1532 p50=1536 p95=1976 max=2200
chunks_per_req avg=3.5 p50=4.0 p95=4.0 max=4.0
inter_chunk_ms avg=816.7 p50=1036.0 p95=1985.4 max=2087.6
tail_emit_speed_x avg=1.26 p50=1.17 p95=1.81 max=2.11
timeline_to_first_batch_ms: avg=39.86 p50=40.06 p95=45.62 max=47.21
timeline_to_first_chunk_emitted_ms: avg=1518.84 p50=1522.73 p95=1952.86 max=2176.56
timeline_total_ms: avg=3586.04 p50=3668.39 p95=3690.47 max=3692.91
llm_time_in_queue_ms: avg=0.06 p50=0.06 p95=0.12 max=0.14
llm_scheduler_ms: avg=29.38 p50=26.26 p95=33.46 max=33.46
llm_first_token_ms: avg=37.79 p50=37.99 p95=44.72 max=46.40
llm_request_lifecycle_ms: avg=1590.34 p50=1610.20 p95=1756.50 max=1886.55
llm_generation_wall_ms: avg=3564.99 p50=3668.27 p95=3690.42 max=3692.87
llm_time_per_token_ms: avg=16.17 p50=16.02 p95=18.18 max=20.06
llm_batch_wall_ms_total: avg=2398.95 p50=2234.14 p95=3579.05 max=3669.73
llm_decode_wall_ms_total: avg=2359.12 p50=2190.37 p95=3535.99 max=3632.44
tokens_per_batch_p50: avg=1.00 p50=1.00 p95=1.00 max=1.00
bicodec_decode_calls: avg=2.53 p50=3.00 p95=3.00 max=3.00
bicodec_decode_wall_ms_total: avg=3057.97 p50=3141.73 p95=3170.45 max=3177.87
bicodec_decode_interval_ms_p50: avg=1185.80 p50=1255.62 p95=1966.86 max=2015.62
latency_over_timeline_ms: avg=104.25 p50=26.41 p95=322.00 max=895.46
timeline_share: llm=99.4% bicodec=85.3% other=0.0%
gpu: util_avg=63% util_max=94% mem_avg=24380MB mem_max=24863MB

results saved: /tmp/stream_vllm_baseline_detailed.json
ERROR 02-18 01:08:46 [core_client.py:605] Engine core proc EngineCore_DP0 died unexpectedly, shutting down client.


**Planning exit check implementation**


Output:
Chunk ID: f8f5af
Wall time: 0.0506 seconds
Process exited with code 0
Original token count: 0
Output:


**Tuning vLLM config knobs**


cd /home/ubuntu/veenaModal && python3 scripts/stress_test_streaming_runtime.py --levels 64 --text short --max-tokens 1024 --timeout 240 --requests-multiplier 1 --min-requests 8 --gpu-memory 0.4 --stream-output-kind delta --max-num-batched-tokens 8192 --max-num-seqs 128 --stream-decode-interval 64 --stream-window-size 96 --stream-min-semantic-first 8 --output /tmp/stream_vllm_tuned_detailed.json


Output:
Chunk ID: d9aa14
Wall time: 1.0011 seconds
Process running with session ID 45777
Original token count: 0
Output:


Summary: auto


Output:
Chunk ID: e6cd06
Wall time: 5.0008 seconds
Process running with session ID 45777
Original token count: 1277
Output:
[2026-02-18 01:09:09] INFO tts_runtime.py:183: Loading SparkTTS model with vLLM...
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
🚀 Initializing Spark TTS Model
📁 Model path: /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
🗣  Speakers: 12 predefined (lipakshi, vardan, reet, Nandini, krishna, anika, adarsh, Nilay, Aarvi, Asha, Bittu, Mira)
🎭 Emotions: 10 tags in [bracket] format

📝 Loading tokenizer from model...
✅ Tokenizer loaded: 165168 tokens

🔍 Validating 12 speakers for Spark TTS...
  lipakshi → <|speaker_0|>: 7 tokens
  vardan → <|speaker_1|>: 7 tokens
  reet → <|speaker_2|>: 7 tokens
  Nandini → <|speaker_3|>: 7 tokens
  krishna → <|speaker_4|>: 7 tokens
  anika → <|speaker_5|>: 7 tokens
  adarsh → <|speaker_6|>: 7 tokens
  Nilay → <|speaker_7|>: 7 tokens
  Aarvi → <|speaker_8|>: 7 tokens
  Asha → <|speaker_9|>: 7 tokens
  Bittu → <|speaker_10|>: 8 tokens
  Mira → <|speaker_11|>: 8 tokens
✅ All 12 speakers validated with mapping

🔍 Validating 10 emotion tags for Spark TTS...
✅ All 10 emotion tags validated

📋 Sample emotion tags:
  [angry]: 1 tokens
  [curious]: 1 tokens
  [excited]: 1 tokens
  [giggle]: 1 tokens
  [laughs harder]: 1 tokens
  ...

🔧 Initializing vLLM engine with optimizations...
  - dtype: bfloat16
  - max_model_len: 4096
  - gpu_memory_utilization: 0.4
  - enable_prefix_caching: True
  - enforce_eager (disable CUDA graphs): False
INFO 02-18 01:09:10 [model.py:541] Resolved architecture: Qwen2ForCausalLM
INFO 02-18 01:09:10 [model.py:1561] Using max model len 4096
INFO 02-18 01:09:10 [scheduler.py:226] Chunked prefill is enabled with max_num_batched_tokens=8192.
INFO 02-18 01:09:10 [vllm.py:624] Asynchronous scheduling is enabled.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(EngineCore_DP0 pid=4121517) INFO 02-18 01:09:11 [core.py:96] Initializing a V1 LLM engine (v0.15.1) with config: model='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', speculative_config=None, tokenizer='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 256, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': True}, 'local_cache_dir': None, 'static_all_moe_layers': []}


Output:
Chunk ID: 11c2c4
Wall time: 5.0012 seconds
Process running with session ID 45777
Original token count: 511
Output:
(EngineCore_DP0 pid=4121517) INFO 02-18 01:09:13 [parallel_state.py:1212] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://216.81.248.184:55523 backend=nccl
(EngineCore_DP0 pid=4121517) INFO 02-18 01:09:13 [parallel_state.py:1423] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
(EngineCore_DP0 pid=4121517) INFO 02-18 01:09:13 [gpu_model_runner.py:4033] Starting to load model /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM...
(EngineCore_DP0 pid=4121517) INFO 02-18 01:09:14 [cuda.py:364] Using FLASH_ATTN attention backend out of potential backends: ('FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION')
(EngineCore_DP0 pid=4121517) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
(EngineCore_DP0 pid=4121517) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
(EngineCore_DP0 pid=4121517)
Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
(EngineCore_DP0 pid=4121517)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:01<00:00,  1.29s/it]
(EngineCore_DP0 pid=4121517)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:01<00:00,  1.29s/it]
(EngineCore_DP0 pid=4121517)
(EngineCore_DP0 pid=4121517) INFO 02-18 01:09:15 [default_loader.py:291] Loading weights took 1.37 seconds
(EngineCore_DP0 pid=4121517) INFO 02-18 01:09:16 [gpu_model_runner.py:4130] Model loading took 0.95 GiB memory and 1.868593 seconds
(EngineCore_DP0 pid=4121517) INFO 02-18 01:09:20 [backends.py:812] Using cache directory: /home/ubuntu/.cache/vllm/torch_compile_cache/363c797fe7/rank_0_0/backbone for vLLM's torch.compile
(EngineCore_DP0 pid=4121517) INFO 02-18 01:09:20 [backends.py:872] Dynamo bytecode transform time: 4.21 s


Output:
Chunk ID: 523a51
Wall time: 5.0005 seconds
Process running with session ID 45777
Original token count: 206
Output:
(EngineCore_DP0 pid=4121517) INFO 02-18 01:09:25 [backends.py:302] Cache the graph of compile range (1, 8192) for later use
(EngineCore_DP0 pid=4121517) INFO 02-18 01:09:28 [backends.py:319] Compiling a graph for compile range (1, 8192) takes 5.07 s
(EngineCore_DP0 pid=4121517) INFO 02-18 01:09:28 [monitor.py:34] torch.compile takes 9.28 s in total
(EngineCore_DP0 pid=4121517) INFO 02-18 01:09:29 [gpu_worker.py:356] Available KV cache memory: 29.94 GiB
(EngineCore_DP0 pid=4121517) INFO 02-18 01:09:29 [kv_cache_utils.py:1307] GPU KV cache size: 2,615,968 tokens
(EngineCore_DP0 pid=4121517) INFO 02-18 01:09:29 [kv_cache_utils.py:1312] Maximum concurrency for 4,096 tokens per request: 638.66x
(EngineCore_DP0 pid=4121517)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   0%|          | 0/35 [00:00<?, ?it/s]


Output:
Chunk ID: f828a4
Wall time: 5.0009 seconds
Process running with session ID 45777
Original token count: 603
Output:

Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  11%|█▏        | 4/35 [00:00<00:00, 31.30it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  23%|██▎       | 8/35 [00:00<00:00, 33.83it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  34%|███▍      | 12/35 [00:00<00:00, 34.47it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  46%|████▌     | 16/35 [00:00<00:00, 34.98it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  57%|█████▋    | 20/35 [00:00<00:00, 35.28it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  69%|██████▊   | 24/35 [00:00<00:00, 35.61it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  80%|████████  | 28/35 [00:00<00:00, 35.16it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  91%|█████████▏| 32/35 [00:00<00:00, 33.59it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 35/35 [00:01<00:00, 33.58it/s]
(EngineCore_DP0 pid=4121517)
Capturing CUDA graphs (decode, FULL):   0%|          | 0/19 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL):  16%|█▌        | 3/19 [00:00<00:00, 27.21it/s]
Capturing CUDA graphs (decode, FULL):  37%|███▋      | 7/19 [00:00<00:00, 31.62it/s]
Capturing CUDA graphs (decode, FULL):  58%|█████▊    | 11/19 [00:00<00:00, 33.89it/s]
Capturing CUDA graphs (decode, FULL):  79%|███████▉  | 15/19 [00:00<00:00, 34.81it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 35.54it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 34.29it/s]
(EngineCore_DP0 pid=4121517) INFO 02-18 01:09:31 [gpu_model_runner.py:5063] Graph capturing finished in 2 secs, took 0.28 GiB
(EngineCore_DP0 pid=4121517) INFO 02-18 01:09:31 [core.py:272] init engine (profile, create kv cache, warmup model) took 15.15 seconds
[2026-02-18 01:09:33] INFO tts_runtime.py:192: Initializing prompt builder...
[2026-02-18 01:09:33] INFO tts_runtime.py:199: Initializing BiCodec decoder...
/home/ubuntu/.local/lib/python3.10/site-packages/torch/nn/utils/weight_norm.py:144: FutureWarning: `torch.nn.utils.weight_norm` is deprecated in favor of `torch.nn.utils.parametrizations.weight_norm`.
  WeightNorm.apply(module, name, dim)


Output:
Chunk ID: 83900c
Wall time: 5.0015 seconds
Process running with session ID 45777
Original token count: 127
Output:
[2026-02-18 01:09:41] INFO tts_runtime.py:206: Initializing TTS pipeline...
[2026-02-18 01:09:41] INFO tts_runtime.py:217: Initializing streaming pipeline...
[2026-02-18 01:09:41] INFO long_text_processor.py:112: LongTextProcessor initialized
[2026-02-18 01:09:41] INFO tts_runtime.py:274: ✅ TTS runtime initialized in 37767ms
[2026-02-18 01:09:41] INFO tts_runtime.py:275:    Model version: LLM
[2026-02-18 01:09:41] INFO tts_runtime.py:286: Speaker globals cache startup precompute skipped (disabled)


**Polling final results**


Output:
Chunk ID: b247b8
Wall time: 0.0510 seconds
Process exited with code 0
Original token count: 545
Output:
✅ vLLM engine initialized successfully!

🎉 Spark TTS Model ready for inference!

🎵 Loading BiCodec audio tokenizer from /home/ubuntu/veenaModal/models/spark_tts_4speaker...
   Device: cuda
Missing tensor: mel_transformer.spectrogram.window
Missing tensor: mel_transformer.mel_scale.fb
✅ BiCodec decoder initialized (sample rate: 16000Hz)
🚀 SparkTTSPipeline initialized
🌊 Veena3SlidingWindowPipeline initialized (sliding window: 28 tokens)

=== streaming concurrency=64 requests=64 ===
success=64/64 (100%)
wall=3.82s throughput=16.77 req/s
latency_ms avg=3813 p50=3815 p95=3816 max=3816
ttfb_ms avg=1575 p50=1575 p95=1992 max=2631
chunks_per_req avg=3.6 p50=4.0 p95=4.0 max=4.0
inter_chunk_ms avg=810.7 p50=39.5 p95=2122.5 max=2193.1
tail_emit_speed_x avg=1.45 p50=1.38 p95=1.75 max=2.14
timeline_to_first_batch_ms: avg=40.78 p50=40.98 p95=45.81 max=47.28
timeline_to_first_chunk_emitted_ms: avg=1560.57 p50=1561.58 p95=1967.86 max=2604.89
timeline_total_ms: avg=3637.94 p50=3785.68 p95=3807.51 max=3809.74
llm_time_in_queue_ms: avg=0.06 p50=0.06 p95=0.12 max=0.14
llm_scheduler_ms: avg=29.58 p50=28.91 p95=32.77 max=32.77
llm_first_token_ms: avg=38.72 p50=38.73 p95=44.56 max=46.21
llm_request_lifecycle_ms: avg=1618.41 p50=1640.35 p95=1758.59 max=1884.62
llm_generation_wall_ms: avg=3613.23 p50=3785.50 p95=3807.46 max=3809.70
llm_time_per_token_ms: avg=16.00 p50=15.98 p95=18.56 max=20.59
llm_batch_wall_ms_total: avg=2694.85 p50=3056.39 p95=3664.66 max=3758.46
llm_decode_wall_ms_total: avg=2654.11 p50=3009.53 p95=3619.95 max=3714.08
tokens_per_batch_p50: avg=1.00 p50=1.00 p95=1.00 max=1.00
bicodec_decode_calls: avg=2.56 p50=3.00 p95=3.00 max=3.00
bicodec_decode_wall_ms_total: avg=2994.06 p50=3133.48 p95=3167.89 max=3180.76
bicodec_decode_interval_ms_p50: avg=943.14 p50=630.68 p95=1944.68 max=2149.12
latency_over_timeline_ms: avg=175.32 p50=28.95 p95=668.64 max=1083.82
timeline_share: llm=99.3% bicodec=82.3% other=0.0%
gpu: util_avg=57% util_max=97% mem_avg=36548MB mem_max=36959MB

results saved: /tmp/stream_vllm_tuned_detailed.json
ERROR 02-18 01:09:47 [core_client.py:605] Engine core proc EngineCore_DP0 died unexpectedly, shutting down client.


**Testing vLLM batching parameters**


cd /home/ubuntu/veenaModal && python3 scripts/stress_test_streaming_runtime.py --levels 64 --text short --max-tokens 1024 --timeout 240 --requests-multiplier 1 --min-requests 8 --gpu-memory 0.4 --stream-output-kind delta --max-num-batched-tokens 8192 --max-num-seqs 128 --output /tmp/stream_vllm_tuned_vllm_only.json


Output:
Chunk ID: 26a612
Wall time: 1.0001 seconds
Process running with session ID 37870
Original token count: 0
Output:


**Implementing polling loop**


Output:
Chunk ID: 273c94
Wall time: 5.0009 seconds
Process running with session ID 37870
Original token count: 1787
Output:
[2026-02-18 01:10:02] INFO tts_runtime.py:183: Loading SparkTTS model with vLLM...
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
🚀 Initializing Spark TTS Model
📁 Model path: /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
🗣  Speakers: 12 predefined (lipakshi, vardan, reet, Nandini, krishna, anika, adarsh, Nilay, Aarvi, Asha, Bittu, Mira)
🎭 Emotions: 10 tags in [bracket] format

📝 Loading tokenizer from model...
✅ Tokenizer loaded: 165168 tokens

🔍 Validating 12 speakers for Spark TTS...
  lipakshi → <|speaker_0|>: 7 tokens
  vardan → <|speaker_1|>: 7 tokens
  reet → <|speaker_2|>: 7 tokens
  Nandini → <|speaker_3|>: 7 tokens
  krishna → <|speaker_4|>: 7 tokens
  anika → <|speaker_5|>: 7 tokens
  adarsh → <|speaker_6|>: 7 tokens
  Nilay → <|speaker_7|>: 7 tokens
  Aarvi → <|speaker_8|>: 7 tokens
  Asha → <|speaker_9|>: 7 tokens
  Bittu → <|speaker_10|>: 8 tokens
  Mira → <|speaker_11|>: 8 tokens
✅ All 12 speakers validated with mapping

🔍 Validating 10 emotion tags for Spark TTS...
✅ All 10 emotion tags validated

📋 Sample emotion tags:
  [angry]: 1 tokens
  [curious]: 1 tokens
  [excited]: 1 tokens
  [giggle]: 1 tokens
  [laughs harder]: 1 tokens
  ...

🔧 Initializing vLLM engine with optimizations...
  - dtype: bfloat16
  - max_model_len: 4096
  - gpu_memory_utilization: 0.4
  - enable_prefix_caching: True
  - enforce_eager (disable CUDA graphs): False
INFO 02-18 01:10:03 [model.py:541] Resolved architecture: Qwen2ForCausalLM
INFO 02-18 01:10:03 [model.py:1561] Using max model len 4096
INFO 02-18 01:10:03 [scheduler.py:226] Chunked prefill is enabled with max_num_batched_tokens=8192.
INFO 02-18 01:10:03 [vllm.py:624] Asynchronous scheduling is enabled.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(EngineCore_DP0 pid=4122239) INFO 02-18 01:10:04 [core.py:96] Initializing a V1 LLM engine (v0.15.1) with config: model='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', speculative_config=None, tokenizer='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 256, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': True}, 'local_cache_dir': None, 'static_all_moe_layers': []}
(EngineCore_DP0 pid=4122239) INFO 02-18 01:10:06 [parallel_state.py:1212] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://216.81.248.184:55987 backend=nccl
(EngineCore_DP0 pid=4122239) INFO 02-18 01:10:06 [parallel_state.py:1423] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
(EngineCore_DP0 pid=4122239) INFO 02-18 01:10:07 [gpu_model_runner.py:4033] Starting to load model /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM...
(EngineCore_DP0 pid=4122239) INFO 02-18 01:10:07 [cuda.py:364] Using FLASH_ATTN attention backend out of potential backends: ('FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION')
(EngineCore_DP0 pid=4122239) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
(EngineCore_DP0 pid=4122239) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
(EngineCore_DP0 pid=4122239)
Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
(EngineCore_DP0 pid=4122239)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.79it/s]
(EngineCore_DP0 pid=4122239)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.79it/s]
(EngineCore_DP0 pid=4122239)
(EngineCore_DP0 pid=4122239) INFO 02-18 01:10:08 [default_loader.py:291] Loading weights took 0.63 seconds
(EngineCore_DP0 pid=4122239) INFO 02-18 01:10:09 [gpu_model_runner.py:4130] Model loading took 0.95 GiB memory and 1.167926 seconds
(EngineCore_DP0 pid=4122239) INFO 02-18 01:10:13 [backends.py:812] Using cache directory: /home/ubuntu/.cache/vllm/torch_compile_cache/363c797fe7/rank_0_0/backbone for vLLM's torch.compile
(EngineCore_DP0 pid=4122239) INFO 02-18 01:10:13 [backends.py:872] Dynamo bytecode transform time: 4.23 s


**Considering completion polling**


Output:
Chunk ID: c62c14
Wall time: 5.0048 seconds
Process running with session ID 37870
Original token count: 757
Output:
(EngineCore_DP0 pid=4122239) INFO 02-18 01:10:16 [backends.py:267] Directly load the compiled graph(s) for compile range (1, 8192) from the cache, took 0.663 s
(EngineCore_DP0 pid=4122239) INFO 02-18 01:10:16 [monitor.py:34] torch.compile takes 4.89 s in total
(EngineCore_DP0 pid=4122239) INFO 02-18 01:10:17 [gpu_worker.py:356] Available KV cache memory: 29.94 GiB
(EngineCore_DP0 pid=4122239) INFO 02-18 01:10:17 [kv_cache_utils.py:1307] GPU KV cache size: 2,615,888 tokens
(EngineCore_DP0 pid=4122239) INFO 02-18 01:10:17 [kv_cache_utils.py:1312] Maximum concurrency for 4,096 tokens per request: 638.64x
(EngineCore_DP0 pid=4122239)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   0%|          | 0/35 [00:00<?, ?it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   9%|▊         | 3/35 [00:00<00:01, 28.53it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  20%|██        | 7/35 [00:00<00:00, 32.35it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  31%|███▏      | 11/35 [00:00<00:00, 33.73it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  43%|████▎     | 15/35 [00:00<00:00, 33.19it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  54%|█████▍    | 19/35 [00:00<00:00, 33.52it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  66%|██████▌   | 23/35 [00:00<00:00, 33.37it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  77%|███████▋  | 27/35 [00:00<00:00, 33.35it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  89%|████████▊ | 31/35 [00:00<00:00, 32.07it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 35/35 [00:01<00:00, 30.56it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 35/35 [00:01<00:00, 31.98it/s]
(EngineCore_DP0 pid=4122239)
Capturing CUDA graphs (decode, FULL):   0%|          | 0/19 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL):  16%|█▌        | 3/19 [00:00<00:00, 28.03it/s]
Capturing CUDA graphs (decode, FULL):  37%|███▋      | 7/19 [00:00<00:00, 31.32it/s]
Capturing CUDA graphs (decode, FULL):  58%|█████▊    | 11/19 [00:00<00:00, 32.27it/s]
Capturing CUDA graphs (decode, FULL):  79%|███████▉  | 15/19 [00:00<00:00, 32.39it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 32.26it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 31.95it/s]
(EngineCore_DP0 pid=4122239) INFO 02-18 01:10:20 [gpu_model_runner.py:5063] Graph capturing finished in 3 secs, took 0.28 GiB
(EngineCore_DP0 pid=4122239) INFO 02-18 01:10:20 [core.py:272] init engine (profile, create kv cache, warmup model) took 11.04 seconds
[2026-02-18 01:10:21] INFO tts_runtime.py:192: Initializing prompt builder...
[2026-02-18 01:10:21] INFO tts_runtime.py:199: Initializing BiCodec decoder...


Output:
Chunk ID: c81239
Wall time: 5.0010 seconds
Process running with session ID 37870
Original token count: 186
Output:
/home/ubuntu/.local/lib/python3.10/site-packages/torch/nn/utils/weight_norm.py:144: FutureWarning: `torch.nn.utils.weight_norm` is deprecated in favor of `torch.nn.utils.parametrizations.weight_norm`.
  WeightNorm.apply(module, name, dim)
[2026-02-18 01:10:28] INFO tts_runtime.py:206: Initializing TTS pipeline...
[2026-02-18 01:10:28] INFO tts_runtime.py:217: Initializing streaming pipeline...
[2026-02-18 01:10:28] INFO long_text_processor.py:112: LongTextProcessor initialized
[2026-02-18 01:10:28] INFO tts_runtime.py:274: ✅ TTS runtime initialized in 31778ms
[2026-02-18 01:10:28] INFO tts_runtime.py:275:    Model version: LLM
[2026-02-18 01:10:28] INFO tts_runtime.py:286: Speaker globals cache startup precompute skipped (disabled)


**Noting potential quick completion**


Output:
Chunk ID: 1d5c7f
Wall time: 3.7676 seconds
Process exited with code 0
Original token count: 546
Output:
✅ vLLM engine initialized successfully!

🎉 Spark TTS Model ready for inference!

🎵 Loading BiCodec audio tokenizer from /home/ubuntu/veenaModal/models/spark_tts_4speaker...
   Device: cuda
Missing tensor: mel_transformer.spectrogram.window
Missing tensor: mel_transformer.mel_scale.fb
✅ BiCodec decoder initialized (sample rate: 16000Hz)
🚀 SparkTTSPipeline initialized
🌊 Veena3SlidingWindowPipeline initialized (sliding window: 28 tokens)

=== streaming concurrency=64 requests=64 ===
success=64/64 (100%)
wall=4.06s throughput=15.77 req/s
latency_ms avg=4053 p50=4058 p95=4059 max=4059
ttfb_ms avg=1653 p50=1650 p95=2161 max=2430
chunks_per_req avg=3.5 p50=4.0 p95=4.0 max=4.0
inter_chunk_ms avg=919.2 p50=362.1 p95=2314.9 max=2518.2
tail_emit_speed_x avg=1.12 p50=1.05 p95=1.60 max=1.88
timeline_to_first_batch_ms: avg=51.86 p50=52.06 p95=58.23 max=59.84
timeline_to_first_chunk_emitted_ms: avg=1639.57 p50=1637.03 p95=2137.78 max=2404.19
timeline_total_ms: avg=3951.85 p50=4030.62 p95=4053.08 max=4055.63
llm_time_in_queue_ms: avg=0.08 p50=0.08 p95=0.15 max=0.17
llm_scheduler_ms: avg=35.80 p50=27.82 p95=44.91 max=44.91
llm_first_token_ms: avg=49.79 p50=49.98 p95=56.93 max=58.66
llm_request_lifecycle_ms: avg=1656.06 p50=1666.72 p95=1832.68 max=1976.87
llm_generation_wall_ms: avg=3927.71 p50=4030.57 p95=4053.04 max=4055.52
llm_time_per_token_ms: avg=17.76 p50=17.77 p95=19.62 max=21.80
llm_batch_wall_ms_total: avg=2776.84 p50=2443.29 p95=3923.49 max=4002.59
llm_decode_wall_ms_total: avg=2725.01 p50=2383.48 p95=3872.85 max=3948.84
tokens_per_batch_p50: avg=1.00 p50=1.00 p95=1.00 max=1.00
bicodec_decode_calls: avg=2.52 p50=3.00 p95=3.00 max=3.00
bicodec_decode_wall_ms_total: avg=3389.40 p50=3469.23 p95=3499.55 max=3508.25
bicodec_decode_interval_ms_p50: avg=1204.37 p50=1364.82 p95=2292.75 max=2325.54
latency_over_timeline_ms: avg=101.16 p50=26.10 p95=369.78 max=890.39
timeline_share: llm=99.4% bicodec=85.8% other=0.0%
gpu: util_avg=54% util_max=91% mem_avg=36526MB mem_max=37013MB

results saved: /tmp/stream_vllm_tuned_vllm_only.json
ERROR 02-18 01:10:34 [core_client.py:605] Engine core proc EngineCore_DP0 died unexpectedly, shutting down client.


**Testing decode interval tuning**


cd /home/ubuntu/veenaModal && python3 scripts/stress_test_streaming_runtime.py --levels 64 --text short --max-tokens 1024 --timeout 240 --requests-multiplier 1 --min-requests 8 --gpu-memory 0.25 --stream-output-kind delta --stream-decode-interval 64 --stream-window-size 96 --stream-min-semantic-first 8 --output /tmp/stream_decode_tuned.json


Output:
Chunk ID: 03ec68
Wall time: 1.0011 seconds
Process running with session ID 89470
Original token count: 0
Output:


Output:
Chunk ID: c7abc8
Wall time: 5.0028 seconds
Process running with session ID 89470
Original token count: 1277
Output:
[2026-02-18 01:10:49] INFO tts_runtime.py:183: Loading SparkTTS model with vLLM...
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
🚀 Initializing Spark TTS Model
📁 Model path: /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
🗣  Speakers: 12 predefined (lipakshi, vardan, reet, Nandini, krishna, anika, adarsh, Nilay, Aarvi, Asha, Bittu, Mira)
🎭 Emotions: 10 tags in [bracket] format

📝 Loading tokenizer from model...
✅ Tokenizer loaded: 165168 tokens

🔍 Validating 12 speakers for Spark TTS...
  lipakshi → <|speaker_0|>: 7 tokens
  vardan → <|speaker_1|>: 7 tokens
  reet → <|speaker_2|>: 7 tokens
  Nandini → <|speaker_3|>: 7 tokens
  krishna → <|speaker_4|>: 7 tokens
  anika → <|speaker_5|>: 7 tokens
  adarsh → <|speaker_6|>: 7 tokens
  Nilay → <|speaker_7|>: 7 tokens
  Aarvi → <|speaker_8|>: 7 tokens
  Asha → <|speaker_9|>: 7 tokens
  Bittu → <|speaker_10|>: 8 tokens
  Mira → <|speaker_11|>: 8 tokens
✅ All 12 speakers validated with mapping

🔍 Validating 10 emotion tags for Spark TTS...
✅ All 10 emotion tags validated

📋 Sample emotion tags:
  [angry]: 1 tokens
  [curious]: 1 tokens
  [excited]: 1 tokens
  [giggle]: 1 tokens
  [laughs harder]: 1 tokens
  ...

🔧 Initializing vLLM engine with optimizations...
  - dtype: bfloat16
  - max_model_len: 4096
  - gpu_memory_utilization: 0.25
  - enable_prefix_caching: True
  - enforce_eager (disable CUDA graphs): False
INFO 02-18 01:10:49 [model.py:541] Resolved architecture: Qwen2ForCausalLM
INFO 02-18 01:10:49 [model.py:1561] Using max model len 4096
INFO 02-18 01:10:49 [scheduler.py:226] Chunked prefill is enabled with max_num_batched_tokens=4096.
INFO 02-18 01:10:49 [vllm.py:624] Asynchronous scheduling is enabled.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(EngineCore_DP0 pid=4122733) INFO 02-18 01:10:50 [core.py:96] Initializing a V1 LLM engine (v0.15.1) with config: model='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', speculative_config=None, tokenizer='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [4096], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 256, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': True}, 'local_cache_dir': None, 'static_all_moe_layers': []}


Output:
Chunk ID: 7a9f9c
Wall time: 5.0010 seconds
Process running with session ID 89470
Original token count: 511
Output:
(EngineCore_DP0 pid=4122733) INFO 02-18 01:10:52 [parallel_state.py:1212] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://216.81.248.184:43301 backend=nccl
(EngineCore_DP0 pid=4122733) INFO 02-18 01:10:52 [parallel_state.py:1423] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
(EngineCore_DP0 pid=4122733) INFO 02-18 01:10:53 [gpu_model_runner.py:4033] Starting to load model /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM...
(EngineCore_DP0 pid=4122733) INFO 02-18 01:10:54 [cuda.py:364] Using FLASH_ATTN attention backend out of potential backends: ('FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION')
(EngineCore_DP0 pid=4122733) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
(EngineCore_DP0 pid=4122733) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
(EngineCore_DP0 pid=4122733)
Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
(EngineCore_DP0 pid=4122733)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.50it/s]
(EngineCore_DP0 pid=4122733)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.50it/s]
(EngineCore_DP0 pid=4122733)
(EngineCore_DP0 pid=4122733) INFO 02-18 01:10:55 [default_loader.py:291] Loading weights took 0.74 seconds
(EngineCore_DP0 pid=4122733) INFO 02-18 01:10:55 [gpu_model_runner.py:4130] Model loading took 0.95 GiB memory and 1.268532 seconds
(EngineCore_DP0 pid=4122733) INFO 02-18 01:11:00 [backends.py:812] Using cache directory: /home/ubuntu/.cache/vllm/torch_compile_cache/f3ff13511c/rank_0_0/backbone for vLLM's torch.compile
(EngineCore_DP0 pid=4122733) INFO 02-18 01:11:00 [backends.py:872] Dynamo bytecode transform time: 4.24 s


Output:
Chunk ID: 58a135
Wall time: 5.0013 seconds
Process running with session ID 89470
Original token count: 727
Output:
(EngineCore_DP0 pid=4122733) INFO 02-18 01:11:03 [backends.py:267] Directly load the compiled graph(s) for compile range (1, 4096) from the cache, took 0.657 s
(EngineCore_DP0 pid=4122733) INFO 02-18 01:11:03 [monitor.py:34] torch.compile takes 4.90 s in total
(EngineCore_DP0 pid=4122733) INFO 02-18 01:11:04 [gpu_worker.py:356] Available KV cache memory: 18.06 GiB
(EngineCore_DP0 pid=4122733) INFO 02-18 01:11:04 [kv_cache_utils.py:1307] GPU KV cache size: 1,577,712 tokens
(EngineCore_DP0 pid=4122733) INFO 02-18 01:11:04 [kv_cache_utils.py:1312] Maximum concurrency for 4,096 tokens per request: 385.18x
(EngineCore_DP0 pid=4122733)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   0%|          | 0/35 [00:00<?, ?it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  11%|█▏        | 4/35 [00:00<00:00, 32.49it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  23%|██▎       | 8/35 [00:00<00:00, 34.73it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  34%|███▍      | 12/35 [00:00<00:00, 35.63it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  46%|████▌     | 16/35 [00:00<00:00, 35.93it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  57%|█████▋    | 20/35 [00:00<00:00, 35.84it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  69%|██████▊   | 24/35 [00:00<00:00, 35.80it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  80%|████████  | 28/35 [00:00<00:00, 35.73it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  91%|█████████▏| 32/35 [00:00<00:00, 34.38it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 35/35 [00:01<00:00, 34.39it/s]
(EngineCore_DP0 pid=4122733)
Capturing CUDA graphs (decode, FULL):   0%|          | 0/19 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL):  16%|█▌        | 3/19 [00:00<00:00, 28.27it/s]
Capturing CUDA graphs (decode, FULL):  37%|███▋      | 7/19 [00:00<00:00, 32.78it/s]
Capturing CUDA graphs (decode, FULL):  58%|█████▊    | 11/19 [00:00<00:00, 34.54it/s]
Capturing CUDA graphs (decode, FULL):  79%|███████▉  | 15/19 [00:00<00:00, 35.00it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 35.48it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 34.58it/s]
(EngineCore_DP0 pid=4122733) INFO 02-18 01:11:06 [gpu_model_runner.py:5063] Graph capturing finished in 2 secs, took 0.28 GiB
(EngineCore_DP0 pid=4122733) INFO 02-18 01:11:06 [core.py:272] init engine (profile, create kv cache, warmup model) took 10.73 seconds
[2026-02-18 01:11:07] INFO tts_runtime.py:192: Initializing prompt builder...
[2026-02-18 01:11:07] INFO tts_runtime.py:199: Initializing BiCodec decoder...


Output:
Chunk ID: 9143e9
Wall time: 5.0019 seconds
Process running with session ID 89470
Original token count: 186
Output:
/home/ubuntu/.local/lib/python3.10/site-packages/torch/nn/utils/weight_norm.py:144: FutureWarning: `torch.nn.utils.weight_norm` is deprecated in favor of `torch.nn.utils.parametrizations.weight_norm`.
  WeightNorm.apply(module, name, dim)
[2026-02-18 01:11:16] INFO tts_runtime.py:206: Initializing TTS pipeline...
[2026-02-18 01:11:16] INFO tts_runtime.py:217: Initializing streaming pipeline...
[2026-02-18 01:11:16] INFO long_text_processor.py:112: LongTextProcessor initialized
[2026-02-18 01:11:16] INFO tts_runtime.py:274: ✅ TTS runtime initialized in 33145ms
[2026-02-18 01:11:16] INFO tts_runtime.py:275:    Model version: LLM
[2026-02-18 01:11:16] INFO tts_runtime.py:286: Speaker globals cache startup precompute skipped (disabled)


**Confirming poll completion**


Output:
Chunk ID: d82cf6
Wall time: 2.1490 seconds
Process exited with code 0
Original token count: 543
Output:
✅ vLLM engine initialized successfully!

🎉 Spark TTS Model ready for inference!

🎵 Loading BiCodec audio tokenizer from /home/ubuntu/veenaModal/models/spark_tts_4speaker...
   Device: cuda
Missing tensor: mel_transformer.spectrogram.window
Missing tensor: mel_transformer.mel_scale.fb
✅ BiCodec decoder initialized (sample rate: 16000Hz)
🚀 SparkTTSPipeline initialized
🌊 Veena3SlidingWindowPipeline initialized (sliding window: 28 tokens)

=== streaming concurrency=64 requests=64 ===
success=64/64 (100%)
wall=3.71s throughput=17.25 req/s
latency_ms avg=3707 p50=3709 p95=3710 max=3710
ttfb_ms avg=1502 p50=1496 p95=1907 max=2109
chunks_per_req avg=3.6 p50=4.0 p95=4.0 max=4.0
inter_chunk_ms avg=795.4 p50=34.1 p95=2077.7 max=2310.0
tail_emit_speed_x avg=1.47 p50=1.40 p95=1.79 max=2.49
timeline_to_first_batch_ms: avg=38.80 p50=39.01 p95=44.66 max=46.21
timeline_to_first_chunk_emitted_ms: avg=1488.99 p50=1482.56 p95=1884.10 max=2084.56
timeline_total_ms: avg=3527.13 p50=3670.36 p95=3704.03 max=3705.23
llm_time_in_queue_ms: avg=0.08 p50=0.08 p95=0.15 max=0.16
llm_scheduler_ms: avg=28.76 p50=25.74 p95=32.69 max=32.69
llm_first_token_ms: avg=36.98 p50=37.16 p95=43.60 max=45.26
llm_request_lifecycle_ms: avg=1550.36 p50=1569.68 p95=1721.03 max=1855.88
llm_generation_wall_ms: avg=3502.99 p50=3640.78 p95=3703.99 max=3705.19
llm_time_per_token_ms: avg=15.91 p50=15.94 p95=18.26 max=20.14
llm_batch_wall_ms_total: avg=2587.16 p50=2822.98 p95=3559.65 max=3675.37
llm_decode_wall_ms_total: avg=2548.39 p50=2794.27 p95=3518.01 max=3634.72
tokens_per_batch_p50: avg=1.00 p50=1.00 p95=1.00 max=1.00
bicodec_decode_calls: avg=2.56 p50=3.00 p95=3.00 max=3.00
bicodec_decode_wall_ms_total: avg=2898.82 p50=3023.48 p95=3071.71 max=3073.11
bicodec_decode_interval_ms_p50: avg=939.96 p50=680.29 p95=1903.96 max=2111.62
latency_over_timeline_ms: avg=179.45 p50=37.57 p95=685.70 max=1058.72
timeline_share: llm=99.3% bicodec=82.2% other=0.0%
gpu: util_avg=60% util_max=94% mem_avg=24412MB mem_max=24865MB

results saved: /tmp/stream_decode_tuned.json
ERROR 02-18 01:11:22 [core_client.py:605] Engine core proc EngineCore_DP0 died unexpectedly, shutting down client.


**Running baseline concurrency tests**


cd /home/ubuntu/veenaModal && python3 scripts/stress_test_streaming_runtime.py --levels 32,64,96 --text short --max-tokens 1024 --timeout 240 --requests-multiplier 1 --min-requests 8 --gpu-memory 0.25 --stream-output-kind delta --output /tmp/stream_baseline_levels_detailed.json


Output:
Chunk ID: d61339
Wall time: 1.0011 seconds
Process running with session ID 37308
Original token count: 0
Output:


**Designing periodic commentary polling**


Output:
Chunk ID: 215c48
Wall time: 5.0018 seconds
Process running with session ID 37308
Original token count: 1277
Output:
[2026-02-18 01:11:35] INFO tts_runtime.py:183: Loading SparkTTS model with vLLM...
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
🚀 Initializing Spark TTS Model
📁 Model path: /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
🗣  Speakers: 12 predefined (lipakshi, vardan, reet, Nandini, krishna, anika, adarsh, Nilay, Aarvi, Asha, Bittu, Mira)
🎭 Emotions: 10 tags in [bracket] format

📝 Loading tokenizer from model...
✅ Tokenizer loaded: 165168 tokens

🔍 Validating 12 speakers for Spark TTS...
  lipakshi → <|speaker_0|>: 7 tokens
  vardan → <|speaker_1|>: 7 tokens
  reet → <|speaker_2|>: 7 tokens
  Nandini → <|speaker_3|>: 7 tokens
  krishna → <|speaker_4|>: 7 tokens
  anika → <|speaker_5|>: 7 tokens
  adarsh → <|speaker_6|>: 7 tokens
  Nilay → <|speaker_7|>: 7 tokens
  Aarvi → <|speaker_8|>: 7 tokens
  Asha → <|speaker_9|>: 7 tokens
  Bittu → <|speaker_10|>: 8 tokens
  Mira → <|speaker_11|>: 8 tokens
✅ All 12 speakers validated with mapping

🔍 Validating 10 emotion tags for Spark TTS...
✅ All 10 emotion tags validated

📋 Sample emotion tags:
  [angry]: 1 tokens
  [curious]: 1 tokens
  [excited]: 1 tokens
  [giggle]: 1 tokens
  [laughs harder]: 1 tokens
  ...

🔧 Initializing vLLM engine with optimizations...
  - dtype: bfloat16
  - max_model_len: 4096
  - gpu_memory_utilization: 0.25
  - enable_prefix_caching: True
  - enforce_eager (disable CUDA graphs): False
INFO 02-18 01:11:36 [model.py:541] Resolved architecture: Qwen2ForCausalLM
INFO 02-18 01:11:36 [model.py:1561] Using max model len 4096
INFO 02-18 01:11:36 [scheduler.py:226] Chunked prefill is enabled with max_num_batched_tokens=4096.
INFO 02-18 01:11:36 [vllm.py:624] Asynchronous scheduling is enabled.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(EngineCore_DP0 pid=4123290) INFO 02-18 01:11:37 [core.py:96] Initializing a V1 LLM engine (v0.15.1) with config: model='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', speculative_config=None, tokenizer='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [4096], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 256, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': True}, 'local_cache_dir': None, 'static_all_moe_layers': []}


Output:
Chunk ID: a92c75
Wall time: 5.0010 seconds
Process running with session ID 37308
Original token count: 437
Output:
(EngineCore_DP0 pid=4123290) INFO 02-18 01:11:39 [parallel_state.py:1212] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://216.81.248.184:43295 backend=nccl
(EngineCore_DP0 pid=4123290) INFO 02-18 01:11:39 [parallel_state.py:1423] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
(EngineCore_DP0 pid=4123290) INFO 02-18 01:11:40 [gpu_model_runner.py:4033] Starting to load model /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM...
(EngineCore_DP0 pid=4123290) INFO 02-18 01:11:41 [cuda.py:364] Using FLASH_ATTN attention backend out of potential backends: ('FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION')
(EngineCore_DP0 pid=4123290) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
(EngineCore_DP0 pid=4123290) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
(EngineCore_DP0 pid=4123290)
Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
(EngineCore_DP0 pid=4123290)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:01<00:00,  1.37s/it]
(EngineCore_DP0 pid=4123290)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:01<00:00,  1.37s/it]
(EngineCore_DP0 pid=4123290)
(EngineCore_DP0 pid=4123290) INFO 02-18 01:11:42 [default_loader.py:291] Loading weights took 1.45 seconds
(EngineCore_DP0 pid=4123290) INFO 02-18 01:11:43 [gpu_model_runner.py:4130] Model loading took 0.95 GiB memory and 1.996533 seconds


**Scheduling periodic commentary updates**


Baseline multi-level run is in progress (32/64/96). Once it completes, I’ll pull the per-level queue/scheduler/LLM-vs-BiCodec shares so we can pinpoint the exact concurrency knee and choose the next optimization target.


Output:
Chunk ID: b0f782
Wall time: 5.0005 seconds
Process running with session ID 37308
Original token count: 762
Output:
(EngineCore_DP0 pid=4123290) INFO 02-18 01:11:48 [backends.py:812] Using cache directory: /home/ubuntu/.cache/vllm/torch_compile_cache/f3ff13511c/rank_0_0/backbone for vLLM's torch.compile
(EngineCore_DP0 pid=4123290) INFO 02-18 01:11:48 [backends.py:872] Dynamo bytecode transform time: 4.22 s
(EngineCore_DP0 pid=4123290) INFO 02-18 01:11:50 [backends.py:267] Directly load the compiled graph(s) for compile range (1, 4096) from the cache, took 0.662 s
(EngineCore_DP0 pid=4123290) INFO 02-18 01:11:50 [monitor.py:34] torch.compile takes 4.88 s in total
(EngineCore_DP0 pid=4123290) INFO 02-18 01:11:51 [gpu_worker.py:356] Available KV cache memory: 18.06 GiB
(EngineCore_DP0 pid=4123290) INFO 02-18 01:11:51 [kv_cache_utils.py:1307] GPU KV cache size: 1,577,712 tokens
(EngineCore_DP0 pid=4123290) INFO 02-18 01:11:51 [kv_cache_utils.py:1312] Maximum concurrency for 4,096 tokens per request: 385.18x
(EngineCore_DP0 pid=4123290)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   0%|          | 0/35 [00:00<?, ?it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  11%|█▏        | 4/35 [00:00<00:00, 31.90it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  23%|██▎       | 8/35 [00:00<00:00, 32.61it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  34%|███▍      | 12/35 [00:00<00:00, 34.08it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  46%|████▌     | 16/35 [00:00<00:00, 34.53it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  57%|█████▋    | 20/35 [00:00<00:00, 35.36it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  69%|██████▊   | 24/35 [00:00<00:00, 35.82it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  80%|████████  | 28/35 [00:00<00:00, 35.92it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  91%|█████████▏| 32/35 [00:00<00:00, 34.51it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 35/35 [00:01<00:00, 33.94it/s]
(EngineCore_DP0 pid=4123290)
Capturing CUDA graphs (decode, FULL):   0%|          | 0/19 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL):  16%|█▌        | 3/19 [00:00<00:00, 28.65it/s]
Capturing CUDA graphs (decode, FULL):  37%|███▋      | 7/19 [00:00<00:00, 31.43it/s]
Capturing CUDA graphs (decode, FULL):  58%|█████▊    | 11/19 [00:00<00:00, 32.31it/s]
Capturing CUDA graphs (decode, FULL):  79%|███████▉  | 15/19 [00:00<00:00, 32.75it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 33.22it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 32.59it/s]
(EngineCore_DP0 pid=4123290) INFO 02-18 01:11:54 [gpu_model_runner.py:5063] Graph capturing finished in 2 secs, took 0.28 GiB
(EngineCore_DP0 pid=4123290) INFO 02-18 01:11:54 [core.py:272] init engine (profile, create kv cache, warmup model) took 10.90 seconds


Output:
Chunk ID: cb08df
Wall time: 5.0013 seconds
Process running with session ID 37308
Original token count: 99
Output:
[2026-02-18 01:11:55] INFO tts_runtime.py:192: Initializing prompt builder...
[2026-02-18 01:11:55] INFO tts_runtime.py:199: Initializing BiCodec decoder...
/home/ubuntu/.local/lib/python3.10/site-packages/torch/nn/utils/weight_norm.py:144: FutureWarning: `torch.nn.utils.weight_norm` is deprecated in favor of `torch.nn.utils.parametrizations.weight_norm`.
  WeightNorm.apply(module, name, dim)


Output:
Chunk ID: 4a40c8
Wall time: 5.0009 seconds
Process running with session ID 37308
Original token count: 127
Output:
[2026-02-18 01:12:04] INFO tts_runtime.py:206: Initializing TTS pipeline...
[2026-02-18 01:12:04] INFO tts_runtime.py:217: Initializing streaming pipeline...
[2026-02-18 01:12:04] INFO long_text_processor.py:112: LongTextProcessor initialized
[2026-02-18 01:12:04] INFO tts_runtime.py:274: ✅ TTS runtime initialized in 34191ms
[2026-02-18 01:12:04] INFO tts_runtime.py:275:    Model version: LLM
[2026-02-18 01:12:04] INFO tts_runtime.py:286: Speaker globals cache startup precompute skipped (disabled)


**Planning progressive result polling**


Output:
Chunk ID: 6341f9
Wall time: 5.0013 seconds
Process running with session ID 37308
Original token count: 0
Output: