Output:
Chunk ID: 616e39
Wall time: 5.0022 seconds
Process running with session ID 84421
Original token count: 1400
Output:
Total output lines: 52

[2026-02-18 18:54:07] INFO tts_runtime.py:183: Loading SparkTTS model with vLLM...
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
🚀 Initializing Spark TTS Model
📁 Model path: /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
🗣  Speakers: 12 predefined (lipakshi, vardan, reet, Nandini, krishna, anika, adarsh, Nilay, Aarvi, Asha, Bittu, Mira)
🎭 Emotions: 10 tags in [bracket] format

📝 Loading tokenizer from model...
✅ Tokenizer loaded: 165168 tokens

🔍 Validating 12 speakers for Spark TTS...
  lipakshi → <|speaker_0|>: 7 tokens
  vardan → <|speaker_1|>: 7 tokens
  reet → <|speaker_2|>: 7 tokens
  Nandini → <|speaker_3|>: 7 tokens
  krishna → <|speaker_4|>: 7 tokens
  anika → <|speaker_5|>: 7 tokens
  adarsh → <|speaker_6|>: 7 tokens
  Nilay → <|speaker_7|>: 7 tokens
  Aarvi → <|speaker_8|>: 7 tokens
  Asha → <|speaker_9|>: 7 tokens
  Bittu → <|speaker_10|>: 8 tokens
  Mira → <|speaker_11|>: 8 tokens
✅ All 12 speakers validated with mapping

🔍 Validating 10 emotion tags for Spark TTS...
✅ All 10 emotion tags validated

📋 Sample emotion tags:
  [angry]: 1 tokens
  [curious]: 1 tokens
  [excited]: 1 tokens
  [giggle]: 1 tokens
  [laughs harder]: 1 tokens
  ...

🔧 Initializing vLLM engine with optimizations...
  - dtype: bfloat16
  - max_model_len: 4096
  - gpu_memory_utilization: 0.25
  - enable_prefix_caching: True
  - enforce_eager (disable CUDA graphs): False
INFO 02-18 18:54:08 [model.py:541] Resolved architecture: Qwen2ForCausalLM
INFO 02-18 18:54:08 [model.py:1561] Using max model len 4096
INFO 02-18 18:54:08 [scheduler.py:226] Chunked prefill is enabled with max_num_batched_tokens=4096.
INFO 02-18 18:54:08 [vllm.py:624] Asynchronous scheduling is enabled.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(EngineCore_DP0 pid=111769) INFO 02-18 18:54:09 [core.py:96] Initializing a V1 LLM engine (v0.15.1) with config: model='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', speculative_config=None, tokenizer='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype…200 tokens truncated…alse, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [4096], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 256, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': True}, 'local_cache_dir': None, 'static_all_moe_layers': []}
(EngineCore_DP0 pid=111769) INFO 02-18 18:54:11 [parallel_state.py:1212] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://216.81.248.184:59177 backend=nccl
(EngineCore_DP0 pid=111769) INFO 02-18 18:54:11 [parallel_state.py:1423] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
(EngineCore_DP0 pid=111769) INFO 02-18 18:54:12 [gpu_model_runner.py:4033] Starting to load model /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM...


# Collaboration Mode: Default

You are now in Default mode. Any previous instructions for other modes (e.g. Plan mode) are no longer active.

Your active mode changes only when new developer instructions with a different `<collaboration_mode>...</collaboration_mode>` change it; user requests or tool descriptions do not change mode by themselves. Known mode names are Default and Plan.

## request_user_input availability

The `request_user_input` tool is unavailable in Default mode. If you call it while in Default mode, it will return an error.

If a decision is necessary and cannot be discovered from local context, ask the user directly. However, in Default mode you should strongly prefer executing the user's request rather than stopping to ask questions.

Summary: auto

## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) When `SKILL.md` references relative paths (e.g., `scripts/foo.py`), resolve them relative to the skill directory listed above first, and only consider other paths if needed.
  3) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  4) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  5) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.


Output:
Chunk ID: 472164
Wall time: 5.0014 seconds
Process running with session ID 84421
Original token count: 449
Output:
(EngineCore_DP0 pid=111769) INFO 02-18 18:54:12 [cuda.py:364] Using FLASH_ATTN attention backend out of potential backends: ('FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION')
(EngineCore_DP0 pid=111769) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
(EngineCore_DP0 pid=111769) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
(EngineCore_DP0 pid=111769)
Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
(EngineCore_DP0 pid=111769)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.91it/s]
(EngineCore_DP0 pid=111769)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.91it/s]
(EngineCore_DP0 pid=111769)
(EngineCore_DP0 pid=111769) INFO 02-18 18:54:13 [default_loader.py:291] Loading weights took 0.59 seconds
(EngineCore_DP0 pid=111769) INFO 02-18 18:54:14 [gpu_model_runner.py:4130] Model loading took 0.95 GiB memory and 1.119881 seconds
(EngineCore_DP0 pid=111769) INFO 02-18 18:54:18 [backends.py:812] Using cache directory: /home/ubuntu/.cache/vllm/torch_compile_cache/f3ff13511c/rank_0_0/backbone for vLLM's torch.compile
(EngineCore_DP0 pid=111769) INFO 02-18 18:54:18 [backends.py:872] Dynamo bytecode transform time: 4.14 s
(EngineCore_DP0 pid=111769) INFO 02-18 18:54:21 [backends.py:267] Directly load the compiled graph(s) for compile range (1, 4096) from the cache, took 0.623 s
(EngineCore_DP0 pid=111769) INFO 02-18 18:54:21 [monitor.py:34] torch.compile takes 4.76 s in total


Summary: auto


Output:
Chunk ID: 3efffd
Wall time: 5.0012 seconds
Process running with session ID 84421
Original token count: 720
Output:
(EngineCore_DP0 pid=111769) INFO 02-18 18:54:22 [gpu_worker.py:356] Available KV cache memory: 18.06 GiB
(EngineCore_DP0 pid=111769) INFO 02-18 18:54:22 [kv_cache_utils.py:1307] GPU KV cache size: 1,577,712 tokens
(EngineCore_DP0 pid=111769) INFO 02-18 18:54:22 [kv_cache_utils.py:1312] Maximum concurrency for 4,096 tokens per request: 385.18x
(EngineCore_DP0 pid=111769)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   0%|          | 0/35 [00:00<?, ?it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  11%|█▏        | 4/35 [00:00<00:00, 31.61it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  23%|██▎       | 8/35 [00:00<00:00, 34.62it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  34%|███▍      | 12/35 [00:00<00:00, 35.94it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  46%|████▌     | 16/35 [00:00<00:00, 36.75it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  57%|█████▋    | 20/35 [00:00<00:00, 36.58it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  69%|██████▊   | 24/35 [00:00<00:00, 36.27it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  80%|████████  | 28/35 [00:00<00:00, 36.30it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  91%|█████████▏| 32/35 [00:00<00:00, 34.74it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 35/35 [00:01<00:00, 34.78it/s]
(EngineCore_DP0 pid=111769)
Capturing CUDA graphs (decode, FULL):   0%|          | 0/19 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL):  16%|█▌        | 3/19 [00:00<00:00, 29.57it/s]
Capturing CUDA graphs (decode, FULL):  37%|███▋      | 7/19 [00:00<00:00, 33.76it/s]
Capturing CUDA graphs (decode, FULL):  58%|█████▊    | 11/19 [00:00<00:00, 34.76it/s]
Capturing CUDA graphs (decode, FULL):  79%|███████▉  | 15/19 [00:00<00:00, 35.17it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 35.84it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 35.02it/s]
(EngineCore_DP0 pid=111769) INFO 02-18 18:54:24 [gpu_model_runner.py:5063] Graph capturing finished in 2 secs, took 0.28 GiB
(EngineCore_DP0 pid=111769) INFO 02-18 18:54:24 [core.py:272] init engine (profile, create kv cache, warmup model) took 10.58 seconds
[2026-02-18 18:54:26] INFO tts_runtime.py:192: Initializing prompt builder...
[2026-02-18 18:54:26] INFO tts_runtime.py:199: Initializing BiCodec decoder...
/home/ubuntu/.local/lib/python3.10/site-packages/torch/nn/utils/weight_norm.py:144: FutureWarning: `torch.nn.utils.weight_norm` is deprecated in favor of `torch.nn.utils.parametrizations.weight_norm`.
  WeightNorm.apply(module, name, dim)


Output:
Chunk ID: 5c16ab
Wall time: 5.0005 seconds
Process running with session ID 84421
Original token count: 127
Output:
[2026-02-18 18:54:33] INFO tts_runtime.py:209: Initializing TTS pipeline...
[2026-02-18 18:54:33] INFO tts_runtime.py:220: Initializing streaming pipeline...
[2026-02-18 18:54:33] INFO long_text_processor.py:112: LongTextProcessor initialized
[2026-02-18 18:54:33] INFO tts_runtime.py:277: ✅ TTS runtime initialized in 31021ms
[2026-02-18 18:54:33] INFO tts_runtime.py:278:    Model version: LLM
[2026-02-18 18:54:33] INFO tts_runtime.py:289: Speaker globals cache startup precompute skipped (disabled)


The `scale_pending=72` run is in its benchmark phase. I’ll share the final numbers and a direct comparison against `w1` and `scale64` right after completion.


Output:
Chunk ID: 27f6da
Wall time: 2.6546 seconds
Process exited with code 0
Original token count: 1241
Output:
Total output lines: 87

✅ vLLM engine initialized successfully!

🎉 Spark TTS Model ready for inference!

🎵 Loading BiCodec audio tokenizer from /home/ubuntu/veenaModal/models/spark_tts_4speaker...
   Device: cuda
Missing tensor: mel_transformer.spectrogram.window
Missing tensor: mel_transformer.mel_scale.fb
✅ BiCodec decoder initialized (sample rate: 16000Hz)
   ℹ  Micro-batching enabled (workers=2, scale_pending=72, max_batch=16, timeout_ms=6.0)
🚀 SparkTTSPipeline initialized
🌊 Veena3SlidingWindowPipeline initialized (sliding window: 28 tokens)

=== streaming concurrency=96 requests=96 ===
success=96/96 (100%)
wall=4.65s throughput=20.63 req/s
latency_ms avg=4058 p50=4353 p95=4631 max=4651
ttfb_ms avg=1164 p50=1209 p95=1344 max=1406
chunks_per_req avg=5.0 p50=5.0 p95=5.0 max=6.0
inter_chunk_ms avg=701.0 p50=710.5 p95=1945.9 max=2240.9
tail_emit_speed_x avg=1.37 p50=1.29 p95=1.73 max=2.22
timeline_to_first_batch_ms: avg=58.62 p50=59.43 p95=67.60 max=69.66
timeline_to_first_chunk_emitted_ms: avg=1143.70 p50=1179.81 p95=1311.51 max=1369.06
timeline_total_ms: avg=3955.02 p50=4165.33 p95=4605.13 max=4619.11
llm_time_in_queue_ms: avg=0.13 p50=0.12 p95=0.30 max=0.33
llm_scheduler_ms: avg=37.86 p50=30.25 p95=53.40 max=53.40
llm_first_token_ms: avg=55.91 p50=56.90 p95=66.58 max=68.77
llm_request_lifecycle_ms: avg=2593.64 p50=2636.70 p95=2839.54 max=2906.44
llm_generation_wall_ms: avg=3937.17 p50=4152.88 p95=4605.09 max=4619.06
llm_time_per_token_ms: avg=17.14 p50=17.70 p95=19.94 max=20.66
llm_batch_wall_ms_total: avg=3506.41 p50=3408.71 p95=4296.72 max=4416.24
llm_decode_wall_ms_total: avg=3447.82 p50=3346.40 p95=4245.22 max=4364.74
tokens_per_batch_p50: avg=1.00 p50=1.00 p95=1.00 max=1.00
bicodec_decode_calls: avg=4.01 p50=4.00 p95=4.00 max=5.00
bicodec_decode_wall_ms_total: avg=2822.99 p50=3000.26 p95=3840.14 max=3922.45
bicodec_decode_interval_ms_p50: avg=841.90 p50=872.40 p95=1149.16 max=1199.75
decode_interval_applied_tokens_p50: avg=48.00 p50=48.00 p95=48.00 max=48.00
decode_pending_requests_p50: avg=0.00 p50=0.00 p95=0.00 max=0.00
batch_avg_size: avg=9.50 p50=8.97 p95=11.75 max=11.75
batch_max_seen: avg=16.00 p50=16.00 p95=16.00 max=16.00
batch_workers_live: avg=2.00 p50=2.00 p95=2.00 max=2.00
batch_workers_target: avg=2.00 p50=2.00 p95=2.00 max=2.00
batch_queue_wait_ms_avg: avg=499.27 p50=546.79 p95=636.67 max=636.67
batch_compute_ms_avg: avg=195.47 p50=182.86 …41 tokens truncated…share: llm=99.5% bicodec=71.4% other=0.0%
gpu: util_avg=65% util_max=97% mem_avg=48193MB mem_max=49995MB

=== streaming concurrency=128 requests=128 ===
success=128/128 (100%)
wall=5.93s throughput=21.60 req/s
latency_ms avg=4921 p50=4985 p95=5922 max=5923
ttfb_ms avg=1534 p50=1452 p95=1867 max=1867
chunks_per_req avg=5.0 p50=5.0 p95=6.0 max=6.0
inter_chunk_ms avg=840.7 p50=858.9 p95=2128.5 max=2348.1
tail_emit_speed_x avg=1.19 p50=1.14 p95=1.55 max=2.90
timeline_to_first_batch_ms: avg=63.81 p50=64.51 p95=74.81 max=77.14
timeline_to_first_chunk_emitted_ms: avg=1507.54 p50=1432.33 p95=1824.76 max=1826.27
timeline_total_ms: avg=4857.33 p50=4944.46 p95=5886.36 max=5897.29
llm_time_in_queue_ms: avg=0.13 p50=0.08 p95=0.35 max=0.40
llm_scheduler_ms: avg=43.54 p50=38.89 p95=57.03 max=57.03
llm_first_token_ms: avg=60.21 p50=60.76 p95=71.07 max=72.87
llm_request_lifecycle_ms: avg=3090.45 p50=3037.19 p95=3428.47 max=3608.55
llm_generation_wall_ms: avg=4839.26 p50=4944.42 p95=5886.32 max=5897.25
llm_time_per_token_ms: avg=21.26 p50=21.78 p95=24.87 max=26.60
llm_batch_wall_ms_total: avg=4332.88 p50=4265.64 p95=5478.55 max=5791.96
llm_decode_wall_ms_total: avg=4269.11 p50=4190.20 p95=5419.82 max=5736.16
tokens_per_batch_p50: avg=1.00 p50=1.00 p95=1.00 max=1.00
bicodec_decode_calls: avg=3.98 p50=4.00 p95=5.00 max=5.00
bicodec_decode_wall_ms_total: avg=3727.83 p50=3921.63 p95=4926.62 max=4928.80
bicodec_decode_interval_ms_p50: avg=1136.15 p50=1095.27 p95=1751.26 max=1813.29
decode_interval_applied_tokens_p50: avg=48.00 p50=48.00 p95=48.00 max=48.00
decode_pending_requests_p50: avg=0.00 p50=0.00 p95=0.00 max=0.00
batch_avg_size: avg=8.61 p50=8.32 p95=9.92 max=10.02
batch_max_seen: avg=16.00 p50=16.00 p95=16.00 max=16.00
batch_workers_live: avg=2.00 p50=2.00 p95=2.00 max=2.00
batch_workers_target: avg=2.00 p50=2.00 p95=2.00 max=2.00
batch_queue_wait_ms_avg: avg=687.87 p50=685.02 p95=775.47 max=789.19
batch_compute_ms_avg: avg=159.65 p50=156.22 p95=181.79 max=185.30
batch_queue_depth_avg: avg=63.66 p50=63.81 p95=66.63 max=66.64
latency_over_timeline_ms: avg=63.93 p50=42.93 p95=183.99 max=305.56
timeline_share: llm=99.6% bicodec=76.7% other=0.0%
gpu: util_avg=79% util_max=97% mem_avg=50126MB mem_max=50227MB

results saved: /tmp/stream_workers2_scale72_seeded.json
ERROR 02-18 18:54:47 [core_client.py:605] Engine core proc EngineCore_DP0 died unexpectedly, shutting down client.


**Implementing dynamic worker gating**


`scale64` is best at `c=128`, but `c=96` still regresses because worker 2 stays active once promoted. I’m implementing true dynamic gating (extra workers idle when not needed, no cancellation) so we can scale up/down safely within a run.


Input:
*** Begin Patch
*** Update File: veena3modal/core/bicodec_decoder.py
@@
     def _ensure_batch_worker(self, loop: asyncio.AbstractEventLoop) -> asyncio.Queue:
         key = id(loop)
         with self._batch_map_lock:
             queue = self._batch_queues.get(key)
             workers = self._batch_workers.get(key, [])
@@
             if queue is None:
                 queue = asyncio.Queue(maxsize=max(self.batch_max_size * 8, 256))
                 self._batch_queues[key] = queue

-            target_workers = self.batch_worker_count
-            if self.batch_worker_count > 1 and self.batch_worker_scale_pending > 0:
-                pending_now = queue.qsize() + self._batch_inflight_requests
-                if pending_now < self.batch_worker_scale_pending:
-                    target_workers = 1
+            target_workers = self._desired_worker_count(queue)
             while len(live_workers) < target_workers:
                 worker_idx = len(live_workers)
                 worker = loop.create_task(
                     self._batch_worker(loop, queue, worker_idx),
                     name=f"bicodec-batch-worker-{key}-{worker_idx}",
                 )
                 live_workers.append(worker)

             self._batch_workers_target_last = len(live_workers)
             self._batch_workers[key] = live_workers
             return queue
+
+    def _desired_worker_count(self, queue: Optional[asyncio.Queue] = None) -> int:
+        if self.batch_worker_count <= 1:
+            return 1
+        if self.batch_worker_scale_pending <= 0:
+            return self.batch_worker_count
+        with self._batch_map_lock:
+            pending_now = self._batch_inflight_requests
+        if queue is not None:
+            pending_now += queue.qsize()
+        if pending_now >= self.batch_worker_scale_pending:
+            return self.batch_worker_count
+        return 1
@@
     async def _batch_worker(
         self,
         loop: asyncio.AbstractEventLoop,
         queue: asyncio.Queue,
         worker_idx: int,
     ) -> None:
         while True:
+            target_workers = self._desired_worker_count(queue)
+            self._batch_workers_target_last = target_workers
+            if worker_idx >= target_workers:
+                await asyncio.sleep(min(self.batch_timeout_s, 0.002))
+                continue
+
             first: _BatchDecodeRequest = await queue.get()
*** End Patch


Output: {"output":"Success. Updated the following files:\nM veena3modal/core/bicodec_decoder.py\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


**Patching lock recursion to avoid deadlock**


I found and fixed a lock-order issue in that patch (`_ensure_batch_worker` calling a lock-taking helper while already locked). I’m applying the correction now before benchmarking.


Input:
*** Begin Patch
*** Update File: veena3modal/core/bicodec_decoder.py
@@
-            target_workers = self._desired_worker_count(queue)
+            target_workers = self.batch_worker_count
+            if self.batch_worker_count > 1 and self.batch_worker_scale_pending > 0:
+                pending_now = queue.qsize() + self._batch_inflight_requests
+                if pending_now < self.batch_worker_scale_pending:
+                    target_workers = 1
             while len(live_workers) < target_workers:
                 worker_idx = len(live_workers)
                 worker = loop.create_task(
                     self._batch_worker(loop, queue, worker_idx),
                     name=f"bicodec-batch-worker-{key}-{worker_idx}",
*** End Patch


Output: {"output":"Success. Updated the following files:\nM veena3modal/core/bicodec_decoder.py\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


python3 -m py_compile veena3modal/core/bicodec_decoder.py


Output:
Chunk ID: bce417
Wall time: 0.0503 seconds
Process exited with code 0
Original token count: 0
Output:


**Running scaled dynamic gating tests**


The lock fix is compiled. I’m rerunning `scale_pending=64` with the new dynamic gating logic to validate whether `c=96` recovers while keeping the strong `c=128` gains.


python3 scripts/stress_test_streaming_runtime.py --levels 96,128 --text short --max-tokens 1024 --seed-base 2026 --timeout 240 --requests-multiplier 1 --min-requests 8 --gpu-memory 0.25 --stream-output-kind delta --stream-decode-interval 48 --stream-window-size 128 --bicodec-batch-max 16 --bicodec-batch-timeout-ms 6 --bicodec-batch-workers 2 --bicodec-batch-scale-pending 64 --output /tmp/stream_workers2_scale64_gated_seeded.json


Output:
Chunk ID: db149c
Wall time: 1.0007 seconds
Process running with session ID 59889
Original token count: 0
Output:


**Planning update polling**


Output:
Chunk ID: 69c7cf
Wall time: 5.0011 seconds
Process running with session ID 59889
Original token count: 1362
Output:
Total output lines: 51

[2026-02-18 18:55:38] INFO tts_runtime.py:183: Loading SparkTTS model with vLLM...
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
🚀 Initializing Spark TTS Model
📁 Model path: /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
🗣  Speakers: 12 predefined (lipakshi, vardan, reet, Nandini, krishna, anika, adarsh, Nilay, Aarvi, Asha, Bittu, Mira)
🎭 Emotions: 10 tags in [bracket] format

📝 Loading tokenizer from model...
✅ Tokenizer loaded: 165168 tokens

🔍 Validating 12 speakers for Spark TTS...
  lipakshi → <|speaker_0|>: 7 tokens
  vardan → <|speaker_1|>: 7 tokens
  reet → <|speaker_2|>: 7 tokens
  Nandini → <|speaker_3|>: 7 tokens
  krishna → <|speaker_4|>: 7 tokens
  anika → <|speaker_5|>: 7 tokens
  adarsh → <|speaker_6|>: 7 tokens
  Nilay → <|speaker_7|>: 7 tokens
  Aarvi → <|speaker_8|>: 7 tokens
  Asha → <|speaker_9|>: 7 tokens
  Bittu → <|speaker_10|>: 8 tokens
  Mira → <|speaker_11|>: 8 tokens
✅ All 12 speakers validated with mapping

🔍 Validating 10 emotion tags for Spark TTS...
✅ All 10 emotion tags validated

📋 Sample emotion tags:
  [angry]: 1 tokens
  [curious]: 1 tokens
  [excited]: 1 tokens
  [giggle]: 1 tokens
  [laughs harder]: 1 tokens
  ...

🔧 Initializing vLLM engine with optimizations...
  - dtype: bfloat16
  - max_model_len: 4096
  - gpu_memory_utilization: 0.25
  - enable_prefix_caching: True
  - enforce_eager (disable CUDA graphs): False
INFO 02-18 18:55:38 [model.py:541] Resolved architecture: Qwen2ForCausalLM
INFO 02-18 18:55:38 [model.py:1561] Using max model len 4096
INFO 02-18 18:55:39 [scheduler.py:226] Chunked prefill is enabled with max_num_batched_tokens=4096.
INFO 02-18 18:55:39 [vllm.py:624] Asynchronous scheduling is enabled.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(EngineCore_DP0 pid=113257) INFO 02-18 18:55:39 [core.py:96] Initializing a V1 LLM engine (v0.15.1) with config: model='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', speculative_config=None, tokenizer='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype…162 tokens truncated…ailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [4096], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 256, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': True}, 'local_cache_dir': None, 'static_all_moe_layers': []}
(EngineCore_DP0 pid=113257) INFO 02-18 18:55:42 [parallel_state.py:1212] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://216.81.248.184:51843 backend=nccl
(EngineCore_DP0 pid=113257) INFO 02-18 18:55:42 [parallel_state.py:1423] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A


Output:
Chunk ID: 423e62
Wall time: 5.0011 seconds
Process running with session ID 59889
Original token count: 423
Output:
(EngineCore_DP0 pid=113257) INFO 02-18 18:55:42 [gpu_model_runner.py:4033] Starting to load model /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM...
(EngineCore_DP0 pid=113257) INFO 02-18 18:55:43 [cuda.py:364] Using FLASH_ATTN attention backend out of potential backends: ('FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION')
(EngineCore_DP0 pid=113257) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
(EngineCore_DP0 pid=113257) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
(EngineCore_DP0 pid=113257)
Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
(EngineCore_DP0 pid=113257)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.71it/s]
(EngineCore_DP0 pid=113257)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.71it/s]
(EngineCore_DP0 pid=113257)
(EngineCore_DP0 pid=113257) INFO 02-18 18:55:44 [default_loader.py:291] Loading weights took 0.66 seconds
(EngineCore_DP0 pid=113257) INFO 02-18 18:55:45 [gpu_model_runner.py:4130] Model loading took 0.95 GiB memory and 1.213121 seconds
(EngineCore_DP0 pid=113257) INFO 02-18 18:55:49 [backends.py:812] Using cache directory: /home/ubuntu/.cache/vllm/torch_compile_cache/f3ff13511c/rank_0_0/backbone for vLLM's torch.compile
(EngineCore_DP0 pid=113257) INFO 02-18 18:55:49 [backends.py:872] Dynamo bytecode transform time: 4.28 s


Output:
Chunk ID: 631537
Wall time: 5.0010 seconds
Process running with session ID 59889
Original token count: 760
Output:
(EngineCore_DP0 pid=113257) INFO 02-18 18:55:52 [backends.py:267] Directly load the compiled graph(s) for compile range (1, 4096) from the cache, took 0.624 s
(EngineCore_DP0 pid=113257) INFO 02-18 18:55:52 [monitor.py:34] torch.compile takes 4.90 s in total
(EngineCore_DP0 pid=113257) INFO 02-18 18:55:53 [gpu_worker.py:356] Available KV cache memory: 18.06 GiB
(EngineCore_DP0 pid=113257) INFO 02-18 18:55:53 [kv_cache_utils.py:1307] GPU KV cache size: 1,577,712 tokens
(EngineCore_DP0 pid=113257) INFO 02-18 18:55:53 [kv_cache_utils.py:1312] Maximum concurrency for 4,096 tokens per request: 385.18x
(EngineCore_DP0 pid=113257)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   0%|          | 0/35 [00:00<?, ?it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  11%|█▏        | 4/35 [00:00<00:00, 32.14it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  23%|██▎       | 8/35 [00:00<00:00, 34.98it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  34%|███▍      | 12/35 [00:00<00:00, 36.07it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  46%|████▌     | 16/35 [00:00<00:00, 36.55it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  57%|█████▋    | 20/35 [00:00<00:00, 36.94it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  69%|██████▊   | 24/35 [00:00<00:00, 36.95it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  80%|████████  | 28/35 [00:00<00:00, 35.79it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  91%|█████████▏| 32/35 [00:00<00:00, 34.05it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 35/35 [00:01<00:00, 34.58it/s]
(EngineCore_DP0 pid=113257)
Capturing CUDA graphs (decode, FULL):   0%|          | 0/19 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL):  21%|██        | 4/19 [00:00<00:00, 31.52it/s]
Capturing CUDA graphs (decode, FULL):  42%|████▏     | 8/19 [00:00<00:00, 34.48it/s]
Capturing CUDA graphs (decode, FULL):  63%|██████▎   | 12/19 [00:00<00:00, 35.30it/s]
Capturing CUDA graphs (decode, FULL):  84%|████████▍ | 16/19 [00:00<00:00, 35.78it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 35.50it/s]
(EngineCore_DP0 pid=113257) INFO 02-18 18:55:56 [gpu_model_runner.py:5063] Graph capturing finished in 2 secs, took 0.28 GiB
(EngineCore_DP0 pid=113257) INFO 02-18 18:55:56 [core.py:272] init engine (profile, create kv cache, warmup model) took 10.98 seconds
[2026-02-18 18:55:57] INFO tts_runtime.py:192: Initializing prompt builder...
[2026-02-18 18:55:57] INFO tts_runtime.py:199: Initializing BiCodec decoder...
/home/ubuntu/.local/lib/python3.10/site-packages/torch/nn/utils/weight_norm.py:144: FutureWarning: `torch.nn.utils.weight_norm` is deprecated in favor of `torch.nn.utils.parametrizations.weight_norm`.
  WeightNorm.apply(module, name, dim)


Still running cleanly with the gated scheduler. I’m waiting for the final `c=96/c=128` metrics to see whether this dynamic activation reduces the mid-concurrency penalty.


Output:
Chunk ID: 5e673f
Wall time: 5.0017 seconds
Process running with session ID 59889
Original token count: 127
Output:
[2026-02-18 18:56:06] INFO tts_runtime.py:209: Initializing TTS pipeline...
[2026-02-18 18:56:06] INFO tts_runtime.py:220: Initializing streaming pipeline...
[2026-02-18 18:56:06] INFO long_text_processor.py:112: LongTextProcessor initialized
[2026-02-18 18:56:06] INFO tts_runtime.py:277: ✅ TTS runtime initialized in 34003ms
[2026-02-18 18:56:06] INFO tts_runtime.py:278:    Model version: LLM
[2026-02-18 18:56:06] INFO tts_runtime.py:289: Speaker globals cache startup precompute skipped (disabled)


Output:
Chunk ID: 336203
Wall time: 5.0014 seconds
Process running with session ID 59889
Original token count: 1242
Output:
Total output lines: 87

✅ vLLM engine initialized successfully!

🎉 Spark TTS Model ready for inference!

🎵 Loading BiCodec audio tokenizer from /home/ubuntu/veenaModal/models/spark_tts_4speaker...
   Device: cuda
Missing tensor: mel_transformer.spectrogram.window
Missing tensor: mel_transformer.mel_scale.fb
✅ BiCodec decoder initialized (sample rate: 16000Hz)
   ℹ  Micro-batching enabled (workers=2, scale_pending=64, max_batch=16, timeout_ms=6.0)
🚀 SparkTTSPipeline initialized
🌊 Veena3SlidingWindowPipeline initialized (sliding window: 28 tokens)

=== streaming concurrency=96 requests=96 ===
success=96/96 (100%)
wall=4.40s throughput=21.80 req/s
latency_ms avg=3780 p50=3907 p95=4402 max=4402
ttfb_ms avg=1118 p50=1083 p95=1321 max=1322
chunks_per_req avg=5.1 p50=5.0 p95=5.0 max=6.0
inter_chunk_ms avg=642.6 p50=609.0 p95=1803.7 max=1954.9
tail_emit_speed_x avg=1.48 p50=1.42 p95=1.99 max=2.15
timeline_to_first_batch_ms: avg=53.47 p50=53.27 p95=61.43 max=63.24
timeline_to_first_chunk_emitted_ms: avg=1097.72 p50=1071.84 p95=1285.91 max=1287.03
timeline_total_ms: avg=3701.76 p50=3870.43 p95=4364.08 max=4367.66
llm_time_in_queue_ms: avg=0.07 p50=0.07 p95=0.15 max=0.19
llm_scheduler_ms: avg=37.57 p50=37.62 p95=45.06 max=45.06
llm_first_token_ms: avg=50.97 p50=50.68 p95=59.20 max=61.12
llm_request_lifecycle_ms: avg=2560.95 p50=2587.78 p95=2745.67 max=2835.08
llm_generation_wall_ms: avg=3684.75 p50=3838.10 p95=4364.04 max=4367.60
llm_time_per_token_ms: avg=16.10 p50=16.62 p95=18.42 max=19.00
llm_batch_wall_ms_total: avg=3248.31 p50=3158.85 p95=3932.89 max=4207.61
llm_decode_wall_ms_total: avg=3194.87 p50=3103.17 p95=3884.27 max=4148.66
tokens_per_batch_p50: avg=1.00 p50=1.00 p95=1.00 max=1.00
bicodec_decode_calls: avg=4.05 p50=4.00 p95=4.00 max=5.00
bicodec_decode_wall_ms_total: avg=2330.50 p50=2413.84 p95=3561.30 max=3684.10
bicodec_decode_interval_ms_p50: avg=721.77 p50=709.77 p95=952.91 max=1073.82
decode_interval_applied_tokens_p50: avg=48.00 p50=48.00 p95=48.00 max=48.00
decode_pending_requests_p50: avg=0.00 p50=0.00 p95=0.00 max=0.00
batch_avg_size: avg=10.34 p50=9.00 p95=13.05 max=13.05
batch_max_seen: avg=16.00 p50=16.00 p95=16.00 max=16.00
batch_workers_live: avg=2.00 p50=2.00 p95=2.00 max=2.00
batch_workers_target: avg=1.34 p50=1.00 p95=2.00 max=2.00
batch_queue_wait_ms_avg: avg=394.50 p50=413.04 p95=504.94 max=504.94
batch_compute_ms_avg: avg=174.52 p50=159.20 …42 tokens truncated…: llm=99.5% bicodec=63.0% other=0.0%
gpu: util_avg=76% util_max=99% mem_avg=47791MB mem_max=49629MB

=== streaming concurrency=128 requests=128 ===
success=128/128 (100%)
wall=5.57s throughput=22.99 req/s
latency_ms avg=4599 p50=4691 p95=5554 max=5566
ttfb_ms avg=1605 p50=1529 p95=2106 max=2107
chunks_per_req avg=4.7 p50=5.0 p95=5.0 max=5.0
inter_chunk_ms avg=797.5 p50=806.0 p95=2187.8 max=2428.0
tail_emit_speed_x avg=1.35 p50=1.24 p95=2.56 max=2.84
timeline_to_first_batch_ms: avg=61.38 p50=61.44 p95=70.43 max=72.65
timeline_to_first_chunk_emitted_ms: avg=1577.59 p50=1508.52 p95=2058.25 max=2059.68
timeline_total_ms: avg=4537.26 p50=4623.18 p95=5530.58 max=5545.18
llm_time_in_queue_ms: avg=0.12 p50=0.11 p95=0.26 max=0.29
llm_scheduler_ms: avg=41.67 p50=45.20 p95=53.14 max=53.14
llm_first_token_ms: avg=58.28 p50=58.58 p95=67.76 max=70.42
llm_request_lifecycle_ms: avg=2844.83 p50=2910.64 p95=3156.78 max=3277.39
llm_generation_wall_ms: avg=4520.13 p50=4610.91 p95=5530.54 max=5545.08
llm_time_per_token_ms: avg=20.11 p50=20.35 p95=24.64 max=25.92
llm_batch_wall_ms_total: avg=4003.29 p50=4005.76 p95=5305.55 max=5488.78
llm_decode_wall_ms_total: avg=3941.95 p50=3933.36 p95=5244.63 max=5436.95
tokens_per_batch_p50: avg=1.00 p50=1.00 p95=1.00 max=1.00
bicodec_decode_calls: avg=3.71 p50=4.00 p95=4.00 max=4.00
bicodec_decode_wall_ms_total: avg=3339.79 p50=3592.81 p95=4428.00 max=4488.28
bicodec_decode_interval_ms_p50: avg=1024.49 p50=1117.11 p95=1450.11 max=1612.72
decode_interval_applied_tokens_p50: avg=48.00 p50=48.00 p95=48.00 max=48.00
decode_pending_requests_p50: avg=0.00 p50=0.00 p95=0.00 max=0.00
batch_avg_size: avg=8.23 p50=8.10 p95=9.62 max=9.79
batch_max_seen: avg=16.00 p50=16.00 p95=16.00 max=16.00
batch_workers_live: avg=2.00 p50=2.00 p95=2.00 max=2.00
batch_workers_target: avg=1.52 p50=2.00 p95=2.00 max=2.00
batch_queue_wait_ms_avg: avg=616.23 p50=613.25 p95=726.46 max=726.88
batch_compute_ms_avg: avg=130.25 p50=130.31 p95=148.76 max=150.82
batch_queue_depth_avg: avg=64.07 p50=63.62 p95=67.01 max=67.21
latency_over_timeline_ms: avg=61.41 p50=42.15 p95=152.11 max=299.47
timeline_share: llm=99.6% bicodec=73.6% other=0.0%
gpu: util_avg=66% util_max=99% mem_avg=49629MB mem_max=49629MB

results saved: /tmp/stream_workers2_scale64_gated_seeded.json
ERROR 02-18 18:56:20 [core_client.py:605] Engine core proc EngineCore_DP0 died unexpectedly, shutting down client.


**Evaluating gating thresholds**


The gated `scale64` policy was too conservative: it reduced always-on penalty but also lost the big `c=128` gain. I’m doing one last pass with gated `scale_pending=56` to test a better balance.


python3 scripts/stress_test_streaming_runtime.py --levels 96,128 --text short --max-tokens 1024 --seed-base 2026 --timeout 240 --requests-multiplier 1 --min-requests 8 --gpu-memory 0.25 --stream-output-kind delta --stream-decode-interval 48 --stream-window-size 128 --bicodec-batch-max 16 --bicodec-batch-timeout-ms 6 --bicodec-batch-workers 2 --bicodec-batch-scale-pending 56 --output /tmp/stream_workers2_scale56_gated_seeded.json


Output:
Chunk ID: 14f653
Wall time: 1.0011 seconds
Process running with session ID 65679
Original token count: 0
Output:


Output:
Chunk ID: d38ba0
Wall time: 5.0011 seconds
Process running with session ID 65679
Original token count: 1277
Output:
Total output lines: 49

[2026-02-18 18:56:35] INFO tts_runtime.py:183: Loading SparkTTS model with vLLM...
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
🚀 Initializing Spark TTS Model
📁 Model path: /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
🗣  Speakers: 12 predefined (lipakshi, vardan, reet, Nandini, krishna, anika, adarsh, Nilay, Aarvi, Asha, Bittu, Mira)
🎭 Emotions: 10 tags in [bracket] format

📝 Loading tokenizer from model...
✅ Tokenizer loaded: 165168 tokens

🔍 Validating 12 speakers for Spark TTS...
  lipakshi → <|speaker_0|>: 7 tokens
  vardan → <|speaker_1|>: 7 tokens
  reet → <|speaker_2|>: 7 tokens
  Nandini → <|speaker_3|>: 7 tokens
  krishna → <|speaker_4|>: 7 tokens
  anika → <|speaker_5|>: 7 tokens
  adarsh → <|speaker_6|>: 7 tokens
  Nilay → <|speaker_7|>: 7 tokens
  Aarvi → <|speaker_8|>: 7 tokens
  Asha → <|speaker_9|>: 7 tokens
  Bittu → <|speaker_10|>: 8 tokens
  Mira → <|speaker_11|>: 8 tokens
✅ All 12 speakers validated with mapping

🔍 Validating 10 emotion tags for Spark TTS...
✅ All 10 emotion tags validated

📋 Sample emotion tags:
  [angry]: 1 tokens
  [curious]: 1 tokens
  [excited]: 1 tokens
  [giggle]: 1 tokens
  [laughs harder]: 1 tokens
  ...

🔧 Initializing vLLM engine with optimizations...
  - dtype: bfloat16
  - max_model_len: 4096
  - gpu_memory_utilization: 0.25
  - enable_prefix_caching: True
  - enforce_eager (disable CUDA graphs): False
INFO 02-18 18:56:35 [model.py:541] Resolved architecture: Qwen2ForCausalLM
INFO 02-18 18:56:35 [model.py:1561] Using max model len 4096
INFO 02-18 18:56:35 [scheduler.py:226] Chunked prefill is enabled with max_num_batched_tokens=4096.
INFO 02-18 18:56:35 [vllm.py:624] Asynchronous scheduling is enabled.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(EngineCore_DP0 pid=114283) INFO 02-18 18:56:36 [core.py:96] Initializing a V1 LLM engine (v0.15.1) with config: model='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', speculative_config=None, tokenizer='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype…77 tokens truncated…tputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [4096], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 256, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': True}, 'local_cache_dir': None, 'static_all_moe_layers': []}


Output:
Chunk ID: f7ccc3
Wall time: 5.0012 seconds
Process running with session ID 65679
Original token count: 507
Output:
(EngineCore_DP0 pid=114283) INFO 02-18 18:56:39 [parallel_state.py:1212] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://216.81.248.184:42719 backend=nccl
(EngineCore_DP0 pid=114283) INFO 02-18 18:56:39 [parallel_state.py:1423] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
(EngineCore_DP0 pid=114283) INFO 02-18 18:56:39 [gpu_model_runner.py:4033] Starting to load model /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM...
(EngineCore_DP0 pid=114283) INFO 02-18 18:56:40 [cuda.py:364] Using FLASH_ATTN attention backend out of potential backends: ('FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION')
(EngineCore_DP0 pid=114283) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
(EngineCore_DP0 pid=114283) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
(EngineCore_DP0 pid=114283)
Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
(EngineCore_DP0 pid=114283)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.21it/s]
(EngineCore_DP0 pid=114283)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.21it/s]
(EngineCore_DP0 pid=114283)
(EngineCore_DP0 pid=114283) INFO 02-18 18:56:41 [default_loader.py:291] Loading weights took 0.90 seconds
(EngineCore_DP0 pid=114283) INFO 02-18 18:56:42 [gpu_model_runner.py:4130] Model loading took 0.95 GiB memory and 1.444260 seconds
(EngineCore_DP0 pid=114283) INFO 02-18 18:56:46 [backends.py:812] Using cache directory: /home/ubuntu/.cache/vllm/torch_compile_cache/f3ff13511c/rank_0_0/backbone for vLLM's torch.compile
(EngineCore_DP0 pid=114283) INFO 02-18 18:56:46 [backends.py:872] Dynamo bytecode transform time: 4.22 s


Output:
Chunk ID: 9b8a1b
Wall time: 5.0005 seconds
Process running with session ID 65679
Original token count: 785
Output:
(EngineCore_DP0 pid=114283) INFO 02-18 18:56:49 [backends.py:267] Directly load the compiled graph(s) for compile range (1, 4096) from the cache, took 0.663 s
(EngineCore_DP0 pid=114283) INFO 02-18 18:56:49 [monitor.py:34] torch.compile takes 4.89 s in total
(EngineCore_DP0 pid=114283) INFO 02-18 18:56:50 [gpu_worker.py:356] Available KV cache memory: 18.06 GiB
(EngineCore_DP0 pid=114283) INFO 02-18 18:56:50 [kv_cache_utils.py:1307] GPU KV cache size: 1,577,712 tokens
(EngineCore_DP0 pid=114283) INFO 02-18 18:56:50 [kv_cache_utils.py:1312] Maximum concurrency for 4,096 tokens per request: 385.18x
(EngineCore_DP0 pid=114283)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   0%|          | 0/35 [00:00<?, ?it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  11%|█▏        | 4/35 [00:00<00:00, 33.04it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  23%|██▎       | 8/35 [00:00<00:00, 35.59it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  34%|███▍      | 12/35 [00:00<00:00, 36.76it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  46%|████▌     | 16/35 [00:00<00:00, 36.95it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  57%|█████▋    | 20/35 [00:00<00:00, 37.14it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  69%|██████▊   | 24/35 [00:00<00:00, 37.26it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  80%|████████  | 28/35 [00:00<00:00, 36.73it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  91%|█████████▏| 32/35 [00:00<00:00, 35.18it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 35/35 [00:00<00:00, 35.26it/s]
(EngineCore_DP0 pid=114283)
Capturing CUDA graphs (decode, FULL):   0%|          | 0/19 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL):  16%|█▌        | 3/19 [00:00<00:00, 28.99it/s]
Capturing CUDA graphs (decode, FULL):  37%|███▋      | 7/19 [00:00<00:00, 32.95it/s]
Capturing CUDA graphs (decode, FULL):  58%|█████▊    | 11/19 [00:00<00:00, 34.16it/s]
Capturing CUDA graphs (decode, FULL):  79%|███████▉  | 15/19 [00:00<00:00, 34.69it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 35.38it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 34.49it/s]
(EngineCore_DP0 pid=114283) INFO 02-18 18:56:52 [gpu_model_runner.py:5063] Graph capturing finished in 2 secs, took 0.28 GiB
(EngineCore_DP0 pid=114283) INFO 02-18 18:56:53 [core.py:272] init engine (profile, create kv cache, warmup model) took 10.84 seconds
[2026-02-18 18:56:54] INFO tts_runtime.py:192: Initializing prompt builder...
[2026-02-18 18:56:54] INFO tts_runtime.py:199: Initializing BiCodec decoder...
/home/ubuntu/.local/lib/python3.10/site-packages/torch/nn/utils/weight_norm.py:144: FutureWarning: `torch.nn.utils.weight_norm` is deprecated in favor of `torch.nn.utils.parametrizations.weight_norm`.
  WeightNorm.apply(module, name, dim)


Output:
Chunk ID: b5cbc8
Wall time: 5.0006 seconds
Process running with session ID 65679
Original token count: 127
Output:
[2026-02-18 18:56:59] INFO tts_runtime.py:209: Initializing TTS pipeline...
[2026-02-18 18:56:59] INFO tts_runtime.py:220: Initializing streaming pipeline...
[2026-02-18 18:56:59] INFO long_text_processor.py:112: LongTextProcessor initialized
[2026-02-18 18:56:59] INFO tts_runtime.py:277: ✅ TTS runtime initialized in 29872ms
[2026-02-18 18:56:59] INFO tts_runtime.py:278:    Model version: LLM
[2026-02-18 18:56:59] INFO tts_runtime.py:289: Speaker globals cache startup precompute skipped (disabled)


Output:
Chunk ID: 930785
Wall time: 5.0001 seconds
Process running with session ID 65679
Original token count: 1242
Output:
Total output lines: 87

✅ vLLM engine initialized successfully!

🎉 Spark TTS Model ready for inference!

🎵 Loading BiCodec audio tokenizer from /home/ubuntu/veenaModal/models/spark_tts_4speaker...
   Device: cuda
Missing tensor: mel_transformer.spectrogram.window
Missing tensor: mel_transformer.mel_scale.fb
✅ BiCodec decoder initialized (sample rate: 16000Hz)
   ℹ  Micro-batching enabled (workers=2, scale_pending=56, max_batch=16, timeout_ms=6.0)
🚀 SparkTTSPipeline initialized
🌊 Veena3SlidingWindowPipeline initialized (sliding window: 28 tokens)

=== streaming concurrency=96 requests=96 ===
success=96/96 (100%)
wall=4.20s throughput=22.85 req/s
latency_ms avg=3698 p50=3939 p95=4200 max=4200
ttfb_ms avg=1167 p50=1138 p95=1488 max=1489
chunks_per_req avg=5.0 p50=5.0 p95=5.0 max=5.0
inter_chunk_ms avg=622.2 p50=553.6 p95=1797.0 max=1865.4
tail_emit_speed_x avg=1.56 p50=1.45 p95=1.98 max=2.51
timeline_to_first_batch_ms: avg=60.37 p50=60.85 p95=69.51 max=71.54
timeline_to_first_chunk_emitted_ms: avg=1145.44 p50=1125.48 p95=1451.35 max=1452.46
timeline_total_ms: avg=3634.40 p50=3900.81 p95=4165.11 max=4167.99
llm_time_in_queue_ms: avg=0.11 p50=0.11 p95=0.23 max=0.26
llm_scheduler_ms: avg=39.96 p50=31.92 p95=54.54 max=54.54
llm_first_token_ms: avg=57.58 p50=58.20 p95=68.34 max=70.50
llm_request_lifecycle_ms: avg=2513.65 p50=2531.91 p95=2694.94 max=2802.45
llm_generation_wall_ms: avg=3618.79 p50=3888.11 p95=4165.07 max=4167.94
llm_time_per_token_ms: avg=15.83 p50=16.52 p95=17.95 max=19.12
llm_batch_wall_ms_total: avg=3242.13 p50=3190.66 p95=3829.83 max=3901.02
llm_decode_wall_ms_total: avg=3181.79 p50=3126.80 p95=3780.42 max=3853.11
tokens_per_batch_p50: avg=1.00 p50=1.00 p95=1.00 max=1.00
bicodec_decode_calls: avg=4.00 p50=4.00 p95=4.00 max=4.00
bicodec_decode_wall_ms_total: avg=2397.00 p50=2521.91 p95=3467.24 max=3468.01
bicodec_decode_interval_ms_p50: avg=669.46 p50=687.85 p95=905.35 max=1012.36
decode_interval_applied_tokens_p50: avg=48.00 p50=48.00 p95=48.00 max=48.00
decode_pending_requests_p50: avg=0.00 p50=0.00 p95=0.00 max=0.00
batch_avg_size: avg=9.40 p50=8.64 p95=11.65 max=12.33
batch_max_seen: avg=16.00 p50=16.00 p95=16.00 max=16.00
batch_workers_live: avg=2.00 p50=2.00 p95=2.00 max=2.00
batch_workers_target: avg=1.46 p50=1.00 p95=2.00 max=2.00
batch_queue_wait_ms_avg: avg=440.82 p50=499.75 p95=530.80 max=530.80
batch_compute_ms_avg: avg=174.01 p50=162.53 p…42 tokens truncated…: llm=99.6% bicodec=66.0% other=0.0%
gpu: util_avg=73% util_max=97% mem_avg=47939MB mem_max=49919MB

=== streaming concurrency=128 requests=128 ===
success=128/128 (100%)
wall=5.34s throughput=23.96 req/s
latency_ms avg=4552 p50=4633 p95=5338 max=5339
ttfb_ms avg=1361 p50=1312 p95=1616 max=1617
chunks_per_req avg=5.1 p50=5.0 p95=6.0 max=6.0
inter_chunk_ms avg=771.6 p50=781.6 p95=1976.3 max=2176.4
tail_emit_speed_x avg=1.24 p50=1.16 p95=1.66 max=1.80
timeline_to_first_batch_ms: avg=70.85 p50=72.79 p95=83.17 max=84.41
timeline_to_first_chunk_emitted_ms: avg=1334.31 p50=1293.55 p95=1569.52 max=1571.31
timeline_total_ms: avg=4474.87 p50=4586.51 p95=5289.78 max=5296.70
llm_time_in_queue_ms: avg=0.19 p50=0.17 p95=0.41 max=0.46
llm_scheduler_ms: avg=44.24 p50=35.59 p95=67.59 max=67.59
llm_first_token_ms: avg=67.43 p50=69.59 p95=80.93 max=83.48
llm_request_lifecycle_ms: avg=3063.19 p50=3069.46 p95=3326.18 max=3446.89
llm_generation_wall_ms: avg=4459.32 p50=4573.60 p95=5289.74 max=5296.65
llm_time_per_token_ms: avg=19.57 p50=20.14 p95=22.55 max=24.14
llm_batch_wall_ms_total: avg=3982.78 p50=3976.42 p95=4847.91 max=5088.65
llm_decode_wall_ms_total: avg=3911.95 p50=3905.13 p95=4765.97 max=5004.61
tokens_per_batch_p50: avg=1.00 p50=1.00 p95=1.00 max=1.00
bicodec_decode_calls: avg=4.07 p50=4.00 p95=5.00 max=5.00
bicodec_decode_wall_ms_total: avg=2960.87 p50=3219.19 p95=4294.50 max=4369.60
bicodec_decode_interval_ms_p50: avg=957.46 p50=966.39 p95=1382.20 max=1524.73
decode_interval_applied_tokens_p50: avg=48.00 p50=48.00 p95=48.00 max=48.00
decode_pending_requests_p50: avg=0.00 p50=0.00 p95=0.00 max=0.00
batch_avg_size: avg=9.59 p50=9.42 p95=10.37 max=10.47
batch_max_seen: avg=16.00 p50=16.00 p95=16.00 max=16.00
batch_workers_live: avg=2.00 p50=2.00 p95=2.00 max=2.00
batch_workers_target: avg=1.62 p50=2.00 p95=2.00 max=2.00
batch_queue_wait_ms_avg: avg=538.89 p50=560.11 p95=605.65 max=605.65
batch_compute_ms_avg: avg=147.78 p50=145.81 p95=157.81 max=159.83
batch_queue_depth_avg: avg=57.12 p50=57.18 p95=59.19 max=59.19
latency_over_timeline_ms: avg=77.26 p50=47.12 p95=187.64 max=316.27
timeline_share: llm=99.7% bicodec=66.2% other=0.0%
gpu: util_avg=70% util_max=99% mem_avg=50640MB mem_max=50949MB

results saved: /tmp/stream_workers2_scale56_gated_seeded.json
ERROR 02-18 18:57:13 [core_client.py:605] Engine core proc EngineCore_DP0 died unexpectedly, shutting down client.


**Comparing all runs for best result**


python3 - <<'PY'
import json
profiles={
 'w1':'/tmp/stream_workers1_seeded.json',
 'w2':'/tmp/stream_workers2_seeded.json',
 'w2_scale56':'/tmp/stream_workers2_scale56_seeded.json',
 'w2_scale64':'/tmp/stream_workers2_scale64_seeded.json',
 'w2_scale72':'/tmp/stream_workers2_scale72_seeded.json',
 'w2_scale64_g':'/tmp/stream_workers2_scale64_gated_seeded.json',
 'w2_scale56_g':'/tmp/stream_workers2_scale56_gated_seeded.json',
}
res={}
for k,p in profiles.items():
    try:
        d=json.load(open(p))['levels']
    except Exception:
        continue
    res[k]={r['concurrency']:r for r in d}
for c in [96,128]:
    print(f"\n=== c={c} ===")
    rows=[]
    for k,by in res.items():
        if c not in by: continue
        r=by[c]; t=r['timing']
        rows.append((r['throughput_rps'],k,r['latency_ms']['avg'],r['ttfb_ms']['avg'],t['batch_queue_wait_ms_avg']['avg'],t.get('batch_workers_target',{}).get('avg',0)))
    for thr,k,lat,ttfb,qw,wt in sorted(rows, reverse=True):
        print(f"{k:13s} thr={thr:.2f} lat={lat:.0f} ttfb={ttfb:.0f} qwait={qw:.0f} w_target={wt:.2f}")
PY


Output:
Chunk ID: 871616
Wall time: 0.0513 seconds
Process exited with code 0
Original token count: 242
Output:

=== c=96 ===
w1            thr=23.05 lat=3627 ttfb=1155 qwait=478 w_target=0.00
w2_scale56_g  thr=22.85 lat=3698 ttfb=1167 qwait=441 w_target=1.46
w2            thr=22.25 lat=3763 ttfb=1151 qwait=502 w_target=0.00
w2_scale56    thr=22.14 lat=3743 ttfb=1133 qwait=377 w_target=2.00
w2_scale64    thr=22.01 lat=3710 ttfb=1115 qwait=372 w_target=2.00
w2_scale64_g  thr=21.80 lat=3780 ttfb=1118 qwait=395 w_target=1.34
w2_scale72    thr=20.63 lat=4058 ttfb=1164 qwait=499 w_target=2.00

=== c=128 ===
w2_scale64    thr=26.81 lat=4217 ttfb=1227 qwait=463 w_target=2.00
w2_scale56_g  thr=23.96 lat=4552 ttfb=1361 qwait=539 w_target=1.62
w2_scale56    thr=23.37 lat=4549 ttfb=1363 qwait=564 w_target=2.00
w2            thr=23.12 lat=4565 ttfb=1383 qwait=576 w_target=0.00
w2_scale64_g  thr=22.99 lat=4599 ttfb=1605 qwait=616 w_target=1.52
w1            thr=22.79 lat=4619 ttfb=1520 qwait=694 w_target=0.00
w2_scale72    thr=21.60 lat=4921 ttfb=1534 qwait=688 w_target=2.00


Summary: auto


**Designing configurable batch scaling modes**


I’m adding an explicit scale mode so we can choose between `sticky` (once scaled up, keep extra worker active) and `dynamic` (workers can idle). This lets us keep a high-throughput profile for peak load and a safer profile for mixed load.


Input:
*** Begin Patch
*** Update File: veena3modal/core/bicodec_decoder.py
@@
         self.batch_timeout_s = _env_float("VEENA3_BICODEC_BATCH_TIMEOUT_MS", 6.0, minimum=0.1) / 1000.0
         self.batch_worker_count = _env_int("VEENA3_BICODEC_BATCH_WORKERS", 1, minimum=1)
         self.batch_worker_scale_pending = _env_int("VEENA3_BICODEC_BATCH_SCALE_PENDING", 0, minimum=0)
+        self.batch_worker_scale_mode = os.environ.get("VEENA3_BICODEC_BATCH_SCALE_MODE", "sticky").strip().lower()
+        if self.batch_worker_scale_mode not in {"sticky", "dynamic"}:
+            self.batch_worker_scale_mode = "sticky"
@@
         self._batch_queues: Dict[int, asyncio.Queue] = {}
         self._batch_workers: Dict[int, List[asyncio.Task]] = {}
+        self._batch_scaled_loops: Dict[int, bool] = {}
@@
                 "   ℹ  Micro-batching enabled "
-                f"(workers={self.batch_worker_count}, scale_pending={self.batch_worker_scale_pending}, "
+                f"(workers={self.batch_worker_count}, scale_mode={self.batch_worker_scale_mode}, "
+                f"scale_pending={self.batch_worker_scale_pending}, "
                 f"max_batch={self.batch_max_size}, timeout_ms={self.batch_timeout_s * 1000:.1f})"
             )
@@
             if queue is None:
                 queue = asyncio.Queue(maxsize=max(self.batch_max_size * 8, 256))
                 self._batch_queues[key] = queue

             target_workers = self.batch_worker_count
             if self.batch_worker_count > 1 and self.batch_worker_scale_pending > 0:
                 pending_now = queue.qsize() + self._batch_inflight_requests
-                if pending_now < self.batch_worker_scale_pending:
-                    target_workers = 1
+                if self.batch_worker_scale_mode == "sticky":
+                    already_scaled = self._batch_scaled_loops.get(key, False)
+                    should_scale = already_scaled or pending_now >= self.batch_worker_scale_pending
+                    self._batch_scaled_loops[key] = should_scale
+                    if not should_scale:
+                        target_workers = 1
+                elif pending_now < self.batch_worker_scale_pending:
+                    target_workers = 1
             while len(live_workers) < target_workers:
                 worker_idx = len(live_workers)
                 worker = loop.create_task(
                     self._batch_worker(loop, queue, worker_idx),
                     name=f"bicodec-batch-worker-{key}-{worker_idx}",
@@
     async def _batch_worker(
         self,
         loop: asyncio.AbstractEventLoop,
         queue: asyncio.Queue,
         worker_idx: int,
     ) -> None:
         while True:
-            target_workers = self._desired_worker_count(queue)
-            self._batch_workers_target_last = target_workers
-            if worker_idx >= target_workers:
-                await asyncio.sleep(min(self.batch_timeout_s, 0.002))
-                continue
+            if self.batch_worker_scale_mode == "dynamic":
+                target_workers = self._desired_worker_count(queue)
+                self._batch_workers_target_last = target_workers
+                if worker_idx >= target_workers:
+                    await asyncio.sleep(min(self.batch_timeout_s, 0.002))
+                    continue

             first: _BatchDecodeRequest = await queue.get()
@@
             "batch_workers_configured": float(self.batch_worker_count),
             "batch_workers_live": float(workers_live),
             "batch_workers_target": float(self._batch_workers_target_last),
             "batch_worker_scale_pending": float(self.batch_worker_scale_pending),
+            "batch_worker_scale_mode": self.batch_worker_scale_mode,
             "batch_queue_wait_ms_avg": float(self._batch_queue_wait_ms_total / reqs),
*** End Patch


Output: {"output":"Success. Updated the following files:\nM veena3modal/core/bicodec_decoder.py\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


**Planning argument and config addition**


Input:
*** Begin Patch
*** Update File: scripts/stress_test_streaming_runtime.py
@@
             "bicodec_batch_workers": args.bicodec_batch_workers,
             "bicodec_batch_scale_pending": args.bicodec_batch_scale_pending,
+            "bicodec_batch_scale_mode": args.bicodec_batch_scale_mode,
             "timestamp": int(time.time()),
         },
@@
     parser.add_argument("--bicodec-batch-timeout-ms", type=float, default=6.0)
     parser.add_argument("--bicodec-batch-workers", type=int, default=1)
     parser.add_argument("--bicodec-batch-scale-pending", type=int, default=0)
+    parser.add_argument("--bicodec-batch-scale-mode", choices=["sticky", "dynamic"], default="sticky")
     parser.add_argument("--model-path", type=str, default="")
@@
     os.environ["VEENA3_BICODEC_BATCH_TIMEOUT_MS"] = str(args.bicodec_batch_timeout_ms)
     os.environ["VEENA3_BICODEC_BATCH_WORKERS"] = str(args.bicodec_batch_workers)
     os.environ["VEENA3_BICODEC_BATCH_SCALE_PENDING"] = str(args.bicodec_batch_scale_pending)
+    os.environ["VEENA3_BICODEC_BATCH_SCALE_MODE"] = str(args.bicodec_batch_scale_mode)
*** End Patch


Output: {"output":"Success. Updated the following files:\nM scripts/stress_test_streaming_runtime.py\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


python3 -m py_compile veena3modal/core/bicodec_decoder.py scripts/stress_test_streaming_runtime.py


Output:
Chunk ID: 71a144
Wall time: 0.0512 seconds
Process exited with code 0
Original token count: 0
Output:


**Running benchmark for sticky scale64**


I’ve added explicit `sticky/dynamic` mode selection and compiled. I’m running a confirmation pass for `sticky + scale_pending=64` (the highest-throughput candidate) to lock final recommendations.


python3 scripts/stress_test_streaming_runtime.py --levels 96,128 --text short --max-tokens 1024 --seed-base 2026 --timeout 240 --requests-multiplier 1 --min-requests 8 --gpu-memory 0.25 --stream-output-kind delta --stream-decode-interval 48 --stream-window-size 128 --bicodec-batch-max 16 --bicodec-batch-timeout-ms 6 --bicodec-batch-workers 2 --bicodec-batch-scale-pending 64 --bicodec-batch-scale-mode sticky --output /tmp/stream_workers2_scale64_sticky_seeded.json


Output:
Chunk ID: 88474a
Wall time: 1.0011 seconds
Process running with session ID 4317
Original token count: 0
Output:


**Scheduling periodic updates**


Output:
Chunk ID: cc2e40
Wall time: 5.0010 seconds
Process running with session ID 4317
Original token count: 1362
Output:
Total output lines: 51

[2026-02-18 18:58:22] INFO tts_runtime.py:183: Loading SparkTTS model with vLLM...
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
🚀 Initializing Spark TTS Model
📁 Model path: /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
🗣  Speakers: 12 predefined (lipakshi, vardan, reet, Nandini, krishna, anika, adarsh, Nilay, Aarvi, Asha, Bittu, Mira)
🎭 Emotions: 10 tags in [bracket] format

📝 Loading tokenizer from model...
✅ Tokenizer loaded: 165168 tokens

🔍 Validating 12 speakers for Spark TTS...
  lipakshi → <|speaker_0|>: 7 tokens
  vardan → <|speaker_1|>: 7 tokens
  reet → <|speaker_2|>: 7 tokens
  Nandini → <|speaker_3|>: 7 tokens
  krishna → <|speaker_4|>: 7 tokens
  anika → <|speaker_5|>: 7 tokens
  adarsh → <|speaker_6|>: 7 tokens
  Nilay → <|speaker_7|>: 7 tokens
  Aarvi → <|speaker_8|>: 7 tokens
  Asha → <|speaker_9|>: 7 tokens
  Bittu → <|speaker_10|>: 8 tokens
  Mira → <|speaker_11|>: 8 tokens
✅ All 12 speakers validated with mapping

🔍 Validating 10 emotion tags for Spark TTS...
✅ All 10 emotion tags validated

📋 Sample emotion tags:
  [angry]: 1 tokens
  [curious]: 1 tokens
  [excited]: 1 tokens
  [giggle]: 1 tokens
  [laughs harder]: 1 tokens
  ...

🔧 Initializing vLLM engine with optimizations...
  - dtype: bfloat16
  - max_model_len: 4096
  - gpu_memory_utilization: 0.25
  - enable_prefix_caching: True
  - enforce_eager (disable CUDA graphs): False
INFO 02-18 18:58:23 [model.py:541] Resolved architecture: Qwen2ForCausalLM
INFO 02-18 18:58:23 [model.py:1561] Using max model len 4096
INFO 02-18 18:58:23 [scheduler.py:226] Chunked prefill is enabled with max_num_batched_tokens=4096.
INFO 02-18 18:58:23 [vllm.py:624] Asynchronous scheduling is enabled.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(EngineCore_DP0 pid=116006) INFO 02-18 18:58:24 [core.py:96] Initializing a V1 LLM engine (v0.15.1) with config: model='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', speculative_config=None, tokenizer='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype…162 tokens truncated…ailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [4096], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 256, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': True}, 'local_cache_dir': None, 'static_all_moe_layers': []}
(EngineCore_DP0 pid=116006) INFO 02-18 18:58:26 [parallel_state.py:1212] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://216.81.248.184:57657 backend=nccl
(EngineCore_DP0 pid=116006) INFO 02-18 18:58:26 [parallel_state.py:1423] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A


Output:
Chunk ID: 396704
Wall time: 5.0010 seconds
Process running with session ID 4317
Original token count: 423
Output:
(EngineCore_DP0 pid=116006) INFO 02-18 18:58:27 [gpu_model_runner.py:4033] Starting to load model /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM...
(EngineCore_DP0 pid=116006) INFO 02-18 18:58:28 [cuda.py:364] Using FLASH_ATTN attention backend out of potential backends: ('FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION')
(EngineCore_DP0 pid=116006) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
(EngineCore_DP0 pid=116006) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
(EngineCore_DP0 pid=116006)
Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
(EngineCore_DP0 pid=116006)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:01<00:00,  1.09s/it]
(EngineCore_DP0 pid=116006)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:01<00:00,  1.10s/it]
(EngineCore_DP0 pid=116006)
(EngineCore_DP0 pid=116006) INFO 02-18 18:58:29 [default_loader.py:291] Loading weights took 1.17 seconds
(EngineCore_DP0 pid=116006) INFO 02-18 18:58:30 [gpu_model_runner.py:4130] Model loading took 0.95 GiB memory and 1.720871 seconds
(EngineCore_DP0 pid=116006) INFO 02-18 18:58:34 [backends.py:812] Using cache directory: /home/ubuntu/.cache/vllm/torch_compile_cache/f3ff13511c/rank_0_0/backbone for vLLM's torch.compile
(EngineCore_DP0 pid=116006) INFO 02-18 18:58:34 [backends.py:872] Dynamo bytecode transform time: 4.23 s


Output:
Chunk ID: 9ac505
Wall time: 5.0017 seconds
Process running with session ID 4317
Original token count: 785
Output:
(EngineCore_DP0 pid=116006) INFO 02-18 18:58:37 [backends.py:267] Directly load the compiled graph(s) for compile range (1, 4096) from the cache, took 0.666 s
(EngineCore_DP0 pid=116006) INFO 02-18 18:58:37 [monitor.py:34] torch.compile takes 4.89 s in total
(EngineCore_DP0 pid=116006) INFO 02-18 18:58:38 [gpu_worker.py:356] Available KV cache memory: 18.06 GiB
(EngineCore_DP0 pid=116006) INFO 02-18 18:58:38 [kv_cache_utils.py:1307] GPU KV cache size: 1,577,712 tokens
(EngineCore_DP0 pid=116006) INFO 02-18 18:58:38 [kv_cache_utils.py:1312] Maximum concurrency for 4,096 tokens per request: 385.18x
(EngineCore_DP0 pid=116006)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   0%|          | 0/35 [00:00<?, ?it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  11%|█▏        | 4/35 [00:00<00:00, 31.82it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  23%|██▎       | 8/35 [00:00<00:00, 34.77it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  34%|███▍      | 12/35 [00:00<00:00, 35.35it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  46%|████▌     | 16/35 [00:00<00:00, 35.51it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  57%|█████▋    | 20/35 [00:00<00:00, 35.79it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  69%|██████▊   | 24/35 [00:00<00:00, 35.74it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  80%|████████  | 28/35 [00:00<00:00, 35.68it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  91%|█████████▏| 32/35 [00:00<00:00, 34.11it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 35/35 [00:01<00:00, 34.01it/s]
(EngineCore_DP0 pid=116006)
Capturing CUDA graphs (decode, FULL):   0%|          | 0/19 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL):  16%|█▌        | 3/19 [00:00<00:00, 29.19it/s]
Capturing CUDA graphs (decode, FULL):  37%|███▋      | 7/19 [00:00<00:00, 33.48it/s]
Capturing CUDA graphs (decode, FULL):  58%|█████▊    | 11/19 [00:00<00:00, 34.58it/s]
Capturing CUDA graphs (decode, FULL):  79%|███████▉  | 15/19 [00:00<00:00, 34.84it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 35.51it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 34.71it/s]
(EngineCore_DP0 pid=116006) INFO 02-18 18:58:41 [gpu_model_runner.py:5063] Graph capturing finished in 2 secs, took 0.28 GiB
(EngineCore_DP0 pid=116006) INFO 02-18 18:58:41 [core.py:272] init engine (profile, create kv cache, warmup model) took 11.03 seconds
[2026-02-18 18:58:43] INFO tts_runtime.py:192: Initializing prompt builder...
[2026-02-18 18:58:43] INFO tts_runtime.py:199: Initializing BiCodec decoder...
/home/ubuntu/.local/lib/python3.10/site-packages/torch/nn/utils/weight_norm.py:144: FutureWarning: `torch.nn.utils.weight_norm` is deprecated in favor of `torch.nn.utils.parametrizations.weight_norm`.
  WeightNorm.apply(module, name, dim)


Output:
Chunk ID: a72ff1
Wall time: 5.0010 seconds
Process running with session ID 4317
Original token count: 127
Output:
[2026-02-18 18:58:52] INFO tts_runtime.py:209: Initializing TTS pipeline...
[2026-02-18 18:58:52] INFO tts_runtime.py:220: Initializing streaming pipeline...
[2026-02-18 18:58:52] INFO long_text_processor.py:112: LongTextProcessor initialized
[2026-02-18 18:58:52] INFO tts_runtime.py:277: ✅ TTS runtime initialized in 34820ms
[2026-02-18 18:58:52] INFO tts_runtime.py:278:    Model version: LLM
[2026-02-18 18:58:52] INFO tts_runtime.py:289: Speaker globals cache startup precompute skipped (disabled)


Output:
Chunk ID: 5a6418
Wall time: 5.0010 seconds
Process running with session ID 4317
Original token count: 1247
Output:
Total output lines: 87

✅ vLLM engine initialized successfully!

🎉 Spark TTS Model ready for inference!

🎵 Loading BiCodec audio tokenizer from /home/ubuntu/veenaModal/models/spark_tts_4speaker...
   Device: cuda
Missing tensor: mel_transformer.spectrogram.window
Missing tensor: mel_transformer.mel_scale.fb
✅ BiCodec decoder initialized (sample rate: 16000Hz)
   ℹ  Micro-batching enabled (workers=2, scale_mode=sticky, scale_pending=64, max_batch=16, timeout_ms=6.0)
🚀 SparkTTSPipeline initialized
🌊 Veena3SlidingWindowPipeline initialized (sliding window: 28 tokens)

=== streaming concurrency=96 requests=96 ===
success=96/96 (100%)
wall=4.44s throughput=21.62 req/s
latency_ms avg=3775 p50=4109 p95=4438 max=4438
ttfb_ms avg=1103 p50=1064 p95=1306 max=1307
chunks_per_req avg=5.1 p50=5.0 p95=6.0 max=6.0
inter_chunk_ms avg=634.4 p50=587.1 p95=1707.4 max=1928.9
tail_emit_speed_x avg=1.48 p50=1.37 p95=2.08 max=2.60
timeline_to_first_batch_ms: avg=51.11 p50=50.77 p95=58.60 max=59.43
timeline_to_first_chunk_emitted_ms: avg=1082.66 p50=1053.33 p95=1271.15 max=1272.15
timeline_total_ms: avg=3699.44 p50=3869.90 p95=4403.78 max=4407.66
llm_time_in_queue_ms: avg=0.09 p50=0.08 p95=0.21 max=0.23
llm_scheduler_ms: avg=35.03 p50=36.54 p95=43.01 max=43.01
llm_first_token_ms: avg=48.58 p50=48.37 p95=56.86 max=58.58
llm_request_lifecycle_ms: avg=2463.93 p50=2492.07 p95=2660.66 max=2752.87
llm_generation_wall_ms: avg=3680.85 p50=3857.02 p95=4403.74 max=4407.60
llm_time_per_token_ms: avg=16.21 p50=16.74 p95=19.03 max=19.33
llm_batch_wall_ms_total: avg=3288.45 p50=3300.09 p95=4102.94 max=4256.93
llm_decode_wall_ms_total: avg=3237.38 p50=3251.54 p95=4048.42 max=4199.35
tokens_per_batch_p50: avg=1.00 p50=1.00 p95=1.00 max=1.00
bicodec_decode_calls: avg=4.12 p50=4.00 p95=5.00 max=5.00
bicodec_decode_wall_ms_total: avg=2534.15 p50=2640.82 p95=3641.43 max=3719.64
bicodec_decode_interval_ms_p50: avg=751.34 p50=689.44 p95=1276.21 max=1325.32
decode_interval_applied_tokens_p50: avg=48.00 p50=48.00 p95=48.00 max=48.00
decode_pending_requests_p50: avg=0.00 p50=0.00 p95=0.00 max=0.00
batch_avg_size: avg=8.90 p50=8.54 p95=11.35 max=11.35
batch_max_seen: avg=16.00 p50=16.00 p95=16.00 max=16.00
batch_workers_live: avg=2.00 p50=2.00 p95=2.00 max=2.00
batch_workers_target: avg=2.00 p50=2.00 p95=2.00 max=2.00
batch_queue_wait_ms_avg: avg=437.05 p50=483.70 p95=557.41 max=560.70
batch_compute_ms_avg: avg…47 tokens truncated…llm=99.5% bicodec=68.5% other=0.0%
gpu: util_avg=78% util_max=100% mem_avg=47755MB mem_max=49527MB

=== streaming concurrency=128 requests=128 ===
success=128/128 (100%)
wall=5.39s throughput=23.77 req/s
latency_ms avg=4585 p50=4757 p95=5382 max=5382
ttfb_ms avg=1412 p50=1336 p95=1868 max=1869
chunks_per_req avg=5.0 p50=5.0 p95=5.0 max=6.0
inter_chunk_ms avg=787.4 p50=851.8 p95=2067.8 max=2412.0
tail_emit_speed_x avg=1.24 p50=1.20 p95=1.64 max=2.59
timeline_to_first_batch_ms: avg=64.88 p50=66.93 p95=76.55 max=78.09
timeline_to_first_chunk_emitted_ms: avg=1385.09 p50=1313.44 p95=1820.69 max=1822.21
timeline_total_ms: avg=4516.42 p50=4663.38 p95=5343.76 max=5352.20
llm_time_in_queue_ms: avg=0.22 p50=0.19 p95=0.48 max=0.52
llm_scheduler_ms: avg=42.95 p50=36.00 p95=60.37 max=60.37
llm_first_token_ms: avg=61.12 p50=63.80 p95=74.20 max=77.09
llm_request_lifecycle_ms: avg=2999.06 p50=3018.62 p95=3278.61 max=3368.04
llm_generation_wall_ms: avg=4500.51 p50=4649.51 p95=5343.72 max=5352.14
llm_time_per_token_ms: avg=19.79 p50=20.38 p95=22.92 max=25.54
llm_batch_wall_ms_total: avg=4035.23 p50=4055.62 p95=4963.93 max=5207.52
llm_decode_wall_ms_total: avg=3970.38 p50=3985.50 p95=4914.46 max=5135.78
tokens_per_batch_p50: avg=1.00 p50=1.00 p95=1.00 max=1.00
bicodec_decode_calls: avg=3.98 p50=4.00 p95=4.00 max=5.00
bicodec_decode_wall_ms_total: avg=3231.61 p50=3424.60 p95=4388.96 max=4452.89
bicodec_decode_interval_ms_p50: avg=1012.20 p50=1025.63 p95=1459.56 max=1624.31
decode_interval_applied_tokens_p50: avg=48.00 p50=48.00 p95=48.00 max=48.00
decode_pending_requests_p50: avg=0.00 p50=0.00 p95=0.00 max=0.00
batch_avg_size: avg=8.34 p50=8.25 p95=9.52 max=9.52
batch_max_seen: avg=16.00 p50=16.00 p95=16.00 max=16.00
batch_workers_live: avg=2.00 p50=2.00 p95=2.00 max=2.00
batch_workers_target: avg=2.00 p50=2.00 p95=2.00 max=2.00
batch_queue_wait_ms_avg: avg=596.86 p50=605.41 p95=683.22 max=686.16
batch_compute_ms_avg: avg=143.57 p50=141.43 p95=160.37 max=161.92
batch_queue_depth_avg: avg=58.99 p50=59.88 p95=61.36 max=61.45
latency_over_timeline_ms: avg=68.99 p50=43.96 p95=181.42 max=226.26
timeline_share: llm=99.6% bicodec=71.6% other=0.0%
gpu: util_avg=71% util_max=98% mem_avg=50428MB mem_max=50655MB

results saved: /tmp/stream_workers2_scale64_sticky_seeded.json
ERROR 02-18 18:59:05 [core_client.py:605] Engine core proc EngineCore_DP0 died unexpectedly, shutting down client.


**Evaluating run variability and baseline rerun**


I’m running one fresh `workers=1` baseline on the current code so the final comparison is fair against the new scale-mode implementation (same seeds, same load).


python3 scripts/stress_test_streaming_runtime.py --levels 96,128 --text short --max-tokens 1024 --seed-base 2026 --timeout 240 --requests-multiplier 1 --min-requests 8 --gpu-memory 0.25 --stream-output-kind delta --stream-decode-interval 48 --stream-window-size 128 --bicodec-batch-max 16 --bicodec-batch-timeout-ms 6 --bicodec-batch-workers 1 --bicodec-batch-scale-pending 0 --bicodec-batch-scale-mode sticky --output /tmp/stream_workers1_current_seeded.json


Output:
Chunk ID: 160b2b
Wall time: 1.0007 seconds
Process running with session ID 21146
Original token count: 0
Output:


**Planning update polling**


Output:
Chunk ID: c3ff7e
Wall time: 5.0010 seconds
Process running with session ID 21146
Original token count: 1362
Output:
Total output lines: 51

[2026-02-18 18:59:26] INFO tts_runtime.py:183: Loading SparkTTS model with vLLM...
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
🚀 Initializing Spark TTS Model
📁 Model path: /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
🗣  Speakers: 12 predefined (lipakshi, vardan, reet, Nandini, krishna, anika, adarsh, Nilay, Aarvi, Asha, Bittu, Mira)
🎭 Emotions: 10 tags in [bracket] format

📝 Loading tokenizer from model...
✅ Tokenizer loaded: 165168 tokens

🔍 Validating 12 speakers for Spark TTS...
  lipakshi → <|speaker_0|>: 7 tokens
  vardan → <|speaker_1|>: 7 tokens
  reet → <|speaker_2|>: 7 tokens
  Nandini → <|speaker_3|>: 7 tokens
  krishna → <|speaker_4|>: 7 tokens
  anika → <|speaker_5|>: 7 tokens
  adarsh → <|speaker_6|>: 7 tokens
  Nilay → <|speaker_7|>: 7 tokens
  Aarvi → <|speaker_8|>: 7 tokens
  Asha → <|speaker_9|>: 7 tokens
  Bittu → <|speaker_10|>: 8 tokens
  Mira → <|speaker_11|>: 8 tokens
✅ All 12 speakers validated with mapping

🔍 Validating 10 emotion tags for Spark TTS...
✅ All 10 emotion tags validated

📋 Sample emotion tags:
  [angry]: 1 tokens
  [curious]: 1 tokens
  [excited]: 1 tokens
  [giggle]: 1 tokens
  [laughs harder]: 1 tokens
  ...

🔧 Initializing vLLM engine with optimizations...
  - dtype: bfloat16
  - max_model_len: 4096
  - gpu_memory_utilization: 0.25
  - enable_prefix_caching: True
  - enforce_eager (disable CUDA graphs): False
INFO 02-18 18:59:26 [model.py:541] Resolved architecture: Qwen2ForCausalLM
INFO 02-18 18:59:26 [model.py:1561] Using max model len 4096
INFO 02-18 18:59:27 [scheduler.py:226] Chunked prefill is enabled with max_num_batched_tokens=4096.
INFO 02-18 18:59:27 [vllm.py:624] Asynchronous scheduling is enabled.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(EngineCore_DP0 pid=117118) INFO 02-18 18:59:28 [core.py:96] Initializing a V1 LLM engine (v0.15.1) with config: model='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', speculative_config=None, tokenizer='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype…162 tokens truncated…ailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [4096], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 256, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': True}, 'local_cache_dir': None, 'static_all_moe_layers': []}
(EngineCore_DP0 pid=117118) INFO 02-18 18:59:30 [parallel_state.py:1212] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://216.81.248.184:42797 backend=nccl
(EngineCore_DP0 pid=117118) INFO 02-18 18:59:30 [parallel_state.py:1423] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A


Output:
Chunk ID: 6c0852
Wall time: 5.0014 seconds
Process running with session ID 21146
Original token count: 1108
Output:
Total output lines: 21

(EngineCore_DP0 pid=117118) INFO 02-18 18:59:30 [gpu_model_runner.py:4033] Starting to load model /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM...
(EngineCore_DP0 pid=117118) INFO 02-18 18:59:31 [cuda.py:364] Using FLASH_ATTN attention backend out of potential backends: ('FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION')
(EngineCore_DP0 pid=117118) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
(EngineCore_DP0 pid=117118) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
(EngineCore_DP0 pid=117118)
Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
(EngineCore_DP0 pid=117118)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.15it/s]
(EngineCore_DP0 pid=117118)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.15it/s]
(EngineCore_DP0 pid=117118)
(EngineCore_DP0 pid=117118) INFO 02-18 18:59:32 [default_loader.py:291] Loading weights took 0.94 seconds
(EngineCore_DP0 pid=117118) INFO 02-18 18:59:33 [gpu_model_runner.py:4130] Model loading took 0.95 GiB memory and 1.487972 seconds
(EngineCore_DP0 pid=117118) INFO 02-18 18:59:37 [backends.py:812] Using cache directory: /home/ubuntu/.cache/vllm/torch_compile_cache/f3ff13511c/rank_0_0/backbone for vLLM's torch.compile
(EngineCore_DP0 …308 tokens truncated… | 16/35 [00:00<00:00, 36.70it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  57%|█████▋    | 20/35 [00:00<00:00, 36.20it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  69%|██████▊   | 24/35 [00:00<00:00, 36.59it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  80%|████████  | 28/35 [00:00<00:00, 36.12it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  91%|█████████▏| 32/35 [00:00<00:00, 34.67it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 35/35 [00:01<00:00, 34.79it/s]
(EngineCore_DP0 pid=117118)
Capturing CUDA graphs (decode, FULL):   0%|          | 0/19 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL):  16%|█▌        | 3/19 [00:00<00:00, 29.28it/s]
Capturing CUDA graphs (decode, FULL):  37%|███▋      | 7/19 [00:00<00:00, 32.43it/s]
Capturing CUDA graphs (decode, FULL):  58%|█████▊    | 11/19 [00:00<00:00, 34.21it/s]
Capturing CUDA graphs (decode, FULL):  79%|███████▉  | 15/19 [00:00<00:00, 35.02it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 35.71it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 34.69it/s]
(EngineCore_DP0 pid=117118) INFO 02-18 18:59:44 [gpu_model_runner.py:5063] Graph capturing finished in 2 secs, took 0.28 GiB
(EngineCore_DP0 pid=117118) INFO 02-18 18:59:44 [core.py:272] init engine (profile, create kv cache, warmup model) took 10.82 seconds


Output:
Chunk ID: 8ee580
Wall time: 5.0010 seconds
Process running with session ID 21146
Original token count: 226
Output:
[2026-02-18 18:59:45] INFO tts_runtime.py:192: Initializing prompt builder...
[2026-02-18 18:59:45] INFO tts_runtime.py:199: Initializing BiCodec decoder...
/home/ubuntu/.local/lib/python3.10/site-packages/torch/nn/utils/weight_norm.py:144: FutureWarning: `torch.nn.utils.weight_norm` is deprecated in favor of `torch.nn.utils.parametrizations.weight_norm`.
  WeightNorm.apply(module, name, dim)
[2026-02-18 18:59:55] INFO tts_runtime.py:209: Initializing TTS pipeline...
[2026-02-18 18:59:55] INFO tts_runtime.py:220: Initializing streaming pipeline...
[2026-02-18 18:59:55] INFO long_text_processor.py:112: LongTextProcessor initialized
[2026-02-18 18:59:55] INFO tts_runtime.py:277: ✅ TTS runtime initialized in 34257ms
[2026-02-18 18:59:55] INFO tts_runtime.py:278:    Model version: LLM
[2026-02-18 18:59:55] INFO tts_runtime.py:289: Speaker globals cache startup precompute skipped (disabled)


Baseline run is now in the benchmark phase. I’ll use this result as the final reference point for deciding whether to run one-worker or scaled multi-worker in production.


Output:
Chunk ID: a056b3
Wall time: 5.0010 seconds
Process running with session ID 21146
Original token count: 0
Output:


Output:
Chunk ID: 328d80
Wall time: 1.6690 seconds
Process exited with code 0
Original token count: 1244
Output:
Total output lines: 87

✅ vLLM engine initialized successfully!

🎉 Spark TTS Model ready for inference!

🎵 Loading BiCodec audio tokenizer from /home/ubuntu/veenaModal/models/spark_tts_4speaker...
   Device: cuda
Missing tensor: mel_transformer.spectrogram.window
Missing tensor: mel_transformer.mel_scale.fb
✅ BiCodec decoder initialized (sample rate: 16000Hz)
   ℹ  Micro-batching enabled (workers=1, scale_mode=sticky, scale_pending=0, max_batch=16, timeout_ms=6.0)
🚀 SparkTTSPipeline initialized
🌊 Veena3SlidingWindowPipeline initialized (sliding window: 28 tokens)

=== streaming concurrency=96 requests=96 ===
success=96/96 (100%)
wall=4.66s throughput=20.59 req/s
latency_ms avg=3838 p50=4034 p95=4661 max=4661
ttfb_ms avg=1139 p50=1111 p95=1313 max=1314
chunks_per_req avg=5.2 p50=5.0 p95=6.0 max=6.0
inter_chunk_ms avg=640.8 p50=479.4 p95=1874.8 max=2233.7
tail_emit_speed_x avg=1.52 p50=1.34 p95=2.42 max=2.62
timeline_to_first_batch_ms: avg=53.76 p50=53.60 p95=61.60 max=63.37
timeline_to_first_chunk_emitted_ms: avg=1118.54 p50=1093.73 p95=1278.79 max=1279.86
timeline_total_ms: avg=3788.72 p50=3997.15 p95=4627.77 max=4631.08
llm_time_in_queue_ms: avg=0.08 p50=0.07 p95=0.16 max=0.18
llm_scheduler_ms: avg=37.72 p50=38.23 p95=44.65 max=44.65
llm_first_token_ms: avg=51.17 p50=51.01 p95=59.16 max=61.06
llm_request_lifecycle_ms: avg=2442.39 p50=2469.11 p95=2621.71 max=2689.34
llm_generation_wall_ms: avg=3770.77 p50=3963.00 p95=4627.73 max=4631.03
llm_time_per_token_ms: avg=16.45 p50=17.15 p95=19.70 max=20.05
llm_batch_wall_ms_total: avg=3278.54 p50=3275.30 p95=4249.12 max=4446.74
llm_decode_wall_ms_total: avg=3224.81 p50=3224.37 p95=4202.61 max=4385.51
tokens_per_batch_p50: avg=1.00 p50=1.00 p95=1.00 max=1.00
bicodec_decode_calls: avg=4.17 p50=4.00 p95=5.00 max=5.00
bicodec_decode_wall_ms_total: avg=2550.57 p50=2564.80 p95=3849.21 max=3850.67
bicodec_decode_interval_ms_p50: avg=731.29 p50=557.43 p95=1404.39 max=1462.32
decode_interval_applied_tokens_p50: avg=48.00 p50=48.00 p95=48.00 max=48.00
decode_pending_requests_p50: avg=0.00 p50=0.00 p95=0.00 max=0.00
batch_avg_size: avg=8.95 p50=7.95 p95=12.67 max=13.29
batch_max_seen: avg=16.00 p50=16.00 p95=16.00 max=16.00
batch_workers_live: avg=1.00 p50=1.00 p95=1.00 max=1.00
batch_workers_target: avg=1.00 p50=1.00 p95=1.00 max=1.00
batch_queue_wait_ms_avg: avg=483.56 p50=511.29 p95=628.50 max=628.50
batch_compute_ms_avg: avg=…44 tokens truncated…line_share: llm=99.5% bicodec=67.3% other=0.0%
gpu: util_avg=67% util_max=96% mem_avg=46839MB mem_max=47715MB

=== streaming concurrency=128 requests=128 ===
success=128/128 (100%)
wall=5.62s throughput=22.79 req/s
latency_ms avg=4631 p50=4740 p95=5613 max=5614
ttfb_ms avg=1587 p50=1549 p95=2011 max=2011
chunks_per_req avg=4.7 p50=5.0 p95=5.0 max=5.0
inter_chunk_ms avg=816.5 p50=913.2 p95=2262.3 max=2427.1
tail_emit_speed_x avg=1.36 p50=1.23 p95=2.43 max=2.90
timeline_to_first_batch_ms: avg=64.87 p50=65.88 p95=76.19 max=78.29
timeline_to_first_chunk_emitted_ms: avg=1559.77 p50=1524.23 p95=1963.18 max=1965.56
timeline_total_ms: avg=4583.51 p50=4711.58 p95=5597.27 max=5606.71
llm_time_in_queue_ms: avg=0.16 p50=0.15 p95=0.34 max=0.38
llm_scheduler_ms: avg=44.34 p50=39.51 p95=57.94 max=57.94
llm_first_token_ms: avg=61.63 p50=63.09 p95=73.97 max=75.44
llm_request_lifecycle_ms: avg=2785.53 p50=2826.28 p95=3045.65 max=3206.25
llm_generation_wall_ms: avg=4571.00 p50=4678.44 p95=5597.22 max=5606.65
llm_time_per_token_ms: avg=19.95 p50=20.38 p95=24.46 max=25.81
llm_batch_wall_ms_total: avg=3976.47 p50=3880.70 p95=5207.38 max=5429.47
llm_decode_wall_ms_total: avg=3911.64 p50=3813.48 p95=5147.65 max=5381.23
tokens_per_batch_p50: avg=1.00 p50=1.00 p95=1.00 max=1.00
bicodec_decode_calls: avg=3.70 p50=4.00 p95=4.00 max=4.00
bicodec_decode_wall_ms_total: avg=3483.74 p50=3698.61 p95=4435.24 max=4572.78
bicodec_decode_interval_ms_p50: avg=1000.76 p50=1026.83 p95=1417.45 max=1567.69
decode_interval_applied_tokens_p50: avg=48.00 p50=48.00 p95=48.00 max=48.00
decode_pending_requests_p50: avg=0.00 p50=0.00 p95=0.00 max=0.00
batch_avg_size: avg=7.23 p50=6.92 p95=8.73 max=8.74
batch_max_seen: avg=16.00 p50=16.00 p95=16.00 max=16.00
batch_workers_live: avg=1.00 p50=1.00 p95=1.00 max=1.00
batch_workers_target: avg=1.00 p50=1.00 p95=1.00 max=1.00
batch_queue_wait_ms_avg: avg=741.54 p50=733.84 p95=856.38 max=856.38
batch_compute_ms_avg: avg=64.32 p50=61.54 p95=79.64 max=79.85
batch_queue_depth_avg: avg=68.20 p50=68.14 p95=71.13 max=71.16
latency_over_timeline_ms: avg=47.51 p50=36.13 p95=115.66 max=185.97
timeline_share: llm=99.7% bicodec=76.0% other=0.0%
gpu: util_avg=74% util_max=97% mem_avg=47715MB mem_max=47715MB

results saved: /tmp/stream_workers1_current_seeded.json
ERROR 02-18 19:00:09 [core_client.py:605] Engine core proc EngineCore_DP0 died unexpectedly, shutting down client.


**Preparing to compare states**


python3 - <<'PY'
import json
paths={
 'w1_cur':'/tmp/stream_workers1_current_seeded.json',
 'w2s64_sticky':'/tmp/stream_workers2_scale64_sticky_seeded.json',
 'w2s56_g':'/tmp/stream_workers2_scale56_gated_seeded.json',
 'w2s64_g':'/tmp/stream_workers2_scale64_gated_seeded.json',
}
res={k:{r['concurrency']:r for r in json.load(open(p))['levels']} for k,p in paths.items()}
for c in [96,128]:
    print('\n=== c',c,'===')
    for k in paths:
        r=res[k][c];t=r['timing']
        print(f"{k:12s} thr={r['throughput_rps']:.2f} lat={r['latency_ms']['avg']:.0f} ttfb={r['ttfb_ms']['avg']:.0f} qwait={t['batch_queue_wait_ms_avg']['avg']:.0f} w_target={t.get('batch_workers_target',{}).get('avg',0):.2f}")
PY


Output:
Chunk ID: b6d4ef
Wall time: 0.0514 seconds
Process exited with code 0
Original token count: 140
Output:

=== c 96 ===
w1_cur       thr=20.59 lat=3838 ttfb=1139 qwait=484 w_target=1.00
w2s64_sticky thr=21.62 lat=3775 ttfb=1103 qwait=437 w_target=2.00
w2s56_g      thr=22.85 lat=3698 ttfb=1167 qwait=441 w_target=1.46
w2s64_g      thr=21.80 lat=3780 ttfb=1118 qwait=395 w_target=1.34

=== c 128 ===
w1_cur       thr=22.79 lat=4631 ttfb=1587 qwait=742 w_target=1.00
w2s64_sticky thr=23.77 lat=4585 ttfb=1412 qwait=597 w_target=2.00
w2s56_g      thr=23.96 lat=4552 ttfb=1361 qwait=539 w_target=1.62
w2s64_g      thr=22.99 lat=4599 ttfb=1605 qwait=616 w_target=1.52


**Validating dynamic gating consistency**


Given the refreshed baseline, the best result so far is the gated profile around `scale_pending=56`. I’m doing one final confirmation run on current code with explicit `--bicodec-batch-scale-mode dynamic --bicodec-batch-scale-pending 56`.


python3 scripts/stress_test_streaming_runtime.py --levels 96,128 --text short --max-tokens 1024 --seed-base 2026 --timeout 240 --requests-multiplier 1 --min-requests 8 --gpu-memory 0.25 --stream-output-kind delta --stream-decode-interval 48 --stream-window-size 128 --bicodec-batch-max 16 --bicodec-batch-timeout-ms 6 --bicodec-batch-workers 2 --bicodec-batch-scale-pending 56 --bicodec-batch-scale-mode dynamic --output /tmp/stream_workers2_scale56_dynamic_final_seeded.json


Output:
Chunk ID: ad9978
Wall time: 1.0013 seconds
Process running with session ID 97160
Original token count: 0
Output:


Summary: auto


**Planning rapid update polling**