**Summarizing benchmarking outcomes and proposing optimizations**


I now have the full baseline (32/64/96) plus tuned A/B runs and the detailed per-request timelines. I’m summarizing the bottleneck with concrete numbers next, including whether vLLM tuning actually beats current behavior.


Output:
Chunk ID: 9002ce
Wall time: 0.0512 seconds
Process exited with code 0
Original token count: 199
Output:
baseline_c64
{
  "throughput_rps": 17.311740257459824,
  "lat_avg": 3690.2919153799303,
  "ttfb_avg": 1532.1709263662342,
  "llm_tok_ms": 16.17468517616502,
  "queue_ms": 0.0617728364886716,
  "sched_ms": 29.37699915491976,
  "bicodec_ms": 3057.9734236671356
}
vllm_tuned_c64
{
  "throughput_rps": 15.766053315314457,
  "lat_avg": 4053.0143312062137,
  "ttfb_avg": 1653.1602599279722,
  "llm_tok_ms": 17.759664347839315,
  "queue_ms": 0.07771598757244647,
  "sched_ms": 35.803992839646526,
  "bicodec_ms": 3389.3982312147273
}
decode_tuned_c64
{
  "throughput_rps": 17.248739056683277,
  "lat_avg": 3706.5788207401056,
  "ttfb_avg": 1501.9540293869795,
  "llm_tok_ms": 15.908238983844084,
  "queue_ms": 0.07888044638093561,
  "sched_ms": 28.757382693584077,
  "bicodec_ms": 2898.823211507988
}


# Collaboration Mode: Default

You are now in Default mode. Any previous instructions for other modes (e.g. Plan mode) are no longer active.

Your active mode changes only when new developer instructions with a different `<collaboration_mode>...</collaboration_mode>` change it; user requests or tool descriptions do not change mode by themselves. Known mode names are Default and Plan.

## request_user_input availability

The `request_user_input` tool is unavailable in Default mode. If you call it while in Default mode, it will return an error.

If a decision is necessary and cannot be discovered from local context, ask the user directly. However, in Default mode you should strongly prefer executing the user's request rather than stopping to ask questions.

Summary: auto

## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) When `SKILL.md` references relative paths (e.g., `scripts/foo.py`), resolve them relative to the skill directory listed above first, and only consider other paths if needed.
  3) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  4) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  5) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.


**Noting baseline reference levels**


Output:
Chunk ID: 582f37
Wall time: 0.0502 seconds
Process exited with code 0
Original token count: 222
Output:
{
  "c": 32,
  "rps": 12.557102792804068,
  "lat_avg": 2546.459506091196,
  "ttfb_avg": 1046.6992497385945,
  "llm_tok_ms": 10.622712277418243,
  "queue_ms": 0.07949935388751328,
  "sched_ms": 28.118634596467018,
  "bicodec_ms": 1943.453844927717,
  "decode_interval_p50_ms": 383.45315831247717
}
{
  "c": 64,
  "rps": 16.771739232775897,
  "lat_avg": 3811.935635196278,
  "ttfb_avg": 1377.9939615633339,
  "llm_tok_ms": 16.554943799508603,
  "queue_ms": 0.08436896314378828,
  "sched_ms": 32.890047776163556,
  "bicodec_ms": 3147.9037028038874,
  "decode_interval_p50_ms": 920.7632235629717
}
{
  "c": 96,
  "rps": 18.921961611771835,
  "lat_avg": 5069.451287776853,
  "ttfb_avg": 1979.0018753071006,
  "llm_tok_ms": 22.094432296068348,
  "queue_ms": 0.1253390801139176,
  "sched_ms": 35.79173667822033,
  "bicodec_ms": 4293.123280726529,
  "decode_interval_p50_ms": 1824.746812383334
}


Summary: auto


**Summarizing benchmark findings and next steps**


Detailed streaming instrumentation is now in and benchmarked.

**What I changed**
- Added per-request streaming timeline + vLLM + decode metrics in `veena3modal/core/streaming_pipeline.py:312`.
- Added stream tuning env knobs (`decode_interval`, `window_size`, etc.) in `veena3modal/core/streaming_pipeline.py:54`.
- Forwarded metrics-only stream events through runtime without changing audio payload behavior in `veena3modal/services/tts_runtime.py:786`.
- Extended the streaming stress harness with:
  - vLLM scheduler/init knobs
  - per-metric aggregation
  - concurrency bottleneck reporting
  in `scripts/stress_test_streaming_runtime.py:218` and `scripts/stress_test_streaming_runtime.py:399`.

**Answer to your vLLM question (with data)**
For your current stack, vLLM tuning alone did **not** beat the current baseline.

- Baseline (`c64`, delta, gpu_mem=0.25): `17.31 req/s`, `3690 ms` latency avg, `1532 ms` TTFB avg.
- vLLM-tuned (`gpu_mem=0.4`, `max_num_batched_tokens=8192`, `max_num_seqs=128`): `15.77 req/s`, `4053 ms`, `1653 ms` (worse).
- Decode-tuned (`decode_interval=64`, `window=96`): `17.25 req/s`, `3707 ms`, `1502 ms` (TTFB slightly better, throughput flat).

So yes, we can do better, but the bottleneck is not queue/scheduler in vLLM.

**Bottleneck at higher concurrency (baseline 32/64/96)**
- Queue time stays tiny: `~0.08 -> 0.13 ms`.
- Scheduler time stays modest: `~28 -> 36 ms`.
- LLM ms/token degrades hard: `10.62 -> 16.55 -> 22.09 ms`.
- BiCodec decode wall (per request) grows hard: `1943 -> 3148 -> 4293 ms`.
- Decode cadence collapses: interval p50 `383 -> 921 -> 1825 ms`.
- Throughput rises (`12.56 -> 16.77 -> 18.92 req/s`) but latency/TTFB blow up (`2.55s -> 3.81s -> 5.07s`, `1.05s -> 1.38s -> 1.98s`).

**Conclusion**
vLLM is not the primary limiter now. The biggest win path is decode-side scheduling/batching across concurrent streams (BiCodec stage), not more vLLM knob-tweaking.

**Result files**
- `/tmp/stream_vllm_baseline_detailed.json`
- `/tmp/stream_vllm_tuned_vllm_only.json`
- `/tmp/stream_decode_tuned.json`
- `/tmp/stream_baseline_levels_detailed.json`

1. If you want, I’ll implement cross-request BiCodec micro-batching next (shared decode queue) and rerun the same c32/c64/c96 suite.


<permissions instructions>
Filesystem sandboxing defines which files can be read or written. `sandbox_mode` is `danger-full-access`: No filesystem sandboxing - all commands are permitted. Network access is enabled.
Approval policy is currently never. Do not provide the `sandbox_permissions` for any reason, commands will be rejected.
</permissions instructions>


<collaboration_mode># Collaboration Mode: Default

You are now in Default mode. Any previous instructions for other modes (e.g. Plan mode) are no longer active.

Your active mode changes only when new developer instructions with a different `<collaboration_mode>...</collaboration_mode>` change it; user requests or tool descriptions do not change mode by themselves. Known mode names are Default and Plan.

## request_user_input availability

The `request_user_input` tool is unavailable in Default mode. If you call it while in Default mode, it will return an error.

If a decision is necessary and cannot be discovered from local context, ask the user directly. However, in Default mode you should strongly prefer executing the user's request rather than stopping to ask questions.
</collaboration_mode>


# AGENTS.md instructions for /home/ubuntu/veenaModal

<INSTRUCTIONS>
## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) When `SKILL.md` references relative paths (e.g., `scripts/foo.py`), resolve them relative to the skill directory listed above first, and only consider other paths if needed.
  3) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  4) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  5) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.
</INSTRUCTIONS>


<environment_context>
  <cwd>/home/ubuntu/veenaModal</cwd>
  <shell>bash</shell>
</environment_context>


what does these big AI labs do to increase or maximize the throughput of the LLM token generation throughout their inference stack. and what exactly are we missing here to make this productionizable with max saturation of a GPU with it's memory and performance or utilisation.


**Summarizing scaling best practices**


Big labs don’t optimize “the model” in isolation. They optimize the full inference system as a queuing + scheduling + kernel problem.

What they do to maximize LLM token throughput:
1. Continuous batching at token-step granularity (not request-level batching only).
2. Prefill/decode-aware scheduling with admission control and SLO tiers.
3. Aggressive KV cache optimization (paged KV, prefix sharing, eviction policy).
4. Kernel-level optimizations (fused ops, CUDA graphs, custom attention kernels, quantization where safe).
5. Pipeline disaggregation (separate prefill/decode or separate model/vocoder stages).
6. Micro-batching downstream stages (vocoder/decoder/post-processing), not just LLM.
7. Tight observability loops: per-step timings, queue delay, tokens/s, GPU busy, memory BW, tail latency.

What your current data says you’re missing:
1. vLLM queue/scheduler is not the main bottleneck.
`llm_time_in_queue_ms` is ~0.08–0.13 ms and `llm_scheduler_ms` ~28–36 ms.
2. Throughput rises with concurrency, but latency/TTFB blow up.
At c32/c64/c96: ~12.56 / 16.77 / 18.92 req/s, while latency goes ~2.55s / 3.81s / 5.07s.
3. Decoder side is the choke point under load.
`bicodec_decode_wall_ms_total` grows ~1943 -> 3148 -> 4293 ms.
4. Decode cadence collapses with concurrency.
`bicodec_decode_interval_ms_p50` grows ~383 -> 921 -> 1825 ms.
5. Effective LLM decode batching is weak in practice.
`tokens_per_batch_p50` is ~1.0, so you’re mostly stepping per token/per request, not harvesting large batch gains.
6. GPU is not fully saturated in a productive way (util mostly ~60s), meaning pipeline stalls/serialization are leaving performance on the table.

Productionization gaps (most important first):
1. Shared cross-request BiCodec micro-batching in the active streaming path.
You have batch-decoder code (`veena3modal/core/bicodec_batch_decoder.py`) but main streaming path still effectively decodes per request.
2. Stronger separation of stages (LLM token generation vs decode) with bounded queues and backpressure.
3. Load-adaptive decode policy (interval/window/chunk sizing changes with inflight load).
4. Hard SLO controls: max inflight, drop/defer policy, timeout budgets by tier.
5. Reliability hardening (your engine core process exiting after runs must be fixed before prod confidence).

Net: you’re already instrumented well now (`veena3modal/core/streaming_pipeline.py:312`, `scripts/stress_test_streaming_runtime.py:399`). The next big win is decode-side batching/scheduling, not more vLLM knob tuning.


Big labs don’t optimize “the model” in isolation. They optimize the full inference system as a queuing + scheduling + kernel problem.

What they do to maximize LLM token throughput:

Continuous batching at token-step granularity (not request-level batching only).
Prefill/decode-aware scheduling with admission control and SLO tiers.
Aggressive KV cache optimization (paged KV, prefix sharing, eviction policy).
Kernel-level optimizations (fused ops, CUDA graphs, custom attention kernels, quantization where safe).
Pipeline disaggregation (separate prefill/decode or separate model/vocoder stages).
Micro-batching downstream stages (vocoder/decoder/post-processing), not just LLM.
Tight observability loops: per-step timings, queue delay, tokens/s, GPU busy, memory BW, tail latency.
What your current data says you’re missing:

vLLM queue/scheduler is not the main bottleneck.
llm_time_in_queue_ms is ~0.08–0.13 ms and llm_scheduler_ms ~28–36 ms.
Throughput rises with concurrency, but latency/TTFB blow up.
At c32/c64/c96: ~12.56 / 16.77 / 18.92 req/s, while latency goes ~2.55s / 3.81s / 5.07s.
Decoder side is the choke point under load.
bicodec_decode_wall_ms_total grows ~1943 -> 3148 -> 4293 ms.
Decode cadence collapses with concurrency.
bicodec_decode_interval_ms_p50 grows ~383 -> 921 -> 1825 ms.
Effective LLM decode batching is weak in practice.
tokens_per_batch_p50 is ~1.0, so you’re mostly stepping per token/per request, not harvesting large batch gains.
GPU is not fully saturated in a productive way (util mostly ~60s), meaning pipeline stalls/serialization are leaving performance on the table.
Productionization gaps (most important first):

Shared cross-request BiCodec micro-batching in the active streaming path.
You have batch-decoder code (bicodec_batch_decoder.py) but main streaming path still effectively decodes per request.
Stronger separation of stages (LLM token generation vs decode) with bounded queues and backpressure.
Load-adaptive decode policy (interval/window/chunk sizing changes with inflight load).
Hard SLO controls: max inflight, drop/defer policy, timeout budgets by tier.
Reliability hardening (your engine core process exiting after runs must be fixed before prod confidence).
Net: you’re already instrumented well now (streaming_pipeline.py (line 312), stress_test_streaming_runtime.py (line 399)). The next big win is decode-side batching/scheduling, not more vLLM knob tuning.

Can we keep optimising. like go towards how these big AI labs optimize their inference stack. what is the max throughput efficiency or max concurrency attainable with what settings or what optimisations or what kind of right batching.


**Planning micro-batching implementation**


You’re right. I’ll push further toward a lab-style inference stack by implementing real cross-request BiCodec micro-batching in the active streaming path, then run new concurrency sweeps to quantify the new max throughput/latency knee and recommended settings.


cd /home/ubuntu/veenaModal && nl -ba veena3modal/core/bicodec_decoder.py | sed -n '1,320p'


cd /home/ubuntu/veenaModal && nl -ba veena3modal/services/tts_runtime.py | sed -n '150,260p'


cd /home/ubuntu/veenaModal && rg -n "enable_batching|decode_single_async|bicodec_decoder" veena3modal/core/streaming_pipeline.py veena3modal/services/tts_runtime.py -n


Output:
Chunk ID: 8ca473
Wall time: 0.0509 seconds
Process exited with code 0
Original token count: 3598
Output:
     1	"""
     2	Spark TTS BiCodec Decoder
     3
     4	Decodes BiCodec audio tokens (semantic + global) to audio waveforms.
     5	Replaces SNAC decoder for Spark TTS model.
     6	"""
     7
     8	import torch
     9	import numpy as np
    10	import sys
    11	import asyncio
    12	from pathlib import Path
    13	from typing import List, Optional, Tuple
    14
    15	# Ensure `sparktts` (vendored under external/) is importable in local/dev runs.
    16	# In Modal, PYTHONPATH already includes `/root/external/sparktts`.
    17	try:
    18	    from sparktts.models.audio_tokenizer import BiCodecTokenizer
    19	except ImportError:  # pragma: no cover - environment-dependent
    20	    _this_file = Path(__file__).resolve()
    21	    _sparktts_path = None
    22	    for _p in [_this_file.parent] + list(_this_file.parents):
    23	        _candidate = _p / "external" / "sparktts"
    24	        if _candidate.is_dir():
    25	            _sparktts_path = str(_candidate)
    26	            break
    27	    if _sparktts_path and _sparktts_path not in sys.path:
    28	        sys.path.insert(0, _sparktts_path)
    29	    from sparktts.models.audio_tokenizer import BiCodecTokenizer
    30
    31	from veena3modal.core.constants import (
    32	    BICODEC_TOKENIZER_PATH,
    33	    AUDIO_SAMPLE_RATE,
    34	)
    35
    36
    37	class BiCodecDecoder:
    38	    """
    39	    BiCodec Decoder for Spark TTS.
    40
    41	    Decodes semantic and global tokens to audio waveforms using BiCodec.
    42	    This replaces the SNAC decoder used in the old Indic Orpheus model.
    43
    44	    Supports streaming with sliding window approach (like SNAC).
    45	    """
    46
    47	    def __init__(
    48	        self,
    49	        device: str = "cuda",
    50	        model_path: str = BICODEC_TOKENIZER_PATH,
    51	        enable_batching: bool = False,  # For compatibility with streaming pipeline
    52	    ):
    53	        """
    54	        Initialize BiCodec decoder.
    55
    56	        Args:
    57	            device: Device for BiCodec model (cuda/cpu)
    58	            model_path: Path to BiCodec model checkpoint
    59	            enable_batching: Enable async batching for streaming (not used yet)
    60	        """
    61	        self.device = torch.device(device if torch.cuda.is_available() else "cpu")
    62	        self.model_path = model_path
    63	        self.enable_batching = enable_batching  # For streaming pipeline compatibility
    64
    65	        print(f"🎵 Loading BiCodec audio tokenizer from {model_path}...")
    66	        print(f"   Device: {self.device}")
    67
    68	        # Initialize BiCodec tokenizer
    69	        self.audio_tokenizer = BiCodecTokenizer(model_path, device=str(self.device))
    70
    71	        # OPTIMIZATION: Ensure model is on device once at init, not per-decode call
    72	        self.audio_tokenizer.device = self.device
    73	        self.audio_tokenizer.model.to(self.device)
    74
    75	        # NOTE: torch.compile disabled for BiCodec -- dynamic semantic sequence lengths
    76	        # cause repeated recompilation (~120s each), worse than eager mode.
    77	        # torch.compile works best with static shapes. If needed, pre-warm with
    78	        # representative lengths or use mode="default" instead of "reduce-overhead".
    79	        # The multi-engine approach (Tier 3) provides more impactful gains.
    80
    81	        print(f"✅ BiCodec decoder initialized (sample rate: {AUDIO_SAMPLE_RATE}Hz)")
    82	        if enable_batching:
    83	            print(f"   ℹ  Batching enabled (for streaming support)")
    84
    85	    def decode(
    86	        self,
    87	        semantic_ids: List[int],
    88	        global_ids: List[int],
    89	    ) -> Optional[np.ndarray]:
    90	        """
    91	        Decode BiCodec tokens to audio waveform.
    92
    93	        CRITICAL: BiCodec expects EXACTLY 32 global tokens!
    94	        The decoder pools them via speaker_encoder → d_vector, then broadcasts
    95	        via d_vector.unsqueeze(-1) across the time dimension.
    96
    97	        Args:
    98	            semantic_ids: List of semantic token IDs (variable length)
    99	            global_ids: List of global token IDs (MUST be exactly 32!)
   100
   101	        Returns:
   102	            Audio waveform as numpy array (float32, 16kHz mono)
   103	            Shape: (samples,)
   104	            Returns None if decode fails
   105	        """
   106	        if not semantic_ids or not global_ids:
   107	            print(f"⚠  Empty token lists: semantic={len(semantic_ids)}, global={len(global_ids)}")
   108	            return None
   109
   110	        # Validate global token count
   111	        EXPECTED_GLOBAL_TOKENS = 32
   112	        if len(global_ids) != EXPECTED_GLOBAL_TOKENS:
   113	            print(f"❌ BiCodec requires EXACTLY {EXPECTED_GLOBAL_TOKENS} global tokens, got {len(global_ids)}")
   114	            # Try to fix it
   115	            if len(global_ids) > EXPECTED_GLOBAL_TOKENS:
   116	                global_ids = global_ids[:EXPECTED_GLOBAL_TOKENS]
   117	                print(f"   └─ Truncated to {EXPECTED_GLOBAL_TOKENS}")
   118	            else:
   119	                global_ids = global_ids + [0] * (EXPECTED_GLOBAL_TOKENS - len(global_ids))
   120	                print(f"   └─ Padded to {EXPECTED_GLOBAL_TOKENS}")
   121
   122	        try:
   123	            # Convert to tensors following official Spark-TTS implementation
   124	            # global_token_ids shape: (1, 32) - ALWAYS 32!
   125	            # pred_semantic_ids shape: (1, N) - variable length
   126	            pred_semantic = torch.tensor(semantic_ids).long().unsqueeze(0).to(self.device)
   127	            pred_global = torch.tensor(global_ids).long().unsqueeze(0).to(self.device)
   128
   129	            # Device migration done once in __init__, not per-call
   130	            # Decode through BiCodec
   131	            # BiCodecTokenizer.detokenize() expects:
   132	            #   - global_tokens: (batch, global_dim) -> will unsqueeze(1) internally
   133	            #   - semantic_tokens: (batch, latent_dim)
   134	            with torch.inference_mode():
   135	                wav_np = self.audio_tokenizer.detokenize(
   136	                    pred_global,  # (1, seq) - correct shape!
   137	                    pred_semantic  # (1, seq)
   138	                )
   139
   140	            return wav_np
   141
   142	        except Exception as e:
   143	            print(f"❌ BiCodec decode error: {e}")
   144	            import traceback
   145	            traceback.print_exc()
   146	            return None
   147
   148	    def decode_to_bytes(
   149	        self,
   150	        semantic_ids: List[int],
   151	        global_ids: List[int],
   152	    ) -> Optional[bytes]:
   153	        """
   154	        Decode BiCodec tokens to audio bytes (int16 PCM).
   155
   156	        Args:
   157	            semantic_ids: List of semantic token IDs
   158	            global_ids: List of global token IDs
   159
   160	        Returns:
   161	            Audio as bytes (int16 PCM, 16kHz mono)
   162	            Returns None if decode fails
   163	        """
   164	        audio = self.decode(semantic_ids, global_ids)
   165
   166	        if audio is None:
   167	            return None
   168
   169	        # Clip and convert float32 to int16 PCM
   170	        audio = np.clip(audio, -1.0, 1.0)
   171	        audio_int16 = (audio * 32767).astype(np.int16)
   172
   173	        return audio_int16.tobytes()
   174
   175	    async def decode_to_bytes_async(
   176	        self,
   177	        semantic_ids: List[int],
   178	        global_ids: List[int],
   179	    ) -> Optional[bytes]:
   180	        """
   181	        Async wrapper for decode_to_bytes.
   182
   183	        Runs decode in the default executor so the event loop stays responsive
   184	        under high request concurrency.
   185	        """
   186	        loop = asyncio.get_running_loop()
   187	        return await loop.run_in_executor(
   188	            None,
   189	            self.decode_to_bytes,
   190	            semantic_ids,
   191	            global_ids,
   192	        )
   193
   194	    def validate_tokens(self, semantic_ids: List[int], global_ids: List[int]) -> bool:
   195	        """
   196	        Validate BiCodec tokens before decoding.
   197
   198	        Args:
   199	            semantic_ids: List of semantic token IDs
   200	            global_ids: List of global token IDs
   201
   202	        Returns:
   203	            True if valid, False otherwise
   204	        """
   205	        # Check minimum length
   206	        if not semantic_ids:
   207	            print(f"❌ No semantic tokens")
   208	            return False
   209
   210	        if not global_ids:
   211	            print(f"❌ No global tokens")
   212	            return False
   213
   214	        # Basic sanity checks
   215	        if len(semantic_ids) < 1:
   216	            print(f"❌ Too few semantic tokens: {len(semantic_ids)}")
   217	            return False
   218
   219	        if len(global_ids) < 1:
   220	            print(f"❌ Too few global tokens: {len(global_ids)}")
   221	            return False
   222
   223	        return True
   224
   225	    def get_audio_duration(self, semantic_ids: List[int]) -> float:
   226	        """
   227	        Estimate audio duration from semantic tokens.
   228
   229	        Args:
   230	            semantic_ids: List of semantic token IDs
   231
   232	        Returns:
   233	            Estimated duration in seconds
   234	        """
   235	        # BiCodec generates audio based on semantic tokens
   236	        # Approximate duration calculation (may need tuning)
   237	        # This is an estimate and actual duration depends on model output
   238	        estimated_samples = len(semantic_ids) * 320  # Rough estimate
   239	        return estimated_samples / AUDIO_SAMPLE_RATE
   240
   241	    def decode_streaming(
   242	        self,
   243	        semantic_ids: List[int],
   244	        global_ids: List[int],
   245	        use_sliding_window: bool = False,
   246	        trim_warmup: bool = False,
   247	    ) -> Optional[bytes]:
   248	        """
   249	        Decode BiCodec tokens with streaming support.
   250
   251	        CRITICAL: BiCodec decoder expects EXACTLY 32 global tokens always!
   252	        The decoder internally broadcasts d_vector via d_vector.unsqueeze(-1).
   253
   254	        From sparktts/models/bicodec.py line 184-186:
   255	            d_vector = self.speaker_encoder.detokenize(global_tokens)  # Expects 32 tokens
   256	            x = self.prenet(z_q, d_vector)
   257	            x = x + d_vector.unsqueeze(-1)  # Broadcasts across time automatically!
   258
   259	        Args:
   260	            semantic_ids: List of semantic token IDs (variable length, 50 TPS)
   261	            global_ids: List of global token IDs (MUST be exactly 32 tokens!)
   262	            use_sliding_window: If True, return only middle samples
   263	            trim_warmup: Legacy parameter (not used for BiCodec)
   264
   265	        Returns:
   266	            Audio bytes (int16 PCM) or None if decode fails
   267	        """
   268	        # Minimum semantic tokens needed for stable decode
   269	        MIN_SEMANTIC_TOKENS = 8
   270
   271	        if len(semantic_ids) < MIN_SEMANTIC_TOKENS:
   272	            print(f"⚠  Too few semantic tokens ({len(semantic_ids)}) for decode, need >= {MIN_SEMANTIC_TOKENS}")
   273	            return None
   274
   275	        # CRITICAL: Ensure we have exactly 32 global tokens
   276	        EXPECTED_GLOBAL_TOKENS = 32
   277	        if len(global_ids) != EXPECTED_GLOBAL_TOKENS:
   278	            print(f"⚠  WARNING: Got {len(global_ids)} global tokens, expected {EXPECTED_GLOBAL_TOKENS}")
   279	            if len(global_ids) > EXPECTED_GLOBAL_TOKENS:
   280	                # Truncate to first 32
   281	                global_ids = global_ids[:EXPECTED_GLOBAL_TOKENS]
   282	                print(f"   └─ Truncated to {EXPECTED_GLOBAL_TOKENS} tokens")
   283	            else:
   284	                # Pad with zeros if we somehow have fewer
   285	                global_ids = global_ids + [0] * (EXPECTED_GLOBAL_TOKENS - len(global_ids))
   286	                print(f"   └─ Padded to {EXPECTED_GLOBAL_TOKENS} tokens")
   287
   288	        # Decode with EXACTLY 32 global tokens + variable semantic tokens
   289	        # The decoder handles broadcasting internally!
   290	        audio = self.decode(semantic_ids, global_ids)
   291
   292	        if audio is None:
   293	            return None
   294
   295	        # If using sliding window, return only the middle portion
   296	        if use_sliding_window and len(audio) > 4096:
   297	            # Keep middle samples for overlap-add streaming
   298	            # BiCodec produces ~320 samples per semantic token (16kHz / 50 TPS)
   299	            total_samples = len(audio)
   300	            keep_samples = min(4096, total_samples // 2)
   301	            start = (total_samples - keep_samples) // 2
   302	            end = start + keep_samples
   303	            audio = audio[start:end]
   304
   305	        # Convert to bytes
   306	        audio = np.clip(audio, -1.0, 1.0)
   307	        audio_int16 = (audio * 32767).astype(np.int16)
   308
   309	        return audio_int16.tobytes()
   310
   311	    async def decode_single_async(
   312	        self,
   313	        semantic_ids: List[int],
   314	        global_ids: List[int],
   315	        trim_warmup: bool = False,
   316	        use_sliding_window: bool = False,
   317	    ) -> Optional[bytes]:
   318	        """
   319	        Async wrapper for streaming decode - runs in executor to avoid blocking event loop.
   320


Output:
Chunk ID: 24e7e7
Wall time: 0.0515 seconds
Process exited with code 0
Original token count: 1433
Output:
   150	        from veena3modal.processing.prompt_builder import IndicPromptBuilder
   151	        from veena3modal.processing.long_text_processor import LongTextProcessor
   152
   153	        engine_kwargs: Dict[str, Any] = {}
   154	        if isinstance(max_num_batched_tokens, int) and max_num_batched_tokens > 0:
   155	            engine_kwargs["max_num_batched_tokens"] = max_num_batched_tokens
   156	        if isinstance(max_num_seqs, int) and max_num_seqs > 0:
   157	            engine_kwargs["max_num_seqs"] = max_num_seqs
   158	        if enable_chunked_prefill is not None:
   159	            engine_kwargs["enable_chunked_prefill"] = bool(enable_chunked_prefill)
   160	        if enable_prefix_caching is not None:
   161	            engine_kwargs["enable_prefix_caching"] = bool(enable_prefix_caching)
   162	        if disable_log_stats is not None:
   163	            engine_kwargs["disable_log_stats"] = bool(disable_log_stats)
   164	        if enforce_eager is not None:
   165	            engine_kwargs["enforce_eager"] = bool(enforce_eager)
   166
   167	        # Load SparkTTS model with vLLM engine
   168	        if num_engines > 1:
   169	            # TIER 3 OPTIMIZATION: Multiple vLLM engines on same GPU
   170	            # Each engine gets gpu_memory_utilization / num_engines to share GPU fairly.
   171	            # E.g., 3 engines at 0.08 each = 0.24 total, ~8GB per engine, ~24GB total.
   172	            from veena3modal.core.multi_engine import create_multi_engine_model
   173	            per_engine_mem = gpu_memory_utilization / num_engines
   174	            logger.info(f"Loading {num_engines} vLLM engines ({per_engine_mem:.2f} GPU mem each)...")
   175	            model = create_multi_engine_model(
   176	                model_path=model_path,
   177	                num_engines=num_engines,
   178	                hf_token=hf_token,
   179	                gpu_memory_per_engine=per_engine_mem,
   180	                **engine_kwargs,
   181	            )
   182	        else:
   183	            logger.info("Loading SparkTTS model with vLLM...")
   184	            model = SparkTTSModel(
   185	                model_path=model_path,
   186	                hf_token=hf_token,
   187	                gpu_memory_utilization=gpu_memory_utilization,
   188	                **engine_kwargs,
   189	            )
   190
   191	        # Initialize prompt builder
   192	        logger.info("Initializing prompt builder...")
   193	        prompt_builder = IndicPromptBuilder(
   194	            tokenizer=model.tokenizer,
   195	            model=model,
   196	        )
   197
   198	        # Initialize BiCodec decoder
   199	        logger.info("Initializing BiCodec decoder...")
   200	        bicodec_decoder = BiCodecDecoder(
   201	            device=device,
   202	            model_path=bicodec_path,
   203	        )
   204
   205	        # Initialize pipeline
   206	        logger.info("Initializing TTS pipeline...")
   207	        pipeline = SparkTTSPipeline(
   208	            model=model,
   209	            prompt_builder=prompt_builder,
   210	            bicodec_decoder=bicodec_decoder,
   211	        )
   212
   213	        # Initialize streaming pipeline (for M4)
   214	        streaming_pipeline = None
   215	        try:
   216	            from veena3modal.core.streaming_pipeline import Veena3SlidingWindowPipeline
   217	            logger.info("Initializing streaming pipeline...")
   218	            # NOTE: Parameter is named 'snac_decoder' for legacy reasons, but works with BiCodecDecoder
   219	            streaming_pipeline = Veena3SlidingWindowPipeline(
   220	                model=model,
   221	                prompt_builder=prompt_builder,
   222	                snac_decoder=bicodec_decoder,  # BiCodecDecoder is interface-compatible
   223	            )
   224	        except ImportError as e:
   225	            logger.warning(f"Streaming pipeline not available: {e}")
   226
   227	        # Initialize long text processor once (avoid per-request construction/logging)
   228	        long_text_processor = LongTextProcessor(
   229	            pipeline=pipeline,
   230	            streaming_pipeline=streaming_pipeline,
   231	        )
   232
   233	        # Initialize super-resolution (optional)
   234	        sr_service = None
   235	        if enable_sr and sr_checkpoint_dir:
   236	            try:
   237	                from veena3modal.core.super_resolution import SuperResolutionService
   238	                logger.info(f"Initializing super-resolution from {sr_checkpoint_dir}...")
   239	                sr_service = SuperResolutionService(checkpoint_dir=sr_checkpoint_dir)
   240	                # Load the model explicitly
   241	                if sr_service.load_model(device=device):
   242	                    logger.info("✅ Super-resolution model loaded successfully")
   243	                else:
   244	                    logger.warning("Super-resolution model failed to load")
   245	                    sr_service = None
   246	            except Exception as e:
   247	                logger.warning(f"Super-resolution not available: {e}")
   248
   249	        # Determine model version
   250	        model_version = os.path.basename(model_path.rstrip('/'))
   251	        if not model_version:
   252	            model_version = "spark-tts"
   253
   254	        load_time = (time.time() - start_time) * 1000
   255
   256	        # Create runtime
   257	        _runtime = TTSRuntime(
   258	            model=model,
   259	            pipeline=pipeline,
   260	            streaming_pipeline=streaming_pipeline,


Output:
Chunk ID: 2c2a56
Wall time: 0.0514 seconds
Process exited with code 0
Original token count: 346
Output:
veena3modal/services/tts_runtime.py:43:    bicodec_decoder: Any = None
veena3modal/services/tts_runtime.py:149:        from veena3modal.core.bicodec_decoder import BiCodecDecoder
veena3modal/services/tts_runtime.py:200:        bicodec_decoder = BiCodecDecoder(
veena3modal/services/tts_runtime.py:210:            bicodec_decoder=bicodec_decoder,
veena3modal/services/tts_runtime.py:222:                snac_decoder=bicodec_decoder,  # BiCodecDecoder is interface-compatible
veena3modal/services/tts_runtime.py:263:            bicodec_decoder=bicodec_decoder,
veena3modal/core/streaming_pipeline.py:266:                        if self.snac_decoder.enable_batching:
veena3modal/core/streaming_pipeline.py:267:                            audio_bytes = await self.snac_decoder.decode_single_async(
veena3modal/core/streaming_pipeline.py:499:                        audio_bytes = await self.snac_decoder.decode_single_async(
veena3modal/core/streaming_pipeline.py:848:                        if self.snac_decoder.enable_batching:
veena3modal/core/streaming_pipeline.py:849:                            audio_bytes = await self.snac_decoder.decode_single_async(
veena3modal/core/streaming_pipeline.py:1064:                    if self.snac_decoder.enable_batching:
veena3modal/core/streaming_pipeline.py:1065:                        audio_bytes = await self.snac_decoder.decode_single_async(


**Designing internal microbatcher**


cd /home/ubuntu/veenaModal && nl -ba veena3modal/core/bicodec_decoder.py | sed -n '300,430p'


Output:
Chunk ID: c9a099
Wall time: 0.0502 seconds
Process exited with code 0
Original token count: 472
Output:
   300	            keep_samples = min(4096, total_samples // 2)
   301	            start = (total_samples - keep_samples) // 2
   302	            end = start + keep_samples
   303	            audio = audio[start:end]
   304
   305	        # Convert to bytes
   306	        audio = np.clip(audio, -1.0, 1.0)
   307	        audio_int16 = (audio * 32767).astype(np.int16)
   308
   309	        return audio_int16.tobytes()
   310
   311	    async def decode_single_async(
   312	        self,
   313	        semantic_ids: List[int],
   314	        global_ids: List[int],
   315	        trim_warmup: bool = False,
   316	        use_sliding_window: bool = False,
   317	    ) -> Optional[bytes]:
   318	        """
   319	        Async wrapper for streaming decode - runs in executor to avoid blocking event loop.
   320
   321	        OPTIMIZATION: Previously ran synchronously, blocking all concurrent coroutines
   322	        during GPU decode (~20-50ms). Now uses run_in_executor to unblock the event loop
   323	        so other streams can progress while this decode runs on GPU.
   324
   325	        Args:
   326	            semantic_ids: List of semantic token IDs
   327	            global_ids: List of global token IDs
   328	            trim_warmup: Legacy parameter (not used)
   329	            use_sliding_window: Use sliding window mode
   330
   331	        Returns:
   332	            Audio bytes (int16 PCM) or None
   333	        """
   334	        import asyncio
   335	        loop = asyncio.get_event_loop()
   336	        return await loop.run_in_executor(
   337	            None,  # Default ThreadPoolExecutor
   338	            self.decode_streaming,
   339	            semantic_ids,
   340	            global_ids,
   341	            use_sliding_window,
   342	            trim_warmup,
   343	        )
   344


Summary: auto


**Designing async batch decode worker**


I’ve identified the core gap: `decode_single_async` still does per-request executor decode. I’m replacing that with a real async micro-batch worker (shared queue, timeout window, batched BiCodec forward) and then I’ll benchmark with batching on/off to quantify gains.


Input:
*** Begin Patch
*** Update File: /home/ubuntu/veenaModal/veena3modal/core/bicodec_decoder.py
@@
-import torch
-import numpy as np
-import sys
-import asyncio
+import asyncio
+import logging
+import os
+import sys
+import threading
+import time
+from dataclasses import dataclass
 from pathlib import Path
-from typing import List, Optional, Tuple
+from typing import Dict, List, Optional, Tuple
+
+import numpy as np
+import torch
@@
 from veena3modal.core.constants import (
     BICODEC_TOKENIZER_PATH,
     AUDIO_SAMPLE_RATE,
 )

+logger = logging.getLogger(__name__)
+
+
+def _env_flag(name: str, default: bool) -> bool:
+    raw = os.environ.get(name, "")
+    if not raw:
+        return default
+    return raw.strip().lower() in {"1", "true", "yes", "on"}
+
+
+def _env_int(name: str, default: int, minimum: int = 1) -> int:
+    raw = os.environ.get(name, "")
+    if not raw:
+        return default
+    try:
+        value = int(raw)
+    except ValueError:
+        return default
+    return max(minimum, value)
+
+
+def _env_float(name: str, default: float, minimum: float = 0.0) -> float:
+    raw = os.environ.get(name, "")
+    if not raw:
+        return default
+    try:
+        value = float(raw)
+    except ValueError:
+        return default
+    return max(minimum, value)
+
+
+@dataclass
+class _BatchDecodeRequest:
+    semantic_ids: List[int]
+    global_ids: List[int]
+    use_sliding_window: bool
+    trim_warmup: bool
+    future: asyncio.Future
+    submitted_at: float
+

 class BiCodecDecoder:
@@
         self.device = torch.device(device if torch.cuda.is_available() else "cpu")
         self.model_path = model_path
-        self.enable_batching = enable_batching  # For streaming pipeline compatibility
+        requested_batching = bool(enable_batching)
+        if _env_flag("VEENA3_BICODEC_BATCHING_FORCE_ON", False):
+            requested_batching = True
+        if _env_flag("VEENA3_BICODEC_BATCHING_FORCE_OFF", False):
+            requested_batching = False
+        self.enable_batching = requested_batching and self.device.type == "cuda"
+        self.batch_max_size = _env_int("VEENA3_BICODEC_BATCH_MAX", 16, minimum=1)
+        self.batch_timeout_s = _env_float("VEENA3_BICODEC_BATCH_TIMEOUT_MS", 6.0, minimum=0.1) / 1000.0
+
+        # Async micro-batcher state (per event loop).
+        self._batch_map_lock = threading.Lock()
+        self._batch_queues: Dict[int, asyncio.Queue] = {}
+        self._batch_workers: Dict[int, asyncio.Task] = {}
+        self._batch_total_batches = 0
+        self._batch_total_requests = 0
+        self._batch_max_seen = 0

         print(f"🎵 Loading BiCodec audio tokenizer from {model_path}...")
         print(f"   Device: {self.device}")
@@
         # The multi-engine approach (Tier 3) provides more impactful gains.

         print(f"✅ BiCodec decoder initialized (sample rate: {AUDIO_SAMPLE_RATE}Hz)")
-        if enable_batching:
-            print(f"   ℹ  Batching enabled (for streaming support)")
+        if self.enable_batching:
+            print(
+                "   ℹ  Micro-batching enabled "
+                f"(max_batch={self.batch_max_size}, timeout_ms={self.batch_timeout_s * 1000:.1f})"
+            )
@@
     async def decode_to_bytes_async(
         self,
         semantic_ids: List[int],
         global_ids: List[int],
@@
             global_ids,
         )
+
+    @staticmethod
+    def _normalize_global_ids(global_ids: List[int]) -> List[int]:
+        expected = 32
+        if len(global_ids) >= expected:
+            return list(global_ids[:expected])
+        return list(global_ids) + [0] * (expected - len(global_ids))
+
+    def _decode_streaming_batch_sync(
+        self,
+        batch_items: List[Tuple[List[int], List[int], bool, bool]],
+    ) -> List[Optional[bytes]]:
+        """
+        Run a single batched BiCodec forward pass for multiple stream decode requests.
+
+        Falls back to sequential decode_streaming if shapes or settings are not batch-safe.
+        """
+        if not batch_items:
+            return []
+        if len(batch_items) == 1:
+            sem, glob, use_sw, trim = batch_items[0]
+            return [self.decode_streaming(sem, glob, use_sliding_window=use_sw, trim_warmup=trim)]
+
+        # Sliding window path uses special trimming behavior; keep it sequential.
+        if any(use_sw for _, _, use_sw, _ in batch_items):
+            return [
+                self.decode_streaming(sem, glob, use_sliding_window=use_sw, trim_warmup=trim)
+                for sem, glob, use_sw, trim in batch_items
+            ]
+
+        # Decoder needs at least 8 semantic tokens. Use fallback if any request is below.
+        if any(len(sem) < 8 for sem, _, _, _ in batch_items):
+            return [
+                self.decode_streaming(sem, glob, use_sliding_window=use_sw, trim_warmup=trim)
+                for sem, glob, use_sw, trim in batch_items
+            ]
+
+        try:
+            batch_size = len(batch_items)
+            max_sem_len = max(len(sem) for sem, _, _, _ in batch_items)
+
+            semantic_batch = torch.zeros(
+                (batch_size, max_sem_len),
+                dtype=torch.long,
+                device=self.device,
+            )
+            global_batch = torch.zeros(
+                (batch_size, 32),
+                dtype=torch.long,
+                device=self.device,
+            )
+            semantic_lengths: List[int] = []
+
+            for idx, (semantic_ids, global_ids, _, _) in enumerate(batch_items):
+                sem_len = len(semantic_ids)
+                semantic_lengths.append(sem_len)
+                semantic_batch[idx, :sem_len] = torch.tensor(
+                    semantic_ids,
+                    dtype=torch.long,
+                    device=self.device,
+                )
+                global_batch[idx] = torch.tensor(
+                    self._normalize_global_ids(global_ids),
+                    dtype=torch.long,
+                    device=self.device,
+                )
+
+            with torch.inference_mode():
+                wav_batch = self.audio_tokenizer.detokenize(global_batch, semantic_batch)
+
+            results: List[Optional[bytes]] = []
+            for idx, sem_len in enumerate(semantic_lengths):
+                audio = wav_batch[idx] if getattr(wav_batch, "ndim", 1) > 1 else wav_batch
+                if isinstance(audio, torch.Tensor):
+                    audio = audio.detach().cpu().numpy()
+                audio_np = np.asarray(audio).reshape(-1)
+                expected_samples = sem_len * 320
+                if expected_samples > 0 and len(audio_np) > expected_samples:
+                    audio_np = audio_np[:expected_samples]
+                audio_np = np.clip(audio_np, -1.0, 1.0)
+                audio_int16 = (audio_np * 32767).astype(np.int16)
+                results.append(audio_int16.tobytes())
+            return results
+        except Exception:
+            logger.exception("BiCodec batched decode failed; falling back to sequential decode")
+            return [
+                self.decode_streaming(sem, glob, use_sliding_window=use_sw, trim_warmup=trim)
+                for sem, glob, use_sw, trim in batch_items
+            ]
+
+    def _ensure_batch_worker(self, loop: asyncio.AbstractEventLoop) -> asyncio.Queue:
+        key = id(loop)
+        with self._batch_map_lock:
+            queue = self._batch_queues.get(key)
+            worker = self._batch_workers.get(key)
+            if queue is not None and worker is not None and not worker.done():
+                return queue
+
+            queue = asyncio.Queue(maxsize=max(self.batch_max_size * 8, 256))
+            worker = loop.create_task(self._batch_worker(loop, queue), name=f"bicodec-batch-worker-{key}")
+            self._batch_queues[key] = queue
+            self._batch_workers[key] = worker
+            return queue
+
+    async def _batch_worker(self, loop: asyncio.AbstractEventLoop, queue: asyncio.Queue) -> None:
+        key = id(loop)
+        try:
+            while True:
+                first: _BatchDecodeRequest = await queue.get()
+                batch: List[_BatchDecodeRequest] = [first]
+                deadline = loop.time() + self.batch_timeout_s
+
+                while len(batch) < self.batch_max_size:
+                    remaining = deadline - loop.time()
+                    if remaining <= 0:
+                        break
+                    try:
+                        req: _BatchDecodeRequest = await asyncio.wait_for(queue.get(), timeout=remaining)
+                    except asyncio.TimeoutError:
+                        break
+                    batch.append(req)
+
+                batch_inputs = [
+                    (req.semantic_ids, req.global_ids, req.use_sliding_window, req.trim_warmup)
+                    for req in batch
+                ]
+                t0 = time.perf_counter()
+                try:
+                    results = await loop.run_in_executor(None, self._decode_streaming_batch_sync, batch_inputs)
+                    if len(results) != len(batch):
+                        raise RuntimeError("batch decode result size mismatch")
+                    for req, result in zip(batch, results):
+                        if not req.future.done():
+                            req.future.set_result(result)
+                except Exception as exc:
+                    for req in batch:
+                        if not req.future.done():
+                            req.future.set_exception(exc)
+                finally:
+                    _ = (time.perf_counter() - t0) * 1000.0
+                    self._batch_total_batches += 1
+                    self._batch_total_requests += len(batch)
+                    if len(batch) > self._batch_max_seen:
+                        self._batch_max_seen = len(batch)
+        finally:
+            with self._batch_map_lock:
+                self._batch_queues.pop(key, None)
+                self._batch_workers.pop(key, None)
+
+    def get_batching_stats(self) -> Dict[str, float]:
+        batches = max(1, self._batch_total_batches)
+        return {
+            "enabled": 1.0 if self.enable_batching else 0.0,
+            "batch_total_batches": float(self._batch_total_batches),
+            "batch_total_requests": float(self._batch_total_requests),
+            "batch_avg_size": float(self._batch_total_requests / batches),
+            "batch_max_seen": float(self._batch_max_seen),
+        }
@@
     async def decode_single_async(
         self,
         semantic_ids: List[int],
         global_ids: List[int],
         trim_warmup: bool = False,
@@
-        import asyncio
-        loop = asyncio.get_event_loop()
-        return await loop.run_in_executor(
-            None,  # Default ThreadPoolExecutor
-            self.decode_streaming,
-            semantic_ids,
-            global_ids,
-            use_sliding_window,
-            trim_warmup,
-        )
+        loop = asyncio.get_running_loop()
+        if self.enable_batching and not use_sliding_window:
+            queue = self._ensure_batch_worker(loop)
+            future: asyncio.Future = loop.create_future()
+            req = _BatchDecodeRequest(
+                semantic_ids=list(semantic_ids),
+                global_ids=list(global_ids),
+                use_sliding_window=use_sliding_window,
+                trim_warmup=trim_warmup,
+                future=future,
+                submitted_at=time.perf_counter(),
+            )
+            await queue.put(req)
+            return await future
+
+        return await loop.run_in_executor(
+            None,
+            self.decode_streaming,
+            semantic_ids,
+            global_ids,
+            use_sliding_window,
+            trim_warmup,
+        )
*** End Patch


Output: {"output":"Success. Updated the following files:\nM /home/ubuntu/veenaModal/veena3modal/core/bicodec_decoder.py\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


**Adding decoder batching stats to metrics**


Micro-batching logic is now in `BiCodecDecoder` (shared async worker + batched detokenize). I’m adding batch-size observability into streaming metrics next so we can verify whether we’re actually achieving multi-request decode batches under load.


Input:
*** Begin Patch
*** Update File: /home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py
@@
         perf: Dict[str, Any] = {
@@
             "timeline_markers": list(timeline_marks),
             "timeline_total_ms": float((time.perf_counter() - t_request_start) * 1000.0),
         }
+
+        decoder_batch_stats = {}
+        if hasattr(self.snac_decoder, "get_batching_stats"):
+            try:
+                decoder_batch_stats = self.snac_decoder.get_batching_stats() or {}
+            except Exception:
+                decoder_batch_stats = {}
+        if decoder_batch_stats:
+            perf.update(decoder_batch_stats)
@@
         if emit_progress:
             perf: Dict[str, Any] = {
@@
                 "stream_window_size_tokens": int(WINDOW_SIZE),
                 "stream_windowed_decode": bool(_STREAM_WINDOWED_DECODE),
             }
+            decoder_batch_stats = {}
+            if hasattr(self.snac_decoder, "get_batching_stats"):
+                try:
+                    decoder_batch_stats = self.snac_decoder.get_batching_stats() or {}
+                except Exception:
+                    decoder_batch_stats = {}
+            if decoder_batch_stats:
+                perf.update(decoder_batch_stats)
             yield (b"", perf)
*** End Patch


Output: {"output":"Success. Updated the following files:\nM /home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


**Enabling batching via environment variable**


Input:
*** Begin Patch
*** Update File: /home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py
@@
         # Initialize BiCodec decoder
         logger.info("Initializing BiCodec decoder...")
+        batch_raw = os.environ.get("VEENA3_BICODEC_BATCHING", "true").strip().lower()
+        enable_bicodec_batching = batch_raw in {"1", "true", "yes", "on"}
         bicodec_decoder = BiCodecDecoder(
             device=device,
             model_path=bicodec_path,
+            enable_batching=enable_bicodec_batching,
         )
*** End Patch


Output: {"output":"Success. Updated the following files:\nM /home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


**Extending stress script for batching metrics**


Input:
*** Begin Patch
*** Update File: /home/ubuntu/veenaModal/scripts/stress_test_streaming_runtime.py
@@
     timing_keys = [
@@
         "bicodec_decode_calls",
         "bicodec_decode_wall_ms_total",
         "bicodec_decode_interval_ms_p50",
+        "batch_avg_size",
+        "batch_max_seen",
     ]
*** End Patch


Output: {"output":"Success. Updated the following files:\nM /home/ubuntu/veenaModal/scripts/stress_test_streaming_runtime.py\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


Input:
*** Begin Patch
*** Update File: /home/ubuntu/veenaModal/scripts/stress_test_streaming_runtime.py
@@
             "disable_windowed_decode": args.disable_windowed_decode,
+            "disable_bicodec_batching": args.disable_bicodec_batching,
+            "bicodec_batch_max": args.bicodec_batch_max,
+            "bicodec_batch_timeout_ms": args.bicodec_batch_timeout_ms,
             "timestamp": int(time.time()),
         },
@@
                     "bicodec_decode_calls": lvl.timing_stats("bicodec_decode_calls"),
                     "bicodec_decode_wall_ms_total": lvl.timing_stats("bicodec_decode_wall_ms_total"),
                     "bicodec_decode_interval_ms_p50": lvl.timing_stats("bicodec_decode_interval_ms_p50"),
+                    "batch_avg_size": lvl.timing_stats("batch_avg_size"),
+                    "batch_max_seen": lvl.timing_stats("batch_max_seen"),
                     "latency_over_timeline_ms": lvl.latency_over_metric_stats("timeline_total_ms"),
                 },
*** End Patch


Output: {"output":"Success. Updated the following files:\nM /home/ubuntu/veenaModal/scripts/stress_test_streaming_runtime.py\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


**Adding parser arguments and environment settings**


Input:
*** Begin Patch
*** Update File: /home/ubuntu/veenaModal/scripts/stress_test_streaming_runtime.py
@@
     parser.add_argument("--stream-min-semantic-first", type=int, default=10)
     parser.add_argument("--stream-crossfade-ms", type=int, default=50)
     parser.add_argument("--disable-windowed-decode", action="store_true", default=False)
+    parser.add_argument("--disable-bicodec-batching", action="store_true", default=False)
+    parser.add_argument("--bicodec-batch-max", type=int, default=16)
+    parser.add_argument("--bicodec-batch-timeout-ms", type=float, default=6.0)
     parser.add_argument("--model-path", type=str, default="")
     parser.add_argument("--output", default="stress_streaming_runtime.json")
@@
     os.environ["VEENA3_STREAM_MIN_SEMANTIC_FIRST"] = str(args.stream_min_semantic_first)
     os.environ["VEENA3_STREAM_CROSSFADE_MS"] = str(args.stream_crossfade_ms)
     os.environ["VEENA3_STREAM_WINDOWED_DECODE"] = "0" if args.disable_windowed_decode else "1"
+    os.environ["VEENA3_BICODEC_BATCHING"] = "0" if args.disable_bicodec_batching else "1"
+    os.environ["VEENA3_BICODEC_BATCH_MAX"] = str(args.bicodec_batch_max)
+    os.environ["VEENA3_BICODEC_BATCH_TIMEOUT_MS"] = str(args.bicodec_batch_timeout_ms)
*** End Patch


Output: {"output":"Success. Updated the following files:\nM /home/ubuntu/veenaModal/scripts/stress_test_streaming_runtime.py\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


**Considering batch metrics inclusion**


cd /home/ubuntu/veenaModal && python3 -m py_compile veena3modal/core/bicodec_decoder.py veena3modal/core/streaming_pipeline.py veena3modal/services/tts_runtime.py scripts/stress_test_streaming_runtime.py


Output:
Chunk ID: 6f48b5
Wall time: 0.0513 seconds
Process exited with code 0
Original token count: 0
Output:


**Planning benchmark comparisons**


Code compiles. I’m running a controlled A/B now at `c64` with identical settings: first with BiCodec micro-batching disabled, then enabled, so we can measure the real throughput/latency gain from batching alone.


cd /home/ubuntu/veenaModal && python3 scripts/stress_test_streaming_runtime.py --levels 64 --text short --max-tokens 1024 --timeout 240 --requests-multiplier 1 --min-requests 8 --gpu-memory 0.25 --stream-output-kind delta --disable-bicodec-batching --output /tmp/stream_c64_no_bicodec_batch.json


Output:
Chunk ID: 1d0363
Wall time: 1.0011 seconds
Process running with session ID 54156
Original token count: 0
Output:


**Noting need for polling**


Output:
Chunk ID: 443c92
Wall time: 5.0013 seconds
Process running with session ID 54156
Original token count: 1361
Output:
[2026-02-18 18:21:26] INFO tts_runtime.py:183: Loading SparkTTS model with vLLM...
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
🚀 Initializing Spark TTS Model
📁 Model path: /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
🗣  Speakers: 12 predefined (lipakshi, vardan, reet, Nandini, krishna, anika, adarsh, Nilay, Aarvi, Asha, Bittu, Mira)
🎭 Emotions: 10 tags in [bracket] format

📝 Loading tokenizer from model...
✅ Tokenizer loaded: 165168 tokens

🔍 Validating 12 speakers for Spark TTS...
  lipakshi → <|speaker_0|>: 7 tokens
  vardan → <|speaker_1|>: 7 tokens
  reet → <|speaker_2|>: 7 tokens
  Nandini → <|speaker_3|>: 7 tokens
  krishna → <|speaker_4|>: 7 tokens
  anika → <|speaker_5|>: 7 tokens
  adarsh → <|speaker_6|>: 7 tokens
  Nilay → <|speaker_7|>: 7 tokens
  Aarvi → <|speaker_8|>: 7 tokens
  Asha → <|speaker_9|>: 7 tokens
  Bittu → <|speaker_10|>: 8 tokens
  Mira → <|speaker_11|>: 8 tokens
✅ All 12 speakers validated with mapping

🔍 Validating 10 emotion tags for Spark TTS...
✅ All 10 emotion tags validated

📋 Sample emotion tags:
  [angry]: 1 tokens
  [curious]: 1 tokens
  [excited]: 1 tokens
  [giggle]: 1 tokens
  [laughs harder]: 1 tokens
  ...

🔧 Initializing vLLM engine with optimizations...
  - dtype: bfloat16
  - max_model_len: 4096
  - gpu_memory_utilization: 0.25
  - enable_prefix_caching: True
  - enforce_eager (disable CUDA graphs): False
INFO 02-18 18:21:27 [model.py:541] Resolved architecture: Qwen2ForCausalLM
INFO 02-18 18:21:27 [model.py:1561] Using max model len 4096
INFO 02-18 18:21:27 [scheduler.py:226] Chunked prefill is enabled with max_num_batched_tokens=4096.
INFO 02-18 18:21:27 [vllm.py:624] Asynchronous scheduling is enabled.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(EngineCore_DP0 pid=75220) INFO 02-18 18:21:28 [core.py:96] Initializing a V1 LLM engine (v0.15.1) with config: model='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', speculative_config=None, tokenizer='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [4096], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 256, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': True}, 'local_cache_dir': None, 'static_all_moe_layers': []}
(EngineCore_DP0 pid=75220) INFO 02-18 18:21:31 [parallel_state.py:1212] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://216.81.248.184:58693 backend=nccl
(EngineCore_DP0 pid=75220) INFO 02-18 18:21:31 [parallel_state.py:1423] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A


Output:
Chunk ID: be3941
Wall time: 5.0010 seconds
Process running with session ID 54156
Original token count: 420
Output:
(EngineCore_DP0 pid=75220) INFO 02-18 18:21:31 [gpu_model_runner.py:4033] Starting to load model /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM...
(EngineCore_DP0 pid=75220) INFO 02-18 18:21:32 [cuda.py:364] Using FLASH_ATTN attention backend out of potential backends: ('FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION')
(EngineCore_DP0 pid=75220) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
(EngineCore_DP0 pid=75220) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
(EngineCore_DP0 pid=75220)
Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
(EngineCore_DP0 pid=75220)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.23it/s]
(EngineCore_DP0 pid=75220)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.23it/s]
(EngineCore_DP0 pid=75220)
(EngineCore_DP0 pid=75220) INFO 02-18 18:21:33 [default_loader.py:291] Loading weights took 0.88 seconds
(EngineCore_DP0 pid=75220) INFO 02-18 18:21:34 [gpu_model_runner.py:4130] Model loading took 0.95 GiB memory and 1.437075 seconds
(EngineCore_DP0 pid=75220) INFO 02-18 18:21:38 [backends.py:812] Using cache directory: /home/ubuntu/.cache/vllm/torch_compile_cache/f3ff13511c/rank_0_0/backbone for vLLM's torch.compile
(EngineCore_DP0 pid=75220) INFO 02-18 18:21:38 [backends.py:872] Dynamo bytecode transform time: 4.24 s


Output:
Chunk ID: da1d16
Wall time: 5.0018 seconds
Process running with session ID 54156
Original token count: 782
Output:
(EngineCore_DP0 pid=75220) INFO 02-18 18:21:41 [backends.py:267] Directly load the compiled graph(s) for compile range (1, 4096) from the cache, took 0.668 s
(EngineCore_DP0 pid=75220) INFO 02-18 18:21:41 [monitor.py:34] torch.compile takes 4.91 s in total
(EngineCore_DP0 pid=75220) INFO 02-18 18:21:42 [gpu_worker.py:356] Available KV cache memory: 18.06 GiB
(EngineCore_DP0 pid=75220) INFO 02-18 18:21:42 [kv_cache_utils.py:1307] GPU KV cache size: 1,577,712 tokens
(EngineCore_DP0 pid=75220) INFO 02-18 18:21:42 [kv_cache_utils.py:1312] Maximum concurrency for 4,096 tokens per request: 385.18x
(EngineCore_DP0 pid=75220)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   0%|          | 0/35 [00:00<?, ?it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  11%|█▏        | 4/35 [00:00<00:00, 32.54it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  23%|██▎       | 8/35 [00:00<00:00, 35.40it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  34%|███▍      | 12/35 [00:00<00:00, 36.67it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  46%|████▌     | 16/35 [00:00<00:00, 36.90it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  57%|█████▋    | 20/35 [00:00<00:00, 36.89it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  69%|██████▊   | 24/35 [00:00<00:00, 36.58it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  80%|████████  | 28/35 [00:00<00:00, 36.37it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  91%|█████████▏| 32/35 [00:00<00:00, 34.76it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 35/35 [00:01<00:00, 34.90it/s]
(EngineCore_DP0 pid=75220)
Capturing CUDA graphs (decode, FULL):   0%|          | 0/19 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL):  16%|█▌        | 3/19 [00:00<00:00, 28.05it/s]
Capturing CUDA graphs (decode, FULL):  37%|███▋      | 7/19 [00:00<00:00, 32.37it/s]
Capturing CUDA graphs (decode, FULL):  58%|█████▊    | 11/19 [00:00<00:00, 34.18it/s]
Capturing CUDA graphs (decode, FULL):  79%|███████▉  | 15/19 [00:00<00:00, 34.60it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 34.88it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 34.09it/s]
(EngineCore_DP0 pid=75220) INFO 02-18 18:21:45 [gpu_model_runner.py:5063] Graph capturing finished in 2 secs, took 0.28 GiB
(EngineCore_DP0 pid=75220) INFO 02-18 18:21:45 [core.py:272] init engine (profile, create kv cache, warmup model) took 10.96 seconds
[2026-02-18 18:21:46] INFO tts_runtime.py:192: Initializing prompt builder...
[2026-02-18 18:21:46] INFO tts_runtime.py:199: Initializing BiCodec decoder...
/home/ubuntu/.local/lib/python3.10/site-packages/torch/nn/utils/weight_norm.py:144: FutureWarning: `torch.nn.utils.weight_norm` is deprecated in favor of `torch.nn.utils.parametrizations.weight_norm`.
  WeightNorm.apply(module, name, dim)


Output:
Chunk ID: 996f72
Wall time: 5.0007 seconds
Process running with session ID 54156
Original token count: 127
Output:
[2026-02-18 18:21:54] INFO tts_runtime.py:209: Initializing TTS pipeline...
[2026-02-18 18:21:54] INFO tts_runtime.py:220: Initializing streaming pipeline...
[2026-02-18 18:21:54] INFO long_text_processor.py:112: LongTextProcessor initialized
[2026-02-18 18:21:54] INFO tts_runtime.py:277: ✅ TTS runtime initialized in 32738ms
[2026-02-18 18:21:54] INFO tts_runtime.py:278:    Model version: LLM
[2026-02-18 18:21:54] INFO tts_runtime.py:289: Speaker globals cache startup precompute skipped (disabled)


Output:
Chunk ID: dbab1d
Wall time: 1.5332 seconds
Process exited with code 0
Original token count: 572
Output:
✅ vLLM engine initialized successfully!

🎉 Spark TTS Model ready for inference!

🎵 Loading BiCodec audio tokenizer from /home/ubuntu/veenaModal/models/spark_tts_4speaker...
   Device: cuda
Missing tensor: mel_transformer.spectrogram.window
Missing tensor: mel_transformer.mel_scale.fb
✅ BiCodec decoder initialized (sample rate: 16000Hz)
🚀 SparkTTSPipeline initialized
🌊 Veena3SlidingWindowPipeline initialized (sliding window: 28 tokens)

=== streaming concurrency=64 requests=64 ===
success=64/64 (100%)
wall=3.88s throughput=16.47 req/s
latency_ms avg=3883 p50=3883 p95=3884 max=3884
ttfb_ms avg=1544 p50=1527 p95=2016 max=2242
chunks_per_req avg=3.5 p50=4.0 p95=4.0 max=4.0
inter_chunk_ms avg=894.2 p50=1237.2 p95=2213.9 max=2338.2
tail_emit_speed_x avg=1.16 p50=1.11 p95=1.65 max=1.79
timeline_to_first_batch_ms: avg=50.60 p50=50.61 p95=56.73 max=58.31
timeline_to_first_chunk_emitted_ms: avg=1530.66 p50=1514.37 p95=1992.43 max=2216.90
timeline_total_ms: avg=3807.98 p50=3856.55 p95=3878.18 max=3880.36
llm_time_in_queue_ms: avg=0.08 p50=0.08 p95=0.16 max=0.17
llm_scheduler_ms: avg=34.77 p50=35.86 p95=42.37 max=42.37
llm_first_token_ms: avg=48.41 p50=48.36 p95=55.24 max=56.94
llm_request_lifecycle_ms: avg=1617.38 p50=1626.32 p95=1734.43 max=1915.08
llm_generation_wall_ms: avg=3788.16 p50=3856.00 p95=3878.14 max=3880.32
llm_time_per_token_ms: avg=16.97 p50=16.94 p95=19.45 max=22.04
llm_batch_wall_ms_total: avg=2469.51 p50=2292.32 p95=3748.75 max=3846.26
llm_decode_wall_ms_total: avg=2418.94 p50=2237.09 p95=3698.50 max=3792.29
tokens_per_batch_p50: avg=1.00 p50=1.00 p95=1.00 max=1.00
bicodec_decode_calls: avg=2.55 p50=3.00 p95=3.00 max=3.00
bicodec_decode_wall_ms_total: avg=3266.85 p50=3316.46 p95=3337.53 max=3345.02
bicodec_decode_interval_ms_p50: avg=1300.46 p50=1305.37 p95=2213.85 max=2338.19
batch_avg_size: avg=0.00 p50=0.00 p95=0.00 max=0.00
batch_max_seen: avg=0.00 p50=0.00 p95=0.00 max=0.00
latency_over_timeline_ms: avg=75.26 p50=26.21 p95=301.45 max=894.14
timeline_share: llm=99.5% bicodec=85.8% other=0.0%
gpu: util_avg=65% util_max=94% mem_avg=24406MB mem_max=24931MB

results saved: /tmp/stream_c64_no_bicodec_batch.json
ERROR 02-18 18:21:59 [core_client.py:605] Engine core proc EngineCore_DP0 died unexpectedly, shutting down client.


**Enabling batching by default**


No-batching baseline is captured. I’m now running the same `c64` workload with BiCodec micro-batching enabled to measure throughput, latency, and actual batch-size realized (`batch_avg_size`, `batch_max_seen`).


cd /home/ubuntu/veenaModal && python3 scripts/stress_test_streaming_runtime.py --levels 64 --text short --max-tokens 1024 --timeout 240 --requests-multiplier 1 --min-requests 8 --gpu-memory 0.25 --stream-output-kind delta --bicodec-batch-max 16 --bicodec-batch-timeout-ms 6 --output /tmp/stream_c64_with_bicodec_batch.json


Output:
Chunk ID: c5f817
Wall time: 1.0002 seconds
Process running with session ID 65791
Original token count: 0
Output:


Summary: auto


Output:
Chunk ID: 404c58
Wall time: 5.0014 seconds
Process running with session ID 65791
Original token count: 1399
Output:
[2026-02-18 18:22:11] INFO tts_runtime.py:183: Loading SparkTTS model with vLLM...
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
🚀 Initializing Spark TTS Model
📁 Model path: /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
🗣  Speakers: 12 predefined (lipakshi, vardan, reet, Nandini, krishna, anika, adarsh, Nilay, Aarvi, Asha, Bittu, Mira)
🎭 Emotions: 10 tags in [bracket] format

📝 Loading tokenizer from model...
✅ Tokenizer loaded: 165168 tokens

🔍 Validating 12 speakers for Spark TTS...
  lipakshi → <|speaker_0|>: 7 tokens
  vardan → <|speaker_1|>: 7 tokens
  reet → <|speaker_2|>: 7 tokens
  Nandini → <|speaker_3|>: 7 tokens
  krishna → <|speaker_4|>: 7 tokens
  anika → <|speaker_5|>: 7 tokens
  adarsh → <|speaker_6|>: 7 tokens
  Nilay → <|speaker_7|>: 7 tokens
  Aarvi → <|speaker_8|>: 7 tokens
  Asha → <|speaker_9|>: 7 tokens
  Bittu → <|speaker_10|>: 8 tokens
  Mira → <|speaker_11|>: 8 tokens
✅ All 12 speakers validated with mapping

🔍 Validating 10 emotion tags for Spark TTS...
✅ All 10 emotion tags validated

📋 Sample emotion tags:
  [angry]: 1 tokens
  [curious]: 1 tokens
  [excited]: 1 tokens
  [giggle]: 1 tokens
  [laughs harder]: 1 tokens
  ...

🔧 Initializing vLLM engine with optimizations...
  - dtype: bfloat16
  - max_model_len: 4096
  - gpu_memory_utilization: 0.25
  - enable_prefix_caching: True
  - enforce_eager (disable CUDA graphs): False
INFO 02-18 18:22:12 [model.py:541] Resolved architecture: Qwen2ForCausalLM
INFO 02-18 18:22:12 [model.py:1561] Using max model len 4096
INFO 02-18 18:22:12 [scheduler.py:226] Chunked prefill is enabled with max_num_batched_tokens=4096.
INFO 02-18 18:22:12 [vllm.py:624] Asynchronous scheduling is enabled.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(EngineCore_DP0 pid=75945) INFO 02-18 18:22:13 [core.py:96] Initializing a V1 LLM engine (v0.15.1) with config: model='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', speculative_config=None, tokenizer='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [4096], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 256, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': True}, 'local_cache_dir': None, 'static_all_moe_layers': []}
(EngineCore_DP0 pid=75945) INFO 02-18 18:22:16 [parallel_state.py:1212] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://216.81.248.184:40245 backend=nccl
(EngineCore_DP0 pid=75945) INFO 02-18 18:22:16 [parallel_state.py:1423] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
(EngineCore_DP0 pid=75945) INFO 02-18 18:22:16 [gpu_model_runner.py:4033] Starting to load model /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM...


Output:
Chunk ID: 992c56
Wall time: 5.0014 seconds
Process running with session ID 65791
Original token count: 445
Output:
(EngineCore_DP0 pid=75945) INFO 02-18 18:22:17 [cuda.py:364] Using FLASH_ATTN attention backend out of potential backends: ('FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION')
(EngineCore_DP0 pid=75945) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
(EngineCore_DP0 pid=75945) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
(EngineCore_DP0 pid=75945)
Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
(EngineCore_DP0 pid=75945)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.30it/s]
(EngineCore_DP0 pid=75945)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.30it/s]
(EngineCore_DP0 pid=75945)
(EngineCore_DP0 pid=75945) INFO 02-18 18:22:18 [default_loader.py:291] Loading weights took 0.84 seconds
(EngineCore_DP0 pid=75945) INFO 02-18 18:22:19 [gpu_model_runner.py:4130] Model loading took 0.95 GiB memory and 1.382176 seconds
(EngineCore_DP0 pid=75945) INFO 02-18 18:22:23 [backends.py:812] Using cache directory: /home/ubuntu/.cache/vllm/torch_compile_cache/f3ff13511c/rank_0_0/backbone for vLLM's torch.compile
(EngineCore_DP0 pid=75945) INFO 02-18 18:22:23 [backends.py:872] Dynamo bytecode transform time: 4.20 s
(EngineCore_DP0 pid=75945) INFO 02-18 18:22:26 [backends.py:267] Directly load the compiled graph(s) for compile range (1, 4096) from the cache, took 0.658 s
(EngineCore_DP0 pid=75945) INFO 02-18 18:22:26 [monitor.py:34] torch.compile takes 4.85 s in total


Output:
Chunk ID: 5abc72
Wall time: 5.0010 seconds
Process running with session ID 65791
Original token count: 718
Output:
(EngineCore_DP0 pid=75945) INFO 02-18 18:22:27 [gpu_worker.py:356] Available KV cache memory: 18.06 GiB
(EngineCore_DP0 pid=75945) INFO 02-18 18:22:27 [kv_cache_utils.py:1307] GPU KV cache size: 1,577,712 tokens
(EngineCore_DP0 pid=75945) INFO 02-18 18:22:27 [kv_cache_utils.py:1312] Maximum concurrency for 4,096 tokens per request: 385.18x
(EngineCore_DP0 pid=75945)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   0%|          | 0/35 [00:00<?, ?it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  11%|█▏        | 4/35 [00:00<00:00, 32.73it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  23%|██▎       | 8/35 [00:00<00:00, 35.57it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  34%|███▍      | 12/35 [00:00<00:00, 36.63it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  46%|████▌     | 16/35 [00:00<00:00, 37.12it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  57%|█████▋    | 20/35 [00:00<00:00, 36.84it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  69%|██████▊   | 24/35 [00:00<00:00, 36.52it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  80%|████████  | 28/35 [00:00<00:00, 36.41it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  91%|█████████▏| 32/35 [00:00<00:00, 34.94it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 35/35 [00:00<00:00, 35.08it/s]
(EngineCore_DP0 pid=75945)
Capturing CUDA graphs (decode, FULL):   0%|          | 0/19 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL):  16%|█▌        | 3/19 [00:00<00:00, 29.54it/s]
Capturing CUDA graphs (decode, FULL):  37%|███▋      | 7/19 [00:00<00:00, 33.60it/s]
Capturing CUDA graphs (decode, FULL):  58%|█████▊    | 11/19 [00:00<00:00, 35.17it/s]
Capturing CUDA graphs (decode, FULL):  79%|███████▉  | 15/19 [00:00<00:00, 35.65it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 36.11it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 35.27it/s]
(EngineCore_DP0 pid=75945) INFO 02-18 18:22:30 [gpu_model_runner.py:5063] Graph capturing finished in 2 secs, took 0.28 GiB
(EngineCore_DP0 pid=75945) INFO 02-18 18:22:30 [core.py:272] init engine (profile, create kv cache, warmup model) took 10.84 seconds
[2026-02-18 18:22:31] INFO tts_runtime.py:192: Initializing prompt builder...
[2026-02-18 18:22:31] INFO tts_runtime.py:199: Initializing BiCodec decoder...
/home/ubuntu/.local/lib/python3.10/site-packages/torch/nn/utils/weight_norm.py:144: FutureWarning: `torch.nn.utils.weight_norm` is deprecated in favor of `torch.nn.utils.parametrizations.weight_norm`.
  WeightNorm.apply(module, name, dim)


Output:
Chunk ID: 3b6f2c
Wall time: 5.0013 seconds
Process running with session ID 65791
Original token count: 714
Output:
[2026-02-18 18:22:40] INFO tts_runtime.py:209: Initializing TTS pipeline...
[2026-02-18 18:22:40] INFO tts_runtime.py:220: Initializing streaming pipeline...
[2026-02-18 18:22:40] INFO long_text_processor.py:112: LongTextProcessor initialized
[2026-02-18 18:22:40] INFO tts_runtime.py:277: ✅ TTS runtime initialized in 33840ms
[2026-02-18 18:22:40] INFO tts_runtime.py:278:    Model version: LLM
[2026-02-18 18:22:40] INFO tts_runtime.py:289: Speaker globals cache startup precompute skipped (disabled)
✅ vLLM engine initialized successfully!

🎉 Spark TTS Model ready for inference!

🎵 Loading BiCodec audio tokenizer from /home/ubuntu/veenaModal/models/spark_tts_4speaker...
   Device: cuda
Missing tensor: mel_transformer.spectrogram.window
Missing tensor: mel_transformer.mel_scale.fb
✅ BiCodec decoder initialized (sample rate: 16000Hz)
   ℹ  Micro-batching enabled (max_batch=16, timeout_ms=6.0)
🚀 SparkTTSPipeline initialized
🌊 Veena3SlidingWindowPipeline initialized (sliding window: 28 tokens)

=== streaming concurrency=64 requests=64 ===
success=64/64 (100%)
wall=3.23s throughput=19.78 req/s
latency_ms avg=2735 p50=3044 p95=3221 max=3233
ttfb_ms avg=898 p50=874 p95=1013 max=1013
chunks_per_req avg=5.0 p50=5.0 p95=5.0 max=6.0
inter_chunk_ms avg=441.4 p50=313.5 p95=1487.0 max=1596.8
tail_emit_speed_x avg=2.18 p50=1.97 p95=3.13 max=5.21
timeline_to_first_batch_ms: avg=52.95 p50=52.72 p95=59.45 max=61.07
timeline_to_first_chunk_emitted_ms: avg=884.35 p50=864.05 p95=991.14 max=992.11
timeline_total_ms: avg=2656.91 p50=2752.08 p95=3205.72 max=3210.21
llm_time_in_queue_ms: avg=0.10 p50=0.09 p95=0.17 max=0.22
llm_scheduler_ms: avg=36.46 p50=45.03 p95=45.03 max=45.03
llm_first_token_ms: avg=51.14 p50=50.81 p95=58.39 max=60.11
llm_request_lifecycle_ms: avg=1840.65 p50=1854.71 p95=1971.47 max=2121.99
llm_generation_wall_ms: avg=2637.09 p50=2737.67 p95=3202.66 max=3210.17
llm_time_per_token_ms: avg=11.66 p50=12.06 p95=13.80 max=14.88
llm_batch_wall_ms_total: avg=2455.67 p50=2411.57 p95=3007.23 max=3177.76
llm_decode_wall_ms_total: avg=2402.75 p50=2361.92 p95=2960.54 max=3122.20
tokens_per_batch_p50: avg=1.00 p50=1.00 p95=1.00 max=1.00
bicodec_decode_calls: avg=4.02 p50=4.00 p95=4.00 max=5.00
bicodec_decode_wall_ms_total: avg=1622.52 p50=1771.04 p95=2490.68 max=2659.50
bicodec_decode_interval_ms_p50: avg=444.04 p50=450.50 p95=560.45 max=640.88
batch_avg_size: avg=10.12 p50=9.14 p95=11.71 max=12.46
batch_max_seen: avg=16.00 p50=16.00 p95=16.00 max=16.00
latency_over_timeline_ms: avg=78.56 p50=45.00 p95=257.18 max=327.47
timeline_share: llm=99.3% bicodec=61.1% other=0.0%
gpu: util_avg=64% util_max=97% mem_avg=25722MB mem_max=26903MB

results saved: /tmp/stream_c64_with_bicodec_batch.json
ERROR 02-18 18:22:45 [core_client.py:605] Engine core proc EngineCore_DP0 died unexpectedly, shutting down client.


**Planning multi-level batching tests**


cd /home/ubuntu/veenaModal && python3 scripts/stress_test_streaming_runtime.py --levels 32,64,96 --text short --max-tokens 1024 --timeout 240 --requests-multiplier 1 --min-requests 8 --gpu-memory 0.25 --stream-output-kind delta --bicodec-batch-max 16 --bicodec-batch-timeout-ms 6 --output /tmp/stream_levels_with_bicodec_batch.json


Output:
Chunk ID: 38b671
Wall time: 1.0011 seconds
Process running with session ID 75249
Original token count: 0
Output:


**Requesting poll and commentary**


This first batched run is a strong jump: `c64` went from `16.47` to `19.78 req/s`, and TTFB dropped from `~1544ms` to `~898ms`. I’m running a full 32/64/96 sweep now to find the new concurrency knee with batching enabled.


Output:
Chunk ID: 03e216
Wall time: 5.0006 seconds
Process running with session ID 75249
Original token count: 1585
Output:
[2026-02-18 18:22:58] INFO tts_runtime.py:183: Loading SparkTTS model with vLLM...
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
🚀 Initializing Spark TTS Model
📁 Model path: /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
🗣  Speakers: 12 predefined (lipakshi, vardan, reet, Nandini, krishna, anika, adarsh, Nilay, Aarvi, Asha, Bittu, Mira)
🎭 Emotions: 10 tags in [bracket] format

📝 Loading tokenizer from model...
✅ Tokenizer loaded: 165168 tokens

🔍 Validating 12 speakers for Spark TTS...
  lipakshi → <|speaker_0|>: 7 tokens
  vardan → <|speaker_1|>: 7 tokens
  reet → <|speaker_2|>: 7 tokens
  Nandini → <|speaker_3|>: 7 tokens
  krishna → <|speaker_4|>: 7 tokens
  anika → <|speaker_5|>: 7 tokens
  adarsh → <|speaker_6|>: 7 tokens
  Nilay → <|speaker_7|>: 7 tokens
  Aarvi → <|speaker_8|>: 7 tokens
  Asha → <|speaker_9|>: 7 tokens
  Bittu → <|speaker_10|>: 8 tokens
  Mira → <|speaker_11|>: 8 tokens
✅ All 12 speakers validated with mapping

🔍 Validating 10 emotion tags for Spark TTS...
✅ All 10 emotion tags validated

📋 Sample emotion tags:
  [angry]: 1 tokens
  [curious]: 1 tokens
  [excited]: 1 tokens
  [giggle]: 1 tokens
  [laughs harder]: 1 tokens
  ...

🔧 Initializing vLLM engine with optimizations...
  - dtype: bfloat16
  - max_model_len: 4096
  - gpu_memory_utilization: 0.25
  - enable_prefix_caching: True
  - enforce_eager (disable CUDA graphs): False
INFO 02-18 18:22:59 [model.py:541] Resolved architecture: Qwen2ForCausalLM
INFO 02-18 18:22:59 [model.py:1561] Using max model len 4096
INFO 02-18 18:22:59 [scheduler.py:226] Chunked prefill is enabled with max_num_batched_tokens=4096.
INFO 02-18 18:22:59 [vllm.py:624] Asynchronous scheduling is enabled.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(EngineCore_DP0 pid=76657) INFO 02-18 18:23:00 [core.py:96] Initializing a V1 LLM engine (v0.15.1) with config: model='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', speculative_config=None, tokenizer='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [4096], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 256, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': True}, 'local_cache_dir': None, 'static_all_moe_layers': []}
(EngineCore_DP0 pid=76657) INFO 02-18 18:23:02 [parallel_state.py:1212] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://216.81.248.184:55665 backend=nccl
(EngineCore_DP0 pid=76657) INFO 02-18 18:23:02 [parallel_state.py:1423] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
(EngineCore_DP0 pid=76657) INFO 02-18 18:23:03 [gpu_model_runner.py:4033] Starting to load model /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM...
(EngineCore_DP0 pid=76657) INFO 02-18 18:23:04 [cuda.py:364] Using FLASH_ATTN attention backend out of potential backends: ('FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION')
(EngineCore_DP0 pid=76657) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
(EngineCore_DP0 pid=76657) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
(EngineCore_DP0 pid=76657)
Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]