Output:
Chunk ID: 7882db
Wall time: 0.0515 seconds
Process exited with code 0
Original token count: 4384
Output:
   520
   521	        # BiCodec token buffers
   522	        semantic_buffer = []
   523	        global_buffer = []
   524	        processed_token_count = 0  # Track how many vLLM tokens we've parsed
   525	        total_audio_chunks = 0
   526
   527	        # BiCodec streaming configuration
   528	        EXPECTED_GLOBAL_COUNT = 32
   529	        MIN_SEMANTIC_FOR_FIRST_CHUNK = 10  # OPTIMIZATION: Lowered from 16 (decoder min is 8)
   530	        DECODE_INTERVAL = 48  # OPTIMIZATION: Increased from 24 to reduce O(n) re-decode calls
   531	        CROSSFADE_MS = 50
   532
   533	        # Generate unique request ID
   534	        import uuid
   535	        request_id = f"bicodec-first-{uuid.uuid4().hex[:8]}-{int(time.time() * 1000000)}"
   536
   537	        # Reuse pre-warmed token parser (singleton, created once in __init__)
   538	        token_parser = self.token_parser
   539
   540	        results_generator = self.model.engine.generate(
   541	            prompt=prompt,
   542	            sampling_params=sampling_params,
   543	            request_id=request_id,
   544	        )
   545
   546	        # Track state
   547	        last_decode_count = 0
   548	        total_samples_emitted_to_user = 0
   549	        previous_chunk_tail: Optional[bytes] = None
   550	        captured_globals: List[int] = []  # Will be populated once we have 32 globals
   551
   552	        # Stream tokens with BiCodec extraction
   553	        async for request_output in results_generator:
   554	            generated_ids = request_output.outputs[0].token_ids
   555
   556	            # OPTIMIZATION: Incremental parsing (O(1) per new token)
   557	            new_token_ids = generated_ids[processed_token_count:]
   558	            processed_token_count = len(generated_ids)
   559	            token_parser.parse_incremental(new_token_ids, semantic_buffer, global_buffer)
   560
   561	            # Capture global tokens once we have all 32
   562	            if not captured_globals and len(global_buffer) >= EXPECTED_GLOBAL_COUNT:
   563	                captured_globals = global_buffer[:EXPECTED_GLOBAL_COUNT]
   564
   565	            # Process if we have all global tokens
   566	            if len(global_buffer) >= EXPECTED_GLOBAL_COUNT:
   567	                semantic_count = len(semantic_buffer)
   568	                if semantic_count >= MIN_SEMANTIC_FOR_FIRST_CHUNK:
   569	                    if semantic_count - last_decode_count >= DECODE_INTERVAL:
   570	                        last_decode_count = semantic_count
   571
   572	                        all_semantic = semantic_buffer
   573	                        window_global = global_buffer[:EXPECTED_GLOBAL_COUNT]
   574
   575	                        if self.snac_decoder.enable_batching:
   576	                            audio_bytes = await self.snac_decoder.decode_single_async(
   577	                                semantic_ids=all_semantic,
   578	                                global_ids=window_global,
   579	                                trim_warmup=False,
   580	                                use_sliding_window=False
   581	                            )
   582	                        else:
   583	                            audio_bytes = self.snac_decoder.decode_streaming(
   584	                                semantic_ids=all_semantic,
   585	                                global_ids=window_global,
   586	                                use_sliding_window=False,
   587	                                trim_warmup=False
   588	                            )
   589
   590	                        if audio_bytes:
   591	                            total_samples_decoded = len(audio_bytes) // 2
   592
   593	                            if total_samples_decoded > total_samples_emitted_to_user:
   594	                                new_bytes_start = total_samples_emitted_to_user * 2
   595	                                new_audio_bytes = audio_bytes[new_bytes_start:]
   596
   597	                                to_emit, previous_chunk_tail = crossfade_bytes_int16(
   598	                                    previous_chunk_tail,
   599	                                    new_audio_bytes,
   600	                                    sample_rate_hz=16000,
   601	                                    crossfade_ms=CROSSFADE_MS,
   602	                                )
   603
   604	                                samples_emitted_in_this_chunk = len(to_emit) // 2
   605	                                total_samples_emitted_to_user += samples_emitted_in_this_chunk
   606
   607	                                if to_emit:
   608	                                    total_audio_chunks += 1
   609	                                    # Yield tuple: (audio_bytes, global_ids)
   610	                                    yield (to_emit, captured_globals)
   611
   612	        # Final chunk processing
   613	        if len(semantic_buffer) > last_decode_count and len(global_buffer) >= EXPECTED_GLOBAL_COUNT:
   614	            audio_bytes = self.snac_decoder.decode_streaming(
   615	                semantic_ids=semantic_buffer,
   616	                global_ids=global_buffer[:EXPECTED_GLOBAL_COUNT],
   617	                use_sliding_window=False,
   618	                trim_warmup=False
   619	            )
   620	            if audio_bytes:
   621	                total_samples_decoded = len(audio_bytes) // 2
   622
   623	                if total_samples_decoded > total_samples_emitted_to_user:
   624	                    new_bytes_start = total_samples_emitted_to_user * 2
   625	                    new_audio_bytes = audio_bytes[new_bytes_start:]
   626
   627	                    to_emit, previous_chunk_tail = crossfade_bytes_int16(
   628	                        previous_chunk_tail,
   629	                        new_audio_bytes,
   630	                        sample_rate_hz=16000,
   631	                        crossfade_ms=CROSSFADE_MS,
   632	                    )
   633
   634	                    samples_emitted_in_final = len(to_emit) // 2
   635	                    total_samples_emitted_to_user += samples_emitted_in_final
   636
   637	                    if to_emit:
   638	                        total_audio_chunks += 1
   639	                        yield (to_emit, captured_globals)
   640
   641	        # Flush remaining tail
   642	        if previous_chunk_tail:
   643	            total_audio_chunks += 1
   644	            tail_samples = len(previous_chunk_tail) // 2
   645	            total_samples_emitted_to_user += tail_samples
   646	            yield (previous_chunk_tail, captured_globals)
   647
   648	        t_end = time.time()
   649	        audio_duration_s = total_samples_emitted_to_user / 16000
   650	        print(f"🎵 First chunk complete: {audio_duration_s:.2f}s audio, {total_audio_chunks} chunks, captured {len(captured_globals)} globals")
   651
   652	    async def generate_speech_stream_indic_continuation(
   653	        self,
   654	        speaker: str,
   655	        text: str,
   656	        global_ids: List[int],
   657	        temperature: float = DEFAULT_TEMPERATURE,
   658	        top_k: int = DEFAULT_TOP_K,
   659	        top_p: float = DEFAULT_TOP_P,
   660	        max_tokens: int = DEFAULT_MAX_TOKENS,
   661	        repetition_penalty: float = DEFAULT_REPETITION_PENALTY,
   662	        seed: Optional[int] = None,
   663	    ) -> AsyncGenerator[bytes, None]:
   664	        """
   665	        Generate speech for CONTINUATION chunks using pre-captured global tokens.
   666
   667	        This is the key method for voice consistency in chunked generation:
   668	        - Uses global_ids from the first chunk to maintain identical voice
   669	        - Model generates only semantic tokens (skips global token generation)
   670	        - Ensures no voice drift across chunks
   671
   672	        CRITICAL for production:
   673	        - global_ids is request-scoped, passed explicitly (no shared state)
   674	        - Thread-safe: each request has its own state
   675	        - No global caching that could cause cross-request contamination
   676
   677	        Args:
   678	            speaker: Speaker name (must match first chunk)
   679	            text: Continuation text chunk to synthesize
   680	            global_ids: 32 global token IDs captured from first chunk
   681	            temperature: Sampling temperature
   682	            top_k: Top-k sampling
   683	            top_p: Nucleus sampling
   684	            max_tokens: Max tokens to generate
   685	            repetition_penalty: Prevent repetition
   686	            seed: Random seed (use same as first chunk for consistency)
   687
   688	        Yields:
   689	            Audio bytes (int16 PCM, 16kHz mono)
   690
   691	        Raises:
   692	            ValueError: If global_ids doesn't contain exactly 32 tokens
   693
   694	        Thread Safety:
   695	            Fully thread-safe. All state is request-scoped and passed explicitly.
   696	        """
   697	        import time
   698	        t_start = time.time()
   699
   700	        # Validate global tokens
   701	        EXPECTED_GLOBAL_COUNT = 32
   702	        if len(global_ids) != EXPECTED_GLOBAL_COUNT:
   703	            raise ValueError(
   704	                f"Expected exactly {EXPECTED_GLOBAL_COUNT} global tokens, got {len(global_ids)}. "
   705	                f"global_ids must be captured from first chunk generation."
   706	            )
   707
   708	        # Build prompt WITH pre-filled global tokens
   709	        # This tells the model to skip global generation and go straight to semantic tokens
   710	        prompt = self.prompt_builder.build_prefix_with_globals(speaker, text, global_ids)
   711
   712	        # Configure sampling
   713	        # OPTIMIZATION: Use Spark TTS stop token, not legacy SNAC CODE_END_TOKEN_ID
   714	        sampling_params = SamplingParams(
   715	            temperature=temperature,
   716	            top_k=top_k,
   717	            top_p=top_p,
   718	            max_tokens=max_tokens,
   719	            min_tokens=DEFAULT_MIN_TOKENS,
   720	            repetition_penalty=repetition_penalty,
   721	            stop=TRAINING_STOP_TOKEN_IDS,  # "<|im_end|>" - matches non-streaming pipeline
   722	            skip_special_tokens=False,
   723	            seed=seed,
   724	        )
   725
   726	        # BiCodec token buffers
   727	        semantic_buffer = []
   728	        global_buffer_unused = []  # Not used in continuation, but needed for parser API
   729	        processed_token_count = 0  # Track how many vLLM tokens we've parsed
   730	        total_audio_chunks = 0
   731
   732	        # BiCodec streaming configuration
   733	        MIN_SEMANTIC_FOR_FIRST_CHUNK = 10  # OPTIMIZATION: Lowered from 16 (decoder min is 8)
   734	        DECODE_INTERVAL = 48  # OPTIMIZATION: Increased from 24 to reduce O(n) re-decode calls
   735	        CROSSFADE_MS = 50
   736
   737	        # Generate unique request ID
   738	        import uuid
   739	        request_id = f"bicodec-cont-{uuid.uuid4().hex[:8]}-{int(time.time() * 1000000)}"
   740
   741	        # Reuse pre-warmed token parser (singleton, created once in __init__)
   742	        token_parser = self.token_parser
   743
   744	        results_generator = self.model.engine.generate(
   745	            prompt=prompt,
   746	            sampling_params=sampling_params,
   747	            request_id=request_id,
   748	        )
   749
   750	        # Track state
   751	        last_decode_count = 0
   752	        total_samples_emitted_to_user = 0
   753	        previous_chunk_tail: Optional[bytes] = None
   754
   755	        # Stream tokens - model will generate ONLY semantic tokens (globals are pre-filled)
   756	        async for request_output in results_generator:
   757	            generated_ids = request_output.outputs[0].token_ids
   758
   759	            # OPTIMIZATION: Incremental parsing (O(1) per new token)
   760	            # Note: In continuation, model only generates semantic tokens (globals pre-filled in prompt)
   761	            new_token_ids = generated_ids[processed_token_count:]
   762	            processed_token_count = len(generated_ids)
   763	            token_parser.parse_incremental(new_token_ids, semantic_buffer, global_buffer_unused)
   764
   765	            semantic_count = len(semantic_buffer)
   766	            if semantic_count >= MIN_SEMANTIC_FOR_FIRST_CHUNK:
   767	                if semantic_count - last_decode_count >= DECODE_INTERVAL:
   768	                    last_decode_count = semantic_count
   769
   770	                    # Use pre-captured global tokens for decoding
   771	                    if self.snac_decoder.enable_batching:
   772	                        audio_bytes = await self.snac_decoder.decode_single_async(
   773	                            semantic_ids=semantic_buffer,
   774	                            global_ids=global_ids,  # Use captured globals
   775	                            trim_warmup=False,
   776	                            use_sliding_window=False
   777	                        )
   778	                    else:
   779	                        audio_bytes = self.snac_decoder.decode_streaming(
   780	                            semantic_ids=semantic_buffer,
   781	                            global_ids=global_ids,  # Use captured globals
   782	                            use_sliding_window=False,
   783	                            trim_warmup=False
   784	                        )
   785
   786	                    if audio_bytes:
   787	                        total_samples_decoded = len(audio_bytes) // 2
   788
   789	                        if total_samples_decoded > total_samples_emitted_to_user:
   790	                            new_bytes_start = total_samples_emitted_to_user * 2
   791	                            new_audio_bytes = audio_bytes[new_bytes_start:]
   792
   793	                            to_emit, previous_chunk_tail = crossfade_bytes_int16(
   794	                                previous_chunk_tail,
   795	                                new_audio_bytes,
   796	                                sample_rate_hz=16000,
   797	                                crossfade_ms=CROSSFADE_MS,
   798	                            )
   799
   800	                            samples_emitted_in_this_chunk = len(to_emit) // 2
   801	                            total_samples_emitted_to_user += samples_emitted_in_this_chunk
   802
   803	                            if to_emit:
   804	                                total_audio_chunks += 1
   805	                                yield to_emit
   806
   807	        # Final chunk processing
   808	        if len(semantic_buffer) > last_decode_count:
   809	            audio_bytes = self.snac_decoder.decode_streaming(
   810	                semantic_ids=semantic_buffer,
   811	                global_ids=global_ids,  # Use captured globals
   812	                use_sliding_window=False,
   813	                trim_warmup=False
   814	            )
   815	            if audio_bytes:
   816	                total_samples_decoded = len(audio_bytes) // 2
   817
   818	                if total_samples_decoded > total_samples_emitted_to_user:
   819	                    new_bytes_start = total_samples_emitted_to_user * 2
   820	                    new_audio_bytes = audio_bytes[new_bytes_start:]
   821
   822	                    to_emit, previous_chunk_tail = crossfade_bytes_int16(
   823	                        previous_chunk_tail,
   824	                        new_audio_bytes,
   825	                        sample_rate_hz=16000,
   826	                        crossfade_ms=CROSSFADE_MS,
   827	                    )
   828
   829	                    samples_emitted_in_final = len(to_emit) // 2
   830	                    total_samples_emitted_to_user += samples_emitted_in_final
   831
   832	                    if to_emit:
   833	                        total_audio_chunks += 1
   834	                        yield to_emit
   835
   836	        # Flush remaining tail
   837	        if previous_chunk_tail:
   838	            total_audio_chunks += 1
   839	            tail_samples = len(previous_chunk_tail) // 2
   840	            total_samples_emitted_to_user += tail_samples
   841	            yield previous_chunk_tail
   842
   843	        t_end = time.time()
   844	        audio_duration_s = total_samples_emitted_to_user / 16000
   845	        print(f"🎵 Continuation chunk complete: {audio_duration_s:.2f}s audio, {total_audio_chunks} chunks (using cached globals)")
   846


Output:
Chunk ID: 7b1548
Wall time: 0.0509 seconds
Process exited with code 0
Original token count: 420
Output:
121:        sampling_params = SamplingParams(
145:        results_generator = self.model.engine.generate(
255:        sampling_params = SamplingParams(
270:        processed_token_count = 0  # Track how many vLLM tokens we've parsed
293:        results_generator = self.model.engine.generate(
328:            new_token_ids = generated_ids[processed_token_count:]
329:            processed_token_count = len(generated_ids)
332:            token_parser.parse_incremental(new_token_ids, semantic_buffer, global_buffer)
456:    async def generate_speech_stream_indic_first_chunk(
472:        yield and passes them to generate_speech_stream_indic_continuation() for subsequent chunks.
476:            async for audio_bytes, global_ids in pipeline.generate_speech_stream_indic_first_chunk(...):
509:        sampling_params = SamplingParams(
524:        processed_token_count = 0  # Track how many vLLM tokens we've parsed
540:        results_generator = self.model.engine.generate(
557:            new_token_ids = generated_ids[processed_token_count:]
558:            processed_token_count = len(generated_ids)
559:            token_parser.parse_incremental(new_token_ids, semantic_buffer, global_buffer)
652:    async def generate_speech_stream_indic_continuation(
714:        sampling_params = SamplingParams(
729:        processed_token_count = 0  # Track how many vLLM tokens we've parsed
744:        results_generator = self.model.engine.generate(
761:            new_token_ids = generated_ids[processed_token_count:]
762:            processed_token_count = len(generated_ids)
763:            token_parser.parse_incremental(new_token_ids, semantic_buffer, global_buffer_unused)


# Collaboration Mode: Default

You are now in Default mode. Any previous instructions for other modes (e.g. Plan mode) are no longer active.

Your active mode changes only when new developer instructions with a different `<collaboration_mode>...</collaboration_mode>` change it; user requests or tool descriptions do not change mode by themselves. Known mode names are Default and Plan.

## request_user_input availability

The `request_user_input` tool is unavailable in Default mode. If you call it while in Default mode, it will return an error.

If a decision is necessary and cannot be discovered from local context, ask the user directly. However, in Default mode you should strongly prefer executing the user's request rather than stopping to ask questions.

Summary: auto

## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) When `SKILL.md` references relative paths (e.g., `scripts/foo.py`), resolve them relative to the skill directory listed above first, and only consider other paths if needed.
  3) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  4) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  5) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.


**Implementing delta output streaming and benchmarking plan**


I’m implementing the streaming-specific vLLM optimization now: `DELTA` output mode plus streaming-loop handling for delta tokens. Then I’ll run direct streaming stress benchmarks (cumulative vs delta) to quantify the gain.


Input:
*** Begin Patch
*** Update File: /home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py
@@
 import asyncio
+import logging
+import os
 import re
 from typing import AsyncGenerator, Optional, List
 from vllm import SamplingParams
+from vllm.sampling_params import RequestOutputKind
 from veena3modal.audio.crossfade import crossfade_bytes_int16
 from veena3modal.core.token_utils import BiCodecTokenParser
@@
 from veena3modal.core.constants import (
@@
 )

+logger = logging.getLogger(__name__)
+_STREAM_OUTPUT_KIND_RAW = os.environ.get("VEENA3_STREAM_OUTPUT_KIND", "delta").strip().lower()
+_STREAM_USE_DELTA_OUTPUT = _STREAM_OUTPUT_KIND_RAW in {"delta", "d", "1", "true", "yes", "on"}
+

 class Veena3SlidingWindowPipeline:
@@
         # Reuse pre-warmed token parser (singleton, created once in __init__)
         token_parser = self.token_parser
@@
         results_generator = self.model.engine.generate(
             prompt=prompt,
             sampling_params=sampling_params,
             request_id=request_id,
         )
@@
         if previous_chunk_tail:
             total_audio_chunks += 1
             tail_samples = len(previous_chunk_tail) // 2
             total_samples_emitted_to_user += tail_samples
             yield previous_chunk_tail

         # Log completion summary
         t_end = time.time()
         audio_duration_s = total_samples_emitted_to_user / 16000  # 16kHz sample rate
         ttfb_ms = (t_first_chunk_decoded - t_start) * 1000 if t_first_chunk_decoded else 0
-        print(f"🎵 Streaming complete: {audio_duration_s:.2f}s audio, TTFB: {ttfb_ms:.0f}ms, {total_audio_chunks} chunks")
+        logger.debug(
+            "streaming complete: audio=%.2fs ttfb=%.0fms chunks=%d",
+            audio_duration_s,
+            ttfb_ms,
+            total_audio_chunks,
+        )
@@
         sampling_params = SamplingParams(
             temperature=temperature,
             top_k=top_k,
             top_p=top_p,
             max_tokens=max_tokens,
             min_tokens=DEFAULT_MIN_TOKENS,
             repetition_penalty=repetition_penalty,
             stop=TRAINING_STOP_TOKEN_IDS,  # "<|im_end|>" - matches non-streaming pipeline
             skip_special_tokens=False,  # Keep BiCodec tokens in output
             seed=seed,
+            output_kind=(RequestOutputKind.DELTA if _STREAM_USE_DELTA_OUTPUT else RequestOutputKind.CUMULATIVE),
         )
@@
-        processed_token_count = 0  # Track how many vLLM tokens we've parsed
+        processed_token_count = 0  # Used for cumulative mode; ignored for delta mode.
@@
         # Stream tokens with BiCodec extraction
         async for request_output in results_generator:
@@
-            generated_ids = request_output.outputs[0].token_ids
-
-            # OPTIMIZATION: Only process NEW tokens (incremental parsing)
-            # Old code: tokenizer.decode(ALL) + re.findall(ALL) = O(n²) per stream
-            # New code: parse only new tokens = O(1) per token
-            new_token_ids = generated_ids[processed_token_count:]
-            processed_token_count = len(generated_ids)
+            generated_ids = request_output.outputs[0].token_ids
+
+            # In DELTA mode token_ids are already incremental.
+            if _STREAM_USE_DELTA_OUTPUT:
+                new_token_ids = generated_ids
+            else:
+                new_token_ids = generated_ids[processed_token_count:]
+                processed_token_count = len(generated_ids)
@@
-        print(f"🎵 First chunk complete: {audio_duration_s:.2f}s audio, {total_audio_chunks} chunks, captured {len(captured_globals)} globals")
+        logger.debug(
+            "first chunk complete: audio=%.2fs chunks=%d captured_globals=%d",
+            audio_duration_s,
+            total_audio_chunks,
+            len(captured_globals),
+        )
@@
         sampling_params = SamplingParams(
             temperature=temperature,
             top_k=top_k,
             top_p=top_p,
             max_tokens=max_tokens,
             min_tokens=DEFAULT_MIN_TOKENS,
             repetition_penalty=repetition_penalty,
             stop=TRAINING_STOP_TOKEN_IDS,  # "<|im_end|>" - matches non-streaming pipeline
             skip_special_tokens=False,
             seed=seed,
+            output_kind=(RequestOutputKind.DELTA if _STREAM_USE_DELTA_OUTPUT else RequestOutputKind.CUMULATIVE),
         )
@@
-        processed_token_count = 0  # Track how many vLLM tokens we've parsed
+        processed_token_count = 0  # Used for cumulative mode; ignored for delta mode.
@@
         # Stream tokens - model will generate ONLY semantic tokens (globals are pre-filled)
         async for request_output in results_generator:
             generated_ids = request_output.outputs[0].token_ids

-            # OPTIMIZATION: Incremental parsing (O(1) per new token)
-            # Note: In continuation, model only generates semantic tokens (globals pre-filled in prompt)
-            new_token_ids = generated_ids[processed_token_count:]
-            processed_token_count = len(generated_ids)
+            # In DELTA mode token_ids are already incremental.
+            if _STREAM_USE_DELTA_OUTPUT:
+                new_token_ids = generated_ids
+            else:
+                new_token_ids = generated_ids[processed_token_count:]
+                processed_token_count = len(generated_ids)
             token_parser.parse_incremental(new_token_ids, semantic_buffer, global_buffer_unused)
@@
-        print(f"🎵 Continuation chunk complete: {audio_duration_s:.2f}s audio, {total_audio_chunks} chunks (using cached globals)")
+        logger.debug(
+            "continuation chunk complete: audio=%.2fs chunks=%d",
+            audio_duration_s,
+            total_audio_chunks,
+        )
*** End Patch


Output:
apply_patch verification failed: Failed to find expected lines in /home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:
        sampling_params = SamplingParams(
            temperature=temperature,
            top_k=top_k,
            top_p=top_p,
            max_tokens=max_tokens,
            min_tokens=DEFAULT_MIN_TOKENS,
            repetition_penalty=repetition_penalty,
            stop=TRAINING_STOP_TOKEN_IDS,  # "<|im_end|>" - matches non-streaming pipeline
            skip_special_tokens=False,  # Keep BiCodec tokens in output
            seed=seed,
        )


Summary: auto


**Inspecting source context for patch**


cd /home/ubuntu/veenaModal && sed -n '1,80p' veena3modal/core/streaming_pipeline.py


cd /home/ubuntu/veenaModal && sed -n '240,340p' veena3modal/core/streaming_pipeline.py


cd /home/ubuntu/veenaModal && sed -n '680,760p' veena3modal/core/streaming_pipeline.py


Output:
Chunk ID: db0ce6
Wall time: 0.0514 seconds
Process exited with code 0
Original token count: 585
Output:
"""
Veena3 Streaming Pipeline - Sliding Window Approach

Implements streaming for both SNAC and BiCodec tokens.

For BiCodec (Spark TTS):
1. Generate tokens from vLLM
2. Parse NEW token IDs incrementally (O(1) per token via cache)
3. Buffer semantic + global tokens
4. Apply sliding window (every N token pairs)
5. Decode and stream audio chunks

OPTIMIZATION (Dec 2025):
- Replaced O(n²) pattern (decode-all + regex-all per iteration)
- Now uses incremental token parsing with BiCodecTokenParser
- ~10x CPU reduction in streaming hot loop

For SNAC (legacy):
1. Filter SNAC token IDs directly
2. Apply sliding window
3. Decode and stream
"""

import asyncio
import re
from typing import AsyncGenerator, Optional, List
from vllm import SamplingParams
from veena3modal.audio.crossfade import crossfade_bytes_int16
from veena3modal.core.token_utils import BiCodecTokenParser

from veena3modal.core.constants import (
    CODE_END_TOKEN_ID,
    CODE_START_TOKEN_ID,
    SNAC_MIN_ID,
    SNAC_MAX_ID,
    TRAINING_STOP_TOKEN_IDS,
    DEFAULT_TEMPERATURE,
    DEFAULT_TOP_K,
    DEFAULT_TOP_P,
    DEFAULT_MAX_TOKENS,
    DEFAULT_MIN_TOKENS,
    DEFAULT_REPETITION_PENALTY,
    DEFAULT_SEED,
)

class Veena3SlidingWindowPipeline:
    """
    Streaming TTS pipeline using sliding window approach.

    This eliminates choppy audio and popping artifacts by:
    - Decoding overlapping 28-token windows (4 frames)
    - Keeping only the middle 2048 samples from each decode
    - Creating natural continuity between chunks

    Based on the official Canopy Labs implementation.
    """

    def __init__(
        self,
        model,
        prompt_builder,
        snac_decoder,
    ):
        """
        Initialize sliding window streaming pipeline.

        Args:
            model: Veena3Model instance
            prompt_builder: Veena3PromptBuilder instance
            snac_decoder: SNACDecoder instance (with batching enabled)
        """
        self.model = model
        self.prompt_builder = prompt_builder
        self.snac_decoder = snac_decoder

        # OPTIMIZATION: Pre-warm BiCodecTokenParser once at init, not per-request
        # Saves ~123ms per streaming TTFB by avoiding repeated 166K-entry vocab iteration
        tokenizer = getattr(model, "tokenizer", None)
        if tokenizer is None:


Output:
Chunk ID: 938159
Wall time: 0.0517 seconds
Process exited with code 0
Original token count: 1165
Output:
            repetition_penalty: Prevent loops
            seed: Random seed for reproducibility

        Yields:
            Audio bytes (int16 PCM, 16kHz mono - BiCodec)
        """
        import time
        t_start = time.time()

        # Build prompt using Indic prompt builder
        prompt = self.prompt_builder.build_prefix(speaker, text)

        # Configure sampling (matching non-streaming pipeline parameters)
        # OPTIMIZATION: Use Spark TTS stop token, not legacy SNAC CODE_END_TOKEN_ID (128258)
        # which doesn't exist in the Spark vocab and would cause generation to run to max_tokens
        sampling_params = SamplingParams(
            temperature=temperature,
            top_k=top_k,
            top_p=top_p,
            max_tokens=max_tokens,
            min_tokens=DEFAULT_MIN_TOKENS,
            repetition_penalty=repetition_penalty,
            stop=TRAINING_STOP_TOKEN_IDS,  # "<|im_end|>" - matches non-streaming pipeline
            skip_special_tokens=False,  # Keep BiCodec tokens in output
            seed=seed,
        )

        # BiCodec token buffers (separate for semantic and global)
        semantic_buffer = []
        global_buffer = []
        processed_token_count = 0  # Track how many vLLM tokens we've parsed
        total_audio_chunks = 0

        # BiCodec streaming configuration
        EXPECTED_GLOBAL_COUNT = 32  # BiCodec uses 32 fixed global tokens (per paper)
        MIN_SEMANTIC_FOR_FIRST_CHUNK = 10  # OPTIMIZATION: Lowered from 16 (decoder min is 8, -120ms TTFB)
        # OPTIMIZATION: Increased from 24 to 48 to reduce O(n) re-decode calls by 2x.
        # At 50 TPS, 48 tokens = ~960ms audio per chunk. Cuts total decode calls roughly in half.
        DECODE_INTERVAL = 48
        CROSSFADE_MS = 50
        # OPTIMIZATION: Windowed decode - only decode last WINDOW_SIZE tokens instead of ALL.
        # BiCodec WaveGenerator has finite receptive field (~64 tokens based on kernel sizes).
        # Window must be >= receptive_field + DECODE_INTERVAL to avoid boundary artifacts.
        WINDOW_SIZE = 128  # 128 tokens covers receptive field + margin for crossfade

        # Generate tokens with vLLM (streaming)
        import uuid
        request_id = f"bicodec-stream-{uuid.uuid4().hex[:8]}-{int(time.time() * 1000000)}"

        # Reuse pre-warmed token parser (singleton, created once in __init__)
        token_parser = self.token_parser

        t_gen_start = time.time()
        results_generator = self.model.engine.generate(
            prompt=prompt,
            sampling_params=sampling_params,
            request_id=request_id,
        )

        # Track timing
        t_first_token = None
        t_first_chunk_ready = None
        t_first_chunk_decoded = None
        iteration_count = 0
        last_decode_count = 0

        # Track audio samples EMITTED to the user (not just decoded)
        # BiCodec produces ~320 samples per semantic token at 16kHz
        # CRITICAL: This tracks samples that have been YIELDED, not held in tail
        total_samples_emitted_to_user = 0
        # Keep a tail to enable crossfading between consecutive emitted chunks
        previous_chunk_tail: Optional[bytes] = None

        # Stream tokens with BiCodec extraction
        async for request_output in results_generator:
            iteration_count += 1
            t_iter = time.time()

            # Track first token received
            if t_first_token is None:
                t_first_token = t_iter

            # Extract generated token IDs
            generated_ids = request_output.outputs[0].token_ids

            # OPTIMIZATION: Only process NEW tokens (incremental parsing)
            # Old code: tokenizer.decode(ALL) + re.findall(ALL) = O(n²) per stream
            # New code: parse only new tokens = O(1) per token
            new_token_ids = generated_ids[processed_token_count:]
            processed_token_count = len(generated_ids)

            # Parse new tokens and append to buffers (in-place modification)
            token_parser.parse_incremental(new_token_ids, semantic_buffer, global_buffer)

            # TRUE STREAMING with BiCodec two-phase generation:
            # Phase 1: Generate 32 global tokens (pre-roll)
            # Phase 2: Stream semantic tokens incrementally

            # Check if we have the 32 global tokens (pre-roll complete)
            if len(global_buffer) >= EXPECTED_GLOBAL_COUNT:
                # Pre-roll complete! Now we can stream semantic tokens


Output:
Chunk ID: 27caac
Wall time: 0.0508 seconds
Process exited with code 0
Original token count: 850
Output:
            global_ids: 32 global token IDs captured from first chunk
            temperature: Sampling temperature
            top_k: Top-k sampling
            top_p: Nucleus sampling
            max_tokens: Max tokens to generate
            repetition_penalty: Prevent repetition
            seed: Random seed (use same as first chunk for consistency)

        Yields:
            Audio bytes (int16 PCM, 16kHz mono)

        Raises:
            ValueError: If global_ids doesn't contain exactly 32 tokens

        Thread Safety:
            Fully thread-safe. All state is request-scoped and passed explicitly.
        """
        import time
        t_start = time.time()

        # Validate global tokens
        EXPECTED_GLOBAL_COUNT = 32
        if len(global_ids) != EXPECTED_GLOBAL_COUNT:
            raise ValueError(
                f"Expected exactly {EXPECTED_GLOBAL_COUNT} global tokens, got {len(global_ids)}. "
                f"global_ids must be captured from first chunk generation."
            )

        # Build prompt WITH pre-filled global tokens
        # This tells the model to skip global generation and go straight to semantic tokens
        prompt = self.prompt_builder.build_prefix_with_globals(speaker, text, global_ids)

        # Configure sampling
        # OPTIMIZATION: Use Spark TTS stop token, not legacy SNAC CODE_END_TOKEN_ID
        sampling_params = SamplingParams(
            temperature=temperature,
            top_k=top_k,
            top_p=top_p,
            max_tokens=max_tokens,
            min_tokens=DEFAULT_MIN_TOKENS,
            repetition_penalty=repetition_penalty,
            stop=TRAINING_STOP_TOKEN_IDS,  # "<|im_end|>" - matches non-streaming pipeline
            skip_special_tokens=False,
            seed=seed,
        )

        # BiCodec token buffers
        semantic_buffer = []
        global_buffer_unused = []  # Not used in continuation, but needed for parser API
        processed_token_count = 0  # Track how many vLLM tokens we've parsed
        total_audio_chunks = 0

        # BiCodec streaming configuration
        MIN_SEMANTIC_FOR_FIRST_CHUNK = 10  # OPTIMIZATION: Lowered from 16 (decoder min is 8)
        DECODE_INTERVAL = 48  # OPTIMIZATION: Increased from 24 to reduce O(n) re-decode calls
        CROSSFADE_MS = 50

        # Generate unique request ID
        import uuid
        request_id = f"bicodec-cont-{uuid.uuid4().hex[:8]}-{int(time.time() * 1000000)}"

        # Reuse pre-warmed token parser (singleton, created once in __init__)
        token_parser = self.token_parser

        results_generator = self.model.engine.generate(
            prompt=prompt,
            sampling_params=sampling_params,
            request_id=request_id,
        )

        # Track state
        last_decode_count = 0
        total_samples_emitted_to_user = 0
        previous_chunk_tail: Optional[bytes] = None

        # Stream tokens - model will generate ONLY semantic tokens (globals are pre-filled)
        async for request_output in results_generator:
            generated_ids = request_output.outputs[0].token_ids

            # OPTIMIZATION: Incremental parsing (O(1) per new token)
            # Note: In continuation, model only generates semantic tokens (globals pre-filled in prompt)


**Planning stepwise patching**


cd /home/ubuntu/veenaModal && rg -n "🎵|Streaming complete|First chunk complete|Continuation chunk complete|Sliding window streaming generation|Prompt built|Sampling:" veena3modal/core/streaming_pipeline.py


Output:
Chunk ID: 69b7eb
Wall time: 0.0518 seconds
Process exited with code 0
Original token count: 189
Output:
112:        print(f"\n🌊 Sliding window streaming generation")
118:        print(f"✅ Prompt built ({len(prompt)} chars)")
131:        print(f"🎲 Sampling: temp={temperature}, top_p={top_p}, sliding_window=28 tokens")
188:                                print(f"🎵 First chunk decoded ({len(audio_bytes)} bytes, sliding window)")
454:        print(f"🎵 Streaming complete: {audio_duration_s:.2f}s audio, TTFB: {ttfb_ms:.0f}ms, {total_audio_chunks} chunks")
650:        print(f"🎵 First chunk complete: {audio_duration_s:.2f}s audio, {total_audio_chunks} chunks, captured {len(captured_globals)} globals")
845:        print(f"🎵 Continuation chunk complete: {audio_duration_s:.2f}s audio, {total_audio_chunks} chunks (using cached globals)")


**Extending debug print patching**


Input:
*** Begin Patch
*** Update File: /home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py
@@
 import asyncio
+import logging
+import os
 import re
 from typing import AsyncGenerator, Optional, List
 from vllm import SamplingParams
+from vllm.sampling_params import RequestOutputKind
 from veena3modal.audio.crossfade import crossfade_bytes_int16
 from veena3modal.core.token_utils import BiCodecTokenParser
@@
 from veena3modal.core.constants import (
@@
     DEFAULT_SEED,
 )

+logger = logging.getLogger(__name__)
+_STREAM_OUTPUT_KIND_RAW = os.environ.get("VEENA3_STREAM_OUTPUT_KIND", "delta").strip().lower()
+_STREAM_USE_DELTA_OUTPUT = _STREAM_OUTPUT_KIND_RAW in {"delta", "d", "1", "true", "yes", "on"}
+
*** End Patch


Output: {"output":"Success. Updated the following files:\nM /home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


**Patching sampling params with output_kind**


Input:
*** Begin Patch
*** Update File: /home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py
@@
         sampling_params = SamplingParams(
             temperature=temperature,
             top_k=top_k,
             top_p=top_p,
             max_tokens=max_tokens,
             min_tokens=DEFAULT_MIN_TOKENS,
             repetition_penalty=repetition_penalty,
             stop=TRAINING_STOP_TOKEN_IDS,  # "<|im_end|>" - matches non-streaming pipeline
             skip_special_tokens=False,  # Keep BiCodec tokens in output
             seed=seed,
+            output_kind=(RequestOutputKind.DELTA if _STREAM_USE_DELTA_OUTPUT else RequestOutputKind.CUMULATIVE),
         )
*** End Patch


Output: {"output":"Success. Updated the following files:\nM /home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


**Adjusting sampling parameters**


Input:
*** Begin Patch
*** Update File: /home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py
@@
         sampling_params = SamplingParams(
             temperature=temperature,
             top_k=top_k,
             top_p=top_p,
             max_tokens=max_tokens,
             min_tokens=DEFAULT_MIN_TOKENS,
             repetition_penalty=repetition_penalty,
             stop=TRAINING_STOP_TOKEN_IDS,  # "<|im_end|>" - matches non-streaming pipeline
             skip_special_tokens=False,
             seed=seed,
+            output_kind=(RequestOutputKind.DELTA if _STREAM_USE_DELTA_OUTPUT else RequestOutputKind.CUMULATIVE),
         )
*** End Patch


Output: {"output":"Success. Updated the following files:\nM /home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


Input:
*** Begin Patch
*** Update File: /home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py
@@
         sampling_params = SamplingParams(
             temperature=temperature,
             top_k=top_k,
             top_p=top_p,
             max_tokens=max_tokens,
             min_tokens=DEFAULT_MIN_TOKENS,
             repetition_penalty=repetition_penalty,
             stop=TRAINING_STOP_TOKEN_IDS,  # "<|im_end|>" - matches non-streaming pipeline
             skip_special_tokens=False,
             seed=seed,
+            output_kind=(RequestOutputKind.DELTA if _STREAM_USE_DELTA_OUTPUT else RequestOutputKind.CUMULATIVE),
         )
*** End Patch


Output: {"output":"Success. Updated the following files:\nM /home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


**Fixing token count logic**


Input:
*** Begin Patch
*** Update File: /home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py
@@
-        processed_token_count = 0  # Track how many vLLM tokens we've parsed
+        processed_token_count = 0  # Used for cumulative mode; ignored for delta mode.
@@
-            # OPTIMIZATION: Only process NEW tokens (incremental parsing)
-            # Old code: tokenizer.decode(ALL) + re.findall(ALL) = O(n²) per stream
-            # New code: parse only new tokens = O(1) per token
-            new_token_ids = generated_ids[processed_token_count:]
-            processed_token_count = len(generated_ids)
+            # In DELTA mode token_ids are already incremental.
+            if _STREAM_USE_DELTA_OUTPUT:
+                new_token_ids = generated_ids
+            else:
+                new_token_ids = generated_ids[processed_token_count:]
+                processed_token_count = len(generated_ids)
*** End Patch


Output: {"output":"Success. Updated the following files:\nM /home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


**Identifying loop patch points**


Input:
*** Begin Patch
*** Update File: /home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py
@@
-        processed_token_count = 0  # Track how many vLLM tokens we've parsed
+        processed_token_count = 0  # Used for cumulative mode; ignored for delta mode.
@@
-            # OPTIMIZATION: Incremental parsing (O(1) per new token)
-            new_token_ids = generated_ids[processed_token_count:]
-            processed_token_count = len(generated_ids)
+            # In DELTA mode token_ids are already incremental.
+            if _STREAM_USE_DELTA_OUTPUT:
+                new_token_ids = generated_ids
+            else:
+                new_token_ids = generated_ids[processed_token_count:]
+                processed_token_count = len(generated_ids)
             token_parser.parse_incremental(new_token_ids, semantic_buffer, global_buffer)
*** End Patch


Output: {"output":"Success. Updated the following files:\nM /home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


**Tracing patch continuation logic**


Input:
*** Begin Patch
*** Update File: /home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py
@@
-        processed_token_count = 0  # Track how many vLLM tokens we've parsed
+        processed_token_count = 0  # Used for cumulative mode; ignored for delta mode.
@@
-            # OPTIMIZATION: Incremental parsing (O(1) per new token)
-            # Note: In continuation, model only generates semantic tokens (globals pre-filled in prompt)
-            new_token_ids = generated_ids[processed_token_count:]
-            processed_token_count = len(generated_ids)
+            # In DELTA mode token_ids are already incremental.
+            if _STREAM_USE_DELTA_OUTPUT:
+                new_token_ids = generated_ids
+            else:
+                new_token_ids = generated_ids[processed_token_count:]
+                processed_token_count = len(generated_ids)
             token_parser.parse_incremental(new_token_ids, semantic_buffer, global_buffer_unused)
*** End Patch


Output: {"output":"Success. Updated the following files:\nM /home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


**Converting prints to debug logs**


Input:
*** Begin Patch
*** Update File: /home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py
@@
-        print(f"\n🌊 Sliding window streaming generation")
-        print(f"📝 Description: {description[:80]}...")
-        print(f"💬 Text: {text}")
+        logger.debug("sliding-window streaming generation started")
+        logger.debug("description_prefix=%s", description[:80])
+        logger.debug("text_len=%d", len(text))
@@
-        print(f"✅ Prompt built ({len(prompt)} chars)")
+        logger.debug("prompt built chars=%d", len(prompt))
@@
-        print(f"🎲 Sampling: temp={temperature}, top_p={top_p}, sliding_window=28 tokens")
+        logger.debug("sampling temp=%s top_p=%s sliding_window=28", temperature, top_p)
@@
-                            if total_audio_chunks == 1:
-                                print(f"🎵 First chunk decoded ({len(audio_bytes)} bytes, sliding window)")
+                            if total_audio_chunks == 1:
+                                logger.debug("first sliding-window chunk decoded bytes=%d", len(audio_bytes))
@@
-        print(f"✅ Sliding window streaming complete: {total_tokens_generated} tokens → {total_audio_chunks} audio chunks")
+        logger.debug(
+            "sliding-window complete tokens=%d chunks=%d",
+            total_tokens_generated,
+            total_audio_chunks,
+        )
@@
-        print(f"🎵 Streaming complete: {audio_duration_s:.2f}s audio, TTFB: {ttfb_ms:.0f}ms, {total_audio_chunks} chunks")
+        logger.debug(
+            "streaming complete audio=%.2fs ttfb=%.0fms chunks=%d",
+            audio_duration_s,
+            ttfb_ms,
+            total_audio_chunks,
+        )
@@
-        print(f"🎵 First chunk complete: {audio_duration_s:.2f}s audio, {total_audio_chunks} chunks, captured {len(captured_globals)} globals")
+        logger.debug(
+            "first-chunk complete audio=%.2fs chunks=%d captured_globals=%d",
+            audio_duration_s,
+            total_audio_chunks,
+            len(captured_globals),
+        )
@@
-        print(f"🎵 Continuation chunk complete: {audio_duration_s:.2f}s audio, {total_audio_chunks} chunks (using cached globals)")
+        logger.debug(
+            "continuation complete audio=%.2fs chunks=%d",
+            audio_duration_s,
+            total_audio_chunks,
+        )
*** End Patch


Output: {"output":"Success. Updated the following files:\nM /home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


**Planning streaming benchmark script**


I’m adding a direct streaming stress benchmark script (runtime-level, no HTTP) so we can A/B `cumulative` vs `delta` output mode and measure true streaming KPIs: TTFB, chunk cadence, emission speed, and throughput under concurrency.


Input:
*** Begin Patch
*** Add File: /home/ubuntu/veenaModal/scripts/stress_test_streaming_runtime.py
+#!/usr/bin/env python3
+"""
+Direct-runtime streaming stress benchmark for Veena3 TTS.
+
+Focuses on streaming KPIs:
+- TTFB (time to first chunk)
+- End-to-end latency
+- Chunk cadence (inter-chunk interval)
+- Emission speed (audio seconds emitted per second after first chunk)
+- Throughput under concurrency
+"""
+
+from __future__ import annotations
+
+import argparse
+import asyncio
+import json
+import os
+import statistics
+import subprocess
+import sys
+import threading
+import time
+from dataclasses import dataclass, field
+from typing import Any, Dict, List, Optional
+
+
+# Path setup (same pattern as local_server.py)
+REPO_ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+EXTERNAL_DIR = os.path.join(REPO_ROOT, "external")
+for path in [os.path.join(EXTERNAL_DIR, "sparktts"), os.path.join(EXTERNAL_DIR, "AP-BWE"), REPO_ROOT]:
+    if path not in sys.path:
+        sys.path.insert(0, path)
+
+os.environ["AUTH_BYPASS_MODE"] = "true"
+
+TEST_TEXTS = {
+    "short": "Hello, this is a quick streaming benchmark for voice agents.",
+    "medium": (
+        "This streaming benchmark sentence is designed to evaluate first audio latency, "
+        "chunk cadence, and sustained emission speed under concurrent request load."
+    ),
+    "long": (
+        "In a bustling city, a narrator collected voice notes from people walking through markets, "
+        "subways, and parks. The stream needed to start quickly, stay stable, and deliver clear speech "
+        "without stalls as many listeners connected at once."
+    ),
+}
+
+SPEAKERS = ["lipakshi", "vardan", "reet", "Nandini", "krishna", "anika"]
+
+
+@dataclass
+class GPUSnapshot:
+    timestamp: float
+    memory_used_mb: float
+    memory_total_mb: float
+    gpu_utilization_pct: float
+
+
+class GPUMonitor:
+    def __init__(self, interval_seconds: float = 0.5):
+        self.interval = interval_seconds
+        self.snapshots: List[GPUSnapshot] = []
+        self._running = False
+        self._thread: Optional[threading.Thread] = None
+
+    def start(self) -> None:
+        self._running = True
+        self._thread = threading.Thread(target=self._poll_loop, daemon=True)
+        self._thread.start()
+
+    def stop(self) -> None:
+        self._running = False
+        if self._thread:
+            self._thread.join(timeout=3)
+
+    def _poll_loop(self) -> None:
+        while self._running:
+            try:
+                res = subprocess.run(
+                    [
+                        "nvidia-smi",
+                        "--query-gpu=memory.used,memory.total,utilization.gpu",
+                        "--format=csv,noheader,nounits",
+                    ],
+                    capture_output=True,
+                    text=True,
+                    timeout=5,
+                )
+                if res.returncode == 0 and res.stdout:
+                    parts = [p.strip() for p in res.stdout.strip().split(",")]
+                    if len(parts) >= 3:
+                        self.snapshots.append(
+                            GPUSnapshot(
+                                timestamp=time.time(),
+                                memory_used_mb=float(parts[0]),
+                                memory_total_mb=float(parts[1]),
+                                gpu_utilization_pct=float(parts[2]),
+                            )
+                        )
+            except Exception:
+                pass
+            time.sleep(self.interval)
+
+    def summary(self) -> Dict[str, Any]:
+        if not self.snapshots:
+            return {"samples": 0, "error": "no GPU samples captured"}
+        mem = [s.memory_used_mb for s in self.snapshots]
+        util = [s.gpu_utilization_pct for s in self.snapshots]
+        return {
+            "samples": len(self.snapshots),
+            "memory_mb": {
+                "min": min(mem),
+                "avg": statistics.mean(mem),
+                "max": max(mem),
+                "total": self.snapshots[0].memory_total_mb,
+            },
+            "gpu_util_pct": {
+                "min": min(util),
+                "avg": statistics.mean(util),
+                "max": max(util),
+            },
+        }
+
+
+@dataclass
+class RequestResult:
+    success: bool
+    latency_ms: float
+    ttfb_ms: float
+    chunks_sent: int
+    total_bytes: int
+    audio_seconds: float
+    speaker: str
+    error: Optional[str] = None
+    inter_chunk_ms: List[float] = field(default_factory=list)
+    tail_emit_speed_x: float = 0.0
+    final_metrics: Dict[str, Any] = field(default_factory=dict)
+
+
+@dataclass
+class LevelResult:
+    concurrency: int
+    total_requests: int
+    results: List[RequestResult] = field(default_factory=list)
+    wall_time_s: float = 0.0
+    gpu_summary: Dict[str, Any] = field(default_factory=dict)
+
+    @property
+    def successes(self) -> List[RequestResult]:
+        return [r for r in self.results if r.success]
+
+    @property
+    def failures(self) -> List[RequestResult]:
+        return [r for r in self.results if not r.success]
+
+    @property
+    def success_rate(self) -> float:
+        return len(self.successes) / len(self.results) if self.results else 0.0
+
+    @property
+    def throughput_rps(self) -> float:
+        return self.total_requests / self.wall_time_s if self.wall_time_s > 0 else 0.0
+
+    def _pct(self, vals: List[float], p: int) -> float:
+        if not vals:
+            return 0.0
+        s = sorted(vals)
+        idx = int((len(s) - 1) * (p / 100))
+        return s[idx]
+
+    def _stats(self, vals: List[float]) -> Dict[str, float]:
+        if not vals:
+            return {}
+        return {
+            "avg": statistics.mean(vals),
+            "p50": self._pct(vals, 50),
+            "p95": self._pct(vals, 95),
+            "max": max(vals),
+        }
+
+    def latency_stats(self) -> Dict[str, float]:
+        return self._stats([r.latency_ms for r in self.successes])
+
+    def ttfb_stats(self) -> Dict[str, float]:
+        return self._stats([r.ttfb_ms for r in self.successes if r.ttfb_ms > 0])
+
+    def chunk_count_stats(self) -> Dict[str, float]:
+        return self._stats([float(r.chunks_sent) for r in self.successes])
+
+    def emit_speed_stats(self) -> Dict[str, float]:
+        return self._stats([r.tail_emit_speed_x for r in self.successes if r.tail_emit_speed_x > 0])
+
+    def inter_chunk_stats(self) -> Dict[str, float]:
+        vals: List[float] = []
+        for r in self.successes:
+            vals.extend(r.inter_chunk_ms)
+        return self._stats(vals)
+
+
+def init_runtime(model_path: str, gpu_mem: float, stream_output_kind: str) -> None:
+    from veena3modal.local_server import resolve_model_paths
+    from veena3modal.services.tts_runtime import initialize_runtime, is_initialized
+
+    os.environ["VEENA3_STREAM_OUTPUT_KIND"] = stream_output_kind
+
+    llm_path, bicodec_path = resolve_model_paths(model_path)
+    if not is_initialized():
+        initialize_runtime(
+            model_path=llm_path,
+            bicodec_path=bicodec_path,
+            device="cuda",
+            gpu_memory_utilization=gpu_mem,
+            precompute_speaker_globals=False,
+        )
+
+
+async def run_one_request(
+    text: str,
+    speaker: str,
+    max_tokens: int,
+    timeout_s: float,
+    enable_chunking: bool,
+) -> RequestResult:
+    from veena3modal.services import tts_runtime
+
+    t0 = time.perf_counter()
+    first_chunk_t: Optional[float] = None
+    prev_chunk_t: Optional[float] = None
+    inter_chunk_ms: List[float] = []
+    chunks = 0
+    total_bytes = 0
+    final_metrics: Dict[str, Any] = {}
+    try:
+        gen = tts_runtime.generate_speech_streaming(
+            text=text,
+            speaker=speaker,
+            max_tokens=max_tokens,
+            enable_chunking=enable_chunking,
+        )
+        async for audio_chunk, metrics in asyncio.wait_for(_iterate_stream(gen), timeout=timeout_s):
+            now = time.perf_counter()
+            if first_chunk_t is None:
+                first_chunk_t = now
+            if prev_chunk_t is not None:
+                inter_chunk_ms.append((now - prev_chunk_t) * 1000.0)
+            prev_chunk_t = now
+            chunks += 1
+            total_bytes += len(audio_chunk)
+            final_metrics = dict(metrics or {})
+        end = time.perf_counter()
+        latency_ms = (end - t0) * 1000.0
+        ttfb_ms = ((first_chunk_t - t0) * 1000.0) if first_chunk_t is not None else 0.0
+        audio_seconds = float(final_metrics.get("audio_duration_seconds", 0.0) or 0.0)
+        tail_s = max(1e-6, (latency_ms - ttfb_ms) / 1000.0)
+        tail_emit_speed_x = audio_seconds / tail_s if audio_seconds > 0 else 0.0
+        return RequestResult(
+            success=True,
+            latency_ms=latency_ms,
+            ttfb_ms=ttfb_ms,
+            chunks_sent=chunks,
+            total_bytes=total_bytes,
+            audio_seconds=audio_seconds,
+            speaker=speaker,
+            inter_chunk_ms=inter_chunk_ms,
+            tail_emit_speed_x=tail_emit_speed_x,
+            final_metrics=final_metrics,
+        )
+    except asyncio.TimeoutError:
+        return RequestResult(
+            success=False,
+            latency_ms=(time.perf_counter() - t0) * 1000.0,
+            ttfb_ms=0.0,
+            chunks_sent=chunks,
+            total_bytes=total_bytes,
+            audio_seconds=0.0,
+            speaker=speaker,
+            error="timeout",
+            inter_chunk_ms=inter_chunk_ms,
+            final_metrics=final_metrics,
+        )
+    except Exception as exc:
+        return RequestResult(
+            success=False,
+            latency_ms=(time.perf_counter() - t0) * 1000.0,
+            ttfb_ms=0.0,
+            chunks_sent=chunks,
+            total_bytes=total_bytes,
+            audio_seconds=0.0,
+            speaker=speaker,
+            error=str(exc)[:200],
+            inter_chunk_ms=inter_chunk_ms,
+            final_metrics=final_metrics,
+        )
+
+
+async def _iterate_stream(gen):
+    async for item in gen:
+        yield item
+
+
+async def run_level(
+    concurrency: int,
+    total_requests: int,
+    text: str,
+    max_tokens: int,
+    timeout_s: float,
+    enable_chunking: bool,
+    gpu_monitor: GPUMonitor,
+) -> LevelResult:
+    level = LevelResult(concurrency=concurrency, total_requests=total_requests)
+    semaphore = asyncio.Semaphore(concurrency)
+
+    async def wrapped(i: int) -> RequestResult:
+        async with semaphore:
+            speaker = SPEAKERS[i % len(SPEAKERS)]
+            return await run_one_request(
+                text=text,
+                speaker=speaker,
+                max_tokens=max_tokens,
+                timeout_s=timeout_s,
+                enable_chunking=enable_chunking,
+            )
+
+    gpu_monitor.snapshots.clear()
+    t0 = time.perf_counter()
+    level.results = await asyncio.gather(*[wrapped(i) for i in range(total_requests)])
+    level.wall_time_s = time.perf_counter() - t0
+    level.gpu_summary = gpu_monitor.summary()
+    return level
+
+
+def print_level(level: LevelResult) -> None:
+    print(f"\n=== streaming concurrency={level.concurrency} requests={level.total_requests} ===")
+    print(f"success={len(level.successes)}/{level.total_requests} ({level.success_rate:.0%})")
+    print(f"wall={level.wall_time_s:.2f}s throughput={level.throughput_rps:.2f} req/s")
+    lat = level.latency_stats()
+    if lat:
+        print(f"latency_ms avg={lat['avg']:.0f} p50={lat['p50']:.0f} p95={lat['p95']:.0f} max={lat['max']:.0f}")
+    ttfb = level.ttfb_stats()
+    if ttfb:
+        print(f"ttfb_ms avg={ttfb['avg']:.0f} p50={ttfb['p50']:.0f} p95={ttfb['p95']:.0f} max={ttfb['max']:.0f}")
+    cc = level.chunk_count_stats()
+    if cc:
+        print(f"chunks_per_req avg={cc['avg']:.1f} p50={cc['p50']:.1f} p95={cc['p95']:.1f} max={cc['max']:.1f}")
+    ic = level.inter_chunk_stats()
+    if ic:
+        print(
+            "inter_chunk_ms avg={:.1f} p50={:.1f} p95={:.1f} max={:.1f}".format(
+                ic["avg"], ic["p50"], ic["p95"], ic["max"]
+            )
+        )
+    es = level.emit_speed_stats()
+    if es:
+        print(
+            "tail_emit_speed_x avg={:.2f} p50={:.2f} p95={:.2f} max={:.2f}".format(
+                es["avg"], es["p50"], es["p95"], es["max"]
+            )
+        )
+    if "memory_mb" in level.gpu_summary:
+        g = level.gpu_summary
+        print(
+            "gpu: util_avg={:.0f}% util_max={:.0f}% mem_avg={:.0f}MB mem_max={:.0f}MB".format(
+                g["gpu_util_pct"]["avg"],
+                g["gpu_util_pct"]["max"],
+                g["memory_mb"]["avg"],
+                g["memory_mb"]["max"],
+            )
+        )
+
+
+def save_results(path: str, levels: List[LevelResult], args: argparse.Namespace) -> None:
+    payload = {
+        "config": {
+            "levels": args.levels,
+            "text": args.text,
+            "max_tokens": args.max_tokens,
+            "timeout_s": args.timeout,
+            "requests_multiplier": args.requests_multiplier,
+            "min_requests": args.min_requests,
+            "gpu_memory": args.gpu_memory,
+            "stream_output_kind": args.stream_output_kind,
+            "chunking": args.chunking,
+            "timestamp": int(time.time()),
+        },
+        "levels": [],
+    }
+    for lvl in levels:
+        payload["levels"].append(
+            {
+                "concurrency": lvl.concurrency,
+                "total_requests": lvl.total_requests,
+                "success_rate": lvl.success_rate,
+                "throughput_rps": lvl.throughput_rps,
+                "wall_time_s": lvl.wall_time_s,
+                "latency_ms": lvl.latency_stats(),
+                "ttfb_ms": lvl.ttfb_stats(),
+                "chunks_per_req": lvl.chunk_count_stats(),
+                "inter_chunk_ms": lvl.inter_chunk_stats(),
+                "tail_emit_speed_x": lvl.emit_speed_stats(),
+                "gpu": lvl.gpu_summary,
+                "requests": [
+                    {
+                        "success": r.success,
+                        "latency_ms": r.latency_ms,
+                        "ttfb_ms": r.ttfb_ms,
+                        "chunks_sent": r.chunks_sent,
+                        "total_bytes": r.total_bytes,
+                        "audio_seconds": r.audio_seconds,
+                        "speaker": r.speaker,
+                        "error": r.error,
+                        "inter_chunk_ms": r.inter_chunk_ms,
+                        "tail_emit_speed_x": r.tail_emit_speed_x,
+                        "final_metrics": r.final_metrics,
+                    }
+                    for r in lvl.results
+                ],
+            }
+        )
+    with open(path, "w", encoding="utf-8") as fd:
+        json.dump(payload, fd, indent=2)
+
+
+async def main_async(args: argparse.Namespace) -> None:
+    text = TEST_TEXTS[args.text]
+    levels = [int(x.strip()) for x in args.levels.split(",") if x.strip()]
+    gpu_monitor = GPUMonitor(interval_seconds=0.5)
+    gpu_monitor.start()
+
+    # warmup
+    await run_one_request(
+        text=text,
+        speaker=SPEAKERS[0],
+        max_tokens=args.max_tokens,
+        timeout_s=args.timeout,
+        enable_chunking=args.chunking,
+    )
+
+    results: List[LevelResult] = []
+    for idx, c in enumerate(levels):
+        n = max(args.min_requests, c * args.requests_multiplier)
+        level = await run_level(
+            concurrency=c,
+            total_requests=n,
+            text=text,
+            max_tokens=args.max_tokens,
+            timeout_s=args.timeout,
+            enable_chunking=args.chunking,
+            gpu_monitor=gpu_monitor,
+        )
+        print_level(level)
+        results.append(level)
+        if idx < len(levels) - 1:
+            await asyncio.sleep(2)
+
+    gpu_monitor.stop()
+    save_results(args.output, results, args)
+    print(f"\nresults saved: {args.output}")
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(description="Direct runtime streaming stress benchmark")
+    parser.add_argument("--levels", default="16,32,64", help="Comma-separated concurrency levels")
+    parser.add_argument("--text", choices=["short", "medium", "long"], default="short")
+    parser.add_argument("--max-tokens", type=int, default=512, help="max_tokens sent to streaming generation")
+    parser.add_argument("--timeout", type=float, default=180.0, help="Per-request timeout seconds")
+    parser.add_argument("--requests-multiplier", type=int, default=2, help="Requests per level ~= concurrency * multiplier")
+    parser.add_argument("--min-requests", type=int, default=8)
+    parser.add_argument("--chunking", action="store_true", default=False, help="Enable long-text chunking path")
+    parser.add_argument("--gpu-memory", type=float, default=0.25)
+    parser.add_argument("--stream-output-kind", choices=["delta", "cumulative"], default="delta")
+    parser.add_argument("--model-path", type=str, default="")
+    parser.add_argument("--output", default="stress_streaming_runtime.json")
+    args = parser.parse_args()
+
+    from veena3modal.local_server import DEFAULT_LOCAL_MODEL_DIR
+
+    model_path = args.model_path or DEFAULT_LOCAL_MODEL_DIR
+    init_runtime(model_path=model_path, gpu_mem=args.gpu_memory, stream_output_kind=args.stream_output_kind)
+    asyncio.run(main_async(args))
+
+
+if __name__ == "__main__":
+    main()
+
*** End Patch


Output: {"output":"Success. Updated the following files:\nA /home/ubuntu/veenaModal/scripts/stress_test_streaming_runtime.py\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


**Fixing async generator timeout handling**


Input:
*** Begin Patch
*** Update File: /home/ubuntu/veenaModal/scripts/stress_test_streaming_runtime.py
@@
 async def run_one_request(
@@
 ) -> RequestResult:
     from veena3modal.services import tts_runtime

     t0 = time.perf_counter()
@@
     chunks = 0
     total_bytes = 0
     final_metrics: Dict[str, Any] = {}
+
+    async def _consume_stream() -> None:
+        nonlocal first_chunk_t, prev_chunk_t, chunks, total_bytes, final_metrics
+        gen = tts_runtime.generate_speech_streaming(
+            text=text,
+            speaker=speaker,
+            max_tokens=max_tokens,
+            enable_chunking=enable_chunking,
+        )
+        async for audio_chunk, metrics in gen:
+            now = time.perf_counter()
+            if first_chunk_t is None:
+                first_chunk_t = now
+            if prev_chunk_t is not None:
+                inter_chunk_ms.append((now - prev_chunk_t) * 1000.0)
+            prev_chunk_t = now
+            chunks += 1
+            total_bytes += len(audio_chunk)
+            final_metrics = dict(metrics or {})
+
     try:
-        gen = tts_runtime.generate_speech_streaming(
-            text=text,
-            speaker=speaker,
-            max_tokens=max_tokens,
-            enable_chunking=enable_chunking,
-        )
-        async for audio_chunk, metrics in asyncio.wait_for(_iterate_stream(gen), timeout=timeout_s):
-            now = time.perf_counter()
-            if first_chunk_t is None:
-                first_chunk_t = now
-            if prev_chunk_t is not None:
-                inter_chunk_ms.append((now - prev_chunk_t) * 1000.0)
-            prev_chunk_t = now
-            chunks += 1
-            total_bytes += len(audio_chunk)
-            final_metrics = dict(metrics or {})
+        await asyncio.wait_for(_consume_stream(), timeout=timeout_s)
         end = time.perf_counter()
         latency_ms = (end - t0) * 1000.0
@@
         )

-async def _iterate_stream(gen):
-    async for item in gen:
-        yield item
-
-
 async def run_level(
*** End Patch


Output: {"output":"Success. Updated the following files:\nM /home/ubuntu/veenaModal/scripts/stress_test_streaming_runtime.py\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


Summary: auto


**Checking compile readiness**


cd /home/ubuntu/veenaModal && python3 -m py_compile veena3modal/core/streaming_pipeline.py scripts/stress_test_streaming_runtime.py


cd /home/ubuntu/veenaModal && rg -n "VEENA3_STREAM_OUTPUT_KIND|_STREAM_USE_DELTA_OUTPUT|output_kind=\(" veena3modal/core/streaming_pipeline.py scripts/stress_test_streaming_runtime.py


Output:
Chunk ID: 538f16
Wall time: 0.0509 seconds
Process exited with code 0
Original token count: 0
Output:


Output:
Chunk ID: 40baa0
Wall time: 0.0515 seconds
Process exited with code 0
Original token count: 276
Output:
scripts/stress_test_streaming_runtime.py:206:    os.environ["VEENA3_STREAM_OUTPUT_KIND"] = stream_output_kind
veena3modal/core/streaming_pipeline.py:50:_STREAM_OUTPUT_KIND_RAW = os.environ.get("VEENA3_STREAM_OUTPUT_KIND", "delta").strip().lower()
veena3modal/core/streaming_pipeline.py:51:_STREAM_USE_DELTA_OUTPUT = _STREAM_OUTPUT_KIND_RAW in {"delta", "d", "1", "true", "yes", "on"}
veena3modal/core/streaming_pipeline.py:276:            output_kind=(RequestOutputKind.DELTA if _STREAM_USE_DELTA_OUTPUT else RequestOutputKind.CUMULATIVE),
veena3modal/core/streaming_pipeline.py:338:            if _STREAM_USE_DELTA_OUTPUT:
veena3modal/core/streaming_pipeline.py:537:            output_kind=(RequestOutputKind.DELTA if _STREAM_USE_DELTA_OUTPUT else RequestOutputKind.CUMULATIVE),
veena3modal/core/streaming_pipeline.py:576:            if _STREAM_USE_DELTA_OUTPUT:
veena3modal/core/streaming_pipeline.py:751:            output_kind=(RequestOutputKind.DELTA if _STREAM_USE_DELTA_OUTPUT else RequestOutputKind.CUMULATIVE),
veena3modal/core/streaming_pipeline.py:788:            if _STREAM_USE_DELTA_OUTPUT:


**Planning benchmark runs**


I’ve finished the streaming-path patch. I’m now running an A/B benchmark with identical streaming load, first `cumulative` then `delta`, so we can directly quantify whether vLLM can emit tokens/chunks faster in your true voice-agent path.


cd /home/ubuntu/veenaModal && python3 scripts/stress_test_streaming_runtime.py --levels 32,64,96 --text short --max-tokens 256 --timeout 180 --requests-multiplier 2 --min-requests 8 --gpu-memory 0.25 --stream-output-kind cumulative --output /tmp/stream_runtime_cumulative.json


Output:
Chunk ID: d0a7ea
Wall time: 1.0011 seconds
Process running with session ID 19349
Original token count: 0
Output:


Output:
Chunk ID: 721008
Wall time: 5.0007 seconds
Process running with session ID 19349
Original token count: 1362
Output:
[2026-02-18 00:55:08] INFO tts_runtime.py:183: Loading SparkTTS model with vLLM...
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
🚀 Initializing Spark TTS Model
📁 Model path: /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
🗣  Speakers: 12 predefined (lipakshi, vardan, reet, Nandini, krishna, anika, adarsh, Nilay, Aarvi, Asha, Bittu, Mira)
🎭 Emotions: 10 tags in [bracket] format

📝 Loading tokenizer from model...
✅ Tokenizer loaded: 165168 tokens

🔍 Validating 12 speakers for Spark TTS...
  lipakshi → <|speaker_0|>: 7 tokens
  vardan → <|speaker_1|>: 7 tokens
  reet → <|speaker_2|>: 7 tokens
  Nandini → <|speaker_3|>: 7 tokens
  krishna → <|speaker_4|>: 7 tokens
  anika → <|speaker_5|>: 7 tokens
  adarsh → <|speaker_6|>: 7 tokens
  Nilay → <|speaker_7|>: 7 tokens
  Aarvi → <|speaker_8|>: 7 tokens
  Asha → <|speaker_9|>: 7 tokens
  Bittu → <|speaker_10|>: 8 tokens
  Mira → <|speaker_11|>: 8 tokens
✅ All 12 speakers validated with mapping

🔍 Validating 10 emotion tags for Spark TTS...
✅ All 10 emotion tags validated

📋 Sample emotion tags:
  [angry]: 1 tokens
  [curious]: 1 tokens
  [excited]: 1 tokens
  [giggle]: 1 tokens
  [laughs harder]: 1 tokens
  ...

🔧 Initializing vLLM engine with optimizations...
  - dtype: bfloat16
  - max_model_len: 4096
  - gpu_memory_utilization: 0.25
  - enable_prefix_caching: True
  - enforce_eager (disable CUDA graphs): False
INFO 02-18 00:55:09 [model.py:541] Resolved architecture: Qwen2ForCausalLM
INFO 02-18 00:55:09 [model.py:1561] Using max model len 4096
INFO 02-18 00:55:09 [scheduler.py:226] Chunked prefill is enabled with max_num_batched_tokens=4096.
INFO 02-18 00:55:09 [vllm.py:624] Asynchronous scheduling is enabled.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(EngineCore_DP0 pid=4113112) INFO 02-18 00:55:10 [core.py:96] Initializing a V1 LLM engine (v0.15.1) with config: model='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', speculative_config=None, tokenizer='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [4096], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 256, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': True}, 'local_cache_dir': None, 'static_all_moe_layers': []}
(EngineCore_DP0 pid=4113112) INFO 02-18 00:55:12 [parallel_state.py:1212] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://216.81.248.184:58027 backend=nccl
(EngineCore_DP0 pid=4113112) INFO 02-18 00:55:12 [parallel_state.py:1423] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A


Output:
Chunk ID: 9bbda2
Wall time: 5.0005 seconds
Process running with session ID 19349
Original token count: 426
Output:
(EngineCore_DP0 pid=4113112) INFO 02-18 00:55:13 [gpu_model_runner.py:4033] Starting to load model /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM...
(EngineCore_DP0 pid=4113112) INFO 02-18 00:55:13 [cuda.py:364] Using FLASH_ATTN attention backend out of potential backends: ('FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION')
(EngineCore_DP0 pid=4113112) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
(EngineCore_DP0 pid=4113112) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
(EngineCore_DP0 pid=4113112)
Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
(EngineCore_DP0 pid=4113112)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:01<00:00,  1.28s/it]
(EngineCore_DP0 pid=4113112)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:01<00:00,  1.28s/it]
(EngineCore_DP0 pid=4113112)
(EngineCore_DP0 pid=4113112) INFO 02-18 00:55:15 [default_loader.py:291] Loading weights took 1.36 seconds
(EngineCore_DP0 pid=4113112) INFO 02-18 00:55:15 [gpu_model_runner.py:4130] Model loading took 0.95 GiB memory and 1.885810 seconds
(EngineCore_DP0 pid=4113112) INFO 02-18 00:55:20 [backends.py:812] Using cache directory: /home/ubuntu/.cache/vllm/torch_compile_cache/f3ff13511c/rank_0_0/backbone for vLLM's torch.compile
(EngineCore_DP0 pid=4113112) INFO 02-18 00:55:20 [backends.py:872] Dynamo bytecode transform time: 4.22 s


Output:
Chunk ID: 827d99
Wall time: 5.0012 seconds
Process running with session ID 19349
Original token count: 787
Output:
(EngineCore_DP0 pid=4113112) INFO 02-18 00:55:23 [backends.py:267] Directly load the compiled graph(s) for compile range (1, 4096) from the cache, took 0.644 s
(EngineCore_DP0 pid=4113112) INFO 02-18 00:55:23 [monitor.py:34] torch.compile takes 4.87 s in total
(EngineCore_DP0 pid=4113112) INFO 02-18 00:55:24 [gpu_worker.py:356] Available KV cache memory: 18.06 GiB
(EngineCore_DP0 pid=4113112) INFO 02-18 00:55:24 [kv_cache_utils.py:1307] GPU KV cache size: 1,577,712 tokens
(EngineCore_DP0 pid=4113112) INFO 02-18 00:55:24 [kv_cache_utils.py:1312] Maximum concurrency for 4,096 tokens per request: 385.18x
(EngineCore_DP0 pid=4113112)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   0%|          | 0/35 [00:00<?, ?it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  11%|█▏        | 4/35 [00:00<00:00, 33.34it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  23%|██▎       | 8/35 [00:00<00:00, 36.14it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  34%|███▍      | 12/35 [00:00<00:00, 37.12it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  46%|████▌     | 16/35 [00:00<00:00, 37.45it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  57%|█████▋    | 20/35 [00:00<00:00, 37.55it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  69%|██████▊   | 24/35 [00:00<00:00, 37.73it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  80%|████████  | 28/35 [00:00<00:00, 37.44it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  91%|█████████▏| 32/35 [00:00<00:00, 35.75it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 35/35 [00:00<00:00, 35.82it/s]
(EngineCore_DP0 pid=4113112)
Capturing CUDA graphs (decode, FULL):   0%|          | 0/19 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL):  16%|█▌        | 3/19 [00:00<00:00, 29.94it/s]
Capturing CUDA graphs (decode, FULL):  37%|███▋      | 7/19 [00:00<00:00, 34.09it/s]
Capturing CUDA graphs (decode, FULL):  58%|█████▊    | 11/19 [00:00<00:00, 35.41it/s]
Capturing CUDA graphs (decode, FULL):  79%|███████▉  | 15/19 [00:00<00:00, 35.67it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 36.27it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 35.47it/s]
(EngineCore_DP0 pid=4113112) INFO 02-18 00:55:26 [gpu_model_runner.py:5063] Graph capturing finished in 2 secs, took 0.28 GiB
(EngineCore_DP0 pid=4113112) INFO 02-18 00:55:26 [core.py:272] init engine (profile, create kv cache, warmup model) took 10.73 seconds
[2026-02-18 00:55:28] INFO tts_runtime.py:192: Initializing prompt builder...
[2026-02-18 00:55:28] INFO tts_runtime.py:199: Initializing BiCodec decoder...
/home/ubuntu/.local/lib/python3.10/site-packages/torch/nn/utils/weight_norm.py:144: FutureWarning: `torch.nn.utils.weight_norm` is deprecated in favor of `torch.nn.utils.parametrizations.weight_norm`.
  WeightNorm.apply(module, name, dim)


Output:
Chunk ID: aee09b
Wall time: 5.0007 seconds
Process running with session ID 19349
Original token count: 127
Output:
[2026-02-18 00:55:32] INFO tts_runtime.py:206: Initializing TTS pipeline...
[2026-02-18 00:55:32] INFO tts_runtime.py:217: Initializing streaming pipeline...
[2026-02-18 00:55:33] INFO long_text_processor.py:112: LongTextProcessor initialized
[2026-02-18 00:55:33] INFO tts_runtime.py:274: ✅ TTS runtime initialized in 30105ms
[2026-02-18 00:55:33] INFO tts_runtime.py:275:    Model version: LLM
[2026-02-18 00:55:33] INFO tts_runtime.py:286: Speaker globals cache startup precompute skipped (disabled)


Output:
Chunk ID: c03e93
Wall time: 5.0015 seconds
Process running with session ID 19349
Original token count: 0
Output:


The cumulative-mode streaming benchmark is now in request execution phase. I’ll capture full metrics across 32/64/96 and then run the exact same load with `delta` for a clean A/B.


Output:
Chunk ID: ff8065
Wall time: 5.0015 seconds
Process running with session ID 19349
Original token count: 0
Output:


Output:
Chunk ID: 047e55
Wall time: 2.3527 seconds
Process exited with code 0
Original token count: 467
Output:
✅ vLLM engine initialized successfully!

🎉 Spark TTS Model ready for inference!

🎵 Loading BiCodec audio tokenizer from /home/ubuntu/veenaModal/models/spark_tts_4speaker...
   Device: cuda
Missing tensor: mel_transformer.spectrogram.window
Missing tensor: mel_transformer.mel_scale.fb
✅ BiCodec decoder initialized (sample rate: 16000Hz)
🚀 SparkTTSPipeline initialized
🌊 Veena3SlidingWindowPipeline initialized (sliding window: 28 tokens)

=== streaming concurrency=32 requests=64 ===
success=64/64 (100%)
wall=5.17s throughput=12.37 req/s
latency_ms avg=2552 p50=2443 p95=2736 max=2750
ttfb_ms avg=965 p50=892 p95=1262 max=1654
chunks_per_req avg=4.4 p50=4.0 p95=5.0 max=6.0
inter_chunk_ms avg=411.5 p50=423.9 p95=1023.0 max=1149.8
tail_emit_speed_x avg=2.25 p50=2.08 p95=2.81 max=5.25
gpu: util_avg=75% util_max=99% mem_avg=24796MB mem_max=25125MB

=== streaming concurrency=64 requests=128 ===
success=128/128 (100%)
wall=7.31s throughput=17.51 req/s
latency_ms avg=3650 p50=3637 p95=3674 max=3765
ttfb_ms avg=1380 p50=1347 p95=1923 max=2060
chunks_per_req avg=3.8 p50=4.0 p95=4.0 max=4.0
inter_chunk_ms avg=781.3 p50=382.1 p95=1957.5 max=2100.9
tail_emit_speed_x avg=1.32 p50=1.15 p95=2.15 max=2.44
gpu: util_avg=69% util_max=98% mem_avg=25127MB mem_max=25127MB

=== streaming concurrency=96 requests=192 ===
success=192/192 (100%)
wall=9.65s throughput=19.89 req/s
latency_ms avg=4822 p50=4771 p95=4943 max=4945
ttfb_ms avg=1878 p50=1957 p95=2641 max=2694
chunks_per_req avg=3.5 p50=3.0 p95=4.0 max=4.0
inter_chunk_ms avg=1158.9 p50=1012.6 p95=2911.3 max=3129.9
tail_emit_speed_x avg=0.86 p50=0.79 p95=1.29 max=1.64
gpu: util_avg=61% util_max=96% mem_avg=25128MB mem_max=25129MB

results saved: /tmp/stream_runtime_cumulative.json
ERROR 02-18 00:56:01 [core_client.py:605] Engine core proc EngineCore_DP0 died unexpectedly, shutting down client.


**Running delta with same options**


cd /home/ubuntu/veenaModal && python3 scripts/stress_test_streaming_runtime.py --levels 32,64,96 --text short --max-tokens 256 --timeout 180 --requests-multiplier 2 --min-requests 8 --gpu-memory 0.25 --stream-output-kind delta --output /tmp/stream_runtime_delta.json


Output:
Chunk ID: 2d5ab6
Wall time: 1.0014 seconds
Process running with session ID 20059
Original token count: 0
Output:


Output:
Chunk ID: da5829
Wall time: 5.0015 seconds
Process running with session ID 20059
Original token count: 1277
Output:
[2026-02-18 00:56:13] INFO tts_runtime.py:183: Loading SparkTTS model with vLLM...
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
🚀 Initializing Spark TTS Model
📁 Model path: /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
🗣  Speakers: 12 predefined (lipakshi, vardan, reet, Nandini, krishna, anika, adarsh, Nilay, Aarvi, Asha, Bittu, Mira)
🎭 Emotions: 10 tags in [bracket] format

📝 Loading tokenizer from model...
✅ Tokenizer loaded: 165168 tokens

🔍 Validating 12 speakers for Spark TTS...
  lipakshi → <|speaker_0|>: 7 tokens
  vardan → <|speaker_1|>: 7 tokens
  reet → <|speaker_2|>: 7 tokens
  Nandini → <|speaker_3|>: 7 tokens
  krishna → <|speaker_4|>: 7 tokens
  anika → <|speaker_5|>: 7 tokens
  adarsh → <|speaker_6|>: 7 tokens
  Nilay → <|speaker_7|>: 7 tokens
  Aarvi → <|speaker_8|>: 7 tokens
  Asha → <|speaker_9|>: 7 tokens
  Bittu → <|speaker_10|>: 8 tokens
  Mira → <|speaker_11|>: 8 tokens
✅ All 12 speakers validated with mapping

🔍 Validating 10 emotion tags for Spark TTS...
✅ All 10 emotion tags validated

📋 Sample emotion tags:
  [angry]: 1 tokens
  [curious]: 1 tokens
  [excited]: 1 tokens
  [giggle]: 1 tokens
  [laughs harder]: 1 tokens
  ...

🔧 Initializing vLLM engine with optimizations...
  - dtype: bfloat16
  - max_model_len: 4096
  - gpu_memory_utilization: 0.25
  - enable_prefix_caching: True
  - enforce_eager (disable CUDA graphs): False
INFO 02-18 00:56:14 [model.py:541] Resolved architecture: Qwen2ForCausalLM
INFO 02-18 00:56:14 [model.py:1561] Using max model len 4096
INFO 02-18 00:56:14 [scheduler.py:226] Chunked prefill is enabled with max_num_batched_tokens=4096.
INFO 02-18 00:56:14 [vllm.py:624] Asynchronous scheduling is enabled.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(EngineCore_DP0 pid=4113960) INFO 02-18 00:56:15 [core.py:96] Initializing a V1 LLM engine (v0.15.1) with config: model='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', speculative_config=None, tokenizer='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [4096], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 256, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': True}, 'local_cache_dir': None, 'static_all_moe_layers': []}


Output:
Chunk ID: 9387b6
Wall time: 5.0015 seconds
Process running with session ID 20059
Original token count: 511
Output:
(EngineCore_DP0 pid=4113960) INFO 02-18 00:56:17 [parallel_state.py:1212] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://216.81.248.184:40565 backend=nccl
(EngineCore_DP0 pid=4113960) INFO 02-18 00:56:17 [parallel_state.py:1423] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
(EngineCore_DP0 pid=4113960) INFO 02-18 00:56:18 [gpu_model_runner.py:4033] Starting to load model /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM...
(EngineCore_DP0 pid=4113960) INFO 02-18 00:56:18 [cuda.py:364] Using FLASH_ATTN attention backend out of potential backends: ('FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION')
(EngineCore_DP0 pid=4113960) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
(EngineCore_DP0 pid=4113960) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
(EngineCore_DP0 pid=4113960)
Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
(EngineCore_DP0 pid=4113960)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.33it/s]
(EngineCore_DP0 pid=4113960)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.33it/s]
(EngineCore_DP0 pid=4113960)
(EngineCore_DP0 pid=4113960) INFO 02-18 00:56:19 [default_loader.py:291] Loading weights took 0.82 seconds
(EngineCore_DP0 pid=4113960) INFO 02-18 00:56:20 [gpu_model_runner.py:4130] Model loading took 0.95 GiB memory and 1.336294 seconds
(EngineCore_DP0 pid=4113960) INFO 02-18 00:56:24 [backends.py:812] Using cache directory: /home/ubuntu/.cache/vllm/torch_compile_cache/f3ff13511c/rank_0_0/backbone for vLLM's torch.compile
(EngineCore_DP0 pid=4113960) INFO 02-18 00:56:24 [backends.py:872] Dynamo bytecode transform time: 4.21 s


Output:
Chunk ID: 48c391
Wall time: 5.0017 seconds
Process running with session ID 20059
Original token count: 762
Output:
(EngineCore_DP0 pid=4113960) INFO 02-18 00:56:27 [backends.py:267] Directly load the compiled graph(s) for compile range (1, 4096) from the cache, took 0.647 s
(EngineCore_DP0 pid=4113960) INFO 02-18 00:56:27 [monitor.py:34] torch.compile takes 4.86 s in total
(EngineCore_DP0 pid=4113960) INFO 02-18 00:56:28 [gpu_worker.py:356] Available KV cache memory: 18.06 GiB
(EngineCore_DP0 pid=4113960) INFO 02-18 00:56:28 [kv_cache_utils.py:1307] GPU KV cache size: 1,577,712 tokens
(EngineCore_DP0 pid=4113960) INFO 02-18 00:56:28 [kv_cache_utils.py:1312] Maximum concurrency for 4,096 tokens per request: 385.18x
(EngineCore_DP0 pid=4113960)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   0%|          | 0/35 [00:00<?, ?it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  11%|█▏        | 4/35 [00:00<00:00, 32.64it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  23%|██▎       | 8/35 [00:00<00:00, 34.18it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  34%|███▍      | 12/35 [00:00<00:00, 34.40it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  46%|████▌     | 16/35 [00:00<00:00, 34.69it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  57%|█████▋    | 20/35 [00:00<00:00, 35.47it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  69%|██████▊   | 24/35 [00:00<00:00, 35.74it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  80%|████████  | 28/35 [00:00<00:00, 35.82it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  91%|█████████▏| 32/35 [00:00<00:00, 34.57it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 35/35 [00:01<00:00, 34.29it/s]
(EngineCore_DP0 pid=4113960)
Capturing CUDA graphs (decode, FULL):   0%|          | 0/19 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL):  21%|██        | 4/19 [00:00<00:00, 31.59it/s]
Capturing CUDA graphs (decode, FULL):  42%|████▏     | 8/19 [00:00<00:00, 34.30it/s]
Capturing CUDA graphs (decode, FULL):  63%|██████▎   | 12/19 [00:00<00:00, 35.16it/s]
Capturing CUDA graphs (decode, FULL):  84%|████████▍ | 16/19 [00:00<00:00, 35.48it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 35.20it/s]
(EngineCore_DP0 pid=4113960) INFO 02-18 00:56:31 [gpu_model_runner.py:5063] Graph capturing finished in 2 secs, took 0.28 GiB
(EngineCore_DP0 pid=4113960) INFO 02-18 00:56:31 [core.py:272] init engine (profile, create kv cache, warmup model) took 10.78 seconds
[2026-02-18 00:56:32] INFO tts_runtime.py:192: Initializing prompt builder...
[2026-02-18 00:56:32] INFO tts_runtime.py:199: Initializing BiCodec decoder...
/home/ubuntu/.local/lib/python3.10/site-packages/torch/nn/utils/weight_norm.py:144: FutureWarning: `torch.nn.utils.weight_norm` is deprecated in favor of `torch.nn.utils.parametrizations.weight_norm`.
  WeightNorm.apply(module, name, dim)


Output:
Chunk ID: 68b958
Wall time: 5.0017 seconds
Process running with session ID 20059
Original token count: 127
Output:
[2026-02-18 00:56:41] INFO tts_runtime.py:206: Initializing TTS pipeline...
[2026-02-18 00:56:41] INFO tts_runtime.py:217: Initializing streaming pipeline...
[2026-02-18 00:56:41] INFO long_text_processor.py:112: LongTextProcessor initialized
[2026-02-18 00:56:41] INFO tts_runtime.py:274: ✅ TTS runtime initialized in 33453ms
[2026-02-18 00:56:41] INFO tts_runtime.py:275:    Model version: LLM
[2026-02-18 00:56:41] INFO tts_runtime.py:286: Speaker globals cache startup precompute skipped (disabled)


Output:
Chunk ID: e0dea7
Wall time: 5.0014 seconds
Process running with session ID 20059
Original token count: 0
Output:


The delta-mode run is executing now. Once it completes I’ll produce a direct cumulative-vs-delta table for streaming throughput, TTFB, chunk cadence, and emission speed.


Output:
Chunk ID: d5dd97
Wall time: 5.0011 seconds
Process running with session ID 20059
Original token count: 0
Output: