Output:
Chunk ID: c22b0c
Wall time: 5.0010 seconds
Process running with session ID 63481
Original token count: 409
Output:

Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|█| 35/35 [00:00<00
(EngineCore_DP0 pid=3710572)
Capturing CUDA graphs (decode, FULL):   0%|              | 0/19 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL):  16%|▉     | 3/19 [00:00<00:00, 27.90it/s]
Capturing CUDA graphs (decode, FULL):  37%|██▏   | 7/19 [00:00<00:00, 32.90it/s]
Capturing CUDA graphs (decode, FULL):  58%|██▉  | 11/19 [00:00<00:00, 34.44it/s]
Capturing CUDA graphs (decode, FULL):  79%|███▉ | 15/19 [00:00<00:00, 34.92it/s]
Capturing CUDA graphs (decode, FULL): 100%|█████| 19/19 [00:00<00:00, 35.71it/s]
Capturing CUDA graphs (decode, FULL): 100%|█████| 19/19 [00:00<00:00, 34.66it/s]
(EngineCore_DP0 pid=3710572) INFO 02-17 00:30:35 [gpu_model_runner.py:5063] Graph capturing finished in 2 secs, took 0.28 GiB
(EngineCore_DP0 pid=3710572) INFO 02-17 00:30:35 [core.py:272] init engine (profile, create kv cache, warmup model) took 10.89 seconds
✅ vLLM engine initialized successfully!

🎉 Spark TTS Model ready for inference!

2026-02-17 00:30:36 | INFO     | veena3modal.services.tts_runtime | Initializing prompt builder...
2026-02-17 00:30:36 | INFO     | veena3modal.services.tts_runtime | Initializing BiCodec decoder...
🎵 Loading BiCodec audio tokenizer from /home/ubuntu/veenaModal/models/spark_tts_4speaker...
   Device: cuda
/home/ubuntu/.local/lib/python3.10/site-packages/torch/nn/utils/weight_norm.py:144: FutureWarning: `torch.nn.utils.weight_norm` is deprecated in favor of `torch.nn.utils.parametrizations.weight_norm`.
  WeightNorm.apply(module, name, dim)


# Collaboration Mode: Default

You are now in Default mode. Any previous instructions for other modes (e.g. Plan mode) are no longer active.

Your active mode changes only when new developer instructions with a different `<collaboration_mode>...</collaboration_mode>` change it; user requests or tool descriptions do not change mode by themselves. Known mode names are Default and Plan.

## request_user_input availability

The `request_user_input` tool is unavailable in Default mode. If you call it while in Default mode, it will return an error.

If a decision is necessary and cannot be discovered from local context, ask the user directly. However, in Default mode you should strongly prefer executing the user's request rather than stopping to ask questions.

Summary: auto

## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) When `SKILL.md` references relative paths (e.g., `scripts/foo.py`), resolve them relative to the skill directory listed above first, and only consider other paths if needed.
  3) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  4) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  5) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.


Output:
Chunk ID: 76b04a
Wall time: 5.0009 seconds
Process running with session ID 63481
Original token count: 196
Output:
Missing tensor: mel_transformer.spectrogram.window
Missing tensor: mel_transformer.mel_scale.fb
✅ BiCodec decoder initialized (sample rate: 16000Hz)
2026-02-17 00:30:46 | INFO     | veena3modal.services.tts_runtime | Initializing TTS pipeline...
🚀 SparkTTSPipeline initialized
2026-02-17 00:30:46 | INFO     | veena3modal.services.tts_runtime | Initializing streaming pipeline...
🌊 Veena3SlidingWindowPipeline initialized (sliding window: 28 tokens)
2026-02-17 00:30:46 | INFO     | veena3modal.services.tts_runtime | ✅ TTS runtime initialized in 34719ms
2026-02-17 00:30:46 | INFO     | veena3modal.services.tts_runtime |    Model version: LLM
2026-02-17 00:30:46 | INFO     | veena3modal.services.tts_runtime | Pre-computing global tokens for all speakers...


Summary: auto


Output:
Chunk ID: 6c9af3
Wall time: 5.0012 seconds
Process running with session ID 63481
Original token count: 793
Output:
2026-02-17 00:30:46 | INFO     | veena3modal.services.tts_runtime |   Cached globals for lipakshi
2026-02-17 00:30:47 | INFO     | veena3modal.services.tts_runtime |   Cached globals for vardan
2026-02-17 00:30:47 | INFO     | veena3modal.services.tts_runtime |   Cached globals for reet
2026-02-17 00:30:48 | INFO     | veena3modal.services.tts_runtime |   Cached globals for Nandini
2026-02-17 00:30:48 | INFO     | veena3modal.services.tts_runtime |   Cached globals for krishna
2026-02-17 00:30:48 | INFO     | veena3modal.services.tts_runtime |   Cached globals for anika
2026-02-17 00:30:49 | INFO     | veena3modal.services.tts_runtime |   Cached globals for adarsh
2026-02-17 00:30:49 | INFO     | veena3modal.services.tts_runtime |   Cached globals for Nilay
2026-02-17 00:30:50 | INFO     | veena3modal.services.tts_runtime |   Cached globals for Aarvi
2026-02-17 00:30:50 | INFO     | veena3modal.services.tts_runtime |   Cached globals for Asha
2026-02-17 00:30:50 | INFO     | veena3modal.services.tts_runtime |   Cached globals for Bittu
2026-02-17 00:30:51 | INFO     | veena3modal.services.tts_runtime |   Cached globals for Mira
2026-02-17 00:30:51 | INFO     | veena3modal.services.tts_runtime | Speaker globals cached: 12/12 speakers in 4886ms
2026-02-17 00:30:51 | INFO     | veena3.local | Runtime initialized in 39.6s
2026-02-17 00:30:51 | INFO     | veena3.local |   Model version: LLM
2026-02-17 00:30:51 | INFO     | veena3.local |   Streaming: available
2026-02-17 00:30:51 | INFO     | veena3.local |   Super-resolution: disabled
2026-02-17 00:30:51 | INFO     | veena3.local |
2026-02-17 00:30:51 | INFO     | veena3.local | ============================================================
2026-02-17 00:30:51 | INFO     | veena3.local | Veena3 TTS Local Server starting
2026-02-17 00:30:51 | INFO     | veena3.local |   URL: http://0.0.0.0:8000
2026-02-17 00:30:51 | INFO     | veena3.local |   Health: http://0.0.0.0:8000/v1/tts/health
2026-02-17 00:30:51 | INFO     | veena3.local |   Generate: POST http://0.0.0.0:8000/v1/tts/generate
2026-02-17 00:30:51 | INFO     | veena3.local |   Metrics: http://0.0.0.0:8000/v1/tts/metrics
2026-02-17 00:30:51 | INFO     | veena3.local |   Docs: http://0.0.0.0:8000/docs
2026-02-17 00:30:51 | INFO     | veena3.local | ============================================================
{"timestamp": "2026-02-17T00:30:51.339294+00:00", "level": "INFO", "logger": "veena3modal.api.websocket_handler", "message": "WebSocket route /v1/tts/ws registered"}
2026-02-17 00:30:51 | INFO     | veena3modal.api.websocket_handler | WebSocket route /v1/tts/ws registered
{"timestamp": "2026-02-17T00:30:51.339428+00:00", "level": "INFO", "logger": "veena3modal.api.fastapi_app", "message": "WebSocket support enabled at /v1/tts/ws"}
2026-02-17 00:30:51 | INFO     | veena3modal.api.fastapi_app | WebSocket support enabled at /v1/tts/ws
INFO:     Started server process [3710461]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)


curl -sS -m 45 -X POST http://localhost:8000/v1/tts/generate -H 'Content-Type: application/json' -d '{"text":"hello","speaker":"lipakshi","max_tokens":128,"format":"wav","chunking":false,"stream":false}' -D - -o /tmp/new.wav


Output:
Chunk ID: 037991
Wall time: 1.0009 seconds
Process running with session ID 56952
Original token count: 0
Output:


I’ve got the server running in a visible session. I’m issuing a single warm request now to capture where it blocks in real time from both HTTP client and server logs.


Output:
Chunk ID: 8348d7
Wall time: 5.0015 seconds
Process running with session ID 56952
Original token count: 0
Output:


Output:
Chunk ID: 23aba3
Wall time: 5.0015 seconds
Process running with session ID 63481
Original token count: 113
Output:
{"timestamp": "2026-02-17T00:30:54.164524+00:00", "level": "INFO", "logger": "veena3modal.tts", "message": "request_received", "request_id": "d786de14-a9ef-456b-9b2f-9435fa20c767", "event": "request_received", "text_length": 5, "speaker": "lipakshi", "stream": false, "format": "wav"}
2026-02-17 00:30:54 | INFO     | veena3modal.tts | request_received
2026-02-17 00:30:54 | INFO     | veena3modal.services.tts_runtime | tts_runtime_nonstream_start


Output:
Chunk ID: 33e0e7
Wall time: 5.0010 seconds
Process running with session ID 63481
Original token count: 0
Output:


Output:
Chunk ID: ab77c8
Wall time: 5.0011 seconds
Process running with session ID 56952
Original token count: 0
Output:


Output:
Chunk ID: 80049c
Wall time: 5.0016 seconds
Process running with session ID 56952
Original token count: 0
Output:


Output:
Chunk ID: aa645f
Wall time: 5.0020 seconds
Process running with session ID 63481
Original token count: 0
Output:


Output:
Chunk ID: f457f0
Wall time: 4.5541 seconds
Process exited with code 28
Original token count: 20
Output:
curl: (28) Operation timed out after 45000 milliseconds with 0 bytes received


curl -sS -m 45 -X POST http://localhost:8000/v1/tts/generate -H 'Content-Type: application/json' -d '{"text":"hello","speaker":"lipakshi","max_tokens":128,"format":"wav","chunking":false,"stream":true}' -D -


Output:
Chunk ID: 631ed9
Wall time: 1.0010 seconds
Process running with session ID 47025
Original token count: 70
Output:
HTTP/1.1 200 OK
date: Tue, 17 Feb 2026 00:31:42 GMT
server: uvicorn
x-request-id: 41a746b9-784e-4166-a18f-1e231ba10896
x-model-version: LLM
x-format: wav
x-sample-rate: 16000
x-stream: true
x-chunking-enabled: false
content-type: audio/wav
transfer-encoding: chunked


Output:
Chunk ID: 29d24f
Wall time: 5.0020 seconds
Process running with session ID 47025
Original token count: 0
Output:


Output:
Chunk ID: 517f7b
Wall time: 12.0012 seconds
Process running with session ID 47025
Original token count: 0
Output:


Output:
Chunk ID: ce3561
Wall time: 5.0005 seconds
Process running with session ID 63481
Original token count: 163
Output:
{"timestamp": "2026-02-17T00:31:42.887339+00:00", "level": "INFO", "logger": "veena3modal.tts", "message": "request_received", "request_id": "41a746b9-784e-4166-a18f-1e231ba10896", "event": "request_received", "text_length": 5, "speaker": "lipakshi", "stream": true, "format": "wav"}
2026-02-17 00:31:42 | INFO     | veena3modal.tts | request_received
INFO:     127.0.0.1:53038 - "POST /v1/tts/generate HTTP/1.1" 200 OK
2026-02-17 00:31:42 | INFO     | veena3modal.services.tts_runtime | tts_runtime_stream_start
2026-02-17 00:31:42 | INFO     | veena3modal.processing.long_text_processor | LongTextProcessor initialized


The stream path also stalls immediately after entering the streaming runtime, so I’ll inspect the streaming pipeline for any blocking point that would also affect non-streaming benchmarks and instrument both pipeline and runtime accordingly.


sed -n '1,360p' veena3modal/core/streaming_pipeline.py && sed -n '360,760p' veena3modal/core/streaming_pipeline.py && sed -n '760,1120p' veena3modal/core/streaming_pipeline.py


Output:
Chunk ID: b2045c
Wall time: 0.0509 seconds
Process exited with code 0
Original token count: 12759
Output:
Total output lines: 1122

"""
Veena3 Streaming Pipeline - Sliding Window Approach

Implements streaming for both SNAC and BiCodec tokens.

For BiCodec (Spark TTS):
1. Generate tokens from vLLM
2. Parse NEW token IDs incrementally (O(1) per token via cache)
3. Buffer semantic + global tokens
4. Apply sliding window (every N token pairs)
5. Decode and stream audio chunks

OPTIMIZATION (Dec 2025):
- Replaced O(n²) pattern (decode-all + regex-all per iteration)
- Now uses incremental token parsing with BiCodecTokenParser
- ~10x CPU reduction in streaming hot loop

For SNAC (legacy):
1. Filter SNAC token IDs directly
2. Apply sliding window
3. Decode and stream
"""

import asyncio
import os
import re
import logging
import torch
from typing import AsyncGenerator, Optional, List, Dict, Any
from vllm import SamplingParams
from veena3modal.audio.crossfade import crossfade_bytes_int16
from veena3modal.core.token_utils import BiCodecTokenParser

from veena3modal.core.constants import (
    CODE_END_TOKEN_ID,
    CODE_START_TOKEN_ID,
    SNAC_MIN_ID,
    SNAC_MAX_ID,
    TRAINING_STOP_TOKEN_IDS,
    DEFAULT_TEMPERATURE,
    DEFAULT_TOP_K,
    DEFAULT_TOP_P,
    DEFAULT_MAX_TOKENS,
    DEFAULT_MIN_TOKENS,
    DEFAULT_REPETITION_PENALTY,
    DEFAULT_SEED,
)

logger = logging.getLogger(__name__)

def _detailed_timing_enabled() -> bool:
    """Enable per-batch token/decode timing logs."""
    return os.getenv("VEENA3_TIMING_LOGS", "").lower() in {"1", "true", "yes", "on"}

class Veena3SlidingWindowPipeline:
    """
    Streaming TTS pipeline using sliding window approach.

    This eliminates choppy audio and popping artifacts by:
    - Decoding overlapping 28-token windows (4 frames)
    - Keeping only the middle 2048 samples from each decode
    - Creating natural continuity between chunks

    Based on the official Canopy Labs implementation.
    """

    def __init__(
        self,
        model,
        prompt_builder,
        snac_decoder,
    ):
        """
        Initialize sliding window streaming pipeline.

        Args:
            model: Veena3Model instance
            prompt_builder: Veena3PromptBuilder instance
            snac_decoder: SNACDecoder instance (with batching enabled)
        """
        self.model = model
        self.prompt_builder = prompt_builder
        self.snac_decoder = snac_decoder

        # OPTIMIZATION: Pre-warm BiCodecTokenParser once at init, not per-request
        # Saves ~123ms per streaming TTFB by avoiding repeated 166K-entry vocab iteration
        tokenizer = getattr(model, "tokenizer", None)
        if tokenizer is None:
            tokenizer = getattr(model.engine, "tokenizer", None)
        self.token_parser = BiCodecTokenParser(tokenizer) if tokenizer else None

        print(f"🌊 Veena3SlidingWindowPipeline initialized (sliding window: 28 tokens)")

    async def generate_speech_stream(
        self,
        description: str,
        text: str,
        temperature: float = DEFAULT_TEMPERATURE,
        top_p: float = DEFAULT_TOP_P,
        max_tokens: int = DEFAULT_MAX_TOKENS,
        repetition_penalty: float = DEFAULT_REPETITION_PENALTY,
        seed: Optional[int] = None,
    ) -> AsyncGenerator[bytes, None]:
        """
        Generate speech audio with sliding window streaming.

        Yields audio chunks using overlapping windows for smooth playback.

        Args:
            description: Character/voice description
            text: Text to synthesize (with optional <emotion> tags)
            temperature: Sampling temperature
            top_p: Nucleus sampling
            max_tokens: Max SNAC tokens to generate
            repetition_penalty: Prevent loops

        Yields:
            Audio bytes (int16 PCM, 24kHz mono)
        """
        print(f"\n🌊 Sliding window streaming generation")
        print(f"📝 Description: {description[:80]}...")
        print(f"💬 Text: {text}")

        # Build prompt
        prompt = self.prompt_builder.build_prefix(description, text)
        print(f"✅ Prompt built ({len(prompt)} chars)")

        # Configure sampling
        sampling_params = SamplingParams(
            temperature=temperature,
            top_p=top_p,
            max_tokens=max_tokens,
            min_tokens=DEFAULT_MIN_TOKENS,
            repetition_penalty=repetition_penalty,
            stop_token_ids=[CODE_END_TOKEN_ID],  # Only audio EOS
            seed=seed,  # None = random, int = reproducible
        )

        print(f"🎲 Sampling: temp={temperature}, top_p={top_p}, sliding_window=28 tokens")

        # Token buffer - keeps ALL tokens (not chunked)
        token_buffer = []
        total_tokens_generated = 0
        total_audio_chunks = 0

        # Generate tokens with vLLM (streaming)
        print(f"🔮 Starting token generation...")

        # Generate unique request ID for concurrent streaming support
        import uuid
        import time
        request_id = f"slide-{uuid.uuid4().hex[:8]}-{int(time.time() * 1000000)}"
        results_generator = self.model.engine.generate(
            prompt=prompt,
            sampling_params=sampling_params,
            request_id=request_id,
        )

        # Stream tokens with sliding window
        async for request_output in results_generator:
            # Extract generated token IDs
            generated_ids = request_output.outputs[0].token_ids

            # Process only new tokens
            new_tokens = generated_ids[total_tokens_generated:]
            total_tokens_generated = len(generated_ids)

            # Filter and buffer SNAC tokens
            for token_id in new_tokens:
                if SNAC_MIN_ID <= token_id <= SNAC_MAX_ID:
                    token_buffer.append(token_id)

                    # Process every 7 tokens (1 frame) when we have enough for sliding window
                    # Official approach: once count > 27, take last 28 tokens
                    if len(token_buffer) % 7 == 0 and len(token_buffer) > 27:
                        # Sliding window: take last 28 tokens (4 frames)
                        window_tokens = token_buffer[-28:]

                        # Decode with sliding window mode (returns middle 2048 samples only)
                        if self.snac_decoder.enable_batching:
                            audio_bytes = await self.snac_decoder.decode_single_async(
                                window_tokens,
                                trim_warmup=False,  # Sliding window handles trimming
                                use_sliding_window=True  # CRITICAL: Use sliding window mode
                            )
                        else:
                            audio_bytes = self.snac_decoder.decode_to_bytes(
                                window_tokens,
                                trim_warmup=False,
                                use_sliding_window=True
                            )

                        if audio_bytes:
                            total_audio_chunks += 1
                            if total_audio_chunks == 1:
                                print(f"🎵 First chunk decoded ({len(audio_bytes)} bytes, sliding window)")
                            yield audio_bytes

        # Note: No final chunk processing needed - sliding window handles all tokens
        # as they come in (every 7 tokens after the first 28)

        print(f"✅ Sliding window streaming complete: {total_tokens_generated} tokens → {total_audio_chunks} audio chunks")

    def _extract_bicodec_tokens_from_text(self, text: str) -> tuple[List[int], List[int]]:
        """
        Extract BiCodec semantic and global tokens from generated text.

        Args:
            text: Generated text containing BiCodec token markers

        Returns:
            Tuple of (semantic_ids, global_ids)
        """
        semantic_matches = re.findall(r"<\|bicodec_semantic_(\d+)\|>", text)
        semantic_ids = [int(t) for t in semantic_matches]

        global_matches = re.findall(r"<\|bicodec_global_(\d+)\|>", text)
        global_ids = [int(t) for t in global_matches]

        return semantic_ids, global_ids

    async def generate_speech_stream_indic(
        self,
        speaker: str,
        text: str,
        temperature: float = DEFAULT_TEMPERATURE,
        top_k: int = DEFAULT_TOP_K,  # Added for Spark TTS parity with non-streaming
        top_p: float = DEFAULT_TOP_P,
        max_tokens: int = DEFAULT_MAX_TOKENS,
        repetition_penalty: float = DEFAULT_REPETITION_PENALTY,
        seed: Optional[int] = None,
        request_id: Optional[str] = None,
        request_metrics: Optional[Dict[str, Any]] = None,
    ) -> AsyncGenerator[bytes, None]:
        """
        Generate speech audio with sliding window streaming for Indic model.

        Yields audio chunks using overlapping windows for smooth playback.

        Args:
            speaker: Speaker name (one of 12 predefined speakers)
            text: Text to synthesize with inline emotion tags
                Examples:
                - "Hello! Welcome."
                - "<laugh> Hello there!"
                - "नमस्ते! <excited> आज का दिन बहुत अच्छा है।"
            temperature: Sampling temperature
            top_p: Nucleus sampling
            max_tokens: Max SNAC tokens to generate
            repetition_penalty: Prevent loops
            seed: Random seed for reproducibility

        Yields:
            Audio bytes (int16 PCM, 16kHz mono - BiCodec)
        """
        import statistics
        import time
        t_start = time.time()
        emit_timing = _detailed_timing_enabled()
        request_metrics = request_metrics or {}
        request_metrics.update(
            {
                "request_id": request_id,
                "speaker": speaker,
                "text_length": len(text),
            }
        )

        # Build prompt using Indic prompt builder
        prompt = self.prompt_builder.build_prefix(speaker, text)

        # Configure sampling (matching non-streaming pipeline parameters)
        # OPTIMIZATION: Use Spark TTS stop token, not legacy SNAC CODE_END_TOKEN_ID (128258)
        # which doesn't exist in the Spark vocab and would cause generation to run to max_tokens
        sampling_params = SamplingParams(
            temperature=temperature,
            top_k=top_k,
            top_p=top_p,
            max_tokens=max_tokens,
            min_tokens=DEFAULT_MIN_TOKENS,
            repetition_penalty=repetition_penalty,
            stop=TRAINING_STOP_TOKEN_IDS,  # "<|im_end|>" - matches non-streaming pipeline
            skip_special_tokens=False,  # Keep BiCodec tokens in output
            seed=seed,
        )

        # BiCodec token buffers (separate for semantic and global)
        semantic_buffer = []
        global_buffer = []
        processed_token_count = 0  # Track how many vLLM tokens we've parsed
        total_audio_chunks = 0

        # BiCodec streaming configuration
        EXPECTED_GLOBAL_COUNT = 32  # BiCodec uses 32 fixed global tokens (per paper)
        MIN_SEMANTIC_FOR_FIRST_CHUNK = 10  # OPTIMIZATION: Lowered from 16 (decoder min is 8, -120ms TTFB)
        # OPTIMIZATION: Increased from 24 to 48 to reduce O(n) re-decode calls by 2x.
        # At 50 TPS, 48 tokens = ~960ms audio per chunk. Cuts total decode calls roughly in half.
        DECODE_INTERVAL = 48
        CROSSFADE_MS = 50
        # OPTIMIZATION: Windowed decode - only decode last WINDOW_SIZE tokens instead of ALL.
        # BiCodec WaveGenerator has finite receptive field (~64 tokens based on kernel sizes).
        # Window must be >= receptive_field + DECODE_INTERVAL to avoid boundary artifacts.
        WINDOW_SIZE = 128  # 128 tokens covers receptive field + margin for crossfade

        # Generate tokens with vLLM (streaming)
        if request_id is None:
            import uuid
            request_id = f"bicodec-stream-{uuid.uuid4().hex[:8]}-{int(time.time() * 1000000)}"
            request_metrics["request_id"] = request_id

        # Reuse pre-warmed token parser (singleton, created once in __init__)
        token_parser = self.token_parser
        if token_parser is None:
            raise RuntimeError("BiCodecTokenParser unavailable for streaming")

        t_gen_start = time.time()
        batch_wall_ms: List[float] = []
        batch_token_counts: List[int] = []
        batch_gpu_ms: List[float] = []
        parse_wall_ms_list: List[float] = []
        decode_calls = 0
        decode_wall_ms = 0.0
        decode_gpu_ms = 0.0
        decode_wall_samples: List[float] = []
        decode_gpu_samples: List[float] = []

        results_generator = self.model.engine.generate(
            prompt=prompt,
            sampling_params=sampling_params,
            request_id=request_id,
        )

        # Track timing
        t_first_token = None
        t_first_chunk_ready = None
        t_first_chunk_decoded = None
        iteration_count = 0
        last_decode_count = 0

        # Track audio samples EMITTED to the user (not just decoded)
        # BiCodec produces ~320 samples per semantic token at 16kHz
        # CRITICAL: This tracks samples that have been YIELDED, not held in tail
        total_samples_emitted_to_user = 0
        # Keep a tail to enable crossfading between consecutive emitted chunks
        previous_chunk_tail: Optional[bytes] = None

        # Stream tokens with BiCodec extraction
        async for request_output in results_generator:
            iteration_count += 1
            t_iter = time.time()

            # Track first token received
            if t_first_token is None:
                t_first_token = t_iter

            # Extract generated token IDs
            generated_ids = request_output.outputs[0].token_ids

            parse_start = time.time()
            # OPTIMIZATION: Only process NEW tokens (incremental parsing)
            # Old code: tokenizer.decode(ALL) + re.findall(ALL) = O(n²) per stream
            # New code: parse only new tokens = O(1) per token
            new_token_ids = generated_ids[processed_token_count:]
            processed_token_count = len(generated_ids)

            # Parse new tokens and append to buffers (in-place modification)
            token_parser.parse_incremental(new_token_ids, semantic_buffer, global_buffer)
            parse_wall_ms = (time.time() - parse_start) * 1000
            parse_wall_ms_list.append(parse_wall_ms)
            batch_token_counts.append(len(new_token_ids))
            batch_wall_ms_value = (time.time() - t_iter) * 1000 if t_iter else 0.0
            batch_wall_ms.append(batch_wall_ms_value)
            if emit_timing:
                logger.info(
                    "tts_stream_llm_batch",
                    extra={
                        "speaker": speaker,
                        "request_id": request_id,
                        "batch_index": len(batch_wall_ms),
                        "batch_tokens": len(new_token_ids),
                        "llm_batch_wall_ms": int(batch_wall_ms_value),
                        "llm_batch_gpu_ms": int(batch_gpu_ms[-1]) if batch_gpu_ms else None,
                    },
                )

            # TRUE STREAMING with BiCodec two-phase generation:
            # Phase 1: Generate 32 global tokens (pre-roll)
            # Phase 2: Stream semantic tokens incrementally

            # Check if we have the 32 global tokens (pre-roll complete)
            if len(global_buffer) >= EXPECTED_GLOBAL_COUNT:
                # Pre-roll complete! Now we can stream semantic tokens
                if t_first_chunk_ready is None and len(semantic_buffer) >= MIN_SEMANTIC_FOR_FIRST_CHUNK:
                    t_first_chunk_ready = time.time()

                # Stream semantic tokens incrementally
                # Only decode if we have enough NEW semantic tokens since last decode
                semantic_count = len(semantic_buffer)
                if semantic_count >= MIN_SEMANTIC_FOR_FIRST_CHUNK:
                    # Check if we should decode (every DECODE_INTERVAL new semantic tokens)
                    if semantic_count - last_decode_count >= DECODE_INTERVAL:
                        last_decode_count = semantic_count

                        # OPTIMIZATION: Windowed decode instead of full re-decode
                        # Only decode the last WINDOW_SIZE tokens (O(1) per chunk instead of O(n))
                        # For first few chunks when total tokens < WINDOW_SIZE, decode all
                        window_global = global_buffer[:EXPECTED_GLOBAL_COUNT]

                        if len(semantic_buffer) <= WINDOW_SIZE:
                            # Early phase: decode all tokens (window not yet relevant)
                            decode_semantic = semantic_buffer
                        else:
                            # Windowed: only decode last WINDOW_SIZE tokens
                            decode_semantic = semantic_buffer[-WINDOW_SIZE:]

                        decode_gpu_ms_this = None
                        t_decode_start = time.time()
                        if emit_timing and torch.cuda.is_available():
                            torch.cuda.synchronize()
                            decode_start_event = torch.cuda.Event(enable_timing=True)
                            decode_end_event = torch.cuda.Event(enable_timing=True)
                            decode_start_event.record()
                        audio_bytes = await self.snac_decoder.decode_single_async(
                            semantic_ids=decode_semantic,
                            global_ids=window_global,
                            trim_warmup=False,
                            use_sliding_window=False,
                        )
                        t_decode_end = time.time()
                        decode_calls += 1
                        decode_ms = (t_decode_end - t_decode_start) * 1000
                        decode_wall_ms += decode_ms
                        decode_wall_samples.append(decode_ms)
                        if emit_timing and torch.cuda.is_available():
                            decode_end_event.record()
                            torch.cuda.synchronize()
                            decode_gpu_ms_this = decode_start_event.elapsed_time(decode_end_event)
                            decode_gpu_ms += decode_gpu_ms_this
                            decode_gpu_samples.append(decode_gpu_ms_this)
                            batch_gpu_ms.append(decode_gpu_ms_this)

                        if audio_bytes:
                            total_samples_decoded = len(audio_bytes) // 2

                            if len(semantic_buffer) <= WINDOW_SIZE:
                                # Early phase: extract new samples from full decode
                                if total_samples_decoded > total_samples_emitted_to_user:
                                    new_bytes_start = total_samples_emitted_to_user * 2
                                    new_audio_bytes = audio_bytes[new_bytes_start:]
                                else:
                                    new_audio_bytes = b""
                            else:
                                # Windowed phase: take only the tail portion of the window decode
                                # The last DECODE_INTERVAL tokens' worth of audio is the new content
                                # Each…2759 tokens truncated…       )

        # Track state
        batch_wall_ms: List[float] = []
        batch_gpu_ms: List[float] = []
        batch_token_counts: List[int] = []
        parse_wall_ms_list: List[float] = []
        last_decode_count = 0
        total_samples_emitted_to_user = 0
        previous_chunk_tail: Optional[bytes] = None
        captured_globals: List[int] = []  # Will be populated once we have 32 globals
        decode_calls = 0
        decode_wall_ms = 0.0
        decode_gpu_ms = 0.0
        decode_wall_samples: List[float] = []
        decode_gpu_samples: List[float] = []
        t_first_token = None
        t_first_chunk_ready = None
        t_first_chunk_decoded = None
        iteration_count = 0

        # Stream tokens with BiCodec extraction
        async for request_output in results_generator:
            iteration_count += 1
            t_iter = time.time()
            if t_first_token is None:
                t_first_token = t_iter

            generated_ids = request_output.outputs[0].token_ids

            parse_start = time.time()
            # OPTIMIZATION: Incremental parsing (O(1) per new token)
            new_token_ids = generated_ids[processed_token_count:]
            processed_token_count = len(generated_ids)
            token_parser.parse_incremental(new_token_ids, semantic_buffer, global_buffer)
            parse_ms = (time.time() - parse_start) * 1000
            parse_wall_ms_list.append(parse_ms)

            batch_token_counts.append(len(new_token_ids))
            batch_wall_ms_value = (time.time() - t_iter) * 1000
            batch_wall_ms.append(batch_wall_ms_value)

            if emit_timing:
                logger.info(
                    "tts_stream_llm_batch",
                    extra={
                        "speaker": speaker,
                        "request_id": request_id,
                        "batch_index": iteration_count,
                        "batch_tokens": len(new_token_ids),
                        "llm_batch_wall_ms": int(batch_wall_ms_value),
                        "llm_batch_gpu_ms": int(batch_gpu_ms[-1]) if batch_gpu_ms else None,
                        "text_chunked": True,
                    },
                )

            # Capture global tokens once we have all 32
            if not captured_globals and len(global_buffer) >= EXPECTED_GLOBAL_COUNT:
                captured_globals = global_buffer[:EXPECTED_GLOBAL_COUNT]

            # Process if we have all global tokens
            if len(global_buffer) >= EXPECTED_GLOBAL_COUNT:
                if t_first_chunk_ready is None and len(semantic_buffer) >= MIN_SEMANTIC_FOR_FIRST_CHUNK:
                    t_first_chunk_ready = time.time()
                semantic_count = len(semantic_buffer)
                if semantic_count - last_decode_count >= DECODE_INTERVAL:
                    last_decode_count = semantic_count

                    all_semantic = semantic_buffer
                    window_global = global_buffer[:EXPECTED_GLOBAL_COUNT]
                    decode_gpu_ms_this = None
                    t_decode_start = time.time()
                    t_decode_start = time.time()
                    if emit_timing and torch.cuda.is_available():
                        torch.cuda.synchronize()
                        decode_start_event = torch.cuda.Event(enable_timing=True)
                        decode_end_event = torch.cuda.Event(enable_timing=True)
                        decode_start_event.record()

                    if self.snac_decoder.enable_batching:
                        audio_bytes = await self.snac_decoder.decode_single_async(
                            semantic_ids=all_semantic,
                            global_ids=window_global,
                            trim_warmup=False,
                            use_sliding_window=False
                        )
                    else:
                        audio_bytes = self.snac_decoder.decode_streaming(
                            semantic_ids=all_semantic,
                            global_ids=window_global,
                            use_sliding_window=False,
                            trim_warmup=False
                        )

                    t_decode_end = time.time()
                    decode_calls += 1
                    decode_ms = (t_decode_end - t_decode_start) * 1000
                    decode_wall_ms += decode_ms
                    decode_wall_samples.append(decode_ms)
                    if emit_timing and torch.cuda.is_available():
                        decode_end_event.record()
                        torch.cuda.synchronize()
                        decode_gpu_ms_this = decode_start_event.elapsed_time(decode_end_event)
                        decode_gpu_ms += decode_gpu_ms_this
                        decode_gpu_samples.append(decode_gpu_ms_this)
                        batch_gpu_ms.append(decode_gpu_ms_this)

                    to_emit = b""
                    if audio_bytes:
                        total_samples_decoded = len(audio_bytes) // 2

                        if total_samples_decoded > total_samples_emitted_to_user:
                            new_bytes_start = total_samples_emitted_to_user * 2
                            new_audio_bytes = audio_bytes[new_bytes_start:]

                            to_emit, previous_chunk_tail = crossfade_bytes_int16(
                                previous_chunk_tail,
                                new_audio_bytes,
                                sample_rate_hz=16000,
                                crossfade_ms=CROSSFADE_MS,
                            )

                            samples_emitted_in_this_chunk = len(to_emit) // 2
                            total_samples_emitted_to_user += samples_emitted_in_this_chunk

                    if to_emit:
                        total_audio_chunks += 1
                        if t_first_chunk_decoded is None:
                            t_first_chunk_decoded = t_decode_end
                        if emit_timing:
                            logger.info(
                                "tts_stream_decode_batch",
                                extra={
                                    "speaker": speaker,
                                    "request_id": request_id,
                                    "batch_index": decode_calls,
                                    "decode_wall_ms": int(decode_ms),
                                    "decode_gpu_ms": int(decode_gpu_ms_this)
                                    if decode_gpu_ms_this is not None else None,
                                    "semantic_count": len(semantic_buffer),
                                    "text_chunked": True,
                                },
                            )
                        yield (to_emit, captured_globals)

        # Final chunk processing
        if len(semantic_buffer) > last_decode_count and len(global_buffer) >= EXPECTED_GLOBAL_COUNT:
            t_decode_start = time.time()
            audio_bytes = self.snac_decoder.decode_streaming(
                semantic_ids=semantic_buffer,
                global_ids=global_buffer[:EXPECTED_GLOBAL_COUNT],
                use_sliding_window=False,
                trim_warmup=False
            )
            t_decode_end = time.time()
            decode_ms = (t_decode_end - t_decode_start) * 1000
            decode_calls += 1
            decode_wall_ms += decode_ms
            decode_wall_samples.append(decode_ms)

            to_emit = b""
            if audio_bytes:
                total_samples_decoded = len(audio_bytes) // 2

                if total_samples_decoded > total_samples_emitted_to_user:
                    new_bytes_start = total_samples_emitted_to_user * 2
                    new_audio_bytes = audio_bytes[new_bytes_start:]

                    to_emit, previous_chunk_tail = crossfade_bytes_int16(
                        previous_chunk_tail,
                        new_audio_bytes,
                        sample_rate_hz=16000,
                        crossfade_ms=CROSSFADE_MS,
                    )

                    samples_emitted_in_final = len(to_emit) // 2
                    total_samples_emitted_to_user += samples_emitted_in_final

            if to_emit:
                total_audio_chunks += 1
                if t_first_chunk_decoded is None:
                    t_first_chunk_decoded = t_decode_end
                if emit_timing:
                    logger.info(
                        "tts_stream_decode_batch",
                        extra={
                            "speaker": speaker,
                            "request_id": request_id,
                            "batch_index": decode_calls,
                            "decode_wall_ms": int(decode_ms),
                            "decode_gpu_ms": None,
                            "semantic_count": len(semantic_buffer),
                            "text_chunked": True,
                        },
                    )
                yield (to_emit, captured_globals)

        # Flush remaining tail
        if previous_chunk_tail:
            total_audio_chunks += 1
            tail_samples = len(previous_chunk_tail) // 2
            total_samples_emitted_to_user += tail_samples
            yield (previous_chunk_tail, captured_globals)

        t_end = time.time()
        audio_duration_s = total_samples_emitted_to_user / 16000
        ttfb_ms = (
            (t_first_chunk_decoded - t_start) * 1000
            if t_first_chunk_decoded is not None
            else (t_first_chunk_ready - t_start) * 1000
            if t_first_chunk_ready is not None
            else (t_first_token - t_start) * 1000
            if t_first_token is not None
            else 0
        )
        request_metrics.update(
            {
                "request_id": request_id,
                "speaker": speaker,
                "generated_tokens": len(semantic_buffer) + len(global_buffer),
                "semantic_tokens": len(semantic_buffer),
                "global_tokens": len(captured_globals),
                "llm_batch_count": len(batch_wall_ms),
                "llm_batch_tokens": batch_token_counts,
                "llm_batch_wall_ms_total": int(sum(batch_wall_ms)),
                "llm_batch_wall_ms_min": int(min(batch_wall_ms)) if batch_wall_ms else 0,
                "llm_batch_wall_ms_max": int(max(batch_wall_ms)) if batch_wall_ms else 0,
                "llm_batch_wall_ms_p50": int(statistics.median(batch_wall_ms)) if batch_wall_ms else 0,
                "llm_batch_gpu_ms_total": int(sum(batch_gpu_ms)),
                "llm_batch_gpu_ms_min": int(min(batch_gpu_ms)) if batch_gpu_ms else 0,
                "llm_batch_gpu_ms_max": int(max(batch_gpu_ms)) if batch_gpu_ms else 0,
                "llm_batch_gpu_ms_p50": int(statistics.median(batch_gpu_ms)) if batch_gpu_ms else 0,
                "llm_parse_ms": int(sum(parse_wall_ms_list)),
                "llm_parse_ms_avg": int(sum(parse_wall_ms_list) / len(parse_wall_ms_list)) if parse_wall_ms_list else 0,
                "decode_calls": decode_calls,
                "llm_decode_wall_ms_total": int(decode_wall_ms),
                "llm_decode_wall_ms_min": int(min(decode_wall_samples)) if decode_wall_samples else 0,
                "llm_decode_wall_ms_max": int(max(decode_wall_samples)) if decode_wall_samples else 0,
                "llm_decode_wall_ms_p50": int(statistics.median(decode_wall_samples)) if decode_wall_samples else 0,
                "llm_decode_gpu_ms_total": int(decode_gpu_ms),
                "llm_decode_gpu_ms_min": int(min(decode_gpu_samples)) if decode_gpu_samples else 0,
                "llm_decode_gpu_ms_max": int(max(decode_gpu_samples)) if decode_gpu_samples else 0,
                "llm_decode_gpu_ms_p50": int(statistics.median(decode_gpu_samples)) if decode_gpu_samples else 0,
                "audio_duration_seconds": audio_duration_s,
                "audio_chunks": total_audio_chunks,
                "captured_global_count": len(captured_globals),
                "text_chunked": True,
                "ttfb_ms": int(ttfb_ms),
                "generation_ms": int((t_end - t_start) * 1000),
            }
        )
        if batch_token_counts:
            request_metrics["tokens_per_batch_min"] = min(batch_token_counts)
            request_metrics["tokens_per_batch_max"] = max(batch_token_counts)
            request_metrics["tokens_per_batch_p50"] = int(statistics.median(batch_token_counts))

        logger.info("tts_stream_first_chunk_complete", extra=request_metrics)

    async def generate_speech_stream_indic_continuation(
        self,
        speaker: str,
        text: str,
        global_ids: List[int],
        temperature: float = DEFAULT_TEMPERATURE,
        top_k: int = DEFAULT_TOP_K,
        top_p: float = DEFAULT_TOP_P,
        max_tokens: int = DEFAULT_MAX_TOKENS,
        repetition_penalty: float = DEFAULT_REPETITION_PENALTY,
        seed: Optional[int] = None,
        request_id: Optional[str] = None,
        request_metrics: Optional[Dict[str, Any]] = None,
    ) -> AsyncGenerator[bytes, None]:
        """
        Generate speech for CONTINUATION chunks using pre-captured global tokens.

        This is the key method for voice consistency in chunked generation:
        - Uses global_ids from the first chunk to maintain identical voice
        - Model generates only semantic tokens (skips global token generation)
        - Ensures no voice drift across chunks

        CRITICAL for production:
        - global_ids is request-scoped, passed explicitly (no shared state)
        - Thread-safe: each request has its own state
        - No global caching that could cause cross-request contamination

        Args:
            speaker: Speaker name (must match first chunk)
            text: Continuation text chunk to synthesize
            global_ids: 32 global token IDs captured from first chunk
            temperature: Sampling temperature
            top_k: Top-k sampling
            top_p: Nucleus sampling
            max_tokens: Max tokens to generate
            repetition_penalty: Prevent repetition
            seed: Random seed (use same as first chunk for consistency)

        Yields:
            Audio bytes (int16 PCM, 16kHz mono)

        Raises:
            ValueError: If global_ids doesn't contain exactly 32 tokens

        Thread Safety:
            Fully thread-safe. All state is request-scoped and passed explicitly.
        """
        import time
        import torch

        t_start = time.time()
        emit_timing = _detailed_timing_enabled()
        request_metrics = request_metrics or {}
        request_metrics.update(
            {
                "request_id": request_id,
                "speaker": speaker,
                "text_length": len(text),
                "text_chunked": True,
            }
        )

        # Validate global tokens
        EXPECTED_GLOBAL_COUNT = 32
        if len(global_ids) != EXPECTED_GLOBAL_COUNT:
            raise ValueError(
                f"Expected exactly {EXPECTED_GLOBAL_COUNT} global tokens, got {len(global_ids)}. "
                f"global_ids must be captured from first chunk generation."
            )

        # Build prompt WITH pre-filled global tokens
        # This tells the model to skip global generation and go straight to semantic tokens
        prompt = self.prompt_builder.build_prefix_with_globals(speaker, text, global_ids)

        # Configure sampling
        # OPTIMIZATION: Use Spark TTS stop token, not legacy SNAC CODE_END_TOKEN_ID
        sampling_params = SamplingParams(
            temperature=temperature,
            top_k=top_k,
            top_p=top_p,
            max_tokens=max_tokens,
            min_tokens=DEFAULT_MIN_TOKENS,
            repetition_penalty=repetition_penalty,
            stop=TRAINING_STOP_TOKEN_IDS,  # "<|im_end|>" - matches non-streaming pipeline
            skip_special_tokens=False,
            seed=seed,
        )

        # BiCodec token buffers
        semantic_buffer = []
        global_buffer_unused = []  # Not used in continuation, but needed for parser API
        processed_token_count = 0  # Track how many vLLM tokens we've parsed
        total_audio_chunks = 0

        # BiCodec streaming configuration
        MIN_SEMANTIC_FOR_FIRST_CHUNK = 10  # OPTIMIZATION: Lowered from 16 (decoder min is 8)
        DECODE_INTERVAL = 48  # OPTIMIZATION: Increased from 24 to reduce O(n) re-decode calls
        CROSSFADE_MS = 50

        # Generate unique request ID
        import uuid
        if request_id is None:
            request_id = f"bicodec-cont-{uuid.uuid4().hex[:8]}-{int(time.time() * 1000000)}"
            request_metrics["request_id"] = request_id

        # Reuse pre-warmed token parser (singleton, created once in __init__)
        token_parser = self.token_parser
        if token_parser is None:
            raise RuntimeError("BiCodecTokenParser unavailable for continuation streaming")

        results_generator = self.model.engine.generate(
            prompt=prompt,
            sampling_params=sampling_params,
            request_id=request_id,
        )

        # Track state
        batch_wall_ms: List[float] = []
        batch_gpu_ms: List[float] = []
        batch_token_counts: List[int] = []
        parse_wall_ms_list: List[float] = []
        decode_calls = 0
        decode_wall_ms = 0.0
        decode_gpu_ms = 0.0
        decode_wall_samples: List[float] = []
        decode_gpu_samples: List[float] = []
        t_first_token = None
        t_first_chunk_ready = None
        t_first_chunk_decoded = None
        iteration_count = 0
        last_decode_count = 0
        total_samples_emitted_to_user = 0
        previous_chunk_tail: Optional[bytes] = None

        # Stream tokens - model will generate ONLY semantic tokens (globals are pre-filled)
        async for request_output in results_generator:
            iteration_count += 1
            t_iter = time.time()
            if t_first_token is None:
                t_first_token = t_iter

            generated_ids = request_output.outputs[0].token_ids

            # OPTIMIZATION: Incremental parsing (O(1) per new token)
            # Note: In continuation, model only generates semantic tokens (globals pre-filled in prompt)
            parse_start = time.time()
            new_token_ids = generated_ids[processed_token_count:]
            processed_token_count = len(generated_ids)
            token_parser.parse_incremental(new_token_ids, semantic_buffer, global_buffer_unused)
            parse_ms = (time.time() - parse_start) * 1000
            parse_wall_ms_list.append(parse_ms)
            batch_token_counts.append(len(new_token_ids))
            batch_wall_ms_value = (time.time() - t_iter) * 1000
            batch_wall_ms.append(batch_wall_ms_value)
            if emit_timing:
                logger.info(
                    "tts_stream_llm_batch",
                    extra={
                        "speaker": speaker,
                        "request_id": request_id,
                        "batch_index": iteration_count,
                        "batch_tokens": len(new_token_ids),
                        "llm_batch_wall_ms": int(batch_wall_ms_value),
                        "llm_batch_gpu_ms": int(batch_gpu_ms[-1]) if batch_gpu_ms else None,
                        "text_chunked": True,
                    },
                )

            semantic_count = len(semantic_buffer)
            if semantic_count >= MIN_SEMANTIC_FOR_FIRST_CHUNK:
                if t_first_chunk_ready is None:
                    t_first_chunk_ready = time.time()
                if semantic_count - last_decode_count >= DECODE_INTERVAL:
                    last_decode_count = semantic_count


sed -n '1,260p' veena3modal/processing/prompt_builder.py


Output:
Chunk ID: 8f8277
Wall time: 0.0515 seconds
Process exited with code 0
Original token count: 2374
Output:
"""
Spark TTS Indic Prompt Builder

Builds prompts for Spark TTS with Indic speaker system.
Format: <|task_controllable_tts|><|start_content|>...<|end_content|>...

Migrated from Orpheus to Spark TTS architecture.
"""

import re
from typing import List, Tuple

from veena3modal.core.constants import INDIC_SPEAKERS, INDIC_EMOTION_TAGS, SPEAKER_MAP
from veena3modal.processing.emotion_normalizer import normalize_emotion_tags

class IndicPromptBuilder:
    """
    Builds prompts in Spark TTS format with Indic speaker system.

    Format:
    <|task_controllable_tts|>
    <|start_content|>{text with [emotions]}<|end_content|>
    <|start_style_label|><|speaker_{id}|><|end_style_label|>
    <|start_global_token|>

    CRITICAL: This format MUST match Spark TTS training exactly.

    Model Details:
    - HuggingFace: BayAreaBoys/spark_tts_4speaker
    - Architecture: Qwen2ForCausalLM with BiCodec audio tokenizer
    - Languages: Telugu, Hindi, English, and more
    - Speakers: 12 predefined (case-sensitive!) - must match training map
    - Emotions: 10 tags in [bracket] format
    """

    def __init__(self, tokenizer, model=None):
        """
        Initialize Indic prompt builder for Spark TTS.

        Args:
            tokenizer: Transformers tokenizer
            model: Model instance (optional, kept for backward compatibility)
        """
        self.tokenizer = tokenizer
        self.model = model

    def build_prefix(
        self,
        speaker: str,
        text: str,
        validate: bool = True
    ) -> str:
        """
        Build Spark TTS format prompt with Indic speaker.

        Args:
            speaker: Speaker name (one of 12 predefined speakers, case-sensitive!)
                Examples: 'lipakshi', 'reet', 'Nandini', 'Nilay', 'vardan', 'anika', 'adarsh',
                'krishna', 'Aarvi', 'Asha', 'Bittu', 'Mira'
            text: Text to synthesize with inline emotion tags
                Examples:
                - "Hello! Welcome to the demo."
                - "[laughs] The results were amazing!"
                - "The results were amazing and then [giggle] we celebrated!"
                - "नमस्ते! [excited] आज का दिन बहुत अच्छा है।" (Hindi)
                - "[curious] మీరు ఎలా ఉన్నారు?" (Telugu)
                - "Hello <laugh> this works too!" (will be normalized to [laughs])
            validate: Check format correctness (recommended for first use)

        Returns:
            Formatted prompt string ready for tokenization
            Format: <|task_controllable_tts|><|start_content|>text<|end_content|>...

        Note: Emotion tags should be inline in text. The API will normalize:
        - <laugh> → [laughs] (old format)
        - laughing → [laughs] (natural language)
        """
        # Normalize emotion tags: <emotion> → [emotion] (legacy compatibility)
        text = normalize_emotion_tags(text)

        # Validate if requested
        if validate:
            self._validate_inputs(speaker, text)

        # Get speaker ID from mapping
        speaker_id = SPEAKER_MAP.get(speaker)
        if speaker_id is None:
            raise ValueError(f"Invalid speaker: {speaker}. Valid speakers: {list(SPEAKER_MAP.keys())}")

        # Build Spark TTS format prompt
        prompt = "".join([
            "<|task_controllable_tts|>",
            "<|start_content|>",
            text,
            "<|end_content|>",
            "<|start_style_label|>",
            f"<|speaker_{speaker_id}|>",
            "<|end_style_label|>",
            "<|start_global_token|>"
        ])

        return prompt

    def build_prefix_ids(
        self,
        speaker: str,
        text: str,
        validate: bool = True
    ) -> List[int]:
        """
        Build prefix as token IDs (for testing/debugging).

        Args:
            speaker: Speaker name
            text: Text to synthesize (with inline emotions)
            validate: Check format correctness

        Returns:
            List of token IDs for the prefix
        """
        prompt = self.build_prefix(speaker, text, validate)
        token_ids = self.tokenizer.encode(prompt, add_special_tokens=False)
        return token_ids

    def _validate_inputs(self, speaker: str, text: str):
        """
        Validate speaker and text format.

        Args:
            speaker: Speaker name
            text: Text to synthesize (with inline emotions)

        Raises:
            ValueError: If inputs are invalid
        """
        # Validate speaker (case-sensitive!)
        if speaker not in INDIC_SPEAKERS:
            raise ValueError(
                f"Invalid speaker '{speaker}'. Must be one of (case-sensitive): "
                f"{', '.join(INDIC_SPEAKERS)}"
            )

        # Check text isn't empty
        if not text.strip():
            raise ValueError("Text cannot be empty")

        # Validate inline emotion tags in text (after normalization)
        self._validate_emotion_tags(text)

    def _validate_emotion_tags(self, text: str):
        """
        Validate emotion tags in text are in the allowed set for Spark TTS.

        Warns if unknown emotion tags are found (won't break, but won't work).

        Args:
            text: Text with potential emotion tags
        """
        # Find all [emotion] tags in text (Spark TTS format)
        emotion_pattern = r'\[([a-z\s]+)\]'
        found_emotions = re.findall(emotion_pattern, text)

        if not found_emotions:
            return  # No emotion tags, nothing to validate

        # Check each found emotion
        unknown_tags = []
        for emotion in found_emotions:
            tag = f'[{emotion}]'
            if tag not in INDIC_EMOTION_TAGS:
                unknown_tags.append(tag)

        if unknown_tags:
            print(f"⚠  Warning: Found {len(unknown_tags)} unknown emotion tag(s) for Spark TTS:")
            for tag in unknown_tags:
                print(f"    - {tag} (will be treated as regular text)")
            print(f"\n📋 Valid Spark TTS emotion tags:")
            for tag in INDIC_EMOTION_TAGS:
                print(f"    {tag}")
            print("\n")

    def extract_emotion_tags(self, text: str) -> List[Tuple[str, int]]:
        """
        Extract emotion tags and their positions from text.

        Args:
            text: Text with emotion tags

        Returns:
            List of (emotion_tag, position) tuples
        """
        emotion_pattern = r'\[([a-z\s]+)\]'
        matches = []

        for match in re.finditer(emotion_pattern, text):
            tag = match.group(0)
            if tag in INDIC_EMOTION_TAGS:
                matches.append((tag, match.start()))

        return matches

    def build_prefix_with_globals(
        self,
        speaker: str,
        text: str,
        global_ids: List[int],
        validate: bool = True
    ) -> str:
        """
        Build prompt with pre-generated global tokens for voice consistency in chunked generation.

        This is critical for multi-chunk text processing: the 32 global tokens encode
        speaker identity (voice DNA). By injecting the same global tokens from the first
        chunk into subsequent chunks, we ensure consistent voice across the entire text.

        Use Case:
        - Chunk 1: Use build_prefix() → model generates 32 global tokens + semantic tokens
        - Chunk 2+: Use build_prefix_with_globals() with captured global tokens
                    → model skips global generation, generates only semantic tokens

        Args:
            speaker: Speaker name (same as first chunk)
            text: Text chunk to synthesize
            global_ids: List of 32 global token IDs captured from first chunk
            validate: Check format correctness

        Returns:
            Formatted prompt with pre-filled global tokens

        Raises:
            ValueError: If global_ids doesn't contain exactly 32 tokens

        Thread Safety:
            This method is stateless and thread-safe. Global tokens are passed
            explicitly per-request, not stored in any shared state.
        """
        # Validate global tokens count
        EXPECTED_GLOBAL_COUNT = 32
        if len(global_ids) != EXPECTED_GLOBAL_COUNT:
            raise ValueError(
                f"Expected exactly {EXPECTED_GLOBAL_COUNT} global tokens, got {len(global_ids)}. "
                f"This likely indicates an issue with first chunk generation."
            )

        # Normalize emotion tags: <emotion> → [emotion] (legacy compatibility)
        text = normalize_emotion_tags(text)

        # Validate if requested
        if validate:
            self._validate_inputs(speaker, text)

        # Get speaker ID from mapping
        speaker_id = SPEAKER_MAP.get(speaker)
        if speaker_id is None:
            raise ValueError(f"Invalid speaker: {speaker}. Valid speakers: {list(SPEAKER_MAP.keys())}")

        # Build global tokens string: <|bicodec_global_0|><|bicodec_global_1|>...<|bicodec_global_31|>
        global_tokens_str = "".join([f"<|bicodec_global_{gid}|>" for gid in global_ids])


python - <<'PY'
import asyncio,time
from transformers import AutoTokenizer
from vllm import AsyncLLMEngine, AsyncEngineArgs, SamplingParams

MODEL_PATH='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM'
PROMPT='<|task_controllable_tts|><|start_content|>hello<|end_content|><|start_style_label|><|speaker_0|><|end_style_label|><|start_global_token|>'

async def main():
    print('loading tokenizer...')
    tok=AutoTokenizer.from_pretrained(MODEL_PATH, trust_remote_code=True)
    print('tokenizer loaded', len(tok))
    args=AsyncEngineArgs(
        model=MODEL_PATH,
        tokenizer=MODEL_PATH,
        tensor_parallel_size=1,
        trust_remote_code=True,
        dtype='bfloat16',
        gpu_memory_utilization=0.15,
        max_model_len=4096,
        enable_chunked_prefill=True,
        max_num_batched_tokens=4096,
        enable_prefix_caching=True,
        enforce_eager=False,
        disable_log_stats=True,
    )
    print('init engine...')
    eng=AsyncLLMEngine.from_engine_args(args)
    print('engine ready')

    sp=SamplingParams(temperature=0.8,top_k=50,top_p=1.0,max_tokens=128,repetition_penalty=1.05,stop=['<|im_end|>'],skip_special_tokens=False)

    gen=eng.generate(prompt=PROMPT,sampling_params=sp,request_id='dbg-1')
    it=gen.__aiter__()
    t0=time.time()
    print('waiting for first token...')
    try:
        out=await asyncio.wait_for(it.__anext__(), timeout=20)
        t1=time.time()
        print('got iter1', t1-t0, 'len tokens', len(out.outputs[0].token_ids), 'text_len', len(out.outputs[0].text))
    except Exception as e:
        print('first wait error',type(e),e)

    # if first came, drain few tokens
    try:
        for i in range(3):
            out=await asyncio.wait_for(it.__anext__(), timeout=20)
            print('iter',i+2,'len',len(out.outputs[0].token_ids),'text_tail',len(out.outputs[0].text))
    except Exception as e:
        print('drain err',type(e),e)

    print('done')

asyncio.run(main())
PY


Output:
Chunk ID: 9e0246
Wall time: 0.0512 seconds
Process exited with code 127
Original token count: 12
Output:
/bin/bash: line 1: python: command not found


python3 - <<'PY'
import asyncio,time
from transformers import AutoTokenizer
from vllm import AsyncLLMEngine, AsyncEngineArgs, SamplingParams

MODEL_PATH='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM'
PROMPT='<|task_controllable_tts|><|start_content|>hello<|end_content|><|start_style_label|><|speaker_0|><|end_style_label|><|start_global_token|>'

async def main():
    print('loading tokenizer...')
    tok=AutoTokenizer.from_pretrained(MODEL_PATH, trust_remote_code=True)
    print('tokenizer loaded', len(tok))
    args=AsyncEngineArgs(
        model=MODEL_PATH,
        tokenizer=MODEL_PATH,
        tensor_parallel_size=1,
        trust_remote_code=True,
        dtype='bfloat16',
        gpu_memory_utilization=0.15,
        max_model_len=4096,
        max_num_batched_tokens=4096,
        enable_prefix_caching=True,
        enable_chunked_prefill=True,
        enforce_eager=False,
        disable_log_stats=True,
    )
    print('init engine...')
    eng=AsyncLLMEngine.from_engine_args(args)
    print('engine ready')

    sp=SamplingParams(temperature=0.8,top_k=50,top_p=1.0,max_tokens=128,repetition_penalty=1.05,stop=['<|im_end|>'],skip_special_tokens=False)

    gen=eng.generate(prompt=PROMPT,sampling_params=sp,request_id='dbg-1')
    it=gen.__aiter__()
    t0=time.time()
    print('waiting for first token...')
    try:
        out=await asyncio.wait_for(it.__anext__(), timeout=20)
        t1=time.time()
        print('got iter1', t1-t0, 'len tokens', len(out.outputs[0].token_ids), 'text_len', len(out.outputs[0].text))
    except Exception as e:
        print('first wait error',type(e),e)
        return

    try:
        for i in range(5):
            out=await asyncio.wait_for(it.__anext__(), timeout=20)
            print('iter',i+2,'len',len(out.outputs[0].token_ids),'text_tail',len(out.outputs[0].text))
    except Exception as e:
        print('drain err',type(e),e)

    print('done')

asyncio.run(main())
PY


Output:
Chunk ID: 95b43a
Wall time: 1.0010 seconds
Process running with session ID 57130
Original token count: 0
Output:


Output:
Chunk ID: 07ef83
Wall time: 5.0008 seconds
Process running with session ID 57130
Original token count: 943
Output:
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
loading tokenizer...
tokenizer loaded 165158
init engine...
INFO 02-17 00:32:25 [model.py:541] Resolved architecture: Qwen2ForCausalLM
INFO 02-17 00:32:25 [model.py:1561] Using max model len 4096
INFO 02-17 00:32:26 [scheduler.py:226] Chunked prefill is enabled with max_num_batched_tokens=4096.
INFO 02-17 00:32:26 [vllm.py:624] Asynchronous scheduling is enabled.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(EngineCore_DP0 pid=3711282) INFO 02-17 00:32:26 [core.py:96] Initializing a V1 LLM engine (v0.15.1) with config: model='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', speculative_config=None, tokenizer='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [4096], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 256, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': True}, 'local_cache_dir': None, 'static_all_moe_layers': []}


Output:
Chunk ID: 538d0c
Wall time: 3.0748 seconds
Process exited with code 1
Original token count: 3506
Output:
(EngineCore_DP0 pid=3711282) INFO 02-17 00:32:29 [parallel_state.py:1212] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://216.81.248.184:34793 backend=nccl
(EngineCore_DP0 pid=3711282) INFO 02-17 00:32:29 [parallel_state.py:1423] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
(EngineCore_DP0 pid=3711282) ERROR 02-17 00:32:29 [core.py:946] EngineCore failed to start.
(EngineCore_DP0 pid=3711282) ERROR 02-17 00:32:29 [core.py:946] Traceback (most recent call last):
(EngineCore_DP0 pid=3711282) ERROR 02-17 00:32:29 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 937, in run_engine_core
(EngineCore_DP0 pid=3711282) ERROR 02-17 00:32:29 [core.py:946]     engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore_DP0 pid=3711282) ERROR 02-17 00:32:29 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 691, in __init__
(EngineCore_DP0 pid=3711282) ERROR 02-17 00:32:29 [core.py:946]     super().__init__(
(EngineCore_DP0 pid=3711282) ERROR 02-17 00:32:29 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 105, in __init__
(EngineCore_DP0 pid=3711282) ERROR 02-17 00:32:29 [core.py:946]     self.model_executor = executor_class(vllm_config)
(EngineCore_DP0 pid=3711282) ERROR 02-17 00:32:29 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/executor/abstract.py", line 101, in __init__
(EngineCore_DP0 pid=3711282) ERROR 02-17 00:32:29 [core.py:946]     self._init_executor()
(EngineCore_DP0 pid=3711282) ERROR 02-17 00:32:29 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/executor/uniproc_executor.py", line 47, in _init_executor
(EngineCore_DP0 pid=3711282) ERROR 02-17 00:32:29 [core.py:946]     self.driver_worker.init_device()
(EngineCore_DP0 pid=3711282) ERROR 02-17 00:32:29 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/worker/worker_base.py", line 326, in init_device
(EngineCore_DP0 pid=3711282) ERROR 02-17 00:32:29 [core.py:946]     self.worker.init_device()  # type: ignore
(EngineCore_DP0 pid=3711282) ERROR 02-17 00:32:29 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/worker/gpu_worker.py", line 258, in init_device
(EngineCore_DP0 pid=3711282) ERROR 02-17 00:32:29 [core.py:946]     from vllm.v1.worker.gpu_model_runner import (
(EngineCore_DP0 pid=3711282) ERROR 02-17 00:32:29 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/worker/gpu_model_runner.py", line 153, in <module>
(EngineCore_DP0 pid=3711282) ERROR 02-17 00:32:29 [core.py:946]     from vllm.v1.spec_decode.ngram_proposer import NgramProposer
(EngineCore_DP0 pid=3711282) ERROR 02-17 00:32:29 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/spec_decode/ngram_proposer.py", line 170, in <module>
(EngineCore_DP0 pid=3711282) ERROR 02-17 00:32:29 [core.py:946]     def batch_propose_numba(
(EngineCore_DP0 pid=3711282) ERROR 02-17 00:32:29 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/decorators.py", line 221, in wrapper
(EngineCore_DP0 pid=3711282) ERROR 02-17 00:32:29 [core.py:946]     disp = dispatcher(py_func=func, locals=locals,
(EngineCore_DP0 pid=3711282) ERROR 02-17 00:32:29 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/dispatcher.py", line 772, in __init__
(EngineCore_DP0 pid=3711282) ERROR 02-17 00:32:29 [core.py:946]     self.targetctx = self.targetdescr.target_context
(EngineCore_DP0 pid=3711282) ERROR 02-17 00:32:29 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/registry.py", line 30, in target_context
(EngineCore_DP0 pid=3711282) ERROR 02-17 00:32:29 [core.py:946]     return self._toplevel_target_context
(EngineCore_DP0 pid=3711282) ERROR 02-17 00:32:29 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/utils.py", line 741, in __get__
(EngineCore_DP0 pid=3711282) ERROR 02-17 00:32:29 [core.py:946]     return super().__get__(*args, **kwargs)
(EngineCore_DP0 pid=3711282) ERROR 02-17 00:32:29 [core.py:946]   File "/usr/lib/python3.10/functools.py", line 981, in __get__
(EngineCore_DP0 pid=3711282) ERROR 02-17 00:32:29 [core.py:946]     val = self.func(instance)
(EngineCore_DP0 pid=3711282) ERROR 02-17 00:32:29 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/registry.py", line 18, in _toplevel_target_context
(EngineCore_DP0 pid=3711282) ERROR 02-17 00:32:29 [core.py:946]     return cpu.CPUContext(self.typing_context, self._target_name)
(EngineCore_DP0 pid=3711282) ERROR 02-17 00:32:29 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/cpu.py", line 42, in __init__
(EngineCore_DP0 pid=3711282) ERROR 02-17 00:32:29 [core.py:946]     super().__init__(typingctx, target)
(EngineCore_DP0 pid=3711282) ERROR 02-17 00:32:29 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/base.py", line 254, in __init__
(EngineCore_DP0 pid=3711282) ERROR 02-17 00:32:29 [core.py:946]     self.init()
(EngineCore_DP0 pid=3711282) ERROR 02-17 00:32:29 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/compiler_lock.py", line 35, in _acquire_compile_lock
(EngineCore_DP0 pid=3711282) ERROR 02-17 00:32:29 [core.py:946]     return func(*args, **kwargs)
(EngineCore_DP0 pid=3711282) ERROR 02-17 00:32:29 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/cpu.py", line 51, in init
(EngineCore_DP0 pid=3711282) ERROR 02-17 00:32:29 [core.py:946]     self._internal_codegen = codegen.JITCPUCodegen("numba.exec")
(EngineCore_DP0 pid=3711282) ERROR 02-17 00:32:29 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/codegen.py", line 1172, in __init__
(EngineCore_DP0 pid=3711282) ERROR 02-17 00:32:29 [core.py:946]     initialize_llvm()
(EngineCore_DP0 pid=3711282) ERROR 02-17 00:32:29 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/codegen.py", line 1434, in initialize_llvm
(EngineCore_DP0 pid=3711282) ERROR 02-17 00:32:29 [core.py:946]     ll.initialize()
(EngineCore_DP0 pid=3711282) ERROR 02-17 00:32:29 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/llvmlite/binding/initfini.py", line 17, in initialize
(EngineCore_DP0 pid=3711282) ERROR 02-17 00:32:29 [core.py:946]     raise RuntimeError(
(EngineCore_DP0 pid=3711282) ERROR 02-17 00:32:29 [core.py:946] RuntimeError: llvmlite.binding.initialize() is deprecated and will be removed. LLVM initialization is now handled automatically. Please remove calls to this function from your code and check for other behavioral changes that may have occurred due to LLVM updates.
(EngineCore_DP0 pid=3711282) Process EngineCore_DP0:
(EngineCore_DP0 pid=3711282) Traceback (most recent call last):
(EngineCore_DP0 pid=3711282)   File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore_DP0 pid=3711282)     self.run()
(EngineCore_DP0 pid=3711282)   File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
(EngineCore_DP0 pid=3711282)     self._target(*self._args, **self._kwargs)
(EngineCore_DP0 pid=3711282)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 950, in run_engine_core
(EngineCore_DP0 pid=3711282)     raise e
(EngineCore_DP0 pid=3711282)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 937, in run_engine_core
(EngineCore_DP0 pid=3711282)     engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore_DP0 pid=3711282)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 691, in __init__
(EngineCore_DP0 pid=3711282)     super().__init__(
(EngineCore_DP0 pid=3711282)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 105, in __init__
(EngineCore_DP0 pid=3711282)     self.model_executor = executor_class(vllm_config)
(EngineCore_DP0 pid=3711282)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/executor/abstract.py", line 101, in __init__
(EngineCore_DP0 pid=3711282)     self._init_executor()
(EngineCore_DP0 pid=3711282)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/executor/uniproc_executor.py", line 47, in _init_executor
(EngineCore_DP0 pid=3711282)     self.driver_worker.init_device()
(EngineCore_DP0 pid=3711282)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/worker/worker_base.py", line 326, in init_device
(EngineCore_DP0 pid=3711282)     self.worker.init_device()  # type: ignore
(EngineCore_DP0 pid=3711282)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/worker/gpu_worker.py", line 258, in init_device
(EngineCore_DP0 pid=3711282)     from vllm.v1.worker.gpu_model_runner import (
(EngineCore_DP0 pid=3711282)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/worker/gpu_model_runner.py", line 153, in <module>
(EngineCore_DP0 pid=3711282)     from vllm.v1.spec_decode.ngram_proposer import NgramProposer
(EngineCore_DP0 pid=3711282)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/spec_decode/ngram_proposer.py", line 170, in <module>
(EngineCore_DP0 pid=3711282)     def batch_propose_numba(
(EngineCore_DP0 pid=3711282)   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/decorators.py", line 221, in wrapper
(EngineCore_DP0 pid=3711282)     disp = dispatcher(py_func=func, locals=locals,
(EngineCore_DP0 pid=3711282)   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/dispatcher.py", line 772, in __init__
(EngineCore_DP0 pid=3711282)     self.targetctx = self.targetdescr.target_context
(EngineCore_DP0 pid=3711282)   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/registry.py", line 30, in target_context
(EngineCore_DP0 pid=3711282)     return self._toplevel_target_context
(EngineCore_DP0 pid=3711282)   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/utils.py", line 741, in __get__
(EngineCore_DP0 pid=3711282)     return super().__get__(*args, **kwargs)
(EngineCore_DP0 pid=3711282)   File "/usr/lib/python3.10/functools.py", line 981, in __get__
(EngineCore_DP0 pid=3711282)     val = self.func(instance)
(EngineCore_DP0 pid=3711282)   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/registry.py", line 18, in _toplevel_target_context
(EngineCore_DP0 pid=3711282)     return cpu.CPUContext(self.typing_context, self._target_name)
(EngineCore_DP0 pid=3711282)   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/cpu.py", line 42, in __init__
(EngineCore_DP0 pid=3711282)     super().__init__(typingctx, target)
(EngineCore_DP0 pid=3711282)   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/base.py", line 254, in __init__
(EngineCore_DP0 pid=3711282)     self.init()
(EngineCore_DP0 pid=3711282)   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/compiler_lock.py", line 35, in _acquire_compile_lock
(EngineCore_DP0 pid=3711282)     return func(*args, **kwargs)
(EngineCore_DP0 pid=3711282)   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/cpu.py", line 51, in init
(EngineCore_DP0 pid=3711282)     self._internal_codegen = codegen.JITCPUCodegen("numba.exec")
(EngineCore_DP0 pid=3711282)   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/codegen.py", line 1172, in __init__
(EngineCore_DP0 pid=3711282)     initialize_llvm()
(EngineCore_DP0 pid=3711282)   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/codegen.py", line 1434, in initialize_llvm
(EngineCore_DP0 pid=3711282)     ll.initialize()
(EngineCore_DP0 pid=3711282)   File "/home/ubuntu/.local/lib/python3.10/site-packages/llvmlite/binding/initfini.py", line 17, in initialize
(EngineCore_DP0 pid=3711282)     raise RuntimeError(
(EngineCore_DP0 pid=3711282) RuntimeError: llvmlite.binding.initialize() is deprecated and will be removed. LLVM initialization is now handled automatically. Please remove calls to this function from your code and check for other behavioral changes that may have occurred due to LLVM updates.
Traceback (most recent call last):
  File "<stdin>", line 53, in <module>
  File "/usr/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "<stdin>", line 27, in main
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/async_llm.py", line 257, in from_engine_args
    return cls(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/async_llm.py", line 155, in __init__
    self.engine_core = EngineCoreClient.make_async_mp_client(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 122, in make_async_mp_client
    return AsyncMPClient(*client_args)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 819, in __init__
    super().__init__(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 479, in __init__
    with launch_core_engines(vllm_config, executor_class, log_stats) as (
  File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
    next(self.gen)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/utils.py", line 933, in launch_core_engines
    wait_for_engine_startup(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/utils.py", line 992, in wait_for_engine_startup
    raise RuntimeError(
RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}


I confirmed this error is the known llvmlite/numba crash unless `NUMBA_DISABLE_JIT=1`. I’ll rerun with that env to isolate whether generation itself hangs after engine initialization.


NUMBA_DISABLE_JIT=1 python3 - <<'PY'
import asyncio,time
from transformers import AutoTokenizer
from vllm import AsyncLLMEngine, AsyncEngineArgs, SamplingParams

MODEL_PATH='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM'
PROMPT='<|task_controllable_tts|><|start_content|>hello<|end_content|><|start_style_label|><|speaker_0|><|end_style_label|><|start_global_token|>'

async def main():
    print('loading tokenizer...')
    tok=AutoTokenizer.from_pretrained(MODEL_PATH, trust_remote_code=True)
    print('tokenizer loaded', len(tok))
    args=AsyncEngineArgs(
        model=MODEL_PATH,
        tokenizer=MODEL_PATH,
        tensor_parallel_size=1,
        trust_remote_code=True,
        dtype='bfloat16',
        gpu_memory_utilization=0.15,
        max_model_len=4096,
        max_num_batched_tokens=4096,
        enable_prefix_caching=True,
        enable_chunked_prefill=True,
        enforce_eager=False,
        disable_log_stats=True,
    )
    print('init engine...')
    eng=AsyncLLMEngine.from_engine_args(args)
    print('engine ready')

    sp=SamplingParams(temperature=0.8,top_k=50,top_p=1.0,max_tokens=128,repetition_penalty=1.05,stop=['<|im_end|>'],skip_special_tokens=False)

    gen=eng.generate(prompt=PROMPT,sampling_params=sp,request_id='dbg-1')
    it=gen.__aiter__()
    t0=time.time()
    print('waiting for first token...')
    try:
        out=await asyncio.wait_for(it.__anext__(), timeout=20)
        t1=time.time()
        print('got iter1', t1-t0, 'len tokens', len(out.outputs[0].token_ids), 'text_len', len(out.outputs[0].text))
    except Exception as e:
        print('first wait error',type(e),e)
        return

    try:
        for i in range(5):
            out=await asyncio.wait_for(it.__anext__(), timeout=20)
            print('iter',i+2,'len',len(out.outputs[0].token_ids),'text_tail',len(out.outputs[0].text))
    except Exception as e:
        print('drain err',type(e),e)

    print('done')

asyncio.run(main())
PY


Output:
Chunk ID: d9db2f
Wall time: 1.0004 seconds
Process running with session ID 52179
Original token count: 0
Output:


Output:
Chunk ID: 1fc08a
Wall time: 5.0016 seconds
Process running with session ID 52179
Original token count: 943
Output:
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
loading tokenizer...
tokenizer loaded 165158
init engine...
INFO 02-17 00:32:39 [model.py:541] Resolved architecture: Qwen2ForCausalLM
INFO 02-17 00:32:39 [model.py:1561] Using max model len 4096
INFO 02-17 00:32:39 [scheduler.py:226] Chunked prefill is enabled with max_num_batched_tokens=4096.
INFO 02-17 00:32:39 [vllm.py:624] Asynchronous scheduling is enabled.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(EngineCore_DP0 pid=3711448) INFO 02-17 00:32:40 [core.py:96] Initializing a V1 LLM engine (v0.15.1) with config: model='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', speculative_config=None, tokenizer='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [4096], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 256, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': True}, 'local_cache_dir': None, 'static_all_moe_layers': []}


Output:
Chunk ID: 1972ce
Wall time: 5.0004 seconds
Process running with session ID 52179
Original token count: 437
Output:
(EngineCore_DP0 pid=3711448) INFO 02-17 00:32:42 [parallel_state.py:1212] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://216.81.248.184:36711 backend=nccl
(EngineCore_DP0 pid=3711448) INFO 02-17 00:32:42 [parallel_state.py:1423] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
(EngineCore_DP0 pid=3711448) INFO 02-17 00:32:43 [gpu_model_runner.py:4033] Starting to load model /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM...
(EngineCore_DP0 pid=3711448) INFO 02-17 00:32:44 [cuda.py:364] Using FLASH_ATTN attention backend out of potential backends: ('FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION')
(EngineCore_DP0 pid=3711448) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
(EngineCore_DP0 pid=3711448) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
(EngineCore_DP0 pid=3711448)
Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
(EngineCore_DP0 pid=3711448)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.58it/s]
(EngineCore_DP0 pid=3711448)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.58it/s]
(EngineCore_DP0 pid=3711448)
(EngineCore_DP0 pid=3711448) INFO 02-17 00:32:44 [default_loader.py:291] Loading weights took 0.71 seconds
(EngineCore_DP0 pid=3711448) INFO 02-17 00:32:45 [gpu_model_runner.py:4130] Model loading took 0.95 GiB memory and 1.247482 seconds


Output:
Chunk ID: 2ce8d6
Wall time: 5.0014 seconds
Process running with session ID 52179
Original token count: 398
Output:
(EngineCore_DP0 pid=3711448) INFO 02-17 00:32:50 [backends.py:812] Using cache directory: /home/ubuntu/.cache/vllm/torch_compile_cache/f3ff13511c/rank_0_0/backbone for vLLM's torch.compile
(EngineCore_DP0 pid=3711448) INFO 02-17 00:32:50 [backends.py:872] Dynamo bytecode transform time: 4.19 s
(EngineCore_DP0 pid=3711448) INFO 02-17 00:32:52 [backends.py:267] Directly load the compiled graph(s) for compile range (1, 4096) from the cache, took 0.648 s
(EngineCore_DP0 pid=3711448) INFO 02-17 00:32:52 [monitor.py:34] torch.compile takes 4.84 s in total
(EngineCore_DP0 pid=3711448) INFO 02-17 00:32:53 [gpu_worker.py:356] Available KV cache memory: 10.13 GiB
(EngineCore_DP0 pid=3711448) INFO 02-17 00:32:53 [kv_cache_utils.py:1307] GPU KV cache size: 885,184 tokens
(EngineCore_DP0 pid=3711448) INFO 02-17 00:32:53 [kv_cache_utils.py:1312] Maximum concurrency for 4,096 tokens per request: 216.11x
(EngineCore_DP0 pid=3711448)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   0%|          | 0/35 [00:00<?, ?it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  11%|█▏        | 4/35 [00:00<00:00, 32.77it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  23%|██▎       | 8/35 [00:00<00:00, 35.61it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  34%|███▍      | 12/35 [00:00<00:00, 36.71it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  46%|████▌     | 16/35 [00:00<00:00, 37.11it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  57%|█████▋    | 20/35 [00:00<00:00, 37.44it/s]


Output:
Chunk ID: 4e9e72
Wall time: 3.5094 seconds
Process exited with code 0
Original token count: 422
Output:

Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  69%|██████▊   | 24/35 [00:00<00:00, 37.43it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  80%|████████  | 28/35 [00:00<00:00, 37.07it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  91%|█████████▏| 32/35 [00:00<00:00, 35.42it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 35/35 [00:00<00:00, 35.52it/s]
(EngineCore_DP0 pid=3711448)
Capturing CUDA graphs (decode, FULL):   0%|          | 0/19 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL):  16%|█▌        | 3/19 [00:00<00:00, 29.96it/s]
Capturing CUDA graphs (decode, FULL):  37%|███▋      | 7/19 [00:00<00:00, 33.84it/s]
Capturing CUDA graphs (decode, FULL):  58%|█████▊    | 11/19 [00:00<00:00, 35.15it/s]
Capturing CUDA graphs (decode, FULL):  79%|███████▉  | 15/19 [00:00<00:00, 35.40it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 35.93it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 35.19it/s]
(EngineCore_DP0 pid=3711448) INFO 02-17 00:32:56 [gpu_model_runner.py:5063] Graph capturing finished in 2 secs, took 0.28 GiB
(EngineCore_DP0 pid=3711448) INFO 02-17 00:32:56 [core.py:272] init engine (profile, create kv cache, warmup model) took 10.80 seconds
engine ready
waiting for first token...
got iter1 0.04355049133300781 len tokens 1 text_len 14
iter 2 len 2 text_tail 36
iter 3 len 3 text_tail 59
iter 4 len 4 text_tail 82
iter 5 len 5 text_tail 105
iter 6 len 6 text_tail 128
done


rg -n "def _handle_streaming_request|def _handle_non_streaming_request|return Response\(|Request ID|request_id" veena3modal/api/fastapi_app.py && sed -n '620,980p' veena3modal/api/fastapi_app.py