Output:
Chunk ID: db76bb
Wall time: 0.0510 seconds
Process exited with code 0
Original token count: 37182
Output:
Total output lines: 1530

veena3modal/local_server.py:265:    logger.info(f"  Streaming: {'available' if runtime.streaming_pipeline else 'unavailable'}")
veena3modal/local_server.py:389:        help="Precompute speaker global tokens at startup (streaming-only optimization, experimental)",
veena3modal/app.py:5:- True streaming TTS
veena3modal/app.py:148:    # Prevents CPU throttling during vLLM scheduler + streaming loop
veena3modal/app.py:149:    # Modal default is only 0.125 cores which can bottleneck streaming
veena3modal/app.py:170:    timeout=120,             # 2 min max per request (reduced from 10 min)
veena3modal/app.py:179:# vLLM continuous batching handles concurrent requests efficiently
veena3modal/app.py:244:        Return FastAPI ASGI app for serving requests.
veena3modal/app.py:271:    Note: Model will be loaded on first request (higher TTFB).
veena3modal/services/tts_runtime.py:4:In Modal, each container should load the model once and then handle many requests.
veena3modal/services/tts_runtime.py:40:    streaming_pipeline: Any = None
veena3modal/services/tts_runtime.py:56:    # OPTIMIZATION: Pre-computed global tokens per speaker (eliminates ~110ms pre-roll in streaming)
veena3modal/services/tts_runtime.py:216:        # Initialize streaming pipeline (for M4)
veena3modal/services/tts_runtime.py:217:        streaming_pipeline = None
veena3modal/services/tts_runtime.py:219:            from veena3modal.core.streaming_pipeline import Veena3SlidingWindowPipeline
veena3modal/services/tts_runtime.py:220:            logger.info("Initializing streaming pipeline...")
veena3modal/services/tts_runtime.py:222:            streaming_pipeline = Veena3SlidingWindowPipeline(
veena3modal/services/tts_runtime.py:228:            logger.warning(f"Streaming pipeline not available: {e}")
veena3modal/services/tts_runtime.py:230:        # Initialize long text processor once (avoid per-request construction/logging)
veena3modal/services/tts_runtime.py:233:            streaming_pipeline=streaming_pipeline,
veena3modal/services/tts_runtime.py:263:            streaming_pipeline=streaming_pipeline,
veena3modal/services/tts_runtime.py:311:    OPTIMIZATION: BiCodec streaming requires 32 "global tokens" before any audio can be emitted.
veena3modal/services/tts_runtime.py:314:    and cache them. During streaming, we inject cached globals via build_prefix_with_globals(),
veena3modal/services/tts_runtime.py:317:    This runs synchronously at startup (adds ~5-10s to cold start, saves ~110ms per streaming TTFB).
veena3modal/services/tts_runtime.py:344:        request_id = f"warmup-{speaker_name}-{int(time.time() * 1000)}"
veena3modal/services/tts_runtime.py:347:        async for request_output in runtime.model.engine.generate(
veena3modal/services/tts_runtime.py:350:            request_id=request_id,
veena3modal/services/tts_runtime.py:352:            final_output = request_output
veena3modal/services/tts_runtime.py:404:            streaming_pipeline=runtime.streaming_pipeline,
veena3modal/services/tts_runtime.py:421:    Generate speech audio (non-streaming).
veena3modal/services/tts_runtime.py:474:        metrics["ttfb_ms"] = metrics["generation_ms"]  # Non-streaming: TTFB = total time
veena3modal/services/tts_runtime.py:477:            # Apply super-resolution if requested and available
veena3modal/services/tts_runtime.py:661:            # Apply super-resolution if requested and available
veena3modal/services/tts_runtime.py:692:async def generate_speech_streaming(
veena3modal/services/tts_runtime.py:704:    Generate speech audio with true streaming (yields chunks as they're generated).
veena3modal/services/tts_runtime.py:706:    This is the core streaming implementation for M4.
veena3modal/services/tts_runtime.py:728:        RuntimeError: If runtime not initialized or streaming pipeline unavailable
veena3modal/services/tts_runtime.py:735:    if runtime.streaming_pipeline is None:
veena3modal/services/tts_runtime.py:736:        raise RuntimeError("Streaming pipeline not available")
veena3modal/services/tts_runtime.py:760:        # Chunked streaming with voice consistency (global token caching)
veena3modal/services/tts_runtime.py:762:        async for audio_chunk, chunk_metrics in _stream_chunked_text(
veena3modal/services/tts_runtime.py:770:                # Create streaming WAV header (size=0 for unknown length)
veena3modal/services/tts_runtime.py:781:        # Simple streaming (no text chunking)
veena3modal/services/tts_runtime.py:789:            stream_gen = runtime.streaming_pipeline.generate_speech_stream_indic_continuation(
veena3modal/services/tts_runtime.py:803:            stream_gen = runtime.streaming_pipeline.generate_speech_stream_indic(
veena3modal/services/tts_runtime.py:815:        async for stream_item in stream_gen:
veena3modal/services/tts_runtime.py:816:            stream_metrics: Dict[str, Any] = {}
veena3modal/services/tts_runtime.py:817:            audio_chunk = stream_item
veena3modal/services/tts_runtime.py:819:                isinstance(stream_item, tuple)
veena3modal/services/tts_runtime.py:820:                and len(stream_item) == 2
veena3modal/services/tts_runtime.py:821:                and isinstance(stream_item[1], dict)
veena3modal/services/tts_runtime.py:823:                audio_chunk = stream_item[0]
veena3modal/services/tts_runtime.py:824:                stream_metrics = stream_item[1]
veena3modal/services/tts_runtime.py:826:            if stream_metrics:
veena3modal/services/tts_runtime.py:827:                metrics.update(stream_metrics)
veena3modal/services/tts_runtime.py:830:                # Metrics-only event from streaming pipeline.
veena3modal/services/tts_runtime.py:848:async def _stream_chunked_text(
veena3modal/services/tts_runtime.py:862:    Internal helper: stream chunked text with voice consistency.
veena3modal/services/tts_runtime.py:883:            async for audio_bytes, global_ids in runtime.streaming_pipeline.generate_speech_stream_indic_first_chunk(
veena3modal/services/tts_runtime.py:909:                logger.warning(f"No captured globals for chunk {i+1}, using regular streaming")
veena3modal/services/tts_runtime.py:910:                async for audio_bytes in runtime.streaming_pipeline.generate_speech_stream_indic(
veena3modal/services/tts_runtime.py:930:                async for audio_bytes in runtime.streaming_pipeline.generate_speech_stream_indic_continuation(
veena3modal/services/credits.py:4:Lightweight, framework-agnostic credits calculation for TTS requests.
veena3modal/services/credits.py:34:    Calculate credits consumed for a TTS request.
veena3modal/services/credits.py:84:    Check if user has sufficient credits for a request.
veena3modal/services/credits.py:140:    Estimate credits for a request (before generation).
veena3modal/services/sentence_store.py:2:Sentence Store Service - Supabase storage for TTS requests.
veena3modal/services/sentence_store.py:4:Stores all request text + metadata in Supabase for analytics and auditing.
veena3modal/services/sentence_store.py:21:TABLE_NAME = "tts_requests"
veena3modal/services/sentence_store.py:49:    request_id: str,
veena3modal/services/sentence_store.py:52:    stream: bool = False,
veena3modal/services/sentence_store.py:68:        request_id: Unique request identifier (X-Request-ID)
veena3modal/services/sentence_store.py:71:        stream: Whether streaming was enabled
veena3modal/services/sentence_store.py:87:        "request_id": request_id,
veena3modal/services/sentence_store.py:91:        "stream": stream,
veena3modal/services/sentence_store.py:108:    Service for storing TTS request text and metadata in Supabase.
veena3modal/services/sentence_store.py:144:        request_id: str,
veena3modal/services/sentence_store.py:147:        stream: bool = False,
veena3modal/services/sentence_store.py:160:        Store a TTS request record in Supabase.
veena3modal/services/sentence_store.py:169:            logger.debug(f"Supabase not configured, skipping store for {request_id}")
veena3modal/services/sentence_store.py:174:                request_id=request_id,
veena3modal/services/sentence_store.py:177:                stream=stream,
veena3modal/services/sentence_store.py:198:            logger.debug(f"Stored sentence for request {request_id}")
veena3modal/services/sentence_store.py:202:            logger.warning(f"Failed to store sentence for {request_id}: {e}")
veena3modal/services/sentence_store.py:207:        request_id: str,
veena3modal/services/sentence_store.py:213:        Store a TTS request record without blocking.
veena3modal/services/sentence_store.py:230:                    request_id=request_id,
veena3modal/services/sentence_store.py:237:                logger.warning(f"Background store failed for {request_id}: {e}")
veena3modal/core/token_utils.py:14:This eliminates the O(n²) CPU burn in the streaming hot loop.
veena3modal/core/token_utils.py:69:        This avoids lazy cache misses during streaming.
veena3modal/core/token_utils.py:141:        This is the main method for incremental streaming:
veena3modal/audio/crossfade.py:34:      - to_emit: bytes to stream now
veena3modal/audio/encoder.py:6:- Opus (compressed, good for streaming)
veena3modal/audio/encoder.py:32:        data_size: Size of audio data in bytes (0 = unknown/streaming)
veena3modal/audio/encoder.py:38:        For streaming with unknown size, set data_size=0.
veena3modal/audio/encoder.py:101:    def supports_streaming(self) -> bool:
veena3modal/audio/encoder.py:102:        """Check if this encoder supports streaming."""
veena3modal/audio/encoder.py:111:    Supports streaming since it's uncompressed.
veena3modal/audio/encoder.py:151:    def supports_streaming(self) -> bool:
veena3modal/audio/encoder.py:152:        """WAV supports streaming."""
veena3modal/audio/encoder.py:155:    def create_streaming_header(self) -> bytes:
veena3modal/audio/encoder.py:157:        Create WAV header for streaming (unknown size).
veena3modal/audio/encoder.py:160:            WAV header with data_size=0 for streaming
veena3modal/audio/encoder.py:166:            data_size=0  # Unknown size for streaming
veena3modal/audio/encoder.py:174:    Opus is optimized for internet streaming with low latency.
veena3modal/audio/encoder.py:229:    def supports_streaming(self) -> bool:
veena3modal/audio/encoder.py:230:        """Opus does not support chunk-by-chunk streaming (needs full file)."""
veena3modal/audio/encoder.py:290:    def supports_streaming(self) -> bool:
veena3modal/audio/encoder.py:291:        """MP3 does not support chunk-by-chunk streaming."""
veena3modal/audio/encoder.py:344:    def supports_streaming(self) -> bool:
veena3modal/audio/encoder.py:345:        """mu-law WAV could support streaming but needs resampling first."""
veena3modal/audio/encoder.py:405:    def supports_streaming(self) -> bool:
veena3modal/audio/encoder.py:406:        """FLAC does not support chunk-by-chunk streaming."""
veena3modal/core/bicodec_decoder.py:75:class _BatchDecodeRequest:
veena3modal/core/bicodec_decoder.py:91:    Supports streaming with sliding window approach (like SNAC).
veena3modal/core/bicodec_decoder.py:98:        enable_batching: bool = False,  # For compatibility with streaming pipeline
veena3modal/core/bicodec_decoder.py:106:            enable_batching: Enable async batching for streaming (not used yet)
veena3modal/core/bicodec_decoder.py:110:        requested_batching = bool(enable_batching)
veena3modal/core/bicodec_decoder.py:112:            requested_batching = True
veena3modal/core/bicodec_decoder.py:114:            requested_batching = False
veena3modal/core/bicodec_decoder.py:115:        self.enable_batching = requested_batching and self.device.type == "cuda"
veena3modal/core/bicodec_decoder.py:126:        self._batch_queues: Dict[int, asyncio.Queue] = {}
veena3modal/core/bicodec_decoder.py:130:        self._batch_total_requests = 0
veena3modal/core/bicodec_decoder.py:132:        self._batch_queue_wait_ms_total = 0.0
veena3modal/core/bicodec_decoder.py:133:        self._batch_queue_wait_ms_min: Optional[float] = None
veena3modal/core/bicodec_decoder.py:134:        self._batch_queue_wait_ms_max = 0.0
veena3modal/core/bicodec_decoder.py:138:        self._batch_queue_depth_total = 0
veena3modal/core/bicodec_decoder.py:139:        self._batch_queue_depth_max = 0
veena3modal/core/bicodec_decoder.py:140:        self._batch_inflight_requests = 0
veena3modal/core/bicodec_decoder.py:267:        under high request concurrency.
veena3modal/core/bicodec_decoder.py:284:    def _decode_streaming_batch_sync(
veena3modal/core/bicodec_decoder.py:289:        Run a single batched BiCodec forward pass for multiple stream decode requests.
veena3modal/core/bicodec_decoder.py:291:        Falls back to sequential decode_streaming if shapes or settings are not batch-safe.
veena3modal/core/bicodec_decoder.py:297:            return [self.decode_streaming(sem, glob, use_sliding_window=use_sw, trim_warmup=trim)]
veena3modal/core/bicodec_decoder.py:302:                self.decode_streaming(sem, glob, use_sliding_window=use_sw, trim_warmup=trim)
veena3modal/core/bicodec_decoder.py:306:        # Decoder needs at least 8 semantic tokens. Use fallback if any request is below.
veena3modal/core/bicodec_decoder.py:309:                self.decode_streaming(sem, glob, use_sliding_window=use_sw, trim_warmup=trim)
veena3modal/core/bicodec_decoder.py:362:                self.decode_streaming(sem, glob, use_sliding_window=use_sw, trim_warmup=trim)
veena3modal/core/bicodec_decoder.py:366:    def _ensure_batch_worker(self, loop: asyncio.AbstractEventLoop) -> asyncio.Queue:
veena3modal/core/bicodec_decoder.py:369:            queue = self._batch_queues.get(key)
veena3modal/core/bicodec_decoder.py:373:            if queue is None:
veena3modal/core/bicodec_decoder.py:374:                queue = asyncio.Queue(maxsize=max(self.batch_max_size * 8, 256))
veena3modal/core/bicodec_decoder.py:375:                self._batch_queues[key] = queue
veena3modal/core/bicodec_decoder.py:379:                pending_now = queue.qsize() + self._batch_inflight_requests
veena3modal/core/bicodec_decoder.py:391:                    self._batch_worker(loop, queue, worker_idx),
veena3modal/core/bicodec_decoder.py:398:            return queue
veena3modal/core/bicodec_decoder.py:400:    def _desired_worker_count(self, queue: Optional[asyncio.Queue] = None) -> int:
veena3modal/core/bicodec_decoder.py:406:            pending_now = self._batch_inflight_requests
veena3modal/core/bicodec_decoder.py:407:        if queue is not None:
veena3modal/core/bicodec_decoder.py:408:            pending_now += queue.qsize()
veena3modal/core/bicodec_decoder.py:416:        queue: asyncio.Queue,
veena3modal/core/bicodec_decoder.py:421:                target_workers = self._desired_worker_count(queue)
veena3modal/core/bicodec_decoder.py:427:            first: _BatchDecodeRequest = await queue.get()
veena3modal/core/bicodec_decoder.py:428:            batch: List[_BatchDecodeRequest] = [first]
veena3modal/core/bicodec_decoder.py:436:                    req: _BatchDecodeRequest = await asyncio.wait_for(queue.get(), timeout=remaining)
veena3modal/core/bicodec_decoder.py:446:            queue_wait_ms = [
veena3modal/core/bicodec_decoder.py:450:            queue_depth_now = queue.qsize()
veena3modal/core/bicodec_decoder.py:451:            self._batch_queue_depth_total += queue_depth_now
veena3modal/core/bicodec_decoder.py:452:            if queue_depth_now > self._batch_queue_depth_max:
veena3modal/core/bicodec_decoder.py:453:                self._batch_queue_depth_max = queue_depth_now
veena3modal/core/bicodec_decoder.py:454:            for wait_ms in queue_wait_ms:
veena3modal/core/bicodec_decoder.py:455:                self._batch_queue_wait_ms_total += wait_ms
veena3modal/core/bicodec_decoder.py:456:                if self._batch_queue_wait_ms_min is None or wait_ms < self._batch_queue_wait_ms_min:
veena3modal/core/bicodec_decoder.py:457:                    self._batch_queue_wait_ms_min = wait_ms
veena3modal/core/bicodec_decoder.py:458:                if wait_ms > self._batch_queue_wait_ms_max:
veena3modal/core/bicodec_decoder.py:459:                    self._batch_queue_wait_ms_max = wait_ms
veena3modal/core/bicodec_decoder.py:461:                self._batch_inflight_requests += len(batch)
veena3modal/core/bicodec_decoder.py:465:                results = await loop.run_in_executor(None, self._decode_streaming_batch_sync, batch_inputs)
veena3modal/core/bicodec_decoder.py:486:                self._batch_total_requests += len(batch)
veena3modal/core/bicodec_decoder.py:490:                    self._batch_inflight_requests = max(0, self._batch_inflight_requests - len(batch))
veena3modal/core/bicodec_decoder.py:492:    def get_pending_requests(self) -> int:
veena3modal/core/bicodec_decoder.py:496:            pending = sum(queue.qsize() for queue in self._batch_queues.values())
veena3modal/core/bicodec_decoder.py:497:            return int(max(0, pending + self._batch_inflight_requests))
veena3modal/core/bicodec_decoder.py:501:        reqs = max(1, self._batch_total_requests)
veena3modal/core/bicodec_decoder.py:512:            "batch_total_requests": float(self._batch_total_requests),
veena3modal/core/bicodec_decoder.py:513:            "batch_avg_size": float(self._batch_total_requests / batches),
veena3modal/core/bicodec_decoder.py:520:            "batch_queue_wait_ms_avg": float(self._batch_queue_wait_ms_total / reqs),
veena3modal/core/bicodec_decoder.py:521:            "batch_queue_wait_ms_min": float(self._batch_queue_wait_ms_min or 0.0),
veena3modal/core/bicodec_decoder.py:522:            "batch_queue_wait_ms_max": float(self._batch_queue_wait_ms_max),
veena3modal/core/bicodec_decoder.py:526:            "batch_queue_depth_avg": float(self._batch_queue_depth_total / batches),
veena3modal/core/bicodec_decoder.py:527:            "batch_queue_depth_max": float(self._batch_queue_depth_max),
veena3modal/core/bicodec_decoder.py:528:            "batch_pending_now": float(self.get_pending_requests()),
veena3modal/core/bicodec_decoder.py:578:    def decode_streaming(
veena3modal/core/bicodec_decoder.py:586:        Decode BiCodec tokens with streaming support.
veena3modal/core/bicodec_decoder.py:634:            # Keep middle samples for overlap-add streaming
veena3modal/core/bicodec_decoder.py:656:        Async wrapper for streaming decode - runs in executor to avoid blocking event loop.
veena3modal/core/bicodec_decoder.py:660:        so other streams can progress while this decode runs on GPU.
veena3modal/core/bicodec_decoder.py:673:            queue = self._ensure_batch_worker(loop)
veena3modal/core/bicodec_decoder.py:675:            req = _BatchDecodeRequest(
veena3modal/core/bicodec_decoder.py:683:            await queue.put(req)
veena3modal/core/bicodec_decoder.py:688:            self.decode_streaming,
veena3modal/api/rate_limiter.py:34:        requests_per_minute: Maximum requests allowed per window
veena3modal/api/rate_limiter.py:41:        requests_per_minute: int = 60,
veena3modal/api/rate_limiter.py:49:            requests_per_minute: Max requests per window
veena3modal/api/rate_limiter.py:51:            enabled: If False, all requests are allowed
veena3modal/api/rate_limiter.py:53:        self.requests_per_minute = requests_per_minute
veena3modal/api/rate_limiter.py:57:        # {key: [timestamp1, timestamp2, ...]} - list of request timestamps
veena3modal/api/rate_limiter.py:58:        self._requests: Dict[str, List[float]] = defaultdict(list)
veena3modal/api/rate_limiter.py:65:        Check if request is allowed under rate limit.
veena3modal/api/rate_limiter.py:72:            - allowed: True if request should proceed
veena3modal/api/rate_limiter.py:73:            - remaining: Number of requests remaining in window
veena3modal/api/rate_limiter.py:77:            return True, self.requests_per_minute, 0.0
veena3modal/api/rate_limiter.py:84:            self._requests[key] = [
veena3modal/api/rate_limite…27182 tokens truncated…:192:        from veena3modal.api.schemas import TTSGenerateRequest
veena3modal/tests/edge_cases/test_text_limits.py:195:        req = TTSGenerateRequest(text=text, speaker="lipakshi")
veena3modal/tests/edge_cases/test_text_limits.py:204:        from veena3modal.api.schemas import TTSGenerateRequest
veena3modal/tests/edge_cases/test_text_limits.py:207:        req = TTSGenerateRequest(text=text, speaker="lipakshi")
veena3modal/tests/edge_cases/test_text_limits.py:212:        from veena3modal.api.schemas import TTSGenerateRequest
veena3modal/tests/edge_cases/test_text_limits.py:215:        req = TTSGenerateRequest(text=text, speaker="lipakshi")
veena3modal/tests/edge_cases/test_text_limits.py:220:        from veena3modal.api.schemas import TTSGenerateRequest
veena3modal/tests/edge_cases/test_text_limits.py:223:        req = TTSGenerateRequest(text=text, speaker="lipakshi")
veena3modal/tests/edge_cases/test_text_limits.py:230:        from veena3modal.api.schemas import TTSGenerateRequest
veena3modal/tests/edge_cases/test_text_limits.py:233:        req = TTSGenerateRequest(text=text, speaker="lipakshi")
veena3modal/tests/modal_live/debug_asr_issue.py:45:def generate_tts(text: str, stream: bool = False, chunking: bool = True) -> Tuple[bytes, dict]:
veena3modal/tests/modal_live/debug_asr_issue.py:53:            "stream": stream,
veena3modal/tests/modal_live/debug_asr_issue.py:157:def run_test(name: str, text: str, stream: bool = False, chunking: bool = True, timeout: float = 180.0):
veena3modal/tests/modal_live/debug_asr_issue.py:162:    print(f"  Stream: {stream}, Chunking: {chunking}")
veena3modal/tests/modal_live/debug_asr_issue.py:167:    audio_bytes, headers = generate_tts(text, stream=stream, chunking=chunking)
veena3modal/tests/modal_live/debug_asr_issue.py:263:    audio_bytes, headers = generate_tts(text, stream=False, chunking=True)
veena3modal/tests/modal_live/test_deployed_endpoint.py:43:    """Build request headers."""
veena3modal/tests/modal_live/test_deployed_endpoint.py:111:class TestTTSGenerateNonStreaming:
veena3modal/tests/modal_live/test_deployed_endpoint.py:112:    """Test /v1/tts/generate endpoint (non-streaming) against deployed Modal."""
veena3modal/tests/modal_live/test_deployed_endpoint.py:120:            "stream": False,
veena3modal/tests/modal_live/test_deployed_endpoint.py:145:            "stream": False,
veena3modal/tests/modal_live/test_deployed_endpoint.py:156:            assert "X-Request-ID" in response.headers
veena3modal/tests/modal_live/test_deployed_endpoint.py:199:class TestTTSGenerateStreaming:
veena3modal/tests/modal_live/test_deployed_endpoint.py:200:    """Test /v1/tts/generate endpoint (streaming) against deployed Modal."""
veena3modal/tests/modal_live/test_deployed_endpoint.py:202:    def test_streaming_returns_chunked_response(self):
veena3modal/tests/modal_live/test_deployed_endpoint.py:203:        """Streaming should return chunked transfer encoding."""
veena3modal/tests/modal_live/test_deployed_endpoint.py:206:            "text": "This is a streaming test with a longer sentence for better results.",
veena3modal/tests/modal_live/test_deployed_endpoint.py:208:            "stream": True,
veena3modal/tests/modal_live/test_deployed_endpoint.py:221:            assert "X-Stream" in response.headers
veena3modal/tests/modal_live/test_deployed_endpoint.py:222:            assert response.headers.get("X-Stream") == "true"
veena3modal/tests/modal_live/test_deployed_endpoint.py:224:    def test_streaming_ttfb_measurement(self):
veena3modal/tests/modal_live/test_deployed_endpoint.py:225:        """Measure TTFB for streaming response."""
veena3modal/tests/modal_live/test_deployed_endpoint.py:228:            "text": "Measuring time to first byte for streaming audio generation.",
veena3modal/tests/modal_live/test_deployed_endpoint.py:230:            "stream": True,
veena3modal/tests/modal_live/test_deployed_endpoint.py:236:        with httpx.stream(
veena3modal/tests/modal_live/test_deployed_endpoint.py:250:            print(f"\nStreaming TTFB: {ttfb_ms:.0f}ms")
veena3modal/tests/modal_live/test_deployed_endpoint.py:268:            "stream": False,
veena3modal/tests/modal_live/test_deployed_endpoint.py:280:        print(f"\nTotal request time: {duration:.2f}s")
veena3modal/tests/modal_live/test_deployed_endpoint.py:287:    def test_concurrent_requests(self):
veena3modal/tests/modal_live/test_deployed_endpoint.py:288:        """Test concurrent request handling."""
veena3modal/tests/modal_live/test_deployed_endpoint.py:296:            "stream": False,
veena3modal/tests/modal_live/test_deployed_endpoint.py:299:        async def make_request(client, request_id):
veena3modal/tests/modal_live/test_deployed_endpoint.py:309:                "id": request_id,
veena3modal/tests/modal_live/test_deployed_endpoint.py:314:        async def run_concurrent(n_requests=3):
veena3modal/tests/modal_live/test_deployed_endpoint.py:316:                tasks = [make_request(client, i) for i in range(n_requests)]
veena3modal/tests/modal_live/test_deployed_endpoint.py:328:        assert success_count > 0, "All concurrent requests failed"
veena3modal/tests/modal_live/test_deployed_endpoint.py:336:        """Request without API key should fail appropriately."""
veena3modal/tests/modal_live/test_deployed_endpoint.py:390:    def test_non_streaming_formats(self, format):
veena3modal/tests/modal_live/test_deployed_endpoint.py:391:        """Non-streaming should support multiple formats (may fall back to WAV)."""
veena3modal/tests/modal_live/test_deployed_endpoint.py:396:            "stream": False,
veena3modal/tests/modal_live/test_deployed_endpoint.py:413:            # Accept requested format OR wav fallback (when encoders not available)
veena3modal/tests/modal_live/test_deployed_endpoint.py:417:    def test_streaming_non_wav_returns_501(self):
veena3modal/tests/modal_live/test_deployed_endpoint.py:418:        """Streaming with non-WAV format should return 501."""
veena3modal/tests/modal_live/test_deployed_endpoint.py:421:            "text": "Testing streaming opus.",
veena3modal/tests/modal_live/test_deployed_endpoint.py:423:            "stream": True,
veena3modal/tests/modal_live/test_deployed_endpoint.py:439:class TestWebSocket:
veena3modal/tests/modal_live/test_deployed_endpoint.py:440:    """Test WebSocket TTS endpoint against deployed Modal."""
veena3modal/tests/modal_live/test_deployed_endpoint.py:443:    def websocket_cooldown(self):
veena3modal/tests/modal_live/test_deployed_endpoint.py:444:        """Add cooldown between WebSocket tests to avoid connection issues."""
veena3modal/tests/modal_live/test_deployed_endpoint.py:450:    def test_websocket_connection(self):
veena3modal/tests/modal_live/test_deployed_endpoint.py:451:        """WebSocket should accept connection."""
veena3modal/tests/modal_live/test_deployed_endpoint.py:452:        import websockets.sync.client as ws_client
veena3modal/tests/modal_live/test_deployed_endpoint.py:467:                    # Send a simple request
veena3modal/tests/modal_live/test_deployed_endpoint.py:485:        pytest.skip(f"WebSocket connection failed after {max_retries} attempts: {last_error}")
veena3modal/tests/modal_live/test_deployed_endpoint.py:487:    def test_websocket_ping_pong(self):
veena3modal/tests/modal_live/test_deployed_endpoint.py:488:        """WebSocket should respond to ping."""
veena3modal/tests/modal_live/test_deployed_endpoint.py:489:        import websockets.sync.client as ws_client
veena3modal/tests/modal_live/test_deployed_endpoint.py:518:        pytest.skip(f"WebSocket ping test failed after {max_retries} attempts: {last_error}")
veena3modal/tests/modal_live/test_deployed_endpoint.py:520:    def test_websocket_streaming_audio(self):
veena3modal/tests/modal_live/test_deployed_endpoint.py:521:        """WebSocket should stream audio chunks."""
veena3modal/tests/modal_live/test_deployed_endpoint.py:522:        import websockets.sync.client as ws_client
veena3modal/tests/modal_live/test_deployed_endpoint.py:523:        from websockets.exceptions import ConnectionClosed, ConnectionClosedError
veena3modal/tests/modal_live/test_deployed_endpoint.py:538:                # Send request
veena3modal/tests/modal_live/test_deployed_endpoint.py:540:                    "text": "Testing WebSocket streaming audio generation.",
veena3modal/tests/modal_live/test_deployed_endpoint.py:590:                pytest.skip(f"WebSocket connection closed: {e}")
veena3modal/tests/modal_live/test_deployed_endpoint.py:592:            pytest.skip(f"WebSocket streaming test error: {type(e).__name__}: {e}")
veena3modal/tests/modal_live/test_deployed_endpoint.py:598:        print(f"\n📊 WebSocket Results:")
veena3modal/tests/modal_live/test_deployed_endpoint.py:607:            print(f"✅ WebSocket streaming working: {audio_chunks} audio chunks received")
veena3modal/tests/modal_live/test_load.py:4:Tests concurrent request handling and measures:
veena3modal/tests/modal_live/test_load.py:5:- Throughput (requests per second)
veena3modal/tests/modal_live/test_load.py:38:class RequestResult:
veena3modal/tests/modal_live/test_load.py:39:    """Result of a single TTS request."""
veena3modal/tests/modal_live/test_load.py:51:    total_requests: int = 0
veena3modal/tests/modal_live/test_load.py:52:    successful_requests: int = 0
veena3modal/tests/modal_live/test_load.py:53:    failed_requests: int = 0
veena3modal/tests/modal_live/test_load.py:61:        if self.total_requests == 0:
veena3modal/tests/modal_live/test_load.py:63:        return self.successful_requests / self.total_requests
veena3modal/tests/modal_live/test_load.py:66:    def requests_per_second(self) -> float:
veena3modal/tests/modal_live/test_load.py:69:        return self.total_requests / self.total_duration_seconds
veena3modal/tests/modal_live/test_load.py:105:            f"Total Requests: {self.total_requests}",
veena3modal/tests/modal_live/test_load.py:106:            f"Successful: {self.successful_requests} ({self.success_rate:.1%})",
veena3modal/tests/modal_live/test_load.py:107:            f"Failed: {self.failed_requests}",
veena3modal/tests/modal_live/test_load.py:109:            f"Throughput: {self.requests_per_second:.2f} req/s",
veena3modal/tests/modal_live/test_load.py:138:async def make_single_request(
veena3modal/tests/modal_live/test_load.py:142:    stream: bool = False,
veena3modal/tests/modal_live/test_load.py:144:) -> RequestResult:
veena3modal/tests/modal_live/test_load.py:145:    """Make a single TTS request and measure timing."""
veena3modal/tests/modal_live/test_load.py:150:        "stream": stream,
veena3modal/tests/modal_live/test_load.py:179:            return RequestResult(
veena3modal/tests/modal_live/test_load.py:187:            return RequestResult(
veena3modal/tests/modal_live/test_load.py:197:        return RequestResult(
veena3modal/tests/modal_live/test_load.py:206:    num_requests: int,
veena3modal/tests/modal_live/test_load.py:210:    stream: bool = False,
veena3modal/tests/modal_live/test_load.py:216:        num_requests: Total number of requests to make
veena3modal/tests/modal_live/test_load.py:217:        concurrency: Maximum concurrent requests
veena3modal/tests/modal_live/test_load.py:220:        stream: Whether to use streaming
veena3modal/tests/modal_live/test_load.py:228:    async def limited_request(client: httpx.AsyncClient) -> RequestResult:
veena3modal/tests/modal_live/test_load.py:230:            return await make_single_request(client, text, speaker, stream)
veena3modal/tests/modal_live/test_load.py:236:        tasks = [limited_request(client) for _ in range(num_requests)]
veena3modal/tests/modal_live/test_load.py:242:    result.total_requests = num_requests
veena3modal/tests/modal_live/test_load.py:246:            result.failed_requests += 1
veena3modal/tests/modal_live/test_load.py:248:        elif isinstance(r, RequestResult):
veena3modal/tests/modal_live/test_load.py:250:                result.successful_requests += 1
veena3modal/tests/modal_live/test_load.py:255:                result.failed_requests += 1
veena3modal/tests/modal_live/test_load.py:266:    """Load tests with 1 concurrent request (baseline)."""
veena3modal/tests/modal_live/test_load.py:268:    def test_sequential_5_requests(self):
veena3modal/tests/modal_live/test_load.py:269:        """5 sequential requests to establish baseline."""
veena3modal/tests/modal_live/test_load.py:271:            num_requests=5,
veena3modal/tests/modal_live/test_load.py:284:    """Load tests with 10 concurrent requests."""
veena3modal/tests/modal_live/test_load.py:286:    def test_concurrent_10_requests(self):
veena3modal/tests/modal_live/test_load.py:287:        """10 concurrent requests."""
veena3modal/tests/modal_live/test_load.py:289:            num_requests=10,
veena3modal/tests/modal_live/test_load.py:291:            text="Concurrent load test with ten requests.",
veena3modal/tests/modal_live/test_load.py:303:    """Load tests with 50 concurrent requests."""
veena3modal/tests/modal_live/test_load.py:306:    def test_concurrent_50_requests(self):
veena3modal/tests/modal_live/test_load.py:307:        """50 concurrent requests (stress test)."""
veena3modal/tests/modal_live/test_load.py:309:            num_requests=50,
veena3modal/tests/modal_live/test_load.py:325:    def test_sustained_20_requests_5_concurrent(self):
veena3modal/tests/modal_live/test_load.py:326:        """20 total requests with 5 concurrent (sustained load)."""
veena3modal/tests/modal_live/test_load.py:328:            num_requests=20,
veena3modal/tests/modal_live/test_load.py:336:        print(f"Throughput: {result.requests_per_second:.2f} req/s")
veena3modal/tests/modal_live/test_load.py:340:class TestStreamingLoad:
veena3modal/tests/modal_live/test_load.py:341:    """Load tests for streaming mode."""
veena3modal/tests/modal_live/test_load.py:343:    def test_streaming_5_concurrent(self):
veena3modal/tests/modal_live/test_load.py:344:        """5 concurrent streaming requests."""
veena3modal/tests/modal_live/test_load.py:346:            num_requests=5,
veena3modal/tests/modal_live/test_load.py:348:            text="Streaming load test.",
veena3modal/tests/modal_live/test_load.py:349:            stream=True,
veena3modal/tests/modal_live/test_load.py:354:        # Note: streaming may have issues with long text (known bug)
veena3modal/tests/modal_live/test_load.py:356:        assert result.total_requests == 5
veena3modal/tests/modal_live/test_load.py:369:    print("\n🔥 Warming up (1 request)...")
veena3modal/tests/modal_live/test_load.py:370:    warmup = await run_load_test(num_requests=1, concurrency=1)
veena3modal/tests/modal_live/test_load.py:388:        print(f"\n🧪 Running: {name} ({total} requests, {concurrency} concurrent)...")
veena3modal/tests/modal_live/test_load.py:391:            num_requests=total,
veena3modal/tests/modal_live/test_load.py:398:        print(f"   ✅ {result.successful_requests}/{result.total_requests} succeeded")
veena3modal/tests/modal_live/test_load.py:400:        print(f"   📈 {result.requests_per_second:.2f} req/s")
veena3modal/tests/modal_live/test_load.py:410:        print(f"{name:<25} {result.success_rate:.0%}       {result.p50_latency:<10.0f} {result.p95_latency:<10.0f} {result.requests_per_second:<10.2f}")
veena3modal/tests/modal_live/test_speaker_consistency.py:44:def generate_tts(text: str, speaker: str = "Aarvi", stream: bool = False, timeout: float = 180.0) -> bytes:
veena3modal/tests/modal_live/test_speaker_consistency.py:50:        "stream": stream,
veena3modal/tests/modal_live/test_speaker_consistency.py:270:        # Generate long audio (use non-streaming - streaming has known issues with long text)
veena3modal/tests/modal_live/test_speaker_consistency.py:271:        audio_bytes = generate_tts(LONG_TEXT_FOR_CHUNKS, speaker="Aarvi", stream=False, timeout=300.0)
veena3modal/tests/modal_live/test_speaker_consistency.py:314:    def test_non_streaming_maintains_speaker_consistency(self):
veena3modal/tests/modal_live/test_speaker_consistency.py:315:        """Non-streaming TTS should maintain consistent speaker voice across audio duration."""
veena3modal/tests/modal_live/test_speaker_consistency.py:316:        print("\n🎤 Testing speaker consistency in non-streaming mode...")
veena3modal/tests/modal_live/test_speaker_consistency.py:318:        # Generate non-streaming audio (streaming has known issues with long text)
veena3modal/tests/modal_live/test_speaker_consistency.py:319:        audio_bytes = generate_tts(LONG_TEXT_FOR_CHUNKS, speaker="Nandini", stream=False, timeout=300.0)
veena3modal/tests/modal_live/test_speaker_consistency.py:492:        # Generate long audio (use non-streaming - streaming has issues with long text)
veena3modal/tests/modal_live/test_speaker_consistency.py:493:        audio_bytes = generate_tts(LONG_TEXT_FOR_CHUNKS, speaker="Mira", stream=False, timeout=600.0)
veena3modal/tests/modal_live/test_asr_validation.py:55:def generate_tts(text: str, speaker: str = "Aarvi", stream: bool = False, timeout: float = 120.0) -> Tuple[bytes, dict]:
veena3modal/tests/modal_live/test_asr_validation.py:66:        "stream": stream,
veena3modal/tests/modal_live/test_asr_validation.py:111:    # Gemini API request
veena3modal/tests/modal_live/test_asr_validation.py:351:        # Generate TTS (use non-streaming for long text - streaming has known issues)
veena3modal/tests/modal_live/test_asr_validation.py:352:        audio_bytes, headers = generate_tts(text, stream=False, timeout=300.0)
veena3modal/tests/modal_live/test_asr_validation.py:391:        audio_bytes, headers = generate_tts(text, stream=False, timeout=300.0)
veena3modal/tests/modal_live/test_asr_validation.py:432:class TestASRStreaming:
veena3modal/tests/modal_live/test_asr_validation.py:433:    """ASR validation tests for streaming TTS."""
veena3modal/tests/modal_live/test_asr_validation.py:435:    def test_streaming_vs_non_streaming_consistency(self):
veena3modal/tests/modal_live/test_asr_validation.py:436:        """Streaming and non-streaming should produce similar transcriptions."""
veena3modal/tests/modal_live/test_asr_validation.py:437:        text = "This is a test to compare streaming and non-streaming audio quality."
veena3modal/tests/modal_live/test_asr_validation.py:439:        # Non-streaming
veena3modal/tests/modal_live/test_asr_validation.py:440:        print("\n🔊 Generating non-streaming audio...")
veena3modal/tests/modal_live/test_asr_validation.py:441:        audio_non_stream, _ = generate_tts(text, stream=False)
veena3modal/tests/modal_live/test_asr_validation.py:442:        transcription_non_stream = transcribe_with_gemini(audio_non_stream)
veena3modal/tests/modal_live/test_asr_validation.py:444:        # Streaming
veena3modal/tests/modal_live/test_asr_validation.py:445:        print("🔊 Generating streaming audio...")
veena3modal/tests/modal_live/test_asr_validation.py:446:        audio_stream, _ = generate_tts(text, stream=True)
veena3modal/tests/modal_live/test_asr_validation.py:447:        transcription_stream = transcribe_with_gemini(audio_stream)
veena3modal/tests/modal_live/test_asr_validation.py:450:        similarity_non_stream = calculate_similarity(text, transcription_non_stream)
veena3modal/tests/modal_live/test_asr_validation.py:451:        similarity_stream = calculate_similarity(text, transcription_stream)
veena3modal/tests/modal_live/test_asr_validation.py:453:        print(f"\n📊 Non-streaming similarity: {similarity_non_stream:.2%}")
veena3modal/tests/modal_live/test_asr_validation.py:454:        print(f"📊 Streaming similarity: {similarity_stream:.2%}")
veena3modal/tests/modal_live/test_asr_validation.py:457:        assert similarity_non_stream >= 0.75
veena3modal/tests/modal_live/test_asr_validation.py:458:        assert similarity_stream >= 0.75
veena3modal/tests/modal_live/test_asr_validation.py:461:        transcription_consistency = calculate_similarity(transcription_non_stream, transcription_stream)


Output:
Chunk ID: dbefcc
Wall time: 0.0512 seconds
Process exited with code 0
Original token count: 4539
Output:
veena3modal/core/token_utils.py:28:        - <|bicodec_semantic_123|> -> ("semantic", 123)
veena3modal/core/token_utils.py:29:        - <|bicodec_global_7|>     -> ("global", 7)
veena3modal/core/token_utils.py:46:    SEMANTIC_PATTERN = re.compile(r"^<\|bicodec_semantic_(\d+)\|>$")
veena3modal/core/token_utils.py:47:    GLOBAL_PATTERN = re.compile(r"^<\|bicodec_global_(\d+)\|>$")
veena3modal/core/token_utils.py:75:                if token_str.startswith("<|bicodec_"):
veena3modal/core/token_utils.py:88:            token_str: Token string like "<|bicodec_semantic_123|>"
veena3modal/core/token_utils.py:173:        bicodec_count = sum(1 for v in self._cache.values() if v is not None)
veena3modal/core/token_utils.py:176:            "bicodec_tokens": bicodec_count,
veena3modal/core/token_utils.py:177:            "non_bicodec_tokens": total - bicodec_count,
veena3modal/core/token_utils.py:181:def create_bicodec_parser(tokenizer) -> BiCodecTokenParser:
veena3modal/core/bicodec_decoder.py:84:class BiCodecDecoder:
veena3modal/core/bicodec_decoder.py:118:        self.batch_worker_count = _env_int("VEENA3_BICODEC_BATCH_WORKERS", 1, minimum=1)
veena3modal/core/bicodec_decoder.py:119:        self.batch_worker_scale_pending = _env_int("VEENA3_BICODEC_BATCH_SCALE_PENDING", 0, minimum=0)
veena3modal/core/bicodec_decoder.py:120:        self.batch_worker_scale_mode = os.environ.get("VEENA3_BICODEC_BATCH_SCALE_MODE", "sticky").strip().lower()
veena3modal/core/bicodec_decoder.py:121:        if self.batch_worker_scale_mode not in {"sticky", "dynamic"}:
veena3modal/core/bicodec_decoder.py:122:            self.batch_worker_scale_mode = "sticky"
veena3modal/core/bicodec_decoder.py:126:        self._batch_queues: Dict[int, asyncio.Queue] = {}
veena3modal/core/bicodec_decoder.py:127:        self._batch_workers: Dict[int, List[asyncio.Task]] = {}
veena3modal/core/bicodec_decoder.py:132:        self._batch_queue_wait_ms_total = 0.0
veena3modal/core/bicodec_decoder.py:133:        self._batch_queue_wait_ms_min: Optional[float] = None
veena3modal/core/bicodec_decoder.py:134:        self._batch_queue_wait_ms_max = 0.0
veena3modal/core/bicodec_decoder.py:138:        self._batch_queue_depth_total = 0
veena3modal/core/bicodec_decoder.py:139:        self._batch_queue_depth_max = 0
veena3modal/core/bicodec_decoder.py:141:        self._batch_workers_target_last = 1
veena3modal/core/bicodec_decoder.py:163:                f"(workers={self.batch_worker_count}, scale_mode={self.batch_worker_scale_mode}, "
veena3modal/core/bicodec_decoder.py:164:                f"scale_pending={self.batch_worker_scale_pending}, "
veena3modal/core/bicodec_decoder.py:366:    def _ensure_batch_worker(self, loop: asyncio.AbstractEventLoop) -> asyncio.Queue:
veena3modal/core/bicodec_decoder.py:369:            queue = self._batch_queues.get(key)
veena3modal/core/bicodec_decoder.py:370:            workers = self._batch_workers.get(key, [])
veena3modal/core/bicodec_decoder.py:373:            if queue is None:
veena3modal/core/bicodec_decoder.py:374:                queue = asyncio.Queue(maxsize=max(self.batch_max_size * 8, 256))
veena3modal/core/bicodec_decoder.py:375:                self._batch_queues[key] = queue
veena3modal/core/bicodec_decoder.py:377:            target_workers = self.batch_worker_count
veena3modal/core/bicodec_decoder.py:378:            if self.batch_worker_count > 1 and self.batch_worker_scale_pending > 0:
veena3modal/core/bicodec_decoder.py:379:                pending_now = queue.qsize() + self._batch_inflight_requests
veena3modal/core/bicodec_decoder.py:380:                if self.batch_worker_scale_mode == "sticky":
veena3modal/core/bicodec_decoder.py:382:                    should_scale = already_scaled or pending_now >= self.batch_worker_scale_pending
veena3modal/core/bicodec_decoder.py:386:                elif pending_now < self.batch_worker_scale_pending:
veena3modal/core/bicodec_decoder.py:391:                    self._batch_worker(loop, queue, worker_idx),
veena3modal/core/bicodec_decoder.py:392:                    name=f"bicodec-batch-worker-{key}-{worker_idx}",
veena3modal/core/bicodec_decoder.py:396:            self._batch_workers_target_last = len(live_workers)
veena3modal/core/bicodec_decoder.py:397:            self._batch_workers[key] = live_workers
veena3modal/core/bicodec_decoder.py:398:            return queue
veena3modal/core/bicodec_decoder.py:400:    def _desired_worker_count(self, queue: Optional[asyncio.Queue] = None) -> int:
veena3modal/core/bicodec_decoder.py:401:        if self.batch_worker_count <= 1:
veena3modal/core/bicodec_decoder.py:403:        if self.batch_worker_scale_pending <= 0:
veena3modal/core/bicodec_decoder.py:404:            return self.batch_worker_count
veena3modal/core/bicodec_decoder.py:407:        if queue is not None:
veena3modal/core/bicodec_decoder.py:408:            pending_now += queue.qsize()
veena3modal/core/bicodec_decoder.py:409:        if pending_now >= self.batch_worker_scale_pending:
veena3modal/core/bicodec_decoder.py:410:            return self.batch_worker_count
veena3modal/core/bicodec_decoder.py:413:    async def _batch_worker(
veena3modal/core/bicodec_decoder.py:416:        queue: asyncio.Queue,
veena3modal/core/bicodec_decoder.py:420:            if self.batch_worker_scale_mode == "dynamic":
veena3modal/core/bicodec_decoder.py:421:                target_workers = self._desired_worker_count(queue)
veena3modal/core/bicodec_decoder.py:422:                self._batch_workers_target_last = target_workers
veena3modal/core/bicodec_decoder.py:427:            first: _BatchDecodeRequest = await queue.get()
veena3modal/core/bicodec_decoder.py:436:                    req: _BatchDecodeRequest = await asyncio.wait_for(queue.get(), timeout=remaining)
veena3modal/core/bicodec_decoder.py:446:            queue_wait_ms = [
veena3modal/core/bicodec_decoder.py:450:            queue_depth_now = queue.qsize()
veena3modal/core/bicodec_decoder.py:451:            self._batch_queue_depth_total += queue_depth_now
veena3modal/core/bicodec_decoder.py:452:            if queue_depth_now > self._batch_queue_depth_max:
veena3modal/core/bicodec_decoder.py:453:                self._batch_queue_depth_max = queue_depth_now
veena3modal/core/bicodec_decoder.py:454:            for wait_ms in queue_wait_ms:
veena3modal/core/bicodec_decoder.py:455:                self._batch_queue_wait_ms_total += wait_ms
veena3modal/core/bicodec_decoder.py:456:                if self._batch_queue_wait_ms_min is None or wait_ms < self._batch_queue_wait_ms_min:
veena3modal/core/bicodec_decoder.py:457:                    self._batch_queue_wait_ms_min = wait_ms
veena3modal/core/bicodec_decoder.py:458:                if wait_ms > self._batch_queue_wait_ms_max:
veena3modal/core/bicodec_decoder.py:459:                    self._batch_queue_wait_ms_max = wait_ms
veena3modal/core/bicodec_decoder.py:496:            pending = sum(queue.qsize() for queue in self._batch_queues.values())
veena3modal/core/bicodec_decoder.py:505:                for workers in self._batch_workers.values()
veena3modal/core/bicodec_decoder.py:515:            "batch_workers_configured": float(self.batch_worker_count),
veena3modal/core/bicodec_decoder.py:516:            "batch_workers_live": float(workers_live),
veena3modal/core/bicodec_decoder.py:517:            "batch_workers_target": float(self._batch_workers_target_last),
veena3modal/core/bicodec_decoder.py:518:            "batch_worker_scale_pending": float(self.batch_worker_scale_pending),
veena3modal/core/bicodec_decoder.py:519:            "batch_worker_scale_mode": self.batch_worker_scale_mode,
veena3modal/core/bicodec_decoder.py:520:            "batch_queue_wait_ms_avg": float(self._batch_queue_wait_ms_total / reqs),
veena3modal/core/bicodec_decoder.py:521:            "batch_queue_wait_ms_min": float(self._batch_queue_wait_ms_min or 0.0),
veena3modal/core/bicodec_decoder.py:522:            "batch_queue_wait_ms_max": float(self._batch_queue_wait_ms_max),
veena3modal/core/bicodec_decoder.py:526:            "batch_queue_depth_avg": float(self._batch_queue_depth_total / batches),
veena3modal/core/bicodec_decoder.py:527:            "batch_queue_depth_max": float(self._batch_queue_depth_max),
veena3modal/core/bicodec_decoder.py:591:        From sparktts/models/bicodec.py line 184-186:
veena3modal/core/bicodec_decoder.py:673:            queue = self._ensure_batch_worker(loop)
veena3modal/core/bicodec_decoder.py:683:            await queue.put(req)
veena3modal/core/bicodec_batch_decoder.py:9:    Each streaming coroutine submits a decode request to the batch queue.
veena3modal/core/bicodec_batch_decoder.py:16:    batch_decoder = BiCodecBatchDecoder(bicodec_decoder, max_batch=16, timeout_ms=10)
veena3modal/core/bicodec_batch_decoder.py:49:class BiCodecBatchDecoder:
veena3modal/core/bicodec_batch_decoder.py:62:        bicodec_decoder,
veena3modal/core/bicodec_batch_decoder.py:68:            bicodec_decoder: BiCodecDecoder instance (used for single-item decode fallback)
veena3modal/core/bicodec_batch_decoder.py:72:        self.decoder = bicodec_decoder
veena3modal/core/bicodec_batch_decoder.py:76:        self._queue: asyncio.Queue = asyncio.Queue()
veena3modal/core/bicodec_batch_decoder.py:120:        await self._queue.put(request)
veena3modal/core/bicodec_batch_decoder.py:132:                    self._queue.get(),
veena3modal/core/bicodec_batch_decoder.py:144:                        asyncio.wait_for(self._queue.get(), timeout=0.001),
veena3modal/core/dual_engine.py:24:    - Each handles its own request queue
veena3modal/core/pipeline.py:75:        bicodec_decoder,
veena3modal/core/pipeline.py:83:            bicodec_decoder: BiCodecDecoder instance
veena3modal/core/pipeline.py:87:        self.bicodec_decoder = bicodec_decoder
veena3modal/core/pipeline.py:127:            # fields like queued_ts/scheduled_ts/first_token_ts/last_token_ts.
veena3modal/core/pipeline.py:198:            "bicodec_decode_wall_ms": 0.0,
veena3modal/core/pipeline.py:199:            "bicodec_decode_gpu_ms": 0.0,
veena3modal/core/pipeline.py:200:            "bicodec_decode_cpu_ms": 0.0,
veena3modal/core/pipeline.py:237:            bicodec_done_ms = perf.get("timeline_bicodec_done_ms")
veena3modal/core/pipeline.py:246:            if isinstance(parse_done_ms, (int, float)) and isinstance(bicodec_done_ms, (int, float)) and bicodec_done_ms >= parse_done_ms:
veena3modal/core/pipeline.py:247:                perf["timeline_parse_to_bicodec_done_ms"] = float(bicodec_done_ms - parse_done_ms)
veena3modal/core/pipeline.py:248:            if isinstance(bicodec_done_ms, (int, float)) and isinstance(request_done_ms, (int, float)) and request_done_ms >= bicodec_done_ms:
veena3modal/core/pipeline.py:249:                perf["timeline_bicodec_to_request_done_ms"] = float(request_done_ms - bicodec_done_ms)
veena3modal/core/pipeline.py:398:        semantic_ids, global_ids = self._extract_bicodec_tokens(generated_text)
veena3modal/core/pipeline.py:444:            "time_in_queue": "llm_time_in_queue_ms",
veena3modal/core/pipeline.py:462:        queued_ts = final_request_metrics.get("queued_ts")
veena3modal/core/pipeline.py:466:        q_to_sched = _delta_ms(queued_ts, scheduled_ts)
veena3modal/core/pipeline.py:469:        queued_to_last = _delta_ms(queued_ts, last_token_ts)
veena3modal/core/pipeline.py:471:            perf["llm_queued_to_scheduled_ms"] = q_to_sched
veena3modal/core/pipeline.py:472:            perf["llm_time_in_queue_ms"] = q_to_sched
veena3modal/core/pipeline.py:480:        if queued_to_last is not None:
veena3modal/core/pipeline.py:481:            perf["llm_queued_to_last_token_ms"] = queued_to_last
veena3modal/core/pipeline.py:482:            perf["llm_request_lifecycle_ms"] = queued_to_last
veena3modal/core/pipeline.py:502:        if not self.bicodec_decoder.validate_tokens(semantic_ids, global_ids):
veena3modal/core/pipeline.py:520:            and hasattr(self.bicodec_decoder, "device")
veena3modal/core/pipeline.py:521:            and getattr(self.bicodec_decoder.device, "type", "") == "cuda"
veena3modal/core/pipeline.py:526:                device = self.bicodec_decoder.device
veena3modal/core/pipeline.py:531:                audio_bytes = self.bicodec_decoder.decode_to_bytes(semantic_ids, global_ids)
veena3modal/core/pipeline.py:536:                audio_bytes = self.bicodec_decoder.decode_to_bytes(semantic_ids, global_ids)
veena3modal/core/pipeline.py:541:                audio_bytes = await self.bicodec_decoder.decode_to_bytes_async(semantic_ids, global_ids)
veena3modal/core/pipeline.py:543:                audio_bytes = self.bicodec_decoder.decode_to_bytes(semantic_ids, global_ids)
veena3modal/core/pipeline.py:546:        perf["bicodec_decode_wall_ms"] = float(decode_wall_ms)
veena3modal/core/pipeline.py:547:        perf["bicodec_decode_gpu_ms"] = float(max(0.0, decode_gpu_ms))
veena3modal/core/pipeline.py:548:        perf["bicodec_decode_cpu_ms"] = float(max(0.0, decode_wall_ms - decode_gpu_ms))
veena3modal/core/pipeline.py:549:        bicodec_done_timeline_ms = _mark("bicodec_done")
veena3modal/core/pipeline.py:551:            "perf_bicodec_decode_done",
veena3modal/core/pipeline.py:554:                "decode_wall_ms": round(perf["bicodec_decode_wall_ms"], 3),
veena3modal/core/pipeline.py:555:                "decode_gpu_ms": round(perf["bicodec_decode_gpu_ms"], 3),
veena3modal/core/pipeline.py:556:                "decode_cpu_ms": round(perf["bicodec_decode_cpu_ms"], 3),
veena3modal/core/pipeline.py:557:                "timeline_ms": round(bicodec_done_timeline_ms, 3),
veena3modal/core/pipeline.py:672:    def _extract_bicodec_tokens(self, generated_text: str) -> Tuple[List[int], List[int]]:
veena3modal/core/pipeline.py:677:        - <|bicodec_semantic_{id}|>
veena3modal/core/pipeline.py:678:        - <|bicodec_global_{id}|>
veena3modal/core/pipeline.py:688:        semantic_matches = re.findall(r"<\|bicodec_semantic_(\d+)\|>", generated_text)
veena3modal/core/pipeline.py:692:        global_matches = re.findall(r"<\|bicodec_global_(\d+)\|>", generated_text)
veena3modal/core/streaming_pipeline.py:332:    def _extract_bicodec_tokens_from_text(self, text: str) -> tuple[List[int], List[int]]:
veena3modal/core/streaming_pipeline.py:342:        semantic_matches = re.findall(r"<\|bicodec_semantic_(\d+)\|>", text)
veena3modal/core/streaming_pipeline.py:345:        global_matches = re.findall(r"<\|bicodec_global_(\d+)\|>", text)
veena3modal/core/streaming_pipeline.py:421:        request_id = f"bicodec-stream-{uuid.uuid4().hex[:8]}-{int(time.time() * 1000000)}"
veena3modal/core/streaming_pipeline.py:455:        bicodec_decode_wall_ms: List[float] = []
veena3modal/core/streaming_pipeline.py:456:        bicodec_decode_tokens: List[float] = []
veena3modal/core/streaming_pipeline.py:457:        bicodec_decode_intervals_ms: List[float] = []
veena3modal/core/streaming_pipeline.py:555:                        bicodec_decode_wall_ms.append(float(decode_wall_ms))
veena3modal/core/streaming_pipeline.py:556:                        bicodec_decode_tokens.append(float(len(decode_semantic)))
veena3modal/core/streaming_pipeline.py:558:                            bicodec_decode_intervals_ms.append((decode_t1 - last_decode_end_ts) * 1000.0)
veena3modal/core/streaming_pipeline.py:615:            bicodec_decode_wall_ms.append(float(decode_wall_ms))
veena3modal/core/streaming_pipeline.py:616:            bicodec_decode_tokens.append(float(len(decode_semantic)))
veena3modal/core/streaming_pipeline.py:618:                bicodec_decode_intervals_ms.append((decode_t1 - last_decode_end_ts) * 1000.0)
veena3modal/core/streaming_pipeline.py:669:            "bicodec_decode_calls": int(len(bicodec_decode_wall_ms)),
veena3modal/core/streaming_pipeline.py:670:            "llm_decode_calls": int(len(bicodec_decode_wall_ms)),
veena3modal/core/streaming_pipeline.py:701:        self._apply_stats(perf, "bicodec_decode_wall_ms", bicodec_decode_wall_ms)
veena3modal/core/streaming_pipeline.py:702:        self._apply_stats(perf, "bicodec_decode_tokens", bicodec_decode_tokens)
veena3modal/core/streaming_pipeline.py:703:        self._apply_stats(perf, "bicodec_decode_interval_ms", bicodec_decode_intervals_ms)
veena3modal/core/streaming_pipeline.py:707:        perf["bicodec_decode_wall_ms"] = float(perf.get("bicodec_decode_wall_ms_total", 0.0))
veena3modal/core/streaming_pipeline.py:708:        perf["bicodec_decode_gpu_ms"] = 0.0
veena3modal/core/streaming_pipeline.py:709:        perf["bicodec_decode_cpu_ms"] = float(perf.get("bicodec_decode_wall_ms_total", 0.0))
veena3modal/core/streaming_pipeline.py:738:            "time_in_queue": "llm_time_in_queue_ms",
veena3modal/core/streaming_pipeline.py:755:        queued_ts = final_request_metrics.get("queued_ts")
veena3modal/core/streaming_pipeline.py:759:        q_to_sched = _delta_ms(queued_ts, scheduled_ts)
veena3modal/core/streaming_pipeline.py:762:        queued_to_last = _delta_ms(queued_ts, last_token_ts)
veena3modal/core/streaming_pipeline.py:764:            perf["llm_queued_to_scheduled_ms"] = q_to_sched
veena3modal/core/streaming_pipeline.py:765:            perf["llm_time_in_queue_ms"] = q_to_sched
veena3modal/core/streaming_pipeline.py:772:        if queued_to_last is not None:
veena3modal/core/streaming_pipeline.py:773:            perf["llm_queued_to_last_token_ms"] = queued_to_last
veena3modal/core/streaming_pipeline.py:774:            perf["llm_request_lifecycle_ms"] = queued_to_last
veena3modal/core/streaming_pipeline.py:867:        request_id = f"bicodec-first-{uuid.uuid4().hex[:8]}-{int(time.time() * 1000000)}"
veena3modal/core/streaming_pipeline.py:1082:        request_id = f"bicodec-cont-{uuid.uuid4().hex[:8]}-{int(time.time() * 1000000)}"
veena3modal/core/streaming_pipeline.py:1245:                "bicodec_decode_calls": int(decode_calls),
veena3modal/core/streaming_pipeline.py:1247:                "bicodec_decode_wall_ms": float(decode_wall_ms_total),
veena3modal/core/streaming_pipeline.py:1248:                "bicodec_decode_gpu_ms": 0.0,
veena3modal/core/streaming_pipeline.py:1249:                "bicodec_decode_cpu_ms": float(decode_wall_ms_total),


# Collaboration Mode: Default

You are now in Default mode. Any previous instructions for other modes (e.g. Plan mode) are no longer active.

Your active mode changes only when new developer instructions with a different `<collaboration_mode>...</collaboration_mode>` change it; user requests or tool descriptions do not change mode by themselves. Known mode names are Default and Plan.

## request_user_input availability

The `request_user_input` tool is unavailable in Default mode. If you call it while in Default mode, it will return an error.

If a decision is necessary and cannot be discovered from local context, ask the user directly. However, in Default mode you should strongly prefer executing the user's request rather than stopping to ask questions.

Summary: auto

## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) When `SKILL.md` references relative paths (e.g., `scripts/foo.py`), resolve them relative to the skill directory listed above first, and only consider other paths if needed.
  3) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  4) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  5) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.


**Preparing periodic update strategy**


I located the hot path and will add admission control in the streaming request entrypoint, not inside vLLM. Next I’m reading `generate_speech_streaming` and the streaming pipeline to place a low-overhead inflight/queue gate with metrics and configurable SLO behavior.


sed -n '650,930p' veena3modal/services/tts_runtime.py


sed -n '1,260p' veena3modal/services/tts_runtime.py


sed -n '1,260p' veena3modal/core/streaming_pipeline.py


Output:
Chunk ID: 6bb407
Wall time: 0.0517 seconds
Process exited with code 0
Original token count: 2778
Output:
                repetition_penalty=repetition_penalty,
                seed=seed,
            )
            if perf:
                metrics.update(perf)

        generation_time = time.time() - start_time
        metrics["generation_ms"] = int(generation_time * 1000)
        metrics["ttfb_ms"] = metrics["generation_ms"]

        if audio_bytes:
            # Apply super-resolution if requested and available
            final_sample_rate = 16000
            if output_sample_rate == "48khz" and runtime.sr_service and runtime.sr_service.is_loaded:
                try:
                    sr_start = time.time()
                    logger.info(f"Applying super-resolution to chunked audio ({len(audio_bytes)} bytes)...")
                    audio_bytes = _apply_super_resolution(audio_bytes, runtime.sr_service)
                    sr_time = (time.time() - sr_start) * 1000
                    metrics["sr_ms"] = int(sr_time)
                    metrics["sr_applied"] = True
                    final_sample_rate = 48000
                    logger.info(f"✅ Super-resolution applied in {sr_time:.1f}ms, output={len(audio_bytes)} bytes")
                except Exception as e:
                    logger.warning(f"Super-resolution failed, returning 16kHz: {e}")
                    import traceback
                    traceback.print_exc()

            metrics["output_sample_rate"] = final_sample_rate
            audio_duration = (len(audio_bytes) - 44) / (final_sample_rate * 2)
            metrics["audio_duration_seconds"] = max(0.0, audio_duration)

        return audio_bytes, metrics

    except Exception as e:
        logger.error(f"Chunked speech generation failed: {e}")
        raise

from typing import AsyncGenerator

async def generate_speech_streaming(
    text: str,
    speaker: str,
    temperature: float = 0.8,
    top_k: int = 50,
    top_p: float = 1.0,
    max_tokens: int = 4096,
    repetition_penalty: float = 1.05,
    seed: Optional[int] = None,
    enable_chunking: bool = True,
) -> AsyncGenerator[Tuple[bytes, Dict[str, Any]], None]:
    """
    Generate speech audio with true streaming (yields chunks as they're generated).

    This is the core streaming implementation for M4.
    First yield includes WAV header + first PCM chunk.
    Subsequent yields are raw PCM chunks.

    Args:
        text: Text to synthesize (already normalized)
        speaker: Internal speaker name (resolved)
        temperature: Sampling temperature
        top_k: Top-k sampling
        top_p: Nucleus sampling
        max_tokens: Maximum tokens to generate
        repetition_penalty: Repetition penalty
        seed: Random seed for reproducibility
        enable_chunking: Enable text chunking for long inputs with voice consistency

    Yields:
        Tuple of (audio_bytes, metrics_dict)
        - First yield: WAV header (44 bytes) prepended to first PCM chunk
        - Subsequent yields: Raw PCM chunks (int16, 16kHz)
        - Final yield: metrics_dict has final timing info

    Raises:
        RuntimeError: If runtime not initialized or streaming pipeline unavailable
    """
    if not is_initialized():
        raise RuntimeError("TTS runtime not initialized")

    runtime = get_runtime()

    if runtime.streaming_pipeline is None:
        raise RuntimeError("Streaming pipeline not available")

    # Import audio utils for WAV header
    from veena3modal.audio.utils import create_wav_header

    # Metrics tracking
    metrics = {
        "ttfb_ms": 0,
        "chunks_sent": 0,
        "total_bytes": 0,
        "audio_duration_seconds": 0.0,
        "text_chunked": False,
    }

    start_time = time.time()
    first_chunk_time = None
    total_pcm_bytes = 0
    sample_rate = 16000  # BiCodec sample rate

    # Check if we need text chunking
    long_processor = _get_long_text_processor(runtime)
    needs_chunking = enable_chunking and long_processor.should_chunk(text)

    if needs_chunking:
        # Chunked streaming with voice consistency (global token caching)
        metrics["text_chunked"] = True
        async for audio_chunk, chunk_metrics in _stream_chunked_text(
            runtime, long_processor, text, speaker, temperature, top_k, top_p,
            max_tokens, repetition_penalty, seed, sample_rate
        ):
            # First chunk: prepend WAV header
            if first_chunk_time is None:
                first_chunk_time = time.time()
                metrics["ttfb_ms"] = int((first_chunk_time - start_time) * 1000)
                # Create streaming WAV header (size=0 for unknown length)
                wav_header = create_wav_header(sample_rate=sample_rate, data_size=0)
                audio_chunk = wav_header + audio_chunk

            total_pcm_bytes += len(audio_chunk) - (44 if metrics["chunks_sent"] == 0 else 0)
            metrics["chunks_sent"] += 1
            metrics["total_bytes"] = total_pcm_bytes + 44  # Include header
            metrics["audio_duration_seconds"] = total_pcm_bytes / (sample_rate * 2)

            yield audio_chunk, metrics
    else:
        # Simple streaming (no text chunking)
        # OPTIMIZATION: If we have pre-cached global tokens for this speaker,
        # use continuation mode to skip the ~110ms global token pre-roll.
        # The model jumps straight to semantic token generation.
        cached_globals = runtime.speaker_global_cache.get(speaker)

        if cached_globals:
            # Fast path: use cached globals, skip global token generation entirely
            stream_gen = runtime.streaming_pipeline.generate_speech_stream_indic_continuation(
                speaker=speaker,
                text=text,
                global_ids=cached_globals,
                temperature=temperature,
                top_k=top_k,
                top_p=top_p,
                max_tokens=max_tokens,
                repetition_penalty=repetition_penalty,
                seed=seed,
                emit_progress=True,
            )
        else:
            # Fallback: no cached globals, generate them inline (original path)
            stream_gen = runtime.streaming_pipeline.generate_speech_stream_indic(
                speaker=speaker,
                text=text,
                temperature=temperature,
                top_k=top_k,
                top_p=top_p,
                max_tokens=max_tokens,
                repetition_penalty=repetition_penalty,
                seed=seed,
                emit_progress=True,
            )

        async for stream_item in stream_gen:
            stream_metrics: Dict[str, Any] = {}
            audio_chunk = stream_item
            if (
                isinstance(stream_item, tuple)
                and len(stream_item) == 2
                and isinstance(stream_item[1], dict)
            ):
                audio_chunk = stream_item[0]
                stream_metrics = stream_item[1]

            if stream_metrics:
                metrics.update(stream_metrics)

            if not audio_chunk:
                # Metrics-only event from streaming pipeline.
                continue

            # First chunk: prepend WAV header
            if first_chunk_time is None:
                first_chunk_time = time.time()
                metrics["ttfb_ms"] = int((first_chunk_time - start_time) * 1000)
                wav_header = create_wav_header(sample_rate=sample_rate, data_size=0)
                audio_chunk = wav_header + audio_chunk

            total_pcm_bytes += len(audio_chunk) - (44 if metrics["chunks_sent"] == 0 else 0)
            metrics["chunks_sent"] += 1
            metrics["total_bytes"] = total_pcm_bytes + 44
            metrics["audio_duration_seconds"] = total_pcm_bytes / (sample_rate * 2)

            yield audio_chunk, metrics

async def _stream_chunked_text(
    runtime: TTSRuntime,
    long_processor,
    text: str,
    speaker: str,
    temperature: float,
    top_k: int,
    top_p: float,
    max_tokens: int,
    repetition_penalty: float,
    seed: Optional[int],
    sample_rate: int,
) -> AsyncGenerator[Tuple[bytes, Dict[str, Any]], None]:
    """
    Internal helper: stream chunked text with voice consistency.

    Uses global token caching from first chunk to maintain voice across chunks.
    """
    from veena3modal.audio.crossfade import crossfade_bytes_int16

    # Chunk text using the long text processor's chunker
    chunks = long_processor.chunk_text(text)

    if not chunks:
        return

    captured_globals: Optional[List[int]] = None
    previous_chunk_tail: Optional[bytes] = None
    chunk_metrics = {"chunks_processed": 0}

    for i, chunk_text in enumerate(chunks):
        chunk_metrics["chunks_processed"] = i + 1

        if i == 0:
            # First chunk: capture global tokens for voice consistency
            async for audio_bytes, global_ids in runtime.streaming_pipeline.generate_speech_stream_indic_first_chunk(
                speaker=speaker,
                text=chunk_text,
                temperature=temperature,
                top_k=top_k,
                top_p=top_p,
                max_tokens=max_tokens,
                repetition_penalty=repetition_penalty,
                seed=seed,
            ):
                if captured_globals is None and global_ids:
                    captured_globals = global_ids

                # Crossfade with previous tail (inter-chunk stitching)
                to_emit, previous_chunk_tail = crossfade_bytes_int16(
                    previous_chunk_tail,
                    audio_bytes,
                    sample_rate_hz=sample_rate,
                    crossfade_ms=50,
                )

                if to_emit:
                    yield to_emit, chunk_metrics
        else:
            # Continuation chunks: use captured globals for voice consistency
            if captured_globals is None:
                logger.warning(f"No captured globals for chunk {i+1}, using regular streaming")
                async for audio_bytes in runtime.streaming_pipeline.generate_speech_stream_indic(
                    speaker=speaker,
                    text=chunk_text,
                    temperature=temperature,
                    top_k=top_k,
                    top_p=top_p,
                    max_tokens=max_tokens,
                    repetition_penalty=repetition_penalty,
                    seed=seed,
                ):
                    to_emit, previous_chunk_tail = crossfade_bytes_int16(
                        previous_chunk_tail,
                        audio_bytes,
                        sample_rate_hz=sample_rate,
                        crossfade_ms=50,
                    )
                    if to_emit:
                        yield to_emit, chunk_metrics
            else:
                # Use continuation method with cached globals
                async for audio_bytes in runtime.streaming_pipeline.generate_speech_stream_indic_continuation(


Output:
Chunk ID: 550772
Wall time: 0.0504 seconds
Process exited with code 0
Original token count: 2649
Output:
"""
TTS runtime wrapper (container-scoped singleton).

In Modal, each container should load the model once and then handle many requests.
This wrapper owns:
- SparkTTSModel (vLLM engine)
- IndicPromptBuilder
- BiCodecDecoder
- SparkTTSPipeline + Veena3SlidingWindowPipeline
- optional SuperResolutionService

Imports framework-agnostic inference + processing code from `veena3modal/core`,
`veena3modal/processing`, and `veena3modal/audio`.
"""

from __future__ import annotations

import os
import sys
import time
import logging
from dataclasses import dataclass, field
from typing import Any, Dict, List, Optional, Tuple

logger = logging.getLogger(__name__)

@dataclass
class TTSRuntime:
    """
    Holds long-lived, per-container inference objects.

    Thread Safety:
        This class is designed for async concurrency within a single container.
        The vLLM engine handles internal batching and scheduling.
        Do NOT share across processes without proper synchronization.
    """
    model: Any = None
    pipeline: Any = None
    streaming_pipeline: Any = None
    long_text_processor: Any = None
    prompt_builder: Any = None
    bicodec_decoder: Any = None
    sr_service: Optional[Any] = None

    model_version: str = "not_loaded"
    is_loaded: bool = False
    load_time_ms: float = 0.0

    # Configuration
    model_path: str = ""
    bicodec_path: str = ""
    sr_checkpoint_dir: Optional[str] = None
    device: str = "cuda"

    # OPTIMIZATION: Pre-computed global tokens per speaker (eliminates ~110ms pre-roll in streaming)
    # Maps speaker_name -> list of 32 global token IDs captured at startup
    speaker_global_cache: Dict[str, List[int]] = field(default_factory=dict)

# Module-level singleton (per-container)
_runtime: Optional[TTSRuntime] = None

def get_runtime() -> Optional[TTSRuntime]:
    """Get the current TTS runtime singleton."""
    return _runtime

def is_initialized() -> bool:
    """Check if the runtime is initialized and ready."""
    return _runtime is not None and _runtime.is_loaded

def initialize_runtime(
    model_path: Optional[str] = None,
    bicodec_path: Optional[str] = None,
    sr_checkpoint_dir: Optional[str] = None,
    device: str = "cuda",
    hf_token: Optional[str] = None,
    gpu_memory_utilization: float = 0.25,
    enable_sr: bool = False,
    num_engines: int = 1,
    max_num_batched_tokens: Optional[int] = None,
    max_num_seqs: Optional[int] = None,
    enable_chunked_prefill: Optional[bool] = None,
    enable_prefix_caching: Optional[bool] = None,
    disable_log_stats: Optional[bool] = None,
    enforce_eager: Optional[bool] = None,
    precompute_speaker_globals: Optional[bool] = None,
) -> TTSRuntime:
    """
    Initialize the TTS runtime with all components.

    This should be called once per container (e.g., in Modal's @modal.enter).

    Args:
        model_path: Path to Spark TTS model (env: SPARK_TTS_MODEL_PATH)
        bicodec_path: Path to BiCodec model (env: BICODEC_MODEL_PATH, defaults to model_path)
        sr_checkpoint_dir: Path to super-resolution checkpoints (env: AP_BWE_CHECKPOINT_DIR)
        device: Device for inference (cuda/cpu)
        hf_token: HuggingFace token for private models (env: HF_TOKEN)
        gpu_memory_utilization: vLLM GPU memory fraction (default: 0.25)
        enable_sr: Enable super-resolution service
        num_engines: Number of vLLM engine instances (default: 1, set to 2-3 for Tier 3 optimization)
        max_num_batched_tokens: Optional vLLM scheduler cap override.
        max_num_seqs: Optional vLLM concurrent sequence cap override.
        enable_chunked_prefill: Optional vLLM chunked prefill toggle.
        enable_prefix_caching: Optional vLLM prefix caching toggle.
        disable_log_stats: Optional vLLM internal stats log toggle.
        enforce_eager: Optional vLLM eager-mode toggle (disables CUDA graphs when true).
        precompute_speaker_globals: Whether to warm speaker global tokens at startup.
            If None, reads PRECOMPUTE_SPEAKER_GLOBALS env var (default: false).

    Returns:
        Initialized TTSRuntime instance

    Raises:
        RuntimeError: If initialization fails
    """
    global _runtime

    start_time = time.time()

    # Resolve paths from env vars if not provided
    model_path = model_path or os.environ.get(
        'SPARK_TTS_MODEL_PATH',
        os.environ.get('MODEL_PATH', '/models/spark_tts_4speaker')
    )
    bicodec_path = bicodec_path or os.environ.get(
        'BICODEC_MODEL_PATH',
        model_path  # BiCodec is usually in the same directory
    )
    sr_checkpoint_dir = sr_checkpoint_dir or os.environ.get('AP_BWE_CHECKPOINT_DIR')
    hf_token = hf_token or os.environ.get('HF_TOKEN') or os.environ.get('HUGGING_FACE_HUB_TOKEN')
    if precompute_speaker_globals is None:
        raw_flag = os.environ.get("PRECOMPUTE_SPEAKER_GLOBALS", "false").strip().lower()
        precompute_speaker_globals = raw_flag in {"1", "true", "yes", "on"}

    logger.info(f"Initializing TTS runtime...")
    logger.info(f"  Model path: {model_path}")
    logger.info(f"  BiCodec path: {bicodec_path}")
    logger.info(f"  SR enabled: {enable_sr}, path: {sr_checkpoint_dir}")

    try:
        # Import inference components (vendored into veena3modal for Modal deployment)
        from veena3modal.core.model_loader import SparkTTSModel
        from veena3modal.core.pipeline import SparkTTSPipeline
        from veena3modal.core.bicodec_decoder import BiCodecDecoder
        from veena3modal.processing.prompt_builder import IndicPromptBuilder
        from veena3modal.processing.long_text_processor import LongTextProcessor

        engine_kwargs: Dict[str, Any] = {}
        if isinstance(max_num_batched_tokens, int) and max_num_batched_tokens > 0:
            engine_kwargs["max_num_batched_tokens"] = max_num_batched_tokens
        if isinstance(max_num_seqs, int) and max_num_seqs > 0:
            engine_kwargs["max_num_seqs"] = max_num_seqs
        if enable_chunked_prefill is not None:
            engine_kwargs["enable_chunked_prefill"] = bool(enable_chunked_prefill)
        if enable_prefix_caching is not None:
            engine_kwargs["enable_prefix_caching"] = bool(enable_prefix_caching)
        if disable_log_stats is not None:
            engine_kwargs["disable_log_stats"] = bool(disable_log_stats)
        if enforce_eager is not None:
            engine_kwargs["enforce_eager"] = bool(enforce_eager)

        # Load SparkTTS model with vLLM engine
        if num_engines > 1:
            # TIER 3 OPTIMIZATION: Multiple vLLM engines on same GPU
            # Each engine gets gpu_memory_utilization / num_engines to share GPU fairly.
            # E.g., 3 engines at 0.08 each = 0.24 total, ~8GB per engine, ~24GB total.
            from veena3modal.core.multi_engine import create_multi_engine_model
            per_engine_mem = gpu_memory_utilization / num_engines
            logger.info(f"Loading {num_engines} vLLM engines ({per_engine_mem:.2f} GPU mem each)...")
            model = create_multi_engine_model(
                model_path=model_path,
                num_engines=num_engines,
                hf_token=hf_token,
                gpu_memory_per_engine=per_engine_mem,
                **engine_kwargs,
            )
        else:
            logger.info("Loading SparkTTS model with vLLM...")
            model = SparkTTSModel(
                model_path=model_path,
                hf_token=hf_token,
                gpu_memory_utilization=gpu_memory_utilization,
                **engine_kwargs,
            )

        # Initialize prompt builder
        logger.info("Initializing prompt builder...")
        prompt_builder = IndicPromptBuilder(
            tokenizer=model.tokenizer,
            model=model,
        )

        # Initialize BiCodec decoder
        logger.info("Initializing BiCodec decoder...")
        batch_raw = os.environ.get("VEENA3_BICODEC_BATCHING", "true").strip().lower()
        enable_bicodec_batching = batch_raw in {"1", "true", "yes", "on"}
        bicodec_decoder = BiCodecDecoder(
            device=device,
            model_path=bicodec_path,
            enable_batching=enable_bicodec_batching,
        )

        # Initialize pipeline
        logger.info("Initializing TTS pipeline...")
        pipeline = SparkTTSPipeline(
            model=model,
            prompt_builder=prompt_builder,
            bicodec_decoder=bicodec_decoder,
        )

        # Initialize streaming pipeline (for M4)
        streaming_pipeline = None
        try:
            from veena3modal.core.streaming_pipeline import Veena3SlidingWindowPipeline
            logger.info("Initializing streaming pipeline...")
            # NOTE: Parameter is named 'snac_decoder' for legacy reasons, but works with BiCodecDecoder
            streaming_pipeline = Veena3SlidingWindowPipeline(
                model=model,
                prompt_builder=prompt_builder,
                snac_decoder=bicodec_decoder,  # BiCodecDecoder is interface-compatible
            )
        except ImportError as e:
            logger.warning(f"Streaming pipeline not available: {e}")

        # Initialize long text processor once (avoid per-request construction/logging)
        long_text_processor = LongTextProcessor(
            pipeline=pipeline,
            streaming_pipeline=streaming_pipeline,
        )

        # Initialize super-resolution (optional)
        sr_service = None
        if enable_sr and sr_checkpoint_dir:
            try:
                from veena3modal.core.super_resolution import SuperResolutionService
                logger.info(f"Initializing super-resolution from {sr_checkpoint_dir}...")
                sr_service = SuperResolutionService(checkpoint_dir=sr_checkpoint_dir)
                # Load the model explicitly
                if sr_service.load_model(device=device):
                    logger.info("✅ Super-resolution model loaded successfully")
                else:
                    logger.warning("Super-resolution model failed to load")
                    sr_service = None
            except Exception as e:
                logger.warning(f"Super-resolution not available: {e}")

        # Determine model version
        model_version = os.path.basename(model_path.rstrip('/'))
        if not model_version:
            model_version = "spark-tts"

        load_time = (time.time() - start_time) * 1000

        # Create runtime
        _runtime = TTSRuntime(


Output:
Chunk ID: 7622b5
Wall time: 0.0514 seconds
Process exited with code 0
Original token count: 2208
Output:
"""
Veena3 Streaming Pipeline - Sliding Window Approach

Implements streaming for both SNAC and BiCodec tokens.

For BiCodec (Spark TTS):
1. Generate tokens from vLLM
2. Parse NEW token IDs incrementally (O(1) per token via cache)
3. Buffer semantic + global tokens
4. Apply sliding window (every N token pairs)
5. Decode and stream audio chunks

OPTIMIZATION (Dec 2025):
- Replaced O(n²) pattern (decode-all + regex-all per iteration)
- Now uses incremental token parsing with BiCodecTokenParser
- ~10x CPU reduction in streaming hot loop

For SNAC (legacy):
1. Filter SNAC token IDs directly
2. Apply sliding window
3. Decode and stream
"""

import asyncio
import logging
import os
import re
from typing import Any, AsyncGenerator, Dict, List, Optional, Tuple, Union
from vllm import SamplingParams
from vllm.sampling_params import RequestOutputKind
from veena3modal.audio.crossfade import crossfade_bytes_int16
from veena3modal.core.token_utils import BiCodecTokenParser

from veena3modal.core.constants import (
    CODE_END_TOKEN_ID,
    CODE_START_TOKEN_ID,
    SNAC_MIN_ID,
    SNAC_MAX_ID,
    TRAINING_STOP_TOKEN_IDS,
    DEFAULT_TEMPERATURE,
    DEFAULT_TOP_K,
    DEFAULT_TOP_P,
    DEFAULT_MAX_TOKENS,
    DEFAULT_MIN_TOKENS,
    DEFAULT_REPETITION_PENALTY,
    DEFAULT_SEED,
)

logger = logging.getLogger(__name__)
_STREAM_OUTPUT_KIND_RAW = os.environ.get("VEENA3_STREAM_OUTPUT_KIND", "delta").strip().lower()
_STREAM_USE_DELTA_OUTPUT = _STREAM_OUTPUT_KIND_RAW in {"delta", "d", "1", "true", "yes", "on"}

def _env_int(name: str, default: int, minimum: int = 0) -> int:
    raw = os.environ.get(name, "")
    if not raw:
        return default
    try:
        value = int(raw)
    except ValueError:
        return default
    return max(minimum, value)

def _env_flag(name: str, default: bool) -> bool:
    raw = os.environ.get(name, "")
    if not raw:
        return default
    return raw.strip().lower() in {"1", "true", "yes", "on"}

_STREAM_EXPECTED_GLOBAL_COUNT = 32
_STREAM_MIN_SEMANTIC_FIRST_CHUNK = _env_int("VEENA3_STREAM_MIN_SEMANTIC_FIRST", 10, minimum=1)
_STREAM_DECODE_INTERVAL = _env_int("VEENA3_STREAM_DECODE_INTERVAL", 48, minimum=1)
_STREAM_CROSSFADE_MS = _env_int("VEENA3_STREAM_CROSSFADE_MS", 50, minimum=0)
_STREAM_WINDOW_SIZE = _env_int("VEENA3_STREAM_WINDOW_SIZE", 128, minimum=16)
_STREAM_ADAPTIVE_DECODE = _env_flag("VEENA3_STREAM_ADAPTIVE_DECODE", False)
_STREAM_DECODE_INTERVAL_FIRST = _env_int("VEENA3_STREAM_DECODE_INTERVAL_FIRST", _STREAM_DECODE_INTERVAL, minimum=1)
_STREAM_DECODE_INTERVAL_BUSY = _env_int(
    "VEENA3_STREAM_DECODE_INTERVAL_BUSY",
    max(_STREAM_DECODE_INTERVAL, 64),
    minimum=1,
)
_STREAM_DECODE_BUSY_PENDING = _env_int("VEENA3_STREAM_DECODE_BUSY_PENDING", 32, minimum=1)
_STREAM_WINDOWED_DECODE = os.environ.get("VEENA3_STREAM_WINDOWED_DECODE", "true").strip().lower() in {
    "1",
    "true",
    "yes",
    "on",
}

class Veena3SlidingWindowPipeline:
    """
    Streaming TTS pipeline using sliding window approach.

    This eliminates choppy audio and popping artifacts by:
    - Decoding overlapping 28-token windows (4 frames)
    - Keeping only the middle 2048 samples from each decode
    - Creating natural continuity between chunks

    Based on the official Canopy Labs implementation.
    """

    def __init__(
        self,
        model,
        prompt_builder,
        snac_decoder,
    ):
        """
        Initialize sliding window streaming pipeline.

        Args:
            model: Veena3Model instance
            prompt_builder: Veena3PromptBuilder instance
            snac_decoder: SNACDecoder instance (with batching enabled)
        """
        self.model = model
        self.prompt_builder = prompt_builder
        self.snac_decoder = snac_decoder

        # OPTIMIZATION: Pre-warm BiCodecTokenParser once at init, not per-request
        # Saves ~123ms per streaming TTFB by avoiding repeated 166K-entry vocab iteration
        tokenizer = getattr(model, "tokenizer", None)
        if tokenizer is None:
            tokenizer = getattr(model.engine, "tokenizer", None)
        self.token_parser = BiCodecTokenParser(tokenizer) if tokenizer else None

        print(f"🌊 Veena3SlidingWindowPipeline initialized (sliding window: 28 tokens)")

    @staticmethod
    def _p50(values: List[float]) -> float:
        if not values:
            return 0.0
        ordered = sorted(values)
        return ordered[(len(ordered) - 1) // 2]

    @staticmethod
    def _to_number(value: Any) -> Optional[float]:
        if value is None:
            return None
        if isinstance(value, (int, float)):
            return float(value)
        try:
            return float(value)
        except (TypeError, ValueError):
            return None

    def _extract_request_metrics(self, raw_metrics: Any) -> Dict[str, float]:
        if raw_metrics is None:
            return {}

        numeric: Dict[str, float] = {}
        items: List[Tuple[str, Any]] = []

        if isinstance(raw_metrics, dict):
            items.extend(raw_metrics.items())
        else:
            if hasattr(raw_metrics, "__dict__"):
                try:
                    items.extend(dict(raw_metrics.__dict__).items())
                except Exception:
                    pass
            for name in dir(raw_metrics):
                if name.startswith("_"):
                    continue
                try:
                    value = getattr(raw_metrics, name)
                except Exception:
                    continue
                if callable(value):
                    continue
                items.append((name, value))

        seen: set[str] = set()
        for key, value in items:
            key_s = str(key)
            if key_s in seen:
                continue
            seen.add(key_s)
            parsed = self._to_number(value)
            if parsed is not None:
                numeric[key_s] = parsed
        return numeric

    def _apply_stats(self, perf: Dict[str, Any], prefix: str, values: List[float]) -> None:
        if not values:
            return
        perf[f"{prefix}_total"] = float(sum(values))
        perf[f"{prefix}_min"] = float(min(values))
        perf[f"{prefix}_max"] = float(max(values))
        perf[f"{prefix}_p50"] = float(self._p50(values))

    def _get_decoder_pending_requests(self) -> int:
        getter = getattr(self.snac_decoder, "get_pending_requests", None)
        if not callable(getter):
            return 0
        try:
            return max(0, int(getter()))
        except Exception:
            return 0

    def _resolve_decode_interval(self, first_chunk_emitted: bool) -> Tuple[int, int]:
        interval = int(_STREAM_DECODE_INTERVAL)
        pending = 0
        if not _STREAM_ADAPTIVE_DECODE:
            return interval, pending

        if not first_chunk_emitted:
            interval = int(_STREAM_DECODE_INTERVAL_FIRST)

        pending = self._get_decoder_pending_requests()
        if pending >= _STREAM_DECODE_BUSY_PENDING:
            interval = max(interval, int(_STREAM_DECODE_INTERVAL_BUSY))
        return max(1, interval), pending

    async def generate_speech_stream(
        self,
        description: str,
        text: str,
        temperature: float = DEFAULT_TEMPERATURE,
        top_p: float = DEFAULT_TOP_P,
        max_tokens: int = DEFAULT_MAX_TOKENS,
        repetition_penalty: float = DEFAULT_REPETITION_PENALTY,
        seed: Optional[int] = None,
    ) -> AsyncGenerator[bytes, None]:
        """
        Generate speech audio with sliding window streaming.

        Yields audio chunks using overlapping windows for smooth playback.

        Args:
            description: Character/voice description
            text: Text to synthesize (with optional <emotion> tags)
            temperature: Sampling temperature
            top_p: Nucleus sampling
            max_tokens: Max SNAC tokens to generate
            repetition_penalty: Prevent loops

        Yields:
            Audio bytes (int16 PCM, 24kHz mono)
        """
        logger.debug("sliding-window streaming generation started")
        logger.debug("description_prefix=%s", description[:80])
        logger.debug("text_len=%d", len(text))

        # Build prompt
        prompt = self.prompt_builder.build_prefix(description, text)
        logger.debug("prompt built chars=%d", len(prompt))

        # Configure sampling
        sampling_params = SamplingParams(
            temperature=temperature,
            top_p=top_p,
            max_tokens=max_tokens,
            min_tokens=DEFAULT_MIN_TOKENS,
            repetition_penalty=repetition_penalty,
            stop_token_ids=[CODE_END_TOKEN_ID],  # Only audio EOS
            seed=seed,  # None = random, int = reproducible


Summary: auto


**Investigating streaming pipeline and error mapping**


I’ve confirmed the streaming hot path. Next I’m wiring a global runtime gate around `generate_speech_streaming` so we can cap inflight, optionally queue briefly, and return overload quickly when SLO would be violated. Then I’ll expose these controls in config and benchmark scripts.


rg -n "generate_speech_streaming\(|streaming_pipeline\.generate_speech|StreamingResponse|/v1/tts/generate|HTTPException|429|503" veena3modal -S


sed -n '1,320p' veena3modal/api/endpoints.py


sed -n '320,760p' veena3modal/api/endpoints.py