Output:
Chunk ID: 0f0330
Wall time: 0.0513 seconds
Process exited with code 0
Original token count: 585
Output:
142:    async def generate_with_chunking(
153:        return_timing: bool = False,
179:        llm_batch_wall_ms_values: List[float] = []
180:        llm_batch_gpu_ms_values: List[float] = []
183:        llm_batch_tokens: List[int] = []
244:                return_timing=return_timing,
247:            if return_timing:
252:                        llm_batch_wall_ms_values.append(int(chunk_timing.get("llm_wall_ms", 0)))
253:                        llm_batch_gpu_ms_values.append(int(chunk_timing.get("llm_gpu_batch_ms_total", 0)))
256:                        chunk_tokens = chunk_timing.get("token_batches", [])
258:                            llm_batch_tokens.extend([int(x) for x in chunk_tokens if isinstance(x, int) or str(x).isdigit()])
308:        if return_timing:
310:            chunked_metrics = {
319:                "llm_batch_count": sum(int(m.get("batch_count", 0) or 0) for m in emitted_chunk_metrics),
320:                "llm_batch_wall_ms_total": int(sum(llm_batch_wall_ms_values)),
321:                "llm_batch_gpu_ms_total": int(sum(llm_batch_gpu_ms_values)),
324:                "llm_batch_tokens": llm_batch_tokens,
327:            if llm_batch_wall_ms_values:
328:                chunked_metrics["llm_batch_wall_ms_min"] = int(min(llm_batch_wall_ms_values))
329:                chunked_metrics["llm_batch_wall_ms_max"] = int(max(llm_batch_wall_ms_values))
330:                chunked_metrics["llm_batch_wall_ms_p50"] = int(statistics.median(llm_batch_wall_ms_values))
331:            if llm_batch_gpu_ms_values:
332:                chunked_metrics["llm_batch_gpu_ms_min"] = int(min(llm_batch_gpu_ms_values))
333:                chunked_metrics["llm_batch_gpu_ms_max"] = int(max(llm_batch_gpu_ms_values))
334:                chunked_metrics["llm_batch_gpu_ms_p50"] = int(statistics.median(llm_batch_gpu_ms_values))
335:            if llm_batch_tokens:
336:                chunked_metrics["tokens_per_batch_min"] = min(llm_batch_tokens)
337:                chunked_metrics["tokens_per_batch_max"] = max(llm_batch_tokens)
338:                chunked_metrics["tokens_per_batch_p50"] = int(statistics.median(llm_batch_tokens))
339:            logger.info("tts_chunked_nonstream_complete", extra=chunked_metrics)
340:            return final_audio, chunked_metrics
397:    async def generate_with_chunking_streaming(


# Collaboration Mode: Default

You are now in Default mode. Any previous instructions for other modes (e.g. Plan mode) are no longer active.

Your active mode changes only when new developer instructions with a different `<collaboration_mode>...</collaboration_mode>` change it; user requests or tool descriptions do not change mode by themselves. Known mode names are Default and Plan.

## request_user_input availability

The `request_user_input` tool is unavailable in Default mode. If you call it while in Default mode, it will return an error.

If a decision is necessary and cannot be discovered from local context, ask the user directly. However, in Default mode you should strongly prefer executing the user's request rather than stopping to ask questions.

Summary: auto

## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) When `SKILL.md` references relative paths (e.g., `scripts/foo.py`), resolve them relative to the skill directory listed above first, and only consider other paths if needed.
  3) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  4) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  5) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.


sed -n '120,350p' /home/ubuntu/veenaModal/veena3modal/processing/long_text_processor.py


Output:
Chunk ID: f22d77
Wall time: 0.0512 seconds
Process exited with code 0
Original token count: 2450
Output:
            }
        )

    def should_chunk(self, text: str) -> bool:
        """Determine if text should be chunked."""
        return len(text) > self.CHUNKING_THRESHOLD

    def chunk_text(self, text: str) -> list:
        """
        Chunk text using the internal chunker.

        Delegates to self.chunker.chunk_text() for actual chunking logic.
        This method is used by _stream_chunked_text in tts_runtime.

        Args:
            text: Text to chunk

        Returns:
            List of text chunks
        """
        return self.chunker.chunk_text(text)

    async def generate_with_chunking(
        self,
        text: str,
        speaker: str,
        temperature: float = 0.4,
        top_k: int = 50,  # Added for Spark TTS
        top_p: float = 0.9,
        max_tokens: int = 4096,
        repetition_penalty: float = 1.05,  # Prevent token repetition
        seed: Optional[int] = None,
        sample_rate: int = 16000,  # BiCodec uses 16kHz, not 24kHz
        return_timing: bool = False,
        request_id: Optional[str] = None,
    ) -> Optional[bytes] | tuple[Optional[bytes], dict]:
        """
        Generate audio for long text with chunking and stitching.

        Process:
        1. Chunk text at natural boundaries
        2. Generate audio for each chunk
        3. Stitch audio with crossfade

        Args:
            text: Long text to synthesize
            speaker: Speaker name (lipakshi, reet, etc.)
            temperature: Sampling temperature
            top_p: Nucleus sampling
            max_tokens: Max tokens per chunk
            seed: Random seed (same seed = same voice across chunks)
            sample_rate: Audio sample rate

        Returns:
            Stitched audio bytes (int16 PCM) and optional timing metrics
        """
        # Tracking variables for detailed timing aggregation
        request_start = time.perf_counter()
        emitted_chunk_metrics: List[dict] = []
        llm_batch_wall_ms_values: List[float] = []
        llm_batch_gpu_ms_values: List[float] = []
        llm_decode_cpu_ms_values: List[float] = []
        llm_parse_ms_values: List[float] = []
        llm_batch_tokens: List[int] = []

        # Detect language mix for logging
        lang_info = self.chunker.detect_language_mix(text)

        logger.info(
            f"Processing long text with chunking",
            extra={
                "text_length": len(text),
                "primary_language": lang_info['primary'],
                "lang_mix": lang_info,
                "speaker": speaker,
            }
        )

        # 1. Chunk text
        text_chunks = self.chunker.chunk_text(text)

        logger.info(
            f"Text chunked into {len(text_chunks)} chunks",
            extra={
                "num_chunks": len(text_chunks),
                "chunk_sizes": [len(chunk) for chunk in text_chunks],
            }
        )

        # 2. Generate audio for each chunk
        audio_chunks = []

        # Log summary at start
        logger.info(
            f"🎵 Generating {len(text_chunks)} chunks for {len(text)} chars",
            extra={
                "total_chunks": len(text_chunks),
                "total_chars": len(text),
                "speaker": speaker,
            }
        )

        for i, chunk in enumerate(text_chunks):
            # Calculate appropriate max_tokens for this chunk
            # BiCodec semantic tokens: ~50 tokens/second, ~5 words/second = ~10 tokens/word
            # Rough estimate: 5 chars/word, so ~2 tokens/char for audio generation
            # Add buffer for safety: multiply by 3 to handle variations
            chunk_max_tokens = min(
                int(len(chunk) * 3),  # ~2 tokens/char * 1.5 safety buffer = 3
                4096  # Cap at model's max context length
            )
            chunk_max_tokens = max(chunk_max_tokens, 100)  # Ensure minimum

            # Generate audio for this chunk (returns WAV format with header)
            chunk_result = await self.pipeline.generate_speech_indic(
                speaker=speaker,
                text=chunk,
                temperature=temperature,
                top_k=top_k,  # Added for Spark TTS
                top_p=top_p,
                max_tokens=chunk_max_tokens,  # Use calculated limit, not global max_tokens
                repetition_penalty=repetition_penalty,  # Prevent token repetition
                seed=seed,  # Use same seed for consistent voice across chunks
                request_id=request_id,
                return_timing=return_timing,
            )

            if return_timing:
                if isinstance(chunk_result, tuple):
                    audio_bytes, chunk_timing = chunk_result
                    if isinstance(chunk_timing, dict):
                        emitted_chunk_metrics.append(chunk_timing)
                        llm_batch_wall_ms_values.append(int(chunk_timing.get("llm_wall_ms", 0)))
                        llm_batch_gpu_ms_values.append(int(chunk_timing.get("llm_gpu_batch_ms_total", 0)))
                        llm_decode_cpu_ms_values.append(int(chunk_timing.get("decode_cpu_ms", 0)))
                        llm_parse_ms_values.append(int(chunk_timing.get("parse_ms", 0)))
                        chunk_tokens = chunk_timing.get("token_batches", [])
                        if isinstance(chunk_tokens, list):
                            llm_batch_tokens.extend([int(x) for x in chunk_tokens if isinstance(x, int) or str(x).isdigit()])
                else:
                    audio_bytes = chunk_result
            else:
                audio_bytes = chunk_result

            if audio_bytes is None:
                logger.error(
                    f"❌ Failed to generate audio for chunk {i+1}/{len(text_chunks)}",
                    extra={
                        "chunk_index": i,
                        "chunk_length": len(chunk),
                        "chunk_text_preview": chunk[:150],
                        "total_chunks": len(text_chunks),
                    }
                )
                return None

            # Strip WAV header (44 bytes) to get raw PCM data for stitching
            pcm_data = audio_bytes[44:] if len(audio_bytes) > 44 else audio_bytes
            audio_chunks.append(pcm_data)

        # 3. Stitch audio with crossfade
        stitched_pcm = self.stitch_audio_chunks(
            audio_chunks,
            sample_rate=sample_rate,
            crossfade_ms=self.CROSSFADE_MS,
        )

        # 4. Add WAV header to stitched PCM data
        wav_header = create_wav_header(
            data_size=len(stitched_pcm),
            sample_rate=sample_rate,
            channels=1,
            bits_per_sample=16
        )
        final_audio = wav_header + stitched_pcm

        # Calculate total audio metrics
        total_audio_duration = len(stitched_pcm) / (sample_rate * 2)

        logger.info(
            f"✅ Generated {len(text_chunks)} chunks → {total_audio_duration:.1f}s audio",
            extra={
                "total_chunks": len(text_chunks),
                "audio_duration": total_audio_duration,
                "audio_size_mb": len(final_audio) / (1024 * 1024),
            }
        )

        if return_timing:
            request_ms = (time.perf_counter() - request_start) * 1000
            chunked_metrics = {
                "request_id": request_id,
                "speaker": speaker,
                "text_length": len(text),
                "chunks_processed": len(text_chunks),
                "audio_duration_seconds": total_audio_duration,
                "audio_bytes": len(final_audio),
                "text_chunked": True,
                "token_total": sum(m.get("token_total", 0) for m in emitted_chunk_metrics),
                "llm_batch_count": sum(int(m.get("batch_count", 0) or 0) for m in emitted_chunk_metrics),
                "llm_batch_wall_ms_total": int(sum(llm_batch_wall_ms_values)),
                "llm_batch_gpu_ms_total": int(sum(llm_batch_gpu_ms_values)),
                "llm_decode_cpu_ms": int(sum(llm_decode_cpu_ms_values)),
                "llm_parse_ms": int(sum(llm_parse_ms_values)),
                "llm_batch_tokens": llm_batch_tokens,
                "generation_ms": int(request_ms),
            }
            if llm_batch_wall_ms_values:
                chunked_metrics["llm_batch_wall_ms_min"] = int(min(llm_batch_wall_ms_values))
                chunked_metrics["llm_batch_wall_ms_max"] = int(max(llm_batch_wall_ms_values))
                chunked_metrics["llm_batch_wall_ms_p50"] = int(statistics.median(llm_batch_wall_ms_values))
            if llm_batch_gpu_ms_values:
                chunked_metrics["llm_batch_gpu_ms_min"] = int(min(llm_batch_gpu_ms_values))
                chunked_metrics["llm_batch_gpu_ms_max"] = int(max(llm_batch_gpu_ms_values))
                chunked_metrics["llm_batch_gpu_ms_p50"] = int(statistics.median(llm_batch_gpu_ms_values))
            if llm_batch_tokens:
                chunked_metrics["tokens_per_batch_min"] = min(llm_batch_tokens)
                chunked_metrics["tokens_per_batch_max"] = max(llm_batch_tokens)
                chunked_metrics["tokens_per_batch_p50"] = int(statistics.median(llm_batch_tokens))
            logger.info("tts_chunked_nonstream_complete", extra=chunked_metrics)
            return final_audio, chunked_metrics

        return final_audio

    def stitch_audio_chunks(
        self,
        audio_chunks: List[bytes],
        sample_rate: int = 16000,  # BiCodec uses 16kHz, not 24kHz
        crossfade_ms: int = 50,
    ) -> bytes:
        """


Summary: auto


Output:
Chunk ID: 76e756
Wall time: 0.0506 seconds
Process exited with code 0
Original token count: 770
Output:
311:            request_metrics["request_id"] = request_id
377:                    "tts_stream_llm_batch",
383:                        "llm_batch_wall_ms": int(batch_wall_ms_value),
483:                                    "tts_stream_decode_batch",
538:        ttfb_ms = (
554:                "llm_batch_wall_ms_total": int(sum(batch_wall_ms)),
555:                "llm_batch_wall_ms_min": int(min(batch_wall_ms)) if batch_wall_ms else 0,
556:                "llm_batch_wall_ms_max": int(max(batch_wall_ms)) if batch_wall_ms else 0,
557:                "llm_batch_wall_ms_p50": int(statistics.median(batch_wall_ms)) if batch_wall_ms else 0,
572:                "ttfb_ms": int(ttfb_ms),
582:            logger.info("tts_stream_indic_complete", extra=request_metrics)
584:    async def generate_speech_stream_indic_first_chunk(
678:            request_metrics["request_id"] = request_id
733:                    "tts_stream_llm_batch",
739:                        "llm_batch_wall_ms": int(batch_wall_ms_value),
819:                                "tts_stream_decode_batch",
872:                        "tts_stream_decode_batch",
894:        ttfb_ms = (
912:                "llm_batch_wall_ms_total": int(sum(batch_wall_ms)),
913:                "llm_batch_wall_ms_min": int(min(batch_wall_ms)) if batch_wall_ms else 0,
914:                "llm_batch_wall_ms_max": int(max(batch_wall_ms)) if batch_wall_ms else 0,
915:                "llm_batch_wall_ms_p50": int(statistics.median(batch_wall_ms)) if batch_wall_ms else 0,
935:                "ttfb_ms": int(ttfb_ms),
940:            request_metrics["tokens_per_batch_min"] = min(batch_token_counts)
941:            request_metrics["tokens_per_batch_max"] = max(batch_token_counts)
942:            request_metrics["tokens_per_batch_p50"] = int(statistics.median(batch_token_counts))
944:        logger.info("tts_stream_first_chunk_complete", extra=request_metrics)
946:    async def generate_speech_stream_indic_continuation(
1049:            request_metrics["request_id"] = request_id
1102:                    "tts_stream_llm_batch",
1108:                        "llm_batch_wall_ms": int(batch_wall_ms_value),
1182:                                "tts_stream_decode_batch",
1235:                                "tts_stream_decode_batch",
1256:        ttfb_ms = (
1274:                "llm_batch_wall_ms_total": int(sum(batch_wall_ms)),
1275:                "llm_batch_wall_ms_min": int(min(batch_wall_ms)) if batch_wall_ms else 0,
1276:                "llm_batch_wall_ms_max": int(max(batch_wall_ms)) if batch_wall_ms else 0,
1277:                "llm_batch_wall_ms_p50": int(statistics.median(batch_wall_ms)) if batch_wall_ms else 0,
1296:                "ttfb_ms": int(ttfb_ms),
1301:            request_metrics["tokens_per_batch_min"] = min(batch_token_counts)
1302:            request_metrics["tokens_per_batch_max"] = max(batch_token_counts)
1303:            request_metrics["tokens_per_batch_p50"] = int(statistics.median(batch_token_counts))
1304:        logger.info("tts_stream_continuation_complete", extra=request_metrics)


sed -n '520,610p' /home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py


sed -n '900,980p' /home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py


sed -n '1250,1325p' /home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py


Output:
Chunk ID: d7d16e
Wall time: 0.0509 seconds
Process exited with code 0
Original token count: 1207
Output:

                    samples_emitted_in_final = len(to_emit) // 2
                    total_samples_emitted_to_user += samples_emitted_in_final

                    if to_emit:
                        total_audio_chunks += 1
                        yield to_emit

        # Flush any remaining tail that was held back for crossfade
        if previous_chunk_tail:
            total_audio_chunks += 1
            tail_samples = len(previous_chunk_tail) // 2
            total_samples_emitted_to_user += tail_samples
            yield previous_chunk_tail

        # Log completion summary
        t_end = time.time()
        audio_duration_s = total_samples_emitted_to_user / 16000  # 16kHz sample rate
        ttfb_ms = (
            (t_first_chunk_decoded - t_start) * 1000
            if t_first_chunk_decoded is not None
            else (t_first_token - t_start) * 1000
            if t_first_token is not None
            else 0
        )
        total_wall_ms = (t_end - t_start) * 1000
        request_metrics.update(
            {
                "llm_request_id": request_id,
                "generated_tokens": len(semantic_buffer) + len(global_buffer),
                "semantic_tokens": len(semantic_buffer),
                "global_tokens": len(global_buffer),
                "llm_batch_count": len(batch_wall_ms),
                "llm_batch_tokens": batch_token_counts,
                "llm_batch_wall_ms_total": int(sum(batch_wall_ms)),
                "llm_batch_wall_ms_min": int(min(batch_wall_ms)) if batch_wall_ms else 0,
                "llm_batch_wall_ms_max": int(max(batch_wall_ms)) if batch_wall_ms else 0,
                "llm_batch_wall_ms_p50": int(statistics.median(batch_wall_ms)) if batch_wall_ms else 0,
                "llm_batch_gpu_ms_total": int(sum(batch_gpu_ms)),
                "llm_batch_gpu_ms_min": int(min(batch_gpu_ms)) if batch_gpu_ms else 0,
                "llm_batch_gpu_ms_max": int(max(batch_gpu_ms)) if batch_gpu_ms else 0,
                "llm_batch_gpu_ms_p50": int(statistics.median(batch_gpu_ms)) if batch_gpu_ms else 0,
                "llm_parse_ms": int(sum(parse_wall_ms_list)),
                "llm_parse_ms_avg": int(sum(parse_wall_ms_list) / len(parse_wall_ms_list)) if parse_wall_ms_list else 0,
                "decode_calls": decode_calls,
                "llm_decode_wall_ms_total": int(decode_wall_ms),
                "llm_decode_wall_ms_min": int(min(decode_wall_samples)) if decode_wall_samples else 0,
                "llm_decode_wall_ms_max": int(max(decode_wall_samples)) if decode_wall_samples else 0,
                "llm_decode_wall_ms_p50": int(statistics.median(decode_wall_samples)) if decode_wall_samples else 0,
                "llm_decode_gpu_ms_total": int(decode_gpu_ms),
                "llm_decode_gpu_ms_min": int(min(decode_gpu_samples)) if decode_gpu_samples else 0,
                "llm_decode_gpu_ms_max": int(max(decode_gpu_samples)) if decode_gpu_samples else 0,
                "ttfb_ms": int(ttfb_ms),
                "generation_ms": int(total_wall_ms),
                "audio_duration_seconds": audio_duration_s,
                "audio_chunks": total_audio_chunks,
                "samples_emitted": total_samples_emitted_to_user,
                "text_chunked": False,
                "concurrency_mode": "single",
            }
        )
        if emit_timing:
            logger.info("tts_stream_indic_complete", extra=request_metrics)

    async def generate_speech_stream_indic_first_chunk(
        self,
        speaker: str,
        text: str,
        temperature: float = DEFAULT_TEMPERATURE,
        top_k: int = DEFAULT_TOP_K,
        top_p: float = DEFAULT_TOP_P,
        max_tokens: int = DEFAULT_MAX_TOKENS,
        repetition_penalty: float = DEFAULT_REPETITION_PENALTY,
        seed: Optional[int] = None,
        request_id: Optional[str] = None,
        request_metrics: Optional[Dict[str, Any]] = None,
    ) -> AsyncGenerator[tuple, None]:
        """
        Generate speech for the FIRST chunk of multi-chunk text, capturing global tokens.

        This method is specifically for chunked generation: it yields (audio_bytes, global_ids)
        tuples instead of just audio_bytes. The caller captures global_ids from the first
        yield and passes them to generate_speech_stream_indic_continuation() for subsequent chunks.

        Use Case (chunked streaming with voice consistency):
            globals_captured = None
            async for audio_bytes, global_ids in pipeline.generate_speech_stream_indic_first_chunk(...):
                if globals_captured is None and global_ids:
                    globals_captured = global_ids  # Capture once
                yield audio_bytes
            # Now use globals_captured for subsequent chunks


Output:
Chunk ID: 21c6e8
Wall time: 0.0509 seconds
Process exited with code 0
Original token count: 1111
Output:
            if t_first_token is not None
            else 0
        )
        request_metrics.update(
            {
                "request_id": request_id,
                "speaker": speaker,
                "generated_tokens": len(semantic_buffer) + len(global_buffer),
                "semantic_tokens": len(semantic_buffer),
                "global_tokens": len(captured_globals),
                "llm_batch_count": len(batch_wall_ms),
                "llm_batch_tokens": batch_token_counts,
                "llm_batch_wall_ms_total": int(sum(batch_wall_ms)),
                "llm_batch_wall_ms_min": int(min(batch_wall_ms)) if batch_wall_ms else 0,
                "llm_batch_wall_ms_max": int(max(batch_wall_ms)) if batch_wall_ms else 0,
                "llm_batch_wall_ms_p50": int(statistics.median(batch_wall_ms)) if batch_wall_ms else 0,
                "llm_batch_gpu_ms_total": int(sum(batch_gpu_ms)),
                "llm_batch_gpu_ms_min": int(min(batch_gpu_ms)) if batch_gpu_ms else 0,
                "llm_batch_gpu_ms_max": int(max(batch_gpu_ms)) if batch_gpu_ms else 0,
                "llm_batch_gpu_ms_p50": int(statistics.median(batch_gpu_ms)) if batch_gpu_ms else 0,
                "llm_parse_ms": int(sum(parse_wall_ms_list)),
                "llm_parse_ms_avg": int(sum(parse_wall_ms_list) / len(parse_wall_ms_list)) if parse_wall_ms_list else 0,
                "decode_calls": decode_calls,
                "llm_decode_wall_ms_total": int(decode_wall_ms),
                "llm_decode_wall_ms_min": int(min(decode_wall_samples)) if decode_wall_samples else 0,
                "llm_decode_wall_ms_max": int(max(decode_wall_samples)) if decode_wall_samples else 0,
                "llm_decode_wall_ms_p50": int(statistics.median(decode_wall_samples)) if decode_wall_samples else 0,
                "llm_decode_gpu_ms_total": int(decode_gpu_ms),
                "llm_decode_gpu_ms_min": int(min(decode_gpu_samples)) if decode_gpu_samples else 0,
                "llm_decode_gpu_ms_max": int(max(decode_gpu_samples)) if decode_gpu_samples else 0,
                "llm_decode_gpu_ms_p50": int(statistics.median(decode_gpu_samples)) if decode_gpu_samples else 0,
                "audio_duration_seconds": audio_duration_s,
                "audio_chunks": total_audio_chunks,
                "captured_global_count": len(captured_globals),
                "text_chunked": True,
                "ttfb_ms": int(ttfb_ms),
                "generation_ms": int((t_end - t_start) * 1000),
            }
        )
        if batch_token_counts:
            request_metrics["tokens_per_batch_min"] = min(batch_token_counts)
            request_metrics["tokens_per_batch_max"] = max(batch_token_counts)
            request_metrics["tokens_per_batch_p50"] = int(statistics.median(batch_token_counts))

        logger.info("tts_stream_first_chunk_complete", extra=request_metrics)

    async def generate_speech_stream_indic_continuation(
        self,
        speaker: str,
        text: str,
        global_ids: List[int],
        temperature: float = DEFAULT_TEMPERATURE,
        top_k: int = DEFAULT_TOP_K,
        top_p: float = DEFAULT_TOP_P,
        max_tokens: int = DEFAULT_MAX_TOKENS,
        repetition_penalty: float = DEFAULT_REPETITION_PENALTY,
        seed: Optional[int] = None,
        request_id: Optional[str] = None,
        request_metrics: Optional[Dict[str, Any]] = None,
    ) -> AsyncGenerator[bytes, None]:
        """
        Generate speech for CONTINUATION chunks using pre-captured global tokens.

        This is the key method for voice consistency in chunked generation:
        - Uses global_ids from the first chunk to maintain identical voice
        - Model generates only semantic tokens (skips global token generation)
        - Ensures no voice drift across chunks

        CRITICAL for production:
        - global_ids is request-scoped, passed explicitly (no shared state)
        - Thread-safe: each request has its own state
        - No global caching that could cause cross-request contamination

        Args:
            speaker: Speaker name (must match first chunk)
            text: Continuation text chunk to synthesize
            global_ids: 32 global token IDs captured from first chunk
            temperature: Sampling temperature
            top_k: Top-k sampling
            top_p: Nucleus sampling
            max_tokens: Max tokens to generate


Output:
Chunk ID: 615c49
Wall time: 0.0510 seconds
Process exited with code 0
Original token count: 833
Output:
            total_audio_chunks += 1
            tail_samples = len(previous_chunk_tail) // 2
            total_samples_emitted_to_user += tail_samples
            yield previous_chunk_tail

        t_end = time.time()
        ttfb_ms = (
            (t_first_chunk_decoded - t_start) * 1000
            if t_first_chunk_decoded is not None
            else (t_first_chunk_ready - t_start) * 1000
            if t_first_chunk_ready is not None
            else (t_first_token - t_start) * 1000
            if t_first_token is not None
            else 0
        )
        request_metrics.update(
            {
                "request_id": request_id,
                "speaker": speaker,
                "generated_tokens": len(semantic_buffer) + len(global_ids),
                "semantic_tokens": len(semantic_buffer),
                "global_tokens": len(global_ids),
                "llm_batch_count": len(batch_wall_ms),
                "llm_batch_tokens": batch_token_counts,
                "llm_batch_wall_ms_total": int(sum(batch_wall_ms)),
                "llm_batch_wall_ms_min": int(min(batch_wall_ms)) if batch_wall_ms else 0,
                "llm_batch_wall_ms_max": int(max(batch_wall_ms)) if batch_wall_ms else 0,
                "llm_batch_wall_ms_p50": int(statistics.median(batch_wall_ms)) if batch_wall_ms else 0,
                "llm_batch_gpu_ms_total": int(sum(batch_gpu_ms)),
                "llm_batch_gpu_ms_min": int(min(batch_gpu_ms)) if batch_gpu_ms else 0,
                "llm_batch_gpu_ms_max": int(max(batch_gpu_ms)) if batch_gpu_ms else 0,
                "llm_batch_gpu_ms_p50": int(statistics.median(batch_gpu_ms)) if batch_gpu_ms else 0,
                "llm_parse_ms": int(sum(parse_wall_ms_list)),
                "llm_parse_ms_avg": int(sum(parse_wall_ms_list) / len(parse_wall_ms_list)) if parse_wall_ms_list else 0,
                "decode_calls": decode_calls,
                "llm_decode_wall_ms_total": int(decode_wall_ms),
                "llm_decode_wall_ms_min": int(min(decode_wall_samples)) if decode_wall_samples else 0,
                "llm_decode_wall_ms_max": int(max(decode_wall_samples)) if decode_wall_samples else 0,
                "llm_decode_wall_ms_p50": int(statistics.median(decode_wall_samples)) if decode_wall_samples else 0,
                "llm_decode_gpu_ms_total": int(decode_gpu_ms),
                "llm_decode_gpu_ms_min": int(min(decode_gpu_samples)) if decode_gpu_samples else 0,
                "llm_decode_gpu_ms_max": int(max(decode_gpu_samples)) if decode_gpu_samples else 0,
                "llm_decode_gpu_ms_p50": int(statistics.median(decode_gpu_samples)) if decode_gpu_samples else 0,
                "audio_duration_seconds": total_samples_emitted_to_user / 16000,
                "audio_chunks": total_audio_chunks,
                "text_chunked": True,
                "ttfb_ms": int(ttfb_ms),
                "generation_ms": int((t_end - t_start) * 1000),
            }
        )
        if batch_token_counts:
            request_metrics["tokens_per_batch_min"] = min(batch_token_counts)
            request_metrics["tokens_per_batch_max"] = max(batch_token_counts)
            request_metrics["tokens_per_batch_p50"] = int(statistics.median(batch_token_counts))
        logger.info("tts_stream_continuation_complete", extra=request_metrics)


rg -n "handle_websocket_tts|add_websocket_routes|/v1/tts/ws|WebSocketRoute" /home/ubuntu/veenaModal/veena3modal/api -g "*.py"


Output:
Chunk ID: 3872e3
Wall time: 0.0514 seconds
Process exited with code 0
Original token count: 283
Output:
/home/ubuntu/veenaModal/veena3modal/api/fastapi_app.py:665:        from veena3modal.api.websocket_handler import add_websocket_routes
/home/ubuntu/veenaModal/veena3modal/api/fastapi_app.py:666:        add_websocket_routes(app)
/home/ubuntu/veenaModal/veena3modal/api/fastapi_app.py:667:        logger.info("WebSocket support enabled at /v1/tts/ws")
/home/ubuntu/veenaModal/veena3modal/api/websocket_handler.py:10:1. Client connects to /v1/tts/ws
/home/ubuntu/veenaModal/veena3modal/api/websocket_handler.py:165:async def handle_websocket_tts(websocket, api_key: Optional[str] = None):
/home/ubuntu/veenaModal/veena3modal/api/websocket_handler.py:461:def add_websocket_routes(app):
/home/ubuntu/veenaModal/veena3modal/api/websocket_handler.py:472:    @app.websocket("/v1/tts/ws")
/home/ubuntu/veenaModal/veena3modal/api/websocket_handler.py:481:        ws://host/v1/tts/ws?api_key=your-key
/home/ubuntu/veenaModal/veena3modal/api/websocket_handler.py:497:        await handle_websocket_tts(websocket, api_key)
/home/ubuntu/veenaModal/veena3modal/api/websocket_handler.py:499:    logger.info("WebSocket route /v1/tts/ws registered")


sed -n '440,540p' /home/ubuntu/veenaModal/veena3modal/api/websocket_handler.py


Output:
Chunk ID: f3d3a1
Wall time: 0.0506 seconds
Process exited with code 0
Original token count: 421
Output:
            error_message=str(e),
        )
        metrics_request_failed(
            status_code=500,
            error_code="WEBSOCKET_ERROR",
        )

        try:
            await websocket.send_text(create_error_message(
                code="INTERNAL_ERROR",
                message=str(e),
                request_id=request_id,
            ))
            await websocket.close(code=1011)
        except Exception:
            pass

    finally:
        clear_request_context()

def add_websocket_routes(app):
    """
    Add WebSocket routes to FastAPI app.

    Call this from create_app() to enable WebSocket support.

    Args:
        app: FastAPI application instance
    """
    from fastapi import WebSocket, Query

    @app.websocket("/v1/tts/ws")
    async def websocket_tts(
        websocket: WebSocket,
        api_key: Optional[str] = Query(None, alias="api_key"),
    ):
        """
        WebSocket endpoint for TTS streaming.

        Connect with optional api_key query parameter:
        ws://host/v1/tts/ws?api_key=your-key

        Then send JSON request:
        {"text": "Hello world", "speaker": "male1"}

        Receive:
        - JSON header message
        - Binary audio chunks
        - JSON progress messages (every 10 chunks)
        - JSON completion message
        """
        # Also check Authorization header
        auth_header = websocket.headers.get("authorization", "")
        if auth_header.startswith("Bearer "):
            api_key = auth_header[7:]

        await handle_websocket_tts(websocket, api_key)

    logger.info("WebSocket route /v1/tts/ws registered")


rg -n "uvicorn|modal serve|veena3modal.api.fastapi_app|create_app\(|app.py" /home/ubuntu/veenaModal/scripts /home/ubuntu/veenaModal/README* /home/ubuntu/veenaModal


Output:
Chunk ID: 5bb9de
Wall time: 0.0510 seconds
Process exited with code 0
Original token count: 1813
Output:
/home/ubuntu/veenaModal/README.md:19:modal serve veena3modal/app.py
/home/ubuntu/veenaModal/README.md:21:modal deploy veena3modal/app.py
/home/ubuntu/veenaModal/README.md:73:- **Dev**: `AUTH_BYPASS_MODE=true` bypasses key validation (enabled by default in `veena3modal/app.py` image env for testing).
/home/ubuntu/veenaModal/README.md:88:- Model weights are expected to live on a Modal Volume (see `veena3modal/app.py`).
/home/ubuntu/veenaModal/modal_docs/# Managing deployments.md:3:Once you've finished using `modal run` or `modal serve` to iterate on your Modal
/home/ubuntu/veenaModal/modal_docs/# Job processing.md:133:You can try this app by serving it with `modal serve`:
/home/ubuntu/veenaModal/modal_docs/# Job processing.md:136:modal serve my_job_queue_endpoint.py
/home/ubuntu/veenaModal/modal_docs/# Run a FastRTC app on Modal.md:165:modal serve 07_web_endpoints/fastrtc_flip_webcam.py
/home/ubuntu/veenaModal/modal_docs/# Run a FastRTC app on Modal.md:172:The `modal serve` command produces a hot-reloading development server --
/home/ubuntu/veenaModal/modal_docs/Reference/# modal.config.md:81:  when the App is ephemeral (i.e., created via `modal serve`). Must be a short
/home/ubuntu/veenaModal/modal_docs/Reference/# modal.Image.md:892:    modal.Image.debian_slim().cmd(["python", "app.py"])
/home/ubuntu/veenaModal/modal_docs/# QuiLLMan: Voice Chat with Moshi.md:100:modal serve -m src.moshi
/home/ubuntu/veenaModal/modal_docs/# QuiLLMan: Voice Chat with Moshi.md:107:The frontend is a static React app, found in the `src/frontend` directory and served by `src/app.py`.
/home/ubuntu/veenaModal/modal_docs/# QuiLLMan: Voice Chat with Moshi.md:116:modal serve -m src.app
/home/ubuntu/veenaModal/modal_docs/# QuiLLMan: Voice Chat with Moshi.md:119:Since `src/app.py` imports the `src/moshi.py` module, this `serve` command also serves the Moshi websocket server as its own endpoint.
/home/ubuntu/veenaModal/modal_docs/# Dynamic batching (beta).md:119:[`modal serve`](/docs/reference/cli/serve), submit requests to the endpoint,
/home/ubuntu/veenaModal/scripts/setup_local.sh:102:# FastAPI + uvicorn (not in requirements.txt since Modal bundles them)
/home/ubuntu/veenaModal/scripts/setup_local.sh:105:    "uvicorn[standard]>=0.23.0" \
/home/ubuntu/veenaModal/modal_docs/# Invoking deployed functions.md:23:Let's say you have a script `my_shared_app.py` and this script defines a Modal
/home/ubuntu/veenaModal/modal_docs/# Invoking deployed functions.md:40:% modal deploy shared_app.py
/home/ubuntu/veenaModal/modal_docs/# Invoking deployed functions.md:44:├── 🔨 Mounted /Users/erikbern/modal/shared_app.py.
/home/ubuntu/veenaModal/modal_docs/# Hello world wide web!.md:49:You can turn this function into a web endpoint by running `modal serve basic_web.py`.
/home/ubuntu/veenaModal/modal_docs/# Hello world wide web!.md:58:By running the endpoint with `modal serve`, you created a temporary endpoint that will disappear if you interrupt your terminal.
/home/ubuntu/veenaModal/modal_docs/# Hello world wide web!.md:91:If you are already running `modal serve basic_web.py`, this endpoint will be available at a URL, printed in your terminal, that ends with `greet-dev.modal.run`.
/home/ubuntu/veenaModal/modal_docs/# Hello world wide web!.md:190:The `expensive-secret` endpoint URL will still be printed to the output when you `modal serve` or `modal deploy`,
/home/ubuntu/veenaModal/scripts/setup_local.sh:102:# FastAPI + uvicorn (not in requirements.txt since Modal bundles them)
/home/ubuntu/veenaModal/scripts/setup_local.sh:105:    "uvicorn[standard]>=0.23.0" \
/home/ubuntu/veenaModal/veena3modal/local_server.py:8:4. Serves the FastAPI app via uvicorn
/home/ubuntu/veenaModal/veena3modal/local_server.py:250:    from veena3modal.api.fastapi_app import create_app
/home/ubuntu/veenaModal/veena3modal/local_server.py:251:    return create_app()
/home/ubuntu/veenaModal/veena3modal/local_server.py:350:        help="Number of uvicorn workers (default: 1, each loads full model)",
/home/ubuntu/veenaModal/veena3modal/local_server.py:390:    5. Run uvicorn
/home/ubuntu/veenaModal/veena3modal/local_server.py:436:    # Run uvicorn
/home/ubuntu/veenaModal/veena3modal/local_server.py:437:    # NOTE: We use the factory string so uvicorn can find the app.
/home/ubuntu/veenaModal/veena3modal/local_server.py:438:    # With --reload, uvicorn reimports, but the runtime singleton persists in memory.
/home/ubuntu/veenaModal/veena3modal/local_server.py:439:    import uvicorn
/home/ubuntu/veenaModal/veena3modal/local_server.py:451:    uvicorn.run(
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:261:            from veena3modal.api.fastapi_app import set_model_version
/home/ubuntu/veenaModal/README.md:19:modal serve veena3modal/app.py
/home/ubuntu/veenaModal/README.md:21:modal deploy veena3modal/app.py
/home/ubuntu/veenaModal/README.md:73:- **Dev**: `AUTH_BYPASS_MODE=true` bypasses key validation (enabled by default in `veena3modal/app.py` image env for testing).
/home/ubuntu/veenaModal/README.md:88:- Model weights are expected to live on a Modal Volume (see `veena3modal/app.py`).
/home/ubuntu/veenaModal/veena3modal/api/websocket_handler.py:465:    Call this from create_app() to enable WebSocket support.
/home/ubuntu/veenaModal/veena3modal/tests/integration/test_tts_generate.py:55:    from veena3modal.api.fastapi_app import create_app
/home/ubuntu/veenaModal/veena3modal/tests/integration/test_tts_generate.py:65:    app = create_app()
/home/ubuntu/veenaModal/veena3modal/tests/unit/test_fastapi_app.py:14:        """Ensure create_app() returns a FastAPI application."""
/home/ubuntu/veenaModal/veena3modal/tests/unit/test_fastapi_app.py:15:        from veena3modal.api.fastapi_app import create_app
/home/ubuntu/veenaModal/veena3modal/tests/unit/test_fastapi_app.py:17:        app = create_app()
/home/ubuntu/veenaModal/veena3modal/tests/unit/test_fastapi_app.py:26:        from veena3modal.api.fastapi_app import create_app
/home/ubuntu/veenaModal/veena3modal/tests/unit/test_fastapi_app.py:28:        app = create_app()
/home/ubuntu/veenaModal/veena3modal/tests/unit/test_fastapi_app.py:42:        from veena3modal.api.fastapi_app import create_app
/home/ubuntu/veenaModal/veena3modal/tests/unit/test_fastapi_app.py:44:        app = create_app()
/home/ubuntu/veenaModal/veena3modal/api/fastapi_app.py:89:def create_app():
/home/ubuntu/veenaModal/veena3modal/app.py:12:    modal deploy veena3modal/app.py
/home/ubuntu/veenaModal/veena3modal/app.py:15:    modal serve veena3modal/app.py
/home/ubuntu/veenaModal/veena3modal/app.py:18:    modal run veena3modal/app.py::tts_api
/home/ubuntu/veenaModal/veena3modal/app.py:47:        "uvicorn[standard]>=0.23.0",
/home/ubuntu/veenaModal/veena3modal/app.py:246:        from veena3modal.api.fastapi_app import create_app
/home/ubuntu/veenaModal/veena3modal/app.py:247:        return create_app()
/home/ubuntu/veenaModal/veena3modal/app.py:274:    from veena3modal.api.fastapi_app import create_app
/home/ubuntu/veenaModal/veena3modal/app.py:275:    return create_app()
/home/ubuntu/veenaModal/veena3modal/app.py:304:    Run with: modal run veena3modal/app.py::debug_imports
/home/ubuntu/veenaModal/veena3modal/app.py:365:    Usage: modal run veena3modal/app.py


rg -n "websocket|websocket|/v1/tts/ws|ws://|async for msg|create_complete_message" /home/ubuntu/veenaModal/scripts


Output:
Chunk ID: 67d98a
Wall time: 0.0511 seconds
Process exited with code 0
Original token count: 225
Output:
/home/ubuntu/veenaModal/scripts/validate_modal_endpoint.py:218:def test_websocket():
/home/ubuntu/veenaModal/scripts/validate_modal_endpoint.py:223:        import websockets.sync.client as ws_client
/home/ubuntu/veenaModal/scripts/validate_modal_endpoint.py:225:        print("⚠ websockets not installed, skipping WebSocket test")
/home/ubuntu/veenaModal/scripts/validate_modal_endpoint.py:228:    ws_url = ENDPOINT_URL.replace("https://", "wss://").replace("http://", "ws://")
/home/ubuntu/veenaModal/scripts/validate_modal_endpoint.py:229:    ws_url = f"{ws_url}/v1/tts/ws"
/home/ubuntu/veenaModal/scripts/validate_modal_endpoint.py:323:            results["websocket"] = test_websocket()
/home/ubuntu/veenaModal/scripts/validate_modal_endpoint.py:326:            results["websocket"] = False
/home/ubuntu/veenaModal/scripts/validate_modal_endpoint.py:331:        results["websocket"] = None


sed -n '1,260p' /home/ubuntu/veenaModal/veena3modal/tests/unit/test_fastapi_app.py


Output:
Chunk ID: cfc1d6
Wall time: 0.0517 seconds
Process exited with code 0
Original token count: 1093
Output:
"""
Unit tests for veena3modal FastAPI app.

M1: Tests for /v1/tts/health endpoint and app creation.
"""

import pytest

class TestAppFactory:
    """Test app factory and basic structure."""

    def test_create_app_returns_fastapi_instance(self):
        """Ensure create_app() returns a FastAPI application."""
        from veena3modal.api.fastapi_app import create_app

        app = create_app()

        # Check it's a FastAPI app (has routes, title, version)
        assert hasattr(app, "routes")
        assert app.title == "Veena3 TTS (Modal)"
        assert app.version == "0.1.0"

    def test_create_app_has_health_route(self):
        """Ensure /v1/tts/health route exists."""
        from veena3modal.api.fastapi_app import create_app

        app = create_app()

        # Find health route
        route_paths = [r.path for r in app.routes if hasattr(r, "path")]
        assert "/v1/tts/health" in route_paths

class TestHealthEndpoint:
    """Test /v1/tts/health endpoint responses."""

    @pytest.fixture
    def client(self):
        """Create a test client for the FastAPI app."""
        from fastapi.testclient import TestClient
        from veena3modal.api.fastapi_app import create_app

        app = create_app()
        return TestClient(app)

    def test_health_returns_200(self, client):
        """Health endpoint should return 200 OK."""
        response = client.get("/v1/tts/health")
        assert response.status_code == 200

    def test_health_returns_expected_fields(self, client):
        """Health endpoint should return all required fields."""
        response = client.get("/v1/tts/health")
        data = response.json()

        # Required fields
        assert "status" in data
        assert "model_loaded" in data
        assert "model_version" in data
        assert "uptime_seconds" in data
        assert "gpu_available" in data
        assert "app_version" in data

    def test_health_status_is_valid(self, client):
        """Health status should be one of: healthy, degraded, unhealthy."""
        response = client.get("/v1/tts/health")
        data = response.json()

        assert data["status"] in ["healthy", "degraded", "unhealthy"]

    def test_health_model_not_loaded_initially(self, client):
        """Model should not be loaded by default (no runtime wired yet)."""
        response = client.get("/v1/tts/health")
        data = response.json()

        # Model not loaded until M3 wires the runtime
        assert data["model_loaded"] is False
        assert data["model_version"] == "not_loaded"

    def test_health_uptime_is_positive(self, client):
        """Uptime should be a non-negative number."""
        response = client.get("/v1/tts/health")
        data = response.json()

        assert isinstance(data["uptime_seconds"], (int, float))
        assert data["uptime_seconds"] >= 0

    def test_health_includes_version_headers(self, client):
        """Health response should include version headers."""
        response = client.get("/v1/tts/health")

        assert "X-Model-Version" in response.headers
        assert "X-App-Version" in response.headers
        assert response.headers["X-App-Version"] == "0.1.0"

    def test_health_content_type_is_json(self, client):
        """Health response should be JSON."""
        response = client.get("/v1/tts/health")

        assert "application/json" in response.headers.get("content-type", "")

class TestModelVersionHelpers:
    """Test model version getter/setter helpers."""

    def test_get_model_version_default(self):
        """get_model_version returns 'not_loaded' when not set."""
        from veena3modal.api import fastapi_app

        # Reset state
        fastapi_app._MODEL_VERSION = None

        assert fastapi_app.get_model_version() == "not_loaded"

    def test_set_and_get_model_version(self):
        """set_model_version updates the global, get_model_version retrieves it."""
        from veena3modal.api import fastapi_app

        # Reset and set
        fastapi_app._MODEL_VERSION = None
        fastapi_app.set_model_version("spark-tts-v1.2.3")

        assert fastapi_app.get_model_version() == "spark-tts-v1.2.3"

        # Clean up
        fastapi_app._MODEL_VERSION = None


rg -n "/v1/tts/ws|handle_websocket_tts|complete|progress|audio_chunk|metrics" /home/ubuntu/veenaModal/veena3modal/tests/integration -g "*.py"


Output:
Chunk ID: 9189a1
Wall time: 0.0504 seconds
Process exited with code 0
Original token count: 32
Output:
/home/ubuntu/veenaModal/veena3modal/tests/integration/test_sentence_store.py:123:        # Task should complete without error


I’m keeping modifications server-side plus benchmarking side. I’ll now patch three points in one pass: FastAPI request ID/timing propagation, WebSocket completion payload enrichment, and a dedicated local stress benchmark script that records per-request GPU/LLM timing and concurrency signals from those headers/events.


rg -n "AGENTS\.md|skills" /home/ubuntu/veenaModal -g "AGENTS.md"


Output:
Chunk ID: 80116d
Wall time: 0.0511 seconds
Process exited with code 1
Original token count: 0
Output:


rg -n "def generate_speech_indic|return_timing" /home/ubuntu/veenaModal/veena3modal/core/pipeline.py


Output:
Chunk ID: 04bf98
Wall time: 0.0511 seconds
Process exited with code 0
Original token count: 178
Output:
78:        return_timing: bool = False,
95:            return_timing: If true, returns (audio_bytes, metrics_dict)
99:            If return_timing is true, returns tuple(audio_bytes, metrics).
121:        emit_timing = return_timing or _detailed_timing_enabled()
196:            if return_timing:
239:            if return_timing:
256:            if return_timing:
277:            if return_timing:
317:        if return_timing:
321:    async def generate_speech_indic(
331:        return_timing: bool = False,
352:            return_timing: If true, returns (audio_bytes, metrics_dict)
356:            If return_timing is true, returns tuple(audio_bytes, metrics).
368:            return_timing=return_timing,


sed -n '320,420p' /home/ubuntu/veenaModal/veena3modal/core/pipeline.py


Output:
Chunk ID: 3db6d2
Wall time: 0.0504 seconds
Process exited with code 0
Original token count: 779
Output:

    async def generate_speech_indic(
        self,
        speaker: str,
        text: str,
        temperature: float = DEFAULT_TEMPERATURE,
        top_k: int = DEFAULT_TOP_K,
        top_p: float = DEFAULT_TOP_P,
        max_tokens: int = DEFAULT_MAX_TOKENS,
        repetition_penalty: float = 1.05,
        seed: Optional[int] = None,
        return_timing: bool = False,
        request_id: Optional[str] = None,
    ):
        """
        Generate speech audio for Spark TTS (speaker-based).

        This method is kept for backward compatibility with existing API.
        It delegates to generate_speech() with the same implementation.

        Args:
            speaker: Speaker name (one of 12 predefined speakers)
            text: Text to synthesize with inline emotion tags
                Examples:
                - "Hello! Welcome."
                - "Hello <laugh> this is fun!" (will be normalized)
                - "नमस्ते! [excited] आज का दिन बहुत अच्छा है।"
            temperature: Sampling temperature
            top_k: Top-k sampling
            top_p: Nucleus sampling
            max_tokens: Max BiCodec tokens to generate
            seed: Random seed for reproducibility
            return_timing: If true, returns (audio_bytes, metrics_dict)

        Returns:
            Audio bytes (int16 PCM WAV, 16kHz mono) or None if failed.
            If return_timing is true, returns tuple(audio_bytes, metrics).
        """
        # Delegate to generate_speech() with the same implementation for Spark TTS
        return await self.generate_speech(
            speaker=speaker,
            text=text,
            temperature=temperature,
            top_k=top_k,
            top_p=top_p,
            max_tokens=max_tokens,
            repetition_penalty=repetition_penalty,
            seed=seed,
            return_timing=return_timing,
            request_id=request_id,
        )

    def _extract_bicodec_tokens(self, generated_text: str) -> Tuple[List[int], List[int]]:
        """
        Extract BiCodec semantic and global tokens from generated text using regex.

        Spark TTS generates tokens in the format:
        - <|bicodec_semantic_{id}|>
        - <|bicodec_global_{id}|>

        Args:
            generated_text: Generated text containing BiCodec token markers

        Returns:
            Tuple of (semantic_ids, global_ids)
            Returns ([], []) if no tokens found
        """
        # Extract semantic tokens
        semantic_matches = re.findall(r"<\|bicodec_semantic_(\d+)\|>", generated_text)
        semantic_ids = [int(t) for t in semantic_matches]

        # Extract global tokens
        global_matches = re.findall(r"<\|bicodec_global_(\d+)\|>", generated_text)
        global_ids = [int(t) for t in global_matches]

        # Only log if there's an issue
        if not semantic_ids and not global_ids:
            logger.error(f"❌ No BiCodec tokens found! Generated text (first 1000 chars):\n{generated_text[:1000]}")

        return semantic_ids, global_ids


sed -n '70,170p' /home/ubuntu/veenaModal/veena3modal/core/pipeline.py


Output:
Chunk ID: 31f110
Wall time: 0.0506 seconds
Process exited with code 0
Original token count: 982
Output:
        speaker: str,
        text: str,
        temperature: float = DEFAULT_TEMPERATURE,
        top_k: int = DEFAULT_TOP_K,
        top_p: float = DEFAULT_TOP_P,
        max_tokens: int = DEFAULT_MAX_TOKENS,
        repetition_penalty: float = 1.05,
        seed: Optional[int] = None,
        return_timing: bool = False,
        request_id: Optional[str] = None,
    ) -> Optional[bytes] | tuple[Optional[bytes], Dict[str, Any]]:
        """
        Generate speech audio (non-streaming) using Spark TTS.

        NOTE: This method signature changed from description-based to speaker-based
        to align with Spark TTS architecture.

        Args:
            speaker: Speaker name (one of 12 predefined speakers)
            text: Text to synthesize (with optional [emotion] tags)
            temperature: Sampling temperature (default: 0.8 for Spark TTS)
            top_k: Top-k sampling (default: 50)
            top_p: Nucleus sampling (default: 1.0)
            max_tokens: Max BiCodec tokens to generate (default: 2048)
            seed: Random seed for reproducibility
            return_timing: If true, returns (audio_bytes, metrics_dict)

        Returns:
            Audio bytes (int16 PCM WAV, 16kHz mono) or None if failed.
            If return_timing is true, returns tuple(audio_bytes, metrics).
        """
        # Build prompt using Spark TTS format
        prompt = self.prompt_builder.build_prefix(speaker, text)

        # Configure sampling for Spark TTS
        sampling_params = SamplingParams(
            temperature=temperature,
            top_k=top_k,
            top_p=top_p,
            max_tokens=max_tokens,
            repetition_penalty=repetition_penalty,
            stop=TRAINING_STOP_TOKEN_IDS,
            skip_special_tokens=False,
            seed=seed,
        )

        # Use vLLM engine to generate
        if request_id is None:
            import uuid
            request_id = f"req-{uuid.uuid4().hex[:12]}"

        emit_timing = return_timing or _detailed_timing_enabled()
        results_generator = self.model.engine.generate(
            prompt=prompt,
            sampling_params=sampling_params,
            request_id=request_id,
        )

        # Collect all outputs and track generation timing
        final_output = None
        token_batches: List[int] = []
        batch_wall_ms: List[float] = []
        batch_gpu_ms: List[float] = []
        total_token_ids = 0
        first_token_ms: Optional[float] = None
        t_start = asyncio.get_event_loop().time()

        iterator = results_generator.__aiter__()
        while True:
            wall_start = asyncio.get_event_loop().time()
            start_event = None
            end_event = None
            if torch.cuda.is_available():
                torch.cuda.synchronize()
                start_event = torch.cuda.Event(enable_timing=True)
                end_event = torch.cuda.Event(enable_timing=True)
                start_event.record()

            try:
                request_output = await iterator.__anext__()
            except StopAsyncIteration:
                break

            wall_end = asyncio.get_event_loop().time()
            batch_ms = (wall_end - wall_start) * 1000
            batch_wall_ms.append(batch_ms)

            batch_gpu_ms_value: Optional[float] = None
            if start_event is not None and end_event is not None:
                end_event.record()
                torch.cuda.synchronize()
                batch_gpu_ms_value = start_event.elapsed_time(end_event)
            if batch_gpu_ms_value is not None:
                batch_gpu_ms.append(batch_gpu_ms_value)

            total_generated = len(request_output.outputs[0].token_ids)
            new_tokens = max(0, total_generated - total_token_ids)
            total_token_ids = total_generated
            token_batches.append(new_tokens)

            if first_token_ms is None:


sed -n '170,285p' /home/ubuntu/veenaModal/veena3modal/core/pipeline.py


Output:
Chunk ID: dfd7ee
Wall time: 0.0504 seconds
Process exited with code 0
Original token count: 1151
Output:
            if first_token_ms is None:
                first_token_ms = (wall_end - t_start) * 1000

            if emit_timing:
                logger.info(
                    "tts_llm_batch",
                    extra={
                        "speaker": speaker,
                        "request_id": request_id,
                        "batch_index": len(token_batches),
                        "batch_tokens": new_tokens,
                        "llm_batch_wall_ms": round(batch_ms, 3),
                        "llm_batch_gpu_ms": round(batch_gpu_ms_value, 3)
                        if batch_gpu_ms_value is not None
                        else None,
                    },
                )
            final_output = request_output

        if final_output is None:
            if emit_timing:
                logger.warning(
                    "tts_llm_no_output",
                    extra={"speaker": speaker, "request_id": request_id},
                )
            print(f"❌ Generation failed: no output")
            if return_timing:
                return None, {
                    "request_id": request_id,
                    "ttfb_ms": 0,
                    "token_total": 0,
                    "batch_count": 0,
                }
            return None

        # Extract generated text (contains BiCodec tokens)
        parse_start = asyncio.get_event_loop().time()
        generated_text = final_output.outputs[0].text
        generated_ids = final_output.outputs[0].token_ids

        # Extract BiCodec tokens using regex
        semantic_ids, global_ids = self._extract_bicodec_tokens(generated_text)

        parse_ms = (asyncio.get_event_loop().time() - parse_start) * 1000

        if not semantic_ids or not global_ids:
            if emit_timing:
                logger.error(
                    "tts_bicodec_parse_failed",
                    extra={
                        "speaker": speaker,
                        "request_id": request_id,
                        "semantic_count": len(semantic_ids),
                        "global_count": len(global_ids),
                        "token_total": len(generated_ids),
                    },
                )
            logger.error(
                "❌ No BiCodec tokens found in output",
                extra={
                    "speaker": speaker,
                    "text_length": len(text),
                    "text_preview": text[:100],
                    "generated_tokens": len(generated_ids),
                    "semantic_tokens": len(semantic_ids),
                    "global_tokens": len(global_ids),
                    "generated_text_preview": generated_text[:500],
                }
            )
            if return_timing:
                return None, {
                    "request_id": request_id,
                    "ttfb_ms": int(first_token_ms or 0),
                    "token_total": len(generated_ids),
                    "batch_count": len(token_batches),
                }
            return None

        # Validate tokens
        if not self.bicodec_decoder.validate_tokens(semantic_ids, global_ids):
            logger.error("❌ BiCodec token validation failed", extra={
                "speaker": speaker,
                "text_length": len(text),
                "semantic_tokens": len(semantic_ids),
                "global_tokens": len(global_ids),
            })
            if return_timing:
                return None, {
                    "request_id": request_id,
                    "ttfb_ms": int(first_token_ms or 0),
                    "token_total": len(generated_ids),
                    "batch_count": len(token_batches),
                }
            return None

        # Decode to audio
        decode_start = asyncio.get_event_loop().time()
        audio_bytes = self.bicodec_decoder.decode_to_bytes(semantic_ids, global_ids)
        decode_ms = (asyncio.get_event_loop().time() - decode_start) * 1000

        if audio_bytes is None:
            if emit_timing:
                logger.error("tts_decode_failed", extra={"speaker": speaker, "request_id": request_id})
            logger.error("❌ BiCodec decode failed", extra={
                "speaker": speaker,
                "text_length": len(text),
            })
            if return_timing:
                return None, {
                    "request_id": request_id,
                    "ttfb_ms": int(first_token_ms or 0),
                    "token_total": len(generated_ids),
                    "batch_count": len(token_batches),
                }
            return None


sed -n '285,360p' /home/ubuntu/veenaModal/veena3modal/core/pipeline.py


Output:
Chunk ID: 7622f8
Wall time: 0.0504 seconds
Process exited with code 0
Original token count: 813
Output:

        total_ms = (asyncio.get_event_loop().time() - t_start) * 1000

        timing_metrics: Dict[str, Any] = {
            "request_id": request_id,
            "speaker": speaker,
            "text_length": len(text),
            "batch_count": len(token_batches),
            "token_total": len(generated_ids),
            "token_batches": token_batches,
            "ttfb_ms": int(first_token_ms or 0),
            "llm_wall_ms": int(sum(batch_wall_ms)),
            "llm_avg_batch_ms": int(mean(batch_wall_ms)) if batch_wall_ms else 0,
            "llm_gpu_batch_ms_total": int(sum(batch_gpu_ms)),
            "llm_gpu_batch_ms_min": int(min(batch_gpu_ms)) if batch_gpu_ms else 0,
            "llm_gpu_batch_ms_max": int(max(batch_gpu_ms)) if batch_gpu_ms else 0,
            "llm_gpu_batch_ms_p50": int(median(batch_gpu_ms)) if batch_gpu_ms else 0,
            "decode_cpu_ms": int(decode_ms),
            "parse_ms": int(parse_ms),
            "total_ms": int(total_ms),
        }
        if token_batches:
            timing_metrics["tokens_per_batch_min"] = min(token_batches)
            timing_metrics["tokens_per_batch_max"] = max(token_batches)
            timing_metrics["tokens_per_batch_p50"] = int(median(token_batches))

        if emit_timing:
            logger.info("tts_llm_complete", extra=timing_metrics)

        from veena3modal.audio.utils import add_wav_header
        wav_bytes = add_wav_header(audio_bytes, sample_rate=AUDIO_SAMPLE_RATE)

        if return_timing:
            return wav_bytes, timing_metrics
        return wav_bytes

    async def generate_speech_indic(
        self,
        speaker: str,
        text: str,
        temperature: float = DEFAULT_TEMPERATURE,
        top_k: int = DEFAULT_TOP_K,
        top_p: float = DEFAULT_TOP_P,
        max_tokens: int = DEFAULT_MAX_TOKENS,
        repetition_penalty: float = 1.05,
        seed: Optional[int] = None,
        return_timing: bool = False,
        request_id: Optional[str] = None,
    ):
        """
        Generate speech audio for Spark TTS (speaker-based).

        This method is kept for backward compatibility with existing API.
        It delegates to generate_speech() with the same implementation.

        Args:
            speaker: Speaker name (one of 12 predefined speakers)
            text: Text to synthesize with inline emotion tags
                Examples:
                - "Hello! Welcome."
                - "Hello <laugh> this is fun!" (will be normalized)
                - "नमस्ते! [excited] आज का दिन बहुत अच्छा है।"
            temperature: Sampling temperature
            top_k: Top-k sampling
            top_p: Nucleus sampling
            max_tokens: Max BiCodec tokens to generate
            seed: Random seed for reproducibility
            return_timing: If true, returns (audio_bytes, metrics_dict)

        Returns:
            Audio bytes (int16 PCM WAV, 16kHz mono) or None if failed.
            If return_timing is true, returns tuple(audio_bytes, metrics).
        """
        # Delegate to generate_speech() with the same implementation for Spark TTS
        return await self.generate_speech(
            speaker=speaker,


Summary: auto


sed -n '460,560p' /home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py


Output:
Chunk ID: ea633d
Wall time: 0.0512 seconds
Process exited with code 0
Original token count: 1120
Output:

        generation_time = time.time() - start_time
        metrics["generation_ms"] = int(generation_time * 1000)
        metrics["ttfb_ms"] = metrics["generation_ms"]  # Non-streaming: TTFB = total time

        if pipeline_timings:
            metrics.update({
                "llm_token_total": pipeline_timings.get("token_total", 0),
                "llm_batch_count": pipeline_timings.get("batch_count", 0),
                "llm_batch_gpu_ms_total": pipeline_timings.get("llm_gpu_batch_ms_total", 0),
                "llm_batch_gpu_ms_min": pipeline_timings.get("llm_gpu_batch_ms_min", 0),
                "llm_batch_gpu_ms_max": pipeline_timings.get("llm_gpu_batch_ms_max", 0),
                "llm_batch_gpu_ms_p50": pipeline_timings.get("llm_gpu_batch_ms_p50", 0),
                "llm_decode_cpu_ms": pipeline_timings.get("decode_cpu_ms", 0),
                "llm_parse_ms": pipeline_timings.get("parse_ms", 0),
                "llm_wall_ms": pipeline_timings.get("llm_wall_ms", metrics["generation_ms"]),
                "ttfb_ms": int(pipeline_timings.get("ttfb_ms", metrics["ttfb_ms"])),
                "tokens_per_batch_min": pipeline_timings.get("tokens_per_batch_min"),
                "tokens_per_batch_max": pipeline_timings.get("tokens_per_batch_max"),
                "tokens_per_batch_p50": pipeline_timings.get("tokens_per_batch_p50"),
            })

        if audio_bytes:
            # Apply super-resolution if requested and available
            sample_rate = 16000
            logger.info(f"SR check: output_sample_rate={output_sample_rate!r}, "
                       f"sr_service={runtime.sr_service is not None}, "
                       f"is_loaded={runtime.sr_service.is_loaded if runtime.sr_service else False}")

            if output_sample_rate == "48khz" and runtime.sr_service and runtime.sr_service.is_loaded:
                try:
                    sr_start = time.time()
                    logger.info(f"Applying super-resolution to {len(audio_bytes)} bytes...")
                    audio_bytes = _apply_super_resolution(audio_bytes, runtime.sr_service)
                    sr_time = (time.time() - sr_start) * 1000
                    metrics["sr_ms"] = int(sr_time)
                    metrics["sr_applied"] = True
                    sample_rate = 48000
                    logger.info(f"✅ Super-resolution applied in {sr_time:.1f}ms, output={len(audio_bytes)} bytes")
                except Exception as e:
                    logger.warning(f"Super-resolution failed, returning 16kHz: {e}")
                    import traceback
                    traceback.print_exc()

            metrics["output_sample_rate"] = sample_rate

            # Calculate audio duration
            audio_duration = (len(audio_bytes) - 44) / (sample_rate * 2)  # -44 for WAV header
            metrics["audio_duration_seconds"] = max(0.0, audio_duration)

        return audio_bytes, metrics

    except Exception as e:
        logger.error(f"Speech generation failed: {e}")
        raise
    finally:
        active_requests = _with_inflight_delta(-1)
        if _timing_enabled():
            metrics["request_inflight"] = active_requests
            metrics["active_requests"] = active_requests
            logger.info(
                "tts_runtime_nonstream_complete",
                extra=metrics,
            )
        if _timing_enabled():
            logger.info(
                "tts_runtime_nonstream_inflight",
                extra={
                    "request_id": request_id,
                    "active_requests": active_requests,
                },
            )

def _apply_super_resolution(audio_bytes: bytes, sr_service) -> bytes:
    """
    Apply super-resolution to audio bytes.

    Args:
        audio_bytes: WAV audio at 16kHz
        sr_service: SuperResolutionService instance

    Returns:
        WAV audio at 48kHz
    """
    import numpy as np
    import struct
    import torch

    # Parse WAV header and extract PCM data
    if len(audio_bytes) < 44:
        raise ValueError("Invalid WAV data")

    # Extract PCM data (skip 44-byte WAV header)
    pcm_data = np.frombuffer(audio_bytes[44:], dtype=np.int16)

    # Convert to float32 for SR model
    audio_float = pcm_data.astype(np.float32) / 32768.0

    # Convert numpy to torch tensor for process_chunk
    audio_tensor = torch.from_numpy(audio_float)


apply_patch <<'PATCH'
*** Begin Patch
*** Update File: /home/ubuntu/veenaModal/veena3modal/api/fastapi_app.py
@@
 import time
 import uuid
-from typing import Optional
+import json
+from typing import Dict, Optional
@@
 def _get_sentence_store():
@@
     return _sentence_store
+
+
+def _build_perf_headers(metrics: Dict[str, object]) -> Dict[str, str]:
+    """Build compact timing headers for benchmark tooling."""
+    perf: Dict[str, str] = {}
+    if not metrics:
+        return perf
+
+    def _pick(*keys: str):
+        for key in keys:
+            if key in metrics and metrics[key] is not None:
+                return metrics[key]
+        return None
+
+    def _add(name: str, value: object):
+        if value is None:
+            return
+        if isinstance(value, bool):
+            perf[name] = str(value).lower()
+        else:
+            perf[name] = str(value)
+
+    llm_token_total = _pick("llm_token_total", "token_total")
+    llm_batch_count = _pick("llm_batch_count", "batch_count")
+    generation_ms = _pick("generation_ms") or _pick("total_ms")
+    llm_batch_wall_total = _pick("llm_batch_wall_ms_total", "llm_wall_ms", "total_ms")
+    llm_batch_gpu_total = _pick("llm_batch_gpu_ms_total", "llm_gpu_batch_ms_total")
+    llm_decode_wall_total = _pick("llm_decode_wall_ms_total")
+    llm_decode_gpu_total = _pick("llm_decode_gpu_ms_total")
+    llm_parse_ms = _pick("llm_parse_ms")
+    decode_cpu_ms = _pick("llm_decode_cpu_ms")
+    parse_ms = _pick("parse_ms")
+
+    _add("X-LLM-Token-Total", llm_token_total)
+    _add("X-LLM-Batch-Count", llm_batch_count)
+    _add("X-LLM-Batch-Wall-Ms", llm_batch_wall_total)
+    _add("X-LLM-Batch-GPU-Ms", llm_batch_gpu_total)
+    _add("X-LLM-Batch-Wall-Ms-Min", _pick("llm_batch_wall_ms_min"))
+    _add("X-LLM-Batch-Wall-Ms-Max", _pick("llm_batch_wall_ms_max"))
+    _add("X-LLM-Batch-Wall-Ms-P50", _pick("llm_batch_wall_ms_p50"))
+    _add("X-LLM-Batch-GPU-Ms-Min", _pick("llm_batch_gpu_ms_min"))
+    _add("X-LLM-Batch-GPU-Ms-Max", _pick("llm_batch_gpu_ms_max"))
+    _add("X-LLM-Batch-GPU-Ms-P50", _pick("llm_batch_gpu_ms_p50"))
+    _add("X-LLM-Decode-Wall-Ms", llm_decode_wall_total)
+    _add("X-LLM-Decode-GPU-Ms", llm_decode_gpu_total)
+    _add("X-LLM-Decode-Wall-Ms-Min", _pick("llm_decode_wall_ms_min"))
+    _add("X-LLM-Decode-Wall-Ms-Max", _pick("llm_decode_wall_ms_max"))
+    _add("X-LLM-Decode-Wall-Ms-P50", _pick("llm_decode_wall_ms_p50"))
+    _add("X-LLM-Decode-GPU-Ms-Min", _pick("llm_decode_gpu_ms_min"))
+    _add("X-LLM-Decode-GPU-Ms-Max", _pick("llm_decode_gpu_ms_max"))
+    _add("X-LLM-Decode-GPU-Ms-P50", _pick("llm_decode_gpu_ms_p50"))
+    _add("X-LLM-Decode-Calls", _pick("decode_calls"))
+    _add("X-LLM-Decode-CPU-Ms", decode_cpu_ms)
+    _add("X-LLM-Parse-Ms", llm_parse_ms or parse_ms)
+    _add("X-LLM-Parse-Avg-Ms", _pick("llm_parse_ms_avg"))
+    _add("X-LLM-Tokens-Per-Batch-Min", _pick("tokens_per_batch_min"))
+    _add("X-LLM-Tokens-Per-Batch-Max", _pick("tokens_per_batch_max"))
+    _add("X-LLM-Tokens-Per-Batch-P50", _pick("tokens_per_batch_p50"))
+    _add("X-Generation-Ms", generation_ms)
+    _add("X-LLM-Generation-Ms", _pick("llm_wall_ms", generation_ms))
+    _add("X-Request-Inflight", _pick("request_inflight", "active_requests"))
+    _add("X-LLM-Text-Chunked", _pick("text_chunked"))
+    _add("X-LLM-Chunks-Processed", _pick("chunks_processed"))
+    _add("X-LLM-SR-Applied", _pick("sr_applied"))
+
+    if llm_token_total and generation_ms:
+        try:
+            perf["X-LLM-Time-Per-Token-Ms"] = f"{float(generation_ms) / float(llm_token_total):.3f}"
+        except Exception:
+            pass
+
+    if llm_batch_count:
+        try:
+            perf["X-LLM-Time-Per-Batch-Wall-Ms"] = f"{float(generation_ms or 0) / float(llm_batch_count):.3f}"
+        except Exception:
+            pass
+        try:
+            perf["X-LLM-Time-Per-Batch-GPU-Ms"] = f"{float(llm_batch_gpu_total or 0) / float(llm_batch_count):.3f}"
+        except Exception:
+            pass
+
+    # Include compact JSON snapshot for downstream benchmark tooling.
+    detail_keys = [
+        "request_id",
+        "speaker",
+        "text_length",
+        "llm_token_total",
+        "llm_batch_count",
+        "llm_batch_wall_ms_total",
+        "llm_batch_gpu_ms_total",
+        "llm_batch_wall_ms_min",
+        "llm_batch_wall_ms_max",
+        "llm_batch_wall_ms_p50",
+        "llm_batch_gpu_ms_min",
+        "llm_batch_gpu_ms_max",
+        "llm_batch_gpu_ms_p50",
+        "llm_decode_cpu_ms",
+        "llm_decode_wall_ms_total",
+        "llm_decode_gpu_ms_total",
+        "llm_decode_wall_ms_min",
+        "llm_decode_wall_ms_max",
+        "llm_decode_wall_ms_p50",
+        "llm_decode_gpu_ms_min",
+        "llm_decode_gpu_ms_max",
+        "llm_decode_gpu_ms_p50",
+        "decode_calls",
+        "decode_cpu_ms",
+        "parse_ms",
+        "llm_parse_ms",
+        "llm_parse_ms_avg",
+        "llm_batch_tokens",
+        "tokens_per_batch_min",
+        "tokens_per_batch_max",
+        "tokens_per_batch_p50",
+        "generation_ms",
+        "audio_duration_seconds",
+        "request_inflight",
+        "active_requests",
+        "chunks_processed",
+        "text_chunked",
+        "sr_applied",
+    ]
+    perf["X-Perf-Details"] = json.dumps(
+        {k: metrics[k] for k in detail_keys if k in metrics},
+        separators=(",", ":"),
+    )
+
+    return perf
@@
                 if req.chunking:
                 audio_bytes, metrics = await tts_runtime.generate_speech_chunked(
@@
                     top_p=req.top_p,
                     max_tokens=req.max_tokens,
                     repetition_penalty=req.repetition_penalty,
                     seed=req.seed,
                     sample_rate=req.sample_rate,
                     output_sample_rate=output_sr,
+                    request_id=request_id,
                 )
             else:
                 audio_bytes, metrics = await tts_runtime.generate_speech(
@@
                     top_k=req.top_k,
                     top_p=req.top_p,
                     max_tokens=req.max_tokens,
                     repetition_penalty=req.repetition_penalty,
                     seed=req.seed,
                     output_sample_rate=output_sr,
+                    request_id=request_id,
                 )
@@
             headers = {
                 "X-Request-ID": request_id,
                 "X-Model-Version": model_version,
                 "X-Format": "wav",
                 "X-Sample-Rate": str(output_sample_rate),
                 "X-Stream": "false",
                 "X-Audio-Bytes": str(len(audio_bytes)),
                 "X-Audio-Seconds": f"{audio_duration:.2f}",
                 "X-TTFB-ms": str(metrics.get("ttfb_ms", 0)),
                 "X-RTF": f"{rtf:.3f}",
                 "X-Text-Chunked": str(metrics.get("text_chunked", False)).lower(),
                 "X-SR-Applied": str(metrics.get("sr_applied", False)).lower(),
             }
+            headers.update(_build_perf_headers(metrics))
@@
                 async for audio_chunk, metrics in tts_runtime.generate_speech_streaming(
                     text=text,
                     speaker=speaker,
                     temperature=req.temperature,
                     top_k=req.top_k,
@@
                     max_tokens=req.max_tokens,
                     repetition_penalty=req.repetition_penalty,
                     seed=req.seed,
                     enable_chunking=req.chunking,
+                    request_id=request_id,
                 ):
                     final_metrics = metrics
*** End Patch
PATCH


Warning: apply_patch was requested via exec_command. Use the apply_patch tool instead of exec_command.


Output: {"output":"Success. Updated the following files:\nM /home/ubuntu/veenaModal/veena3modal/api/fastapi_app.py\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


apply_patch <<'PATCH'
*** Begin Patch
*** Update File: /home/ubuntu/veenaModal/veena3modal/api/websocket_handler.py
@@
 def create_complete_message(
@@
     return json.dumps({
         "event": WSMessageType.COMPLETE.value,
         "request_id": request_id,
         "metrics": metrics,
     })
+
+
+_METRIC_FORWARD_KEYS = {
+    "ttfb_ms",
+    "generation_ms",
+    "audio_duration_seconds",
+    "audio_chunks",
+    "total_bytes",
+    "chunks_sent",
+    "request_inflight",
+    "active_requests",
+    "llm_token_total",
+    "llm_batch_count",
+    "llm_batch_wall_ms_total",
+    "llm_batch_wall_ms_min",
+    "llm_batch_wall_ms_max",
+    "llm_batch_wall_ms_p50",
+    "llm_batch_gpu_ms_total",
+    "llm_batch_gpu_ms_min",
+    "llm_batch_gpu_ms_max",
+    "llm_batch_gpu_ms_p50",
+    "llm_decode_cpu_ms",
+    "llm_decode_wall_ms_total",
+    "llm_decode_wall_ms_min",
+    "llm_decode_wall_ms_max",
+    "llm_decode_wall_ms_p50",
+    "llm_decode_gpu_ms_total",
+    "llm_decode_gpu_ms_min",
+    "llm_decode_gpu_ms_max",
+    "llm_decode_gpu_ms_p50",
+    "decode_calls",
+    "parse_ms",
+    "llm_parse_ms",
+    "llm_parse_ms_avg",
+    "tokens_per_batch_min",
+    "tokens_per_batch_max",
+    "tokens_per_batch_p50",
+    "llm_batch_tokens",
+    "generated_tokens",
+    "semantic_tokens",
+    "global_tokens",
+    "text_chunked",
+    "concurrency_mode",
+    "captured_global_count",
+    "chunks_processed",
+}
@@
             async for audio_chunk, metrics in tts_runtime.generate_speech_streaming(
                 text=text,
                 speaker=speaker,
                 temperature=req.temperature,
                 top_k=req.top_k,
@@
                 max_tokens=req.max_tokens,
                 repetition_penalty=req.repetition_penalty,
                 seed=req.seed,
                 enable_chunking=req.chunking,
+                request_id=request_id,
             ):
@@
         completion_metrics = {
             "request_id": request_id,
             "chunks_sent": chunks_sent,
             "total_bytes": total_bytes,
             "audio_duration_seconds": round(audio_duration, 2),
@@
             "rtf": round(rtf, 3),
             "cancelled": cancelled,
         }
+
+        if first_chunk_time is None:
+            completion_metrics["ttfb_ms"] = 0
+
+        # Include request-scoped runtime timing for post-run profiling.
+        for key in sorted(_METRIC_FORWARD_KEYS):
+            if key in final_metrics:
+                completion_metrics[key] = final_metrics[key]
+
+        if completion_metrics.get("audio_duration_seconds") in (0, 0.0) and final_metrics.get("audio_duration_seconds"):
+            completion_metrics["audio_duration_seconds"] = final_metrics.get("audio_duration_seconds")
+
+        token_total = final_metrics.get("llm_token_total") or final_metrics.get("token_total")
+        if token_total and completion_metrics.get("generation_ms"):
+            try:
+                completion_metrics["time_per_token_ms"] = float(completion_metrics["generation_ms"]) / float(token_total)
+            except Exception:
+                pass

         await websocket.send_text(create_complete_message(
             request_id=request_id,
             metrics=completion_metrics,
*** End Patch
PATCH


Output: {"output":"Success. Updated the following files:\nM /home/ubuntu/veenaModal/veena3modal/api/websocket_handler.py\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


I’ve finished the API-side propagation and websocket completion enrichment. Next I’m adding a new local benchmark script for deterministic, per-request timing capture and concurrency stress testing against the optimized path, including GPU snapshots and detailed LLM timing breakdowns from response metrics.