Benchmarks are now running cleanly on the optimized path, with detailed per-request timing captured.

**What I changed to make benchmarking reliable**
- Disabled startup speaker-global precompute by default in `veena3modal/services/tts_runtime.py:74` (it was causing cross-event-loop stalls after warmup).
- Added explicit local toggle `--precompute-globals` in `veena3modal/local_server.py:320`.
- Updated HTTP stress script to send `max_tokens` and timeout explicitly in `scripts/stress_test_optimized_local.py:388`.

**Benchmark runs completed**
- Direct runtime detailed stress:
  - `scripts/stress_test_runtime_detailed.py`
  - levels: `1,2,4,8,16,32,64`
  - text: short, `max_tokens=128`
  - output: `/tmp/stress_runtime_detailed_fixed.json`, `/tmp/stress_runtime_detailed_high.json`
- HTTP stress (with perf headers + inflight):
  - `scripts/stress_test_optimized_local.py`
  - levels: `1,4,8,16`
  - text: short, `max_tokens=128`
  - output: `/tmp/stress_http_detailed_fixed.json`

**Runtime results (core)**
| Concurrency | Success | Req/s | p95 latency (ms) | Avg generation (ms) | Avg LLM wall (ms) | Avg BiCodec wall (ms) | Avg ms/token |
|---|---:|---:|---:|---:|---:|---:|---:|
| 1 | 6/6 | 2.27 | 439 | 440.7 | 428.3 | 11.8 | 3.346 |
| 2 | 6/6 | 4.14 | 507 | 477.0 | 464.8 | 11.6 | 3.632 |
| 4 | 12/12 | 7.97 | 529 | 483.4 | 472.3 | 11.0 | 3.690 |
| 8 | 24/24 | 13.85 | 591 | 533.9 | 522.1 | 11.6 | 4.079 |
| 16 | 48/48 | 22.26 | 732 | 630.1 | 619.3 | 10.7 | 4.839 |
| 32 | 64/64 | 30.95 | 1079 | 834.4 | 822.6 | 11.7 | 6.427 |
| 64 | 128/128 | 40.24 | 1672 | 1182.0 | 1170.1 | 11.8 | 9.142 |

**HTTP results (API path)**
| Concurrency | Success | Req/s | p95 latency (ms) | Avg generation (ms) | Avg LLM wall (ms) | Peak inflight |
|---|---:|---:|---:|---:|---:|---:|
| 1 | 10/10 | 2.31 | 441 | 422.9 | 407.6 | 1 |
| 4 | 12/12 | 7.34 | 576 | 501.1 | 486.9 | 4 |
| 8 | 24/24 | 13.20 | 606 | 531.7 | 520.8 | 8 |
| 16 | 48/48 | 21.07 | 760 | 619.8 | 607.8 | 16 |

**Per-request timing breakdown example**
- c=1 median request:
  - `llm_prompt_token_total=22`
  - `llm_token_total=128`
  - `llm_prefill_wall_ms=11.46`
  - `llm_decode_wall_ms_total=390.47`
  - `llm_batch_wall_ms_total=401.92`
  - `bicodec_decode_wall_ms=11.59` (`gpu=11.36`, `cpu=0.24`)
  - `pipeline_total_ms=414.32`

**Bottleneck findings**
- Primary bottleneck is LLM decode/scheduling, not BiCodec.
- LLM wall time is consistently ~97% to 98% of generation time.
- BiCodec decode stays almost flat (~10 to 12 ms) across concurrency.
- Scaling knee starts around `c=16 to c=32`:
  - Throughput still rises, but sublinearly.
  - `ms/token` rises sharply (`4.84 -> 6.43 -> 9.14` at `16 -> 32 -> 64`).
  - p95 latency jumps significantly (`732 -> 1079 -> 1672 ms`).

**Important limitation**
- vLLM request-level GPU timing fields for LLM (`model_execute_time`/`model_forward_time`) were not populated in this environment, so LLM GPU-vs-CPU split is unavailable.
- You still have accurate LLM wall timings and BiCodec GPU timings per request, plus GPU utilization snapshots.

**Artifacts**
- `/tmp/stress_runtime_detailed_fixed.json`
- `/tmp/stress_runtime_detailed_high.json`
- `/tmp/stress_http_detailed_fixed.json`

1. If you want, next I can run the same suite at `max_tokens=256/512` and mixed text lengths to map where queueing starts for realistic traffic.
2. I can also add percentile exports for `llm_prefill_wall_ms` and per-batch decode interval distributions to pinpoint scheduler pressure more precisely.


<permissions instructions>
Filesystem sandboxing defines which files can be read or written. `sandbox_mode` is `danger-full-access`: No filesystem sandboxing - all commands are permitted. Network access is enabled.
Approval policy is currently never. Do not provide the `sandbox_permissions` for any reason, commands will be rejected.
</permissions instructions>


<collaboration_mode># Collaboration Mode: Default

You are now in Default mode. Any previous instructions for other modes (e.g. Plan mode) are no longer active.

Your active mode changes only when new developer instructions with a different `<collaboration_mode>...</collaboration_mode>` change it; user requests or tool descriptions do not change mode by themselves. Known mode names are Default and Plan.

## request_user_input availability

The `request_user_input` tool is unavailable in Default mode. If you call it while in Default mode, it will return an error.

If a decision is necessary and cannot be discovered from local context, ask the user directly. However, in Default mode you should strongly prefer executing the user's request rather than stopping to ask questions.
</collaboration_mode>


# AGENTS.md instructions for /home/ubuntu/veenaModal

<INSTRUCTIONS>
## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) When `SKILL.md` references relative paths (e.g., `scripts/foo.py`), resolve them relative to the skill directory listed above first, and only consider other paths if needed.
  3) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  4) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  5) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.
</INSTRUCTIONS>


<environment_context>
  <cwd>/home/ubuntu/veenaModal</cwd>
  <shell>bash</shell>
</environment_context>


yes. and also this time, before the benchmarks, can you also like put some timeline markers to spot the latency differences and the time spent, to observe. i feel like a 0.5B auto-regressive model and a decoder at some 500M params, in a 80GB A100 GPU should do better than these. So, i'd like to see where we are spending the max time for each stream. what do you think. we can do better ?


# Collaboration Mode: Default

You are now in Default mode. Any previous instructions for other modes (e.g. Plan mode) are no longer active.

Your active mode changes only when new developer instructions with a different `<collaboration_mode>...</collaboration_mode>` change it; user requests or tool descriptions do not change mode by themselves. Known mode names are Default and Plan.

## request_user_input availability

The `request_user_input` tool is unavailable in Default mode. If you call it while in Default mode, it will return an error.

If a decision is necessary and cannot be discovered from local context, ask the user directly. However, in Default mode you should strongly prefer executing the user's request rather than stopping to ask questions.

Summary: auto

## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) When `SKILL.md` references relative paths (e.g., `scripts/foo.py`), resolve them relative to the skill directory listed above first, and only consider other paths if needed.
  3) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  4) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  5) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.


**Planning timeline markers integration**


You’re right to push on this. I’ll add explicit timeline markers (request-relative stage timestamps and deltas) into the generation path first, expose them in benchmark output, then rerun stress tests to show exactly where time is being spent.


**Scoping files for modification**


sed -n '120,460p' veena3modal/core/pipeline.py


sed -n '120,250p' veena3modal/api/fastapi_app.py


Output:
Chunk ID: 83fbf8
Wall time: 0.0505 seconds
Process exited with code 0
Original token count: 883
Output:
scripts/stress_test_optimized_local.py:231:    def timing_stats(self, key: str) -> Dict[str, float]:
scripts/stress_test_optimized_local.py:648:        llm = level.timing_stats("generation_ms")
scripts/stress_test_optimized_local.py:652:        wall = level.timing_stats("llm_batch_wall_ms_total")
scripts/stress_test_optimized_local.py:656:        gpu = level.timing_stats("llm_batch_gpu_ms_total")
scripts/stress_test_optimized_local.py:660:        parse = level.timing_stats("llm_parse_ms")
scripts/stress_test_optimized_local.py:664:        decode = level.timing_stats("llm_decode_wall_ms_total")
scripts/stress_test_optimized_local.py:668:        bdecode = level.timing_stats("bicodec_decode_wall_ms")
scripts/stress_test_optimized_local.py:672:        bdecode_gpu = level.timing_stats("bicodec_decode_gpu_ms")
scripts/stress_test_optimized_local.py:748:                "generation_ms": level.timing_stats("generation_ms"),
scripts/stress_test_optimized_local.py:749:                "llm_batch_wall_ms_total": level.timing_stats("llm_batch_wall_ms_total"),
scripts/stress_test_optimized_local.py:750:                "llm_batch_gpu_ms_total": level.timing_stats("llm_batch_gpu_ms_total"),
scripts/stress_test_optimized_local.py:751:                "llm_parse_ms": level.timing_stats("llm_parse_ms"),
scripts/stress_test_optimized_local.py:752:                "llm_decode_wall_ms_total": level.timing_stats("llm_decode_wall_ms_total"),
scripts/stress_test_optimized_local.py:753:                "bicodec_decode_wall_ms": level.timing_stats("bicodec_decode_wall_ms"),
scripts/stress_test_optimized_local.py:754:                "bicodec_decode_gpu_ms": level.timing_stats("bicodec_decode_gpu_ms"),
scripts/stress_test_runtime_detailed.py:177:    def timing_stats(self, key: str) -> Dict[str, float]:
scripts/stress_test_runtime_detailed.py:339:        st = level.timing_stats(key)
scripts/stress_test_runtime_detailed.py:386:                    "generation_ms": lvl.timing_stats("generation_ms"),
scripts/stress_test_runtime_detailed.py:387:                    "llm_batch_wall_ms_total": lvl.timing_stats("llm_batch_wall_ms_total"),
scripts/stress_test_runtime_detailed.py:388:                    "llm_batch_gpu_ms_total": lvl.timing_stats("llm_batch_gpu_ms_total"),
scripts/stress_test_runtime_detailed.py:389:                    "llm_decode_wall_ms_total": lvl.timing_stats("llm_decode_wall_ms_total"),
scripts/stress_test_runtime_detailed.py:390:                    "llm_decode_gpu_ms_total": lvl.timing_stats("llm_decode_gpu_ms_total"),
scripts/stress_test_runtime_detailed.py:391:                    "llm_parse_ms": lvl.timing_stats("llm_parse_ms"),
scripts/stress_test_runtime_detailed.py:392:                    "llm_time_per_token_ms": lvl.timing_stats("llm_time_per_token_ms"),
scripts/stress_test_runtime_detailed.py:393:                    "bicodec_decode_wall_ms": lvl.timing_stats("bicodec_decode_wall_ms"),
scripts/stress_test_runtime_detailed.py:394:                    "bicodec_decode_gpu_ms": lvl.timing_stats("bicodec_decode_gpu_ms"),
veena3modal/services/tts_runtime.py:639:async def generate_speech_streaming(
veena3modal/core/pipeline.py:183:            "perf_non_stream_start",
veena3modal/core/pipeline.py:237:                    "perf_first_llm_batch",
veena3modal/api/fastapi_app.py:184:    extra_detail_keys = (
veena3modal/api/fastapi_app.py:203:    for key in extra_detail_keys:
veena3modal/api/fastapi_app.py:715:                async for audio_chunk, metrics in tts_runtime.generate_speech_streaming(


Output:
Chunk ID: 673707
Wall time: 0.0515 seconds
Process exited with code 0
Original token count: 3515
Output:
        self,
        speaker: str,
        text: str,
        temperature: float = DEFAULT_TEMPERATURE,
        top_k: int = DEFAULT_TOP_K,
        top_p: float = DEFAULT_TOP_P,
        max_tokens: int = DEFAULT_MAX_TOKENS,
        repetition_penalty: float = 1.05,
        seed: Optional[int] = None,
    ) -> Tuple[Optional[bytes], Dict[str, Any]]:
        """
        Generate speech with detailed per-request timing metrics.

        Returns:
            (wav_bytes, perf_dict)
        """
        from veena3modal.audio.utils import add_wav_header

        perf: Dict[str, Any] = {
            "llm_prompt_token_total": 0,
            "llm_token_total": 0,
            "semantic_token_total": 0,
            "global_token_total": 0,
            "llm_batch_count": 0,
            "llm_parse_ms": 0.0,
            "llm_decode_calls": 0,
            "bicodec_decode_wall_ms": 0.0,
            "bicodec_decode_gpu_ms": 0.0,
            "bicodec_decode_cpu_ms": 0.0,
            "wav_pack_ms": 0.0,
        }

        t_request_start = time.perf_counter()

        # Stage 1: prompt build
        t0 = time.perf_counter()
        prompt = self.prompt_builder.build_prefix(speaker, text)
        perf["prompt_build_ms"] = (time.perf_counter() - t0) * 1000

        # Prompt token count (for TPS normalization)
        try:
            perf["llm_prompt_token_total"] = len(
                self.model.tokenizer.encode(prompt, add_special_tokens=False)
            )
        except Exception:
            perf["llm_prompt_token_total"] = 0

        # Stage 2: sampling params
        t0 = time.perf_counter()
        sampling_params = SamplingParams(
            temperature=temperature,
            top_k=top_k,
            top_p=top_p,
            max_tokens=max_tokens,
            repetition_penalty=repetition_penalty,
            stop=TRAINING_STOP_TOKEN_IDS,
            skip_special_tokens=False,
            seed=seed,
        )
        perf["sampling_params_ms"] = (time.perf_counter() - t0) * 1000

        request_id = f"req-{uuid.uuid4().hex[:12]}"
        logger.info(
            "perf_non_stream_start",
            extra={
                "request_id": request_id,
                "speaker": speaker,
                "text_length": len(text),
                "max_tokens": max_tokens,
            },
        )
        results_generator = self.model.engine.generate(
            prompt=prompt,
            sampling_params=sampling_params,
            request_id=request_id,
        )

        final_output = None
        final_request_metrics: Dict[str, float] = {}

        llm_batch_wall_ms: List[float] = []
        llm_batch_gpu_ms: List[float] = []
        llm_decode_wall_ms: List[float] = []
        llm_decode_gpu_ms: List[float] = []
        tokens_per_batch: List[float] = []

        prev_generated_count = 0
        prev_gpu_cumulative_s: Optional[float] = None

        t_gen_start = time.perf_counter()
        t_last_yield = t_gen_start
        async for request_output in results_generator:
            now = time.perf_counter()
            batch_wall = (now - t_last_yield) * 1000
            t_last_yield = now

            outputs = request_output.outputs or []
            if not outputs:
                final_output = request_output
                continue

            output = outputs[0]
            generated_count = len(output.token_ids)
            new_tokens = max(0, generated_count - prev_generated_count)
            prev_generated_count = generated_count

            raw_req_metrics = self._extract_request_metrics(getattr(request_output, "metrics", None))
            if raw_req_metrics:
                final_request_metrics = raw_req_metrics

            if new_tokens <= 0:
                final_output = request_output
                continue

            perf["llm_batch_count"] += 1
            if perf["llm_batch_count"] == 1:
                logger.info(
                    "perf_first_llm_batch",
                    extra={
                        "request_id": request_id,
                        "batch_wall_ms": round(batch_wall, 3),
                        "new_tokens": new_tokens,
                    },
                )
            tokens_per_batch.append(float(new_tokens))
            llm_batch_wall_ms.append(float(batch_wall))
            if perf["llm_batch_count"] > 1:
                llm_decode_wall_ms.append(float(batch_wall))

            cumulative_gpu_s: Optional[float] = None
            if raw_req_metrics:
                if "model_execute_time" in raw_req_metrics:
                    cumulative_gpu_s = raw_req_metrics["model_execute_time"]
                elif "model_forward_time" in raw_req_metrics:
                    cumulative_gpu_s = raw_req_metrics["model_forward_time"]

            if cumulative_gpu_s is not None:
                if prev_gpu_cumulative_s is None:
                    gpu_delta_ms = cumulative_gpu_s * 1000
                elif cumulative_gpu_s >= prev_gpu_cumulative_s:
                    gpu_delta_ms = (cumulative_gpu_s - prev_gpu_cumulative_s) * 1000
                else:
                    # Some versions report non-cumulative per-step values.
                    gpu_delta_ms = cumulative_gpu_s * 1000
                prev_gpu_cumulative_s = cumulative_gpu_s

                if gpu_delta_ms >= 0:
                    llm_batch_gpu_ms.append(float(gpu_delta_ms))
                    if perf["llm_batch_count"] > 1:
                        llm_decode_gpu_ms.append(float(gpu_delta_ms))

            final_output = request_output

        perf["llm_generation_wall_ms"] = (time.perf_counter() - t_gen_start) * 1000
        logger.info(
            "perf_llm_generation_done",
            extra={
                "request_id": request_id,
                "llm_generation_wall_ms": round(perf["llm_generation_wall_ms"], 3),
                "llm_batch_count": perf["llm_batch_count"],
            },
        )

        if final_output is None:
            perf["pipeline_total_ms"] = (time.perf_counter() - t_request_start) * 1000
            return None, perf

        generated_text = final_output.outputs[0].text
        generated_ids = final_output.outputs[0].token_ids
        perf["llm_token_total"] = len(generated_ids)

        # Stage 3: parse BiCodec tokens from generated text
        t0 = time.perf_counter()
        semantic_ids, global_ids = self._extract_bicodec_tokens(generated_text)
        perf["llm_parse_ms"] = (time.perf_counter() - t0) * 1000
        perf["semantic_token_total"] = len(semantic_ids)
        perf["global_token_total"] = len(global_ids)
        logger.info(
            "perf_token_parse_done",
            extra={
                "request_id": request_id,
                "parse_ms": round(perf["llm_parse_ms"], 3),
                "semantic_tokens": perf["semantic_token_total"],
                "global_tokens": perf["global_token_total"],
            },
        )

        # Aggregate LLM batch stats
        self._apply_stats(perf, "llm_batch_wall_ms", llm_batch_wall_ms)
        self._apply_stats(perf, "llm_batch_gpu_ms", llm_batch_gpu_ms)
        self._apply_stats(perf, "llm_decode_wall_ms", llm_decode_wall_ms)
        self._apply_stats(perf, "llm_decode_gpu_ms", llm_decode_gpu_ms)
        self._apply_stats(perf, "tokens_per_batch", tokens_per_batch)

        if llm_batch_wall_ms:
            perf["llm_prefill_wall_ms"] = float(llm_batch_wall_ms[0])
            perf["llm_prefill_gpu_ms"] = float(llm_batch_gpu_ms[0]) if llm_batch_gpu_ms else 0.0
        else:
            perf["llm_prefill_wall_ms"] = float(perf["llm_generation_wall_ms"])
            perf["llm_prefill_gpu_ms"] = 0.0

        if not llm_batch_gpu_ms and final_request_metrics.get("model_execute_time") is not None:
            total_gpu_ms = float(final_request_metrics["model_execute_time"]) * 1000
            perf["llm_batch_gpu_ms_total"] = total_gpu_ms
            if perf["llm_batch_count"] > 1:
                perf["llm_decode_gpu_ms_total"] = total_gpu_ms

        if perf["llm_token_total"] > 0:
            perf["llm_time_per_token_ms"] = float(perf["llm_generation_wall_ms"]) / float(perf["llm_token_total"])
        if perf["llm_batch_count"] > 0:
            perf["llm_time_per_batch_wall_ms"] = float(perf.get("llm_batch_wall_ms_total", 0.0)) / float(perf["llm_batch_count"])
            if perf.get("llm_batch_gpu_ms_total", 0.0) > 0:
                perf["llm_time_per_batch_gpu_ms"] = float(perf["llm_batch_gpu_ms_total"]) / float(perf["llm_batch_count"])

        # Pull useful request-level vLLM metrics (seconds -> ms)
        sec_fields = {
            "time_in_queue": "llm_time_in_queue_ms",
            "scheduler_time": "llm_scheduler_ms",
            "model_forward_time": "llm_model_forward_ms",
            "model_execute_time": "llm_model_execute_ms",
        }
        for src, dst in sec_fields.items():
            if src in final_request_metrics:
                perf[dst] = float(final_request_metrics[src]) * 1000

        arrival = final_request_metrics.get("arrival_time")
        first_token = final_request_metrics.get("first_token_time")
        finished = final_request_metrics.get("finished_time")
        if arrival is not None and first_token is not None and first_token >= arrival:
            perf["llm_first_token_ms"] = (first_token - arrival) * 1000
        if arrival is not None and finished is not None and finished >= arrival:
            perf["llm_request_lifecycle_ms"] = (finished - arrival) * 1000

        if not semantic_ids or not global_ids:
            perf["pipeline_total_ms"] = (time.perf_counter() - t_request_start) * 1000
            return None, perf

        # Stage 4: validate BiCodec tokens
        t0 = time.perf_counter()
        if not self.bicodec_decoder.validate_tokens(semantic_ids, global_ids):
            perf["token_validation_ms"] = (time.perf_counter() - t0) * 1000
            perf["pipeline_total_ms"] = (time.perf_counter() - t_request_start) * 1000
            return None, perf
        perf["token_validation_ms"] = (time.perf_counter() - t0) * 1000

        # Stage 5: BiCodec decode (wall + GPU/CPU split)
        decode_t0 = time.perf_counter()
        decode_gpu_ms = 0.0
        perf["llm_decode_calls"] = 1

        use_cuda_timing = (
            torch.cuda.is_available()
            and hasattr(self.bicodec_decoder, "device")
            and getattr(self.bicodec_decoder.device, "type", "") == "cuda"
        )
        audio_bytes: Optional[bytes]
        if use_cuda_timing:
            try:
                device = self.bicodec_decoder.device
                start_event = torch.cuda.Event(enable_timing=True)
                end_event = torch.cuda.Event(enable_timing=True)
                torch.cuda.synchronize(device=device)
                start_event.record()
                audio_bytes = self.bicodec_decoder.decode_to_bytes(semantic_ids, global_ids)
                end_event.record()
                torch.cuda.synchronize(device=device)
                decode_gpu_ms = float(start_event.elapsed_time(end_event))
            except Exception:
                audio_bytes = self.bicodec_decoder.decode_to_bytes(semantic_ids, global_ids)
                decode_gpu_ms = 0.0
        else:
            audio_bytes = self.bicodec_decoder.decode_to_bytes(semantic_ids, global_ids)

        decode_wall_ms = (time.perf_counter() - decode_t0) * 1000
        perf["bicodec_decode_wall_ms"] = float(decode_wall_ms)
        perf["bicodec_decode_gpu_ms"] = float(max(0.0, decode_gpu_ms))
        perf["bicodec_decode_cpu_ms"] = float(max(0.0, decode_wall_ms - decode_gpu_ms))
        logger.info(
            "perf_bicodec_decode_done",
            extra={
                "request_id": request_id,
                "decode_wall_ms": round(perf["bicodec_decode_wall_ms"], 3),
                "decode_gpu_ms": round(perf["bicodec_decode_gpu_ms"], 3),
                "decode_cpu_ms": round(perf["bicodec_decode_cpu_ms"], 3),
            },
        )

        if audio_bytes is None:
            perf["pipeline_total_ms"] = (time.perf_counter() - t_request_start) * 1000
            return None, perf

        # Stage 6: WAV header
        t0 = time.perf_counter()
        wav_bytes = add_wav_header(audio_bytes, sample_rate=AUDIO_SAMPLE_RATE)
        perf["wav_pack_ms"] = (time.perf_counter() - t0) * 1000

        perf["pipeline_total_ms"] = (time.perf_counter() - t_request_start) * 1000
        logger.info(
            "perf_non_stream_done",
            extra={
                "request_id": request_id,
                "pipeline_total_ms": round(perf["pipeline_total_ms"], 3),
                "llm_token_total": perf.get("llm_token_total"),
            },
        )
        return wav_bytes, perf

    async def generate_speech(
        self,
        speaker: str,
        text: str,
        temperature: float = DEFAULT_TEMPERATURE,
        top_k: int = DEFAULT_TOP_K,
        top_p: float = DEFAULT_TOP_P,
        max_tokens: int = DEFAULT_MAX_TOKENS,
        repetition_penalty: float = 1.05,
        seed: Optional[int] = None,
    ) -> Optional[bytes]:
        """
        Generate speech audio (non-streaming) using Spark TTS.

        NOTE: This method signature changed from description-based to speaker-based
        to align with Spark TTS architecture.

        Args:
            speaker: Speaker name (one of 12 predefined speakers)
            text: Text to synthesize (with optional [emotion] tags)
            temperature: Sampling temperature (default: 0.8 for Spark TTS)
            top_k: Top-k sampling (default: 50)
            top_p: Nucleus sampling (default: 1.0)
            max_tokens: Max BiCodec tokens to generate (default: 2048)
            seed: Random seed for reproducibility

        Returns:
            Audio bytes (int16 PCM WAV, 16kHz mono) or None if failed
        """
        wav_bytes, _ = await self.generate_speech_profiled(
            speaker=speaker,
            text=text,
            temperature=temperature,
            top_k=top_k,


Output:
Chunk ID: e6bc27
Wall time: 0.0509 seconds
Process exited with code 0
Original token count: 1189
Output:
    for easy parsing in shell scripts/tools.
    """
    perf_headers: Dict[str, str] = {}
    if not isinstance(metrics, dict):
        metrics = {}

    detail_payload: Dict[str, Any] = {}
    if request_inflight > 0:
        detail_payload["request_inflight"] = request_inflight

    # Expose selected metrics as explicit headers.
    header_map = {
        "x-generation-ms": "generation_ms",
        "x-request-inflight": "request_inflight",
        "x-llm-token-total": "llm_token_total",
        "x-llm-batch-count": "llm_batch_count",
        "x-llm-batch-wall-ms": "llm_batch_wall_ms_total",
        "x-llm-batch-gpu-ms": "llm_batch_gpu_ms_total",
        "x-llm-batch-wall-ms-min": "llm_batch_wall_ms_min",
        "x-llm-batch-wall-ms-max": "llm_batch_wall_ms_max",
        "x-llm-batch-wall-ms-p50": "llm_batch_wall_ms_p50",
        "x-llm-batch-gpu-ms-min": "llm_batch_gpu_ms_min",
        "x-llm-batch-gpu-ms-max": "llm_batch_gpu_ms_max",
        "x-llm-batch-gpu-ms-p50": "llm_batch_gpu_ms_p50",
        "x-llm-decode-wall-ms": "llm_decode_wall_ms_total",
        "x-llm-decode-gpu-ms": "llm_decode_gpu_ms_total",
        "x-llm-decode-wall-ms-min": "llm_decode_wall_ms_min",
        "x-llm-decode-wall-ms-max": "llm_decode_wall_ms_max",
        "x-llm-decode-wall-ms-p50": "llm_decode_wall_ms_p50",
        "x-llm-decode-gpu-ms-min": "llm_decode_gpu_ms_min",
        "x-llm-decode-gpu-ms-max": "llm_decode_gpu_ms_max",
        "x-llm-decode-gpu-ms-p50": "llm_decode_gpu_ms_p50",
        "x-llm-decode-calls": "llm_decode_calls",
        "x-llm-decode-cpu-ms": "bicodec_decode_cpu_ms",
        "x-llm-parse-ms": "llm_parse_ms",
        "x-llm-parse-avg-ms": "llm_parse_ms",
        "x-llm-tokens-per-batch-min": "tokens_per_batch_min",
        "x-llm-tokens-per-batch-max": "tokens_per_batch_max",
        "x-llm-tokens-per-batch-p50": "tokens_per_batch_p50",
        "x-llm-time-per-token-ms": "llm_time_per_token_ms",
        "x-llm-time-per-batch-wall-ms": "llm_time_per_batch_wall_ms",
        "x-llm-time-per-batch-gpu-ms": "llm_time_per_batch_gpu_ms",
        "x-llm-text-chunked": "text_chunked",
        "x-llm-chunks-processed": "chunks_processed",
        "x-bicodec-decode-wall-ms": "bicodec_decode_wall_ms",
        "x-bicodec-decode-gpu-ms": "bicodec_decode_gpu_ms",
        "x-bicodec-decode-cpu-ms": "bicodec_decode_cpu_ms",
    }

    if request_inflight > 0:
        metrics = dict(metrics)
        metrics["request_inflight"] = request_inflight

    for header_key, metric_key in header_map.items():
        if metric_key not in metrics:
            continue
        value = metrics.get(metric_key)
        serialized = _serialize_header_value(value)
        if serialized is None:
            continue
        perf_headers[header_key] = serialized
        detail_payload[metric_key] = value

    # Include additional timing fields (bounded set) in JSON payload only.
    extra_detail_keys = (
        "prompt_build_ms",
        "sampling_params_ms",
        "llm_prefill_wall_ms",
        "llm_prefill_gpu_ms",
        "llm_generation_wall_ms",
        "llm_time_in_queue_ms",
        "llm_scheduler_ms",
        "llm_model_forward_ms",
        "llm_model_execute_ms",
        "llm_first_token_ms",
        "llm_request_lifecycle_ms",
        "llm_prompt_token_total",
        "semantic_token_total",
        "global_token_total",
        "wav_pack_ms",
        "pipeline_total_ms",
        "token_validation_ms",
    )
    for key in extra_detail_keys:
        if key in metrics:
            detail_payload[key] = metrics[key]

    perf_headers["x-perf-details"] = json.dumps(
        detail_payload,
        separators=(",", ":"),
        ensure_ascii=True,
    )
    return perf_headers

def get_model_version() -> str:
    """Return the current model version, or 'not_loaded' if not initialized."""
    return _MODEL_VERSION or "not_loaded"

def set_model_version(version: str) -> None:
    """Set the model version after loading."""
    global _MODEL_VERSION
    _MODEL_VERSION = version

def create_app():
    """
    Factory function to create FastAPI app.

    Import inside factory so local tooling can import this module without FastAPI installed.
    """
    from fastapi import FastAPI, HTTPException, Request
    from fastapi.responses import JSONResponse, Response

    global _startup_time
    _startup_time = time.time()

    app = FastAPI(
        title="Veena3 TTS (Modal)",
        version=_APP_VERSION,
        description="High-quality multilingual Text-to-Speech API with true streaming support.",
    )

    @app.get("/v1/tts/health")
    async def tts_health():
        """
        Health check endpoint.

        Returns:
            - status: "healthy" | "degraded" | "unhealthy"


Summary: auto


**Inspecting streaming pipeline timeline markers**


sed -n '620,900p' veena3modal/services/tts_runtime.py


sed -n '1,320p' veena3modal/core/streaming_pipeline.py


sed -n '320,760p' veena3modal/core/streaming_pipeline.py


Output:
Chunk ID: de12a2
Wall time: 0.0515 seconds
Process exited with code 0
Original token count: 2567
Output:
                except Exception as e:
                    logger.warning(f"Super-resolution failed, returning 16kHz: {e}")
                    import traceback
                    traceback.print_exc()

            metrics["output_sample_rate"] = final_sample_rate
            audio_duration = (len(audio_bytes) - 44) / (final_sample_rate * 2)
            metrics["audio_duration_seconds"] = max(0.0, audio_duration)

        return audio_bytes, metrics

    except Exception as e:
        logger.error(f"Chunked speech generation failed: {e}")
        raise

from typing import AsyncGenerator

async def generate_speech_streaming(
    text: str,
    speaker: str,
    temperature: float = 0.8,
    top_k: int = 50,
    top_p: float = 1.0,
    max_tokens: int = 4096,
    repetition_penalty: float = 1.05,
    seed: Optional[int] = None,
    enable_chunking: bool = True,
) -> AsyncGenerator[Tuple[bytes, Dict[str, Any]], None]:
    """
    Generate speech audio with true streaming (yields chunks as they're generated).

    This is the core streaming implementation for M4.
    First yield includes WAV header + first PCM chunk.
    Subsequent yields are raw PCM chunks.

    Args:
        text: Text to synthesize (already normalized)
        speaker: Internal speaker name (resolved)
        temperature: Sampling temperature
        top_k: Top-k sampling
        top_p: Nucleus sampling
        max_tokens: Maximum tokens to generate
        repetition_penalty: Repetition penalty
        seed: Random seed for reproducibility
        enable_chunking: Enable text chunking for long inputs with voice consistency

    Yields:
        Tuple of (audio_bytes, metrics_dict)
        - First yield: WAV header (44 bytes) prepended to first PCM chunk
        - Subsequent yields: Raw PCM chunks (int16, 16kHz)
        - Final yield: metrics_dict has final timing info

    Raises:
        RuntimeError: If runtime not initialized or streaming pipeline unavailable
    """
    if not is_initialized():
        raise RuntimeError("TTS runtime not initialized")

    runtime = get_runtime()

    if runtime.streaming_pipeline is None:
        raise RuntimeError("Streaming pipeline not available")

    # Import audio utils for WAV header
    from veena3modal.audio.utils import create_wav_header

    # Metrics tracking
    metrics = {
        "ttfb_ms": 0,
        "chunks_sent": 0,
        "total_bytes": 0,
        "audio_duration_seconds": 0.0,
        "text_chunked": False,
    }

    start_time = time.time()
    first_chunk_time = None
    total_pcm_bytes = 0
    sample_rate = 16000  # BiCodec sample rate

    # Check if we need text chunking
    from veena3modal.processing.long_text_processor import LongTextProcessor
    long_processor = LongTextProcessor(pipeline=runtime.pipeline)
    needs_chunking = enable_chunking and long_processor.should_chunk(text)

    if needs_chunking:
        # Chunked streaming with voice consistency (global token caching)
        metrics["text_chunked"] = True
        async for audio_chunk, chunk_metrics in _stream_chunked_text(
            runtime, long_processor, text, speaker, temperature, top_k, top_p,
            max_tokens, repetition_penalty, seed, sample_rate
        ):
            # First chunk: prepend WAV header
            if first_chunk_time is None:
                first_chunk_time = time.time()
                metrics["ttfb_ms"] = int((first_chunk_time - start_time) * 1000)
                # Create streaming WAV header (size=0 for unknown length)
                wav_header = create_wav_header(sample_rate=sample_rate, data_size=0)
                audio_chunk = wav_header + audio_chunk

            total_pcm_bytes += len(audio_chunk) - (44 if metrics["chunks_sent"] == 0 else 0)
            metrics["chunks_sent"] += 1
            metrics["total_bytes"] = total_pcm_bytes + 44  # Include header
            metrics["audio_duration_seconds"] = total_pcm_bytes / (sample_rate * 2)

            yield audio_chunk, metrics
    else:
        # Simple streaming (no text chunking)
        # OPTIMIZATION: If we have pre-cached global tokens for this speaker,
        # use continuation mode to skip the ~110ms global token pre-roll.
        # The model jumps straight to semantic token generation.
        cached_globals = runtime.speaker_global_cache.get(speaker)

        if cached_globals:
            # Fast path: use cached globals, skip global token generation entirely
            stream_gen = runtime.streaming_pipeline.generate_speech_stream_indic_continuation(
                speaker=speaker,
                text=text,
                global_ids=cached_globals,
                temperature=temperature,
                top_k=top_k,
                top_p=top_p,
                max_tokens=max_tokens,
                repetition_penalty=repetition_penalty,
                seed=seed,
            )
        else:
            # Fallback: no cached globals, generate them inline (original path)
            stream_gen = runtime.streaming_pipeline.generate_speech_stream_indic(
                speaker=speaker,
                text=text,
                temperature=temperature,
                top_k=top_k,
                top_p=top_p,
                max_tokens=max_tokens,
                repetition_penalty=repetition_penalty,
                seed=seed,
            )

        async for audio_chunk in stream_gen:
            # First chunk: prepend WAV header
            if first_chunk_time is None:
                first_chunk_time = time.time()
                metrics["ttfb_ms"] = int((first_chunk_time - start_time) * 1000)
                wav_header = create_wav_header(sample_rate=sample_rate, data_size=0)
                audio_chunk = wav_header + audio_chunk

            total_pcm_bytes += len(audio_chunk) - (44 if metrics["chunks_sent"] == 0 else 0)
            metrics["chunks_sent"] += 1
            metrics["total_bytes"] = total_pcm_bytes + 44
            metrics["audio_duration_seconds"] = total_pcm_bytes / (sample_rate * 2)

            yield audio_chunk, metrics

async def _stream_chunked_text(
    runtime: TTSRuntime,
    long_processor,
    text: str,
    speaker: str,
    temperature: float,
    top_k: int,
    top_p: float,
    max_tokens: int,
    repetition_penalty: float,
    seed: Optional[int],
    sample_rate: int,
) -> AsyncGenerator[Tuple[bytes, Dict[str, Any]], None]:
    """
    Internal helper: stream chunked text with voice consistency.

    Uses global token caching from first chunk to maintain voice across chunks.
    """
    from veena3modal.audio.crossfade import crossfade_bytes_int16

    # Chunk text using the long text processor's chunker
    chunks = long_processor.chunk_text(text)

    if not chunks:
        return

    captured_globals: Optional[List[int]] = None
    previous_chunk_tail: Optional[bytes] = None
    chunk_metrics = {"chunks_processed": 0}

    for i, chunk_text in enumerate(chunks):
        chunk_metrics["chunks_processed"] = i + 1

        if i == 0:
            # First chunk: capture global tokens for voice consistency
            async for audio_bytes, global_ids in runtime.streaming_pipeline.generate_speech_stream_indic_first_chunk(
                speaker=speaker,
                text=chunk_text,
                temperature=temperature,
                top_k=top_k,
                top_p=top_p,
                max_tokens=max_tokens,
                repetition_penalty=repetition_penalty,
                seed=seed,
            ):
                if captured_globals is None and global_ids:
                    captured_globals = global_ids

                # Crossfade with previous tail (inter-chunk stitching)
                to_emit, previous_chunk_tail = crossfade_bytes_int16(
                    previous_chunk_tail,
                    audio_bytes,
                    sample_rate_hz=sample_rate,
                    crossfade_ms=50,
                )

                if to_emit:
                    yield to_emit, chunk_metrics
        else:
            # Continuation chunks: use captured globals for voice consistency
            if captured_globals is None:
                logger.warning(f"No captured globals for chunk {i+1}, using regular streaming")
                async for audio_bytes in runtime.streaming_pipeline.generate_speech_stream_indic(
                    speaker=speaker,
                    text=chunk_text,
                    temperature=temperature,
                    top_k=top_k,
                    top_p=top_p,
                    max_tokens=max_tokens,
                    repetition_penalty=repetition_penalty,
                    seed=seed,
                ):
                    to_emit, previous_chunk_tail = crossfade_bytes_int16(
                        previous_chunk_tail,
                        audio_bytes,
                        sample_rate_hz=sample_rate,
                        crossfade_ms=50,
                    )
                    if to_emit:
                        yield to_emit, chunk_metrics
            else:
                # Use continuation method with cached globals
                async for audio_bytes in runtime.streaming_pipeline.generate_speech_stream_indic_continuation(
                    speaker=speaker,
                    text=chunk_text,
                    global_ids=captured_globals,
                    temperature=temperature,
                    top_k=top_k,
                    top_p=top_p,
                    max_tokens=max_tokens,
                    repetition_penalty=repetition_penalty,
                    seed=seed,
                ):
                    to_emit, previous_chunk_tail = crossfade_bytes_int16(
                        previous_chunk_tail,
                        audio_bytes,
                        sample_rate_hz=sample_rate,
                        crossfade_ms=50,
                    )
                    if to_emit:
                        yield to_emit, chunk_metrics

    # Flush remaining tail
    if previous_chunk_tail:
        yield previous_chunk_tail, chunk_metrics


Output:
Chunk ID: be6b5f
Wall time: 0.0506 seconds
Process exited with code 0
Original token count: 3206
Output:
"""
Veena3 Streaming Pipeline - Sliding Window Approach

Implements streaming for both SNAC and BiCodec tokens.

For BiCodec (Spark TTS):
1. Generate tokens from vLLM
2. Parse NEW token IDs incrementally (O(1) per token via cache)
3. Buffer semantic + global tokens
4. Apply sliding window (every N token pairs)
5. Decode and stream audio chunks

OPTIMIZATION (Dec 2025):
- Replaced O(n²) pattern (decode-all + regex-all per iteration)
- Now uses incremental token parsing with BiCodecTokenParser
- ~10x CPU reduction in streaming hot loop

For SNAC (legacy):
1. Filter SNAC token IDs directly
2. Apply sliding window
3. Decode and stream
"""

import asyncio
import re
from typing import AsyncGenerator, Optional, List
from vllm import SamplingParams
from veena3modal.audio.crossfade import crossfade_bytes_int16
from veena3modal.core.token_utils import BiCodecTokenParser

from veena3modal.core.constants import (
    CODE_END_TOKEN_ID,
    CODE_START_TOKEN_ID,
    SNAC_MIN_ID,
    SNAC_MAX_ID,
    TRAINING_STOP_TOKEN_IDS,
    DEFAULT_TEMPERATURE,
    DEFAULT_TOP_K,
    DEFAULT_TOP_P,
    DEFAULT_MAX_TOKENS,
    DEFAULT_MIN_TOKENS,
    DEFAULT_REPETITION_PENALTY,
    DEFAULT_SEED,
)

class Veena3SlidingWindowPipeline:
    """
    Streaming TTS pipeline using sliding window approach.

    This eliminates choppy audio and popping artifacts by:
    - Decoding overlapping 28-token windows (4 frames)
    - Keeping only the middle 2048 samples from each decode
    - Creating natural continuity between chunks

    Based on the official Canopy Labs implementation.
    """

    def __init__(
        self,
        model,
        prompt_builder,
        snac_decoder,
    ):
        """
        Initialize sliding window streaming pipeline.

        Args:
            model: Veena3Model instance
            prompt_builder: Veena3PromptBuilder instance
            snac_decoder: SNACDecoder instance (with batching enabled)
        """
        self.model = model
        self.prompt_builder = prompt_builder
        self.snac_decoder = snac_decoder

        # OPTIMIZATION: Pre-warm BiCodecTokenParser once at init, not per-request
        # Saves ~123ms per streaming TTFB by avoiding repeated 166K-entry vocab iteration
        tokenizer = getattr(model, "tokenizer", None)
        if tokenizer is None:
            tokenizer = getattr(model.engine, "tokenizer", None)
        self.token_parser = BiCodecTokenParser(tokenizer) if tokenizer else None

        print(f"🌊 Veena3SlidingWindowPipeline initialized (sliding window: 28 tokens)")

    async def generate_speech_stream(
        self,
        description: str,
        text: str,
        temperature: float = DEFAULT_TEMPERATURE,
        top_p: float = DEFAULT_TOP_P,
        max_tokens: int = DEFAULT_MAX_TOKENS,
        repetition_penalty: float = DEFAULT_REPETITION_PENALTY,
        seed: Optional[int] = None,
    ) -> AsyncGenerator[bytes, None]:
        """
        Generate speech audio with sliding window streaming.

        Yields audio chunks using overlapping windows for smooth playback.

        Args:
            description: Character/voice description
            text: Text to synthesize (with optional <emotion> tags)
            temperature: Sampling temperature
            top_p: Nucleus sampling
            max_tokens: Max SNAC tokens to generate
            repetition_penalty: Prevent loops

        Yields:
            Audio bytes (int16 PCM, 24kHz mono)
        """
        print(f"\n🌊 Sliding window streaming generation")
        print(f"📝 Description: {description[:80]}...")
        print(f"💬 Text: {text}")

        # Build prompt
        prompt = self.prompt_builder.build_prefix(description, text)
        print(f"✅ Prompt built ({len(prompt)} chars)")

        # Configure sampling
        sampling_params = SamplingParams(
            temperature=temperature,
            top_p=top_p,
            max_tokens=max_tokens,
            min_tokens=DEFAULT_MIN_TOKENS,
            repetition_penalty=repetition_penalty,
            stop_token_ids=[CODE_END_TOKEN_ID],  # Only audio EOS
            seed=seed,  # None = random, int = reproducible
        )

        print(f"🎲 Sampling: temp={temperature}, top_p={top_p}, sliding_window=28 tokens")

        # Token buffer - keeps ALL tokens (not chunked)
        token_buffer = []
        total_tokens_generated = 0
        total_audio_chunks = 0

        # Generate tokens with vLLM (streaming)
        print(f"🔮 Starting token generation...")

        # Generate unique request ID for concurrent streaming support
        import uuid
        import time
        request_id = f"slide-{uuid.uuid4().hex[:8]}-{int(time.time() * 1000000)}"
        results_generator = self.model.engine.generate(
            prompt=prompt,
            sampling_params=sampling_params,
            request_id=request_id,
        )

        # Stream tokens with sliding window
        async for request_output in results_generator:
            # Extract generated token IDs
            generated_ids = request_output.outputs[0].token_ids

            # Process only new tokens
            new_tokens = generated_ids[total_tokens_generated:]
            total_tokens_generated = len(generated_ids)

            # Filter and buffer SNAC tokens
            for token_id in new_tokens:
                if SNAC_MIN_ID <= token_id <= SNAC_MAX_ID:
                    token_buffer.append(token_id)

                    # Process every 7 tokens (1 frame) when we have enough for sliding window
                    # Official approach: once count > 27, take last 28 tokens
                    if len(token_buffer) % 7 == 0 and len(token_buffer) > 27:
                        # Sliding window: take last 28 tokens (4 frames)
                        window_tokens = token_buffer[-28:]

                        # Decode with sliding window mode (returns middle 2048 samples only)
                        if self.snac_decoder.enable_batching:
                            audio_bytes = await self.snac_decoder.decode_single_async(
                                window_tokens,
                                trim_warmup=False,  # Sliding window handles trimming
                                use_sliding_window=True  # CRITICAL: Use sliding window mode
                            )
                        else:
                            audio_bytes = self.snac_decoder.decode_to_bytes(
                                window_tokens,
                                trim_warmup=False,
                                use_sliding_window=True
                            )

                        if audio_bytes:
                            total_audio_chunks += 1
                            if total_audio_chunks == 1:
                                print(f"🎵 First chunk decoded ({len(audio_bytes)} bytes, sliding window)")
                            yield audio_bytes

        # Note: No final chunk processing needed - sliding window handles all tokens
        # as they come in (every 7 tokens after the first 28)

        print(f"✅ Sliding window streaming complete: {total_tokens_generated} tokens → {total_audio_chunks} audio chunks")

    def _extract_bicodec_tokens_from_text(self, text: str) -> tuple[List[int], List[int]]:
        """
        Extract BiCodec semantic and global tokens from generated text.

        Args:
            text: Generated text containing BiCodec token markers

        Returns:
            Tuple of (semantic_ids, global_ids)
        """
        semantic_matches = re.findall(r"<\|bicodec_semantic_(\d+)\|>", text)
        semantic_ids = [int(t) for t in semantic_matches]

        global_matches = re.findall(r"<\|bicodec_global_(\d+)\|>", text)
        global_ids = [int(t) for t in global_matches]

        return semantic_ids, global_ids

    async def generate_speech_stream_indic(
        self,
        speaker: str,
        text: str,
        temperature: float = DEFAULT_TEMPERATURE,
        top_k: int = DEFAULT_TOP_K,  # Added for Spark TTS parity with non-streaming
        top_p: float = DEFAULT_TOP_P,
        max_tokens: int = DEFAULT_MAX_TOKENS,
        repetition_penalty: float = DEFAULT_REPETITION_PENALTY,
        seed: Optional[int] = None,
    ) -> AsyncGenerator[bytes, None]:
        """
        Generate speech audio with sliding window streaming for Indic model.

        Yields audio chunks using overlapping windows for smooth playback.

        Args:
            speaker: Speaker name (one of 12 predefined speakers)
            text: Text to synthesize with inline emotion tags
                Examples:
                - "Hello! Welcome."
                - "<laugh> Hello there!"
                - "नमस्ते! <excited> आज का दिन बहुत अच्छा है।"
            temperature: Sampling temperature
            top_p: Nucleus sampling
            max_tokens: Max SNAC tokens to generate
            repetition_penalty: Prevent loops
            seed: Random seed for reproducibility

        Yields:
            Audio bytes (int16 PCM, 16kHz mono - BiCodec)
        """
        import time
        t_start = time.time()

        # Build prompt using Indic prompt builder
        prompt = self.prompt_builder.build_prefix(speaker, text)

        # Configure sampling (matching non-streaming pipeline parameters)
        # OPTIMIZATION: Use Spark TTS stop token, not legacy SNAC CODE_END_TOKEN_ID (128258)
        # which doesn't exist in the Spark vocab and would cause generation to run to max_tokens
        sampling_params = SamplingParams(
            temperature=temperature,
            top_k=top_k,
            top_p=top_p,
            max_tokens=max_tokens,
            min_tokens=DEFAULT_MIN_TOKENS,
            repetition_penalty=repetition_penalty,
            stop=TRAINING_STOP_TOKEN_IDS,  # "<|im_end|>" - matches non-streaming pipeline
            skip_special_tokens=False,  # Keep BiCodec tokens in output
            seed=seed,
        )

        # BiCodec token buffers (separate for semantic and global)
        semantic_buffer = []
        global_buffer = []
        processed_token_count = 0  # Track how many vLLM tokens we've parsed
        total_audio_chunks = 0

        # BiCodec streaming configuration
        EXPECTED_GLOBAL_COUNT = 32  # BiCodec uses 32 fixed global tokens (per paper)
        MIN_SEMANTIC_FOR_FIRST_CHUNK = 10  # OPTIMIZATION: Lowered from 16 (decoder min is 8, -120ms TTFB)
        # OPTIMIZATION: Increased from 24 to 48 to reduce O(n) re-decode calls by 2x.
        # At 50 TPS, 48 tokens = ~960ms audio per chunk. Cuts total decode calls roughly in half.
        DECODE_INTERVAL = 48
        CROSSFADE_MS = 50
        # OPTIMIZATION: Windowed decode - only decode last WINDOW_SIZE tokens instead of ALL.
        # BiCodec WaveGenerator has finite receptive field (~64 tokens based on kernel sizes).
        # Window must be >= receptive_field + DECODE_INTERVAL to avoid boundary artifacts.
        WINDOW_SIZE = 128  # 128 tokens covers receptive field + margin for crossfade

        # Generate tokens with vLLM (streaming)
        import uuid
        request_id = f"bicodec-stream-{uuid.uuid4().hex[:8]}-{int(time.time() * 1000000)}"

        # Reuse pre-warmed token parser (singleton, created once in __init__)
        token_parser = self.token_parser

        t_gen_start = time.time()
        results_generator = self.model.engine.generate(
            prompt=prompt,
            sampling_params=sampling_params,
            request_id=request_id,
        )

        # Track timing
        t_first_token = None
        t_first_chunk_ready = None
        t_first_chunk_decoded = None
        iteration_count = 0
        last_decode_count = 0

        # Track audio samples EMITTED to the user (not just decoded)
        # BiCodec produces ~320 samples per semantic token at 16kHz
        # CRITICAL: This tracks samples that have been YIELDED, not held in tail
        total_samples_emitted_to_user = 0
        # Keep a tail to enable crossfading between consecutive emitted chunks
        previous_chunk_tail: Optional[bytes] = None

        # Stream tokens with BiCodec extraction
        async for request_output in results_generator:
            iteration_count += 1
            t_iter = time.time()

            # Track first token received
            if t_first_token is None:
                t_first_token = t_iter


Output:
Chunk ID: 418fda
Wall time: 0.0518 seconds
Process exited with code 0
Original token count: 5352
Output:
                t_first_token = t_iter

            # Extract generated token IDs
            generated_ids = request_output.outputs[0].token_ids

            # OPTIMIZATION: Only process NEW tokens (incremental parsing)
            # Old code: tokenizer.decode(ALL) + re.findall(ALL) = O(n²) per stream
            # New code: parse only new tokens = O(1) per token
            new_token_ids = generated_ids[processed_token_count:]
            processed_token_count = len(generated_ids)

            # Parse new tokens and append to buffers (in-place modification)
            token_parser.parse_incremental(new_token_ids, semantic_buffer, global_buffer)

            # TRUE STREAMING with BiCodec two-phase generation:
            # Phase 1: Generate 32 global tokens (pre-roll)
            # Phase 2: Stream semantic tokens incrementally

            # Check if we have the 32 global tokens (pre-roll complete)
            if len(global_buffer) >= EXPECTED_GLOBAL_COUNT:
                # Pre-roll complete! Now we can stream semantic tokens
                if t_first_chunk_ready is None and len(semantic_buffer) >= MIN_SEMANTIC_FOR_FIRST_CHUNK:
                    t_first_chunk_ready = time.time()

                # Stream semantic tokens incrementally
                # Only decode if we have enough NEW semantic tokens since last decode
                semantic_count = len(semantic_buffer)
                if semantic_count >= MIN_SEMANTIC_FOR_FIRST_CHUNK:
                    # Check if we should decode (every DECODE_INTERVAL new semantic tokens)
                    if semantic_count - last_decode_count >= DECODE_INTERVAL:
                        last_decode_count = semantic_count

                        # OPTIMIZATION: Windowed decode instead of full re-decode
                        # Only decode the last WINDOW_SIZE tokens (O(1) per chunk instead of O(n))
                        # For first few chunks when total tokens < WINDOW_SIZE, decode all
                        window_global = global_buffer[:EXPECTED_GLOBAL_COUNT]

                        if len(semantic_buffer) <= WINDOW_SIZE:
                            # Early phase: decode all tokens (window not yet relevant)
                            decode_semantic = semantic_buffer
                        else:
                            # Windowed: only decode last WINDOW_SIZE tokens
                            decode_semantic = semantic_buffer[-WINDOW_SIZE:]

                        t_decode_start = time.time()
                        audio_bytes = await self.snac_decoder.decode_single_async(
                            semantic_ids=decode_semantic,
                            global_ids=window_global,
                            trim_warmup=False,
                            use_sliding_window=False,
                        )
                        t_decode_end = time.time()

                        if audio_bytes:
                            total_samples_decoded = len(audio_bytes) // 2

                            if len(semantic_buffer) <= WINDOW_SIZE:
                                # Early phase: extract new samples from full decode
                                if total_samples_decoded > total_samples_emitted_to_user:
                                    new_bytes_start = total_samples_emitted_to_user * 2
                                    new_audio_bytes = audio_bytes[new_bytes_start:]
                                else:
                                    new_audio_bytes = b""
                            else:
                                # Windowed phase: take only the tail portion of the window decode
                                # The last DECODE_INTERVAL tokens' worth of audio is the new content
                                # Each token produces ~320 samples at 16kHz/50TPS
                                new_samples_approx = DECODE_INTERVAL * 320
                                if total_samples_decoded > new_samples_approx:
                                    new_bytes_start = (total_samples_decoded - new_samples_approx) * 2
                                else:
                                    new_bytes_start = 0
                                new_audio_bytes = audio_bytes[new_bytes_start:]

                            if new_audio_bytes:
                                to_emit, previous_chunk_tail = crossfade_bytes_int16(
                                    previous_chunk_tail,
                                    new_audio_bytes,
                                    sample_rate_hz=16000,
                                    crossfade_ms=CROSSFADE_MS,
                                )

                                samples_emitted_in_this_chunk = len(to_emit) // 2
                                total_samples_emitted_to_user += samples_emitted_in_this_chunk

                                if to_emit:
                                    total_audio_chunks += 1
                                    if t_first_chunk_decoded is None:
                                        t_first_chunk_decoded = t_decode_end

                                    yield to_emit

        # Final chunk: decode any remaining semantic tokens
        # Use the same 32 global tokens (they get pooled)
        if len(semantic_buffer) > last_decode_count and len(global_buffer) >= EXPECTED_GLOBAL_COUNT:
            audio_bytes = self.snac_decoder.decode_streaming(
                semantic_ids=semantic_buffer,  # All semantic tokens
                global_ids=global_buffer[:EXPECTED_GLOBAL_COUNT],  # Use first 32 global tokens
                use_sliding_window=False,  # Get all samples
                trim_warmup=False
            )
            if audio_bytes:
                # Only yield NEW samples
                total_samples_decoded = len(audio_bytes) // 2

                if total_samples_decoded > total_samples_emitted_to_user:
                    new_bytes_start = total_samples_emitted_to_user * 2
                    new_audio_bytes = audio_bytes[new_bytes_start:]

                    to_emit, previous_chunk_tail = crossfade_bytes_int16(
                        previous_chunk_tail,
                        new_audio_bytes,
                        sample_rate_hz=16000,
                        crossfade_ms=CROSSFADE_MS,
                    )

                    samples_emitted_in_final = len(to_emit) // 2
                    total_samples_emitted_to_user += samples_emitted_in_final

                    if to_emit:
                        total_audio_chunks += 1
                        yield to_emit

        # Flush any remaining tail that was held back for crossfade
        if previous_chunk_tail:
            total_audio_chunks += 1
            tail_samples = len(previous_chunk_tail) // 2
            total_samples_emitted_to_user += tail_samples
            yield previous_chunk_tail

        # Log completion summary
        t_end = time.time()
        audio_duration_s = total_samples_emitted_to_user / 16000  # 16kHz sample rate
        ttfb_ms = (t_first_chunk_decoded - t_start) * 1000 if t_first_chunk_decoded else 0
        print(f"🎵 Streaming complete: {audio_duration_s:.2f}s audio, TTFB: {ttfb_ms:.0f}ms, {total_audio_chunks} chunks")

    async def generate_speech_stream_indic_first_chunk(
        self,
        speaker: str,
        text: str,
        temperature: float = DEFAULT_TEMPERATURE,
        top_k: int = DEFAULT_TOP_K,
        top_p: float = DEFAULT_TOP_P,
        max_tokens: int = DEFAULT_MAX_TOKENS,
        repetition_penalty: float = DEFAULT_REPETITION_PENALTY,
        seed: Optional[int] = None,
    ) -> AsyncGenerator[tuple, None]:
        """
        Generate speech for the FIRST chunk of multi-chunk text, capturing global tokens.

        This method is specifically for chunked generation: it yields (audio_bytes, global_ids)
        tuples instead of just audio_bytes. The caller captures global_ids from the first
        yield and passes them to generate_speech_stream_indic_continuation() for subsequent chunks.

        Use Case (chunked streaming with voice consistency):
            globals_captured = None
            async for audio_bytes, global_ids in pipeline.generate_speech_stream_indic_first_chunk(...):
                if globals_captured is None and global_ids:
                    globals_captured = global_ids  # Capture once
                yield audio_bytes
            # Now use globals_captured for subsequent chunks

        Args:
            speaker: Speaker name
            text: First text chunk to synthesize
            temperature: Sampling temperature
            top_k: Top-k sampling
            top_p: Nucleus sampling
            max_tokens: Max tokens to generate
            repetition_penalty: Prevent repetition
            seed: Random seed for reproducibility

        Yields:
            Tuple of (audio_bytes, global_ids)
            - audio_bytes: Raw PCM audio (int16, 16kHz)
            - global_ids: List of 32 global token IDs (populated after first decode, else empty)

        Thread Safety:
            This method is stateless and thread-safe. Each call creates its own
            request-scoped state. No global state is modified or shared.
        """
        import time
        t_start = time.time()

        # Build prompt using Indic prompt builder
        prompt = self.prompt_builder.build_prefix(speaker, text)

        # Configure sampling
        # OPTIMIZATION: Use Spark TTS stop token, not legacy SNAC CODE_END_TOKEN_ID
        sampling_params = SamplingParams(
            temperature=temperature,
            top_k=top_k,
            top_p=top_p,
            max_tokens=max_tokens,
            min_tokens=DEFAULT_MIN_TOKENS,
            repetition_penalty=repetition_penalty,
            stop=TRAINING_STOP_TOKEN_IDS,  # "<|im_end|>" - matches non-streaming pipeline
            skip_special_tokens=False,
            seed=seed,
        )

        # BiCodec token buffers
        semantic_buffer = []
        global_buffer = []
        processed_token_count = 0  # Track how many vLLM tokens we've parsed
        total_audio_chunks = 0

        # BiCodec streaming configuration
        EXPECTED_GLOBAL_COUNT = 32
        MIN_SEMANTIC_FOR_FIRST_CHUNK = 10  # OPTIMIZATION: Lowered from 16 (decoder min is 8)
        DECODE_INTERVAL = 48  # OPTIMIZATION: Increased from 24 to reduce O(n) re-decode calls
        CROSSFADE_MS = 50

        # Generate unique request ID
        import uuid
        request_id = f"bicodec-first-{uuid.uuid4().hex[:8]}-{int(time.time() * 1000000)}"

        # Reuse pre-warmed token parser (singleton, created once in __init__)
        token_parser = self.token_parser

        results_generator = self.model.engine.generate(
            prompt=prompt,
            sampling_params=sampling_params,
            request_id=request_id,
        )

        # Track state
        last_decode_count = 0
        total_samples_emitted_to_user = 0
        previous_chunk_tail: Optional[bytes] = None
        captured_globals: List[int] = []  # Will be populated once we have 32 globals

        # Stream tokens with BiCodec extraction
        async for request_output in results_generator:
            generated_ids = request_output.outputs[0].token_ids

            # OPTIMIZATION: Incremental parsing (O(1) per new token)
            new_token_ids = generated_ids[processed_token_count:]
            processed_token_count = len(generated_ids)
            token_parser.parse_incremental(new_token_ids, semantic_buffer, global_buffer)

            # Capture global tokens once we have all 32
            if not captured_globals and len(global_buffer) >= EXPECTED_GLOBAL_COUNT:
                captured_globals = global_buffer[:EXPECTED_GLOBAL_COUNT]

            # Process if we have all global tokens
            if len(global_buffer) >= EXPECTED_GLOBAL_COUNT:
                semantic_count = len(semantic_buffer)
                if semantic_count >= MIN_SEMANTIC_FOR_FIRST_CHUNK:
                    if semantic_count - last_decode_count >= DECODE_INTERVAL:
                        last_decode_count = semantic_count

                        all_semantic = semantic_buffer
                        window_global = global_buffer[:EXPECTED_GLOBAL_COUNT]

                        if self.snac_decoder.enable_batching:
                            audio_bytes = await self.snac_decoder.decode_single_async(
                                semantic_ids=all_semantic,
                                global_ids=window_global,
                                trim_warmup=False,
                                use_sliding_window=False
                            )
                        else:
                            audio_bytes = self.snac_decoder.decode_streaming(
                                semantic_ids=all_semantic,
                                global_ids=window_global,
                                use_sliding_window=False,
                                trim_warmup=False
                            )

                        if audio_bytes:
                            total_samples_decoded = len(audio_bytes) // 2

                            if total_samples_decoded > total_samples_emitted_to_user:
                                new_bytes_start = total_samples_emitted_to_user * 2
                                new_audio_bytes = audio_bytes[new_bytes_start:]

                                to_emit, previous_chunk_tail = crossfade_bytes_int16(
                                    previous_chunk_tail,
                                    new_audio_bytes,
                                    sample_rate_hz=16000,
                                    crossfade_ms=CROSSFADE_MS,
                                )

                                samples_emitted_in_this_chunk = len(to_emit) // 2
                                total_samples_emitted_to_user += samples_emitted_in_this_chunk

                                if to_emit:
                                    total_audio_chunks += 1
                                    # Yield tuple: (audio_bytes, global_ids)
                                    yield (to_emit, captured_globals)

        # Final chunk processing
        if len(semantic_buffer) > last_decode_count and len(global_buffer) >= EXPECTED_GLOBAL_COUNT:
            audio_bytes = self.snac_decoder.decode_streaming(
                semantic_ids=semantic_buffer,
                global_ids=global_buffer[:EXPECTED_GLOBAL_COUNT],
                use_sliding_window=False,
                trim_warmup=False
            )
            if audio_bytes:
                total_samples_decoded = len(audio_bytes) // 2

                if total_samples_decoded > total_samples_emitted_to_user:
                    new_bytes_start = total_samples_emitted_to_user * 2
                    new_audio_bytes = audio_bytes[new_bytes_start:]

                    to_emit, previous_chunk_tail = crossfade_bytes_int16(
                        previous_chunk_tail,
                        new_audio_bytes,
                        sample_rate_hz=16000,
                        crossfade_ms=CROSSFADE_MS,
                    )

                    samples_emitted_in_final = len(to_emit) // 2
                    total_samples_emitted_to_user += samples_emitted_in_final

                    if to_emit:
                        total_audio_chunks += 1
                        yield (to_emit, captured_globals)

        # Flush remaining tail
        if previous_chunk_tail:
            total_audio_chunks += 1
            tail_samples = len(previous_chunk_tail) // 2
            total_samples_emitted_to_user += tail_samples
            yield (previous_chunk_tail, captured_globals)

        t_end = time.time()
        audio_duration_s = total_samples_emitted_to_user / 16000
        print(f"🎵 First chunk complete: {audio_duration_s:.2f}s audio, {total_audio_chunks} chunks, captured {len(captured_globals)} globals")

    async def generate_speech_stream_indic_continuation(
        self,
        speaker: str,
        text: str,
        global_ids: List[int],
        temperature: float = DEFAULT_TEMPERATURE,
        top_k: int = DEFAULT_TOP_K,
        top_p: float = DEFAULT_TOP_P,
        max_tokens: int = DEFAULT_MAX_TOKENS,
        repetition_penalty: float = DEFAULT_REPETITION_PENALTY,
        seed: Optional[int] = None,
    ) -> AsyncGenerator[bytes, None]:
        """
        Generate speech for CONTINUATION chunks using pre-captured global tokens.

        This is the key method for voice consistency in chunked generation:
        - Uses global_ids from the first chunk to maintain identical voice
        - Model generates only semantic tokens (skips global token generation)
        - Ensures no voice drift across chunks

        CRITICAL for production:
        - global_ids is request-scoped, passed explicitly (no shared state)
        - Thread-safe: each request has its own state
        - No global caching that could cause cross-request contamination

        Args:
            speaker: Speaker name (must match first chunk)
            text: Continuation text chunk to synthesize
            global_ids: 32 global token IDs captured from first chunk
            temperature: Sampling temperature
            top_k: Top-k sampling
            top_p: Nucleus sampling
            max_tokens: Max tokens to generate
            repetition_penalty: Prevent repetition
            seed: Random seed (use same as first chunk for consistency)

        Yields:
            Audio bytes (int16 PCM, 16kHz mono)

        Raises:
            ValueError: If global_ids doesn't contain exactly 32 tokens

        Thread Safety:
            Fully thread-safe. All state is request-scoped and passed explicitly.
        """
        import time
        t_start = time.time()

        # Validate global tokens
        EXPECTED_GLOBAL_COUNT = 32
        if len(global_ids) != EXPECTED_GLOBAL_COUNT:
            raise ValueError(
                f"Expected exactly {EXPECTED_GLOBAL_COUNT} global tokens, got {len(global_ids)}. "
                f"global_ids must be captured from first chunk generation."
            )

        # Build prompt WITH pre-filled global tokens
        # This tells the model to skip global generation and go straight to semantic tokens
        prompt = self.prompt_builder.build_prefix_with_globals(speaker, text, global_ids)

        # Configure sampling
        # OPTIMIZATION: Use Spark TTS stop token, not legacy SNAC CODE_END_TOKEN_ID
        sampling_params = SamplingParams(
            temperature=temperature,
            top_k=top_k,
            top_p=top_p,
            max_tokens=max_tokens,
            min_tokens=DEFAULT_MIN_TOKENS,
            repetition_penalty=repetition_penalty,
            stop=TRAINING_STOP_TOKEN_IDS,  # "<|im_end|>" - matches non-streaming pipeline
            skip_special_tokens=False,
            seed=seed,
        )

        # BiCodec token buffers
        semantic_buffer = []
        global_buffer_unused = []  # Not used in continuation, but needed for parser API
        processed_token_count = 0  # Track how many vLLM tokens we've parsed
        total_audio_chunks = 0

        # BiCodec streaming configuration
        MIN_SEMANTIC_FOR_FIRST_CHUNK = 10  # OPTIMIZATION: Lowered from 16 (decoder min is 8)
        DECODE_INTERVAL = 48  # OPTIMIZATION: Increased from 24 to reduce O(n) re-decode calls
        CROSSFADE_MS = 50

        # Generate unique request ID
        import uuid
        request_id = f"bicodec-cont-{uuid.uuid4().hex[:8]}-{int(time.time() * 1000000)}"

        # Reuse pre-warmed token parser (singleton, created once in __init__)
        token_parser = self.token_parser

        results_generator = self.model.engine.generate(
            prompt=prompt,
            sampling_params=sampling_params,
            request_id=request_id,
        )

        # Track state
        last_decode_count = 0
        total_samples_emitted_to_user = 0
        previous_chunk_tail: Optional[bytes] = None

        # Stream tokens - model will generate ONLY semantic tokens (globals are pre-filled)
        async for request_output in results_generator:
            generated_ids = request_output.outputs[0].token_ids

            # OPTIMIZATION: Incremental parsing (O(1) per new token)
            # Note: In continuation, model only generates semantic tokens (globals pre-filled in prompt)


**Designing detailed timeline markers**


I’m editing the core pipeline now to add stage-by-stage timeline markers (`request_start -> first_batch -> llm_done -> parse -> bicodec -> done`) with per-stage deltas, so every request carries a full latency timeline.


Input:
*** Begin Patch
*** Update File: veena3modal/core/pipeline.py
@@
         perf: Dict[str, Any] = {
             "llm_prompt_token_total": 0,
             "llm_token_total": 0,
             "semantic_token_total": 0,
             "global_token_total": 0,
@@
             "wav_pack_ms": 0.0,
         }

         t_request_start = time.perf_counter()
+        timeline_marks: List[Dict[str, float]] = [{"stage": "request_start", "t_ms": 0.0, "dt_ms": 0.0}]
+        last_mark_ms = 0.0
+        perf["timeline_markers"] = timeline_marks
+        perf["timeline_request_start_ms"] = 0.0
+
+        def _mark(stage: str) -> float:
+            nonlocal last_mark_ms
+            now_ms = (time.perf_counter() - t_request_start) * 1000
+            dt_ms = now_ms - last_mark_ms
+            last_mark_ms = now_ms
+            timeline_marks.append(
+                {
+                    "stage": stage,
+                    "t_ms": float(now_ms),
+                    "dt_ms": float(dt_ms),
+                }
+            )
+            perf[f"timeline_{stage}_ms"] = float(now_ms)
+            perf[f"timeline_{stage}_delta_ms"] = float(dt_ms)
+            return float(now_ms)
+
+        def _finalize_timeline() -> None:
+            if not timeline_marks or timeline_marks[-1].get("stage") != "request_done":
+                _mark("request_done")
+
+            # Keep a compact, ordered timeline payload for downstream benchmarking.
+            perf["timeline_markers"] = list(timeline_marks)
+            perf["timeline_total_ms"] = float(perf.get("timeline_request_done_ms", 0.0))
+
+            first_batch_ms = perf.get("timeline_llm_first_batch_ms")
+            llm_done_ms = perf.get("timeline_llm_done_ms")
+            parse_done_ms = perf.get("timeline_parse_done_ms")
+            bicodec_done_ms = perf.get("timeline_bicodec_done_ms")
+            request_done_ms = perf.get("timeline_request_done_ms")
+
+            if isinstance(first_batch_ms, (int, float)):
+                perf["timeline_to_first_batch_ms"] = float(first_batch_ms)
+            if isinstance(first_batch_ms, (int, float)) and isinstance(llm_done_ms, (int, float)) and llm_done_ms >= first_batch_ms:
+                perf["timeline_first_batch_to_llm_done_ms"] = float(llm_done_ms - first_batch_ms)
+            if isinstance(llm_done_ms, (int, float)) and isinstance(request_done_ms, (int, float)) and request_done_ms >= llm_done_ms:
+                perf["timeline_post_llm_ms"] = float(request_done_ms - llm_done_ms)
+            if isinstance(parse_done_ms, (int, float)) and isinstance(bicodec_done_ms, (int, float)) and bicodec_done_ms >= parse_done_ms:
+                perf["timeline_parse_to_bicodec_done_ms"] = float(bicodec_done_ms - parse_done_ms)
+            if isinstance(bicodec_done_ms, (int, float)) and isinstance(request_done_ms, (int, float)) and request_done_ms >= bicodec_done_ms:
+                perf["timeline_bicodec_to_request_done_ms"] = float(request_done_ms - bicodec_done_ms)

         # Stage 1: prompt build
         t0 = time.perf_counter()
         prompt = self.prompt_builder.build_prefix(speaker, text)
         perf["prompt_build_ms"] = (time.perf_counter() - t0) * 1000
+        _mark("prompt_ready")
@@
         sampling_params = SamplingParams(
             temperature=temperature,
             top_k=top_k,
             top_p=top_p,
             max_tokens=max_tokens,
@@
             seed=seed,
         )
         perf["sampling_params_ms"] = (time.perf_counter() - t0) * 1000
+        _mark("sampling_ready")
@@
         prev_generated_count = 0
         prev_gpu_cumulative_s: Optional[float] = None
+        first_batch_timeline_marked = False

         t_gen_start = time.perf_counter()
         t_last_yield = t_gen_start
         async for request_output in results_generator:
@@
             perf["llm_batch_count"] += 1
             if perf["llm_batch_count"] == 1:
+                first_batch_timeline_ms = _mark("llm_first_batch")
+                first_batch_timeline_marked = True
                 logger.info(
                     "perf_first_llm_batch",
                     extra={
                         "request_id": request_id,
                         "batch_wall_ms": round(batch_wall, 3),
                         "new_tokens": new_tokens,
+                        "timeline_ms": round(first_batch_timeline_ms, 3),
                     },
                 )
@@
         perf["llm_generation_wall_ms"] = (time.perf_counter() - t_gen_start) * 1000
+        llm_done_timeline_ms = _mark("llm_done")
         logger.info(
             "perf_llm_generation_done",
             extra={
                 "request_id": request_id,
                 "llm_generation_wall_ms": round(perf["llm_generation_wall_ms"], 3),
                 "llm_batch_count": perf["llm_batch_count"],
+                "timeline_ms": round(llm_done_timeline_ms, 3),
             },
         )

         if final_output is None:
             perf["pipeline_total_ms"] = (time.perf_counter() - t_request_start) * 1000
+            _finalize_timeline()
             return None, perf
@@
         semantic_ids, global_ids = self._extract_bicodec_tokens(generated_text)
         perf["llm_parse_ms"] = (time.perf_counter() - t0) * 1000
         perf["semantic_token_total"] = len(semantic_ids)
         perf["global_token_total"] = len(global_ids)
+        parse_done_timeline_ms = _mark("parse_done")
         logger.info(
             "perf_token_parse_done",
             extra={
                 "request_id": request_id,
                 "parse_ms": round(perf["llm_parse_ms"], 3),
                 "semantic_tokens": perf["semantic_token_total"],
                 "global_tokens": perf["global_token_total"],
+                "timeline_ms": round(parse_done_timeline_ms, 3),
             },
         )
@@
         if not semantic_ids or not global_ids:
             perf["pipeline_total_ms"] = (time.perf_counter() - t_request_start) * 1000
+            _finalize_timeline()
             return None, perf

         # Stage 4: validate BiCodec tokens
         t0 = time.perf_counter()
         if not self.bicodec_decoder.validate_tokens(semantic_ids, global_ids):
             perf["token_validation_ms"] = (time.perf_counter() - t0) * 1000
+            _mark("validation_done")
             perf["pipeline_total_ms"] = (time.perf_counter() - t_request_start) * 1000
+            _finalize_timeline()
             return None, perf
         perf["token_validation_ms"] = (time.perf_counter() - t0) * 1000
+        _mark("validation_done")
@@
         decode_wall_ms = (time.perf_counter() - decode_t0) * 1000
         perf["bicodec_decode_wall_ms"] = float(decode_wall_ms)
         perf["bicodec_decode_gpu_ms"] = float(max(0.0, decode_gpu_ms))
         perf["bicodec_decode_cpu_ms"] = float(max(0.0, decode_wall_ms - decode_gpu_ms))
+        bicodec_done_timeline_ms = _mark("bicodec_done")
         logger.info(
             "perf_bicodec_decode_done",
             extra={
                 "request_id": request_id,
                 "decode_wall_ms": round(perf["bicodec_decode_wall_ms"], 3),
                 "decode_gpu_ms": round(perf["bicodec_decode_gpu_ms"], 3),
                 "decode_cpu_ms": round(perf["bicodec_decode_cpu_ms"], 3),
+                "timeline_ms": round(bicodec_done_timeline_ms, 3),
             },
         )

         if audio_bytes is None:
             perf["pipeline_total_ms"] = (time.perf_counter() - t_request_start) * 1000
+            _finalize_timeline()
             return None, perf

         # Stage 6: WAV header
         t0 = time.perf_counter()
         wav_bytes = add_wav_header(audio_bytes, sample_rate=AUDIO_SAMPLE_RATE)
         perf["wav_pack_ms"] = (time.perf_counter() - t0) * 1000
+        _mark("wav_done")

         perf["pipeline_total_ms"] = (time.perf_counter() - t_request_start) * 1000
+        _finalize_timeline()
         logger.info(
             "perf_non_stream_done",
             extra={
                 "request_id": request_id,
                 "pipeline_total_ms": round(perf["pipeline_total_ms"], 3),
                 "llm_token_total": perf.get("llm_token_total"),
+                "timeline_total_ms": round(float(perf.get("timeline_total_ms", 0.0)), 3),
             },
         )
         return wav_bytes, perf
*** End Patch


Output: {"output":"Success. Updated the following files:\nM veena3modal/core/pipeline.py\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


**Removing unused variable**


Input:
*** Begin Patch
*** Update File: veena3modal/core/pipeline.py
@@
         prev_generated_count = 0
         prev_gpu_cumulative_s: Optional[float] = None
-        first_batch_timeline_marked = False

         t_gen_start = time.perf_counter()
         t_last_yield = t_gen_start
         async for request_output in results_generator:
@@
             perf["llm_batch_count"] += 1
             if perf["llm_batch_count"] == 1:
                 first_batch_timeline_ms = _mark("llm_first_batch")
-                first_batch_timeline_marked = True
                 logger.info(
*** End Patch


Output: {"output":"Success. Updated the following files:\nM veena3modal/core/pipeline.py\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


**Patching FastAPI for extra keys**


I’ve added the timeline fields to core generation. Next I’m exposing these markers through the API perf payload/headers so both HTTP and runtime benchmark scripts can aggregate them directly.


Input:
*** Begin Patch
*** Update File: veena3modal/api/fastapi_app.py
@@
     header_map = {
         "x-generation-ms": "generation_ms",
         "x-request-inflight": "request_inflight",
+        "x-timeline-first-batch-ms": "timeline_llm_first_batch_ms",
+        "x-timeline-llm-done-ms": "timeline_llm_done_ms",
+        "x-timeline-bicodec-done-ms": "timeline_bicodec_done_ms",
+        "x-timeline-total-ms": "timeline_total_ms",
         "x-llm-token-total": "llm_token_total",
         "x-llm-batch-count": "llm_batch_count",
@@
     extra_detail_keys = (
         "prompt_build_ms",
         "sampling_params_ms",
+        "timeline_markers",
+        "timeline_prompt_ready_ms",
+        "timeline_sampling_ready_ms",
+        "timeline_llm_first_batch_ms",
+        "timeline_llm_done_ms",
+        "timeline_parse_done_ms",
+        "timeline_validation_done_ms",
+        "timeline_bicodec_done_ms",
+        "timeline_wav_done_ms",
+        "timeline_request_done_ms",
+        "timeline_to_first_batch_ms",
+        "timeline_first_batch_to_llm_done_ms",
+        "timeline_post_llm_ms",
+        "timeline_parse_to_bicodec_done_ms",
+        "timeline_bicodec_to_request_done_ms",
+        "timeline_total_ms",
         "llm_prefill_wall_ms",
         "llm_prefill_gpu_ms",
         "llm_generation_wall_ms",
*** End Patch


Output: {"output":"Success. Updated the following files:\nM veena3modal/api/fastapi_app.py\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


**Planning header parsing patch**


sed -n '280,430p' scripts/stress_test_optimized_local.py


Output:
Chunk ID: 60be7d
Wall time: 0.0512 seconds
Process exited with code 0
Original token count: 1380
Output:
        return None
    try:
        if "." in v:
            return float(v)
        return float(int(v))
    except ValueError:
        try:
            return float(v)
        except ValueError:
            return None

def _normalize_time_ms(raw: Optional[str]) -> float:
    value = _coerce_number(raw)
    if value is None:
        return 0.0
    return float(value)

def _parse_perf_headers(headers: Any) -> Dict[str, Any]:
    """Parse server timing headers into numeric metric fields."""
    metrics: Dict[str, Any] = {}

    # Explicit compact payload.
    detail_payload = headers.get("x-perf-details") if hasattr(headers, "get") else None
    if detail_payload:
        try:
            parsed = json.loads(detail_payload)
            if isinstance(parsed, dict):
                metrics.update(parsed)
        except Exception:
            pass

    header_map = {
        "x-llm-token-total": "llm_token_total",
        "x-llm-batch-count": "llm_batch_count",
        "x-llm-batch-wall-ms": "llm_batch_wall_ms_total",
        "x-llm-batch-gpu-ms": "llm_batch_gpu_ms_total",
        "x-llm-batch-wall-ms-min": "llm_batch_wall_ms_min",
        "x-llm-batch-wall-ms-max": "llm_batch_wall_ms_max",
        "x-llm-batch-wall-ms-p50": "llm_batch_wall_ms_p50",
        "x-llm-batch-gpu-ms-min": "llm_batch_gpu_ms_min",
        "x-llm-batch-gpu-ms-max": "llm_batch_gpu_ms_max",
        "x-llm-batch-gpu-ms-p50": "llm_batch_gpu_ms_p50",
        "x-llm-decode-wall-ms": "llm_decode_wall_ms_total",
        "x-llm-decode-gpu-ms": "llm_decode_gpu_ms_total",
        "x-llm-decode-wall-ms-min": "llm_decode_wall_ms_min",
        "x-llm-decode-wall-ms-max": "llm_decode_wall_ms_max",
        "x-llm-decode-wall-ms-p50": "llm_decode_wall_ms_p50",
        "x-llm-decode-gpu-ms-min": "llm_decode_gpu_ms_min",
        "x-llm-decode-gpu-ms-max": "llm_decode_gpu_ms_max",
        "x-llm-decode-gpu-ms-p50": "llm_decode_gpu_ms_p50",
        "x-llm-decode-calls": "decode_calls",
        "x-llm-decode-cpu-ms": "llm_decode_cpu_ms",
        "x-llm-parse-ms": "llm_parse_ms",
        "x-llm-parse-avg-ms": "llm_parse_ms_avg",
        "x-llm-tokens-per-batch-min": "tokens_per_batch_min",
        "x-llm-tokens-per-batch-max": "tokens_per_batch_max",
        "x-llm-tokens-per-batch-p50": "tokens_per_batch_p50",
        "x-generation-ms": "generation_ms",
        "x-request-inflight": "request_inflight",
        "x-llm-text-chunked": "text_chunked",
        "x-llm-chunks-processed": "chunks_processed",
        "x-llm-time-per-token-ms": "llm_time_per_token_ms",
        "x-llm-time-per-batch-wall-ms": "llm_time_per_batch_wall_ms",
        "x-llm-time-per-batch-gpu-ms": "llm_time_per_batch_gpu_ms",
        "x-llm-parse-avg-ms": "llm_parse_ms_avg",
        "x-bicodec-decode-wall-ms": "bicodec_decode_wall_ms",
        "x-bicodec-decode-gpu-ms": "bicodec_decode_gpu_ms",
        "x-bicodec-decode-cpu-ms": "bicodec_decode_cpu_ms",
    }

    if hasattr(headers, "items"):
        for raw_key, raw_val in headers.items():
            lk = str(raw_key).lower()
            target_key = header_map.get(lk)
            if target_key is None:
                continue
            v = _coerce_number(raw_val)
            if v is not None:
                if target_key in {
                    "text_chunked",
                }:
                    metrics[target_key] = bool(v)
                else:
                    metrics[target_key] = int(v) if float(v).is_integer() else float(v)

    if "llm_token_total" in metrics and "generation_ms" in metrics and metrics["llm_token_total"]:
        try:
            metrics["llm_time_per_token_ms"] = float(metrics["generation_ms"]) / float(metrics["llm_token_total"])
        except Exception:
            pass

    if "llm_batch_count" in metrics and "generation_ms" in metrics and metrics["llm_batch_count"]:
        try:
            metrics["llm_time_per_batch_wall_ms"] = float(metrics["generation_ms"]) / float(metrics["llm_batch_count"])
        except Exception:
            pass

    if "llm_batch_count" in metrics and "llm_batch_gpu_ms_total" in metrics and metrics["llm_batch_count"]:
        try:
            metrics["llm_time_per_batch_gpu_ms"] = float(metrics["llm_batch_gpu_ms_total"]) / float(metrics["llm_batch_count"])
        except Exception:
            pass

    return metrics

async def make_request(
    client: httpx.AsyncClient,
    base_url: str,
    text: str,
    speaker: str,
    chunking: bool,
    max_tokens: int,
    stream: bool = False,
    timeout: float = 120.0,
) -> RequestResult:
    url = f"{base_url}{GENERATE_PATH}"
    payload = {
        "text": text,
        "speaker": speaker,
        "stream": stream,
        "chunking": chunking,
        "format": "wav",
        "max_tokens": max_tokens,
    }

    start = time.time()
    ttfb_time = 0.0

    try:
        if stream:
            async with client.stream(
                "POST",
                url,
                json=payload,
                headers={"Content-Type": "application/json"},
                timeout=timeout,
            ) as response:
                first = True
                total_bytes = 0
                async for chunk in response.aiter_bytes():
                    if first:
                        ttfb_time = (time.time() - start) * 1000
                        first = False
                    total_bytes += len(chunk)

                latency = (time.time() - start) * 1000
                timing = _parse_perf_headers(response.headers)


sed -n '620,820p' scripts/stress_test_optimized_local.py