Output:
Chunk ID: 6dde12
Wall time: 5.0009 seconds
Process running with session ID 31949
Original token count: 603
Output:

Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  11%|█▏        | 4/35 [00:00<00:00, 32.43it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  23%|██▎       | 8/35 [00:00<00:00, 34.01it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  34%|███▍      | 12/35 [00:00<00:00, 35.71it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  46%|████▌     | 16/35 [00:00<00:00, 36.07it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  57%|█████▋    | 20/35 [00:00<00:00, 36.57it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  69%|██████▊   | 24/35 [00:00<00:00, 36.34it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  80%|████████  | 28/35 [00:00<00:00, 36.07it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  91%|█████████▏| 32/35 [00:00<00:00, 34.54it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 35/35 [00:01<00:00, 34.51it/s]
(EngineCore_DP0 pid=4092599)
Capturing CUDA graphs (decode, FULL):   0%|          | 0/19 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL):  16%|█▌        | 3/19 [00:00<00:00, 28.57it/s]
Capturing CUDA graphs (decode, FULL):  37%|███▋      | 7/19 [00:00<00:00, 32.50it/s]
Capturing CUDA graphs (decode, FULL):  58%|█████▊    | 11/19 [00:00<00:00, 33.58it/s]
Capturing CUDA graphs (decode, FULL):  79%|███████▉  | 15/19 [00:00<00:00, 34.37it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 35.32it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 34.26it/s]
(EngineCore_DP0 pid=4092599) INFO 02-18 00:10:27 [gpu_model_runner.py:5063] Graph capturing finished in 2 secs, took 0.28 GiB
(EngineCore_DP0 pid=4092599) INFO 02-18 00:10:27 [core.py:272] init engine (profile, create kv cache, warmup model) took 10.85 seconds
[2026-02-18 00:10:29] INFO tts_runtime.py:192: Initializing prompt builder...
[2026-02-18 00:10:29] INFO tts_runtime.py:199: Initializing BiCodec decoder...
/home/ubuntu/.local/lib/python3.10/site-packages/torch/nn/utils/weight_norm.py:144: FutureWarning: `torch.nn.utils.weight_norm` is deprecated in favor of `torch.nn.utils.parametrizations.weight_norm`.
  WeightNorm.apply(module, name, dim)


# Collaboration Mode: Default

You are now in Default mode. Any previous instructions for other modes (e.g. Plan mode) are no longer active.

Your active mode changes only when new developer instructions with a different `<collaboration_mode>...</collaboration_mode>` change it; user requests or tool descriptions do not change mode by themselves. Known mode names are Default and Plan.

## request_user_input availability

The `request_user_input` tool is unavailable in Default mode. If you call it while in Default mode, it will return an error.

If a decision is necessary and cannot be discovered from local context, ask the user directly. However, in Default mode you should strongly prefer executing the user's request rather than stopping to ask questions.

Summary: auto

## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) When `SKILL.md` references relative paths (e.g., `scripts/foo.py`), resolve them relative to the skill directory listed above first, and only consider other paths if needed.
  3) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  4) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  5) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.


**Planning persistent polling**


Baseline confirmation run is still in progress; I’ll report the exact `96c` delta immediately after it completes and then run the matching `no-stats` confirmation so we can lock a production-safe winner.


Output:
Chunk ID: d187de
Wall time: 5.0016 seconds
Process running with session ID 31949
Original token count: 127
Output:
[2026-02-18 00:10:37] INFO tts_runtime.py:206: Initializing TTS pipeline...
[2026-02-18 00:10:37] INFO tts_runtime.py:217: Initializing streaming pipeline...
[2026-02-18 00:10:37] INFO long_text_processor.py:112: LongTextProcessor initialized
[2026-02-18 00:10:37] INFO tts_runtime.py:274: ✅ TTS runtime initialized in 33479ms
[2026-02-18 00:10:37] INFO tts_runtime.py:275:    Model version: LLM
[2026-02-18 00:10:37] INFO tts_runtime.py:286: Speaker globals cache startup precompute skipped (disabled)


Summary: auto


Output:
Chunk ID: 25acb9
Wall time: 1.3521 seconds
Process exited with code 0
Original token count: 484
Output:
✅ vLLM engine initialized successfully!

🎉 Spark TTS Model ready for inference!

🎵 Loading BiCodec audio tokenizer from /home/ubuntu/veenaModal/models/spark_tts_4speaker...
   Device: cuda
Missing tensor: mel_transformer.spectrogram.window
Missing tensor: mel_transformer.mel_scale.fb
✅ BiCodec decoder initialized (sample rate: 16000Hz)
🚀 SparkTTSPipeline initialized
🌊 Veena3SlidingWindowPipeline initialized (sliding window: 28 tokens)

=== concurrency=96 requests=384 ===
success=384/384 (100%)
wall=8.40s throughput=45.71 req/s
latency_ms avg=2034 p50=2158 p95=2169 max=2179
generation_ms: avg=1514.91 p50=1511.00 p95=2035.00 max=2103.00
timeline_llm_first_batch_ms: avg=55.30 p50=37.92 p95=76.16 max=1217.88
timeline_llm_done_ms: avg=1503.31 p50=1499.97 p95=2025.64 max=2092.87
timeline_parse_done_ms: avg=1503.38 p50=1500.04 p95=2025.71 max=2092.94
timeline_bicodec_done_ms: avg=1515.33 p50=1511.56 p95=2035.77 max=2103.29
timeline_total_ms: avg=1515.35 p50=1511.58 p95=2035.79 max=2103.31
llm_batch_wall_ms_total: avg=1503.16 p50=1499.83 p95=2025.50 max=2092.70
llm_decode_wall_ms_total: avg=1448.00 p50=1461.31 p95=1990.65 max=2066.46
llm_parse_ms: avg=0.06 p50=0.06 p95=0.07 max=0.10
llm_time_per_token_ms: avg=11.74 p50=11.72 p95=15.82 max=16.35
bicodec_decode_wall_ms: avg=11.90 p50=11.93 p95=14.69 max=19.68
bicodec_decode_gpu_ms: avg=11.83 p50=11.85 p95=14.60 max=19.58
latency_over_timeline_ms: avg=518.69 p50=482.15 p95=1048.44 max=1175.59
latency_over_generation_ms: avg=519.13 p50=482.13 p95=1048.38 max=1176.36
timeline_share: llm=99.2% bicodec=0.8% other=0.0%
timeline_sample_ms: first_batch=33.1 llm_done=1599.9 parse_done=1599.9 bicodec_done=1612.3 total=1612.3
gpu: util_avg=75% util_max=100% mem_avg=23602MB mem_max=23603MB

results saved: /tmp/runtime_c96_baseline_confirm.json
ERROR 02-18 00:10:47 [core_client.py:605] Engine core proc EngineCore_DP0 died unexpectedly, shutting down client.


**Running test command**


python3 /home/ubuntu/veenaModal/scripts/stress_test_runtime_detailed.py --levels 96 --text short --max-tokens 128 --timeout 180 --requests-multiplier 4 --min-requests 8 --gpu-memory 0.25 --num-engines 1 --disable-engine-stats-logs --output /tmp/runtime_c96_nostats_confirm.json


Output:
Chunk ID: 5753e3
Wall time: 1.0011 seconds
Process running with session ID 50348
Original token count: 0
Output:


Output:
Chunk ID: 2011c1
Wall time: 5.0007 seconds
Process running with session ID 50348
Original token count: 1714
Output:
[2026-02-18 00:11:09] INFO tts_runtime.py:183: Loading SparkTTS model with vLLM...
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
🚀 Initializing Spark TTS Model
📁 Model path: /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
🗣  Speakers: 12 predefined (lipakshi, vardan, reet, Nandini, krishna, anika, adarsh, Nilay, Aarvi, Asha, Bittu, Mira)
🎭 Emotions: 10 tags in [bracket] format

📝 Loading tokenizer from model...
✅ Tokenizer loaded: 165168 tokens

🔍 Validating 12 speakers for Spark TTS...
  lipakshi → <|speaker_0|>: 7 tokens
  vardan → <|speaker_1|>: 7 tokens
  reet → <|speaker_2|>: 7 tokens
  Nandini → <|speaker_3|>: 7 tokens
  krishna → <|speaker_4|>: 7 tokens
  anika → <|speaker_5|>: 7 tokens
  adarsh → <|speaker_6|>: 7 tokens
  Nilay → <|speaker_7|>: 7 tokens
  Aarvi → <|speaker_8|>: 7 tokens
  Asha → <|speaker_9|>: 7 tokens
  Bittu → <|speaker_10|>: 8 tokens
  Mira → <|speaker_11|>: 8 tokens
✅ All 12 speakers validated with mapping

🔍 Validating 10 emotion tags for Spark TTS...
✅ All 10 emotion tags validated

📋 Sample emotion tags:
  [angry]: 1 tokens
  [curious]: 1 tokens
  [excited]: 1 tokens
  [giggle]: 1 tokens
  [laughs harder]: 1 tokens
  ...

🔧 Initializing vLLM engine with optimizations...
  - dtype: bfloat16
  - max_model_len: 4096
  - gpu_memory_utilization: 0.25
  - enable_prefix_caching: True
  - enforce_eager (disable CUDA graphs): False
INFO 02-18 00:11:10 [model.py:541] Resolved architecture: Qwen2ForCausalLM
INFO 02-18 00:11:10 [model.py:1561] Using max model len 4096
INFO 02-18 00:11:10 [scheduler.py:226] Chunked prefill is enabled with max_num_batched_tokens=4096.
INFO 02-18 00:11:10 [vllm.py:624] Asynchronous scheduling is enabled.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(EngineCore_DP0 pid=4092939) INFO 02-18 00:11:11 [core.py:96] Initializing a V1 LLM engine (v0.15.1) with config: model='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', speculative_config=None, tokenizer='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [4096], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 256, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': True}, 'local_cache_dir': None, 'static_all_moe_layers': []}
(EngineCore_DP0 pid=4092939) INFO 02-18 00:11:13 [parallel_state.py:1212] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://216.81.248.184:55643 backend=nccl
(EngineCore_DP0 pid=4092939) INFO 02-18 00:11:13 [parallel_state.py:1423] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
(EngineCore_DP0 pid=4092939) INFO 02-18 00:11:13 [gpu_model_runner.py:4033] Starting to load model /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM...
(EngineCore_DP0 pid=4092939) INFO 02-18 00:11:14 [cuda.py:364] Using FLASH_ATTN attention backend out of potential backends: ('FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION')
(EngineCore_DP0 pid=4092939) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
(EngineCore_DP0 pid=4092939) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
(EngineCore_DP0 pid=4092939)
Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
(EngineCore_DP0 pid=4092939)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.97it/s]
(EngineCore_DP0 pid=4092939)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.97it/s]
(EngineCore_DP0 pid=4092939)
(EngineCore_DP0 pid=4092939) INFO 02-18 00:11:15 [default_loader.py:291] Loading weights took 0.59 seconds
(EngineCore_DP0 pid=4092939) INFO 02-18 00:11:16 [gpu_model_runner.py:4130] Model loading took 0.95 GiB memory and 1.135077 seconds


The current TTS application here is an implementation in Modal labs. Which has autoscaling and other rules self scaling rules and everything suitable for my target audience. Now, i need you to create a local entry point or implementation point. then we will plan for optimising the pipeline, then we will discuss on stress testing, then we will discuss on scaling laws post optimisations locally. @veena3modal/app.py

the above part is performed by a different agent. and benchmarks are made. Now can you do the benchmarks on this newly optimized path to understand the concurrency and stress testing.

Now i need you to capture detailed time logs for each request happening in the GPU. like what happens with 1req, time per LLM token, time per batch decoding, and everything else. detailed time about where we are spending each minute in the GPU per generation.

and when you have the details for per request. you can obviously add more logging to get details of concurrency. so you understand what's the bottleneck for the peak concurrency. and then we will decide on effective methods in optimising this TTS engine.

yes. and also this time, before the benchmarks, can you also like put some timeline markers to spot the latency differences and the time spent, to observe. i feel like a 0.5B auto-regressive model and a decoder at some 500M params, in a 80GB A100 GPU should do better than these. So, i'd like to see where we are spending the max time for each stream. what do you think. we can do better ?

yes. onto next steps.

;et

<permissions instructions>
Filesystem sandboxing defines which files can be read or written. `sandbox_mode` is `danger-full-access`: No filesystem sandboxing - all commands are permitted. Network access is enabled.
Approval policy is currently never. Do not provide the `sandbox_permissions` for any reason, commands will be rejected.
</permissions instructions>

<collaboration_mode># Collaboration Mode: Default

You are now in Default mode. Any previous instructions for other modes (e.g. Plan mode) are no longer active.

Your active mode changes only when new developer instructions with a different `<collaboration_mode>...</collaboration_mode>` change it; user requests or tool descriptions do not change mode by themselves. Known mode names are Default and Plan.

## request_user_input availability

The `request_user_input` tool is unavailable in Default mode. If you call it while in Default mode, it will return an error.

If a decision is necessary and cannot be discovered from local context, ask the user directly. However, in Default mode you should strongly prefer executing the user's request rather than stopping to ask questions.
</collaboration_mode>

# AGENTS.md instructions for /home/ubuntu/veenaModal

<INSTRUCTIONS>
## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) When `SKILL.md` references relative paths (e.g., `scripts/foo.py`), resolve them relative to the skill directory listed above first, and only consider other paths if needed.
  3) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  4) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  5) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.
</INSTRUCTIONS>

<environment_context>
  <cwd>/home/ubuntu/veenaModal</cwd>
  <shell>bash</shell>
</environment_context>

let's go towards achieving the biggest optimisation in decoding/schediling/queing behaviours which increases per GPU throughput and reduces latency overheads and has best overall throughput also with higher concurrency


**Acknowledging optimization request and polling benchmark**


I’m continuing from the benchmark tuning run and first I’ll finish the pending high-concurrency confirmation job, then I’ll compare the results and apply the next decoding/scheduling/queueing optimization directly in the runtime path.


Output:
Chunk ID: 5b7f58
Wall time: 0.0506 seconds
Process exited with code 0
Original token count: 1471
Output:
(EngineCore_DP0 pid=4092939) INFO 02-18 00:11:20 [backends.py:812] Using cache directory: /home/ubuntu/.cache/vllm/torch_compile_cache/f3ff13511c/rank_0_0/backbone for vLLM's torch.compile
(EngineCore_DP0 pid=4092939) INFO 02-18 00:11:20 [backends.py:872] Dynamo bytecode transform time: 4.22 s
(EngineCore_DP0 pid=4092939) INFO 02-18 00:11:23 [backends.py:267] Directly load the compiled graph(s) for compile range (1, 4096) from the cache, took 0.669 s
(EngineCore_DP0 pid=4092939) INFO 02-18 00:11:23 [monitor.py:34] torch.compile takes 4.89 s in total
(EngineCore_DP0 pid=4092939) INFO 02-18 00:11:24 [gpu_worker.py:356] Available KV cache memory: 18.06 GiB
(EngineCore_DP0 pid=4092939) INFO 02-18 00:11:24 [kv_cache_utils.py:1307] GPU KV cache size: 1,577,712 tokens
(EngineCore_DP0 pid=4092939) INFO 02-18 00:11:24 [kv_cache_utils.py:1312] Maximum concurrency for 4,096 tokens per request: 385.18x
(EngineCore_DP0 pid=4092939)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   0%|          | 0/35 [00:00<?, ?it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  11%|█▏        | 4/35 [00:00<00:00, 32.24it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  23%|██▎       | 8/35 [00:00<00:00, 35.04it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  34%|███▍      | 12/35 [00:00<00:00, 36.15it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  46%|████▌     | 16/35 [00:00<00:00, 36.35it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  57%|█████▋    | 20/35 [00:00<00:00, 36.07it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  69%|██████▊   | 24/35 [00:00<00:00, 35.39it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  80%|████████  | 28/35 [00:00<00:00, 34.73it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  91%|█████████▏| 32/35 [00:00<00:00, 32.95it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 35/35 [00:01<00:00, 33.73it/s]
(EngineCore_DP0 pid=4092939)
Capturing CUDA graphs (decode, FULL):   0%|          | 0/19 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL):  16%|█▌        | 3/19 [00:00<00:00, 28.10it/s]
Capturing CUDA graphs (decode, FULL):  37%|███▋      | 7/19 [00:00<00:00, 32.16it/s]
Capturing CUDA graphs (decode, FULL):  58%|█████▊    | 11/19 [00:00<00:00, 33.88it/s]
Capturing CUDA graphs (decode, FULL):  79%|███████▉  | 15/19 [00:00<00:00, 34.15it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 34.80it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 33.92it/s]
(EngineCore_DP0 pid=4092939) INFO 02-18 00:11:27 [gpu_model_runner.py:5063] Graph capturing finished in 2 secs, took 0.28 GiB
(EngineCore_DP0 pid=4092939) INFO 02-18 00:11:27 [core.py:272] init engine (profile, create kv cache, warmup model) took 10.95 seconds
[2026-02-18 00:11:28] INFO tts_runtime.py:192: Initializing prompt builder...
[2026-02-18 00:11:28] INFO tts_runtime.py:199: Initializing BiCodec decoder...
/home/ubuntu/.local/lib/python3.10/site-packages/torch/nn/utils/weight_norm.py:144: FutureWarning: `torch.nn.utils.weight_norm` is deprecated in favor of `torch.nn.utils.parametrizations.weight_norm`.
  WeightNorm.apply(module, name, dim)
[2026-02-18 00:11:35] INFO tts_runtime.py:206: Initializing TTS pipeline...
[2026-02-18 00:11:35] INFO tts_runtime.py:217: Initializing streaming pipeline...
[2026-02-18 00:11:35] INFO long_text_processor.py:112: LongTextProcessor initialized
[2026-02-18 00:11:35] INFO tts_runtime.py:274: ✅ TTS runtime initialized in 31726ms
[2026-02-18 00:11:35] INFO tts_runtime.py:275:    Model version: LLM
[2026-02-18 00:11:35] INFO tts_runtime.py:286: Speaker globals cache startup precompute skipped (disabled)
✅ vLLM engine initialized successfully!

🎉 Spark TTS Model ready for inference!

🎵 Loading BiCodec audio tokenizer from /home/ubuntu/veenaModal/models/spark_tts_4speaker...
   Device: cuda
Missing tensor: mel_transformer.spectrogram.window
Missing tensor: mel_transformer.mel_scale.fb
✅ BiCodec decoder initialized (sample rate: 16000Hz)
🚀 SparkTTSPipeline initialized
🌊 Veena3SlidingWindowPipeline initialized (sliding window: 28 tokens)

=== concurrency=96 requests=384 ===
success=384/384 (100%)
wall=8.16s throughput=47.08 req/s
latency_ms avg=1922 p50=2139 p95=2197 max=2197
generation_ms: avg=1478.45 p50=1429.00 p95=2033.00 max=2147.00
timeline_llm_first_batch_ms: avg=122.13 p50=37.89 p95=933.85 max=1255.85
timeline_llm_done_ms: avg=1466.57 p50=1419.07 p95=2023.34 max=2134.62
timeline_parse_done_ms: avg=1466.64 p50=1419.13 p95=2023.40 max=2134.68
timeline_bicodec_done_ms: avg=1478.91 p50=1429.47 p95=2033.58 max=2147.41
timeline_total_ms: avg=1478.93 p50=1429.48 p95=2033.60 max=2147.42
llm_batch_wall_ms_total: avg=1466.42 p50=1418.94 p95=2023.20 max=2134.47
llm_decode_wall_ms_total: avg=1351.47 p50=1368.93 p95=1989.75 max=2111.02
llm_parse_ms: avg=0.06 p50=0.06 p95=0.07 max=0.27
llm_time_per_token_ms: avg=11.46 p50=11.09 p95=15.81 max=16.68
bicodec_decode_wall_ms: avg=12.23 p50=12.44 p95=15.15 max=17.92
bicodec_decode_gpu_ms: avg=12.15 p50=12.37 p95=15.09 max=17.79
latency_over_timeline_ms: avg=442.60 p50=371.03 p95=1022.48 max=1163.39
latency_over_generation_ms: avg=443.08 p50=371.01 p95=1023.11 max=1164.04
timeline_share: llm=99.2% bicodec=0.8% other=0.0%
timeline_sample_ms: first_batch=36.7 llm_done=1141.5 parse_done=1141.5 bicodec_done=1154.5 total=1154.5
gpu: util_avg=79% util_max=100% mem_avg=23601MB mem_max=23601MB

results saved: /tmp/runtime_c96_nostats_confirm.json
ERROR 02-18 00:11:45 [core_client.py:605] Engine core proc EngineCore_DP0 died unexpectedly, shutting down client.


**Identifying queueing bottleneck and next optimizations**


The pending high-concurrency confirmation finished: disabling vLLM stats logs consistently improved throughput at concurrency 96 (from ~45.7 to ~47.1 req/s) and reduced average latency overhead. Next I’m inspecting the runtime queue/scheduler path to remove the remaining per-request queue delay, which is now the biggest latency bucket outside decode.


cd /home/ubuntu/veenaModal && rg -n "latency_over_timeline|timeline_total|generation_ms|gpu_semaphore|semaphore|queue|schedule|batch|first_batch|token" scripts/stress_test_runtime_detailed.py veena3modal/services/tts_runtime.py veena3modal/core/multi_engine.py veena3modal/app.py veena3modal/local_server.py


cd /home/ubuntu/veenaModal && sed -n '1,260p' veena3modal/services/tts_runtime.py


cd /home/ubuntu/veenaModal && sed -n '260,620p' veena3modal/services/tts_runtime.py


Output:
Chunk ID: 325dbb
Wall time: 0.0517 seconds
Process exited with code 0
Original token count: 3185
Output:
veena3modal/local_server.py:28:    HF_TOKEN                - HuggingFace token for private models
veena3modal/local_server.py:80:    hf_token: str | None = None,
veena3modal/local_server.py:86:    1. Try private repo (BayAreaBoys/spark_tts_4speaker) with HF token
veena3modal/local_server.py:115:            token=hf_token,
veena3modal/local_server.py:130:                token=hf_token,
veena3modal/local_server.py:163:        BiCodec/      <- Audio tokenizer/decoder
veena3modal/local_server.py:198:    hf_token: str | None = None,
veena3modal/local_server.py:200:    max_num_batched_tokens: int | None = None,
veena3modal/local_server.py:228:    if max_num_batched_tokens is not None:
veena3modal/local_server.py:229:        logger.info(f"  Max batched tokens: {max_num_batched_tokens}")
veena3modal/local_server.py:249:        hf_token=hf_token,
veena3modal/local_server.py:253:        max_num_batched_tokens=max_num_batched_tokens,
veena3modal/local_server.py:311:        "--hf-token",
veena3modal/local_server.py:314:        help="HuggingFace token (default: from HF_TOKEN env var)",
veena3modal/local_server.py:344:        "--max-num-batched-tokens",
veena3modal/local_server.py:347:        help="Override vLLM max_num_batched_tokens scheduler limit (default: model config)",
veena3modal/local_server.py:353:        help="Override vLLM max_num_seqs scheduler limit (default: vLLM auto)",
veena3modal/local_server.py:389:        help="Precompute speaker global tokens at startup (streaming-only optimization, experimental)",
veena3modal/local_server.py:497:            hf_token=args.hf_token,
veena3modal/local_server.py:510:        hf_token=args.hf_token,
veena3modal/local_server.py:512:        max_num_batched_tokens=args.max_num_batched_tokens,
veena3modal/core/multi_engine.py:97:        .tokenizer -> shared tokenizer (same across all engines)
veena3modal/core/multi_engine.py:109:        # Shared tokenizer (identical across all engines)
veena3modal/core/multi_engine.py:110:        self.tokenizer = self._primary.tokenizer
veena3modal/core/multi_engine.py:113:        self.hf_token = self._primary.hf_token
veena3modal/core/multi_engine.py:122:    def get_tokenizer(self):
veena3modal/core/multi_engine.py:123:        return self.tokenizer
veena3modal/core/multi_engine.py:132:    hf_token: str = None,
veena3modal/core/multi_engine.py:148:        hf_token: HuggingFace token
veena3modal/core/multi_engine.py:164:            hf_token=hf_token,
veena3modal/app.py:7:- Per-container concurrency (vLLM batching)
veena3modal/app.py:148:    # Prevents CPU throttling during vLLM scheduler + streaming loop
veena3modal/app.py:179:# vLLM continuous batching handles concurrent requests efficiently
veena3modal/app.py:341:        "sparktts.models.audio_tokenizer",
veena3modal/services/tts_runtime.py:35:        The vLLM engine handles internal batching and scheduling.
veena3modal/services/tts_runtime.py:56:    # OPTIMIZATION: Pre-computed global tokens per speaker (eliminates ~110ms pre-roll in streaming)
veena3modal/services/tts_runtime.py:57:    # Maps speaker_name -> list of 32 global token IDs captured at startup
veena3modal/services/tts_runtime.py:80:    hf_token: Optional[str] = None,
veena3modal/services/tts_runtime.py:84:    max_num_batched_tokens: Optional[int] = None,
veena3modal/services/tts_runtime.py:102:        hf_token: HuggingFace token for private models (env: HF_TOKEN)
veena3modal/services/tts_runtime.py:106:        max_num_batched_tokens: Optional vLLM scheduler cap override.
veena3modal/services/tts_runtime.py:112:        precompute_speaker_globals: Whether to warm speaker global tokens at startup.
veena3modal/services/tts_runtime.py:135:    hf_token = hf_token or os.environ.get('HF_TOKEN') or os.environ.get('HUGGING_FACE_HUB_TOKEN')
veena3modal/services/tts_runtime.py:154:        if isinstance(max_num_batched_tokens, int) and max_num_batched_tokens > 0:
veena3modal/services/tts_runtime.py:155:            engine_kwargs["max_num_batched_tokens"] = max_num_batched_tokens
veena3modal/services/tts_runtime.py:178:                hf_token=hf_token,
veena3modal/services/tts_runtime.py:186:                hf_token=hf_token,
veena3modal/services/tts_runtime.py:194:            tokenizer=model.tokenizer,
veena3modal/services/tts_runtime.py:306:    Pre-compute and cache global tokens for all 12 speakers at startup.
veena3modal/services/tts_runtime.py:308:    OPTIMIZATION: BiCodec streaming requires 32 "global tokens" before any audio can be emitted.
veena3modal/services/tts_runtime.py:310:    we generate one short utterance per speaker at startup, capture the 32 global tokens,
veena3modal/services/tts_runtime.py:312:    skipping the ~110ms global token pre-roll phase entirely.
veena3modal/services/tts_runtime.py:324:    logger.info("Pre-computing global tokens for all speakers...")
veena3modal/services/tts_runtime.py:333:        max_tokens=128,  # Only need ~60 tokens (32 global + some semantic)
veena3modal/services/tts_runtime.py:335:        skip_special_tokens=False,
veena3modal/services/tts_runtime.py:339:        """Generate a short utterance and capture the 32 global tokens."""
veena3modal/services/tts_runtime.py:412:    max_tokens: int = 4096,
veena3modal/services/tts_runtime.py:426:        max_tokens: Maximum tokens to generate
veena3modal/services/tts_runtime.py:445:        "generation_ms": 0,
veena3modal/services/tts_runtime.py:446:        "tokens_generated": 0,
veena3modal/services/tts_runtime.py:462:            max_tokens=max_tokens,
veena3modal/services/tts_runtime.py:470:        metrics["generation_ms"] = int(generation_time * 1000)
veena3modal/services/tts_runtime.py:471:        metrics["ttfb_ms"] = metrics["generation_ms"]  # Non-streaming: TTFB = total time
veena3modal/services/tts_runtime.py:541:    # process_chunk expects [batch, samples] or [samples] and returns [batch, samples]
veena3modal/services/tts_runtime.py:544:    # Convert back to numpy, squeeze batch dim if present
veena3modal/services/tts_runtime.py:584:    max_tokens: int = 4096,
veena3modal/services/tts_runtime.py:611:        "generation_ms": 0,
veena3modal/services/tts_runtime.py:633:                max_tokens=max_tokens,
veena3modal/services/tts_runtime.py:646:                max_tokens=max_tokens,
veena3modal/services/tts_runtime.py:654:        metrics["generation_ms"] = int(generation_time * 1000)
veena3modal/services/tts_runtime.py:655:        metrics["ttfb_ms"] = metrics["generation_ms"]
veena3modal/services/tts_runtime.py:695:    max_tokens: int = 4096,
veena3modal/services/tts_runtime.py:713:        max_tokens: Maximum tokens to generate
veena3modal/services/tts_runtime.py:757:        # Chunked streaming with voice consistency (global token caching)
veena3modal/services/tts_runtime.py:761:            max_tokens, repetition_penalty, seed, sample_rate
veena3modal/services/tts_runtime.py:779:        # OPTIMIZATION: If we have pre-cached global tokens for this speaker,
veena3modal/services/tts_runtime.py:780:        # use continuation mode to skip the ~110ms global token pre-roll.
veena3modal/services/tts_runtime.py:781:        # The model jumps straight to semantic token generation.
veena3modal/services/tts_runtime.py:785:            # Fast path: use cached globals, skip global token generation entirely
veena3modal/services/tts_runtime.py:793:                max_tokens=max_tokens,
veena3modal/services/tts_runtime.py:805:                max_tokens=max_tokens,
veena3modal/services/tts_runtime.py:834:    max_tokens: int,
veena3modal/services/tts_runtime.py:842:    Uses global token caching from first chunk to maintain voice across chunks.
veena3modal/services/tts_runtime.py:860:            # First chunk: capture global tokens for voice consistency
veena3modal/services/tts_runtime.py:867:                max_tokens=max_tokens,
veena3modal/services/tts_runtime.py:894:                    max_tokens=max_tokens,
veena3modal/services/tts_runtime.py:915:                    max_tokens=max_tokens,
scripts/stress_test_runtime_detailed.py:39:        "This benchmark sentence checks token generation speed, decode latency, "
scripts/stress_test_runtime_detailed.py:223:    max_num_batched_tokens: Optional[int],
scripts/stress_test_runtime_detailed.py:241:            max_num_batched_tokens=max_num_batched_tokens,
scripts/stress_test_runtime_detailed.py:254:    max_tokens: int,
scripts/stress_test_runtime_detailed.py:266:                max_tokens=max_tokens,
scripts/stress_test_runtime_detailed.py:272:                max_tokens=max_tokens,
scripts/stress_test_runtime_detailed.py:324:    max_tokens: int,
scripts/stress_test_runtime_detailed.py:330:    semaphore = asyncio.Semaphore(concurrency)
scripts/stress_test_runtime_detailed.py:333:        async with semaphore:
scripts/stress_test_runtime_detailed.py:338:                max_tokens=max_tokens,
scripts/stress_test_runtime_detailed.py:360:        "generation_ms",
scripts/stress_test_runtime_detailed.py:361:        "timeline_llm_first_batch_ms",
scripts/stress_test_runtime_detailed.py:365:        "timeline_total_ms",
scripts/stress_test_runtime_detailed.py:366:        "llm_batch_wall_ms_total",
scripts/stress_test_runtime_detailed.py:367:        "llm_batch_gpu_ms_total",
scripts/stress_test_runtime_detailed.py:371:        "llm_time_per_token_ms",
scripts/stress_test_runtime_detailed.py:380:    tl_total = level.timing_stats("timeline_total_ms")
scripts/stress_test_runtime_detailed.py:381:    llm_wall = level.timing_stats("llm_batch_wall_ms_total")
scripts/stress_test_runtime_detailed.py:383:    latency_over_timeline = level.latency_over_metric_stats("timeline_total_ms")
scripts/stress_test_runtime_detailed.py:384:    latency_over_generation = level.latency_over_metric_stats("generation_ms")
scripts/stress_test_runtime_detailed.py:385:    if latency_over_timeline:
scripts/stress_test_runtime_detailed.py:387:            "latency_over_timeline_ms: avg={:.2f} p50={:.2f} p95={:.2f} max={:.2f}".format(
scripts/stress_test_runtime_detailed.py:388:                latency_over_timeline["avg"],
scripts/stress_test_runtime_detailed.py:389:                latency_over_timeline["p50"],
scripts/stress_test_runtime_detailed.py:390:                latency_over_timeline["p95"],
scripts/stress_test_runtime_detailed.py:391:                latency_over_timeline["max"],
scripts/stress_test_runtime_detailed.py:396:            "latency_over_generation_ms: avg={:.2f} p50={:.2f} p95={:.2f} max={:.2f}".format(
scripts/stress_test_runtime_detailed.py:418:            if isinstance(r.timing.get("timeline_total_ms"), (int, float))
scripts/stress_test_runtime_detailed.py:424:                "timeline_sample_ms: first_batch={:.1f} llm_done={:.1f} parse_done={:.1f} "
scripts/stress_test_runtime_detailed.py:426:                    float(t.get("timeline_llm_first_batch_ms", 0.0)),
scripts/stress_test_runtime_detailed.py:430:                    float(t.get("timeline_total_ms", 0.0)),
scripts/stress_test_runtime_detailed.py:458:            "max_tokens": args.max_tokens,
scripts/stress_test_runtime_detailed.py:464:            "max_num_batched_tokens": args.max_num_batched_tokens,
scripts/stress_test_runtime_detailed.py:486:                    "generation_ms": lvl.timing_stats("generation_ms"),
scripts/stress_test_runtime_detailed.py:487:                    "llm_batch_wall_ms_total": lvl.timing_stats("llm_batch_wall_ms_total"),
scripts/stress_test_runtime_detailed.py:488:                    "llm_batch_gpu_ms_total": lvl.timing_stats("llm_batch_gpu_ms_total"),
scripts/stress_test_runtime_detailed.py:492:                    "llm_time_per_token_ms": lvl.timing_stats("llm_time_per_token_ms"),
scripts/stress_test_runtime_detailed.py:495:                    "timeline_llm_first_batch_ms": lvl.timing_stats("timeline_llm_first_batch_ms"),
scripts/stress_test_runtime_detailed.py:499:                    "timeline_total_ms": lvl.timing_stats("timeline_total_ms"),
scripts/stress_test_runtime_detailed.py:500:                    "latency_over_timeline_ms": lvl.latency_over_metric_stats("timeline_total_ms"),
scripts/stress_test_runtime_detailed.py:501:                    "latency_over_generation_ms": lvl.latency_over_metric_stats("generation_ms"),
scripts/stress_test_runtime_detailed.py:532:        max_tokens=args.max_tokens,
scripts/stress_test_runtime_detailed.py:544:            max_tokens=args.max_tokens,
scripts/stress_test_runtime_detailed.py:563:    parser.add_argument("--max-tokens", type=int, default=128, help="max_tokens sent to generation")
scripts/stress_test_runtime_detailed.py:570:    parser.add_argument("--max-num-batched-tokens", type=int, default=None)
scripts/stress_test_runtime_detailed.py:590:        max_num_batched_tokens=args.max_num_batched_tokens,


Output:
Chunk ID: e461d0
Wall time: 0.0513 seconds
Process exited with code 0
Original token count: 2623
Output:
"""
TTS runtime wrapper (container-scoped singleton).

In Modal, each container should load the model once and then handle many requests.
This wrapper owns:
- SparkTTSModel (vLLM engine)
- IndicPromptBuilder
- BiCodecDecoder
- SparkTTSPipeline + Veena3SlidingWindowPipeline
- optional SuperResolutionService

Imports framework-agnostic inference + processing code from `veena3modal/core`,
`veena3modal/processing`, and `veena3modal/audio`.
"""

from __future__ import annotations

import os
import sys
import time
import logging
from dataclasses import dataclass, field
from typing import Any, Dict, List, Optional, Tuple

logger = logging.getLogger(__name__)

@dataclass
class TTSRuntime:
    """
    Holds long-lived, per-container inference objects.

    Thread Safety:
        This class is designed for async concurrency within a single container.
        The vLLM engine handles internal batching and scheduling.
        Do NOT share across processes without proper synchronization.
    """
    model: Any = None
    pipeline: Any = None
    streaming_pipeline: Any = None
    long_text_processor: Any = None
    prompt_builder: Any = None
    bicodec_decoder: Any = None
    sr_service: Optional[Any] = None

    model_version: str = "not_loaded"
    is_loaded: bool = False
    load_time_ms: float = 0.0

    # Configuration
    model_path: str = ""
    bicodec_path: str = ""
    sr_checkpoint_dir: Optional[str] = None
    device: str = "cuda"

    # OPTIMIZATION: Pre-computed global tokens per speaker (eliminates ~110ms pre-roll in streaming)
    # Maps speaker_name -> list of 32 global token IDs captured at startup
    speaker_global_cache: Dict[str, List[int]] = field(default_factory=dict)

# Module-level singleton (per-container)
_runtime: Optional[TTSRuntime] = None

def get_runtime() -> Optional[TTSRuntime]:
    """Get the current TTS runtime singleton."""
    return _runtime

def is_initialized() -> bool:
    """Check if the runtime is initialized and ready."""
    return _runtime is not None and _runtime.is_loaded

def initialize_runtime(
    model_path: Optional[str] = None,
    bicodec_path: Optional[str] = None,
    sr_checkpoint_dir: Optional[str] = None,
    device: str = "cuda",
    hf_token: Optional[str] = None,
    gpu_memory_utilization: float = 0.25,
    enable_sr: bool = False,
    num_engines: int = 1,
    max_num_batched_tokens: Optional[int] = None,
    max_num_seqs: Optional[int] = None,
    enable_chunked_prefill: Optional[bool] = None,
    enable_prefix_caching: Optional[bool] = None,
    disable_log_stats: Optional[bool] = None,
    enforce_eager: Optional[bool] = None,
    precompute_speaker_globals: Optional[bool] = None,
) -> TTSRuntime:
    """
    Initialize the TTS runtime with all components.

    This should be called once per container (e.g., in Modal's @modal.enter).

    Args:
        model_path: Path to Spark TTS model (env: SPARK_TTS_MODEL_PATH)
        bicodec_path: Path to BiCodec model (env: BICODEC_MODEL_PATH, defaults to model_path)
        sr_checkpoint_dir: Path to super-resolution checkpoints (env: AP_BWE_CHECKPOINT_DIR)
        device: Device for inference (cuda/cpu)
        hf_token: HuggingFace token for private models (env: HF_TOKEN)
        gpu_memory_utilization: vLLM GPU memory fraction (default: 0.25)
        enable_sr: Enable super-resolution service
        num_engines: Number of vLLM engine instances (default: 1, set to 2-3 for Tier 3 optimization)
        max_num_batched_tokens: Optional vLLM scheduler cap override.
        max_num_seqs: Optional vLLM concurrent sequence cap override.
        enable_chunked_prefill: Optional vLLM chunked prefill toggle.
        enable_prefix_caching: Optional vLLM prefix caching toggle.
        disable_log_stats: Optional vLLM internal stats log toggle.
        enforce_eager: Optional vLLM eager-mode toggle (disables CUDA graphs when true).
        precompute_speaker_globals: Whether to warm speaker global tokens at startup.
            If None, reads PRECOMPUTE_SPEAKER_GLOBALS env var (default: false).

    Returns:
        Initialized TTSRuntime instance

    Raises:
        RuntimeError: If initialization fails
    """
    global _runtime

    start_time = time.time()

    # Resolve paths from env vars if not provided
    model_path = model_path or os.environ.get(
        'SPARK_TTS_MODEL_PATH',
        os.environ.get('MODEL_PATH', '/models/spark_tts_4speaker')
    )
    bicodec_path = bicodec_path or os.environ.get(
        'BICODEC_MODEL_PATH',
        model_path  # BiCodec is usually in the same directory
    )
    sr_checkpoint_dir = sr_checkpoint_dir or os.environ.get('AP_BWE_CHECKPOINT_DIR')
    hf_token = hf_token or os.environ.get('HF_TOKEN') or os.environ.get('HUGGING_FACE_HUB_TOKEN')
    if precompute_speaker_globals is None:
        raw_flag = os.environ.get("PRECOMPUTE_SPEAKER_GLOBALS", "false").strip().lower()
        precompute_speaker_globals = raw_flag in {"1", "true", "yes", "on"}

    logger.info(f"Initializing TTS runtime...")
    logger.info(f"  Model path: {model_path}")
    logger.info(f"  BiCodec path: {bicodec_path}")
    logger.info(f"  SR enabled: {enable_sr}, path: {sr_checkpoint_dir}")

    try:
        # Import inference components (vendored into veena3modal for Modal deployment)
        from veena3modal.core.model_loader import SparkTTSModel
        from veena3modal.core.pipeline import SparkTTSPipeline
        from veena3modal.core.bicodec_decoder import BiCodecDecoder
        from veena3modal.processing.prompt_builder import IndicPromptBuilder
        from veena3modal.processing.long_text_processor import LongTextProcessor

        engine_kwargs: Dict[str, Any] = {}
        if isinstance(max_num_batched_tokens, int) and max_num_batched_tokens > 0:
            engine_kwargs["max_num_batched_tokens"] = max_num_batched_tokens
        if isinstance(max_num_seqs, int) and max_num_seqs > 0:
            engine_kwargs["max_num_seqs"] = max_num_seqs
        if enable_chunked_prefill is not None:
            engine_kwargs["enable_chunked_prefill"] = bool(enable_chunked_prefill)
        if enable_prefix_caching is not None:
            engine_kwargs["enable_prefix_caching"] = bool(enable_prefix_caching)
        if disable_log_stats is not None:
            engine_kwargs["disable_log_stats"] = bool(disable_log_stats)
        if enforce_eager is not None:
            engine_kwargs["enforce_eager"] = bool(enforce_eager)

        # Load SparkTTS model with vLLM engine
        if num_engines > 1:
            # TIER 3 OPTIMIZATION: Multiple vLLM engines on same GPU
            # Each engine gets gpu_memory_utilization / num_engines to share GPU fairly.
            # E.g., 3 engines at 0.08 each = 0.24 total, ~8GB per engine, ~24GB total.
            from veena3modal.core.multi_engine import create_multi_engine_model
            per_engine_mem = gpu_memory_utilization / num_engines
            logger.info(f"Loading {num_engines} vLLM engines ({per_engine_mem:.2f} GPU mem each)...")
            model = create_multi_engine_model(
                model_path=model_path,
                num_engines=num_engines,
                hf_token=hf_token,
                gpu_memory_per_engine=per_engine_mem,
                **engine_kwargs,
            )
        else:
            logger.info("Loading SparkTTS model with vLLM...")
            model = SparkTTSModel(
                model_path=model_path,
                hf_token=hf_token,
                gpu_memory_utilization=gpu_memory_utilization,
                **engine_kwargs,
            )

        # Initialize prompt builder
        logger.info("Initializing prompt builder...")
        prompt_builder = IndicPromptBuilder(
            tokenizer=model.tokenizer,
            model=model,
        )

        # Initialize BiCodec decoder
        logger.info("Initializing BiCodec decoder...")
        bicodec_decoder = BiCodecDecoder(
            device=device,
            model_path=bicodec_path,
        )

        # Initialize pipeline
        logger.info("Initializing TTS pipeline...")
        pipeline = SparkTTSPipeline(
            model=model,
            prompt_builder=prompt_builder,
            bicodec_decoder=bicodec_decoder,
        )

        # Initialize streaming pipeline (for M4)
        streaming_pipeline = None
        try:
            from veena3modal.core.streaming_pipeline import Veena3SlidingWindowPipeline
            logger.info("Initializing streaming pipeline...")
            # NOTE: Parameter is named 'snac_decoder' for legacy reasons, but works with BiCodecDecoder
            streaming_pipeline = Veena3SlidingWindowPipeline(
                model=model,
                prompt_builder=prompt_builder,
                snac_decoder=bicodec_decoder,  # BiCodecDecoder is interface-compatible
            )
        except ImportError as e:
            logger.warning(f"Streaming pipeline not available: {e}")

        # Initialize long text processor once (avoid per-request construction/logging)
        long_text_processor = LongTextProcessor(
            pipeline=pipeline,
            streaming_pipeline=streaming_pipeline,
        )

        # Initialize super-resolution (optional)
        sr_service = None
        if enable_sr and sr_checkpoint_dir:
            try:
                from veena3modal.core.super_resolution import SuperResolutionService
                logger.info(f"Initializing super-resolution from {sr_checkpoint_dir}...")
                sr_service = SuperResolutionService(checkpoint_dir=sr_checkpoint_dir)
                # Load the model explicitly
                if sr_service.load_model(device=device):
                    logger.info("✅ Super-resolution model loaded successfully")
                else:
                    logger.warning("Super-resolution model failed to load")
                    sr_service = None
            except Exception as e:
                logger.warning(f"Super-resolution not available: {e}")

        # Determine model version
        model_version = os.path.basename(model_path.rstrip('/'))
        if not model_version:
            model_version = "spark-tts"

        load_time = (time.time() - start_time) * 1000

        # Create runtime
        _runtime = TTSRuntime(
            model=model,
            pipeline=pipeline,
            streaming_pipeline=streaming_pipeline,


Output:
Chunk ID: 6cd4f8
Wall time: 0.0503 seconds
Process exited with code 0
Original token count: 3251
Output:
            streaming_pipeline=streaming_pipeline,
            long_text_processor=long_text_processor,
            prompt_builder=prompt_builder,
            bicodec_decoder=bicodec_decoder,
            sr_service=sr_service,
            model_version=model_version,
            is_loaded=True,
            load_time_ms=load_time,
            model_path=model_path,
            bicodec_path=bicodec_path,
            sr_checkpoint_dir=sr_checkpoint_dir,
            device=device,
        )

        logger.info(f"✅ TTS runtime initialized in {load_time:.0f}ms")
        logger.info(f"   Model version: {model_version}")

        # Pre-compute speaker globals only when explicitly enabled.
        # Startup warmup runs async generation and can bind AsyncLLMEngine state
        # to a non-serving event loop if executed in a different loop context.
        if precompute_speaker_globals and num_engines <= 1:
            logger.info("Speaker globals startup precompute enabled")
            _precompute_speaker_globals(_runtime)
        elif num_engines > 1:
            logger.info("Speaker globals cache startup precompute skipped (multi-engine mode)")
        else:
            logger.info("Speaker globals cache startup precompute skipped (disabled)")

        # Update FastAPI app with model version
        try:
            from veena3modal.api.fastapi_app import set_model_version
            set_model_version(model_version)
        except ImportError:
            pass

        return _runtime

    except Exception as e:
        logger.error(f"❌ Failed to initialize TTS runtime: {e}")
        import traceback
        traceback.print_exc()
        raise RuntimeError(f"TTS runtime initialization failed: {e}") from e

def _precompute_speaker_globals(runtime: TTSRuntime) -> None:
    """
    Pre-compute and cache global tokens for all 12 speakers at startup.

    OPTIMIZATION: BiCodec streaming requires 32 "global tokens" before any audio can be emitted.
    These encode speaker identity via FSQ quantization. Since we have only 12 fixed speakers,
    we generate one short utterance per speaker at startup, capture the 32 global tokens,
    and cache them. During streaming, we inject cached globals via build_prefix_with_globals(),
    skipping the ~110ms global token pre-roll phase entirely.

    This runs synchronously at startup (adds ~5-10s to cold start, saves ~110ms per streaming TTFB).
    """
    import asyncio
    import re
    from vllm import SamplingParams
    from veena3modal.core.constants import (
        SPEAKER_MAP, TRAINING_STOP_TOKEN_IDS,
        DEFAULT_TEMPERATURE, DEFAULT_TOP_K, DEFAULT_TOP_P,
    )

    logger.info("Pre-computing global tokens for all speakers...")
    t_start = time.time()

    test_text = "Hello, this is a voice test."

    sampling_params = SamplingParams(
        temperature=DEFAULT_TEMPERATURE,
        top_k=DEFAULT_TOP_K,
        top_p=DEFAULT_TOP_P,
        max_tokens=128,  # Only need ~60 tokens (32 global + some semantic)
        stop=TRAINING_STOP_TOKEN_IDS,
        skip_special_tokens=False,
    )

    async def _capture_globals_for_speaker(speaker_name: str) -> Optional[List[int]]:
        """Generate a short utterance and capture the 32 global tokens."""
        prompt = runtime.prompt_builder.build_prefix(speaker_name, test_text)
        request_id = f"warmup-{speaker_name}-{int(time.time() * 1000)}"

        final_output = None
        async for request_output in runtime.model.engine.generate(
            prompt=prompt,
            sampling_params=sampling_params,
            request_id=request_id,
        ):
            final_output = request_output

        if final_output is None:
            return None

        generated_text = final_output.outputs[0].text
        global_matches = re.findall(r"<\|bicodec_global_(\d+)\|>", generated_text)
        global_ids = [int(t) for t in global_matches]

        if len(global_ids) >= 32:
            return global_ids[:32]
        return None

    async def _run_all():
        for speaker_name in SPEAKER_MAP.keys():
            try:
                global_ids = await _capture_globals_for_speaker(speaker_name)
                if global_ids and len(global_ids) == 32:
                    runtime.speaker_global_cache[speaker_name] = global_ids
                    logger.info(f"  Cached globals for {speaker_name}")
                else:
                    logger.warning(f"  Failed to capture globals for {speaker_name}")
            except Exception as e:
                logger.warning(f"  Error caching globals for {speaker_name}: {e}")

    # Run async warmup in event loop
    try:
        loop = asyncio.get_event_loop()
        if loop.is_running():
            # Already in async context (unlikely at startup, but handle it)
            import concurrent.futures
            with concurrent.futures.ThreadPoolExecutor() as pool:
                pool.submit(lambda: asyncio.run(_run_all())).result()
        else:
            loop.run_until_complete(_run_all())
    except RuntimeError:
        # No event loop exists yet
        asyncio.run(_run_all())

    elapsed = (time.time() - t_start) * 1000
    cached = len(runtime.speaker_global_cache)
    total = len(SPEAKER_MAP)
    logger.info(f"Speaker globals cached: {cached}/{total} speakers in {elapsed:.0f}ms")

def _get_long_text_processor(runtime: TTSRuntime):
    """Return cached LongTextProcessor, creating it lazily if needed."""
    if runtime.long_text_processor is None:
        from veena3modal.processing.long_text_processor import LongTextProcessor

        runtime.long_text_processor = LongTextProcessor(
            pipeline=runtime.pipeline,
            streaming_pipeline=runtime.streaming_pipeline,
        )
    return runtime.long_text_processor

async def generate_speech(
    text: str,
    speaker: str,
    temperature: float = 0.8,
    top_k: int = 50,
    top_p: float = 1.0,
    max_tokens: int = 4096,
    repetition_penalty: float = 1.05,
    seed: Optional[int] = None,
    output_sample_rate: str = "16khz",
) -> Tuple[Optional[bytes], Dict[str, Any]]:
    """
    Generate speech audio (non-streaming).

    Args:
        text: Text to synthesize (already normalized)
        speaker: Internal speaker name (resolved)
        temperature: Sampling temperature
        top_k: Top-k sampling
        top_p: Nucleus sampling
        max_tokens: Maximum tokens to generate
        repetition_penalty: Repetition penalty
        seed: Random seed for reproducibility
        output_sample_rate: "16khz" or "48khz" (triggers super-resolution)

    Returns:
        Tuple of (audio_bytes, metrics_dict)
        audio_bytes: WAV audio bytes (16kHz or 48kHz, 16-bit PCM) or None if failed
        metrics_dict: Dictionary with timing metrics

    Raises:
        RuntimeError: If runtime not initialized
    """
    if not is_initialized():
        raise RuntimeError("TTS runtime not initialized")

    runtime = get_runtime()
    metrics = {
        "ttfb_ms": 0,
        "generation_ms": 0,
        "tokens_generated": 0,
        "audio_duration_seconds": 0.0,
        "sr_applied": False,
        "output_sample_rate": 16000,
    }

    start_time = time.time()

    try:
        # Generate audio using pipeline (always at 16kHz)
        audio_bytes, perf = await runtime.pipeline.generate_speech_profiled(
            speaker=speaker,
            text=text,
            temperature=temperature,
            top_k=top_k,
            top_p=top_p,
            max_tokens=max_tokens,
            repetition_penalty=repetition_penalty,
            seed=seed,
        )
        if perf:
            metrics.update(perf)

        generation_time = time.time() - start_time
        metrics["generation_ms"] = int(generation_time * 1000)
        metrics["ttfb_ms"] = metrics["generation_ms"]  # Non-streaming: TTFB = total time

        if audio_bytes:
            # Apply super-resolution if requested and available
            sample_rate = 16000
            if output_sample_rate == "48khz" or logger.isEnabledFor(logging.DEBUG):
                logger.debug(
                    "SR check: output_sample_rate=%r, sr_service=%s, is_loaded=%s",
                    output_sample_rate,
                    runtime.sr_service is not None,
                    runtime.sr_service.is_loaded if runtime.sr_service else False,
                )

            if output_sample_rate == "48khz" and runtime.sr_service and runtime.sr_service.is_loaded:
                try:
                    sr_start = time.time()
                    logger.info(f"Applying super-resolution to {len(audio_bytes)} bytes...")
                    audio_bytes = _apply_super_resolution(audio_bytes, runtime.sr_service)
                    sr_time = (time.time() - sr_start) * 1000
                    metrics["sr_ms"] = int(sr_time)
                    metrics["sr_applied"] = True
                    sample_rate = 48000
                    logger.info(f"✅ Super-resolution applied in {sr_time:.1f}ms, output={len(audio_bytes)} bytes")
                except Exception as e:
                    logger.warning(f"Super-resolution failed, returning 16kHz: {e}")
                    import traceback
                    traceback.print_exc()

            metrics["output_sample_rate"] = sample_rate

            # Calculate audio duration
            audio_duration = (len(audio_bytes) - 44) / (sample_rate * 2)  # -44 for WAV header
            metrics["audio_duration_seconds"] = max(0.0, audio_duration)

        return audio_bytes, metrics

    except Exception as e:
        logger.error(f"Speech generation failed: {e}")
        raise

def _apply_super_resolution(audio_bytes: bytes, sr_service) -> bytes:
    """
    Apply super-resolution to audio bytes.

    Args:
        audio_bytes: WAV audio at 16kHz
        sr_service: SuperResolutionService instance

    Returns:
        WAV audio at 48kHz
    """
    import numpy as np
    import struct
    import torch

    # Parse WAV header and extract PCM data
    if len(audio_bytes) < 44:
        raise ValueError("Invalid WAV data")

    # Extract PCM data (skip 44-byte WAV header)
    pcm_data = np.frombuffer(audio_bytes[44:], dtype=np.int16)

    # Convert to float32 for SR model
    audio_float = pcm_data.astype(np.float32) / 32768.0

    # Convert numpy to torch tensor for process_chunk
    audio_tensor = torch.from_numpy(audio_float)

    # Apply super-resolution using process_chunk (16kHz -> 48kHz)
    # process_chunk expects [batch, samples] or [samples] and returns [batch, samples]
    upsampled_tensor = sr_service.process_chunk(audio_tensor)

    # Convert back to numpy, squeeze batch dim if present
    upsampled = upsampled_tensor.squeeze().cpu().numpy()

    # Convert back to int16
    upsampled_int16 = np.clip(upsampled * 32768.0, -32768, 32767).astype(np.int16)

    # Create new WAV header for 48kHz
    sample_rate = 48000
    num_channels = 1
    bits_per_sample = 16
    byte_rate = sample_rate * num_channels * bits_per_sample // 8
    block_align = num_channels * bits_per_sample // 8
    data_size = len(upsampled_int16) * 2

    wav_header = struct.pack(
        '<4sI4s4sIHHIIHH4sI',
        b'RIFF',
        36 + data_size,
        b'WAVE',
        b'fmt ',
        16,  # fmt chunk size
        1,   # PCM format
        num_channels,
        sample_rate,
        byte_rate,
        block_align,
        bits_per_sample,
        b'data',
        data_size,
    )

    return wav_header + upsampled_int16.tobytes()

async def generate_speech_chunked(
    text: str,
    speaker: str,
    temperature: float = 0.8,
    top_k: int = 50,
    top_p: float = 1.0,
    max_tokens: int = 4096,
    repetition_penalty: float = 1.05,
    seed: Optional[int] = None,
    sample_rate: int = 16000,
    output_sample_rate: str = "16khz",
) -> Tuple[Optional[bytes], Dict[str, Any]]:
    """
    Generate speech with automatic text chunking for long inputs.

    Uses LongTextProcessor to split text and stitch audio.

    Args:
        text: Text to synthesize
        speaker: Internal speaker name
        ... (same as generate_speech)
        sample_rate: Internal generation sample rate (always 16000)
        output_sample_rate: "16khz" or "48khz" (triggers super-resolution)

    Returns:
        Tuple of (audio_bytes, metrics_dict)
    """
    if not is_initialized():
        raise RuntimeError("TTS runtime not initialized")

    runtime = get_runtime()
    metrics = {
        "ttfb_ms": 0,
        "generation_ms": 0,
        "chunks_processed": 0,
        "audio_duration_seconds": 0.0,
        "text_chunked": False,
        "sr_applied": False,
        "output_sample_rate": 16000,
    }

    start_time = time.time()


**Investigating streaming pipeline performance**


cd /home/ubuntu/veenaModal && rg -n "generate_speech_profiled|timeline_|llm_batch|decode_wall|first_batch|perf|request_output|async for|generate\(" veena3modal/core -g'*.py'


cd /home/ubuntu/veenaModal && sed -n '1,320p' veena3modal/core/pipeline.py


cd /home/ubuntu/veenaModal && sed -n '1,360p' veena3modal/core/model_loader.py


Output:
Chunk ID: 4421aa
Wall time: 0.0507 seconds
Process exited with code 0
Original token count: 4022
Output:
veena3modal/core/super_resolution.py:296:        # Track performance
veena3modal/core/super_resolution.py:301:        # Log performance occasionally
veena3modal/core/super_resolution.py:305:                f"SR performance: {self.total_chunks_processed} chunks, "
veena3modal/core/super_resolution.py:312:        """Get performance statistics."""
veena3modal/core/pipeline.py:26:def _log_perf_stage(event: str, extra: Dict[str, Any]) -> None:
veena3modal/core/pipeline.py:27:    """Emit hot-path perf logs only when explicitly enabled or DEBUG logging is active."""
veena3modal/core/pipeline.py:93:        Convert vLLM request_output.metrics into a numeric dictionary.
veena3modal/core/pipeline.py:125:    def _apply_stats(self, perf: Dict[str, Any], prefix: str, values: List[float]) -> None:
veena3modal/core/pipeline.py:129:        perf[f"{prefix}_total"] = float(sum(values))
veena3modal/core/pipeline.py:130:        perf[f"{prefix}_min"] = float(min(values))
veena3modal/core/pipeline.py:131:        perf[f"{prefix}_max"] = float(max(values))
veena3modal/core/pipeline.py:132:        perf[f"{prefix}_p50"] = float(self._p50(values))
veena3modal/core/pipeline.py:134:    async def generate_speech_profiled(
veena3modal/core/pipeline.py:149:            (wav_bytes, perf_dict)
veena3modal/core/pipeline.py:153:        perf: Dict[str, Any] = {
veena3modal/core/pipeline.py:158:            "llm_batch_count": 0,
veena3modal/core/pipeline.py:161:            "bicodec_decode_wall_ms": 0.0,
veena3modal/core/pipeline.py:167:        t_request_start = time.perf_counter()
veena3modal/core/pipeline.py:168:        timeline_marks: List[Dict[str, float]] = [{"stage": "request_start", "t_ms": 0.0, "dt_ms": 0.0}]
veena3modal/core/pipeline.py:170:        perf["timeline_markers"] = timeline_marks
veena3modal/core/pipeline.py:171:        perf["timeline_request_start_ms"] = 0.0
veena3modal/core/pipeline.py:175:            now_ms = (time.perf_counter() - t_request_start) * 1000
veena3modal/core/pipeline.py:178:            timeline_marks.append(
veena3modal/core/pipeline.py:185:            perf[f"timeline_{stage}_ms"] = float(now_ms)
veena3modal/core/pipeline.py:186:            perf[f"timeline_{stage}_delta_ms"] = float(dt_ms)
veena3modal/core/pipeline.py:190:            if not timeline_marks or timeline_marks[-1].get("stage") != "request_done":
veena3modal/core/pipeline.py:194:            perf["timeline_markers"] = list(timeline_marks)
veena3modal/core/pipeline.py:195:            perf["timeline_total_ms"] = float(perf.get("timeline_request_done_ms", 0.0))
veena3modal/core/pipeline.py:197:            first_batch_ms = perf.get("timeline_llm_first_batch_ms")
veena3modal/core/pipeline.py:198:            llm_done_ms = perf.get("timeline_llm_done_ms")
veena3modal/core/pipeline.py:199:            parse_done_ms = perf.get("timeline_parse_done_ms")
veena3modal/core/pipeline.py:200:            bicodec_done_ms = perf.get("timeline_bicodec_done_ms")
veena3modal/core/pipeline.py:201:            request_done_ms = perf.get("timeline_request_done_ms")
veena3modal/core/pipeline.py:203:            if isinstance(first_batch_ms, (int, float)):
veena3modal/core/pipeline.py:204:                perf["timeline_to_first_batch_ms"] = float(first_batch_ms)
veena3modal/core/pipeline.py:205:            if isinstance(first_batch_ms, (int, float)) and isinstance(llm_done_ms, (int, float)) and llm_done_ms >= first_batch_ms:
veena3modal/core/pipeline.py:206:                perf["timeline_first_batch_to_llm_done_ms"] = float(llm_done_ms - first_batch_ms)
veena3modal/core/pipeline.py:208:                perf["timeline_post_llm_ms"] = float(request_done_ms - llm_done_ms)
veena3modal/core/pipeline.py:210:                perf["timeline_parse_to_bicodec_done_ms"] = float(bicodec_done_ms - parse_done_ms)
veena3modal/core/pipeline.py:212:                perf["timeline_bicodec_to_request_done_ms"] = float(request_done_ms - bicodec_done_ms)
veena3modal/core/pipeline.py:215:        t0 = time.perf_counter()
veena3modal/core/pipeline.py:217:        perf["prompt_build_ms"] = (time.perf_counter() - t0) * 1000
veena3modal/core/pipeline.py:222:            perf["llm_prompt_token_total"] = len(
veena3modal/core/pipeline.py:226:            perf["llm_prompt_token_total"] = 0
veena3modal/core/pipeline.py:229:        t0 = time.perf_counter()
veena3modal/core/pipeline.py:240:        perf["sampling_params_ms"] = (time.perf_counter() - t0) * 1000
veena3modal/core/pipeline.py:244:        _log_perf_stage(
veena3modal/core/pipeline.py:245:            "perf_non_stream_start",
veena3modal/core/pipeline.py:253:        results_generator = self.model.engine.generate(
veena3modal/core/pipeline.py:262:        llm_batch_wall_ms: List[float] = []
veena3modal/core/pipeline.py:263:        llm_batch_gpu_ms: List[float] = []
veena3modal/core/pipeline.py:264:        llm_decode_wall_ms: List[float] = []
veena3modal/core/pipeline.py:271:        t_gen_start = time.perf_counter()
veena3modal/core/pipeline.py:273:        async for request_output in results_generator:
veena3modal/core/pipeline.py:274:            now = time.perf_counter()
veena3modal/core/pipeline.py:278:            outputs = request_output.outputs or []
veena3modal/core/pipeline.py:280:                final_output = request_output
veena3modal/core/pipeline.py:288:            raw_req_metrics = self._extract_request_metrics(getattr(request_output, "metrics", None))
veena3modal/core/pipeline.py:293:                final_output = request_output
veena3modal/core/pipeline.py:296:            perf["llm_batch_count"] += 1
veena3modal/core/pipeline.py:297:            if perf["llm_batch_count"] == 1:
veena3modal/core/pipeline.py:298:                first_batch_timeline_ms = _mark("llm_first_batch")
veena3modal/core/pipeline.py:299:                _log_perf_stage(
veena3modal/core/pipeline.py:300:                    "perf_first_llm_batch",
veena3modal/core/pipeline.py:305:                        "timeline_ms": round(first_batch_timeline_ms, 3),
veena3modal/core/pipeline.py:309:            llm_batch_wall_ms.append(float(batch_wall))
veena3modal/core/pipeline.py:310:            if perf["llm_batch_count"] > 1:
veena3modal/core/pipeline.py:311:                llm_decode_wall_ms.append(float(batch_wall))
veena3modal/core/pipeline.py:331:                    llm_batch_gpu_ms.append(float(gpu_delta_ms))
veena3modal/core/pipeline.py:332:                    if perf["llm_batch_count"] > 1:
veena3modal/core/pipeline.py:335:            final_output = request_output
veena3modal/core/pipeline.py:337:        perf["llm_generation_wall_ms"] = (time.perf_counter() - t_gen_start) * 1000
veena3modal/core/pipeline.py:338:        llm_done_timeline_ms = _mark("llm_done")
veena3modal/core/pipeline.py:339:        _log_perf_stage(
veena3modal/core/pipeline.py:340:            "perf_llm_generation_done",
veena3modal/core/pipeline.py:343:                "llm_generation_wall_ms": round(perf["llm_generation_wall_ms"], 3),
veena3modal/core/pipeline.py:344:                "llm_batch_count": perf["llm_batch_count"],
veena3modal/core/pipeline.py:345:                "timeline_ms": round(llm_done_timeline_ms, 3),
veena3modal/core/pipeline.py:350:            perf["pipeline_total_ms"] = (time.perf_counter() - t_request_start) * 1000
veena3modal/core/pipeline.py:352:            return None, perf
veena3modal/core/pipeline.py:356:        perf["llm_token_total"] = len(generated_ids)
veena3modal/core/pipeline.py:359:        t0 = time.perf_counter()
veena3modal/core/pipeline.py:361:        perf["llm_parse_ms"] = (time.perf_counter() - t0) * 1000
veena3modal/core/pipeline.py:362:        perf["semantic_token_total"] = len(semantic_ids)
veena3modal/core/pipeline.py:363:        perf["global_token_total"] = len(global_ids)
veena3modal/core/pipeline.py:364:        parse_done_timeline_ms = _mark("parse_done")
veena3modal/core/pipeline.py:365:        _log_perf_stage(
veena3modal/core/pipeline.py:366:            "perf_token_parse_done",
veena3modal/core/pipeline.py:369:                "parse_ms": round(perf["llm_parse_ms"], 3),
veena3modal/core/pipeline.py:370:                "semantic_tokens": perf["semantic_token_total"],
veena3modal/core/pipeline.py:371:                "global_tokens": perf["global_token_total"],
veena3modal/core/pipeline.py:372:                "timeline_ms": round(parse_done_timeline_ms, 3),
veena3modal/core/pipeline.py:377:        self._apply_stats(perf, "llm_batch_wall_ms", llm_batch_wall_ms)
veena3modal/core/pipeline.py:378:        self._apply_stats(perf, "llm_batch_gpu_ms", llm_batch_gpu_ms)
veena3modal/core/pipeline.py:379:        self._apply_stats(perf, "llm_decode_wall_ms", llm_decode_wall_ms)
veena3modal/core/pipeline.py:380:        self._apply_stats(perf, "llm_decode_gpu_ms", llm_decode_gpu_ms)
veena3modal/core/pipeline.py:381:        self._apply_stats(perf, "tokens_per_batch", tokens_per_batch)
veena3modal/core/pipeline.py:383:        if llm_batch_wall_ms:
veena3modal/core/pipeline.py:384:            perf["llm_prefill_wall_ms"] = float(llm_batch_wall_ms[0])
veena3modal/core/pipeline.py:385:            perf["llm_prefill_gpu_ms"] = float(llm_batch_gpu_ms[0]) if llm_batch_gpu_ms else 0.0
veena3modal/core/pipeline.py:387:            perf["llm_prefill_wall_ms"] = float(perf["llm_generation_wall_ms"])
veena3modal/core/pipeline.py:388:            perf["llm_prefill_gpu_ms"] = 0.0
veena3modal/core/pipeline.py:390:        if not llm_batch_gpu_ms and final_request_metrics.get("model_execute_time") is not None:
veena3modal/core/pipeline.py:392:            perf["llm_batch_gpu_ms_total"] = total_gpu_ms
veena3modal/core/pipeline.py:393:            if perf["llm_batch_count"] > 1:
veena3modal/core/pipeline.py:394:                perf["llm_decode_gpu_ms_total"] = total_gpu_ms
veena3modal/core/pipeline.py:396:        if perf["llm_token_total"] > 0:
veena3modal/core/pipeline.py:397:            perf["llm_time_per_token_ms"] = float(perf["llm_generation_wall_ms"]) / float(perf["llm_token_total"])
veena3modal/core/pipeline.py:398:        if perf["llm_batch_count"] > 0:
veena3modal/core/pipeline.py:399:            perf["llm_time_per_batch_wall_ms"] = float(perf.get("llm_batch_wall_ms_total", 0.0)) / float(perf["llm_batch_count"])
veena3modal/core/pipeline.py:400:            if perf.get("llm_batch_gpu_ms_total", 0.0) > 0:
veena3modal/core/pipeline.py:401:                perf["llm_time_per_batch_gpu_ms"] = float(perf["llm_batch_gpu_ms_total"]) / float(perf["llm_batch_count"])
veena3modal/core/pipeline.py:412:                perf[dst] = float(final_request_metrics[src]) * 1000
veena3modal/core/pipeline.py:418:            perf["llm_first_token_ms"] = (first_token - arrival) * 1000
veena3modal/core/pipeline.py:420:            perf["llm_request_lifecycle_ms"] = (finished - arrival) * 1000
veena3modal/core/pipeline.py:423:            perf["pipeline_total_ms"] = (time.perf_counter() - t_request_start) * 1000
veena3modal/core/pipeline.py:425:            return None, perf
veena3modal/core/pipeline.py:428:        t0 = time.perf_counter()
veena3modal/core/pipeline.py:430:            perf["token_validation_ms"] = (time.perf_counter() - t0) * 1000
veena3modal/core/pipeline.py:432:            perf["pipeline_total_ms"] = (time.perf_counter() - t_request_start) * 1000
veena3modal/core/pipeline.py:434:            return None, perf
veena3modal/core/pipeline.py:435:        perf["token_validation_ms"] = (time.perf_counter() - t0) * 1000
veena3modal/core/pipeline.py:439:        decode_t0 = time.perf_counter()
veena3modal/core/pipeline.py:441:        perf["llm_decode_calls"] = 1
veena3modal/core/pipeline.py:466:        decode_wall_ms = (time.perf_counter() - decode_t0) * 1000
veena3modal/core/pipeline.py:467:        perf["bicodec_decode_wall_ms"] = float(decode_wall_ms)
veena3modal/core/pipeline.py:468:        perf["bicodec_decode_gpu_ms"] = float(max(0.0, decode_gpu_ms))
veena3modal/core/pipeline.py:469:        perf["bicodec_decode_cpu_ms"] = float(max(0.0, decode_wall_ms - decode_gpu_ms))
veena3modal/core/pipeline.py:470:        bicodec_done_timeline_ms = _mark("bicodec_done")
veena3modal/core/pipeline.py:471:        _log_perf_stage(
veena3modal/core/pipeline.py:472:            "perf_bicodec_decode_done",
veena3modal/core/pipeline.py:475:                "decode_wall_ms": round(perf["bicodec_decode_wall_ms"], 3),
veena3modal/core/pipeline.py:476:                "decode_gpu_ms": round(perf["bicodec_decode_gpu_ms"], 3),
veena3modal/core/pipeline.py:477:                "decode_cpu_ms": round(perf["bicodec_decode_cpu_ms"], 3),
veena3modal/core/pipeline.py:478:                "timeline_ms": round(bicodec_done_timeline_ms, 3),
veena3modal/core/pipeline.py:483:            perf["pipeline_total_ms"] = (time.perf_counter() - t_request_start) * 1000
veena3modal/core/pipeline.py:485:            return None, perf
veena3modal/core/pipeline.py:488:        t0 = time.perf_counter()
veena3modal/core/pipeline.py:490:        perf["wav_pack_ms"] = (time.perf_counter() - t0) * 1000
veena3modal/core/pipeline.py:493:        perf["pipeline_total_ms"] = (time.perf_counter() - t_request_start) * 1000
veena3modal/core/pipeline.py:495:        _log_perf_stage(
veena3modal/core/pipeline.py:496:            "perf_non_stream_done",
veena3modal/core/pipeline.py:499:                "pipeline_total_ms": round(perf["pipeline_total_ms"], 3),
veena3modal/core/pipeline.py:500:                "llm_token_total": perf.get("llm_token_total"),
veena3modal/core/pipeline.py:501:                "timeline_total_ms": round(float(perf.get("timeline_total_ms", 0.0)), 3),
veena3modal/core/pipeline.py:504:        return wav_bytes, perf
veena3modal/core/pipeline.py:535:        wav_bytes, _ = await self.generate_speech_profiled(
veena3modal/core/dual_engine.py:42:    results = engine.generate(prompt, sampling_params, request_id)
veena3modal/core/dual_engine.py:127:#     Replace: self.model.engine.generate(...)
veena3modal/core/dual_engine.py:128:#     With:    router.get_engine().engine.generate(...)
veena3modal/core/streaming_pipeline.py:145:        results_generator = self.model.engine.generate(
veena3modal/core/streaming_pipeline.py:152:        async for request_output in results_generator:
veena3modal/core/streaming_pipeline.py:154:            generated_ids = request_output.outputs[0].token_ids
veena3modal/core/streaming_pipeline.py:293:        results_generator = self.model.engine.generate(
veena3modal/core/streaming_pipeline.py:314:        async for request_output in results_generator:
veena3modal/core/streaming_pipeline.py:323:            generated_ids = request_output.outputs[0].token_ids
veena3modal/core/streaming_pipeline.py:476:            async for audio_bytes, global_ids in pipeline.generate_speech_stream_indic_first_chunk(...):
veena3modal/core/streaming_pipeline.py:540:        results_generator = self.model.engine.generate(
veena3modal/core/streaming_pipeline.py:553:        async for request_output in results_generator:
veena3modal/core/streaming_pipeline.py:554:            generated_ids = request_output.outputs[0].token_ids
veena3modal/core/streaming_pipeline.py:744:        results_generator = self.model.engine.generate(
veena3modal/core/streaming_pipeline.py:756:        async for request_output in results_generator:
veena3modal/core/streaming_pipeline.py:757:            generated_ids = request_output.outputs[0].token_ids
veena3modal/core/multi_engine.py:9:The wrapper is transparent to pipeline code: it exposes the same `.engine.generate()`
veena3modal/core/multi_engine.py:17:    model.engine.generate(prompt, sampling_params, request_id)  # routed to engine N
veena3modal/core/multi_engine.py:63:    def generate(self, *args, **kwargs) -> AsyncGenerator:
veena3modal/core/multi_engine.py:64:        """Route generate() to the least-loaded engine and release on completion."""
veena3modal/core/multi_engine.py:67:            base_generator = engine.generate(*args, **kwargs)
veena3modal/core/multi_engine.py:74:                async for item in base_generator:
veena3modal/core/multi_engine.py:96:        .engine    -> LoadAwareEngine (dispatches generate() calls)


Output:
Chunk ID: 27b433
Wall time: 0.0513 seconds
Process exited with code 0
Original token count: 2889
Output:
"""
Spark TTS Generation Pipeline

End-to-end pipeline for TTS generation with BiCodec audio tokenizer.
Migrated from Veena3/Orpheus (SNAC) to Spark TTS (BiCodec).
"""

import logging
import os
import re
import time
import uuid
from typing import Optional, List, Dict, Any, Tuple
from vllm import SamplingParams
import torch

logger = logging.getLogger(__name__)
_PERF_STAGE_LOG_ENABLED = os.environ.get("VEENA3_PERF_STAGE_LOG", "").strip().lower() in {
    "1",
    "true",
    "yes",
    "on",
}

def _log_perf_stage(event: str, extra: Dict[str, Any]) -> None:
    """Emit hot-path perf logs only when explicitly enabled or DEBUG logging is active."""
    if _PERF_STAGE_LOG_ENABLED:
        logger.info(event, extra=extra)
    elif logger.isEnabledFor(logging.DEBUG):
        logger.debug(event, extra=extra)

from veena3modal.core.constants import (
    TRAINING_STOP_TOKEN_IDS,
    DEFAULT_TEMPERATURE,
    DEFAULT_TOP_K,
    DEFAULT_TOP_P,
    DEFAULT_MAX_TOKENS,
    DEFAULT_SEED,
    AUDIO_SAMPLE_RATE,
)

class SparkTTSPipeline:
    """
    End-to-end TTS pipeline for Spark TTS with BiCodec.

    Replaces SNAC-based token extraction with BiCodec regex-based extraction.
    """

    def __init__(
        self,
        model,
        prompt_builder,
        bicodec_decoder,
    ):
        """
        Initialize pipeline.

        Args:
            model: SparkTTS Model instance
            prompt_builder: IndicPromptBuilder instance (with Spark TTS format)
            bicodec_decoder: BiCodecDecoder instance
        """
        self.model = model
        self.prompt_builder = prompt_builder
        self.bicodec_decoder = bicodec_decoder

        print(f"🚀 SparkTTSPipeline initialized")

    @staticmethod
    def _p50(values: List[float]) -> float:
        """Compute a stable p50 for small lists."""
        if not values:
            return 0.0
        ordered = sorted(values)
        return ordered[(len(ordered) - 1) // 2]

    @staticmethod
    def _to_number(value: Any) -> Optional[float]:
        """Best-effort conversion of scalar values to float."""
        if value is None:
            return None
        if isinstance(value, (int, float)):
            return float(value)
        try:
            return float(value)
        except (TypeError, ValueError):
            return None

    def _extract_request_metrics(self, raw_metrics: Any) -> Dict[str, float]:
        """
        Convert vLLM request_output.metrics into a numeric dictionary.

        vLLM may return a dataclass-like object or a plain dict depending on version.
        """
        if raw_metrics is None:
            return {}

        numeric: Dict[str, float] = {}
        if isinstance(raw_metrics, dict):
            items = raw_metrics.items()
        else:
            names = (
                "arrival_time",
                "first_scheduled_time",
                "first_token_time",
                "finished_time",
                "time_in_queue",
                "scheduler_time",
                "model_forward_time",
                "model_execute_time",
            )
            items = []
            for name in names:
                if hasattr(raw_metrics, name):
                    items.append((name, getattr(raw_metrics, name)))

        for key, value in items:
            parsed = self._to_number(value)
            if parsed is not None:
                numeric[str(key)] = parsed
        return numeric

    def _apply_stats(self, perf: Dict[str, Any], prefix: str, values: List[float]) -> None:
        """Populate total/min/max/p50 stats for a list of ms samples."""
        if not values:
            return
        perf[f"{prefix}_total"] = float(sum(values))
        perf[f"{prefix}_min"] = float(min(values))
        perf[f"{prefix}_max"] = float(max(values))
        perf[f"{prefix}_p50"] = float(self._p50(values))

    async def generate_speech_profiled(
        self,
        speaker: str,
        text: str,
        temperature: float = DEFAULT_TEMPERATURE,
        top_k: int = DEFAULT_TOP_K,
        top_p: float = DEFAULT_TOP_P,
        max_tokens: int = DEFAULT_MAX_TOKENS,
        repetition_penalty: float = 1.05,
        seed: Optional[int] = None,
    ) -> Tuple[Optional[bytes], Dict[str, Any]]:
        """
        Generate speech with detailed per-request timing metrics.

        Returns:
            (wav_bytes, perf_dict)
        """
        from veena3modal.audio.utils import add_wav_header

        perf: Dict[str, Any] = {
            "llm_prompt_token_total": 0,
            "llm_token_total": 0,
            "semantic_token_total": 0,
            "global_token_total": 0,
            "llm_batch_count": 0,
            "llm_parse_ms": 0.0,
            "llm_decode_calls": 0,
            "bicodec_decode_wall_ms": 0.0,
            "bicodec_decode_gpu_ms": 0.0,
            "bicodec_decode_cpu_ms": 0.0,
            "wav_pack_ms": 0.0,
        }

        t_request_start = time.perf_counter()
        timeline_marks: List[Dict[str, float]] = [{"stage": "request_start", "t_ms": 0.0, "dt_ms": 0.0}]
        last_mark_ms = 0.0
        perf["timeline_markers"] = timeline_marks
        perf["timeline_request_start_ms"] = 0.0

        def _mark(stage: str) -> float:
            nonlocal last_mark_ms
            now_ms = (time.perf_counter() - t_request_start) * 1000
            dt_ms = now_ms - last_mark_ms
            last_mark_ms = now_ms
            timeline_marks.append(
                {
                    "stage": stage,
                    "t_ms": float(now_ms),
                    "dt_ms": float(dt_ms),
                }
            )
            perf[f"timeline_{stage}_ms"] = float(now_ms)
            perf[f"timeline_{stage}_delta_ms"] = float(dt_ms)
            return float(now_ms)

        def _finalize_timeline() -> None:
            if not timeline_marks or timeline_marks[-1].get("stage") != "request_done":
                _mark("request_done")

            # Keep a compact, ordered timeline payload for downstream benchmarking.
            perf["timeline_markers"] = list(timeline_marks)
            perf["timeline_total_ms"] = float(perf.get("timeline_request_done_ms", 0.0))

            first_batch_ms = perf.get("timeline_llm_first_batch_ms")
            llm_done_ms = perf.get("timeline_llm_done_ms")
            parse_done_ms = perf.get("timeline_parse_done_ms")
            bicodec_done_ms = perf.get("timeline_bicodec_done_ms")
            request_done_ms = perf.get("timeline_request_done_ms")

            if isinstance(first_batch_ms, (int, float)):
                perf["timeline_to_first_batch_ms"] = float(first_batch_ms)
            if isinstance(first_batch_ms, (int, float)) and isinstance(llm_done_ms, (int, float)) and llm_done_ms >= first_batch_ms:
                perf["timeline_first_batch_to_llm_done_ms"] = float(llm_done_ms - first_batch_ms)
            if isinstance(llm_done_ms, (int, float)) and isinstance(request_done_ms, (int, float)) and request_done_ms >= llm_done_ms:
                perf["timeline_post_llm_ms"] = float(request_done_ms - llm_done_ms)
            if isinstance(parse_done_ms, (int, float)) and isinstance(bicodec_done_ms, (int, float)) and bicodec_done_ms >= parse_done_ms:
                perf["timeline_parse_to_bicodec_done_ms"] = float(bicodec_done_ms - parse_done_ms)
            if isinstance(bicodec_done_ms, (int, float)) and isinstance(request_done_ms, (int, float)) and request_done_ms >= bicodec_done_ms:
                perf["timeline_bicodec_to_request_done_ms"] = float(request_done_ms - bicodec_done_ms)

        # Stage 1: prompt build
        t0 = time.perf_counter()
        prompt = self.prompt_builder.build_prefix(speaker, text)
        perf["prompt_build_ms"] = (time.perf_counter() - t0) * 1000
        _mark("prompt_ready")

        # Prompt token count (for TPS normalization)
        try:
            perf["llm_prompt_token_total"] = len(
                self.model.tokenizer.encode(prompt, add_special_tokens=False)
            )
        except Exception:
            perf["llm_prompt_token_total"] = 0

        # Stage 2: sampling params
        t0 = time.perf_counter()
        sampling_params = SamplingParams(
            temperature=temperature,
            top_k=top_k,
            top_p=top_p,
            max_tokens=max_tokens,
            repetition_penalty=repetition_penalty,
            stop=TRAINING_STOP_TOKEN_IDS,
            skip_special_tokens=False,
            seed=seed,
        )
        perf["sampling_params_ms"] = (time.perf_counter() - t0) * 1000
        _mark("sampling_ready")

        request_id = f"req-{uuid.uuid4().hex[:12]}"
        _log_perf_stage(
            "perf_non_stream_start",
            extra={
                "request_id": request_id,
                "speaker": speaker,
                "text_length": len(text),
                "max_tokens": max_tokens,
            },
        )
        results_generator = self.model.engine.generate(
            prompt=prompt,
            sampling_params=sampling_params,
            request_id=request_id,
        )

        final_output = None
        final_request_metrics: Dict[str, float] = {}

        llm_batch_wall_ms: List[float] = []
        llm_batch_gpu_ms: List[float] = []
        llm_decode_wall_ms: List[float] = []
        llm_decode_gpu_ms: List[float] = []
        tokens_per_batch: List[float] = []

        prev_generated_count = 0
        prev_gpu_cumulative_s: Optional[float] = None

        t_gen_start = time.perf_counter()
        t_last_yield = t_gen_start
        async for request_output in results_generator:
            now = time.perf_counter()
            batch_wall = (now - t_last_yield) * 1000
            t_last_yield = now

            outputs = request_output.outputs or []
            if not outputs:
                final_output = request_output
                continue

            output = outputs[0]
            generated_count = len(output.token_ids)
            new_tokens = max(0, generated_count - prev_generated_count)
            prev_generated_count = generated_count

            raw_req_metrics = self._extract_request_metrics(getattr(request_output, "metrics", None))
            if raw_req_metrics:
                final_request_metrics = raw_req_metrics

            if new_tokens <= 0:
                final_output = request_output
                continue

            perf["llm_batch_count"] += 1
            if perf["llm_batch_count"] == 1:
                first_batch_timeline_ms = _mark("llm_first_batch")
                _log_perf_stage(
                    "perf_first_llm_batch",
                    extra={
                        "request_id": request_id,
                        "batch_wall_ms": round(batch_wall, 3),
                        "new_tokens": new_tokens,
                        "timeline_ms": round(first_batch_timeline_ms, 3),
                    },
                )
            tokens_per_batch.append(float(new_tokens))
            llm_batch_wall_ms.append(float(batch_wall))
            if perf["llm_batch_count"] > 1:
                llm_decode_wall_ms.append(float(batch_wall))

            cumulative_gpu_s: Optional[float] = None
            if raw_req_metrics:
                if "model_execute_time" in raw_req_metrics:
                    cumulative_gpu_s = raw_req_metrics["model_execute_time"]
                elif "model_forward_time" in raw_req_metrics:
                    cumulative_gpu_s = raw_req_metrics["model_forward_time"]

            if cumulative_gpu_s is not None: