Model/runtime initialization is still in progress for this benchmark run; once it clears cold start I’ll report the steady-state concurrency metrics from the actual request levels.


Output:
Chunk ID: eed2e9
Wall time: 5.0011 seconds
Process running with session ID 22576
Original token count: 762
Output:
(EngineCore_DP0 pid=3810286) INFO 02-17 04:31:09 [backends.py:267] Directly load the compiled graph(s) for compile range (1, 4096) from the cache, took 0.651 s
(EngineCore_DP0 pid=3810286) INFO 02-17 04:31:09 [monitor.py:34] torch.compile takes 4.88 s in total
(EngineCore_DP0 pid=3810286) INFO 02-17 04:31:10 [gpu_worker.py:356] Available KV cache memory: 18.06 GiB
(EngineCore_DP0 pid=3810286) INFO 02-17 04:31:10 [kv_cache_utils.py:1307] GPU KV cache size: 1,577,712 tokens
(EngineCore_DP0 pid=3810286) INFO 02-17 04:31:10 [kv_cache_utils.py:1312] Maximum concurrency for 4,096 tokens per request: 385.18x
(EngineCore_DP0 pid=3810286)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   0%|          | 0/35 [00:00<?, ?it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  11%|█▏        | 4/35 [00:00<00:00, 33.48it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  23%|██▎       | 8/35 [00:00<00:00, 35.60it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  34%|███▍      | 12/35 [00:00<00:00, 36.60it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  46%|████▌     | 16/35 [00:00<00:00, 36.97it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  57%|█████▋    | 20/35 [00:00<00:00, 37.19it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  69%|██████▊   | 24/35 [00:00<00:00, 36.87it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  80%|████████  | 28/35 [00:00<00:00, 36.26it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  91%|█████████▏| 32/35 [00:00<00:00, 35.00it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 35/35 [00:00<00:00, 35.25it/s]
(EngineCore_DP0 pid=3810286)
Capturing CUDA graphs (decode, FULL):   0%|          | 0/19 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL):  21%|██        | 4/19 [00:00<00:00, 31.18it/s]
Capturing CUDA graphs (decode, FULL):  42%|████▏     | 8/19 [00:00<00:00, 34.67it/s]
Capturing CUDA graphs (decode, FULL):  63%|██████▎   | 12/19 [00:00<00:00, 36.11it/s]
Capturing CUDA graphs (decode, FULL):  84%|████████▍ | 16/19 [00:00<00:00, 36.51it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 36.15it/s]
(EngineCore_DP0 pid=3810286) INFO 02-17 04:31:12 [gpu_model_runner.py:5063] Graph capturing finished in 2 secs, took 0.28 GiB
(EngineCore_DP0 pid=3810286) INFO 02-17 04:31:13 [core.py:272] init engine (profile, create kv cache, warmup model) took 10.72 seconds
[2026-02-17 04:31:14] INFO tts_runtime.py:162: Initializing prompt builder...
[2026-02-17 04:31:14] INFO tts_runtime.py:169: Initializing BiCodec decoder...
/home/ubuntu/.local/lib/python3.10/site-packages/torch/nn/utils/weight_norm.py:144: FutureWarning: `torch.nn.utils.weight_norm` is deprecated in favor of `torch.nn.utils.parametrizations.weight_norm`.
  WeightNorm.apply(module, name, dim)


# Collaboration Mode: Default

You are now in Default mode. Any previous instructions for other modes (e.g. Plan mode) are no longer active.

Your active mode changes only when new developer instructions with a different `<collaboration_mode>...</collaboration_mode>` change it; user requests or tool descriptions do not change mode by themselves. Known mode names are Default and Plan.

## request_user_input availability

The `request_user_input` tool is unavailable in Default mode. If you call it while in Default mode, it will return an error.

If a decision is necessary and cannot be discovered from local context, ask the user directly. However, in Default mode you should strongly prefer executing the user's request rather than stopping to ask questions.

Summary: auto

## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) When `SKILL.md` references relative paths (e.g., `scripts/foo.py`), resolve them relative to the skill directory listed above first, and only consider other paths if needed.
  3) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  4) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  5) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.


Output:
Chunk ID: c2c8c0
Wall time: 5.0010 seconds
Process running with session ID 22576
Original token count: 1014
Output:
[2026-02-17 04:31:20] INFO tts_runtime.py:176: Initializing TTS pipeline...
[2026-02-17 04:31:20] INFO tts_runtime.py:187: Initializing streaming pipeline...
[2026-02-17 04:31:20] INFO tts_runtime.py:237: ✅ TTS runtime initialized in 31176ms
[2026-02-17 04:31:20] INFO tts_runtime.py:238:    Model version: LLM
[2026-02-17 04:31:20] INFO tts_runtime.py:249: Speaker globals cache startup precompute skipped (disabled)
[2026-02-17 04:31:20] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:20] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:20] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:20] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:21] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:21] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:21] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:21] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:21] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:22] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:22] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:22] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:22] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:22] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:22] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:22] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:22] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:22] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:22] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:22] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:22] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:22] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:22] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:23] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:23] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:23] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:23] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:23] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:23] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:23] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:23] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:23] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:23] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:23] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:23] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:23] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:23] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:24] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:24] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:24] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:24] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:24] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:24] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:24] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:24] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:24] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:24] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:24] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:24] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False


Summary: auto


**Setting up output capture**


Output:
Chunk ID: d47a1f
Wall time: 5.0011 seconds
Process running with session ID 22576
Original token count: 3636
Output:
[2026-02-17 04:31:26] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:26] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:26] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:26] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:27] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:27] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:27] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:27] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:27] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:27] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:27] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:27] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:27] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:27] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:27] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:27] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:27] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:27] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:27] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:27] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:27] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:27] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:27] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:27] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:27] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:27] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:27] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:27] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:27] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:27] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:27] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:27] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:27] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:27] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:28] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:28] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:28] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:28] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:28] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:28] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:28] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:28] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:30] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:30] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:30] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:30] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:30] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:30] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:30] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:30] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:30] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:30] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:30] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:30] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:30] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:30] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:30] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:30] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:30] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:30] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:30] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:30] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:30] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:30] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:30] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:30] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:30] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:30] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:30] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:30] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:30] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:30] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:30] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:30] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:30] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:30] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:30] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:30] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:30] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:30] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:31] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:31] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:31] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:31] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:31] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:31] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:31] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:31] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:31] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:31] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:31] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:31] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:31] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:31] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:31] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:31] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:31] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:31] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:31] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:31] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:31] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:31] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:31] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:31] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:31] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:31] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:31] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:31] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:31] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:31] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:31] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:31] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:31] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:31] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:31] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:31] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:31] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:31] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:31] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:31] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:31] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:31] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:31] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:31] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:31] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:31] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:33] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:33] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:33] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:33] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:33] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:33] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:33] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:33] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:33] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:33] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:33] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:33] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:33] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:33] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:33] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:33] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:34] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:34] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:34] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:34] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:34] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:34] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:34] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:34] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:34] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:34] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:34] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:34] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:34] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:34] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:34] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:34] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:34] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:34] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:34] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:34] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:34] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:34] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:34] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:34] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:34] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:34] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:34] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:34] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:34] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:34] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:34] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:34] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:34] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:34] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:34] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:34] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:34] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:34] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:34] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:34] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:34] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:34] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:34] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:34] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:34] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:34] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:34] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:34] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:34] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:34] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:34] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:34] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:34] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:34] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:34] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:34] INFO pipeline.py:236: perf_first_llm_batch


**Scheduling extended polling for summary**


Output:
Chunk ID: 0105b4
Wall time: 4.0407 seconds
Process exited with code 0
Original token count: 8984
Output:
[2026-02-17 04:31:34] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:34] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:34] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:34] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:34] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:34] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:34] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:34] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:34] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:34] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:34] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:34] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:34] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:34] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:34] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:34] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:34] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:34] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:34] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:34] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:34] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:34] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:34] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:34] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:34] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:34] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:34] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:34] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:34] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:34] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:34] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:34] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:34] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:34] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:34] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:34] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:34] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:34] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:34] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:34] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:34] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:34] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:34] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:34] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:34] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:34] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:34] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:34] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:34] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:34] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:34] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:34] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:34] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:34] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:34] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:34] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:35] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:35] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:35] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:35] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:35] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:35] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:35] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:35] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:35] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:35] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:35] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:35] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:35] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:35] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:35] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:35] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:35] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:35] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:35] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:35] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:35] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:35] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:35] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:35] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:35] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:35] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:35] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:35] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:35] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:35] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:35] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:35] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:35] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:35] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:35] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:35] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:35] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:35] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:35] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:35] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:37] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:37] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:37] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:37] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:37] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:37] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:37] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:37] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:37] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:37] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:37] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:37] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:37] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:37] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:37] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:37] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:37] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:37] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:37] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:37] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:37] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:37] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:37] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:37] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:37] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:37] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:37] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:37] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:37] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:37] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:37] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:37] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:37] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:37] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:37] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:37] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:37] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:37] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:37] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:37] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:37] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:37] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:37] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:37] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:37] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:37] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:37] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:37] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:37] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:37] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:37] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:37] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:37] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:37] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:37] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:37] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:37] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:37] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:37] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:37] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:37] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:37] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:37] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:37] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:37] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:37] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:37] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:37] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:37] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:37] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:37] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:37] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:37] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:37] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:37] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:37] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:37] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:37] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:37] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:37] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:37] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:37] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:37] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:37] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:37] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:37] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:37] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:37] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:37] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:37] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:37] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:37] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:37] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:37] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:37] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:37] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:37] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:37] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:37] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:37] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:37] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:37] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:37] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:37] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:37] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:37] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:37] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:37] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:37] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:37] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:37] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:37] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:37] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:37] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:37] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:37] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:37] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:37] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:37] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:37] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:37] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:37] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:37] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:37] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:37] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:37] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:37] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:37] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:38] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:38] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:38] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:38] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:38] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:38] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:38] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:38] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:38] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:38] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:38] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:38] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:38] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:38] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:38] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:38] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:38] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:38] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:38] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:38] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:38] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:38] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:38] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:38] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:38] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:38] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:38] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:38] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:38] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:38] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:38] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:38] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:38] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:38] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:38] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:38] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:38] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:38] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:38] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:38] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:38] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:38] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:38] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:38] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:38] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:38] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:38] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:38] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:38] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:38] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:38] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:38] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:38] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:38] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:38] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:38] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:38] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:38] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:38] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:38] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:38] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:38] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:38] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:38] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:38] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:38] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:38] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:38] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:38] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:38] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:38] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:38] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:38] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:38] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:38] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:38] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:38] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:38] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:38] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:38] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:38] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:38] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:38] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:38] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:38] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:38] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:38] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:38] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:38] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:38] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:38] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:38] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:38] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:38] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:38] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:38] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:38] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:38] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:38] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:38] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:38] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:38] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:38] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:38] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:38] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:38] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:38] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:38] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:38] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:38] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:38] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:38] INFO pipeline.py:182: perf_non_stream_start
[2026-02-17 04:31:38] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:38] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:38] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:38] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:38] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:38] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:38] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:38] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:38] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:38] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:38] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:38] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:38] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:38] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:38] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:38] INFO pipeline.py:236: perf_first_llm_batch
[2026-02-17 04:31:39] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:39] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:39] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:39] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:39] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:39] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:39] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:39] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:39] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:39] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:39] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:39] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:39] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:39] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:39] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:39] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:39] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:39] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:39] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:39] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:39] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:39] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:39] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:39] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:39] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:39] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:39] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:39] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:39] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:39] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:39] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:39] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:39] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:39] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:39] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:39] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:39] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:39] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:39] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:39] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:39] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:39] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:39] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:39] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:39] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:39] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:39] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:39] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:39] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:39] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:39] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:39] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:39] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:39] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:39] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:39] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:39] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:39] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:39] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:39] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:39] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:39] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:39] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:39] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:39] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:39] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:39] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:39] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:39] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:39] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:39] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:39] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:39] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:39] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:39] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
[2026-02-17 04:31:39] INFO pipeline.py:274: perf_llm_generation_done
[2026-02-17 04:31:39] INFO pipeline.py:297: perf_token_parse_done
[2026-02-17 04:31:39] INFO pipeline.py:397: perf_bicodec_decode_done
[2026-02-17 04:31:39] INFO pipeline.py:417: perf_non_stream_done
[2026-02-17 04:31:39] INFO tts_runtime.py:427: SR check: output_sample_rate='16khz', sr_service=False, is_loaded=False
✅ vLLM engine initialized successfully!

🎉 Spark TTS Model ready for inference!

🎵 Loading BiCodec audio tokenizer from /home/ubuntu/veenaModal/models/spark_tts_4speaker...
   Device: cuda
Missing tensor: mel_transformer.spectrogram.window
Missing tensor: mel_transformer.mel_scale.fb
✅ BiCodec decoder initialized (sample rate: 16000Hz)
🚀 SparkTTSPipeline initialized
🌊 Veena3SlidingWindowPipeline initialized (sliding window: 28 tokens)

=== concurrency=1 requests=6 ===
success=6/6 (100%)
wall=2.65s throughput=2.27 req/s
latency_ms avg=441 p50=413 p95=439 max=560
generation_ms: avg=440.67 p50=413.00 p95=438.00 max=559.00
llm_batch_wall_ms_total: avg=428.32 p50=400.58 p95=425.74 max=546.37
llm_decode_wall_ms_total: avg=416.73 p50=389.41 p95=414.92 max=532.50
llm_parse_ms: avg=0.08 p50=0.08 p95=0.08 max=0.08
llm_time_per_token_ms: avg=3.35 p50=3.13 p95=3.33 max=4.27
bicodec_decode_wall_ms: avg=11.76 p50=11.63 p95=12.15 max=12.22
bicodec_decode_gpu_ms: avg=11.55 p50=11.45 p95=11.89 max=11.98
gpu: util_avg=57% util_max=69% mem_avg=23601MB mem_max=23601MB

=== concurrency=2 requests=6 ===
success=6/6 (100%)
wall=1.45s throughput=4.14 req/s
latency_ms avg=482 p50=467 p95=507 max=507
generation_ms: avg=477.00 p50=466.00 p95=496.00 max=505.00
llm_batch_wall_ms_total: avg=464.85 p50=453.65 p95=479.99 max=494.71
llm_decode_wall_ms_total: avg=448.91 p50=438.60 p95=464.25 max=475.93
llm_parse_ms: avg=0.07 p50=0.07 p95=0.08 max=0.09
llm_time_per_token_ms: avg=3.63 p50=3.54 p95=3.75 max=3.87
bicodec_decode_wall_ms: avg=11.65 p50=10.13 p95=12.40 max=14.96
bicodec_decode_gpu_ms: avg=11.48 p50=10.04 p95=12.17 max=14.71
gpu: util_avg=58% util_max=69% mem_avg=23601MB mem_max=23601MB

=== concurrency=4 requests=12 ===
success=12/12 (100%)
wall=1.51s throughput=7.97 req/s
latency_ms avg=499 p50=494 p95=529 max=529
generation_ms: avg=483.42 p50=479.00 p95=514.00 max=524.00
llm_batch_wall_ms_total: avg=472.27 p50=469.20 p95=503.76 max=514.07
llm_decode_wall_ms_total: avg=454.77 p50=451.61 p95=484.42 max=495.21
llm_parse_ms: avg=0.07 p50=0.07 p95=0.10 max=0.12
llm_time_per_token_ms: avg=3.69 p50=3.67 p95=3.94 max=4.02
bicodec_decode_wall_ms: avg=11.04 p50=10.44 p95=13.08 max=13.83
bicodec_decode_gpu_ms: avg=10.93 p50=10.35 p95=12.84 max=13.61
gpu: util_avg=70% util_max=71% mem_avg=23601MB mem_max=23601MB

=== concurrency=8 requests=24 ===
success=24/24 (100%)
wall=1.73s throughput=13.85 req/s
latency_ms avg=572 p50=580 p95=591 max=591
generation_ms: avg=533.92 p50=528.00 p95=571.00 max=586.00
llm_batch_wall_ms_total: avg=522.10 p50=515.61 p95=558.77 max=571.17
llm_decode_wall_ms_total: avg=500.88 p50=498.35 p95=541.47 max=554.35
llm_parse_ms: avg=0.07 p50=0.06 p95=0.09 max=0.12
llm_time_per_token_ms: avg=4.08 p50=4.03 p95=4.37 max=4.46
bicodec_decode_wall_ms: avg=11.65 p50=10.50 p95=14.70 max=16.35
bicodec_decode_gpu_ms: avg=11.54 p50=10.43 p95=14.59 max=16.25
gpu: util_avg=50% util_max=78% mem_avg=23601MB mem_max=23601MB

=== concurrency=16 requests=48 ===
success=48/48 (100%)
wall=2.16s throughput=22.26 req/s
latency_ms avg=715 p50=720 p95=732 max=732
generation_ms: avg=630.15 p50=630.00 p95=702.00 max=722.00
llm_batch_wall_ms_total: avg=619.34 p50=619.52 p95=691.50 max=712.29
llm_decode_wall_ms_total: avg=598.46 p50=596.62 p95=671.87 max=693.59
llm_parse_ms: avg=0.07 p50=0.07 p95=0.09 max=0.12
llm_time_per_token_ms: avg=4.84 p50=4.84 p95=5.40 max=5.56
bicodec_decode_wall_ms: avg=10.75 p50=10.43 p95=12.46 max=13.00
bicodec_decode_gpu_ms: avg=10.65 p50=10.35 p95=12.27 max=12.73
gpu: util_avg=49% util_max=67% mem_avg=23601MB mem_max=23601MB

results saved: /tmp/stress_runtime_detailed_fixed.json
ERROR 02-17 04:31:39 [core_client.py:605] Engine core proc EngineCore_DP0 died unexpectedly, shutting down client.


**Planning local server benchmarking**


The direct-runtime stress run completed successfully with full timing breakdown through `16` concurrent requests. I’m now running one HTTP-path benchmark (with perf headers and in-flight counters) so we can also analyze request-level concurrency bottlenecks from the API surface.


**Starting server with precompute disabled**


nohup python3 -m veena3modal.local_server --model-path /home/ubuntu/veenaModal/models/spark_tts_4speaker --gpu-memory 0.25 --port 8000 > /tmp/veena_local_server.log 2>&1 & echo $!


Output:
Chunk ID: 4852f0
Wall time: 0.0511 seconds
Process exited with code 0
Original token count: 2
Output:
3810636


**Implementing health check loop**


for i in $(seq 1 120); do if curl -sf http://localhost:8000/v1/tts/health >/tmp/health.json; then cat /tmp/health.json; exit 0; fi; sleep 1; done; echo 'health timeout'; tail -n 80 /tmp/veena_local_server.log; exit 1


Output:
Chunk ID: 2aaed3
Wall time: 1.0011 seconds
Process running with session ID 98390
Original token count: 0
Output:


**Requesting clarification for next step**


Output:
Chunk ID: ed14ee
Wall time: 5.0013 seconds
Process running with session ID 98390
Original token count: 0
Output:


**Extending wait time for initialization**


Output:
Chunk ID: 27fb79
Wall time: 20.0009 seconds
Process running with session ID 98390
Original token count: 0
Output:


**Waiting for process readiness**


The local server is warming up in the background; I’m waiting for health to flip before starting the HTTP stress run so we capture clean header-level timing/inflight data.


Output:
Chunk ID: 5151d9
Wall time: 30.0006 seconds
Process running with session ID 98390
Original token count: 0
Output:


**Waiting for command completion**


Output:
Chunk ID: d7ccf9
Wall time: 30.0015 seconds
Process running with session ID 98390
Original token count: 0
Output:


**Waiting to poll again**


Output:
Chunk ID: e9ea89
Wall time: 17.9879 seconds
Process exited with code 1
Original token count: 477
Output:
health timeout
2026-02-17 04:31:49 | INFO     | veena3.local | Auth bypass enabled (local mode). Use --auth to require API keys.
2026-02-17 04:31:49 | INFO     | veena3.local | Resolved model paths:
2026-02-17 04:31:49 | INFO     | veena3.local |   LLM (vLLM):    /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
2026-02-17 04:31:49 | INFO     | veena3.local |   BiCodec:       /home/ubuntu/veenaModal/models/spark_tts_4speaker
2026-02-17 04:31:49 | INFO     | veena3.local | ============================================================
2026-02-17 04:31:49 | INFO     | veena3.local | Initializing Veena3 TTS Runtime (local)
2026-02-17 04:31:49 | INFO     | veena3.local |   Model:       /home/ubuntu/veenaModal/models/spark_tts_4speaker
2026-02-17 04:31:49 | INFO     | veena3.local |   LLM path:    /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
2026-02-17 04:31:49 | INFO     | veena3.local |   BiCodec:     /home/ubuntu/veenaModal/models/spark_tts_4speaker
2026-02-17 04:31:49 | INFO     | veena3.local |   Device:      cuda
2026-02-17 04:31:49 | INFO     | veena3.local |   GPU Memory:  25% (total)
2026-02-17 04:31:49 | INFO     | veena3.local |   Engines:     1
2026-02-17 04:31:49 | INFO     | veena3.local |   Precompute globals: disabled
2026-02-17 04:31:49 | INFO     | veena3.local |   SR:          disabled
2026-02-17 04:31:49 | INFO     | veena3.local | ============================================================
2026-02-17 04:31:49 | INFO     | veena3modal.services.tts_runtime | Initializing TTS runtime...
2026-02-17 04:31:49 | INFO     | veena3modal.services.tts_runtime |   Model path: /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
2026-02-17 04:31:49 | INFO     | veena3modal.services.tts_runtime |   BiCodec path: /home/ubuntu/veenaModal/models/spark_tts_4speaker
2026-02-17 04:31:49 | INFO     | veena3modal.services.tts_runtime |   SR enabled: False, path: None


**Investigating GPU occupancy and server hang**


ps -eo pid,ppid,cmd | rg "veena3modal.local_server|VLLM::EngineCore|python3 scripts/stress_test_runtime_detailed|uvicorn" -n


Output:
Chunk ID: 4d1781
Wall time: 0.0519 seconds
Process exited with code 0
Original token count: 70
Output:
312:3811565 3675569 /bin/bash -c ps -eo pid,ppid,cmd | rg "veena3modal.local_server|VLLM::EngineCore|python3 scripts/stress_test_runtime_detailed|uvicorn" -n
314:3811568 3811565 rg veena3modal.local_server|VLLM::EngineCore|python3 scripts/stress_test_runtime_detailed|uvicorn -n


Output:
Chunk ID: 986650
Wall time: 0.0508 seconds
Process exited with code 0
Original token count: 4
Output:
0, 1, 81920, 29


Output:
Chunk ID: e3019c
Wall time: 0.0508 seconds
Process exited with code 0
Original token count: 473
Output:
2026-02-17 04:31:49 | INFO     | veena3.local | Auth bypass enabled (local mode). Use --auth to require API keys.
2026-02-17 04:31:49 | INFO     | veena3.local | Resolved model paths:
2026-02-17 04:31:49 | INFO     | veena3.local |   LLM (vLLM):    /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
2026-02-17 04:31:49 | INFO     | veena3.local |   BiCodec:       /home/ubuntu/veenaModal/models/spark_tts_4speaker
2026-02-17 04:31:49 | INFO     | veena3.local | ============================================================
2026-02-17 04:31:49 | INFO     | veena3.local | Initializing Veena3 TTS Runtime (local)
2026-02-17 04:31:49 | INFO     | veena3.local |   Model:       /home/ubuntu/veenaModal/models/spark_tts_4speaker
2026-02-17 04:31:49 | INFO     | veena3.local |   LLM path:    /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
2026-02-17 04:31:49 | INFO     | veena3.local |   BiCodec:     /home/ubuntu/veenaModal/models/spark_tts_4speaker
2026-02-17 04:31:49 | INFO     | veena3.local |   Device:      cuda
2026-02-17 04:31:49 | INFO     | veena3.local |   GPU Memory:  25% (total)
2026-02-17 04:31:49 | INFO     | veena3.local |   Engines:     1
2026-02-17 04:31:49 | INFO     | veena3.local |   Precompute globals: disabled
2026-02-17 04:31:49 | INFO     | veena3.local |   SR:          disabled
2026-02-17 04:31:49 | INFO     | veena3.local | ============================================================
2026-02-17 04:31:49 | INFO     | veena3modal.services.tts_runtime | Initializing TTS runtime...
2026-02-17 04:31:49 | INFO     | veena3modal.services.tts_runtime |   Model path: /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
2026-02-17 04:31:49 | INFO     | veena3modal.services.tts_runtime |   BiCodec path: /home/ubuntu/veenaModal/models/spark_tts_4speaker
2026-02-17 04:31:49 | INFO     | veena3modal.services.tts_runtime |   SR enabled: False, path: None


Summary: auto


**Diagnosing silent server crash**


The background HTTP server process exited during startup before binding the port, so I’m launching it in the foreground once to capture the exact startup error and then rerunning the HTTP benchmark.


python3 -m veena3modal.local_server --model-path /home/ubuntu/veenaModal/models/spark_tts_4speaker --gpu-memory 0.25 --port 8000


Output:
Chunk ID: c35b0a
Wall time: 1.0011 seconds
Process running with session ID 52189
Original token count: 473
Output:
2026-02-17 04:34:04 | INFO     | veena3.local | Auth bypass enabled (local mode). Use --auth to require API keys.
2026-02-17 04:34:04 | INFO     | veena3.local | Resolved model paths:
2026-02-17 04:34:04 | INFO     | veena3.local |   LLM (vLLM):    /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
2026-02-17 04:34:04 | INFO     | veena3.local |   BiCodec:       /home/ubuntu/veenaModal/models/spark_tts_4speaker
2026-02-17 04:34:04 | INFO     | veena3.local | ============================================================
2026-02-17 04:34:04 | INFO     | veena3.local | Initializing Veena3 TTS Runtime (local)
2026-02-17 04:34:04 | INFO     | veena3.local |   Model:       /home/ubuntu/veenaModal/models/spark_tts_4speaker
2026-02-17 04:34:04 | INFO     | veena3.local |   LLM path:    /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
2026-02-17 04:34:04 | INFO     | veena3.local |   BiCodec:     /home/ubuntu/veenaModal/models/spark_tts_4speaker
2026-02-17 04:34:04 | INFO     | veena3.local |   Device:      cuda
2026-02-17 04:34:04 | INFO     | veena3.local |   GPU Memory:  25% (total)
2026-02-17 04:34:04 | INFO     | veena3.local |   Engines:     1
2026-02-17 04:34:04 | INFO     | veena3.local |   Precompute globals: disabled
2026-02-17 04:34:04 | INFO     | veena3.local |   SR:          disabled
2026-02-17 04:34:04 | INFO     | veena3.local | ============================================================
2026-02-17 04:34:04 | INFO     | veena3modal.services.tts_runtime | Initializing TTS runtime...
2026-02-17 04:34:04 | INFO     | veena3modal.services.tts_runtime |   Model path: /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
2026-02-17 04:34:04 | INFO     | veena3modal.services.tts_runtime |   BiCodec path: /home/ubuntu/veenaModal/models/spark_tts_4speaker
2026-02-17 04:34:04 | INFO     | veena3modal.services.tts_runtime |   SR enabled: False, path: None


**Continuing polling for updates**


Output:
Chunk ID: 846d73
Wall time: 5.0014 seconds
Process running with session ID 52189
Original token count: 1368
Output:
2026-02-17 04:34:10 | INFO     | veena3modal.services.tts_runtime | Loading SparkTTS model with vLLM...
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
🚀 Initializing Spark TTS Model
📁 Model path: /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
🗣  Speakers: 12 predefined (lipakshi, vardan, reet, Nandini, krishna, anika, adarsh, Nilay, Aarvi, Asha, Bittu, Mira)
🎭 Emotions: 10 tags in [bracket] format

📝 Loading tokenizer from model...
✅ Tokenizer loaded: 165168 tokens

🔍 Validating 12 speakers for Spark TTS...
  lipakshi → <|speaker_0|>: 7 tokens
  vardan → <|speaker_1|>: 7 tokens
  reet → <|speaker_2|>: 7 tokens
  Nandini → <|speaker_3|>: 7 tokens
  krishna → <|speaker_4|>: 7 tokens
  anika → <|speaker_5|>: 7 tokens
  adarsh → <|speaker_6|>: 7 tokens
  Nilay → <|speaker_7|>: 7 tokens
  Aarvi → <|speaker_8|>: 7 tokens
  Asha → <|speaker_9|>: 7 tokens
  Bittu → <|speaker_10|>: 8 tokens
  Mira → <|speaker_11|>: 8 tokens
✅ All 12 speakers validated with mapping

🔍 Validating 10 emotion tags for Spark TTS...
✅ All 10 emotion tags validated

📋 Sample emotion tags:
  [angry]: 1 tokens
  [curious]: 1 tokens
  [excited]: 1 tokens
  [giggle]: 1 tokens
  [laughs harder]: 1 tokens
  ...

🔧 Initializing vLLM engine with optimizations...
  - dtype: bfloat16
  - max_model_len: 4096
  - gpu_memory_utilization: 0.25
  - enable_prefix_caching: True
  - enforce_eager (disable CUDA graphs): False
INFO 02-17 04:34:11 [model.py:541] Resolved architecture: Qwen2ForCausalLM
INFO 02-17 04:34:11 [model.py:1561] Using max model len 4096
INFO 02-17 04:34:11 [scheduler.py:226] Chunked prefill is enabled with max_num_batched_tokens=4096.
INFO 02-17 04:34:11 [vllm.py:624] Asynchronous scheduling is enabled.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(EngineCore_DP0 pid=3811746) INFO 02-17 04:34:12 [core.py:96] Initializing a V1 LLM engine (v0.15.1) with config: model='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', speculative_config=None, tokenizer='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [4096], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 256, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': True}, 'local_cache_dir': None, 'static_all_moe_layers': []}
(EngineCore_DP0 pid=3811746) INFO 02-17 04:34:14 [parallel_state.py:1212] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://216.81.248.184:41725 backend=nccl
(EngineCore_DP0 pid=3811746) INFO 02-17 04:34:14 [parallel_state.py:1423] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A


**Assessing startup timing issue**


Output:
Chunk ID: 015350
Wall time: 5.0004 seconds
Process running with session ID 52189
Original token count: 578
Output:
(EngineCore_DP0 pid=3811746) INFO 02-17 04:34:14 [gpu_model_runner.py:4033] Starting to load model /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM...
(EngineCore_DP0 pid=3811746) INFO 02-17 04:34:15 [cuda.py:364] Using FLASH_ATTN attention backend out of potential backends: ('FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION')
(EngineCore_DP0 pid=3811746) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
(EngineCore_DP0 pid=3811746) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
(EngineCore_DP0 pid=3811746)
Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
(EngineCore_DP0 pid=3811746)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.82it/s]
(EngineCore_DP0 pid=3811746)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.82it/s]
(EngineCore_DP0 pid=3811746)
(EngineCore_DP0 pid=3811746) INFO 02-17 04:34:16 [default_loader.py:291] Loading weights took 0.61 seconds
(EngineCore_DP0 pid=3811746) INFO 02-17 04:34:17 [gpu_model_runner.py:4130] Model loading took 0.95 GiB memory and 1.152603 seconds
(EngineCore_DP0 pid=3811746) INFO 02-17 04:34:21 [backends.py:812] Using cache directory: /home/ubuntu/.cache/vllm/torch_compile_cache/f3ff13511c/rank_0_0/backbone for vLLM's torch.compile
(EngineCore_DP0 pid=3811746) INFO 02-17 04:34:21 [backends.py:872] Dynamo bytecode transform time: 4.35 s
(EngineCore_DP0 pid=3811746) INFO 02-17 04:34:24 [backends.py:267] Directly load the compiled graph(s) for compile range (1, 4096) from the cache, took 0.651 s
(EngineCore_DP0 pid=3811746) INFO 02-17 04:34:24 [monitor.py:34] torch.compile takes 5.00 s in total
(EngineCore_DP0 pid=3811746) INFO 02-17 04:34:25 [gpu_worker.py:356] Available KV cache memory: 18.06 GiB
(EngineCore_DP0 pid=3811746) INFO 02-17 04:34:25 [kv_cache_utils.py:1307] GPU KV cache size: 1,577,712 tokens
(EngineCore_DP0 pid=3811746) INFO 02-17 04:34:25 [kv_cache_utils.py:1312] Maximum concurrency for 4,096 tokens per request: 385.18x


**Waiting for server start**


Output:
Chunk ID: cb7649
Wall time: 5.0013 seconds
Process running with session ID 52189
Original token count: 645
Output:
(EngineCore_DP0 pid=3811746)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   0%|          | 0/35 [00:00<?, ?it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  11%|█▏        | 4/35 [00:00<00:00, 32.42it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  23%|██▎       | 8/35 [00:00<00:00, 34.88it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  34%|███▍      | 12/35 [00:00<00:00, 35.88it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  46%|████▌     | 16/35 [00:00<00:00, 35.84it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  57%|█████▋    | 20/35 [00:00<00:00, 36.09it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  69%|██████▊   | 24/35 [00:00<00:00, 35.98it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  80%|████████  | 28/35 [00:00<00:00, 35.66it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  91%|█████████▏| 32/35 [00:00<00:00, 34.20it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 35/35 [00:01<00:00, 34.41it/s]
(EngineCore_DP0 pid=3811746)
Capturing CUDA graphs (decode, FULL):   0%|          | 0/19 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL):  16%|█▌        | 3/19 [00:00<00:00, 27.94it/s]
Capturing CUDA graphs (decode, FULL):  37%|███▋      | 7/19 [00:00<00:00, 31.42it/s]
Capturing CUDA graphs (decode, FULL):  58%|█████▊    | 11/19 [00:00<00:00, 32.90it/s]
Capturing CUDA graphs (decode, FULL):  79%|███████▉  | 15/19 [00:00<00:00, 34.05it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 34.69it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 33.62it/s]
(EngineCore_DP0 pid=3811746) INFO 02-17 04:34:28 [gpu_model_runner.py:5063] Graph capturing finished in 2 secs, took 0.28 GiB
(EngineCore_DP0 pid=3811746) INFO 02-17 04:34:28 [core.py:272] init engine (profile, create kv cache, warmup model) took 11.00 seconds
2026-02-17 04:34:29 | INFO     | veena3modal.services.tts_runtime | Initializing prompt builder...
2026-02-17 04:34:29 | INFO     | veena3modal.services.tts_runtime | Initializing BiCodec decoder...
/home/ubuntu/.local/lib/python3.10/site-packages/torch/nn/utils/weight_norm.py:144: FutureWarning: `torch.nn.utils.weight_norm` is deprecated in favor of `torch.nn.utils.parametrizations.weight_norm`.
  WeightNorm.apply(module, name, dim)


Output:
Chunk ID: e3b8e2
Wall time: 5.0015 seconds
Process running with session ID 52189
Original token count: 701
Output:
2026-02-17 04:34:37 | INFO     | veena3modal.services.tts_runtime | Initializing TTS pipeline...
2026-02-17 04:34:37 | INFO     | veena3modal.services.tts_runtime | Initializing streaming pipeline...
2026-02-17 04:34:37 | INFO     | veena3modal.services.tts_runtime | ✅ TTS runtime initialized in 32894ms
2026-02-17 04:34:37 | INFO     | veena3modal.services.tts_runtime |    Model version: LLM
2026-02-17 04:34:37 | INFO     | veena3modal.services.tts_runtime | Speaker globals cache startup precompute skipped (disabled)
2026-02-17 04:34:37 | INFO     | veena3.local | Runtime initialized in 32.9s
2026-02-17 04:34:37 | INFO     | veena3.local |   Model version: LLM
2026-02-17 04:34:37 | INFO     | veena3.local |   Streaming: available
2026-02-17 04:34:37 | INFO     | veena3.local |   Super-resolution: disabled
2026-02-17 04:34:37 | INFO     | veena3.local |
2026-02-17 04:34:37 | INFO     | veena3.local | ============================================================
2026-02-17 04:34:37 | INFO     | veena3.local | Veena3 TTS Local Server starting
2026-02-17 04:34:37 | INFO     | veena3.local |   URL: http://0.0.0.0:8000
2026-02-17 04:34:37 | INFO     | veena3.local |   Health: http://0.0.0.0:8000/v1/tts/health
2026-02-17 04:34:37 | INFO     | veena3.local |   Generate: POST http://0.0.0.0:8000/v1/tts/generate
2026-02-17 04:34:37 | INFO     | veena3.local |   Metrics: http://0.0.0.0:8000/v1/tts/metrics
2026-02-17 04:34:37 | INFO     | veena3.local |   Docs: http://0.0.0.0:8000/docs
2026-02-17 04:34:37 | INFO     | veena3.local | ============================================================
✅ vLLM engine initialized successfully!

🎉 Spark TTS Model ready for inference!

🎵 Loading BiCodec audio tokenizer from /home/ubuntu/veenaModal/models/spark_tts_4speaker...
   Device: cuda
Missing tensor: mel_transformer.spectrogram.window
Missing tensor: mel_transformer.mel_scale.fb
✅ BiCodec decoder initialized (sample rate: 16000Hz)
🚀 SparkTTSPipeline initialized
🌊 Veena3SlidingWindowPipeline initialized (sliding window: 28 tokens)
{"timestamp": "2026-02-17T04:34:37.796503+00:00", "level": "INFO", "logger": "veena3modal.api.websocket_handler", "message": "WebSocket route /v1/tts/ws registered"}
2026-02-17 04:34:37 | INFO     | veena3modal.api.websocket_handler | WebSocket route /v1/tts/ws registered
2026-02-17 04:34:37 | INFO     | veena3modal.api.fastapi_app | WebSocket support enabled at /v1/tts/ws
{"timestamp": "2026-02-17T04:34:37.796645+00:00", "level": "INFO", "logger": "veena3modal.api.fastapi_app", "message": "WebSocket support enabled at /v1/tts/ws"}
INFO:     Started server process [3811619]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)


**Running concurrent benchmark tests**


python3 scripts/stress_test_optimized_local.py --url http://localhost:8000 --levels 1,4,8,16 --text short --max-tokens 128 --timeout 90 --output /tmp/stress_http_detailed_fixed.json


Output:
Chunk ID: 7e252d
Wall time: 1.0017 seconds
Process running with session ID 11146
Original token count: 0
Output:


**Planning poll benchmark**


The HTTP benchmark is now running against the live local server with explicit `max_tokens=128` and timing headers enabled. I’ll extract concurrency and inflight bottlenecks from this run next.


Output:
Chunk ID: 569a36
Wall time: 5.0016 seconds
Process running with session ID 11146
Original token count: 0
Output:


Output:
Chunk ID: 5091ca
Wall time: 9.3027 seconds
Process exited with code 0
Original token count: 1225
Output:
  Health: healthy
  Version: LLM
  GPU: True

Warming up (1 request, non-streaming)...
  warmup ok: 100%

========================================================================
  1C-SHORT-NON-STREAMING
  10 requests, 1 concurrent, non-streaming, chunking=True
========================================================================
  Success:     10/10 (100%)
  Wall time:   4.32s
  Throughput:  2.31 req/s
  Audio total: 18.78s
  Eff. RTF:    0.230
  Peak inflight: 1

  Latency (ms):
    min=415 avg=427 p50=428 p95=441 p99=441 max=441

  Server TTFB (ms):
    min=411 avg=423 p50=423 p95=436 max=436

  Generation (ms):
    min=411 avg=423 p50=423 p95=436 max=436

  LLM batch wall (ms):
    min=392 avg=408 p50=409 p95=423 max=423

  Parse (ms):
    avg=0 p95=0

  Decode (ms):
    avg=395 p95=409

  BiCodec decode (ms):
    avg=15 p95=34

  BiCodec decode GPU (ms):
    avg=14 p95=34
  Token timing: avg 3303.906 us/token

  GPU:
    Memory: 23601MB avg, 23601MB peak / 81920MB
    Util:   61% avg, 70% peak
    Temp:   32C avg, 33C peak
  cooldown 3s...

========================================================================
  4C-SHORT-NON-STREAMING
  12 requests, 4 concurrent, non-streaming, chunking=True
========================================================================
  Success:     12/12 (100%)
  Wall time:   1.63s
  Throughput:  7.34 req/s
  Audio total: 22.46s
  Eff. RTF:    0.073
  Peak inflight: 4

  Latency (ms):
    min=472 avg=525 p50=537 p95=576 p99=576 max=576

  Server TTFB (ms):
    min=467 avg=501 p50=501 p95=532 max=532

  Generation (ms):
    min=467 avg=501 p50=501 p95=532 max=532

  LLM batch wall (ms):
    min=448 avg=487 p50=490 p95=520 max=520

  Parse (ms):
    avg=0 p95=0

  Decode (ms):
    avg=469 p95=503

  BiCodec decode (ms):
    avg=14 p95=42

  BiCodec decode GPU (ms):
    avg=14 p95=42
  Token timing: avg 3924.935 us/token

  GPU:
    Memory: 23601MB avg, 23601MB peak / 81920MB
    Util:   39% avg, 58% peak
    Temp:   32C avg, 33C peak
  cooldown 3s...

========================================================================
  8C-SHORT-NON-STREAMING
  24 requests, 8 concurrent, non-streaming, chunking=True
========================================================================
  Success:     24/24 (100%)
  Wall time:   1.82s
  Throughput:  13.20 req/s
  Audio total: 45.10s
  Eff. RTF:    0.040
  Peak inflight: 8

  Latency (ms):
    min=501 avg=585 p50=583 p95=606 p99=607 max=607

  Server TTFB (ms):
    min=488 avg=532 p50=535 p95=569 max=579

  Generation (ms):
    min=488 avg=532 p50=535 p95=569 max=579

  LLM batch wall (ms):
    min=474 avg=521 p50=525 p95=558 max=569

  Parse (ms):
    avg=0 p95=0

  Decode (ms):
    avg=503 p95=542

  BiCodec decode (ms):
    avg=11 p95=12

  BiCodec decode GPU (ms):
    avg=11 p95=12
  Token timing: avg 4153.971 us/token

  GPU:
    Memory: 23601MB avg, 23601MB peak / 81920MB
    Util:   57% avg, 59% peak
    Temp:   33C avg, 33C peak
  cooldown 3s...

========================================================================
  16C-SHORT-NON-STREAMING
  48 requests, 16 concurrent, non-streaming, chunking=True
========================================================================
  Success:     48/48 (100%)
  Wall time:   2.28s
  Throughput:  21.07 req/s
  Audio total: 90.24s
  Eff. RTF:    0.025
  Peak inflight: 16

  Latency (ms):
    min=571 avg=726 p50=740 p95=760 p99=889 max=889

  Server TTFB (ms):
    min=529 avg=620 p50=621 p95=699 max=710

  Generation (ms):
    min=529 avg=620 p50=621 p95=699 max=710

  LLM batch wall (ms):
    min=512 avg=608 p50=608 p95=686 max=699

  Parse (ms):
    avg=0 p95=0

  Decode (ms):
    avg=583 p95=669

  BiCodec decode (ms):
    avg=12 p95=15

  BiCodec decode GPU (ms):
    avg=12 p95=15
  Token timing: avg 4841.960 us/token

  GPU:
    Memory: 23601MB avg, 23601MB peak / 81920MB
    Util:   53% avg, 77% peak
    Temp:   33C avg, 34C peak

==============================================================================================================
  STRESS BENCHMARK SUMMARY
==============================================================================================================
Level                        OK%     RPS     p50     p95  GPU%   GPUmem    RTF Inflight
--------------------------------------------------------------------------------------------------------------
1c-short-non-streaming     100%    2.31     428     441   61%   23601MB  0.230        1
4c-short-non-streaming     100%    7.34     537     576   39%   23601MB  0.073        4
8c-short-non-streaming     100%   13.20     583     606   57%   23601MB  0.040        8
16c-short-non-streaming    100%   21.07     740     760   53%   23601MB  0.025       16
==============================================================================================================

Results written to /tmp/stress_http_detailed_fixed.json


**Sending Ctrl-C to stop server**


Output: write_stdin failed: stdin is closed for this session; rerun exec_command with tty=true to keep stdin open


**Terminating unresponsive processes**


kill 3811619 3811746 3810636 || true; sleep 1; ps -eo pid,ppid,cmd | rg "3811619|3811746|veena3modal.local_server|VLLM::EngineCore" -n || true


Output:
Chunk ID: 48dbc5
Wall time: 0.9155 seconds
Process exited with code 0
Original token count: 79
Output:
/bin/bash: line 1: kill: (3810636) - No such process
311:3812172 3675569 /bin/bash -c kill 3811619 3811746 3810636 || true; sleep 1; ps -eo pid,ppid,cmd | rg "3811619|3811746|veena3modal.local_server|VLLM::EngineCore" -n || true
316:3812178 3812172 rg 3811619|3811746|veena3modal.local_server|VLLM::EngineCore -n


**Planning detailed benchmark analysis**


python3 - <<'PY'
import json, statistics, pathlib

runtime_path = pathlib.Path('/tmp/stress_runtime_detailed_fixed.json')
http_path = pathlib.Path('/tmp/stress_http_detailed_fixed.json')

print('runtime exists', runtime_path.exists())
print('http exists', http_path.exists())

rt = json.loads(runtime_path.read_text())
print('\nRUNTIME LEVELS:')
for lvl in rt['levels']:
    c=lvl['concurrency']
    succ=[r for r in lvl['requests'] if r['success']]
    gen=[r['timing'].get('generation_ms') for r in succ if r.get('timing') and r['timing'].get('generation_ms') is not None]
    llm=[r['timing'].get('llm_batch_wall_ms_total') for r in succ if r.get('timing') and r['timing'].get('llm_batch_wall_ms_total') is not None]
    bdc=[r['timing'].get('bicodec_decode_wall_ms') for r in succ if r.get('timing') and r['timing'].get('bicodec_decode_wall_ms') is not None]
    tok=[r['timing'].get('llm_token_total') for r in succ if r.get('timing') and r['timing'].get('llm_token_total') is not None]
    tpt=[r['timing'].get('llm_time_per_token_ms') for r in succ if r.get('timing') and r['timing'].get('llm_time_per_token_ms') is not None]
    if not succ:
        print(c, 'no successes')
        continue
    avg_gen = statistics.mean(gen) if gen else 0
    avg_llm = statistics.mean(llm) if llm else 0
    avg_bdc = statistics.mean(bdc) if bdc else 0
    avg_tok = statistics.mean(tok) if tok else 0
    avg_tpt = statistics.mean(tpt) if tpt else 0
    share_llm = (avg_llm/avg_gen*100) if avg_gen else 0
    share_bdc = (avg_bdc/avg_gen*100) if avg_gen else 0
    print(f"c={c:>2} succ={len(succ):>2}/{lvl['total_requests']} thr={lvl['throughput_rps']:.2f} avg_gen={avg_gen:.1f}ms avg_llm={avg_llm:.1f}ms ({share_llm:.1f}%) avg_bicodec={avg_bdc:.1f}ms ({share_bdc:.1f}%) avg_tok={avg_tok:.1f} avg_tpt={avg_tpt:.3f}ms")

hp = json.loads(http_path.read_text())
print('\nHTTP LEVELS:')
for lvl in hp:
    c=lvl['concurrency']
    succ=[r for r in lvl['requests'] if r['success']]
    gen=[r['timing'].get('generation_ms') for r in succ if r.get('timing') and r['timing'].get('generation_ms') is not None]
    llm=[r['timing'].get('llm_batch_wall_ms_total') for r in succ if r.get('timing') and r['timing'].get('llm_batch_wall_ms_total') is not None]
    bdc=[r['timing'].get('bicodec_decode_wall_ms') for r in succ if r.get('timing') and r['timing'].get('bicodec_decode_wall_ms') is not None]
    infl=[r.get('request_inflight') for r in succ if isinstance(r.get('request_inflight'), int)]
    if not succ:
        print(c, 'no successes')
        continue
    avg_gen = statistics.mean(gen) if gen else 0
    avg_llm = statistics.mean(llm) if llm else 0
    avg_bdc = statistics.mean(bdc) if bdc else 0
    peak_infl=max(infl) if infl else 0
    share_llm = (avg_llm/avg_gen*100) if avg_gen else 0
    share_bdc = (avg_bdc/avg_gen*100) if avg_gen else 0
    print(f"c={c:>2} succ={len(succ):>2}/{lvl['concurrency']*3 if lvl['concurrency']*3>10 else 10} rps={lvl['throughput_rps']:.2f} avg_gen={avg_gen:.1f}ms llm={avg_llm:.1f}ms ({share_llm:.1f}%) bicodec={avg_bdc:.1f}ms ({share_bdc:.1f}%) inflight_peak={peak_infl}")
PY


Output:
Chunk ID: 14ddc7
Wall time: 0.0515 seconds
Process exited with code 0
Original token count: 268
Output:
runtime exists True
http exists True

RUNTIME LEVELS:
c= 1 succ= 6/6 thr=2.27 avg_gen=440.7ms avg_llm=428.3ms (97.2%) avg_bicodec=11.8ms (2.7%) avg_tok=128.0 avg_tpt=3.346ms
c= 2 succ= 6/6 thr=4.14 avg_gen=477.0ms avg_llm=464.8ms (97.5%) avg_bicodec=11.6ms (2.4%) avg_tok=128.0 avg_tpt=3.632ms
c= 4 succ=12/12 thr=7.97 avg_gen=483.4ms avg_llm=472.3ms (97.7%) avg_bicodec=11.0ms (2.3%) avg_tok=128.0 avg_tpt=3.690ms
c= 8 succ=24/24 thr=13.85 avg_gen=533.9ms avg_llm=522.1ms (97.8%) avg_bicodec=11.6ms (2.2%) avg_tok=128.0 avg_tpt=4.079ms
c=16 succ=48/48 thr=22.26 avg_gen=630.1ms avg_llm=619.3ms (98.3%) avg_bicodec=10.7ms (1.7%) avg_tok=128.0 avg_tpt=4.839ms

HTTP LEVELS:
c= 1 succ=10/10 rps=2.31 avg_gen=422.9ms llm=407.6ms (96.4%) bicodec=14.7ms (3.5%) inflight_peak=1
c= 4 succ=12/12 rps=7.34 avg_gen=501.1ms llm=486.9ms (97.2%) bicodec=13.9ms (2.8%) inflight_peak=4
c= 8 succ=24/24 rps=13.20 avg_gen=531.7ms llm=520.8ms (98.0%) bicodec=10.7ms (2.0%) inflight_peak=8
c=16 succ=48/48 rps=21.07 avg_gen=619.8ms llm=607.8ms (98.1%) bicodec=11.9ms (1.9%) inflight_peak=16


**Computing decode and prefill metrics**


python3 - <<'PY'
import json, statistics
rt=json.load(open('/tmp/stress_runtime_detailed_fixed.json'))
print('Prefill/decode split (runtime):')
for lvl in rt['levels']:
    succ=[r for r in lvl['requests'] if r['success']]
    pre=[r['timing'].get('llm_prefill_wall_ms') for r in succ if r['timing'].get('llm_prefill_wall_ms') is not None]
    dec=[r['timing'].get('llm_decode_wall_ms_total') for r in succ if r['timing'].get('llm_decode_wall_ms_total') is not None]
    gen=[r['timing'].get('llm_generation_wall_ms') for r in succ if r['timing'].get('llm_generation_wall_ms') is not None]
    if not succ: continue
    ap=statistics.mean(pre) if pre else 0
    ad=statistics.mean(dec) if dec else 0
    ag=statistics.mean(gen) if gen else 0
    print(f"c={lvl['concurrency']:>2} prefill={ap:.1f}ms decode={ad:.1f}ms gen={ag:.1f}ms prefill_share={(ap/ag*100 if ag else 0):.1f}%")
PY


Output:
Chunk ID: f3dfe7
Wall time: 0.0504 seconds
Process exited with code 0
Original token count: 91
Output:
Prefill/decode split (runtime):
c= 1 prefill=11.6ms decode=416.7ms gen=428.3ms prefill_share=2.7%
c= 2 prefill=15.9ms decode=448.9ms gen=464.9ms prefill_share=3.4%
c= 4 prefill=17.5ms decode=454.8ms gen=472.3ms prefill_share=3.7%
c= 8 prefill=21.2ms decode=500.9ms gen=522.1ms prefill_share=4.1%
c=16 prefill=20.9ms decode=598.5ms gen=619.4ms prefill_share=3.4%


Summary: auto


**Checking queue and scheduler metrics**


I have both benchmark artifacts now and I’m extracting queue/scheduler/forward timing from request-level metrics to identify where concurrency pressure starts accumulating inside vLLM.


python3 - <<'PY'
import json, statistics
rt=json.load(open('/tmp/stress_runtime_detailed_fixed.json'))
keys=['llm_time_in_queue_ms','llm_scheduler_ms','llm_model_forward_ms','llm_model_execute_ms','llm_first_token_ms','llm_request_lifecycle_ms']
for lvl in rt['levels']:
    succ=[r for r in lvl['requests'] if r['success']]
    print(f"\nconcurrency={lvl['concurrency']} count={len(succ)}")
    for k in keys:
        vals=[r['timing'].get(k) for r in succ if isinstance(r.get('timing',{}).get(k),(int,float))]
        if vals:
            print(f"  {k}: avg={statistics.mean(vals):.2f} p95={sorted(vals)[int(0.95*(len(vals)-1))]:.2f} max={max(vals):.2f}")
PY


Output:
Chunk ID: 099ec9
Wall time: 0.0514 seconds
Process exited with code 0
Original token count: 30
Output:

concurrency=1 count=6

concurrency=2 count=6

concurrency=4 count=12

concurrency=8 count=24

concurrency=16 count=48


**Planning high concurrency tests**


python3 scripts/stress_test_runtime_detailed.py --levels 16,32,64 --text short --max-tokens 128 --timeout 120 --requests-multiplier 2 --min-requests 16 --gpu-memory 0.25 --output /tmp/stress_runtime_detailed_high.json


Output:
Chunk ID: 23cc0b
Wall time: 1.0014 seconds
Process running with session ID 99881
Original token count: 0
Output:


**Considering long polling**


I’m extending the runtime sweep to higher concurrency (`16,32,64`) to find the scaling knee and whether failures/timeouts appear at peak load.