Output:
Chunk ID: fa88d2
Wall time: 9.3960 seconds
Process exited with code 0
Original token count: 1391
Output:
✅ vLLM engine initialized successfully!

🎉 Spark TTS Model ready for inference!

🎵 Loading BiCodec audio tokenizer from /home/ubuntu/veenaModal/models/spark_tts_4speaker...
   Device: cuda
Missing tensor: mel_transformer.spectrogram.window
Missing tensor: mel_transformer.mel_scale.fb
✅ BiCodec decoder initialized (sample rate: 16000Hz)
   ℹ  Micro-batching enabled (workers=2, scale_mode=dynamic, scale_pending=56, max_batch=16, timeout_ms=6.0)
🚀 SparkTTSPipeline initialized
🌊 Veena3SlidingWindowPipeline initialized (sliding window: 28 tokens)

=== streaming concurrency=96 requests=576 ===
success=576/576 (100%)
wall=23.83s throughput=24.17 req/s
latency_ms avg=3727 p50=3614 p95=4859 max=5298
ttfb_ms avg=1872 p50=1846 p95=2651 max=3045
chunks_per_req avg=4.5 p50=5.0 p95=5.0 max=6.0
inter_chunk_ms avg=520.9 p50=506.5 p95=1580.6 max=2549.6
tail_emit_speed_x avg=2.47 p50=2.22 p95=5.10 max=8.75
timeline_to_first_batch_ms: avg=69.32 p50=44.47 p95=231.56 max=690.46
timeline_to_first_chunk_emitted_ms: avg=1844.62 p50=1820.51 p95=2638.50 max=3043.65
timeline_total_ms: avg=3647.16 p50=3510.58 p95=4821.73 max=5198.30
llm_time_in_queue_ms: avg=0.04 p50=0.01 p95=0.16 max=0.30
llm_scheduler_ms: avg=27.41 p50=26.48 p95=52.43 max=52.43
llm_first_token_ms: avg=62.02 p50=41.66 p95=176.29 max=356.96
llm_request_lifecycle_ms: avg=2666.87 p50=2661.81 p95=3182.78 max=3734.59
llm_generation_wall_ms: avg=3629.46 p50=3501.01 p95=4821.69 max=5198.26
llm_time_per_token_ms: avg=16.05 p50=15.64 p95=20.61 max=25.17
llm_batch_wall_ms_total: avg=3266.81 p50=3267.45 p95=4140.15 max=5020.35
llm_decode_wall_ms_total: avg=3197.53 p50=3206.10 p95=4092.79 max=4983.70
tokens_per_batch_p50: avg=1.26 p50=1.00 p95=1.00 max=70.00
bicodec_decode_calls: avg=3.46 p50=4.00 p95=4.00 max=5.00
bicodec_decode_wall_ms_total: avg=2478.01 p50=2426.95 p95=3848.04 max=4173.25
bicodec_decode_interval_ms_p50: avg=614.83 p50=635.40 p95=1187.39 max=1889.80
decode_interval_applied_tokens_p50: avg=48.00 p50=48.00 p95=48.00 max=48.00
decode_pending_requests_p50: avg=0.00 p50=0.00 p95=0.00 max=0.00
admission_wait_ms: avg=0.00 p50=0.00 p95=0.00 max=0.01
admission_inflight_on_grant: avg=83.97 p50=91.00 p95=96.00 max=96.00
admission_waiters_on_grant: avg=0.00 p50=0.00 p95=0.00 max=0.00
admission_queue_depth_on_entry: avg=0.00 p50=0.00 p95=0.00 max=0.00
batch_avg_size: avg=7.34 p50=6.95 p95=10.15 max=11.43
batch_max_seen: avg=16.00 p50=16.00 p95=16.00 max=16.00
batch_workers_live: avg=2.00 p50=2.00 p95=2.00 max=2.00
batch_workers_target: avg=1.69 p50=2.00 p95=2.00 max=2.00
batch_queue_wait_ms_avg: avg=632.07 p50=662.81 p95=702.94 max=717.67
batch_compute_ms_avg: avg=137.67 p50=132.44 p95=184.99 max=203.77
batch_queue_depth_avg: avg=51.17 p50=51.98 p95=53.13 max=53.88
latency_over_timeline_ms: avg=79.38 p50=44.41 p95=265.13 max=534.29
timeline_share: llm=99.5% bicodec=67.9% other=0.0%
gpu: util_avg=90% util_max=98% mem_avg=49509MB mem_max=49983MB

=== streaming concurrency=128 requests=768 ===
success=465/768 (61%)
wall=18.03s throughput=42.59 req/s
failures: streaming admission wait timeout:303
latency_ms avg=4191 p50=4272 p95=5549 max=6114
ttfb_ms avg=2058 p50=2197 p95=3076 max=3339
chunks_per_req avg=4.4 p50=4.0 p95=5.0 max=6.0
inter_chunk_ms avg=605.0 p50=546.4 p95=1737.7 max=2532.1
tail_emit_speed_x avg=2.39 p50=1.75 p95=5.83 max=8.69
timeline_to_first_batch_ms: avg=91.41 p50=63.19 p95=292.05 max=427.90
timeline_to_first_chunk_emitted_ms: avg=1851.48 p50=1936.71 p95=2726.84 max=2912.13
timeline_total_ms: avg=3935.73 p50=3869.54 p95=5358.27 max=5665.03
llm_time_in_queue_ms: avg=0.06 p50=0.02 p95=0.33 max=0.44
llm_scheduler_ms: avg=28.30 p50=25.72 p95=61.75 max=61.75
llm_first_token_ms: avg=79.64 p50=58.66 p95=236.80 max=352.72
llm_request_lifecycle_ms: avg=2778.73 p50=2786.99 p95=3225.39 max=3797.19
llm_generation_wall_ms: avg=3920.31 p50=3842.21 p95=5358.23 max=5664.98
llm_time_per_token_ms: avg=17.39 p50=17.52 p95=22.90 max=25.11
llm_batch_wall_ms_total: avg=3403.78 p50=3360.66 p95=4582.40 max=5260.22
llm_decode_wall_ms_total: avg=3312.41 p50=3285.22 p95=4462.09 max=5232.70
tokens_per_batch_p50: avg=1.02 p50=1.00 p95=1.00 max=2.00
bicodec_decode_calls: avg=3.45 p50=3.00 p95=4.00 max=5.00
bicodec_decode_wall_ms_total: avg=2706.01 p50=2731.21 p95=4128.78 max=4657.43
bicodec_decode_interval_ms_p50: avg=745.33 p50=786.92 p95=1390.55 max=1703.99
decode_interval_applied_tokens_p50: avg=48.00 p50=48.00 p95=48.00 max=48.00
decode_pending_requests_p50: avg=0.00 p50=0.00 p95=0.00 max=0.00
admission_wait_ms: avg=181.94 p50=44.50 p95=545.76 max=648.79
admission_inflight_on_grant: avg=96.10 p50=108.00 p95=112.00 max=112.00
admission_waiters_on_grant: avg=7.48 p50=8.00 p95=15.00 max=16.00
admission_queue_depth_on_entry: avg=5.49 p50=2.00 p95=16.00 max=16.00
batch_avg_size: avg=7.15 p50=7.14 p95=7.31 max=7.37
batch_max_seen: avg=16.00 p50=16.00 p95=16.00 max=16.00
batch_workers_live: avg=2.00 p50=2.00 p95=2.00 max=2.00
batch_workers_target: avg=1.75 p50=2.00 p95=2.00 max=2.00
batch_queue_wait_ms_avg: avg=662.25 p50=669.18 p95=701.68 max=704.12
batch_compute_ms_avg: avg=125.99 p50=125.18 p95=129.00 max=129.30
batch_queue_depth_avg: avg=53.22 p50=53.81 p95=55.41 max=55.46
latency_over_timeline_ms: avg=254.89 p50=194.82 p95=611.48 max=813.69
timeline_share: llm=99.6% bicodec=68.8% other=0.0%
gpu: util_avg=90% util_max=100% mem_avg=51003MB mem_max=51109MB

results saved: /tmp/stream_admission_cap112.json
ERROR 02-19 04:20:22 [core_client.py:605] Engine core proc EngineCore_DP0 died unexpectedly, shutting down client.


# Collaboration Mode: Default

You are now in Default mode. Any previous instructions for other modes (e.g. Plan mode) are no longer active.

Your active mode changes only when new developer instructions with a different `<collaboration_mode>...</collaboration_mode>` change it; user requests or tool descriptions do not change mode by themselves. Known mode names are Default and Plan.

## request_user_input availability

The `request_user_input` tool is unavailable in Default mode. If you call it while in Default mode, it will return an error.

If a decision is necessary and cannot be discovered from local context, ask the user directly. However, in Default mode you should strongly prefer executing the user's request rather than stopping to ask questions.

Summary: auto

## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) When `SKILL.md` references relative paths (e.g., `scripts/foo.py`), resolve them relative to the skill directory listed above first, and only consider other paths if needed.
  3) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  4) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  5) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.


**Planning latency and throughput metrics**


python3 - <<'PY'
import json, statistics, math
files = {
    'baseline': '/tmp/stream_admission_off.json',
    'cap96': '/tmp/stream_admission_cap96.json',
    'cap112': '/tmp/stream_admission_cap112.json',
}

def pct(vals, p):
    if not vals:
        return 0.0
    vals = sorted(vals)
    i = int((len(vals)-1)*p/100.0)
    return vals[i]

for name,path in files.items():
    with open(path) as f:
        data=json.load(f)
    print(f"\n== {name} ==")
    for lvl in data['levels']:
        c=lvl['concurrency']
        reqs=lvl['requests']
        succ=[r for r in reqs if r['success']]
        fails=[r for r in reqs if not r['success']]
        lat=[r['latency_ms'] for r in succ]
        ttfb=[r['ttfb_ms'] for r in succ if r['ttfb_ms']>0]
        inter=[]
        for r in succ:
            inter.extend(r.get('inter_chunk_ms') or [])
        wall=float(lvl.get('wall_time_s') or 0.0)
        succ_rps=(len(succ)/wall) if wall>0 else 0
        reasons={}
        for r in fails:
            e=(r.get('error') or 'unknown').lower().strip()
            reasons[e]=reasons.get(e,0)+1
        r_keys=sorted(reasons.items(), key=lambda kv: kv[1], reverse=True)
        reason_summary=', '.join(f"{k}:{v}" for k,v in r_keys[:3])
        print(
            f"c={c} succ={len(succ)}/{len(reqs)} succ_rps={succ_rps:.2f} "
            f"lat_p95={pct(lat,95):.0f} lat_p99={pct(lat,99):.0f} "
            f"ttfb_p95={pct(ttfb,95):.0f} ttfb_p99={pct(ttfb,99):.0f} "
            f"inter_p95={pct(inter,95):.0f} inter_p99={pct(inter,99):.0f} "
            f"fail={reason_summary or '-'}"
        )
PY


Output:
Chunk ID: 1659a8
Wall time: 0.2074 seconds
Process exited with code 0
Original token count: 215
Output:

== baseline ==
c=96 succ=576/576 succ_rps=24.47 lat_p95=4539 lat_p99=4601 ttfb_p95=2490 ttfb_p99=2846 inter_p95=1591 inter_p99=2112 fail=-
c=128 succ=768/768 succ_rps=27.43 lat_p95=5627 lat_p99=5838 ttfb_p95=3012 ttfb_p99=3284 inter_p95=1874 inter_p99=2517 fail=-

== cap96 ==
c=96 succ=576/576 succ_rps=24.82 lat_p95=4635 lat_p99=4828 ttfb_p95=2586 ttfb_p99=3006 inter_p95=1480 inter_p99=2050 fail=-
c=128 succ=212/768 succ_rps=24.34 lat_p95=4722 lat_p99=4754 ttfb_p95=2358 ttfb_p99=2525 inter_p95=1621 inter_p99=1959 fail=streaming admission wait timeout:556

== cap112 ==
c=96 succ=576/576 succ_rps=24.17 lat_p95=4859 lat_p99=5094 ttfb_p95=2651 ttfb_p99=3012 inter_p95=1581 inter_p99=2190 fail=-
c=128 succ=465/768 succ_rps=25.79 lat_p95=5549 lat_p99=5577 ttfb_p95=3076 ttfb_p99=3300 inter_p95=1738 inter_p99=2243 fail=streaming admission wait timeout:303


Summary: auto


**Evaluating admission cap120 configuration**


I’m doing one more intermediate run (`max_inflight=120`) to identify a practical operating point with moderate protection and higher admitted throughput than the strict cap.


python3 scripts/stress_test_streaming_runtime.py --levels 96,128 --text short --max-tokens 1024 --seed-base 2026 --requests-multiplier 6 --min-requests 256 --stream-decode-interval 48 --stream-window-size 128 --stream-min-semantic-first 10 --stream-crossfade-ms 50 --bicodec-batch-max 16 --bicodec-batch-timeout-ms 6 --bicodec-batch-workers 2 --bicodec-batch-scale-pending 56 --bicodec-batch-scale-mode dynamic --stream-admission-max-inflight 120 --stream-admission-max-queue 32 --stream-admission-max-wait-ms 250 --stream-admission-poll-ms 2 --output /tmp/stream_admission_cap120.json


Output:
Chunk ID: 84278c
Wall time: 1.0011 seconds
Process running with session ID 22028
Original token count: 0
Output:


**Considering timing for polling**


Output:
Chunk ID: ffcc81
Wall time: 5.0006 seconds
Process running with session ID 22028
Original token count: 1277
Output:
[2026-02-19 04:20:51] INFO tts_runtime.py:408: Loading SparkTTS model with vLLM...
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
🚀 Initializing Spark TTS Model
📁 Model path: /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
🗣  Speakers: 12 predefined (lipakshi, vardan, reet, Nandini, krishna, anika, adarsh, Nilay, Aarvi, Asha, Bittu, Mira)
🎭 Emotions: 10 tags in [bracket] format

📝 Loading tokenizer from model...
✅ Tokenizer loaded: 165168 tokens

🔍 Validating 12 speakers for Spark TTS...
  lipakshi → <|speaker_0|>: 7 tokens
  vardan → <|speaker_1|>: 7 tokens
  reet → <|speaker_2|>: 7 tokens
  Nandini → <|speaker_3|>: 7 tokens
  krishna → <|speaker_4|>: 7 tokens
  anika → <|speaker_5|>: 7 tokens
  adarsh → <|speaker_6|>: 7 tokens
  Nilay → <|speaker_7|>: 7 tokens
  Aarvi → <|speaker_8|>: 7 tokens
  Asha → <|speaker_9|>: 7 tokens
  Bittu → <|speaker_10|>: 8 tokens
  Mira → <|speaker_11|>: 8 tokens
✅ All 12 speakers validated with mapping

🔍 Validating 10 emotion tags for Spark TTS...
✅ All 10 emotion tags validated

📋 Sample emotion tags:
  [angry]: 1 tokens
  [curious]: 1 tokens
  [excited]: 1 tokens
  [giggle]: 1 tokens
  [laughs harder]: 1 tokens
  ...

🔧 Initializing vLLM engine with optimizations...
  - dtype: bfloat16
  - max_model_len: 4096
  - gpu_memory_utilization: 0.25
  - enable_prefix_caching: True
  - enforce_eager (disable CUDA graphs): False
INFO 02-19 04:20:52 [model.py:541] Resolved architecture: Qwen2ForCausalLM
INFO 02-19 04:20:52 [model.py:1561] Using max model len 4096
INFO 02-19 04:20:52 [scheduler.py:226] Chunked prefill is enabled with max_num_batched_tokens=4096.
INFO 02-19 04:20:52 [vllm.py:624] Asynchronous scheduling is enabled.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(EngineCore_DP0 pid=583227) INFO 02-19 04:20:53 [core.py:96] Initializing a V1 LLM engine (v0.15.1) with config: model='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', speculative_config=None, tokenizer='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [4096], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 256, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': True}, 'local_cache_dir': None, 'static_all_moe_layers': []}


Intermediate cap run is underway; this should give us a clearer “high admission rate + bounded tails” point to use as the default production policy.


Output:
Chunk ID: 161cb8
Wall time: 5.0006 seconds
Process running with session ID 22028
Original token count: 434
Output:
(EngineCore_DP0 pid=583227) INFO 02-19 04:20:55 [parallel_state.py:1212] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://216.81.248.184:51659 backend=nccl
(EngineCore_DP0 pid=583227) INFO 02-19 04:20:55 [parallel_state.py:1423] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
(EngineCore_DP0 pid=583227) INFO 02-19 04:20:56 [gpu_model_runner.py:4033] Starting to load model /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM...
(EngineCore_DP0 pid=583227) INFO 02-19 04:20:56 [cuda.py:364] Using FLASH_ATTN attention backend out of potential backends: ('FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION')
(EngineCore_DP0 pid=583227) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
(EngineCore_DP0 pid=583227) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
(EngineCore_DP0 pid=583227)
Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
(EngineCore_DP0 pid=583227)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:01<00:00,  1.08s/it]
(EngineCore_DP0 pid=583227)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:01<00:00,  1.08s/it]
(EngineCore_DP0 pid=583227)
(EngineCore_DP0 pid=583227) INFO 02-19 04:20:58 [default_loader.py:291] Loading weights took 1.16 seconds
(EngineCore_DP0 pid=583227) INFO 02-19 04:20:58 [gpu_model_runner.py:4130] Model loading took 0.95 GiB memory and 1.701518 seconds


**Extending poll duration**


Output:
Chunk ID: f21be7
Wall time: 5.0016 seconds
Process running with session ID 22028
Original token count: 759
Output:
(EngineCore_DP0 pid=583227) INFO 02-19 04:21:03 [backends.py:812] Using cache directory: /home/ubuntu/.cache/vllm/torch_compile_cache/f3ff13511c/rank_0_0/backbone for vLLM's torch.compile
(EngineCore_DP0 pid=583227) INFO 02-19 04:21:03 [backends.py:872] Dynamo bytecode transform time: 4.21 s
(EngineCore_DP0 pid=583227) INFO 02-19 04:21:06 [backends.py:267] Directly load the compiled graph(s) for compile range (1, 4096) from the cache, took 0.653 s
(EngineCore_DP0 pid=583227) INFO 02-19 04:21:06 [monitor.py:34] torch.compile takes 4.86 s in total
(EngineCore_DP0 pid=583227) INFO 02-19 04:21:07 [gpu_worker.py:356] Available KV cache memory: 18.06 GiB
(EngineCore_DP0 pid=583227) INFO 02-19 04:21:07 [kv_cache_utils.py:1307] GPU KV cache size: 1,577,712 tokens
(EngineCore_DP0 pid=583227) INFO 02-19 04:21:07 [kv_cache_utils.py:1312] Maximum concurrency for 4,096 tokens per request: 385.18x
(EngineCore_DP0 pid=583227)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   0%|          | 0/35 [00:00<?, ?it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  11%|█▏        | 4/35 [00:00<00:00, 32.47it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  23%|██▎       | 8/35 [00:00<00:00, 35.18it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  34%|███▍      | 12/35 [00:00<00:00, 36.27it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  46%|████▌     | 16/35 [00:00<00:00, 37.05it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  57%|█████▋    | 20/35 [00:00<00:00, 36.67it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  69%|██████▊   | 24/35 [00:00<00:00, 36.24it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  80%|████████  | 28/35 [00:00<00:00, 35.83it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  91%|█████████▏| 32/35 [00:00<00:00, 34.28it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 35/35 [00:01<00:00, 34.51it/s]
(EngineCore_DP0 pid=583227)
Capturing CUDA graphs (decode, FULL):   0%|          | 0/19 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL):  16%|█▌        | 3/19 [00:00<00:00, 28.45it/s]
Capturing CUDA graphs (decode, FULL):  37%|███▋      | 7/19 [00:00<00:00, 32.65it/s]
Capturing CUDA graphs (decode, FULL):  58%|█████▊    | 11/19 [00:00<00:00, 34.66it/s]
Capturing CUDA graphs (decode, FULL):  79%|███████▉  | 15/19 [00:00<00:00, 35.29it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 35.58it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 34.69it/s]
(EngineCore_DP0 pid=583227) INFO 02-19 04:21:09 [gpu_model_runner.py:5063] Graph capturing finished in 2 secs, took 0.28 GiB
(EngineCore_DP0 pid=583227) INFO 02-19 04:21:09 [core.py:272] init engine (profile, create kv cache, warmup model) took 10.58 seconds


**Considering repeated polling after initialization**


Output:
Chunk ID: 9a422a
Wall time: 5.0009 seconds
Process running with session ID 22028
Original token count: 256
Output:
[2026-02-19 04:21:10] INFO tts_runtime.py:417: Initializing prompt builder...
[2026-02-19 04:21:10] INFO tts_runtime.py:424: Initializing BiCodec decoder...
/home/ubuntu/.local/lib/python3.10/site-packages/torch/nn/utils/weight_norm.py:144: FutureWarning: `torch.nn.utils.weight_norm` is deprecated in favor of `torch.nn.utils.parametrizations.weight_norm`.
  WeightNorm.apply(module, name, dim)
[2026-02-19 04:21:16] INFO tts_runtime.py:434: Initializing TTS pipeline...
[2026-02-19 04:21:16] INFO tts_runtime.py:445: Initializing streaming pipeline...
[2026-02-19 04:21:16] INFO long_text_processor.py:112: LongTextProcessor initialized
[2026-02-19 04:21:16] INFO tts_runtime.py:504: ✅ TTS runtime initialized in 30842ms
[2026-02-19 04:21:16] INFO tts_runtime.py:505:    Model version: LLM
[2026-02-19 04:21:16] INFO tts_runtime.py:507:    Streaming admission: inflight<=120 queue<=32 wait<=250.0ms poll=2.0ms
[2026-02-19 04:21:16] INFO tts_runtime.py:526: Speaker globals cache startup precompute skipped (disabled)


**Running benchmark tests**


Output:
Chunk ID: 35c323
Wall time: 30.0009 seconds
Process running with session ID 22028
Original token count: 0
Output:


Output:
Chunk ID: 1720b5
Wall time: 10.3437 seconds
Process exited with code 0
Original token count: 1390
Output:
✅ vLLM engine initialized successfully!

🎉 Spark TTS Model ready for inference!

🎵 Loading BiCodec audio tokenizer from /home/ubuntu/veenaModal/models/spark_tts_4speaker...
   Device: cuda
Missing tensor: mel_transformer.spectrogram.window
Missing tensor: mel_transformer.mel_scale.fb
✅ BiCodec decoder initialized (sample rate: 16000Hz)
   ℹ  Micro-batching enabled (workers=2, scale_mode=dynamic, scale_pending=56, max_batch=16, timeout_ms=6.0)
🚀 SparkTTSPipeline initialized
🌊 Veena3SlidingWindowPipeline initialized (sliding window: 28 tokens)

=== streaming concurrency=96 requests=576 ===
success=576/576 (100%)
wall=23.80s throughput=24.20 req/s
latency_ms avg=3758 p50=3675 p95=4942 max=5784
ttfb_ms avg=1747 p50=1764 p95=2791 max=3437
chunks_per_req avg=4.6 p50=5.0 p95=5.0 max=6.0
inter_chunk_ms avg=549.6 p50=511.5 p95=1821.0 max=2301.8
tail_emit_speed_x avg=2.83 p50=1.86 p95=6.08 max=127.46
timeline_to_first_batch_ms: avg=61.01 p50=40.72 p95=128.18 max=869.25
timeline_to_first_chunk_emitted_ms: avg=1718.58 p50=1742.13 p95=2774.32 max=3318.66
timeline_total_ms: avg=3676.61 p50=3609.94 p95=4863.81 max=5776.45
llm_time_in_queue_ms: avg=0.04 p50=0.02 p95=0.17 max=0.30
llm_scheduler_ms: avg=26.33 p50=24.04 p95=49.72 max=49.72
llm_first_token_ms: avg=53.41 p50=38.36 p95=113.59 max=530.26
llm_request_lifecycle_ms: avg=2648.36 p50=2673.69 p95=3119.99 max=3469.79
llm_generation_wall_ms: avg=3658.50 p50=3591.61 p95=4823.37 max=5776.41
llm_time_per_token_ms: avg=16.17 p50=16.04 p95=21.16 max=25.02
llm_batch_wall_ms_total: avg=3288.65 p50=3284.89 p95=4236.51 max=4922.33
llm_decode_wall_ms_total: avg=3227.68 p50=3221.67 p95=4196.27 max=4877.90
tokens_per_batch_p50: avg=1.67 p50=1.00 p95=1.00 max=78.00
bicodec_decode_calls: avg=3.56 p50=4.00 p95=4.00 max=5.00
bicodec_decode_wall_ms_total: avg=2329.83 p50=2332.04 p95=3693.12 max=4236.01
bicodec_decode_interval_ms_p50: avg=669.85 p50=643.74 p95=1305.93 max=1612.36
decode_interval_applied_tokens_p50: avg=48.00 p50=48.00 p95=48.00 max=48.00
decode_pending_requests_p50: avg=0.00 p50=0.00 p95=0.00 max=0.00
admission_wait_ms: avg=0.00 p50=0.00 p95=0.00 max=0.00
admission_inflight_on_grant: avg=83.47 p50=91.00 p95=96.00 max=96.00
admission_waiters_on_grant: avg=0.00 p50=0.00 p95=0.00 max=0.00
admission_queue_depth_on_entry: avg=0.00 p50=0.00 p95=0.00 max=0.00
batch_avg_size: avg=8.47 p50=8.17 p95=12.48 max=13.00
batch_max_seen: avg=16.00 p50=16.00 p95=16.00 max=16.00
batch_workers_live: avg=2.00 p50=2.00 p95=2.00 max=2.00
batch_workers_target: avg=1.59 p50=2.00 p95=2.00 max=2.00
batch_queue_wait_ms_avg: avg=502.64 p50=550.68 p95=592.60 max=598.10
batch_compute_ms_avg: avg=140.75 p50=133.79 p95=201.34 max=218.22
batch_queue_depth_avg: avg=44.09 p50=45.70 p95=48.04 max=48.35
latency_over_timeline_ms: avg=81.56 p50=45.80 p95=265.64 max=532.85
timeline_share: llm=99.5% bicodec=63.4% other=0.0%
gpu: util_avg=90% util_max=100% mem_avg=49224MB mem_max=49941MB

=== streaming concurrency=128 requests=768 ===
success=525/768 (68%)
wall=19.63s throughput=39.12 req/s
failures: streaming admission wait timeout:243
latency_ms avg=4283 p50=4080 p95=5941 max=6468
ttfb_ms avg=2151 p50=2318 p95=3003 max=3375
chunks_per_req avg=4.3 p50=4.0 p95=5.0 max=6.0
inter_chunk_ms avg=635.3 p50=646.1 p95=1895.4 max=2823.9
tail_emit_speed_x avg=2.30 p50=2.05 p95=4.75 max=6.15
timeline_to_first_batch_ms: avg=81.32 p50=52.64 p95=240.68 max=626.07
timeline_to_first_chunk_emitted_ms: avg=2056.32 p50=2195.78 p95=2905.74 max=3279.88
timeline_total_ms: avg=4141.36 p50=3935.34 p95=5772.75 max=6263.89
llm_time_in_queue_ms: avg=0.03 p50=0.02 p95=0.13 max=0.23
llm_scheduler_ms: avg=27.72 p50=25.46 p95=50.67 max=50.67
llm_first_token_ms: avg=73.98 p50=49.87 p95=218.49 max=591.38
llm_request_lifecycle_ms: avg=2804.39 p50=2801.38 p95=3312.33 max=3778.12
llm_generation_wall_ms: avg=4125.05 p50=3908.05 p95=5772.71 max=6263.84
llm_time_per_token_ms: avg=18.34 p50=17.57 p95=24.49 max=27.07
llm_batch_wall_ms_total: avg=3605.22 p50=3533.35 p95=4718.99 max=5576.39
llm_decode_wall_ms_total: avg=3523.95 p50=3474.86 p95=4670.74 max=5516.56
tokens_per_batch_p50: avg=1.11 p50=1.00 p95=1.00 max=10.00
bicodec_decode_calls: avg=3.28 p50=3.00 p95=4.00 max=5.00
bicodec_decode_wall_ms_total: avg=2908.38 p50=2806.77 p95=4424.24 max=5428.82
bicodec_decode_interval_ms_p50: avg=744.96 p50=781.92 p95=1603.87 max=2041.03
decode_interval_applied_tokens_p50: avg=48.00 p50=48.00 p95=48.00 max=48.00
decode_pending_requests_p50: avg=0.00 p50=0.00 p95=0.00 max=0.00
admission_wait_ms: avg=73.41 p50=0.00 p95=347.87 max=464.45
admission_inflight_on_grant: avg=103.74 p50=116.00 p95=120.00 max=120.00
admission_waiters_on_grant: avg=2.54 p50=1.00 p95=7.00 max=8.00
admission_queue_depth_on_entry: avg=2.16 p50=0.00 p95=8.00 max=8.00
batch_avg_size: avg=7.25 p50=7.17 p95=7.98 max=8.02
batch_max_seen: avg=16.00 p50=16.00 p95=16.00 max=16.00
batch_workers_live: avg=2.00 p50=2.00 p95=2.00 max=2.00
batch_workers_target: avg=1.82 p50=2.00 p95=2.00 max=2.00
batch_queue_wait_ms_avg: avg=657.61 p50=664.43 p95=713.49 max=716.03
batch_compute_ms_avg: avg=121.13 p50=121.32 p95=124.33 max=125.88
batch_queue_depth_avg: avg=55.26 p50=56.72 p95=58.14 max=58.20
latency_over_timeline_ms: avg=142.02 p50=100.56 p95=414.01 max=705.75
timeline_share: llm=99.6% bicodec=70.2% other=0.0%
gpu: util_avg=91% util_max=100% mem_avg=49943MB mem_max=49943MB

results saved: /tmp/stream_admission_cap120.json
ERROR 02-19 04:22:04 [core_client.py:605] Engine core proc EngineCore_DP0 died unexpectedly, shutting down client.


**Calculating performance metrics**


python3 - <<'PY'
import json
files=[
 ('baseline','/tmp/stream_admission_off.json'),
 ('cap96','/tmp/stream_admission_cap96.json'),
 ('cap112','/tmp/stream_admission_cap112.json'),
 ('cap120','/tmp/stream_admission_cap120.json'),
]

def pct(vals,p):
    if not vals:return 0.0
    vals=sorted(vals)
    i=int((len(vals)-1)*p/100)
    return vals[i]

print('name,c,success,total,success_rate,wall_s,req_rps,success_rps,lat_p95,lat_p99,ttfb_p95,ttfb_p99,inter_p95,inter_p99')
for name,path in files:
  d=json.load(open(path))
  for lvl in d['levels']:
    reqs=lvl['requests'];succ=[r for r in reqs if r['success']]
    inter=[]
    for r in succ: inter.extend(r.get('inter_chunk_ms') or [])
    lat=[r['latency_ms'] for r in succ]
    ttfb=[r['ttfb_ms'] for r in succ if r['ttfb_ms']>0]
    wall=float(lvl['wall_time_s'])
    print(','.join(map(str,[
      name,lvl['concurrency'],len(succ),len(reqs),
      round(len(succ)/len(reqs),4),round(wall,2),round(len(reqs)/wall,2),round(len(succ)/wall,2),
      round(pct(lat,95),1),round(pct(lat,99),1),round(pct(ttfb,95),1),round(pct(ttfb,99),1),
      round(pct(inter,95),1),round(pct(inter,99),1)
    ])))
PY


Output:
Chunk ID: 21d8d0
Wall time: 0.2969 seconds
Process exited with code 0
Original token count: 195
Output:
name,c,success,total,success_rate,wall_s,req_rps,success_rps,lat_p95,lat_p99,ttfb_p95,ttfb_p99,inter_p95,inter_p99
baseline,96,576,576,1.0,23.54,24.47,24.47,4539.2,4601.3,2490.2,2845.5,1590.7,2111.8
baseline,128,768,768,1.0,28.0,27.43,27.43,5627.3,5837.6,3011.7,3284.2,1874.3,2517.1
cap96,96,576,576,1.0,23.21,24.82,24.82,4635.3,4828.0,2586.0,3006.5,1480.2,2049.8
cap96,128,212,768,0.276,8.71,88.19,24.34,4721.8,4753.8,2357.9,2524.5,1620.8,1958.7
cap112,96,576,576,1.0,23.83,24.17,24.17,4858.6,5094.3,2650.6,3012.0,1580.6,2189.7
cap112,128,465,768,0.6055,18.03,42.59,25.79,5549.0,5576.7,3075.8,3300.1,1737.7,2242.7
cap120,96,576,576,1.0,23.8,24.2,24.2,4942.5,5222.7,2791.4,3346.7,1821.0,2112.9
cap120,128,525,768,0.6836,19.63,39.12,26.74,5940.7,6220.9,3003.0,3273.0,1895.4,2520.5


**Testing intermediate admission cap**


Current data shows strict caps are the only ones that materially improve tail latency; softer caps mostly reintroduce baseline tails. I’m running one bridging profile (`max_inflight=104`) to find whether we can keep most of the p99 gain with less rejection.


python3 scripts/stress_test_streaming_runtime.py --levels 96,128 --text short --max-tokens 1024 --seed-base 2026 --requests-multiplier 6 --min-requests 256 --stream-decode-interval 48 --stream-window-size 128 --stream-min-semantic-first 10 --stream-crossfade-ms 50 --bicodec-batch-max 16 --bicodec-batch-timeout-ms 6 --bicodec-batch-workers 2 --bicodec-batch-scale-pending 56 --bicodec-batch-scale-mode dynamic --stream-admission-max-inflight 104 --stream-admission-max-queue 32 --stream-admission-max-wait-ms 250 --stream-admission-poll-ms 2 --output /tmp/stream_admission_cap104.json


Output:
Chunk ID: caa25e
Wall time: 1.0006 seconds
Process running with session ID 63897
Original token count: 0
Output:


Output:
Chunk ID: 027dba
Wall time: 5.0012 seconds
Process running with session ID 63897
Original token count: 1400
Output:
[2026-02-19 04:22:33] INFO tts_runtime.py:408: Loading SparkTTS model with vLLM...
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
🚀 Initializing Spark TTS Model
📁 Model path: /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
🗣  Speakers: 12 predefined (lipakshi, vardan, reet, Nandini, krishna, anika, adarsh, Nilay, Aarvi, Asha, Bittu, Mira)
🎭 Emotions: 10 tags in [bracket] format

📝 Loading tokenizer from model...
✅ Tokenizer loaded: 165168 tokens

🔍 Validating 12 speakers for Spark TTS...
  lipakshi → <|speaker_0|>: 7 tokens
  vardan → <|speaker_1|>: 7 tokens
  reet → <|speaker_2|>: 7 tokens
  Nandini → <|speaker_3|>: 7 tokens
  krishna → <|speaker_4|>: 7 tokens
  anika → <|speaker_5|>: 7 tokens
  adarsh → <|speaker_6|>: 7 tokens
  Nilay → <|speaker_7|>: 7 tokens
  Aarvi → <|speaker_8|>: 7 tokens
  Asha → <|speaker_9|>: 7 tokens
  Bittu → <|speaker_10|>: 8 tokens
  Mira → <|speaker_11|>: 8 tokens
✅ All 12 speakers validated with mapping

🔍 Validating 10 emotion tags for Spark TTS...
✅ All 10 emotion tags validated

📋 Sample emotion tags:
  [angry]: 1 tokens
  [curious]: 1 tokens
  [excited]: 1 tokens
  [giggle]: 1 tokens
  [laughs harder]: 1 tokens
  ...

🔧 Initializing vLLM engine with optimizations...
  - dtype: bfloat16
  - max_model_len: 4096
  - gpu_memory_utilization: 0.25
  - enable_prefix_caching: True
  - enforce_eager (disable CUDA graphs): False
INFO 02-19 04:22:34 [model.py:541] Resolved architecture: Qwen2ForCausalLM
INFO 02-19 04:22:34 [model.py:1561] Using max model len 4096
INFO 02-19 04:22:34 [scheduler.py:226] Chunked prefill is enabled with max_num_batched_tokens=4096.
INFO 02-19 04:22:34 [vllm.py:624] Asynchronous scheduling is enabled.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(EngineCore_DP0 pid=584888) INFO 02-19 04:22:35 [core.py:96] Initializing a V1 LLM engine (v0.15.1) with config: model='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', speculative_config=None, tokenizer='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [4096], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 256, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': True}, 'local_cache_dir': None, 'static_all_moe_layers': []}
(EngineCore_DP0 pid=584888) INFO 02-19 04:22:37 [parallel_state.py:1212] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://216.81.248.184:32851 backend=nccl
(EngineCore_DP0 pid=584888) INFO 02-19 04:22:37 [parallel_state.py:1423] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
(EngineCore_DP0 pid=584888) INFO 02-19 04:22:37 [gpu_model_runner.py:4033] Starting to load model /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM...


**Continuing poll with update pending**


Output:
Chunk ID: d4986f
Wall time: 5.0016 seconds
Process running with session ID 63897
Original token count: 535
Output:
(EngineCore_DP0 pid=584888) INFO 02-19 04:22:38 [cuda.py:364] Using FLASH_ATTN attention backend out of potential backends: ('FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION')
(EngineCore_DP0 pid=584888) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
(EngineCore_DP0 pid=584888) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
(EngineCore_DP0 pid=584888)
Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
(EngineCore_DP0 pid=584888)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.00it/s]
(EngineCore_DP0 pid=584888)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.00it/s]
(EngineCore_DP0 pid=584888)
(EngineCore_DP0 pid=584888) INFO 02-19 04:22:39 [default_loader.py:291] Loading weights took 1.07 seconds
(EngineCore_DP0 pid=584888) INFO 02-19 04:22:40 [gpu_model_runner.py:4130] Model loading took 0.95 GiB memory and 1.617781 seconds
(EngineCore_DP0 pid=584888) INFO 02-19 04:22:44 [backends.py:812] Using cache directory: /home/ubuntu/.cache/vllm/torch_compile_cache/f3ff13511c/rank_0_0/backbone for vLLM's torch.compile
(EngineCore_DP0 pid=584888) INFO 02-19 04:22:44 [backends.py:872] Dynamo bytecode transform time: 4.15 s
(EngineCore_DP0 pid=584888) INFO 02-19 04:22:47 [backends.py:267] Directly load the compiled graph(s) for compile range (1, 4096) from the cache, took 0.654 s
(EngineCore_DP0 pid=584888) INFO 02-19 04:22:47 [monitor.py:34] torch.compile takes 4.81 s in total
(EngineCore_DP0 pid=584888) INFO 02-19 04:22:48 [gpu_worker.py:356] Available KV cache memory: 18.06 GiB
(EngineCore_DP0 pid=584888) INFO 02-19 04:22:48 [kv_cache_utils.py:1307] GPU KV cache size: 1,577,712 tokens
(EngineCore_DP0 pid=584888) INFO 02-19 04:22:48 [kv_cache_utils.py:1312] Maximum concurrency for 4,096 tokens per request: 385.18x


Bridge-cap run is still executing; once it completes I’ll calculate success-throughput plus p95/p99 for all five profiles and pick the recommended admission defaults.


Output:
Chunk ID: 6e0131
Wall time: 5.0003 seconds
Process running with session ID 63897
Original token count: 634
Output:
(EngineCore_DP0 pid=584888)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   0%|          | 0/35 [00:00<?, ?it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  11%|█▏        | 4/35 [00:00<00:00, 32.73it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  23%|██▎       | 8/35 [00:00<00:00, 35.20it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  34%|███▍      | 12/35 [00:00<00:00, 36.37it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  46%|████▌     | 16/35 [00:00<00:00, 36.77it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  57%|█████▋    | 20/35 [00:00<00:00, 37.05it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  69%|██████▊   | 24/35 [00:00<00:00, 37.04it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  80%|████████  | 28/35 [00:00<00:00, 36.81it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  91%|█████████▏| 32/35 [00:00<00:00, 35.12it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 35/35 [00:00<00:00, 35.07it/s]
(EngineCore_DP0 pid=584888)
Capturing CUDA graphs (decode, FULL):   0%|          | 0/19 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL):  16%|█▌        | 3/19 [00:00<00:00, 28.75it/s]
Capturing CUDA graphs (decode, FULL):  37%|███▋      | 7/19 [00:00<00:00, 33.38it/s]
Capturing CUDA graphs (decode, FULL):  58%|█████▊    | 11/19 [00:00<00:00, 34.81it/s]
Capturing CUDA graphs (decode, FULL):  79%|███████▉  | 15/19 [00:00<00:00, 35.35it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 36.06it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 35.07it/s]
(EngineCore_DP0 pid=584888) INFO 02-19 04:22:50 [gpu_model_runner.py:5063] Graph capturing finished in 2 secs, took 0.28 GiB
(EngineCore_DP0 pid=584888) INFO 02-19 04:22:50 [core.py:272] init engine (profile, create kv cache, warmup model) took 10.56 seconds
[2026-02-19 04:22:52] INFO tts_runtime.py:417: Initializing prompt builder...
[2026-02-19 04:22:52] INFO tts_runtime.py:424: Initializing BiCodec decoder...
/home/ubuntu/.local/lib/python3.10/site-packages/torch/nn/utils/weight_norm.py:144: FutureWarning: `torch.nn.utils.weight_norm` is deprecated in favor of `torch.nn.utils.parametrizations.weight_norm`.
  WeightNorm.apply(module, name, dim)


Output:
Chunk ID: c88243
Wall time: 5.0011 seconds
Process running with session ID 63897
Original token count: 157
Output:
[2026-02-19 04:23:02] INFO tts_runtime.py:434: Initializing TTS pipeline...
[2026-02-19 04:23:02] INFO tts_runtime.py:445: Initializing streaming pipeline...
[2026-02-19 04:23:02] INFO long_text_processor.py:112: LongTextProcessor initialized
[2026-02-19 04:23:02] INFO tts_runtime.py:504: ✅ TTS runtime initialized in 34109ms
[2026-02-19 04:23:02] INFO tts_runtime.py:505:    Model version: LLM
[2026-02-19 04:23:02] INFO tts_runtime.py:507:    Streaming admission: inflight<=104 queue<=32 wait<=250.0ms poll=2.0ms
[2026-02-19 04:23:02] INFO tts_runtime.py:526: Speaker globals cache startup precompute skipped (disabled)


**Benchmarking with extended polling**


Output:
Chunk ID: a2c989
Wall time: 30.0001 seconds
Process running with session ID 63897
Original token count: 0
Output:


Summary: auto


Output:
Chunk ID: 18e3e5
Wall time: 0.0509 seconds
Process exited with code 0
Original token count: 1390
Output:
✅ vLLM engine initialized successfully!

🎉 Spark TTS Model ready for inference!

🎵 Loading BiCodec audio tokenizer from /home/ubuntu/veenaModal/models/spark_tts_4speaker...
   Device: cuda
Missing tensor: mel_transformer.spectrogram.window
Missing tensor: mel_transformer.mel_scale.fb
✅ BiCodec decoder initialized (sample rate: 16000Hz)
   ℹ  Micro-batching enabled (workers=2, scale_mode=dynamic, scale_pending=56, max_batch=16, timeout_ms=6.0)
🚀 SparkTTSPipeline initialized
🌊 Veena3SlidingWindowPipeline initialized (sliding window: 28 tokens)

=== streaming concurrency=96 requests=576 ===
success=576/576 (100%)
wall=23.75s throughput=24.26 req/s
latency_ms avg=3755 p50=3624 p95=4860 max=6010
ttfb_ms avg=1816 p50=1819 p95=2551 max=3703
chunks_per_req avg=4.4 p50=5.0 p95=5.0 max=6.0
inter_chunk_ms avg=548.3 p50=520.8 p95=1719.8 max=2247.1
tail_emit_speed_x avg=2.45 p50=2.08 p95=5.60 max=7.98
timeline_to_first_batch_ms: avg=72.13 p50=41.65 p95=190.21 max=811.08
timeline_to_first_chunk_emitted_ms: avg=1791.72 p50=1796.48 p95=2511.86 max=3701.50
timeline_total_ms: avg=3678.55 p50=3566.09 p95=4822.04 max=6001.06
llm_time_in_queue_ms: avg=0.03 p50=0.02 p95=0.15 max=0.25
llm_scheduler_ms: avg=26.20 p50=24.33 p95=50.44 max=50.44
llm_first_token_ms: avg=65.05 p50=38.91 p95=171.29 max=660.42
llm_request_lifecycle_ms: avg=2625.69 p50=2641.67 p95=3063.93 max=3427.40
llm_generation_wall_ms: avg=3661.24 p50=3552.45 p95=4807.20 max=6001.01
llm_time_per_token_ms: avg=16.22 p50=16.03 p95=20.83 max=25.04
llm_batch_wall_ms_total: avg=3269.11 p50=3300.09 p95=4236.16 max=4896.53
llm_decode_wall_ms_total: avg=3197.03 p50=3232.35 p95=4203.75 max=4854.84
tokens_per_batch_p50: avg=1.94 p50=1.00 p95=1.00 max=102.00
bicodec_decode_calls: avg=3.44 p50=4.00 p95=4.00 max=5.00
bicodec_decode_wall_ms_total: avg=2419.57 p50=2343.01 p95=3665.35 max=4387.51
bicodec_decode_interval_ms_p50: avg=674.73 p50=667.91 p95=1380.07 max=1746.26
decode_interval_applied_tokens_p50: avg=48.00 p50=48.00 p95=48.00 max=48.00
decode_pending_requests_p50: avg=0.00 p50=0.00 p95=0.00 max=0.00
admission_wait_ms: avg=0.00 p50=0.00 p95=0.00 max=0.01
admission_inflight_on_grant: avg=83.84 p50=91.00 p95=96.00 max=96.00
admission_waiters_on_grant: avg=0.00 p50=0.00 p95=0.00 max=0.00
admission_queue_depth_on_entry: avg=0.00 p50=0.00 p95=0.00 max=0.00
batch_avg_size: avg=8.12 p50=7.38 p95=12.43 max=12.95
batch_max_seen: avg=16.00 p50=16.00 p95=16.00 max=16.00
batch_workers_live: avg=2.00 p50=2.00 p95=2.00 max=2.00
batch_workers_target: avg=1.63 p50=2.00 p95=2.00 max=2.00
batch_queue_wait_ms_avg: avg=554.14 p50=620.70 p95=653.55 max=659.02
batch_compute_ms_avg: avg=144.31 p50=133.79 p95=202.96 max=226.22
batch_queue_depth_avg: avg=47.05 p50=49.16 p95=50.67 max=50.89
latency_over_timeline_ms: avg=76.54 p50=42.07 p95=224.72 max=609.33
timeline_share: llm=99.5% bicodec=65.8% other=0.0%
gpu: util_avg=89% util_max=100% mem_avg=48885MB mem_max=49403MB

=== streaming concurrency=128 requests=768 ===
success=263/768 (34%)
wall=10.67s throughput=71.95 req/s
failures: streaming admission wait timeout:505
latency_ms avg=3938 p50=3849 p95=5440 max=5779
ttfb_ms avg=1804 p50=1896 p95=2673 max=3124
chunks_per_req avg=4.6 p50=5.0 p95=5.0 max=6.0
inter_chunk_ms avg=585.2 p50=606.5 p95=1871.1 max=2523.6
tail_emit_speed_x avg=2.39 p50=1.93 p95=4.91 max=7.66
timeline_to_first_batch_ms: avg=88.22 p50=57.32 p95=340.44 max=549.63
timeline_to_first_chunk_emitted_ms: avg=1701.28 p50=1852.80 p95=2482.49 max=2844.62
timeline_total_ms: avg=3790.71 p50=3691.37 p95=5410.72 max=5764.76
llm_time_in_queue_ms: avg=0.06 p50=0.02 p95=0.24 max=0.32
llm_scheduler_ms: avg=29.01 p50=27.84 p95=51.05 max=51.05
llm_first_token_ms: avg=77.52 p50=55.10 p95=196.89 max=374.18
llm_request_lifecycle_ms: avg=2688.26 p50=2662.18 p95=3173.77 max=3898.03
llm_generation_wall_ms: avg=3774.37 p50=3672.19 p95=5410.66 max=5764.71
llm_time_per_token_ms: avg=16.68 p50=16.35 p95=22.35 max=23.79
llm_batch_wall_ms_total: avg=3388.18 p50=3286.15 p95=4426.20 max=5058.69
llm_decode_wall_ms_total: avg=3300.01 p50=3190.12 p95=4383.90 max=4993.79
tokens_per_batch_p50: avg=1.03 p50=1.00 p95=1.00 max=8.00
bicodec_decode_calls: avg=3.57 p50=4.00 p95=4.00 max=5.00
bicodec_decode_wall_ms_total: avg=2601.69 p50=2567.35 p95=4348.32 max=4867.16
bicodec_decode_interval_ms_p50: avg=713.43 p50=715.44 p95=1615.87 max=1772.24
decode_interval_applied_tokens_p50: avg=48.00 p50=48.00 p95=48.00 max=48.00
decode_pending_requests_p50: avg=0.00 p50=0.00 p95=0.00 max=0.00
admission_wait_ms: avg=81.50 p50=0.00 p95=342.41 max=472.09
admission_inflight_on_grant: avg=81.77 p50=98.00 p95=104.00 max=104.00
admission_waiters_on_grant: avg=8.83 p50=4.00 p95=23.00 max=24.00
admission_queue_depth_on_entry: avg=6.07 p50=0.00 p95=24.00 max=24.00
batch_avg_size: avg=7.31 p50=7.27 p95=7.57 max=7.57
batch_max_seen: avg=16.00 p50=16.00 p95=16.00 max=16.00
batch_workers_live: avg=2.00 p50=2.00 p95=2.00 max=2.00
batch_workers_target: avg=1.70 p50=2.00 p95=2.00 max=2.00
batch_queue_wait_ms_avg: avg=629.84 p50=640.93 p95=647.20 max=650.97
batch_compute_ms_avg: avg=129.53 p50=129.50 p95=131.46 max=132.04
batch_queue_depth_avg: avg=50.04 p50=50.19 p95=51.03 max=51.22
latency_over_timeline_ms: avg=146.99 p50=98.07 p95=384.83 max=557.76
timeline_share: llm=99.6% bicodec=68.6% other=0.0%
gpu: util_avg=92% util_max=100% mem_avg=50406MB mem_max=50517MB

results saved: /tmp/stream_admission_cap104.json
ERROR 02-19 04:23:40 [core_client.py:605] Engine core proc EngineCore_DP0 died unexpectedly, shutting down client.


**Planning compute comparison including cap104**


python3 - <<'PY'
import json
files=[
 ('baseline','/tmp/stream_admission_off.json'),
 ('cap96','/tmp/stream_admission_cap96.json'),
 ('cap104','/tmp/stream_admission_cap104.json'),
 ('cap112','/tmp/stream_admission_cap112.json'),
 ('cap120','/tmp/stream_admission_cap120.json'),
]

def pct(vals,p):
    if not vals:return 0.0
    vals=sorted(vals)
    i=int((len(vals)-1)*p/100)
    return vals[i]
print('name,c,succ,total,succ_rate,success_rps,lat_p95,lat_p99,ttfb_p95,ttfb_p99,inter_p95,inter_p99')
for name,path in files:
    d=json.load(open(path))
    for lvl in d['levels']:
      reqs=lvl['requests'];succ=[r for r in reqs if r['success']]
      wall=float(lvl['wall_time_s'])
      lat=[r['latency_ms'] for r in succ]
      ttfb=[r['ttfb_ms'] for r in succ if r['ttfb_ms']>0]
      inter=[]
      for r in succ: inter.extend(r.get('inter_chunk_ms') or [])
      print(','.join(map(str,[name,lvl['concurrency'],len(succ),len(reqs),round(len(succ)/len(reqs),4),round(len(succ)/wall,2),round(pct(lat,95),1),round(pct(lat,99),1),round(pct(ttfb,95),1),round(pct(ttfb,99),1),round(pct(inter,95),1),round(pct(inter,99),1)])))
PY


Output:
Chunk ID: f7192d
Wall time: 0.3650 seconds
Process exited with code 0
Original token count: 203
Output:
name,c,succ,total,succ_rate,success_rps,lat_p95,lat_p99,ttfb_p95,ttfb_p99,inter_p95,inter_p99
baseline,96,576,576,1.0,24.47,4539.2,4601.3,2490.2,2845.5,1590.7,2111.8
baseline,128,768,768,1.0,27.43,5627.3,5837.6,3011.7,3284.2,1874.3,2517.1
cap96,96,576,576,1.0,24.82,4635.3,4828.0,2586.0,3006.5,1480.2,2049.8
cap96,128,212,768,0.276,24.34,4721.8,4753.8,2357.9,2524.5,1620.8,1958.7
cap104,96,576,576,1.0,24.26,4860.2,5390.3,2550.6,3555.1,1719.8,2108.5
cap104,128,263,768,0.3424,24.64,5440.4,5634.7,2672.9,3124.1,1871.1,2427.2
cap112,96,576,576,1.0,24.17,4858.6,5094.3,2650.6,3012.0,1580.6,2189.7
cap112,128,465,768,0.6055,25.79,5549.0,5576.7,3075.8,3300.1,1737.7,2242.7
cap120,96,576,576,1.0,24.2,4942.5,5222.7,2791.4,3346.7,1821.0,2112.9
cap120,128,525,768,0.6836,26.74,5940.7,6220.9,3003.0,3273.0,1895.4,2520.5


**Evaluating capacity and queue strategies**


Output:
Chunk ID: df99a3
Wall time: 0.0512 seconds
Process exited with code 0
Original token count: 193
Output:
 D CLEANUP_PLAN.md
 D PROGRESS.md
 D migration.md
 M veena3modal/api/fastapi_app.py
 M veena3modal/api/websocket_handler.py
 M veena3modal/core/bicodec_decoder.py
 M veena3modal/core/constants.py
 M veena3modal/core/pipeline.py
 M veena3modal/core/streaming_pipeline.py
 M veena3modal/services/tts_runtime.py
?? scripts/profile_pipeline.py
?? scripts/setup_local.sh
?? scripts/stress_test_local.py
?? scripts/stress_test_optimized_local.py
?? scripts/stress_test_production.py
?? scripts/stress_test_runtime_detailed.py
?? scripts/stress_test_streaming_runtime.py
?? stress_results_opt_local.json
?? veena3modal/__main__.py
?? veena3modal/core/bicodec_batch_decoder.py
?? veena3modal/core/dual_engine.py
?? veena3modal/core/multi_engine.py
?? veena3modal/local_server.py


**Gathering changed line numbers**


rg -n "STREAMING_OVERLOADED|acquire_streaming_slot|X-Admission|BackgroundTask|generate_speech_streaming\(" veena3modal/api/fastapi_app.py -n


rg -n "STREAMING_OVERLOADED|acquire_streaming_slot|release_streaming_slot|generate_speech_streaming\(" veena3modal/api/websocket_handler.py -n


rg -n "stream-admission|max-inflight|admission_wait_ms|failure_reason_counts|VEENA3_STREAM_ADMISSION" scripts/stress_test_streaming_runtime.py -n


Output:
Chunk ID: 025c69
Wall time: 0.0515 seconds
Process exited with code 0
Original token count: 236
Output:
64:class StreamingAdmissionError(RuntimeError):
81:class StreamingAdmissionController:
112:    def from_env(cls) -> "StreamingAdmissionController":
177:                        raise StreamingAdmissionError(
194:                raise StreamingAdmissionError(
242:    stream_admission: Optional[StreamingAdmissionController] = None
259:def _get_stream_admission_controller(runtime: Optional[TTSRuntime]) -> StreamingAdmissionController:
265:        controller = StreamingAdmissionController.from_env()
282:async def acquire_streaming_slot() -> StreamingAdmissionLease:
283:    """Acquire one streaming admission slot or raise StreamingAdmissionError."""
291:def release_streaming_slot(lease: Optional[StreamingAdmissionLease]) -> None:
483:        stream_admission = StreamingAdmissionController.from_env()
926:async def generate_speech_streaming(
978:        lease = await acquire_streaming_slot()
1106:            release_streaming_slot(lease)


Output:
Chunk ID: ca6072
Wall time: 0.0509 seconds
Process exited with code 0
Original token count: 167
Output:
748:        from starlette.background import BackgroundTask
785:            admission_lease = await tts_runtime.acquire_streaming_slot()
793:                error_code="STREAMING_OVERLOADED",
798:                error_code="STREAMING_OVERLOADED",
806:                        "code": "STREAMING_OVERLOADED",
826:                async for audio_chunk, metrics in tts_runtime.generate_speech_streaming(
931:            "X-Admission-Wait-ms": str(int(round(float(admission_lease.wait_ms)))),
932:            "X-Admission-Queued": str(bool(admission_lease.queued)).lower(),
943:            background=BackgroundTask(tts_runtime.release_streaming_slot, admission_lease),


Output:
Chunk ID: ef1b15
Wall time: 0.0509 seconds
Process exited with code 0
Original token count: 115
Output:
293:            admission_lease = await tts_runtime.acquire_streaming_slot()
298:                error_code="STREAMING_OVERLOADED",
303:                error_code="STREAMING_OVERLOADED",
307:                code="STREAMING_OVERLOADED",
365:            async for audio_chunk, metrics in tts_runtime.generate_speech_streaming(
414:            tts_runtime.release_streaming_slot(admission_lease)
484:        tts_runtime.release_streaming_slot(admission_lease)


Output:
Chunk ID: 047aae
Wall time: 0.0510 seconds
Process exited with code 0
Original token count: 255
Output:
217:    def failure_reason_counts(self) -> Dict[str, int]:
390:        reasons = level.failure_reason_counts()
435:        "admission_wait_ms",
568:                    "admission_wait_ms": lvl.timing_stats("admission_wait_ms"),
581:                "failure_reasons": lvl.failure_reason_counts(),
678:    parser.add_argument("--stream-admission-max-inflight", type=int, default=0)
679:    parser.add_argument("--stream-admission-max-queue", type=int, default=0)
680:    parser.add_argument("--stream-admission-max-wait-ms", type=float, default=0.0)
681:    parser.add_argument("--stream-admission-poll-ms", type=float, default=2.0)
706:    os.environ["VEENA3_STREAM_ADMISSION_MAX_INFLIGHT"] = str(args.stream_admission_max_inflight)
707:    os.environ["VEENA3_STREAM_ADMISSION_MAX_QUEUE"] = str(args.stream_admission_max_queue)
708:    os.environ["VEENA3_STREAM_ADMISSION_MAX_WAIT_MS"] = str(args.stream_admission_max_wait_ms)
709:    os.environ["VEENA3_STREAM_ADMISSION_POLL_MS"] = str(args.stream_admission_poll_ms)


**Calculating performance improvements**


python3 - <<'PY'
import json
base=json.load(open('/tmp/stream_admission_off.json'))
cap96=json.load(open('/tmp/stream_admission_cap96.json'))
cap104=json.load(open('/tmp/stream_admission_cap104.json'))
cap112=json.load(open('/tmp/stream_admission_cap112.json'))
cap120=json.load(open('/tmp/stream_admission_cap120.json'))

runs={'cap96':cap96,'cap104':cap104,'cap112':cap112,'cap120':cap120}

def pct(vals,p):
    if not vals:return 0
    vals=sorted(vals)
    return vals[int((len(vals)-1)*p/100)]

def stats(d,c):
    lvl=next(l for l in d['levels'] if l['concurrency']==c)
    reqs=lvl['requests'];succ=[r for r in reqs if r['success']]
    wall=lvl['wall_time_s']
    lat=[r['latency_ms'] for r in succ]
    ttfb=[r['ttfb_ms'] for r in succ if r['ttfb_ms']>0]
    inter=[]
    for r in succ: inter.extend(r.get('inter_chunk_ms') or [])
    return {
      'succ':len(succ),'total':len(reqs),'succ_rate':len(succ)/len(reqs),
      'succ_rps':len(succ)/wall,
      'lat_p95':pct(lat,95),'lat_p99':pct(lat,99),
      'ttfb_p95':pct(ttfb,95),'ttfb_p99':pct(ttfb,99),
      'inter_p95':pct(inter,95),'inter_p99':pct(inter,99),
    }

b=stats(base,128)
print('baseline',b)
for name,d in runs.items():
    s=stats(d,128)
    print(name,s,
      'd_lat_p99%',round((s['lat_p99']-b['lat_p99'])*100/b['lat_p99'],1),
      'd_ttfb_p99%',round((s['ttfb_p99']-b['ttfb_p99'])*100/b['ttfb_p99'],1),
      'd_succ_rps%',round((s['succ_rps']-b['succ_rps'])*100/b['succ_rps'],1)
    )
PY


Output:
Chunk ID: 1d8ad9
Wall time: 0.3960 seconds
Process exited with code 0
Original token count: 402
Output:
baseline {'succ': 768, 'total': 768, 'succ_rate': 1.0, 'succ_rps': 27.42932934325126, 'lat_p95': 5627.344579435885, 'lat_p99': 5837.604188360274, 'ttfb_p95': 3011.747382581234, 'ttfb_p99': 3284.20312423259, 'inter_p95': 1874.252705834806, 'inter_p99': 2517.0574774965644}
cap96 {'succ': 212, 'total': 768, 'succ_rate': 0.2760416666666667, 'succ_rps': 24.344446206746635, 'lat_p95': 4721.754211001098, 'lat_p99': 4753.828031942248, 'ttfb_p95': 2357.9447688534856, 'ttfb_p99': 2524.54995829612, 'inter_p95': 1620.7841681316495, 'inter_p99': 1958.7297635152936} d_lat_p99% -18.6 d_ttfb_p99% -23.1 d_succ_rps% -11.2
cap104 {'succ': 263, 'total': 768, 'succ_rate': 0.3424479166666667, 'succ_rps': 24.64016958671135, 'lat_p95': 5440.389947965741, 'lat_p99': 5634.655867703259, 'ttfb_p95': 2672.9462686926126, 'ttfb_p99': 3124.0521762520075, 'inter_p95': 1871.0687831044197, 'inter_p99': 2427.1683460101485} d_lat_p99% -3.5 d_ttfb_p99% -4.9 d_succ_rps% -10.2
cap112 {'succ': 465, 'total': 768, 'succ_rate': 0.60546875, 'succ_rps': 25.785884369876186, 'lat_p95': 5548.950818367302, 'lat_p99': 5576.654167845845, 'ttfb_p95': 3075.8378505706787, 'ttfb_p99': 3300.066319294274, 'inter_p95': 1737.7153234556317, 'inter_p99': 2242.732218466699} d_lat_p99% -4.5 d_ttfb_p99% 0.5 d_succ_rps% -6.0
cap120 {'succ': 525, 'total': 768, 'succ_rate': 0.68359375, 'succ_rps': 26.742899014107838, 'lat_p95': 5940.6818440184, 'lat_p99': 6220.911921933293, 'ttfb_p95': 3002.957433462143, 'ttfb_p99': 3273.003085516393, 'inter_p95': 1895.3794268891215, 'inter_p99': 2520.515133626759} d_lat_p99% 6.6 d_ttfb_p99% -0.3 d_succ_rps% -2.5


**Summarizing implementation and benchmark results**


Implemented next step: strict streaming admission/backpressure with overload signaling, then benchmarked it on the optimized decode path.

**What I changed**
- Added shared streaming admission controller (inflight cap, bounded wait queue, wait timeout, counters, snapshots) in `veena3modal/services/tts_runtime.py:64`, `veena3modal/services/tts_runtime.py:282`, `veena3modal/services/tts_runtime.py:926`.
- Streaming generation now records admission timing/queue metrics per request (`admission_wait_ms`, `admission_inflight_on_grant`, etc.) in `veena3modal/services/tts_runtime.py:926`.
- HTTP streaming route now pre-acquires admission and returns clean `429` with `Retry-After` on overload in `veena3modal/api/fastapi_app.py:785`; also exposes admission headers in `veena3modal/api/fastapi_app.py:931`.
- WebSocket path now applies same gate and returns overload error + close `1013` in `veena3modal/api/websocket_handler.py:293`.
- Stress harness now supports admission knobs and reports failure reasons + admission timing in `scripts/stress_test_streaming_runtime.py:678`, `scripts/stress_test_streaming_runtime.py:435`, `scripts/stress_test_streaming_runtime.py:217`.

**Benchmark artifacts**
- Baseline (no admission): `/tmp/stream_admission_off.json`
- Cap 96: `/tmp/stream_admission_cap96.json`
- Cap 104: `/tmp/stream_admission_cap104.json`
- Cap 112: `/tmp/stream_admission_cap112.json`
- Cap 120: `/tmp/stream_admission_cap120.json`

**Key result (c=128, optimized decode config held constant)**
- `baseline`: success `768/768`, success throughput `27.43 req/s`, latency p99 `5838 ms`, TTFB p99 `3284 ms`.
- `cap96` (`inflight=96 queue=32 wait=250ms`): success `212/768`, success throughput `24.34 req/s`, latency p99 `4754 ms` (`-18.6%`), TTFB p99 `2525 ms` (`-23.1%`).
- `cap104`: success `263/768`, success throughput `24.64 req/s`, latency p99 `5635 ms` (`-3.5%`).
- `cap112`: success `465/768`, success throughput `25.79 req/s`, latency p99 `5577 ms` (`-4.5%`).
- `cap120`: success `525/768`, success throughput `26.74 req/s`, latency p99 `6221 ms` (`+6.6%`, worse than baseline).

**Interpretation**
- Yes, this is real progress: we now have production-style overload control with measurable SLO tradeoffs.
- On a single A100, you can’t simultaneously keep baseline admitted throughput at `c=128` and strongly reduce tails.
- Strict caps improve tail latency for admitted streams; softer caps recover acceptance but lose most tail gains.

**Recommended direction**
1. Use tiered admission policy:
1. `realtime` tier: strict cap (close to `cap96`) + immediate reroute to another replica/GPU.
2. `best_effort` tier: softer cap or no cap.
2. Add external load-aware routing across replicas; admission then protects per-GPU latency while cluster keeps total success high.
3. Next code step: implement SLO tiers in request schema + policy map, then rerun the same sweep.