Output:
Chunk ID: 322d76
Wall time: 0.0510 seconds
Process exited with code 0
Original token count: 1325
Output:
✅ vLLM engine initialized successfully!

🎉 Spark TTS Model ready for inference!

🎵 Loading BiCodec audio tokenizer from /home/ubuntu/veenaModal/models/spark_tts_4speaker...
   Device: cuda
Missing tensor: mel_transformer.spectrogram.window
Missing tensor: mel_transformer.mel_scale.fb
✅ BiCodec decoder initialized (sample rate: 16000Hz)
🚀 SparkTTSPipeline initialized
🌊 Veena3SlidingWindowPipeline initialized (sliding window: 28 tokens)

=== streaming concurrency=32 requests=32 ===
success=32/32 (100%)
wall=2.55s throughput=12.56 req/s
latency_ms avg=2546 p50=2548 p95=2548 max=2548
ttfb_ms avg=1047 p50=927 p95=1319 max=1346
chunks_per_req avg=4.1 p50=4.0 p95=5.0 max=5.0
inter_chunk_ms avg=428.9 p50=498.5 p95=1069.7 max=1100.7
tail_emit_speed_x avg=2.45 p50=2.12 p95=3.20 max=3.36
timeline_to_first_batch_ms: avg=38.79 p50=38.55 p95=43.35 max=44.13
timeline_to_first_chunk_emitted_ms: avg=1039.76 p50=919.49 p95=1307.52 max=1335.87
timeline_total_ms: avg=2379.96 p50=2425.08 p95=2540.20 max=2542.00
llm_time_in_queue_ms: avg=0.08 p50=0.07 p95=0.15 max=0.18
llm_scheduler_ms: avg=28.12 p50=27.58 p95=27.58 max=36.13
llm_first_token_ms: avg=37.67 p50=37.75 p95=42.45 max=43.29
llm_request_lifecycle_ms: avg=1398.59 p50=1397.40 p95=1515.59 max=1698.71
llm_generation_wall_ms: avg=2354.02 p50=2425.04 p95=2538.83 max=2541.96
llm_time_per_token_ms: avg=10.62 p50=10.76 p95=12.09 max=12.86
llm_batch_wall_ms_total: avg=1979.60 p50=2010.67 p95=2462.64 max=2508.70
llm_decode_wall_ms_total: avg=1940.84 p50=1971.87 p95=2421.19 max=2468.04
tokens_per_batch_p50: avg=1.00 p50=1.00 p95=1.00 max=1.00
bicodec_decode_calls: avg=3.12 p50=3.00 p95=4.00 max=4.00
bicodec_decode_wall_ms_total: avg=1943.45 p50=1985.32 p95=2099.40 max=2109.01
bicodec_decode_interval_ms_p50: avg=383.45 p50=519.68 p95=747.60 max=777.98
latency_over_timeline_ms: avg=166.50 p50=121.86 p95=415.73 max=504.42
timeline_share: llm=98.9% bicodec=81.7% other=0.0%
gpu: util_avg=67% util_max=99% mem_avg=24374MB mem_max=25007MB

=== streaming concurrency=64 requests=64 ===
success=64/64 (100%)
wall=3.82s throughput=16.77 req/s
latency_ms avg=3812 p50=3814 p95=3815 max=3815
ttfb_ms avg=1378 p50=1342 p95=1815 max=2084
chunks_per_req avg=3.8 p50=4.0 p95=4.0 max=4.0
inter_chunk_ms avg=833.5 p50=672.5 p95=1972.7 max=2083.7
tail_emit_speed_x avg=1.21 p50=1.08 p95=1.93 max=2.19
timeline_to_first_batch_ms: avg=47.48 p50=47.24 p95=53.20 max=54.81
timeline_to_first_chunk_emitted_ms: avg=1364.16 p50=1328.46 p95=1794.00 max=2059.52
timeline_total_ms: avg=3708.54 p50=3796.63 p95=3805.51 max=3808.58
llm_time_in_queue_ms: avg=0.08 p50=0.08 p95=0.16 max=0.19
llm_scheduler_ms: avg=32.89 p50=33.17 p95=39.28 max=39.28
llm_first_token_ms: avg=45.72 p50=45.52 p95=52.23 max=53.94
llm_request_lifecycle_ms: avg=1716.83 p50=1741.83 p95=1882.49 max=2041.70
llm_generation_wall_ms: avg=3687.29 p50=3796.59 p95=3805.47 max=3808.54
llm_time_per_token_ms: avg=16.55 p50=16.54 p95=18.83 max=20.34
llm_batch_wall_ms_total: avg=2666.93 p50=2393.51 p95=3659.22 max=3770.81
llm_decode_wall_ms_total: avg=2619.48 p50=2339.96 p95=3614.98 max=3727.95
tokens_per_batch_p50: avg=1.00 p50=1.00 p95=1.00 max=1.00
bicodec_decode_calls: avg=2.81 p50=3.00 p95=3.00 max=3.00
bicodec_decode_wall_ms_total: avg=3147.90 p50=3199.08 p95=3251.78 max=3254.29
bicodec_decode_interval_ms_p50: avg=920.76 p50=1029.63 p95=1955.55 max=1972.71
latency_over_timeline_ms: avg=103.40 p50=17.54 p95=376.68 max=921.03
timeline_share: llm=99.4% bicodec=84.9% other=0.0%
gpu: util_avg=67% util_max=97% mem_avg=25009MB mem_max=25009MB

=== streaming concurrency=96 requests=96 ===
success=96/96 (100%)
wall=5.07s throughput=18.92 req/s
latency_ms avg=5069 p50=5070 p95=5072 max=5072
ttfb_ms avg=1979 p50=2085 p95=2790 max=2814
chunks_per_req avg=3.5 p50=3.0 p95=4.0 max=4.0
inter_chunk_ms avg=1221.2 p50=364.3 p95=3058.1 max=3233.4
tail_emit_speed_x avg=0.81 p50=0.73 p95=1.18 max=1.48
timeline_to_first_batch_ms: avg=52.13 p50=53.01 p95=60.91 max=63.12
timeline_to_first_chunk_emitted_ms: avg=1958.75 p50=2065.95 p95=2755.43 max=2777.47
timeline_total_ms: avg=4960.94 p50=5031.90 p95=5063.43 max=5065.59
llm_time_in_queue_ms: avg=0.13 p50=0.12 p95=0.25 max=0.28
llm_scheduler_ms: avg=35.79 p50=29.75 p95=47.36 max=47.36
llm_first_token_ms: avg=49.54 p50=50.58 p95=59.86 max=62.18
llm_request_lifecycle_ms: avg=1910.08 p50=1919.08 p95=2125.36 max=2420.69
llm_generation_wall_ms: avg=4941.34 p50=5031.68 p95=5063.39 max=5065.52
llm_time_per_token_ms: avg=22.09 p50=21.92 p95=25.66 max=26.62
llm_batch_wall_ms_total: avg=3112.91 p50=2755.68 p95=4911.72 max=5024.93
llm_decode_wall_ms_total: avg=3060.81 p50=2716.89 p95=4853.08 max=4969.24
tokens_per_batch_p50: avg=1.00 p50=1.00 p95=1.00 max=1.00
bicodec_decode_calls: avg=2.46 p50=2.00 p95=3.00 max=3.00
bicodec_decode_wall_ms_total: avg=4293.12 p50=4374.01 p95=4389.59 max=4392.36
bicodec_decode_interval_ms_p50: avg=1824.75 p50=2260.08 p95=2931.62 max=3019.06
latency_over_timeline_ms: avg=108.51 p50=38.14 p95=380.42 max=1501.01
timeline_share: llm=99.6% bicodec=86.5% other=0.0%
gpu: util_avg=64% util_max=95% mem_avg=25010MB mem_max=25011MB

results saved: /tmp/stream_baseline_levels_detailed.json
ERROR 02-18 01:12:21 [core_client.py:605] Engine core proc EngineCore_DP0 died unexpectedly, shutting down client.


# Collaboration Mode: Default

You are now in Default mode. Any previous instructions for other modes (e.g. Plan mode) are no longer active.

Your active mode changes only when new developer instructions with a different `<collaboration_mode>...</collaboration_mode>` change it; user requests or tool descriptions do not change mode by themselves. Known mode names are Default and Plan.

## request_user_input availability

The `request_user_input` tool is unavailable in Default mode. If you call it while in Default mode, it will return an error.

If a decision is necessary and cannot be discovered from local context, ask the user directly. However, in Default mode you should strongly prefer executing the user's request rather than stopping to ask questions.

Summary: auto

## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) When `SKILL.md` references relative paths (e.g., `scripts/foo.py`), resolve them relative to the skill directory listed above first, and only consider other paths if needed.
  3) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  4) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  5) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.


**Extracting key metrics with jq**


jq '.levels[0] | {throughput_rps, latency_ms, ttfb_ms, chunks_per_req, timing}' /tmp/stream_vllm_baseline_detailed.json


jq '.levels[0] | {throughput_rps, latency_ms, ttfb_ms, chunks_per_req, timing}' /tmp/stream_vllm_tuned_vllm_only.json


jq '.levels[0] | {throughput_rps, latency_ms, ttfb_ms, chunks_per_req, timing}' /tmp/stream_decode_tuned.json


jq '.levels[] | {concurrency, throughput_rps, latency_ms, ttfb_ms, timing: {llm_time_in_queue_ms, llm_scheduler_ms, llm_time_per_token_ms, bicodec_decode_wall_ms_total, bicodec_decode_interval_ms_p50}}' /tmp/stream_baseline_levels_detailed.json


Output:
Chunk ID: 9d9f38
Wall time: 0.0517 seconds
Process exited with code 0
Original token count: 753
Output:
{
  "throughput_rps": 17.311740257459824,
  "latency_ms": {
    "avg": 3690.2919153799303,
    "p50": 3695.0781773775816,
    "p95": 3695.929305627942,
    "max": 3696.0813840851188
  },
  "ttfb_ms": {
    "avg": 1532.1709263662342,
    "p50": 1536.0623486340046,
    "p95": 1975.5387287586927,
    "max": 2200.360244140029
  },
  "chunks_per_req": {
    "avg": 3.53125,
    "p50": 4,
    "p95": 4,
    "max": 4
  },
  "timing": {
    "timeline_to_first_batch_ms": {
      "avg": 39.86460994929075,
      "p50": 40.056051686406136,
      "p95": 45.62344681471586,
      "max": 47.2054909914732
    },
    "timeline_to_first_chunk_emitted_ms": {
      "avg": 1518.8397584715858,
      "p50": 1522.7289590984583,
      "p95": 1952.8557993471622,
      "max": 2176.557693630457
    },
    "timeline_total_ms": {
      "avg": 3586.043624600279,
      "p50": 3668.394595384598,
      "p95": 3690.473041497171,
      "max": 3692.90916249156
    },
    "llm_time_in_queue_ms": {
      "avg": 0.0617728364886716,
      "p50": 0.0570882111787796,
      "p95": 0.12011639773845673,
      "max": 0.13591721653938293
    },
    "llm_scheduler_ms": {
      "avg": 29.37699915491976,
      "p50": 26.255400851368904,
      "p95": 33.459863625466824,
      "max": 33.459863625466824
    },
    "llm_first_token_ms": {
      "avg": 37.788134068250656,
      "p50": 37.98794746398926,
      "p95": 44.72470283508301,
      "max": 46.40460014343262
    },
    "llm_request_lifecycle_ms": {
      "avg": 1590.336157460115,
      "p50": 1610.199049115181,
      "p95": 1756.5001072362065,
      "max": 1886.5547459572554
    },
    "llm_generation_wall_ms": {
      "avg": 3564.985583361704,
      "p50": 3668.271694332361,
      "p95": 3690.422055311501,
      "max": 3692.8681656718254
    },
    "llm_time_per_token_ms": {
      "avg": 16.17468517616502,
      "p50": 16.019027663890974,
      "p95": 18.17859614747438,
      "max": 20.06373952443157
    },
    "llm_batch_wall_ms_total": {
      "avg": 2398.9512129774084,
      "p50": 2234.1446299105883,
      "p95": 3579.045762307942,
      "max": 3669.7257552295923
    },
    "llm_decode_wall_ms_total": {
      "avg": 2359.1156284091994,
      "p50": 2190.3699403628707,
      "p95": 3535.98727658391,
      "max": 3632.4357949197292
    },
    "tokens_per_batch_p50": {
      "avg": 1,
      "p50": 1,
      "p95": 1,
      "max": 1
    },
    "bicodec_decode_calls": {
      "avg": 2.53125,
      "p50": 3,
      "p95": 3,
      "max": 3
    },
    "bicodec_decode_wall_ms_total": {
      "avg": 3057.9734236671356,
      "p50": 3141.73262193799,
      "p95": 3170.445444062352,
      "max": 3177.872314117849
    },
    "bicodec_decode_interval_ms_p50": {
      "avg": 1185.8032489399193,
      "p50": 1255.6158797815442,
      "p95": 1966.8617192655802,
      "max": 2015.6202567741275
    },
    "latency_over_timeline_ms": {
      "avg": 104.24829077965114,
      "p50": 26.4116358011961,
      "p95": 322.00404070317745,
      "max": 895.4648254439235
    }
  }
}


Output:
Chunk ID: 45854f
Wall time: 0.0510 seconds
Process exited with code 0
Original token count: 754
Output:
{
  "throughput_rps": 15.766053315314457,
  "latency_ms": {
    "avg": 4053.0143312062137,
    "p50": 4057.766165584326,
    "p95": 4058.5516039282084,
    "max": 4058.744768612087
  },
  "ttfb_ms": {
    "avg": 1653.1602599279722,
    "p50": 1650.4119569435716,
    "p95": 2161.4510091021657,
    "max": 2429.5830130577087
  },
  "chunks_per_req": {
    "avg": 3.515625,
    "p50": 4,
    "p95": 4,
    "max": 4
  },
  "timing": {
    "timeline_to_first_batch_ms": {
      "avg": 51.863698405213654,
      "p50": 52.05592233687639,
      "p95": 58.2311786711216,
      "max": 59.84123423695564
    },
    "timeline_to_first_chunk_emitted_ms": {
      "avg": 1639.5670478814282,
      "p50": 1637.0308678597212,
      "p95": 2137.7829760313034,
      "max": 2404.1909789666533
    },
    "timeline_total_ms": {
      "avg": 3951.8545484752394,
      "p50": 4030.622740276158,
      "p95": 4053.079890087247,
      "max": 4055.6297851726413
    },
    "llm_time_in_queue_ms": {
      "avg": 0.07771598757244647,
      "p50": 0.07571280002593994,
      "p95": 0.14848075807094574,
      "max": 0.1691998913884163
    },
    "llm_scheduler_ms": {
      "avg": 35.803992839646526,
      "p50": 27.82270312309265,
      "p95": 44.91256829351187,
      "max": 44.91256829351187
    },
    "llm_first_token_ms": {
      "avg": 49.79321360588074,
      "p50": 49.98373985290527,
      "p95": 56.93244934082031,
      "max": 58.66074562072754
    },
    "llm_request_lifecycle_ms": {
      "avg": 1656.0566241969354,
      "p50": 1666.7237086221576,
      "p95": 1832.677325233817,
      "max": 1976.873212493956
    },
    "llm_generation_wall_ms": {
      "avg": 3927.7114704746054,
      "p50": 4030.565571039915,
      "p95": 4053.0410073697567,
      "max": 4055.518804118037
    },
    "llm_time_per_token_ms": {
      "avg": 17.759664347839315,
      "p50": 17.766599461697695,
      "p95": 19.6180409550959,
      "max": 21.803486256583316
    },
    "llm_batch_wall_ms_total": {
      "avg": 2776.8393334845314,
      "p50": 2443.287312053144,
      "p95": 3923.4910383820534,
      "max": 4002.589608542621
    },
    "llm_decode_wall_ms_total": {
      "avg": 2725.0070511654485,
      "p50": 2383.4789507091045,
      "p95": 3872.854059562087,
      "max": 3948.844608850777
    },
    "tokens_per_batch_p50": {
      "avg": 1,
      "p50": 1,
      "p95": 1,
      "max": 1
    },
    "bicodec_decode_calls": {
      "avg": 2.515625,
      "p50": 3,
      "p95": 3,
      "max": 3
    },
    "bicodec_decode_wall_ms_total": {
      "avg": 3389.3982312147273,
      "p50": 3469.2275198176503,
      "p95": 3499.5473865419626,
      "max": 3508.252347819507
    },
    "bicodec_decode_interval_ms_p50": {
      "avg": 1204.371656247531,
      "p50": 1364.819354377687,
      "p95": 2292.754329741001,
      "max": 2325.5380298942327
    },
    "latency_over_timeline_ms": {
      "avg": 101.15978273097426,
      "p50": 26.0980986058712,
      "p95": 369.7821320965886,
      "max": 890.3919430449605
    }
  }
}


Output:
Chunk ID: af1264
Wall time: 0.0510 seconds
Process exited with code 0
Original token count: 754
Output:
{
  "throughput_rps": 17.248739056683277,
  "latency_ms": {
    "avg": 3706.5788207401056,
    "p50": 3708.891457878053,
    "p95": 3709.657806903124,
    "max": 3709.7078217193484
  },
  "ttfb_ms": {
    "avg": 1501.9540293869795,
    "p50": 1496.3948698714375,
    "p95": 1906.6688576713204,
    "max": 2108.566482551396
  },
  "chunks_per_req": {
    "avg": 3.5625,
    "p50": 4,
    "p95": 4,
    "max": 4
  },
  "timing": {
    "timeline_to_first_batch_ms": {
      "avg": 38.79740458796732,
      "p50": 39.00954686105251,
      "p95": 44.65858452022076,
      "max": 46.209780499339104
    },
    "timeline_to_first_chunk_emitted_ms": {
      "avg": 1488.9853868808132,
      "p50": 1482.5581600889564,
      "p95": 1884.097071364522,
      "max": 2084.5641000196338
    },
    "timeline_total_ms": {
      "avg": 3527.130705755553,
      "p50": 3670.3555658459663,
      "p95": 3704.025079496205,
      "max": 3705.2293391898274
    },
    "llm_time_in_queue_ms": {
      "avg": 0.07888044638093561,
      "p50": 0.07896963506937027,
      "p95": 0.15040487051010132,
      "max": 0.16134604811668396
    },
    "llm_scheduler_ms": {
      "avg": 28.757382693584077,
      "p50": 25.742605328559875,
      "p95": 32.69196301698685,
      "max": 32.69196301698685
    },
    "llm_first_token_ms": {
      "avg": 36.98468953371048,
      "p50": 37.16468811035156,
      "p95": 43.59889030456543,
      "max": 45.263051986694336
    },
    "llm_request_lifecycle_ms": {
      "avg": 1550.3597745409934,
      "p50": 1569.6776835247874,
      "p95": 1721.0342632606626,
      "max": 1855.8770390227437
    },
    "llm_generation_wall_ms": {
      "avg": 3502.9877730848966,
      "p50": 3640.783876180649,
      "p95": 3703.988059423864,
      "max": 3705.1867078989744
    },
    "llm_time_per_token_ms": {
      "avg": 15.908238983844084,
      "p50": 15.940670327190713,
      "p95": 18.256014803110965,
      "max": 20.142059233752104
    },
    "llm_batch_wall_ms_total": {
      "avg": 2587.15904186829,
      "p50": 2822.9790506884456,
      "p95": 3559.650088660419,
      "max": 3675.36713834852
    },
    "llm_decode_wall_ms_total": {
      "avg": 2548.3904693392105,
      "p50": 2794.265093281865,
      "p95": 3518.0147225037217,
      "max": 3634.7237508744
    },
    "tokens_per_batch_p50": {
      "avg": 1,
      "p50": 1,
      "p95": 1,
      "max": 1
    },
    "bicodec_decode_calls": {
      "avg": 2.5625,
      "p50": 3,
      "p95": 3,
      "max": 3
    },
    "bicodec_decode_wall_ms_total": {
      "avg": 2898.823211507988,
      "p50": 3023.4756814315915,
      "p95": 3071.713289245963,
      "max": 3073.1056444346905
    },
    "bicodec_decode_interval_ms_p50": {
      "avg": 939.9609970569145,
      "p50": 680.2855059504509,
      "p95": 1903.9592891931534,
      "max": 2111.618955619633
    },
    "latency_over_timeline_ms": {
      "avg": 179.44811498455238,
      "p50": 37.56709862500429,
      "p95": 685.6983788311481,
      "max": 1058.7234506383538
    }
  }
}


Output:
Chunk ID: 824dbf
Wall time: 0.0515 seconds
Process exited with code 0
Original token count: 414
Output:
{
  "concurrency": 32,
  "throughput_rps": 12.557102792804068,
  "latency_ms": {
    "avg": 2546.459506091196,
    "p50": 2547.595084644854,
    "p95": 2547.835980542004,
    "max": 2547.974002547562
  },
  "ttfb_ms": {
    "avg": 1046.6992497385945,
    "p50": 926.6489669680595,
    "p95": 1319.3976432085037,
    "max": 1345.9673514589667
  },
  "timing": {
    "llm_time_in_queue_ms": null,
    "llm_scheduler_ms": null,
    "llm_time_per_token_ms": null,
    "bicodec_decode_wall_ms_total": null,
    "bicodec_decode_interval_ms_p50": null
  }
}
{
  "concurrency": 64,
  "throughput_rps": 16.771739232775897,
  "latency_ms": {
    "avg": 3811.935635196278,
    "p50": 3814.2220182344317,
    "p95": 3814.9786507710814,
    "max": 3815.1524988934398
  },
  "ttfb_ms": {
    "avg": 1377.9939615633339,
    "p50": 1342.1410182490945,
    "p95": 1814.6885884925723,
    "max": 2083.6309837177396
  },
  "timing": {
    "llm_time_in_queue_ms": null,
    "llm_scheduler_ms": null,
    "llm_time_per_token_ms": null,
    "bicodec_decode_wall_ms_total": null,
    "bicodec_decode_interval_ms_p50": null
  }
}
{
  "concurrency": 96,
  "throughput_rps": 18.921961611771835,
  "latency_ms": {
    "avg": 5069.451287776853,
    "p50": 5070.159585215151,
    "p95": 5072.340160608292,
    "max": 5072.465548291802
  },
  "ttfb_ms": {
    "avg": 1979.0018753071006,
    "p50": 2085.385232232511,
    "p95": 2789.6522199735045,
    "max": 2814.453484490514
  },
  "timing": {
    "llm_time_in_queue_ms": null,
    "llm_scheduler_ms": null,
    "llm_time_per_token_ms": null,
    "bicodec_decode_wall_ms_total": null,
    "bicodec_decode_interval_ms_p50": null
  }
}


Summary: auto


**Parsing nested timing keys**


jq '.levels[] | {c:.concurrency, rps:.throughput_rps, lat_avg:.latency_ms.avg, ttfb_avg:.ttfb_ms.avg, queue_avg:.timing.llm_time_in_queue_ms.avg, sched_avg:.timing.llm_scheduler_ms.avg, tok_ms:.timing.llm_time_per_token_ms.avg, bicodec_ms:.timing.bicodec_decode_wall_ms_total.avg, decode_interval_p50:.timing.bicodec_decode_interval_ms_p50.avg}' /tmp/stream_baseline_levels_detailed.json


Output:
Chunk ID: 6d1bfc
Wall time: 0.0514 seconds
Process exited with code 0
Original token count: 219
Output:
{
  "c": 32,
  "rps": 12.557102792804068,
  "lat_avg": 2546.459506091196,
  "ttfb_avg": 1046.6992497385945,
  "queue_avg": 0.07949935388751328,
  "sched_avg": 28.118634596467018,
  "tok_ms": 10.622712277418243,
  "bicodec_ms": 1943.453844927717,
  "decode_interval_p50": 383.45315831247717
}
{
  "c": 64,
  "rps": 16.771739232775897,
  "lat_avg": 3811.935635196278,
  "ttfb_avg": 1377.9939615633339,
  "queue_avg": 0.08436896314378828,
  "sched_avg": 32.890047776163556,
  "tok_ms": 16.554943799508603,
  "bicodec_ms": 3147.9037028038874,
  "decode_interval_p50": 920.7632235629717
}
{
  "c": 96,
  "rps": 18.921961611771835,
  "lat_avg": 5069.451287776853,
  "ttfb_avg": 1979.0018753071006,
  "queue_avg": 0.1253390801139176,
  "sched_avg": 35.79173667822033,
  "tok_ms": 22.094432296068348,
  "bicodec_ms": 4293.123280726529,
  "decode_interval_p50": 1824.746812383334
}


**Planning diff analysis and verification**


cd /home/ubuntu/veenaModal && git status --short


Output:
Chunk ID: ce4a8b
Wall time: 0.0513 seconds
Process exited with code 0
Original token count: 183
Output:
 D CLEANUP_PLAN.md
 D PROGRESS.md
 D migration.md
 M veena3modal/api/fastapi_app.py
 M veena3modal/core/bicodec_decoder.py
 M veena3modal/core/constants.py
 M veena3modal/core/pipeline.py
 M veena3modal/core/streaming_pipeline.py
 M veena3modal/services/tts_runtime.py
?? scripts/profile_pipeline.py
?? scripts/setup_local.sh
?? scripts/stress_test_local.py
?? scripts/stress_test_optimized_local.py
?? scripts/stress_test_production.py
?? scripts/stress_test_runtime_detailed.py
?? scripts/stress_test_streaming_runtime.py
?? stress_results_opt_local.json
?? veena3modal/__main__.py
?? veena3modal/core/bicodec_batch_decoder.py
?? veena3modal/core/dual_engine.py
?? veena3modal/core/multi_engine.py
?? veena3modal/local_server.py


**Extracting change line numbers**


cd /home/ubuntu/veenaModal && nl -ba veena3modal/core/streaming_pipeline.py | sed -n '1,180p'


cd /home/ubuntu/veenaModal && nl -ba veena3modal/core/streaming_pipeline.py | sed -n '225,560p'


cd /home/ubuntu/veenaModal && nl -ba veena3modal/core/streaming_pipeline.py | sed -n '679,940p'


Output:
Chunk ID: ec6b0f
Wall time: 0.0504 seconds
Process exited with code 0
Original token count: 1756
Output:
     1	"""
     2	Veena3 Streaming Pipeline - Sliding Window Approach
     3
     4	Implements streaming for both SNAC and BiCodec tokens.
     5
     6	For BiCodec (Spark TTS):
     7	1. Generate tokens from vLLM
     8	2. Parse NEW token IDs incrementally (O(1) per token via cache)
     9	3. Buffer semantic + global tokens
    10	4. Apply sliding window (every N token pairs)
    11	5. Decode and stream audio chunks
    12
    13	OPTIMIZATION (Dec 2025):
    14	- Replaced O(n²) pattern (decode-all + regex-all per iteration)
    15	- Now uses incremental token parsing with BiCodecTokenParser
    16	- ~10x CPU reduction in streaming hot loop
    17
    18	For SNAC (legacy):
    19	1. Filter SNAC token IDs directly
    20	2. Apply sliding window
    21	3. Decode and stream
    22	"""
    23
    24	import asyncio
    25	import logging
    26	import os
    27	import re
    28	from typing import Any, AsyncGenerator, Dict, List, Optional, Tuple, Union
    29	from vllm import SamplingParams
    30	from vllm.sampling_params import RequestOutputKind
    31	from veena3modal.audio.crossfade import crossfade_bytes_int16
    32	from veena3modal.core.token_utils import BiCodecTokenParser
    33
    34	from veena3modal.core.constants import (
    35	    CODE_END_TOKEN_ID,
    36	    CODE_START_TOKEN_ID,
    37	    SNAC_MIN_ID,
    38	    SNAC_MAX_ID,
    39	    TRAINING_STOP_TOKEN_IDS,
    40	    DEFAULT_TEMPERATURE,
    41	    DEFAULT_TOP_K,
    42	    DEFAULT_TOP_P,
    43	    DEFAULT_MAX_TOKENS,
    44	    DEFAULT_MIN_TOKENS,
    45	    DEFAULT_REPETITION_PENALTY,
    46	    DEFAULT_SEED,
    47	)
    48
    49	logger = logging.getLogger(__name__)
    50	_STREAM_OUTPUT_KIND_RAW = os.environ.get("VEENA3_STREAM_OUTPUT_KIND", "delta").strip().lower()
    51	_STREAM_USE_DELTA_OUTPUT = _STREAM_OUTPUT_KIND_RAW in {"delta", "d", "1", "true", "yes", "on"}
    52
    53
    54	def _env_int(name: str, default: int, minimum: int = 0) -> int:
    55	    raw = os.environ.get(name, "")
    56	    if not raw:
    57	        return default
    58	    try:
    59	        value = int(raw)
    60	    except ValueError:
    61	        return default
    62	    return max(minimum, value)
    63
    64
    65	_STREAM_EXPECTED_GLOBAL_COUNT = 32
    66	_STREAM_MIN_SEMANTIC_FIRST_CHUNK = _env_int("VEENA3_STREAM_MIN_SEMANTIC_FIRST", 10, minimum=1)
    67	_STREAM_DECODE_INTERVAL = _env_int("VEENA3_STREAM_DECODE_INTERVAL", 48, minimum=1)
    68	_STREAM_CROSSFADE_MS = _env_int("VEENA3_STREAM_CROSSFADE_MS", 50, minimum=0)
    69	_STREAM_WINDOW_SIZE = _env_int("VEENA3_STREAM_WINDOW_SIZE", 128, minimum=16)
    70	_STREAM_WINDOWED_DECODE = os.environ.get("VEENA3_STREAM_WINDOWED_DECODE", "true").strip().lower() in {
    71	    "1",
    72	    "true",
    73	    "yes",
    74	    "on",
    75	}
    76
    77
    78	class Veena3SlidingWindowPipeline:
    79	    """
    80	    Streaming TTS pipeline using sliding window approach.
    81
    82	    This eliminates choppy audio and popping artifacts by:
    83	    - Decoding overlapping 28-token windows (4 frames)
    84	    - Keeping only the middle 2048 samples from each decode
    85	    - Creating natural continuity between chunks
    86
    87	    Based on the official Canopy Labs implementation.
    88	    """
    89
    90	    def __init__(
    91	        self,
    92	        model,
    93	        prompt_builder,
    94	        snac_decoder,
    95	    ):
    96	        """
    97	        Initialize sliding window streaming pipeline.
    98
    99	        Args:
   100	            model: Veena3Model instance
   101	            prompt_builder: Veena3PromptBuilder instance
   102	            snac_decoder: SNACDecoder instance (with batching enabled)
   103	        """
   104	        self.model = model
   105	        self.prompt_builder = prompt_builder
   106	        self.snac_decoder = snac_decoder
   107
   108	        # OPTIMIZATION: Pre-warm BiCodecTokenParser once at init, not per-request
   109	        # Saves ~123ms per streaming TTFB by avoiding repeated 166K-entry vocab iteration
   110	        tokenizer = getattr(model, "tokenizer", None)
   111	        if tokenizer is None:
   112	            tokenizer = getattr(model.engine, "tokenizer", None)
   113	        self.token_parser = BiCodecTokenParser(tokenizer) if tokenizer else None
   114
   115	        print(f"🌊 Veena3SlidingWindowPipeline initialized (sliding window: 28 tokens)")
   116
   117	    @staticmethod
   118	    def _p50(values: List[float]) -> float:
   119	        if not values:
   120	            return 0.0
   121	        ordered = sorted(values)
   122	        return ordered[(len(ordered) - 1) // 2]
   123
   124	    @staticmethod
   125	    def _to_number(value: Any) -> Optional[float]:
   126	        if value is None:
   127	            return None
   128	        if isinstance(value, (int, float)):
   129	            return float(value)
   130	        try:
   131	            return float(value)
   132	        except (TypeError, ValueError):
   133	            return None
   134
   135	    def _extract_request_metrics(self, raw_metrics: Any) -> Dict[str, float]:
   136	        if raw_metrics is None:
   137	            return {}
   138
   139	        numeric: Dict[str, float] = {}
   140	        items: List[Tuple[str, Any]] = []
   141
   142	        if isinstance(raw_metrics, dict):
   143	            items.extend(raw_metrics.items())
   144	        else:
   145	            if hasattr(raw_metrics, "__dict__"):
   146	                try:
   147	                    items.extend(dict(raw_metrics.__dict__).items())
   148	                except Exception:
   149	                    pass
   150	            for name in dir(raw_metrics):
   151	                if name.startswith("_"):
   152	                    continue
   153	                try:
   154	                    value = getattr(raw_metrics, name)
   155	                except Exception:
   156	                    continue
   157	                if callable(value):
   158	                    continue
   159	                items.append((name, value))
   160
   161	        seen: set[str] = set()
   162	        for key, value in items:
   163	            key_s = str(key)
   164	            if key_s in seen:
   165	                continue
   166	            seen.add(key_s)
   167	            parsed = self._to_number(value)
   168	            if parsed is not None:
   169	                numeric[key_s] = parsed
   170	        return numeric
   171
   172	    def _apply_stats(self, perf: Dict[str, Any], prefix: str, values: List[float]) -> None:
   173	        if not values:
   174	            return
   175	        perf[f"{prefix}_total"] = float(sum(values))
   176	        perf[f"{prefix}_min"] = float(min(values))
   177	        perf[f"{prefix}_max"] = float(max(values))
   178	        perf[f"{prefix}_p50"] = float(self._p50(values))
   179
   180	    async def generate_speech_stream(


Output:
Chunk ID: 8625cf
Wall time: 0.0507 seconds
Process exited with code 0
Original token count: 4487
Output:
   225	        logger.debug("sampling temp=%s top_p=%s sliding_window=28", temperature, top_p)
   226
   227	        # Token buffer - keeps ALL tokens (not chunked)
   228	        token_buffer = []
   229	        total_tokens_generated = 0
   230	        total_audio_chunks = 0
   231
   232	        # Generate tokens with vLLM (streaming)
   233	        print(f"🔮 Starting token generation...")
   234
   235	        # Generate unique request ID for concurrent streaming support
   236	        import uuid
   237	        import time
   238	        request_id = f"slide-{uuid.uuid4().hex[:8]}-{int(time.time() * 1000000)}"
   239	        results_generator = self.model.engine.generate(
   240	            prompt=prompt,
   241	            sampling_params=sampling_params,
   242	            request_id=request_id,
   243	        )
   244
   245	        # Stream tokens with sliding window
   246	        async for request_output in results_generator:
   247	            # Extract generated token IDs
   248	            generated_ids = request_output.outputs[0].token_ids
   249
   250	            # Process only new tokens
   251	            new_tokens = generated_ids[total_tokens_generated:]
   252	            total_tokens_generated = len(generated_ids)
   253
   254	            # Filter and buffer SNAC tokens
   255	            for token_id in new_tokens:
   256	                if SNAC_MIN_ID <= token_id <= SNAC_MAX_ID:
   257	                    token_buffer.append(token_id)
   258
   259	                    # Process every 7 tokens (1 frame) when we have enough for sliding window
   260	                    # Official approach: once count > 27, take last 28 tokens
   261	                    if len(token_buffer) % 7 == 0 and len(token_buffer) > 27:
   262	                        # Sliding window: take last 28 tokens (4 frames)
   263	                        window_tokens = token_buffer[-28:]
   264
   265	                        # Decode with sliding window mode (returns middle 2048 samples only)
   266	                        if self.snac_decoder.enable_batching:
   267	                            audio_bytes = await self.snac_decoder.decode_single_async(
   268	                                window_tokens,
   269	                                trim_warmup=False,  # Sliding window handles trimming
   270	                                use_sliding_window=True  # CRITICAL: Use sliding window mode
   271	                            )
   272	                        else:
   273	                            audio_bytes = self.snac_decoder.decode_to_bytes(
   274	                                window_tokens,
   275	                                trim_warmup=False,
   276	                                use_sliding_window=True
   277	                            )
   278
   279	                        if audio_bytes:
   280	                            total_audio_chunks += 1
   281	                            if total_audio_chunks == 1:
   282	                                logger.debug("first sliding-window chunk decoded bytes=%d", len(audio_bytes))
   283	                            yield audio_bytes
   284
   285	        # Note: No final chunk processing needed - sliding window handles all tokens
   286	        # as they come in (every 7 tokens after the first 28)
   287
   288	        logger.debug(
   289	            "sliding-window complete tokens=%d chunks=%d",
   290	            total_tokens_generated,
   291	            total_audio_chunks,
   292	        )
   293
   294	    def _extract_bicodec_tokens_from_text(self, text: str) -> tuple[List[int], List[int]]:
   295	        """
   296	        Extract BiCodec semantic and global tokens from generated text.
   297
   298	        Args:
   299	            text: Generated text containing BiCodec token markers
   300
   301	        Returns:
   302	            Tuple of (semantic_ids, global_ids)
   303	        """
   304	        semantic_matches = re.findall(r"<\|bicodec_semantic_(\d+)\|>", text)
   305	        semantic_ids = [int(t) for t in semantic_matches]
   306
   307	        global_matches = re.findall(r"<\|bicodec_global_(\d+)\|>", text)
   308	        global_ids = [int(t) for t in global_matches]
   309
   310	        return semantic_ids, global_ids
   311
   312	    async def generate_speech_stream_indic(
   313	        self,
   314	        speaker: str,
   315	        text: str,
   316	        temperature: float = DEFAULT_TEMPERATURE,
   317	        top_k: int = DEFAULT_TOP_K,  # Added for Spark TTS parity with non-streaming
   318	        top_p: float = DEFAULT_TOP_P,
   319	        max_tokens: int = DEFAULT_MAX_TOKENS,
   320	        repetition_penalty: float = DEFAULT_REPETITION_PENALTY,
   321	        seed: Optional[int] = None,
   322	        emit_progress: bool = False,
   323	    ) -> AsyncGenerator[Union[bytes, Tuple[bytes, Dict[str, Any]]], None]:
   324	        """
   325	        Generate speech audio with sliding window streaming for Indic model.
   326
   327	        Yields audio chunks using overlapping windows for smooth playback.
   328
   329	        Args:
   330	            speaker: Speaker name (one of 12 predefined speakers)
   331	            text: Text to synthesize with inline emotion tags
   332	                Examples:
   333	                - "Hello! Welcome."
   334	                - "<laugh> Hello there!"
   335	                - "नमस्ते! <excited> आज का दिन बहुत अच्छा है।"
   336	            temperature: Sampling temperature
   337	            top_p: Nucleus sampling
   338	            max_tokens: Max SNAC tokens to generate
   339	            repetition_penalty: Prevent loops
   340	            seed: Random seed for reproducibility
   341
   342	        Yields:
   343	            Audio bytes (int16 PCM, 16kHz mono - BiCodec)
   344	        """
   345	        import time
   346	        import uuid
   347
   348	        t_request_start = time.perf_counter()
   349	        timeline_marks: List[Dict[str, float]] = [{"stage": "request_start", "t_ms": 0.0, "dt_ms": 0.0}]
   350	        last_mark_ms = 0.0
   351	        mark_stage_set: set[str] = {"request_start"}
   352
   353	        def _mark(stage: str) -> float:
   354	            nonlocal last_mark_ms
   355	            now_ms = (time.perf_counter() - t_request_start) * 1000.0
   356	            dt_ms = now_ms - last_mark_ms
   357	            last_mark_ms = now_ms
   358	            timeline_marks.append({"stage": stage, "t_ms": float(now_ms), "dt_ms": float(dt_ms)})
   359	            mark_stage_set.add(stage)
   360	            return float(now_ms)
   361
   362	        prompt_t0 = time.perf_counter()
   363	        prompt = self.prompt_builder.build_prefix(speaker, text)
   364	        prompt_build_ms = (time.perf_counter() - prompt_t0) * 1000.0
   365	        _mark("prompt_ready")
   366
   367	        sampling_t0 = time.perf_counter()
   368	        sampling_params = SamplingParams(
   369	            temperature=temperature,
   370	            top_k=top_k,
   371	            top_p=top_p,
   372	            max_tokens=max_tokens,
   373	            min_tokens=DEFAULT_MIN_TOKENS,
   374	            repetition_penalty=repetition_penalty,
   375	            stop=TRAINING_STOP_TOKEN_IDS,
   376	            skip_special_tokens=False,
   377	            seed=seed,
   378	            output_kind=(RequestOutputKind.DELTA if _STREAM_USE_DELTA_OUTPUT else RequestOutputKind.CUMULATIVE),
   379	        )
   380	        sampling_params_ms = (time.perf_counter() - sampling_t0) * 1000.0
   381	        _mark("sampling_ready")
   382
   383	        request_id = f"bicodec-stream-{uuid.uuid4().hex[:8]}-{int(time.time() * 1000000)}"
   384	        token_parser = self.token_parser
   385	        if token_parser is None:
   386	            tokenizer = getattr(self.model, "tokenizer", None) or getattr(self.model.engine, "tokenizer", None)
   387	            token_parser = BiCodecTokenParser(tokenizer) if tokenizer else None
   388	        if token_parser is None:
   389	            raise RuntimeError("BiCodecTokenParser unavailable for streaming")
   390
   391	        results_generator = self.model.engine.generate(
   392	            prompt=prompt,
   393	            sampling_params=sampling_params,
   394	            request_id=request_id,
   395	        )
   396
   397	        semantic_buffer: List[int] = []
   398	        global_buffer: List[int] = []
   399	        processed_token_count = 0
   400	        total_generated_tokens = 0
   401	        total_audio_chunks = 0
   402	        total_samples_emitted_to_user = 0
   403	        previous_chunk_tail: Optional[bytes] = None
   404	        last_decode_count = 0
   405	        last_decode_end_ts: Optional[float] = None
   406	        t_first_chunk_decode_done: Optional[float] = None
   407	        t_first_chunk_emitted: Optional[float] = None
   408	        final_request_metrics: Dict[str, float] = {}
   409	        prev_gpu_cumulative_s: Optional[float] = None
   410
   411	        llm_batch_wall_ms: List[float] = []
   412	        llm_batch_gpu_ms: List[float] = []
   413	        llm_decode_wall_ms: List[float] = []
   414	        llm_decode_gpu_ms: List[float] = []
   415	        llm_parse_ms_samples: List[float] = []
   416	        tokens_per_batch: List[float] = []
   417	        bicodec_decode_wall_ms: List[float] = []
   418	        bicodec_decode_tokens: List[float] = []
   419	        bicodec_decode_intervals_ms: List[float] = []
   420
   421	        t_llm_start = time.perf_counter()
   422	        t_last_batch = t_llm_start
   423
   424	        async for request_output in results_generator:
   425	            now = time.perf_counter()
   426	            batch_wall = (now - t_last_batch) * 1000.0
   427	            t_last_batch = now
   428
   429	            outputs = request_output.outputs or []
   430	            if not outputs:
   431	                continue
   432
   433	            generated_ids = outputs[0].token_ids
   434	            if _STREAM_USE_DELTA_OUTPUT:
   435	                new_token_ids = generated_ids
   436	                processed_token_count += len(new_token_ids)
   437	            else:
   438	                new_token_ids = generated_ids[processed_token_count:]
   439	                processed_token_count = len(generated_ids)
   440
   441	            new_token_count = len(new_token_ids)
   442	            if new_token_count <= 0:
   443	                continue
   444
   445	            total_generated_tokens += new_token_count
   446	            llm_batch_wall_ms.append(float(batch_wall))
   447	            tokens_per_batch.append(float(new_token_count))
   448
   449	            if "llm_first_batch" not in mark_stage_set:
   450	                _mark("llm_first_batch")
   451	            elif llm_batch_wall_ms:
   452	                llm_decode_wall_ms.append(float(batch_wall))
   453
   454	            raw_req_metrics = self._extract_request_metrics(getattr(request_output, "metrics", None))
   455	            if raw_req_metrics:
   456	                final_request_metrics = raw_req_metrics
   457	                cumulative_gpu_s: Optional[float] = None
   458	                if "model_execute_time" in raw_req_metrics:
   459	                    cumulative_gpu_s = raw_req_metrics["model_execute_time"]
   460	                elif "model_forward_time" in raw_req_metrics:
   461	                    cumulative_gpu_s = raw_req_metrics["model_forward_time"]
   462
   463	                if cumulative_gpu_s is not None:
   464	                    if prev_gpu_cumulative_s is None:
   465	                        gpu_delta_ms = cumulative_gpu_s * 1000.0
   466	                    elif cumulative_gpu_s >= prev_gpu_cumulative_s:
   467	                        gpu_delta_ms = (cumulative_gpu_s - prev_gpu_cumulative_s) * 1000.0
   468	                    else:
   469	                        gpu_delta_ms = cumulative_gpu_s * 1000.0
   470	                    prev_gpu_cumulative_s = cumulative_gpu_s
   471	                    if gpu_delta_ms >= 0.0:
   472	                        llm_batch_gpu_ms.append(float(gpu_delta_ms))
   473	                        if len(llm_batch_wall_ms) > 1:
   474	                            llm_decode_gpu_ms.append(float(gpu_delta_ms))
   475
   476	            parse_t0 = time.perf_counter()
   477	            token_parser.parse_incremental(new_token_ids, semantic_buffer, global_buffer)
   478	            llm_parse_ms_samples.append((time.perf_counter() - parse_t0) * 1000.0)
   479
   480	            if len(global_buffer) >= _STREAM_EXPECTED_GLOBAL_COUNT:
   481	                semantic_count = len(semantic_buffer)
   482	                if semantic_count >= _STREAM_MIN_SEMANTIC_FIRST_CHUNK:
   483	                    if "first_chunk_ready" not in mark_stage_set:
   484	                        _mark("first_chunk_ready")
   485
   486	                    if semantic_count - last_decode_count >= _STREAM_DECODE_INTERVAL:
   487	                        last_decode_count = semantic_count
   488	                        decode_global = global_buffer[:_STREAM_EXPECTED_GLOBAL_COUNT]
   489
   490	                        if (
   491	                            _STREAM_WINDOWED_DECODE
   492	                            and semantic_count > _STREAM_WINDOW_SIZE
   493	                        ):
   494	                            decode_semantic = semantic_buffer[-_STREAM_WINDOW_SIZE:]
   495	                        else:
   496	                            decode_semantic = semantic_buffer
   497
   498	                        decode_t0 = time.perf_counter()
   499	                        audio_bytes = await self.snac_decoder.decode_single_async(
   500	                            semantic_ids=decode_semantic,
   501	                            global_ids=decode_global,
   502	                            trim_warmup=False,
   503	                            use_sliding_window=False,
   504	                        )
   505	                        decode_t1 = time.perf_counter()
   506
   507	                        decode_wall_ms = (decode_t1 - decode_t0) * 1000.0
   508	                        bicodec_decode_wall_ms.append(float(decode_wall_ms))
   509	                        bicodec_decode_tokens.append(float(len(decode_semantic)))
   510	                        if last_decode_end_ts is not None:
   511	                            bicodec_decode_intervals_ms.append((decode_t1 - last_decode_end_ts) * 1000.0)
   512	                        last_decode_end_ts = decode_t1
   513
   514	                        if audio_bytes:
   515	                            total_samples_decoded = len(audio_bytes) // 2
   516	                            if (
   517	                                _STREAM_WINDOWED_DECODE
   518	                                and semantic_count > _STREAM_WINDOW_SIZE
   519	                            ):
   520	                                new_samples_approx = _STREAM_DECODE_INTERVAL * 320
   521	                                if total_samples_decoded > new_samples_approx:
   522	                                    new_bytes_start = (total_samples_decoded - new_samples_approx) * 2
   523	                                else:
   524	                                    new_bytes_start = 0
   525	                                new_audio_bytes = audio_bytes[new_bytes_start:]
   526	                            else:
   527	                                if total_samples_decoded > total_samples_emitted_to_user:
   528	                                    new_bytes_start = total_samples_emitted_to_user * 2
   529	                                    new_audio_bytes = audio_bytes[new_bytes_start:]
   530	                                else:
   531	                                    new_audio_bytes = b""
   532
   533	                            if new_audio_bytes:
   534	                                to_emit, previous_chunk_tail = crossfade_bytes_int16(
   535	                                    previous_chunk_tail,
   536	                                    new_audio_bytes,
   537	                                    sample_rate_hz=16000,
   538	                                    crossfade_ms=_STREAM_CROSSFADE_MS,
   539	                                )
   540	                                if to_emit:
   541	                                    total_audio_chunks += 1
   542	                                    total_samples_emitted_to_user += len(to_emit) // 2
   543	                                    if t_first_chunk_decode_done is None:
   544	                                        t_first_chunk_decode_done = decode_t1
   545	                                        _mark("first_chunk_decoded")
   546	                                    if t_first_chunk_emitted is None:
   547	                                        t_first_chunk_emitted = time.perf_counter()
   548	                                        _mark("first_chunk_emitted")
   549	                                    if emit_progress:
   550	                                        yield (to_emit, {})
   551	                                    else:
   552	                                        yield to_emit
   553
   554	        llm_generation_wall_ms = (time.perf_counter() - t_llm_start) * 1000.0
   555	        _mark("llm_done")
   556
   557	        if len(semantic_buffer) > last_decode_count and len(global_buffer) >= _STREAM_EXPECTED_GLOBAL_COUNT:
   558	            decode_semantic = semantic_buffer
   559	            decode_t0 = time.perf_counter()
   560	            audio_bytes = self.snac_decoder.decode_streaming(


Output:
Chunk ID: 57d2e8
Wall time: 0.0515 seconds
Process exited with code 0
Original token count: 3436
Output:
   679	            "model_execute_time": "llm_model_execute_ms",
   680	            "first_token_latency": "llm_first_token_ms",
   681	        }
   682	        for src, dst in sec_fields.items():
   683	            if src in final_request_metrics:
   684	                perf[dst] = float(final_request_metrics[src]) * 1000.0
   685
   686	        def _delta_ms(start_s: Optional[float], end_s: Optional[float]) -> Optional[float]:
   687	            if start_s is None or end_s is None:
   688	                return None
   689	            if end_s < start_s:
   690	                return None
   691	            return (end_s - start_s) * 1000.0
   692
   693	        queued_ts = final_request_metrics.get("queued_ts")
   694	        scheduled_ts = final_request_metrics.get("scheduled_ts")
   695	        first_token_ts = final_request_metrics.get("first_token_ts")
   696	        last_token_ts = final_request_metrics.get("last_token_ts")
   697	        q_to_sched = _delta_ms(queued_ts, scheduled_ts)
   698	        sched_to_first = _delta_ms(scheduled_ts, first_token_ts)
   699	        first_to_last = _delta_ms(first_token_ts, last_token_ts)
   700	        queued_to_last = _delta_ms(queued_ts, last_token_ts)
   701	        if q_to_sched is not None:
   702	            perf["llm_queued_to_scheduled_ms"] = q_to_sched
   703	            perf["llm_time_in_queue_ms"] = q_to_sched
   704	        if sched_to_first is not None:
   705	            perf["llm_scheduled_to_first_token_ms"] = sched_to_first
   706	            perf["llm_scheduler_ms"] = sched_to_first
   707	        if first_to_last is not None:
   708	            perf["llm_first_to_last_token_ms"] = first_to_last
   709	            perf["llm_model_execute_ms"] = first_to_last
   710	        if queued_to_last is not None:
   711	            perf["llm_queued_to_last_token_ms"] = queued_to_last
   712	            perf["llm_request_lifecycle_ms"] = queued_to_last
   713
   714	        logger.debug(
   715	            "streaming complete audio=%.2fs ttfb=%.0fms chunks=%d llm_tokens=%d",
   716	            perf["audio_duration_seconds"],
   717	            perf["ttfb_ms"],
   718	            perf["chunks_sent"],
   719	            perf["llm_token_total"],
   720	        )
   721
   722	        if emit_progress:
   723	            yield (b"", perf)
   724
   725	    async def generate_speech_stream_indic_first_chunk(
   726	        self,
   727	        speaker: str,
   728	        text: str,
   729	        temperature: float = DEFAULT_TEMPERATURE,
   730	        top_k: int = DEFAULT_TOP_K,
   731	        top_p: float = DEFAULT_TOP_P,
   732	        max_tokens: int = DEFAULT_MAX_TOKENS,
   733	        repetition_penalty: float = DEFAULT_REPETITION_PENALTY,
   734	        seed: Optional[int] = None,
   735	    ) -> AsyncGenerator[tuple, None]:
   736	        """
   737	        Generate speech for the FIRST chunk of multi-chunk text, capturing global tokens.
   738
   739	        This method is specifically for chunked generation: it yields (audio_bytes, global_ids)
   740	        tuples instead of just audio_bytes. The caller captures global_ids from the first
   741	        yield and passes them to generate_speech_stream_indic_continuation() for subsequent chunks.
   742
   743	        Use Case (chunked streaming with voice consistency):
   744	            globals_captured = None
   745	            async for audio_bytes, global_ids in pipeline.generate_speech_stream_indic_first_chunk(...):
   746	                if globals_captured is None and global_ids:
   747	                    globals_captured = global_ids  # Capture once
   748	                yield audio_bytes
   749	            # Now use globals_captured for subsequent chunks
   750
   751	        Args:
   752	            speaker: Speaker name
   753	            text: First text chunk to synthesize
   754	            temperature: Sampling temperature
   755	            top_k: Top-k sampling
   756	            top_p: Nucleus sampling
   757	            max_tokens: Max tokens to generate
   758	            repetition_penalty: Prevent repetition
   759	            seed: Random seed for reproducibility
   760
   761	        Yields:
   762	            Tuple of (audio_bytes, global_ids)
   763	            - audio_bytes: Raw PCM audio (int16, 16kHz)
   764	            - global_ids: List of 32 global token IDs (populated after first decode, else empty)
   765
   766	        Thread Safety:
   767	            This method is stateless and thread-safe. Each call creates its own
   768	            request-scoped state. No global state is modified or shared.
   769	        """
   770	        import time
   771	        t_start = time.time()
   772
   773	        # Build prompt using Indic prompt builder
   774	        prompt = self.prompt_builder.build_prefix(speaker, text)
   775
   776	        # Configure sampling
   777	        # OPTIMIZATION: Use Spark TTS stop token, not legacy SNAC CODE_END_TOKEN_ID
   778	        sampling_params = SamplingParams(
   779	            temperature=temperature,
   780	            top_k=top_k,
   781	            top_p=top_p,
   782	            max_tokens=max_tokens,
   783	            min_tokens=DEFAULT_MIN_TOKENS,
   784	            repetition_penalty=repetition_penalty,
   785	            stop=TRAINING_STOP_TOKEN_IDS,  # "<|im_end|>" - matches non-streaming pipeline
   786	            skip_special_tokens=False,
   787	            seed=seed,
   788	            output_kind=(RequestOutputKind.DELTA if _STREAM_USE_DELTA_OUTPUT else RequestOutputKind.CUMULATIVE),
   789	        )
   790
   791	        # BiCodec token buffers
   792	        semantic_buffer = []
   793	        global_buffer = []
   794	        processed_token_count = 0  # Used for cumulative mode; ignored for delta mode.
   795	        total_audio_chunks = 0
   796
   797	        # BiCodec streaming configuration
   798	        EXPECTED_GLOBAL_COUNT = 32
   799	        MIN_SEMANTIC_FOR_FIRST_CHUNK = 10  # OPTIMIZATION: Lowered from 16 (decoder min is 8)
   800	        DECODE_INTERVAL = 48  # OPTIMIZATION: Increased from 24 to reduce O(n) re-decode calls
   801	        CROSSFADE_MS = 50
   802
   803	        # Generate unique request ID
   804	        import uuid
   805	        request_id = f"bicodec-first-{uuid.uuid4().hex[:8]}-{int(time.time() * 1000000)}"
   806
   807	        # Reuse pre-warmed token parser (singleton, created once in __init__)
   808	        token_parser = self.token_parser
   809
   810	        results_generator = self.model.engine.generate(
   811	            prompt=prompt,
   812	            sampling_params=sampling_params,
   813	            request_id=request_id,
   814	        )
   815
   816	        # Track state
   817	        last_decode_count = 0
   818	        total_samples_emitted_to_user = 0
   819	        previous_chunk_tail: Optional[bytes] = None
   820	        captured_globals: List[int] = []  # Will be populated once we have 32 globals
   821
   822	        # Stream tokens with BiCodec extraction
   823	        async for request_output in results_generator:
   824	            generated_ids = request_output.outputs[0].token_ids
   825
   826	            # In DELTA mode token_ids are already incremental.
   827	            if _STREAM_USE_DELTA_OUTPUT:
   828	                new_token_ids = generated_ids
   829	            else:
   830	                new_token_ids = generated_ids[processed_token_count:]
   831	                processed_token_count = len(generated_ids)
   832	            token_parser.parse_incremental(new_token_ids, semantic_buffer, global_buffer)
   833
   834	            # Capture global tokens once we have all 32
   835	            if not captured_globals and len(global_buffer) >= EXPECTED_GLOBAL_COUNT:
   836	                captured_globals = global_buffer[:EXPECTED_GLOBAL_COUNT]
   837
   838	            # Process if we have all global tokens
   839	            if len(global_buffer) >= EXPECTED_GLOBAL_COUNT:
   840	                semantic_count = len(semantic_buffer)
   841	                if semantic_count >= MIN_SEMANTIC_FOR_FIRST_CHUNK:
   842	                    if semantic_count - last_decode_count >= DECODE_INTERVAL:
   843	                        last_decode_count = semantic_count
   844
   845	                        all_semantic = semantic_buffer
   846	                        window_global = global_buffer[:EXPECTED_GLOBAL_COUNT]
   847
   848	                        if self.snac_decoder.enable_batching:
   849	                            audio_bytes = await self.snac_decoder.decode_single_async(
   850	                                semantic_ids=all_semantic,
   851	                                global_ids=window_global,
   852	                                trim_warmup=False,
   853	                                use_sliding_window=False
   854	                            )
   855	                        else:
   856	                            audio_bytes = self.snac_decoder.decode_streaming(
   857	                                semantic_ids=all_semantic,
   858	                                global_ids=window_global,
   859	                                use_sliding_window=False,
   860	                                trim_warmup=False
   861	                            )
   862
   863	                        if audio_bytes:
   864	                            total_samples_decoded = len(audio_bytes) // 2
   865
   866	                            if total_samples_decoded > total_samples_emitted_to_user:
   867	                                new_bytes_start = total_samples_emitted_to_user * 2
   868	                                new_audio_bytes = audio_bytes[new_bytes_start:]
   869
   870	                                to_emit, previous_chunk_tail = crossfade_bytes_int16(
   871	                                    previous_chunk_tail,
   872	                                    new_audio_bytes,
   873	                                    sample_rate_hz=16000,
   874	                                    crossfade_ms=CROSSFADE_MS,
   875	                                )
   876
   877	                                samples_emitted_in_this_chunk = len(to_emit) // 2
   878	                                total_samples_emitted_to_user += samples_emitted_in_this_chunk
   879
   880	                                if to_emit:
   881	                                    total_audio_chunks += 1
   882	                                    # Yield tuple: (audio_bytes, global_ids)
   883	                                    yield (to_emit, captured_globals)
   884
   885	        # Final chunk processing
   886	        if len(semantic_buffer) > last_decode_count and len(global_buffer) >= EXPECTED_GLOBAL_COUNT:
   887	            audio_bytes = self.snac_decoder.decode_streaming(
   888	                semantic_ids=semantic_buffer,
   889	                global_ids=global_buffer[:EXPECTED_GLOBAL_COUNT],
   890	                use_sliding_window=False,
   891	                trim_warmup=False
   892	            )
   893	            if audio_bytes:
   894	                total_samples_decoded = len(audio_bytes) // 2
   895
   896	                if total_samples_decoded > total_samples_emitted_to_user:
   897	                    new_bytes_start = total_samples_emitted_to_user * 2
   898	                    new_audio_bytes = audio_bytes[new_bytes_start:]
   899
   900	                    to_emit, previous_chunk_tail = crossfade_bytes_int16(
   901	                        previous_chunk_tail,
   902	                        new_audio_bytes,
   903	                        sample_rate_hz=16000,
   904	                        crossfade_ms=CROSSFADE_MS,
   905	                    )
   906
   907	                    samples_emitted_in_final = len(to_emit) // 2
   908	                    total_samples_emitted_to_user += samples_emitted_in_final
   909
   910	                    if to_emit:
   911	                        total_audio_chunks += 1
   912	                        yield (to_emit, captured_globals)
   913
   914	        # Flush remaining tail
   915	        if previous_chunk_tail:
   916	            total_audio_chunks += 1
   917	            tail_samples = len(previous_chunk_tail) // 2
   918	            total_samples_emitted_to_user += tail_samples
   919	            yield (previous_chunk_tail, captured_globals)
   920
   921	        t_end = time.time()
   922	        audio_duration_s = total_samples_emitted_to_user / 16000
   923	        logger.debug(
   924	            "first-chunk complete audio=%.2fs chunks=%d captured_globals=%d",
   925	            audio_duration_s,
   926	            total_audio_chunks,
   927	            len(captured_globals),
   928	        )
   929
   930	    async def generate_speech_stream_indic_continuation(
   931	        self,
   932	        speaker: str,
   933	        text: str,
   934	        global_ids: List[int],
   935	        temperature: float = DEFAULT_TEMPERATURE,
   936	        top_k: int = DEFAULT_TOP_K,
   937	        top_p: float = DEFAULT_TOP_P,
   938	        max_tokens: int = DEFAULT_MAX_TOKENS,
   939	        repetition_penalty: float = DEFAULT_REPETITION_PENALTY,
   940	        seed: Optional[int] = None,


**Planning further code extraction**


cd /home/ubuntu/veenaModal && nl -ba veena3modal/core/streaming_pipeline.py | sed -n '930,1130p'


Output:
Chunk ID: 2e2387
Wall time: 0.0512 seconds
Process exited with code 0
Original token count: 2728
Output:
   930	    async def generate_speech_stream_indic_continuation(
   931	        self,
   932	        speaker: str,
   933	        text: str,
   934	        global_ids: List[int],
   935	        temperature: float = DEFAULT_TEMPERATURE,
   936	        top_k: int = DEFAULT_TOP_K,
   937	        top_p: float = DEFAULT_TOP_P,
   938	        max_tokens: int = DEFAULT_MAX_TOKENS,
   939	        repetition_penalty: float = DEFAULT_REPETITION_PENALTY,
   940	        seed: Optional[int] = None,
   941	        emit_progress: bool = False,
   942	    ) -> AsyncGenerator[Union[bytes, Tuple[bytes, Dict[str, Any]]], None]:
   943	        """
   944	        Generate speech for CONTINUATION chunks using pre-captured global tokens.
   945
   946	        This is the key method for voice consistency in chunked generation:
   947	        - Uses global_ids from the first chunk to maintain identical voice
   948	        - Model generates only semantic tokens (skips global token generation)
   949	        - Ensures no voice drift across chunks
   950
   951	        CRITICAL for production:
   952	        - global_ids is request-scoped, passed explicitly (no shared state)
   953	        - Thread-safe: each request has its own state
   954	        - No global caching that could cause cross-request contamination
   955
   956	        Args:
   957	            speaker: Speaker name (must match first chunk)
   958	            text: Continuation text chunk to synthesize
   959	            global_ids: 32 global token IDs captured from first chunk
   960	            temperature: Sampling temperature
   961	            top_k: Top-k sampling
   962	            top_p: Nucleus sampling
   963	            max_tokens: Max tokens to generate
   964	            repetition_penalty: Prevent repetition
   965	            seed: Random seed (use same as first chunk for consistency)
   966
   967	        Yields:
   968	            Audio bytes (int16 PCM, 16kHz mono)
   969
   970	        Raises:
   971	            ValueError: If global_ids doesn't contain exactly 32 tokens
   972
   973	        Thread Safety:
   974	            Fully thread-safe. All state is request-scoped and passed explicitly.
   975	        """
   976	        import time
   977	        t_start = time.time()
   978
   979	        # Validate global tokens
   980	        if len(global_ids) != _STREAM_EXPECTED_GLOBAL_COUNT:
   981	            raise ValueError(
   982	                f"Expected exactly {_STREAM_EXPECTED_GLOBAL_COUNT} global tokens, got {len(global_ids)}. "
   983	                f"global_ids must be captured from first chunk generation."
   984	            )
   985
   986	        # Build prompt WITH pre-filled global tokens
   987	        # This tells the model to skip global generation and go straight to semantic tokens
   988	        prompt = self.prompt_builder.build_prefix_with_globals(speaker, text, global_ids)
   989
   990	        # Configure sampling
   991	        # OPTIMIZATION: Use Spark TTS stop token, not legacy SNAC CODE_END_TOKEN_ID
   992	        sampling_params = SamplingParams(
   993	            temperature=temperature,
   994	            top_k=top_k,
   995	            top_p=top_p,
   996	            max_tokens=max_tokens,
   997	            min_tokens=DEFAULT_MIN_TOKENS,
   998	            repetition_penalty=repetition_penalty,
   999	            stop=TRAINING_STOP_TOKEN_IDS,  # "<|im_end|>" - matches non-streaming pipeline
  1000	            skip_special_tokens=False,
  1001	            seed=seed,
  1002	            output_kind=(RequestOutputKind.DELTA if _STREAM_USE_DELTA_OUTPUT else RequestOutputKind.CUMULATIVE),
  1003	        )
  1004
  1005	        # BiCodec token buffers
  1006	        semantic_buffer = []
  1007	        global_buffer_unused = []  # Not used in continuation, but needed for parser API
  1008	        processed_token_count = 0  # Used for cumulative mode; ignored for delta mode.
  1009	        total_audio_chunks = 0
  1010	        total_generated_tokens = 0
  1011
  1012	        # BiCodec streaming configuration
  1013	        MIN_SEMANTIC_FOR_FIRST_CHUNK = _STREAM_MIN_SEMANTIC_FIRST_CHUNK
  1014	        DECODE_INTERVAL = _STREAM_DECODE_INTERVAL
  1015	        CROSSFADE_MS = _STREAM_CROSSFADE_MS
  1016	        WINDOW_SIZE = _STREAM_WINDOW_SIZE
  1017
  1018	        # Generate unique request ID
  1019	        import uuid
  1020	        request_id = f"bicodec-cont-{uuid.uuid4().hex[:8]}-{int(time.time() * 1000000)}"
  1021
  1022	        # Reuse pre-warmed token parser (singleton, created once in __init__)
  1023	        token_parser = self.token_parser
  1024
  1025	        results_generator = self.model.engine.generate(
  1026	            prompt=prompt,
  1027	            sampling_params=sampling_params,
  1028	            request_id=request_id,
  1029	        )
  1030
  1031	        # Track state
  1032	        last_decode_count = 0
  1033	        total_samples_emitted_to_user = 0
  1034	        previous_chunk_tail: Optional[bytes] = None
  1035	        t_first_chunk_emitted: Optional[float] = None
  1036	        decode_calls = 0
  1037	        decode_wall_ms_total = 0.0
  1038
  1039	        # Stream tokens - model will generate ONLY semantic tokens (globals are pre-filled)
  1040	        async for request_output in results_generator:
  1041	            generated_ids = request_output.outputs[0].token_ids
  1042
  1043	            # In DELTA mode token_ids are already incremental.
  1044	            if _STREAM_USE_DELTA_OUTPUT:
  1045	                new_token_ids = generated_ids
  1046	                processed_token_count += len(new_token_ids)
  1047	            else:
  1048	                new_token_ids = generated_ids[processed_token_count:]
  1049	                processed_token_count = len(generated_ids)
  1050	            total_generated_tokens += len(new_token_ids)
  1051	            token_parser.parse_incremental(new_token_ids, semantic_buffer, global_buffer_unused)
  1052
  1053	            semantic_count = len(semantic_buffer)
  1054	            if semantic_count >= MIN_SEMANTIC_FOR_FIRST_CHUNK:
  1055	                if semantic_count - last_decode_count >= DECODE_INTERVAL:
  1056	                    last_decode_count = semantic_count
  1057	                    if _STREAM_WINDOWED_DECODE and semantic_count > WINDOW_SIZE:
  1058	                        decode_semantic = semantic_buffer[-WINDOW_SIZE:]
  1059	                    else:
  1060	                        decode_semantic = semantic_buffer
  1061
  1062	                    # Use pre-captured global tokens for decoding
  1063	                    decode_t0 = time.perf_counter()
  1064	                    if self.snac_decoder.enable_batching:
  1065	                        audio_bytes = await self.snac_decoder.decode_single_async(
  1066	                            semantic_ids=decode_semantic,
  1067	                            global_ids=global_ids,  # Use captured globals
  1068	                            trim_warmup=False,
  1069	                            use_sliding_window=False
  1070	                        )
  1071	                    else:
  1072	                        audio_bytes = self.snac_decoder.decode_streaming(
  1073	                            semantic_ids=decode_semantic,
  1074	                            global_ids=global_ids,  # Use captured globals
  1075	                            use_sliding_window=False,
  1076	                            trim_warmup=False
  1077	                        )
  1078	                    decode_calls += 1
  1079	                    decode_wall_ms_total += (time.perf_counter() - decode_t0) * 1000.0
  1080
  1081	                    if audio_bytes:
  1082	                        total_samples_decoded = len(audio_bytes) // 2
  1083	                        if _STREAM_WINDOWED_DECODE and semantic_count > WINDOW_SIZE:
  1084	                            new_samples_approx = DECODE_INTERVAL * 320
  1085	                            if total_samples_decoded > new_samples_approx:
  1086	                                new_bytes_start = (total_samples_decoded - new_samples_approx) * 2
  1087	                            else:
  1088	                                new_bytes_start = 0
  1089	                            new_audio_bytes = audio_bytes[new_bytes_start:]
  1090	                        elif total_samples_decoded > total_samples_emitted_to_user:
  1091	                            new_bytes_start = total_samples_emitted_to_user * 2
  1092	                            new_audio_bytes = audio_bytes[new_bytes_start:]
  1093	                        else:
  1094	                            new_audio_bytes = b""
  1095
  1096	                        if new_audio_bytes:
  1097	                            to_emit, previous_chunk_tail = crossfade_bytes_int16(
  1098	                                previous_chunk_tail,
  1099	                                new_audio_bytes,
  1100	                                sample_rate_hz=16000,
  1101	                                crossfade_ms=CROSSFADE_MS,
  1102	                            )
  1103
  1104	                            if to_emit:
  1105	                                total_audio_chunks += 1
  1106	                                total_samples_emitted_to_user += len(to_emit) // 2
  1107	                                if t_first_chunk_emitted is None:
  1108	                                    t_first_chunk_emitted = time.perf_counter()
  1109	                                if emit_progress:
  1110	                                    yield (to_emit, {})
  1111	                                else:
  1112	                                    yield to_emit
  1113
  1114	        # Final chunk processing
  1115	        if len(semantic_buffer) > last_decode_count:
  1116	            decode_t0 = time.perf_counter()
  1117	            audio_bytes = self.snac_decoder.decode_streaming(
  1118	                semantic_ids=semantic_buffer,
  1119	                global_ids=global_ids,  # Use captured globals
  1120	                use_sliding_window=False,
  1121	                trim_warmup=False
  1122	            )
  1123	            decode_calls += 1
  1124	            decode_wall_ms_total += (time.perf_counter() - decode_t0) * 1000.0
  1125	            if audio_bytes:
  1126	                total_samples_decoded = len(audio_bytes) // 2
  1127
  1128	                if total_samples_decoded > total_samples_emitted_to_user:
  1129	                    new_bytes_start = total_samples_emitted_to_user * 2
  1130	                    new_audio_bytes = audio_bytes[new_bytes_start:]


**Planning final metrics extraction**


cd /home/ubuntu/veenaModal && nl -ba veena3modal/core/streaming_pipeline.py | sed -n '1130,1210p'


Output:
Chunk ID: e032d6
Wall time: 0.0513 seconds
Process exited with code 0
Original token count: 821
Output:
  1130	                    new_audio_bytes = audio_bytes[new_bytes_start:]
  1131
  1132	                    to_emit, previous_chunk_tail = crossfade_bytes_int16(
  1133	                        previous_chunk_tail,
  1134	                        new_audio_bytes,
  1135	                        sample_rate_hz=16000,
  1136	                        crossfade_ms=CROSSFADE_MS,
  1137	                    )
  1138
  1139	                    samples_emitted_in_final = len(to_emit) // 2
  1140	                    total_samples_emitted_to_user += samples_emitted_in_final
  1141
  1142	                    if to_emit:
  1143	                        total_audio_chunks += 1
  1144	                        total_samples_emitted_to_user += len(to_emit) // 2
  1145	                        if t_first_chunk_emitted is None:
  1146	                            t_first_chunk_emitted = time.perf_counter()
  1147	                        if emit_progress:
  1148	                            yield (to_emit, {})
  1149	                        else:
  1150	                            yield to_emit
  1151
  1152	        # Flush remaining tail
  1153	        if previous_chunk_tail:
  1154	            total_audio_chunks += 1
  1155	            tail_samples = len(previous_chunk_tail) // 2
  1156	            total_samples_emitted_to_user += tail_samples
  1157	            if t_first_chunk_emitted is None:
  1158	                t_first_chunk_emitted = time.perf_counter()
  1159	            if emit_progress:
  1160	                yield (previous_chunk_tail, {})
  1161	            else:
  1162	                yield previous_chunk_tail
  1163
  1164	        t_end = time.time()
  1165	        audio_duration_s = total_samples_emitted_to_user / 16000
  1166	        logger.debug(
  1167	            "continuation complete audio=%.2fs chunks=%d",
  1168	            audio_duration_s,
  1169	            total_audio_chunks,
  1170	        )
  1171
  1172	        if emit_progress:
  1173	            perf: Dict[str, Any] = {
  1174	                "llm_token_total": int(total_generated_tokens),
  1175	                "semantic_token_total": int(len(semantic_buffer)),
  1176	                "global_token_total": int(len(global_ids)),
  1177	                "bicodec_decode_calls": int(decode_calls),
  1178	                "llm_decode_calls": int(decode_calls),
  1179	                "bicodec_decode_wall_ms": float(decode_wall_ms_total),
  1180	                "bicodec_decode_gpu_ms": 0.0,
  1181	                "bicodec_decode_cpu_ms": float(decode_wall_ms_total),
  1182	                "chunks_sent": int(total_audio_chunks),
  1183	                "audio_duration_seconds": float(audio_duration_s),
  1184	                "ttfb_ms": float((t_first_chunk_emitted - t_start) * 1000.0 if t_first_chunk_emitted else 0.0),
  1185	                "stream_output_kind": "delta" if _STREAM_USE_DELTA_OUTPUT else "cumulative",
  1186	                "stream_decode_interval_tokens": int(DECODE_INTERVAL),
  1187	                "stream_window_size_tokens": int(WINDOW_SIZE),
  1188	                "stream_windowed_decode": bool(_STREAM_WINDOWED_DECODE),
  1189	            }
  1190	            yield (b"", perf)


**Fixing duplicate sample count addition**


Input:
*** Begin Patch
*** Update File: /home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py
@@
-                    samples_emitted_in_final = len(to_emit) // 2
-                    total_samples_emitted_to_user += samples_emitted_in_final
-
                     if to_emit:
                         total_audio_chunks += 1
                         total_samples_emitted_to_user += len(to_emit) // 2
*** End Patch


Output: {"output":"Success. Updated the following files:\nM /home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


**Recompiling after patch**


cd /home/ubuntu/veenaModal && python3 -m py_compile veena3modal/core/streaming_pipeline.py veena3modal/services/tts_runtime.py scripts/stress_test_streaming_runtime.py


Output:
Chunk ID: 832768
Wall time: 0.0513 seconds
Process exited with code 0
Original token count: 0
Output:


**Extracting line numbers for references**


cd /home/ubuntu/veenaModal && nl -ba veena3modal/services/tts_runtime.py | sed -n '770,870p'


cd /home/ubuntu/veenaModal && nl -ba scripts/stress_test_streaming_runtime.py | sed -n '180,390p'


cd /home/ubuntu/veenaModal && nl -ba scripts/stress_test_streaming_runtime.py | sed -n '390,560p'


Output:
Chunk ID: 28da38
Wall time: 0.0512 seconds
Process exited with code 0
Original token count: 1128
Output:
   770
   771	            total_pcm_bytes += len(audio_chunk) - (44 if metrics["chunks_sent"] == 0 else 0)
   772	            metrics["chunks_sent"] += 1
   773	            metrics["total_bytes"] = total_pcm_bytes + 44  # Include header
   774	            metrics["audio_duration_seconds"] = total_pcm_bytes / (sample_rate * 2)
   775
   776	            yield audio_chunk, metrics
   777	    else:
   778	        # Simple streaming (no text chunking)
   779	        # OPTIMIZATION: If we have pre-cached global tokens for this speaker,
   780	        # use continuation mode to skip the ~110ms global token pre-roll.
   781	        # The model jumps straight to semantic token generation.
   782	        cached_globals = runtime.speaker_global_cache.get(speaker)
   783
   784	        if cached_globals:
   785	            # Fast path: use cached globals, skip global token generation entirely
   786	            stream_gen = runtime.streaming_pipeline.generate_speech_stream_indic_continuation(
   787	                speaker=speaker,
   788	                text=text,
   789	                global_ids=cached_globals,
   790	                temperature=temperature,
   791	                top_k=top_k,
   792	                top_p=top_p,
   793	                max_tokens=max_tokens,
   794	                repetition_penalty=repetition_penalty,
   795	                seed=seed,
   796	                emit_progress=True,
   797	            )
   798	        else:
   799	            # Fallback: no cached globals, generate them inline (original path)
   800	            stream_gen = runtime.streaming_pipeline.generate_speech_stream_indic(
   801	                speaker=speaker,
   802	                text=text,
   803	                temperature=temperature,
   804	                top_k=top_k,
   805	                top_p=top_p,
   806	                max_tokens=max_tokens,
   807	                repetition_penalty=repetition_penalty,
   808	                seed=seed,
   809	                emit_progress=True,
   810	            )
   811
   812	        async for stream_item in stream_gen:
   813	            stream_metrics: Dict[str, Any] = {}
   814	            audio_chunk = stream_item
   815	            if (
   816	                isinstance(stream_item, tuple)
   817	                and len(stream_item) == 2
   818	                and isinstance(stream_item[1], dict)
   819	            ):
   820	                audio_chunk = stream_item[0]
   821	                stream_metrics = stream_item[1]
   822
   823	            if stream_metrics:
   824	                metrics.update(stream_metrics)
   825
   826	            if not audio_chunk:
   827	                # Metrics-only event from streaming pipeline.
   828	                continue
   829
   830	            # First chunk: prepend WAV header
   831	            if first_chunk_time is None:
   832	                first_chunk_time = time.time()
   833	                metrics["ttfb_ms"] = int((first_chunk_time - start_time) * 1000)
   834	                wav_header = create_wav_header(sample_rate=sample_rate, data_size=0)
   835	                audio_chunk = wav_header + audio_chunk
   836
   837	            total_pcm_bytes += len(audio_chunk) - (44 if metrics["chunks_sent"] == 0 else 0)
   838	            metrics["chunks_sent"] += 1
   839	            metrics["total_bytes"] = total_pcm_bytes + 44
   840	            metrics["audio_duration_seconds"] = total_pcm_bytes / (sample_rate * 2)
   841
   842	            yield audio_chunk, metrics
   843
   844
   845	async def _stream_chunked_text(
   846	    runtime: TTSRuntime,
   847	    long_processor,
   848	    text: str,
   849	    speaker: str,
   850	    temperature: float,
   851	    top_k: int,
   852	    top_p: float,
   853	    max_tokens: int,
   854	    repetition_penalty: float,
   855	    seed: Optional[int],
   856	    sample_rate: int,
   857	) -> AsyncGenerator[Tuple[bytes, Dict[str, Any]], None]:
   858	    """
   859	    Internal helper: stream chunked text with voice consistency.
   860
   861	    Uses global token caching from first chunk to maintain voice across chunks.
   862	    """
   863	    from veena3modal.audio.crossfade import crossfade_bytes_int16
   864
   865	    # Chunk text using the long text processor's chunker
   866	    chunks = long_processor.chunk_text(text)
   867
   868	    if not chunks:
   869	        return
   870


Output:
Chunk ID: 52a71a
Wall time: 0.0503 seconds
Process exited with code 0
Original token count: 2288
Output:
   180	            "max": max(vals),
   181	        }
   182
   183	    def latency_stats(self) -> Dict[str, float]:
   184	        return self._stats([r.latency_ms for r in self.successes])
   185
   186	    def ttfb_stats(self) -> Dict[str, float]:
   187	        return self._stats([r.ttfb_ms for r in self.successes if r.ttfb_ms > 0])
   188
   189	    def chunk_count_stats(self) -> Dict[str, float]:
   190	        return self._stats([float(r.chunks_sent) for r in self.successes])
   191
   192	    def emit_speed_stats(self) -> Dict[str, float]:
   193	        return self._stats([r.tail_emit_speed_x for r in self.successes if r.tail_emit_speed_x > 0])
   194
   195	    def inter_chunk_stats(self) -> Dict[str, float]:
   196	        vals: List[float] = []
   197	        for r in self.successes:
   198	            vals.extend(r.inter_chunk_ms)
   199	        return self._stats(vals)
   200
   201	    def timing_stats(self, key: str) -> Dict[str, float]:
   202	        vals: List[float] = []
   203	        for r in self.successes:
   204	            v = r.final_metrics.get(key)
   205	            if isinstance(v, (int, float)):
   206	                vals.append(float(v))
   207	        return self._stats(vals)
   208
   209	    def latency_over_metric_stats(self, key: str) -> Dict[str, float]:
   210	        vals: List[float] = []
   211	        for r in self.successes:
   212	            base = r.final_metrics.get(key)
   213	            if isinstance(base, (int, float)):
   214	                vals.append(max(0.0, float(r.latency_ms) - float(base)))
   215	        return self._stats(vals)
   216
   217
   218	def init_runtime(
   219	    model_path: str,
   220	    gpu_mem: float,
   221	    stream_output_kind: str,
   222	    num_engines: int,
   223	    max_num_batched_tokens: Optional[int],
   224	    max_num_seqs: Optional[int],
   225	    enable_chunked_prefill: Optional[bool],
   226	    enable_prefix_caching: Optional[bool],
   227	    disable_log_stats: Optional[bool],
   228	    enforce_eager: Optional[bool],
   229	) -> None:
   230	    from veena3modal.local_server import resolve_model_paths
   231	    from veena3modal.services.tts_runtime import initialize_runtime, is_initialized
   232
   233	    os.environ["VEENA3_STREAM_OUTPUT_KIND"] = stream_output_kind
   234
   235	    llm_path, bicodec_path = resolve_model_paths(model_path)
   236	    if not is_initialized():
   237	        initialize_runtime(
   238	            model_path=llm_path,
   239	            bicodec_path=bicodec_path,
   240	            device="cuda",
   241	            gpu_memory_utilization=gpu_mem,
   242	            num_engines=num_engines,
   243	            max_num_batched_tokens=max_num_batched_tokens,
   244	            max_num_seqs=max_num_seqs,
   245	            enable_chunked_prefill=enable_chunked_prefill,
   246	            enable_prefix_caching=enable_prefix_caching,
   247	            disable_log_stats=disable_log_stats,
   248	            enforce_eager=enforce_eager,
   249	            precompute_speaker_globals=False,
   250	        )
   251
   252
   253	async def run_one_request(
   254	    text: str,
   255	    speaker: str,
   256	    max_tokens: int,
   257	    timeout_s: float,
   258	    enable_chunking: bool,
   259	) -> RequestResult:
   260	    from veena3modal.services import tts_runtime
   261
   262	    t0 = time.perf_counter()
   263	    first_chunk_t: Optional[float] = None
   264	    prev_chunk_t: Optional[float] = None
   265	    inter_chunk_ms: List[float] = []
   266	    chunks = 0
   267	    total_bytes = 0
   268	    final_metrics: Dict[str, Any] = {}
   269
   270	    async def _consume_stream() -> None:
   271	        nonlocal first_chunk_t, prev_chunk_t, chunks, total_bytes, final_metrics
   272	        gen = tts_runtime.generate_speech_streaming(
   273	            text=text,
   274	            speaker=speaker,
   275	            max_tokens=max_tokens,
   276	            enable_chunking=enable_chunking,
   277	        )
   278	        async for audio_chunk, metrics in gen:
   279	            now = time.perf_counter()
   280	            if first_chunk_t is None:
   281	                first_chunk_t = now
   282	            if prev_chunk_t is not None:
   283	                inter_chunk_ms.append((now - prev_chunk_t) * 1000.0)
   284	            prev_chunk_t = now
   285	            chunks += 1
   286	            total_bytes += len(audio_chunk)
   287	            if isinstance(metrics, dict):
   288	                # Keep reference; runtime may mutate this dict after the final chunk
   289	                # when internal stream generation completes.
   290	                final_metrics = metrics
   291
   292	    try:
   293	        await asyncio.wait_for(_consume_stream(), timeout=timeout_s)
   294	        end = time.perf_counter()
   295	        latency_ms = (end - t0) * 1000.0
   296	        ttfb_ms = ((first_chunk_t - t0) * 1000.0) if first_chunk_t is not None else 0.0
   297	        audio_seconds = float(final_metrics.get("audio_duration_seconds", 0.0) or 0.0)
   298	        tail_s = max(1e-6, (latency_ms - ttfb_ms) / 1000.0)
   299	        tail_emit_speed_x = audio_seconds / tail_s if audio_seconds > 0 else 0.0
   300	        return RequestResult(
   301	            success=True,
   302	            latency_ms=latency_ms,
   303	            ttfb_ms=ttfb_ms,
   304	            chunks_sent=chunks,
   305	            total_bytes=total_bytes,
   306	            audio_seconds=audio_seconds,
   307	            speaker=speaker,
   308	            inter_chunk_ms=inter_chunk_ms,
   309	            tail_emit_speed_x=tail_emit_speed_x,
   310	            final_metrics=dict(final_metrics),
   311	        )
   312	    except asyncio.TimeoutError:
   313	        return RequestResult(
   314	            success=False,
   315	            latency_ms=(time.perf_counter() - t0) * 1000.0,
   316	            ttfb_ms=0.0,
   317	            chunks_sent=chunks,
   318	            total_bytes=total_bytes,
   319	            audio_seconds=0.0,
   320	            speaker=speaker,
   321	            error="timeout",
   322	            inter_chunk_ms=inter_chunk_ms,
   323	            final_metrics=dict(final_metrics),
   324	        )
   325	    except Exception as exc:
   326	        return RequestResult(
   327	            success=False,
   328	            latency_ms=(time.perf_counter() - t0) * 1000.0,
   329	            ttfb_ms=0.0,
   330	            chunks_sent=chunks,
   331	            total_bytes=total_bytes,
   332	            audio_seconds=0.0,
   333	            speaker=speaker,
   334	            error=str(exc)[:200],
   335	            inter_chunk_ms=inter_chunk_ms,
   336	            final_metrics=dict(final_metrics),
   337	        )
   338
   339
   340	async def run_level(
   341	    concurrency: int,
   342	    total_requests: int,
   343	    text: str,
   344	    max_tokens: int,
   345	    timeout_s: float,
   346	    enable_chunking: bool,
   347	    gpu_monitor: GPUMonitor,
   348	) -> LevelResult:
   349	    level = LevelResult(concurrency=concurrency, total_requests=total_requests)
   350	    semaphore = asyncio.Semaphore(concurrency)
   351
   352	    async def wrapped(i: int) -> RequestResult:
   353	        async with semaphore:
   354	            speaker = SPEAKERS[i % len(SPEAKERS)]
   355	            return await run_one_request(
   356	                text=text,
   357	                speaker=speaker,
   358	                max_tokens=max_tokens,
   359	                timeout_s=timeout_s,
   360	                enable_chunking=enable_chunking,
   361	            )
   362
   363	    gpu_monitor.snapshots.clear()
   364	    t0 = time.perf_counter()
   365	    level.results = await asyncio.gather(*[wrapped(i) for i in range(total_requests)])
   366	    level.wall_time_s = time.perf_counter() - t0
   367	    level.gpu_summary = gpu_monitor.summary()
   368	    return level
   369
   370
   371	def print_level(level: LevelResult) -> None:
   372	    print(f"\n=== streaming concurrency={level.concurrency} requests={level.total_requests} ===")
   373	    print(f"success={len(level.successes)}/{level.total_requests} ({level.success_rate:.0%})")
   374	    print(f"wall={level.wall_time_s:.2f}s throughput={level.throughput_rps:.2f} req/s")
   375	    lat = level.latency_stats()
   376	    if lat:
   377	        print(f"latency_ms avg={lat['avg']:.0f} p50={lat['p50']:.0f} p95={lat['p95']:.0f} max={lat['max']:.0f}")
   378	    ttfb = level.ttfb_stats()
   379	    if ttfb:
   380	        print(f"ttfb_ms avg={ttfb['avg']:.0f} p50={ttfb['p50']:.0f} p95={ttfb['p95']:.0f} max={ttfb['max']:.0f}")
   381	    cc = level.chunk_count_stats()
   382	    if cc:
   383	        print(f"chunks_per_req avg={cc['avg']:.1f} p50={cc['p50']:.1f} p95={cc['p95']:.1f} max={cc['max']:.1f}")
   384	    ic = level.inter_chunk_stats()
   385	    if ic:
   386	        print(
   387	            "inter_chunk_ms avg={:.1f} p50={:.1f} p95={:.1f} max={:.1f}".format(
   388	                ic["avg"], ic["p50"], ic["p95"], ic["max"]
   389	            )
   390	        )


Output:
Chunk ID: bbaa52
Wall time: 0.0517 seconds
Process exited with code 0
Original token count: 2166
Output:
   390	        )
   391	    es = level.emit_speed_stats()
   392	    if es:
   393	        print(
   394	            "tail_emit_speed_x avg={:.2f} p50={:.2f} p95={:.2f} max={:.2f}".format(
   395	                es["avg"], es["p50"], es["p95"], es["max"]
   396	            )
   397	        )
   398
   399	    timing_keys = [
   400	        "timeline_to_first_batch_ms",
   401	        "timeline_to_first_chunk_emitted_ms",
   402	        "timeline_total_ms",
   403	        "llm_time_in_queue_ms",
   404	        "llm_scheduler_ms",
   405	        "llm_first_token_ms",
   406	        "llm_request_lifecycle_ms",
   407	        "llm_generation_wall_ms",
   408	        "llm_time_per_token_ms",
   409	        "llm_batch_wall_ms_total",
   410	        "llm_decode_wall_ms_total",
   411	        "tokens_per_batch_p50",
   412	        "bicodec_decode_calls",
   413	        "bicodec_decode_wall_ms_total",
   414	        "bicodec_decode_interval_ms_p50",
   415	    ]
   416	    for key in timing_keys:
   417	        st = level.timing_stats(key)
   418	        if st:
   419	            print(f"{key}: avg={st['avg']:.2f} p50={st['p50']:.2f} p95={st['p95']:.2f} max={st['max']:.2f}")
   420
   421	    latency_over_timeline = level.latency_over_metric_stats("timeline_total_ms")
   422	    if latency_over_timeline:
   423	        print(
   424	            "latency_over_timeline_ms: avg={:.2f} p50={:.2f} p95={:.2f} max={:.2f}".format(
   425	                latency_over_timeline["avg"],
   426	                latency_over_timeline["p50"],
   427	                latency_over_timeline["p95"],
   428	                latency_over_timeline["max"],
   429	            )
   430	        )
   431
   432	    tl_total = level.timing_stats("timeline_total_ms")
   433	    llm_total = level.timing_stats("llm_generation_wall_ms")
   434	    bicodec_total = level.timing_stats("bicodec_decode_wall_ms_total")
   435	    if tl_total and llm_total:
   436	        total_avg = tl_total.get("avg", 0.0)
   437	        llm_avg = llm_total.get("avg", 0.0)
   438	        bicodec_avg = bicodec_total.get("avg", 0.0) if bicodec_total else 0.0
   439	        other_avg = max(0.0, total_avg - llm_avg - bicodec_avg)
   440	        if total_avg > 0:
   441	            print(
   442	                "timeline_share: llm={:.1f}% bicodec={:.1f}% other={:.1f}%".format(
   443	                    (llm_avg / total_avg) * 100,
   444	                    (bicodec_avg / total_avg) * 100,
   445	                    (other_avg / total_avg) * 100,
   446	                )
   447	            )
   448
   449	    if "memory_mb" in level.gpu_summary:
   450	        g = level.gpu_summary
   451	        print(
   452	            "gpu: util_avg={:.0f}% util_max={:.0f}% mem_avg={:.0f}MB mem_max={:.0f}MB".format(
   453	                g["gpu_util_pct"]["avg"],
   454	                g["gpu_util_pct"]["max"],
   455	                g["memory_mb"]["avg"],
   456	                g["memory_mb"]["max"],
   457	            )
   458	        )
   459
   460
   461	def save_results(path: str, levels: List[LevelResult], args: argparse.Namespace) -> None:
   462	    payload = {
   463	        "config": {
   464	            "levels": args.levels,
   465	            "text": args.text,
   466	            "max_tokens": args.max_tokens,
   467	            "timeout_s": args.timeout,
   468	            "requests_multiplier": args.requests_multiplier,
   469	            "min_requests": args.min_requests,
   470	            "gpu_memory": args.gpu_memory,
   471	            "stream_output_kind": args.stream_output_kind,
   472	            "chunking": args.chunking,
   473	            "num_engines": args.num_engines,
   474	            "max_num_batched_tokens": args.max_num_batched_tokens,
   475	            "max_num_seqs": args.max_num_seqs,
   476	            "enable_chunked_prefill": args.enable_chunked_prefill,
   477	            "disable_chunked_prefill": args.disable_chunked_prefill,
   478	            "disable_prefix_caching": args.disable_prefix_caching,
   479	            "disable_engine_stats_logs": args.disable_engine_stats_logs,
   480	            "enforce_eager": args.enforce_eager,
   481	            "stream_decode_interval": args.stream_decode_interval,
   482	            "stream_window_size": args.stream_window_size,
   483	            "stream_min_semantic_first": args.stream_min_semantic_first,
   484	            "stream_crossfade_ms": args.stream_crossfade_ms,
   485	            "disable_windowed_decode": args.disable_windowed_decode,
   486	            "timestamp": int(time.time()),
   487	        },
   488	        "levels": [],
   489	    }
   490	    for lvl in levels:
   491	        payload["levels"].append(
   492	            {
   493	                "concurrency": lvl.concurrency,
   494	                "total_requests": lvl.total_requests,
   495	                "success_rate": lvl.success_rate,
   496	                "throughput_rps": lvl.throughput_rps,
   497	                "wall_time_s": lvl.wall_time_s,
   498	                "latency_ms": lvl.latency_stats(),
   499	                "ttfb_ms": lvl.ttfb_stats(),
   500	                "chunks_per_req": lvl.chunk_count_stats(),
   501	                "inter_chunk_ms": lvl.inter_chunk_stats(),
   502	                "tail_emit_speed_x": lvl.emit_speed_stats(),
   503	                "gpu": lvl.gpu_summary,
   504	                "timing": {
   505	                    "timeline_to_first_batch_ms": lvl.timing_stats("timeline_to_first_batch_ms"),
   506	                    "timeline_to_first_chunk_emitted_ms": lvl.timing_stats("timeline_to_first_chunk_emitted_ms"),
   507	                    "timeline_total_ms": lvl.timing_stats("timeline_total_ms"),
   508	                    "llm_time_in_queue_ms": lvl.timing_stats("llm_time_in_queue_ms"),
   509	                    "llm_scheduler_ms": lvl.timing_stats("llm_scheduler_ms"),
   510	                    "llm_first_token_ms": lvl.timing_stats("llm_first_token_ms"),
   511	                    "llm_request_lifecycle_ms": lvl.timing_stats("llm_request_lifecycle_ms"),
   512	                    "llm_generation_wall_ms": lvl.timing_stats("llm_generation_wall_ms"),
   513	                    "llm_time_per_token_ms": lvl.timing_stats("llm_time_per_token_ms"),
   514	                    "llm_batch_wall_ms_total": lvl.timing_stats("llm_batch_wall_ms_total"),
   515	                    "llm_decode_wall_ms_total": lvl.timing_stats("llm_decode_wall_ms_total"),
   516	                    "tokens_per_batch_p50": lvl.timing_stats("tokens_per_batch_p50"),
   517	                    "bicodec_decode_calls": lvl.timing_stats("bicodec_decode_calls"),
   518	                    "bicodec_decode_wall_ms_total": lvl.timing_stats("bicodec_decode_wall_ms_total"),
   519	                    "bicodec_decode_interval_ms_p50": lvl.timing_stats("bicodec_decode_interval_ms_p50"),
   520	                    "latency_over_timeline_ms": lvl.latency_over_metric_stats("timeline_total_ms"),
   521	                },
   522	                "requests": [
   523	                    {
   524	                        "success": r.success,
   525	                        "latency_ms": r.latency_ms,
   526	                        "ttfb_ms": r.ttfb_ms,
   527	                        "chunks_sent": r.chunks_sent,
   528	                        "total_bytes": r.total_bytes,
   529	                        "audio_seconds": r.audio_seconds,
   530	                        "speaker": r.speaker,
   531	                        "error": r.error,
   532	                        "inter_chunk_ms": r.inter_chunk_ms,
   533	                        "tail_emit_speed_x": r.tail_emit_speed_x,
   534	                        "final_metrics": r.final_metrics,
   535	                    }
   536	                    for r in lvl.results
   537	                ],
   538	            }
   539	        )
   540	    with open(path, "w", encoding="utf-8") as fd:
   541	        json.dump(payload, fd, indent=2)
   542
   543
   544	async def main_async(args: argparse.Namespace) -> None:
   545	    text = TEST_TEXTS[args.text]
   546	    levels = [int(x.strip()) for x in args.levels.split(",") if x.strip()]
   547	    gpu_monitor = GPUMonitor(interval_seconds=0.5)
   548	    gpu_monitor.start()
   549
   550	    # warmup
   551	    await run_one_request(
   552	        text=text,
   553	        speaker=SPEAKERS[0],
   554	        max_tokens=args.max_tokens,
   555	        timeout_s=args.timeout,
   556	        enable_chunking=args.chunking,
   557	    )
   558
   559	    results: List[LevelResult] = []
   560	    for idx, c in enumerate(levels):


Summary: auto