Output:
Chunk ID: ce5a01
Wall time: 5.0012 seconds
Process running with session ID 38524
Original token count: 1579
Output:
Total output lines: 58

The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
🚀 Initializing Spark TTS Model
📁 Model path: /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
🗣  Speakers: 12 predefined (lipakshi, vardan, reet, Nandini, krishna, anika, adarsh, Nilay, Aarvi, Asha, Bittu, Mira)
🎭 Emotions: 10 tags in [bracket] format

📝 Loading tokenizer from model...
✅ Tokenizer loaded: 165168 tokens

🔍 Validating 12 speakers for Spark TTS...
  lipakshi → <|speaker_0|>: 7 tokens
  vardan → <|speaker_1|>: 7 tokens
  reet → <|speaker_2|>: 7 tokens
  Nandini → <|speaker_3|>: 7 tokens
  krishna → <|speaker_4|>: 7 tokens
  anika → <|speaker_5|>: 7 tokens
  adarsh → <|speaker_6|>: 7 tokens
  Nilay → <|speaker_7|>: 7 tokens
  Aarvi → <|speaker_8|>: 7 tokens
  Asha → <|speaker_9|>: 7 tokens
  Bittu → <|speaker_10|>: 8 tokens
  Mira → <|speaker_11|>: 8 tokens
✅ All 12 speakers validated with mapping

🔍 Validating 10 emotion tags for Spark TTS...
✅ All 10 emotion tags validated

📋 Sample emotion tags:
  [angry]: 1 tokens
  [curious]: 1 tokens
  [excited]: 1 tokens
  [giggle]: 1 tokens
  [laughs harder]: 1 tokens
  ...

🔧 Initializing vLLM engine with optimizations...
  - dtype: bfloat16
  - max_model_len: 4096
  - gpu_memory_utilization: 0.25
  - enable_prefix_caching: True
  - enforce_eager (disable CUDA graphs): False
INFO 02-17 00:16:52 [model.py:541] Resolved architecture: Qwen2ForCausalLM
INFO 02-17 00:16:52 [model.py:1561] Using max model len 4096
INFO 02-17 00:16:53 [scheduler.py:226] Chunked prefill is enabled with max_num_batched_tokens=4096.
INFO 02-17 00:16:53 [vllm.py:624] Asynchronous scheduling is enabled.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(EngineCore_DP0 pid=3699871) INFO 02-17 00:16:54 [core.py:96] Initializing a V1 LLM engine (v0.15.1) with config: model='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', speculative_config=None, tokenizer='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_par…379 tokens truncated…', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [4096], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 256, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': True}, 'local_cache_dir': None, 'static_all_moe_layers': []}
(EngineCore_DP0 pid=3699871) INFO 02-17 00:16:56 [parallel_state.py:1212] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://216.81.248.184:55075 backend=nccl
(EngineCore_DP0 pid=3699871) INFO 02-17 00:16:56 [parallel_state.py:1423] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
(EngineCore_DP0 pid=3699871) INFO 02-17 00:16:57 [gpu_model_runner.py:4033] Starting to load model /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM...
(EngineCore_DP0 pid=3699871) INFO 02-17 00:16:57 [cuda.py:364] Using FLASH_ATTN attention backend out of potential backends: ('FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION')
(EngineCore_DP0 pid=3699871)
Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
(EngineCore_DP0 pid=3699871)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:01<00:00,  1.98s/it]
(EngineCore_DP0 pid=3699871)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:01<00:00,  1.98s/it]
(EngineCore_DP0 pid=3699871)
(EngineCore_DP0 pid=3699871) INFO 02-17 00:17:00 [default_loader.py:291] Loading weights took 2.08 seconds
(EngineCore_DP0 pid=3699871) INFO 02-17 00:17:00 [gpu_model_runner.py:4130] Model loading took 0.95 GiB memory and 2.662055 seconds


# Collaboration Mode: Default

You are now in Default mode. Any previous instructions for other modes (e.g. Plan mode) are no longer active.

Your active mode changes only when new developer instructions with a different `<collaboration_mode>...</collaboration_mode>` change it; user requests or tool descriptions do not change mode by themselves. Known mode names are Default and Plan.

## request_user_input availability

The `request_user_input` tool is unavailable in Default mode. If you call it while in Default mode, it will return an error.

If a decision is necessary and cannot be discovered from local context, ask the user directly. However, in Default mode you should strongly prefer executing the user's request rather than stopping to ask questions.

Summary: auto

## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) When `SKILL.md` references relative paths (e.g., `scripts/foo.py`), resolve them relative to the skill directory listed above first, and only consider other paths if needed.
  3) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  4) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  5) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.


Output:
Chunk ID: b39c18
Wall time: 5.0066 seconds
Process running with session ID 38524
Original token count: 74
Output:
(EngineCore_DP0 pid=3699871) INFO 02-17 00:17:05 [backends.py:812] Using cache directory: /home/ubuntu/.cache/vllm/torch_compile_cache/c3bd5d9fb9/rank_0_0/backbone for vLLM's torch.compile
(EngineCore_DP0 pid=3699871) INFO 02-17 00:17:05 [backends.py:872] Dynamo bytecode transform time: 4.67 s


Summary: auto


Output:
Chunk ID: 8b6395
Wall time: 5.0011 seconds
Process running with session ID 38524
Original token count: 688
Output:
(EngineCore_DP0 pid=3699871) INFO 02-17 00:17:08 [backends.py:267] Directly load the compiled graph(s) for compile range (1, 4096) from the cache, took 0.692 s
(EngineCore_DP0 pid=3699871) INFO 02-17 00:17:08 [monitor.py:34] torch.compile takes 5.36 s in total
(EngineCore_DP0 pid=3699871) INFO 02-17 00:17:09 [gpu_worker.py:356] Available KV cache memory: 18.06 GiB
(EngineCore_DP0 pid=3699871) INFO 02-17 00:17:09 [kv_cache_utils.py:1307] GPU KV cache size: 1,577,712 tokens
(EngineCore_DP0 pid=3699871) INFO 02-17 00:17:09 [kv_cache_utils.py:1312] Maximum concurrency for 4,096 tokens per request: 385.18x
(EngineCore_DP0 pid=3699871)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   0%|          | 0/35 [00:00<?, ?it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  11%|█▏        | 4/35 [00:00<00:00, 33.25it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  23%|██▎       | 8/35 [00:00<00:00, 30.37it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  34%|███▍      | 12/35 [00:00<00:00, 31.68it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  46%|████▌     | 16/35 [00:00<00:00, 33.77it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  57%|█████▋    | 20/35 [00:00<00:00, 32.10it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  69%|██████▊   | 24/35 [00:00<00:00, 32.60it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  80%|████████  | 28/35 [00:00<00:00, 33.82it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  91%|█████████▏| 32/35 [00:00<00:00, 32.74it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 35/35 [00:01<00:00, 31.63it/s]
(EngineCore_DP0 pid=3699871)
Capturing CUDA graphs (decode, FULL):   0%|          | 0/19 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL):  16%|█▌        | 3/19 [00:00<00:00, 29.48it/s]
Capturing CUDA graphs (decode, FULL):  37%|███▋      | 7/19 [00:00<00:00, 33.88it/s]
Capturing CUDA graphs (decode, FULL):  58%|█████▊    | 11/19 [00:00<00:00, 35.35it/s]
Capturing CUDA graphs (decode, FULL):  79%|███████▉  | 15/19 [00:00<00:00, 36.01it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 36.48it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 35.56it/s]
(EngineCore_DP0 pid=3699871) INFO 02-17 00:17:12 [gpu_model_runner.py:5063] Graph capturing finished in 2 secs, took 0.28 GiB
(EngineCore_DP0 pid=3699871) INFO 02-17 00:17:12 [core.py:272] init engine (profile, create kv cache, warmup model) took 11.44 seconds


Output:
Chunk ID: 1bcf18
Wall time: 5.0020 seconds
Process running with session ID 38524
Original token count: 112
Output:
2026-02-17 00:17:13 | INFO     | veena3modal.services.tts_runtime | Initializing prompt builder...
2026-02-17 00:17:13 | INFO     | veena3modal.services.tts_runtime | Initializing BiCodec decoder...
/home/ubuntu/veenaModal/venv/lib/python3.10/site-packages/torch/nn/utils/weight_norm.py:144: FutureWarning: `torch.nn.utils.weight_norm` is deprecated in favor of `torch.nn.utils.parametrizations.weight_norm`.
  WeightNorm.apply(module, name, dim)


Output:
Chunk ID: 1d701f
Wall time: 5.0029 seconds
Process running with session ID 38524
Original token count: 177
Output:
2026-02-17 00:17:24 | INFO     | veena3modal.services.tts_runtime | Initializing TTS pipeline...
2026-02-17 00:17:24 | INFO     | veena3modal.services.tts_runtime | Initializing streaming pipeline...
2026-02-17 00:17:24 | INFO     | veena3modal.services.tts_runtime | ✅ TTS runtime initialized in 39120ms
2026-02-17 00:17:24 | INFO     | veena3modal.services.tts_runtime |    Model version: LLM
2026-02-17 00:17:24 | INFO     | veena3modal.services.tts_runtime | Pre-computing global tokens for all speakers...
2026-02-17 00:17:25 | INFO     | veena3modal.services.tts_runtime |   Cached globals for lipakshi
2026-02-17 00:17:25 | INFO     | veena3modal.services.tts_runtime |   Cached globals for vardan


Output:
Chunk ID: deb885
Wall time: 5.0039 seconds
Process running with session ID 38524
Original token count: 341
Output:
2026-02-17 00:17:26 | INFO     | veena3modal.services.tts_runtime |   Cached globals for reet
2026-02-17 00:17:27 | INFO     | veena3modal.services.tts_runtime |   Cached globals for Nandini
2026-02-17 00:17:27 | INFO     | veena3modal.services.tts_runtime |   Cached globals for krishna
2026-02-17 00:17:28 | INFO     | veena3modal.services.tts_runtime |   Cached globals for anika
2026-02-17 00:17:29 | INFO     | veena3modal.services.tts_runtime |   Cached globals for adarsh
2026-02-17 00:17:29 | INFO     | veena3modal.services.tts_runtime |   Cached globals for Nilay
2026-02-17 00:17:30 | INFO     | veena3modal.services.tts_runtime |   Cached globals for Aarvi
2026-02-17 00:17:31 | INFO     | veena3modal.services.tts_runtime |   Cached globals for Asha
2026-02-17 00:17:31 | INFO     | veena3modal.services.tts_runtime |   Cached globals for Bittu
2026-02-17 00:17:32 | INFO     | veena3modal.services.tts_runtime |   Cached globals for Mira
2026-02-17 00:17:32 | INFO     | veena3modal.services.tts_runtime | Speaker globals cached: 12/12 speakers in 7839ms
2026-02-17 00:17:32 | INFO     | veena3.local | Runtime initialized in 47.0s
2026-02-17 00:17:32 | INFO     | veena3.local |   Model version: LLM
2026-02-17 00:17:32 | INFO     | veena3.local |   Streaming: available
2026-02-17 00:17:32 | INFO     | veena3.local |   Super-resolution: disabled


Output:
Chunk ID: c275fd
Wall time: 5.0010 seconds
Process running with session ID 38524
Original token count: 496
Output:
✅ vLLM engine initialized successfully!

🎉 Spark TTS Model ready for inference!

🎵 Loading BiCodec audio tokenizer from /home/ubuntu/veenaModal/models/spark_tts_4speaker...
   Device: cuda
Missing tensor: mel_transformer.spectrogram.window
Missing tensor: mel_transformer.mel_scale.fb
✅ BiCodec decoder initialized (sample rate: 16000Hz)
🚀 SparkTTSPipeline initialized
🌊 Veena3SlidingWindowPipeline initialized (sliding window: 28 tokens)
2026-02-17 00:17:32 | INFO     | veena3.local |
2026-02-17 00:17:32 | INFO     | veena3.local | ============================================================
2026-02-17 00:17:32 | INFO     | veena3.local | Veena3 TTS Local Server starting
2026-02-17 00:17:32 | INFO     | veena3.local |   URL: http://0.0.0.0:8000
2026-02-17 00:17:32 | INFO     | veena3.local |   Health: http://0.0.0.0:8000/v1/tts/health
2026-02-17 00:17:32 | INFO     | veena3.local |   Generate: POST http://0.0.0.0:8000/v1/tts/generate
2026-02-17 00:17:32 | INFO     | veena3.local |   Metrics: http://0.0.0.0:8000/v1/tts/metrics
2026-02-17 00:17:32 | INFO     | veena3.local |   Docs: http://0.0.0.0:8000/docs
2026-02-17 00:17:32 | INFO     | veena3.local | ============================================================
{"timestamp": "2026-02-17T00:17:32.529451+00:00", "level": "INFO", "logger": "veena3modal.api.websocket_handler", "message": "WebSocket route /v1/tts/ws registered"}
2026-02-17 00:17:32 | INFO     | veena3modal.api.websocket_handler | WebSocket route /v1/tts/ws registered
2026-02-17 00:17:32 | INFO     | veena3modal.api.fastapi_app | WebSocket support enabled at /v1/tts/ws
{"timestamp": "2026-02-17T00:17:32.529642+00:00", "level": "INFO", "logger": "veena3modal.api.fastapi_app", "message": "WebSocket support enabled at /v1/tts/ws"}
INFO:     Started server process [3699753]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)


cd /home/ubuntu/veenaModal && source venv/bin/activate && python3 scripts/stress_test_optimized_local.py --levels 1,5,10 --text short --output /home/ubuntu/veenaModal/stress_results_opt_local.json


Output:
Chunk ID: 64671f
Wall time: 1.0010 seconds
Process running with session ID 49061
Original token count: 0
Output:


Output:
Chunk ID: ad1c02
Wall time: 5.0011 seconds
Process running with session ID 49061
Original token count: 0
Output:


Output:
Chunk ID: fa720a
Wall time: 5.0012 seconds
Process running with session ID 49061
Original token count: 0
Output:


Output:
Chunk ID: fe982d
Wall time: 5.0003 seconds
Process running with session ID 49061
Original token count: 0
Output:


Output:
Chunk ID: e22352
Wall time: 5.0006 seconds
Process running with session ID 49061
Original token count: 0
Output:


Output:
Chunk ID: 1fc346
Wall time: 5.0009 seconds
Process running with session ID 49061
Original token count: 0
Output:


The benchmark command appears to be running but not producing output yet; I’m checking server-side logs to verify whether requests are being accepted and to capture the exact failure point if any.


Output:
Chunk ID: c42c80
Wall time: 5.0038 seconds
Process running with session ID 38524
Original token count: 156
Output:
INFO:     127.0.0.1:38140 - "GET /v1/tts/health HTTP/1.1" 200 OK
{"timestamp": "2026-02-17T00:17:41.222740+00:00", "level": "INFO", "logger": "veena3modal.tts", "message": "request_received", "request_id": "114bab3a-8576-4bba-8e1d-b67530ee5a7d", "event": "request_received", "text_length": 28, "speaker": "lipakshi", "stream": false, "format": "wav"}
2026-02-17 00:17:41 | INFO     | veena3modal.tts | request_received
2026-02-17 00:17:41 | INFO     | veena3modal.services.tts_runtime | tts_runtime_chunking_start
2026-02-17 00:17:41 | INFO     | veena3modal.processing.long_text_processor | LongTextProcessor initialized


Output:
Chunk ID: 478d69
Wall time: 5.0012 seconds
Process running with session ID 38524
Original token count: 0
Output:


Output:
Chunk ID: 141bf6
Wall time: 5.0005 seconds
Process running with session ID 38524
Original token count: 0
Output:


Output:
Chunk ID: 22b192
Wall time: 5.0006 seconds
Process running with session ID 49061
Original token count: 0
Output:


Output:
Chunk ID: ae65ed
Wall time: 5.0007 seconds
Process running with session ID 38524
Original token count: 0
Output:


cd /home/ubuntu/veenaModal && source venv/bin/activate && python3 - <<'PY'
import httpx
import time
payload={'text':'Hello, this is a quick test.','speaker':'lipakshi','stream':False,'chunking':True,'format':'wav'}
start=time.time()
try:
    r=httpx.post('http://127.0.0.1:8000/v1/tts/generate', json=payload, timeout=60)
    print('status',r.status_code,'ms', (time.time()-start)*1000)
    print('x-llm-token-total',r.headers.get('x-llm-token-total'))
    print('x-llm-batch-gpu-ms',r.headers.get('x-llm-batch-gpu-ms'))
    print('len',len(r.content))
    print(r.text[:120])
except Exception as e:
    print('err',e)
PY


Output:
Chunk ID: a0ddf3
Wall time: 10.0004 seconds
Process running with session ID 99757
Original token count: 0
Output:


Output:
Chunk ID: 688b17
Wall time: 5.0008 seconds
Process running with session ID 99757
Original token count: 0
Output:


Output:
Chunk ID: 34b648
Wall time: 5.0014 seconds
Process running with session ID 99757
Original token count: 0
Output:


Output:
Chunk ID: 2906c8
Wall time: 5.0006 seconds
Process running with session ID 99757
Original token count: 0
Output:


Output:
Chunk ID: 1f4282
Wall time: 0.0512 seconds
Process exited with code 0
Original token count: 3
Output:
33, 46994


cd /home/ubuntu/veenaModal && source venv/bin/activate && python3 - <<'PY'
import httpx, time
payload={'text':'Hello, this is a quick test.','speaker':'lipakshi','stream':False,'chunking':False,'format':'wav'}
start=time.time()
try:
    r=httpx.post('http://127.0.0.1:8000/v1/tts/generate', json=payload, timeout=10)
    print('status',r.status_code,'ms', (time.time()-start)*1000)
    print('len',len(r.content))
except Exception as e:
    print('err',type(e),e,'elapsed', (time.time()-start)*1000)
PY


Output:
Chunk ID: ebd124
Wall time: 10.0008 seconds
Process running with session ID 81293
Original token count: 0
Output:


Output:
Chunk ID: 8346c1
Wall time: 0.0532 seconds
Process exited with code 0
Original token count: 18
Output:
err <class 'httpx.ReadTimeout'> timed out elapsed 10103.291511535645


Output:
Chunk ID: 515f81
Wall time: 5.0011 seconds
Process running with session ID 38524
Original token count: 391
Output:
{"timestamp": "2026-02-17T00:18:50.740813+00:00", "level": "INFO", "logger": "veena3modal.tts", "message": "request_received", "request_id": "98bda89b-c839-452c-85d5-154f76f3d670", "event": "request_received", "text_length": 28, "speaker": "lipakshi", "stream": false, "format": "wav"}
2026-02-17 00:18:50 | INFO     | veena3modal.tts | request_received
2026-02-17 00:18:50 | INFO     | veena3modal.services.tts_runtime | tts_runtime_chunking_start
2026-02-17 00:18:50 | INFO     | veena3modal.processing.long_text_processor | LongTextProcessor initialized
2026-02-17 00:19:24 | INFO     | veena3modal.tts | request_received
2026-02-17 00:19:24 | INFO     | veena3modal.services.tts_runtime | tts_runtime_nonstream_start
{"timestamp": "2026-02-17T00:19:24.655512+00:00", "level": "INFO", "logger": "veena3modal.tts", "message": "request_received", "request_id": "d1fe0d1c-90e5-45ea-8c84-7134e6146c68", "event": "request_received", "text_length": 28, "speaker": "lipakshi", "stream": false, "format": "wav"}
2026-02-17 00:19:41 | INFO     | veena3modal.tts | request_received
2026-02-17 00:19:41 | INFO     | veena3modal.services.tts_runtime | tts_runtime_chunking_start
2026-02-17 00:19:41 | INFO     | veena3modal.processing.long_text_processor | LongTextProcessor initialized
{"timestamp": "2026-02-17T00:19:41.333577+00:00", "level": "INFO", "logger": "veena3modal.tts", "message": "request_received", "request_id": "7ef00896-504a-4d9c-8939-6d5b867684f9", "event": "request_received", "text_length": 28, "speaker": "lipakshi", "stream": false, "format": "wav"}


Output:
Chunk ID: fea866
Wall time: 0.0512 seconds
Process exited with code 0
Original token count: 76
Output:
142:    async def generate_with_chunking(
154:        request_id: Optional[str] = None,
234:            chunk_result = await self.pipeline.generate_speech_indic(
243:                request_id=request_id,
311:                "request_id": request_id,
397:    async def generate_with_chunking_streaming(


I found the issue now appears to be that non-streamed chunking mode is not completing requests (likely waiting in chunking path), so I’m inspecting `LongTextProcessor` for argument-path behavior before choosing benchmark mode.


sed -n '130,280p' /home/ubuntu/veenaModal/veena3modal/processing/long_text_processor.py


Output:
Chunk ID: 8c153f
Wall time: 0.0508 seconds
Process exited with code 0
Original token count: 1551
Output:

        Delegates to self.chunker.chunk_text() for actual chunking logic.
        This method is used by _stream_chunked_text in tts_runtime.

        Args:
            text: Text to chunk

        Returns:
            List of text chunks
        """
        return self.chunker.chunk_text(text)

    async def generate_with_chunking(
        self,
        text: str,
        speaker: str,
        temperature: float = 0.4,
        top_k: int = 50,  # Added for Spark TTS
        top_p: float = 0.9,
        max_tokens: int = 4096,
        repetition_penalty: float = 1.05,  # Prevent token repetition
        seed: Optional[int] = None,
        sample_rate: int = 16000,  # BiCodec uses 16kHz, not 24kHz
        return_timing: bool = False,
        request_id: Optional[str] = None,
    ) -> Optional[bytes] | tuple[Optional[bytes], dict]:
        """
        Generate audio for long text with chunking and stitching.

        Process:
        1. Chunk text at natural boundaries
        2. Generate audio for each chunk
        3. Stitch audio with crossfade

        Args:
            text: Long text to synthesize
            speaker: Speaker name (lipakshi, reet, etc.)
            temperature: Sampling temperature
            top_p: Nucleus sampling
            max_tokens: Max tokens per chunk
            seed: Random seed (same seed = same voice across chunks)
            sample_rate: Audio sample rate

        Returns:
            Stitched audio bytes (int16 PCM) and optional timing metrics
        """
        # Tracking variables for detailed timing aggregation
        request_start = time.perf_counter()
        emitted_chunk_metrics: List[dict] = []
        llm_batch_wall_ms_values: List[float] = []
        llm_batch_gpu_ms_values: List[float] = []
        llm_decode_cpu_ms_values: List[float] = []
        llm_parse_ms_values: List[float] = []
        llm_batch_tokens: List[int] = []

        # Detect language mix for logging
        lang_info = self.chunker.detect_language_mix(text)

        logger.info(
            f"Processing long text with chunking",
            extra={
                "text_length": len(text),
                "primary_language": lang_info['primary'],
                "lang_mix": lang_info,
                "speaker": speaker,
            }
        )

        # 1. Chunk text
        text_chunks = self.chunker.chunk_text(text)

        logger.info(
            f"Text chunked into {len(text_chunks)} chunks",
            extra={
                "num_chunks": len(text_chunks),
                "chunk_sizes": [len(chunk) for chunk in text_chunks],
            }
        )

        # 2. Generate audio for each chunk
        audio_chunks = []

        # Log summary at start
        logger.info(
            f"🎵 Generating {len(text_chunks)} chunks for {len(text)} chars",
            extra={
                "total_chunks": len(text_chunks),
                "total_chars": len(text),
                "speaker": speaker,
            }
        )

        for i, chunk in enumerate(text_chunks):
            # Calculate appropriate max_tokens for this chunk
            # BiCodec semantic tokens: ~50 tokens/second, ~5 words/second = ~10 tokens/word
            # Rough estimate: 5 chars/word, so ~2 tokens/char for audio generation
            # Add buffer for safety: multiply by 3 to handle variations
            chunk_max_tokens = min(
                int(len(chunk) * 3),  # ~2 tokens/char * 1.5 safety buffer = 3
                4096  # Cap at model's max context length
            )
            chunk_max_tokens = max(chunk_max_tokens, 100)  # Ensure minimum

            # Generate audio for this chunk (returns WAV format with header)
            chunk_result = await self.pipeline.generate_speech_indic(
                speaker=speaker,
                text=chunk,
                temperature=temperature,
                top_k=top_k,  # Added for Spark TTS
                top_p=top_p,
                max_tokens=chunk_max_tokens,  # Use calculated limit, not global max_tokens
                repetition_penalty=repetition_penalty,  # Prevent token repetition
                seed=seed,  # Use same seed for consistent voice across chunks
                request_id=request_id,
                return_timing=return_timing,
            )

            if return_timing:
                if isinstance(chunk_result, tuple):
                    audio_bytes, chunk_timing = chunk_result
                    if isinstance(chunk_timing, dict):
                        emitted_chunk_metrics.append(chunk_timing)
                        llm_batch_wall_ms_values.append(int(chunk_timing.get("llm_wall_ms", 0)))
                        llm_batch_gpu_ms_values.append(int(chunk_timing.get("llm_gpu_batch_ms_total", 0)))
                        llm_decode_cpu_ms_values.append(int(chunk_timing.get("decode_cpu_ms", 0)))
                        llm_parse_ms_values.append(int(chunk_timing.get("parse_ms", 0)))
                        chunk_tokens = chunk_timing.get("token_batches", [])
                        if isinstance(chunk_tokens, list):
                            llm_batch_tokens.extend([int(x) for x in chunk_tokens if isinstance(x, int) or str(x).isdigit()])
                else:
                    audio_bytes = chunk_result
            else:
                audio_bytes = chunk_result

            if audio_bytes is None:
                logger.error(
                    f"❌ Failed to generate audio for chunk {i+1}/{len(text_chunks)}",
                    extra={
                        "chunk_index": i,
                        "chunk_length": len(chunk),
                        "chunk_text_preview": chunk[:150],
                        "total_chunks": len(text_chunks),
                    }
                )
                return None

            # Strip WAV header (44 bytes) to get raw PCM data for stitching
            pcm_data = audio_bytes[44:] if len(audio_bytes) > 44 else audio_bytes
            audio_chunks.append(pcm_data)

        # 3. Stitch audio with crossfade


sed -n '70,180p' /home/ubuntu/veenaModal/veena3modal/core/pipeline.py


Output:
Chunk ID: ec7a61
Wall time: 0.0508 seconds
Process exited with code 0
Original token count: 1079
Output:
        speaker: str,
        text: str,
        temperature: float = DEFAULT_TEMPERATURE,
        top_k: int = DEFAULT_TOP_K,
        top_p: float = DEFAULT_TOP_P,
        max_tokens: int = DEFAULT_MAX_TOKENS,
        repetition_penalty: float = 1.05,
        seed: Optional[int] = None,
        return_timing: bool = False,
        request_id: Optional[str] = None,
    ) -> Optional[bytes] | tuple[Optional[bytes], Dict[str, Any]]:
        """
        Generate speech audio (non-streaming) using Spark TTS.

        NOTE: This method signature changed from description-based to speaker-based
        to align with Spark TTS architecture.

        Args:
            speaker: Speaker name (one of 12 predefined speakers)
            text: Text to synthesize (with optional [emotion] tags)
            temperature: Sampling temperature (default: 0.8 for Spark TTS)
            top_k: Top-k sampling (default: 50)
            top_p: Nucleus sampling (default: 1.0)
            max_tokens: Max BiCodec tokens to generate (default: 2048)
            seed: Random seed for reproducibility
            return_timing: If true, returns (audio_bytes, metrics_dict)

        Returns:
            Audio bytes (int16 PCM WAV, 16kHz mono) or None if failed.
            If return_timing is true, returns tuple(audio_bytes, metrics).
        """
        # Build prompt using Spark TTS format
        prompt = self.prompt_builder.build_prefix(speaker, text)

        # Configure sampling for Spark TTS
        sampling_params = SamplingParams(
            temperature=temperature,
            top_k=top_k,
            top_p=top_p,
            max_tokens=max_tokens,
            repetition_penalty=repetition_penalty,
            stop=TRAINING_STOP_TOKEN_IDS,
            skip_special_tokens=False,
            seed=seed,
        )

        # Use vLLM engine to generate
        if request_id is None:
            import uuid
            request_id = f"req-{uuid.uuid4().hex[:12]}"

        emit_timing = return_timing or _detailed_timing_enabled()
        results_generator = self.model.engine.generate(
            prompt=prompt,
            sampling_params=sampling_params,
            request_id=request_id,
        )

        # Collect all outputs and track generation timing
        final_output = None
        token_batches: List[int] = []
        batch_wall_ms: List[float] = []
        batch_gpu_ms: List[float] = []
        total_token_ids = 0
        first_token_ms: Optional[float] = None
        t_start = asyncio.get_event_loop().time()

        iterator = results_generator.__aiter__()
        while True:
            wall_start = asyncio.get_event_loop().time()
            start_event = None
            end_event = None
            if torch.cuda.is_available():
                torch.cuda.synchronize()
                start_event = torch.cuda.Event(enable_timing=True)
                end_event = torch.cuda.Event(enable_timing=True)
                start_event.record()

            try:
                request_output = await iterator.__anext__()
            except StopAsyncIteration:
                break

            wall_end = asyncio.get_event_loop().time()
            batch_ms = (wall_end - wall_start) * 1000
            batch_wall_ms.append(batch_ms)

            batch_gpu_ms_value: Optional[float] = None
            if start_event is not None and end_event is not None:
                end_event.record()
                torch.cuda.synchronize()
                batch_gpu_ms_value = start_event.elapsed_time(end_event)
            if batch_gpu_ms_value is not None:
                batch_gpu_ms.append(batch_gpu_ms_value)

            total_generated = len(request_output.outputs[0].token_ids)
            new_tokens = max(0, total_generated - total_token_ids)
            total_token_ids = total_generated
            token_batches.append(new_tokens)

            if first_token_ms is None:
                first_token_ms = (wall_end - t_start) * 1000

            if emit_timing:
                logger.info(
                    "tts_llm_batch",
                    extra={
                        "speaker": speaker,
                        "request_id": request_id,
                        "batch_index": len(token_batches),
                        "batch_tokens": new_tokens,


I confirmed non-stream and chunked paths now both stall on generation, so this looks like a generation timeout configuration issue rather than chunk routing. I’m checking request schema defaults (especially `max_tokens`) to prevent over-long requests before rerunning the benchmark with constrained generation caps.


sed -n '1,260p' /home/ubuntu/veenaModal/veena3modal/api/schemas.py


Output:
Chunk ID: 13aeff
Wall time: 0.0507 seconds
Process exited with code 0
Original token count: 1906
Output:
"""
Pydantic schemas for TTS API requests/responses.

Ported from veena3srv/apps/api/serializers.py with full validation parity.
No Django dependencies - pure Pydantic for Modal deployment.
"""

from __future__ import annotations

import re
from enum import Enum
from typing import Any, Dict, List, Optional, Union
from uuid import UUID

from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator

# === Constants (mirrored from veena3srv/apps/inference/constants.py) ===
# These are duplicated here to avoid Django import chain; must stay in sync.

MAX_TEXT_LENGTH = 50000

# Spark TTS Speaker System - 12 Speakers
# Model: BayAreaBoys/spark_tts_4speaker (HuggingFace) has 12 speaker tokens
SPEAKER_MAP = {
    "lipakshi": 0, "vardan": 1, "reet": 2, "Nandini": 3,
    "krishna": 4, "anika": 5, "adarsh": 6, "Nilay": 7,
    "Aarvi": 8, "Asha": 9, "Bittu": 10, "Mira": 11,
}

# Friendly speaker name mappings (user-facing → internal)
FRIENDLY_SPEAKER_MAP = {
    "Mitra": "lipakshi", "Aaranya": "reet", "Taru": "Nandini",
    "Neer": "Nilay", "Dhruva": "vardan", "Ira": "anika",
    "Veda": "adarsh", "Aria": "krishna",
    "Aarvi": "Aarvi", "Asha": "Asha", "Bittu": "Bittu", "Mira": "Mira",
}

INDIC_SPEAKERS = list(SPEAKER_MAP.keys())
ALL_SPEAKER_NAMES = INDIC_SPEAKERS + list(FRIENDLY_SPEAKER_MAP.keys())

# Emotion tags (Spark TTS bracket format)
INDIC_EMOTION_TAGS = [
    "[angry]", "[curious]", "[excited]", "[giggle]", "[laughs harder]",
    "[laughs]", "[screams]", "[sighs]", "[sings]", "[whispers]"
]

# Legacy emotion tag mapping (<angle> → [bracket])
LEGACY_EMOTION_MAP = {
    "<angry>": "[angry]", "<curious>": "[curious]", "<excited>": "[excited]",
    "<giggle>": "[giggle]", "<laugh_harder>": "[laughs harder]",
    "<laugh>": "[laughs]", "<scream>": "[screams]", "<sigh>": "[sighs]",
    "<sing>": "[sings]", "<whisper>": "[whispers]"
}

# === Helper Functions ===

def resolve_speaker_name(name: str) -> str:
    """
    Resolve friendly speaker name to internal name.

    Examples:
        resolve_speaker_name("Mitra") -> "lipakshi"
        resolve_speaker_name("lipakshi") -> "lipakshi"

    Raises:
        ValueError: if name is not a valid speaker name
    """
    if name in FRIENDLY_SPEAKER_MAP:
        return FRIENDLY_SPEAKER_MAP[name]
    if name in INDIC_SPEAKERS:
        return name
    raise ValueError(
        f"Invalid speaker name: {name}. Valid names: {', '.join(ALL_SPEAKER_NAMES)}"
    )

# Emotion normalization mapping (various formats → [emotion])
_EMOTION_TO_TAG = {
    **{k.strip('<>'): v for k, v in LEGACY_EMOTION_MAP.items()},
    'laughing': '[laughs]', 'laugh': '[laughs]', 'laughs harder': '[laughs harder]',
    'laugh harder': '[laughs harder]', 'laugh_harder': '[laughs harder]',
    'sighs': '[sighs]', 'sigh': '[sighs]', 'giggles': '[giggle]', 'giggle': '[giggle]',
    'angry': '[angry]', 'excited': '[excited]', 'whispers': '[whispers]',
    'whisper': '[whispers]', 'screaming': '[screams]', 'scream': '[screams]',
    'screams': '[screams]', 'singing': '[sings]', 'sing': '[sings]',
    'sings': '[sings]', 'curious': '[curious]',
}

def normalize_emotion_tags(text: str) -> str:
    """
    Normalize emotion tags to Spark TTS bracket format.

    Conversions:
    - <laugh> → [laughs]
    - [laughing] → [laughs]
    - etc.
    """
    # Convert old <emotion> format to [emotion]
    angle_pattern = r'<([a-z_]+)>'

    def replace_angle(match):
        emotion = match.group(1).lower().strip()
        if emotion in _EMOTION_TO_TAG:
            return _EMOTION_TO_TAG[emotion]
        old_tag = f'<{emotion}>'
        if old_tag in LEGACY_EMOTION_MAP:
            return LEGACY_EMOTION_MAP[old_tag]
        return f'[{emotion}]'  # Unknown, convert to bracket anyway

    text = re.sub(angle_pattern, replace_angle, text)

    # Normalize [emotion] variants
    bracket_pattern = r'\[([^\]]+)\]'

    def replace_bracket(match):
        emotion = match.group(1).lower().strip()
        bracket_tag = f'[{emotion}]'
        if bracket_tag in INDIC_EMOTION_TAGS:
            return bracket_tag
        if emotion in _EMOTION_TO_TAG:
            return _EMOTION_TO_TAG[emotion]
        # Try singular/plural
        if emotion.endswith('s') and emotion[:-1] in _EMOTION_TO_TAG:
            return _EMOTION_TO_TAG[emotion[:-1]]
        if (emotion + 's') in _EMOTION_TO_TAG:
            return _EMOTION_TO_TAG[emotion + 's']
        return bracket_tag  # Keep as-is

    return re.sub(bracket_pattern, replace_bracket, text)

# === Enums ===

class AudioFormat(str, Enum):
    """Supported audio output formats."""
    WAV = "wav"
    OPUS = "opus"
    MP3 = "mp3"
    MULAW = "mulaw"
    FLAC = "flac"

class OutputSampleRate(str, Enum):
    """Output sample rate options."""
    SR_16KHZ = "16khz"
    SR_48KHZ = "48khz"

# === Request Schema ===

class TTSGenerateRequest(BaseModel):
    """
    Request schema for /v1/tts/generate.

    Mirrors validation rules from veena3srv/apps/api/serializers.py.
    """

    # Required
    text: str = Field(
        ...,
        min_length=1,
        max_length=MAX_TEXT_LENGTH,
        description="Text to synthesize (max 50,000 characters)"
    )

    # Speaker (required for indic_speakers model)
    speaker: Optional[str] = Field(
        None,
        max_length=50,
        description="Speaker name (internal or friendly)"
    )

    # Voice design (not currently supported)
    description: Optional[str] = Field(
        None,
        max_length=2000,
        description="Voice description (for voiceDesign model, NOT currently supported)"
    )
    voice_id: Optional[UUID] = Field(
        None,
        description="UUID of pre-created voice profile (for voiceDesign model)"
    )

    # Generation options
    model: Optional[str] = Field(
        None,
        max_length=50,
        description="Model to use (optional). Default: indic_speakers"
    )
    seed: Optional[int] = Field(
        None,
        ge=0,
        le=2**31 - 1,
        description="Random seed for reproducibility (0-2147483647)"
    )
    stream: bool = Field(
        False,
        description="Stream audio as it's generated"
    )
    format: AudioFormat = Field(
        AudioFormat.WAV,
        description="Audio output format"
    )
    sample_rate: int = Field(
        16000,
        description="Sample rate in Hz (default: 16000)"
    )

    # Advanced parameters
    temperature: float = Field(
        0.4,
        ge=0.0,
        le=2.0,
        description="Sampling temperature (0.0-2.0)"
    )
    top_k: int = Field(
        50,
        ge=1,
        le=100,
        description="Top-k sampling (1-100)"
    )
    top_p: float = Field(
        1.0,
        ge=0.0,
        le=1.0,
        description="Nucleus sampling (0.0-1.0)"
    )
    max_tokens: int = Field(
        4096,
        ge=128,
        le=4096,
        description="Maximum BiCodec tokens to generate"
    )
    repetition_penalty: float = Field(
        1.05,
        ge=1.0,
        le=2.0,
        description="Repetition penalty (1.0-2.0)"
    )

    # Preprocessing toggles
    normalize: bool = Field(
        True,
        description="Apply text normalization before TTS"
    )
    normalize_verbose: bool = Field(
        False,
        description="Return normalized text in X-Normalized-Text header"
    )
    chunking: bool = Field(
        True,
        description="Enable intelligent text chunking for long inputs"
    )

    # Output options
    output: OutputSampleRate = Field(
        OutputSampleRate.SR_16KHZ,


Input:
*** Begin Patch
*** Update File: /home/ubuntu/veenaModal/scripts/stress_test_optimized_local.py
@@
 async def make_request(
     client: httpx.AsyncClient,
     base_url: str,
     text: str,
     speaker: str,
     chunking: bool,
+    max_tokens: int,
     stream: bool = False,
     timeout: float = 120.0,
 ) -> RequestResult:
@@
         "chunking": chunking,
+        "max_tokens": max_tokens,
         "format": "wav",
     }
@@
 async def run_level(
@@
     mode: str,
     stream: bool,
+    max_tokens: int,
     gpu_monitor: Optional[GPUMonitor] = None,
 ) -> LevelResult:
@@
     async def limited_request(i: int, client: httpx.AsyncClient) -> RequestResult:
         async with semaphore:
             text = texts[i % len(texts)]
             speaker = SPEAKERS[i % len(SPEAKERS)]
             return await make_request(
                 client=client,
                 base_url=base_url,
                 text=text,
                 speaker=speaker,
                 chunking=chunking,
+                max_tokens=max_tokens,
                 stream=stream,
             )
@@
     concurrency = 10,  # keep low for local GPU in single-proc mode
     text="short",
     chunking=True,
+    max_tokens=256,
     stream=False,
     output="stress_test_optimized_local.json",
 )
@@
     args = parser.parse_args()
@@
                 mode=mode,
                 text_category=args.text,
                 chunking=args.chunking,
                 stream=args.stream,
+                max_tokens=args.max_tokens,
                 gpu_monitor=gpu_monitor,
             )
*** End Patch


Output:
apply_patch verification failed: Failed to find expected lines in /home/ubuntu/veenaModal/scripts/stress_test_optimized_local.py:
    mode: str,
    stream: bool,
    gpu_monitor: Optional[GPUMonitor] = None,
) -> LevelResult:


rg -n "def make_request|def run_level|def main|payload = \{|mode: str|text_category" /home/ubuntu/veenaModal/scripts/stress_test_optimized_local.py


Output:
Chunk ID: 4d6be7
Wall time: 0.0507 seconds
Process exited with code 0
Original token count: 128
Output:
165:    mode: str
166:    text_category: str
412:async def make_request(
422:    payload = {
561:async def run_level(
565:    text_category: str,
572:        level_name=f"{concurrency}c-{text_category}-{mode}",
576:        text_category=text_category,
582:    if text_category == "mixed":
585:        texts = [TEST_TEXTS.get(text_category, TEST_TEXTS["short"])]
746:            "text_category": level.text_category,
812:        text_category=args.text,
826:            text_category=args.text,
845:def main():


sed -n '398,470p' /home/ubuntu/veenaModal/scripts/stress_test_optimized_local.py && sed -n '560,700p' /home/ubuntu/veenaModal/scripts/stress_test_optimized_local.py && sed -n '700,910p' /home/ubuntu/veenaModal/scripts/stress_test_optimized_local.py


Output:
Chunk ID: 28ca09
Wall time: 0.0512 seconds
Process exited with code 0
Original token count: 3478
Output:
        try:
            metrics["llm_time_per_batch_wall_ms"] = float(metrics["generation_ms"]) / float(metrics["llm_batch_count"])
        except Exception:
            pass

    if "llm_batch_count" in metrics and "llm_batch_gpu_ms_total" in metrics and metrics["llm_batch_count"]:
        try:
            metrics["llm_time_per_batch_gpu_ms"] = float(metrics["llm_batch_gpu_ms_total"]) / float(metrics["llm_batch_count"])
        except Exception:
            pass

    return metrics

async def make_request(
    client: httpx.AsyncClient,
    base_url: str,
    text: str,
    speaker: str,
    chunking: bool,
    stream: bool = False,
    timeout: float = 120.0,
) -> RequestResult:
    url = f"{base_url}{GENERATE_PATH}"
    payload = {
        "text": text,
        "speaker": speaker,
        "stream": stream,
        "chunking": chunking,
        "format": "wav",
    }

    start = time.time()
    ttfb_time = 0.0

    try:
        if stream:
            async with client.stream(
                "POST",
                url,
                json=payload,
                headers={"Content-Type": "application/json"},
                timeout=timeout,
            ) as response:
                first = True
                total_bytes = 0
                async for chunk in response.aiter_bytes():
                    if first:
                        ttfb_time = (time.time() - start) * 1000
                        first = False
                    total_bytes += len(chunk)

                latency = (time.time() - start) * 1000
                timing = _parse_perf_headers(response.headers)

                if response.status_code == 200:
                    pcm_bytes = max(0, total_bytes - 44)
                    audio_seconds = pcm_bytes / (16000 * 2)
                    return RequestResult(
                        success=True,
                        status_code=200,
                        latency_ms=latency,
                        ttfb_ms=ttfb_time,
                        audio_bytes=total_bytes,
                        audio_seconds=audio_seconds,
                        request_id=response.headers.get("x-request-id", ""),
                        speaker=speaker,
                        text_length=len(text),
                        stream=True,
                        timing=timing,
                        request_inflight=timing.get("request_inflight"),
                        server_ttfb_ms=_normalize_time_ms(response.headers.get("x-ttfb-ms")),
                    )

async def run_level(
    base_url: str,
    num_requests: int,
    concurrency: int,
    text_category: str,
    chunking: bool,
    stream: bool,
    gpu_monitor: Optional[GPUMonitor] = None,
) -> LevelResult:
    mode = "streaming" if stream else "non-streaming"
    level = LevelResult(
        level_name=f"{concurrency}c-{text_category}-{mode}",
        concurrency=concurrency,
        total_requests=num_requests,
        mode=mode,
        text_category=text_category,
        chunking=chunking,
    )

    semaphore = asyncio.Semaphore(concurrency)

    if text_category == "mixed":
        texts = list(TEST_TEXTS.values())
    else:
        texts = [TEST_TEXTS.get(text_category, TEST_TEXTS["short"])]

    async def limited_request(i: int, client: httpx.AsyncClient) -> RequestResult:
        async with semaphore:
            text = texts[i % len(texts)]
            speaker = SPEAKERS[i % len(SPEAKERS)]
            return await make_request(
                client=client,
                base_url=base_url,
                text=text,
                speaker=speaker,
                chunking=chunking,
                stream=stream,
            )

    if gpu_monitor:
        gpu_monitor.snapshots.clear()

    start_time = time.time()
    limits = httpx.Limits(
        max_connections=concurrency + 10,
        max_keepalive_connections=concurrency,
    )
    async with httpx.AsyncClient(limits=limits) as client:
        tasks = [limited_request(i, client) for i in range(num_requests)]
        responses = await asyncio.gather(*tasks, return_exceptions=True)

    level.wall_time_seconds = time.time() - start_time

    for item in responses:
        if isinstance(item, RequestResult):
            level.results.append(item)
        else:
            level.results.append(
                RequestResult(
                    success=False,
                    status_code=0,
                    latency_ms=0,
                    error=str(item)[:200],
                    stream=stream,
                )
            )

    if gpu_monitor:
        level.gpu_summary = gpu_monitor.summary()

    return level

def print_level_report(level: LevelResult):
    print(f"\n{'=' * 72}")
    print(f"  {level.level_name.upper()}")
    print(f"  {level.total_requests} requests, {level.concurrency} concurrent, {level.mode}, chunking={level.chunking}")
    print(f"{'=' * 72}")

    print(f"  Success:     {len(level.successes)}/{level.total_requests} ({level.success_rate:.0%})")
    print(f"  Wall time:   {level.wall_time_seconds:.2f}s")
    print(f"  Throughput:  {level.throughput_rps:.2f} req/s")
    print(f"  Audio total: {level.audio_duration_total:.2f}s")
    print(f"  Eff. RTF:    {level.effective_rtf:.3f}")

    if level.observed_inflight_max() is not None:
        print(f"  Peak inflight: {level.observed_inflight_max()}")

    latency = level.latency_stats
    if latency:
        print("\n  Latency (ms):")
        print(
            f"    min={latency['min']:.0f} avg={latency['avg']:.0f} "
            f"p50={latency['p50']:.0f} p95={latency['p95']:.0f} p99={latency['p99']:.0f} max={latency['max']:.0f}"
        )

    ttfb = level.ttfb_stats
    if ttfb and not level.mode.startswith("stream"):
        print("\n  Server TTFB (ms):")
        print(
            f"    min={ttfb['min']:.0f} avg={ttfb['avg']:.0f} "
            f"p50={ttfb['p50']:.0f} p95={ttfb['p95']:.0f} max={ttfb['max']:.0f}"
        )

    if level.mode == "non-streaming":
        llm = level.timing_stats("generation_ms")
        if llm:
            print("\n  Generation (ms):")
            print(f"    min={llm['min']:.0f} avg={llm['avg']:.0f} p50={llm['p50']:.0f} p95={llm['p95']:.0f} max={llm['max']:.0f}")
        wall = level.timing_stats("llm_batch_wall_ms_total")
        if wall:
            print("\n  LLM batch wall (ms):")
            print(f"    min={wall['min']:.0f} avg={wall['avg']:.0f} p50={wall['p50']:.0f} p95={wall['p95']:.0f} max={wall['max']:.0f}")
        gpu = level.timing_stats("llm_batch_gpu_ms_total")
        if gpu:
            print("\n  LLM batch GPU (ms):")
            print(f"    min={gpu['min']:.0f} avg={gpu['avg']:.0f} p50={gpu['p50']:.0f} p95={gpu['p95']:.0f} max={gpu['max']:.0f}")
        parse = level.timing_stats("llm_parse_ms")
        if parse:
            print("\n  Parse (ms):")
            print(f"    avg={parse['avg']:.0f} p95={parse['p95']:.0f}")
        decode = level.timing_stats("llm_decode_wall_ms_total")
        if decode:
            print("\n  Decode (ms):")
            print(f"    avg={decode['avg']:.0f} p95={decode['p95']:.0f}")

        token_ms, token_count = level.timing_rate("generation_ms", "llm_token_total")
        if token_count:
            print(f"  Token timing: avg {token_ms / token_count * 1000:.3f} us/token")

    if level.gpu_summary and "memory_mb" in level.gpu_summary:
        gpu_summary = level.gpu_summary
        mem = gpu_summary["memory_mb"]
        util = gpu_summary["gpu_util_pct"]
        temp = gpu_summary["temperature_c"]
        print("\n  GPU:")
        print(f"    Memory: {mem['avg']:.0f}MB avg, {mem['max']:.0f}MB peak / {mem['total']:.0f}MB")
        print(f"    Util:   {util['avg']:.0f}% avg, {util['max']:.0f}% peak")
        print(f"    Temp:   {temp['avg']:.0f}C avg, {temp['max']:.0f}C peak")

    errors = level.error_summary()
    if errors:
        print("\n  Errors:")
        for key, count in sorted(errors.items(), key=lambda kv: -kv[1]):
            print(f"    [{count}x] {key}")

def print_summary_table(levels: List[LevelResult]):
    print(f"\n{'=' * 110}")
    print("  STRESS BENCHMARK SUMMARY")
    print(f"{'=' * 110}")
    header = f"{'Level':<26} {'OK%':>5} {'RPS':>7} {'p50':>7} {'p95':>7} {'GPU%':>5} {'GPUmem':>8} {'RTF':>6} {'Inflight':>8}"
    print(header)
    print("-" * 110)

    for level in levels:
        lat = level.latency_stats
        p50 = f"{lat.get('p50', 0):.0f}" if lat else "---"
        p95 = f"{lat.get('p95', 0):.0f}" if lat else "---"
        gpu_util = level.gpu_summary.get("gpu_util_pct", {}).get("avg", 0)
        gpu_mem = level.gpu_summary.get("memory_mb", {}).get("max", 0)
        inflight = level.observed_inflight_max() or 0
        print(
            f"{level.level_name:<26} "
            f"{level.success_rate:>4.0%} "
            f"{level.throughput_rps:>7.2f} "
            f"{p50:>7} "
            f"{p95:>7} "
            f"{gpu_util:>4.0f}% "
            f"{gpu_mem:>7.0f}MB "
            f"{level.effective_rtf:>6.3f} "
            f"{inflight:>8d}"
        )

    print("=" * 110)

def save_results(levels: List[LevelResult], path: str):
    payload = []
    for level in levels:
        payload.append({
            "level": level.level_name,
            "concurrency": level.concurrency,
            "mode": level.mode,
            "chunking": level.chunking,
            "text_category": level.text_category,
            "success_rate": level.success_rate,
            "throughput_rps": level.throughput_rps,
            "wall_time_s": level.wall_time_seconds,
            "latency_ms": level.latency_stats,
            "ttfb_ms": level.ttfb_stats,
            "effective_rtf": level.effective_rtf,
            "audio_total_seconds": level.audio_duration_total,
            "observed_inflight_max": level.observed_inflight_max(),
            "errors": level.error_summary(),
            "gpu": level.gpu_summary,
            "timing_averages": {
                "generation_ms": level.timing_stats("generation_ms"),
                "llm_batch_wall_ms_total": level.timing_stats("llm_batch_wall_ms_total"),
                "llm_batch_gpu_ms_total": level.timing_stats("llm_batch_gpu_ms_total"),
                "llm_parse_ms": level.timing_stats("llm_parse_ms"),
                "llm_decode_wall_ms_total": level.timing_stats("llm_decode_wall_ms_total"),
            },
            "requests": [
                {
                    "request_id": r.request_id,
                    "success": r.success,
                    "status_code": r.status_code,
                    "latency_ms": r.latency_ms,
                    "ttfb_ms": r.ttfb_ms,
                    "server_ttfb_ms": r.server_ttfb_ms,
                    "audio_bytes": r.audio_bytes,
                    "audio_seconds": r.audio_seconds,
                    "request_inflight": r.request_inflight,
                    "error": r.error,
                    "timing": r.timing,
                }
                for r in level.results
            ],
        })

    with open(path, "w") as fd:
        json.dump(payload, fd, indent=2)

async def run_stress_test(args):
    base_url = args.url

    try:
        async with httpx.AsyncClient() as client:
            response = await client.get(f"{base_url}/v1/tts/health", timeout=10)
            health = response.json()
            print(f"  Health: {health.get('status')}")
            print(f"  Version: {health.get('model_version')}")
            print(f"  GPU: {health.get('gpu_available')}")
    except Exception as e:
        print(f"ERROR: server not reachable at {base_url}: {e}")
        sys.exit(1)

    levels = [int(v) for v in args.levels.split(",") if v.strip()]
    gpu_monitor = GPUMonitor(interval_seconds=0.5)
    gpu_monitor.start()

    mode_label = "streaming" if args.stream else "non-streaming"
    results: List[LevelResult] = []

    print(f"\nWarming up (1 request, non-streaming)...")
    warmup = await run_level(
        base_url=base_url,
        num_requests=1,
        concurrency=1,
        text_category=args.text,
        chunking=args.chunking,
        stream=False,
        gpu_monitor=gpu_monitor,
    )
    print(f"  warmup ok: {warmup.success_rate:.0%}")

    for conc in levels:
        num_requests = max(conc * 3, 10)
        num_requests = min(num_requests, 200)
        level = await run_level(
            base_url=base_url,
            num_requests=num_requests,
            concurrency=conc,
            text_category=args.text,
            chunking=args.chunking,
            stream=args.stream,
            gpu_monitor=gpu_monitor,
        )
        print_level_report(level)
        results.append(level)
        if conc != levels[-1]:
            print("  cooldown 3s...")
            await asyncio.sleep(3)

    gpu_monitor.stop()
    print_summary_table(results)

    if args.output:
        save_results(results, args.output)
        print(f"\nResults written to {args.output}")

def main():
    parser = argparse.ArgumentParser(description="Detailed local optimized-path TTS benchmark")
    parser.add_argument("--url", default=DEFAULT_URL, help=f"Server URL (default: {DEFAULT_URL})")
    parser.add_argument("--levels", default="1,5,10,20,50", help="Comma-separated concurrency levels")
    parser.add_argument("--text", default="short", choices=["short", "medium", "long", "mixed"], help="Input text category")
    parser.add_argument("--chunking", action="store_true", help="Enable server-side chunking", default=True)
    parser.add_argument("--no-chunking", action="store_false", dest="chunking", help="Disable chunking")
    parser.add_argument("--stream", action="store_true", help="Use streaming endpoint")
    parser.add_argument("--output", default="stress_test_optimized_local.json", help="Output JSON path")

    args = parser.parse_args()
    asyncio.run(run_stress_test(args))

if __name__ == "__main__":
    main()


Input:
*** Begin Patch
*** Update File: /home/ubuntu/veenaModal/scripts/stress_test_optimized_local.py
@@
 async def make_request(
     client: httpx.AsyncClient,
     base_url: str,
     text: str,
     speaker: str,
     chunking: bool,
+    max_tokens: int,
     stream: bool = False,
     timeout: float = 120.0,
 ) -> RequestResult:
@@
         "text": text,
         "speaker": speaker,
         "stream": stream,
         "chunking": chunking,
+        "max_tokens": max_tokens,
         "format": "wav",
     }
@@
 async def run_level(
@@
     text_category: str,
     chunking: bool,
     stream: bool,
+    max_tokens: int,
     gpu_monitor: Optional[GPUMonitor] = None,
 ) -> LevelResult:
@@
             return await make_request(
                 client=client,
                 base_url=base_url,
                 text=text,
                 speaker=speaker,
                 chunking=chunking,
+                max_tokens=max_tokens,
                 stream=stream,
             )
@@
     warmup = await run_level(
         base_url=base_url,
         num_requests=1,
         concurrency=1,
         text_category=args.text,
         chunking=args.chunking,
         stream=False,
+        max_tokens=args.max_tokens,
         gpu_monitor=gpu_monitor,
     )
@@
         level = await run_level(
             base_url=base_url,
             num_requests=num_requests,
             concurrency=conc,
             text_category=args.text,
             chunking=args.chunking,
             stream=args.stream,
+            max_tokens=args.max_tokens,
             gpu_monitor=gpu_monitor,
         )
@@
     parser.add_argument("--chunking", action="store_true", help="Enable server-side chunking", default=True)
     parser.add_argument("--no-chunking", action="store_false", dest="chunking", help="Disable chunking")
+    parser.add_argument(
+        "--max-tokens",
+        type=int,
+        default=256,
+        help="max_tokens passed to /v1/tts/generate (default: 256)",
+    )
     parser.add_argument("--stream", action="store_true", help="Use streaming endpoint")
*** End Patch


Output: {"output":"Success. Updated the following files:\nM /home/ubuntu/veenaModal/scripts/stress_test_optimized_local.py\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


Output: write_stdin failed: stdin is closed for this session; rerun exec_command with tty=true to keep stdin open


I added `--max-tokens` to the stress script (default 256) so short requests terminate quickly under local load. I’ll rerun benchmark at 1/5/10 concurrency and capture fresh timing JSON.


cd /home/ubuntu/veenaModal && source venv/bin/activate && python3 scripts/stress_test_optimized_local.py --levels 1,5,10 --text short --max-tokens 256 --output /home/ubuntu/veenaModal/stress_results_opt_local.json


Output:
Chunk ID: 789a9c
Wall time: 1.0013 seconds
Process running with session ID 3063
Original token count: 0
Output:


Output:
Chunk ID: 0efaa9
Wall time: 5.0006 seconds
Process running with session ID 3063
Original token count: 0
Output:


Output:
Chunk ID: 7cf67e
Wall time: 5.0025 seconds
Process running with session ID 3063
Original token count: 0
Output:


Output:
Chunk ID: 3fd852
Wall time: 5.0004 seconds
Process running with session ID 38524
Original token count: 156
Output:
INFO:     127.0.0.1:52194 - "GET /v1/tts/health HTTP/1.1" 200 OK
{"timestamp": "2026-02-17T00:20:07.130059+00:00", "level": "INFO", "logger": "veena3modal.tts", "message": "request_received", "request_id": "29ca9766-9436-442a-84d6-882bd5ce1b36", "event": "request_received", "text_length": 28, "speaker": "lipakshi", "stream": false, "format": "wav"}
2026-02-17 00:20:07 | INFO     | veena3modal.tts | request_received
2026-02-17 00:20:07 | INFO     | veena3modal.services.tts_runtime | tts_runtime_chunking_start
2026-02-17 00:20:07 | INFO     | veena3modal.processing.long_text_processor | LongTextProcessor initialized


I still see request starts entering chunking path and not returning, so I’m checking the chunking threshold in `LongTextProcessor` (it may treat short payloads as chunked and call multiple model runs). This likely explains the stall and must be aligned with the benchmark harness.


rg -n "def should_chunk|chunk_text\(|MIN_CHARS|max_length|length" /home/ubuntu/veenaModal/veena3modal/processing/long_text_processor.py


Output:
Chunk ID: 9baee2
Wall time: 0.0518 seconds
Process exited with code 0
Original token count: 253
Output:
10:- Unlimited text: Can process any length text
79:    # Maximum safe input length for model (characters)
85:    # Trigger chunking when text approaches max safe length
90:    # Use 75% of max safe length to ensure each chunk processes cleanly
111:        self.chunker = IndicSentenceChunker(max_chunk_length=self.CHUNK_SIZE)
116:                "max_model_input_length": self.MAX_MODEL_INPUT_LENGTH,
123:    def should_chunk(self, text: str) -> bool:
127:    def chunk_text(self, text: str) -> list:
131:        Delegates to self.chunker.chunk_text() for actual chunking logic.
140:        return self.chunker.chunk_text(text)
191:                "text_length": len(text),
199:        text_chunks = self.chunker.chunk_text(text)
229:                4096  # Cap at model's max context length
269:                        "chunk_length": len(chunk),
313:                "text_length": len(text),
391:                    "current_length_bytes": len(result),
432:        text_chunks = self.chunker.chunk_text(text)


sed -n '70,150p' /home/ubuntu/veenaModal/veena3modal/processing/long_text_processor.py


Output:
Chunk ID: 605f54
Wall time: 0.0505 seconds
Process exited with code 0
Original token count: 853
Output:
    - SNAC token sliding window is separate (already implemented)

    Max Input Length Reference (post-normalization):
    "इसके निर्माण में मुख्य वास्तुकार उस्ताद अहमद लाहौरी के नेतृत्व में लगभग
    20,000 कारीगरों और शिल्पकारों ने दिन-रात मेहनत की थी। इमारत के सामने बना
    'चारबाग' शैली का उद्यान और पानी की नहरें इसकी सुंदरता में चार चाँद लगा देते हैं।"
    = ~232 characters after normalization (numbers expanded to words)
    """

    # Maximum safe input length for model (characters)
    # Based on reference Hindi sentence (~232 chars after normalization)
    # Model struggles with longer inputs, causes token repetition
    MAX_MODEL_INPUT_LENGTH = 230  # ~232 chars is the reference max

    # Threshold for when to enable chunking (characters)
    # Trigger chunking when text approaches max safe length
    # Using 95% of max to leave some buffer
    CHUNKING_THRESHOLD = 220  # ~95% of MAX_MODEL_INPUT_LENGTH

    # Chunk size for long texts (characters)
    # Use 75% of max safe length to ensure each chunk processes cleanly
    # This gives model headroom for complex multilingual content
    CHUNK_SIZE = 170  # ~75% of MAX_MODEL_INPUT_LENGTH

    # Crossfade duration (milliseconds)
    CROSSFADE_MS = 50  # 50ms smooth transition

    def __init__(
        self,
        pipeline: SparkTTSPipeline,
        streaming_pipeline: Optional[Veena3SlidingWindowPipeline] = None
    ):
        """
        Initialize long text processor.

        Args:
            pipeline: Non-streaming TTS pipeline
            streaming_pipeline: Optional streaming pipeline (for streaming mode)
        """
        self.pipeline = pipeline
        self.streaming_pipeline = streaming_pipeline
        self.chunker = IndicSentenceChunker(max_chunk_length=self.CHUNK_SIZE)

        logger.info(
            f"LongTextProcessor initialized",
            extra={
                "max_model_input_length": self.MAX_MODEL_INPUT_LENGTH,
                "chunking_threshold": self.CHUNKING_THRESHOLD,
                "chunk_size": self.CHUNK_SIZE,
                "crossfade_ms": self.CROSSFADE_MS,
            }
        )

    def should_chunk(self, text: str) -> bool:
        """Determine if text should be chunked."""
        return len(text) > self.CHUNKING_THRESHOLD

    def chunk_text(self, text: str) -> list:
        """
        Chunk text using the internal chunker.

        Delegates to self.chunker.chunk_text() for actual chunking logic.
        This method is used by _stream_chunked_text in tts_runtime.

        Args:
            text: Text to chunk

        Returns:
            List of text chunks
        """
        return self.chunker.chunk_text(text)

    async def generate_with_chunking(
        self,
        text: str,
        speaker: str,
        temperature: float = 0.4,
        top_k: int = 50,  # Added for Spark TTS
        top_p: float = 0.9,
        max_tokens: int = 4096,
        repetition_penalty: float = 1.05,  # Prevent token repetition


Summary: auto


cd /home/ubuntu/veenaModal && source venv/bin/activate && python3 - <<'PY'
import httpx, time
payload={'text':'Hello, this is a quick test.','speaker':'lipakshi','stream':False,'chunking':True,'format':'wav','max_tokens':256}
start=time.time()
try:
    r=httpx.post('http://127.0.0.1:8000/v1/tts/generate', json=payload, timeout=120)
    print('status',r.status_code,'ms', (time.time()-start)*1000)
    print('x-llm-token-total',r.headers.get('x-llm-token-total'))
    print('x-llm-batch-gpu-ms',r.headers.get('x-llm-batch-gpu-ms'))
    print('len',len(r.content))
    print('error' , r.text[:120])
except Exception as e:
    print('err',type(e),e,'elapsed', (time.time()-start)*1000)
PY


Output:
Chunk ID: d368b1
Wall time: 1.0012 seconds
Process running with session ID 34726
Original token count: 0
Output:


Output:
Chunk ID: d66047
Wall time: 5.0010 seconds
Process running with session ID 34726
Original token count: 0
Output:


Output:
Chunk ID: f76a8e
Wall time: 5.0017 seconds
Process running with session ID 34726
Original token count: 0
Output:


Output:
Chunk ID: f726aa
Wall time: 5.0006 seconds
Process running with session ID 34726
Original token count: 0
Output:


Output:
Chunk ID: 9d52c1
Wall time: 5.0016 seconds
Process running with session ID 38524
Original token count: 140
Output:
{"timestamp": "2026-02-17T00:20:37.738031+00:00", "level": "INFO", "logger": "veena3modal.tts", "message": "request_received", "request_id": "a693f5c6-13ec-4972-859f-6a38351ec48e", "event": "request_received", "text_length": 28, "speaker": "lipakshi", "stream": false, "format": "wav"}
2026-02-17 00:20:37 | INFO     | veena3modal.tts | request_received
2026-02-17 00:20:37 | INFO     | veena3modal.services.tts_runtime | tts_runtime_chunking_start
2026-02-17 00:20:37 | INFO     | veena3modal.processing.long_text_processor | LongTextProcessor initialized


sed -n '380,760p' /home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py


Output:
Chunk ID: bd3939
Wall time: 0.0509 seconds
Process exited with code 0
Original token count: 3656
Output:
    Args:
        text: Text to synthesize (already normalized)
        speaker: Internal speaker name (resolved)
        temperature: Sampling temperature
        top_k: Top-k sampling
        top_p: Nucleus sampling
        max_tokens: Maximum tokens to generate
        repetition_penalty: Repetition penalty
        seed: Random seed for reproducibility
        output_sample_rate: "16khz" or "48khz" (triggers super-resolution)

    Returns:
        Tuple of (audio_bytes, metrics_dict)
        audio_bytes: WAV audio bytes (16kHz or 48kHz, 16-bit PCM) or None if failed
        metrics_dict: Dictionary with timing metrics

    Raises:
        RuntimeError: If runtime not initialized
    """
    if not is_initialized():
        raise RuntimeError("TTS runtime not initialized")

    runtime = get_runtime()
    metrics = {
        "request_id": request_id,
        "ttfb_ms": 0,
        "generation_ms": 0,
        "tokens_generated": 0,
        "audio_duration_seconds": 0.0,
        "sr_applied": False,
        "output_sample_rate": 16000,
    }

    if request_id is None:
        import uuid
        request_id = f"runtime-ns-{uuid.uuid4().hex[:10]}"
        metrics["request_id"] = request_id

    active_requests = _with_inflight_delta(1)
    start_time = time.time()

    pipeline_timings = {}

    try:
        logger.info(
            "tts_runtime_nonstream_start",
            extra={
                "speaker": speaker,
                "request_id": request_id,
                "text_length": len(text),
                "request_inflight": active_requests,
                "temperature": temperature,
                "top_k": top_k,
                "top_p": top_p,
                "max_tokens": max_tokens,
            },
        )

        gen_kwargs = dict(
            speaker=speaker,
            text=text,
            temperature=temperature,
            top_k=top_k,
            top_p=top_p,
            max_tokens=max_tokens,
            repetition_penalty=repetition_penalty,
            seed=seed,
        )

        if _timing_enabled():
            audio_bytes, pipeline_timings = await runtime.pipeline.generate_speech_indic(
                **gen_kwargs,
                return_timing=True,
                request_id=request_id,
            )
        else:
            audio_bytes = await runtime.pipeline.generate_speech_indic(
                **gen_kwargs,
                request_id=request_id,
            )

        generation_time = time.time() - start_time
        metrics["generation_ms"] = int(generation_time * 1000)
        metrics["ttfb_ms"] = metrics["generation_ms"]  # Non-streaming: TTFB = total time

        if pipeline_timings:
            metrics.update({
                "llm_token_total": pipeline_timings.get("token_total", 0),
                "llm_batch_count": pipeline_timings.get("batch_count", 0),
                "llm_batch_gpu_ms_total": pipeline_timings.get("llm_gpu_batch_ms_total", 0),
                "llm_batch_gpu_ms_min": pipeline_timings.get("llm_gpu_batch_ms_min", 0),
                "llm_batch_gpu_ms_max": pipeline_timings.get("llm_gpu_batch_ms_max", 0),
                "llm_batch_gpu_ms_p50": pipeline_timings.get("llm_gpu_batch_ms_p50", 0),
                "llm_decode_cpu_ms": pipeline_timings.get("decode_cpu_ms", 0),
                "llm_parse_ms": pipeline_timings.get("parse_ms", 0),
                "llm_wall_ms": pipeline_timings.get("llm_wall_ms", metrics["generation_ms"]),
                "ttfb_ms": int(pipeline_timings.get("ttfb_ms", metrics["ttfb_ms"])),
                "tokens_per_batch_min": pipeline_timings.get("tokens_per_batch_min"),
                "tokens_per_batch_max": pipeline_timings.get("tokens_per_batch_max"),
                "tokens_per_batch_p50": pipeline_timings.get("tokens_per_batch_p50"),
            })

        if audio_bytes:
            # Apply super-resolution if requested and available
            sample_rate = 16000
            logger.info(f"SR check: output_sample_rate={output_sample_rate!r}, "
                       f"sr_service={runtime.sr_service is not None}, "
                       f"is_loaded={runtime.sr_service.is_loaded if runtime.sr_service else False}")

            if output_sample_rate == "48khz" and runtime.sr_service and runtime.sr_service.is_loaded:
                try:
                    sr_start = time.time()
                    logger.info(f"Applying super-resolution to {len(audio_bytes)} bytes...")
                    audio_bytes = _apply_super_resolution(audio_bytes, runtime.sr_service)
                    sr_time = (time.time() - sr_start) * 1000
                    metrics["sr_ms"] = int(sr_time)
                    metrics["sr_applied"] = True
                    sample_rate = 48000
                    logger.info(f"✅ Super-resolution applied in {sr_time:.1f}ms, output={len(audio_bytes)} bytes")
                except Exception as e:
                    logger.warning(f"Super-resolution failed, returning 16kHz: {e}")
                    import traceback
                    traceback.print_exc()

            metrics["output_sample_rate"] = sample_rate

            # Calculate audio duration
            audio_duration = (len(audio_bytes) - 44) / (sample_rate * 2)  # -44 for WAV header
            metrics["audio_duration_seconds"] = max(0.0, audio_duration)

        return audio_bytes, metrics

    except Exception as e:
        logger.error(f"Speech generation failed: {e}")
        raise
    finally:
        active_requests = _with_inflight_delta(-1)
        if _timing_enabled():
            metrics["request_inflight"] = active_requests
            metrics["active_requests"] = active_requests
            logger.info(
                "tts_runtime_nonstream_complete",
                extra=metrics,
            )
        if _timing_enabled():
            logger.info(
                "tts_runtime_nonstream_inflight",
                extra={
                    "request_id": request_id,
                    "active_requests": active_requests,
                },
            )

def _apply_super_resolution(audio_bytes: bytes, sr_service) -> bytes:
    """
    Apply super-resolution to audio bytes.

    Args:
        audio_bytes: WAV audio at 16kHz
        sr_service: SuperResolutionService instance

    Returns:
        WAV audio at 48kHz
    """
    import numpy as np
    import struct
    import torch

    # Parse WAV header and extract PCM data
    if len(audio_bytes) < 44:
        raise ValueError("Invalid WAV data")

    # Extract PCM data (skip 44-byte WAV header)
    pcm_data = np.frombuffer(audio_bytes[44:], dtype=np.int16)

    # Convert to float32 for SR model
    audio_float = pcm_data.astype(np.float32) / 32768.0

    # Convert numpy to torch tensor for process_chunk
    audio_tensor = torch.from_numpy(audio_float)

    # Apply super-resolution using process_chunk (16kHz -> 48kHz)
    # process_chunk expects [batch, samples] or [samples] and returns [batch, samples]
    upsampled_tensor = sr_service.process_chunk(audio_tensor)

    # Convert back to numpy, squeeze batch dim if present
    upsampled = upsampled_tensor.squeeze().cpu().numpy()

    # Convert back to int16
    upsampled_int16 = np.clip(upsampled * 32768.0, -32768, 32767).astype(np.int16)

    # Create new WAV header for 48kHz
    sample_rate = 48000
    num_channels = 1
    bits_per_sample = 16
    byte_rate = sample_rate * num_channels * bits_per_sample // 8
    block_align = num_channels * bits_per_sample // 8
    data_size = len(upsampled_int16) * 2

    wav_header = struct.pack(
        '<4sI4s4sIHHIIHH4sI',
        b'RIFF',
        36 + data_size,
        b'WAVE',
        b'fmt ',
        16,  # fmt chunk size
        1,   # PCM format
        num_channels,
        sample_rate,
        byte_rate,
        block_align,
        bits_per_sample,
        b'data',
        data_size,
    )

    return wav_header + upsampled_int16.tobytes()

async def generate_speech_chunked(
    text: str,
    speaker: str,
    temperature: float = 0.8,
    top_k: int = 50,
    top_p: float = 1.0,
    max_tokens: int = 4096,
    repetition_penalty: float = 1.05,
    seed: Optional[int] = None,
    sample_rate: int = 16000,
    output_sample_rate: str = "16khz",
    request_id: Optional[str] = None,
) -> Tuple[Optional[bytes], Dict[str, Any]]:
    """
    Generate speech with automatic text chunking for long inputs.

    Uses LongTextProcessor to split text and stitch audio.

    Args:
        text: Text to synthesize
        speaker: Internal speaker name
        ... (same as generate_speech)
        sample_rate: Internal generation sample rate (always 16000)
        output_sample_rate: "16khz" or "48khz" (triggers super-resolution)

    Returns:
        Tuple of (audio_bytes, metrics_dict)
    """
    if not is_initialized():
        raise RuntimeError("TTS runtime not initialized")

    runtime = get_runtime()
    if request_id is None:
        import uuid
        request_id = f"runtime-chunk-{uuid.uuid4().hex[:10]}"
    metrics = {
        "request_id": request_id,
        "ttfb_ms": 0,
        "generation_ms": 0,
        "chunks_processed": 0,
        "audio_duration_seconds": 0.0,
        "text_chunked": False,
        "sr_applied": False,
        "output_sample_rate": 16000,
    }

    active_requests = _with_inflight_delta(1)
    start_time = time.time()

    try:
        if _timing_enabled():
            logger.info(
                "tts_runtime_chunking_start",
                extra={
                    "speaker": speaker,
                    "request_id": request_id,
                    "text_length": len(text),
                    "request_inflight": active_requests,
                    "chunking": True,
                },
            )
        # Import long text processor
        from veena3modal.processing.long_text_processor import LongTextProcessor

        long_processor = LongTextProcessor(pipeline=runtime.pipeline)

        # Check if chunking is needed
        if long_processor.should_chunk(text):
            metrics["text_chunked"] = True
            if _timing_enabled():
                chunk_result = await long_processor.generate_with_chunking(
                    text=text,
                    speaker=speaker,
                    temperature=temperature,
                    top_k=top_k,
                    top_p=top_p,
                    max_tokens=max_tokens,
                    repetition_penalty=repetition_penalty,
                    seed=seed,
                    sample_rate=sample_rate,
                    request_id=request_id,
                    return_timing=True,
                )
                if isinstance(chunk_result, tuple):
                    audio_bytes, chunk_timing = chunk_result
                else:
                    audio_bytes = chunk_result
                    chunk_timing = {}
            else:
                audio_bytes = await long_processor.generate_with_chunking(
                    text=text,
                    speaker=speaker,
                    temperature=temperature,
                    top_k=top_k,
                    top_p=top_p,
                    max_tokens=max_tokens,
                    repetition_penalty=repetition_penalty,
                    seed=seed,
                    sample_rate=sample_rate,
                    request_id=request_id,
                )
                chunk_timing = {}
        else:
            # Short text: direct generation
            gen_kwargs = dict(
                speaker=speaker,
                text=text,
                temperature=temperature,
                top_k=top_k,
                top_p=top_p,
                max_tokens=max_tokens,
                repetition_penalty=repetition_penalty,
                seed=seed,
            )
            if _timing_enabled():
                audio_bytes, pipeline_timings = await runtime.pipeline.generate_speech_indic(
                    **gen_kwargs,
                    return_timing=True,
                    request_id=request_id,
                )
                chunk_timing = pipeline_timings
            else:
                audio_bytes = await runtime.pipeline.generate_speech_indic(
                    **gen_kwargs,
                    request_id=request_id,
                )
                chunk_timing = {}

        generation_time = time.time() - start_time
        metrics["generation_ms"] = int(generation_time * 1000)
        metrics["ttfb_ms"] = metrics["generation_ms"]

        if chunk_timing:
            metrics.update({
                "llm_token_total": chunk_timing.get("token_total", 0),
                "llm_batch_count": chunk_timing.get("batch_count", 0),
                "llm_batch_gpu_ms_total": chunk_timing.get("llm_gpu_batch_ms_total", 0),
                "llm_batch_wall_ms_total": chunk_timing.get("llm_wall_ms", metrics["generation_ms"]),
                "llm_parse_ms": chunk_timing.get("parse_ms", 0),
                "llm_decode_cpu_ms": chunk_timing.get("decode_cpu_ms", 0),
                "llm_batch_wall_ms_min": chunk_timing.get("llm_batch_wall_ms_min", 0),
                "llm_batch_wall_ms_max": chunk_timing.get("llm_batch_wall_ms_max", 0),
                "llm_batch_gpu_ms_min": chunk_timing.get("llm_batch_gpu_ms_min", 0),
                "llm_batch_gpu_ms_max": chunk_timing.get("llm_batch_gpu_ms_max", 0),
                "llm_batch_gpu_ms_p50": chunk_timing.get("llm_batch_gpu_ms_p50", 0),
                "tokens_per_batch_min": chunk_timing.get("tokens_per_batch_min"),
                "tokens_per_batch_max": chunk_timing.get("tokens_per_batch_max"),
                "tokens_per_batch_p50": chunk_timing.get("tokens_per_batch_p50"),
                "llm_batch_tokens": chunk_timing.get("token_batches", []),
            })
            if metrics["text_chunked"]:
                metrics["chunks_processed"] = chunk_timing.get("chunks_processed", 0)

        if audio_bytes:
            # Apply super-resolution if requested and available
            final_sample_rate = 16000
            if output_sample_rate == "48khz" and runtime.sr_service and runtime.sr_service.is_loaded:
                try:
                    sr_start = time.time()
                    logger.info(f"Applying super-resolution to chunked audio ({len(audio_bytes)} bytes)...")
                    audio_bytes = _apply_super_resolution(audio_bytes, runtime.sr_service)


rg -n "TRAINING_STOP_TOKEN_IDS|stop token|MAX_OUTPUT_TOKENS|max_tokens" /home/ubuntu/veenaModal/veena3modal/core/pipeline.py /home/ubuntu/veenaModal/veena3modal/core/*.py


Output:
Chunk ID: 0d0c02
Wall time: 0.0514 seconds
Process exited with code 0
Original token count: 1059
Output:
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:39:    TRAINING_STOP_TOKEN_IDS,
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:102:        max_tokens: int = DEFAULT_MAX_TOKENS,
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:116:            max_tokens: Max SNAC tokens to generate
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:134:            max_tokens=max_tokens,
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:231:        max_tokens: int = DEFAULT_MAX_TOKENS,
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:251:            max_tokens: Max SNAC tokens to generate
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:275:        # OPTIMIZATION: Use Spark TTS stop token, not legacy SNAC CODE_END_TOKEN_ID (128258)
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:276:        # which doesn't exist in the Spark vocab and would cause generation to run to max_tokens
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:281:            max_tokens=max_tokens,
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:284:            stop=TRAINING_STOP_TOKEN_IDS,  # "<|im_end|>" - matches non-streaming pipeline
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:591:        max_tokens: int = DEFAULT_MAX_TOKENS,
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:618:            max_tokens: Max tokens to generate
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:654:            max_tokens=max_tokens,
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:657:            stop=TRAINING_STOP_TOKEN_IDS,  # "<|im_end|>" - matches non-streaming pipeline
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:954:        max_tokens: int = DEFAULT_MAX_TOKENS,
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:980:            max_tokens: Max tokens to generate
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:1021:        # OPTIMIZATION: Use Spark TTS stop token, not legacy SNAC CODE_END_TOKEN_ID
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:1026:            max_tokens=max_tokens,
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:1029:            stop=TRAINING_STOP_TOKEN_IDS,  # "<|im_end|>" - matches non-streaming pipeline
/home/ubuntu/veenaModal/veena3modal/core/pipeline.py:21:    TRAINING_STOP_TOKEN_IDS,
/home/ubuntu/veenaModal/veena3modal/core/pipeline.py:75:        max_tokens: int = DEFAULT_MAX_TOKENS,
/home/ubuntu/veenaModal/veena3modal/core/pipeline.py:93:            max_tokens: Max BiCodec tokens to generate (default: 2048)
/home/ubuntu/veenaModal/veena3modal/core/pipeline.py:109:            max_tokens=max_tokens,
/home/ubuntu/veenaModal/veena3modal/core/pipeline.py:111:            stop=TRAINING_STOP_TOKEN_IDS,
/home/ubuntu/veenaModal/veena3modal/core/pipeline.py:328:        max_tokens: int = DEFAULT_MAX_TOKENS,
/home/ubuntu/veenaModal/veena3modal/core/pipeline.py:350:            max_tokens: Max BiCodec tokens to generate
/home/ubuntu/veenaModal/veena3modal/core/pipeline.py:365:            max_tokens=max_tokens,
/home/ubuntu/veenaModal/veena3modal/core/pipeline.py:21:    TRAINING_STOP_TOKEN_IDS,
/home/ubuntu/veenaModal/veena3modal/core/pipeline.py:75:        max_tokens: int = DEFAULT_MAX_TOKENS,
/home/ubuntu/veenaModal/veena3modal/core/pipeline.py:93:            max_tokens: Max BiCodec tokens to generate (default: 2048)
/home/ubuntu/veenaModal/veena3modal/core/pipeline.py:109:            max_tokens=max_tokens,
/home/ubuntu/veenaModal/veena3modal/core/pipeline.py:111:            stop=TRAINING_STOP_TOKEN_IDS,
/home/ubuntu/veenaModal/veena3modal/core/pipeline.py:328:        max_tokens: int = DEFAULT_MAX_TOKENS,
/home/ubuntu/veenaModal/veena3modal/core/pipeline.py:350:            max_tokens: Max BiCodec tokens to generate
/home/ubuntu/veenaModal/veena3modal/core/pipeline.py:365:            max_tokens=max_tokens,
/home/ubuntu/veenaModal/veena3modal/core/constants.py:11:# Spark TTS uses <|im_end|> as stop token
/home/ubuntu/veenaModal/veena3modal/core/constants.py:12:TRAINING_STOP_TOKEN_IDS = ["<|im_end|>"]  # String-based stop token for Spark TTS


cd /home/ubuntu/veenaModal && source venv/bin/activate && python3 - <<'PY'
import httpx, time
payload={'text':'Hello, this is a quick test.','speaker':'lipakshi','stream':False,'chunking':False,'format':'wav','max_tokens':20}
start=time.time()
try:
    r=httpx.post('http://127.0.0.1:8000/v1/tts/generate', json=payload, timeout=120)
    print('status',r.status_code,'ms', (time.time()-start)*1000)
    print('len',len(r.content))
    print('x-llm-token-total',r.headers.get('x-llm-token-total'))
    print('x-llm-batch-count',r.headers.get('x-llm-batch-count'))
except Exception as e:
    print('err',type(e),e,'elapsed', (time.time()-start)*1000)
PY


Output:
Chunk ID: bc692f
Wall time: 0.1531 seconds
Process exited with code 0
Original token count: 22
Output:
status 400 ms 95.25012969970703
len 350
x-llm-token-total None
x-llm-batch-count None


cd /home/ubuntu/veenaModal && source venv/bin/activate && python3 - <<'PY'
import httpx
payload={'text':'Hello, this is a quick test.','speaker':'lipakshi','stream':False,'chunking':False,'format':'wav','max_tokens':20}
r=httpx.post('http://127.0.0.1:8000/v1/tts/generate', json=payload, timeout=20)
print(r.status_code)
print(r.text)
print(r.headers.get('x-request-id'))
PY


Output:
Chunk ID: 5fa131
Wall time: 0.2316 seconds
Process exited with code 0
Original token count: 98
Output:
400
{"error":{"code":"VALIDATION_ERROR","message":"1 validation error for TTSGenerateRequest\nmax_tokens\n  Input should be greater than or equal to 128 [type=greater_than_equal, input_value=20, input_type=int]\n    For further information visit https://errors.pydantic.dev/2.12/v/greater_than_equal","request_id":"4112c521-3fe1-4e82-af8f-ec4ca8b21d59"}}
4112c521-3fe1-4e82-af8f-ec4ca8b21d59


cd /home/ubuntu/veenaModal && source venv/bin/activate && python3 - <<'PY'
import httpx, time
payload={'text':'Hello, this is a quick test.','speaker':'lipakshi','stream':False,'chunking':False,'format':'wav','max_tokens':128}
start=time.time()
try:
    r=httpx.post('http://127.0.0.1:8000/v1/tts/generate', json=payload, timeout=120)
    print('status',r.status_code,'ms', (time.time()-start)*1000)
    print('len',len(r.content))
    print('error', r.text[:200])
except Exception as e:
    print('err',type(e),e,'elapsed', (time.time()-start)*1000)
PY


Output:
Chunk ID: ef6cda
Wall time: 10.0057 seconds
Process running with session ID 71366
Original token count: 0
Output:


Output:
Chunk ID: c126c8
Wall time: 5.0007 seconds
Process running with session ID 71366
Original token count: 0
Output:


Output:
Chunk ID: 75938a
Wall time: 5.0047 seconds
Process running with session ID 38524
Original token count: 290
Output:
INFO:     127.0.0.1:60846 - "POST /v1/tts/generate HTTP/1.1" 400 Bad Request
INFO:     127.0.0.1:44742 - "POST /v1/tts/generate HTTP/1.1" 400 Bad Request
{"timestamp": "2026-02-17T00:21:18.745843+00:00", "level": "INFO", "logger": "veena3modal.tts", "message": "request_received", "request_id": "1dae57be-0671-4bf8-924d-951d26076e8e", "event": "request_received", "text_length": 28, "speaker": "lipakshi", "stream": false, "format": "wav"}
2026-02-17 00:21:18 | INFO     | veena3modal.tts | request_received
2026-02-17 00:21:18 | INFO     | veena3modal.services.tts_runtime | tts_runtime_nonstream_start
{"timestamp": "2026-02-17T00:21:41.385723+00:00", "level": "INFO", "logger": "veena3modal.tts", "message": "request_received", "request_id": "c2a1744f-ed61-45d3-82e6-64c0efd69ae8", "event": "request_received", "text_length": 28, "speaker": "vardan", "stream": false, "format": "wav"}
2026-02-17 00:21:41 | INFO     | veena3modal.tts | request_received
2026-02-17 00:21:41 | INFO     | veena3modal.services.tts_runtime | tts_runtime_chunking_start
2026-02-17 00:21:41 | INFO     | veena3modal.processing.long_text_processor | LongTextProcessor initialized


Output:
Chunk ID: 8f06a8
Wall time: 0.0516 seconds
Process exited with code 0
Original token count: 4
Output:
8, 46994, 31


pkill -f 'veena3modal.local_server' || true; pkill -f 'local_server:create_local_app' || true; sleep 1; lsof -i :8000 || true


Output:
Chunk ID: 05c222
Wall time: 0.0512 seconds
Process exited with code -1
Original token count: 0
Output:


cd /home/ubuntu/veenaModal && source venv/bin/activate && python3 -m veena3modal.local_server --skip-download --model-path /home/ubuntu/veenaModal/models/spark_tts_4speaker --port 8000 --gpu-memory 0.25 --num-engines 1 --log-level INFO


Output:
Chunk ID: 86898a
Wall time: 1.0009 seconds
Process running with session ID 95901
Original token count: 454
Output:
2026-02-17 00:21:48 | INFO     | veena3.local | Auth bypass enabled (local mode). Use --auth to require API keys.
2026-02-17 00:21:48 | INFO     | veena3.local | Resolved model paths:
2026-02-17 00:21:48 | INFO     | veena3.local |   LLM (vLLM):    /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
2026-02-17 00:21:48 | INFO     | veena3.local |   BiCodec:       /home/ubuntu/veenaModal/models/spark_tts_4speaker
2026-02-17 00:21:48 | INFO     | veena3.local | ============================================================
2026-02-17 00:21:48 | INFO     | veena3.local | Initializing Veena3 TTS Runtime (local)
2026-02-17 00:21:48 | INFO     | veena3.local |   Model:       /home/ubuntu/veenaModal/models/spark_tts_4speaker
2026-02-17 00:21:48 | INFO     | veena3.local |   LLM path:    /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
2026-02-17 00:21:48 | INFO     | veena3.local |   BiCodec:     /home/ubuntu/veenaModal/models/spark_tts_4speaker
2026-02-17 00:21:48 | INFO     | veena3.local |   Device:      cuda
2026-02-17 00:21:48 | INFO     | veena3.local |   GPU Memory:  25% (total)
2026-02-17 00:21:48 | INFO     | veena3.local |   Engines:     1
2026-02-17 00:21:48 | INFO     | veena3.local |   SR:          disabled
2026-02-17 00:21:48 | INFO     | veena3.local | ============================================================
2026-02-17 00:21:48 | INFO     | veena3modal.services.tts_runtime | Initializing TTS runtime...
2026-02-17 00:21:48 | INFO     | veena3modal.services.tts_runtime |   Model path: /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
2026-02-17 00:21:48 | INFO     | veena3modal.services.tts_runtime |   BiCodec path: /home/ubuntu/veenaModal/models/spark_tts_4speaker
2026-02-17 00:21:48 | INFO     | veena3modal.services.tts_runtime |   SR enabled: False, path: None