Output:
Chunk ID: f69b33
Wall time: 30.0012 seconds
Process running with session ID 92042
Original token count: 2505
Output:
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
🚀 Initializing Spark TTS Model
📁 Model path: /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
🗣  Speakers: 12 predefined (lipakshi, vardan, reet, Nandini, krishna, anika, adarsh, Nilay, Aarvi, Asha, Bittu, Mira)
🎭 Emotions: 10 tags in [bracket] format

📝 Loading tokenizer from model...
✅ Tokenizer loaded: 165168 tokens

🔍 Validating 12 speakers for Spark TTS...
  lipakshi → <|speaker_0|>: 7 tokens
  vardan → <|speaker_1|>: 7 tokens
  reet → <|speaker_2|>: 7 tokens
  Nandini → <|speaker_3|>: 7 tokens
  krishna → <|speaker_4|>: 7 tokens
  anika → <|speaker_5|>: 7 tokens
  adarsh → <|speaker_6|>: 7 tokens
  Nilay → <|speaker_7|>: 7 tokens
  Aarvi → <|speaker_8|>: 7 tokens
  Asha → <|speaker_9|>: 7 tokens
  Bittu → <|speaker_10|>: 8 tokens
  Mira → <|speaker_11|>: 8 tokens
✅ All 12 speakers validated with mapping

🔍 Validating 10 emotion tags for Spark TTS...
✅ All 10 emotion tags validated

📋 Sample emotion tags:
  [angry]: 1 tokens
  [curious]: 1 tokens
  [excited]: 1 tokens
  [giggle]: 1 tokens
  [laughs harder]: 1 tokens
  ...

🔧 Initializing vLLM engine with optimizations...
  - dtype: bfloat16
  - max_model_len: 4096
  - gpu_memory_utilization: 0.15
  - enable_prefix_caching: True
  - enforce_eager (disable CUDA graphs): False
INFO 02-17 00:52:49 [model.py:541] Resolved architecture: Qwen2ForCausalLM
INFO 02-17 00:52:49 [model.py:1561] Using max model len 4096
INFO 02-17 00:52:49 [scheduler.py:226] Chunked prefill is enabled with max_num_batched_tokens=4096.
INFO 02-17 00:52:49 [vllm.py:624] Asynchronous scheduling is enabled.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(EngineCore_DP0 pid=3720359) INFO 02-17 00:52:50 [core.py:96] Initializing a V1 LLM engine (v0.15.1) with config: model='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', speculative_config=None, tokenizer='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [4096], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 256, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': True}, 'local_cache_dir': None, 'static_all_moe_layers': []}
(EngineCore_DP0 pid=3720359) INFO 02-17 00:52:53 [parallel_state.py:1212] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://216.81.248.184:34085 backend=nccl
(EngineCore_DP0 pid=3720359) INFO 02-17 00:52:53 [parallel_state.py:1423] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
(EngineCore_DP0 pid=3720359) INFO 02-17 00:52:53 [gpu_model_runner.py:4033] Starting to load model /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM...
(EngineCore_DP0 pid=3720359) INFO 02-17 00:52:54 [cuda.py:364] Using FLASH_ATTN attention backend out of potential backends: ('FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION')
(EngineCore_DP0 pid=3720359) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
(EngineCore_DP0 pid=3720359) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
(EngineCore_DP0 pid=3720359)
Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
(EngineCore_DP0 pid=3720359)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.81it/s]
(EngineCore_DP0 pid=3720359)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.81it/s]
(EngineCore_DP0 pid=3720359)
(EngineCore_DP0 pid=3720359) INFO 02-17 00:52:55 [default_loader.py:291] Loading weights took 0.62 seconds
(EngineCore_DP0 pid=3720359) INFO 02-17 00:52:55 [gpu_model_runner.py:4130] Model loading took 0.95 GiB memory and 1.127297 seconds
(EngineCore_DP0 pid=3720359) INFO 02-17 00:53:00 [backends.py:812] Using cache directory: /home/ubuntu/.cache/vllm/torch_compile_cache/f3ff13511c/rank_0_0/backbone for vLLM's torch.compile
(EngineCore_DP0 pid=3720359) INFO 02-17 00:53:00 [backends.py:872] Dynamo bytecode transform time: 4.17 s
(EngineCore_DP0 pid=3720359) INFO 02-17 00:53:02 [backends.py:267] Directly load the compiled graph(s) for compile range (1, 4096) from the cache, took 0.609 s
(EngineCore_DP0 pid=3720359) INFO 02-17 00:53:02 [monitor.py:34] torch.compile takes 4.78 s in total
(EngineCore_DP0 pid=3720359) INFO 02-17 00:53:03 [gpu_worker.py:356] Available KV cache memory: 10.13 GiB
(EngineCore_DP0 pid=3720359) INFO 02-17 00:53:03 [kv_cache_utils.py:1307] GPU KV cache size: 885,184 tokens
(EngineCore_DP0 pid=3720359) INFO 02-17 00:53:03 [kv_cache_utils.py:1312] Maximum concurrency for 4,096 tokens per request: 216.11x
(EngineCore_DP0 pid=3720359)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   0%|          | 0/35 [00:00<?, ?it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  11%|█▏        | 4/35 [00:00<00:00, 33.69it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  23%|██▎       | 8/35 [00:00<00:00, 36.33it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  34%|███▍      | 12/35 [00:00<00:00, 37.46it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  46%|████▌     | 16/35 [00:00<00:00, 37.87it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  57%|█████▋    | 20/35 [00:00<00:00, 37.98it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  69%|██████▊   | 24/35 [00:00<00:00, 37.90it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  80%|████████  | 28/35 [00:00<00:00, 37.88it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  91%|█████████▏| 32/35 [00:00<00:00, 36.17it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 35/35 [00:00<00:00, 36.18it/s]
(EngineCore_DP0 pid=3720359)
Capturing CUDA graphs (decode, FULL):   0%|          | 0/19 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL):  21%|██        | 4/19 [00:00<00:00, 31.77it/s]
Capturing CUDA graphs (decode, FULL):  42%|████▏     | 8/19 [00:00<00:00, 34.96it/s]
Capturing CUDA graphs (decode, FULL):  63%|██████▎   | 12/19 [00:00<00:00, 35.28it/s]
Capturing CUDA graphs (decode, FULL):  84%|████████▍ | 16/19 [00:00<00:00, 35.63it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 35.49it/s]
(EngineCore_DP0 pid=3720359) INFO 02-17 00:53:06 [gpu_model_runner.py:5063] Graph capturing finished in 2 secs, took 0.28 GiB
(EngineCore_DP0 pid=3720359) INFO 02-17 00:53:06 [core.py:272] init engine (profile, create kv cache, warmup model) took 10.35 seconds
/home/ubuntu/.local/lib/python3.10/site-packages/torch/nn/utils/weight_norm.py:144: FutureWarning: `torch.nn.utils.weight_norm` is deprecated in favor of `torch.nn.utils.parametrizations.weight_norm`.
  WeightNorm.apply(module, name, dim)
[2026-02-17 00:53:13] INFO pipeline.py:149: tts_llm_generate_start


# Collaboration Mode: Default

You are now in Default mode. Any previous instructions for other modes (e.g. Plan mode) are no longer active.

Your active mode changes only when new developer instructions with a different `<collaboration_mode>...</collaboration_mode>` change it; user requests or tool descriptions do not change mode by themselves. Known mode names are Default and Plan.

## request_user_input availability

The `request_user_input` tool is unavailable in Default mode. If you call it while in Default mode, it will return an error.

If a decision is necessary and cannot be discovered from local context, ask the user directly. However, in Default mode you should strongly prefer executing the user's request rather than stopping to ask questions.

Summary: auto

## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) When `SKILL.md` references relative paths (e.g., `scripts/foo.py`), resolve them relative to the skill directory listed above first, and only consider other paths if needed.
  3) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  4) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  5) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.


Output:
Chunk ID: c4a027
Wall time: 1.2539 seconds
Process exited with code 0
Original token count: 1412
Output:
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:13] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:14] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:14] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:14] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:14] INFO pipeline.py:216: tts_llm_batch
[2026-02-17 00:53:14] INFO pipeline.py:354: tts_llm_complete
✅ vLLM engine initialized successfully!

🎉 Spark TTS Model ready for inference!

🎵 Loading BiCodec audio tokenizer from /home/ubuntu/veenaModal/models/spark_tts_4speaker...
   Device: cuda
Missing tensor: mel_transformer.spectrogram.window
Missing tensor: mel_transformer.mel_scale.fb
✅ BiCodec decoder initialized (sample rate: 16000Hz)
🚀 SparkTTSPipeline initialized
ok 34604 89 89
elapsed 1.2629315853118896


Summary: auto


I found that the standalone pipeline path itself can complete quickly, so I’m now checking the FastAPI handler and `generate_speech_chunked` call path for any request-layer differences (e.g., hidden defaults, payload coercion, or incorrect request flags).


cd /home/ubuntu/veenaModal && rg -n "@app.post\(\"/v1/tts/generate\"|async def tts_generate|generate_speech_chunked\(|generate_speech\(" veena3modal/api/fastapi_app.py


cd /home/ubuntu/veenaModal && sed -n '1,220p' veena3modal/api/fastapi_app.py


cd /home/ubuntu/veenaModal && sed -n '220,560p' veena3modal/api/fastapi_app.py


Output:
Chunk ID: b37147
Wall time: 0.0516 seconds
Process exited with code 0
Original token count: 63
Output:
302:    @app.post("/v1/tts/generate")
303:    async def tts_generate(request: Request):
446:                audio_bytes, metrics = await tts_runtime.generate_speech_chunked(
460:                audio_bytes, metrics = await tts_runtime.generate_speech(


Output:
Chunk ID: ab2996
Wall time: 0.0504 seconds
Process exited with code 0
Original token count: 1877
Output:
"""
FastAPI ASGI app factory for Modal deployment.

Phase 1: expose `/v1/tts/generate` and `/v1/tts/health` while reusing existing
Veena3 inference/streaming code from the Django project.

Phase 2+: extract shared code into `veena3modal/shared/` and remove Django coupling.
"""

# NOTE: Intentionally NOT using `from __future__ import annotations` here
# to ensure FastAPI recognizes Request type annotations properly

import time
import uuid
import json
from typing import Dict, Optional

# Structured logging (JSON formatted, request-scoped)
from veena3modal.shared.logging import (
    get_logger,
    set_request_context,
    clear_request_context,
    log_request_received,
    log_first_audio_emitted,
    log_request_completed,
    log_request_failed,
)

# Prometheus metrics
from veena3modal.shared.metrics import (
    record_request_received as metrics_request_received,
    record_request_completed as metrics_request_completed,
    record_request_failed as metrics_request_failed,
    record_ttfb,
    record_rtf,
    record_audio_duration,
    record_chunks_sent,
    get_metrics_text,
    get_content_type as get_metrics_content_type,
)

# Auth and rate limiting
from veena3modal.api.auth import (
    get_api_validator,
    extract_api_key,
    hash_api_key,
)
from veena3modal.api.rate_limiter import get_rate_limiter
from veena3modal.api.error_handlers import (
    ErrorCode,
    create_error_response,
    get_error_status,
    is_gpu_fault,
    handle_gpu_fault,
    FeatureFlags,
)

logger = get_logger(__name__)

# Sentence storage (fire-and-forget, non-blocking)
# Imported at module level for singleton access
_sentence_store = None

def _get_sentence_store():
    """Lazy load sentence store singleton."""
    global _sentence_store
    if _sentence_store is None:
        from veena3modal.services.sentence_store import get_sentence_store
        _sentence_store = get_sentence_store()
    return _sentence_store

def _build_perf_headers(metrics: Dict[str, object]) -> Dict[str, str]:
    """Build compact timing headers for benchmark tooling."""
    perf: Dict[str, str] = {}
    if not metrics:
        return perf

    def _pick(*keys: str):
        for key in keys:
            if key in metrics and metrics[key] is not None:
                return metrics[key]
        return None

    def _add(name: str, value: object):
        if value is None:
            return
        if isinstance(value, bool):
            perf[name] = str(value).lower()
        else:
            perf[name] = str(value)

    llm_token_total = _pick("llm_token_total", "token_total")
    llm_batch_count = _pick("llm_batch_count", "batch_count")
    generation_ms = _pick("generation_ms") or _pick("total_ms")
    llm_batch_wall_total = _pick("llm_batch_wall_ms_total", "llm_wall_ms", "total_ms")
    llm_batch_gpu_total = _pick("llm_batch_gpu_ms_total", "llm_gpu_batch_ms_total")
    llm_decode_wall_total = _pick("llm_decode_wall_ms_total")
    llm_decode_gpu_total = _pick("llm_decode_gpu_ms_total")
    llm_parse_ms = _pick("llm_parse_ms")
    decode_cpu_ms = _pick("llm_decode_cpu_ms")
    parse_ms = _pick("parse_ms")

    _add("X-LLM-Token-Total", llm_token_total)
    _add("X-LLM-Batch-Count", llm_batch_count)
    _add("X-LLM-Batch-Wall-Ms", llm_batch_wall_total)
    _add("X-LLM-Batch-GPU-Ms", llm_batch_gpu_total)
    _add("X-LLM-Batch-Wall-Ms-Min", _pick("llm_batch_wall_ms_min"))
    _add("X-LLM-Batch-Wall-Ms-Max", _pick("llm_batch_wall_ms_max"))
    _add("X-LLM-Batch-Wall-Ms-P50", _pick("llm_batch_wall_ms_p50"))
    _add("X-LLM-Batch-GPU-Ms-Min", _pick("llm_batch_gpu_ms_min"))
    _add("X-LLM-Batch-GPU-Ms-Max", _pick("llm_batch_gpu_ms_max"))
    _add("X-LLM-Batch-GPU-Ms-P50", _pick("llm_batch_gpu_ms_p50"))
    _add("X-LLM-Decode-Wall-Ms", llm_decode_wall_total)
    _add("X-LLM-Decode-GPU-Ms", llm_decode_gpu_total)
    _add("X-LLM-Decode-Wall-Ms-Min", _pick("llm_decode_wall_ms_min"))
    _add("X-LLM-Decode-Wall-Ms-Max", _pick("llm_decode_wall_ms_max"))
    _add("X-LLM-Decode-Wall-Ms-P50", _pick("llm_decode_wall_ms_p50"))
    _add("X-LLM-Decode-GPU-Ms-Min", _pick("llm_decode_gpu_ms_min"))
    _add("X-LLM-Decode-GPU-Ms-Max", _pick("llm_decode_gpu_ms_max"))
    _add("X-LLM-Decode-GPU-Ms-P50", _pick("llm_decode_gpu_ms_p50"))
    _add("X-LLM-Decode-Calls", _pick("decode_calls"))
    _add("X-LLM-Decode-CPU-Ms", decode_cpu_ms)
    _add("X-LLM-Parse-Ms", llm_parse_ms or parse_ms)
    _add("X-LLM-Parse-Avg-Ms", _pick("llm_parse_ms_avg"))
    _add("X-LLM-Tokens-Per-Batch-Min", _pick("tokens_per_batch_min"))
    _add("X-LLM-Tokens-Per-Batch-Max", _pick("tokens_per_batch_max"))
    _add("X-LLM-Tokens-Per-Batch-P50", _pick("tokens_per_batch_p50"))
    _add("X-Generation-Ms", generation_ms)
    _add("X-LLM-Generation-Ms", _pick("llm_wall_ms", generation_ms))
    _add("X-Request-Inflight", _pick("request_inflight", "active_requests"))
    _add("X-LLM-Text-Chunked", _pick("text_chunked"))
    _add("X-LLM-Chunks-Processed", _pick("chunks_processed"))
    _add("X-LLM-SR-Applied", _pick("sr_applied"))

    if llm_token_total and generation_ms:
        try:
            perf["X-LLM-Time-Per-Token-Ms"] = f"{float(generation_ms) / float(llm_token_total):.3f}"
        except Exception:
            pass

    if llm_batch_count:
        try:
            perf["X-LLM-Time-Per-Batch-Wall-Ms"] = f"{float(generation_ms or 0) / float(llm_batch_count):.3f}"
        except Exception:
            pass
        try:
            perf["X-LLM-Time-Per-Batch-GPU-Ms"] = f"{float(llm_batch_gpu_total or 0) / float(llm_batch_count):.3f}"
        except Exception:
            pass

    # Include compact JSON snapshot for downstream benchmark tooling.
    detail_keys = [
        "request_id",
        "speaker",
        "text_length",
        "llm_token_total",
        "llm_batch_count",
        "llm_batch_wall_ms_total",
        "llm_batch_gpu_ms_total",
        "llm_batch_wall_ms_min",
        "llm_batch_wall_ms_max",
        "llm_batch_wall_ms_p50",
        "llm_batch_gpu_ms_min",
        "llm_batch_gpu_ms_max",
        "llm_batch_gpu_ms_p50",
        "llm_decode_cpu_ms",
        "llm_decode_wall_ms_total",
        "llm_decode_gpu_ms_total",
        "llm_decode_wall_ms_min",
        "llm_decode_wall_ms_max",
        "llm_decode_wall_ms_p50",
        "llm_decode_gpu_ms_min",
        "llm_decode_gpu_ms_max",
        "llm_decode_gpu_ms_p50",
        "decode_calls",
        "decode_cpu_ms",
        "parse_ms",
        "llm_parse_ms",
        "llm_parse_ms_avg",
        "llm_batch_tokens",
        "tokens_per_batch_min",
        "tokens_per_batch_max",
        "tokens_per_batch_p50",
        "generation_ms",
        "audio_duration_seconds",
        "request_inflight",
        "active_requests",
        "chunks_processed",
        "text_chunked",
        "sr_applied",
    ]
    perf["X-Perf-Details"] = json.dumps(
        {k: metrics[k] for k in detail_keys if k in metrics},
        separators=(",", ":"),
    )

    return perf

# Version info (will be set dynamically after model load)
_APP_VERSION = "0.1.0"
_MODEL_VERSION: Optional[str] = None
_startup_time: Optional[float] = None

def get_model_version() -> str:
    """Return the current model version, or 'not_loaded' if not initialized."""
    return _MODEL_VERSION or "not_loaded"

def set_model_version(version: str) -> None:
    """Set the model version after loading."""
    global _MODEL_VERSION
    _MODEL_VERSION = version

def create_app():
    """
    Factory function to create FastAPI app.


Output:
Chunk ID: fdb328
Wall time: 0.0512 seconds
Process exited with code 0
Original token count: 3355
Output:
    Factory function to create FastAPI app.

    Import inside factory so local tooling can import this module without FastAPI installed.
    """
    from fastapi import FastAPI, HTTPException, Request
    from fastapi.responses import JSONResponse, Response

    global _startup_time
    _startup_time = time.time()

    app = FastAPI(
        title="Veena3 TTS (Modal)",
        version=_APP_VERSION,
        description="High-quality multilingual Text-to-Speech API with true streaming support.",
    )

    @app.get("/v1/tts/health")
    async def tts_health():
        """
        Health check endpoint.

        Returns:
            - status: "healthy" | "degraded" | "unhealthy"
            - model_loaded: whether TTS model is initialized
            - model_version: version string of loaded model
            - uptime_seconds: time since app startup
            - gpu_available: whether GPU is detected (best-effort check)
        """
        # Check model status via runtime
        from veena3modal.services import tts_runtime
        model_loaded = tts_runtime.is_initialized()

        # Use runtime's model version if available
        runtime = tts_runtime.get_runtime()
        model_version = runtime.model_version if runtime else get_model_version()

        # Check GPU availability (best-effort, non-blocking)
        gpu_available = False
        try:
            import torch
            gpu_available = torch.cuda.is_available()
        except ImportError:
            pass  # torch not installed, skip GPU check

        # Compute uptime
        uptime = time.time() - _startup_time if _startup_time else 0

        # Determine overall status
        if model_loaded and gpu_available:
            status = "healthy"
        elif model_loaded or gpu_available:
            status = "degraded"
        else:
            status = "degraded"

        return JSONResponse(
            content={
                "status": status,
                "model_loaded": model_loaded,
                "model_version": model_version,
                "uptime_seconds": round(uptime, 2),
                "gpu_available": gpu_available,
                "app_version": _APP_VERSION,
            },
            headers={
                "X-Model-Version": model_version,
                "X-App-Version": _APP_VERSION,
            },
        )

    @app.get("/v1/tts/metrics")
    async def tts_metrics():
        """
        Prometheus metrics endpoint.

        Returns metrics in Prometheus text format for scraping.
        """
        return Response(
            content=get_metrics_text(),
            media_type=get_metrics_content_type(),
        )

    @app.post("/v1/tts/generate")
    async def tts_generate(request: Request):
        """
        Generate speech from text.

        Supports streaming (M4) and non-streaming (M3) modes.
        Currently implements non-streaming WAV generation only.
        """
        from veena3modal.api.schemas import TTSGenerateRequest, AudioFormat
        from veena3modal.services import tts_runtime

        request_id = str(uuid.uuid4())
        start_time = time.time()

        # === AUTH CHECK ===
        if FeatureFlags.is_auth_enabled():
            # Extract API key from headers
            headers_dict = dict(request.headers)
            api_key = extract_api_key(headers_dict)

            if not api_key:
                return JSONResponse(
                    status_code=401,
                    content=create_error_response(
                        code=ErrorCode.INVALID_API_KEY,
                        message="API key required. Use 'Authorization: Bearer <key>' or 'X-API-Key: <key>' header.",
                        request_id=request_id,
                    ),
                    headers={"X-Request-ID": request_id},
                )

            # Validate API key
            validator = get_api_validator()
            key_hash = hash_api_key(api_key)
            auth_result = validator.validate(key_hash)

            if not auth_result.is_valid:
                status_code = 401 if auth_result.error_code == "INVALID_API_KEY" else 403
                return JSONResponse(
                    status_code=status_code,
                    content=create_error_response(
                        code=ErrorCode(auth_result.error_code),
                        message=auth_result.error_message or "Authentication failed",
                        request_id=request_id,
                    ),
                    headers={"X-Request-ID": request_id},
                )

            # === RATE LIMIT CHECK ===
            if FeatureFlags.is_rate_limiting_enabled():
                rate_limiter = get_rate_limiter()
                allowed, remaining, reset_after = rate_limiter.check(key_hash)

                if not allowed:
                    headers = rate_limiter.get_headers(allowed, remaining, reset_after)
                    headers["X-Request-ID"] = request_id
                    return JSONResponse(
                        status_code=429,
                        content=create_error_response(
                            code=ErrorCode.RATE_LIMIT_EXCEEDED,
                            message=f"Rate limit exceeded. Retry after {int(reset_after) + 1} seconds.",
                            request_id=request_id,
                        ),
                        headers=headers,
                    )

        # Parse and validate request body
        try:
            body = await request.json()
        except Exception as e:
            raise HTTPException(status_code=400, detail=f"Invalid JSON: {e}")

        try:
            req = TTSGenerateRequest(**body)
        except Exception as e:
            error_msg = str(e)
            # Extract validation error details
            return JSONResponse(
                status_code=400,
                content={
                    "error": {
                        "code": "VALIDATION_ERROR",
                        "message": error_msg,
                        "request_id": request_id,
                    }
                },
                headers={"X-Request-ID": request_id},
            )

        # Check if runtime is initialized
        if not tts_runtime.is_initialized():
            return JSONResponse(
                status_code=503,
                content={
                    "error": {
                        "code": "MODEL_NOT_LOADED",
                        "message": "TTS model not initialized. Please wait for warmup.",
                        "request_id": request_id,
                    }
                },
                headers={"X-Request-ID": request_id},
            )

        # Get resolved speaker
        speaker = req.get_resolved_speaker()

        # Get normalized text (apply text normalization if enabled)
        normalizer_func = None
        if req.normalize:
            try:
                from veena3modal.processing.text_normalizer import normalize_text
                normalizer_func = lambda t: normalize_text(t, verbose=req.normalize_verbose)
            except ImportError:
                logger.warning("TextNormalizer not available, skipping text normalization")

        text = req.get_normalized_text(normalizer_func=normalizer_func)

        # Set request context for structured logging
        set_request_context(request_id=request_id)

        # Log request received (lifecycle event + metrics)
        log_request_received(
            request_id=request_id,
            text_length=len(text),
            speaker=speaker,
            stream=req.stream,
            format=req.format,
        )
        metrics_request_received(speaker=speaker, stream=req.stream, format=req.format)

        # Streaming mode (M4)
        if req.stream:
            return await _handle_streaming_request(req, request_id, text, speaker, start_time)

        # Format encoding will be applied after generation (non-streaming only)
        # For streaming, only WAV is supported
        target_format = req.format

        try:
            # Get output sample rate - use_enum_values=True means req.output is already a string
            output_sr = req.output if hasattr(req, 'output') else "16khz"

            # Generate speech (non-streaming)
            if req.chunking:
                audio_bytes, metrics = await tts_runtime.generate_speech_chunked(
                    text=text,
                    speaker=speaker,
                    temperature=req.temperature,
                    top_k=req.top_k,
                    top_p=req.top_p,
                    max_tokens=req.max_tokens,
                    repetition_penalty=req.repetition_penalty,
                    seed=req.seed,
                    sample_rate=req.sample_rate,
                    output_sample_rate=output_sr,
                    request_id=request_id,
                )
            else:
                audio_bytes, metrics = await tts_runtime.generate_speech(
                    text=text,
                    speaker=speaker,
                    temperature=req.temperature,
                    top_k=req.top_k,
                    top_p=req.top_p,
                    max_tokens=req.max_tokens,
                    repetition_penalty=req.repetition_penalty,
                    seed=req.seed,
                    output_sample_rate=output_sr,
                    request_id=request_id,
                )

            if audio_bytes is None:
                return JSONResponse(
                    status_code=500,
                    content={
                        "error": {
                            "code": "GENERATION_FAILED",
                            "message": "Audio generation failed. Check logs for details.",
                            "request_id": request_id,
                        }
                    },
                    headers={"X-Request-ID": request_id},
                )

            # Calculate RTF
            total_time = time.time() - start_time
            audio_duration = metrics.get("audio_duration_seconds", 0)
            rtf = total_time / audio_duration if audio_duration > 0 else 0

            # Build response headers
            runtime = tts_runtime.get_runtime()
            model_version = runtime.model_version if runtime else get_model_version()

            # Use actual output sample rate from metrics
            output_sample_rate = metrics.get("output_sample_rate", req.sample_rate)

            headers = {
                "X-Request-ID": request_id,
                "X-Model-Version": model_version,
                "X-Format": "wav",
                "X-Sample-Rate": str(output_sample_rate),
                "X-Stream": "false",
                "X-Audio-Bytes": str(len(audio_bytes)),
                "X-Audio-Seconds": f"{audio_duration:.2f}",
                "X-TTFB-ms": str(metrics.get("ttfb_ms", 0)),
                "X-RTF": f"{rtf:.3f}",
                "X-Text-Chunked": str(metrics.get("text_chunked", False)).lower(),
                "X-SR-Applied": str(metrics.get("sr_applied", False)).lower(),
            }
            headers.update(_build_perf_headers(metrics))

            if req.seed is not None:
                headers["X-Seed"] = str(req.seed)

            # Fire-and-forget sentence storage (non-blocking, after generation)
            sentence_store = _get_sentence_store()
            sentence_store.store_fire_and_forget(
                request_id=request_id,
                text=text,
                speaker=speaker,
                stream=False,
                format=req.format,
                temperature=req.temperature,
                top_k=req.top_k,
                top_p=req.top_p,
                max_tokens=req.max_tokens,
                repetition_penalty=req.repetition_penalty,
                seed=req.seed,
                text_chunked=metrics.get("text_chunked", False),
                ttfb_ms=metrics.get("ttfb_ms"),
                audio_duration_seconds=audio_duration,
            )

            # Log successful completion (lifecycle event + metrics)
            total_duration_ms = int(total_time * 1000)
            log_request_completed(
                request_id=request_id,
                status_code=200,
                total_duration_ms=total_duration_ms,
                audio_duration_seconds=audio_duration,
                rtf=rtf,
                chunks_sent=1,
                **metrics,
            )
            metrics_request_completed(
                status_code=200,
                duration_seconds=total_time,
                speaker=speaker,
                stream=False,
            )
            record_ttfb(ttfb_seconds=metrics.get("ttfb_ms", 0) / 1000, speaker=speaker, stream=False)
            record_rtf(rtf=rtf, speaker=speaker)
            record_audio_duration(duration_seconds=audio_duration, speaker=speaker)

            clear_request_context()

            # Apply format encoding if not WAV
            media_type = "audio/wav"
            if target_format != AudioFormat.WAV.value:


cd /home/ubuntu/veenaModal && sed -n '560,760p' veena3modal/api/fastapi_app.py


Output:
Chunk ID: 411115
Wall time: 0.0511 seconds
Process exited with code 0
Original token count: 2228
Output:
            if target_format != AudioFormat.WAV.value:
                try:
                    from veena3modal.audio.encoder import (
                        OpusEncoder, MP3Encoder, MuLawEncoder, FLACEncoder
                    )

                    # Get sample rate from output (after potential SR)
                    sample_rate = metrics.get("output_sample_rate", 16000)

                    # Strip WAV header to get PCM data
                    pcm_data = audio_bytes[44:] if len(audio_bytes) > 44 else audio_bytes

                    if target_format == AudioFormat.OPUS.value:
                        encoder = OpusEncoder(sample_rate=sample_rate)
                        audio_bytes = encoder.encode(pcm_data, sample_rate)
                        media_type = "audio/opus"
                    elif target_format == AudioFormat.MP3.value:
                        encoder = MP3Encoder(sample_rate=sample_rate)
                        audio_bytes = encoder.encode(pcm_data, sample_rate)
                        media_type = "audio/mpeg"
                    elif target_format == AudioFormat.MULAW.value:
                        encoder = MuLawEncoder()
                        audio_bytes = encoder.encode(pcm_data, sample_rate)
                        media_type = "audio/x-wav"
                    elif target_format == AudioFormat.FLAC.value:
                        encoder = FLACEncoder(sample_rate=sample_rate)
                        audio_bytes = encoder.encode(pcm_data, sample_rate)
                        media_type = "audio/flac"

                    headers["X-Audio-Bytes"] = str(len(audio_bytes))
                    logger.info(f"Encoded audio to {target_format}: {len(audio_bytes)} bytes")
                except ImportError as e:
                    logger.warning(f"Audio encoder not available: {e}")
                    # Fall back to WAV
                except Exception as e:
                    logger.error(f"Format encoding failed: {e}")
                    # Fall back to WAV

            return Response(
                content=audio_bytes,
                media_type=media_type,
                headers=headers,
            )

        except Exception as e:
            logger.exception(f"TTS generation error: {e}")

            # Log failure (lifecycle event + metrics)
            log_request_failed(
                request_id=request_id,
                status_code=500,
                error_code="INTERNAL_ERROR",
                error_message=str(e),
            )
            metrics_request_failed(
                status_code=500,
                error_code="INTERNAL_ERROR",
                speaker=speaker,
            )
            clear_request_context()

            return JSONResponse(
                status_code=500,
                content={
                    "error": {
                        "code": "INTERNAL_ERROR",
                        "message": str(e),
                        "request_id": request_id,
                    }
                },
                headers={"X-Request-ID": request_id},
            )

    async def _handle_streaming_request(req, request_id: str, text: str, speaker: str, start_time: float):
        """
        Handle streaming TTS request.

        Returns a StreamingResponse that yields WAV header + PCM chunks.
        True streaming: first bytes sent ASAP before full audio is generated.
        """
        from fastapi.responses import StreamingResponse
        from veena3modal.api.schemas import AudioFormat
        from veena3modal.services import tts_runtime

        # Check if streaming pipeline is available
        runtime = tts_runtime.get_runtime()
        if runtime is None or runtime.streaming_pipeline is None:
            return JSONResponse(
                status_code=503,
                content={
                    "error": {
                        "code": "STREAMING_UNAVAILABLE",
                        "message": "Streaming pipeline not initialized.",
                        "request_id": request_id,
                    }
                },
                headers={"X-Request-ID": request_id},
            )

        # Non-WAV formats not implemented yet (M5)
        if req.format != AudioFormat.WAV.value:
            return JSONResponse(
                status_code=501,
                content={
                    "error": {
                        "code": "FORMAT_NOT_IMPLEMENTED",
                        "message": f"Streaming format '{req.format}' not yet implemented. Use format=wav.",
                        "request_id": request_id,
                    }
                },
                headers={"X-Request-ID": request_id},
            )

        # Set request context for structured logging
        set_request_context(request_id=request_id)

        # Create async generator for streaming response
        async def audio_stream_generator():
            """Yields audio chunks as they're generated."""
            final_metrics = {}
            first_chunk_sent = False
            chunks_count = 0
            try:
                async for audio_chunk, metrics in tts_runtime.generate_speech_streaming(
                    text=text,
                    speaker=speaker,
                    temperature=req.temperature,
                    top_k=req.top_k,
                    top_p=req.top_p,
                    max_tokens=req.max_tokens,
                    repetition_penalty=req.repetition_penalty,
                    seed=req.seed,
                    enable_chunking=req.chunking,
                    request_id=request_id,
                ):
                    final_metrics = metrics
                    chunks_count += 1

                    # Fire-and-forget sentence storage after first chunk (don't block TTFB)
                    if not first_chunk_sent:
                        first_chunk_sent = True
                        ttfb_ms = metrics.get("ttfb_ms", 0)

                        # Log first audio emitted (TTFB marker)
                        log_first_audio_emitted(
                            request_id=request_id,
                            ttfb_ms=ttfb_ms,
                            chunk_size_bytes=len(audio_chunk),
                        )
                        record_ttfb(ttfb_seconds=ttfb_ms / 1000, speaker=speaker, stream=True)

                        sentence_store = _get_sentence_store()
                        sentence_store.store_fire_and_forget(
                            request_id=request_id,
                            text=text,
                            speaker=speaker,
                            stream=True,
                            format=req.format,
                            temperature=req.temperature,
                            top_k=req.top_k,
                            top_p=req.top_p,
                            max_tokens=req.max_tokens,
                            repetition_penalty=req.repetition_penalty,
                            seed=req.seed,
                            text_chunked=metrics.get("text_chunked", False),
                            ttfb_ms=ttfb_ms,
                            # audio_duration_seconds not known yet for streaming
                        )

                    yield audio_chunk

                # Stream completed - log metrics
                total_time = time.time() - start_time
                audio_duration = final_metrics.get("audio_duration_seconds", 0)
                rtf = total_time / audio_duration if audio_duration > 0 else 0

                log_request_completed(
                    request_id=request_id,
                    status_code=200,
                    total_duration_ms=int(total_time * 1000),
                    audio_duration_seconds=audio_duration,
                    rtf=rtf,
                    chunks_sent=chunks_count,
                    **final_metrics,
                )
                metrics_request_completed(
                    status_code=200,
                    duration_seconds=total_time,
                    speaker=speaker,
                    stream=True,
                )
                record_rtf(rtf=rtf, speaker=speaker)
                record_audio_duration(duration_seconds=audio_duration, speaker=speaker)
                record_chunks_sent(chunks=chunks_count, speaker=speaker)
                clear_request_context()
            except Exception as e:
                logger.exception(f"Streaming error: {e}")
                # Log streaming failure
                log_request_failed(
                    request_id=request_id,
                    status_code=500,


cd /home/ubuntu/veenaModal && python3 - <<'PY'
import asyncio
from veena3modal.services import tts_runtime
from veena3modal.services.tts_runtime import initialize_runtime

async def main():
    if not tts_runtime.is_initialized():
        initialize_runtime(
            model_path='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM',
            bicodec_path='/home/ubuntu/veenaModal/models/spark_tts_4speaker',
            device='cuda',
            gpu_memory_utilization=0.15,
            num_engines=1,
        )
    try:
        audio, metrics = await tts_runtime.generate_speech(
            text='Hello',
            speaker='lipakshi',
            temperature=0.8,
            top_k=50,
            top_p=1.0,
            max_tokens=128,
            repetition_penalty=1.05,
            seed=None,
            output_sample_rate='16khz',
            request_id='manual-test',
        )
        print('result', audio is not None, len(audio) if audio else None, list(metrics.items())[:5])
    except Exception as e:
        import traceback
        traceback.print_exc()
        print(type(e).__name__,e)

asyncio.run(main())
PY


Output:
Chunk ID: 9c32d2
Wall time: 11.3118 seconds
Process exited with code 1
Original token count: 5390
Output:
[2026-02-17 00:53:30] INFO tts_runtime.py:179: Loading SparkTTS model with vLLM...
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
🚀 Initializing Spark TTS Model
📁 Model path: /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
🗣  Speakers: 12 predefined (lipakshi, vardan, reet, Nandini, krishna, anika, adarsh, Nilay, Aarvi, Asha, Bittu, Mira)
🎭 Emotions: 10 tags in [bracket] format

📝 Loading tokenizer from model...
✅ Tokenizer loaded: 165168 tokens

🔍 Validating 12 speakers for Spark TTS...
  lipakshi → <|speaker_0|>: 7 tokens
  vardan → <|speaker_1|>: 7 tokens
  reet → <|speaker_2|>: 7 tokens
  Nandini → <|speaker_3|>: 7 tokens
  krishna → <|speaker_4|>: 7 tokens
  anika → <|speaker_5|>: 7 tokens
  adarsh → <|speaker_6|>: 7 tokens
  Nilay → <|speaker_7|>: 7 tokens
  Aarvi → <|speaker_8|>: 7 tokens
  Asha → <|speaker_9|>: 7 tokens
  Bittu → <|speaker_10|>: 8 tokens
  Mira → <|speaker_11|>: 8 tokens
✅ All 12 speakers validated with mapping

🔍 Validating 10 emotion tags for Spark TTS...
✅ All 10 emotion tags validated

📋 Sample emotion tags:
  [angry]: 1 tokens
  [curious]: 1 tokens
  [excited]: 1 tokens
  [giggle]: 1 tokens
  [laughs harder]: 1 tokens
  ...

🔧 Initializing vLLM engine with optimizations...
  - dtype: bfloat16
  - max_model_len: 4096
  - gpu_memory_utilization: 0.15
  - enable_prefix_caching: True
  - enforce_eager (disable CUDA graphs): False
INFO 02-17 00:53:31 [model.py:541] Resolved architecture: Qwen2ForCausalLM
INFO 02-17 00:53:31 [model.py:1561] Using max model len 4096
INFO 02-17 00:53:31 [scheduler.py:226] Chunked prefill is enabled with max_num_batched_tokens=4096.
INFO 02-17 00:53:31 [vllm.py:624] Asynchronous scheduling is enabled.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(EngineCore_DP0 pid=3720696) INFO 02-17 00:53:32 [core.py:96] Initializing a V1 LLM engine (v0.15.1) with config: model='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', speculative_config=None, tokenizer='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [4096], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 256, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': True}, 'local_cache_dir': None, 'static_all_moe_layers': []}
(EngineCore_DP0 pid=3720696) INFO 02-17 00:53:34 [parallel_state.py:1212] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://216.81.248.184:38291 backend=nccl
(EngineCore_DP0 pid=3720696) INFO 02-17 00:53:34 [parallel_state.py:1423] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
(EngineCore_DP0 pid=3720696) ERROR 02-17 00:53:34 [core.py:946] EngineCore failed to start.
(EngineCore_DP0 pid=3720696) ERROR 02-17 00:53:34 [core.py:946] Traceback (most recent call last):
(EngineCore_DP0 pid=3720696) ERROR 02-17 00:53:34 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 937, in run_engine_core
(EngineCore_DP0 pid=3720696) ERROR 02-17 00:53:34 [core.py:946]     engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore_DP0 pid=3720696) ERROR 02-17 00:53:34 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 691, in __init__
(EngineCore_DP0 pid=3720696) ERROR 02-17 00:53:34 [core.py:946]     super().__init__(
(EngineCore_DP0 pid=3720696) ERROR 02-17 00:53:34 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 105, in __init__
(EngineCore_DP0 pid=3720696) ERROR 02-17 00:53:34 [core.py:946]     self.model_executor = executor_class(vllm_config)
(EngineCore_DP0 pid=3720696) ERROR 02-17 00:53:34 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/executor/abstract.py", line 101, in __init__
(EngineCore_DP0 pid=3720696) ERROR 02-17 00:53:34 [core.py:946]     self._init_executor()
(EngineCore_DP0 pid=3720696) ERROR 02-17 00:53:34 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/executor/uniproc_executor.py", line 47, in _init_executor
(EngineCore_DP0 pid=3720696) ERROR 02-17 00:53:34 [core.py:946]     self.driver_worker.init_device()
(EngineCore_DP0 pid=3720696) ERROR 02-17 00:53:34 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/worker/worker_base.py", line 326, in init_device
(EngineCore_DP0 pid=3720696) ERROR 02-17 00:53:34 [core.py:946]     self.worker.init_device()  # type: ignore
(EngineCore_DP0 pid=3720696) ERROR 02-17 00:53:34 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/worker/gpu_worker.py", line 258, in init_device
(EngineCore_DP0 pid=3720696) ERROR 02-17 00:53:34 [core.py:946]     from vllm.v1.worker.gpu_model_runner import (
(EngineCore_DP0 pid=3720696) ERROR 02-17 00:53:34 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/worker/gpu_model_runner.py", line 153, in <module>
(EngineCore_DP0 pid=3720696) ERROR 02-17 00:53:34 [core.py:946]     from vllm.v1.spec_decode.ngram_proposer import NgramProposer
(EngineCore_DP0 pid=3720696) ERROR 02-17 00:53:34 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/spec_decode/ngram_proposer.py", line 170, in <module>
(EngineCore_DP0 pid=3720696) ERROR 02-17 00:53:34 [core.py:946]     def batch_propose_numba(
(EngineCore_DP0 pid=3720696) ERROR 02-17 00:53:34 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/decorators.py", line 221, in wrapper
(EngineCore_DP0 pid=3720696) ERROR 02-17 00:53:34 [core.py:946]     disp = dispatcher(py_func=func, locals=locals,
(EngineCore_DP0 pid=3720696) ERROR 02-17 00:53:34 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/dispatcher.py", line 772, in __init__
(EngineCore_DP0 pid=3720696) ERROR 02-17 00:53:34 [core.py:946]     self.targetctx = self.targetdescr.target_context
(EngineCore_DP0 pid=3720696) ERROR 02-17 00:53:34 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/registry.py", line 30, in target_context
(EngineCore_DP0 pid=3720696) ERROR 02-17 00:53:34 [core.py:946]     return self._toplevel_target_context
(EngineCore_DP0 pid=3720696) ERROR 02-17 00:53:34 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/utils.py", line 741, in __get__
(EngineCore_DP0 pid=3720696) ERROR 02-17 00:53:34 [core.py:946]     return super().__get__(*args, **kwargs)
(EngineCore_DP0 pid=3720696) ERROR 02-17 00:53:34 [core.py:946]   File "/usr/lib/python3.10/functools.py", line 981, in __get__
(EngineCore_DP0 pid=3720696) ERROR 02-17 00:53:34 [core.py:946]     val = self.func(instance)
(EngineCore_DP0 pid=3720696) ERROR 02-17 00:53:34 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/registry.py", line 18, in _toplevel_target_context
(EngineCore_DP0 pid=3720696) ERROR 02-17 00:53:34 [core.py:946]     return cpu.CPUContext(self.typing_context, self._target_name)
(EngineCore_DP0 pid=3720696) ERROR 02-17 00:53:34 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/cpu.py", line 42, in __init__
(EngineCore_DP0 pid=3720696) ERROR 02-17 00:53:34 [core.py:946]     super().__init__(typingctx, target)
(EngineCore_DP0 pid=3720696) ERROR 02-17 00:53:34 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/base.py", line 254, in __init__
(EngineCore_DP0 pid=3720696) ERROR 02-17 00:53:34 [core.py:946]     self.init()
(EngineCore_DP0 pid=3720696) ERROR 02-17 00:53:34 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/compiler_lock.py", line 35, in _acquire_compile_lock
(EngineCore_DP0 pid=3720696) ERROR 02-17 00:53:34 [core.py:946]     return func(*args, **kwargs)
(EngineCore_DP0 pid=3720696) ERROR 02-17 00:53:34 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/cpu.py", line 51, in init
(EngineCore_DP0 pid=3720696) ERROR 02-17 00:53:34 [core.py:946]     self._internal_codegen = codegen.JITCPUCodegen("numba.exec")
(EngineCore_DP0 pid=3720696) ERROR 02-17 00:53:34 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/codegen.py", line 1172, in __init__
(EngineCore_DP0 pid=3720696) ERROR 02-17 00:53:34 [core.py:946]     initialize_llvm()
(EngineCore_DP0 pid=3720696) ERROR 02-17 00:53:34 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/codegen.py", line 1434, in initialize_llvm
(EngineCore_DP0 pid=3720696) ERROR 02-17 00:53:34 [core.py:946]     ll.initialize()
(EngineCore_DP0 pid=3720696) ERROR 02-17 00:53:34 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/llvmlite/binding/initfini.py", line 17, in initialize
(EngineCore_DP0 pid=3720696) ERROR 02-17 00:53:34 [core.py:946]     raise RuntimeError(
(EngineCore_DP0 pid=3720696) ERROR 02-17 00:53:34 [core.py:946] RuntimeError: llvmlite.binding.initialize() is deprecated and will be removed. LLVM initialization is now handled automatically. Please remove calls to this function from your code and check for other behavioral changes that may have occurred due to LLVM updates.
(EngineCore_DP0 pid=3720696) Process EngineCore_DP0:
(EngineCore_DP0 pid=3720696) Traceback (most recent call last):
(EngineCore_DP0 pid=3720696)   File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore_DP0 pid=3720696)     self.run()
(EngineCore_DP0 pid=3720696)   File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
(EngineCore_DP0 pid=3720696)     self._target(*self._args, **self._kwargs)
(EngineCore_DP0 pid=3720696)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 950, in run_engine_core
(EngineCore_DP0 pid=3720696)     raise e
(EngineCore_DP0 pid=3720696)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 937, in run_engine_core
(EngineCore_DP0 pid=3720696)     engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore_DP0 pid=3720696)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 691, in __init__
(EngineCore_DP0 pid=3720696)     super().__init__(
(EngineCore_DP0 pid=3720696)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 105, in __init__
(EngineCore_DP0 pid=3720696)     self.model_executor = executor_class(vllm_config)
(EngineCore_DP0 pid=3720696)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/executor/abstract.py", line 101, in __init__
(EngineCore_DP0 pid=3720696)     self._init_executor()
(EngineCore_DP0 pid=3720696)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/executor/uniproc_executor.py", line 47, in _init_executor
(EngineCore_DP0 pid=3720696)     self.driver_worker.init_device()
(EngineCore_DP0 pid=3720696)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/worker/worker_base.py", line 326, in init_device
(EngineCore_DP0 pid=3720696)     self.worker.init_device()  # type: ignore
(EngineCore_DP0 pid=3720696)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/worker/gpu_worker.py", line 258, in init_device
(EngineCore_DP0 pid=3720696)     from vllm.v1.worker.gpu_model_runner import (
(EngineCore_DP0 pid=3720696)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/worker/gpu_model_runner.py", line 153, in <module>
(EngineCore_DP0 pid=3720696)     from vllm.v1.spec_decode.ngram_proposer import NgramProposer
(EngineCore_DP0 pid=3720696)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/spec_decode/ngram_proposer.py", line 170, in <module>
(EngineCore_DP0 pid=3720696)     def batch_propose_numba(
(EngineCore_DP0 pid=3720696)   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/decorators.py", line 221, in wrapper
(EngineCore_DP0 pid=3720696)     disp = dispatcher(py_func=func, locals=locals,
(EngineCore_DP0 pid=3720696)   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/dispatcher.py", line 772, in __init__
(EngineCore_DP0 pid=3720696)     self.targetctx = self.targetdescr.target_context
(EngineCore_DP0 pid=3720696)   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/registry.py", line 30, in target_context
(EngineCore_DP0 pid=3720696)     return self._toplevel_target_context
(EngineCore_DP0 pid=3720696)   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/utils.py", line 741, in __get__
(EngineCore_DP0 pid=3720696)     return super().__get__(*args, **kwargs)
(EngineCore_DP0 pid=3720696)   File "/usr/lib/python3.10/functools.py", line 981, in __get__
(EngineCore_DP0 pid=3720696)     val = self.func(instance)
(EngineCore_DP0 pid=3720696)   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/registry.py", line 18, in _toplevel_target_context
(EngineCore_DP0 pid=3720696)     return cpu.CPUContext(self.typing_context, self._target_name)
(EngineCore_DP0 pid=3720696)   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/cpu.py", line 42, in __init__
(EngineCore_DP0 pid=3720696)     super().__init__(typingctx, target)
(EngineCore_DP0 pid=3720696)   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/base.py", line 254, in __init__
(EngineCore_DP0 pid=3720696)     self.init()
(EngineCore_DP0 pid=3720696)   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/compiler_lock.py", line 35, in _acquire_compile_lock
(EngineCore_DP0 pid=3720696)     return func(*args, **kwargs)
(EngineCore_DP0 pid=3720696)   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/cpu.py", line 51, in init
(EngineCore_DP0 pid=3720696)     self._internal_codegen = codegen.JITCPUCodegen("numba.exec")
(EngineCore_DP0 pid=3720696)   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/codegen.py", line 1172, in __init__
(EngineCore_DP0 pid=3720696)     initialize_llvm()
(EngineCore_DP0 pid=3720696)   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/codegen.py", line 1434, in initialize_llvm
(EngineCore_DP0 pid=3720696)     ll.initialize()
(EngineCore_DP0 pid=3720696)   File "/home/ubuntu/.local/lib/python3.10/site-packages/llvmlite/binding/initfini.py", line 17, in initialize
(EngineCore_DP0 pid=3720696)     raise RuntimeError(
(EngineCore_DP0 pid=3720696) RuntimeError: llvmlite.binding.initialize() is deprecated and will be removed. LLVM initialization is now handled automatically. Please remove calls to this function from your code and check for other behavioral changes that may have occurred due to LLVM updates.
[2026-02-17 00:53:35] ERROR tts_runtime.py:282: ❌ Failed to initialize TTS runtime: Engine core initialization failed. See root cause above. Failed core proc(s): {}
Traceback (most recent call last):
  File "/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py", line 180, in initialize_runtime
    model = SparkTTSModel(
  File "/home/ubuntu/veenaModal/veena3modal/core/model_loader.py", line 128, in __init__
    self.engine = AsyncLLMEngine.from_engine_args(engine_args)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/async_llm.py", line 257, in from_engine_args
    return cls(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/async_llm.py", line 155, in __init__
    self.engine_core = EngineCoreClient.make_async_mp_client(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 122, in make_async_mp_client
    return AsyncMPClient(*client_args)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 819, in __init__
    super().__init__(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 479, in __init__
    with launch_core_engines(vllm_config, executor_class, log_stats) as (
  File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
    next(self.gen)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/utils.py", line 933, in launch_core_engines
    wait_for_engine_startup(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/utils.py", line 992, in wait_for_engine_startup
    raise RuntimeError(
RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
Traceback (most recent call last):
  File "/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py", line 180, in initialize_runtime
    model = SparkTTSModel(
  File "/home/ubuntu/veenaModal/veena3modal/core/model_loader.py", line 128, in __init__
    self.engine = AsyncLLMEngine.from_engine_args(engine_args)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/async_llm.py", line 257, in from_engine_args
    return cls(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/async_llm.py", line 155, in __init__
    self.engine_core = EngineCoreClient.make_async_mp_client(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 122, in make_async_mp_client
    return AsyncMPClient(*client_args)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 819, in __init__
    super().__init__(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 479, in __init__
    with launch_core_engines(vllm_config, executor_class, log_stats) as (
  File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
    next(self.gen)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/utils.py", line 933, in launch_core_engines
    wait_for_engine_startup(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/utils.py", line 992, in wait_for_engine_startup
    raise RuntimeError(
RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 33, in <module>
  File "/usr/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "<stdin>", line 7, in main
  File "/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py", line 285, in initialize_runtime
    raise RuntimeError(f"TTS runtime initialization failed: {e}") from e
RuntimeError: TTS runtime initialization failed: Engine core initialization failed. See root cause above. Failed core proc(s): {}


cd /home/ubuntu/veenaModal && sed -n '1,260p' veena3modal/api/schemas.py


Output:
Chunk ID: 06584e
Wall time: 0.0504 seconds
Process exited with code 0
Original token count: 1906
Output:
"""
Pydantic schemas for TTS API requests/responses.

Ported from veena3srv/apps/api/serializers.py with full validation parity.
No Django dependencies - pure Pydantic for Modal deployment.
"""

from __future__ import annotations

import re
from enum import Enum
from typing import Any, Dict, List, Optional, Union
from uuid import UUID

from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator

# === Constants (mirrored from veena3srv/apps/inference/constants.py) ===
# These are duplicated here to avoid Django import chain; must stay in sync.

MAX_TEXT_LENGTH = 50000

# Spark TTS Speaker System - 12 Speakers
# Model: BayAreaBoys/spark_tts_4speaker (HuggingFace) has 12 speaker tokens
SPEAKER_MAP = {
    "lipakshi": 0, "vardan": 1, "reet": 2, "Nandini": 3,
    "krishna": 4, "anika": 5, "adarsh": 6, "Nilay": 7,
    "Aarvi": 8, "Asha": 9, "Bittu": 10, "Mira": 11,
}

# Friendly speaker name mappings (user-facing → internal)
FRIENDLY_SPEAKER_MAP = {
    "Mitra": "lipakshi", "Aaranya": "reet", "Taru": "Nandini",
    "Neer": "Nilay", "Dhruva": "vardan", "Ira": "anika",
    "Veda": "adarsh", "Aria": "krishna",
    "Aarvi": "Aarvi", "Asha": "Asha", "Bittu": "Bittu", "Mira": "Mira",
}

INDIC_SPEAKERS = list(SPEAKER_MAP.keys())
ALL_SPEAKER_NAMES = INDIC_SPEAKERS + list(FRIENDLY_SPEAKER_MAP.keys())

# Emotion tags (Spark TTS bracket format)
INDIC_EMOTION_TAGS = [
    "[angry]", "[curious]", "[excited]", "[giggle]", "[laughs harder]",
    "[laughs]", "[screams]", "[sighs]", "[sings]", "[whispers]"
]

# Legacy emotion tag mapping (<angle> → [bracket])
LEGACY_EMOTION_MAP = {
    "<angry>": "[angry]", "<curious>": "[curious]", "<excited>": "[excited]",
    "<giggle>": "[giggle]", "<laugh_harder>": "[laughs harder]",
    "<laugh>": "[laughs]", "<scream>": "[screams]", "<sigh>": "[sighs]",
    "<sing>": "[sings]", "<whisper>": "[whispers]"
}

# === Helper Functions ===

def resolve_speaker_name(name: str) -> str:
    """
    Resolve friendly speaker name to internal name.

    Examples:
        resolve_speaker_name("Mitra") -> "lipakshi"
        resolve_speaker_name("lipakshi") -> "lipakshi"

    Raises:
        ValueError: if name is not a valid speaker name
    """
    if name in FRIENDLY_SPEAKER_MAP:
        return FRIENDLY_SPEAKER_MAP[name]
    if name in INDIC_SPEAKERS:
        return name
    raise ValueError(
        f"Invalid speaker name: {name}. Valid names: {', '.join(ALL_SPEAKER_NAMES)}"
    )

# Emotion normalization mapping (various formats → [emotion])
_EMOTION_TO_TAG = {
    **{k.strip('<>'): v for k, v in LEGACY_EMOTION_MAP.items()},
    'laughing': '[laughs]', 'laugh': '[laughs]', 'laughs harder': '[laughs harder]',
    'laugh harder': '[laughs harder]', 'laugh_harder': '[laughs harder]',
    'sighs': '[sighs]', 'sigh': '[sighs]', 'giggles': '[giggle]', 'giggle': '[giggle]',
    'angry': '[angry]', 'excited': '[excited]', 'whispers': '[whispers]',
    'whisper': '[whispers]', 'screaming': '[screams]', 'scream': '[screams]',
    'screams': '[screams]', 'singing': '[sings]', 'sing': '[sings]',
    'sings': '[sings]', 'curious': '[curious]',
}

def normalize_emotion_tags(text: str) -> str:
    """
    Normalize emotion tags to Spark TTS bracket format.

    Conversions:
    - <laugh> → [laughs]
    - [laughing] → [laughs]
    - etc.
    """
    # Convert old <emotion> format to [emotion]
    angle_pattern = r'<([a-z_]+)>'

    def replace_angle(match):
        emotion = match.group(1).lower().strip()
        if emotion in _EMOTION_TO_TAG:
            return _EMOTION_TO_TAG[emotion]
        old_tag = f'<{emotion}>'
        if old_tag in LEGACY_EMOTION_MAP:
            return LEGACY_EMOTION_MAP[old_tag]
        return f'[{emotion}]'  # Unknown, convert to bracket anyway

    text = re.sub(angle_pattern, replace_angle, text)

    # Normalize [emotion] variants
    bracket_pattern = r'\[([^\]]+)\]'

    def replace_bracket(match):
        emotion = match.group(1).lower().strip()
        bracket_tag = f'[{emotion}]'
        if bracket_tag in INDIC_EMOTION_TAGS:
            return bracket_tag
        if emotion in _EMOTION_TO_TAG:
            return _EMOTION_TO_TAG[emotion]
        # Try singular/plural
        if emotion.endswith('s') and emotion[:-1] in _EMOTION_TO_TAG:
            return _EMOTION_TO_TAG[emotion[:-1]]
        if (emotion + 's') in _EMOTION_TO_TAG:
            return _EMOTION_TO_TAG[emotion + 's']
        return bracket_tag  # Keep as-is

    return re.sub(bracket_pattern, replace_bracket, text)

# === Enums ===

class AudioFormat(str, Enum):
    """Supported audio output formats."""
    WAV = "wav"
    OPUS = "opus"
    MP3 = "mp3"
    MULAW = "mulaw"
    FLAC = "flac"

class OutputSampleRate(str, Enum):
    """Output sample rate options."""
    SR_16KHZ = "16khz"
    SR_48KHZ = "48khz"

# === Request Schema ===

class TTSGenerateRequest(BaseModel):
    """
    Request schema for /v1/tts/generate.

    Mirrors validation rules from veena3srv/apps/api/serializers.py.
    """

    # Required
    text: str = Field(
        ...,
        min_length=1,
        max_length=MAX_TEXT_LENGTH,
        description="Text to synthesize (max 50,000 characters)"
    )

    # Speaker (required for indic_speakers model)
    speaker: Optional[str] = Field(
        None,
        max_length=50,
        description="Speaker name (internal or friendly)"
    )

    # Voice design (not currently supported)
    description: Optional[str] = Field(
        None,
        max_length=2000,
        description="Voice description (for voiceDesign model, NOT currently supported)"
    )
    voice_id: Optional[UUID] = Field(
        None,
        description="UUID of pre-created voice profile (for voiceDesign model)"
    )

    # Generation options
    model: Optional[str] = Field(
        None,
        max_length=50,
        description="Model to use (optional). Default: indic_speakers"
    )
    seed: Optional[int] = Field(
        None,
        ge=0,
        le=2**31 - 1,
        description="Random seed for reproducibility (0-2147483647)"
    )
    stream: bool = Field(
        False,
        description="Stream audio as it's generated"
    )
    format: AudioFormat = Field(
        AudioFormat.WAV,
        description="Audio output format"
    )
    sample_rate: int = Field(
        16000,
        description="Sample rate in Hz (default: 16000)"
    )

    # Advanced parameters
    temperature: float = Field(
        0.4,
        ge=0.0,
        le=2.0,
        description="Sampling temperature (0.0-2.0)"
    )
    top_k: int = Field(
        50,
        ge=1,
        le=100,
        description="Top-k sampling (1-100)"
    )
    top_p: float = Field(
        1.0,
        ge=0.0,
        le=1.0,
        description="Nucleus sampling (0.0-1.0)"
    )
    max_tokens: int = Field(
        4096,
        ge=128,
        le=4096,
        description="Maximum BiCodec tokens to generate"
    )
    repetition_penalty: float = Field(
        1.05,
        ge=1.0,
        le=2.0,
        description="Repetition penalty (1.0-2.0)"
    )

    # Preprocessing toggles
    normalize: bool = Field(
        True,
        description="Apply text normalization before TTS"
    )
    normalize_verbose: bool = Field(
        False,
        description="Return normalized text in X-Normalized-Text header"
    )
    chunking: bool = Field(
        True,
        description="Enable intelligent text chunking for long inputs"
    )

    # Output options
    output: OutputSampleRate = Field(
        OutputSampleRate.SR_16KHZ,


cd /home/ubuntu/veenaModal && rg -n "def get_resolved_speaker|get_normalized_text|speaker" veena3modal/api/schemas.py


Output:
Chunk ID: 1e12fb
Wall time: 0.0515 seconds
Process exited with code 0
Original token count: 464
Output:
24:# Model: BayAreaBoys/spark_tts_4speaker (HuggingFace) has 12 speaker tokens
31:# Friendly speaker name mappings (user-facing → internal)
59:def resolve_speaker_name(name: str) -> str:
61:    Resolve friendly speaker name to internal name.
64:        resolve_speaker_name("Mitra") -> "lipakshi"
65:        resolve_speaker_name("lipakshi") -> "lipakshi"
68:        ValueError: if name is not a valid speaker name
75:        f"Invalid speaker name: {name}. Valid names: {', '.join(ALL_SPEAKER_NAMES)}"
169:    # Speaker (required for indic_speakers model)
170:    speaker: Optional[str] = Field(
191:        description="Model to use (optional). Default: indic_speakers"
266:    _resolved_speaker: Optional[str] = None
308:        Cross-field validation: speaker required, speaker resolution, mu-law sample rate.
310:        NOTE: We assume indic_speakers model type (voiceDesign not supported yet).
312:        has_speaker = bool(self.speaker)
316:        # For indic_speakers model: speaker is required
317:        if not has_speaker:
319:                "Parameter 'speaker' is required for indic_speakers model. "
323:        # Validate and resolve speaker name
324:        if self.speaker not in ALL_SPEAKER_NAMES:
326:                f"Invalid speaker '{self.speaker}'. Valid names: {', '.join(ALL_SPEAKER_NAMES)}"
329:        # Store resolved speaker
330:        self._resolved_speaker = resolve_speaker_name(self.speaker)
336:                "Current model type is 'indic_speakers'. Use 'speaker' instead."
345:    def get_resolved_speaker(self) -> str:
346:        """Return the resolved internal speaker name."""
347:        if self._resolved_speaker:
348:            return self._resolved_speaker
349:        return resolve_speaker_name(self.speaker) if self.speaker else ""
351:    def get_normalized_text(self, normalizer_func=None) -> str:


cd /home/ubuntu/veenaModal && sed -n '250,380p' veena3modal/api/schemas.py


Output:
Chunk ID: bf6926
Wall time: 0.0514 seconds
Process exited with code 0
Original token count: 1177
Output:
        False,
        description="Return normalized text in X-Normalized-Text header"
    )
    chunking: bool = Field(
        True,
        description="Enable intelligent text chunking for long inputs"
    )

    # Output options
    output: OutputSampleRate = Field(
        OutputSampleRate.SR_16KHZ,
        description="Output sample rate: '16khz' or '48khz' (super-resolution)"
    )

    # Internal fields (populated during validation)
    _original_text: Optional[str] = None
    _resolved_speaker: Optional[str] = None

    model_config = ConfigDict(use_enum_values=True)  # Serialize enums as strings

    @field_validator('text')
    @classmethod
    def validate_text(cls, v: str) -> str:
        """Validate text field: no empty, no control chars."""
        stripped = v.strip()
        if not stripped:
            raise ValueError("Text cannot be empty or whitespace only")

        # Check for control characters (except newlines/tabs)
        control_chars = [c for c in v if ord(c) < 32 and c not in '\n\t\r']
        if control_chars:
            raise ValueError("Text contains invalid control characters")

        return v

    @field_validator('description')
    @classmethod
    def validate_description(cls, v: Optional[str]) -> Optional[str]:
        """Validate description field."""
        if not v:
            return v
        control_chars = [c for c in v if ord(c) < 32 and c not in '\n\t\r']
        if control_chars:
            raise ValueError("Description contains invalid control characters")
        return v

    @field_validator('sample_rate')
    @classmethod
    def validate_sample_rate(cls, v: int) -> int:
        """Validate sample rate is one of the allowed values."""
        valid_rates = [8000, 16000, 22050, 24000, 44100, 48000]
        if v not in valid_rates:
            raise ValueError(f"Invalid sample rate. Valid options: {valid_rates}")
        return v

    @model_validator(mode='after')
    def cross_field_validation(self) -> 'TTSGenerateRequest':
        """
        Cross-field validation: speaker required, speaker resolution, mu-law sample rate.

        NOTE: We assume indic_speakers model type (voiceDesign not supported yet).
        """
        has_speaker = bool(self.speaker)
        has_description = bool(self.description)
        has_voice_id = self.voice_id is not None

        # For indic_speakers model: speaker is required
        if not has_speaker:
            raise ValueError(
                "Parameter 'speaker' is required for indic_speakers model. "
                f"Must be one of: {', '.join(ALL_SPEAKER_NAMES)}"
            )

        # Validate and resolve speaker name
        if self.speaker not in ALL_SPEAKER_NAMES:
            raise ValueError(
                f"Invalid speaker '{self.speaker}'. Valid names: {', '.join(ALL_SPEAKER_NAMES)}"
            )

        # Store resolved speaker
        self._resolved_speaker = resolve_speaker_name(self.speaker)

        # Warn if description/voice_id provided (not supported for indic model)
        if has_description or has_voice_id:
            raise ValueError(
                "Parameters 'description' and 'voice_id' are only valid for voiceDesign model. "
                "Current model type is 'indic_speakers'. Use 'speaker' instead."
            )

        # mu-law typically uses 8kHz
        if self.format == AudioFormat.MULAW and self.sample_rate != 8000:
            self.sample_rate = 8000

        return self

    def get_resolved_speaker(self) -> str:
        """Return the resolved internal speaker name."""
        if self._resolved_speaker:
            return self._resolved_speaker
        return resolve_speaker_name(self.speaker) if self.speaker else ""

    def get_normalized_text(self, normalizer_func=None) -> str:
        """
        Return normalized + emotion-normalized text.

        Args:
            normalizer_func: Optional custom normalizer (for testing).
                             If None, returns text with only emotion normalization.
        """
        text = self.text

        # Apply text normalization if enabled
        if self.normalize and normalizer_func:
            self._original_text = text
            text = normalizer_func(text)

        # Always apply emotion tag normalization
        text = normalize_emotion_tags(text)

        return text

# === Response Schemas ===

class TTSGenerateResponse(BaseModel):
    """Response metadata for non-streaming /v1/tts/generate."""

    request_id: str
    model: str
    format: str
    sample_rate: int


cd /home/ubuntu/veenaModal && NUMBA_DISABLE_JIT=1 python3 - <<'PY'
import asyncio, time
from veena3modal.services import tts_runtime
from veena3modal.services.tts_runtime import initialize_runtime

async def main():
    print('init',tts_runtime.is_initialized())
    if not tts_runtime.is_initialized():
      initialize_runtime('/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM','/home/ubuntu/veenaModal/models/spark_tts_4speaker',device='cuda',gpu_memory_utilization=0.15,num_engines=1)
    t=time.time()
    audio, metrics = await tts_runtime.generate_speech_chunked(text='Hello', speaker='lipakshi', temperature=0.8, top_k=50, top_p=1.0, max_tokens=128, repetition_penalty=1.05, request_id='x')
    print('done',len(audio),metrics.get('llm_batch_count'),metrics.get('token_total'))
    print('t',time.time()-t)

asyncio.run(main())
PY


Output:
Chunk ID: f3843c
Wall time: 30.0020 seconds
Process running with session ID 63734
Original token count: 2551
Output:
[2026-02-17 00:53:55] INFO tts_runtime.py:179: Loading SparkTTS model with vLLM...
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
init False
🚀 Initializing Spark TTS Model
📁 Model path: /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
🗣  Speakers: 12 predefined (lipakshi, vardan, reet, Nandini, krishna, anika, adarsh, Nilay, Aarvi, Asha, Bittu, Mira)
🎭 Emotions: 10 tags in [bracket] format

📝 Loading tokenizer from model...
✅ Tokenizer loaded: 165168 tokens

🔍 Validating 12 speakers for Spark TTS...
  lipakshi → <|speaker_0|>: 7 tokens
  vardan → <|speaker_1|>: 7 tokens
  reet → <|speaker_2|>: 7 tokens
  Nandini → <|speaker_3|>: 7 tokens
  krishna → <|speaker_4|>: 7 tokens
  anika → <|speaker_5|>: 7 tokens
  adarsh → <|speaker_6|>: 7 tokens
  Nilay → <|speaker_7|>: 7 tokens
  Aarvi → <|speaker_8|>: 7 tokens
  Asha → <|speaker_9|>: 7 tokens
  Bittu → <|speaker_10|>: 8 tokens
  Mira → <|speaker_11|>: 8 tokens
✅ All 12 speakers validated with mapping

🔍 Validating 10 emotion tags for Spark TTS...
✅ All 10 emotion tags validated

📋 Sample emotion tags:
  [angry]: 1 tokens
  [curious]: 1 tokens
  [excited]: 1 tokens
  [giggle]: 1 tokens
  [laughs harder]: 1 tokens
  ...

🔧 Initializing vLLM engine with optimizations...
  - dtype: bfloat16
  - max_model_len: 4096
  - gpu_memory_utilization: 0.15
  - enable_prefix_caching: True
  - enforce_eager (disable CUDA graphs): False
INFO 02-17 00:53:56 [model.py:541] Resolved architecture: Qwen2ForCausalLM
INFO 02-17 00:53:56 [model.py:1561] Using max model len 4096
INFO 02-17 00:53:56 [scheduler.py:226] Chunked prefill is enabled with max_num_batched_tokens=4096.
INFO 02-17 00:53:56 [vllm.py:624] Asynchronous scheduling is enabled.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(EngineCore_DP0 pid=3720983) INFO 02-17 00:53:57 [core.py:96] Initializing a V1 LLM engine (v0.15.1) with config: model='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', speculative_config=None, tokenizer='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [4096], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 256, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': True}, 'local_cache_dir': None, 'static_all_moe_layers': []}
(EngineCore_DP0 pid=3720983) INFO 02-17 00:53:59 [parallel_state.py:1212] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://216.81.248.184:40295 backend=nccl
(EngineCore_DP0 pid=3720983) INFO 02-17 00:53:59 [parallel_state.py:1423] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
(EngineCore_DP0 pid=3720983) INFO 02-17 00:54:00 [gpu_model_runner.py:4033] Starting to load model /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM...
(EngineCore_DP0 pid=3720983) INFO 02-17 00:54:01 [cuda.py:364] Using FLASH_ATTN attention backend out of potential backends: ('FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION')
(EngineCore_DP0 pid=3720983) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
(EngineCore_DP0 pid=3720983) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
(EngineCore_DP0 pid=3720983)
Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
(EngineCore_DP0 pid=3720983)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.64it/s]
(EngineCore_DP0 pid=3720983)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.64it/s]
(EngineCore_DP0 pid=3720983)
(EngineCore_DP0 pid=3720983) INFO 02-17 00:54:01 [default_loader.py:291] Loading weights took 0.67 seconds
(EngineCore_DP0 pid=3720983) INFO 02-17 00:54:02 [gpu_model_runner.py:4130] Model loading took 0.95 GiB memory and 1.180564 seconds
(EngineCore_DP0 pid=3720983) INFO 02-17 00:54:07 [backends.py:812] Using cache directory: /home/ubuntu/.cache/vllm/torch_compile_cache/f3ff13511c/rank_0_0/backbone for vLLM's torch.compile
(EngineCore_DP0 pid=3720983) INFO 02-17 00:54:07 [backends.py:872] Dynamo bytecode transform time: 4.21 s
(EngineCore_DP0 pid=3720983) INFO 02-17 00:54:09 [backends.py:267] Directly load the compiled graph(s) for compile range (1, 4096) from the cache, took 0.621 s
(EngineCore_DP0 pid=3720983) INFO 02-17 00:54:09 [monitor.py:34] torch.compile takes 4.83 s in total
(EngineCore_DP0 pid=3720983) INFO 02-17 00:54:10 [gpu_worker.py:356] Available KV cache memory: 10.13 GiB
(EngineCore_DP0 pid=3720983) INFO 02-17 00:54:10 [kv_cache_utils.py:1307] GPU KV cache size: 885,184 tokens
(EngineCore_DP0 pid=3720983) INFO 02-17 00:54:10 [kv_cache_utils.py:1312] Maximum concurrency for 4,096 tokens per request: 216.11x
(EngineCore_DP0 pid=3720983)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   0%|          | 0/35 [00:00<?, ?it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  11%|█▏        | 4/35 [00:00<00:00, 32.86it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  23%|██▎       | 8/35 [00:00<00:00, 35.37it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  34%|███▍      | 12/35 [00:00<00:00, 36.44it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  46%|████▌     | 16/35 [00:00<00:00, 36.36it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  57%|█████▋    | 20/35 [00:00<00:00, 36.86it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  69%|██████▊   | 24/35 [00:00<00:00, 37.44it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  80%|████████  | 28/35 [00:00<00:00, 37.25it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  91%|█████████▏| 32/35 [00:00<00:00, 35.64it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 35/35 [00:00<00:00, 35.46it/s]
(EngineCore_DP0 pid=3720983)
Capturing CUDA graphs (decode, FULL):   0%|          | 0/19 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL):  21%|██        | 4/19 [00:00<00:00, 31.85it/s]
Capturing CUDA graphs (decode, FULL):  42%|████▏     | 8/19 [00:00<00:00, 35.28it/s]
Capturing CUDA graphs (decode, FULL):  63%|██████▎   | 12/19 [00:00<00:00, 36.42it/s]
Capturing CUDA graphs (decode, FULL):  84%|████████▍ | 16/19 [00:00<00:00, 36.88it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 36.54it/s]
(EngineCore_DP0 pid=3720983) INFO 02-17 00:54:13 [gpu_model_runner.py:5063] Graph capturing finished in 2 secs, took 0.28 GiB
(EngineCore_DP0 pid=3720983) INFO 02-17 00:54:13 [core.py:272] init engine (profile, create kv cache, warmup model) took 10.75 seconds
[2026-02-17 00:54:14] INFO tts_runtime.py:187: Initializing prompt builder...
[2026-02-17 00:54:14] INFO tts_runtime.py:194: Initializing BiCodec decoder...
/home/ubuntu/.local/lib/python3.10/site-packages/torch/nn/utils/weight_norm.py:144: FutureWarning: `torch.nn.utils.weight_norm` is deprecated in favor of `torch.nn.utils.parametrizations.weight_norm`.
  WeightNorm.apply(module, name, dim)


Output:
Chunk ID: 2fbf58
Wall time: 30.0009 seconds
Process running with session ID 63734
Original token count: 102
Output:
[2026-02-17 00:54:22] INFO tts_runtime.py:201: Initializing TTS pipeline...
[2026-02-17 00:54:22] INFO tts_runtime.py:212: Initializing streaming pipeline...
[2026-02-17 00:54:23] INFO tts_runtime.py:262: ✅ TTS runtime initialized in 32532ms
[2026-02-17 00:54:23] INFO tts_runtime.py:263:    Model version: LLM
[2026-02-17 00:54:23] INFO tts_runtime.py:308: Pre-computing global tokens for all speakers...


Output:
Chunk ID: ec54bd
Wall time: 30.0011 seconds
Process running with session ID 63734
Original token count: 0
Output:


Output:
Chunk ID: c22f28
Wall time: 30.0012 seconds
Process running with session ID 63734
Original token count: 0
Output:


Output:
Chunk ID: d9a1a5
Wall time: 30.0017 seconds
Process running with session ID 63734
Original token count: 0
Output:


Output:
Chunk ID: 0e9b8c
Wall time: 30.0011 seconds
Process running with session ID 63734
Original token count: 0
Output:


I identified a major blocker: with `VEENA3_PIPELINE_BATCH_TIMEOUT_SECONDS` and request timeout wrappers active, API calls are cancelled at 120s while direct pipeline code can complete when used in-process. I’m going to run one more controlled server with those per-batch/overall timeouts removed to see whether the route itself is the bottleneck or if the integration still stalls.


pkill -f 'python3 -m veena3modal.local_server' || true
ps -ef | rg -n 'python3 -m veena3modal.local_server' || true


Output:
Chunk ID: 68a4e1
Wall time: 0.0505 seconds
Process exited with code -1
Original token count: 0
Output: