Output:
Chunk ID: e0b580
Wall time: 24.9880 seconds
Process exited with code 0
Original token count: 2147
Output:
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
INFO 02-17 00:49:13 [model.py:541] Resolved architecture: Qwen2ForCausalLM
INFO 02-17 00:49:13 [model.py:1561] Using max model len 4096
INFO 02-17 00:49:13 [scheduler.py:226] Chunked prefill is enabled with max_num_batched_tokens=2048.
INFO 02-17 00:49:13 [vllm.py:624] Asynchronous scheduling is enabled.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(EngineCore_DP0 pid=3718849) INFO 02-17 00:49:13 [core.py:96] Initializing a V1 LLM engine (v0.15.1) with config: model='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', speculative_config=None, tokenizer='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 256, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': True}, 'local_cache_dir': None, 'static_all_moe_layers': []}
(EngineCore_DP0 pid=3718849) INFO 02-17 00:49:16 [parallel_state.py:1212] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://216.81.248.184:56593 backend=nccl
(EngineCore_DP0 pid=3718849) INFO 02-17 00:49:16 [parallel_state.py:1423] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
(EngineCore_DP0 pid=3718849) INFO 02-17 00:49:16 [gpu_model_runner.py:4033] Starting to load model /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM...
(EngineCore_DP0 pid=3718849) INFO 02-17 00:49:17 [cuda.py:364] Using FLASH_ATTN attention backend out of potential backends: ('FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION')
(EngineCore_DP0 pid=3718849) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
(EngineCore_DP0 pid=3718849) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
(EngineCore_DP0 pid=3718849)
Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
(EngineCore_DP0 pid=3718849)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.89it/s]
(EngineCore_DP0 pid=3718849)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.89it/s]
(EngineCore_DP0 pid=3718849)
(EngineCore_DP0 pid=3718849) INFO 02-17 00:49:18 [default_loader.py:291] Loading weights took 0.61 seconds
(EngineCore_DP0 pid=3718849) INFO 02-17 00:49:18 [gpu_model_runner.py:4130] Model loading took 0.95 GiB memory and 1.125535 seconds
(EngineCore_DP0 pid=3718849) INFO 02-17 00:49:23 [backends.py:812] Using cache directory: /home/ubuntu/.cache/vllm/torch_compile_cache/5591bfed59/rank_0_0/backbone for vLLM's torch.compile
(EngineCore_DP0 pid=3718849) INFO 02-17 00:49:23 [backends.py:872] Dynamo bytecode transform time: 4.18 s
(EngineCore_DP0 pid=3718849) INFO 02-17 00:49:26 [backends.py:267] Directly load the compiled graph(s) for compile range (1, 2048) from the cache, took 0.655 s
(EngineCore_DP0 pid=3718849) INFO 02-17 00:49:26 [monitor.py:34] torch.compile takes 4.84 s in total
(EngineCore_DP0 pid=3718849) INFO 02-17 00:49:27 [gpu_worker.py:356] Available KV cache memory: 10.13 GiB
(EngineCore_DP0 pid=3718849) INFO 02-17 00:49:27 [kv_cache_utils.py:1307] GPU KV cache size: 885,424 tokens
(EngineCore_DP0 pid=3718849) INFO 02-17 00:49:27 [kv_cache_utils.py:1312] Maximum concurrency for 4,096 tokens per request: 216.17x
(EngineCore_DP0 pid=3718849)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   0%|          | 0/35 [00:00<?, ?it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  11%|█▏        | 4/35 [00:00<00:00, 32.64it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  23%|██▎       | 8/35 [00:00<00:00, 35.61it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  34%|███▍      | 12/35 [00:00<00:00, 36.83it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  46%|████▌     | 16/35 [00:00<00:00, 37.33it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  57%|█████▋    | 20/35 [00:00<00:00, 37.55it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  69%|██████▊   | 24/35 [00:00<00:00, 37.70it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  80%|████████  | 28/35 [00:00<00:00, 37.36it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  91%|█████████▏| 32/35 [00:00<00:00, 35.50it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 35/35 [00:00<00:00, 35.56it/s]
(EngineCore_DP0 pid=3718849)
Capturing CUDA graphs (decode, FULL):   0%|          | 0/19 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL):  16%|█▌        | 3/19 [00:00<00:00, 27.99it/s]
Capturing CUDA graphs (decode, FULL):  37%|███▋      | 7/19 [00:00<00:00, 33.32it/s]
Capturing CUDA graphs (decode, FULL):  58%|█████▊    | 11/19 [00:00<00:00, 34.95it/s]
Capturing CUDA graphs (decode, FULL):  79%|███████▉  | 15/19 [00:00<00:00, 35.65it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 36.40it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 35.26it/s]
(EngineCore_DP0 pid=3718849) INFO 02-17 00:49:29 [gpu_model_runner.py:5063] Graph capturing finished in 2 secs, took 0.28 GiB
(EngineCore_DP0 pid=3718849) INFO 02-17 00:49:29 [core.py:272] init engine (profile, create kv cache, warmup model) took 10.74 seconds
chunk x False 1
chunk x False 2
chunk x False 3
elapsed 0.05040097236633301 chunks 3


# Collaboration Mode: Default

You are now in Default mode. Any previous instructions for other modes (e.g. Plan mode) are no longer active.

Your active mode changes only when new developer instructions with a different `<collaboration_mode>...</collaboration_mode>` change it; user requests or tool descriptions do not change mode by themselves. Known mode names are Default and Plan.

## request_user_input availability

The `request_user_input` tool is unavailable in Default mode. If you call it while in Default mode, it will return an error.

If a decision is necessary and cannot be discovered from local context, ask the user directly. However, in Default mode you should strongly prefer executing the user's request rather than stopping to ask questions.

Summary: auto

## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) When `SKILL.md` references relative paths (e.g., `scripts/foo.py`), resolve them relative to the skill directory listed above first, and only consider other paths if needed.
  3) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  4) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  5) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.


cd /home/ubuntu/veenaModal && NUMBA_DISABLE_JIT=1 python3 - <<'PY'
import asyncio, types
from vllm import AsyncLLMEngine, AsyncEngineArgs, SamplingParams

async def main():
    engine=AsyncLLMEngine.from_engine_args(AsyncEngineArgs(model='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', trust_remote_code=True, dtype='bfloat16', tensor_parallel_size=1, max_model_len=4096, gpu_memory_utilization=0.15, enable_chunked_prefill=True, enable_prefix_caching=True))
    sp=SamplingParams(max_tokens=16)
    gen=engine.generate(prompt='hello', sampling_params=sp, request_id='x')
    print('type',type(gen))
    print('is awaitable',hasattr(gen,'__await__'))
    print('is async generator',isinstance(gen, types.AsyncGeneratorType))
    try:
        r=await gen
        print('await returned',type(r))
    except Exception as e:
        print('await error',type(e).__name__,e)
    await engine.aclose()

asyncio.run(main())
PY


Output:
Chunk ID: 7c560b
Wall time: 24.9551 seconds
Process exited with code 1
Original token count: 2290
Output:
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
INFO 02-17 00:49:41 [model.py:541] Resolved architecture: Qwen2ForCausalLM
INFO 02-17 00:49:41 [model.py:1561] Using max model len 4096
INFO 02-17 00:49:41 [scheduler.py:226] Chunked prefill is enabled with max_num_batched_tokens=2048.
INFO 02-17 00:49:41 [vllm.py:624] Asynchronous scheduling is enabled.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(EngineCore_DP0 pid=3719100) INFO 02-17 00:49:42 [core.py:96] Initializing a V1 LLM engine (v0.15.1) with config: model='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', speculative_config=None, tokenizer='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 256, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': True}, 'local_cache_dir': None, 'static_all_moe_layers': []}
(EngineCore_DP0 pid=3719100) INFO 02-17 00:49:44 [parallel_state.py:1212] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://216.81.248.184:57703 backend=nccl
(EngineCore_DP0 pid=3719100) INFO 02-17 00:49:44 [parallel_state.py:1423] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
(EngineCore_DP0 pid=3719100) INFO 02-17 00:49:45 [gpu_model_runner.py:4033] Starting to load model /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM...
(EngineCore_DP0 pid=3719100) INFO 02-17 00:49:45 [cuda.py:364] Using FLASH_ATTN attention backend out of potential backends: ('FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION')
(EngineCore_DP0 pid=3719100) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
(EngineCore_DP0 pid=3719100) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
(EngineCore_DP0 pid=3719100)
Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
(EngineCore_DP0 pid=3719100)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.90it/s]
(EngineCore_DP0 pid=3719100)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.90it/s]
(EngineCore_DP0 pid=3719100)
(EngineCore_DP0 pid=3719100) INFO 02-17 00:49:46 [default_loader.py:291] Loading weights took 0.60 seconds
(EngineCore_DP0 pid=3719100) INFO 02-17 00:49:47 [gpu_model_runner.py:4130] Model loading took 0.95 GiB memory and 1.113292 seconds
(EngineCore_DP0 pid=3719100) INFO 02-17 00:49:51 [backends.py:812] Using cache directory: /home/ubuntu/.cache/vllm/torch_compile_cache/5591bfed59/rank_0_0/backbone for vLLM's torch.compile
(EngineCore_DP0 pid=3719100) INFO 02-17 00:49:51 [backends.py:872] Dynamo bytecode transform time: 4.14 s
(EngineCore_DP0 pid=3719100) INFO 02-17 00:49:54 [backends.py:267] Directly load the compiled graph(s) for compile range (1, 2048) from the cache, took 0.652 s
(EngineCore_DP0 pid=3719100) INFO 02-17 00:49:54 [monitor.py:34] torch.compile takes 4.79 s in total
(EngineCore_DP0 pid=3719100) INFO 02-17 00:49:55 [gpu_worker.py:356] Available KV cache memory: 10.13 GiB
(EngineCore_DP0 pid=3719100) INFO 02-17 00:49:55 [kv_cache_utils.py:1307] GPU KV cache size: 885,424 tokens
(EngineCore_DP0 pid=3719100) INFO 02-17 00:49:55 [kv_cache_utils.py:1312] Maximum concurrency for 4,096 tokens per request: 216.17x
(EngineCore_DP0 pid=3719100)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   0%|          | 0/35 [00:00<?, ?it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  11%|█▏        | 4/35 [00:00<00:00, 32.52it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  23%|██▎       | 8/35 [00:00<00:00, 34.29it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  34%|███▍      | 12/35 [00:00<00:00, 35.70it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  46%|████▌     | 16/35 [00:00<00:00, 36.62it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  57%|█████▋    | 20/35 [00:00<00:00, 36.41it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  69%|██████▊   | 24/35 [00:00<00:00, 36.56it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  80%|████████  | 28/35 [00:00<00:00, 36.53it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  91%|█████████▏| 32/35 [00:00<00:00, 34.72it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 35/35 [00:01<00:00, 34.69it/s]
(EngineCore_DP0 pid=3719100)
Capturing CUDA graphs (decode, FULL):   0%|          | 0/19 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL):  16%|█▌        | 3/19 [00:00<00:00, 28.90it/s]
Capturing CUDA graphs (decode, FULL):  37%|███▋      | 7/19 [00:00<00:00, 33.81it/s]
Capturing CUDA graphs (decode, FULL):  58%|█████▊    | 11/19 [00:00<00:00, 35.07it/s]
Capturing CUDA graphs (decode, FULL):  79%|███████▉  | 15/19 [00:00<00:00, 35.54it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 36.09it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 35.21it/s]
(EngineCore_DP0 pid=3719100) INFO 02-17 00:49:58 [gpu_model_runner.py:5063] Graph capturing finished in 2 secs, took 0.28 GiB
(EngineCore_DP0 pid=3719100) INFO 02-17 00:49:58 [core.py:272] init engine (profile, create kv cache, warmup model) took 10.72 seconds
type <class 'async_generator'>
is awaitable False
is async generator True
await error TypeError object async_generator can't be used in 'await' expression
Traceback (most recent call last):
  File "<stdin>", line 18, in <module>
  File "/usr/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "<stdin>", line 16, in main
AttributeError: 'AsyncLLM' object has no attribute 'aclose'
ERROR 02-17 00:49:59 [core_client.py:605] Engine core proc EngineCore_DP0 died unexpectedly, shutting down client.


Summary: auto


cd /home/ubuntu/veenaModal && python3 - <<'PY'
import requests, time
url='http://localhost:8000/v1/tts/generate'
payload={"text":"Hello","speaker":"lipakshi","max_tokens":128,"stream":True,"chunking":False}
with requests.post(url,json=payload,stream=True,timeout=(5,180)) as r:
    print('status',r.status_code,'headers', {k:v for k,v in r.headers.items() if k.startswith('x-')})
    t0=time.time()
    count=0
    for chunk in r.iter_content(chunk_size=1024):
        if not chunk:
            continue
        count += 1
        print('chunk',count,'len',len(chunk),'time',time.time()-t0)
        if count>=10:
            break
    print('done',count,'elapsed',time.time()-t0)
PY


Output:
Chunk ID: 1f1402
Wall time: 30.0010 seconds
Process running with session ID 47578
Original token count: 0
Output:


Output:
Chunk ID: 284264
Wall time: 30.0006 seconds
Process running with session ID 47578
Original token count: 0
Output:


Output:
Chunk ID: 19ef68
Wall time: 30.0010 seconds
Process running with session ID 37173
Original token count: 156
Output:
{"timestamp": "2026-02-17T00:50:03.945967+00:00", "level": "INFO", "logger": "veena3modal.tts", "message": "request_received", "request_id": "dfe5c442-4157-4218-979b-07b557458885", "event": "request_received", "text_length": 5, "speaker": "lipakshi", "stream": true, "format": "wav"}
2026-02-17 00:50:03 | INFO     | veena3modal.tts | request_received
INFO:     127.0.0.1:41238 - "POST /v1/tts/generate HTTP/1.1" 200 OK
2026-02-17 00:50:03 | INFO     | veena3modal.services.tts_runtime | tts_runtime_stream_start
2026-02-17 00:50:03 | INFO     | veena3modal.processing.long_text_processor | LongTextProcessor initialized


I need one more quick isolation check: whether the server route is being called with the expected payload contract. I’m checking the local FastAPI docs JSON to pull the request schema examples used by `/v1/tts/generate`.


python3 - <<'PY'
import requests, re
html=requests.get('http://localhost:8000/docs',timeout=10).text
for p in ['max_tokens','chunking','stream','output_sample_rate','speaker','text','TTSGenerateRequest','generate']:
    if p in html:
        print(p,'found')
print('len',len(html))
print(html[:500])
PY


Output:
Chunk ID: bf36da
Wall time: 0.0513 seconds
Process exited with code 0
Original token count: 131
Output:
text found
len 1017

    <!DOCTYPE html>
    <html>
    <head>
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <link type="text/css" rel="stylesheet" href="https://cdn.jsdelivr.net/npm/swagger-ui-dist@5/swagger-ui.css">
    <link rel="shortcut icon" href="https://fastapi.tiangolo.com/img/favicon.png">
    <title>Veena3 TTS (Modal) - Swagger UI</title>
    </head>
    <body>
    <div id="swagger-ui">
    </div>
    <script src="https://cdn.jsdelivr.net/npm/swagger-ui-dist@5/swagger-ui-


cd /home/ubuntu/veenaModal && rg -n "VEENA3_PIPELINE_BATCH_TIMEOUT_SECONDS|_batch_wait_timeout_seconds|tts_llm_batch_timeout|wait_for\(" veena3modal/core/pipeline.py


cd /home/ubuntu/veenaModal && rg -n "VEENA3_REQUEST_TIMEOUT_SECONDS|tts_runtime_nonstream|tts_runtime_chunk|wait_for\(" veena3modal/services/tts_runtime.py


Output:
Chunk ID: 622510
Wall time: 0.0506 seconds
Process exited with code 0
Original token count: 75
Output:
41:def _batch_wait_timeout_seconds() -> float:
43:    raw = os.getenv("VEENA3_PIPELINE_BATCH_TIMEOUT_SECONDS", "").strip()
133:        batch_timeout = _batch_wait_timeout_seconds()
172:                    request_output = await asyncio.wait_for(
181:                        "tts_llm_batch_timeout",


Output:
Chunk ID: 8529c4
Wall time: 0.0516 seconds
Process exited with code 0
Original token count: 173
Output:
41:    raw = os.getenv("VEENA3_REQUEST_TIMEOUT_SECONDS", "").strip()
438:            "tts_runtime_nonstream_start",
472:                    audio_bytes, pipeline_timings = await asyncio.wait_for(
489:                    audio_bytes = await asyncio.wait_for(
554:            "tts_runtime_nonstream_error",
568:                "tts_runtime_nonstream_complete",
573:                "tts_runtime_nonstream_inflight",
699:                "tts_runtime_chunking_start",
770:                        audio_bytes, pipeline_timings = await asyncio.wait_for(
788:                        audio_bytes = await asyncio.wait_for(
850:            logger.info("tts_runtime_chunking_complete", extra=metrics)


cd /home/ubuntu/veenaModal && sed -n '35,70p' veena3modal/core/pipeline.py


cd /home/ubuntu/veenaModal && sed -n '120,210p' veena3modal/core/pipeline.py


cd /home/ubuntu/veenaModal && sed -n '470,820p' veena3modal/services/tts_runtime.py


Output:
Chunk ID: 74eb4a
Wall time: 0.0509 seconds
Process exited with code 0
Original token count: 239
Output:
    Controlled by environment variable:
      VEENA3_TIMING_LOGS=true|1
    """
    return os.getenv("VEENA3_TIMING_LOGS", "").lower() in {"1", "true", "yes", "on"}

def _batch_wait_timeout_seconds() -> float:
    """Optional hard timeout (seconds) for waiting on each vLLM iterator batch."""
    raw = os.getenv("VEENA3_PIPELINE_BATCH_TIMEOUT_SECONDS", "").strip()
    if not raw:
        return 0.0
    try:
        return max(0.0, float(raw))
    except ValueError:
        return 0.0

class SparkTTSPipeline:
    """
    End-to-end TTS pipeline for Spark TTS with BiCodec.

    Replaces SNAC-based token extraction with BiCodec regex-based extraction.
    """

    def __init__(
        self,
        model,
        prompt_builder,
        bicodec_decoder,
    ):
        """
        Initialize pipeline.

        Args:
            model: SparkTTS Model instance
            prompt_builder: IndicPromptBuilder instance (with Spark TTS format)


Output:
Chunk ID: 62ae0c
Wall time: 0.0516 seconds
Process exited with code 0
Original token count: 876
Output:
            max_tokens=max_tokens,
            repetition_penalty=repetition_penalty,
            stop=TRAINING_STOP_TOKEN_IDS,
            skip_special_tokens=False,
            seed=seed,
        )

        # Use vLLM engine to generate
        if request_id is None:
            import uuid
            request_id = f"req-{uuid.uuid4().hex[:12]}"

        emit_timing = return_timing or _detailed_timing_enabled()
        batch_timeout = _batch_wait_timeout_seconds()
        results_generator = self.model.engine.generate(
            prompt=prompt,
            sampling_params=sampling_params,
            request_id=request_id,
        )

        # Collect all outputs and track generation timing
        final_output = None
        token_batches: List[int] = []
        batch_wall_ms: List[float] = []
        batch_gpu_ms: List[float] = []
        total_token_ids = 0
        first_token_ms: Optional[float] = None
        t_start = asyncio.get_event_loop().time()
        if emit_timing:
            logger.info(
                "tts_llm_generate_start",
                extra={
                    "speaker": speaker,
                    "request_id": request_id,
                    "max_tokens": max_tokens,
                    "batch_timeout_seconds": batch_timeout,
                },
            )

        iterator = results_generator.__aiter__()
        while True:
            wall_start = asyncio.get_event_loop().time()
            start_event = None
            end_event = None
            if torch.cuda.is_available():
                torch.cuda.synchronize()
                start_event = torch.cuda.Event(enable_timing=True)
                end_event = torch.cuda.Event(enable_timing=True)
                start_event.record()

            try:
                if batch_timeout:
                    request_output = await asyncio.wait_for(
                        iterator.__anext__(),
                        timeout=batch_timeout,
                    )
                else:
                    request_output = await iterator.__anext__()
            except asyncio.TimeoutError as exc:
                if emit_timing:
                    logger.error(
                        "tts_llm_batch_timeout",
                        extra={
                            "speaker": speaker,
                            "request_id": request_id,
                            "batch_index": len(token_batches) + 1,
                            "batch_timeout_seconds": batch_timeout,
                        },
                    )
                raise TimeoutError(
                    f"LLM batch wait timed out after {batch_timeout:.2f}s"
                ) from exc
            except StopAsyncIteration:
                break

            wall_end = asyncio.get_event_loop().time()
            batch_ms = (wall_end - wall_start) * 1000
            batch_wall_ms.append(batch_ms)

            batch_gpu_ms_value: Optional[float] = None
            if start_event is not None and end_event is not None:
                end_event.record()
                torch.cuda.synchronize()
                batch_gpu_ms_value = start_event.elapsed_time(end_event)
            if batch_gpu_ms_value is not None:
                batch_gpu_ms.append(batch_gpu_ms_value)

            total_generated = len(request_output.outputs[0].token_ids)
            new_tokens = max(0, total_generated - total_token_ids)
            total_token_ids = total_generated
            token_batches.append(new_tokens)


Output:
Chunk ID: 65432c
Wall time: 0.0506 seconds
Process exited with code 0
Original token count: 3538
Output:
            if generation_timeout:
                try:
                    audio_bytes, pipeline_timings = await asyncio.wait_for(
                        generation_coro,
                        timeout=generation_timeout,
                    )
                except asyncio.TimeoutError as exc:
                    raise TimeoutError(
                        f"Non-stream generation timed out after {generation_timeout:.2f}s"
                    ) from exc
            else:
                audio_bytes, pipeline_timings = await generation_coro
        else:
            generation_coro = runtime.pipeline.generate_speech_indic(
                **gen_kwargs,
                request_id=request_id,
            )
            if generation_timeout:
                try:
                    audio_bytes = await asyncio.wait_for(
                        generation_coro,
                        timeout=generation_timeout,
                    )
                except asyncio.TimeoutError as exc:
                    raise TimeoutError(
                        f"Non-stream generation timed out after {generation_timeout:.2f}s"
                    ) from exc
            else:
                audio_bytes = await generation_coro

        generation_time = time.time() - start_time
        metrics["generation_ms"] = int(generation_time * 1000)
        metrics["ttfb_ms"] = metrics["generation_ms"]  # Non-streaming: TTFB = total time

        if pipeline_timings:
            metrics.update({
                "llm_token_total": pipeline_timings.get("token_total", 0),
                "llm_batch_count": pipeline_timings.get("batch_count", 0),
                "llm_batch_gpu_ms_total": pipeline_timings.get("llm_gpu_batch_ms_total", 0),
                "llm_batch_gpu_ms_min": pipeline_timings.get("llm_gpu_batch_ms_min", 0),
                "llm_batch_gpu_ms_max": pipeline_timings.get("llm_gpu_batch_ms_max", 0),
                "llm_batch_gpu_ms_p50": pipeline_timings.get("llm_gpu_batch_ms_p50", 0),
                "llm_decode_cpu_ms": pipeline_timings.get("decode_cpu_ms", 0),
                "llm_parse_ms": pipeline_timings.get("parse_ms", 0),
                "llm_wall_ms": pipeline_timings.get("llm_wall_ms", metrics["generation_ms"]),
                "ttfb_ms": int(pipeline_timings.get("ttfb_ms", metrics["ttfb_ms"])),
                "tokens_per_batch_min": pipeline_timings.get("tokens_per_batch_min"),
                "tokens_per_batch_max": pipeline_timings.get("tokens_per_batch_max"),
                "tokens_per_batch_p50": pipeline_timings.get("tokens_per_batch_p50"),
            })

        if audio_bytes:
            # Apply super-resolution if requested and available
            sample_rate = 16000
            logger.info(f"SR check: output_sample_rate={output_sample_rate!r}, "
                       f"sr_service={runtime.sr_service is not None}, "
                       f"is_loaded={runtime.sr_service.is_loaded if runtime.sr_service else False}")

            if output_sample_rate == "48khz" and runtime.sr_service and runtime.sr_service.is_loaded:
                try:
                    sr_start = time.time()
                    logger.info(f"Applying super-resolution to {len(audio_bytes)} bytes...")
                    audio_bytes = _apply_super_resolution(audio_bytes, runtime.sr_service)
                    sr_time = (time.time() - sr_start) * 1000
                    metrics["sr_ms"] = int(sr_time)
                    metrics["sr_applied"] = True
                    sample_rate = 48000
                    logger.info(f"✅ Super-resolution applied in {sr_time:.1f}ms, output={len(audio_bytes)} bytes")
                except Exception as e:
                    logger.warning(f"Super-resolution failed, returning 16kHz: {e}")
                    import traceback
                    traceback.print_exc()

            metrics["output_sample_rate"] = sample_rate

            # Calculate audio duration
            audio_duration = (len(audio_bytes) - 44) / (sample_rate * 2)  # -44 for WAV header
            metrics["audio_duration_seconds"] = max(0.0, audio_duration)

        return audio_bytes, metrics

    except Exception as e:
        logger.error(f"Speech generation failed: {e}")
        logger.exception(
            "tts_runtime_nonstream_error",
            extra={
                "speaker": speaker,
                "request_id": request_id,
                "error": str(e),
            },
        )
        raise
    finally:
        active_requests = _with_inflight_delta(-1)
        if _timing_enabled():
            metrics["request_inflight"] = active_requests
            metrics["active_requests"] = active_requests
            logger.info(
                "tts_runtime_nonstream_complete",
                extra=metrics,
            )
        if _timing_enabled():
            logger.info(
                "tts_runtime_nonstream_inflight",
                extra={
                    "request_id": request_id,
                    "active_requests": active_requests,
                },
            )

def _apply_super_resolution(audio_bytes: bytes, sr_service) -> bytes:
    """
    Apply super-resolution to audio bytes.

    Args:
        audio_bytes: WAV audio at 16kHz
        sr_service: SuperResolutionService instance

    Returns:
        WAV audio at 48kHz
    """
    import numpy as np
    import struct
    import torch

    # Parse WAV header and extract PCM data
    if len(audio_bytes) < 44:
        raise ValueError("Invalid WAV data")

    # Extract PCM data (skip 44-byte WAV header)
    pcm_data = np.frombuffer(audio_bytes[44:], dtype=np.int16)

    # Convert to float32 for SR model
    audio_float = pcm_data.astype(np.float32) / 32768.0

    # Convert numpy to torch tensor for process_chunk
    audio_tensor = torch.from_numpy(audio_float)

    # Apply super-resolution using process_chunk (16kHz -> 48kHz)
    # process_chunk expects [batch, samples] or [samples] and returns [batch, samples]
    upsampled_tensor = sr_service.process_chunk(audio_tensor)

    # Convert back to numpy, squeeze batch dim if present
    upsampled = upsampled_tensor.squeeze().cpu().numpy()

    # Convert back to int16
    upsampled_int16 = np.clip(upsampled * 32768.0, -32768, 32767).astype(np.int16)

    # Create new WAV header for 48kHz
    sample_rate = 48000
    num_channels = 1
    bits_per_sample = 16
    byte_rate = sample_rate * num_channels * bits_per_sample // 8
    block_align = num_channels * bits_per_sample // 8
    data_size = len(upsampled_int16) * 2

    wav_header = struct.pack(
        '<4sI4s4sIHHIIHH4sI',
        b'RIFF',
        36 + data_size,
        b'WAVE',
        b'fmt ',
        16,  # fmt chunk size
        1,   # PCM format
        num_channels,
        sample_rate,
        byte_rate,
        block_align,
        bits_per_sample,
        b'data',
        data_size,
    )

    return wav_header + upsampled_int16.tobytes()

async def generate_speech_chunked(
    text: str,
    speaker: str,
    temperature: float = 0.8,
    top_k: int = 50,
    top_p: float = 1.0,
    max_tokens: int = 4096,
    repetition_penalty: float = 1.05,
    seed: Optional[int] = None,
    sample_rate: int = 16000,
    output_sample_rate: str = "16khz",
    request_id: Optional[str] = None,
) -> Tuple[Optional[bytes], Dict[str, Any]]:
    """
    Generate speech with automatic text chunking for long inputs.

    Uses LongTextProcessor to split text and stitch audio.

    Args:
        text: Text to synthesize
        speaker: Internal speaker name
        ... (same as generate_speech)
        sample_rate: Internal generation sample rate (always 16000)
        output_sample_rate: "16khz" or "48khz" (triggers super-resolution)

    Returns:
        Tuple of (audio_bytes, metrics_dict)
    """
    if not is_initialized():
        raise RuntimeError("TTS runtime not initialized")

    runtime = get_runtime()
    if request_id is None:
        import uuid
        request_id = f"runtime-chunk-{uuid.uuid4().hex[:10]}"
    metrics = {
        "request_id": request_id,
        "ttfb_ms": 0,
        "generation_ms": 0,
        "chunks_processed": 0,
        "audio_duration_seconds": 0.0,
        "text_chunked": False,
        "sr_applied": False,
        "output_sample_rate": 16000,
    }

    active_requests = _with_inflight_delta(1)
    start_time = time.time()

    try:
        if _timing_enabled():
            logger.info(
                "tts_runtime_chunking_start",
                extra={
                    "speaker": speaker,
                    "request_id": request_id,
                    "text_length": len(text),
                    "request_inflight": active_requests,
                    "chunking": True,
                },
            )
        # Import long text processor
        from veena3modal.processing.long_text_processor import LongTextProcessor

        long_processor = LongTextProcessor(pipeline=runtime.pipeline)
        generation_timeout = _generation_timeout_seconds()

        # Check if chunking is needed
        if long_processor.should_chunk(text):
            metrics["text_chunked"] = True
            if _timing_enabled():
                chunk_result = await long_processor.generate_with_chunking(
                    text=text,
                    speaker=speaker,
                    temperature=temperature,
                    top_k=top_k,
                    top_p=top_p,
                    max_tokens=max_tokens,
                    repetition_penalty=repetition_penalty,
                    seed=seed,
                    sample_rate=sample_rate,
                    request_id=request_id,
                    return_timing=True,
                )
                if isinstance(chunk_result, tuple):
                    audio_bytes, chunk_timing = chunk_result
                else:
                    audio_bytes = chunk_result
                    chunk_timing = {}
            else:
                audio_bytes = await long_processor.generate_with_chunking(
                    text=text,
                    speaker=speaker,
                    temperature=temperature,
                    top_k=top_k,
                    top_p=top_p,
                    max_tokens=max_tokens,
                    repetition_penalty=repetition_penalty,
                    seed=seed,
                    sample_rate=sample_rate,
                    request_id=request_id,
                )
                chunk_timing = {}
        else:
            # Short text: direct generation
            gen_kwargs = dict(
                speaker=speaker,
                text=text,
                temperature=temperature,
                top_k=top_k,
                top_p=top_p,
                max_tokens=max_tokens,
                repetition_penalty=repetition_penalty,
                seed=seed,
            )
            if _timing_enabled():
                generation_coro = runtime.pipeline.generate_speech_indic(
                    **gen_kwargs,
                    return_timing=True,
                    request_id=request_id,
                )
                if generation_timeout:
                    try:
                        audio_bytes, pipeline_timings = await asyncio.wait_for(
                            generation_coro,
                            timeout=generation_timeout,
                        )
                    except asyncio.TimeoutError as exc:
                        raise TimeoutError(
                            f"Chunked generation timed out after {generation_timeout:.2f}s"
                        ) from exc
                else:
                    audio_bytes, pipeline_timings = await generation_coro
                chunk_timing = pipeline_timings
            else:
                generation_coro = runtime.pipeline.generate_speech_indic(
                    **gen_kwargs,
                    request_id=request_id,
                )
                if generation_timeout:
                    try:
                        audio_bytes = await asyncio.wait_for(
                            generation_coro,
                            timeout=generation_timeout,
                        )
                    except asyncio.TimeoutError as exc:
                        raise TimeoutError(
                            f"Chunked generation timed out after {generation_timeout:.2f}s"
                        ) from exc
                else:
                    audio_bytes = await generation_coro
                chunk_timing = {}

        generation_time = time.time() - start_time
        metrics["generation_ms"] = int(generation_time * 1000)
        metrics["ttfb_ms"] = metrics["generation_ms"]

        if chunk_timing:
            metrics.update({
                "llm_token_total": chunk_timing.get("token_total", 0),
                "llm_batch_count": chunk_timing.get("batch_count", 0),
                "llm_batch_gpu_ms_total": chunk_timing.get("llm_gpu_batch_ms_total", 0),
                "llm_batch_wall_ms_total": chunk_timing.get("llm_wall_ms", metrics["generation_ms"]),
                "llm_parse_ms": chunk_timing.get("parse_ms", 0),
                "llm_decode_cpu_ms": chunk_timing.get("decode_cpu_ms", 0),
                "llm_batch_wall_ms_min": chunk_timing.get("llm_batch_wall_ms_min", 0),
                "llm_batch_wall_ms_max": chunk_timing.get("llm_batch_wall_ms_max", 0),
                "llm_batch_gpu_ms_min": chunk_timing.get("llm_batch_gpu_ms_min", 0),
                "llm_batch_gpu_ms_max": chunk_timing.get("llm_batch_gpu_ms_max", 0),
                "llm_batch_gpu_ms_p50": chunk_timing.get("llm_batch_gpu_ms_p50", 0),
                "tokens_per_batch_min": chunk_timing.get("tokens_per_batch_min"),
                "tokens_per_batch_max": chunk_timing.get("tokens_per_batch_max"),
                "tokens_per_batch_p50": chunk_timing.get("tokens_per_batch_p50"),
                "llm_batch_tokens": chunk_timing.get("token_batches", []),


cd /home/ubuntu/veenaModal && sed -n '30,70p' veena3modal/services/tts_runtime.py


Output:
Chunk ID: daf60d
Wall time: 0.0512 seconds
Process exited with code 0
Original token count: 303
Output:
_inflight_counter = 0
_inflight_lock = Lock()

def _timing_enabled() -> bool:
    """Return True when detailed request timing should be emitted."""
    return os.getenv(_PROFILE_REQUEST_TIMING, "").lower() in {"1", "true", "yes", "on"}

def _generation_timeout_seconds() -> float:
    """Optional hard timeout (seconds) for each generation call."""
    raw = os.getenv("VEENA3_REQUEST_TIMEOUT_SECONDS", "").strip()
    if not raw:
        return 0.0
    try:
        value = float(raw)
    except ValueError:
        return 0.0
    return max(0.0, value)

def _with_inflight_delta(delta: int) -> int:
    """Update in-flight request counter and return the new value."""
    global _inflight_counter
    with _inflight_lock:
        _inflight_counter = max(0, _inflight_counter + delta)
        return _inflight_counter

@dataclass
class TTSRuntime:
    """
    Holds long-lived, per-container inference objects.

    Thread Safety:
        This class is designed for async concurrency within a single container.
        The vLLM engine handles internal batching and scheduling.
        Do NOT share across processes without proper synchronization.
    """
    model: Any = None
    pipeline: Any = None


cd /home/ubuntu/veenaModal && sed -n '210,320p' veena3modal/core/pipeline.py


Output:
Chunk ID: 240a78
Wall time: 0.0512 seconds
Process exited with code 0
Original token count: 1099
Output:
            token_batches.append(new_tokens)

            if first_token_ms is None:
                first_token_ms = (wall_end - t_start) * 1000

            if emit_timing:
                logger.info(
                    "tts_llm_batch",
                    extra={
                        "speaker": speaker,
                        "request_id": request_id,
                        "batch_index": len(token_batches),
                        "batch_tokens": new_tokens,
                        "llm_batch_wall_ms": round(batch_ms, 3),
                        "llm_batch_gpu_ms": round(batch_gpu_ms_value, 3)
                        if batch_gpu_ms_value is not None
                        else None,
                    },
                )
            final_output = request_output

        if final_output is None:
            if emit_timing:
                logger.warning(
                    "tts_llm_no_output",
                    extra={"speaker": speaker, "request_id": request_id},
                )
            print(f"❌ Generation failed: no output")
            if return_timing:
                return None, {
                    "request_id": request_id,
                    "ttfb_ms": 0,
                    "token_total": 0,
                    "batch_count": 0,
                }
            return None

        # Extract generated text (contains BiCodec tokens)
        parse_start = asyncio.get_event_loop().time()
        generated_text = final_output.outputs[0].text
        generated_ids = final_output.outputs[0].token_ids

        # Extract BiCodec tokens using regex
        semantic_ids, global_ids = self._extract_bicodec_tokens(generated_text)

        parse_ms = (asyncio.get_event_loop().time() - parse_start) * 1000

        if not semantic_ids or not global_ids:
            if emit_timing:
                logger.error(
                    "tts_bicodec_parse_failed",
                    extra={
                        "speaker": speaker,
                        "request_id": request_id,
                        "semantic_count": len(semantic_ids),
                        "global_count": len(global_ids),
                        "token_total": len(generated_ids),
                    },
                )
            logger.error(
                "❌ No BiCodec tokens found in output",
                extra={
                    "speaker": speaker,
                    "text_length": len(text),
                    "text_preview": text[:100],
                    "generated_tokens": len(generated_ids),
                    "semantic_tokens": len(semantic_ids),
                    "global_tokens": len(global_ids),
                    "generated_text_preview": generated_text[:500],
                }
            )
            if return_timing:
                return None, {
                    "request_id": request_id,
                    "ttfb_ms": int(first_token_ms or 0),
                    "token_total": len(generated_ids),
                    "batch_count": len(token_batches),
                }
            return None

        # Validate tokens
        if not self.bicodec_decoder.validate_tokens(semantic_ids, global_ids):
            logger.error("❌ BiCodec token validation failed", extra={
                "speaker": speaker,
                "text_length": len(text),
                "semantic_tokens": len(semantic_ids),
                "global_tokens": len(global_ids),
            })
            if return_timing:
                return None, {
                    "request_id": request_id,
                    "ttfb_ms": int(first_token_ms or 0),
                    "token_total": len(generated_ids),
                    "batch_count": len(token_batches),
                }
            return None

        # Decode to audio
        decode_start = asyncio.get_event_loop().time()
        audio_bytes = self.bicodec_decoder.decode_to_bytes(semantic_ids, global_ids)
        decode_ms = (asyncio.get_event_loop().time() - decode_start) * 1000

        if audio_bytes is None:
            if emit_timing:
                logger.error("tts_decode_failed", extra={"speaker": speaker, "request_id": request_id})
            logger.error("❌ BiCodec decode failed", extra={
                "speaker": speaker,
                "text_length": len(text),
            })
            if return_timing:
                return None, {


cd /home/ubuntu/veenaModal && sed -n '320,390p' veena3modal/core/pipeline.py


Output:
Chunk ID: bfd5e9
Wall time: 0.0506 seconds
Process exited with code 0
Original token count: 746
Output:
                return None, {
                    "request_id": request_id,
                    "ttfb_ms": int(first_token_ms or 0),
                    "token_total": len(generated_ids),
                    "batch_count": len(token_batches),
                }
            return None

        total_ms = (asyncio.get_event_loop().time() - t_start) * 1000

        timing_metrics: Dict[str, Any] = {
            "request_id": request_id,
            "speaker": speaker,
            "text_length": len(text),
            "batch_count": len(token_batches),
            "token_total": len(generated_ids),
            "token_batches": token_batches,
            "ttfb_ms": int(first_token_ms or 0),
            "llm_wall_ms": int(sum(batch_wall_ms)),
            "llm_avg_batch_ms": int(mean(batch_wall_ms)) if batch_wall_ms else 0,
            "llm_gpu_batch_ms_total": int(sum(batch_gpu_ms)),
            "llm_gpu_batch_ms_min": int(min(batch_gpu_ms)) if batch_gpu_ms else 0,
            "llm_gpu_batch_ms_max": int(max(batch_gpu_ms)) if batch_gpu_ms else 0,
            "llm_gpu_batch_ms_p50": int(median(batch_gpu_ms)) if batch_gpu_ms else 0,
            "decode_cpu_ms": int(decode_ms),
            "parse_ms": int(parse_ms),
            "total_ms": int(total_ms),
        }
        if token_batches:
            timing_metrics["tokens_per_batch_min"] = min(token_batches)
            timing_metrics["tokens_per_batch_max"] = max(token_batches)
            timing_metrics["tokens_per_batch_p50"] = int(median(token_batches))

        if emit_timing:
            logger.info("tts_llm_complete", extra=timing_metrics)

        from veena3modal.audio.utils import add_wav_header
        wav_bytes = add_wav_header(audio_bytes, sample_rate=AUDIO_SAMPLE_RATE)

        if return_timing:
            return wav_bytes, timing_metrics
        return wav_bytes

    async def generate_speech_indic(
        self,
        speaker: str,
        text: str,
        temperature: float = DEFAULT_TEMPERATURE,
        top_k: int = DEFAULT_TOP_K,
        top_p: float = DEFAULT_TOP_P,
        max_tokens: int = DEFAULT_MAX_TOKENS,
        repetition_penalty: float = 1.05,
        seed: Optional[int] = None,
        return_timing: bool = False,
        request_id: Optional[str] = None,
    ):
        """
        Generate speech audio for Spark TTS (speaker-based).

        This method is kept for backward compatibility with existing API.
        It delegates to generate_speech() with the same implementation.

        Args:
            speaker: Speaker name (one of 12 predefined speakers)
            text: Text to synthesize with inline emotion tags
                Examples:
                - "Hello! Welcome."
                - "Hello <laugh> this is fun!" (will be normalized)
                - "नमस्ते! [excited] आज का दिन बहुत अच्छा है।"
            temperature: Sampling temperature
            top_k: Top-k sampling


cd /home/ubuntu/veenaModal && rg -n "class .*Prompt|build_prompt|PromptBuilder|train|SparkPrompt|IndicPromptBuilder" veena3modal -g"*.py"


Output:
Chunk ID: 22a6de
Wall time: 0.0511 seconds
Process exited with code 0
Original token count: 457
Output:
veena3modal/local_server.py:116:            ignore_patterns=["*.bin", "training_args.bin", "optimizer.pt"],
veena3modal/services/tts_runtime.py:7:- IndicPromptBuilder
veena3modal/services/tts_runtime.py:162:        from veena3modal.processing.prompt_builder import IndicPromptBuilder
veena3modal/services/tts_runtime.py:188:        prompt_builder = IndicPromptBuilder(
veena3modal/core/pipeline.py:70:            prompt_builder: IndicPromptBuilder instance (with Spark TTS format)
veena3modal/core/model_loader.py:30:    Speakers: 12 predefined (mapped to speaker_0-11 as per training)
veena3modal/core/model_loader.py:89:        self.tokenizer = AutoTokenizer.from_pretrained(
veena3modal/core/streaming_pipeline.py:80:            prompt_builder: Veena3PromptBuilder instance
veena3modal/processing/prompt_builder.py:17:class IndicPromptBuilder:
veena3modal/processing/prompt_builder.py:27:    CRITICAL: This format MUST match Spark TTS training exactly.
veena3modal/processing/prompt_builder.py:33:    - Speakers: 12 predefined (case-sensitive!) - must match training map
veena3modal/tests/unit/test_prompt_builder.py:3:from veena3modal.processing.prompt_builder import IndicPromptBuilder
veena3modal/tests/unit/test_prompt_builder.py:13:    pb = IndicPromptBuilder(tokenizer=_DummyTokenizer())
veena3modal/tests/unit/test_prompt_builder.py:23:    pb = IndicPromptBuilder(tokenizer=_DummyTokenizer())
veena3modal/tests/unit/test_prompt_builder.py:29:    pb = IndicPromptBuilder(tokenizer=_DummyTokenizer())
veena3modal/tests/unit/test_prompt_builder.py:35:    pb = IndicPromptBuilder(tokenizer=_DummyTokenizer())
veena3modal/tests/unit/test_prompt_builder.py:46:    pb = IndicPromptBuilder(tokenizer=_DummyTokenizer())
veena3modal/tests/unit/test_prompt_builder.py:52:    pb = IndicPromptBuilder(tokenizer=_DummyTokenizer())


cd /home/ubuntu/veenaModal && sed -n '1,220p' veena3modal/processing/prompt_builder.py


Output:
Chunk ID: 874857
Wall time: 0.0510 seconds
Process exited with code 0
Original token count: 1960
Output:
"""
Spark TTS Indic Prompt Builder

Builds prompts for Spark TTS with Indic speaker system.
Format: <|task_controllable_tts|><|start_content|>...<|end_content|>...

Migrated from Orpheus to Spark TTS architecture.
"""

import re
from typing import List, Tuple

from veena3modal.core.constants import INDIC_SPEAKERS, INDIC_EMOTION_TAGS, SPEAKER_MAP
from veena3modal.processing.emotion_normalizer import normalize_emotion_tags

class IndicPromptBuilder:
    """
    Builds prompts in Spark TTS format with Indic speaker system.

    Format:
    <|task_controllable_tts|>
    <|start_content|>{text with [emotions]}<|end_content|>
    <|start_style_label|><|speaker_{id}|><|end_style_label|>
    <|start_global_token|>

    CRITICAL: This format MUST match Spark TTS training exactly.

    Model Details:
    - HuggingFace: BayAreaBoys/spark_tts_4speaker
    - Architecture: Qwen2ForCausalLM with BiCodec audio tokenizer
    - Languages: Telugu, Hindi, English, and more
    - Speakers: 12 predefined (case-sensitive!) - must match training map
    - Emotions: 10 tags in [bracket] format
    """

    def __init__(self, tokenizer, model=None):
        """
        Initialize Indic prompt builder for Spark TTS.

        Args:
            tokenizer: Transformers tokenizer
            model: Model instance (optional, kept for backward compatibility)
        """
        self.tokenizer = tokenizer
        self.model = model

    def build_prefix(
        self,
        speaker: str,
        text: str,
        validate: bool = True
    ) -> str:
        """
        Build Spark TTS format prompt with Indic speaker.

        Args:
            speaker: Speaker name (one of 12 predefined speakers, case-sensitive!)
                Examples: 'lipakshi', 'reet', 'Nandini', 'Nilay', 'vardan', 'anika', 'adarsh',
                'krishna', 'Aarvi', 'Asha', 'Bittu', 'Mira'
            text: Text to synthesize with inline emotion tags
                Examples:
                - "Hello! Welcome to the demo."
                - "[laughs] The results were amazing!"
                - "The results were amazing and then [giggle] we celebrated!"
                - "नमस्ते! [excited] आज का दिन बहुत अच्छा है।" (Hindi)
                - "[curious] మీరు ఎలా ఉన్నారు?" (Telugu)
                - "Hello <laugh> this works too!" (will be normalized to [laughs])
            validate: Check format correctness (recommended for first use)

        Returns:
            Formatted prompt string ready for tokenization
            Format: <|task_controllable_tts|><|start_content|>text<|end_content|>...

        Note: Emotion tags should be inline in text. The API will normalize:
        - <laugh> → [laughs] (old format)
        - laughing → [laughs] (natural language)
        """
        # Normalize emotion tags: <emotion> → [emotion] (legacy compatibility)
        text = normalize_emotion_tags(text)

        # Validate if requested
        if validate:
            self._validate_inputs(speaker, text)

        # Get speaker ID from mapping
        speaker_id = SPEAKER_MAP.get(speaker)
        if speaker_id is None:
            raise ValueError(f"Invalid speaker: {speaker}. Valid speakers: {list(SPEAKER_MAP.keys())}")

        # Build Spark TTS format prompt
        prompt = "".join([
            "<|task_controllable_tts|>",
            "<|start_content|>",
            text,
            "<|end_content|>",
            "<|start_style_label|>",
            f"<|speaker_{speaker_id}|>",
            "<|end_style_label|>",
            "<|start_global_token|>"
        ])

        return prompt

    def build_prefix_ids(
        self,
        speaker: str,
        text: str,
        validate: bool = True
    ) -> List[int]:
        """
        Build prefix as token IDs (for testing/debugging).

        Args:
            speaker: Speaker name
            text: Text to synthesize (with inline emotions)
            validate: Check format correctness

        Returns:
            List of token IDs for the prefix
        """
        prompt = self.build_prefix(speaker, text, validate)
        token_ids = self.tokenizer.encode(prompt, add_special_tokens=False)
        return token_ids

    def _validate_inputs(self, speaker: str, text: str):
        """
        Validate speaker and text format.

        Args:
            speaker: Speaker name
            text: Text to synthesize (with inline emotions)

        Raises:
            ValueError: If inputs are invalid
        """
        # Validate speaker (case-sensitive!)
        if speaker not in INDIC_SPEAKERS:
            raise ValueError(
                f"Invalid speaker '{speaker}'. Must be one of (case-sensitive): "
                f"{', '.join(INDIC_SPEAKERS)}"
            )

        # Check text isn't empty
        if not text.strip():
            raise ValueError("Text cannot be empty")

        # Validate inline emotion tags in text (after normalization)
        self._validate_emotion_tags(text)

    def _validate_emotion_tags(self, text: str):
        """
        Validate emotion tags in text are in the allowed set for Spark TTS.

        Warns if unknown emotion tags are found (won't break, but won't work).

        Args:
            text: Text with potential emotion tags
        """
        # Find all [emotion] tags in text (Spark TTS format)
        emotion_pattern = r'\[([a-z\s]+)\]'
        found_emotions = re.findall(emotion_pattern, text)

        if not found_emotions:
            return  # No emotion tags, nothing to validate

        # Check each found emotion
        unknown_tags = []
        for emotion in found_emotions:
            tag = f'[{emotion}]'
            if tag not in INDIC_EMOTION_TAGS:
                unknown_tags.append(tag)

        if unknown_tags:
            print(f"⚠  Warning: Found {len(unknown_tags)} unknown emotion tag(s) for Spark TTS:")
            for tag in unknown_tags:
                print(f"    - {tag} (will be treated as regular text)")
            print(f"\n📋 Valid Spark TTS emotion tags:")
            for tag in INDIC_EMOTION_TAGS:
                print(f"    {tag}")
            print("\n")

    def extract_emotion_tags(self, text: str) -> List[Tuple[str, int]]:
        """
        Extract emotion tags and their positions from text.

        Args:
            text: Text with emotion tags

        Returns:
            List of (emotion_tag, position) tuples
        """
        emotion_pattern = r'\[([a-z\s]+)\]'
        matches = []

        for match in re.finditer(emotion_pattern, text):
            tag = match.group(0)
            if tag in INDIC_EMOTION_TAGS:
                matches.append((tag, match.start()))

        return matches

    def build_prefix_with_globals(
        self,
        speaker: str,
        text: str,
        global_ids: List[int],
        validate: bool = True
    ) -> str:
        """
        Build prompt with pre-generated global tokens for voice consistency in chunked generation.

        This is critical for multi-chunk text processing: the 32 global tokens encode
        speaker identity (voice DNA). By injecting the same global tokens from the first
        chunk into subsequent chunks, we ensure consistent voice across the entire text.

        Use Case:
        - Chunk 1: Use build_prefix() → model generates 32 global tokens + semantic tokens
        - Chunk 2+: Use build_prefix_with_globals() with captured global tokens
                    → model skips global generation, generates only semantic tokens


cd /home/ubuntu/veenaModal && sed -n '70,140p' veena3modal/core/pipeline.py


Output:
Chunk ID: f97fbd
Wall time: 0.0511 seconds
Process exited with code 0
Original token count: 675
Output:
            prompt_builder: IndicPromptBuilder instance (with Spark TTS format)
            bicodec_decoder: BiCodecDecoder instance
        """
        self.model = model
        self.prompt_builder = prompt_builder
        self.bicodec_decoder = bicodec_decoder

        print(f"🚀 SparkTTSPipeline initialized")

    async def generate_speech(
        self,
        speaker: str,
        text: str,
        temperature: float = DEFAULT_TEMPERATURE,
        top_k: int = DEFAULT_TOP_K,
        top_p: float = DEFAULT_TOP_P,
        max_tokens: int = DEFAULT_MAX_TOKENS,
        repetition_penalty: float = 1.05,
        seed: Optional[int] = None,
        return_timing: bool = False,
        request_id: Optional[str] = None,
    ) -> Optional[bytes] | tuple[Optional[bytes], Dict[str, Any]]:
        """
        Generate speech audio (non-streaming) using Spark TTS.

        NOTE: This method signature changed from description-based to speaker-based
        to align with Spark TTS architecture.

        Args:
            speaker: Speaker name (one of 12 predefined speakers)
            text: Text to synthesize (with optional [emotion] tags)
            temperature: Sampling temperature (default: 0.8 for Spark TTS)
            top_k: Top-k sampling (default: 50)
            top_p: Nucleus sampling (default: 1.0)
            max_tokens: Max BiCodec tokens to generate (default: 2048)
            seed: Random seed for reproducibility
            return_timing: If true, returns (audio_bytes, metrics_dict)

        Returns:
            Audio bytes (int16 PCM WAV, 16kHz mono) or None if failed.
            If return_timing is true, returns tuple(audio_bytes, metrics).
        """
        # Build prompt using Spark TTS format
        prompt = self.prompt_builder.build_prefix(speaker, text)

        # Configure sampling for Spark TTS
        sampling_params = SamplingParams(
            temperature=temperature,
            top_k=top_k,
            top_p=top_p,
            max_tokens=max_tokens,
            repetition_penalty=repetition_penalty,
            stop=TRAINING_STOP_TOKEN_IDS,
            skip_special_tokens=False,
            seed=seed,
        )

        # Use vLLM engine to generate
        if request_id is None:
            import uuid
            request_id = f"req-{uuid.uuid4().hex[:12]}"

        emit_timing = return_timing or _detailed_timing_enabled()
        batch_timeout = _batch_wait_timeout_seconds()
        results_generator = self.model.engine.generate(
            prompt=prompt,
            sampling_params=sampling_params,
            request_id=request_id,
        )

        # Collect all outputs and track generation timing


cd /home/ubuntu/veenaModal && NUMBA_DISABLE_JIT=1 python3 - <<'PY'
import asyncio, time
from transformers import AutoTokenizer
from vllm import AsyncLLMEngine, AsyncEngineArgs, SamplingParams
from veena3modal.processing.prompt_builder import IndicPromptBuilder
from veena3modal.core.constants import TRAINING_STOP_TOKEN_IDS

async def main():
    tokenizer=AutoTokenizer.from_pretrained('/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', trust_remote_code=True)
    pb=IndicPromptBuilder(tokenizer=tokenizer)
    prompt=pb.build_prefix('lipakshi','Hello')
    print('prompt len',len(prompt))
    print('tokens', len(tokenizer.encode(prompt, add_special_tokens=False)))
    print('prompt preview',prompt[:120])

    engine=AsyncLLMEngine.from_engine_args(AsyncEngineArgs(model='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', tokenizer='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', dtype='bfloat16', max_model_len=4096, gpu_memory_utilization=0.15, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, trust_remote_code=True, enforce_eager=False, enable_prefix_caching=True, enable_chunked_prefill=True))
    sp=SamplingParams(temperature=0.8, top_k=50, top_p=1.0, max_tokens=128, repetition_penalty=1.05, stop=TRAINING_STOP_TOKEN_IDS, skip_special_tokens=False)
    t=time.time()
    gen=engine.generate(prompt=prompt, sampling_params=sp, request_id='probe')
    chunks=0
    async for out in gen:
        chunks += 1
        r=out.outputs[0]
        print('chunk',chunks,'finished',out.finished,'token_count',len(r.token_ids),'new',len(r.token_ids))
        if out.finished:
            text=r.text
            print('finished len text',len(text))
            print('last tokens',len(r.token_ids))
            break
        if chunks>20:
            break
    print('elapsed',time.time()-t)
    await engine.abort('probe')

asyncio.run(main())
PY


Output:
Chunk ID: 003d35
Wall time: 26.5662 seconds
Process exited with code 0
Original token count: 2407
Output:
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
prompt len 137
tokens 14
prompt preview <|task_controllable_tts|><|start_content|>Hello<|end_content|><|start_style_label|><|speaker_0|><|end_style_label|><|sta
INFO 02-17 00:52:03 [model.py:541] Resolved architecture: Qwen2ForCausalLM
INFO 02-17 00:52:03 [model.py:1561] Using max model len 4096
INFO 02-17 00:52:03 [scheduler.py:226] Chunked prefill is enabled with max_num_batched_tokens=2048.
INFO 02-17 00:52:03 [vllm.py:624] Asynchronous scheduling is enabled.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(EngineCore_DP0 pid=3719914) INFO 02-17 00:52:04 [core.py:96] Initializing a V1 LLM engine (v0.15.1) with config: model='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', speculative_config=None, tokenizer='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 256, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': True}, 'local_cache_dir': None, 'static_all_moe_layers': []}
(EngineCore_DP0 pid=3719914) INFO 02-17 00:52:07 [parallel_state.py:1212] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://216.81.248.184:52959 backend=nccl
(EngineCore_DP0 pid=3719914) INFO 02-17 00:52:07 [parallel_state.py:1423] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
(EngineCore_DP0 pid=3719914) INFO 02-17 00:52:07 [gpu_model_runner.py:4033] Starting to load model /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM...
(EngineCore_DP0 pid=3719914) INFO 02-17 00:52:08 [cuda.py:364] Using FLASH_ATTN attention backend out of potential backends: ('FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION')
(EngineCore_DP0 pid=3719914) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
(EngineCore_DP0 pid=3719914) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
(EngineCore_DP0 pid=3719914)
Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
(EngineCore_DP0 pid=3719914)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.18it/s]
(EngineCore_DP0 pid=3719914)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.18it/s]
(EngineCore_DP0 pid=3719914)
(EngineCore_DP0 pid=3719914) INFO 02-17 00:52:09 [default_loader.py:291] Loading weights took 0.92 seconds
(EngineCore_DP0 pid=3719914) INFO 02-17 00:52:10 [gpu_model_runner.py:4130] Model loading took 0.95 GiB memory and 1.446719 seconds
(EngineCore_DP0 pid=3719914) INFO 02-17 00:52:14 [backends.py:812] Using cache directory: /home/ubuntu/.cache/vllm/torch_compile_cache/5591bfed59/rank_0_0/backbone for vLLM's torch.compile
(EngineCore_DP0 pid=3719914) INFO 02-17 00:52:14 [backends.py:872] Dynamo bytecode transform time: 4.22 s
(EngineCore_DP0 pid=3719914) INFO 02-17 00:52:17 [backends.py:267] Directly load the compiled graph(s) for compile range (1, 2048) from the cache, took 0.666 s
(EngineCore_DP0 pid=3719914) INFO 02-17 00:52:17 [monitor.py:34] torch.compile takes 4.88 s in total
(EngineCore_DP0 pid=3719914) INFO 02-17 00:52:18 [gpu_worker.py:356] Available KV cache memory: 10.13 GiB
(EngineCore_DP0 pid=3719914) INFO 02-17 00:52:18 [kv_cache_utils.py:1307] GPU KV cache size: 885,424 tokens
(EngineCore_DP0 pid=3719914) INFO 02-17 00:52:18 [kv_cache_utils.py:1312] Maximum concurrency for 4,096 tokens per request: 216.17x
(EngineCore_DP0 pid=3719914)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   0%|          | 0/35 [00:00<?, ?it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  11%|█▏        | 4/35 [00:00<00:00, 32.38it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  23%|██▎       | 8/35 [00:00<00:00, 34.87it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  34%|███▍      | 12/35 [00:00<00:00, 35.79it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  46%|████▌     | 16/35 [00:00<00:00, 36.31it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  57%|█████▋    | 20/35 [00:00<00:00, 36.85it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  69%|██████▊   | 24/35 [00:00<00:00, 36.80it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  80%|████████  | 28/35 [00:00<00:00, 36.54it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  91%|█████████▏| 32/35 [00:00<00:00, 35.11it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 35/35 [00:01<00:00, 34.98it/s]
(EngineCore_DP0 pid=3719914)
Capturing CUDA graphs (decode, FULL):   0%|          | 0/19 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL):  16%|█▌        | 3/19 [00:00<00:00, 29.64it/s]
Capturing CUDA graphs (decode, FULL):  37%|███▋      | 7/19 [00:00<00:00, 33.87it/s]
Capturing CUDA graphs (decode, FULL):  58%|█████▊    | 11/19 [00:00<00:00, 35.03it/s]
Capturing CUDA graphs (decode, FULL):  79%|███████▉  | 15/19 [00:00<00:00, 35.44it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 35.89it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 35.14it/s]
(EngineCore_DP0 pid=3719914) INFO 02-17 00:52:21 [gpu_model_runner.py:5063] Graph capturing finished in 2 secs, took 0.28 GiB
(EngineCore_DP0 pid=3719914) INFO 02-17 00:52:21 [core.py:272] init engine (profile, create kv cache, warmup model) took 11.00 seconds
chunk 1 finished False token_count 1 new 1
chunk 2 finished False token_count 2 new 2
chunk 3 finished False token_count 3 new 3
chunk 4 finished False token_count 4 new 4
chunk 5 finished False token_count 5 new 5
chunk 6 finished False token_count 6 new 6
chunk 7 finished False token_count 7 new 7
chunk 8 finished False token_count 8 new 8
chunk 9 finished False token_count 9 new 9
chunk 10 finished False token_count 10 new 10
chunk 11 finished False token_count 11 new 11
chunk 12 finished False token_count 12 new 12
chunk 13 finished False token_count 13 new 13
chunk 14 finished False token_count 14 new 14
chunk 15 finished False token_count 15 new 15
chunk 16 finished False token_count 16 new 16
chunk 17 finished False token_count 17 new 17
chunk 18 finished False token_count 18 new 18
chunk 19 finished False token_count 19 new 19
chunk 20 finished False token_count 20 new 20
chunk 21 finished False token_count 21 new 21
elapsed 0.11228442192077637


cd /home/ubuntu/veenaModal && rg -n "BiCodecDecoder|decode_to_bytes|bicodec" veena3modal/services/tts_runtime.py veena3modal/core -g"*.py"


Output:
Chunk ID: c095b8
Wall time: 0.0510 seconds
Process exited with code 0
Original token count: 1286
Output:
veena3modal/services/tts_runtime.py:8:- BiCodecDecoder
veena3modal/services/tts_runtime.py:73:    bicodec_decoder: Any = None
veena3modal/services/tts_runtime.py:82:    bicodec_path: str = ""
veena3modal/services/tts_runtime.py:107:    bicodec_path: Optional[str] = None,
veena3modal/services/tts_runtime.py:122:        bicodec_path: Path to BiCodec model (env: BICODEC_MODEL_PATH, defaults to model_path)
veena3modal/services/tts_runtime.py:145:    bicodec_path = bicodec_path or os.environ.get(
veena3modal/services/tts_runtime.py:154:    logger.info(f"  BiCodec path: {bicodec_path}")
veena3modal/services/tts_runtime.py:161:        from veena3modal.core.bicodec_decoder import BiCodecDecoder
veena3modal/services/tts_runtime.py:195:        bicodec_decoder = BiCodecDecoder(
veena3modal/services/tts_runtime.py:197:            model_path=bicodec_path,
veena3modal/services/tts_runtime.py:205:            bicodec_decoder=bicodec_decoder,
veena3modal/services/tts_runtime.py:213:            # NOTE: Parameter is named 'snac_decoder' for legacy reasons, but works with BiCodecDecoder
veena3modal/services/tts_runtime.py:217:                snac_decoder=bicodec_decoder,  # BiCodecDecoder is interface-compatible
veena3modal/services/tts_runtime.py:251:            bicodec_decoder=bicodec_decoder,
veena3modal/services/tts_runtime.py:257:            bicodec_path=bicodec_path,
veena3modal/services/tts_runtime.py:339:        global_matches = re.findall(r"<\|bicodec_global_(\d+)\|>", generated_text)
veena3modal/core/token_utils.py:28:        - <|bicodec_semantic_123|> -> ("semantic", 123)
veena3modal/core/token_utils.py:29:        - <|bicodec_global_7|>     -> ("global", 7)
veena3modal/core/token_utils.py:46:    SEMANTIC_PATTERN = re.compile(r"^<\|bicodec_semantic_(\d+)\|>$")
veena3modal/core/token_utils.py:47:    GLOBAL_PATTERN = re.compile(r"^<\|bicodec_global_(\d+)\|>$")
veena3modal/core/token_utils.py:75:                if token_str.startswith("<|bicodec_"):
veena3modal/core/token_utils.py:88:            token_str: Token string like "<|bicodec_semantic_123|>"
veena3modal/core/token_utils.py:173:        bicodec_count = sum(1 for v in self._cache.values() if v is not None)
veena3modal/core/token_utils.py:176:            "bicodec_tokens": bicodec_count,
veena3modal/core/token_utils.py:177:            "non_bicodec_tokens": total - bicodec_count,
veena3modal/core/token_utils.py:181:def create_bicodec_parser(tokenizer) -> BiCodecTokenParser:
veena3modal/core/bicodec_decoder.py:36:class BiCodecDecoder:
veena3modal/core/bicodec_decoder.py:147:    def decode_to_bytes(
veena3modal/core/bicodec_decoder.py:234:        From sparktts/models/bicodec.py line 184-186:
veena3modal/core/pipeline.py:63:        bicodec_decoder,
veena3modal/core/pipeline.py:71:            bicodec_decoder: BiCodecDecoder instance
veena3modal/core/pipeline.py:75:        self.bicodec_decoder = bicodec_decoder
veena3modal/core/pipeline.py:253:        semantic_ids, global_ids = self._extract_bicodec_tokens(generated_text)
veena3modal/core/pipeline.py:260:                    "tts_bicodec_parse_failed",
veena3modal/core/pipeline.py:291:        if not self.bicodec_decoder.validate_tokens(semantic_ids, global_ids):
veena3modal/core/pipeline.py:309:        audio_bytes = self.bicodec_decoder.decode_to_bytes(semantic_ids, global_ids)
veena3modal/core/pipeline.py:414:    def _extract_bicodec_tokens(self, generated_text: str) -> Tuple[List[int], List[int]]:
veena3modal/core/pipeline.py:419:        - <|bicodec_semantic_{id}|>
veena3modal/core/pipeline.py:420:        - <|bicodec_global_{id}|>
veena3modal/core/pipeline.py:430:        semantic_matches = re.findall(r"<\|bicodec_semantic_(\d+)\|>", generated_text)
veena3modal/core/pipeline.py:434:        global_matches = re.findall(r"<\|bicodec_global_(\d+)\|>", generated_text)
veena3modal/core/streaming_pipeline.py:189:                            audio_bytes = self.snac_decoder.decode_to_bytes(
veena3modal/core/streaming_pipeline.py:206:    def _extract_bicodec_tokens_from_text(self, text: str) -> tuple[List[int], List[int]]:
veena3modal/core/streaming_pipeline.py:216:        semantic_matches = re.findall(r"<\|bicodec_semantic_(\d+)\|>", text)
veena3modal/core/streaming_pipeline.py:219:        global_matches = re.findall(r"<\|bicodec_global_(\d+)\|>", text)
veena3modal/core/streaming_pipeline.py:310:            request_id = f"bicodec-stream-{uuid.uuid4().hex[:8]}-{int(time.time() * 1000000)}"
veena3modal/core/streaming_pipeline.py:677:            request_id = f"bicodec-first-{uuid.uuid4().hex[:8]}-{int(time.time() * 1000000)}"
veena3modal/core/streaming_pipeline.py:1048:            request_id = f"bicodec-cont-{uuid.uuid4().hex[:8]}-{int(time.time() * 1000000)}"
veena3modal/core/bicodec_batch_decoder.py:16:    batch_decoder = BiCodecBatchDecoder(bicodec_decoder, max_batch=16, timeout_ms=10)
veena3modal/core/bicodec_batch_decoder.py:62:        bicodec_decoder,
veena3modal/core/bicodec_batch_decoder.py:68:            bicodec_decoder: BiCodecDecoder instance (used for single-item decode fallback)
veena3modal/core/bicodec_batch_decoder.py:72:        self.decoder = bicodec_decoder


Summary: auto


I’m now reproducing the same pipeline path directly (same `IndicPromptBuilder + SparkTTSPipeline.generate_speech`) to verify whether the block is in the pipeline consumer loop itself versus something in FastAPI/request handling.


cd /home/ubuntu/veenaModal && NUMBA_DISABLE_JIT=1 python3 - <<'PY'
import asyncio, time
from veena3modal.core.model_loader import SparkTTSModel
from veena3modal.processing.prompt_builder import IndicPromptBuilder
from veena3modal.core.bicodec_decoder import BiCodecDecoder
from veena3modal.core.pipeline import SparkTTSPipeline
from vllm import AsyncEngineArgs

async def main():
    model = SparkTTSModel(model_path='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', bicodec_path='/home/ubuntu/veenaModal/models/spark_tts_4speaker', device='cuda', gpu_memory_utilization=0.15, num_engines=1)
    pb = IndicPromptBuilder(tokenizer=model.tokenizer)
    decoder = BiCodecDecoder(model_path='/home/ubuntu/veenaModal/models/spark_tts_4speaker')
    pipe = SparkTTSPipeline(model=model, prompt_builder=pb, bicodec_decoder=decoder)
    t=time.time()
    try:
        out = await asyncio.wait_for(pipe.generate_speech('lipakshi','Hello', return_timing=True), timeout=120)
        print(type(out), out[0] is not None if isinstance(out, tuple) else None)
        if isinstance(out, tuple):
            wav, metrics = out
            print('metrics keys', list(metrics.keys())[:20])
            print('token_total', metrics.get('token_total'), 'batches', metrics.get('batch_count'))
            print('len', len(wav) if wav else None)
    except Exception as e:
        print(type(e).__name__, e)
    print('elapsed', time.time()-t)

asyncio.run(main())
PY


Output:
Chunk ID: cca579
Wall time: 7.1281 seconds
Process exited with code 1
Original token count: 469
Output:
🚀 Initializing Spark TTS Model
📁 Model path: /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
🗣  Speakers: 12 predefined (lipakshi, vardan, reet, Nandini, krishna, anika, adarsh, Nilay, Aarvi, Asha, Bittu, Mira)
🎭 Emotions: 10 tags in [bracket] format

📝 Loading tokenizer from model...
✅ Tokenizer loaded: 165168 tokens

🔍 Validating 12 speakers for Spark TTS...
  lipakshi → <|speaker_0|>: 7 tokens
  vardan → <|speaker_1|>: 7 tokens
  reet → <|speaker_2|>: 7 tokens
  Nandini → <|speaker_3|>: 7 tokens
  krishna → <|speaker_4|>: 7 tokens
  anika → <|speaker_5|>: 7 tokens
  adarsh → <|speaker_6|>: 7 tokens
  Nilay → <|speaker_7|>: 7 tokens
  Aarvi → <|speaker_8|>: 7 tokens
  Asha → <|speaker_9|>: 7 tokens
  Bittu → <|speaker_10|>: 8 tokens
  Mira → <|speaker_11|>: 8 tokens
✅ All 12 speakers validated with mapping

🔍 Validating 10 emotion tags for Spark TTS...
✅ All 10 emotion tags validated

📋 Sample emotion tags:
  [angry]: 1 tokens
  [curious]: 1 tokens
  [excited]: 1 tokens
  [giggle]: 1 tokens
  [laughs harder]: 1 tokens
  ...

🔧 Initializing vLLM engine with optimizations...
  - dtype: bfloat16
  - max_model_len: 4096
  - gpu_memory_utilization: 0.15
  - enable_prefix_caching: True
  - enforce_eager (disable CUDA graphs): False
Traceback (most recent call last):
  File "<stdin>", line 26, in <module>
  File "/usr/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "<stdin>", line 9, in main
  File "/home/ubuntu/veenaModal/veena3modal/core/model_loader.py", line 127, in __init__
    engine_args = AsyncEngineArgs(**engine_kwargs_dict)
TypeError: AsyncEngineArgs.__init__() got an unexpected keyword argument 'bicodec_path'


cd /home/ubuntu/veenaModal && sed -n '1,190p' veena3modal/core/model_loader.py


Output:
Chunk ID: c93fc7
Wall time: 0.0512 seconds
Process exited with code 0
Original token count: 1793
Output:
"""
Spark TTS Model Loader

Loads Spark TTS model with vLLM engine and validates speakers/emotions.
Migrated from Veena3/Orpheus to Spark TTS architecture.
"""

import os
import torch
from typing import Optional, Dict, Any
from transformers import AutoTokenizer
from vllm import AsyncLLMEngine, AsyncEngineArgs, SamplingParams

from veena3modal.core.constants import (
    INDIC_EMOTION_TAGS,
    INDIC_SPEAKERS,
    SPEAKER_MAP,
    DEFAULT_MODEL_PATH,
    DEFAULT_MAX_MODEL_LEN,
    VLLM_CONFIG,
)

class SparkTTSModel:
    """
    Spark TTS Model with vLLM inference engine and BiCodec audio tokenizer.

    Model: BayAreaBoys/spark_tts_4speaker
    Architecture: Qwen2ForCausalLM with BiCodec audio tokenization
    Speakers: 12 predefined (mapped to speaker_0-11 as per training)
    Emotions: 10 tags in [bracket] format

    CRITICAL REQUIREMENTS:
    - Load tokenizer with special Spark TTS tokens
    - Configure vLLM with optimizations (prefix caching, CUDA graphs)
    - Support speaker mapping (lipakshi → speaker_0, etc.)
    """

    def __init__(
        self,
        model_path: str = None,
        model_type: str = "indic_speakers",  # Default to indic_speakers for backward compatibility
        hf_token: str = None,
        **engine_kwargs
    ):
        """
        Initialize Spark TTS model with vLLM.

        Args:
            model_path: Path to model directory (local or HF). If None, uses env var or default.
            model_type: Model type (kept for compatibility, always uses indic_speakers)
            hf_token: HuggingFace token for private models
            **engine_kwargs: Additional vLLM engine arguments (override defaults from VLLM_CONFIG)
        """
        self.model_type = "indic_speakers"  # Spark TTS only supports speaker-based

        # Use provided path, or check environment variable, or use default
        if model_path is None:
            model_path = os.environ.get(
                'SPARK_TTS_MODEL_PATH',
                DEFAULT_MODEL_PATH
            )

        self.model_path = model_path
        self.hf_token = hf_token

        print(f"🚀 Initializing Spark TTS Model")
        print(f"📁 Model path: {model_path}")
        print("🗣  Speakers: 12 predefined "
              "(lipakshi, vardan, reet, Nandini, krishna, anika, adarsh, "
              "Nilay, Aarvi, Asha, Bittu, Mira)")
        print(f"🎭 Emotions: 10 tags in [bracket] format")

        # Load tokenizer
        print(f"\n📝 Loading tokenizer from model...")
        tokenizer_kwargs = {
            'trust_remote_code': True,
        }

        # Add HF token for private repos
        if hf_token:
            tokenizer_kwargs['token'] = hf_token

        # Add emotion tags as special tokens
        special_tokens = {
            'additional_special_tokens': INDIC_EMOTION_TAGS
        }

        self.tokenizer = AutoTokenizer.from_pretrained(
            model_path,
            **tokenizer_kwargs
        )

        # Add special tokens
        self.tokenizer.add_special_tokens(special_tokens)

        print(f"✅ Tokenizer loaded: {len(self.tokenizer)} tokens")

        # Validate speakers and emotions
        self._validate_speakers()
        self._validate_emotion_tags()

        # Initialize vLLM engine with Spark TTS optimizations
        print(f"\n🔧 Initializing vLLM engine with optimizations...")

        # Start with default VLLM_CONFIG and override with kwargs
        engine_kwargs_dict = VLLM_CONFIG.copy()
        engine_kwargs_dict.update({
            'model': model_path,
            'tokenizer': model_path,
        })

        # Add HF token for private models
        if hf_token:
            os.environ['HUGGING_FACE_HUB_TOKEN'] = hf_token

        # Override with any user-provided kwargs
        engine_kwargs_dict.update(engine_kwargs)

        # Print configuration
        print(f"  - dtype: {engine_kwargs_dict['dtype']}")
        print(f"  - max_model_len: {engine_kwargs_dict['max_model_len']}")
        print(f"  - gpu_memory_utilization: {engine_kwargs_dict['gpu_memory_utilization']}")
        print(f"  - enable_prefix_caching: {engine_kwargs_dict.get('enable_prefix_caching', False)}")
        print(f"  - enforce_eager (disable CUDA graphs): {engine_kwargs_dict.get('enforce_eager', True)}")

        engine_args = AsyncEngineArgs(**engine_kwargs_dict)
        self.engine = AsyncLLMEngine.from_engine_args(engine_args)

        print(f"✅ vLLM engine initialized successfully!")
        print(f"\n🎉 Spark TTS Model ready for inference!\n")

    def _validate_emotion_tags(self):
        """
        Validate that emotion tags are properly configured for Spark TTS.

        For Spark TTS, emotion tags use [bracket] format and should be
        added as special tokens to the tokenizer.
        """
        print(f"\n🔍 Validating {len(INDIC_EMOTION_TAGS)} emotion tags for Spark TTS...")

        # Check if emotion tags can be tokenized
        for tag in INDIC_EMOTION_TAGS:
            token_ids = self.tokenizer.encode(tag, add_special_tokens=False)
            # Spark TTS emotion tags may be multiple tokens, that's okay
            if not token_ids:
                print(f"⚠  Warning: Emotion tag {tag} produced no tokens")

        print(f"✅ All {len(INDIC_EMOTION_TAGS)} emotion tags validated")

        # Print sample emotion tags
        print(f"\n📋 Sample emotion tags:")
        for i, tag in enumerate(INDIC_EMOTION_TAGS[:5]):
            token_ids = self.tokenizer.encode(tag, add_special_tokens=False)
            print(f"  {tag}: {len(token_ids)} tokens")
        if len(INDIC_EMOTION_TAGS) > 5:
            print(f"  ...")

    def _validate_speakers(self):
        """
        Validate that all speakers are properly mapped for Spark TTS.

        Spark TTS uses speaker tokens like <|speaker_0|> to <|speaker_7|>.
        User-facing names (lipakshi, vardan, etc.) are mapped to these IDs.
        """
        print(f"\n🔍 Validating {len(INDIC_SPEAKERS)} speakers for Spark TTS...")

        # Validate speaker mapping
        for speaker_name in INDIC_SPEAKERS:
            if speaker_name not in SPEAKER_MAP:
                print(f"⚠  Warning: Speaker {speaker_name} not in SPEAKER_MAP")
                continue

            speaker_id = SPEAKER_MAP[speaker_name]
            speaker_token = f'<|speaker_{speaker_id}|>'
            token_ids = self.tokenizer.encode(speaker_token, add_special_tokens=False)
            print(f"  {speaker_name} → {speaker_token}: {len(token_ids)} tokens")

        print(f"✅ All {len(INDIC_SPEAKERS)} speakers validated with mapping")

    def get_model_type(self) -> str:
        """Get the model type (always 'indic_speakers' for Spark TTS)."""
        return self.model_type

    def get_tokenizer(self):
        """Get the tokenizer."""
        return self.tokenizer

    def get_engine(self):
        """Get the vLLM engine."""


cd /home/ubuntu/veenaModal && NUMBA_DISABLE_JIT=1 python3 - <<'PY'
import asyncio, time
from veena3modal.core.model_loader import SparkTTSModel
from veena3modal.processing.prompt_builder import IndicPromptBuilder
from veena3modal.core.bicodec_decoder import BiCodecDecoder
from veena3modal.core.pipeline import SparkTTSPipeline

async def main():
    model = SparkTTSModel(model_path='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', gpu_memory_utilization=0.15)
    pb = IndicPromptBuilder(tokenizer=model.tokenizer)
    decoder = BiCodecDecoder(model_path='/home/ubuntu/veenaModal/models/spark_tts_4speaker')
    pipe = SparkTTSPipeline(model=model, prompt_builder=pb, bicodec_decoder=decoder)
    t=time.time()
    try:
        out = await asyncio.wait_for(pipe.generate_speech('lipakshi','Hello', return_timing=True), timeout=120)
        wav, metrics = out
        print('ok',len(wav), metrics.get('token_total'), metrics.get('batch_count'))
    except Exception as e:
        print(type(e).__name__,e)
    print('elapsed',time.time()-t)

asyncio.run(main())
PY