Input:
*** Begin Patch
*** Update File: scripts/stress_test_optimized_local.py
@@
         "x-llm-time-per-token-ms": "llm_time_per_token_ms",
         "x-llm-time-per-batch-wall-ms": "llm_time_per_batch_wall_ms",
         "x-llm-time-per-batch-gpu-ms": "llm_time_per_batch_gpu_ms",
         "x-llm-parse-avg-ms": "llm_parse_ms_avg",
+        "x-bicodec-decode-wall-ms": "bicodec_decode_wall_ms",
+        "x-bicodec-decode-gpu-ms": "bicodec_decode_gpu_ms",
+        "x-bicodec-decode-cpu-ms": "bicodec_decode_cpu_ms",
     }
@@
         decode = level.timing_stats("llm_decode_wall_ms_total")
         if decode:
             print("\n  Decode (ms):")
             print(f"    avg={decode['avg']:.0f} p95={decode['p95']:.0f}")
+        bdecode = level.timing_stats("bicodec_decode_wall_ms")
+        if bdecode:
+            print("\n  BiCodec decode (ms):")
+            print(f"    avg={bdecode['avg']:.0f} p95={bdecode['p95']:.0f}")
+        bdecode_gpu = level.timing_stats("bicodec_decode_gpu_ms")
+        if bdecode_gpu:
+            print("\n  BiCodec decode GPU (ms):")
+            print(f"    avg={bdecode_gpu['avg']:.0f} p95={bdecode_gpu['p95']:.0f}")

-        token_ms, token_count = level.timing_rate("generation_ms", "llm_token_total")
+        token_ms, token_count = level.timing_rate("generation_ms", "llm_token_total")
         if token_count:
-            print(f"  Token timing: avg {token_ms / token_count * 1000:.3f} us/token")
+            print(f"  Token timing: avg {token_ms * 1000:.3f} us/token")
@@
             "timing_averages": {
                 "generation_ms": level.timing_stats("generation_ms"),
                 "llm_batch_wall_ms_total": level.timing_stats("llm_batch_wall_ms_total"),
                 "llm_batch_gpu_ms_total": level.timing_stats("llm_batch_gpu_ms_total"),
                 "llm_parse_ms": level.timing_stats("llm_parse_ms"),
                 "llm_decode_wall_ms_total": level.timing_stats("llm_decode_wall_ms_total"),
+                "bicodec_decode_wall_ms": level.timing_stats("bicodec_decode_wall_ms"),
+                "bicodec_decode_gpu_ms": level.timing_stats("bicodec_decode_gpu_ms"),
             },
*** End Patch


Output: {"output":"Success. Updated the following files:\nM scripts/stress_test_optimized_local.py\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


# Collaboration Mode: Default

You are now in Default mode. Any previous instructions for other modes (e.g. Plan mode) are no longer active.

Your active mode changes only when new developer instructions with a different `<collaboration_mode>...</collaboration_mode>` change it; user requests or tool descriptions do not change mode by themselves. Known mode names are Default and Plan.

## request_user_input availability

The `request_user_input` tool is unavailable in Default mode. If you call it while in Default mode, it will return an error.

If a decision is necessary and cannot be discovered from local context, ask the user directly. However, in Default mode you should strongly prefer executing the user's request rather than stopping to ask questions.

Summary: auto

## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) When `SKILL.md` references relative paths (e.g., `scripts/foo.py`), resolve them relative to the skill directory listed above first, and only consider other paths if needed.
  3) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  4) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  5) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.


**Planning to add compilation script**


python3 -m py_compile scripts/stress_test_optimized_local.py


Output:
Chunk ID: 497f65
Wall time: 0.0511 seconds
Process exited with code 0
Original token count: 0
Output:


Summary: auto


**Planning server restart and tests**


ps -ef | rg 'veena3modal.local_server|uvicorn|v1/tts' -n


Output:
Chunk ID: a791f7
Wall time: 0.0512 seconds
Process exited with code 0
Original token count: 108
Output:
289:ubuntu   3732972 3675569  0 01:16 ?        00:01:24 python3 -m veena3modal.local_server --model-path /home/ubuntu/veenaModal/models/spark_tts_4speaker --gpu-memory 0.5 --log-level INFO --port 8000
315:ubuntu   3791086 3675569  0 03:38 ?        00:00:00 /bin/bash -c ps -ef | rg 'veena3modal.local_server|uvicorn|v1/tts' -n
317:ubuntu   3791088 3791086  0 03:38 ?        00:00:00 rg veena3modal.local_server|uvicorn|v1/tts -n


**Planning safe server restart**


Next step is a controlled local server restart so the new instrumentation code is loaded; then I’ll issue a single request to verify `x-perf-details`/timing headers before running the full stress sweep.


**Restarting server process with logging**


pkill -f 'python3 -m veena3modal.local_server --model-path /home/ubuntu/veenaModal/models/spark_tts_4speaker --gpu-memory 0.5 --log-level INFO --port 8000' || true && sleep 2 && nohup python3 -m veena3modal.local_server --model-path /home/ubuntu/veenaModal/models/spark_tts_4speaker --gpu-memory 0.5 --log-level INFO --port 8000 > /tmp/veena_local_server.log 2>&1 & echo $!


Output:
Chunk ID: 112506
Wall time: 0.0512 seconds
Process exited with code 0
Original token count: 2
Output:
3791135


**Planning health check polling**


for i in $(seq 1 120); do if curl -sS -m 2 http://localhost:8000/v1/tts/health >/tmp/health.json 2>/dev/null; then echo READY; cat /tmp/health.json; exit 0; fi; sleep 2; done; echo TIMEOUT; tail -n 80 /tmp/veena_local_server.log; exit 1


Output:
Chunk ID: ad71c7
Wall time: 30.0011 seconds
Process running with session ID 35708
Original token count: 0
Output:


**Monitoring session status**


Output:
Chunk ID: f58aaf
Wall time: 30.0010 seconds
Process running with session ID 35708
Original token count: 0
Output:


**Waiting for readiness before proceeding**


Output:
Chunk ID: ba05f6
Wall time: 30.0010 seconds
Process running with session ID 35708
Original token count: 0
Output:


**Updating due to delay**


Server restart is in progress and still warming the runtime; I’m polling health and will verify headers as soon as it comes up.


Output:
Chunk ID: a85fac
Wall time: 30.0017 seconds
Process running with session ID 35708
Original token count: 0
Output:


**Continuing polling with extended timeout**


Output:
Chunk ID: 6817bd
Wall time: 30.0018 seconds
Process running with session ID 35708
Original token count: 0
Output:


**Inspecting server startup logs**


Output:
Chunk ID: 1b0e0b
Wall time: 0.0513 seconds
Process exited with code 1
Original token count: 22
Output:
tail: cannot open '/tmp/veena_local_server.log' for reading: No such file or directory


**Checking server process and logs**


ps -ef | rg 'veena3modal.local_server' -n || true


Output:
Chunk ID: 6e83fb
Wall time: 0.0504 seconds
Process exited with code 0
Original token count: 102
Output:
289:ubuntu   3732972 3675569  0 01:16 ?        00:01:24 python3 -m veena3modal.local_server --model-path /home/ubuntu/veenaModal/models/spark_tts_4speaker --gpu-memory 0.5 --log-level INFO --port 8000
314:ubuntu   3792199 3675569  0 03:41 ?        00:00:00 /bin/bash -c ps -ef | rg 'veena3modal.local_server' -n || true
316:ubuntu   3792201 3792199  0 03:41 ?        00:00:00 rg veena3modal.local_server -n


Summary: auto


**Investigating lingering server process and health check**


Output:
Chunk ID: 39bf47
Wall time: 5.0015 seconds
Process running with session ID 35708
Original token count: 0
Output:


**Investigating hanging process issue**


ps -ef | rg 'localhost:8000/v1/tts/health|/tmp/health.json|seq 1 120|curl -sS -m 2' -n


Output:
Chunk ID: fccac2
Wall time: 0.0512 seconds
Process exited with code 0
Original token count: 149
Output:
308:ubuntu   3791160 3675569  0 03:38 ?        00:00:00 /bin/bash -c for i in $(seq 1 120); do if curl -sS -m 2 http://localhost:8000/v1/tts/health >/tmp/health.json 2>/dev/null; then echo READY; cat /tmp/health.json; exit 0; fi; sleep 2; done; echo TIMEOUT; tail -n 80 /tmp/veena_local_server.log; exit 1
315:ubuntu   3792324 3675569  0 03:41 ?        00:00:00 /bin/bash -c ps -ef | rg 'localhost:8000/v1/tts/health|/tmp/health.json|seq 1 120|curl -sS -m 2' -n
317:ubuntu   3792326 3792324  0 03:41 ?        00:00:00 rg localhost:8000/v1/tts/health|/tmp/health.json|seq 1 120|curl -sS -m 2 -n


**Investigating potential curl hang**


curl -v -m 3 http://localhost:8000/v1/tts/health | head -c 200


Output:
Chunk ID: 4d76ac
Wall time: 0.0512 seconds
Process exited with code 0
Original token count: 166
Output:
*   Trying 127.0.0.1:8000...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* connect to 127.0.0.1 port 8000 failed: Connection refused
*   Trying ::1:8000...
* connect to ::1 port 8000 failed: Connection refused
* Failed to connect to localhost port 8000 after 0 ms: Connection refused

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
* Closing connection 0
curl: (7) Failed to connect to localhost port 8000 after 0 ms: Connection refused


**Checking defunct process status**


Output:
Chunk ID: 817d46
Wall time: 0.0515 seconds
Process exited with code 0
Original token count: 64
Output:
UID          PID    PPID  C STIME TTY          TIME CMD
ubuntu   3732972 3675569  0 01:16 ?        00:01:24 python3 -m veena3modal.local_server --model-path /home/ubuntu/veenaModal/models/spark_tts_4speaker --gpu-memory 0.5 --log-level INFO --port 8000


**Handling hung server processes**


pkill -f 'veena3modal.local_server' || true; pkill -f 'seq 1 120' || true; sleep 1; ps -ef | rg 'veena3modal.local_server|seq 1 120' -n || true


Output:
Chunk ID: 1a89d7
Wall time: 0.0514 seconds
Process exited with code -1
Original token count: 0
Output:


**Investigating potential shell process termination**


ps -ef | rg 'veena3modal.local_server|seq 1 120|stress_test_optimized_local.py' -n || true


Output:
Chunk ID: 2e1924
Wall time: 0.0511 seconds
Process exited with code 0
Original token count: 199
Output:
289:ubuntu   3732972 3675569  0 01:16 ?        00:01:24 python3 -m veena3modal.local_server --model-path /home/ubuntu/veenaModal/models/spark_tts_4speaker --gpu-memory 0.5 --log-level INFO --port 8000
308:ubuntu   3791160 3675569  0 03:38 ?        00:00:00 /bin/bash -c for i in $(seq 1 120); do if curl -sS -m 2 http://localhost:8000/v1/tts/health >/tmp/health.json 2>/dev/null; then echo READY; cat /tmp/health.json; exit 0; fi; sleep 2; done; echo TIMEOUT; tail -n 80 /tmp/veena_local_server.log; exit 1
315:ubuntu   3792491 3675569  0 03:41 ?        00:00:00 /bin/bash -c ps -ef | rg 'veena3modal.local_server|seq 1 120|stress_test_optimized_local.py' -n || true
317:ubuntu   3792493 3792491  0 03:41 ?        00:00:00 rg veena3modal.local_server|seq 1 120|stress_test_optimized_local.py -n


**Switching to manual kill commands**


Output:
Chunk ID: 0b467f
Wall time: 0.9151 seconds
Process exited with code 0
Original token count: 15
Output:
UID          PID    PPID  C STIME TTY      STAT   TIME CMD


**Starting server with nohup**


nohup python3 -m veena3modal.local_server --model-path /home/ubuntu/veenaModal/models/spark_tts_4speaker --gpu-memory 0.5 --log-level INFO --port 8000 > /tmp/veena_local_server.log 2>&1 & echo $!


Output:
Chunk ID: 07dc03
Wall time: 0.0514 seconds
Process exited with code 0
Original token count: 2
Output:
3792581


**Implementing health poll with timeout**


python3 - <<'PY'
import time,requests,sys
url='http://localhost:8000/v1/tts/health'
for i in range(180):
    try:
        r=requests.get(url,timeout=2)
        if r.ok:
            print('READY',r.text)
            sys.exit(0)
    except Exception:
        pass
    if i % 10 == 0:
        print('wait',i,flush=True)
    time.sleep(2)
print('TIMEOUT')
sys.exit(1)
PY


Output:
Chunk ID: c53454
Wall time: 30.0012 seconds
Process running with session ID 56749
Original token count: 4
Output:
wait 0
wait 10


**Waiting for output completion**


Output:
Chunk ID: 7429d4
Wall time: 30.0007 seconds
Process running with session ID 56749
Original token count: 4
Output:
wait 20
wait 30


Summary: auto


Output:
Chunk ID: b8a650
Wall time: 30.0007 seconds
Process running with session ID 56749
Original token count: 2
Output:
wait 40


Restart succeeded and the process is actively loading models; health checks are still pending while the runtime warms up on GPU. I’ll validate the new headers immediately once it reports ready.


Output:
Chunk ID: aba28a
Wall time: 30.0006 seconds
Process running with session ID 56749
Original token count: 4
Output:
wait 50
wait 60


Output:
Chunk ID: dcb8fa
Wall time: 30.0012 seconds
Process running with session ID 56749
Original token count: 4
Output:
wait 70
wait 80


**Checking startup and log progress**


Output:
Chunk ID: 809a9e
Wall time: 0.0503 seconds
Process exited with code 0
Original token count: 454
Output:
2026-02-17 03:42:06 | INFO     | veena3.local | Auth bypass enabled (local mode). Use --auth to require API keys.
2026-02-17 03:42:06 | INFO     | veena3.local | Resolved model paths:
2026-02-17 03:42:06 | INFO     | veena3.local |   LLM (vLLM):    /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
2026-02-17 03:42:06 | INFO     | veena3.local |   BiCodec:       /home/ubuntu/veenaModal/models/spark_tts_4speaker
2026-02-17 03:42:06 | INFO     | veena3.local | ============================================================
2026-02-17 03:42:06 | INFO     | veena3.local | Initializing Veena3 TTS Runtime (local)
2026-02-17 03:42:06 | INFO     | veena3.local |   Model:       /home/ubuntu/veenaModal/models/spark_tts_4speaker
2026-02-17 03:42:06 | INFO     | veena3.local |   LLM path:    /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
2026-02-17 03:42:06 | INFO     | veena3.local |   BiCodec:     /home/ubuntu/veenaModal/models/spark_tts_4speaker
2026-02-17 03:42:06 | INFO     | veena3.local |   Device:      cuda
2026-02-17 03:42:06 | INFO     | veena3.local |   GPU Memory:  50% (total)
2026-02-17 03:42:06 | INFO     | veena3.local |   Engines:     1
2026-02-17 03:42:06 | INFO     | veena3.local |   SR:          disabled
2026-02-17 03:42:06 | INFO     | veena3.local | ============================================================
2026-02-17 03:42:06 | INFO     | veena3modal.services.tts_runtime | Initializing TTS runtime...
2026-02-17 03:42:06 | INFO     | veena3modal.services.tts_runtime |   Model path: /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
2026-02-17 03:42:06 | INFO     | veena3modal.services.tts_runtime |   BiCodec path: /home/ubuntu/veenaModal/models/spark_tts_4speaker
2026-02-17 03:42:06 | INFO     | veena3modal.services.tts_runtime |   SR enabled: False, path: None


**Waiting for model initialization**


Output:
Chunk ID: 09a3fc
Wall time: 30.0011 seconds
Process running with session ID 56749
Original token count: 5
Output:
wait 90
wait 100


Output:
Chunk ID: c2310b
Wall time: 30.0013 seconds
Process running with session ID 56749
Original token count: 3
Output:
wait 110


Output:
Chunk ID: c35641
Wall time: 30.0014 seconds
Process running with session ID 56749
Original token count: 5
Output:
wait 120
wait 130


**Checking GPU memory usage**


Output:
Chunk ID: 804a9f
Wall time: 0.0505 seconds
Process exited with code 0
Original token count: 9
Output:
3733081, VLLM::EngineCore, 41442


**Observing model warmup delay**


Runtime load is still executing (vLLM engine is resident on GPU), and the startup warmup/precompute phase is the current delay. I’m continuing to poll until it’s ready, then I’ll run request-level validation and stress tests.


Output:
Chunk ID: f84839
Wall time: 30.0014 seconds
Process running with session ID 56749
Original token count: 5
Output:
wait 140
wait 150


Output:
Chunk ID: 256340
Wall time: 30.0010 seconds
Process running with session ID 56749
Original token count: 5
Output:
wait 160
wait 170


**Planning timeout polling**


Output:
Chunk ID: bfa97f
Wall time: 6.6988 seconds
Process exited with code 1
Original token count: 2
Output:
TIMEOUT


**Inspecting server startup logs**


Output:
Chunk ID: 74bf63
Wall time: 0.0511 seconds
Process exited with code 0
Original token count: 454
Output:
2026-02-17 03:42:06 | INFO     | veena3.local | Auth bypass enabled (local mode). Use --auth to require API keys.
2026-02-17 03:42:06 | INFO     | veena3.local | Resolved model paths:
2026-02-17 03:42:06 | INFO     | veena3.local |   LLM (vLLM):    /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
2026-02-17 03:42:06 | INFO     | veena3.local |   BiCodec:       /home/ubuntu/veenaModal/models/spark_tts_4speaker
2026-02-17 03:42:06 | INFO     | veena3.local | ============================================================
2026-02-17 03:42:06 | INFO     | veena3.local | Initializing Veena3 TTS Runtime (local)
2026-02-17 03:42:06 | INFO     | veena3.local |   Model:       /home/ubuntu/veenaModal/models/spark_tts_4speaker
2026-02-17 03:42:06 | INFO     | veena3.local |   LLM path:    /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
2026-02-17 03:42:06 | INFO     | veena3.local |   BiCodec:     /home/ubuntu/veenaModal/models/spark_tts_4speaker
2026-02-17 03:42:06 | INFO     | veena3.local |   Device:      cuda
2026-02-17 03:42:06 | INFO     | veena3.local |   GPU Memory:  50% (total)
2026-02-17 03:42:06 | INFO     | veena3.local |   Engines:     1
2026-02-17 03:42:06 | INFO     | veena3.local |   SR:          disabled
2026-02-17 03:42:06 | INFO     | veena3.local | ============================================================
2026-02-17 03:42:06 | INFO     | veena3modal.services.tts_runtime | Initializing TTS runtime...
2026-02-17 03:42:06 | INFO     | veena3modal.services.tts_runtime |   Model path: /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
2026-02-17 03:42:06 | INFO     | veena3modal.services.tts_runtime |   BiCodec path: /home/ubuntu/veenaModal/models/spark_tts_4speaker
2026-02-17 03:42:06 | INFO     | veena3modal.services.tts_runtime |   SR enabled: False, path: None


**Identifying GPU-hang due to orphan process**


Output:
Chunk ID: 1b1b42
Wall time: 0.0513 seconds
Process exited with code 0
Original token count: 32
Output:
UID          PID    PPID  C STIME TTY          TIME CMD
ubuntu   3733081       1  1 01:16 ?        00:02:16 VLLM::EngineCore


**Identifying hanging server process**


Output:
Chunk ID: ee73c7
Wall time: 0.0504 seconds
Process exited with code 1
Original token count: 14
Output:
UID          PID    PPID  C STIME TTY          TIME CMD


**Investigating missing engine process**


ps -ef | rg 'local_server|VLLM::EngineCore|python3 -m veena3modal|uvicorn|engine_core|vllm' -n


Output:
Chunk ID: 56c116
Wall time: 0.0514 seconds
Process exited with code 0
Original token count: 95
Output:
289:ubuntu   3733081       1  1 01:16 ?        00:02:16 VLLM::EngineCore
315:ubuntu   3794818 3675569  0 03:48 ?        00:00:00 /bin/bash -c ps -ef | rg 'local_server|VLLM::EngineCore|python3 -m veena3modal|uvicorn|engine_core|vllm' -n
317:ubuntu   3794820 3794818  0 03:48 ?        00:00:00 rg local_server|VLLM::EngineCore|python3 -m veena3modal|uvicorn|engine_core|vllm -n


**Killing existing engine core process**


kill -9 3733081 && sleep 1; nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv,noheader,nounits || true


Output:
Chunk ID: 65978d
Wall time: 0.9222 seconds
Process exited with code 0
Original token count: 0
Output:


**Restarting local server with logging**


Output:
Chunk ID: b83542
Wall time: 0.0512 seconds
Process exited with code 0
Original token count: 2
Output:
3794879


**Planning dynamic log loading**


sleep 2; tail -n 40 /tmp/veena_local_server.log


Output:
Chunk ID: a26ec4
Wall time: 1.9052 seconds
Process exited with code 0
Original token count: 454
Output:
2026-02-17 03:48:48 | INFO     | veena3.local | Auth bypass enabled (local mode). Use --auth to require API keys.
2026-02-17 03:48:48 | INFO     | veena3.local | Resolved model paths:
2026-02-17 03:48:48 | INFO     | veena3.local |   LLM (vLLM):    /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
2026-02-17 03:48:48 | INFO     | veena3.local |   BiCodec:       /home/ubuntu/veenaModal/models/spark_tts_4speaker
2026-02-17 03:48:48 | INFO     | veena3.local | ============================================================
2026-02-17 03:48:48 | INFO     | veena3.local | Initializing Veena3 TTS Runtime (local)
2026-02-17 03:48:48 | INFO     | veena3.local |   Model:       /home/ubuntu/veenaModal/models/spark_tts_4speaker
2026-02-17 03:48:48 | INFO     | veena3.local |   LLM path:    /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
2026-02-17 03:48:48 | INFO     | veena3.local |   BiCodec:     /home/ubuntu/veenaModal/models/spark_tts_4speaker
2026-02-17 03:48:48 | INFO     | veena3.local |   Device:      cuda
2026-02-17 03:48:48 | INFO     | veena3.local |   GPU Memory:  50% (total)
2026-02-17 03:48:48 | INFO     | veena3.local |   Engines:     1
2026-02-17 03:48:48 | INFO     | veena3.local |   SR:          disabled
2026-02-17 03:48:48 | INFO     | veena3.local | ============================================================
2026-02-17 03:48:48 | INFO     | veena3modal.services.tts_runtime | Initializing TTS runtime...
2026-02-17 03:48:48 | INFO     | veena3modal.services.tts_runtime |   Model path: /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
2026-02-17 03:48:48 | INFO     | veena3modal.services.tts_runtime |   BiCodec path: /home/ubuntu/veenaModal/models/spark_tts_4speaker
2026-02-17 03:48:48 | INFO     | veena3modal.services.tts_runtime |   SR enabled: False, path: None


Summary: auto


**Checking process status after halt**


Output:
Chunk ID: 8ab8f5
Wall time: 0.0513 seconds
Process exited with code 1
Original token count: 14
Output:
UID          PID    PPID  C STIME TTY          TIME CMD


**Planning to capture crash output**


python3 -m veena3modal.local_server --model-path /home/ubuntu/veenaModal/models/spark_tts_4speaker --gpu-memory 0.5 --log-level INFO --port 8000


Output:
Chunk ID: d13d54
Wall time: 11.6717 seconds
Process exited with code 1
Original token count: 5917
Output:
2026-02-17 03:49:05 | INFO     | veena3.local | Auth bypass enabled (local mode). Use --auth to require API keys.
2026-02-17 03:49:05 | INFO     | veena3.local | Resolved model paths:
2026-02-17 03:49:05 | INFO     | veena3.local |   LLM (vLLM):    /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
2026-02-17 03:49:05 | INFO     | veena3.local |   BiCodec:       /home/ubuntu/veenaModal/models/spark_tts_4speaker
2026-02-17 03:49:05 | INFO     | veena3.local | ============================================================
2026-02-17 03:49:05 | INFO     | veena3.local | Initializing Veena3 TTS Runtime (local)
2026-02-17 03:49:05 | INFO     | veena3.local |   Model:       /home/ubuntu/veenaModal/models/spark_tts_4speaker
2026-02-17 03:49:05 | INFO     | veena3.local |   LLM path:    /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
2026-02-17 03:49:05 | INFO     | veena3.local |   BiCodec:     /home/ubuntu/veenaModal/models/spark_tts_4speaker
2026-02-17 03:49:05 | INFO     | veena3.local |   Device:      cuda
2026-02-17 03:49:05 | INFO     | veena3.local |   GPU Memory:  50% (total)
2026-02-17 03:49:05 | INFO     | veena3.local |   Engines:     1
2026-02-17 03:49:05 | INFO     | veena3.local |   SR:          disabled
2026-02-17 03:49:05 | INFO     | veena3.local | ============================================================
2026-02-17 03:49:05 | INFO     | veena3modal.services.tts_runtime | Initializing TTS runtime...
2026-02-17 03:49:05 | INFO     | veena3modal.services.tts_runtime |   Model path: /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
2026-02-17 03:49:05 | INFO     | veena3modal.services.tts_runtime |   BiCodec path: /home/ubuntu/veenaModal/models/spark_tts_4speaker
2026-02-17 03:49:05 | INFO     | veena3modal.services.tts_runtime |   SR enabled: False, path: None
2026-02-17 03:49:10 | INFO     | veena3modal.services.tts_runtime | Loading SparkTTS model with vLLM...
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
🚀 Initializing Spark TTS Model
📁 Model path: /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
🗣  Speakers: 12 predefined (lipakshi, vardan, reet, Nandini, krishna, anika, adarsh, Nilay, Aarvi, Asha, Bittu, Mira)
🎭 Emotions: 10 tags in [bracket] format

📝 Loading tokenizer from model...
✅ Tokenizer loaded: 165168 tokens

🔍 Validating 12 speakers for Spark TTS...
  lipakshi → <|speaker_0|>: 7 tokens
  vardan → <|speaker_1|>: 7 tokens
  reet → <|speaker_2|>: 7 tokens
  Nandini → <|speaker_3|>: 7 tokens
  krishna → <|speaker_4|>: 7 tokens
  anika → <|speaker_5|>: 7 tokens
  adarsh → <|speaker_6|>: 7 tokens
  Nilay → <|speaker_7|>: 7 tokens
  Aarvi → <|speaker_8|>: 7 tokens
  Asha → <|speaker_9|>: 7 tokens
  Bittu → <|speaker_10|>: 8 tokens
  Mira → <|speaker_11|>: 8 tokens
✅ All 12 speakers validated with mapping

🔍 Validating 10 emotion tags for Spark TTS...
✅ All 10 emotion tags validated

📋 Sample emotion tags:
  [angry]: 1 tokens
  [curious]: 1 tokens
  [excited]: 1 tokens
  [giggle]: 1 tokens
  [laughs harder]: 1 tokens
  ...

🔧 Initializing vLLM engine with optimizations...
  - dtype: bfloat16
  - max_model_len: 4096
  - gpu_memory_utilization: 0.5
  - enable_prefix_caching: True
  - enforce_eager (disable CUDA graphs): False
INFO 02-17 03:49:11 [model.py:541] Resolved architecture: Qwen2ForCausalLM
INFO 02-17 03:49:11 [model.py:1561] Using max model len 4096
INFO 02-17 03:49:11 [scheduler.py:226] Chunked prefill is enabled with max_num_batched_tokens=4096.
INFO 02-17 03:49:11 [vllm.py:624] Asynchronous scheduling is enabled.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(EngineCore_DP0 pid=3795049) INFO 02-17 03:49:12 [core.py:96] Initializing a V1 LLM engine (v0.15.1) with config: model='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', speculative_config=None, tokenizer='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [4096], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 256, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': True}, 'local_cache_dir': None, 'static_all_moe_layers': []}
(EngineCore_DP0 pid=3795049) INFO 02-17 03:49:14 [parallel_state.py:1212] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://216.81.248.184:42381 backend=nccl
(EngineCore_DP0 pid=3795049) INFO 02-17 03:49:14 [parallel_state.py:1423] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
(EngineCore_DP0 pid=3795049) ERROR 02-17 03:49:15 [core.py:946] EngineCore failed to start.
(EngineCore_DP0 pid=3795049) ERROR 02-17 03:49:15 [core.py:946] Traceback (most recent call last):
(EngineCore_DP0 pid=3795049) ERROR 02-17 03:49:15 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 937, in run_engine_core
(EngineCore_DP0 pid=3795049) ERROR 02-17 03:49:15 [core.py:946]     engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore_DP0 pid=3795049) ERROR 02-17 03:49:15 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 691, in __init__
(EngineCore_DP0 pid=3795049) ERROR 02-17 03:49:15 [core.py:946]     super().__init__(
(EngineCore_DP0 pid=3795049) ERROR 02-17 03:49:15 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 105, in __init__
(EngineCore_DP0 pid=3795049) ERROR 02-17 03:49:15 [core.py:946]     self.model_executor = executor_class(vllm_config)
(EngineCore_DP0 pid=3795049) ERROR 02-17 03:49:15 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/executor/abstract.py", line 101, in __init__
(EngineCore_DP0 pid=3795049) ERROR 02-17 03:49:15 [core.py:946]     self._init_executor()
(EngineCore_DP0 pid=3795049) ERROR 02-17 03:49:15 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/executor/uniproc_executor.py", line 47, in _init_executor
(EngineCore_DP0 pid=3795049) ERROR 02-17 03:49:15 [core.py:946]     self.driver_worker.init_device()
(EngineCore_DP0 pid=3795049) ERROR 02-17 03:49:15 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/worker/worker_base.py", line 326, in init_device
(EngineCore_DP0 pid=3795049) ERROR 02-17 03:49:15 [core.py:946]     self.worker.init_device()  # type: ignore
(EngineCore_DP0 pid=3795049) ERROR 02-17 03:49:15 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/worker/gpu_worker.py", line 258, in init_device
(EngineCore_DP0 pid=3795049) ERROR 02-17 03:49:15 [core.py:946]     from vllm.v1.worker.gpu_model_runner import (
(EngineCore_DP0 pid=3795049) ERROR 02-17 03:49:15 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/worker/gpu_model_runner.py", line 153, in <module>
(EngineCore_DP0 pid=3795049) ERROR 02-17 03:49:15 [core.py:946]     from vllm.v1.spec_decode.ngram_proposer import NgramProposer
(EngineCore_DP0 pid=3795049) ERROR 02-17 03:49:15 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/spec_decode/ngram_proposer.py", line 170, in <module>
(EngineCore_DP0 pid=3795049) ERROR 02-17 03:49:15 [core.py:946]     def batch_propose_numba(
(EngineCore_DP0 pid=3795049) ERROR 02-17 03:49:15 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/decorators.py", line 221, in wrapper
(EngineCore_DP0 pid=3795049) ERROR 02-17 03:49:15 [core.py:946]     disp = dispatcher(py_func=func, locals=locals,
(EngineCore_DP0 pid=3795049) ERROR 02-17 03:49:15 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/dispatcher.py", line 772, in __init__
(EngineCore_DP0 pid=3795049) ERROR 02-17 03:49:15 [core.py:946]     self.targetctx = self.targetdescr.target_context
(EngineCore_DP0 pid=3795049) ERROR 02-17 03:49:15 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/registry.py", line 30, in target_context
(EngineCore_DP0 pid=3795049) ERROR 02-17 03:49:15 [core.py:946]     return self._toplevel_target_context
(EngineCore_DP0 pid=3795049) ERROR 02-17 03:49:15 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/utils.py", line 741, in __get__
(EngineCore_DP0 pid=3795049) ERROR 02-17 03:49:15 [core.py:946]     return super().__get__(*args, **kwargs)
(EngineCore_DP0 pid=3795049) ERROR 02-17 03:49:15 [core.py:946]   File "/usr/lib/python3.10/functools.py", line 981, in __get__
(EngineCore_DP0 pid=3795049) ERROR 02-17 03:49:15 [core.py:946]     val = self.func(instance)
(EngineCore_DP0 pid=3795049) ERROR 02-17 03:49:15 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/registry.py", line 18, in _toplevel_target_context
(EngineCore_DP0 pid=3795049) ERROR 02-17 03:49:15 [core.py:946]     return cpu.CPUContext(self.typing_context, self._target_name)
(EngineCore_DP0 pid=3795049) ERROR 02-17 03:49:15 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/cpu.py", line 42, in __init__
(EngineCore_DP0 pid=3795049) ERROR 02-17 03:49:15 [core.py:946]     super().__init__(typingctx, target)
(EngineCore_DP0 pid=3795049) ERROR 02-17 03:49:15 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/base.py", line 254, in __init__
(EngineCore_DP0 pid=3795049) ERROR 02-17 03:49:15 [core.py:946]     self.init()
(EngineCore_DP0 pid=3795049) ERROR 02-17 03:49:15 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/compiler_lock.py", line 35, in _acquire_compile_lock
(EngineCore_DP0 pid=3795049) ERROR 02-17 03:49:15 [core.py:946]     return func(*args, **kwargs)
(EngineCore_DP0 pid=3795049) ERROR 02-17 03:49:15 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/cpu.py", line 51, in init
(EngineCore_DP0 pid=3795049) ERROR 02-17 03:49:15 [core.py:946]     self._internal_codegen = codegen.JITCPUCodegen("numba.exec")
(EngineCore_DP0 pid=3795049) ERROR 02-17 03:49:15 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/codegen.py", line 1172, in __init__
(EngineCore_DP0 pid=3795049) ERROR 02-17 03:49:15 [core.py:946]     initialize_llvm()
(EngineCore_DP0 pid=3795049) ERROR 02-17 03:49:15 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/codegen.py", line 1434, in initialize_llvm
(EngineCore_DP0 pid=3795049) ERROR 02-17 03:49:15 [core.py:946]     ll.initialize()
(EngineCore_DP0 pid=3795049) ERROR 02-17 03:49:15 [core.py:946]   File "/home/ubuntu/.local/lib/python3.10/site-packages/llvmlite/binding/initfini.py", line 17, in initialize
(EngineCore_DP0 pid=3795049) ERROR 02-17 03:49:15 [core.py:946]     raise RuntimeError(
(EngineCore_DP0 pid=3795049) ERROR 02-17 03:49:15 [core.py:946] RuntimeError: llvmlite.binding.initialize() is deprecated and will be removed. LLVM initialization is now handled automatically. Please remove calls to this function from your code and check for other behavioral changes that may have occurred due to LLVM updates.
(EngineCore_DP0 pid=3795049) Process EngineCore_DP0:
(EngineCore_DP0 pid=3795049) Traceback (most recent call last):
(EngineCore_DP0 pid=3795049)   File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore_DP0 pid=3795049)     self.run()
(EngineCore_DP0 pid=3795049)   File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
(EngineCore_DP0 pid=3795049)     self._target(*self._args, **self._kwargs)
(EngineCore_DP0 pid=3795049)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 950, in run_engine_core
(EngineCore_DP0 pid=3795049)     raise e
(EngineCore_DP0 pid=3795049)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 937, in run_engine_core
(EngineCore_DP0 pid=3795049)     engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore_DP0 pid=3795049)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 691, in __init__
(EngineCore_DP0 pid=3795049)     super().__init__(
(EngineCore_DP0 pid=3795049)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 105, in __init__
(EngineCore_DP0 pid=3795049)     self.model_executor = executor_class(vllm_config)
(EngineCore_DP0 pid=3795049)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/executor/abstract.py", line 101, in __init__
(EngineCore_DP0 pid=3795049)     self._init_executor()
(EngineCore_DP0 pid=3795049)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/executor/uniproc_executor.py", line 47, in _init_executor
(EngineCore_DP0 pid=3795049)     self.driver_worker.init_device()
(EngineCore_DP0 pid=3795049)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/worker/worker_base.py", line 326, in init_device
(EngineCore_DP0 pid=3795049)     self.worker.init_device()  # type: ignore
(EngineCore_DP0 pid=3795049)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/worker/gpu_worker.py", line 258, in init_device
(EngineCore_DP0 pid=3795049)     from vllm.v1.worker.gpu_model_runner import (
(EngineCore_DP0 pid=3795049)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/worker/gpu_model_runner.py", line 153, in <module>
(EngineCore_DP0 pid=3795049)     from vllm.v1.spec_decode.ngram_proposer import NgramProposer
(EngineCore_DP0 pid=3795049)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/spec_decode/ngram_proposer.py", line 170, in <module>
(EngineCore_DP0 pid=3795049)     def batch_propose_numba(
(EngineCore_DP0 pid=3795049)   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/decorators.py", line 221, in wrapper
(EngineCore_DP0 pid=3795049)     disp = dispatcher(py_func=func, locals=locals,
(EngineCore_DP0 pid=3795049)   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/dispatcher.py", line 772, in __init__
(EngineCore_DP0 pid=3795049)     self.targetctx = self.targetdescr.target_context
(EngineCore_DP0 pid=3795049)   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/registry.py", line 30, in target_context
(EngineCore_DP0 pid=3795049)     return self._toplevel_target_context
(EngineCore_DP0 pid=3795049)   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/utils.py", line 741, in __get__
(EngineCore_DP0 pid=3795049)     return super().__get__(*args, **kwargs)
(EngineCore_DP0 pid=3795049)   File "/usr/lib/python3.10/functools.py", line 981, in __get__
(EngineCore_DP0 pid=3795049)     val = self.func(instance)
(EngineCore_DP0 pid=3795049)   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/registry.py", line 18, in _toplevel_target_context
(EngineCore_DP0 pid=3795049)     return cpu.CPUContext(self.typing_context, self._target_name)
(EngineCore_DP0 pid=3795049)   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/cpu.py", line 42, in __init__
(EngineCore_DP0 pid=3795049)     super().__init__(typingctx, target)
(EngineCore_DP0 pid=3795049)   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/base.py", line 254, in __init__
(EngineCore_DP0 pid=3795049)     self.init()
(EngineCore_DP0 pid=3795049)   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/compiler_lock.py", line 35, in _acquire_compile_lock
(EngineCore_DP0 pid=3795049)     return func(*args, **kwargs)
(EngineCore_DP0 pid=3795049)   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/cpu.py", line 51, in init
(EngineCore_DP0 pid=3795049)     self._internal_codegen = codegen.JITCPUCodegen("numba.exec")
(EngineCore_DP0 pid=3795049)   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/codegen.py", line 1172, in __init__
(EngineCore_DP0 pid=3795049)     initialize_llvm()
(EngineCore_DP0 pid=3795049)   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/codegen.py", line 1434, in initialize_llvm
(EngineCore_DP0 pid=3795049)     ll.initialize()
(EngineCore_DP0 pid=3795049)   File "/home/ubuntu/.local/lib/python3.10/site-packages/llvmlite/binding/initfini.py", line 17, in initialize
(EngineCore_DP0 pid=3795049)     raise RuntimeError(
(EngineCore_DP0 pid=3795049) RuntimeError: llvmlite.binding.initialize() is deprecated and will be removed. LLVM initialization is now handled automatically. Please remove calls to this function from your code and check for other behavioral changes that may have occurred due to LLVM updates.
2026-02-17 03:49:15 | ERROR    | veena3modal.services.tts_runtime | ❌ Failed to initialize TTS runtime: Engine core initialization failed. See root cause above. Failed core proc(s): {}
Traceback (most recent call last):
  File "/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py", line 149, in initialize_runtime
    model = SparkTTSModel(
  File "/home/ubuntu/veenaModal/veena3modal/core/model_loader.py", line 128, in __init__
    self.engine = AsyncLLMEngine.from_engine_args(engine_args)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/async_llm.py", line 257, in from_engine_args
    return cls(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/async_llm.py", line 155, in __init__
    self.engine_core = EngineCoreClient.make_async_mp_client(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 122, in make_async_mp_client
    return AsyncMPClient(*client_args)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 819, in __init__
    super().__init__(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 479, in __init__
    with launch_core_engines(vllm_config, executor_class, log_stats) as (
  File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
    next(self.gen)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/utils.py", line 933, in launch_core_engines
    wait_for_engine_startup(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/utils.py", line 992, in wait_for_engine_startup
    raise RuntimeError(
RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
Traceback (most recent call last):
  File "/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py", line 149, in initialize_runtime
    model = SparkTTSModel(
  File "/home/ubuntu/veenaModal/veena3modal/core/model_loader.py", line 128, in __init__
    self.engine = AsyncLLMEngine.from_engine_args(engine_args)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/async_llm.py", line 257, in from_engine_args
    return cls(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/async_llm.py", line 155, in __init__
    self.engine_core = EngineCoreClient.make_async_mp_client(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 122, in make_async_mp_client
    return AsyncMPClient(*client_args)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 819, in __init__
    super().__init__(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 479, in __init__
    with launch_core_engines(vllm_config, executor_class, log_stats) as (
  File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
    next(self.gen)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/utils.py", line 933, in launch_core_engines
    wait_for_engine_startup(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/utils.py", line 992, in wait_for_engine_startup
    raise RuntimeError(
RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/ubuntu/veenaModal/veena3modal/local_server.py", line 465, in <module>
    main()
  File "/home/ubuntu/veenaModal/veena3modal/local_server.py", line 426, in main
    initialize_local_runtime(
  File "/home/ubuntu/veenaModal/veena3modal/local_server.py", line 224, in initialize_local_runtime
    runtime = initialize_runtime(
  File "/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py", line 254, in initialize_runtime
    raise RuntimeError(f"TTS runtime initialization failed: {e}") from e
RuntimeError: TTS runtime initialization failed: Engine core initialization failed. See root cause above. Failed core proc(s): {}


**Diagnosing environment dependency issue**


VLLM_USE_V1=0 python3 -m veena3modal.local_server --model-path /home/ubuntu/veenaModal/models/spark_tts_4speaker --gpu-memory 0.5 --log-level INFO --port 8000


Output:
Chunk ID: a16efc
Wall time: 11.3989 seconds
Process exited with code 1
Original token count: 5917
Output:
Total output lines: 231

2026-02-17 03:49:27 | INFO     | veena3.local | Auth bypass enabled (local mode). Use --auth to require API keys.
2026-02-17 03:49:27 | INFO     | veena3.local | Resolved model paths:
2026-02-17 03:49:27 | INFO     | veena3.local |   LLM (vLLM):    /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
2026-02-17 03:49:27 | INFO     | veena3.local |   BiCodec:       /home/ubuntu/veenaModal/models/spark_tts_4speaker
2026-02-17 03:49:27 | INFO     | veena3.local | ============================================================
2026-02-17 03:49:27 | INFO     | veena3.local | Initializing Veena3 TTS Runtime (local)
2026-02-17 03:49:27 | INFO     | veena3.local |   Model:       /home/ubuntu/veenaModal/models/spark_tts_4speaker
2026-02-17 03:49:27 | INFO     | veena3.local |   LLM path:    /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
2026-02-17 03:49:27 | INFO     | veena3.local |   BiCodec:     /home/ubuntu/veenaModal/models/spark_tts_4speaker
2026-02-17 03:49:27 | INFO     | veena3.local |   Device:      cuda
2026-02-17 03:49:27 | INFO     | veena3.local |   GPU Memory:  50% (total)
2026-02-17 03:49:27 | INFO     | veena3.local |   Engines:     1
2026-02-17 03:49:27 | INFO     | veena3.local |   SR:          disabled
2026-02-17 03:49:27 | INFO     | veena3.local | ============================================================
2026-02-17 03:49:27 | INFO     | veena3modal.services.tts_runtime | Initializing TTS runtime...
2026-02-17 03:49:27 | INFO     | veena3modal.services.tts_runtime |   Model path: /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
2026-02-17 03:49:27 | INFO     | veena3modal.services.tts_runtime |   BiCodec path: /home/ubuntu/veenaModal/models/spark_tts_4speaker
2026-02-17 03:49:27 | INFO     | veena3modal.services.tts_runtime |   SR enabled: False, path: None
2026-02-17 03:49:32 | INFO     | veena3modal.services.tts_runtime | Loading SparkTTS model with vLLM...
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
🚀 Initializing Spark TTS Model
📁 Model path: /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
🗣  Speakers: 12 predefined (lipakshi, vardan, reet, Nandini, krishna, anika, adarsh, Nilay, Aarvi, Asha, Bittu, Mira)
🎭 Emotions: 10 tags in [bracket] format

📝 Loading tokenizer from model...
✅ Tokenizer loaded: 165168 tokens

🔍 Validating 12 speakers for Spark TTS...
  lipakshi → <|speaker_0|>: 7 tokens
  vardan → <|speaker_1|>: 7 tokens
  reet → <|speaker_2|>: 7 tokens
  Nandini → <|speaker_3|>: 7 tokens
  krishna → <|speaker_4|>: 7 tokens
  anika → <|speaker_5|>: 7 tokens
  adarsh → <|speaker_6|>: 7 tokens
  Nilay → <|speaker_7|>: 7 tokens
  Aarvi → <|speaker_8|>: 7 tokens
  Asha → <|speaker_9|>: 7 tokens
  Bittu → <|speaker_10|>: 8 tokens
  Mira → <|speaker_11|>: 8 tokens
✅ All 12 speakers validated with mapping

🔍 Validating 10 emotion tags for Spark TTS...
✅ All 10 emotion tags validated

📋 Sample emotion tags:
  [angry]: 1 tokens
  [curious]: 1 tokens
  [excited]: 1 tokens
  [giggle]: 1 tokens
  [laughs harder]: 1 tokens
  ...

🔧 Initializing vLLM engine with optimizations...
  - dtype: bfloat16
  - max_model_len: 4096
  - gpu_memory_utilization: 0.5
  - enable_prefix_caching: True
  - enforce_eager (disable CUDA graphs): False
INFO 02-17 03:49:33 [model.py:541] Resolved architecture: Qwen2ForCausalLM
INFO 02-17 03:49:33 [model.py:1561] Using max model len 4096
INFO 02-17 03:49:33 [scheduler.py:226] Chunked prefill is enabled with max_num_batched_tokens=4096.
INFO 02-17 03:49:33 [vllm.py:624] Asynchronous scheduling is enabled.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(EngineCore_DP0 pid=3795245) INFO 02-17 03:49:34 [core.py:96] Initializing a V1 LLM engine (v0.15.1) with config: model='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', speculative_config=None, tokenizer='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [4096], 'ind…2917 tokens truncated…urn cpu.CPUContext(self.typing_context, self._target_name)
(EngineCore_DP0 pid=3795245)   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/cpu.py", line 42, in __init__
(EngineCore_DP0 pid=3795245)     super().__init__(typingctx, target)
(EngineCore_DP0 pid=3795245)   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/base.py", line 254, in __init__
(EngineCore_DP0 pid=3795245)     self.init()
(EngineCore_DP0 pid=3795245)   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/compiler_lock.py", line 35, in _acquire_compile_lock
(EngineCore_DP0 pid=3795245)     return func(*args, **kwargs)
(EngineCore_DP0 pid=3795245)   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/cpu.py", line 51, in init
(EngineCore_DP0 pid=3795245)     self._internal_codegen = codegen.JITCPUCodegen("numba.exec")
(EngineCore_DP0 pid=3795245)   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/codegen.py", line 1172, in __init__
(EngineCore_DP0 pid=3795245)     initialize_llvm()
(EngineCore_DP0 pid=3795245)   File "/home/ubuntu/.local/lib/python3.10/site-packages/numba/core/codegen.py", line 1434, in initialize_llvm
(EngineCore_DP0 pid=3795245)     ll.initialize()
(EngineCore_DP0 pid=3795245)   File "/home/ubuntu/.local/lib/python3.10/site-packages/llvmlite/binding/initfini.py", line 17, in initialize
(EngineCore_DP0 pid=3795245)     raise RuntimeError(
(EngineCore_DP0 pid=3795245) RuntimeError: llvmlite.binding.initialize() is deprecated and will be removed. LLVM initialization is now handled automatically. Please remove calls to this function from your code and check for other behavioral changes that may have occurred due to LLVM updates.
2026-02-17 03:49:37 | ERROR    | veena3modal.services.tts_runtime | ❌ Failed to initialize TTS runtime: Engine core initialization failed. See root cause above. Failed core proc(s): {}
Traceback (most recent call last):
  File "/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py", line 149, in initialize_runtime
    model = SparkTTSModel(
  File "/home/ubuntu/veenaModal/veena3modal/core/model_loader.py", line 128, in __init__
    self.engine = AsyncLLMEngine.from_engine_args(engine_args)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/async_llm.py", line 257, in from_engine_args
    return cls(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/async_llm.py", line 155, in __init__
    self.engine_core = EngineCoreClient.make_async_mp_client(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 122, in make_async_mp_client
    return AsyncMPClient(*client_args)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 819, in __init__
    super().__init__(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 479, in __init__
    with launch_core_engines(vllm_config, executor_class, log_stats) as (
  File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
    next(self.gen)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/utils.py", line 933, in launch_core_engines
    wait_for_engine_startup(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/utils.py", line 992, in wait_for_engine_startup
    raise RuntimeError(
RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
Traceback (most recent call last):
  File "/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py", line 149, in initialize_runtime
    model = SparkTTSModel(
  File "/home/ubuntu/veenaModal/veena3modal/core/model_loader.py", line 128, in __init__
    self.engine = AsyncLLMEngine.from_engine_args(engine_args)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/async_llm.py", line 257, in from_engine_args
    return cls(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/async_llm.py", line 155, in __init__
    self.engine_core = EngineCoreClient.make_async_mp_client(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 122, in make_async_mp_client
    return AsyncMPClient(*client_args)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 819, in __init__
    super().__init__(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 479, in __init__
    with launch_core_engines(vllm_config, executor_class, log_stats) as (
  File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
    next(self.gen)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/utils.py", line 933, in launch_core_engines
    wait_for_engine_startup(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/utils.py", line 992, in wait_for_engine_startup
    raise RuntimeError(
RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/ubuntu/veenaModal/veena3modal/local_server.py", line 465, in <module>
    main()
  File "/home/ubuntu/veenaModal/veena3modal/local_server.py", line 426, in main
    initialize_local_runtime(
  File "/home/ubuntu/veenaModal/veena3modal/local_server.py", line 224, in initialize_local_runtime
    runtime = initialize_runtime(
  File "/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py", line 254, in initialize_runtime
    raise RuntimeError(f"TTS runtime initialization failed: {e}") from e
RuntimeError: TTS runtime initialization failed: Engine core initialization failed. See root cause above. Failed core proc(s): {}


**Diagnosing numba/llvmlite mismatch**


python3 - <<'PY'
import numba, llvmlite
print('numba', numba.__version__)
print('llvmlite', llvmlite.__version__)
PY


python3 -m pip show numba llvmlite | sed -n '1,120p'


Output:
Chunk ID: f513eb
Wall time: 0.2268 seconds
Process exited with code 0
Original token count: 8
Output:
numba 0.61.2
llvmlite 0.46.0


Output:
Chunk ID: f9f411
Wall time: 0.6182 seconds
Process exited with code 0
Original token count: 128
Output:
Name: numba
Version: 0.61.2
Summary: compiling Python code using LLVM
Home-page: https://numba.pydata.org
Author:
Author-email:
License: BSD
Location: /home/ubuntu/.local/lib/python3.10/site-packages
Requires: llvmlite, numpy
Required-by: vllm
---
Name: llvmlite
Version: 0.44.0
Summary: lightweight wrapper around basic LLVM functionality
Home-page: http://llvmlite.readthedocs.io
Author:
Author-email:
License: BSD
Location: /home/ubuntu/.local/lib/python3.10/site-packages
Requires:
Required-by: numba


**Investigating conflicting package versions**


python3 - <<'PY'
import numba, llvmlite, sys
print('numba', numba.__version__, numba.__file__)
print('llvmlite', llvmlite.__version__, llvmlite.__file__)
print('python', sys.executable)
PY


Output:
Chunk ID: 5aa886
Wall time: 0.2010 seconds
Process exited with code 0
Original token count: 48
Output:
numba 0.61.2 /home/ubuntu/.local/lib/python3.10/site-packages/numba/__init__.py
llvmlite 0.46.0 /home/ubuntu/.local/lib/python3.10/site-packages/llvmlite/__init__.py
python /usr/bin/python3


**Resolving llvmlite-numba version mismatch**


python3 -m pip install --user --force-reinstall 'llvmlite==0.44.0'


Output:
Chunk ID: 906d68
Wall time: 2.1254 seconds
Process exited with code 0
Original token count: 142
Output:
Looking in indexes: https://mcache-dsm.massedcompute.com/simple, https://pypi.org/simple
Collecting llvmlite==0.44.0
  Using cached llvmlite-0.44.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.8 kB)
Using cached llvmlite-0.44.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (42.4 MB)
Installing collected packages: llvmlite
  Attempting uninstall: llvmlite
    Found existing installation: llvmlite 0.44.0
    Uninstalling llvmlite-0.44.0:
      Successfully uninstalled llvmlite-0.44.0
Successfully installed llvmlite-0.44.0


**Checking imported versions**


python3 - <<'PY'
import numba, llvmlite
print(numba.__version__, llvmlite.__version__)
print(llvmlite.__file__)
PY


Output:
Chunk ID: 85be88
Wall time: 0.1756 seconds
Process exited with code 0
Original token count: 21
Output:
0.61.2 0.44.0
/home/ubuntu/.local/lib/python3.10/site-packages/llvmlite/__init__.py


**Restarting server to apply fix**


ps -ef | rg 'VLLM::EngineCore|veena3modal.local_server|python3 -m veena3modal.local_server' -n || true


Output:
Chunk ID: b91413
Wall time: 0.0513 seconds
Process exited with code 0
Original token count: 78
Output:
318:ubuntu   3795612 3675569  0 03:50 ?        00:00:00 /bin/bash -c ps -ef | rg 'VLLM::EngineCore|veena3modal.local_server|python3 -m veena3modal.local_server' -n || true
320:ubuntu   3795614 3795612  0 03:50 ?        00:00:00 rg VLLM::EngineCore|veena3modal.local_server|python3 -m veena3modal.local_server -n


Summary: auto


**Restarting server to test**


Output:
Chunk ID: 67d7b0
Wall time: 0.0503 seconds
Process exited with code 0
Original token count: 2
Output:
3795645


**Extending readiness wait time**


python3 - <<'PY'
import time,requests,sys
url='http://localhost:8000/v1/tts/health'
for i in range(240):
    try:
        r=requests.get(url,timeout=2)
        if r.ok:
            print('READY',r.text)
            sys.exit(0)
    except Exception:
        pass
    if i % 10 == 0:
        print('wait',i,flush=True)
    time.sleep(1)
print('TIMEOUT')
sys.exit(1)
PY


Output:
Chunk ID: 5adad9
Wall time: 30.0013 seconds
Process running with session ID 91433
Original token count: 6
Output:
wait 0
wait 10
wait 20


Output:
Chunk ID: f2294d
Wall time: 30.0010 seconds
Process running with session ID 91433
Original token count: 8
Output:
wait 30
wait 40
wait 50
wait 60


Output:
Chunk ID: 4116f9
Wall time: 30.0009 seconds
Process running with session ID 91433
Original token count: 9
Output:
wait 70
wait 80
wait 90
wait 100


**Checking precompute logs for stall**


Output:
Chunk ID: 5039c2
Wall time: 0.0504 seconds
Process exited with code 0
Original token count: 454
Output:
2026-02-17 03:50:24 | INFO     | veena3.local | Auth bypass enabled (local mode). Use --auth to require API keys.
2026-02-17 03:50:24 | INFO     | veena3.local | Resolved model paths:
2026-02-17 03:50:24 | INFO     | veena3.local |   LLM (vLLM):    /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
2026-02-17 03:50:24 | INFO     | veena3.local |   BiCodec:       /home/ubuntu/veenaModal/models/spark_tts_4speaker
2026-02-17 03:50:24 | INFO     | veena3.local | ============================================================
2026-02-17 03:50:24 | INFO     | veena3.local | Initializing Veena3 TTS Runtime (local)
2026-02-17 03:50:24 | INFO     | veena3.local |   Model:       /home/ubuntu/veenaModal/models/spark_tts_4speaker
2026-02-17 03:50:24 | INFO     | veena3.local |   LLM path:    /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
2026-02-17 03:50:24 | INFO     | veena3.local |   BiCodec:     /home/ubuntu/veenaModal/models/spark_tts_4speaker
2026-02-17 03:50:24 | INFO     | veena3.local |   Device:      cuda
2026-02-17 03:50:24 | INFO     | veena3.local |   GPU Memory:  50% (total)
2026-02-17 03:50:24 | INFO     | veena3.local |   Engines:     1
2026-02-17 03:50:24 | INFO     | veena3.local |   SR:          disabled
2026-02-17 03:50:24 | INFO     | veena3.local | ============================================================
2026-02-17 03:50:24 | INFO     | veena3modal.services.tts_runtime | Initializing TTS runtime...
2026-02-17 03:50:24 | INFO     | veena3modal.services.tts_runtime |   Model path: /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
2026-02-17 03:50:24 | INFO     | veena3modal.services.tts_runtime |   BiCodec path: /home/ubuntu/veenaModal/models/spark_tts_4speaker
2026-02-17 03:50:24 | INFO     | veena3modal.services.tts_runtime |   SR enabled: False, path: None


**Checking process liveliness due to output buffering**


ps -fp 3795645; echo '---'; ps -ef | rg 'VLLM::EngineCore' -n || true


Output:
Chunk ID: cd6cc4
Wall time: 0.0503 seconds
Process exited with code 0
Original token count: 70
Output:
UID          PID    PPID  C STIME TTY          TIME CMD
---
319:ubuntu   3796353 3675569  0 03:52 ?        00:00:00 /bin/bash -c ps -fp 3795645; echo '---'; ps -ef | rg 'VLLM::EngineCore' -n || true
321:ubuntu   3796356 3796353  0 03:52 ?        00:00:00 rg VLLM::EngineCore -n


**Confirming crash with fixed dependency**


Output:
Chunk ID: e0cc6b
Wall time: 30.0018 seconds
Process running with session ID 61574
Original token count: 3146
Output:
2026-02-17 03:52:28 | INFO     | veena3.local | Auth bypass enabled (local mode). Use --auth to require API keys.
2026-02-17 03:52:28 | INFO     | veena3.local | Resolved model paths:
2026-02-17 03:52:28 | INFO     | veena3.local |   LLM (vLLM):    /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
2026-02-17 03:52:28 | INFO     | veena3.local |   BiCodec:       /home/ubuntu/veenaModal/models/spark_tts_4speaker
2026-02-17 03:52:28 | INFO     | veena3.local | ============================================================
2026-02-17 03:52:28 | INFO     | veena3.local | Initializing Veena3 TTS Runtime (local)
2026-02-17 03:52:28 | INFO     | veena3.local |   Model:       /home/ubuntu/veenaModal/models/spark_tts_4speaker
2026-02-17 03:52:28 | INFO     | veena3.local |   LLM path:    /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
2026-02-17 03:52:28 | INFO     | veena3.local |   BiCodec:     /home/ubuntu/veenaModal/models/spark_tts_4speaker
2026-02-17 03:52:28 | INFO     | veena3.local |   Device:      cuda
2026-02-17 03:52:28 | INFO     | veena3.local |   GPU Memory:  50% (total)
2026-02-17 03:52:28 | INFO     | veena3.local |   Engines:     1
2026-02-17 03:52:28 | INFO     | veena3.local |   SR:          disabled
2026-02-17 03:52:28 | INFO     | veena3.local | ============================================================
2026-02-17 03:52:28 | INFO     | veena3modal.services.tts_runtime | Initializing TTS runtime...
2026-02-17 03:52:28 | INFO     | veena3modal.services.tts_runtime |   Model path: /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
2026-02-17 03:52:28 | INFO     | veena3modal.services.tts_runtime |   BiCodec path: /home/ubuntu/veenaModal/models/spark_tts_4speaker
2026-02-17 03:52:28 | INFO     | veena3modal.services.tts_runtime |   SR enabled: False, path: None
2026-02-17 03:52:34 | INFO     | veena3modal.services.tts_runtime | Loading SparkTTS model with vLLM...
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
🚀 Initializing Spark TTS Model
📁 Model path: /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
🗣  Speakers: 12 predefined (lipakshi, vardan, reet, Nandini, krishna, anika, adarsh, Nilay, Aarvi, Asha, Bittu, Mira)
🎭 Emotions: 10 tags in [bracket] format

📝 Loading tokenizer from model...
✅ Tokenizer loaded: 165168 tokens

🔍 Validating 12 speakers for Spark TTS...
  lipakshi → <|speaker_0|>: 7 tokens
  vardan → <|speaker_1|>: 7 tokens
  reet → <|speaker_2|>: 7 tokens
  Nandini → <|speaker_3|>: 7 tokens
  krishna → <|speaker_4|>: 7 tokens
  anika → <|speaker_5|>: 7 tokens
  adarsh → <|speaker_6|>: 7 tokens
  Nilay → <|speaker_7|>: 7 tokens
  Aarvi → <|speaker_8|>: 7 tokens
  Asha → <|speaker_9|>: 7 tokens
  Bittu → <|speaker_10|>: 8 tokens
  Mira → <|speaker_11|>: 8 tokens
✅ All 12 speakers validated with mapping

🔍 Validating 10 emotion tags for Spark TTS...
✅ All 10 emotion tags validated

📋 Sample emotion tags:
  [angry]: 1 tokens
  [curious]: 1 tokens
  [excited]: 1 tokens
  [giggle]: 1 tokens
  [laughs harder]: 1 tokens
  ...

🔧 Initializing vLLM engine with optimizations...
  - dtype: bfloat16
  - max_model_len: 4096
  - gpu_memory_utilization: 0.5
  - enable_prefix_caching: True
  - enforce_eager (disable CUDA graphs): False
INFO 02-17 03:52:34 [model.py:541] Resolved architecture: Qwen2ForCausalLM
INFO 02-17 03:52:34 [model.py:1561] Using max model len 4096
INFO 02-17 03:52:34 [scheduler.py:226] Chunked prefill is enabled with max_num_batched_tokens=4096.
INFO 02-17 03:52:34 [vllm.py:624] Asynchronous scheduling is enabled.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(EngineCore_DP0 pid=3796506) INFO 02-17 03:52:35 [core.py:96] Initializing a V1 LLM engine (v0.15.1) with config: model='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', speculative_config=None, tokenizer='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [4096], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 256, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': True}, 'local_cache_dir': None, 'static_all_moe_layers': []}
(EngineCore_DP0 pid=3796506) INFO 02-17 03:52:38 [parallel_state.py:1212] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://216.81.248.184:39483 backend=nccl
(EngineCore_DP0 pid=3796506) INFO 02-17 03:52:38 [parallel_state.py:1423] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
(EngineCore_DP0 pid=3796506) INFO 02-17 03:52:38 [gpu_model_runner.py:4033] Starting to load model /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM...
(EngineCore_DP0 pid=3796506) INFO 02-17 03:52:39 [cuda.py:364] Using FLASH_ATTN attention backend out of potential backends: ('FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION')
(EngineCore_DP0 pid=3796506) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
(EngineCore_DP0 pid=3796506) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
(EngineCore_DP0 pid=3796506)
Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
(EngineCore_DP0 pid=3796506)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.94it/s]
(EngineCore_DP0 pid=3796506)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.94it/s]
(EngineCore_DP0 pid=3796506)
(EngineCore_DP0 pid=3796506) INFO 02-17 03:52:40 [default_loader.py:291] Loading weights took 0.58 seconds
(EngineCore_DP0 pid=3796506) INFO 02-17 03:52:41 [gpu_model_runner.py:4130] Model loading took 0.95 GiB memory and 1.123326 seconds
(EngineCore_DP0 pid=3796506) INFO 02-17 03:52:45 [backends.py:812] Using cache directory: /home/ubuntu/.cache/vllm/torch_compile_cache/f3ff13511c/rank_0_0/backbone for vLLM's torch.compile
(EngineCore_DP0 pid=3796506) INFO 02-17 03:52:45 [backends.py:872] Dynamo bytecode transform time: 4.20 s
(EngineCore_DP0 pid=3796506) INFO 02-17 03:52:48 [backends.py:267] Directly load the compiled graph(s) for compile range (1, 4096) from the cache, took 0.629 s
(EngineCore_DP0 pid=3796506) INFO 02-17 03:52:48 [monitor.py:34] torch.compile takes 4.83 s in total
(EngineCore_DP0 pid=3796506) INFO 02-17 03:52:49 [gpu_worker.py:356] Available KV cache memory: 37.87 GiB
(EngineCore_DP0 pid=3796506) INFO 02-17 03:52:49 [kv_cache_utils.py:1307] GPU KV cache size: 3,309,040 tokens
(EngineCore_DP0 pid=3796506) INFO 02-17 03:52:49 [kv_cache_utils.py:1312] Maximum concurrency for 4,096 tokens per request: 807.87x
(EngineCore_DP0 pid=3796506)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   0%|          | 0/35 [00:00<?, ?it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  11%|█▏        | 4/35 [00:00<00:00, 32.60it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  23%|██▎       | 8/35 [00:00<00:00, 35.38it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  34%|███▍      | 12/35 [00:00<00:00, 36.62it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  46%|████▌     | 16/35 [00:00<00:00, 37.40it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  57%|█████▋    | 20/35 [00:00<00:00, 37.61it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  69%|██████▊   | 24/35 [00:00<00:00, 37.81it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  80%|████████  | 28/35 [00:00<00:00, 36.85it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  91%|█████████▏| 32/35 [00:00<00:00, 35.53it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 35/35 [00:00<00:00, 35.58it/s]
(EngineCore_DP0 pid=3796506)
Capturing CUDA graphs (decode, FULL):   0%|          | 0/19 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL):  21%|██        | 4/19 [00:00<00:00, 31.48it/s]
Capturing CUDA graphs (decode, FULL):  42%|████▏     | 8/19 [00:00<00:00, 34.48it/s]
Capturing CUDA graphs (decode, FULL):  63%|██████▎   | 12/19 [00:00<00:00, 35.42it/s]
Capturing CUDA graphs (decode, FULL):  84%|████████▍ | 16/19 [00:00<00:00, 35.44it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 35.42it/s]
(EngineCore_DP0 pid=3796506) INFO 02-17 03:52:51 [gpu_model_runner.py:5063] Graph capturing finished in 2 secs, took 0.28 GiB
(EngineCore_DP0 pid=3796506) INFO 02-17 03:52:51 [core.py:272] init engine (profile, create kv cache, warmup model) took 10.66 seconds
2026-02-17 03:52:53 | INFO     | veena3modal.services.tts_runtime | Initializing prompt builder...
2026-02-17 03:52:53 | INFO     | veena3modal.services.tts_runtime | Initializing BiCodec decoder...
/home/ubuntu/.local/lib/python3.10/site-packages/torch/nn/utils/weight_norm.py:144: FutureWarning: `torch.nn.utils.weight_norm` is deprecated in favor of `torch.nn.utils.parametrizations.weight_norm`.
  WeightNorm.apply(module, name, dim)
2026-02-17 03:52:58 | INFO     | veena3modal.services.tts_runtime | Initializing TTS pipeline...
2026-02-17 03:52:58 | INFO     | veena3modal.services.tts_runtime | Initializing streaming pipeline...
2026-02-17 03:52:58 | INFO     | veena3modal.services.tts_runtime | ✅ TTS runtime initialized in 30036ms
2026-02-17 03:52:58 | INFO     | veena3modal.services.tts_runtime |    Model version: LLM
2026-02-17 03:52:58 | INFO     | veena3modal.services.tts_runtime | Pre-computing global tokens for all speakers...


**Keeping process session active for polling**


Output:
Chunk ID: 4fa0e2
Wall time: 30.0012 seconds
Process running with session ID 61574
Original token count: 901
Output:
2026-02-17 03:52:59 | INFO     | veena3modal.services.tts_runtime |   Cached globals for lipakshi
2026-02-17 03:52:59 | INFO     | veena3modal.services.tts_runtime |   Cached globals for vardan
2026-02-17 03:53:00 | INFO     | veena3modal.services.tts_runtime |   Cached globals for reet
2026-02-17 03:53:00 | INFO     | veena3modal.services.tts_runtime |   Cached globals for Nandini
2026-02-17 03:53:00 | INFO     | veena3modal.services.tts_runtime |   Cached globals for krishna
2026-02-17 03:53:01 | INFO     | veena3modal.services.tts_runtime |   Cached globals for anika
2026-02-17 03:53:01 | INFO     | veena3modal.services.tts_runtime |   Cached globals for adarsh
2026-02-17 03:53:02 | INFO     | veena3modal.services.tts_runtime |   Cached globals for Nilay
2026-02-17 03:53:02 | INFO     | veena3modal.services.tts_runtime |   Cached globals for Aarvi
2026-02-17 03:53:02 | INFO     | veena3modal.services.tts_runtime |   Cached globals for Asha
2026-02-17 03:53:03 | INFO     | veena3modal.services.tts_runtime |   Cached globals for Bittu
2026-02-17 03:53:03 | INFO     | veena3modal.services.tts_runtime |   Cached globals for Mira
2026-02-17 03:53:03 | INFO     | veena3modal.services.tts_runtime | Speaker globals cached: 12/12 speakers in 4871ms
2026-02-17 03:53:03 | INFO     | veena3.local | Runtime initialized in 34.9s
2026-02-17 03:53:03 | INFO     | veena3.local |   Model version: LLM
2026-02-17 03:53:03 | INFO     | veena3.local |   Streaming: available
2026-02-17 03:53:03 | INFO     | veena3.local |   Super-resolution: disabled
2026-02-17 03:53:03 | INFO     | veena3.local |
2026-02-17 03:53:03 | INFO     | veena3.local | ============================================================
2026-02-17 03:53:03 | INFO     | veena3.local | Veena3 TTS Local Server starting
2026-02-17 03:53:03 | INFO     | veena3.local |   URL: http://0.0.0.0:8000
2026-02-17 03:53:03 | INFO     | veena3.local |   Health: http://0.0.0.0:8000/v1/tts/health
2026-02-17 03:53:03 | INFO     | veena3.local |   Generate: POST http://0.0.0.0:8000/v1/tts/generate
2026-02-17 03:53:03 | INFO     | veena3.local |   Metrics: http://0.0.0.0:8000/v1/tts/metrics
2026-02-17 03:53:03 | INFO     | veena3.local |   Docs: http://0.0.0.0:8000/docs
2026-02-17 03:53:03 | INFO     | veena3.local | ============================================================
✅ vLLM engine initialized successfully!

🎉 Spark TTS Model ready for inference!

🎵 Loading BiCodec audio tokenizer from /home/ubuntu/veenaModal/models/spark_tts_4speaker...
   Device: cuda
Missing tensor: mel_transformer.spectrogram.window
Missing tensor: mel_transformer.mel_scale.fb
✅ BiCodec decoder initialized (sample rate: 16000Hz)
🚀 SparkTTSPipeline initialized
🌊 Veena3SlidingWindowPipeline initialized (sliding window: 28 tokens)
{"timestamp": "2026-02-17T03:53:03.768335+00:00", "level": "INFO", "logger": "veena3modal.api.websocket_handler", "message": "WebSocket route /v1/tts/ws registered"}
2026-02-17 03:53:03 | INFO     | veena3modal.api.websocket_handler | WebSocket route /v1/tts/ws registered
2026-02-17 03:53:03 | INFO     | veena3modal.api.fastapi_app | WebSocket support enabled at /v1/tts/ws
{"timestamp": "2026-02-17T03:53:03.768487+00:00", "level": "INFO", "logger": "veena3modal.api.fastapi_app", "message": "WebSocket support enabled at /v1/tts/ws"}
INFO:     Started server process [3796383]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
INFO:     127.0.0.1:48582 - "GET /v1/tts/health HTTP/1.1" 200 OK


**Verifying response headers**


curl -sS -D /tmp/resp_headers.txt -o /tmp/resp_audio.wav -X POST http://localhost:8000/v1/tts/generate -H 'Content-Type: application/json' --data '{"text":"Hello, this is a quick benchmark test.","speaker":"lipakshi","stream":false,"chunking":false,"format":"wav"}' && sed -n '1,120p' /tmp/resp_headers.txt && ls -lh /tmp/resp_audio.wav


Output:
Chunk ID: 152556
Wall time: 10.0012 seconds
Process running with session ID 70959
Original token count: 0
Output:


**Polling long-running command**


Output:
Chunk ID: 4d538b
Wall time: 30.0012 seconds
Process running with session ID 70959
Original token count: 0
Output:


**Waiting for delayed output**


Output:
Chunk ID: a4c1fc
Wall time: 30.0013 seconds
Process running with session ID 70959
Original token count: 0
Output:


**Polling server session for progress**


Output:
Chunk ID: 5c6609
Wall time: 5.0014 seconds
Process running with session ID 61574
Original token count: 89
Output:
{"timestamp": "2026-02-17T03:53:45.194482+00:00", "level": "INFO", "logger": "veena3modal.tts", "message": "request_received", "request_id": "bcea1cb6-ae0b-4324-8c05-d684fa2113f1", "event": "request_received", "text_length": 38, "speaker": "lipakshi", "stream": false, "format": "wav"}
2026-02-17 03:53:45 | INFO     | veena3modal.tts | request_received


**Waiting for response delay**


The patched server is now running and receiving requests, but the first test request is still executing. I’m watching server-side logs to confirm completion and verify the emitted perf headers once it returns.


Output:
Chunk ID: 293ff6
Wall time: 30.0008 seconds
Process running with session ID 61574
Original token count: 0
Output:


**Investigating potential deadlock in generation**


Output:
Chunk ID: 466b71
Wall time: 30.0007 seconds
Process running with session ID 70959
Original token count: 0
Output:


Summary: auto


**Investigating potential server request hang**


Output:
Chunk ID: 873206
Wall time: 5.0019 seconds
Process running with session ID 61574
Original token count: 0
Output:


**Checking GPU utilization for blockage**


Output:
Chunk ID: e1c52c
Wall time: 0.0513 seconds
Process exited with code 0
Original token count: 3
Output:
0, 43695


**Assessing GPU inactivity原因**


curl -sS -m 5 http://localhost:8000/v1/tts/health


Output:
Chunk ID: 57ecf0
Wall time: 0.0513 seconds
Process exited with code 0
Original token count: 33
Output:
{"status":"healthy","model_loaded":true,"model_version":"LLM","uptime_seconds":223.21,"gpu_available":true,"app_version":"0.1.0"}


**Assessing request processing delays**


ps -ef | rg 'EngineCore|vllm|local_server|uvicorn' -n


Output:
Chunk ID: 90df08
Wall time: 0.0511 seconds
Process exited with code 0
Original token count: 124
Output:
303:ubuntu   3796383 3675569 25 03:52 ?        00:01:05 python3 -m veena3modal.local_server --model-path /home/ubuntu/veenaModal/models/spark_tts_4speaker --gpu-memory 0.5 --log-level INFO --port 8000
304:ubuntu   3796506 3796383 12 03:52 ?        00:00:31 VLLM::EngineCore
322:ubuntu   3798023 3675569  0 03:56 ?        00:00:00 /bin/bash -c ps -ef | rg 'EngineCore|vllm|local_server|uvicorn' -n
324:ubuntu   3798025 3798023  0 03:56 ?        00:00:00 rg EngineCore|vllm|local_server|uvicorn -n


**Planning to abort stuck request**


ps -ef | rg "curl -sS -D /tmp/resp_headers.txt" -n || true


Output:
Chunk ID: 8f98fa
Wall time: 0.0504 seconds
Process exited with code 0
Original token count: 237
Output:
309:ubuntu   3796925 3675569  0 03:53 ?        00:00:00 /bin/bash -c curl -sS -D /tmp/resp_headers.txt -o /tmp/resp_audio.wav -X POST http://localhost:8000/v1/tts/generate -H 'Content-Type: application/json' --data '{"text":"Hello, this is a quick benchmark test.","speaker":"lipakshi","stream":false,"chunking":false,"format":"wav"}' && sed -n '1,120p' /tmp/resp_headers.txt && ls -lh /tmp/resp_audio.wav
310:ubuntu   3796926 3796925  0 03:53 ?        00:00:00 curl -sS -D /tmp/resp_headers.txt -o /tmp/resp_audio.wav -X POST http://localhost:8000/v1/tts/generate -H Content-Type: application/json --data {"text":"Hello, this is a quick benchmark test.","speaker":"lipakshi","stream":false,"chunking":false,"format":"wav"}
323:ubuntu   3798061 3675569  0 03:56 ?        00:00:00 /bin/bash -c ps -ef | rg "curl -sS -D /tmp/resp_headers.txt" -n || true
325:ubuntu   3798063 3798061  0 03:56 ?        00:00:00 rg curl -sS -D /tmp/resp_headers.txt -n


**Terminating running process**


Output:
Chunk ID: dc4a4c
Wall time: 0.0512 seconds
Process exited with code 0
Original token count: 0
Output:


**Planning session termination**


Output:
Chunk ID: 11401a
Wall time: 0.0510 seconds
Process exited with code -1
Original token count: 0
Output:


**Planning server-side session polling**


Output:
Chunk ID: 00af57
Wall time: 5.0011 seconds
Process running with session ID 61574
Original token count: 17
Output:
INFO:     127.0.0.1:33906 - "GET /v1/tts/health HTTP/1.1" 200 OK


**Testing request with token limit**


python3 - <<'PY'
import requests, json, time
url='http://localhost:8000/v1/tts/generate'
payload={"text":"Hello test.","speaker":"lipakshi","stream":False,"chunking":False,"format":"wav","max_tokens":256}
t=time.time()
resp=requests.post(url,json=payload,timeout=120)
print('status',resp.status_code,'elapsed',time.time()-t,'len',len(resp.content))
keys=[k for k in resp.headers.keys() if k.lower().startswith('x-')]
for k in sorted(keys):
    if k.lower() in {'x-perf-details','x-request-id','x-ttfb-ms','x-generation-ms','x-llm-token-total','x-llm-batch-count','x-llm-batch-wall-ms','x-llm-batch-gpu-ms','x-llm-decode-wall-ms','x-llm-decode-gpu-ms','x-bicodec-decode-wall-ms','x-bicodec-decode-gpu-ms','x-request-inflight'}:
        print(k+':', resp.headers.get(k))
print('x-perf-details size',len(resp.headers.get('x-perf-details','')))
PY


Output:
Chunk ID: cee2e4
Wall time: 30.0014 seconds
Process running with session ID 4573
Original token count: 0
Output:


**Checking for server deadlock**


Output:
Chunk ID: da6ecc
Wall time: 5.0024 seconds
Process running with session ID 61574
Original token count: 89
Output:
{"timestamp": "2026-02-17T03:57:22.508312+00:00", "level": "INFO", "logger": "veena3modal.tts", "message": "request_received", "request_id": "e423aeaa-4e20-45b2-b1f9-3770b92fb6a6", "event": "request_received", "text_length": 11, "speaker": "lipakshi", "stream": false, "format": "wav"}
2026-02-17 03:57:22 | INFO     | veena3modal.tts | request_received


**Waiting for generation completion**


Output:
Chunk ID: 9d0e1e
Wall time: 30.0009 seconds
Process running with session ID 61574
Original token count: 0
Output:


**Assessing generation latency and GPU utilization**


Output:
Chunk ID: 63c2bc
Wall time: 0.0512 seconds
Process exited with code 0
Original token count: 4
Output:
0, 43695, 29


Summary: auto


**Investigating GPU inactivity and potential engine hang**


ps -o pid,pcpu,pmem,stat,etime,cmd -p 3796383,3796506


Output:
Chunk ID: 18fc7e
Wall time: 0.0514 seconds
Process exited with code 0
Original token count: 68
Output:
    PID %CPU %MEM STAT     ELAPSED CMD
3796383 16.9  1.3 Ssl        06:28 python3 -m veena3modal.local_server --model-path /home/ubuntu/veenaModal/models/spark_tts_4speaker --gpu-memory 0.5 --log-level INFO --port 8000
3796506  8.5  1.6 Sl         06:21 VLLM::EngineCore


**Extending server polling to capture logs**


Output:
Chunk ID: 26b0c2
Wall time: 30.0010 seconds
Process running with session ID 61574
Original token count: 0
Output:


**Diagnosing server hang cause**


ps -ef | rg "python3 - <<'PY'|requests.post\(url" -n || true


Output:
Chunk ID: ecce38
Wall time: 0.0514 seconds
Process exited with code 0
Original token count: 57
Output:
321:ubuntu   3799051 3675569  0 03:59 ?        00:00:00 /bin/bash -c ps -ef | rg "python3 - <<'PY'|requests.post\(url" -n || true
323:ubuntu   3799053 3799051  0 03:59 ?        00:00:00 rg python3 - <<'PY'|requests.post\(url -n


**Polling active session status**


Output:
Chunk ID: 8aa904
Wall time: 0.0503 seconds
Process exited with code 1
Original token count: 752
Output:
Traceback (most recent call last):
  File "/home/ubuntu/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 534, in _make_request
    response = conn.getresponse()
  File "/home/ubuntu/.local/lib/python3.10/site-packages/urllib3/connection.py", line 571, in getresponse
    httplib_response = super().getresponse()
  File "/usr/lib/python3.10/http/client.py", line 1375, in getresponse
    response.begin()
  File "/usr/lib/python3.10/http/client.py", line 318, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.10/http/client.py", line 279, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib/python3.10/socket.py", line 705, in readinto
    return self._sock.recv_into(b)
TimeoutError: timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ubuntu/.local/lib/python3.10/site-packages/requests/adapters.py", line 644, in send
    resp = conn.urlopen(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 841, in urlopen
    retries = retries.increment(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/urllib3/util/retry.py", line 490, in increment
    raise reraise(type(error), error, _stacktrace)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/urllib3/util/util.py", line 39, in reraise
    raise value
  File "/home/ubuntu/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    response = self._make_request(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 536, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 367, in _raise_timeout
    raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='localhost', port=8000): Read timed out. (read timeout=120)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 5, in <module>
  File "/home/ubuntu/.local/lib/python3.10/site-packages/requests/api.py", line 115, in post
    return request("post", url, data=data, json=json, **kwargs)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/requests/adapters.py", line 690, in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPConnectionPool(host='localhost', port=8000): Read timed out. (read timeout=120)


**Reducing max tokens to prevent timeouts**


kill -9 3796383 3796506 || true; sleep 1; ps -ef | rg 'local_server|EngineCore' -n || true


Output:
Chunk ID: 3f8652
Wall time: 0.9162 seconds
Process exited with code 0
Original token count: 62
Output:
314:ubuntu   3799166 3675569  0 04:00 ?        00:00:00 /bin/bash -c kill -9 3796383 3796506 || true; sleep 1; ps -ef | rg 'local_server|EngineCore' -n || true
316:ubuntu   3799170 3799166  0 04:00 ?        00:00:00 rg local_server|EngineCore -n


**Closing potentially killed session**


Output:
Chunk ID: 667825
Wall time: 0.0515 seconds
Process exited with code -1
Original token count: 0
Output:


**Restarting server with nohup**


nohup python3 -m veena3modal.local_server --model-path /home/ubuntu/veenaModal/models/spark_tts_4speaker --gpu-memory 0.5 --log-level INFO --port 8000 > /tmp/veena_local_server.log 2>&1 & echo $!


Output:
Chunk ID: 2f072a
Wall time: 0.0503 seconds
Process exited with code 0
Original token count: 2
Output:
3799203


**Adjusting readiness wait timing**


python3 - <<'PY'
import time,requests,sys
for i in range(180):
    try:
        r=requests.get('http://localhost:8000/v1/tts/health',timeout=2)
        if r.ok:
            print(r.text)
            sys.exit(0)
    except Exception:
        pass
    time.sleep(1)
print('not_ready')
sys.exit(1)
PY


Output:
Chunk ID: e5a068
Wall time: 30.0012 seconds
Process running with session ID 73237
Original token count: 0
Output:


Output:
Chunk ID: a3ffdc
Wall time: 30.0006 seconds
Process running with session ID 73237
Original token count: 0
Output:


**Waiting for script response**


I restarted cleanly to clear the long-running generation requests; once health comes back I’ll test with constrained `max_tokens` so we can get deterministic per-request timings instead of timeout-heavy runs.


Output:
Chunk ID: df85d0
Wall time: 30.0013 seconds
Process running with session ID 73237
Original token count: 0
Output:


**Checking process logs**


Output:
Chunk ID: 1c7926
Wall time: 0.0513 seconds
Process exited with code 0
Original token count: 454
Output:
2026-02-17 04:00:11 | INFO     | veena3.local | Auth bypass enabled (local mode). Use --auth to require API keys.
2026-02-17 04:00:11 | INFO     | veena3.local | Resolved model paths:
2026-02-17 04:00:11 | INFO     | veena3.local |   LLM (vLLM):    /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
2026-02-17 04:00:11 | INFO     | veena3.local |   BiCodec:       /home/ubuntu/veenaModal/models/spark_tts_4speaker
2026-02-17 04:00:11 | INFO     | veena3.local | ============================================================
2026-02-17 04:00:11 | INFO     | veena3.local | Initializing Veena3 TTS Runtime (local)
2026-02-17 04:00:11 | INFO     | veena3.local |   Model:       /home/ubuntu/veenaModal/models/spark_tts_4speaker
2026-02-17 04:00:11 | INFO     | veena3.local |   LLM path:    /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
2026-02-17 04:00:11 | INFO     | veena3.local |   BiCodec:     /home/ubuntu/veenaModal/models/spark_tts_4speaker
2026-02-17 04:00:11 | INFO     | veena3.local |   Device:      cuda
2026-02-17 04:00:11 | INFO     | veena3.local |   GPU Memory:  50% (total)
2026-02-17 04:00:11 | INFO     | veena3.local |   Engines:     1
2026-02-17 04:00:11 | INFO     | veena3.local |   SR:          disabled
2026-02-17 04:00:11 | INFO     | veena3.local | ============================================================
2026-02-17 04:00:11 | INFO     | veena3modal.services.tts_runtime | Initializing TTS runtime...
2026-02-17 04:00:11 | INFO     | veena3modal.services.tts_runtime |   Model path: /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
2026-02-17 04:00:11 | INFO     | veena3modal.services.tts_runtime |   BiCodec path: /home/ubuntu/veenaModal/models/spark_tts_4speaker
2026-02-17 04:00:11 | INFO     | veena3modal.services.tts_runtime |   SR enabled: False, path: None


Summary: auto


**Checking process termination cause**


Output:
Chunk ID: 4f2a41
Wall time: 0.0507 seconds
Process exited with code 1
Original token count: 14
Output:
UID          PID    PPID  C STIME TTY          TIME CMD


**Diagnosing server crash causes**


python3 -u -m veena3modal.local_server --model-path /home/ubuntu/veenaModal/models/spark_tts_4speaker --gpu-memory 0.5 --log-level INFO --port 8000