Output:
Chunk ID: 243bc2
Wall time: 0.0514 seconds
Process exited with code 0
Original token count: 724
Output:
        "timeline_bicodec_done_ms",
        "timeline_wav_done_ms",
        "timeline_request_done_ms",
        "timeline_to_first_batch_ms",
        "timeline_first_batch_to_llm_done_ms",
        "timeline_post_llm_ms",
        "timeline_parse_to_bicodec_done_ms",
        "timeline_bicodec_to_request_done_ms",
        "timeline_total_ms",
        "llm_prefill_wall_ms",
        "llm_prefill_gpu_ms",
        "llm_generation_wall_ms",
        "llm_time_in_queue_ms",
        "llm_scheduler_ms",
        "llm_model_forward_ms",
        "llm_model_execute_ms",
        "llm_first_token_ms",
        "llm_request_lifecycle_ms",
        "llm_prompt_token_total",
        "semantic_token_total",
        "global_token_total",
        "wav_pack_ms",
        "pipeline_total_ms",
        "token_validation_ms",
    )
    for key in extra_detail_keys:
        if key in metrics:
            detail_payload[key] = metrics[key]

    perf_headers["x-perf-details"] = json.dumps(
        detail_payload,
        separators=(",", ":"),
        ensure_ascii=True,
    )
    return perf_headers

def get_model_version() -> str:
    """Return the current model version, or 'not_loaded' if not initialized."""
    return _MODEL_VERSION or "not_loaded"

def set_model_version(version: str) -> None:
    """Set the model version after loading."""
    global _MODEL_VERSION
    _MODEL_VERSION = version

def create_app():
    """
    Factory function to create FastAPI app.

    Import inside factory so local tooling can import this module without FastAPI installed.
    """
    from fastapi import FastAPI, HTTPException, Request
    from fastapi.responses import JSONResponse, Response

    global _startup_time
    _startup_time = time.time()

    app = FastAPI(
        title="Veena3 TTS (Modal)",
        version=_APP_VERSION,
        description="High-quality multilingual Text-to-Speech API with true streaming support.",
    )

    @app.get("/v1/tts/health")
    async def tts_health():
        """
        Health check endpoint.

        Returns:
            - status: "healthy" | "degraded" | "unhealthy"
            - model_loaded: whether TTS model is initialized
            - model_version: version string of loaded model
            - uptime_seconds: time since app startup
            - gpu_available: whether GPU is detected (best-effort check)
        """
        # Check model status via runtime
        from veena3modal.services import tts_runtime
        model_loaded = tts_runtime.is_initialized()

        # Use runtime's model version if available
        runtime = tts_runtime.get_runtime()
        model_version = runtime.model_version if runtime else get_model_version()

        # Check GPU availability (best-effort, non-blocking)
        gpu_available = False
        try:
            import torch
            gpu_available = torch.cuda.is_available()


# Collaboration Mode: Default

You are now in Default mode. Any previous instructions for other modes (e.g. Plan mode) are no longer active.

Your active mode changes only when new developer instructions with a different `<collaboration_mode>...</collaboration_mode>` change it; user requests or tool descriptions do not change mode by themselves. Known mode names are Default and Plan.

## request_user_input availability

The `request_user_input` tool is unavailable in Default mode. If you call it while in Default mode, it will return an error.

If a decision is necessary and cannot be discovered from local context, ask the user directly. However, in Default mode you should strongly prefer executing the user's request rather than stopping to ask questions.

Summary: auto

## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) When `SKILL.md` references relative paths (e.g., `scripts/foo.py`), resolve them relative to the skill directory listed above first, and only consider other paths if needed.
  3) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  4) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  5) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.


**Planning explicit headers addition**


Input:
*** Begin Patch
*** Update File: /home/ubuntu/veenaModal/veena3modal/api/fastapi_app.py
@@
         "x-llm-time-per-token-ms": "llm_time_per_token_ms",
         "x-llm-time-per-batch-wall-ms": "llm_time_per_batch_wall_ms",
         "x-llm-time-per-batch-gpu-ms": "llm_time_per_batch_gpu_ms",
+        "x-llm-time-in-queue-ms": "llm_time_in_queue_ms",
+        "x-llm-scheduler-ms": "llm_scheduler_ms",
+        "x-llm-first-token-ms": "llm_first_token_ms",
+        "x-llm-lifecycle-ms": "llm_request_lifecycle_ms",
         "x-llm-text-chunked": "text_chunked",
         "x-llm-chunks-processed": "chunks_processed",
         "x-bicodec-decode-wall-ms": "bicodec_decode_wall_ms",
         "x-bicodec-decode-gpu-ms": "bicodec_decode_gpu_ms",
         "x-bicodec-decode-cpu-ms": "bicodec_decode_cpu_ms",
@@
         "llm_time_in_queue_ms",
         "llm_scheduler_ms",
         "llm_model_forward_ms",
         "llm_model_execute_ms",
         "llm_first_token_ms",
         "llm_request_lifecycle_ms",
+        "llm_queued_to_scheduled_ms",
+        "llm_scheduled_to_first_token_ms",
+        "llm_first_to_last_token_ms",
+        "llm_queued_to_last_token_ms",
         "llm_prompt_token_total",
         "semantic_token_total",
         "global_token_total",
*** End Patch


Output: {"output":"Success. Updated the following files:\nM /home/ubuntu/veenaModal/veena3modal/api/fastapi_app.py\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


Summary: auto


**Adding GPU timing flag**


Input:
*** Begin Patch
*** Update File: /home/ubuntu/veenaModal/scripts/stress_test_runtime_detailed.py
@@
             "disable_engine_stats_logs": args.disable_engine_stats_logs,
             "enforce_eager": args.enforce_eager,
+            "enable_gpu_decode_timing": args.enable_gpu_decode_timing,
             "timestamp": int(time.time()),
         },
         "levels": [],
@@
     parser.add_argument("--disable-prefix-caching", action="store_true", default=False)
     parser.add_argument("--disable-engine-stats-logs", action="store_true", default=False)
     parser.add_argument("--enforce-eager", action="store_true", default=False)
+    parser.add_argument(
+        "--enable-gpu-decode-timing",
+        action="store_true",
+        default=False,
+        help="Enable precise BiCodec GPU timing (adds synchronize overhead)",
+    )
     parser.add_argument("--model-path", type=str, default="")
     parser.add_argument("--output", default="stress_runtime_detailed.json")
     args = parser.parse_args()

     from veena3modal.local_server import DEFAULT_LOCAL_MODEL_DIR
     model_path = args.model_path or DEFAULT_LOCAL_MODEL_DIR
+    if args.enable_gpu_decode_timing:
+        os.environ["VEENA3_PERF_GPU_TIMING"] = "1"
+    else:
+        os.environ.setdefault("VEENA3_PERF_GPU_TIMING", "0")
     enable_chunked_prefill = args.enable_chunked_prefill
*** End Patch


Output: {"output":"Success. Updated the following files:\nM /home/ubuntu/veenaModal/scripts/stress_test_runtime_detailed.py\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


**Planning GPU timing flag**


cd /home/ubuntu/veenaModal && sed -n '180,320p' veena3modal/local_server.py


cd /home/ubuntu/veenaModal && sed -n '320,560p' veena3modal/local_server.py


Output:
Chunk ID: 49ee3c
Wall time: 0.0507 seconds
Process exited with code 0
Original token count: 1375
Output:
        llm_path = model_path

    # BiCodec path is always the root (it contains BiCodec/ subdir)
    bicodec_path = model_path

    logger.info(f"Resolved model paths:")
    logger.info(f"  LLM (vLLM):    {llm_path}")
    logger.info(f"  BiCodec:       {bicodec_path}")

    return llm_path, bicodec_path

def initialize_local_runtime(
    model_path: str,
    device: str = "cuda",
    gpu_memory_utilization: float = 0.85,
    enable_sr: bool = False,
    sr_checkpoint_dir: str | None = None,
    hf_token: str | None = None,
    num_engines: int = 1,
    max_num_batched_tokens: int | None = None,
    max_num_seqs: int | None = None,
    enable_chunked_prefill: bool | None = None,
    enable_prefix_caching: bool | None = None,
    disable_log_stats: bool | None = None,
    enforce_eager: bool | None = None,
    precompute_speaker_globals: bool = False,
):
    """
    Initialize TTS runtime for local serving.

    This replaces Modal's @modal.enter lifecycle hook.
    Loads: vLLM engine, BiCodec decoder, prompt builder, pipelines.
    """
    from veena3modal.services.tts_runtime import initialize_runtime

    # Resolve LLM vs BiCodec paths from model directory
    llm_path, bicodec_path = resolve_model_paths(model_path)

    start = time.time()
    logger.info("=" * 60)
    logger.info("Initializing Veena3 TTS Runtime (local)")
    logger.info(f"  Model:       {model_path}")
    logger.info(f"  LLM path:    {llm_path}")
    logger.info(f"  BiCodec:     {bicodec_path}")
    logger.info(f"  Device:      {device}")
    logger.info(f"  GPU Memory:  {gpu_memory_utilization:.0%} (total)")
    logger.info(f"  Engines:     {num_engines}")
    if max_num_batched_tokens is not None:
        logger.info(f"  Max batched tokens: {max_num_batched_tokens}")
    if max_num_seqs is not None:
        logger.info(f"  Max seqs:    {max_num_seqs}")
    if enable_chunked_prefill is not None:
        logger.info(f"  Chunked prefill: {'enabled' if enable_chunked_prefill else 'disabled'}")
    if enable_prefix_caching is not None:
        logger.info(f"  Prefix caching: {'enabled' if enable_prefix_caching else 'disabled'}")
    if disable_log_stats is not None:
        logger.info(f"  Engine stats logs: {'disabled' if disable_log_stats else 'enabled'}")
    if enforce_eager is not None:
        logger.info(f"  Enforce eager: {'enabled' if enforce_eager else 'disabled'}")
    logger.info(f"  Precompute globals: {'enabled' if precompute_speaker_globals else 'disabled'}")
    logger.info(f"  SR:          {'enabled' if enable_sr else 'disabled'}")
    logger.info("=" * 60)

    runtime = initialize_runtime(
        model_path=llm_path,
        bicodec_path=bicodec_path,
        sr_checkpoint_dir=sr_checkpoint_dir if enable_sr else None,
        device=device,
        hf_token=hf_token,
        gpu_memory_utilization=gpu_memory_utilization,
        enable_sr=enable_sr,
        num_engines=num_engines,
        max_num_batched_tokens=max_num_batched_tokens,
        max_num_seqs=max_num_seqs,
        enable_chunked_prefill=enable_chunked_prefill,
        enable_prefix_caching=enable_prefix_caching,
        disable_log_stats=disable_log_stats,
        enforce_eager=enforce_eager,
        precompute_speaker_globals=precompute_speaker_globals,
    )

    elapsed = time.time() - start
    logger.info(f"Runtime initialized in {elapsed:.1f}s")
    logger.info(f"  Model version: {runtime.model_version}")
    logger.info(f"  Streaming: {'available' if runtime.streaming_pipeline else 'unavailable'}")
    logger.info(f"  Super-resolution: {'loaded' if runtime.sr_service and runtime.sr_service.is_loaded else 'disabled'}")

    return runtime

def create_local_app():
    """
    Create the FastAPI app for local serving.

    Same app factory as Modal deployment, but called from local context.
    """
    from veena3modal.api.fastapi_app import create_app
    return create_app()

def parse_args():
    """Parse CLI arguments for local server configuration."""
    parser = argparse.ArgumentParser(
        description="Veena3 TTS Local Server - GPU-accelerated text-to-speech",
        formatter_class=argparse.RawDescriptionHelpFormatter,
        epilog="""
Examples:
  python -m veena3modal.local_server                          # Auto-download model, serve on :8000
  python -m veena3modal.local_server --port 8080              # Custom port
  python -m veena3modal.local_server --model-path ./my_model  # Use local model
  python -m veena3modal.local_server --gpu-memory 0.5         # Limit GPU memory
  python -m veena3modal.local_server --workers 2              # Multiple workers (careful: each loads model)
        """,
    )

    # Model configuration
    model_group = parser.add_argument_group("Model")
    model_group.add_argument(
        "--model-path",
        type=str,
        default=os.environ.get("MODEL_PATH", DEFAULT_LOCAL_MODEL_DIR),
        help=f"Path to Spark TTS model (default: {DEFAULT_LOCAL_MODEL_DIR})",
    )
    model_group.add_argument(
        "--hf-repo",
        type=str,
        default=PRIVATE_HF_REPO,
        help=f"HuggingFace repo to download from (default: {PRIVATE_HF_REPO})",
    )
    model_group.add_argument(
        "--hf-token",
        type=str,
        default=os.environ.get("HF_TOKEN"),
        help="HuggingFace token (default: from HF_TOKEN env var)",
    )
    model_group.add_argument(
        "--skip-download",
        action="store_true",
        help="Don't auto-download model (fail if not found)",
    )


Output:
Chunk ID: fa56fc
Wall time: 0.0508 seconds
Process exited with code 0
Original token count: 1842
Output:
    )

    # GPU configuration
    gpu_group = parser.add_argument_group("GPU")
    gpu_group.add_argument(
        "--device",
        type=str,
        default="cuda",
        choices=["cuda", "cpu"],
        help="Device for inference (default: cuda)",
    )
    gpu_group.add_argument(
        "--gpu-memory",
        type=float,
        default=float(os.environ.get("GPU_MEMORY_UTILIZATION", "0.25")),
        help="vLLM GPU memory utilization fraction (default: 0.25)",
    )
    gpu_group.add_argument(
        "--num-engines",
        type=int,
        default=int(os.environ.get("NUM_ENGINES", "1")),
        help="Number of vLLM engine instances (default: 1, use 2-3 for multi-engine mode)",
    )
    gpu_group.add_argument(
        "--max-num-batched-tokens",
        type=int,
        default=None,
        help="Override vLLM max_num_batched_tokens scheduler limit (default: model config)",
    )
    gpu_group.add_argument(
        "--max-num-seqs",
        type=int,
        default=None,
        help="Override vLLM max_num_seqs scheduler limit (default: vLLM auto)",
    )
    gpu_group.add_argument(
        "--enable-chunked-prefill",
        action="store_true",
        default=None,
        help="Force-enable vLLM chunked prefill",
    )
    gpu_group.add_argument(
        "--disable-chunked-prefill",
        action="store_true",
        default=False,
        help="Force-disable vLLM chunked prefill",
    )
    gpu_group.add_argument(
        "--disable-prefix-caching",
        action="store_true",
        default=False,
        help="Disable vLLM prefix caching",
    )
    gpu_group.add_argument(
        "--disable-engine-stats-logs",
        action="store_true",
        default=False,
        help="Disable vLLM periodic stats logging",
    )
    gpu_group.add_argument(
        "--enforce-eager",
        action="store_true",
        default=False,
        help="Enable eager mode (disables CUDA graphs)",
    )
    gpu_group.add_argument(
        "--precompute-globals",
        action="store_true",
        default=False,
        help="Precompute speaker global tokens at startup (streaming-only optimization, experimental)",
    )

    # Super-resolution
    sr_group = parser.add_argument_group("Super-Resolution")
    sr_group.add_argument(
        "--enable-sr",
        action="store_true",
        default=False,
        help="Enable AP-BWE super-resolution (16kHz -> 48kHz)",
    )
    sr_group.add_argument(
        "--sr-path",
        type=str,
        default=os.environ.get("AP_BWE_CHECKPOINT_DIR"),
        help="Path to AP-BWE checkpoint directory",
    )

    # Server configuration
    server_group = parser.add_argument_group("Server")
    server_group.add_argument(
        "--host",
        type=str,
        default="0.0.0.0",
        help="Bind address (default: 0.0.0.0)",
    )
    server_group.add_argument(
        "--port",
        type=int,
        default=int(os.environ.get("PORT", "8000")),
        help="Port to serve on (default: 8000)",
    )
    server_group.add_argument(
        "--workers",
        type=int,
        default=1,
        help="Number of uvicorn workers (default: 1, each loads full model)",
    )
    server_group.add_argument(
        "--reload",
        action="store_true",
        default=False,
        help="Enable auto-reload for development (incompatible with --workers > 1)",
    )

    # Auth
    auth_group = parser.add_argument_group("Auth")
    auth_group.add_argument(
        "--auth",
        action="store_true",
        default=False,
        help="Enable API key authentication (default: disabled for local)",
    )

    # Logging
    log_group = parser.add_argument_group("Logging")
    log_group.add_argument(
        "--log-level",
        type=str,
        default="INFO",
        choices=["DEBUG", "INFO", "WARNING", "ERROR"],
        help="Log level (default: INFO)",
    )

    return parser.parse_args()

def main():
    """
    Main entry point for local TTS server.

    Lifecycle:
    1. Parse args + configure logging
    2. Download model if needed
    3. Initialize TTS runtime (vLLM engine, decoders, pipelines)
    4. Create FastAPI app
    5. Run uvicorn
    """
    args = parse_args()

    # Configure logging
    logging.basicConfig(
        level=getattr(logging, args.log_level),
        format="%(asctime)s | %(levelname)-8s | %(name)s | %(message)s",
        datefmt="%Y-%m-%d %H:%M:%S",
    )

    # Auth bypass for local development (unless --auth flag is set)
    if not args.auth:
        os.environ["AUTH_BYPASS_MODE"] = "true"
        logger.info("Auth bypass enabled (local mode). Use --auth to require API keys.")
    else:
        os.environ.setdefault("AUTH_BYPASS_MODE", "false")

    # Resolve model path
    # Spark TTS root has config.yaml; LLM subdir has config.json; fine-tuned may have config.json at root
    model_path = args.model_path
    model_exists = any(
        os.path.exists(os.path.join(model_path, marker))
        for marker in ("config.json", "config.yaml", "LLM/config.json")
    )
    if not model_exists:
        if args.skip_download:
            logger.error(f"Model not found at {model_path} and --skip-download is set")
            sys.exit(1)
        model_path = download_model(
            hf_repo=args.hf_repo,
            local_dir=model_path,
            hf_token=args.hf_token,
        )

    # Initialize TTS runtime (this is the expensive step - loads model onto GPU)
    enable_chunked_prefill = args.enable_chunked_prefill
    if args.disable_chunked_prefill:
        enable_chunked_prefill = False
    initialize_local_runtime(
        model_path=model_path,
        device=args.device,
        gpu_memory_utilization=args.gpu_memory,
        enable_sr=args.enable_sr,
        sr_checkpoint_dir=args.sr_path,
        hf_token=args.hf_token,
        num_engines=args.num_engines,
        max_num_batched_tokens=args.max_num_batched_tokens,
        max_num_seqs=args.max_num_seqs,
        enable_chunked_prefill=enable_chunked_prefill,
        enable_prefix_caching=(False if args.disable_prefix_caching else None),
        disable_log_stats=(True if args.disable_engine_stats_logs else None),
        enforce_eager=(True if args.enforce_eager else None),
        precompute_speaker_globals=args.precompute_globals,
    )

    # Run uvicorn
    # NOTE: We use the factory string so uvicorn can find the app.
    # With --reload, uvicorn reimports, but the runtime singleton persists in memory.
    import uvicorn

    logger.info("")
    logger.info("=" * 60)
    logger.info(f"Veena3 TTS Local Server starting")
    logger.info(f"  URL: http://{args.host}:{args.port}")
    logger.info(f"  Health: http://{args.host}:{args.port}/v1/tts/health")
    logger.info(f"  Generate: POST http://{args.host}:{args.port}/v1/tts/generate")
    logger.info(f"  Metrics: http://{args.host}:{args.port}/v1/tts/metrics")
    logger.info(f"  Docs: http://{args.host}:{args.port}/docs")
    logger.info("=" * 60)

    uvicorn.run(
        "veena3modal.local_server:create_local_app",
        factory=True,
        host=args.host,
        port=args.port,
        workers=args.workers,
        reload=args.reload,
        log_level=args.log_level.lower(),
        # Performance: keep-alive for connection reuse under load
        timeout_keep_alive=30,
    )

if __name__ == "__main__":
    main()


**Considering optional instrumentation flag**


cd /home/ubuntu/veenaModal && python3 -m py_compile veena3modal/core/pipeline.py veena3modal/core/bicodec_decoder.py scripts/stress_test_runtime_detailed.py veena3modal/api/fastapi_app.py


cd /home/ubuntu/veenaModal && git status --short


Output:
Chunk ID: 9f8134
Wall time: 0.0516 seconds
Process exited with code 0
Original token count: 0
Output:


Output:
Chunk ID: 0eb254
Wall time: 0.0511 seconds
Process exited with code 0
Original token count: 172
Output:
 D CLEANUP_PLAN.md
 D PROGRESS.md
 D migration.md
 M veena3modal/api/fastapi_app.py
 M veena3modal/core/bicodec_decoder.py
 M veena3modal/core/constants.py
 M veena3modal/core/pipeline.py
 M veena3modal/core/streaming_pipeline.py
 M veena3modal/services/tts_runtime.py
?? scripts/profile_pipeline.py
?? scripts/setup_local.sh
?? scripts/stress_test_local.py
?? scripts/stress_test_optimized_local.py
?? scripts/stress_test_production.py
?? scripts/stress_test_runtime_detailed.py
?? stress_results_opt_local.json
?? veena3modal/__main__.py
?? veena3modal/core/bicodec_batch_decoder.py
?? veena3modal/core/dual_engine.py
?? veena3modal/core/multi_engine.py
?? veena3modal/local_server.py


**Planning benchmark runs and cleanup**


I’ve finished the code changes and compile checks passed. Next I’m running an A/B stress benchmark on the updated runtime to quantify impact on throughput, queueing delay, and latency at concurrency 96.


cd /home/ubuntu/veenaModal && python3 scripts/stress_test_runtime_detailed.py --levels 96 --text short --max-tokens 128 --timeout 180 --requests-multiplier 4 --min-requests 8 --gpu-memory 0.25 --num-engines 1 --output /tmp/runtime_c96_after_patch_stats_on.json


Output:
Chunk ID: b1c2ef
Wall time: 0.0513 seconds
Process exited with code 0
Original token count: 0
Output:


Output:
Chunk ID: ae1576
Wall time: 1.0007 seconds
Process running with session ID 48913
Original token count: 0
Output:


**Initiating frequent polling**


Output:
Chunk ID: 071dc0
Wall time: 5.0010 seconds
Process running with session ID 48913
Original token count: 1362
Output:
[2026-02-18 00:18:10] INFO tts_runtime.py:183: Loading SparkTTS model with vLLM...
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
🚀 Initializing Spark TTS Model
📁 Model path: /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
🗣  Speakers: 12 predefined (lipakshi, vardan, reet, Nandini, krishna, anika, adarsh, Nilay, Aarvi, Asha, Bittu, Mira)
🎭 Emotions: 10 tags in [bracket] format

📝 Loading tokenizer from model...
✅ Tokenizer loaded: 165168 tokens

🔍 Validating 12 speakers for Spark TTS...
  lipakshi → <|speaker_0|>: 7 tokens
  vardan → <|speaker_1|>: 7 tokens
  reet → <|speaker_2|>: 7 tokens
  Nandini → <|speaker_3|>: 7 tokens
  krishna → <|speaker_4|>: 7 tokens
  anika → <|speaker_5|>: 7 tokens
  adarsh → <|speaker_6|>: 7 tokens
  Nilay → <|speaker_7|>: 7 tokens
  Aarvi → <|speaker_8|>: 7 tokens
  Asha → <|speaker_9|>: 7 tokens
  Bittu → <|speaker_10|>: 8 tokens
  Mira → <|speaker_11|>: 8 tokens
✅ All 12 speakers validated with mapping

🔍 Validating 10 emotion tags for Spark TTS...
✅ All 10 emotion tags validated

📋 Sample emotion tags:
  [angry]: 1 tokens
  [curious]: 1 tokens
  [excited]: 1 tokens
  [giggle]: 1 tokens
  [laughs harder]: 1 tokens
  ...

🔧 Initializing vLLM engine with optimizations...
  - dtype: bfloat16
  - max_model_len: 4096
  - gpu_memory_utilization: 0.25
  - enable_prefix_caching: True
  - enforce_eager (disable CUDA graphs): False
INFO 02-18 00:18:11 [model.py:541] Resolved architecture: Qwen2ForCausalLM
INFO 02-18 00:18:11 [model.py:1561] Using max model len 4096
INFO 02-18 00:18:11 [scheduler.py:226] Chunked prefill is enabled with max_num_batched_tokens=4096.
INFO 02-18 00:18:11 [vllm.py:624] Asynchronous scheduling is enabled.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(EngineCore_DP0 pid=4095414) INFO 02-18 00:18:12 [core.py:96] Initializing a V1 LLM engine (v0.15.1) with config: model='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', speculative_config=None, tokenizer='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [4096], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 256, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': True}, 'local_cache_dir': None, 'static_all_moe_layers': []}
(EngineCore_DP0 pid=4095414) INFO 02-18 00:18:14 [parallel_state.py:1212] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://216.81.248.184:59145 backend=nccl
(EngineCore_DP0 pid=4095414) INFO 02-18 00:18:14 [parallel_state.py:1423] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A


**Updating run status commentary**


The patched benchmark is running now; model startup is complete and I’m waiting for the full 96-concurrency run output to compare queue and decode behavior against the previous baseline.


Output:
Chunk ID: bf6c59
Wall time: 5.0017 seconds
Process running with session ID 48913
Original token count: 426
Output:
(EngineCore_DP0 pid=4095414) INFO 02-18 00:18:15 [gpu_model_runner.py:4033] Starting to load model /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM...
(EngineCore_DP0 pid=4095414) INFO 02-18 00:18:16 [cuda.py:364] Using FLASH_ATTN attention backend out of potential backends: ('FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION')
(EngineCore_DP0 pid=4095414) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
(EngineCore_DP0 pid=4095414) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
(EngineCore_DP0 pid=4095414)
Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
(EngineCore_DP0 pid=4095414)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.09it/s]
(EngineCore_DP0 pid=4095414)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.09it/s]
(EngineCore_DP0 pid=4095414)
(EngineCore_DP0 pid=4095414) INFO 02-18 00:18:17 [default_loader.py:291] Loading weights took 0.98 seconds
(EngineCore_DP0 pid=4095414) INFO 02-18 00:18:17 [gpu_model_runner.py:4130] Model loading took 0.95 GiB memory and 1.503054 seconds
(EngineCore_DP0 pid=4095414) INFO 02-18 00:18:22 [backends.py:812] Using cache directory: /home/ubuntu/.cache/vllm/torch_compile_cache/f3ff13511c/rank_0_0/backbone for vLLM's torch.compile
(EngineCore_DP0 pid=4095414) INFO 02-18 00:18:22 [backends.py:872] Dynamo bytecode transform time: 4.21 s


I’m still in the same run and watching for the post-start benchmark section; once it completes I’ll immediately run the no-stats high-throughput variant for a direct A/B.


Output:
Chunk ID: 568ed2
Wall time: 5.0009 seconds
Process running with session ID 48913
Original token count: 762
Output:
(EngineCore_DP0 pid=4095414) INFO 02-18 00:18:25 [backends.py:267] Directly load the compiled graph(s) for compile range (1, 4096) from the cache, took 0.621 s
(EngineCore_DP0 pid=4095414) INFO 02-18 00:18:25 [monitor.py:34] torch.compile takes 4.83 s in total
(EngineCore_DP0 pid=4095414) INFO 02-18 00:18:26 [gpu_worker.py:356] Available KV cache memory: 18.06 GiB
(EngineCore_DP0 pid=4095414) INFO 02-18 00:18:26 [kv_cache_utils.py:1307] GPU KV cache size: 1,577,712 tokens
(EngineCore_DP0 pid=4095414) INFO 02-18 00:18:26 [kv_cache_utils.py:1312] Maximum concurrency for 4,096 tokens per request: 385.18x
(EngineCore_DP0 pid=4095414)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   0%|          | 0/35 [00:00<?, ?it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  11%|█▏        | 4/35 [00:00<00:00, 34.06it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  23%|██▎       | 8/35 [00:00<00:00, 36.33it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  34%|███▍      | 12/35 [00:00<00:00, 37.38it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  46%|████▌     | 16/35 [00:00<00:00, 37.95it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  57%|█████▋    | 20/35 [00:00<00:00, 38.04it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  69%|██████▊   | 24/35 [00:00<00:00, 38.18it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  80%|████████  | 28/35 [00:00<00:00, 38.16it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  91%|█████████▏| 32/35 [00:00<00:00, 36.41it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 35/35 [00:00<00:00, 36.33it/s]
(EngineCore_DP0 pid=4095414)
Capturing CUDA graphs (decode, FULL):   0%|          | 0/19 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL):  21%|██        | 4/19 [00:00<00:00, 32.26it/s]
Capturing CUDA graphs (decode, FULL):  42%|████▏     | 8/19 [00:00<00:00, 35.28it/s]
Capturing CUDA graphs (decode, FULL):  63%|██████▎   | 12/19 [00:00<00:00, 36.24it/s]
Capturing CUDA graphs (decode, FULL):  84%|████████▍ | 16/19 [00:00<00:00, 36.72it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 36.38it/s]
(EngineCore_DP0 pid=4095414) INFO 02-18 00:18:28 [gpu_model_runner.py:5063] Graph capturing finished in 2 secs, took 0.28 GiB
(EngineCore_DP0 pid=4095414) INFO 02-18 00:18:28 [core.py:272] init engine (profile, create kv cache, warmup model) took 10.53 seconds
[2026-02-18 00:18:29] INFO tts_runtime.py:192: Initializing prompt builder...
[2026-02-18 00:18:29] INFO tts_runtime.py:199: Initializing BiCodec decoder...
/home/ubuntu/.local/lib/python3.10/site-packages/torch/nn/utils/weight_norm.py:144: FutureWarning: `torch.nn.utils.weight_norm` is deprecated in favor of `torch.nn.utils.parametrizations.weight_norm`.
  WeightNorm.apply(module, name, dim)


Output:
Chunk ID: 83d692
Wall time: 5.0008 seconds
Process running with session ID 48913
Original token count: 127
Output:
[2026-02-18 00:18:38] INFO tts_runtime.py:206: Initializing TTS pipeline...
[2026-02-18 00:18:38] INFO tts_runtime.py:217: Initializing streaming pipeline...
[2026-02-18 00:18:38] INFO long_text_processor.py:112: LongTextProcessor initialized
[2026-02-18 00:18:38] INFO tts_runtime.py:274: ✅ TTS runtime initialized in 32771ms
[2026-02-18 00:18:38] INFO tts_runtime.py:275:    Model version: LLM
[2026-02-18 00:18:38] INFO tts_runtime.py:286: Speaker globals cache startup precompute skipped (disabled)


**Scheduling next poll**


Output:
Chunk ID: f01437
Wall time: 5.0014 seconds
Process running with session ID 48913
Original token count: 608
Output:
✅ vLLM engine initialized successfully!

🎉 Spark TTS Model ready for inference!

🎵 Loading BiCodec audio tokenizer from /home/ubuntu/veenaModal/models/spark_tts_4speaker...
   Device: cuda
Missing tensor: mel_transformer.spectrogram.window
Missing tensor: mel_transformer.mel_scale.fb
✅ BiCodec decoder initialized (sample rate: 16000Hz)
🚀 SparkTTSPipeline initialized
🌊 Veena3SlidingWindowPipeline initialized (sliding window: 28 tokens)

=== concurrency=96 requests=384 ===
success=384/384 (100%)
wall=10.30s throughput=37.30 req/s
latency_ms avg=2420 p50=2549 p95=2888 max=3446
generation_ms: avg=2411.90 p50=2548.00 p95=2850.00 max=3401.00
llm_time_in_queue_ms: avg=0.02 p50=0.01 p95=0.10 max=0.15
llm_scheduler_ms: avg=18.09 p50=16.27 p95=27.49 max=27.49
llm_first_token_ms: avg=27.76 p50=26.18 p95=40.52 max=50.51
llm_request_lifecycle_ms: avg=885.61 p50=887.83 p95=927.67 max=930.70
llm_queued_to_scheduled_ms: avg=0.02 p50=0.01 p95=0.10 max=0.15
llm_scheduled_to_first_token_ms: avg=18.09 p50=16.27 p95=27.49 max=27.49
llm_first_to_last_token_ms: avg=867.50 p50=872.22 p95=901.10 max=907.76
llm_queued_to_last_token_ms: avg=885.61 p50=887.83 p95=927.67 max=930.70
timeline_llm_first_batch_ms: avg=31.08 p50=29.71 p95=43.02 max=57.14
timeline_llm_done_ms: avg=902.17 p50=898.92 p95=959.32 max=964.79
timeline_parse_done_ms: avg=902.29 p50=899.06 p95=959.38 max=964.86
timeline_bicodec_done_ms: avg=2412.30 p50=2548.83 p95=2850.02 max=3401.20
timeline_total_ms: avg=2412.34 p50=2548.86 p95=2850.05 max=3401.23
llm_batch_wall_ms_total: avg=901.30 p50=897.08 p95=959.16 max=964.60
llm_decode_wall_ms_total: avg=871.06 p50=870.42 p95=920.24 max=923.01
llm_parse_ms: avg=0.10 p50=0.11 p95=0.16 max=0.24
llm_time_per_token_ms: avg=7.04 p50=7.01 p95=7.49 max=7.54
bicodec_decode_wall_ms: avg=1509.94 p50=1651.35 p95=1909.58 max=2454.94
bicodec_decode_gpu_ms: avg=0.00 p50=0.00 p95=0.00 max=0.00
latency_over_timeline_ms: avg=7.47 p50=1.42 p95=38.22 max=48.52
latency_over_generation_ms: avg=7.92 p50=1.84 p95=38.26 max=49.20
timeline_share: llm=37.4% bicodec=62.6% other=0.0%
timeline_sample_ms: first_batch=31.2 llm_done=901.4 parse_done=901.5 bicodec_done=2551.0 total=2551.0
gpu: util_avg=81% util_max=98% mem_avg=24772MB mem_max=24937MB

results saved: /tmp/runtime_c96_after_patch_stats_on.json
ERROR 02-18 00:18:50 [core_client.py:605] Engine core proc EngineCore_DP0 died unexpectedly, shutting down client.


**Reverting async decode due to threadpool contention**


The first patched run exposed a critical regression: async offloaded BiCodec decode created a decode queue and inflated `bicodec_decode_wall_ms` (~1.5s), which cut throughput. I’m immediately switching decode back to synchronous by default (with async kept as an opt-in flag), then rerunning the same benchmark.


Input:
*** Begin Patch
*** Update File: /home/ubuntu/veenaModal/veena3modal/core/pipeline.py
@@
 _PERF_GPU_TIMING_ENABLED = os.environ.get("VEENA3_PERF_GPU_TIMING", "").strip().lower() in {
     "1",
     "true",
     "yes",
     "on",
 }
+_ASYNC_BICODEC_DECODE_ENABLED = os.environ.get("VEENA3_ASYNC_BICODEC_DECODE", "").strip().lower() in {
+    "1",
+    "true",
+    "yes",
+    "on",
+}
@@
         else:
-            # Keep the event loop responsive under load by offloading decode.
-            audio_bytes = await self.bicodec_decoder.decode_to_bytes_async(semantic_ids, global_ids)
+            if _ASYNC_BICODEC_DECODE_ENABLED:
+                # Optional: keep event loop responsive by offloading decode.
+                audio_bytes = await self.bicodec_decoder.decode_to_bytes_async(semantic_ids, global_ids)
+            else:
+                audio_bytes = self.bicodec_decoder.decode_to_bytes(semantic_ids, global_ids)
*** End Patch


Output: {"output":"Success. Updated the following files:\nM /home/ubuntu/veenaModal/veena3modal/core/pipeline.py\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


Summary: auto


**Running py_compile and benchmarks**


cd /home/ubuntu/veenaModal && python3 -m py_compile veena3modal/core/pipeline.py


Output:
Chunk ID: 3d6d45
Wall time: 0.0516 seconds
Process exited with code 0
Original token count: 0
Output:


Output:
Chunk ID: 1a6bae
Wall time: 0.0517 seconds
Process exited with code 0
Original token count: 0
Output:


**Running benchmark with new output**


cd /home/ubuntu/veenaModal && python3 scripts/stress_test_runtime_detailed.py --levels 96 --text short --max-tokens 128 --timeout 180 --requests-multiplier 4 --min-requests 8 --gpu-memory 0.25 --num-engines 1 --output /tmp/runtime_c96_after_patch_stats_on_v2.json


Output:
Chunk ID: 8db478
Wall time: 1.0013 seconds
Process running with session ID 58889
Original token count: 0
Output:


**Planning poll with commentary updates**


Output:
Chunk ID: 4694ed
Wall time: 5.0012 seconds
Process running with session ID 58889
Original token count: 1589
Output:
[2026-02-18 00:19:28] INFO tts_runtime.py:183: Loading SparkTTS model with vLLM...
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
🚀 Initializing Spark TTS Model
📁 Model path: /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
🗣  Speakers: 12 predefined (lipakshi, vardan, reet, Nandini, krishna, anika, adarsh, Nilay, Aarvi, Asha, Bittu, Mira)
🎭 Emotions: 10 tags in [bracket] format

📝 Loading tokenizer from model...
✅ Tokenizer loaded: 165168 tokens

🔍 Validating 12 speakers for Spark TTS...
  lipakshi → <|speaker_0|>: 7 tokens
  vardan → <|speaker_1|>: 7 tokens
  reet → <|speaker_2|>: 7 tokens
  Nandini → <|speaker_3|>: 7 tokens
  krishna → <|speaker_4|>: 7 tokens
  anika → <|speaker_5|>: 7 tokens
  adarsh → <|speaker_6|>: 7 tokens
  Nilay → <|speaker_7|>: 7 tokens
  Aarvi → <|speaker_8|>: 7 tokens
  Asha → <|speaker_9|>: 7 tokens
  Bittu → <|speaker_10|>: 8 tokens
  Mira → <|speaker_11|>: 8 tokens
✅ All 12 speakers validated with mapping

🔍 Validating 10 emotion tags for Spark TTS...
✅ All 10 emotion tags validated

📋 Sample emotion tags:
  [angry]: 1 tokens
  [curious]: 1 tokens
  [excited]: 1 tokens
  [giggle]: 1 tokens
  [laughs harder]: 1 tokens
  ...

🔧 Initializing vLLM engine with optimizations...
  - dtype: bfloat16
  - max_model_len: 4096
  - gpu_memory_utilization: 0.25
  - enable_prefix_caching: True
  - enforce_eager (disable CUDA graphs): False
INFO 02-18 00:19:28 [model.py:541] Resolved architecture: Qwen2ForCausalLM
INFO 02-18 00:19:28 [model.py:1561] Using max model len 4096
INFO 02-18 00:19:29 [scheduler.py:226] Chunked prefill is enabled with max_num_batched_tokens=4096.
INFO 02-18 00:19:29 [vllm.py:624] Asynchronous scheduling is enabled.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(EngineCore_DP0 pid=4096175) INFO 02-18 00:19:29 [core.py:96] Initializing a V1 LLM engine (v0.15.1) with config: model='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', speculative_config=None, tokenizer='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [4096], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 256, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': True}, 'local_cache_dir': None, 'static_all_moe_layers': []}
(EngineCore_DP0 pid=4096175) INFO 02-18 00:19:32 [parallel_state.py:1212] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://216.81.248.184:36597 backend=nccl
(EngineCore_DP0 pid=4096175) INFO 02-18 00:19:32 [parallel_state.py:1423] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
(EngineCore_DP0 pid=4096175) INFO 02-18 00:19:32 [gpu_model_runner.py:4033] Starting to load model /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM...
(EngineCore_DP0 pid=4096175) INFO 02-18 00:19:33 [cuda.py:364] Using FLASH_ATTN attention backend out of potential backends: ('FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION')
(EngineCore_DP0 pid=4096175) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
(EngineCore_DP0 pid=4096175) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
(EngineCore_DP0 pid=4096175)
Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]


Output:
Chunk ID: 8573a9
Wall time: 5.0006 seconds
Process running with session ID 58889
Original token count: 199
Output:
(EngineCore_DP0 pid=4096175)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:01<00:00,  1.34s/it]
(EngineCore_DP0 pid=4096175)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:01<00:00,  1.34s/it]
(EngineCore_DP0 pid=4096175)
(EngineCore_DP0 pid=4096175) INFO 02-18 00:19:35 [default_loader.py:291] Loading weights took 1.42 seconds
(EngineCore_DP0 pid=4096175) INFO 02-18 00:19:35 [gpu_model_runner.py:4130] Model loading took 0.95 GiB memory and 1.957965 seconds
(EngineCore_DP0 pid=4096175) INFO 02-18 00:19:40 [backends.py:812] Using cache directory: /home/ubuntu/.cache/vllm/torch_compile_cache/f3ff13511c/rank_0_0/backbone for vLLM's torch.compile
(EngineCore_DP0 pid=4096175) INFO 02-18 00:19:40 [backends.py:872] Dynamo bytecode transform time: 4.35 s


This rerun is progressing normally with decode reverted to sync; once it finishes I’ll compare it against the prior (regressed) run and proceed to the optimized no-stats run.


Output:
Chunk ID: 9d03ef
Wall time: 5.0007 seconds
Process running with session ID 58889
Original token count: 787
Output:
(EngineCore_DP0 pid=4096175) INFO 02-18 00:19:43 [backends.py:267] Directly load the compiled graph(s) for compile range (1, 4096) from the cache, took 0.676 s
(EngineCore_DP0 pid=4096175) INFO 02-18 00:19:43 [monitor.py:34] torch.compile takes 5.02 s in total
(EngineCore_DP0 pid=4096175) INFO 02-18 00:19:44 [gpu_worker.py:356] Available KV cache memory: 18.06 GiB
(EngineCore_DP0 pid=4096175) INFO 02-18 00:19:44 [kv_cache_utils.py:1307] GPU KV cache size: 1,577,712 tokens
(EngineCore_DP0 pid=4096175) INFO 02-18 00:19:44 [kv_cache_utils.py:1312] Maximum concurrency for 4,096 tokens per request: 385.18x
(EngineCore_DP0 pid=4096175)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   0%|          | 0/35 [00:00<?, ?it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  11%|█▏        | 4/35 [00:00<00:00, 32.14it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  23%|██▎       | 8/35 [00:00<00:00, 34.29it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  34%|███▍      | 12/35 [00:00<00:00, 35.74it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  46%|████▌     | 16/35 [00:00<00:00, 36.14it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  57%|█████▋    | 20/35 [00:00<00:00, 36.33it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  69%|██████▊   | 24/35 [00:00<00:00, 36.55it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  80%|████████  | 28/35 [00:00<00:00, 36.53it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  91%|█████████▏| 32/35 [00:00<00:00, 34.87it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 35/35 [00:01<00:00, 34.70it/s]
(EngineCore_DP0 pid=4096175)
Capturing CUDA graphs (decode, FULL):   0%|          | 0/19 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL):  16%|█▌        | 3/19 [00:00<00:00, 27.28it/s]
Capturing CUDA graphs (decode, FULL):  37%|███▋      | 7/19 [00:00<00:00, 32.90it/s]
Capturing CUDA graphs (decode, FULL):  58%|█████▊    | 11/19 [00:00<00:00, 34.56it/s]
Capturing CUDA graphs (decode, FULL):  79%|███████▉  | 15/19 [00:00<00:00, 35.12it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 35.83it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 34.73it/s]
(EngineCore_DP0 pid=4096175) INFO 02-18 00:19:46 [gpu_model_runner.py:5063] Graph capturing finished in 2 secs, took 0.28 GiB
(EngineCore_DP0 pid=4096175) INFO 02-18 00:19:46 [core.py:272] init engine (profile, create kv cache, warmup model) took 11.09 seconds
[2026-02-18 00:19:48] INFO tts_runtime.py:192: Initializing prompt builder...
[2026-02-18 00:19:48] INFO tts_runtime.py:199: Initializing BiCodec decoder...
/home/ubuntu/.local/lib/python3.10/site-packages/torch/nn/utils/weight_norm.py:144: FutureWarning: `torch.nn.utils.weight_norm` is deprecated in favor of `torch.nn.utils.parametrizations.weight_norm`.
  WeightNorm.apply(module, name, dim)


Output:
Chunk ID: 373923
Wall time: 5.0010 seconds
Process running with session ID 58889
Original token count: 127
Output:
[2026-02-18 00:19:54] INFO tts_runtime.py:206: Initializing TTS pipeline...
[2026-02-18 00:19:54] INFO tts_runtime.py:217: Initializing streaming pipeline...
[2026-02-18 00:19:54] INFO long_text_processor.py:112: LongTextProcessor initialized
[2026-02-18 00:19:54] INFO tts_runtime.py:274: ✅ TTS runtime initialized in 31712ms
[2026-02-18 00:19:54] INFO tts_runtime.py:275:    Model version: LLM
[2026-02-18 00:19:54] INFO tts_runtime.py:286: Speaker globals cache startup precompute skipped (disabled)


Output:
Chunk ID: 6b942e
Wall time: 5.0006 seconds
Process running with session ID 58889
Original token count: 617
Output:
✅ vLLM engine initialized successfully!

🎉 Spark TTS Model ready for inference!

🎵 Loading BiCodec audio tokenizer from /home/ubuntu/veenaModal/models/spark_tts_4speaker...
   Device: cuda
Missing tensor: mel_transformer.spectrogram.window
Missing tensor: mel_transformer.mel_scale.fb
✅ BiCodec decoder initialized (sample rate: 16000Hz)
🚀 SparkTTSPipeline initialized
🌊 Veena3SlidingWindowPipeline initialized (sliding window: 28 tokens)

=== concurrency=96 requests=384 ===
success=384/384 (100%)
wall=8.34s throughput=46.03 req/s
latency_ms avg=2047 p50=2147 p95=2173 max=2173
generation_ms: avg=1509.72 p50=1501.00 p95=2041.00 max=2116.00
llm_time_in_queue_ms: avg=0.06 p50=0.05 p95=0.13 max=0.17
llm_scheduler_ms: avg=23.81 p50=24.01 p95=31.00 max=31.00
llm_first_token_ms: avg=60.78 p50=39.93 p95=67.84 max=1186.89
llm_request_lifecycle_ms: avg=912.42 p50=917.34 p95=932.13 max=935.01
llm_queued_to_scheduled_ms: avg=0.06 p50=0.05 p95=0.13 max=0.17
llm_scheduled_to_first_token_ms: avg=23.81 p50=24.01 p95=31.00 max=31.00
llm_first_to_last_token_ms: avg=888.55 p50=890.22 p95=907.55 max=911.51
llm_queued_to_last_token_ms: avg=912.42 p50=917.34 p95=932.13 max=935.01
timeline_llm_first_batch_ms: avg=65.45 p50=43.08 p95=72.07 max=1250.66
timeline_llm_done_ms: avg=1498.30 p50=1484.82 p95=2031.47 max=2106.62
timeline_parse_done_ms: avg=1498.36 p50=1484.88 p95=2031.53 max=2106.69
timeline_bicodec_done_ms: avg=1510.14 p50=1501.47 p95=2041.40 max=2116.69
timeline_total_ms: avg=1510.16 p50=1501.51 p95=2041.42 max=2116.71
llm_batch_wall_ms_total: avg=1498.11 p50=1484.65 p95=2031.29 max=2106.44
llm_decode_wall_ms_total: avg=1455.56 p50=1451.58 p95=1996.16 max=2075.19
llm_parse_ms: avg=0.06 p50=0.06 p95=0.08 max=0.10
llm_time_per_token_ms: avg=11.70 p50=11.60 p95=15.87 max=16.46
bicodec_decode_wall_ms: avg=11.72 p50=12.33 p95=14.87 max=16.99
bicodec_decode_gpu_ms: avg=0.00 p50=0.00 p95=0.00 max=0.00
latency_over_timeline_ms: avg=536.40 p50=513.35 p95=1097.19 max=1197.99
latency_over_generation_ms: avg=536.84 p50=514.02 p95=1097.81 max=1198.34
timeline_share: llm=99.2% bicodec=0.8% other=0.0%
timeline_sample_ms: first_batch=69.5 llm_done=1049.3 parse_done=1049.3 bicodec_done=1062.1 total=1062.1
gpu: util_avg=73% util_max=99% mem_avg=23601MB mem_max=23601MB

results saved: /tmp/runtime_c96_after_patch_stats_on_v2.json
ERROR 02-18 00:20:04 [core_client.py:605] Engine core proc EngineCore_DP0 died unexpectedly, shutting down client.


**Running no-stats variant benchmark**


The regression is fixed with sync decode restored. I’m now running the optimized throughput configuration (`--disable-engine-stats-logs`) on the same workload to see the best post-patch concurrency result.


cd /home/ubuntu/veenaModal && python3 scripts/stress_test_runtime_detailed.py --levels 96 --text short --max-tokens 128 --timeout 180 --requests-multiplier 4 --min-requests 8 --gpu-memory 0.25 --num-engines 1 --disable-engine-stats-logs --output /tmp/runtime_c96_after_patch_nostats.json


Output:
Chunk ID: e6e8ec
Wall time: 1.0013 seconds
Process running with session ID 17901
Original token count: 0
Output:


Output:
Chunk ID: 81da01
Wall time: 5.0012 seconds
Process running with session ID 17901
Original token count: 1589
Output:
[2026-02-18 00:20:16] INFO tts_runtime.py:183: Loading SparkTTS model with vLLM...
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
🚀 Initializing Spark TTS Model
📁 Model path: /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
🗣  Speakers: 12 predefined (lipakshi, vardan, reet, Nandini, krishna, anika, adarsh, Nilay, Aarvi, Asha, Bittu, Mira)
🎭 Emotions: 10 tags in [bracket] format

📝 Loading tokenizer from model...
✅ Tokenizer loaded: 165168 tokens

🔍 Validating 12 speakers for Spark TTS...
  lipakshi → <|speaker_0|>: 7 tokens
  vardan → <|speaker_1|>: 7 tokens
  reet → <|speaker_2|>: 7 tokens
  Nandini → <|speaker_3|>: 7 tokens
  krishna → <|speaker_4|>: 7 tokens
  anika → <|speaker_5|>: 7 tokens
  adarsh → <|speaker_6|>: 7 tokens
  Nilay → <|speaker_7|>: 7 tokens
  Aarvi → <|speaker_8|>: 7 tokens
  Asha → <|speaker_9|>: 7 tokens
  Bittu → <|speaker_10|>: 8 tokens
  Mira → <|speaker_11|>: 8 tokens
✅ All 12 speakers validated with mapping

🔍 Validating 10 emotion tags for Spark TTS...
✅ All 10 emotion tags validated

📋 Sample emotion tags:
  [angry]: 1 tokens
  [curious]: 1 tokens
  [excited]: 1 tokens
  [giggle]: 1 tokens
  [laughs harder]: 1 tokens
  ...

🔧 Initializing vLLM engine with optimizations...
  - dtype: bfloat16
  - max_model_len: 4096
  - gpu_memory_utilization: 0.25
  - enable_prefix_caching: True
  - enforce_eager (disable CUDA graphs): False
INFO 02-18 00:20:17 [model.py:541] Resolved architecture: Qwen2ForCausalLM
INFO 02-18 00:20:17 [model.py:1561] Using max model len 4096
INFO 02-18 00:20:17 [scheduler.py:226] Chunked prefill is enabled with max_num_batched_tokens=4096.
INFO 02-18 00:20:17 [vllm.py:624] Asynchronous scheduling is enabled.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(EngineCore_DP0 pid=4096784) INFO 02-18 00:20:18 [core.py:96] Initializing a V1 LLM engine (v0.15.1) with config: model='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', speculative_config=None, tokenizer='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [4096], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 256, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': True}, 'local_cache_dir': None, 'static_all_moe_layers': []}
(EngineCore_DP0 pid=4096784) INFO 02-18 00:20:20 [parallel_state.py:1212] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://216.81.248.184:43681 backend=nccl
(EngineCore_DP0 pid=4096784) INFO 02-18 00:20:20 [parallel_state.py:1423] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
(EngineCore_DP0 pid=4096784) INFO 02-18 00:20:20 [gpu_model_runner.py:4033] Starting to load model /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM...
(EngineCore_DP0 pid=4096784) INFO 02-18 00:20:21 [cuda.py:364] Using FLASH_ATTN attention backend out of potential backends: ('FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION')
(EngineCore_DP0 pid=4096784) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
(EngineCore_DP0 pid=4096784) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
(EngineCore_DP0 pid=4096784)
Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]


Output:
Chunk ID: cd124a
Wall time: 5.0013 seconds
Process running with session ID 17901
Original token count: 199
Output:
(EngineCore_DP0 pid=4096784)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.06it/s]
(EngineCore_DP0 pid=4096784)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.06it/s]
(EngineCore_DP0 pid=4096784)
(EngineCore_DP0 pid=4096784) INFO 02-18 00:20:22 [default_loader.py:291] Loading weights took 1.02 seconds
(EngineCore_DP0 pid=4096784) INFO 02-18 00:20:23 [gpu_model_runner.py:4130] Model loading took 0.95 GiB memory and 1.568571 seconds
(EngineCore_DP0 pid=4096784) INFO 02-18 00:20:28 [backends.py:812] Using cache directory: /home/ubuntu/.cache/vllm/torch_compile_cache/f3ff13511c/rank_0_0/backbone for vLLM's torch.compile
(EngineCore_DP0 pid=4096784) INFO 02-18 00:20:28 [backends.py:872] Dynamo bytecode transform time: 4.20 s


The no-stats run is in progress; after this finishes I’ll extract a side-by-side of throughput, latency, and queue/scheduler markers from the new metrics fields.


Output:
Chunk ID: d328f9
Wall time: 5.0009 seconds
Process running with session ID 17901
Original token count: 787
Output:
(EngineCore_DP0 pid=4096784) INFO 02-18 00:20:31 [backends.py:267] Directly load the compiled graph(s) for compile range (1, 4096) from the cache, took 0.671 s
(EngineCore_DP0 pid=4096784) INFO 02-18 00:20:31 [monitor.py:34] torch.compile takes 4.87 s in total
(EngineCore_DP0 pid=4096784) INFO 02-18 00:20:32 [gpu_worker.py:356] Available KV cache memory: 18.06 GiB
(EngineCore_DP0 pid=4096784) INFO 02-18 00:20:32 [kv_cache_utils.py:1307] GPU KV cache size: 1,577,712 tokens
(EngineCore_DP0 pid=4096784) INFO 02-18 00:20:32 [kv_cache_utils.py:1312] Maximum concurrency for 4,096 tokens per request: 385.18x
(EngineCore_DP0 pid=4096784)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   0%|          | 0/35 [00:00<?, ?it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  11%|█▏        | 4/35 [00:00<00:00, 32.87it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  23%|██▎       | 8/35 [00:00<00:00, 35.57it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  34%|███▍      | 12/35 [00:00<00:00, 35.79it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  46%|████▌     | 16/35 [00:00<00:00, 36.00it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  57%|█████▋    | 20/35 [00:00<00:00, 35.77it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  69%|██████▊   | 24/35 [00:00<00:00, 35.83it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  80%|████████  | 28/35 [00:00<00:00, 35.12it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  91%|█████████▏| 32/35 [00:00<00:00, 33.72it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 35/35 [00:01<00:00, 34.09it/s]
(EngineCore_DP0 pid=4096784)
Capturing CUDA graphs (decode, FULL):   0%|          | 0/19 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL):  16%|█▌        | 3/19 [00:00<00:00, 29.52it/s]
Capturing CUDA graphs (decode, FULL):  37%|███▋      | 7/19 [00:00<00:00, 34.11it/s]
Capturing CUDA graphs (decode, FULL):  58%|█████▊    | 11/19 [00:00<00:00, 35.48it/s]
Capturing CUDA graphs (decode, FULL):  79%|███████▉  | 15/19 [00:00<00:00, 35.62it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 36.14it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 35.37it/s]
(EngineCore_DP0 pid=4096784) INFO 02-18 00:20:34 [gpu_model_runner.py:5063] Graph capturing finished in 2 secs, took 0.28 GiB
(EngineCore_DP0 pid=4096784) INFO 02-18 00:20:34 [core.py:272] init engine (profile, create kv cache, warmup model) took 10.88 seconds
[2026-02-18 00:20:36] INFO tts_runtime.py:192: Initializing prompt builder...
[2026-02-18 00:20:36] INFO tts_runtime.py:199: Initializing BiCodec decoder...
/home/ubuntu/.local/lib/python3.10/site-packages/torch/nn/utils/weight_norm.py:144: FutureWarning: `torch.nn.utils.weight_norm` is deprecated in favor of `torch.nn.utils.parametrizations.weight_norm`.
  WeightNorm.apply(module, name, dim)


Output:
Chunk ID: fa15e2
Wall time: 5.0008 seconds
Process running with session ID 17901
Original token count: 127
Output:
[2026-02-18 00:20:41] INFO tts_runtime.py:206: Initializing TTS pipeline...
[2026-02-18 00:20:41] INFO tts_runtime.py:217: Initializing streaming pipeline...
[2026-02-18 00:20:41] INFO long_text_processor.py:112: LongTextProcessor initialized
[2026-02-18 00:20:41] INFO tts_runtime.py:274: ✅ TTS runtime initialized in 30525ms
[2026-02-18 00:20:41] INFO tts_runtime.py:275:    Model version: LLM
[2026-02-18 00:20:41] INFO tts_runtime.py:286: Speaker globals cache startup precompute skipped (disabled)


Output:
Chunk ID: 3bba48
Wall time: 1.2551 seconds
Process exited with code 0
Original token count: 484
Output:
✅ vLLM engine initialized successfully!

🎉 Spark TTS Model ready for inference!

🎵 Loading BiCodec audio tokenizer from /home/ubuntu/veenaModal/models/spark_tts_4speaker...
   Device: cuda
Missing tensor: mel_transformer.spectrogram.window
Missing tensor: mel_transformer.mel_scale.fb
✅ BiCodec decoder initialized (sample rate: 16000Hz)
🚀 SparkTTSPipeline initialized
🌊 Veena3SlidingWindowPipeline initialized (sliding window: 28 tokens)

=== concurrency=96 requests=384 ===
success=384/384 (100%)
wall=8.14s throughput=47.15 req/s
latency_ms avg=1925 p50=2091 p95=2476 max=2476
generation_ms: avg=1469.20 p50=1429.00 p95=2020.00 max=2152.00
timeline_llm_first_batch_ms: avg=114.19 p50=40.67 p95=1022.41 max=1251.90
timeline_llm_done_ms: avg=1457.52 p50=1416.97 p95=2010.71 max=2136.96
timeline_parse_done_ms: avg=1457.59 p50=1417.02 p95=2010.77 max=2137.03
timeline_bicodec_done_ms: avg=1469.63 p50=1429.12 p95=2020.67 max=2152.07
timeline_total_ms: avg=1469.65 p50=1429.14 p95=2020.68 max=2152.10
llm_batch_wall_ms_total: avg=1457.38 p50=1416.84 p95=2010.57 max=2136.84
llm_decode_wall_ms_total: avg=1357.46 p50=1354.61 p95=1981.64 max=2111.10
llm_parse_ms: avg=0.06 p50=0.06 p95=0.07 max=0.24
llm_time_per_token_ms: avg=11.39 p50=11.07 p95=15.71 max=16.69
bicodec_decode_wall_ms: avg=11.99 p50=12.47 p95=14.92 max=18.26
bicodec_decode_gpu_ms: avg=0.00 p50=0.00 p95=0.00 max=0.00
latency_over_timeline_ms: avg=455.66 p50=375.46 p95=1075.29 max=1252.31
latency_over_generation_ms: avg=456.11 p50=375.72 p95=1075.89 max=1252.89
timeline_share: llm=99.2% bicodec=0.8% other=0.0%
timeline_sample_ms: first_batch=32.4 llm_done=2044.5 parse_done=2044.5 bicodec_done=2054.6 total=2054.6
gpu: util_avg=79% util_max=100% mem_avg=23601MB mem_max=23601MB

results saved: /tmp/runtime_c96_after_patch_nostats.json
ERROR 02-18 00:20:51 [core_client.py:605] Engine core proc EngineCore_DP0 died unexpectedly, shutting down client.


**Analyzing detailed latency discrepancies**


I have the post-patch datasets now. I’m running a quick numeric comparison to isolate the true bottleneck split: vLLM engine lifecycle vs Python-side wall time vs end-to-end request latency.


python3 - <<'PY'
import json, statistics
paths=[
('/tmp/runtime_c96_baseline_confirm.json','baseline_old_statson'),
('/tmp/runtime_c96_nostats_confirm.json','baseline_old_nostats'),
('/tmp/runtime_c96_after_patch_stats_on_v2.json','after_patch_statson'),
('/tmp/runtime_c96_after_patch_nostats.json','after_patch_nostats'),
]

for path,label in paths:
    with open(path) as f: d=json.load(f)
    lvl=d['levels'][0]
    reqs=[r for r in lvl['requests'] if r['success']]
    def vals(key):
        return [float(r['timing'][key]) for r in reqs if isinstance(r['timing'].get(key),(int,float))]
    def stat(v):
        if not v:return None
        s=sorted(v)
        return {'avg':statistics.mean(v),'p50':s[len(s)//2],'p95':s[int(len(s)*0.95)-1],'max':s[-1]}
    print('\n##',label)
    print('throughput',round(lvl['throughput_rps'],3),'lat_avg',round(lvl['latency_ms']['avg'],2))
    for k in ['generation_ms','timeline_total_ms','timeline_llm_done_ms','timeline_llm_first_batch_ms','llm_batch_wall_ms_total','llm_time_per_token_ms','bicodec_decode_wall_ms']:
        st=stat(vals(k))
        if st: print(k, {m:round(st[m],2) for m in st})
    if vals('llm_queued_to_last_token_ms'):
        st=stat(vals('llm_queued_to_last_token_ms'))
        print('llm_queued_to_last_token_ms', {m:round(st[m],2) for m in st})
    if vals('llm_scheduler_ms'):
        st=stat(vals('llm_scheduler_ms'))
        print('llm_scheduler_ms', {m:round(st[m],2) for m in st})
    if vals('llm_time_in_queue_ms'):
        st=stat(vals('llm_time_in_queue_ms'))
        print('llm_time_in_queue_ms', {m:round(st[m],2) for m in st})
    # derived gaps
    tl=vals('timeline_total_ms')
    life=vals('llm_queued_to_last_token_ms')
    if tl and life and len(tl)==len(life):
        gap=[max(0.0,a-b) for a,b in zip(tl,life)]
        st=stat(gap)
        print('gap_timeline_vs_engine_lifecycle_ms', {m:round(st[m],2) for m in st})
    lat=[float(r['latency_ms']) for r in reqs]
    gen=vals('generation_ms')
    if lat and gen and len(lat)==len(gen):
        gap=[max(0.0,a-b) for a,b in zip(lat,gen)]
        st=stat(gap)
        print('gap_latency_vs_generation_ms', {m:round(st[m],2) for m in st})
PY


Output:
Chunk ID: 87e97f
Wall time: 0.0512 seconds
Process exited with code 0
Original token count: 812
Output:

## baseline_old_statson
throughput 45.714 lat_avg 2034.04
generation_ms {'avg': 1514.91, 'p50': 1512.0, 'p95': 2035.0, 'max': 2103.0}
timeline_total_ms {'avg': 1515.35, 'p50': 1512.58, 'p95': 2035.79, 'max': 2103.31}
timeline_llm_done_ms {'avg': 1503.31, 'p50': 1500.93, 'p95': 2025.64, 'max': 2092.87}
timeline_llm_first_batch_ms {'avg': 55.3, 'p50': 37.94, 'p95': 76.16, 'max': 1217.88}
llm_batch_wall_ms_total {'avg': 1503.16, 'p50': 1500.79, 'p95': 2025.5, 'max': 2092.7}
llm_time_per_token_ms {'avg': 11.74, 'p50': 11.73, 'p95': 15.82, 'max': 16.35}
bicodec_decode_wall_ms {'avg': 11.9, 'p50': 11.98, 'p95': 14.69, 'max': 19.68}
gap_latency_vs_generation_ms {'avg': 519.13, 'p50': 486.34, 'p95': 1048.38, 'max': 1176.36}

## baseline_old_nostats
throughput 47.078 lat_avg 1921.53
generation_ms {'avg': 1478.45, 'p50': 1430.0, 'p95': 2033.0, 'max': 2147.0}
timeline_total_ms {'avg': 1478.93, 'p50': 1429.99, 'p95': 2033.6, 'max': 2147.42}
timeline_llm_done_ms {'avg': 1466.57, 'p50': 1419.23, 'p95': 2023.34, 'max': 2134.62}
timeline_llm_first_batch_ms {'avg': 122.13, 'p50': 38.01, 'p95': 933.85, 'max': 1255.85}
llm_batch_wall_ms_total {'avg': 1466.42, 'p50': 1419.09, 'p95': 2023.2, 'max': 2134.47}
llm_time_per_token_ms {'avg': 11.46, 'p50': 11.09, 'p95': 15.81, 'max': 16.68}
bicodec_decode_wall_ms {'avg': 12.23, 'p50': 12.46, 'p95': 15.15, 'max': 17.92}
gap_latency_vs_generation_ms {'avg': 443.08, 'p50': 372.97, 'p95': 1023.11, 'max': 1164.04}

## after_patch_statson
throughput 46.026 lat_avg 2046.56
generation_ms {'avg': 1509.72, 'p50': 1504.0, 'p95': 2041.0, 'max': 2116.0}
timeline_total_ms {'avg': 1510.16, 'p50': 1503.97, 'p95': 2041.42, 'max': 2116.71}
timeline_llm_done_ms {'avg': 1498.3, 'p50': 1490.16, 'p95': 2031.47, 'max': 2106.62}
timeline_llm_first_batch_ms {'avg': 65.45, 'p50': 43.16, 'p95': 72.07, 'max': 1250.66}
llm_batch_wall_ms_total {'avg': 1498.11, 'p50': 1489.98, 'p95': 2031.29, 'max': 2106.44}
llm_time_per_token_ms {'avg': 11.7, 'p50': 11.64, 'p95': 15.87, 'max': 16.46}
bicodec_decode_wall_ms {'avg': 11.72, 'p50': 12.43, 'p95': 14.87, 'max': 16.99}
llm_queued_to_last_token_ms {'avg': 912.42, 'p50': 917.34, 'p95': 932.13, 'max': 935.01}
llm_scheduler_ms {'avg': 23.81, 'p50': 24.01, 'p95': 31.0, 'max': 31.0}
llm_time_in_queue_ms {'avg': 0.06, 'p50': 0.05, 'p95': 0.13, 'max': 0.17}
gap_timeline_vs_engine_lifecycle_ms {'avg': 597.75, 'p50': 586.54, 'p95': 1134.31, 'max': 1205.76}
gap_latency_vs_generation_ms {'avg': 536.84, 'p50': 516.27, 'p95': 1097.81, 'max': 1198.34}

## after_patch_nostats
throughput 47.151 lat_avg 1925.3
generation_ms {'avg': 1469.2, 'p50': 1435.0, 'p95': 2020.0, 'max': 2152.0}
timeline_total_ms {'avg': 1469.65, 'p50': 1435.3, 'p95': 2020.68, 'max': 2152.1}
timeline_llm_done_ms {'avg': 1457.52, 'p50': 1424.18, 'p95': 2010.71, 'max': 2136.96}
timeline_llm_first_batch_ms {'avg': 114.19, 'p50': 40.72, 'p95': 1022.41, 'max': 1251.9}
llm_batch_wall_ms_total {'avg': 1457.38, 'p50': 1424.06, 'p95': 2010.57, 'max': 2136.84}
llm_time_per_token_ms {'avg': 11.39, 'p50': 11.13, 'p95': 15.71, 'max': 16.69}
bicodec_decode_wall_ms {'avg': 11.99, 'p50': 12.48, 'p95': 14.92, 'max': 18.26}
gap_latency_vs_generation_ms {'avg': 456.11, 'p50': 376.37, 'p95': 1075.89, 'max': 1252.89}


**Identifying streaming overhead and output_kind optimization**


python3 - <<'PY'
from vllm import SamplingParams
import inspect
sig=inspect.signature(SamplingParams)
print('params:', list(sig.parameters.keys())[:80])
print('has output_kind', 'output_kind' in sig.parameters)
if 'output_kind' in sig.parameters:
    print('default', sig.parameters['output_kind'].default)
    try:
        from vllm.sampling_params import RequestOutputKind
        print('RequestOutputKind', list(RequestOutputKind))
    except Exception as e:
        print('enum import error',e)
PY


Output:
Chunk ID: 40a5b8
Wall time: 4.9012 seconds
Process exited with code 0
Original token count: 197
Output:
params: ['n', 'presence_penalty', 'frequency_penalty', 'repetition_penalty', 'temperature', 'top_p', 'top_k', 'min_p', 'seed', 'stop', 'stop_token_ids', 'ignore_eos', 'max_tokens', 'min_tokens', 'logprobs', 'prompt_logprobs', 'flat_logprobs', 'detokenize', 'skip_special_tokens', 'spaces_between_special_tokens', 'logits_processors', 'include_stop_str_in_output', 'truncate_prompt_tokens', 'output_kind', 'skip_clone', 'output_text_buffer_length', '_all_stop_token_ids', 'structured_outputs', 'logit_bias', 'allowed_token_ids', 'extra_args', 'bad_words', '_bad_words_token_ids', 'skip_reading_prefix_cache']
has output_kind True
default RequestOutputKind.CUMULATIVE
RequestOutputKind [<RequestOutputKind.CUMULATIVE: 0>, <RequestOutputKind.DELTA: 1>, <RequestOutputKind.FINAL_ONLY: 2>]


**Planning final-only output optimization**


Input:
*** Begin Patch
*** Update File: /home/ubuntu/veenaModal/veena3modal/core/pipeline.py
@@
 from dataclasses import asdict, is_dataclass
 from typing import Optional, List, Dict, Any, Tuple
 from vllm import SamplingParams
+from vllm.sampling_params import RequestOutputKind
 import torch
@@
 _ASYNC_BICODEC_DECODE_ENABLED = os.environ.get("VEENA3_ASYNC_BICODEC_DECODE", "").strip().lower() in {
     "1",
     "true",
     "yes",
     "on",
 }
+_NON_STREAM_FINAL_ONLY_ENABLED = os.environ.get("VEENA3_NON_STREAM_FINAL_ONLY", "").strip().lower() in {
+    "1",
+    "true",
+    "yes",
+    "on",
+}
@@
         sampling_params = SamplingParams(
             temperature=temperature,
             top_k=top_k,
             top_p=top_p,
             max_tokens=max_tokens,
             repetition_penalty=repetition_penalty,
             stop=TRAINING_STOP_TOKEN_IDS,
             skip_special_tokens=False,
             seed=seed,
+            output_kind=(RequestOutputKind.FINAL_ONLY if _NON_STREAM_FINAL_ONLY_ENABLED else RequestOutputKind.CUMULATIVE),
         )
*** End Patch


Output: {"output":"Success. Updated the following files:\nM /home/ubuntu/veenaModal/veena3modal/core/pipeline.py\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


Summary: auto


**Adding non-stream-final-only flag**


I’m adding one more high-impact scheduler optimization switch: non-streaming `FINAL_ONLY` output from vLLM. This cuts per-token IPC/driver overhead and should increase throughput; I’ll benchmark it next against the current best run.


Input:
*** Begin Patch
*** Update File: /home/ubuntu/veenaModal/scripts/stress_test_runtime_detailed.py
@@
             "disable_engine_stats_logs": args.disable_engine_stats_logs,
             "enforce_eager": args.enforce_eager,
             "enable_gpu_decode_timing": args.enable_gpu_decode_timing,
+            "non_stream_final_only": args.non_stream_final_only,
             "timestamp": int(time.time()),
         },
         "levels": [],
@@
     parser.add_argument("--disable-engine-stats-logs", action="store_true", default=False)
     parser.add_argument("--enforce-eager", action="store_true", default=False)
+    parser.add_argument(
+        "--non-stream-final-only",
+        action="store_true",
+        default=False,
+        help="Use vLLM FINAL_ONLY output mode (less per-token driver overhead, less granular timeline)",
+    )
     parser.add_argument(
         "--enable-gpu-decode-timing",
         action="store_true",
         default=False,
         help="Enable precise BiCodec GPU timing (adds synchronize overhead)",
@@
     if args.enable_gpu_decode_timing:
         os.environ["VEENA3_PERF_GPU_TIMING"] = "1"
     else:
         os.environ.setdefault("VEENA3_PERF_GPU_TIMING", "0")
+    if args.non_stream_final_only:
+        os.environ["VEENA3_NON_STREAM_FINAL_ONLY"] = "1"
+    else:
+        os.environ.setdefault("VEENA3_NON_STREAM_FINAL_ONLY", "0")
     enable_chunked_prefill = args.enable_chunked_prefill
*** End Patch


Output: {"output":"Success. Updated the following files:\nM /home/ubuntu/veenaModal/scripts/stress_test_runtime_detailed.py\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


**Preparing changed files compilation**


cd /home/ubuntu/veenaModal && python3 -m py_compile veena3modal/core/pipeline.py scripts/stress_test_runtime_detailed.py


Output:
Chunk ID: 77e75e
Wall time: 0.0513 seconds
Process exited with code 0
Original token count: 0
Output:


**Running benchmark with final-only and nostats**


cd /home/ubuntu/veenaModal && python3 scripts/stress_test_runtime_detailed.py --levels 96 --text short --max-tokens 128 --timeout 180 --requests-multiplier 4 --min-requests 8 --gpu-memory 0.25 --num-engines 1 --disable-engine-stats-logs --non-stream-final-only --output /tmp/runtime_c96_after_patch_nostats_finalonly.json


Output:
Chunk ID: cd5e49
Wall time: 1.0007 seconds
Process running with session ID 5883
Original token count: 0
Output:


**Preparing poll session request**


Output:
Chunk ID: 27dd0a
Wall time: 5.0008 seconds
Process running with session ID 5883
Original token count: 1401
Output:
[2026-02-18 00:22:08] INFO tts_runtime.py:183: Loading SparkTTS model with vLLM...
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
🚀 Initializing Spark TTS Model
📁 Model path: /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
🗣  Speakers: 12 predefined (lipakshi, vardan, reet, Nandini, krishna, anika, adarsh, Nilay, Aarvi, Asha, Bittu, Mira)
🎭 Emotions: 10 tags in [bracket] format

📝 Loading tokenizer from model...
✅ Tokenizer loaded: 165168 tokens

🔍 Validating 12 speakers for Spark TTS...
  lipakshi → <|speaker_0|>: 7 tokens
  vardan → <|speaker_1|>: 7 tokens
  reet → <|speaker_2|>: 7 tokens
  Nandini → <|speaker_3|>: 7 tokens
  krishna → <|speaker_4|>: 7 tokens
  anika → <|speaker_5|>: 7 tokens
  adarsh → <|speaker_6|>: 7 tokens
  Nilay → <|speaker_7|>: 7 tokens
  Aarvi → <|speaker_8|>: 7 tokens
  Asha → <|speaker_9|>: 7 tokens
  Bittu → <|speaker_10|>: 8 tokens
  Mira → <|speaker_11|>: 8 tokens
✅ All 12 speakers validated with mapping

🔍 Validating 10 emotion tags for Spark TTS...
✅ All 10 emotion tags validated

📋 Sample emotion tags:
  [angry]: 1 tokens
  [curious]: 1 tokens
  [excited]: 1 tokens
  [giggle]: 1 tokens
  [laughs harder]: 1 tokens
  ...

🔧 Initializing vLLM engine with optimizations...
  - dtype: bfloat16
  - max_model_len: 4096
  - gpu_memory_utilization: 0.25
  - enable_prefix_caching: True
  - enforce_eager (disable CUDA graphs): False
INFO 02-18 00:22:08 [model.py:541] Resolved architecture: Qwen2ForCausalLM
INFO 02-18 00:22:08 [model.py:1561] Using max model len 4096
INFO 02-18 00:22:09 [scheduler.py:226] Chunked prefill is enabled with max_num_batched_tokens=4096.
INFO 02-18 00:22:09 [vllm.py:624] Asynchronous scheduling is enabled.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(EngineCore_DP0 pid=4097625) INFO 02-18 00:22:09 [core.py:96] Initializing a V1 LLM engine (v0.15.1) with config: model='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', speculative_config=None, tokenizer='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [4096], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 256, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': True}, 'local_cache_dir': None, 'static_all_moe_layers': []}
(EngineCore_DP0 pid=4097625) INFO 02-18 00:22:12 [parallel_state.py:1212] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://216.81.248.184:40893 backend=nccl
(EngineCore_DP0 pid=4097625) INFO 02-18 00:22:12 [parallel_state.py:1423] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
(EngineCore_DP0 pid=4097625) INFO 02-18 00:22:12 [gpu_model_runner.py:4033] Starting to load model /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM...


**Planning polling continuation**


Output:
Chunk ID: 653094
Wall time: 5.0013 seconds
Process running with session ID 5883
Original token count: 653
Output:
(EngineCore_DP0 pid=4097625) INFO 02-18 00:22:13 [cuda.py:364] Using FLASH_ATTN attention backend out of potential backends: ('FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION')
(EngineCore_DP0 pid=4097625) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
(EngineCore_DP0 pid=4097625) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
(EngineCore_DP0 pid=4097625)
Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
(EngineCore_DP0 pid=4097625)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.64it/s]
(EngineCore_DP0 pid=4097625)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.64it/s]
(EngineCore_DP0 pid=4097625)
(EngineCore_DP0 pid=4097625) INFO 02-18 00:22:14 [default_loader.py:291] Loading weights took 0.68 seconds
(EngineCore_DP0 pid=4097625) INFO 02-18 00:22:14 [gpu_model_runner.py:4130] Model loading took 0.95 GiB memory and 1.226979 seconds
(EngineCore_DP0 pid=4097625) INFO 02-18 00:22:19 [backends.py:812] Using cache directory: /home/ubuntu/.cache/vllm/torch_compile_cache/f3ff13511c/rank_0_0/backbone for vLLM's torch.compile
(EngineCore_DP0 pid=4097625) INFO 02-18 00:22:19 [backends.py:872] Dynamo bytecode transform time: 4.23 s
(EngineCore_DP0 pid=4097625) INFO 02-18 00:22:22 [backends.py:267] Directly load the compiled graph(s) for compile range (1, 4096) from the cache, took 0.641 s
(EngineCore_DP0 pid=4097625) INFO 02-18 00:22:22 [monitor.py:34] torch.compile takes 4.87 s in total
(EngineCore_DP0 pid=4097625) INFO 02-18 00:22:23 [gpu_worker.py:356] Available KV cache memory: 18.06 GiB
(EngineCore_DP0 pid=4097625) INFO 02-18 00:22:23 [kv_cache_utils.py:1307] GPU KV cache size: 1,577,712 tokens
(EngineCore_DP0 pid=4097625) INFO 02-18 00:22:23 [kv_cache_utils.py:1312] Maximum concurrency for 4,096 tokens per request: 385.18x
(EngineCore_DP0 pid=4097625)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   0%|          | 0/35 [00:00<?, ?it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  11%|█▏        | 4/35 [00:00<00:00, 31.59it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  23%|██▎       | 8/35 [00:00<00:00, 34.14it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  34%|███▍      | 12/35 [00:00<00:00, 35.40it/s]


The `FINAL_ONLY` experiment is still running; this is the key throughput test to see if per-token output streaming is a driver bottleneck on non-streaming requests.


Output:
Chunk ID: 34ebe5
Wall time: 5.0010 seconds
Process running with session ID 5883
Original token count: 647
Output:

Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  46%|████▌     | 16/35 [00:00<00:00, 35.78it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  57%|█████▋    | 20/35 [00:00<00:00, 35.89it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  69%|██████▊   | 24/35 [00:00<00:00, 35.91it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  80%|████████  | 28/35 [00:00<00:00, 35.70it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  91%|█████████▏| 32/35 [00:00<00:00, 34.15it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 35/35 [00:01<00:00, 33.96it/s]
(EngineCore_DP0 pid=4097625)
Capturing CUDA graphs (decode, FULL):   0%|          | 0/19 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL):  16%|█▌        | 3/19 [00:00<00:00, 28.13it/s]
Capturing CUDA graphs (decode, FULL):  37%|███▋      | 7/19 [00:00<00:00, 32.67it/s]
Capturing CUDA graphs (decode, FULL):  58%|█████▊    | 11/19 [00:00<00:00, 34.26it/s]
Capturing CUDA graphs (decode, FULL):  79%|███████▉  | 15/19 [00:00<00:00, 34.30it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 35.04it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 34.17it/s]
(EngineCore_DP0 pid=4097625) INFO 02-18 00:22:25 [gpu_model_runner.py:5063] Graph capturing finished in 2 secs, took 0.28 GiB
(EngineCore_DP0 pid=4097625) INFO 02-18 00:22:25 [core.py:272] init engine (profile, create kv cache, warmup model) took 10.70 seconds
[2026-02-18 00:22:27] INFO tts_runtime.py:192: Initializing prompt builder...
[2026-02-18 00:22:27] INFO tts_runtime.py:199: Initializing BiCodec decoder...
/home/ubuntu/.local/lib/python3.10/site-packages/torch/nn/utils/weight_norm.py:144: FutureWarning: `torch.nn.utils.weight_norm` is deprecated in favor of `torch.nn.utils.parametrizations.weight_norm`.
  WeightNorm.apply(module, name, dim)
[2026-02-18 00:22:32] INFO tts_runtime.py:206: Initializing TTS pipeline...
[2026-02-18 00:22:32] INFO tts_runtime.py:217: Initializing streaming pipeline...
[2026-02-18 00:22:32] INFO long_text_processor.py:112: LongTextProcessor initialized
[2026-02-18 00:22:32] INFO tts_runtime.py:274: ✅ TTS runtime initialized in 29576ms
[2026-02-18 00:22:32] INFO tts_runtime.py:275:    Model version: LLM
[2026-02-18 00:22:32] INFO tts_runtime.py:286: Speaker globals cache startup precompute skipped (disabled)


Output:
Chunk ID: 1b9a57
Wall time: 5.0012 seconds
Process running with session ID 5883
Original token count: 469
Output:
✅ vLLM engine initialized successfully!

🎉 Spark TTS Model ready for inference!

🎵 Loading BiCodec audio tokenizer from /home/ubuntu/veenaModal/models/spark_tts_4speaker...
   Device: cuda
Missing tensor: mel_transformer.spectrogram.window
Missing tensor: mel_transformer.mel_scale.fb
✅ BiCodec decoder initialized (sample rate: 16000Hz)
🚀 SparkTTSPipeline initialized
🌊 Veena3SlidingWindowPipeline initialized (sliding window: 28 tokens)

=== concurrency=96 requests=384 ===
success=384/384 (100%)
wall=8.04s throughput=47.74 req/s
latency_ms avg=1844 p50=2037 p95=2461 max=2461
generation_ms: avg=1453.12 p50=1389.00 p95=2000.00 max=2150.00
timeline_llm_first_batch_ms: avg=1440.81 p50=1375.76 p95=1984.92 max=2137.12
timeline_llm_done_ms: avg=1440.82 p50=1375.77 p95=1984.93 max=2137.13
timeline_parse_done_ms: avg=1440.89 p50=1375.83 p95=1985.01 max=2137.19
timeline_bicodec_done_ms: avg=1453.58 p50=1389.04 p95=1999.95 max=2150.88
timeline_total_ms: avg=1453.60 p50=1389.05 p95=1999.97 max=2150.90
llm_batch_wall_ms_total: avg=1440.67 p50=1375.63 p95=1984.80 max=2136.99
llm_parse_ms: avg=0.06 p50=0.06 p95=0.08 max=0.13
llm_time_per_token_ms: avg=11.26 p50=10.75 p95=15.51 max=16.70
bicodec_decode_wall_ms: avg=12.67 p50=12.72 p95=15.31 max=18.04
bicodec_decode_gpu_ms: avg=0.00 p50=0.00 p95=0.00 max=0.00
latency_over_timeline_ms: avg=390.79 p50=294.80 p95=972.87 max=1226.49
latency_over_generation_ms: avg=391.27 p50=294.98 p95=973.46 max=1227.15
timeline_share: llm=99.1% bicodec=0.9% other=0.0%
timeline_sample_ms: first_batch=1523.1 llm_done=1523.1 parse_done=1523.1 bicodec_done=1536.0 total=1536.0
gpu: util_avg=82% util_max=99% mem_avg=23601MB mem_max=23601MB

results saved: /tmp/runtime_c96_after_patch_nostats_finalonly.json
ERROR 02-18 00:22:42 [core_client.py:605] Engine core proc EngineCore_DP0 died unexpectedly, shutting down client.


**Planning concurrency sweep tests**


`FINAL_ONLY` improved throughput and lowered latency. I’m running a higher-concurrency sweep next to find the real peak throughput point and where latency starts bending.


cd /home/ubuntu/veenaModal && python3 scripts/stress_test_runtime_detailed.py --levels 96,128,160 --text short --max-tokens 128 --timeout 180 --requests-multiplier 3 --min-requests 8 --gpu-memory 0.25 --num-engines 1 --disable-engine-stats-logs --non-stream-final-only --output /tmp/runtime_levels_96_160_finalonly.json


Output:
Chunk ID: 0deb05
Wall time: 1.0012 seconds
Process running with session ID 12978
Original token count: 0
Output:


**Planning regular polling**


Output:
Chunk ID: b3feda
Wall time: 5.0012 seconds
Process running with session ID 12978
Original token count: 1401
Output:
[2026-02-18 00:22:58] INFO tts_runtime.py:183: Loading SparkTTS model with vLLM...
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
🚀 Initializing Spark TTS Model
📁 Model path: /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
🗣  Speakers: 12 predefined (lipakshi, vardan, reet, Nandini, krishna, anika, adarsh, Nilay, Aarvi, Asha, Bittu, Mira)
🎭 Emotions: 10 tags in [bracket] format

📝 Loading tokenizer from model...
✅ Tokenizer loaded: 165168 tokens

🔍 Validating 12 speakers for Spark TTS...
  lipakshi → <|speaker_0|>: 7 tokens
  vardan → <|speaker_1|>: 7 tokens
  reet → <|speaker_2|>: 7 tokens
  Nandini → <|speaker_3|>: 7 tokens
  krishna → <|speaker_4|>: 7 tokens
  anika → <|speaker_5|>: 7 tokens
  adarsh → <|speaker_6|>: 7 tokens
  Nilay → <|speaker_7|>: 7 tokens
  Aarvi → <|speaker_8|>: 7 tokens
  Asha → <|speaker_9|>: 7 tokens
  Bittu → <|speaker_10|>: 8 tokens
  Mira → <|speaker_11|>: 8 tokens
✅ All 12 speakers validated with mapping

🔍 Validating 10 emotion tags for Spark TTS...
✅ All 10 emotion tags validated

📋 Sample emotion tags:
  [angry]: 1 tokens
  [curious]: 1 tokens
  [excited]: 1 tokens
  [giggle]: 1 tokens
  [laughs harder]: 1 tokens
  ...

🔧 Initializing vLLM engine with optimizations...
  - dtype: bfloat16
  - max_model_len: 4096
  - gpu_memory_utilization: 0.25
  - enable_prefix_caching: True
  - enforce_eager (disable CUDA graphs): False
INFO 02-18 00:22:59 [model.py:541] Resolved architecture: Qwen2ForCausalLM
INFO 02-18 00:22:59 [model.py:1561] Using max model len 4096
INFO 02-18 00:22:59 [scheduler.py:226] Chunked prefill is enabled with max_num_batched_tokens=4096.
INFO 02-18 00:22:59 [vllm.py:624] Asynchronous scheduling is enabled.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(EngineCore_DP0 pid=4098048) INFO 02-18 00:23:00 [core.py:96] Initializing a V1 LLM engine (v0.15.1) with config: model='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', speculative_config=None, tokenizer='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [4096], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 256, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': True}, 'local_cache_dir': None, 'static_all_moe_layers': []}
(EngineCore_DP0 pid=4098048) INFO 02-18 00:23:02 [parallel_state.py:1212] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://216.81.248.184:58071 backend=nccl
(EngineCore_DP0 pid=4098048) INFO 02-18 00:23:02 [parallel_state.py:1423] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
(EngineCore_DP0 pid=4098048) INFO 02-18 00:23:03 [gpu_model_runner.py:4033] Starting to load model /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM...