Output:
Chunk ID: d3e548
Wall time: 0.0513 seconds
Process exited with code 0
Original token count: 7445
Output:
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:28:_PROFILE_REQUEST_TIMING = "VEENA3_TIMING_LOGS"
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:33:def _timing_enabled() -> bool:
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:34:    """Return True when detailed request timing should be emitted."""
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:98:    gpu_memory_utilization: float = 0.25,
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:113:        gpu_memory_utilization: vLLM GPU memory fraction (default: 0.25)
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:154:            # Each engine gets gpu_memory_utilization / num_engines to share GPU fairly.
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:157:            per_engine_mem = gpu_memory_utilization / num_engines
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:163:                gpu_memory_per_engine=per_engine_mem,
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:170:                gpu_memory_utilization=gpu_memory_utilization,
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:394:        metrics_dict: Dictionary with timing metrics
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:421:    pipeline_timings = {}
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:449:        if _timing_enabled():
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:450:            audio_bytes, pipeline_timings = await runtime.pipeline.generate_speech_indic(
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:452:                return_timing=True,
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:465:        if pipeline_timings:
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:467:                "llm_token_total": pipeline_timings.get("token_total", 0),
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:468:                "llm_batch_count": pipeline_timings.get("batch_count", 0),
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:469:                "llm_batch_gpu_ms_total": pipeline_timings.get("llm_gpu_batch_ms_total", 0),
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:470:                "llm_batch_gpu_ms_min": pipeline_timings.get("llm_gpu_batch_ms_min", 0),
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:471:                "llm_batch_gpu_ms_max": pipeline_timings.get("llm_gpu_batch_ms_max", 0),
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:472:                "llm_batch_gpu_ms_p50": pipeline_timings.get("llm_gpu_batch_ms_p50", 0),
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:473:                "llm_decode_cpu_ms": pipeline_timings.get("decode_cpu_ms", 0),
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:474:                "llm_parse_ms": pipeline_timings.get("parse_ms", 0),
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:475:                "llm_wall_ms": pipeline_timings.get("llm_wall_ms", metrics["generation_ms"]),
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:476:                "ttfb_ms": int(pipeline_timings.get("ttfb_ms", metrics["ttfb_ms"])),
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:477:                "tokens_per_batch_min": pipeline_timings.get("tokens_per_batch_min"),
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:478:                "tokens_per_batch_max": pipeline_timings.get("tokens_per_batch_max"),
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:479:                "tokens_per_batch_p50": pipeline_timings.get("tokens_per_batch_p50"),
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:517:        if _timing_enabled():
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:524:        if _timing_enabled():
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:650:        if _timing_enabled():
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:669:            if _timing_enabled():
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:681:                    return_timing=True,
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:684:                    audio_bytes, chunk_timing = chunk_result
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:687:                    chunk_timing = {}
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:701:                chunk_timing = {}
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:715:            if _timing_enabled():
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:716:                audio_bytes, pipeline_timings = await runtime.pipeline.generate_speech_indic(
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:718:                    return_timing=True,
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:721:                chunk_timing = pipeline_timings
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:727:                chunk_timing = {}
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:733:        if chunk_timing:
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:735:                "llm_token_total": chunk_timing.get("token_total", 0),
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:736:                "llm_batch_count": chunk_timing.get("batch_count", 0),
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:737:                "llm_batch_gpu_ms_total": chunk_timing.get("llm_gpu_batch_ms_total", 0),
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:738:                "llm_batch_wall_ms_total": chunk_timing.get("llm_wall_ms", metrics["generation_ms"]),
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:739:                "llm_parse_ms": chunk_timing.get("parse_ms", 0),
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:740:                "llm_decode_cpu_ms": chunk_timing.get("decode_cpu_ms", 0),
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:741:                "llm_batch_wall_ms_min": chunk_timing.get("llm_batch_wall_ms_min", 0),
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:742:                "llm_batch_wall_ms_max": chunk_timing.get("llm_batch_wall_ms_max", 0),
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:743:                "llm_batch_gpu_ms_min": chunk_timing.get("llm_batch_gpu_ms_min", 0),
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:744:                "llm_batch_gpu_ms_max": chunk_timing.get("llm_batch_gpu_ms_max", 0),
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:745:                "llm_batch_gpu_ms_p50": chunk_timing.get("llm_batch_gpu_ms_p50", 0),
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:746:                "tokens_per_batch_min": chunk_timing.get("tokens_per_batch_min"),
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:747:                "tokens_per_batch_max": chunk_timing.get("tokens_per_batch_max"),
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:748:                "tokens_per_batch_p50": chunk_timing.get("tokens_per_batch_p50"),
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:749:                "llm_batch_tokens": chunk_timing.get("token_batches", []),
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:752:                metrics["chunks_processed"] = chunk_timing.get("chunks_processed", 0)
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:776:        if _timing_enabled():
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:828:        - Final yield: metrics_dict has final timing info
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:871:    if _timing_enabled():
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:974:        if _timing_enabled():
/home/ubuntu/veenaModal/veena3modal/core/pipeline.py:31:def _detailed_timing_enabled() -> bool:
/home/ubuntu/veenaModal/veena3modal/core/pipeline.py:36:      VEENA3_TIMING_LOGS=true|1
/home/ubuntu/veenaModal/veena3modal/core/pipeline.py:38:    return os.getenv("VEENA3_TIMING_LOGS", "").lower() in {"1", "true", "yes", "on"}
/home/ubuntu/veenaModal/veena3modal/core/pipeline.py:78:        return_timing: bool = False,
/home/ubuntu/veenaModal/veena3modal/core/pipeline.py:95:            return_timing: If true, returns (audio_bytes, metrics_dict)
/home/ubuntu/veenaModal/veena3modal/core/pipeline.py:99:            If return_timing is true, returns tuple(audio_bytes, metrics).
/home/ubuntu/veenaModal/veena3modal/core/pipeline.py:121:        emit_timing = return_timing or _detailed_timing_enabled()
/home/ubuntu/veenaModal/veena3modal/core/pipeline.py:128:        # Collect all outputs and track generation timing
/home/ubuntu/veenaModal/veena3modal/core/pipeline.py:132:        batch_gpu_ms: List[float] = []
/home/ubuntu/veenaModal/veena3modal/core/pipeline.py:144:                start_event = torch.cuda.Event(enable_timing=True)
/home/ubuntu/veenaModal/veena3modal/core/pipeline.py:145:                end_event = torch.cuda.Event(enable_timing=True)
/home/ubuntu/veenaModal/veena3modal/core/pipeline.py:157:            batch_gpu_ms_value: Optional[float] = None
/home/ubuntu/veenaModal/veena3modal/core/pipeline.py:161:                batch_gpu_ms_value = start_event.elapsed_time(end_event)
/home/ubuntu/veenaModal/veena3modal/core/pipeline.py:162:            if batch_gpu_ms_value is not None:
/home/ubuntu/veenaModal/veena3modal/core/pipeline.py:163:                batch_gpu_ms.append(batch_gpu_ms_value)
/home/ubuntu/veenaModal/veena3modal/core/pipeline.py:173:            if emit_timing:
/home/ubuntu/veenaModal/veena3modal/core/pipeline.py:175:                    "tts_llm_batch",
/home/ubuntu/veenaModal/veena3modal/core/pipeline.py:181:                        "llm_batch_wall_ms": round(batch_ms, 3),
/home/ubuntu/veenaModal/veena3modal/core/pipeline.py:182:                        "llm_batch_gpu_ms": round(batch_gpu_ms_value, 3)
/home/ubuntu/veenaModal/veena3modal/core/pipeline.py:183:                        if batch_gpu_ms_value is not None
/home/ubuntu/veenaModal/veena3modal/core/pipeline.py:190:            if emit_timing:
/home/ubuntu/veenaModal/veena3modal/core/pipeline.py:196:            if return_timing:
/home/ubuntu/veenaModal/veena3modal/core/pipeline.py:216:            if emit_timing:
/home/ubuntu/veenaModal/veena3modal/core/pipeline.py:239:            if return_timing:
/home/ubuntu/veenaModal/veena3modal/core/pipeline.py:256:            if return_timing:
/home/ubuntu/veenaModal/veena3modal/core/pipeline.py:271:            if emit_timing:
/home/ubuntu/veenaModal/veena3modal/core/pipeline.py:277:            if return_timing:
/home/ubuntu/veenaModal/veena3modal/core/pipeline.py:288:        timing_metrics: Dict[str, Any] = {
/home/ubuntu/veenaModal/veena3modal/core/pipeline.py:298:            "llm_gpu_batch_ms_total": int(sum(batch_gpu_ms)),
/home/ubuntu/veenaModal/veena3modal/core/pipeline.py:299:            "llm_gpu_batch_ms_min": int(min(batch_gpu_ms)) if batch_gpu_ms else 0,
/home/ubuntu/veenaModal/veena3modal/core/pipeline.py:300:            "llm_gpu_batch_ms_max": int(max(batch_gpu_ms)) if batch_gpu_ms else 0,
/home/ubuntu/veenaModal/veena3modal/core/pipeline.py:301:            "llm_gpu_batch_ms_p50": int(median(batch_gpu_ms)) if batch_gpu_ms else 0,
/home/ubuntu/veenaModal/veena3modal/core/pipeline.py:307:            timing_metrics["tokens_per_batch_min"] = min(token_batches)
/home/ubuntu/veenaModal/veena3modal/core/pipeline.py:308:            timing_metrics["tokens_per_batch_max"] = max(token_batches)
/home/ubuntu/veenaModal/veena3modal/core/pipeline.py:309:            timing_metrics["tokens_per_batch_p50"] = int(median(token_batches))
/home/ubuntu/veenaModal/veena3modal/core/pipeline.py:311:        if emit_timing:
/home/ubuntu/veenaModal/veena3modal/core/pipeline.py:312:            logger.info("tts_llm_complete", extra=timing_metrics)
/home/ubuntu/veenaModal/veena3modal/core/pipeline.py:317:        if return_timing:
/home/ubuntu/veenaModal/veena3modal/core/pipeline.py:318:            return wav_bytes, timing_metrics
/home/ubuntu/veenaModal/veena3modal/core/pipeline.py:331:        return_timing: bool = False,
/home/ubuntu/veenaModal/veena3modal/core/pipeline.py:352:            return_timing: If true, returns (audio_bytes, metrics_dict)
/home/ubuntu/veenaModal/veena3modal/core/pipeline.py:356:            If return_timing is true, returns tuple(audio_bytes, metrics).
/home/ubuntu/veenaModal/veena3modal/core/pipeline.py:368:            return_timing=return_timing,
/home/ubuntu/veenaModal/veena3modal/core/model_loader.py:123:        print(f"  - gpu_memory_utilization: {engine_kwargs_dict['gpu_memory_utilization']}")
/home/ubuntu/veenaModal/veena3modal/core/constants.py:136:# NOTE: gpu_memory_utilization adjusted based on available memory
/home/ubuntu/veenaModal/veena3modal/core/constants.py:143:# Reduced gpu_memory_utilization from 0.85 to 0.25
/home/ubuntu/veenaModal/veena3modal/core/constants.py:150:    "gpu_memory_utilization": 0.25,  # 0.5B model needs <2GB; 0.25 gives ~380 concurrent seqs
/home/ubuntu/veenaModal/veena3modal/core/dual_engine.py:5:With gpu_memory_utilization reduced to 0.25, the 0.5B model uses only ~20GB of 80GB VRAM.
/home/ubuntu/veenaModal/veena3modal/core/dual_engine.py:14:    - Each gets gpu_memory_utilization=0.15 (~100 concurrent seqs each)
/home/ubuntu/veenaModal/veena3modal/core/dual_engine.py:23:    - Run 4-8 separate processes, each with gpu_memory_utilization=0.05
/home/ubuntu/veenaModal/veena3modal/core/dual_engine.py:37:        gpu_memory_per_engine=0.15,
/home/ubuntu/veenaModal/veena3modal/core/dual_engine.py:58:    gpu_memory_utilization: float = 0.15
/home/ubuntu/veenaModal/veena3modal/core/dual_engine.py:81:    gpu_memory_per_engine: float = 0.15
/home/ubuntu/veenaModal/veena3modal/core/dual_engine.py:93:                gpu_memory_utilization=self.gpu_memory_per_engine,
/home/ubuntu/veenaModal/veena3modal/core/dual_engine.py:122:#     router = DualEngineRouter(model_path=model_path, num_engines=2, gpu_memory_per_engine=0.15)
/home/ubuntu/veenaModal/veena3modal/core/multi_engine.py:14:    model = create_multi_engine_model(model_path, num_engines=3, gpu_memory_per_engine=0.12)
/home/ubuntu/veenaModal/veena3modal/core/multi_engine.py:98:    gpu_memory_per_engine: float = 0.12,
/home/ubuntu/veenaModal/veena3modal/core/multi_engine.py:113:        gpu_memory_per_engine: GPU memory fraction per engine
/home/ubuntu/veenaModal/veena3modal/core/multi_engine.py:120:    logger.info(f"Creating multi-engine model: {num_engines} engines x {gpu_memory_per_engine:.0%} GPU each")
/home/ubuntu/veenaModal/veena3modal/core/multi_engine.py:128:            gpu_memory_utilization=gpu_memory_per_engine,
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:52:def _detailed_timing_enabled() -> bool:
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:53:    """Enable per-batch token/decode timing logs."""
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:54:    return os.getenv("VEENA3_TIMING_LOGS", "").lower() in {"1", "true", "yes", "on"}
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:261:        emit_timing = _detailed_timing_enabled()
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:321:        batch_gpu_ms: List[float] = []
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:325:        decode_gpu_ms = 0.0
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:327:        decode_gpu_samples: List[float] = []
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:335:        # Track timing
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:375:            if emit_timing:
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:377:                    "tts_stream_llm_batch",
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:383:                        "llm_batch_wall_ms": int(batch_wall_ms_value),
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:384:                        "llm_batch_gpu_ms": int(batch_gpu_ms[-1]) if batch_gpu_ms else None,
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:418:                        decode_gpu_ms_this = None
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:420:                        if emit_timing and torch.cuda.is_available():
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:422:                            decode_start_event = torch.cuda.Event(enable_timing=True)
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:423:                            decode_end_event = torch.cuda.Event(enable_timing=True)
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:436:                        if emit_timing and torch.cuda.is_available():
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:439:                            decode_gpu_ms_this = decode_start_event.elapsed_time(decode_end_event)
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:440:                            decode_gpu_ms += decode_gpu_ms_this
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:441:                            decode_gpu_samples.append(decode_gpu_ms_this)
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:442:                            batch_gpu_ms.append(decode_gpu_ms_this)
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:481:                            if emit_timing:
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:489:                                        "decode_gpu_ms": int(decode_gpu_ms_this)
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:490:                                        if decode_gpu_ms_this is not None else None,
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:552:                "llm_batch_count": len(batch_wall_ms),
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:553:                "llm_batch_tokens": batch_token_counts,
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:554:                "llm_batch_wall_ms_total": int(sum(batch_wall_ms)),
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:555:                "llm_batch_wall_ms_min": int(min(batch_wall_ms)) if batch_wall_ms else 0,
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:556:                "llm_batch_wall_ms_max": int(max(batch_wall_ms)) if batch_wall_ms else 0,
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:557:                "llm_batch_wall_ms_p50": int(statistics.median(batch_wall_ms)) if batch_wall_ms else 0,
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:558:                "llm_batch_gpu_ms_total": int(sum(batch_gpu_ms)),
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:559:                "llm_batch_gpu_ms_min": int(min(batch_gpu_ms)) if batch_gpu_ms else 0,
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:560:                "llm_batch_gpu_ms_max": int(max(batch_gpu_ms)) if batch_gpu_ms else 0,
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:561:                "llm_batch_gpu_ms_p50": int(statistics.median(batch_gpu_ms)) if batch_gpu_ms else 0,
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:569:                "llm_decode_gpu_ms_total": int(decode_gpu_ms),
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:570:                "llm_decode_gpu_ms_min": int(min(decode_gpu_samples)) if decode_gpu_samples else 0,
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:571:                "llm_decode_gpu_ms_max": int(max(decode_gpu_samples)) if decode_gpu_samples else 0,
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:581:        if emit_timing:
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:635:        emit_timing = _detailed_timing_enabled()
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:693:        batch_gpu_ms: List[float] = []
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:702:        decode_gpu_ms = 0.0
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:704:        decode_gpu_samples: List[float] = []
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:731:            if emit_timing:
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:733:                    "tts_stream_llm_batch",
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:739:                        "llm_batch_wall_ms": int(batch_wall_ms_value),
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:740:                        "llm_batch_gpu_ms": int(batch_gpu_ms[-1]) if batch_gpu_ms else None,
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:759:                    decode_gpu_ms_this = None
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:761:                    if emit_timing and torch.cuda.is_available():
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:763:                        decode_start_event = torch.cuda.Event(enable_timing=True)
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:764:                        decode_end_event = torch.cuda.Event(enable_timing=True)
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:787:                    if emit_timing and torch.cuda.is_available():
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:790:                        decode_gpu_ms_this = decode_start_event.elapsed_time(decode_end_event)
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:791:                        decode_gpu_ms += decode_gpu_ms_this
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:792:                        decode_gpu_samples.append(decode_gpu_ms_this)
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:793:                        batch_gpu_ms.append(decode_gpu_ms_this)
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:817:                        if emit_timing:
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:825:                                    "decode_gpu_ms": int(decode_gpu_ms_this)
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:826:                                    if decode_gpu_ms_this is not None else None,
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:870:                if emit_timing:
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:878:                            "decode_gpu_ms": None,
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:910:                "llm_batch_count": len(batch_wall_ms),
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:911:                "llm_batch_tokens": batch_token_counts,
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:912:                "llm_batch_wall_ms_total": int(sum(batch_wall_ms)),
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:913:                "llm_batch_wall_ms_min": int(min(batch_wall_ms)) if batch_wall_ms else 0,
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:914:                "llm_batch_wall_ms_max": int(max(batch_wall_ms)) if batch_wall_ms else 0,
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:915:                "llm_batch_wall_ms_p50": int(statistics.median(batch_wall_ms)) if batch_wall_ms else 0,
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:916:                "llm_batch_gpu_ms_total": int(sum(batch_gpu_ms)),
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:917:                "llm_batch_gpu_ms_min": int(min(batch_gpu_ms)) if batch_gpu_ms else 0,
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:918:                "llm_batch_gpu_ms_max": int(max(batch_gpu_ms)) if batch_gpu_ms else 0,
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:919:                "llm_batch_gpu_ms_p50": int(statistics.median(batch_gpu_ms)) if batch_gpu_ms else 0,
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:927:                "llm_decode_gpu_ms_total": int(decode_gpu_ms),
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:928:                "llm_decode_gpu_ms_min": int(min(decode_gpu_samples)) if decode_gpu_samples else 0,
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:929:                "llm_decode_gpu_ms_max": int(max(decode_gpu_samples)) if decode_gpu_samples else 0,
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:930:                "llm_decode_gpu_ms_p50": int(statistics.median(decode_gpu_samples)) if decode_gpu_samples else 0,
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:997:        emit_timing = _detailed_timing_enabled()
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:1064:        batch_gpu_ms: List[float] = []
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:1069:        decode_gpu_ms = 0.0
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:1071:        decode_gpu_samples: List[float] = []
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:1100:            if emit_timing:
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:1102:                    "tts_stream_llm_batch",
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:1108:                        "llm_batch_wall_ms": int(batch_wall_ms_value),
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:1109:                        "llm_batch_gpu_ms": int(batch_gpu_ms[-1]) if batch_gpu_ms else None,
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:1121:                    decode_gpu_ms_this = None
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:1123:                    if emit_timing and torch.cuda.is_available():
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:1125:                        decode_start_event = torch.cuda.Event(enable_timing=True)
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:1126:                        decode_end_event = torch.cuda.Event(enable_timing=True)
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:1150:                    if emit_timing and torch.cuda.is_available():
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:1153:                        decode_gpu_ms_this = decode_start_event.elapsed_time(decode_end_event)
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:1154:                        decode_gpu_ms += decode_gpu_ms_this
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:1155:                        decode_gpu_samples.append(decode_gpu_ms_this)
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:1156:                        batch_gpu_ms.append(decode_gpu_ms_this)
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:1180:                        if emit_timing:
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:1188:                                    "decode_gpu_ms": int(decode_gpu_ms_this)
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:1189:                                    if decode_gpu_ms_this is not None else None,
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:1233:                        if emit_timing:
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:1241:                                    "decode_gpu_ms": None,
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:1272:                "llm_batch_count": len(batch_wall_ms),
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:1273:                "llm_batch_tokens": batch_token_counts,
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:1274:                "llm_batch_wall_ms_total": int(sum(batch_wall_ms)),
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:1275:                "llm_batch_wall_ms_min": int(min(batch_wall_ms)) if batch_wall_ms else 0,
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:1276:                "llm_batch_wall_ms_max": int(max(batch_wall_ms)) if batch_wall_ms else 0,
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:1277:                "llm_batch_wall_ms_p50": int(statistics.median(batch_wall_ms)) if batch_wall_ms else 0,
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:1278:                "llm_batch_gpu_ms_total": int(sum(batch_gpu_ms)),
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:1279:                "llm_batch_gpu_ms_min": int(min(batch_gpu_ms)) if batch_gpu_ms else 0,
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:1280:                "llm_batch_gpu_ms_max": int(max(batch_gpu_ms)) if batch_gpu_ms else 0,
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:1281:                "llm_batch_gpu_ms_p50": int(statistics.median(batch_gpu_ms)) if batch_gpu_ms else 0,
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:1289:                "llm_decode_gpu_ms_total": int(decode_gpu_ms),
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:1290:                "llm_decode_gpu_ms_min": int(min(decode_gpu_samples)) if decode_gpu_samples else 0,
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:1291:                "llm_decode_gpu_ms_max": int(max(decode_gpu_samples)) if decode_gpu_samples else 0,
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:1292:                "llm_decode_gpu_ms_p50": int(statistics.median(decode_gpu_samples)) if decode_gpu_samples else 0,


# Collaboration Mode: Default

You are now in Default mode. Any previous instructions for other modes (e.g. Plan mode) are no longer active.

Your active mode changes only when new developer instructions with a different `<collaboration_mode>...</collaboration_mode>` change it; user requests or tool descriptions do not change mode by themselves. Known mode names are Default and Plan.

## request_user_input availability

The `request_user_input` tool is unavailable in Default mode. If you call it while in Default mode, it will return an error.

If a decision is necessary and cannot be discovered from local context, ask the user directly. However, in Default mode you should strongly prefer executing the user's request rather than stopping to ask questions.

Summary: auto

## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) When `SKILL.md` references relative paths (e.g., `scripts/foo.py`), resolve them relative to the skill directory listed above first, and only consider other paths if needed.
  3) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  4) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  5) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.


ls -1 /home/ubuntu/veenaModal/models | sed -n '1,120p'


Output:
Chunk ID: 0fbf63
Wall time: 0.0506 seconds
Process exited with code 0
Original token count: 5
Output:
spark_tts_4speaker


Summary: auto


ls -1 /home/ubuntu/veenaModal/models/spark_tts_4speaker | sed -n '1,80p'


Output:
Chunk ID: dff676
Wall time: 0.0513 seconds
Process exited with code 0
Original token count: 16
Output:
BiCodec
LLM
README.md
config.yaml
src
wav2vec2-large-xlsr-53


I’m starting the local server with optimized-path timing enabled (`VEENA3_TIMING_LOGS=true`) and will then run the local stress script with increasing concurrency and stream the result set.


cd /home/ubuntu/veenaModal && source venv/bin/activate && VEENA3_TIMING_LOGS=true python3 -m veena3modal.local_server --skip-download --model-path /home/ubuntu/veenaModal/models/spark_tts_4speaker --port 8000 --gpu-memory 0.25 --num-engines 1 --log-level INFO


Output:
Chunk ID: 394485
Wall time: 1.0007 seconds
Process running with session ID 19889
Original token count: 454
Output:
2026-02-17 00:14:22 | INFO     | veena3.local | Auth bypass enabled (local mode). Use --auth to require API keys.
2026-02-17 00:14:22 | INFO     | veena3.local | Resolved model paths:
2026-02-17 00:14:22 | INFO     | veena3.local |   LLM (vLLM):    /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
2026-02-17 00:14:22 | INFO     | veena3.local |   BiCodec:       /home/ubuntu/veenaModal/models/spark_tts_4speaker
2026-02-17 00:14:22 | INFO     | veena3.local | ============================================================
2026-02-17 00:14:22 | INFO     | veena3.local | Initializing Veena3 TTS Runtime (local)
2026-02-17 00:14:22 | INFO     | veena3.local |   Model:       /home/ubuntu/veenaModal/models/spark_tts_4speaker
2026-02-17 00:14:22 | INFO     | veena3.local |   LLM path:    /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
2026-02-17 00:14:22 | INFO     | veena3.local |   BiCodec:     /home/ubuntu/veenaModal/models/spark_tts_4speaker
2026-02-17 00:14:22 | INFO     | veena3.local |   Device:      cuda
2026-02-17 00:14:22 | INFO     | veena3.local |   GPU Memory:  25% (total)
2026-02-17 00:14:22 | INFO     | veena3.local |   Engines:     1
2026-02-17 00:14:22 | INFO     | veena3.local |   SR:          disabled
2026-02-17 00:14:22 | INFO     | veena3.local | ============================================================
2026-02-17 00:14:22 | INFO     | veena3modal.services.tts_runtime | Initializing TTS runtime...
2026-02-17 00:14:22 | INFO     | veena3modal.services.tts_runtime |   Model path: /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
2026-02-17 00:14:22 | INFO     | veena3modal.services.tts_runtime |   BiCodec path: /home/ubuntu/veenaModal/models/spark_tts_4speaker
2026-02-17 00:14:22 | INFO     | veena3modal.services.tts_runtime |   SR enabled: False, path: None


Output:
Chunk ID: 6e47f7
Wall time: 5.0036 seconds
Process running with session ID 19889
Original token count: 0
Output:


Output:
Chunk ID: 53fe14
Wall time: 5.0005 seconds
Process running with session ID 19889
Original token count: 1283
Output:
Total output lines: 49

2026-02-17 00:14:31 | INFO     | veena3modal.services.tts_runtime | Loading SparkTTS model with vLLM...
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
🚀 Initializing Spark TTS Model
📁 Model path: /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
🗣  Speakers: 12 predefined (lipakshi, vardan, reet, Nandini, krishna, anika, adarsh, Nilay, Aarvi, Asha, Bittu, Mira)
🎭 Emotions: 10 tags in [bracket] format

📝 Loading tokenizer from model...
✅ Tokenizer loaded: 165168 tokens

🔍 Validating 12 speakers for Spark TTS...
  lipakshi → <|speaker_0|>: 7 tokens
  vardan → <|speaker_1|>: 7 tokens
  reet → <|speaker_2|>: 7 tokens
  Nandini → <|speaker_3|>: 7 tokens
  krishna → <|speaker_4|>: 7 tokens
  anika → <|speaker_5|>: 7 tokens
  adarsh → <|speaker_6|>: 7 tokens
  Nilay → <|speaker_7|>: 7 tokens
  Aarvi → <|speaker_8|>: 7 tokens
  Asha → <|speaker_9|>: 7 tokens
  Bittu → <|speaker_10|>: 8 tokens
  Mira → <|speaker_11|>: 8 tokens
✅ All 12 speakers validated with mapping

🔍 Validating 10 emotion tags for Spark TTS...
✅ All 10 emotion tags validated

📋 Sample emotion tags:
  [angry]: 1 tokens
  [curious]: 1 tokens
  [excited]: 1 tokens
  [giggle]: 1 tokens
  [laughs harder]: 1 tokens
  ...

🔧 Initializing vLLM engine with optimizations...
  - dtype: bfloat16
  - max_model_len: 4096
  - gpu_memory_utilization: 0.25
  - enable_prefix_caching: True
  - enforce_eager (disable CUDA graphs): False
INFO 02-17 00:14:32 [model.py:541] Resolved architecture: Qwen2ForCausalLM
INFO 02-17 00:14:32 [model.py:1561] Using max model len 4096
INFO 02-17 00:14:32 [scheduler.py:226] Chunked prefill is enabled with max_num_batched_tokens=4096.
INFO 02-17 00:14:32 [vllm.py:624] Asynchronous scheduling is enabled.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(EngineCore_DP0 pid=3698538) INFO 02-17 00:14:33 [core.py:96] Initializing a V1 LLM engine (v0.15.1) with config: model='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', speculative_config=None, tokenizer='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_r…83 tokens truncated…tputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [4096], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 256, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': True}, 'local_cache_dir': None, 'static_all_moe_layers': []}


Output:
Chunk ID: 3d7f29
Wall time: 5.0013 seconds
Process running with session ID 19889
Original token count: 290
Output:
(EngineCore_DP0 pid=3698538) INFO 02-17 00:14:36 [parallel_state.py:1212] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://216.81.248.184:57505 backend=nccl
(EngineCore_DP0 pid=3698538) INFO 02-17 00:14:36 [parallel_state.py:1423] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
(EngineCore_DP0 pid=3698538) INFO 02-17 00:14:37 [gpu_model_runner.py:4033] Starting to load model /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM...
(EngineCore_DP0 pid=3698538) INFO 02-17 00:14:38 [cuda.py:364] Using FLASH_ATTN attention backend out of potential backends: ('FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION')
(EngineCore_DP0 pid=3698538)
Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
(EngineCore_DP0 pid=3698538)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:03<00:00,  3.41s/it]
(EngineCore_DP0 pid=3698538)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:03<00:00,  3.41s/it]
(EngineCore_DP0 pid=3698538)
(EngineCore_DP0 pid=3698538) INFO 02-17 00:14:42 [default_loader.py:291] Loading weights took 3.49 seconds


Output:
Chunk ID: 532725
Wall time: 5.0015 seconds
Process running with session ID 19889
Original token count: 33
Output:
(EngineCore_DP0 pid=3698538) INFO 02-17 00:14:43 [gpu_model_runner.py:4130] Model loading took 0.95 GiB memory and 4.601286 seconds


Output:
Chunk ID: a104b6
Wall time: 5.0005 seconds
Process running with session ID 19889
Original token count: 457
Output:
(EngineCore_DP0 pid=3698538) INFO 02-17 00:14:49 [backends.py:812] Using cache directory: /home/ubuntu/.cache/vllm/torch_compile_cache/c3bd5d9fb9/rank_0_0/backbone for vLLM's torch.compile
(EngineCore_DP0 pid=3698538) INFO 02-17 00:14:49 [backends.py:872] Dynamo bytecode transform time: 5.63 s
(EngineCore_DP0 pid=3698538) INFO 02-17 00:14:52 [backends.py:267] Directly load the compiled graph(s) for compile range (1, 4096) from the cache, took 1.196 s
(EngineCore_DP0 pid=3698538) INFO 02-17 00:14:52 [monitor.py:34] torch.compile takes 6.83 s in total
(EngineCore_DP0 pid=3698538) INFO 02-17 00:14:53 [gpu_worker.py:356] Available KV cache memory: 18.06 GiB
(EngineCore_DP0 pid=3698538) INFO 02-17 00:14:53 [kv_cache_utils.py:1307] GPU KV cache size: 1,577,712 tokens
(EngineCore_DP0 pid=3698538) INFO 02-17 00:14:53 [kv_cache_utils.py:1312] Maximum concurrency for 4,096 tokens per request: 385.18x
(EngineCore_DP0 pid=3698538)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   0%|          | 0/35 [00:00<?, ?it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   9%|▊         | 3/35 [00:00<00:01, 26.96it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  20%|██        | 7/35 [00:00<00:00, 30.16it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  31%|███▏      | 11/35 [00:00<00:00, 31.58it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  43%|████▎     | 15/35 [00:00<00:00, 33.74it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  54%|█████▍    | 19/35 [00:00<00:00, 31.98it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  66%|██████▌   | 23/35 [00:00<00:00, 33.94it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  77%|███████▋  | 27/35 [00:00<00:00, 33.29it/s]


Output:
Chunk ID: a2daae
Wall time: 5.0031 seconds
Process running with session ID 19889
Original token count: 443
Output:

Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  89%|████████▊ | 31/35 [00:01<00:00, 28.61it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  97%|█████████▋| 34/35 [00:01<00:00, 24.12it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 35/35 [00:01<00:00, 27.84it/s]
(EngineCore_DP0 pid=3698538)
Capturing CUDA graphs (decode, FULL):   0%|          | 0/19 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL):   5%|▌         | 1/19 [00:00<00:01,  9.57it/s]
Capturing CUDA graphs (decode, FULL):  21%|██        | 4/19 [00:00<00:00, 16.63it/s]
Capturing CUDA graphs (decode, FULL):  42%|████▏     | 8/19 [00:00<00:00, 23.92it/s]
Capturing CUDA graphs (decode, FULL):  63%|██████▎   | 12/19 [00:00<00:00, 28.14it/s]
Capturing CUDA graphs (decode, FULL):  84%|████████▍ | 16/19 [00:00<00:00, 30.99it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 27.34it/s]
(EngineCore_DP0 pid=3698538) INFO 02-17 00:14:56 [gpu_model_runner.py:5063] Graph capturing finished in 3 secs, took 0.28 GiB
(EngineCore_DP0 pid=3698538) INFO 02-17 00:14:56 [core.py:272] init engine (profile, create kv cache, warmup model) took 13.71 seconds
2026-02-17 00:14:58 | INFO     | veena3modal.services.tts_runtime | Initializing prompt builder...
2026-02-17 00:14:58 | INFO     | veena3modal.services.tts_runtime | Initializing BiCodec decoder...
/home/ubuntu/veenaModal/venv/lib/python3.10/site-packages/torch/nn/utils/weight_norm.py:144: FutureWarning: `torch.nn.utils.weight_norm` is deprecated in favor of `torch.nn.utils.parametrizations.weight_norm`.
  WeightNorm.apply(module, name, dim)


Local runtime initialization has progressed through model compile and KV-cache setup and is now initializing prompt builder + BiCodec. I’ll wait for the server to finish binding before launching the benchmark run.


Output:
Chunk ID: e6139a
Wall time: 5.0010 seconds
Process running with session ID 19889
Original token count: 0
Output:


Output:
Chunk ID: 3f11b4
Wall time: 5.0007 seconds
Process running with session ID 19889
Original token count: 201
Output:
2026-02-17 00:15:13 | INFO     | veena3modal.services.tts_runtime | Initializing TTS pipeline...
2026-02-17 00:15:13 | INFO     | veena3modal.services.tts_runtime | Initializing streaming pipeline...
2026-02-17 00:15:13 | INFO     | veena3modal.services.tts_runtime | ✅ TTS runtime initialized in 51608ms
2026-02-17 00:15:13 | INFO     | veena3modal.services.tts_runtime |    Model version: LLM
2026-02-17 00:15:13 | INFO     | veena3modal.services.tts_runtime | Pre-computing global tokens for all speakers...
2026-02-17 00:15:14 | INFO     | veena3modal.services.tts_runtime |   Cached globals for lipakshi
2026-02-17 00:15:15 | INFO     | veena3modal.services.tts_runtime |   Cached globals for vardan
2026-02-17 00:15:15 | INFO     | veena3modal.services.tts_runtime |   Cached globals for reet


Output:
Chunk ID: 02ac89
Wall time: 5.0013 seconds
Process running with session ID 19889
Original token count: 813
Output:
2026-02-17 00:15:16 | INFO     | veena3modal.services.tts_runtime |   Cached globals for Nandini
2026-02-17 00:15:17 | INFO     | veena3modal.services.tts_runtime |   Cached globals for krishna
2026-02-17 00:15:17 | INFO     | veena3modal.services.tts_runtime |   Cached globals for anika
2026-02-17 00:15:18 | INFO     | veena3modal.services.tts_runtime |   Cached globals for adarsh
2026-02-17 00:15:18 | INFO     | veena3modal.services.tts_runtime |   Cached globals for Nilay
2026-02-17 00:15:19 | INFO     | veena3modal.services.tts_runtime |   Cached globals for Aarvi
2026-02-17 00:15:20 | INFO     | veena3modal.services.tts_runtime |   Cached globals for Asha
2026-02-17 00:15:20 | INFO     | veena3modal.services.tts_runtime |   Cached globals for Bittu
2026-02-17 00:15:21 | INFO     | veena3modal.services.tts_runtime |   Cached globals for Mira
2026-02-17 00:15:21 | INFO     | veena3modal.services.tts_runtime | Speaker globals cached: 12/12 speakers in 7438ms
2026-02-17 00:15:21 | INFO     | veena3.local | Runtime initialized in 59.1s
2026-02-17 00:15:21 | INFO     | veena3.local |   Model version: LLM
2026-02-17 00:15:21 | INFO     | veena3.local |   Streaming: available
2026-02-17 00:15:21 | INFO     | veena3.local |   Super-resolution: disabled
2026-02-17 00:15:21 | INFO     | veena3.local |
2026-02-17 00:15:21 | INFO     | veena3.local | ============================================================
2026-02-17 00:15:21 | INFO     | veena3.local | Veena3 TTS Local Server starting
2026-02-17 00:15:21 | INFO     | veena3.local |   URL: http://0.0.0.0:8000
2026-02-17 00:15:21 | INFO     | veena3.local |   Health: http://0.0.0.0:8000/v1/tts/health
2026-02-17 00:15:21 | INFO     | veena3.local |   Generate: POST http://0.0.0.0:8000/v1/tts/generate
2026-02-17 00:15:21 | INFO     | veena3.local |   Metrics: http://0.0.0.0:8000/v1/tts/metrics
2026-02-17 00:15:21 | INFO     | veena3.local |   Docs: http://0.0.0.0:8000/docs
2026-02-17 00:15:21 | INFO     | veena3.local | ============================================================
✅ vLLM engine initialized successfully!

🎉 Spark TTS Model ready for inference!

🎵 Loading BiCodec audio tokenizer from /home/ubuntu/veenaModal/models/spark_tts_4speaker...
   Device: cuda
Missing tensor: mel_transformer.spectrogram.window
Missing tensor: mel_transformer.mel_scale.fb
✅ BiCodec decoder initialized (sample rate: 16000Hz)
🚀 SparkTTSPipeline initialized
🌊 Veena3SlidingWindowPipeline initialized (sliding window: 28 tokens)
{"timestamp": "2026-02-17T00:15:21.431730+00:00", "level": "INFO", "logger": "veena3modal.api.websocket_handler", "message": "WebSocket route /v1/tts/ws registered"}
2026-02-17 00:15:21 | INFO     | veena3modal.api.websocket_handler | WebSocket route /v1/tts/ws registered
2026-02-17 00:15:21 | INFO     | veena3modal.api.fastapi_app | WebSocket support enabled at /v1/tts/ws
{"timestamp": "2026-02-17T00:15:21.431906+00:00", "level": "INFO", "logger": "veena3modal.api.fastapi_app", "message": "WebSocket support enabled at /v1/tts/ws"}
INFO:     Started server process [3698377]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)


cd /home/ubuntu/veenaModal && source venv/bin/activate && python3 scripts/stress_test_optimized_local.py --levels 1,5,10 --text short --output /home/ubuntu/veenaModal/stress_results_opt_local.json


Output:
Chunk ID: be80ce
Wall time: 1.0043 seconds
Process running with session ID 84583
Original token count: 0
Output:


I’ve started the benchmark run against the live local server with short text and multiple concurrency levels. I’m collecting logs now and will stop when the complete run/JSON export is ready.


Output:
Chunk ID: 0e7b71
Wall time: 5.0009 seconds
Process running with session ID 84583
Original token count: 0
Output:


Output:
Chunk ID: 0f2a68
Wall time: 0.0538 seconds
Process exited with code 0
Original token count: 669
Output:
  Health: healthy
  Version: LLM
  GPU: True

Warming up (1 request, non-streaming)...
  warmup ok: 0%

========================================================================
  1C-SHORT-NON-STREAMING
  10 requests, 1 concurrent, non-streaming, chunking=True
========================================================================
  Success:     0/10 (0%)
  Wall time:   0.10s
  Throughput:  102.98 req/s
  Audio total: 0.00s
  Eff. RTF:    0.000

  Errors:
    [10x] {"error":{"code":"INTERNAL_ERROR","message":"veena3modal.core.pipeline.SparkTTSPipeline.generate_spe
  cooldown 3s...

========================================================================
  5C-SHORT-NON-STREAMING
  15 requests, 5 concurrent, non-streaming, chunking=True
========================================================================
  Success:     0/15 (0%)
  Wall time:   0.13s
  Throughput:  117.95 req/s
  Audio total: 0.00s
  Eff. RTF:    0.000

  GPU:
    Memory: 25803MB avg, 25803MB peak / 81920MB
    Util:   35% avg, 35% peak
    Temp:   29C avg, 29C peak

  Errors:
    [15x] {"error":{"code":"INTERNAL_ERROR","message":"veena3modal.core.pipeline.SparkTTSPipeline.generate_spe
  cooldown 3s...

========================================================================
  10C-SHORT-NON-STREAMING
  30 requests, 10 concurrent, non-streaming, chunking=True
========================================================================
  Success:     0/30 (0%)
  Wall time:   0.19s
  Throughput:  155.05 req/s
  Audio total: 0.00s
  Eff. RTF:    0.000

  GPU:
    Memory: 25803MB avg, 25803MB peak / 81920MB
    Util:   32% avg, 32% peak
    Temp:   29C avg, 29C peak

  Errors:
    [30x] {"error":{"code":"INTERNAL_ERROR","message":"veena3modal.core.pipeline.SparkTTSPipeline.generate_spe

==============================================================================================================
  STRESS BENCHMARK SUMMARY
==============================================================================================================
Level                        OK%     RPS     p50     p95  GPU%   GPUmem    RTF Inflight
--------------------------------------------------------------------------------------------------------------
1c-short-non-streaming       0%  102.98     ---     ---    0%       0MB  0.000        0
5c-short-non-streaming       0%  117.95     ---     ---   35%   25803MB  0.000        0
10c-short-non-streaming      0%  155.05     ---     ---   32%   25803MB  0.000        0
==============================================================================================================

Results written to /home/ubuntu/veenaModal/stress_results_opt_local.json


Output:
Chunk ID: 6c9d4e
Wall time: 5.0029 seconds
Process running with session ID 19889
Original token count: 40871
Output:
Total output lines: 897

INFO:     127.0.0.1:45398 - "GET /v1/tts/health HTTP/1.1" 200 OK
{"timestamp": "2026-02-17T00:15:24.170184+00:00", "level": "INFO", "logger": "veena3modal.tts", "message": "request_received", "request_id": "8abf79ab-9245-45fa-9a40-6aa33abc73ac", "event": "request_received", "text_length": 28, "speaker": "lipakshi", "stream": false, "format": "wav"}
2026-02-17 00:15:24 | INFO     | veena3modal.tts | request_received
2026-02-17 00:15:24 | INFO     | veena3modal.services.tts_runtime | tts_runtime_chunking_start
2026-02-17 00:15:24 | INFO     | veena3modal.processing.long_text_processor | LongTextProcessor initialized
2026-02-17 00:15:24 | ERROR    | veena3modal.services.tts_runtime | Chunked speech generation failed: veena3modal.core.pipeline.SparkTTSPipeline.generate_speech_indic() got multiple values for keyword argument 'request_id'
{"timestamp": "2026-02-17T00:15:24.176613+00:00", "level": "ERROR", "logger": "veena3modal.api.fastapi_app", "message": "TTS generation error: veena3modal.core.pipeline.SparkTTSPipeline.generate_speech_indic() got multiple values for keyword argument 'request_id'", "request_id": "8abf79ab-9245-45fa-9a40-6aa33abc73ac", "exception": "Traceback (most recent call last):\n  File \"/home/ubuntu/veenaModal/veena3modal/api/fastapi_app.py\", line 446, in tts_generate\n    audio_bytes, metrics = await tts_runtime.generate_speech_chunked(\n  File \"/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py\", line 716, in generate_speech_chunked\n    audio_bytes, pipeline_timings = await runtime.pipeline.generate_speech_indic(\nTypeError: veena3modal.core.pipeline.SparkTTSPipeline.generate_speech_indic() got multiple values for keyword argument 'request_id'\n"}
2026-02-17 00:15:24 | ERROR    | veena3modal.api.fastapi_app | TTS generation error: veena3modal.core.pipeline.SparkTTSPipeline.generate_speech_indic() got multiple values for keyword argument 'request_id'
Traceback (most recent call last):
  File "/home/ubuntu/veenaModal/veena3modal/api/fastapi_app.py", line 446, in tts_generate
    audio_bytes, metrics = await tts_runtime.generate_speech_chunked(
  File "/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py", line 716, in generate_speech_chunked
    audio_bytes, pipeline_timings = await runtime.pipeline.generate_speech_indic(
TypeError: veena3modal.core.pipeline.SparkTTSPipeline.generate_speech_indic() got multiple values for keyword argument 'request_id'
{"timestamp": "2026-02-17T00:15:24.176949+00:00", "level": "ERROR", "logger": "veena3modal.tts", "message": "request_failed", "request_id": "8abf79ab-9245-45fa-9a40-6aa33abc73ac", "event": "request_failed", "status_code": 500, "error_code": "INTERNAL_ERROR", "error_message": "veena3modal.core.pipeline.SparkTTSPipeline.generate_speech_indic() got multiple values for keyword argument 'request_id'"}
2026-02-17 00:15:24 | ERROR    | veena3modal.tts | request_failed
INFO:     127.0.0.1:45406 - "POST /v1/tts/generate HTTP/1.1" 500 Internal Server Error
{"timestamp": "2026-02-17T00:15:24.253692+00:00", "level": "INFO", "logger": "veena3modal.tts", "message": "request_received", "request_id": "fb212d46-65e4-4200-97dc-d74c8b9c6bfd", "event": "request_received", "text_length": 28, "speaker": "lipakshi", "stream": false, "format": "wav"}
2026-02-17 00:15:24 | INFO     | veena3modal.tts | request_received
2026-02-17 00:15:24 | INFO     | veena3modal.services.tts_runtime | tts_runtime_chunking_start
2026-02-17 00:15:24 | INFO     | veena3modal.processing.long_text_processor | LongTextProcessor initialized
2026-02-17 00:15:24 | ERROR    | veena3modal.services.tts_runtime | Chunked speech generation failed: veena3modal.core.pipeline.SparkTTSPipeline.generate_speech_indic() got multiple values for keyword argument 'request_id'
{"timestamp": "2026-02-17T00:15:24.254179+00:00", "level": "ERROR", "logger": "veena3modal.api.fastapi_app", "message": "TTS generation error: veena3modal.core.pipeline.SparkTTSPipeline.generate_speech_indic() got multiple values for keyword argument 'request_id'", "request_id": "fb212d46-65e4-4200-97dc-d74c8b9c6bfd", "exception": "Traceback (most recent call last):\n  File \"/home/ubuntu/veenaModal/veena3modal/api/fastapi_app.py\", line 446, in tts_generate\n    audio_bytes, metrics = await tts_runtime.generate_speech_chunked(\n  File \"/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py\", line 716, in generate_speech_chunked\n    audio_bytes, pipeline_timings = await runtime.pipeline.generate_speech_indic(\nTypeError: veena3modal.core.pipeline.SparkTTSPipeline.generate_speech_indic() got multiple values for keyword argument 'request_id'\n"}
2026-02-17 00:15:24 | ERROR    | veena3modal.api.fastapi_app | TTS generation error: veena3modal.core.pipeline.SparkTTSPipeline.generate_speech_indic() got multiple values for keyword argument 'request_id'
Traceback (most recent call last):
  File "/home/ubuntu/veenaModal/veena3modal/api/fastapi_app.py", line 446, in tts_generate
    audio_bytes, metrics = await tts_runtime.generate_speech_chunked(
  File "/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py", line 716, in generate_speech_chunked
    audio_bytes, pipeline_timings = await runtime.pipeline.generate_speech_indic(
TypeError: veena3modal.core.pipeline.SparkTTSPipeline.generate_speech_indic() got multiple values for keyword argument 'request_id'
2026-02-17 00:15:24 | ERROR    | veena3modal.tts | request_failed
{"timestamp": "2026-02-17T00:15:24.254532+00:00", "level": "ERROR", "logger": "veena3modal.tts", "message": "request_failed", "request_id": "fb212d46-65e4-4200-97dc-d74c8b9c6bfd", "event": "request_failed", "status_code": 500, "error_code": "INTERNAL_ERROR", "error_message": "veena3modal.core.pipeline.SparkTTSPipeline.generate_speech_indic() got multiple values for keyword argument 'request_id'"}
INFO:     127.0.0.1:45418 - "POST /v1/tts/generate HTTP/1.1" 500 Internal Server Error
{"timestamp": "2026-02-17T00:15:24.256920+00:00", "level": "INFO", "logger": "veena3modal.tts", "message": "request_received", "request_id": "0f35c7fe-735e-440b-a3e7-831e48def7d3", "event": "request_received", "text_length": 28, "speaker": "vardan", "stream": false, "format": "wav"}
2026-02-17 00:15:24 | INFO     | veena3modal.tts | request_received
2026-02-17 00:15:24 | INFO     | veena3modal.services.tts_runtime | tts_runtime_chunking_start
2026-02-17 00:15:24 | INFO     | veena3modal.processing.long_text_processor | LongTextProcessor initialized
2026-02-17 00:15:24 | ERROR    | veena3modal.services.tts_runtime | Chunked speech generation failed: veena3modal.core.pipeline.SparkTTSPipeline.generate_speech_indic() got multiple values for keyword argument 'request_id'
{"timestamp": "2026-02-17T00:15:24.257312+00:00", "level": "ERROR", "logger": "veena3modal.api.fastapi_app", "message": "TTS generation error: veena3modal.core.pipeline.SparkTTSPipeline.generate_speech_indic() got multiple values for keyword argument 'request_id'", "request_id": "0f35c7fe-735e-440b-a3e7-831e48def7d3", "exception": "Traceback (most recent call last):\n  File \"/home/ubuntu/veenaModal/veena3modal/api/fastapi_app.py\", line 446, in tts_generate\n    audio_bytes, metrics = await tts_runtime.generate_speech_chunked(\n  File \"/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py\", line 716, in generate_speech_chunked\n    audio_bytes, pipeline_timings = await runtime.pipeline.generate_speech_indic(\nTypeError: veena3modal.core.pipeline.SparkTTSPipeline.generate_speech_indic() got multiple values for keyword argument 'request_id'\n"}
2026-02-17 00:15:24 | ERROR    | veena3modal.api.fastapi_app | TTS generation error: veena3modal.core.pipeline.SparkTTSPipeline.generate_speech_indic() got multiple values for keyword argument 'request_id'
Traceback (most recent call last):
  File "/home/ubuntu/veenaModal/veena3modal/api/fastapi_app.py", line 446, in tts_generate
    audio_bytes, metrics = await tts_runtime.generate_speech_chunked(
  File "/home/ubuntu/veenaModal/veena3modal/service…36871 tokens truncated…rd argument 'request_id'
{"timestamp": "2026-02-17T00:15:30.603667+00:00", "level": "ERROR", "logger": "veena3modal.api.fastapi_app", "message": "TTS generation error: veena3modal.core.pipeline.SparkTTSPipeline.generate_speech_indic() got multiple values for keyword argument 'request_id'", "request_id": "2038ba8f-8c06-4bea-8446-fefbc5a93a8e", "exception": "Traceback (most recent call last):\n  File \"/home/ubuntu/veenaModal/veena3modal/api/fastapi_app.py\", line 446, in tts_generate\n    audio_bytes, metrics = await tts_runtime.generate_speech_chunked(\n  File \"/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py\", line 716, in generate_speech_chunked\n    audio_bytes, pipeline_timings = await runtime.pipeline.generate_speech_indic(\nTypeError: veena3modal.core.pipeline.SparkTTSPipeline.generate_speech_indic() got multiple values for keyword argument 'request_id'\n"}
2026-02-17 00:15:30 | ERROR    | veena3modal.api.fastapi_app | TTS generation error: veena3modal.core.pipeline.SparkTTSPipeline.generate_speech_indic() got multiple values for keyword argument 'request_id'
Traceback (most recent call last):
  File "/home/ubuntu/veenaModal/veena3modal/api/fastapi_app.py", line 446, in tts_generate
    audio_bytes, metrics = await tts_runtime.generate_speech_chunked(
  File "/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py", line 716, in generate_speech_chunked
    audio_bytes, pipeline_timings = await runtime.pipeline.generate_speech_indic(
TypeError: veena3modal.core.pipeline.SparkTTSPipeline.generate_speech_indic() got multiple values for keyword argument 'request_id'
{"timestamp": "2026-02-17T00:15:30.603833+00:00", "level": "ERROR", "logger": "veena3modal.tts", "message": "request_failed", "request_id": "2038ba8f-8c06-4bea-8446-fefbc5a93a8e", "event": "request_failed", "status_code": 500, "error_code": "INTERNAL_ERROR", "error_message": "veena3modal.core.pipeline.SparkTTSPipeline.generate_speech_indic() got multiple values for keyword argument 'request_id'"}
2026-02-17 00:15:30 | ERROR    | veena3modal.tts | request_failed
INFO:     127.0.0.1:51966 - "POST /v1/tts/generate HTTP/1.1" 500 Internal Server Error
{"timestamp": "2026-02-17T00:15:30.604371+00:00", "level": "INFO", "logger": "veena3modal.tts", "message": "request_received", "request_id": "e855fd4c-0b8c-4687-9ac9-f4357bd365dd", "event": "request_received", "text_length": 28, "speaker": "anika", "stream": false, "format": "wav"}
2026-02-17 00:15:30 | INFO     | veena3modal.tts | request_received
2026-02-17 00:15:30 | INFO     | veena3modal.services.tts_runtime | tts_runtime_chunking_start
2026-02-17 00:15:30 | INFO     | veena3modal.processing.long_text_processor | LongTextProcessor initialized
2026-02-17 00:15:30 | ERROR    | veena3modal.services.tts_runtime | Chunked speech generation failed: veena3modal.core.pipeline.SparkTTSPipeline.generate_speech_indic() got multiple values for keyword argument 'request_id'
{"timestamp": "2026-02-17T00:15:30.604554+00:00", "level": "ERROR", "logger": "veena3modal.api.fastapi_app", "message": "TTS generation error: veena3modal.core.pipeline.SparkTTSPipeline.generate_speech_indic() got multiple values for keyword argument 'request_id'", "request_id": "e855fd4c-0b8c-4687-9ac9-f4357bd365dd", "exception": "Traceback (most recent call last):\n  File \"/home/ubuntu/veenaModal/veena3modal/api/fastapi_app.py\", line 446, in tts_generate\n    audio_bytes, metrics = await tts_runtime.generate_speech_chunked(\n  File \"/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py\", line 716, in generate_speech_chunked\n    audio_bytes, pipeline_timings = await runtime.pipeline.generate_speech_indic(\nTypeError: veena3modal.core.pipeline.SparkTTSPipeline.generate_speech_indic() got multiple values for keyword argument 'request_id'\n"}
2026-02-17 00:15:30 | ERROR    | veena3modal.api.fastapi_app | TTS generation error: veena3modal.core.pipeline.SparkTTSPipeline.generate_speech_indic() got multiple values for keyword argument 'request_id'
Traceback (most recent call last):
  File "/home/ubuntu/veenaModal/veena3modal/api/fastapi_app.py", line 446, in tts_generate
    audio_bytes, metrics = await tts_runtime.generate_speech_chunked(
  File "/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py", line 716, in generate_speech_chunked
    audio_bytes, pipeline_timings = await runtime.pipeline.generate_speech_indic(
TypeError: veena3modal.core.pipeline.SparkTTSPipeline.generate_speech_indic() got multiple values for keyword argument 'request_id'
{"timestamp": "2026-02-17T00:15:30.604722+00:00", "level": "ERROR", "logger": "veena3modal.tts", "message": "request_failed", "request_id": "e855fd4c-0b8c-4687-9ac9-f4357bd365dd", "event": "request_failed", "status_code": 500, "error_code": "INTERNAL_ERROR", "error_message": "veena3modal.core.pipeline.SparkTTSPipeline.generate_speech_indic() got multiple values for keyword argument 'request_id'"}
2026-02-17 00:15:30 | ERROR    | veena3modal.tts | request_failed
INFO:     127.0.0.1:52016 - "POST /v1/tts/generate HTTP/1.1" 500 Internal Server Error
{"timestamp": "2026-02-17T00:15:30.605248+00:00", "level": "INFO", "logger": "veena3modal.tts", "message": "request_received", "request_id": "0349dd86-113d-43bd-a137-c9338ff1402a", "event": "request_received", "text_length": 28, "speaker": "vardan", "stream": false, "format": "wav"}
2026-02-17 00:15:30 | INFO     | veena3modal.tts | request_received
2026-02-17 00:15:30 | INFO     | veena3modal.services.tts_runtime | tts_runtime_chunking_start
2026-02-17 00:15:30 | INFO     | veena3modal.processing.long_text_processor | LongTextProcessor initialized
2026-02-17 00:15:30 | ERROR    | veena3modal.services.tts_runtime | Chunked speech generation failed: veena3modal.core.pipeline.SparkTTSPipeline.generate_speech_indic() got multiple values for keyword argument 'request_id'
{"timestamp": "2026-02-17T00:15:30.605435+00:00", "level": "ERROR", "logger": "veena3modal.api.fastapi_app", "message": "TTS generation error: veena3modal.core.pipeline.SparkTTSPipeline.generate_speech_indic() got multiple values for keyword argument 'request_id'", "request_id": "0349dd86-113d-43bd-a137-c9338ff1402a", "exception": "Traceback (most recent call last):\n  File \"/home/ubuntu/veenaModal/veena3modal/api/fastapi_app.py\", line 446, in tts_generate\n    audio_bytes, metrics = await tts_runtime.generate_speech_chunked(\n  File \"/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py\", line 716, in generate_speech_chunked\n    audio_bytes, pipeline_timings = await runtime.pipeline.generate_speech_indic(\nTypeError: veena3modal.core.pipeline.SparkTTSPipeline.generate_speech_indic() got multiple values for keyword argument 'request_id'\n"}
2026-02-17 00:15:30 | ERROR    | veena3modal.api.fastapi_app | TTS generation error: veena3modal.core.pipeline.SparkTTSPipeline.generate_speech_indic() got multiple values for keyword argument 'request_id'
Traceback (most recent call last):
  File "/home/ubuntu/veenaModal/veena3modal/api/fastapi_app.py", line 446, in tts_generate
    audio_bytes, metrics = await tts_runtime.generate_speech_chunked(
  File "/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py", line 716, in generate_speech_chunked
    audio_bytes, pipeline_timings = await runtime.pipeline.generate_speech_indic(
TypeError: veena3modal.core.pipeline.SparkTTSPipeline.generate_speech_indic() got multiple values for keyword argument 'request_id'
2026-02-17 00:15:30 | ERROR    | veena3modal.tts | request_failed
{"timestamp": "2026-02-17T00:15:30.605603+00:00", "level": "ERROR", "logger": "veena3modal.tts", "message": "request_failed", "request_id": "0349dd86-113d-43bd-a137-c9338ff1402a", "event": "request_failed", "status_code": 500, "error_code": "INTERNAL_ERROR", "error_message": "veena3modal.core.pipeline.SparkTTSPipeline.generate_speech_indic() got multiple values for keyword argument 'request_id'"}
INFO:     127.0.0.1:51968 - "POST /v1/tts/generate HTTP/1.1" 500 Internal Server Error


I hit a runtime regression preventing valid timing data: request failures are from a duplicate `request_id` argument in chunked generation. I’ll patch the offending `tts_runtime` call, relaunch server once, and then rerun the same stress matrix so we actually get per-request timing.


sed -n '660,760p' /home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py


Output:
Chunk ID: e18983
Wall time: 0.0511 seconds
Process exited with code 0
Original token count: 1161
Output:
            )
        # Import long text processor
        from veena3modal.processing.long_text_processor import LongTextProcessor

        long_processor = LongTextProcessor(pipeline=runtime.pipeline)

        # Check if chunking is needed
        if long_processor.should_chunk(text):
            metrics["text_chunked"] = True
            if _timing_enabled():
                chunk_result = await long_processor.generate_with_chunking(
                    text=text,
                    speaker=speaker,
                    temperature=temperature,
                    top_k=top_k,
                    top_p=top_p,
                    max_tokens=max_tokens,
                    repetition_penalty=repetition_penalty,
                    seed=seed,
                    sample_rate=sample_rate,
                    request_id=request_id,
                    return_timing=True,
                )
                if isinstance(chunk_result, tuple):
                    audio_bytes, chunk_timing = chunk_result
                else:
                    audio_bytes = chunk_result
                    chunk_timing = {}
            else:
                audio_bytes = await long_processor.generate_with_chunking(
                    text=text,
                    speaker=speaker,
                    temperature=temperature,
                    top_k=top_k,
                    top_p=top_p,
                    max_tokens=max_tokens,
                    repetition_penalty=repetition_penalty,
                    seed=seed,
                    sample_rate=sample_rate,
                    request_id=request_id,
                )
                chunk_timing = {}
        else:
            # Short text: direct generation
            gen_kwargs = dict(
                speaker=speaker,
                text=text,
                temperature=temperature,
                top_k=top_k,
                top_p=top_p,
                max_tokens=max_tokens,
                repetition_penalty=repetition_penalty,
                seed=seed,
                request_id=request_id,
            )
            if _timing_enabled():
                audio_bytes, pipeline_timings = await runtime.pipeline.generate_speech_indic(
                    **gen_kwargs,
                    return_timing=True,
                    request_id=request_id,
                )
                chunk_timing = pipeline_timings
            else:
                audio_bytes = await runtime.pipeline.generate_speech_indic(
                    **gen_kwargs,
                    request_id=request_id,
                )
                chunk_timing = {}

        generation_time = time.time() - start_time
        metrics["generation_ms"] = int(generation_time * 1000)
        metrics["ttfb_ms"] = metrics["generation_ms"]

        if chunk_timing:
            metrics.update({
                "llm_token_total": chunk_timing.get("token_total", 0),
                "llm_batch_count": chunk_timing.get("batch_count", 0),
                "llm_batch_gpu_ms_total": chunk_timing.get("llm_gpu_batch_ms_total", 0),
                "llm_batch_wall_ms_total": chunk_timing.get("llm_wall_ms", metrics["generation_ms"]),
                "llm_parse_ms": chunk_timing.get("parse_ms", 0),
                "llm_decode_cpu_ms": chunk_timing.get("decode_cpu_ms", 0),
                "llm_batch_wall_ms_min": chunk_timing.get("llm_batch_wall_ms_min", 0),
                "llm_batch_wall_ms_max": chunk_timing.get("llm_batch_wall_ms_max", 0),
                "llm_batch_gpu_ms_min": chunk_timing.get("llm_batch_gpu_ms_min", 0),
                "llm_batch_gpu_ms_max": chunk_timing.get("llm_batch_gpu_ms_max", 0),
                "llm_batch_gpu_ms_p50": chunk_timing.get("llm_batch_gpu_ms_p50", 0),
                "tokens_per_batch_min": chunk_timing.get("tokens_per_batch_min"),
                "tokens_per_batch_max": chunk_timing.get("tokens_per_batch_max"),
                "tokens_per_batch_p50": chunk_timing.get("tokens_per_batch_p50"),
                "llm_batch_tokens": chunk_timing.get("token_batches", []),
            })
            if metrics["text_chunked"]:
                metrics["chunks_processed"] = chunk_timing.get("chunks_processed", 0)

        if audio_bytes:
            # Apply super-resolution if requested and available
            final_sample_rate = 16000
            if output_sample_rate == "48khz" and runtime.sr_service and runtime.sr_service.is_loaded:
                try:
                    sr_start = time.time()
                    logger.info(f"Applying super-resolution to chunked audio ({len(audio_bytes)} bytes)...")


rg -n "generate_speech_indic\(|return_timing=True" /home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py


Output:
Chunk ID: 7f06e8
Wall time: 0.0509 seconds
Process exited with code 0
Original token count: 119
Output:
450:            audio_bytes, pipeline_timings = await runtime.pipeline.generate_speech_indic(
452:                return_timing=True,
456:            audio_bytes = await runtime.pipeline.generate_speech_indic(
681:                    return_timing=True,
716:                audio_bytes, pipeline_timings = await runtime.pipeline.generate_speech_indic(
718:                    return_timing=True,
723:                audio_bytes = await runtime.pipeline.generate_speech_indic(


sed -n '430,470p' /home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py


Output:
Chunk ID: 07a0f0
Wall time: 0.0510 seconds
Process exited with code 0
Original token count: 388
Output:
                "request_inflight": active_requests,
                "temperature": temperature,
                "top_k": top_k,
                "top_p": top_p,
                "max_tokens": max_tokens,
            },
        )

        gen_kwargs = dict(
            speaker=speaker,
            text=text,
            temperature=temperature,
            top_k=top_k,
            top_p=top_p,
            max_tokens=max_tokens,
            repetition_penalty=repetition_penalty,
            seed=seed,
        )

        if _timing_enabled():
            audio_bytes, pipeline_timings = await runtime.pipeline.generate_speech_indic(
                **gen_kwargs,
                return_timing=True,
                request_id=request_id,
            )
        else:
            audio_bytes = await runtime.pipeline.generate_speech_indic(
                **gen_kwargs,
                request_id=request_id,
            )

        generation_time = time.time() - start_time
        metrics["generation_ms"] = int(generation_time * 1000)
        metrics["ttfb_ms"] = metrics["generation_ms"]  # Non-streaming: TTFB = total time

        if pipeline_timings:
            metrics.update({
                "llm_token_total": pipeline_timings.get("token_total", 0),
                "llm_batch_count": pipeline_timings.get("batch_count", 0),
                "llm_batch_gpu_ms_total": pipeline_timings.get("llm_gpu_batch_ms_total", 0),
                "llm_batch_gpu_ms_min": pipeline_timings.get("llm_gpu_batch_ms_min", 0),


Input:
*** Begin Patch
*** Update File: /home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py
@@
         else:
             # Short text: direct generation
             gen_kwargs = dict(
                 speaker=speaker,
                 text=text,
                 temperature=temperature,
                 top_k=top_k,
                 top_p=top_p,
                 max_tokens=max_tokens,
                 repetition_penalty=repetition_penalty,
                 seed=seed,
-                request_id=request_id,
             )
*** End Patch


Output: {"output":"Success. Updated the following files:\nM /home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


I fixed the duplicated `request_id` keyword in non-chunked generation and will restart the server before rerunning the benchmark to capture valid timing traces.


Output: write_stdin failed: stdin is closed for this session; rerun exec_command with tty=true to keep stdin open


Output:
Chunk ID: ebdbfe
Wall time: 1.0007 seconds
Process running with session ID 83735
Original token count: 454
Output:
2026-02-17 00:15:50 | INFO     | veena3.local | Auth bypass enabled (local mode). Use --auth to require API keys.
2026-02-17 00:15:50 | INFO     | veena3.local | Resolved model paths:
2026-02-17 00:15:50 | INFO     | veena3.local |   LLM (vLLM):    /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
2026-02-17 00:15:50 | INFO     | veena3.local |   BiCodec:       /home/ubuntu/veenaModal/models/spark_tts_4speaker
2026-02-17 00:15:50 | INFO     | veena3.local | ============================================================
2026-02-17 00:15:50 | INFO     | veena3.local | Initializing Veena3 TTS Runtime (local)
2026-02-17 00:15:50 | INFO     | veena3.local |   Model:       /home/ubuntu/veenaModal/models/spark_tts_4speaker
2026-02-17 00:15:50 | INFO     | veena3.local |   LLM path:    /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
2026-02-17 00:15:50 | INFO     | veena3.local |   BiCodec:     /home/ubuntu/veenaModal/models/spark_tts_4speaker
2026-02-17 00:15:50 | INFO     | veena3.local |   Device:      cuda
2026-02-17 00:15:50 | INFO     | veena3.local |   GPU Memory:  25% (total)
2026-02-17 00:15:50 | INFO     | veena3.local |   Engines:     1
2026-02-17 00:15:50 | INFO     | veena3.local |   SR:          disabled
2026-02-17 00:15:50 | INFO     | veena3.local | ============================================================
2026-02-17 00:15:50 | INFO     | veena3modal.services.tts_runtime | Initializing TTS runtime...
2026-02-17 00:15:50 | INFO     | veena3modal.services.tts_runtime |   Model path: /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
2026-02-17 00:15:50 | INFO     | veena3modal.services.tts_runtime |   BiCodec path: /home/ubuntu/veenaModal/models/spark_tts_4speaker
2026-02-17 00:15:50 | INFO     | veena3modal.services.tts_runtime |   SR enabled: False, path: None


Output:
Chunk ID: 080d02
Wall time: 5.0006 seconds
Process running with session ID 83735
Original token count: 1283
Output:
Total output lines: 49

2026-02-17 00:15:56 | INFO     | veena3modal.services.tts_runtime | Loading SparkTTS model with vLLM...
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
🚀 Initializing Spark TTS Model
📁 Model path: /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
🗣  Speakers: 12 predefined (lipakshi, vardan, reet, Nandini, krishna, anika, adarsh, Nilay, Aarvi, Asha, Bittu, Mira)
🎭 Emotions: 10 tags in [bracket] format

📝 Loading tokenizer from model...
✅ Tokenizer loaded: 165168 tokens

🔍 Validating 12 speakers for Spark TTS...
  lipakshi → <|speaker_0|>: 7 tokens
  vardan → <|speaker_1|>: 7 tokens
  reet → <|speaker_2|>: 7 tokens
  Nandini → <|speaker_3|>: 7 tokens
  krishna → <|speaker_4|>: 7 tokens
  anika → <|speaker_5|>: 7 tokens
  adarsh → <|speaker_6|>: 7 tokens
  Nilay → <|speaker_7|>: 7 tokens
  Aarvi → <|speaker_8|>: 7 tokens
  Asha → <|speaker_9|>: 7 tokens
  Bittu → <|speaker_10|>: 8 tokens
  Mira → <|speaker_11|>: 8 tokens
✅ All 12 speakers validated with mapping

🔍 Validating 10 emotion tags for Spark TTS...
✅ All 10 emotion tags validated

📋 Sample emotion tags:
  [angry]: 1 tokens
  [curious]: 1 tokens
  [excited]: 1 tokens
  [giggle]: 1 tokens
  [laughs harder]: 1 tokens
  ...

🔧 Initializing vLLM engine with optimizations...
  - dtype: bfloat16
  - max_model_len: 4096
  - gpu_memory_utilization: 0.25
  - enable_prefix_caching: True
  - enforce_eager (disable CUDA graphs): False
INFO 02-17 00:15:57 [model.py:541] Resolved architecture: Qwen2ForCausalLM
INFO 02-17 00:15:57 [model.py:1561] Using max model len 4096
INFO 02-17 00:15:57 [scheduler.py:226] Chunked prefill is enabled with max_num_batched_tokens=4096.
INFO 02-17 00:15:57 [vllm.py:624] Asynchronous scheduling is enabled.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(EngineCore_DP0 pid=3699402) INFO 02-17 00:15:58 [core.py:96] Initializing a V1 LLM engine (v0.15.1) with config: model='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', speculative_config=None, tokenizer='/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_r…83 tokens truncated…tputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=/home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [4096], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 256, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': True}, 'local_cache_dir': None, 'static_all_moe_layers': []}


Output:
Chunk ID: b91c3d
Wall time: 5.0012 seconds
Process running with session ID 83735
Original token count: 323
Output:
(EngineCore_DP0 pid=3699402) INFO 02-17 00:16:00 [parallel_state.py:1212] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://216.81.248.184:33379 backend=nccl
(EngineCore_DP0 pid=3699402) INFO 02-17 00:16:00 [parallel_state.py:1423] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
(EngineCore_DP0 pid=3699402) INFO 02-17 00:16:01 [gpu_model_runner.py:4033] Starting to load model /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM...
(EngineCore_DP0 pid=3699402) INFO 02-17 00:16:01 [cuda.py:364] Using FLASH_ATTN attention backend out of potential backends: ('FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION')
(EngineCore_DP0 pid=3699402)
Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
(EngineCore_DP0 pid=3699402)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:01<00:00,  1.89s/it]
(EngineCore_DP0 pid=3699402)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:01<00:00,  1.89s/it]
(EngineCore_DP0 pid=3699402)
(EngineCore_DP0 pid=3699402) INFO 02-17 00:16:03 [default_loader.py:291] Loading weights took 1.97 seconds
(EngineCore_DP0 pid=3699402) INFO 02-17 00:16:04 [gpu_model_runner.py:4130] Model loading took 0.95 GiB memory and 2.498341 seconds


Output:
Chunk ID: 3c7b3b
Wall time: 5.0025 seconds
Process running with session ID 83735
Original token count: 74
Output:
(EngineCore_DP0 pid=3699402) INFO 02-17 00:16:09 [backends.py:812] Using cache directory: /home/ubuntu/.cache/vllm/torch_compile_cache/c3bd5d9fb9/rank_0_0/backbone for vLLM's torch.compile
(EngineCore_DP0 pid=3699402) INFO 02-17 00:16:09 [backends.py:872] Dynamo bytecode transform time: 4.20 s


Output:
Chunk ID: bea195
Wall time: 5.0008 seconds
Process running with session ID 83735
Original token count: 767
Output:
(EngineCore_DP0 pid=3699402) INFO 02-17 00:16:11 [backends.py:267] Directly load the compiled graph(s) for compile range (1, 4096) from the cache, took 0.701 s
(EngineCore_DP0 pid=3699402) INFO 02-17 00:16:11 [monitor.py:34] torch.compile takes 4.90 s in total
(EngineCore_DP0 pid=3699402) INFO 02-17 00:16:13 [gpu_worker.py:356] Available KV cache memory: 18.06 GiB
(EngineCore_DP0 pid=3699402) INFO 02-17 00:16:13 [kv_cache_utils.py:1307] GPU KV cache size: 1,577,712 tokens
(EngineCore_DP0 pid=3699402) INFO 02-17 00:16:13 [kv_cache_utils.py:1312] Maximum concurrency for 4,096 tokens per request: 385.18x
(EngineCore_DP0 pid=3699402)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   0%|          | 0/35 [00:00<?, ?it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   9%|▊         | 3/35 [00:00<00:01, 23.60it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  20%|██        | 7/35 [00:00<00:00, 29.17it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  31%|███▏      | 11/35 [00:00<00:00, 33.13it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  43%|████▎     | 15/35 [00:00<00:00, 33.26it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  54%|█████▍    | 19/35 [00:00<00:00, 32.33it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  66%|██████▌   | 23/35 [00:00<00:00, 34.32it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  77%|███████▋  | 27/35 [00:00<00:00, 33.17it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  89%|████████▊ | 31/35 [00:00<00:00, 32.14it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 35/35 [00:01<00:00, 31.58it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 35/35 [00:01<00:00, 31.94it/s]
(EngineCore_DP0 pid=3699402)
Capturing CUDA graphs (decode, FULL):   0%|          | 0/19 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL):  16%|█▌        | 3/19 [00:00<00:00, 29.71it/s]
Capturing CUDA graphs (decode, FULL):  37%|███▋      | 7/19 [00:00<00:00, 34.54it/s]
Capturing CUDA graphs (decode, FULL):  58%|█████▊    | 11/19 [00:00<00:00, 35.43it/s]
Capturing CUDA graphs (decode, FULL):  79%|███████▉  | 15/19 [00:00<00:00, 31.81it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 32.44it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 19/19 [00:00<00:00, 32.74it/s]
(EngineCore_DP0 pid=3699402) INFO 02-17 00:16:15 [gpu_model_runner.py:5063] Graph capturing finished in 3 secs, took 0.28 GiB
(EngineCore_DP0 pid=3699402) INFO 02-17 00:16:15 [core.py:272] init engine (profile, create kv cache, warmup model) took 11.40 seconds
2026-02-17 00:16:17 | INFO     | veena3modal.services.tts_runtime | Initializing prompt builder...
2026-02-17 00:16:17 | INFO     | veena3modal.services.tts_runtime | Initializing BiCodec decoder...


Output:
Chunk ID: e433e6
Wall time: 5.0042 seconds
Process running with session ID 83735
Original token count: 62
Output:
/home/ubuntu/veenaModal/venv/lib/python3.10/site-packages/torch/nn/utils/weight_norm.py:144: FutureWarning: `torch.nn.utils.weight_norm` is deprecated in favor of `torch.nn.utils.parametrizations.weight_norm`.
  WeightNorm.apply(module, name, dim)


Output:
Chunk ID: d047e0
Wall time: 5.0007 seconds
Process running with session ID 83735
Original token count: 0
Output:


Output:
Chunk ID: e80a10
Wall time: 5.0041 seconds
Process running with session ID 83735
Original token count: 321
Output:
2026-02-17 00:16:30 | INFO     | veena3modal.services.tts_runtime | Initializing TTS pipeline...
2026-02-17 00:16:30 | INFO     | veena3modal.services.tts_runtime | Initializing streaming pipeline...
2026-02-17 00:16:30 | INFO     | veena3modal.services.tts_runtime | ✅ TTS runtime initialized in 39947ms
2026-02-17 00:16:30 | INFO     | veena3modal.services.tts_runtime |    Model version: LLM
2026-02-17 00:16:30 | INFO     | veena3modal.services.tts_runtime | Pre-computing global tokens for all speakers...
2026-02-17 00:16:30 | INFO     | veena3modal.services.tts_runtime |   Cached globals for lipakshi
2026-02-17 00:16:31 | INFO     | veena3modal.services.tts_runtime |   Cached globals for vardan
2026-02-17 00:16:32 | INFO     | veena3modal.services.tts_runtime |   Cached globals for reet
2026-02-17 00:16:32 | INFO     | veena3modal.services.tts_runtime |   Cached globals for Nandini
2026-02-17 00:16:33 | INFO     | veena3modal.services.tts_runtime |   Cached globals for krishna
2026-02-17 00:16:34 | INFO     | veena3modal.services.tts_runtime |   Cached globals for anika
2026-02-17 00:16:34 | INFO     | veena3modal.services.tts_runtime |   Cached globals for adarsh
2026-02-17 00:16:35 | INFO     | veena3modal.services.tts_runtime |   Cached globals for Nilay


Output:
Chunk ID: 35633d
Wall time: 2.9198 seconds
Process exited with code 1
Original token count: 751
Output:
2026-02-17 00:16:36 | INFO     | veena3modal.services.tts_runtime |   Cached globals for Aarvi
2026-02-17 00:16:36 | INFO     | veena3modal.services.tts_runtime |   Cached globals for Asha
2026-02-17 00:16:37 | INFO     | veena3modal.services.tts_runtime |   Cached globals for Bittu
2026-02-17 00:16:37 | INFO     | veena3modal.services.tts_runtime |   Cached globals for Mira
2026-02-17 00:16:37 | INFO     | veena3modal.services.tts_runtime | Speaker globals cached: 12/12 speakers in 7215ms
2026-02-17 00:16:37 | INFO     | veena3.local | Runtime initialized in 47.2s
2026-02-17 00:16:37 | INFO     | veena3.local |   Model version: LLM
2026-02-17 00:16:37 | INFO     | veena3.local |   Streaming: available
2026-02-17 00:16:37 | INFO     | veena3.local |   Super-resolution: disabled
2026-02-17 00:16:37 | INFO     | veena3.local |
2026-02-17 00:16:37 | INFO     | veena3.local | ============================================================
2026-02-17 00:16:37 | INFO     | veena3.local | Veena3 TTS Local Server starting
2026-02-17 00:16:37 | INFO     | veena3.local |   URL: http://0.0.0.0:8000
2026-02-17 00:16:37 | INFO     | veena3.local |   Health: http://0.0.0.0:8000/v1/tts/health
2026-02-17 00:16:37 | INFO     | veena3.local |   Generate: POST http://0.0.0.0:8000/v1/tts/generate
2026-02-17 00:16:37 | INFO     | veena3.local |   Metrics: http://0.0.0.0:8000/v1/tts/metrics
2026-02-17 00:16:37 | INFO     | veena3.local |   Docs: http://0.0.0.0:8000/docs
2026-02-17 00:16:37 | INFO     | veena3.local | ============================================================
✅ vLLM engine initialized successfully!

🎉 Spark TTS Model ready for inference!

🎵 Loading BiCodec audio tokenizer from /home/ubuntu/veenaModal/models/spark_tts_4speaker...
   Device: cuda
Missing tensor: mel_transformer.spectrogram.window
Missing tensor: mel_transformer.mel_scale.fb
✅ BiCodec decoder initialized (sample rate: 16000Hz)
🚀 SparkTTSPipeline initialized
🌊 Veena3SlidingWindowPipeline initialized (sliding window: 28 tokens)
{"timestamp": "2026-02-17T00:16:37.689667+00:00", "level": "INFO", "logger": "veena3modal.api.websocket_handler", "message": "WebSocket route /v1/tts/ws registered"}
{"timestamp": "2026-02-17T00:16:37.689792+00:00", "level": "INFO", "logger": "veena3modal.api.fastapi_app", "message": "WebSocket support enabled at /v1/tts/ws"}
2026-02-17 00:16:37 | INFO     | veena3modal.api.websocket_handler | WebSocket route /v1/tts/ws registered
2026-02-17 00:16:37 | INFO     | veena3modal.api.fastapi_app | WebSocket support enabled at /v1/tts/ws
INFO:     Started server process [3699266]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
ERROR:    [Errno 98] error while attempting to bind on address ('0.0.0.0', 8000): address already in use
INFO:     Waiting for application shutdown.
INFO:     Application shutdown complete.
ERROR 02-17 00:16:37 [core_client.py:605] Engine core proc EngineCore_DP0 died unexpectedly, shutting down client.


lsof -i :8000 || true && ss -ltnp | rg ':8000' || true


Output:
Chunk ID: 0a0442
Wall time: 0.2279 seconds
Process exited with code 0
Original token count: 62
Output:
COMMAND     PID   USER   FD   TYPE    DEVICE SIZE/OFF NODE NAME
python3 3698377 ubuntu   81u  IPv4 330506371      0t0  TCP *:8000 (LISTEN)
LISTEN 0      2048         0.0.0.0:8000       0.0.0.0:*    users:(("python3",pid=3698377,fd=81))


Output:
Chunk ID: 25446a
Wall time: 1.1026 seconds
Process exited with code 0
Original token count: 0
Output:


There was a port collision from the earlier server process still bound to 8000, so I terminated it and will restart one clean server instance, then rerun the benchmark run.


Output:
Chunk ID: ba3b12
Wall time: 1.0006 seconds
Process running with session ID 38524
Original token count: 454
Output:
2026-02-17 00:16:45 | INFO     | veena3.local | Auth bypass enabled (local mode). Use --auth to require API keys.
2026-02-17 00:16:45 | INFO     | veena3.local | Resolved model paths:
2026-02-17 00:16:45 | INFO     | veena3.local |   LLM (vLLM):    /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
2026-02-17 00:16:45 | INFO     | veena3.local |   BiCodec:       /home/ubuntu/veenaModal/models/spark_tts_4speaker
2026-02-17 00:16:45 | INFO     | veena3.local | ============================================================
2026-02-17 00:16:45 | INFO     | veena3.local | Initializing Veena3 TTS Runtime (local)
2026-02-17 00:16:45 | INFO     | veena3.local |   Model:       /home/ubuntu/veenaModal/models/spark_tts_4speaker
2026-02-17 00:16:45 | INFO     | veena3.local |   LLM path:    /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
2026-02-17 00:16:45 | INFO     | veena3.local |   BiCodec:     /home/ubuntu/veenaModal/models/spark_tts_4speaker
2026-02-17 00:16:45 | INFO     | veena3.local |   Device:      cuda
2026-02-17 00:16:45 | INFO     | veena3.local |   GPU Memory:  25% (total)
2026-02-17 00:16:45 | INFO     | veena3.local |   Engines:     1
2026-02-17 00:16:45 | INFO     | veena3.local |   SR:          disabled
2026-02-17 00:16:45 | INFO     | veena3.local | ============================================================
2026-02-17 00:16:45 | INFO     | veena3modal.services.tts_runtime | Initializing TTS runtime...
2026-02-17 00:16:45 | INFO     | veena3modal.services.tts_runtime |   Model path: /home/ubuntu/veenaModal/models/spark_tts_4speaker/LLM
2026-02-17 00:16:45 | INFO     | veena3modal.services.tts_runtime |   BiCodec path: /home/ubuntu/veenaModal/models/spark_tts_4speaker
2026-02-17 00:16:45 | INFO     | veena3modal.services.tts_runtime |   SR enabled: False, path: None


Output:
Chunk ID: da858a
Wall time: 5.0004 seconds
Process running with session ID 38524
Original token count: 26
Output:
2026-02-17 00:16:52 | INFO     | veena3modal.services.tts_runtime | Loading SparkTTS model with vLLM...