Output: {"output":"Success. Updated the following files:\nM veena3modal/core/streaming_pipeline.py\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


# Collaboration Mode: Default

You are now in Default mode. Any previous instructions for other modes (e.g. Plan mode) are no longer active.

Your active mode changes only when new developer instructions with a different `<collaboration_mode>...</collaboration_mode>` change it; user requests or tool descriptions do not change mode by themselves. Known mode names are Default and Plan.

## request_user_input availability

The `request_user_input` tool is unavailable in Default mode. If you call it while in Default mode, it will return an error.

If a decision is necessary and cannot be discovered from local context, ask the user directly. However, in Default mode you should strongly prefer executing the user's request rather than stopping to ask questions.

Summary: auto

## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) When `SKILL.md` references relative paths (e.g., `scripts/foo.py`), resolve them relative to the skill directory listed above first, and only consider other paths if needed.
  3) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  4) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  5) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.


rg -n "generate_speech_chunked|generate_speech\(" veena3modal/services/tts_runtime.py | head -n 60


Output:
Chunk ID: 5fa8c2
Wall time: 0.0514 seconds
Process exited with code 0
Original token count: 18
Output:
365:async def generate_speech(
596:async def generate_speech_chunked(


Summary: auto


sed -n '596,780p' veena3modal/services/tts_runtime.py


Output:
Chunk ID: 273abe
Wall time: 0.0504 seconds
Process exited with code 0
Original token count: 1927
Output:
async def generate_speech_chunked(
    text: str,
    speaker: str,
    temperature: float = 0.8,
    top_k: int = 50,
    top_p: float = 1.0,
    max_tokens: int = 4096,
    repetition_penalty: float = 1.05,
    seed: Optional[int] = None,
    sample_rate: int = 16000,
    output_sample_rate: str = "16khz",
    request_id: Optional[str] = None,
) -> Tuple[Optional[bytes], Dict[str, Any]]:
    """
    Generate speech with automatic text chunking for long inputs.

    Uses LongTextProcessor to split text and stitch audio.

    Args:
        text: Text to synthesize
        speaker: Internal speaker name
        ... (same as generate_speech)
        sample_rate: Internal generation sample rate (always 16000)
        output_sample_rate: "16khz" or "48khz" (triggers super-resolution)

    Returns:
        Tuple of (audio_bytes, metrics_dict)
    """
    if not is_initialized():
        raise RuntimeError("TTS runtime not initialized")

    runtime = get_runtime()
    if request_id is None:
        import uuid
        request_id = f"runtime-chunk-{uuid.uuid4().hex[:10]}"
    metrics = {
        "request_id": request_id,
        "ttfb_ms": 0,
        "generation_ms": 0,
        "chunks_processed": 0,
        "audio_duration_seconds": 0.0,
        "text_chunked": False,
        "sr_applied": False,
        "output_sample_rate": 16000,
    }

    active_requests = _with_inflight_delta(1)
    start_time = time.time()

    try:
        if _timing_enabled():
            logger.info(
                "tts_runtime_chunking_start",
                extra={
                    "speaker": speaker,
                    "request_id": request_id,
                    "text_length": len(text),
                    "request_inflight": active_requests,
                    "chunking": True,
                },
            )
        # Import long text processor
        from veena3modal.processing.long_text_processor import LongTextProcessor

        long_processor = LongTextProcessor(pipeline=runtime.pipeline)

        # Check if chunking is needed
        if long_processor.should_chunk(text):
            metrics["text_chunked"] = True
            if _timing_enabled():
                chunk_result = await long_processor.generate_with_chunking(
                    text=text,
                    speaker=speaker,
                    temperature=temperature,
                    top_k=top_k,
                    top_p=top_p,
                    max_tokens=max_tokens,
                    repetition_penalty=repetition_penalty,
                    seed=seed,
                    sample_rate=sample_rate,
                    request_id=request_id,
                    return_timing=True,
                )
                if isinstance(chunk_result, tuple):
                    audio_bytes, chunk_timing = chunk_result
                else:
                    audio_bytes = chunk_result
                    chunk_timing = {}
            else:
                audio_bytes = await long_processor.generate_with_chunking(
                    text=text,
                    speaker=speaker,
                    temperature=temperature,
                    top_k=top_k,
                    top_p=top_p,
                    max_tokens=max_tokens,
                    repetition_penalty=repetition_penalty,
                    seed=seed,
                    sample_rate=sample_rate,
                    request_id=request_id,
                )
                chunk_timing = {}
        else:
            # Short text: direct generation
            gen_kwargs = dict(
                speaker=speaker,
                text=text,
                temperature=temperature,
                top_k=top_k,
                top_p=top_p,
                max_tokens=max_tokens,
                repetition_penalty=repetition_penalty,
                seed=seed,
                request_id=request_id,
            )
            if _timing_enabled():
                audio_bytes, pipeline_timings = await runtime.pipeline.generate_speech_indic(
                    **gen_kwargs,
                    return_timing=True,
                    request_id=request_id,
                )
                chunk_timing = pipeline_timings
            else:
                audio_bytes = await runtime.pipeline.generate_speech_indic(
                    **gen_kwargs,
                    request_id=request_id,
                )
                chunk_timing = {}

        generation_time = time.time() - start_time
        metrics["generation_ms"] = int(generation_time * 1000)
        metrics["ttfb_ms"] = metrics["generation_ms"]

        if chunk_timing:
            metrics.update({
                "llm_token_total": chunk_timing.get("token_total", 0),
                "llm_batch_count": chunk_timing.get("batch_count", 0),
                "llm_batch_gpu_ms_total": chunk_timing.get("llm_gpu_batch_ms_total", 0),
                "llm_batch_wall_ms_total": chunk_timing.get("llm_wall_ms", metrics["generation_ms"]),
                "llm_parse_ms": chunk_timing.get("parse_ms", 0),
                "llm_decode_cpu_ms": chunk_timing.get("decode_cpu_ms", 0),
                "llm_batch_wall_ms_min": chunk_timing.get("llm_batch_wall_ms_min", 0),
                "llm_batch_wall_ms_max": chunk_timing.get("llm_batch_wall_ms_max", 0),
                "llm_batch_gpu_ms_min": chunk_timing.get("llm_batch_gpu_ms_min", 0),
                "llm_batch_gpu_ms_max": chunk_timing.get("llm_batch_gpu_ms_max", 0),
                "llm_batch_gpu_ms_p50": chunk_timing.get("llm_batch_gpu_ms_p50", 0),
                "tokens_per_batch_min": chunk_timing.get("tokens_per_batch_min"),
                "tokens_per_batch_max": chunk_timing.get("tokens_per_batch_max"),
                "tokens_per_batch_p50": chunk_timing.get("tokens_per_batch_p50"),
                "llm_batch_tokens": chunk_timing.get("token_batches", []),
            })
            if metrics["text_chunked"]:
                metrics["chunks_processed"] = chunk_timing.get("chunks_processed", 0)

        if audio_bytes:
            # Apply super-resolution if requested and available
            final_sample_rate = 16000
            if output_sample_rate == "48khz" and runtime.sr_service and runtime.sr_service.is_loaded:
                try:
                    sr_start = time.time()
                    logger.info(f"Applying super-resolution to chunked audio ({len(audio_bytes)} bytes)...")
                    audio_bytes = _apply_super_resolution(audio_bytes, runtime.sr_service)
                    sr_time = (time.time() - sr_start) * 1000
                    metrics["sr_ms"] = int(sr_time)
                    metrics["sr_applied"] = True
                    final_sample_rate = 48000
                    logger.info(f"✅ Super-resolution applied in {sr_time:.1f}ms, output={len(audio_bytes)} bytes")
                except Exception as e:
                    logger.warning(f"Super-resolution failed, returning 16kHz: {e}")
                    import traceback
                    traceback.print_exc()

            metrics["output_sample_rate"] = final_sample_rate
            audio_duration = (len(audio_bytes) - 44) / (final_sample_rate * 2)
            metrics["audio_duration_seconds"] = max(0.0, audio_duration)

        if _timing_enabled():
            metrics["active_requests"] = active_requests - 1
            metrics["request_inflight"] = active_requests - 1
            logger.info("tts_runtime_chunking_complete", extra=metrics)
        return audio_bytes, metrics

    except Exception as e:
        logger.error(f"Chunked speech generation failed: {e}")
        raise


sed -n '780,840p' veena3modal/services/tts_runtime.py


Output:
Chunk ID: 4ea71d
Wall time: 0.0511 seconds
Process exited with code 0
Original token count: 493
Output:
        raise
    finally:
        _with_inflight_delta(-1)

from typing import AsyncGenerator

async def generate_speech_streaming(
    text: str,
    speaker: str,
    temperature: float = 0.8,
    top_k: int = 50,
    top_p: float = 1.0,
    max_tokens: int = 4096,
    repetition_penalty: float = 1.05,
    seed: Optional[int] = None,
    enable_chunking: bool = True,
    request_id: Optional[str] = None,
) -> AsyncGenerator[Tuple[bytes, Dict[str, Any]], None]:
    """
    Generate speech audio with true streaming (yields chunks as they're generated).

    This is the core streaming implementation for M4.
    First yield includes WAV header + first PCM chunk.
    Subsequent yields are raw PCM chunks.

    Args:
        text: Text to synthesize (already normalized)
        speaker: Internal speaker name (resolved)
        temperature: Sampling temperature
        top_k: Top-k sampling
        top_p: Nucleus sampling
        max_tokens: Maximum tokens to generate
        repetition_penalty: Repetition penalty
        seed: Random seed for reproducibility
        enable_chunking: Enable text chunking for long inputs with voice consistency

    Yields:
        Tuple of (audio_bytes, metrics_dict)
        - First yield: WAV header (44 bytes) prepended to first PCM chunk
        - Subsequent yields: Raw PCM chunks (int16, 16kHz)
        - Final yield: metrics_dict has final timing info

    Raises:
        RuntimeError: If runtime not initialized or streaming pipeline unavailable
    """
    if not is_initialized():
        raise RuntimeError("TTS runtime not initialized")

    runtime = get_runtime()

    if request_id is None:
        import uuid
        request_id = f"runtime-stream-{uuid.uuid4().hex[:10]}"

    if runtime.streaming_pipeline is None:
        raise RuntimeError("Streaming pipeline not available")

    # Import audio utils for WAV header
    from veena3modal.audio.utils import create_wav_header


sed -n '420,470p' veena3modal/services/tts_runtime.py


Output:
Chunk ID: dea43c
Wall time: 0.0514 seconds
Process exited with code 0
Original token count: 449
Output:

    pipeline_timings = {}

    try:
        logger.info(
            "tts_runtime_nonstream_start",
            extra={
                "speaker": speaker,
                "request_id": request_id,
                "text_length": len(text),
                "request_inflight": active_requests,
                "temperature": temperature,
                "top_k": top_k,
                "top_p": top_p,
                "max_tokens": max_tokens,
            },
        )

        gen_kwargs = dict(
            speaker=speaker,
            text=text,
            temperature=temperature,
            top_k=top_k,
            top_p=top_p,
            max_tokens=max_tokens,
            repetition_penalty=repetition_penalty,
            seed=seed,
        )

        if _timing_enabled():
            audio_bytes, pipeline_timings = await runtime.pipeline.generate_speech_indic(
                **gen_kwargs,
                return_timing=True,
                request_id=request_id,
            )
        else:
            audio_bytes = await runtime.pipeline.generate_speech_indic(
                **gen_kwargs,
                request_id=request_id,
            )

        generation_time = time.time() - start_time
        metrics["generation_ms"] = int(generation_time * 1000)
        metrics["ttfb_ms"] = metrics["generation_ms"]  # Non-streaming: TTFB = total time

        if pipeline_timings:
            metrics.update({
                "llm_token_total": pipeline_timings.get("token_total", 0),
                "llm_batch_count": pipeline_timings.get("batch_count", 0),
                "llm_batch_gpu_ms_total": pipeline_timings.get("llm_gpu_batch_ms_total", 0),
                "llm_batch_gpu_ms_min": pipeline_timings.get("llm_gpu_batch_ms_min", 0),


I’ll now wire request IDs consistently through API handlers, expose key timing metrics as response metadata for benchmarking, and keep websocket completion messages richer with full runtime timing so you can inspect per-request behavior quickly.


sed -n '470,540p' veena3modal/services/tts_runtime.py


Output:
Chunk ID: 025eb1
Wall time: 0.0514 seconds
Process exited with code 0
Original token count: 825
Output:
                "llm_batch_gpu_ms_min": pipeline_timings.get("llm_gpu_batch_ms_min", 0),
                "llm_batch_gpu_ms_max": pipeline_timings.get("llm_gpu_batch_ms_max", 0),
                "llm_batch_gpu_ms_p50": pipeline_timings.get("llm_gpu_batch_ms_p50", 0),
                "llm_decode_cpu_ms": pipeline_timings.get("decode_cpu_ms", 0),
                "llm_parse_ms": pipeline_timings.get("parse_ms", 0),
                "llm_wall_ms": pipeline_timings.get("llm_wall_ms", metrics["generation_ms"]),
                "ttfb_ms": int(pipeline_timings.get("ttfb_ms", metrics["ttfb_ms"])),
                "tokens_per_batch_min": pipeline_timings.get("tokens_per_batch_min"),
                "tokens_per_batch_max": pipeline_timings.get("tokens_per_batch_max"),
                "tokens_per_batch_p50": pipeline_timings.get("tokens_per_batch_p50"),
            })

        if audio_bytes:
            # Apply super-resolution if requested and available
            sample_rate = 16000
            logger.info(f"SR check: output_sample_rate={output_sample_rate!r}, "
                       f"sr_service={runtime.sr_service is not None}, "
                       f"is_loaded={runtime.sr_service.is_loaded if runtime.sr_service else False}")

            if output_sample_rate == "48khz" and runtime.sr_service and runtime.sr_service.is_loaded:
                try:
                    sr_start = time.time()
                    logger.info(f"Applying super-resolution to {len(audio_bytes)} bytes...")
                    audio_bytes = _apply_super_resolution(audio_bytes, runtime.sr_service)
                    sr_time = (time.time() - sr_start) * 1000
                    metrics["sr_ms"] = int(sr_time)
                    metrics["sr_applied"] = True
                    sample_rate = 48000
                    logger.info(f"✅ Super-resolution applied in {sr_time:.1f}ms, output={len(audio_bytes)} bytes")
                except Exception as e:
                    logger.warning(f"Super-resolution failed, returning 16kHz: {e}")
                    import traceback
                    traceback.print_exc()

            metrics["output_sample_rate"] = sample_rate

            # Calculate audio duration
            audio_duration = (len(audio_bytes) - 44) / (sample_rate * 2)  # -44 for WAV header
            metrics["audio_duration_seconds"] = max(0.0, audio_duration)

        if _timing_enabled():
            logger.info("tts_runtime_nonstream_complete", extra=metrics)

        return audio_bytes, metrics

    except Exception as e:
        logger.error(f"Speech generation failed: {e}")
        raise
    finally:
        active_requests = _with_inflight_delta(-1)
        if _timing_enabled():
            logger.info(
                "tts_runtime_nonstream_inflight",
                extra={
                    "request_id": request_id,
                    "active_requests": active_requests,
                },
            )

def _apply_super_resolution(audio_bytes: bytes, sr_service) -> bytes:
    """
    Apply super-resolution to audio bytes.

    Args:
        audio_bytes: WAV audio at 16kHz
        sr_service: SuperResolutionService instance

    Returns:
        WAV audio at 48kHz
    """


Input:
*** Begin Patch
*** Update File: veena3modal/services/tts_runtime.py
@@
     except Exception as e:
         logger.error(f"Speech generation failed: {e}")
         raise
     finally:
         active_requests = _with_inflight_delta(-1)
+        if _timing_enabled():
+            metrics["request_inflight"] = active_requests
+            metrics["active_requests"] = active_requests
+            logger.info(
+                "tts_runtime_nonstream_complete",
+                extra=metrics,
+            )
         if _timing_enabled():
             logger.info(
                 "tts_runtime_nonstream_inflight",
                 extra={
                     "request_id": request_id,
@@
     except Exception as e:
         logger.error(f"Chunked speech generation failed: {e}")
         raise
     finally:
-        _with_inflight_delta(-1)
+        active_requests = _with_inflight_delta(-1)
+        metrics["active_requests"] = active_requests
+        metrics["request_inflight"] = active_requests
*** End Patch


Output: {"output":"Success. Updated the following files:\nM veena3modal/services/tts_runtime.py\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


Input:
*** Begin Patch
*** Update File: veena3modal/services/tts_runtime.py
@@
-        if _timing_enabled():
-            logger.info("tts_runtime_nonstream_complete", extra=metrics)
-
         return audio_bytes, metrics
*** End Patch


Output: {"output":"Success. Updated the following files:\nM veena3modal/services/tts_runtime.py\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


Input:
*** Begin Patch
*** Update File: veena3modal/api/fastapi_app.py
@@
-import time
-import uuid
-from typing import Optional
+import time
+import uuid
+from typing import Optional, Dict, Any
@@
     @app.get("/v1/tts/metrics")
     async def tts_metrics():
@@
         return Response(
             content=get_metrics_text(),
             media_type=get_content_type(),
         )
+
+    def _append_runtime_metrics_headers(headers: Dict[str, str], metrics: Dict[str, Any]) -> None:
+        """Add selected runtime metrics as HTTP headers for benchmark visibility."""
+        metric_to_header = {
+            "generation_ms": "X-Generation-Ms",
+            "llm_token_total": "X-LLM-Token-Total",
+            "llm_batch_count": "X-LLM-Batch-Count",
+            "llm_batch_wall_ms_total": "X-LLM-Batch-Wall-Ms-Total",
+            "llm_batch_gpu_ms_total": "X-LLM-Batch-GPU-Ms-Total",
+            "llm_batch_gpu_ms_p50": "X-LLM-Batch-GPU-Ms-P50",
+            "llm_batch_wall_ms_min": "X-LLM-Batch-Wall-Ms-Min",
+            "llm_batch_wall_ms_max": "X-LLM-Batch-Wall-Ms-Max",
+            "llm_decode_cpu_ms": "X-LLM-Decode-CPU-Ms",
+            "llm_decode_cpu_ms": "X-LLM-Decode-CPU-Ms",
+            "llm_parse_ms": "X-LLM-Parse-Ms",
+            "audio_chunks": "X-Audio-Chunks",
+            "active_requests": "X-Active-Requests",
+            "request_inflight": "X-Request-Inflight",
+        }
+        for metric_key, header_name in metric_to_header.items():
+            value = metrics.get(metric_key)
+            if value is not None:
+                headers[header_name] = str(int(value) if isinstance(value, (int, float)) else value)
@@
                 audio_bytes, metrics = await tts_runtime.generate_speech_chunked(
                     text=text,
                     speaker=speaker,
                     temperature=req.temperature,
                     top_k=req.top_k,
@@
                     repetition_penalty=req.repetition_penalty,
                     seed=req.seed,
                     sample_rate=req.sample_rate,
                     output_sample_rate=output_sr,
+                    request_id=request_id,
                 )
             else:
                 audio_bytes, metrics = await tts_runtime.generate_speech(
                     text=text,
                     speaker=speaker,
@@
                     max_tokens=req.max_tokens,
                     repetition_penalty=req.repetition_penalty,
                     seed=req.seed,
                     output_sample_rate=output_sr,
+                    request_id=request_id,
                 )
@@
-            headers = {
+            headers = {
                 "X-Request-ID": request_id,
                 "X-Model-Version": model_version,
                 "X-Format": "wav",
@@
                 "X-Text-Chunked": str(metrics.get("text_chunked", False)).lower(),
                 "X-SR-Applied": str(metrics.get("sr_applied", False)).lower(),
             }
+
+            _append_runtime_metrics_headers(headers, metrics)
@@
                 async for audio_chunk, metrics in tts_runtime.generate_speech_streaming(
                     text=text,
                     speaker=speaker,
                     temperature=req.temperature,
                     top_k=req.top_k,
@@
                     max_tokens=req.max_tokens,
                     repetition_penalty=req.repetition_penalty,
                     seed=req.seed,
                     enable_chunking=req.chunking,
+                    request_id=request_id,
                 ):
*** End Patch


Output:
apply_patch verification failed: Failed to find expected lines in /home/ubuntu/veenaModal/veena3modal/api/fastapi_app.py:
        return Response(
            content=get_metrics_text(),
            media_type=get_content_type(),
        )


sed -n '1,70p' veena3modal/api/fastapi_app.py


Output:
Chunk ID: 560f37
Wall time: 0.0513 seconds
Process exited with code 0
Original token count: 481
Output:
"""
FastAPI ASGI app factory for Modal deployment.

Phase 1: expose `/v1/tts/generate` and `/v1/tts/health` while reusing existing
Veena3 inference/streaming code from the Django project.

Phase 2+: extract shared code into `veena3modal/shared/` and remove Django coupling.
"""

# NOTE: Intentionally NOT using `from __future__ import annotations` here
# to ensure FastAPI recognizes Request type annotations properly

import time
import uuid
from typing import Optional

# Structured logging (JSON formatted, request-scoped)
from veena3modal.shared.logging import (
    get_logger,
    set_request_context,
    clear_request_context,
    log_request_received,
    log_first_audio_emitted,
    log_request_completed,
    log_request_failed,
)

# Prometheus metrics
from veena3modal.shared.metrics import (
    record_request_received as metrics_request_received,
    record_request_completed as metrics_request_completed,
    record_request_failed as metrics_request_failed,
    record_ttfb,
    record_rtf,
    record_audio_duration,
    record_chunks_sent,
    get_metrics_text,
    get_content_type as get_metrics_content_type,
)

# Auth and rate limiting
from veena3modal.api.auth import (
    get_api_validator,
    extract_api_key,
    hash_api_key,
)
from veena3modal.api.rate_limiter import get_rate_limiter
from veena3modal.api.error_handlers import (
    ErrorCode,
    create_error_response,
    get_error_status,
    is_gpu_fault,
    handle_gpu_fault,
    FeatureFlags,
)

logger = get_logger(__name__)

# Sentence storage (fire-and-forget, non-blocking)
# Imported at module level for singleton access
_sentence_store = None

def _get_sentence_store():
    """Lazy load sentence store singleton."""
    global _sentence_store
    if _sentence_store is None:
        from veena3modal.services.sentence_store import get_sentence_store
        _sentence_store = get_sentence_store()
    return _sentence_store


sed -n '130,230p' veena3modal/api/fastapi_app.py


Output:
Chunk ID: 9fc7ae
Wall time: 0.0511 seconds
Process exited with code 0
Original token count: 948
Output:
            import torch
            gpu_available = torch.cuda.is_available()
        except ImportError:
            pass  # torch not installed, skip GPU check

        # Compute uptime
        uptime = time.time() - _startup_time if _startup_time else 0

        # Determine overall status
        if model_loaded and gpu_available:
            status = "healthy"
        elif model_loaded or gpu_available:
            status = "degraded"
        else:
            status = "degraded"

        return JSONResponse(
            content={
                "status": status,
                "model_loaded": model_loaded,
                "model_version": model_version,
                "uptime_seconds": round(uptime, 2),
                "gpu_available": gpu_available,
                "app_version": _APP_VERSION,
            },
            headers={
                "X-Model-Version": model_version,
                "X-App-Version": _APP_VERSION,
            },
        )

    @app.get("/v1/tts/metrics")
    async def tts_metrics():
        """
        Prometheus metrics endpoint.

        Returns metrics in Prometheus text format for scraping.
        """
        return Response(
            content=get_metrics_text(),
            media_type=get_metrics_content_type(),
        )

    @app.post("/v1/tts/generate")
    async def tts_generate(request: Request):
        """
        Generate speech from text.

        Supports streaming (M4) and non-streaming (M3) modes.
        Currently implements non-streaming WAV generation only.
        """
        from veena3modal.api.schemas import TTSGenerateRequest, AudioFormat
        from veena3modal.services import tts_runtime

        request_id = str(uuid.uuid4())
        start_time = time.time()

        # === AUTH CHECK ===
        if FeatureFlags.is_auth_enabled():
            # Extract API key from headers
            headers_dict = dict(request.headers)
            api_key = extract_api_key(headers_dict)

            if not api_key:
                return JSONResponse(
                    status_code=401,
                    content=create_error_response(
                        code=ErrorCode.INVALID_API_KEY,
                        message="API key required. Use 'Authorization: Bearer <key>' or 'X-API-Key: <key>' header.",
                        request_id=request_id,
                    ),
                    headers={"X-Request-ID": request_id},
                )

            # Validate API key
            validator = get_api_validator()
            key_hash = hash_api_key(api_key)
            auth_result = validator.validate(key_hash)

            if not auth_result.is_valid:
                status_code = 401 if auth_result.error_code == "INVALID_API_KEY" else 403
                return JSONResponse(
                    status_code=status_code,
                    content=create_error_response(
                        code=ErrorCode(auth_result.error_code),
                        message=auth_result.error_message or "Authentication failed",
                        request_id=request_id,
                    ),
                    headers={"X-Request-ID": request_id},
                )

            # === RATE LIMIT CHECK ===
            if FeatureFlags.is_rate_limiting_enabled():
                rate_limiter = get_rate_limiter()
                allowed, remaining, reset_after = rate_limiter.check(key_hash)

                if not allowed:
                    headers = rate_limiter.get_headers(allowed, remaining, reset_after)
                    headers["X-Request-ID"] = request_id
                    return JSONResponse(
                        status_code=429,


<permissions instructions>
Filesystem sandboxing defines which files can be read or written. `sandbox_mode` is `danger-full-access`: No filesystem sandboxing - all commands are permitted. Network access is enabled.
Approval policy is currently never. Do not provide the `sandbox_permissions` for any reason, commands will be rejected.
</permissions instructions>

<collaboration_mode># Collaboration Mode: Default

You are now in Default mode. Any previous instructions for other modes (e.g. Plan mode) are no longer active.

Your active mode changes only when new developer instructions with a different `<collaboration_mode>...</collaboration_mode>` change it; user requests or tool descriptions do not change mode by themselves. Known mode names are Default and Plan.

## request_user_input availability

The `request_user_input` tool is unavailable in Default mode. If you call it while in Default mode, it will return an error.

If a decision is necessary and cannot be discovered from local context, ask the user directly. However, in Default mode you should strongly prefer executing the user's request rather than stopping to ask questions.
</collaboration_mode>

# AGENTS.md instructions for /home/ubuntu/veenaModal

<INSTRUCTIONS>
## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) When `SKILL.md` references relative paths (e.g., `scripts/foo.py`), resolve them relative to the skill directory listed above first, and only consider other paths if needed.
  3) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  4) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  5) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.
</INSTRUCTIONS>

<environment_context>
  <cwd>/home/ubuntu/veenaModal</cwd>
  <shell>bash</shell>
</environment_context>

The current TTS application here is an implementation in Modal labs. Which has autoscaling and other rules self scaling rules and everything suitable for my target audience. Now, i need you to create a local entry point or implementation point. then we will plan for optimising the pipeline, then we will discuss on stress testing, then we will discuss on scaling laws post optimisations locally. @veena3modal/app.py

the above part is performed by a different agent. and benchmarks are made. Now can you do the benchmarks on this newly optimized path to understand the concurrency and stress testing.

Now i need you to capture detailed time logs for each request happening in the GPU. like what happens with 1req, time per LLM token, time per batch decoding, and everything else. detailed time about where we are spending each minute in the GPU per generation.

and when you have the details for per request. you can obviously add more logging to get details of concurrency. so you understand what's the bottleneck for the peak concurrency. and then we will decide on effective methods in optimising this TTS engine.


I’ll finish the observability path end-to-end for benchmarks: first wire a request-scoped ID through HTTP/WebSocket into runtime/pipeline, then add detailed per-request timing + concurrency metrics to responses/events, and finally add a local stress benchmark entrypoint to emit latency breakdowns (per-token/decode/batch + in-flight concurrency) for 1req and ramped concurrency. I’ll patch only the existing API handlers and add a dedicated benchmark script, then summarize how to run and read results.


sed -n '1,260p' /home/ubuntu/veenaModal/veena3modal/api/fastapi_app.py


sed -n '1,300p' /home/ubuntu/veenaModal/veena3modal/api/websocket_handler.py


rg -n "benchmark|stress|locust|wrk|artillery|concurrency|requests" /home/ubuntu/veenaModal -g "*.py"


Output:
Chunk ID: 10d351
Wall time: 0.0569 seconds
Process exited with code 0
Original token count: 2185
Output:
"""
FastAPI ASGI app factory for Modal deployment.

Phase 1: expose `/v1/tts/generate` and `/v1/tts/health` while reusing existing
Veena3 inference/streaming code from the Django project.

Phase 2+: extract shared code into `veena3modal/shared/` and remove Django coupling.
"""

# NOTE: Intentionally NOT using `from __future__ import annotations` here
# to ensure FastAPI recognizes Request type annotations properly

import time
import uuid
from typing import Optional

# Structured logging (JSON formatted, request-scoped)
from veena3modal.shared.logging import (
    get_logger,
    set_request_context,
    clear_request_context,
    log_request_received,
    log_first_audio_emitted,
    log_request_completed,
    log_request_failed,
)

# Prometheus metrics
from veena3modal.shared.metrics import (
    record_request_received as metrics_request_received,
    record_request_completed as metrics_request_completed,
    record_request_failed as metrics_request_failed,
    record_ttfb,
    record_rtf,
    record_audio_duration,
    record_chunks_sent,
    get_metrics_text,
    get_content_type as get_metrics_content_type,
)

# Auth and rate limiting
from veena3modal.api.auth import (
    get_api_validator,
    extract_api_key,
    hash_api_key,
)
from veena3modal.api.rate_limiter import get_rate_limiter
from veena3modal.api.error_handlers import (
    ErrorCode,
    create_error_response,
    get_error_status,
    is_gpu_fault,
    handle_gpu_fault,
    FeatureFlags,
)

logger = get_logger(__name__)

# Sentence storage (fire-and-forget, non-blocking)
# Imported at module level for singleton access
_sentence_store = None

def _get_sentence_store():
    """Lazy load sentence store singleton."""
    global _sentence_store
    if _sentence_store is None:
        from veena3modal.services.sentence_store import get_sentence_store
        _sentence_store = get_sentence_store()
    return _sentence_store

# Version info (will be set dynamically after model load)
_APP_VERSION = "0.1.0"
_MODEL_VERSION: Optional[str] = None
_startup_time: Optional[float] = None

def get_model_version() -> str:
    """Return the current model version, or 'not_loaded' if not initialized."""
    return _MODEL_VERSION or "not_loaded"

def set_model_version(version: str) -> None:
    """Set the model version after loading."""
    global _MODEL_VERSION
    _MODEL_VERSION = version

def create_app():
    """
    Factory function to create FastAPI app.

    Import inside factory so local tooling can import this module without FastAPI installed.
    """
    from fastapi import FastAPI, HTTPException, Request
    from fastapi.responses import JSONResponse, Response

    global _startup_time
    _startup_time = time.time()

    app = FastAPI(
        title="Veena3 TTS (Modal)",
        version=_APP_VERSION,
        description="High-quality multilingual Text-to-Speech API with true streaming support.",
    )

    @app.get("/v1/tts/health")
    async def tts_health():
        """
        Health check endpoint.

        Returns:
            - status: "healthy" | "degraded" | "unhealthy"
            - model_loaded: whether TTS model is initialized
            - model_version: version string of loaded model
            - uptime_seconds: time since app startup
            - gpu_available: whether GPU is detected (best-effort check)
        """
        # Check model status via runtime
        from veena3modal.services import tts_runtime
        model_loaded = tts_runtime.is_initialized()

        # Use runtime's model version if available
        runtime = tts_runtime.get_runtime()
        model_version = runtime.model_version if runtime else get_model_version()

        # Check GPU availability (best-effort, non-blocking)
        gpu_available = False
        try:
            import torch
            gpu_available = torch.cuda.is_available()
        except ImportError:
            pass  # torch not installed, skip GPU check

        # Compute uptime
        uptime = time.time() - _startup_time if _startup_time else 0

        # Determine overall status
        if model_loaded and gpu_available:
            status = "healthy"
        elif model_loaded or gpu_available:
            status = "degraded"
        else:
            status = "degraded"

        return JSONResponse(
            content={
                "status": status,
                "model_loaded": model_loaded,
                "model_version": model_version,
                "uptime_seconds": round(uptime, 2),
                "gpu_available": gpu_available,
                "app_version": _APP_VERSION,
            },
            headers={
                "X-Model-Version": model_version,
                "X-App-Version": _APP_VERSION,
            },
        )

    @app.get("/v1/tts/metrics")
    async def tts_metrics():
        """
        Prometheus metrics endpoint.

        Returns metrics in Prometheus text format for scraping.
        """
        return Response(
            content=get_metrics_text(),
            media_type=get_metrics_content_type(),
        )

    @app.post("/v1/tts/generate")
    async def tts_generate(request: Request):
        """
        Generate speech from text.

        Supports streaming (M4) and non-streaming (M3) modes.
        Currently implements non-streaming WAV generation only.
        """
        from veena3modal.api.schemas import TTSGenerateRequest, AudioFormat
        from veena3modal.services import tts_runtime

        request_id = str(uuid.uuid4())
        start_time = time.time()

        # === AUTH CHECK ===
        if FeatureFlags.is_auth_enabled():
            # Extract API key from headers
            headers_dict = dict(request.headers)
            api_key = extract_api_key(headers_dict)

            if not api_key:
                return JSONResponse(
                    status_code=401,
                    content=create_error_response(
                        code=ErrorCode.INVALID_API_KEY,
                        message="API key required. Use 'Authorization: Bearer <key>' or 'X-API-Key: <key>' header.",
                        request_id=request_id,
                    ),
                    headers={"X-Request-ID": request_id},
                )

            # Validate API key
            validator = get_api_validator()
            key_hash = hash_api_key(api_key)
            auth_result = validator.validate(key_hash)

            if not auth_result.is_valid:
                status_code = 401 if auth_result.error_code == "INVALID_API_KEY" else 403
                return JSONResponse(
                    status_code=status_code,
                    content=create_error_response(
                        code=ErrorCode(auth_result.error_code),
                        message=auth_result.error_message or "Authentication failed",
                        request_id=request_id,
                    ),
                    headers={"X-Request-ID": request_id},
                )

            # === RATE LIMIT CHECK ===
            if FeatureFlags.is_rate_limiting_enabled():
                rate_limiter = get_rate_limiter()
                allowed, remaining, reset_after = rate_limiter.check(key_hash)

                if not allowed:
                    headers = rate_limiter.get_headers(allowed, remaining, reset_after)
                    headers["X-Request-ID"] = request_id
                    return JSONResponse(
                        status_code=429,
                        content=create_error_response(
                            code=ErrorCode.RATE_LIMIT_EXCEEDED,
                            message=f"Rate limit exceeded. Retry after {int(reset_after) + 1} seconds.",
                            request_id=request_id,
                        ),
                        headers=headers,
                    )

        # Parse and validate request body
        try:
            body = await request.json()
        except Exception as e:
            raise HTTPException(status_code=400, detail=f"Invalid JSON: {e}")

        try:
            req = TTSGenerateRequest(**body)
        except Exception as e:
            error_msg = str(e)
            # Extract validation error details
            return JSONResponse(
                status_code=400,
                content={
                    "error": {
                        "code": "VALIDATION_ERROR",
                        "message": error_msg,
                        "request_id": request_id,
                    }
                },
                headers={"X-Request-ID": request_id},
            )


Output:
Chunk ID: a9c308
Wall time: 0.0545 seconds
Process exited with code 0
Original token count: 2432
Output:
"""
WebSocket handler for TTS streaming.

Provides bidirectional WebSocket communication for TTS:
- Client sends JSON request with text and parameters
- Server streams audio chunks as binary frames
- Client can send control messages (cancel, ping)

Protocol:
1. Client connects to /v1/tts/ws
2. Client sends JSON: {"text": "...", "speaker": "...", ...}
3. Server sends binary audio chunks
4. Server sends JSON: {"event": "complete", "metrics": {...}}
5. Connection closes

Control messages:
- {"event": "cancel"} - Stop generation
- {"event": "ping"} - Keep-alive (server responds with pong)
"""

import asyncio
import json
import time
import uuid
from typing import Optional, Dict, Any
from dataclasses import dataclass
from enum import Enum

from veena3modal.shared.logging import (
    get_logger,
    set_request_context,
    clear_request_context,
    log_request_received,
    log_first_audio_emitted,
    log_request_completed,
    log_request_failed,
)
from veena3modal.shared.metrics import (
    record_request_received as metrics_request_received,
    record_request_completed as metrics_request_completed,
    record_request_failed as metrics_request_failed,
    record_ttfb,
    record_rtf,
    record_audio_duration,
    record_chunks_sent,
)

logger = get_logger(__name__)

class WSMessageType(str, Enum):
    """WebSocket message types."""
    # Client -> Server
    REQUEST = "request"
    CANCEL = "cancel"
    PING = "ping"

    # Server -> Client
    AUDIO_CHUNK = "audio_chunk"  # Binary frame
    HEADER = "header"  # WAV header info
    PROGRESS = "progress"  # Progress update
    COMPLETE = "complete"  # Generation complete
    ERROR = "error"  # Error occurred
    PONG = "pong"  # Response to ping

@dataclass
class WSRequest:
    """Parsed WebSocket TTS request."""
    text: str
    speaker: str = "male1"
    temperature: float = 0.8
    top_k: int = 50
    top_p: float = 1.0
    max_tokens: int = 4096
    repetition_penalty: float = 1.05
    seed: Optional[int] = None
    chunking: bool = False
    normalize: bool = True
    format: str = "wav"
    sample_rate: int = 16000

    @classmethod
    def from_dict(cls, data: Dict[str, Any]) -> "WSRequest":
        """Parse request from JSON dict."""
        return cls(
            text=data.get("text", ""),
            speaker=data.get("speaker", "male1"),
            temperature=data.get("temperature", 0.8),
            top_k=data.get("top_k", 50),
            top_p=data.get("top_p", 1.0),
            max_tokens=data.get("max_tokens", 4096),
            repetition_penalty=data.get("repetition_penalty", 1.05),
            seed=data.get("seed"),
            chunking=data.get("chunking", False),
            normalize=data.get("normalize", True),
            format=data.get("format", "wav"),
            sample_rate=data.get("sample_rate", 16000),
        )

    def validate(self) -> Optional[str]:
        """Validate request, return error message if invalid."""
        if not self.text or not self.text.strip():
            return "Text is required and cannot be empty"
        if len(self.text) > 50000:
            return f"Text too long: {len(self.text)} chars (max 50000)"
        if self.format != "wav":
            return f"WebSocket streaming only supports WAV format (got {self.format})"
        return None

def create_error_message(code: str, message: str, request_id: str) -> str:
    """Create JSON error message."""
    return json.dumps({
        "event": WSMessageType.ERROR.value,
        "error": {
            "code": code,
            "message": message,
            "request_id": request_id,
        }
    })

def create_header_message(
    request_id: str,
    sample_rate: int,
    model_version: str,
) -> str:
    """Create JSON header message (sent before audio)."""
    return json.dumps({
        "event": WSMessageType.HEADER.value,
        "request_id": request_id,
        "sample_rate": sample_rate,
        "format": "wav",
        "model_version": model_version,
    })

def create_progress_message(
    chunks_sent: int,
    bytes_sent: int,
    elapsed_ms: int,
) -> str:
    """Create JSON progress message."""
    return json.dumps({
        "event": WSMessageType.PROGRESS.value,
        "chunks_sent": chunks_sent,
        "bytes_sent": bytes_sent,
        "elapsed_ms": elapsed_ms,
    })

def create_complete_message(
    request_id: str,
    metrics: Dict[str, Any],
) -> str:
    """Create JSON completion message."""
    return json.dumps({
        "event": WSMessageType.COMPLETE.value,
        "request_id": request_id,
        "metrics": metrics,
    })

async def handle_websocket_tts(websocket, api_key: Optional[str] = None):
    """
    Handle WebSocket TTS streaming connection.

    This is the main handler called from FastAPI WebSocket route.

    Args:
        websocket: FastAPI WebSocket instance
        api_key: Optional API key for authentication
    """
    from veena3modal.api.auth import get_api_validator, hash_api_key
    from veena3modal.api.rate_limiter import get_rate_limiter
    from veena3modal.api.error_handlers import FeatureFlags
    from veena3modal.api.schemas import resolve_speaker_name
    from veena3modal.services import tts_runtime

    request_id = str(uuid.uuid4())
    set_request_context(request_id=request_id)

    # Accept the WebSocket connection
    await websocket.accept()
    logger.info(f"WebSocket connection accepted: {request_id}")

    # Track cancellation
    cancelled = False

    try:
        # === AUTH CHECK (if enabled) ===
        if FeatureFlags.is_auth_enabled() and api_key:
            validator = get_api_validator()
            key_hash = hash_api_key(api_key)
            auth_result = validator.validate(key_hash)

            if not auth_result.is_valid:
                await websocket.send_text(create_error_message(
                    code=auth_result.error_code or "AUTH_FAILED",
                    message=auth_result.error_message or "Authentication failed",
                    request_id=request_id,
                ))
                await websocket.close(code=1008)  # Policy violation
                return

            # Rate limit check
            if FeatureFlags.is_rate_limiting_enabled():
                rate_limiter = get_rate_limiter()
                allowed, remaining, reset_after = rate_limiter.check(key_hash)
                if not allowed:
                    await websocket.send_text(create_error_message(
                        code="RATE_LIMIT_EXCEEDED",
                        message=f"Rate limit exceeded. Retry after {int(reset_after) + 1} seconds.",
                        request_id=request_id,
                    ))
                    await websocket.close(code=1008)
                    return

        # === WAIT FOR REQUEST ===
        try:
            # Wait for client to send request (with timeout)
            raw_message = await asyncio.wait_for(
                websocket.receive_text(),
                timeout=30.0
            )
            message = json.loads(raw_message)
        except asyncio.TimeoutError:
            await websocket.send_text(create_error_message(
                code="TIMEOUT",
                message="No request received within 30 seconds",
                request_id=request_id,
            ))
            await websocket.close(code=1000)
            return
        except json.JSONDecodeError as e:
            await websocket.send_text(create_error_message(
                code="INVALID_JSON",
                message=f"Invalid JSON: {e}",
                request_id=request_id,
            ))
            await websocket.close(code=1003)  # Unsupported data
            return

        # Handle control messages
        event_type = message.get("event", "request")
        if event_type == "ping":
            await websocket.send_text(json.dumps({"event": "pong"}))
            # Wait for actual request
            raw_message = await websocket.receive_text()
            message = json.loads(raw_message)

        # === PARSE REQUEST ===
        req = WSRequest.from_dict(message)
        validation_error = req.validate()
        if validation_error:
            await websocket.send_text(create_error_message(
                code="VALIDATION_ERROR",
                message=validation_error,
                request_id=request_id,
            ))
            await websocket.close(code=1003)
            return

        # Resolve speaker
        speaker = resolve_speaker_name(req.speaker)

        # Log request
        log_request_received(
            request_id=request_id,
            text_length=len(req.text),
            speaker=speaker,
            stream=True,
            format="wav",
        )
        metrics_request_received(speaker=speaker, stream=True, format="wav")

        # === CHECK MODEL ===
        if not tts_runtime.is_initialized():
            await websocket.send_text(create_error_message(
                code="MODEL_NOT_LOADED",
                message="TTS model not initialized",
                request_id=request_id,
            ))
            await websocket.close(code=1011)  # Internal error
            return

        runtime = tts_runtime.get_runtime()
        model_version = runtime.model_version if runtime else "unknown"

        # === SEND HEADER INFO ===
        await websocket.send_text(create_header_message(
            request_id=request_id,
            sample_rate=req.sample_rate,
            model_version=model_version,
        ))

        # === STREAM AUDIO ===
        start_time = time.time()
        chunks_sent = 0


Output:
Chunk ID: 36c9be
Wall time: 0.0519 seconds
Process exited with code 0
Original token count: 9541
Output:
/home/ubuntu/veenaModal/veena3modal/app.py:7:- Per-container concurrency (vLLM batching)
/home/ubuntu/veenaModal/veena3modal/app.py:179:# vLLM continuous batching handles concurrent requests efficiently
/home/ubuntu/veenaModal/veena3modal/app.py:244:        Return FastAPI ASGI app for serving requests.
/home/ubuntu/veenaModal/scripts/detailed_word_analysis.py:20:import requests
/home/ubuntu/veenaModal/scripts/detailed_word_analysis.py:161:    response = requests.post(
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:4:In Modal, each container should load the model once and then handle many requests.
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:52:        This class is designed for async concurrency within a single container.
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:418:    active_requests = _with_inflight_delta(1)
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:430:                "request_inflight": active_requests,
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:516:        active_requests = _with_inflight_delta(-1)
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:518:            metrics["request_inflight"] = active_requests
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:519:            metrics["active_requests"] = active_requests
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:529:                    "active_requests": active_requests,
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:646:    active_requests = _with_inflight_delta(1)
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:657:                    "request_inflight": active_requests,
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:777:            metrics["active_requests"] = active_requests - 1
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:778:            metrics["request_inflight"] = active_requests - 1
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:786:        active_requests = _with_inflight_delta(-1)
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:787:        metrics["active_requests"] = active_requests
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:788:        metrics["request_inflight"] = active_requests
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:870:    active_requests = _with_inflight_delta(1)
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:877:                "request_inflight": active_requests,
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:970:        active_requests = _with_inflight_delta(-1)
/home/ubuntu/veenaModal/veena3modal/services/tts_runtime.py:980:                    "request_inflight": active_requests,
/home/ubuntu/veenaModal/scripts/measure_ttfb_detailed_v2.py:6:import requests
/home/ubuntu/veenaModal/scripts/measure_ttfb_detailed_v2.py:31:    response = requests.post(API_URL, json=payload, headers=headers, stream=True, timeout=30)
/home/ubuntu/veenaModal/veena3modal/services/credits.py:4:Lightweight, framework-agnostic credits calculation for TTS requests.
/home/ubuntu/veenaModal/veena3modal/services/sentence_store.py:2:Sentence Store Service - Supabase storage for TTS requests.
/home/ubuntu/veenaModal/veena3modal/services/sentence_store.py:21:TABLE_NAME = "tts_requests"
/home/ubuntu/veenaModal/scripts/validate_true_streaming.py:12:import requests
/home/ubuntu/veenaModal/scripts/validate_true_streaming.py:32:    response = requests.post(
/home/ubuntu/veenaModal/scripts/test_true_streaming.py:17:import requests
/home/ubuntu/veenaModal/scripts/test_true_streaming.py:39:    response = requests.post(
/home/ubuntu/veenaModal/scripts/measure_client_ttfb.py:7:import requests
/home/ubuntu/veenaModal/scripts/measure_client_ttfb.py:46:    response = requests.post(url, json=payload, headers=headers, stream=True)
/home/ubuntu/veenaModal/scripts/elevenlabs_asr_analysis.py:13:import requests
/home/ubuntu/veenaModal/scripts/elevenlabs_asr_analysis.py:55:        response = requests.post(url, headers=headers, files=files, data=data, timeout=180)
/home/ubuntu/veenaModal/scripts/test_all_features.py:15:import requests
/home/ubuntu/veenaModal/scripts/test_all_features.py:50:        response = requests.post(
/home/ubuntu/veenaModal/scripts/test_all_features.py:87:    response = requests.post(
/home/ubuntu/veenaModal/scripts/test_all_features.py:206:        ("Health endpoint", lambda: requests.get(f"{API_BASE_URL}/health").status_code == 200),
/home/ubuntu/veenaModal/scripts/test_all_features.py:235:    response = requests.post(
/home/ubuntu/veenaModal/scripts/test_all_features.py:245:    response = requests.post(
/home/ubuntu/veenaModal/scripts/test_all_features.py:256:    response = requests.post(
/home/ubuntu/veenaModal/scripts/test_all_features.py:269:        r = requests.get(f"{API_BASE_URL}/health", timeout=5)
/home/ubuntu/veenaModal/veena3modal/core/pipeline.py:33:    Return True when benchmark logs are enabled.
/home/ubuntu/veenaModal/scripts/stress_test_production.py:12:Runs the same concurrency ramp as the baseline test for direct comparison.
/home/ubuntu/veenaModal/scripts/stress_test_production.py:88:# === GPU Monitor (reused from stress_test_local.py) ===
/home/ubuntu/veenaModal/scripts/stress_test_production.py:227:async def run_level(base_url, num_requests, concurrency, stream, gpu_monitor=None):
/home/ubuntu/veenaModal/scripts/stress_test_production.py:228:    sem = asyncio.Semaphore(concurrency)
/home/ubuntu/veenaModal/scripts/stress_test_production.py:243:    limits = httpx.Limits(max_connections=concurrency + 10, max_keepalive_connections=concurrency)
/home/ubuntu/veenaModal/scripts/stress_test_production.py:245:        tasks = [req(i, client) for i in range(num_requests)]
/home/ubuntu/veenaModal/scripts/stress_test_production.py:260:    rps = num_requests / wall if wall > 0 else 0
/home/ubuntu/veenaModal/scripts/stress_test_production.py:265:    label = f"{concurrency}c-production-{mode}"
/home/ubuntu/veenaModal/scripts/stress_test_production.py:269:    print(f"  {num_requests} requests, {concurrency} concurrent, {mode}")
/home/ubuntu/veenaModal/scripts/stress_test_production.py:272:    print(f"  Success:     {len(successes)}/{num_requests} ({len(successes)/num_requests:.0%})")
/home/ubuntu/veenaModal/scripts/stress_test_production.py:306:        "level": label, "concurrency": concurrency, "requests": num_requests,
/home/ubuntu/veenaModal/scripts/stress_test_production.py:307:        "mode": mode, "success_rate": len(successes) / num_requests if num_requests else 0,
/home/ubuntu/veenaModal/scripts/stress_test_production.py:338:    print(f"\nWarming up (3 requests with real sentences)...")
/home/ubuntu/veenaModal/scripts/stress_test_production.py:406:    parser.add_argument("--output", default="stress_test_production.json")
/home/ubuntu/veenaModal/veena3modal/tests/integration/test_sentence_store.py:138:        """Verify tts_requests table exists in Supabase."""
/home/ubuntu/veenaModal/veena3modal/tests/integration/test_sentence_store.py:144:            result = store.client.table('tts_requests').select('request_id').limit(1).execute()
/home/ubuntu/veenaModal/veena3modal/tests/integration/test_sentence_store.py:149:            pytest.skip(f"tts_requests table not found: {e}")
/home/ubuntu/veenaModal/scripts/wer_test_with_emotions.py:21:import requests
/home/ubuntu/veenaModal/scripts/wer_test_with_emotions.py:155:    response = requests.post(
/home/ubuntu/veenaModal/veena3modal/core/dual_engine.py:12:    - Engine A and Engine B both handle full requests
/home/ubuntu/veenaModal/veena3modal/core/dual_engine.py:66:    Routes requests across multiple vLLM engine instances.
/home/ubuntu/veenaModal/veena3modal/core/dual_engine.py:132:# This doubles prefill capacity, cutting streaming TTFB under concurrency by ~50%.
/home/ubuntu/veenaModal/scripts/comprehensive_validation.py:24:import requests
/home/ubuntu/veenaModal/scripts/comprehensive_validation.py:163:        response = requests.get(f"{API_BASE_URL}/health", timeout=5)
/home/ubuntu/veenaModal/scripts/comprehensive_validation.py:192:        response = requests.post(url, json=payload, headers=headers, timeout=timeout)
/home/ubuntu/veenaModal/scripts/comprehensive_validation.py:365:        response = requests.post(url, json=payload, stream=True, timeout=60)
/home/ubuntu/veenaModal/scripts/comprehensive_validation.py:393:def test_concurrency():
/home/ubuntu/veenaModal/scripts/comprehensive_validation.py:398:    concurrency_levels = [1, 3, 5]
/home/ubuntu/veenaModal/scripts/comprehensive_validation.py:402:    for n_concurrent in concurrency_levels:
/home/ubuntu/veenaModal/scripts/comprehensive_validation.py:403:        log_info(f"\nTesting with {n_concurrent} concurrent requests...")
/home/ubuntu/veenaModal/scripts/comprehensive_validation.py:438:    # Check if all requests succeeded
/home/ubuntu/veenaModal/scripts/comprehensive_validation.py:442:        log_success("All concurrent requests succeeded")
/home/ubuntu/veenaModal/scripts/comprehensive_validation.py:445:        log_warning("Some concurrent requests failed")
/home/ubuntu/veenaModal/scripts/comprehensive_validation.py:501:        ("Concurrency", test_concurrency),
/home/ubuntu/veenaModal/scripts/test_fix.py:7:import requests
/home/ubuntu/veenaModal/scripts/test_fix.py:62:    response = requests.post(
/home/ubuntu/veenaModal/scripts/validation/final_validation.py:7:import requests
/home/ubuntu/veenaModal/scripts/validation/final_validation.py:21:    response = requests.get(f"{BASE_URL}/health")
/home/ubuntu/veenaModal/scripts/validation/final_validation.py:37:    response = requests.post(f"{BASE_URL}/v1/tts/generate", json=payload)
/home/ubuntu/veenaModal/scripts/validation/final_validation.py:54:    response = requests.post(f"{BASE_URL}/v1/tts/generate", json=payload)
/home/ubuntu/veenaModal/scripts/validation/final_validation.py:70:    response = requests.post(f"{BASE_URL}/v1/tts/generate", json=payload)
/home/ubuntu/veenaModal/scripts/validation/final_validation.py:90:    response = requests.post(f"{BASE_URL}/v1/tts/generate", json=payload, timeout=60)
/home/ubuntu/veenaModal/scripts/validation/final_validation.py:125:    response = requests.post(f"{BASE_URL}/v1/tts/generate", json=payload, timeout=180)
/home/ubuntu/veenaModal/veena3modal/tests/performance/__init__.py:1:"""Performance/benchmark tests for `veena3modal` (TTFB/RTF regressions)."""
/home/ubuntu/veenaModal/veena3modal/core/bicodec_batch_decoder.py:2:BiCodec Batch Decoder - Batches concurrent decode requests into single GPU forward passes.
/home/ubuntu/veenaModal/veena3modal/core/bicodec_batch_decoder.py:10:    A background worker collects requests within a timeout window,
/home/ubuntu/veenaModal/veena3modal/core/bicodec_batch_decoder.py:51:    Batches concurrent BiCodec decode requests into single GPU forward passes.
/home/ubuntu/veenaModal/veena3modal/core/bicodec_batch_decoder.py:69:            max_batch: Max requests to batch together
/home/ubuntu/veenaModal/veena3modal/core/bicodec_batch_decoder.py:82:        self.total_requests = 0
/home/ubuntu/veenaModal/veena3modal/core/bicodec_batch_decoder.py:128:            # Collect requests up to max_batch or timeout
/home/ubuntu/veenaModal/veena3modal/core/bicodec_batch_decoder.py:178:            self.total_requests += len(batch)
/home/ubuntu/veenaModal/veena3modal/core/bicodec_batch_decoder.py:250:        avg_batch_size = self.total_requests / self.total_batches if self.total_batches > 0 else 0
/home/ubuntu/veenaModal/veena3modal/core/bicodec_batch_decoder.py:254:            "total_requests": self.total_requests,
/home/ubuntu/veenaModal/scripts/profile_pipeline.py:15:For concurrency: measures contention points under parallel load.
/home/ubuntu/veenaModal/scripts/profile_pipeline.py:21:    # Concurrency profiling (1, 5, 10, 20 parallel requests)
/home/ubuntu/veenaModal/scripts/profile_pipeline.py:22:    python scripts/profile_pipeline.py --concurrency 1,5,10,20
/home/ubuntu/veenaModal/scripts/profile_pipeline.py:547:    num_requests: int,
/home/ubuntu/veenaModal/scripts/profile_pipeline.py:552:    """Profile multiple concurrent requests."""
/home/ubuntu/veenaModal/scripts/profile_pipeline.py:563:    profiles = await asyncio.gather(*[run_one(i) for i in range(num_requests)])
/home/ubuntu/veenaModal/scripts/profile_pipeline.py:567:def print_concurrency_summary(profiles: List[RequestProfile], concurrency: int, stream: bool):
/home/ubuntu/veenaModal/scripts/profile_pipeline.py:576:    print(f"  CONCURRENCY={concurrency} ({mode}), {len(profiles)} requests")
/home/ubuntu/veenaModal/scripts/profile_pipeline.py:599:    print(f"\n  STAGE BREAKDOWN (avg across {len(profiles)} requests):")
/home/ubuntu/veenaModal/scripts/profile_pipeline.py:614:        print(f"\n  BICODEC DECODE (all calls across {len(profiles)} requests):")
/home/ubuntu/veenaModal/scripts/profile_pipeline.py:673:    concurrency_levels = [int(x) for x in args.concurrency.split(",")]
/home/ubuntu/veenaModal/scripts/profile_pipeline.py:676:    if not args.concurrency_only:
/home/ubuntu/veenaModal/scripts/profile_pipeline.py:694:    for conc in concurrency_levels:
/home/ubuntu/veenaModal/scripts/profile_pipeline.py:702:            logger.info(f"\nProfiling {conc} concurrent {'streaming' if stream_mode else 'non-streaming'} requests...")
/home/ubuntu/veenaModal/scripts/profile_pipeline.py:704:            print_concurrency_summary(profiles, conc, stream_mode)
/home/ubuntu/veenaModal/scripts/profile_pipeline.py:714:    parser.add_argument("--concurrency", default="1,5,10,20", help="Concurrency levels (comma-separated)")
/home/ubuntu/veenaModal/scripts/profile_pipeline.py:717:    parser.add_argument("--concurrency-only", action="store_true", help="Skip single-request profiles")
/home/ubuntu/veenaModal/scripts/stress_test_local.py:11:- Higher concurrency ceiling (single-machine, no rate limits)
/home/ubuntu/veenaModal/scripts/stress_test_local.py:16:    python scripts/stress_test_local.py
/home/ubuntu/veenaModal/scripts/stress_test_local.py:19:    python scripts/stress_test_local.py --quick
/home/ubuntu/veenaModal/scripts/stress_test_local.py:22:    python scripts/stress_test_local.py --levels 1,5,10,20
/home/ubuntu/veenaModal/scripts/stress_test_local.py:25:    python scripts/stress_test_local.py --stream-only
/home/ubuntu/veenaModal/scripts/stress_test_local.py:28:    python scripts/stress_test_local.py --no-stream
/home/ubuntu/veenaModal/scripts/stress_test_local.py:31:    python scripts/stress_test_local.py --url http://localhost:8080
/home/ubuntu/veenaModal/scripts/stress_test_local.py:53:# Test texts at different lengths to stress different parts of the pipeline
/home/ubuntu/veenaModal/scripts/stress_test_local.py:180:    concurrency: int
/home/ubuntu/veenaModal/scripts/stress_test_local.py:181:    total_requests: int
/home/ubuntu/veenaModal/scripts/stress_test_local.py:202:        return self.total_requests / self.wall_time_seconds if self.wall_time_seconds > 0 else 0
/home/ubuntu/veenaModal/scripts/stress_test_local.py:400:    num_requests: int,
/home/ubuntu/veenaModal/scripts/stress_test_local.py:401:    concurrency: int,
/home/ubuntu/veenaModal/scripts/stress_test_local.py:408:    level_name = f"{concurrency}c-{text_category}-{mode}"
/home/ubuntu/veenaModal/scripts/stress_test_local.py:412:        concurrency=concurrency,
/home/ubuntu/veenaModal/scripts/stress_test_local.py:413:        total_requests=num_requests,
/home/ubuntu/veenaModal/scripts/stress_test_local.py:418:    semaphore = asyncio.Semaphore(concurrency)
/home/ubuntu/veenaModal/scripts/stress_test_local.py:440:        max_connections=concurrency + 10,
/home/ubuntu/veenaModal/scripts/stress_test_local.py:441:        max_keepalive_connections=concurrency,
/home/ubuntu/veenaModal/scripts/stress_test_local.py:444:        tasks = [limited_request(i, client) for i in range(num_requests)]
/home/ubuntu/veenaModal/scripts/stress_test_local.py:473:    print(f"  {level.total_requests} requests, {level.concurrency} concurrent, {level.mode}")
/home/ubuntu/veenaModal/scripts/stress_test_local.py:476:    print(f"  Success:     {len(level.successes)}/{level.total_requests} ({level.success_rate:.0%})")
/home/ubuntu/veenaModal/scripts/stress_test_local.py:545:async def run_stress_test(args):
/home/ubuntu/veenaModal/scripts/stress_test_local.py:546:    """Run the full stress test suite."""
/home/ubuntu/veenaModal/scripts/stress_test_local.py:564:    # Parse concurrency levels
/home/ubuntu/veenaModal/scripts/stress_test_local.py:581:    print(f"\nWarming up (2 requests)...")
/home/ubuntu/veenaModal/scripts/stress_test_local.py:582:    warmup = await run_level(base_url, num_requests=2, concurrency=1, text_category="short", stream=False)
/home/ubuntu/veenaModal/scripts/stress_test_local.py:597:            # Scale requests: enough to get meaningful stats, cap at reasonable total
/home/ubuntu/veenaModal/scripts/stress_test_local.py:598:            num_requests = max(conc * 3, 10)  # At least 3x concurrency or 10
/home/ubuntu/veenaModal/scripts/stress_test_local.py:599:            num_requests = min(num_requests, 200)  # Cap at 200
/home/ubuntu/veenaModal/scripts/stress_test_local.py:603:            print(f"\n>>> {conc} concurrent, {num_requests} total requests, text={text_cat}, stream={stream}")
/home/ubuntu/veenaModal/scripts/stress_test_local.py:607:                num_requests=num_requests,
/home/ubuntu/veenaModal/scripts/stress_test_local.py:608:                concurrency=conc,
/home/ubuntu/veenaModal/scripts/stress_test_local.py:640:            "concurrency": level.concurrency,
/home/ubuntu/veenaModal/scripts/stress_test_local.py:641:            "total_requests": level.total_requests,
/home/ubuntu/veenaModal/scripts/stress_test_local.py:665:    parser.add_argument("--levels", default="1,5,10,20,50", help="Comma-separated concurrency levels (default: 1,5,10,20,50)")
/home/ubuntu/veenaModal/scripts/stress_test_local.py:670:    parser.add_argument("--output", default="stress_test_results.json", help="Output JSON path")
/home/ubuntu/veenaModal/scripts/stress_test_local.py:677:    asyncio.run(run_stress_test(args))
/home/ubuntu/veenaModal/scripts/test_chunking_no_emotions.py:11:import requests
/home/ubuntu/veenaModal/scripts/test_chunking_no_emotions.py:91:    response = requests.post(
/home/ubuntu/veenaModal/veena3modal/tests/performance/test_benchmarks.py:2:Performance benchmarks for TTS service.
/home/ubuntu/veenaModal/veena3modal/tests/performance/test_benchmarks.py:17:def _can_run_benchmarks() -> Tuple[bool, str]:
/home/ubuntu/veenaModal/veena3modal/tests/performance/test_benchmarks.py:18:    """Check if benchmarks can run."""
/home/ubuntu/veenaModal/veena3modal/tests/performance/test_benchmarks.py:33:CAN_RUN, SKIP_REASON = _can_run_benchmarks()
/home/ubuntu/veenaModal/veena3modal/tests/performance/test_benchmarks.py:38:    """Ensure runtime is initialized for benchmarks."""
/home/ubuntu/veenaModal/veena3modal/tests/performance/test_benchmarks.py:54:    Time-to-First-Byte benchmarks.
/home/ubuntu/veenaModal/veena3modal/tests/performance/test_benchmarks.py:104:    Real-Time Factor benchmarks.
/home/ubuntu/veenaModal/veena3modal/tests/performance/test_benchmarks.py:162:    Throughput benchmarks.
/home/ubuntu/veenaModal/veena3modal/tests/performance/test_benchmarks.py:168:    async def test_concurrent_requests(self, initialized_runtime):
/home/ubuntu/veenaModal/veena3modal/tests/performance/test_benchmarks.py:169:        """Measure throughput with concurrent requests."""
/home/ubuntu/veenaModal/veena3modal/tests/performance/test_benchmarks.py:197:        print(f"\nConcurrent throughput ({len(texts)} requests):")
/home/ubuntu/veenaModal/veena3modal/tests/performance/test_benchmarks.py:208:    Memory usage benchmarks.
/home/ubuntu/veenaModal/veena3modal/tests/performance/test_benchmarks.py:259:    Cache performance benchmarks.
/home/ubuntu/veenaModal/veena3modal/tests/performance/test_benchmarks.py:266:        limiter = RateLimiter(requests_per_minute=1000)
/home/ubuntu/veenaModal/scripts/elevenlabs_verify_fix.py:9:import requests
/home/ubuntu/veenaModal/scripts/elevenlabs_verify_fix.py:85:        response = requests.post(
/home/ubuntu/veenaModal/scripts/elevenlabs_verify_fix.py:98:            response = requests.post(url, files=files, headers=headers)
/home/ubuntu/veenaModal/scripts/final_asr_verification.py:9:import requests
/home/ubuntu/veenaModal/scripts/final_asr_verification.py:86:        response = requests.post(url, files=files, headers=headers)
/home/ubuntu/veenaModal/scripts/test_correct_format.py:6:import requests
/home/ubuntu/veenaModal/scripts/test_correct_format.py:92:response = requests.post(
/home/ubuntu/veenaModal/veena3modal/core/streaming_pipeline.py:578:                "concurrency_mode": "single",
/home/ubuntu/veenaModal/scripts/test_original_bug.py:7:import requests
/home/ubuntu/veenaModal/scripts/test_original_bug.py:96:    response = requests.post(
/home/ubuntu/veenaModal/scripts/measure_ttfb_detailed.py:9:import requests
/home/ubuntu/veenaModal/scripts/measure_ttfb_detailed.py:43:        response = requests.post(API_URL, json=payload, headers=headers, stream=True, timeout=30)
/home/ubuntu/veenaModal/scripts/measure_ttfb_detailed.py:135:        response = requests.post(API_URL, json=payload, headers=headers, timeout=30)
/home/ubuntu/veenaModal/veena3modal/tests/modal_live/test_deployed_endpoint.py:287:    def test_concurrent_requests(self):
/home/ubuntu/veenaModal/veena3modal/tests/modal_live/test_deployed_endpoint.py:314:        async def run_concurrent(n_requests=3):
/home/ubuntu/veenaModal/veena3modal/tests/modal_live/test_deployed_endpoint.py:316:                tasks = [make_request(client, i) for i in range(n_requests)]
/home/ubuntu/veenaModal/veena3modal/tests/modal_live/test_deployed_endpoint.py:328:        assert success_count > 0, "All concurrent requests failed"
/home/ubuntu/veenaModal/external/AP-BWE/train/train_48k.py:24:torch.backends.cudnn.benchmark = True
/home/ubuntu/veenaModal/veena3modal/tests/edge_cases/test_concurrent_access.py:23:        limiter = RateLimiter(requests_per_minute=10, window_seconds=60)
/home/ubuntu/veenaModal/veena3modal/tests/edge_cases/test_concurrent_access.py:31:        # Run 20 concurrent requests
/home/ubuntu/veenaModal/veena3modal/tests/edge_cases/test_concurrent_access.py:45:        limiter = RateLimiter(requests_per_minute=5, window_seconds=60)
/home/ubuntu/veenaModal/veena3modal/tests/edge_cases/test_concurrent_access.py:51:        # 10 different keys, 5 requests each
/home/ubuntu/veenaModal/veena3modal/tests/edge_cases/test_concurrent_access.py:60:        # All should be allowed (each key gets 5 requests, limit is 5)
/home/ubuntu/veenaModal/veena3modal/tests/edge_cases/test_concurrent_access.py:61:        assert all(results), "All requests should be allowed"
/home/ubuntu/veenaModal/veena3modal/tests/edge_cases/test_concurrent_access.py:131:        metrics_module._requests_total = None
/home/ubuntu/veenaModal/veena3modal/tests/edge_cases/test_concurrent_access.py:144:        # Metrics should have recorded all requests
/home/ubuntu/veenaModal/veena3modal/tests/edge_cases/test_concurrent_access.py:185:        limiter = RateLimiter(requests_per_minute=100)
/home/ubuntu/veenaModal/veena3modal/tests/unit/test_schemas.py:193:    """Test unicode handling in requests."""
/home/ubuntu/veenaModal/external/AP-BWE/train/train_16k.py:24:torch.backends.cudnn.benchmark = True
/home/ubuntu/veenaModal/veena3modal/shared/metrics.py:36:_requests_total: Optional['Counter'] = None
/home/ubuntu/veenaModal/veena3modal/shared/metrics.py:37:_requests_completed: Optional['Counter'] = None
/home/ubuntu/veenaModal/veena3modal/shared/metrics.py:38:_requests_failed: Optional['Counter'] = None
/home/ubuntu/veenaModal/veena3modal/shared/metrics.py:63:    global _requests_total, _requests_completed, _requests_failed
/home/ubuntu/veenaModal/veena3modal/shared/metrics.py:71:    _requests_total = Counter(
/home/ubuntu/veenaModal/veena3modal/shared/metrics.py:72:        'veena3_tts_requests_total',
/home/ubuntu/veenaModal/veena3modal/shared/metrics.py:73:        'Total TTS requests received',
/home/ubuntu/veenaModal/veena3modal/shared/metrics.py:78:    _requests_completed = Counter(
/home/ubuntu/veenaModal/veena3modal/shared/metrics.py:79:        'veena3_tts_requests_completed_total',
/home/ubuntu/veenaModal/veena3modal/shared/metrics.py:80:        'Total TTS requests completed successfully',
/home/ubuntu/veenaModal/veena3modal/shared/metrics.py:85:    _requests_failed = Counter(
/home/ubuntu/veenaModal/veena3modal/shared/metrics.py:86:        'veena3_tts_requests_failed_total',
/home/ubuntu/veenaModal/veena3modal/shared/metrics.py:87:        'Total TTS requests failed',
/home/ubuntu/veenaModal/veena3modal/shared/metrics.py:161:    if _requests_total is None:
/home/ubuntu/veenaModal/veena3modal/shared/metrics.py:164:    if _requests_total is not None:
/home/ubuntu/veenaModal/veena3modal/shared/metrics.py:165:        _requests_total.labels(
/home/ubuntu/veenaModal/veena3modal/shared/metrics.py:179:    if _requests_completed is None:
/home/ubuntu/veenaModal/veena3modal/shared/metrics.py:182:    if _requests_completed is not None:
/home/ubuntu/veenaModal/veena3modal/shared/metrics.py:183:        _requests_completed.labels(
/home/ubuntu/veenaModal/veena3modal/shared/metrics.py:196:    if _requests_failed is None:
/home/ubuntu/veenaModal/veena3modal/shared/metrics.py:199:    if _requests_failed is not None:
/home/ubuntu/veenaModal/veena3modal/shared/metrics.py:200:        _requests_failed.labels(
/home/ubuntu/veenaModal/veena3modal/tests/edge_cases/__init__.py:1:"""Edge case tests for `veena3modal` (max size, unicode, cancellations, concurrency)."""
/home/ubuntu/veenaModal/veena3modal/tests/modal_live/test_load.py:5:- Throughput (requests per second)
/home/ubuntu/veenaModal/veena3modal/tests/modal_live/test_load.py:51:    total_requests: int = 0
/home/ubuntu/veenaModal/veena3modal/tests/modal_live/test_load.py:52:    successful_requests: int = 0
/home/ubuntu/veenaModal/veena3modal/tests/modal_live/test_load.py:53:    failed_requests: int = 0
/home/ubuntu/veenaModal/veena3modal/tests/modal_live/test_load.py:61:        if self.total_requests == 0:
/home/ubuntu/veenaModal/veena3modal/tests/modal_live/test_load.py:63:        return self.successful_requests / self.total_requests
/home/ubuntu/veenaModal/veena3modal/tests/modal_live/test_load.py:66:    def requests_per_second(self) -> float:
/home/ubuntu/veenaModal/veena3modal/tests/modal_live/test_load.py:69:        return self.total_requests / self.total_duration_seconds
/home/ubuntu/veenaModal/veena3modal/tests/modal_live/test_load.py:105:            f"Total Requests: {self.total_requests}",
/home/ubuntu/veenaModal/veena3modal/tests/modal_live/test_load.py:106:            f"Successful: {self.successful_requests} ({self.success_rate:.1%})",
/home/ubuntu/veenaModal/veena3modal/tests/modal_live/test_load.py:107:            f"Failed: {self.failed_requests}",
/home/ubuntu/veenaModal/veena3modal/tests/modal_live/test_load.py:109:            f"Throughput: {self.requests_per_second:.2f} req/s",
/home/ubuntu/veenaModal/veena3modal/tests/modal_live/test_load.py:206:    num_requests: int,
/home/ubuntu/veenaModal/veena3modal/tests/modal_live/test_load.py:207:    concurrency: int,
/home/ubuntu/veenaModal/veena3modal/tests/modal_live/test_load.py:213:    Run load test with specified concurrency.
/home/ubuntu/veenaModal/veena3modal/tests/modal_live/test_load.py:216:        num_requests: Total number of requests to make
/home/ubuntu/veenaModal/veena3modal/tests/modal_live/test_load.py:217:        concurrency: Maximum concurrent requests
/home/ubuntu/veenaModal/veena3modal/tests/modal_live/test_load.py:226:    semaphore = asyncio.Semaphore(concurrency)
/home/ubuntu/veenaModal/veena3modal/tests/modal_live/test_load.py:236:        tasks = [limited_request(client) for _ in range(num_requests)]
/home/ubuntu/veenaModal/veena3modal/tests/modal_live/test_load.py:242:    result.total_requests = num_requests
/home/ubuntu/veenaModal/veena3modal/tests/modal_live/test_load.py:246:            result.failed_requests += 1
/home/ubuntu/veenaModal/veena3modal/tests/modal_live/test_load.py:250:                result.successful_requests += 1
/home/ubuntu/veenaModal/veena3modal/tests/modal_live/test_load.py:255:                result.failed_requests += 1
/home/ubuntu/veenaModal/veena3modal/tests/modal_live/test_load.py:268:    def test_sequential_5_requests(self):
/home/ubuntu/veenaModal/veena3modal/tests/modal_live/test_load.py:269:        """5 sequential requests to establish baseline."""
/home/ubuntu/veenaModal/veena3modal/tests/modal_live/test_load.py:271:            num_requests=5,
/home/ubuntu/veenaModal/veena3modal/tests/modal_live/test_load.py:272:            concurrency=1,
/home/ubuntu/veenaModal/veena3modal/tests/modal_live/test_load.py:284:    """Load tests with 10 concurrent requests."""
/home/ubuntu/veenaModal/veena3modal/tests/modal_live/test_load.py:286:    def test_concurrent_10_requests(self):
/home/ubuntu/veenaModal/veena3modal/tests/modal_live/test_load.py:287:        """10 concurrent requests."""
/home/ubuntu/veenaModal/veena3modal/tests/modal_live/test_load.py:289:            num_requests=10,
/home/ubuntu/veenaModal/veena3modal/tests/modal_live/test_load.py:290:            concurrency=10,
/home/ubuntu/veenaModal/veena3modal/tests/modal_live/test_load.py:291:            text="Concurrent load test with ten requests.",
/home/ubuntu/veenaModal/veena3modal/tests/modal_live/test_load.py:297:        # Under concurrency, latency may be higher
/home/ubuntu/veenaModal/veena3modal/tests/modal_live/test_load.py:303:    """Load tests with 50 concurrent requests."""
/home/ubuntu/veenaModal/veena3modal/tests/modal_live/test_load.py:306:    def test_concurrent_50_requests(self):
/home/ubuntu/veenaModal/veena3modal/tests/modal_live/test_load.py:307:        """50 concurrent requests (stress test)."""
/home/ubuntu/veenaModal/veena3modal/tests/modal_live/test_load.py:309:            num_requests=50,
/home/ubuntu/veenaModal/veena3modal/tests/modal_live/test_load.py:310:            concurrency=50,
/home/ubuntu/veenaModal/veena3modal/tests/modal_live/test_load.py:311:            text="High concurrency stress test.",
/home/ubuntu/veenaModal/veena3modal/tests/modal_live/test_load.py:325:    def test_sustained_20_requests_5_concurrent(self):
/home/ubuntu/veenaModal/veena3modal/tests/modal_live/test_load.py:326:        """20 total requests with 5 concurrent (sustained load)."""
/home/ubuntu/veenaModal/veena3modal/tests/modal_live/test_load.py:328:            num_requests=20,
/home/ubuntu/veenaModal/veena3modal/tests/modal_live/test_load.py:329:            concurrency=5,
/home/ubuntu/veenaModal/veena3modal/tests/modal_live/test_load.py:330:            text="Sustained load test with moderate concurrency.",
/home/ubuntu/veenaModal/veena3modal/tests/modal_live/test_load.py:336:        print(f"Throughput: {result.requests_per_second:.2f} req/s")
/home/ubuntu/veenaModal/veena3modal/tests/modal_live/test_load.py:344:        """5 concurrent streaming requests."""
/home/ubuntu/veenaModal/veena3modal/tests/modal_live/test_load.py:346:            num_requests=5,
/home/ubuntu/veenaModal/veena3modal/tests/modal_live/test_load.py:347:            concurrency=5,
/home/ubuntu/veenaModal/veena3modal/tests/modal_live/test_load.py:356:        assert result.total_requests == 5
/home/ubuntu/veenaModal/veena3modal/tests/modal_live/test_load.py:370:    warmup = await run_load_test(num_requests=1, concurrency=1)
/home/ubuntu/veenaModal/veena3modal/tests/modal_live/test_load.py:387:    for total, concurrency, name in configs:
/home/ubuntu/veenaModal/veena3modal/tests/modal_live/test_load.py:388:        print(f"\n🧪 Running: {name} ({total} requests, {concurrency} concurrent)...")
/home/ubuntu/veenaModal/veena3modal/tests/modal_live/test_load.py:391:            num_requests=total,
/home/ubuntu/veenaModal/veena3modal/tests/modal_live/test_load.py:392:            concurrency=concurrency,
/home/ubuntu/veenaModal/veena3modal/tests/modal_live/test_load.py:398:        print(f"   ✅ {result.successful_requests}/{result.total_requests} succeeded")
/home/ubuntu/veenaModal/veena3modal/tests/modal_live/test_load.py:400:        print(f"   📈 {result.requests_per_second:.2f} req/s")
/home/ubuntu/veenaModal/veena3modal/tests/modal_live/test_load.py:410:        print(f"{name:<25} {result.success_rate:.0%}       {result.p50_latency:<10.0f} {result.p95_latency:<10.0f} {result.requests_per_second:<10.2f}")
/home/ubuntu/veenaModal/veena3modal/api/auth.py:84:    rate_limit: int = 60  # Default requests per minute
/home/ubuntu/veenaModal/veena3modal/api/schemas.py:2:Pydantic schemas for TTS API requests/responses.
/home/ubuntu/veenaModal/veena3modal/tests/unit/test_rate_limiter.py:20:        limiter = RateLimiter(requests_per_minute=60)
/home/ubuntu/veenaModal/veena3modal/tests/unit/test_rate_limiter.py:22:        assert limiter.requests_per_minute == 60
/home/ubuntu/veenaModal/veena3modal/tests/unit/test_rate_limiter.py:28:        limiter = RateLimiter(requests_per_minute=60)
/home/ubuntu/veenaModal/veena3modal/tests/unit/test_rate_limiter.py:34:    def test_rate_limiter_tracks_requests(self):
/home/ubuntu/veenaModal/veena3modal/tests/unit/test_rate_limiter.py:38:        limiter = RateLimiter(requests_per_minute=10)
/home/ubuntu/veenaModal/veena3modal/tests/unit/test_rate_limiter.py:49:        limiter = RateLimiter(requests_per_minute=5)
/home/ubuntu/veenaModal/veena3modal/tests/unit/test_rate_limiter.py:66:        limiter = RateLimiter(requests_per_minute=3)
/home/ubuntu/veenaModal/veena3modal/tests/unit/test_rate_limiter.py:85:        limiter = RateLimiter(requests_per_minute=60, window_seconds=1)
/home/ubuntu/veenaModal/veena3modal/tests/unit/test_rate_limiter.py:110:        limiter = RateLimiter(requests_per_minute=60)
/home/ubuntu/veenaModal/veena3modal/tests/unit/test_rate_limiter.py:124:        limiter = RateLimiter(requests_per_minute=1)
/home/ubuntu/veenaModal/veena3modal/tests/unit/test_rate_limiter.py:148:        """Disabled limiter should always allow requests."""
/home/ubuntu/veenaModal/veena3modal/tests/unit/test_rate_limiter.py:153:        # Should allow unlimited requests
/home/ubuntu/veenaModal/veena3modal/tests/unit/test_rate_limiter.py:166:        limiter = RateLimiter(requests_per_minute=60, window_seconds=1)
/home/ubuntu/veenaModal/veena3modal/tests/unit/test_rate_limiter.py:179:        assert len(limiter._requests) < 100
/home/ubuntu/veenaModal/veena3modal/api/error_handlers.py:151:    to prevent the container from accepting more requests while it drains.
/home/ubuntu/veenaModal/veena3modal/api/rate_limiter.py:34:        requests_per_minute: Maximum requests allowed per window
/home/ubuntu/veenaModal/veena3modal/api/rate_limiter.py:41:        requests_per_minute: int = 60,
/home/ubuntu/veenaModal/veena3modal/api/rate_limiter.py:49:            requests_per_minute: Max requests per window
/home/ubuntu/veenaModal/veena3modal/api/rate_limiter.py:51:            enabled: If False, all requests are allowed
/home/ubuntu/veenaModal/veena3modal/api/rate_limiter.py:53:        self.requests_per_minute = requests_per_minute
/home/ubuntu/veenaModal/veena3modal/api/rate_limiter.py:58:        self._requests: Dict[str, List[float]] = defaultdict(list)
/home/ubuntu/veenaModal/veena3modal/api/rate_limiter.py:73:            - remaining: Number of requests remaining in window
/home/ubuntu/veenaModal/veena3modal/api/rate_limiter.py:77:            return True, self.requests_per_minute, 0.0
/home/ubuntu/veenaModal/veena3modal/api/rate_limiter.py:84:            self._requests[key] = [
/home/ubuntu/veenaModal/veena3modal/api/rate_limiter.py:85:                ts for ts in self._requests[key]
/home/ubuntu/veenaModal/veena3modal/api/rate_limiter.py:89:            current_count = len(self._requests[key])
/home/ubuntu/veenaModal/veena3modal/api/rate_limiter.py:91:            if current_count >= self.requests_per_minute:
/home/ubuntu/veenaModal/veena3modal/api/rate_limiter.py:93:                oldest_in_window = min(self._requests[key]) if self._requests[key] else now
/home/ubuntu/veenaModal/veena3modal/api/rate_limiter.py:98:            self._requests[key].append(now)
/home/ubuntu/veenaModal/veena3modal/api/rate_limiter.py:99:            remaining = self.requests_per_minute - current_count - 1
/home/ubuntu/veenaModal/veena3modal/api/rate_limiter.py:111:        for key, timestamps in self._requests.items():
/home/ubuntu/veenaModal/veena3modal/api/rate_limiter.py:117:                self._requests[key] = valid
/home/ubuntu/veenaModal/veena3modal/api/rate_limiter.py:120:            del self._requests[key]
/home/ubuntu/veenaModal/veena3modal/api/rate_limiter.py:144:            remaining: Remaining requests in window
/home/ubuntu/veenaModal/veena3modal/api/rate_limiter.py:151:            "X-RateLimit-Limit": str(self.requests_per_minute),
/home/ubuntu/veenaModal/veena3modal/api/rate_limiter.py:163:            if key in self._requests:
/home/ubuntu/veenaModal/veena3modal/api/rate_limiter.py:164:                del self._requests[key]
/home/ubuntu/veenaModal/veena3modal/api/rate_limiter.py:176:    - RATE_LIMIT_REQUESTS_PER_MINUTE: Max requests (default: 60)
/home/ubuntu/veenaModal/veena3modal/api/rate_limiter.py:185:        requests_per_minute = int(os.environ.get("RATE_LIMIT_REQUESTS_PER_MINUTE", "60"))
/home/ubuntu/veenaModal/veena3modal/api/rate_limiter.py:189:            requests_per_minute=requests_per_minute,
/home/ubuntu/veenaModal/veena3modal/api/rate_limiter.py:196:                "requests_per_minute": requests_per_minute,
/home/ubuntu/veenaModal/veena3modal/tests/unit/test_api_key_cache.py:162:        """Bypass mode should allow all requests."""
/home/ubuntu/veenaModal/veena3modal/tests/unit/test_sentence_store.py:205:            mock_client.table.assert_called_with('tts_requests')
/home/ubuntu/veenaModal/veena3modal/tests/unit/test_credits.py:43:        """Minimum charge should be applied for tiny requests."""


I found existing stress scripts; next I’m targeting existing `/v1/tts/generate` and websocket paths so each request carries a stable `request_id` through runtime and returns structured timing metrics (including in-flight concurrency), then I’ll add a dedicated local stress script that reads and persists per-request timing by concurrency level.


Output:
Chunk ID: 619c2f
Wall time: 0.0508 seconds
Process exited with code 0
Original token count: 1018
Output:
5:Veena3 inference/streaming code from the Django project.
23:    log_first_audio_emitted,
24:    log_request_completed,
104:        description="High-quality multilingual Text-to-Speech API with true streaming support.",
178:        Supports streaming (M4) and non-streaming (M3) modes.
179:        Currently implements non-streaming WAV generation only.
184:        request_id = str(uuid.uuid4())
199:                        request_id=request_id,
201:                    headers={"X-Request-ID": request_id},
216:                        request_id=request_id,
218:                    headers={"X-Request-ID": request_id},
228:                    headers["X-Request-ID"] = request_id
234:                            request_id=request_id,
256:                        "request_id": request_id,
259:                headers={"X-Request-ID": request_id},
270:                        "request_id": request_id,
273:                headers={"X-Request-ID": request_id},
291:        set_request_context(request_id=request_id)
295:            request_id=request_id,
298:            stream=req.stream,
301:        metrics_request_received(speaker=speaker, stream=req.stream, format=req.format)
304:        if req.stream:
305:            return await _handle_streaming_request(req, request_id, text, speaker, start_time)
307:        # Format encoding will be applied after generation (non-streaming only)
308:        # For streaming, only WAV is supported
315:            # Generate speech (non-streaming)
317:                audio_bytes, metrics = await tts_runtime.generate_speech_chunked(
330:                audio_bytes, metrics = await tts_runtime.generate_speech(
349:                            "request_id": request_id,
352:                    headers={"X-Request-ID": request_id},
368:                "X-Request-ID": request_id,
387:                request_id=request_id,
390:                stream=False,
405:            log_request_completed(
406:                request_id=request_id,
417:                stream=False,
419:            record_ttfb(ttfb_seconds=metrics.get("ttfb_ms", 0) / 1000, speaker=speaker, stream=False)
476:                request_id=request_id,
494:                        "request_id": request_id,
497:                headers={"X-Request-ID": request_id},
500:    async def _handle_streaming_request(req, request_id: str, text: str, speaker: str, start_time: float):
502:        Handle streaming TTS request.
505:        True streaming: first bytes sent ASAP before full audio is generated.
511:        # Check if streaming pipeline is available
513:        if runtime is None or runtime.streaming_pipeline is None:
520:                        "request_id": request_id,
523:                headers={"X-Request-ID": request_id},
534:                        "request_id": request_id,
537:                headers={"X-Request-ID": request_id},
541:        set_request_context(request_id=request_id)
543:        # Create async generator for streaming response
544:        async def audio_stream_generator():
550:                async for audio_chunk, metrics in tts_runtime.generate_speech_streaming(
570:                        log_first_audio_emitted(
571:                            request_id=request_id,
575:                        record_ttfb(ttfb_seconds=ttfb_ms / 1000, speaker=speaker, stream=True)
579:                            request_id=request_id,
582:                            stream=True,
592:                            # audio_duration_seconds not known yet for streaming
602:                log_request_completed(
603:                    request_id=request_id,
614:                    stream=True,
622:                # Log streaming failure
624:                    request_id=request_id,
635:                # Can't return error once streaming started
640:        # Note: Some headers can't be set dynamically during streaming
641:        # TTFB and RTF will be logged but not in headers for streaming
645:            "X-Request-ID": request_id,
657:        return StreamingResponse(
658:            audio_stream_generator(),


Output:
Chunk ID: 3771ec
Wall time: 0.0508 seconds
Process exited with code 0
Original token count: 47
Output:
153:def create_complete_message(
342:            async for audio_chunk, metrics in tts_runtime.generate_speech_streaming(
406:        await websocket.send_text(create_complete_message(


sed -n '280,460p' /home/ubuntu/veenaModal/veena3modal/api/fastapi_app.py


sed -n '440,720p' /home/ubuntu/veenaModal/veena3modal/api/fastapi_app.py


sed -n '320,460p' /home/ubuntu/veenaModal/veena3modal/api/websocket_handler.py