Explore the pipecat codebase at /home/ubuntu/pipecat thoroughly. I need to understand:
1. The overall project structure - what kind of application is this?
2. Key entry points and main application files
3. How connections/sessions are managed
4. Any WebSocket or real-time communication patterns
5. How resources (audio/video streams, AI services, etc.) are allocated and cleaned up

List all important files and directories with brief descriptions. Be very thorough - check package.json, requirements.txt, Dockerfiles, main app files, configuration files, etc.


I'll conduct a thorough exploration of the pipecat codebase to understand its structure, architecture, and key components. Let me start with a systematic scan.


Error: EISDIR: illegal operation on a directory, read


Bash

find /home/ubuntu/pipecat -maxdepth 2 -type f -name "*.json" -o -name "*.txt" -o -name "Dockerfile*" -o -name "*.yaml" -o -name "*.yml" -o -name "*.cfg" -o -name "*.ini" -o -name "*.toml" | head -30

Search for configuration and setup files


<tool_use_error>Sibling tool call errored</tool_use_error>


List root directory contents


Let me use the correct approach to explore the directory:


List root directory contents with details


total 188
drwxrwxr-x 17 ubuntu ubuntu  4096 Feb 12 07:13 .
drwxr-x--- 30 ubuntu ubuntu  4096 Feb 12 12:35 ..
-rw-rw-r--  1 ubuntu ubuntu   631 Feb  4 22:39 assign_voices.sql
drwxrwxr-x  2 ubuntu ubuntu  4096 Feb 12 07:13 bot
-rwxrwxr-x  1 ubuntu ubuntu  4065 Feb 12 07:13 check_idle_pods_memory.sh
-rwxrwxr-x  1 ubuntu ubuntu  3540 Feb 12 07:13 check_total_sessions.sh
-rwxrwxr-x  1 ubuntu ubuntu  4412 Feb 12 07:13 cleanup_dead_pods.sh
drwxrwxr-x  3 ubuntu ubuntu  4096 Feb  4 22:39 cloudflare
-rw-rw-r--  1 ubuntu ubuntu   521 Feb  4 22:39 CLUSTER_ACCESS.md
drwxrwxr-x  2 ubuntu ubuntu  4096 Feb  4 22:39 credentials
drwxrwxr-x  2 ubuntu ubuntu  4096 Feb  4 22:39 .cursorrules
-rw-rw-r--  1 ubuntu ubuntu 13348 Feb  4 22:39 daily_stats.json
-rw-rw-r--  1 ubuntu ubuntu   317 Feb  4 22:39 dashboard_stats.json
drwxrwxr-x  6 ubuntu ubuntu  4096 Feb 12 07:13 deployments
-rw-rw-r--  1 ubuntu ubuntu  2440 Feb  4 22:39 DEPLOYMENT_VERSIONS.md
-rw-rw-r--  1 ubuntu ubuntu  2125 Feb 10 20:34 .env
drwxrwxr-x  4 ubuntu ubuntu  4096 Feb 12 10:28 fnHandlers
-rw-rw-r--  1 ubuntu ubuntu  2398 Feb  4 22:39 gcp-new-service-account.json
-rw-rw-r--  1 ubuntu ubuntu  2369 Feb  4 22:39 gcp-service-account.json
drwxrwxr-x  8 ubuntu ubuntu  4096 Feb 12 12:33 .git
-rw-rw-r--  1 ubuntu ubuntu   522 Feb  4 22:39 .gitignore
drwxrwxr-x  2 ubuntu ubuntu  4096 Feb 12 10:28 migrations
-rwxrwxr-x  1 ubuntu ubuntu  3549 Feb 12 07:13 monitor_connections.sh
-rwxrwxr-x  1 ubuntu ubuntu 17876 Feb  4 22:39 pipecat-ctl.sh
-rw-rw-r--  1 ubuntu ubuntu 12130 Feb  4 22:39 pipecat_flow_example.py
drwxrwxr-x  2 ubuntu ubuntu  4096 Feb 10 20:34 prompts
-rw-rw-r--  1 ubuntu ubuntu  5488 Feb  4 22:39 README.md
drwxrwxr-x  2 ubuntu ubuntu  4096 Feb  4 22:39 references
-rw-rw-r--  1 ubuntu ubuntu   857 Feb 10 20:34 requirements.txt
drwxrwxr-x  2 ubuntu ubuntu  4096 Feb 12 07:13 routes
drwxrwxr-x  2 ubuntu ubuntu  4096 Feb 12 10:28 scripts
-rw-rw-r--  1 ubuntu ubuntu  3357 Feb 12 07:13 server.py
drwxrwxr-x  2 ubuntu ubuntu  4096 Feb  4 22:39 services
-rwxrwxr-x  1 ubuntu ubuntu  1222 Feb  4 22:39 test_tts_speakers.sh
drwxrwxr-x  2 ubuntu ubuntu  4096 Feb  4 22:39 tts_samples
drwxrwxr-x  2 ubuntu ubuntu  4096 Feb 12 10:28 utils


/home/ubuntu/pipecat/README.md
/home/ubuntu/pipecat/DEPLOYMENT_VERSIONS.md
/home/ubuntu/pipecat/CLUSTER_ACCESS.md
/home/ubuntu/pipecat/bot/__init__.py
/home/ubuntu/pipecat/bot/image_upload.py
/home/ubuntu/pipecat/dashboard_stats.json
/home/ubuntu/pipecat/cloudflare/status-api/node_modules/.mf/cf.json
/home/ubuntu/pipecat/daily_stats.json
/home/ubuntu/pipecat/bot/observers.py
/home/ubuntu/pipecat/bot/pipeline.py
/home/ubuntu/pipecat/deployments/datadog/datadog.yaml
/home/ubuntu/pipecat/deployments/datadog/python.d/conf.yaml
/home/ubuntu/pipecat/fnHandlers/__init__.py
/home/ubuntu/pipecat/fnHandlers/clients/__init__.py
/home/ubuntu/pipecat/fnHandlers/clients/azure_image_client.py
/home/ubuntu/pipecat/fnHandlers/azure_edit_image_handler.py
/home/ubuntu/pipecat/fnHandlers/cf_image_handler.py
/home/ubuntu/pipecat/deployments/k8s/grafana-ingress.yaml
/home/ubuntu/pipecat/deployments/k8s/grafana-dashboard.json
/home/ubuntu/pipecat/deployments/k8s/maya-pipecat-dashboard.json
/home/ubuntu/pipecat/deployments/k8s/deployment-test.yaml
/home/ubuntu/pipecat/gcp-service-account.json
/home/ubuntu/pipecat/gcp-new-service-account.json
/home/ubuntu/pipecat/fnHandlers/clients/exceptions.py
/home/ubuntu/pipecat/fnHandlers/clients/edit_service_client.py
/home/ubuntu/pipecat/fnHandlers/video_handler.py
/home/ubuntu/pipecat/fnHandlers/edit_image_handler.py
/home/ubuntu/pipecat/fnHandlers/utils/prompt_writer.py
/home/ubuntu/pipecat/fnHandlers/utils/__init__.py
/home/ubuntu/pipecat/fnHandlers/utils/image_describer.py
/home/ubuntu/pipecat/fnHandlers/utils/celebrity_reference.py
/home/ubuntu/pipecat/fnHandlers/utils/rtvi_events.py
/home/ubuntu/pipecat/fnHandlers/utils/handler_helpers.py
/home/ubuntu/pipecat/fnHandlers/utils/limit_handler.py
/home/ubuntu/pipecat/prompts/image_describer_prompt.txt
/home/ubuntu/pipecat/prompts/prompt_writer_template.txt
/home/ubuntu/pipecat/pipecat_flow_example.py
/home/ubuntu/pipecat/scripts/upload_explore_images.py
/home/ubuntu/pipecat/routes/video.py
/home/ubuntu/pipecat/routes/contacts.py
/home/ubuntu/pipecat/routes/device.py
/home/ubuntu/pipecat/routes/support.py
/home/ubuntu/pipecat/routes/app.py
/home/ubuntu/pipecat/routes/prompt.py
/home/ubuntu/pipecat/routes/subscription.py
/home/ubuntu/pipecat/routes/characters.py
/home/ubuntu/pipecat/routes/referral.py
/home/ubuntu/pipecat/routes/explore.py
/home/ubuntu/pipecat/services/mistral_stt.py
/home/ubuntu/pipecat/services/__init__.py
/home/ubuntu/pipecat/utils/character_generator.py
/home/ubuntu/pipecat/utils/__init__.py
/home/ubuntu/pipecat/utils/custom_tts.py
/home/ubuntu/pipecat/utils/session_state.py
/home/ubuntu/pipecat/utils/daily_client.py
/home/ubuntu/pipecat/utils/models.py
/home/ubuntu/pipecat/utils/s3_utils.py
/home/ubuntu/pipecat/deployments/k8s/redis-secret.yaml
/home/ubuntu/pipecat/deployments/k8s/datadog-daemonset.yaml
/home/ubuntu/pipecat/deployments/k8s/ingress.yaml
/home/ubuntu/pipecat/deployments/k8s/deployment.yaml
/home/ubuntu/pipecat/deployments/monitoring/values.yaml
/home/ubuntu/pipecat/deployments/monitoring/dashboards/maya-pipecat-overview.json
/home/ubuntu/pipecat/scripts/seed_characters.py
/home/ubuntu/pipecat/prompts/prompt_global_agent.txt
/home/ubuntu/pipecat/routes/health.py
/home/ubuntu/pipecat/requirements.txt
/home/ubuntu/pipecat/utils/constants.py
/home/ubuntu/pipecat/utils/logging_config.py
/home/ubuntu/pipecat/utils/resource_monitor.py
/home/ubuntu/pipecat/bot/main.py
/home/ubuntu/pipecat/bot/config.py
/home/ubuntu/pipecat/bot/session_handlers.py
/home/ubuntu/pipecat/server.py
/home/ubuntu/pipecat/routes/__init__.py
/home/ubuntu/pipecat/routes/sessions.py
/home/ubuntu/pipecat/routes/bot.py
/home/ubuntu/pipecat/utils/redis_client.py
/home/ubuntu/pipecat/fnHandlers/clients/cf_flux_client.py
/home/ubuntu/pipecat/fnHandlers/image_handler.py
/home/ubuntu/pipecat/utils/config_sync.py
/home/ubuntu/pipecat/utils/cache_layer.py
/home/ubuntu/pipecat/utils/turso_client.py


Good! Now let me examine the key files to understand the project structure:


     1→# Pipecat Voice Agent
     2→
     3→Real-time conversational AI agent powered by Pipecat, integrated with Veena3 TTS.
     4→
     5→## Features
     6→
     7→
     8→- 🎙 **Real-time Voice Conversations** - WebRTC support (WebSocket DISABLED)
     9→- 🗣 **Custom TTS Integration** - Veena3 streaming TTS with zero-buffering
    10→- 🤖 **Multiple LLM Options** - OpenAI, Google Gemini, OpenRouter
    11→- 🎯 **Smart Turn Detection** - Advanced VAD with Silero
    12→- 📊 **RTVI Protocol Support** - Full RTVI compatibility
    13→- 🔄 **User Idle Detection** - Automatic nudges for inactive users
    14→- ⚠ **WebSocket Disabled** - Use WebRTC transport instead
    15→
    16→## Architecture
    17→
    18→```
    19→User → WebRTC (Daily.co) → Pipecat → LLM (OpenRouter/OpenAI/Gemini)
    20→                              ↓
    21→                         STT (Deepgram)
    22→                              ↓
    23→                    TTS (Veena3 API - Zero Buffer)
    24→                              ↓
    25→                    Audio Stream (24kHz PCM)
    26→                              ↓
    27→                         Immediate Yield
    28→```
    29→
    30→**Note**: WebSocket transport is disabled. All connections use WebRTC.
    31→
    32→## Voice Configuration
    33→
    34→**Current Voice:** Australian Female (Late 20s)
    35→- Low-pitch, slightly raspy timbre
    36→- Upbeat pacing, approachable tone
    37→- Full of energy and curiosity
    38→
    39→**TTS Settings:**
    40→- Endpoint: Veena3 Streaming API
    41→- Sample Rate: 24kHz
    42→- Temperature: 0.4
    43→- Top P: 0.9
    44→- Supports emotion tags: `<excited>`, `<curious>`, `<gasp>`, etc.
    45→
    46→## Installation
    47→
    48→### Prerequisites
    49→- Python 3.10+
    50→- Virtual environment
    51→
    52→### Setup
    53→
    54→1. **Clone the repository:**
    55→```bash
    56→git clone https://github.com/MayaResearch/pipecat.git
    57→cd pipecat
    58→```
    59→
    60→2. **Create virtual environment:**
    61→```bash
    62→python -m venv venv
    63→source venv/bin/activate  # On Windows: venv\Scripts\activate
    64→```
    65→
    66→3. **Install dependencies:**
    67→```bash
    68→pip install -r requirements.txt
    69→```
    70→
    71→4. **Configure environment variables:**
    72→Create a `.env` file:
    73→```bash
    74→# TTS API
    75→TTS_API_URL=http://localhost:8000/v1/tts/stream
    76→
    77→# STT (Deepgram)
    78→DEEPGRAM_API_KEY=your_deepgram_api_key
    79→
    80→# LLM (choose one or multiple)
    81→OPENAI_API_KEY=your_openai_api_key
    82→OPENROUTER_API_KEY=your_openrouter_api_key
    83→
    84→# WebRTC (Daily.co)
    85→DAILY_API_KEY=your_daily_api_key
    86→
    87→# Server mode
    88→WEBSOCKET_SERVER=fast_api
    89→```
    90→
    91→## Running the Server
    92→
    93→### Standard Mode (FastAPI + WebSocket):
    94→```bash
    95→python server.py
    96→```
    97→
    98→Server will start on `http://0.0.0.0:3006`
    99→
   100→### Production Mode (with logs):
   101→```bash
   102→mkdir -p logs
   103→nohup python server.py > logs/server.log 2>&1 &
   104→```
   105→
   106→## API Endpoints
   107→
   108→### ⚠ WebSocket NOT Supported
   109→**WS** `/ws`
   110→
   111→WebSocket connections are immediately rejected with an error message.
   112→
   113→**Response:**
   114→```json
   115→{
   116→  "error": "websocket_not_supported",
   117→  "message": "WebSocket transport is not currently supported. Please use WebRTC transport instead.",
   118→  "supported_transports": ["webrtc"],
   119→  "webrtc_endpoint": "/connect"
   120→}
   121→```
   122→
   123→### WebRTC Connection (PRIMARY - Daily.co)
   124→**POST** `/connect`
   125→
   126→Creates a Daily.co room and returns connection details.
   127→
   128→**Request:**
   129→```json
   130→{
   131→  "language": "te"  // Options: "te", "en", "hi"
   132→}
   133→```
   134→
   135→**Response:**
   136→```json
   137→{
   138→  "room_url": "https://maya-c.daily.co/xxxxx",
   139→  "token": "xxxxxxxx"
   140→}
   141→```
   142→
   143→## Configuration Files
   144→
   145→### Bot Implementations
   146→- `bot_fast_api.py` - FastAPI bot (HTTP mode, no transport)
   147→- `bot_webrtc_daily.py` - WebRTC/Daily.co bot (PRIMARY)
   148→- `bot_websocket_server.py` - ⚠ **DEPRECATED** - WebSocket not supported
   149→
   150→### Core Services
   151→- `custom_tts.py` - Veena3 TTS integration (zero-buffering, immediate streaming)
   152→- `tts_verifier.py` - TTS output verification (stub)
   153→- `server.py` - Main FastAPI server (WebSocket disabled, WebRTC only)
   154→
   155→## Customization
   156→
   157→### Change Voice
   158→Edit the `voice_description` in `bot_fast_api.py` or `bot_webrtc_daily.py`:
   159→
   160→```python
   161→tts = CustomTTSService(
   162→    api_url=os.getenv("TTS_API_URL", "http://localhost:8000/v1/tts/stream"),
   163→    voice_description="Your custom voice description here",
   164→    temperature=0.4,
   165→    # ... other params
   166→)
   167→```
   168→
   169→### Change LLM
   170→Uncomment the desired LLM in the bot files:
   171→- OpenAI
   172→- Google Gemini
   173→- OpenRouter (default)
   174→
   175→### Adjust System Prompt
   176→Modify the `SYSTEM_INSTRUCTION` in the bot files to change agent behavior.
   177→
   178→## Logging
   179→
   180→Logs are written to:
   181→- `logs/server.log` - Server startup and connection logs
   182→- `logs/bot.log` - Detailed bot operation logs (DEBUG level)
   183→
   184→## Troubleshooting
   185→
   186→### TTS 404 Errors
   187→Ensure `TTS_API_URL` in `.env` points to a valid endpoint:
   188→```bash
   189→TTS_API_URL=http://localhost:8000/v1/tts/stream
   190→```
   191→
   192→### WebRTC Connection Issues
   193→Verify `DAILY_API_KEY` is set correctly in `.env`
   194→
   195→### Audio Quality Issues
   196→Check sample rate matches between TTS (24kHz) and bot configuration
   197→
   198→## Production Deployment
   199→
   200→For production deployment with nginx:
   201→- Use SSL/TLS certificates
   202→- Configure proper CORS settings
   203→- Enable rate limiting
   204→- Set up monitoring and alerting
   205→
   206→See `COMPLETE_SETUP.md` for full production setup guide.
   207→
   208→## License
   209→
   210→Copyright (c) 2025, Maya Research
   211→
   212→## Important Notes
   213→
   214→### WebSocket Disabled
   215→WebSocket transport has been permanently disabled in favor of WebRTC for:
   216→- Lower latency
   217→- Better audio quality
   218→- NAT traversal support
   219→- Industry-standard implementation
   220→
   221→See `WEBSOCKET_DISABLED.md` for full details on the migration.
   222→
   223→### Zero-Buffering TTS
   224→The TTS integration has been optimized for immediate streaming:
   225→- No 20ms frame buffering
   226→- Chunks sent as received from API
   227→- WebRTC handles packetization
   228→- Minimum latency configuration
   229→
   230→## Support
   231→
   232→For issues and questions, please open an issue on GitHub.
   233→
   234→


     1→# Pipecat and core dependencies
     2→pipecat-ai[daily,openai,google,rnnoise,deepgram,koala]>=0.0.99
     3→
     4→# FastAPI and web server
     5→fastapi==0.115.5
     6→uvicorn[standard]==0.32.1
     7→python-multipart==0.0.18
     8→aiohttp
     9→
    10→# LLM and AI services
    11→google-genai>=1.50.0  # For Gemini support
    12→
    13→# Audio processing
    14→numpy==1.26.4
    15→onnxruntime==1.17.1  # Pinned - newer versions have Silero VAD compatibility issues
    16→
    17→# Utilities
    18→python-dotenv==1.0.1
    19→loguru==0.7.3
    20→Pillow>=10.0.0  # For image compression/resizing
    21→
    22→# Storage and Database
    23→boto3
    24→libsql-client  # Turso database client
    25→redis[hiredis]>=5.0.0  # Redis cache layer (ElastiCache)
    26→
    27→# Datadog (optional - for logging, traces, metrics)
    28→ddtrace  # APM traces
    29→datadog  # Custom metrics
    30→
    31→# Video generation
    32→replicate  # Replicate API for video generation
    33→
    34→# Additional STT providers
    35→deepgram-sdk  # Deepgram STT
    36→mistralai[realtime]>=1.0.0
    37→


     1→"""Maya Global Agent API Server."""
     2→
     3→import asyncio
     4→import time
     5→from contextlib import asynccontextmanager
     6→from dotenv import load_dotenv
     7→from fastapi import FastAPI
     8→from fastapi.middleware.cors import CORSMiddleware
     9→from loguru import logger
    10→
    11→from utils import constants
    12→from utils.logging_config import configure_logging, log_startup_complete, log_shutdown
    13→from utils.config_sync import ConfigSyncManager
    14→from utils.redis_client import init_redis, close_redis
    15→
    16→# Load environment variables
    17→load_dotenv(override=True)
    18→_startup_time = time.time()
    19→configure_logging()
    20→
    21→# Import all route modules
    22→from routes import (
    23→    bot_router,
    24→    health_router,
    25→    explore_router,
    26→    video_router,
    27→    subscription_router,
    28→    app_router,
    29→    device_router,
    30→    prompt_router,
    31→    support_router,
    32→    referral_router,
    33→    contacts_router,
    34→    characters_router,
    35→    sessions_router,
    36→    cleanup_active_sessions,
    37→)
    38→
    39→# Initialize config sync manager
    40→config_sync_manager = ConfigSyncManager(sync_interval=300)
    41→
    42→
    43→@asynccontextmanager
    44→async def lifespan(app: FastAPI):
    45→    """Handles FastAPI startup and shutdown."""
    46→    # Initialize Redis cache layer (before config sync so it can use Redis)
    47→    await init_redis()
    48→
    49→    await config_sync_manager.start()
    50→    startup_duration_ms = (time.time() - _startup_time) * 1000
    51→    log_startup_complete(startup_duration_ms)
    52→    yield
    53→    log_shutdown()
    54→
    55→    # Graceful shutdown: cleanup active bot sessions
    56→    await cleanup_active_sessions()
    57→
    58→    await config_sync_manager.stop()
    59→    await close_redis()
    60→    logger.info("SERVER_STOPPED: status=graceful")
    61→
    62→
    63→# Create FastAPI app
    64→app = FastAPI(
    65→    title="Maya Global Agent API",
    66→    description="Hindi-only voice AI with creative generation capabilities and character system",
    67→    version="2.0.0",
    68→    lifespan=lifespan
    69→)
    70→
    71→# Add CORS middleware
    72→app.add_middleware(
    73→    CORSMiddleware,
    74→    allow_origins=["*"],
    75→    allow_credentials=True,
    76→    allow_methods=["*"],
    77→    allow_headers=["*"],
    78→)
    79→
    80→# Include routers
    81→# Bot connection (no prefix)
    82→app.include_router(bot_router)
    83→
    84→# Health check (no prefix)
    85→app.include_router(health_router)
    86→
    87→# API routes with /api prefix
    88→app.include_router(explore_router, prefix="/api")
    89→app.include_router(video_router, prefix="/api")
    90→app.include_router(subscription_router, prefix="/api")
    91→app.include_router(app_router, prefix="/api")
    92→app.include_router(device_router, prefix="/api")
    93→app.include_router(prompt_router, prefix="/api")
    94→app.include_router(support_router, prefix="/api")
    95→app.include_router(referral_router, prefix="/api")
    96→app.include_router(contacts_router, prefix="/api")
    97→app.include_router(characters_router, prefix="/api")
    98→app.include_router(sessions_router, prefix="/api")
    99→
   100→
   101→async def main():
   102→    """Run the FastAPI server."""
   103→    import uvicorn
   104→    try:
   105→        log_config = {
   106→            "version": 1,
   107→            "disable_existing_loggers": True,
   108→            "handlers": {},
   109→            "root": {"handlers": [], "level": "INFO"}
   110→        }
   111→        config = uvicorn.Config(
   112→            app,
   113→            host=constants.SERVER_HOST,
   114→            port=constants.SERVER_PORT,
   115→            log_config=log_config
   116→        )
   117→        server = uvicorn.Server(config)
   118→        await server.serve()
   119→    except asyncio.CancelledError:
   120→        logger.info("Server cancelled (shutdown).")
   121→
   122→
   123→if __name__ == "__main__":
   124→    asyncio.run(main())
   125→


     1→import asyncio
     2→import time
     3→from dotenv import load_dotenv
     4→from loguru import logger
     5→
     6→from pipecat.pipeline.runner import PipelineRunner
     7→
     8→from utils import constants
     9→from utils.session_state import SessionState
    10→
    11→from bot.config import load_system_instruction, cleanup_session, BotSessionState, ChunkState, TranscriptManager
    12→from bot.pipeline import create_vad_analyzer, create_transport, create_stt_service, create_llm_service, create_tts_service, create_context_and_tools, create_processors, create_pipeline, create_task
    13→from bot.session_handlers import create_audio_data_handler, create_transcript_handler, create_client_message_handler, create_cleanup_if_no_user, create_participant_joined_handler, create_idle_timeout_handler, create_app_review_trigger, create_first_participant_joined_handler, create_participant_left_handler, create_client_disconnected_handler, create_client_ready_handler
    14→from fnHandlers import create_generate_image_handler, create_azure_edit_image_handler, create_cf_image_handler, create_video_handler, create_video_from_image_handler
    15→
    16→load_dotenv()
    17→
    18→
    19→async def run_bot_webrtc_global_agent(room_url: str, token: str, device_id: str = None, room_name: str = None, version: str = None, character_id: str = None):
    20→    bot_start_time = time.time()
    21→    device_id = device_id or "common"
    22→    session_id = room_name
    23→
    24→    session_state = SessionState()
    25→    await session_state.start_queue_processor()
    26→
    27→    # Initialize bot session
    28→    bot_session = BotSessionState(session_id, device_id, bot_start_time)
    29→    bot_session.character_id = character_id
    30→
    31→    # Load character info if character_id is provided
    32→    character_voice = None  # Will use default if None
    33→    if character_id:
    34→        from utils import turso_client
    35→        character = await turso_client.get_character_by_id(character_id)
    36→        if character:
    37→            bot_session.character_name = character.get('display_name')
    38→            character_voice = character.get('voice_id')  # Get character's voice
    39→            # Characters don't have image generation - only conversation
    40→            logger.info(f"SESSION_START: character={character_id} name={character.get('display_name')} voice={character_voice} mode=conversation_only")
    41→        else:
    42→            logger.warning(f"CHARACTER_NOT_FOUND: character={character_id} falling_back_to_maya")
    43→            bot_session.character_id = None  # Fall back to Maya
    44→    else:
    45→        # Maya session - full features (images + conversation)
    46→        logger.info(f"SESSION_START: Maya (global agent) with images and conversation")
    47→
    48→    # Load system instruction based on character
    49→    system_instruction = await load_system_instruction(bot_session.character_id)
    50→
    51→    # Check if video generation is enabled for this user
    52→    # Video is only enabled if: version == "explore" AND global config allows it AND user is video_premium
    53→    from utils.turso_client import is_video_generation_enabled, is_video_premium_user
    54→    video_global_enabled = is_video_generation_enabled()
    55→
    56→    # Only consider video if version is "explore"
    57→    is_explore_session = version == "explore"
    58→
    59→    # Check if user has video_premium access
    60→    is_video_premium = False
    61→    if video_global_enabled and is_explore_session and device_id and device_id != 'common':
    62→        try:
    63→            is_video_premium = await is_video_premium_user(device_id)
    64→        except Exception as e:
    65→            logger.error(f"VIDEO_PREMIUM_CHECK_FAILED: device_id={device_id} error={e}")
    66→            is_video_premium = False
    67→
    68→    # Video enabled only for video_premium users in explore sessions with global config on
    69→    video_enabled = video_global_enabled and is_explore_session and is_video_premium
    70→
    71→    logger.info(f"VIDEO_CHECK: session_id={session_id} device_id={device_id} global_enabled={video_global_enabled} version={version} is_explore={is_explore_session} is_video_premium={is_video_premium} video_enabled={video_enabled}")
    72→
    73→    # Store video_enabled in bot_session so image_upload can check it
    74→    bot_session.video_enabled = video_enabled
    75→
    76→    vad_analyzer = create_vad_analyzer(session_id)
    77→    bot_session.vad_analyzer = vad_analyzer
    78→
    79→    # Create services and store references IMMEDIATELY on bot_session after each creation.
    80→    # This ensures cleanup_session can release them even if a later create_* call throws.
    81→    transport = create_transport(room_url, token, vad_analyzer)
    82→    bot_session.transport = transport
    83→
    84→    stt = create_stt_service()
    85→    bot_session.stt_service = stt
    86→
    87→    llm = create_llm_service(system_instruction)
    88→    bot_session.llm_service = llm
    89→
    90→    tts = create_tts_service(speaker=character_voice)  # Use character's voice if available
    91→    session_state.tts_service = tts
    92→
    93→    # Create context with tools based on character capabilities
    94→    context_aggregator = create_context_and_tools(
    95→        llm,
    96→        system_instruction,
    97→        include_video=video_enabled,
    98→        can_generate_images=bot_session.can_generate_images(),
    99→        can_edit_images=bot_session.can_edit_images()
   100→    )
   101→    bot_session.context_aggregator = context_aggregator
   102→
   103→    rtvi, transcript, audio_buffer = create_processors()
   104→    transcript_manager = TranscriptManager()
   105→    chunk_state = ChunkState()
   106→
   107→    pipeline = create_pipeline(transport, stt, context_aggregator, rtvi, llm, tts, audio_buffer, transcript)
   108→    bot_session.pipeline = pipeline
   109→    task = create_task(pipeline, rtvi, bot_session)
   110→
   111→    audio_buffer.event_handler("on_audio_data")(create_audio_data_handler(bot_session, chunk_state))
   112→
   113→    trigger_app_review_if_eligible = create_app_review_trigger(bot_session, rtvi)
   114→
   115→    # Register image generation handlers only if character can generate images (Maya only)
   116→    if bot_session.can_generate_images():
   117→        llm.register_function("generate_image", create_generate_image_handler(
   118→            session_state, rtvi, bot_session, transcript_manager, task, llm, trigger_app_review_if_eligible
   119→        ))
   120→        logger.debug(f"FUNCTION_REGISTERED: generate_image for maya")
   121→
   122→        try:
   123→            llm.register_function("cf_image", create_cf_image_handler(
   124→                session_state, rtvi, bot_session, transcript_manager, task, llm, trigger_app_review_if_eligible
   125→            ))
   126→            logger.debug(f"FUNCTION_REGISTERED: cf_image for maya")
   127→        except Exception as e:
   128→            logger.error(f"Failed to register cf_image handler: {e}", exc_info=True)
   129→
   130→    # Register edit handlers only if character can edit images (Maya only)
   131→    if bot_session.can_edit_images():
   132→        try:
   133→            llm.register_function("azure_edit_image", create_azure_edit_image_handler(
   134→                session_state, rtvi, llm, bot_session, transcript_manager, task, trigger_app_review_if_eligible
   135→            ))
   136→            logger.debug(f"FUNCTION_REGISTERED: azure_edit_image for maya")
   137→        except Exception as e:
   138→            logger.error(f"Failed to register azure_edit_image handler: {e}", exc_info=True)
   139→
   140→    # Register video handlers if enabled (version="explore" AND global config allows it)
   141→    if video_enabled:
   142→        try:
   143→            # Text-to-Video handler
   144→            llm.register_function("generate_video", create_video_handler(
   145→                session_state, rtvi, bot_session, transcript_manager, task, llm
   146→            ))
   147→            # Image-to-Video handler
   148→            llm.register_function("generate_video_from_image", create_video_from_image_handler(
   149→                session_state, rtvi, bot_session, transcript_manager, task, llm
   150→            ))
   151→            logger.info(f"VIDEO_HANDLERS_REGISTERED: session_id={session_id} (t2v + i2v)")
   152→        except Exception as e:
   153→            logger.error(f"Failed to register video handlers: {e}", exc_info=True)
   154→
   155→    transcript.event_handler("on_transcript_update")(create_transcript_handler(bot_session, transcript_manager, rtvi))
   156→    rtvi.event_handler("on_client_ready")(create_client_ready_handler())
   157→    rtvi.event_handler("on_client_message")(create_client_message_handler(bot_session, rtvi, task, transcript_manager))
   158→
   159→    cleanup_if_no_user = create_cleanup_if_no_user(bot_session, session_state, task)
   160→
   161→    transport.event_handler("on_participant_joined")(create_participant_joined_handler(bot_session, task))
   162→    transport.event_handler("on_first_participant_joined")(create_first_participant_joined_handler(bot_session, rtvi, task, audio_buffer, chunk_state))
   163→    transport.event_handler("on_participant_left")(create_participant_left_handler(bot_session, session_state, transcript_manager, chunk_state, task))
   164→    transport.event_handler("on_client_disconnected")(create_client_disconnected_handler(bot_session, task))
   165→    task.event_handler("on_idle_timeout")(create_idle_timeout_handler(bot_session))
   166→
   167→    runner = PipelineRunner(handle_sigint=False)
   168→
   169→    bot_session.no_user_timeout_task = asyncio.create_task(cleanup_if_no_user())
   170→    logger.info(f"NO_USER_TIMEOUT_STARTED: session_id={session_id} timeout={constants.NO_USER_TIMEOUT}s")
   171→
   172→    try:
   173→        await runner.run(task)
   174→    except Exception as e:
   175→        logger.error(f"PIPELINE_ERROR: session_id={session_id} error={e}", exc_info=True)
   176→        await task.cancel()
   177→    finally:
   178→        logger.debug(f"RUNNER_CLEANUP_START: session_id={session_id}")
   179→
   180→        bot_session.cancel_timeout_task()
   181→        if bot_session.no_user_timeout_task:
   182→            try:
   183→                await bot_session.no_user_timeout_task
   184→            except asyncio.CancelledError:
   185→                pass
   186→
   187→        await cleanup_session(session_state, bot_session, session_id, transcript_manager, chunk_state)
   188→        logger.debug(f"RUNNER_CLEANUP_COMPLETE: session_id={session_id}")
   189→


Excellent! Now let me explore more key files:


     1→import asyncio
     2→import os
     3→import time
     4→from datetime import datetime
     5→from typing import Any, Optional, List, Dict
     6→from loguru import logger
     7→
     8→from utils import constants
     9→
    10→__all__ = [
    11→    'load_system_instruction',
    12→    'cleanup_session',
    13→    'BotSessionState',
    14→    'ChunkState',
    15→    'TranscriptManager',
    16→]
    17→
    18→
    19→class ChunkState:
    20→    def __init__(self):
    21→        self.counter: int = 0
    22→        self.chunks: List[str] = []
    23→        self.session_id: Optional[str] = None
    24→
    25→    def reset(self, session_id: str):
    26→        self.session_id = session_id
    27→        self.counter = 0
    28→        self.chunks = []
    29→
    30→    def clear(self):
    31→        """Clear all chunk data for memory cleanup."""
    32→        self.counter = 0
    33→        self.chunks.clear()
    34→        self.session_id = None
    35→
    36→    def add_chunk(self, chunk_key: str) -> int:
    37→        chunk_num = self.counter
    38→        self.counter += 1
    39→        self.chunks.append(chunk_key)
    40→        return chunk_num
    41→
    42→    def get_chunk_key(self, session_id: str, chunk_num: int) -> str:
    43→        return f"maya-c/{session_id}/chunks/chunk_{chunk_num:03d}.wav"
    44→
    45→    @property
    46→    def total_chunks(self) -> int:
    47→        return self.counter
    48→
    49→    @property
    50→    def chunk_list(self) -> List[str]:
    51→        return self.chunks.copy()
    52→
    53→
    54→class TranscriptManager:
    55→    def __init__(self):
    56→        self._messages: List[Dict] = []
    57→
    58→    def clear(self):
    59→        """Clear all messages for memory cleanup."""
    60→        self._messages.clear()
    61→
    62→    def add_message(self, role: str, content: str, msg_type: str = None, **kwargs):
    63→        entry = {
    64→            "timestamp": datetime.now().isoformat(),
    65→            "role": role,
    66→            "content": content
    67→        }
    68→        if msg_type:
    69→            entry["type"] = msg_type
    70→        entry.update(kwargs)
    71→        self._messages.append(entry)
    72→
    73→    def add_user_message(self, content: str):
    74→        self.add_message("user", content)
    75→
    76→    def add_assistant_message(self, content: str):
    77→        self.add_message("assistant", content)
    78→
    79→    def add_system_message(self, content: str):
    80→        self.add_message("system", content)
    81→
    82→    def add_generation(self, generation_type: str, content: Dict):
    83→        self._messages.append({
    84→            "timestamp": datetime.now().isoformat(),
    85→            "role": "system",
    86→            "type": "generation",
    87→            "generation_type": generation_type,
    88→            "content": content
    89→        })
    90→
    91→    def add_image_upload(self, mime_type: str, size_bytes: int, s3_url: str = None):
    92→        self._messages.append({
    93→            "timestamp": datetime.now().isoformat(),
    94→            "role": "user",
    95→            "type": "image_upload",
    96→            "content": {"mime_type": mime_type, "size_bytes": size_bytes, "s3_url": s3_url}
    97→        })
    98→
    99→    @property
   100→    def messages(self) -> List[Dict]:
   101→        return self._messages
   102→
   103→    @property
   104→    def count(self) -> int:
   105→        return len(self._messages)
   106→
   107→    def get_user_messages(self) -> List[Dict]:
   108→        return [m for m in self._messages if m.get("role") == "user"]
   109→
   110→    def get_assistant_messages(self) -> List[Dict]:
   111→        return [m for m in self._messages if m.get("role") == "assistant"]
   112→
   113→    def get_generations(self) -> List[Dict]:
   114→        return [m for m in self._messages if m.get("type") == "generation"]
   115→
   116→    def get_image_generations_count(self) -> int:
   117→        return len([m for m in self._messages if m.get("generation_type") == "image"])
   118→
   119→    def get_image_edits_count(self) -> int:
   120→        return len([m for m in self._messages if m.get("generation_type") == "image_edit"])
   121→
   122→
   123→class BotSessionState:
   124→    def __init__(self, session_id: str, device_id: str, bot_start_time: float):
   125→        self.session_id = session_id
   126→        self.device_id = device_id
   127→        self.chunk_counter = 0
   128→        self.user_stopped_speaking_time: Optional[float] = None
   129→        self.current_user_id: Optional[str] = None
   130→        self.greeting_sent = False
   131→        self.user_spoke_first = False
   132→        self.audio_recorder = None
   133→        self.user_joined = False
   134→        self.bot_joined = False
   135→        self.bot_idle = False
   136→        self.no_user_timeout_task = None
   137→        self.participants: dict = {}
   138→        self.session_start_time = bot_start_time
   139→        self.bot_start_time = bot_start_time
   140→        self.app_review_eligible = False
   141→        self.vad_analyzer = None
   142→        self.audio_buffer = None
   143→        self._last_upload_time: float = 0
   144→        self.uploaded_image: Optional[dict] = None
   145→        self.last_image: Optional[dict] = None
   146→        self.last_generated_image: Optional[dict] = None
   147→        self.image_batch: list = []  # NEW: Batch of images uploaded together
   148→        self.max_batch_size: int = 2  # NEW: Max images per batch
   149→        self.is_premium = False
   150→        self.daily_limit = 20
   151→        self.today_image_count = 0
   152→        self.remaining_image_limit = 999
   153→        self.current_session_image_count = 0
   154→        self.daily_limit_exceeded = False
   155→        self.session_created_at_start = False
   156→        self.video_enabled = False  # NEW: Track if video generation is enabled for this session
   157→        self.is_processing = False  # NEW: In-memory lock for video/image processing (prevents concurrent requests)
   158→        self.generated_videos: List[Dict] = []  # Store generated videos to save at session end
   159→
   160→        # Pipeline service references for explicit cleanup (prevents memory leaks)
   161→        self.stt_service = None       # STT service (Deepgram/OpenAI/Mistral WebSocket)
   162→        self.llm_service = None       # LLM service (Gemini/OpenAI HTTP client)
   163→        self.transport = None         # Daily WebRTC transport
   164→        self.pipeline = None          # Pipeline object
   165→        self.context_aggregator = None  # LLM context (grows with conversation)
   166→
   167→        # Background task tracking for proper cancellation
   168→        self._background_tasks: List[asyncio.Task] = []
   169→
   170→        # Character support
   171→        self.character_id: Optional[str] = None
   172→        self.character_name: Optional[str] = None
   173→        # Note: Characters don't have image generation - only Maya does
   174→        # If character_id is set, image generation/editing is disabled
   175→
   176→    def get_session_info(self) -> str:
   177→        """Get formatted session info for logging."""
   178→        info = f"session_id={self.session_id} device_id={self.device_id}"
   179→        if self.character_id:
   180→            info += f" character={self.character_id}"
   181→            if self.character_name:
   182→                info += f" character_name={self.character_name}"
   183→        return info
   184→
   185→    def add_participant(self, participant_id: str, name: str, participant_type: str, joined_at: float):
   186→        self.participants[participant_id] = {
   187→            "name": name,
   188→            "type": participant_type,
   189→            "joined_at": joined_at
   190→        }
   191→
   192→    def get_participant(self, participant_id: str) -> dict:
   193→        return self.participants.get(participant_id, {})
   194→
   195→    def increment_image_count(self):
   196→        self.current_session_image_count += 1
   197→        self.today_image_count += 1
   198→
   199→    def set_premium_limits(self):
   200→        self.is_premium = True
   201→        self.daily_limit = 999999
   202→        self.today_image_count = 0
   203→        self.remaining_image_limit = 999999
   204→        self.current_session_image_count = 0
   205→        self.daily_limit_exceeded = False
   206→
   207→    def set_regular_limits(self, limit_per_3hrs: int, recent_count: int):
   208→        from utils import constants
   209→
   210→        # Toggle: Give unlimited to everyone
   211→        if constants.UNLIMITED_FOR_ALL:
   212→            self.is_premium = False  # Keep premium flag false for tracking
   213→            self.daily_limit = 999999
   214→            self.today_image_count = 0
   215→            self.remaining_image_limit = 999999
   216→            self.current_session_image_count = 0
   217→            self.daily_limit_exceeded = False
   218→            return
   219→
   220→        # Regular limits
   221→        self.is_premium = False
   222→        self.daily_limit = limit_per_3hrs
   223→        self.today_image_count = recent_count
   224→        self.remaining_image_limit = max(0, limit_per_3hrs - recent_count)
   225→        self.current_session_image_count = 0
   226→        self.daily_limit_exceeded = self.remaining_image_limit <= 0
   227→
   228→    def set_fallback_limits(self):
   229→        self.is_premium = False
   230→        self.daily_limit_exceeded = False
   231→        self.remaining_image_limit = 999
   232→        self.current_session_image_count = 0
   233→
   234→    def can_upload_image(self, cooldown: float = 2.0) -> bool:
   235→        now = time.time()
   236→        if now - self._last_upload_time < cooldown:
   237→            return False
   238→        self._last_upload_time = now
   239→        return True
   240→
   241→    def set_uploaded_image(self, image_info: dict):
   242→        """Set single uploaded image (backward compatible)."""
   243→        # Clear stale generated/edited images when user uploads fresh
   244→        self.last_generated_image = None
   245→        self.image_batch = []
   246→        # Set new upload
   247→        self.uploaded_image = image_info
   248→        self.last_image = image_info
   249→
   250→    def set_image_batch(self, images: list):
   251→        """Store a batch of images uploaded together."""
   252→        # Clear stale generated/edited images when user uploads fresh batch
   253→        self.last_generated_image = None
   254→        self.uploaded_image = None
   255→        # Set new batch
   256→        self.image_batch = images[:self.max_batch_size]  # Limit to max
   257→        if self.image_batch:
   258→            self.last_image = self.image_batch[0]
   259→            self.uploaded_image = self.image_batch[0]  # Backward compatibility
   260→
   261→    def get_image_batch(self) -> list:
   262→        """Get current image batch."""
   263→        return self.image_batch
   264→
   265→    def get_batch_size(self) -> int:
   266→        """Get number of images in current batch."""
   267→        return len(self.image_batch)
   268→
   269→    def clear_image_batch(self):
   270→        """Clear image batch."""
   271→        self.image_batch = []
   272→        self.uploaded_image = None
   273→
   274→    def set_generated_image(self, image_info: dict):
   275→        """Store generated image for iteration."""
   276→        self.last_generated_image = image_info
   277→        self.last_image = image_info
   278→
   279→    def add_generated_video(self, video_type: str, metadata: Dict, prompt: str = None):
   280→        """Add a generated video to the list for saving at session end."""
   281→        self.generated_videos.append({
   282→            "video_type": video_type,
   283→            "metadata": metadata,
   284→            "prompt": prompt
   285→        })
   286→
   287→    def get_generated_videos(self) -> List[Dict]:
   288→        """Get all generated videos for this session."""
   289→        return self.generated_videos
   290→
   291→    def set_edited_image(self, image_info: dict):
   292→        """Store edited image for iteration and clear uploads."""
   293→        self.last_generated_image = image_info
   294→        self.last_image = image_info
   295→        # Clear uploads after edit (ready for new uploads)
   296→        self.uploaded_image = None
   297→        self.image_batch = []
   298→
   299→    def clear_uploaded_image(self):
   300→        """Clear single uploaded image (backward compatible)."""
   301→        self.uploaded_image = None
   302→
   303→    def clear_all_images(self):
   304→        """Clear ALL image data for memory cleanup at session end."""
   305→        self.uploaded_image = None
   306→        self.last_image = None
   307→        self.last_generated_image = None
   308→        self.image_batch = []
   309→
   310→    def get_image_to_edit(self) -> Optional[dict]:
   311→        """Get the image to edit (supports iteration)."""
   312→        if self.last_image:
   313→            return self.last_image
   314→        return self.uploaded_image or self.last_generated_image
   315→
   316→    def get_uploaded_image(self) -> Optional[dict]:
   317→        """Get the currently uploaded image (for i2v video generation).
   318→        Also returns last_generated_image if no uploaded image is available,
   319→        since users may want to create video from a generated image too.
   320→        """
   321→        return self.uploaded_image or self.last_generated_image
   322→
   323→    def track_background_task(self, task: asyncio.Task):
   324→        """Track a fire-and-forget background task for cleanup cancellation."""
   325→        self._background_tasks.append(task)
   326→        # Auto-remove when done to avoid accumulation
   327→        def _on_task_done(t):
   328→            try:
   329→                self._background_tasks.remove(t)
   330→            except ValueError:
   331→                pass  # Already removed by cancel_background_tasks
   332→        task.add_done_callback(_on_task_done)
   333→
   334→    async def cancel_background_tasks(self):
   335→        """Cancel all tracked background tasks. Call during session cleanup."""
   336→        if not self._background_tasks:
   337→            return
   338→
   339→        tasks_to_cancel = list(self._background_tasks)
   340→        for task in tasks_to_cancel:
   341→            if not task.done():
   342→                task.cancel()
   343→
   344→        # Wait briefly for cancellation to propagate
   345→        if tasks_to_cancel:
   346→            await asyncio.gather(*tasks_to_cancel, return_exceptions=True)
   347→            logger.debug(f"CLEANUP: cancelled {len(tasks_to_cancel)} background tasks session_id={self.session_id}")
   348→
   349→        self._background_tasks.clear()
   350→
   351→    def cancel_timeout_task(self):
   352→        if self.no_user_timeout_task and not self.no_user_timeout_task.done():
   353→            self.no_user_timeout_task.cancel()
   354→
   355→    def can_generate_images(self) -> bool:
   356→        """Check if current mode allows image generation. Only Maya can generate images."""
   357→        return self.character_id is None  # Only Maya (no character_id)
   358→
   359→    def can_edit_images(self) -> bool:
   360→        """Check if current mode allows image editing. Only Maya can edit images."""
   361→        return self.character_id is None  # Only Maya (no character_id)
   362→
   363→    def is_maya(self) -> bool:
   364→        """Check if current session is with Maya (not a character)."""
   365→        return self.character_id is None
   366→
   367→
   368→async def load_system_instruction(character_id: Optional[str] = None) -> str:
   369→    """Load the system instruction from file or database based on character."""
   370→    try:
   371→        # If character_id is provided, try to load character-specific prompt from DB
   372→        if character_id:
   373→            from utils import turso_client
   374→            prompt = await turso_client.get_character_prompt(character_id)
   375→            if prompt:
   376→                logger.info(f"PROMPT_LOADED: character={character_id} source=database length={len(prompt)}")
   377→                return prompt
   378→            else:
   379→                logger.warning(f"PROMPT_NOT_FOUND: character={character_id} using_fallback=maya")
   380→                # Fall through to load Maya's prompt
   381→
   382→        # Load Maya's prompt from file
   383→        prompt_path = os.path.join(os.path.dirname(os.path.dirname(__file__)), 'prompts', 'prompt_global_agent.txt')
   384→
   385→        if os.path.exists(prompt_path):
   386→            with open(prompt_path, 'r', encoding='utf-8') as f:
   387→                content = f.read().strip()
   388→            logger.debug("PROMPT_LOADED: source=file synced_from_db=true")
   389→            return content
   390→    except Exception as e:
   391→        logger.error(f"Error loading prompt: {e}")
   392→
   393→    logger.warning("PROMPT_LOAD_FAILED: using default fallback")
   394→    return constants.DEFAULT_SYSTEM_PROMPT
   395→
   396→
   397→async def cleanup_session(session_state, bot_session: BotSessionState, session_id: str,
   398→                         transcript_manager: "TranscriptManager" = None,
   399→                         chunk_state: "ChunkState" = None):
   400→    """Comprehensive session cleanup to free memory."""
   401→
   402→    # 1. Cancel all tracked background tasks first (prevents them from holding references)
   403→    try:
   404→        await bot_session.cancel_background_tasks()
   405→    except Exception as e:
   406→        logger.error(f"CLEANUP_ERROR: cancel_background_tasks failed: {e}")
   407→
   408→    # 2. Stop queue processors
   409→    try:
   410→        await session_state.stop_queue_processor()
   411→    except Exception as e:
   412→        logger.error(f"CLEANUP_ERROR: stop_queue_processor failed: {e}")
   413→
   414→    # 3. Close TTS service (HTTP session + connector)
   415→    if session_state.tts_service:
   416→        try:
   417→            await session_state.tts_service.close()
   418→        except Exception as e:
   419→            logger.error(f"CLEANUP_ERROR: tts_service.close failed: {e}")
   420→        finally:
   421→            session_state.tts_service = None
   422→
   423→    # 4. Close STT service (WebSocket connections to Deepgram/OpenAI/Mistral)
   424→    if bot_session.stt_service:
   425→        try:
   426→            if hasattr(bot_session.stt_service, 'close'):
   427→                await bot_session.stt_service.close()
   428→            elif hasattr(bot_session.stt_service, 'cleanup'):
   429→                await bot_session.stt_service.cleanup()
   430→            logger.debug(f"CLEANUP: STT service closed session_id={session_id}")
   431→        except Exception as e:
   432→            logger.error(f"CLEANUP_ERROR: stt_service close failed: {e}")
   433→        finally:
   434→            bot_session.stt_service = None
   435→
   436→    # 5. Close LLM service (HTTP client connections to Gemini/OpenAI)
   437→    if bot_session.llm_service:
   438→        try:
   439→            if hasattr(bot_session.llm_service, 'close'):
   440→                await bot_session.llm_service.close()
   441→            elif hasattr(bot_session.llm_service, 'cleanup'):
   442→                await bot_session.llm_service.cleanup()
   443→            logger.debug(f"CLEANUP: LLM service closed session_id={session_id}")
   444→        except Exception as e:
   445→            logger.error(f"CLEANUP_ERROR: llm_service close failed: {e}")
   446→        finally:
   447→            bot_session.llm_service = None
   448→
   449→    # 6. Clear LLM context aggregator (conversation history - can be several MB)
   450→    try:
   451→        if bot_session.context_aggregator:
   452→            bot_session.context_aggregator = None
   453→            logger.debug(f"CLEANUP: context aggregator released session_id={session_id}")
   454→    except Exception as e:
   455→        logger.error(f"CLEANUP_ERROR: context_aggregator clear failed: {e}")
   456→
   457→    # 7. Cleanup pipeline first (this releases transport and all native threads)
   458→    # IMPORTANT: pipeline.cleanup() must be called BEFORE setting pipeline=None
   459→    # This ensures Daily SDK native C++ threads (WebRTC, MediaSoup, etc.) are properly released
   460→    # Each session creates ~7-8 native threads that MUST be cleaned up to prevent thread leaks
   461→    pipeline_cleanup_success = False
   462→    try:
   463→        if bot_session.pipeline:
   464→            # Pipeline.cleanup() internally calls transport.cleanup() and releases all native resources
   465→            if hasattr(bot_session.pipeline, 'cleanup'):
   466→                await bot_session.pipeline.cleanup()
   467→                pipeline_cleanup_success = True
   468→                logger.debug(f"CLEANUP: pipeline.cleanup() called session_id={session_id}")
   469→            bot_session.pipeline = None
   470→            logger.debug(f"CLEANUP: pipeline released session_id={session_id}")
   471→    except Exception as e:
   472→        logger.error(f"CLEANUP_ERROR: pipeline cleanup failed: {e}", exc_info=True)
   473→        # Still try to release reference even if cleanup failed
   474→        try:
   475→            bot_session.pipeline = None
   476→        except:
   477→            pass
   478→
   479→    # 8. Cleanup transport explicitly (in case pipeline.cleanup() didn't handle it or failed)
   480→    # Daily SDK native threads must be released via transport.cleanup()
   481→    # This releases WebRTC threads: rtp_send_contro, TaskQueuePacedS, mediasoup-manag,
   482→    # SoupSignalling, DailyCustomAudi, AudioEncoder, etc.
   483→    # Without this, threads accumulate (~147 sessions worth = 1130 threads = ~7GB memory)
   484→    try:
   485→        if bot_session.transport:
   486→            # Only call transport.cleanup() if pipeline.cleanup() didn't succeed
   487→            # (pipeline.cleanup() typically calls transport.cleanup() internally)
   488→            if not pipeline_cleanup_success:
   489→                if hasattr(bot_session.transport, 'cleanup'):
   490→                    await bot_session.transport.cleanup()
   491→                    logger.debug(f"CLEANUP: transport.cleanup() called (pipeline cleanup failed) session_id={session_id}")
   492→                elif hasattr(bot_session.transport, 'close'):
   493→                    await bot_session.transport.close()
   494→                    logger.debug(f"CLEANUP: transport.close() called (pipeline cleanup failed) session_id={session_id}")
   495→            bot_session.transport = None
   496→            logger.debug(f"CLEANUP: transport released session_id={session_id}")
   497→    except Exception as e:
   498→        logger.error(f"CLEANUP_ERROR: transport cleanup failed: {e}", exc_info=True)
   499→        # Still try to release reference even if cleanup failed
   500→        try:
   501→            bot_session.transport = None
   502→        except:
   503→            pass
   504→
   505→    # 9. Destroy VAD analyzer (ONNX model - frees C++ allocations)
   506→    if bot_session.vad_analyzer:
   507→        try:
   508→            del bot_session.vad_analyzer
   509→            logger.info(f"VAD_DESTROYED: session_id={session_id} (memory freed)")
   510→        except Exception as e:
   511→            logger.error(f"CLEANUP_ERROR: VAD destruction failed: {e}")
   512→        finally:
   513→            bot_session.vad_analyzer = None
   514→
   515→    # 10. Clear all image base64 data (2-10MB per session)
   516→    try:
   517→        bot_session.clear_all_images()
   518→        logger.debug(f"CLEANUP: images cleared session_id={session_id}")
   519→    except Exception as e:
   520→        logger.error(f"CLEANUP_ERROR: clear_all_images failed: {e}")
   521→
   522→    # 11. Clear audio buffer
   523→    try:
   524→        if bot_session.audio_buffer:
   525→            bot_session.audio_buffer = None
   526→    except Exception as e:
   527→        logger.error(f"CLEANUP_ERROR: audio_buffer clear failed: {e}")
   528→
   529→    # 12. Clear participants dict
   530→    try:
   531→        bot_session.participants.clear()
   532→    except Exception as e:
   533→        logger.error(f"CLEANUP_ERROR: participants clear failed: {e}")
   534→
   535→    # 13. Clear generated videos list
   536→    try:
   537→        bot_session.generated_videos.clear()
   538→    except Exception as e:
   539→        logger.error(f"CLEANUP_ERROR: generated_videos clear failed: {e}")
   540→
   541→    # 14. Clear transcript messages (only after saved to S3)
   542→    try:
   543→        if transcript_manager:
   544→            transcript_manager.clear()
   545→            logger.debug(f"CLEANUP: transcript cleared session_id={session_id}")
   546→    except Exception as e:
   547→        logger.error(f"CLEANUP_ERROR: transcript clear failed: {e}")
   548→
   549→    # 15. Clear chunk state (only after manifest saved)
   550→    try:
   551→        if chunk_state:
   552→            chunk_state.clear()
   553→            logger.debug(f"CLEANUP: chunks cleared session_id={session_id}")
   554→    except Exception as e:
   555→        logger.error(f"CLEANUP_ERROR: chunk_state clear failed: {e}")
   556→
   557→    logger.info(f"SESSION_CLEANUP_COMPLETE: session_id={session_id}")


     1→"""Pipeline creation for bot."""
     2→
     3→from loguru import logger
     4→from pipecat.services.deepgram.stt import DeepgramSTTService
     5→from pipecat.pipeline.pipeline import Pipeline
     6→from pipecat.pipeline.task import PipelineParams, PipelineTask
     7→from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext
     8→from pipecat.processors.audio.audio_buffer_processor import AudioBufferProcessor
     9→from pipecat.processors.frameworks.rtvi import RTVIConfig, RTVIProcessor
    10→from pipecat.processors.transcript_processor import TranscriptProcessor
    11→from pipecat.services.openai.stt import OpenAISTTService
    12→from services.mistral_stt import MistralSTTService
    13→from pipecat.transcriptions.language import Language
    14→from pipecat.services.openai.llm import OpenAILLMService
    15→from pipecat.services.google.llm_vertex import GoogleVertexLLMService
    16→from pipecat.transports.daily.transport import DailyParams, DailyTransport
    17→from pipecat.adapters.schemas.tools_schema import ToolsSchema
    18→from pipecat.audio.vad.silero import SileroVADAnalyzer, VADParams
    19→# from pipecat.audio.filters.rnnoise_filter import RNNoiseFilter
    20→from pipecat.observers.loggers.user_bot_latency_log_observer import UserBotLatencyLogObserver
    21→
    22→from utils import constants
    23→from utils.custom_tts import CustomTTSService
    24→from bot.observers import InfoMetricsObserver, SafeRTVIObserver
    25→from fnHandlers import generate_image_function, azure_edit_image_function, cf_image_function, generate_video_function, generate_video_from_image_function
    26→
    27→
    28→def create_vad_analyzer(session_id: str):
    29→    """Create VAD analyzer with configured parameters."""
    30→    vad = SileroVADAnalyzer(
    31→        params=VADParams(
    32→            confidence=constants.VAD_CONFIDENCE,
    33→            stop_secs=constants.VAD_STOP_SECS,
    34→            start_secs=constants.VAD_START_SECS,
    35→            min_volume=constants.VAD_MIN_VOLUME,
    36→        )
    37→    )
    38→    logger.info(f"VAD_CREATED: session_id={session_id}")
    39→    return vad
    40→
    41→
    42→def create_transport(room_url: str, token: str, vad_analyzer):
    43→    """Create Daily transport with configured parameters."""
    44→    return DailyTransport(
    45→        room_url,
    46→        token,
    47→        "Maya Global Agent Bot",
    48→        DailyParams(
    49→            audio_in_enabled=True,
    50→            audio_out_enabled=True,
    51→            video_in_enabled=False,
    52→            transcription_enabled=False,
    53→            audio_in_sample_rate=constants.SAMPLE_RATE_16K,
    54→            audio_out_sample_rate=constants.SAMPLE_RATE_16K,
    55→            # audio_in_filter=RNNoiseFilter(resampler_quality="MQ"),
    56→            vad_analyzer=vad_analyzer,
    57→            cam_enabled=True,
    58→        ),
    59→    )
    60→
    61→
    62→def create_stt_service():
    63→    """Create STT service. Uses cached DB config to switch between Deepgram, OpenAI, Mistral, or random.
    64→
    65→    Modes:
    66→    - "deepgram": Always use Deepgram STT
    67→    - "openai": Always use OpenAI STT
    68→    - "mistral": Always use Mistral Voxtral STT (realtime WebSocket)
    69→    - "random": Randomly choose between Deepgram and OpenAI based on percentage
    70→    """
    71→    import random
    72→    from deepgram import LiveOptions
    73→    from utils.turso_client import get_stt_mode, get_stt_random_deepgram_percentage
    74→
    75→    # Get STT mode from cache (no DB call - refreshed periodically by server)
    76→    mode = get_stt_mode()
    77→
    78→    # Determine which STT to use
    79→    if mode == "random":
    80→        deepgram_percentage = get_stt_random_deepgram_percentage()
    81→        use_deepgram = random.randint(1, 100) <= deepgram_percentage
    82→        chosen = "deepgram" if use_deepgram else "openai"
    83→        logger.info(f"STT_RANDOM_SELECTION: mode=random deepgram_percentage={deepgram_percentage} chosen={chosen}")
    84→    else:
    85→        chosen = mode if mode in ["deepgram", "openai", "mistral"] else "deepgram"  # Default to deepgram
    86→    # chosen = "mistral"
    87→    if chosen == "deepgram":
    88→        # Deepgram STT - using LiveOptions for language configuration
    89→        stt = DeepgramSTTService(
    90→            api_key=constants.DEEPGRAM_API_KEY,
    91→            live_options=LiveOptions(
    92→                model="nova-2",
    93→                language=constants.STT_LANGUAGE,  # "hi" for Hindi
    94→                smart_format=True,
    95→                punctuate=True,
    96→            ),
    97→        )
    98→        logger.info(f"STT_SERVICE_CREATED: provider=deepgram mode={mode} model=nova-2 language={constants.STT_LANGUAGE}")
    99→    elif chosen == "mistral":
   100→        # Mistral Voxtral STT - realtime WebSocket transcription
   101→        # Map STT_LANGUAGE string to Language enum
   102→        language_map = {
   103→            "hi": Language.HI,
   104→            "en": Language.EN,
   105→            "es": Language.ES,
   106→            "fr": Language.FR,
   107→            "de": Language.DE,
   108→            "pt": Language.PT,
   109→            "zh": Language.ZH,
   110→            "ja": Language.JA,
   111→            "ko": Language.KO,
   112→            "ar": Language.AR,
   113→            "ru": Language.RU,
   114→        }
   115→        lang = language_map.get(constants.STT_LANGUAGE, Language.HI)
   116→
   117→        stt = MistralSTTService(
   118→            api_key=constants.MISTRAL_API_KEY,
   119→            model=constants.VOXTRAL_MODEL,
   120→            ws_url=constants.VOXTRAL_WS_URL,
   121→            language=lang,
   122→            sample_rate=constants.SAMPLE_RATE_16K,
   123→        )
   124→        logger.info(f"STT_SERVICE_CREATED: provider=mistral mode={mode} model={constants.VOXTRAL_MODEL} ws_url={constants.VOXTRAL_WS_URL} language={constants.STT_LANGUAGE}")
   125→    else:
   126→        # OpenAI STT
   127→        stt = OpenAISTTService(
   128→            api_key=constants.OPENAI_API_KEY,
   129→            model=constants.STT_MODEL,
   130→            language=constants.STT_LANGUAGE,
   131→        )
   132→        logger.info(f"STT_SERVICE_CREATED: provider=openai mode={mode} model={constants.STT_MODEL}")
   133→
   134→    return stt
   135→
   136→
   137→def create_llm_service(system_instruction: str):
   138→    """Create LLM service with system instruction. Uses cached DB config to switch providers.
   139→
   140→    LLM Provider Modes:
   141→    - "gemini": Use Google Vertex AI Gemini (with GCP account switching)
   142→    - "openai": Use OpenAI GPT-5-mini
   143→    - "random": Randomly choose between Gemini and OpenAI based on percentage
   144→
   145→    GCP Account Modes (only for gemini):
   146→    - "old": Always use mayaresearchgcp (old account)
   147→    - "new": Always use custom-frame-484619-j2 (new account)
   148→    - "random": Randomly choose between old and new based on percentage
   149→    """
   150→    import random
   151→    from utils.turso_client import get_llm_provider, get_llm_random_openai_percentage, get_gcp_mode, get_gcp_new_account_percentage
   152→
   153→    # Get LLM provider from cache (defaults to "gemini" if not set)
   154→    llm_provider = get_llm_provider()
   155→
   156→    # Determine which LLM to use
   157→    if llm_provider == "random":
   158→        openai_percentage = get_llm_random_openai_percentage()
   159→        use_openai = random.randint(1, 100) <= openai_percentage
   160→        chosen_llm = "openai" if use_openai else "gemini"
   161→        logger.info(f"LLM_RANDOM_SELECTION: mode=random openai_percentage={openai_percentage} chosen={chosen_llm}")
   162→    else:
   163→        chosen_llm = llm_provider
   164→    # Use OpenAI GPT-4o-mini
   165→    if chosen_llm == "openai":
   166→        logger.info(f"🤖 LLM_PROVIDER: OpenAI GPT-4o-mini active")
   167→        return OpenAILLMService(
   168→            api_key=constants.OPENAI_API_KEY,
   169→            model="gpt-4o-mini",
   170→            params=OpenAILLMService.InputParams(
   171→                max_completion_tokens=constants.LLM_MAX_OUTPUT_TOKENS,
   172→                temperature=constants.LLM_TEMPERATURE,
   173→            ),
   174→        )
   175→
   176→    # Use Google Vertex AI Gemini (default)
   177→    # Get GCP mode from cache (no DB call - refreshed periodically by server)
   178→    mode = get_gcp_mode()
   179→
   180→    # Determine which account to use
   181→    if mode == "random":
   182→        new_percentage = get_gcp_new_account_percentage()
   183→        use_new_account = random.randint(1, 100) <= new_percentage
   184→        chosen = "new" if use_new_account else "old"
   185→        logger.info(f"GCP_RANDOM_SELECTION: mode=random new_percentage={new_percentage} chosen={chosen}")
   186→    else:
   187→        use_new_account = (mode == "new")
   188→        chosen = mode
   189→
   190→    if use_new_account:
   191→        project_id = "custom-frame-484619-j2"
   192→        credentials_path = "gcp-new-service-account.json"
   193→        account_tag = "NEW"
   194→        logger.info(f"🆕 GCP_ACCOUNT: NEW account active | project={project_id} | credentials={credentials_path}")
   195→    else:
   196→        project_id = "mayaresearchgcp"
   197→        credentials_path = "gcp-service-account.json"
   198→        account_tag = "OLD"
   199→        logger.info(f"🔄 GCP_ACCOUNT: OLD account active | project={project_id} | credentials={credentials_path}")
   200→
   201→    # Use global endpoint for better capacity (Google recommends this for Standard PayGo)
   202→    # Global endpoint routes to region with most availability, reducing 429 errors
   203→    return GoogleVertexLLMService(
   204→        name=f"GeminiLLM-{account_tag}",  # Shows in error logs which account caused the error
   205→        project_id=project_id,
   206→        location="global",  # Global endpoint for multi-region capacity pool
   207→        model=constants.GEMINI_MODEL,
   208→        credentials_path=credentials_path,
   209→        system_instruction=system_instruction,
   210→        params=GoogleVertexLLMService.InputParams(
   211→            max_output_tokens=constants.LLM_MAX_OUTPUT_TOKENS,
   212→            temperature=constants.LLM_TEMPERATURE,
   213→        ),
   214→    )
   215→
   216→
   217→def create_tts_service(speaker: str = None):
   218→    """Create TTS service with optional custom speaker/voice.
   219→
   220→    Args:
   221→        speaker: TTS voice to use. If None, uses DEFAULT_TTS_SPEAKER.
   222→    """
   223→    voice = speaker or constants.DEFAULT_TTS_SPEAKER
   224→    logger.info(f"TTS_SERVICE_CREATED: speaker={voice}")
   225→    return CustomTTSService(
   226→        api_url=constants.TTS_API_URL,
   227→        api_key=constants.VEENA3_API_KEY,
   228→        speaker=voice,
   229→        sample_rate=constants.SAMPLE_RATE_16K,
   230→    )
   231→
   232→
   233→def create_context_and_tools(llm, system_instruction: str, include_video: bool = False, can_generate_images: bool = True, can_edit_images: bool = True):
   234→    """Create LLM context with tools based on capabilities.
   235→
   236→    Args:
   237→        llm: The LLM service
   238→        system_instruction: The system prompt/instruction for the LLM
   239→        include_video: Whether to include video generation functions (t2v and i2v)
   240→                      (True only if video_generation_global=true AND user has video_premium)
   241→        can_generate_images: Whether to include image generation tools (False for characters)
   242→        can_edit_images: Whether to include image editing tools (Maya only)
   243→    """
   244→    # Build tools list based on capabilities
   245→    standard_tools = []
   246→
   247→    if can_generate_images:
   248→        standard_tools.append(generate_image_function)
   249→        standard_tools.append(cf_image_function)
   250→
   251→    if can_edit_images:
   252→        standard_tools.append(azure_edit_image_function)
   253→
   254→    # Conditionally add video functions (both t2v and i2v)
   255→    if include_video and can_generate_images:  # Video only makes sense if images are enabled
   256→        standard_tools.append(generate_video_function)  # text-to-video
   257→        standard_tools.append(generate_video_from_image_function)  # image-to-video
   258→        logger.info("PIPELINE_TOOLS: Video generation functions ENABLED (t2v + i2v)")
   259→
   260→    logger.debug(f"TOOLS_REGISTERED: generate={can_generate_images} edit={can_edit_images} video={include_video} total={len(standard_tools)}")
   261→
   262→    tools = ToolsSchema(standard_tools=standard_tools) if standard_tools else None
   263→
   264→    # Build messages with system instruction for OpenAI compatibility
   265→    messages = [
   266→        {"role": "system", "content": system_instruction},
   267→        {"role": "user", "content": "Start the Conversation :"}
   268→    ]
   269→
   270→    context = OpenAILLMContext(
   271→        messages=messages,
   272→        tools=tools
   273→    )
   274→    context_aggregator = llm.create_context_aggregator(context)
   275→    return context_aggregator
   276→
   277→
   278→def create_processors():
   279→    """Create RTVI, transcript, and audio buffer processors."""
   280→    rtvi = RTVIProcessor(config=RTVIConfig(config=[]))
   281→    transcript = TranscriptProcessor()
   282→
   283→    audio_buffer = AudioBufferProcessor(
   284→        sample_rate=constants.SAMPLE_RATE,
   285→        num_channels=1,
   286→        buffer_size=constants.SAMPLE_RATE * 2 * constants.CHUNK_DURATION,
   287→    )
   288→
   289→    return rtvi, transcript, audio_buffer
   290→
   291→
   292→def create_pipeline(transport, stt, context_aggregator, rtvi, llm, tts, audio_buffer, transcript):
   293→    """Create the processing pipeline."""
   294→    pipeline = Pipeline([
   295→        transport.input(),
   296→        stt,
   297→        transcript.user(),
   298→        context_aggregator.user(),
   299→        rtvi,
   300→        llm,
   301→        tts,
   302→        transport.output(),
   303→        audio_buffer,
   304→        transcript.assistant(),
   305→        context_aggregator.assistant(),
   306→    ])
   307→    logger.info("PIPELINE: Traditional STT mode (Audio → STT → Text → Gemini → TTS)")
   308→    return pipeline
   309→
   310→
   311→def create_task(pipeline, rtvi, bot_session):
   312→    """Create pipeline task with observers."""
   313→    return PipelineTask(
   314→        pipeline,
   315→        params=PipelineParams(
   316→            enable_metrics=True,
   317→            enable_usage_metrics=True,
   318→            allow_interruptions=True
   319→        ),
   320→        observers=[
   321→            SafeRTVIObserver(rtvi),  # Use SafeRTVIObserver to handle startup race condition
   322→            InfoMetricsObserver(bot_session),
   323→            UserBotLatencyLogObserver()
   324→        ],
   325→    )
   326→
   327→


     1→"""Session event handlers for bot."""
     2→
     3→import asyncio
     4→import io
     5→import json
     6→import time
     7→import wave
     8→from datetime import datetime
     9→from loguru import logger
    10→
    11→from pipecat.frames.frames import LLMRunFrame
    12→
    13→from utils import constants
    14→from utils.s3_utils import upload_to_s3, save_json_to_s3
    15→from utils.turso_client import save_session_to_turso, save_session_complete, save_character_session
    16→
    17→from typing import TYPE_CHECKING
    18→if TYPE_CHECKING:
    19→    from bot.config import BotSessionState, ChunkState, TranscriptManager
    20→
    21→
    22→def create_audio_data_handler(bot_session: "BotSessionState", chunk_state: "ChunkState"):
    23→    async def on_audio_data(buffer, audio, sample_rate, num_channels):
    24→        if not chunk_state.session_id:
    25→            chunk_state.session_id = bot_session.session_id or "unknown"
    26→
    27→        session_id = chunk_state.session_id
    28→        chunk_key = chunk_state.get_chunk_key(session_id, chunk_state.counter)
    29→        chunk_num = chunk_state.add_chunk(chunk_key)
    30→
    31→        try:
    32→            wav_buffer = io.BytesIO()
    33→            with wave.open(wav_buffer, 'wb') as wav_file:
    34→                wav_file.setnchannels(num_channels)
    35→                wav_file.setsampwidth(2)
    36→                wav_file.setframerate(sample_rate)
    37→                wav_file.writeframes(audio)
    38→
    39→            wav_buffer.seek(0)
    40→            wav_data = wav_buffer.read()
    41→
    42→            await upload_to_s3(wav_data, chunk_key, "audio/wav")
    43→            logger.debug(f"Chunk {chunk_num} uploaded: {chunk_key}")
    44→        except Exception as e:
    45→            logger.error(f"Failed to upload chunk {chunk_num}: {e}")
    46→    return on_audio_data
    47→
    48→
    49→def create_transcript_handler(bot_session: "BotSessionState", transcript_manager: "TranscriptManager", rtvi):
    50→    async def on_transcript_update(processor, frame):
    51→        try:
    52→            for message in frame.messages:
    53→                transcript_entry = {
    54→                    "timestamp": message.timestamp,
    55→                    "role": message.role,
    56→                    "content": message.content
    57→                }
    58→                transcript_manager.add_message(message.role, message.content)
    59→
    60→                if message.role == "user":
    61→                    bot_session.user_spoke_first = True
    62→
    63→                try:
    64→                    await rtvi.send_server_message({
    65→                        "type": "bot-transcript-update",
    66→                        "data": transcript_entry
    67→                    })
    68→                except Exception as e:
    69→                    logger.error(f"Failed to send transcript to RTVI: {e}")
    70→        except Exception as e:
    71→            logger.error(f"Error in transcript handler: {e}", exc_info=True)
    72→    return on_transcript_update
    73→
    74→
    75→def create_client_message_handler(bot_session: "BotSessionState", rtvi, task, transcript_manager: "TranscriptManager"):
    76→    from bot.image_upload import handle_image_upload, handle_image_batch_upload, handle_load_image_from_url
    77→    async def on_client_message(rtvi_proc, message):
    78→        try:
    79→            msg_type = getattr(message, 'type', None)
    80→            msg_data = getattr(message, 'data', None)
    81→            logger.debug(f"RTVI message: {msg_type}")
    82→
    83→            if msg_type == "user_image_upload":
    84→                data = msg_data if isinstance(msg_data, dict) else {}
    85→                image_data = data.get("image")
    86→                mime_type = data.get("mime_type", "image/jpeg")
    87→
    88→                if image_data:
    89→                    await handle_image_upload(
    90→                        image_data, mime_type, bot_session,
    91→                        rtvi, task, transcript_manager, source="rtvi"
    92→                    )
    93→
    94→            elif msg_type == "user_image_batch_upload":
    95→                data = msg_data if isinstance(msg_data, dict) else {}
    96→                images_data = data.get("images", [])
    97→
    98→                if images_data:
    99→                    await handle_image_batch_upload(
   100→                        images_data, bot_session, rtvi, task, transcript_manager


Let me continue exploring the routes and key utility files:


     1→"""Bot connection API endpoints."""
     2→
     3→import asyncio
     4→import time
     5→from typing import Any, Dict
     6→
     7→from fastapi import APIRouter, Request, HTTPException
     8→from loguru import logger
     9→
    10→from bot import run_bot_webrtc_global_agent
    11→from utils.daily_client import DailyClient, DailyAPIError
    12→from utils.logging_config import log_error
    13→from utils.redis_client import add_active_session, remove_active_session, cleanup_pod_sessions, redis_available, _get_pod_name
    14→
    15→router = APIRouter(tags=["bot"])
    16→
    17→# Daily client instance
    18→daily_client = DailyClient()
    19→
    20→# Track active sessions: {task: room_name}
    21→active_sessions: dict[asyncio.Task, str] = {}
    22→
    23→
    24→def get_active_sessions() -> dict:
    25→    """Get reference to active sessions for cleanup."""
    26→    return active_sessions
    27→
    28→
    29→async def cleanup_active_sessions():
    30→    """Cleanup active sessions on shutdown."""
    31→    if active_sessions:
    32→        logger.info(f"SHUTDOWN_CLEANUP: active_sessions={len(active_sessions)}")
    33→        rooms_to_delete = list(active_sessions.values())
    34→
    35→        # Cancel all bot tasks
    36→        for task in list(active_sessions.keys()):
    37→            task.cancel()
    38→        await asyncio.gather(*active_sessions.keys(), return_exceptions=True)
    39→
    40→        # Delete all rooms from Daily
    41→        for room_name in rooms_to_delete:
    42→            await daily_client.delete_room(room_name)
    43→
    44→        logger.info(f"SHUTDOWN_CLEANUP_DONE: deleted_rooms={len(rooms_to_delete)}")
    45→
    46→    # Cleanup Redis sessions for this pod
    47→    if redis_available():
    48→        pod_name = _get_pod_name()
    49→        if pod_name:
    50→            cleaned_count = await cleanup_pod_sessions(pod_name)
    51→            logger.info(f"REDIS_SHUTDOWN_CLEANUP: pod={pod_name} sessions_cleaned={cleaned_count}")
    52→
    53→
    54→@router.post("/connect-webrtc-global-agent")
    55→async def bot_connect_webrtc_global_agent(request: Request) -> Dict[Any, Any]:
    56→    """WebRTC (Daily) connection endpoint for Global Agent Bot or Character."""
    57→    device_id = None
    58→    try:
    59→        version = None
    60→        character_id = None
    61→        try:
    62→            body = await request.json()
    63→            device_id = body.get("device_id")
    64→            version = body.get("version")  # Optional: "explore" to enable video
    65→            character_id = body.get("character_id")  # Optional: character ID for character sessions
    66→        except:
    67→            pass
    68→
    69→        # Log session start with character info if provided
    70→        if character_id:
    71→            logger.info(f"SESSION_START: device_id={device_id or 'common'} version={version} character_id={character_id}")
    72→        else:
    73→            logger.info(f"SESSION_START: device_id={device_id or 'common'} version={version} (Maya)")
    74→
    75→        room_creation_start = time.time()
    76→        room_url, room_name, token = await daily_client.create_room_with_token()
    77→        room_token_time = time.time() - room_creation_start
    78→
    79→        logger.info(f"ROOM_CREATED: device_id={device_id or 'common'} room_name={room_name} character={character_id or 'maya'} time={room_token_time:.3f}s")
    80→


     1→"""Session tracking API endpoints."""
     2→
     3→from fastapi import APIRouter
     4→from loguru import logger
     5→
     6→from utils.redis_client import get_all_pod_sessions, redis_available
     7→
     8→router = APIRouter(tags=["sessions"])
     9→
    10→
    11→@router.get("/sessions/pods")
    12→async def get_pods_sessions():
    13→    """Get all pods and their active sessions with metadata.
    14→
    15→    Returns:
    16→        JSON with total_sessions, total_pods, and list of pods with their sessions
    17→    """
    18→    if not redis_available():
    19→        return {
    20→            "total_sessions": 0,
    21→            "total_pods": 0,
    22→            "pods": [],
    23→            "error": "Redis not available"
    24→        }
    25→
    26→    try:
    27→        all_pod_sessions = await get_all_pod_sessions()
    28→
    29→        total_sessions = 0
    30→        pods_list = []
    31→
    32→        for pod_name, sessions in all_pod_sessions.items():
    33→            total_sessions += len(sessions)
    34→            pods_list.append({
    35→                "pod_name": pod_name,
    36→                "session_count": len(sessions),
    37→                "sessions": sessions
    38→            })
    39→
    40→        # Sort pods by session count (descending)
    41→        pods_list.sort(key=lambda x: x["session_count"], reverse=True)
    42→
    43→        return {
    44→            "total_sessions": total_sessions,
    45→            "total_pods": len(pods_list),
    46→            "pods": pods_list
    47→        }
    48→    except Exception as e:
    49→        logger.error(f"SESSIONS_ENDPOINT_ERROR: error={e}")
    50→        return {
    51→            "total_sessions": 0,
    52→            "total_pods": 0,
    53→            "pods": [],
    54→            "error": str(e)
    55→        }
    56→


     1→"""Session state management for tracking generation requests and queue processing."""
     2→
     3→import asyncio
     4→import time
     5→import traceback
     6→from loguru import logger
     7→
     8→queue_logger = logger.bind(queue=True)
     9→
    10→
    11→class GenerationRequest:
    12→    def __init__(self, request_type: str, request_id: str, handler_func, params):
    13→        self.request_type = request_type
    14→        self.request_id = request_id
    15→        self.handler_func = handler_func
    16→        self.params = params
    17→        self.queued_at = time.time()
    18→
    19→    def __repr__(self):
    20→        return f"GenerationRequest({self.request_type}, {self.request_id})"
    21→
    22→
    23→class SessionState:
    24→    def __init__(self):
    25→        self.is_generating = False
    26→        self.last_generation_time = 0
    27→        self.current_language = "hi"
    28→        self.tts_service = None
    29→
    30→        self.image_queue = asyncio.Queue()
    31→        self.image_processor_task = None
    32→        self.is_image_busy = False
    33→
    34→        self.audio_queue = asyncio.Queue()
    35→        self.audio_processor_task = None
    36→        self.is_audio_busy = False
    37→
    38→        self.total_queued = 0
    39→        self.total_processed = 0
    40→
    41→    async def start_queue_processor(self):
    42→        if self.image_processor_task is None:
    43→            self.image_processor_task = asyncio.create_task(self._process_image_queue())
    44→
    45→        if self.audio_processor_task is None:
    46→            self.audio_processor_task = asyncio.create_task(self._process_audio_queue())
    47→
    48→    async def stop_queue_processor(self):
    49→        for task in [self.image_processor_task, self.audio_processor_task]:
    50→            if task and not task.done():
    51→                task.cancel()
    52→                try:
    53→                    await task
    54→                except asyncio.CancelledError:
    55→                    pass
    56→        self.image_processor_task = None
    57→        self.audio_processor_task = None
    58→
    59→        # Drain and clear queues to free any pending requests
    60→        self._drain_queue(self.image_queue)
    61→        self._drain_queue(self.audio_queue)
    62→
    63→    def _drain_queue(self, queue: asyncio.Queue):
    64→        """Drain a queue to free pending items."""
    65→        try:
    66→            while not queue.empty():
    67→                try:
    68→                    queue.get_nowait()
    69→                    queue.task_done()
    70→                except asyncio.QueueEmpty:
    71→                    break
    72→        except Exception:
    73→            pass
    74→
    75→    async def _process_image_queue(self):
    76→        while True:
    77→            try:
    78→                request: GenerationRequest = await self.image_queue.get()
    79→
    80→                self.is_image_busy = True
    81→                try:
    82→                    await request.handler_func()
    83→                    self.total_processed += 1
    84→                except Exception as e:
    85→                    queue_logger.error(f"[IMAGE] FAILED: {request.request_type.upper()} - {type(e).__name__}: {str(e)}\n{traceback.format_exc()}")
    86→                finally:
    87→                    self.is_image_busy = False
    88→                    self.image_queue.task_done()
    89→
    90→            except asyncio.CancelledError:
    91→                break
    92→            except Exception as e:
    93→                queue_logger.error(f"Image queue error: {type(e).__name__}: {str(e)}\n{traceback.format_exc()}")
    94→                await asyncio.sleep(0.1)
    95→
    96→    async def _process_audio_queue(self):
    97→        while True:
    98→            try:
    99→                request: GenerationRequest = await self.audio_queue.get()
   100→
   101→                self.is_audio_busy = True
   102→                try:
   103→                    await request.handler_func()
   104→                    self.total_processed += 1
   105→                except Exception as e:
   106→                    queue_logger.error(f"[AUDIO] FAILED: {request.request_type.upper()} - {type(e).__name__}: {str(e)}\n{traceback.format_exc()}")
   107→                finally:
   108→                    self.is_audio_busy = False
   109→                    self.audio_queue.task_done()
   110→
   111→            except asyncio.CancelledError:
   112→                break
   113→            except Exception as e:
   114→                queue_logger.error(f"Audio queue error: {type(e).__name__}: {str(e)}\n{traceback.format_exc()}")
   115→                await asyncio.sleep(0.1)
   116→
   117→    async def add_to_queue(self, request_type: str, handler_func, params) -> tuple[bool, str, int]:
   118→        request_id = f"{request_type}_{int(time.time() * 1000)}"
   119→        request = GenerationRequest(request_type, request_id, handler_func, params)
   120→
   121→        if request_type in ["image", "analysis"]:
   122→            target_queue = self.image_queue
   123→            is_busy = self.is_image_busy
   124→        elif request_type == "audio":
   125→            target_queue = self.audio_queue
   126→            is_busy = self.is_audio_busy
   127→        else:
   128→            target_queue = self.image_queue
   129→            is_busy = self.is_image_busy
   130→
   131→        await target_queue.put(request)
   132→        self.total_queued += 1
   133→
   134→        queue_position = target_queue.qsize()
   135→
   136→        if queue_position == 1 and not is_busy:
   137→            return True, "Processing now...", 0
   138→        else:
   139→            return True, f"Added to queue (position: {queue_position})", queue_position
   140→
   141→    def get_queue_status(self) -> dict:
   142→        self.is_generating = self.is_image_busy or self.is_audio_busy
   143→
   144→        return {
   145→            "image_queue": {
   146→                "size": self.image_queue.qsize(),
   147→                "is_busy": self.is_image_busy
   148→            },
   149→            "audio_queue": {
   150→                "size": self.audio_queue.qsize(),
   151→                "is_busy": self.is_audio_busy
   152→            },
   153→            "total_queued": self.total_queued,
   154→            "total_processed": self.total_processed,
   155→            "any_busy": self.is_generating
   156→        }
   157→


Now let me check more utility files and resources:


     1→"""Shared Redis client for cross-pod caching (ElastiCache).
     2→
     3→All pods connect to the same Redis instance. This eliminates redundant
     4→DB reads and makes config/device data available cluster-wide.
     5→
     6→Usage:
     7→    from utils.redis_client import get_redis, redis_available
     8→
     9→    if redis_available():
    10→        value = await get_redis().get("maya:some_key")
    11→"""
    12→
    13→import asyncio
    14→import json
    15→import os
    16→import time
    17→from typing import Any, Dict, List, Optional
    18→
    19→from loguru import logger
    20→
    21→from utils import constants
    22→
    23→
    24→# ---------------------------------------------------------------------------
    25→# Module-level singleton
    26→# ---------------------------------------------------------------------------
    27→_redis_pool = None
    28→_redis_available = False
    29→_init_attempted = False
    30→
    31→
    32→async def init_redis() -> bool:
    33→    """Initialize the async Redis connection pool. Safe to call multiple times."""
    34→    global _redis_pool, _redis_available, _init_attempted
    35→
    36→    if _init_attempted:
    37→        return _redis_available
    38→
    39→    _init_attempted = True
    40→
    41→    if not constants.REDIS_ENABLED or not constants.REDIS_URL:
    42→        logger.info("REDIS_DISABLED: reason=not_configured")
    43→        return False
    44→
    45→    try:
    46→        import redis.asyncio as aioredis
    47→
    48→        _redis_pool = aioredis.from_url(
    49→            constants.REDIS_URL,
    50→            socket_connect_timeout=constants.REDIS_CONNECT_TIMEOUT,
    51→            socket_timeout=constants.REDIS_SOCKET_TIMEOUT,
    52→            max_connections=constants.REDIS_MAX_CONNECTIONS,
    53→            decode_responses=True,  # Return str instead of bytes
    54→            retry_on_timeout=True,
    55→        )
    56→
    57→        # Verify connectivity
    58→        await _redis_pool.ping()
    59→        _redis_available = True
    60→        logger.info(f"REDIS_CONNECTED: url={constants.REDIS_URL.split('@')[-1] if '@' in constants.REDIS_URL else constants.REDIS_URL.replace('redis://', '')}")
    61→        return True
    62→
    63→    except ImportError:
    64→        logger.warning("REDIS_DISABLED: reason=redis_package_not_installed")
    65→        return False
    66→    except Exception as e:
    67→        logger.error(f"REDIS_CONNECT_FAILED: error={e} error_type={type(e).__name__}")
    68→        _redis_available = False
    69→        return False
    70→
    71→
    72→def redis_available() -> bool:
    73→    """Check if Redis is connected and available."""
    74→    return _redis_available
    75→
    76→
    77→def get_redis():
    78→    """Get the Redis connection pool. Returns None if Redis is unavailable."""
    79→    return _redis_pool if _redis_available else None
    80→
    81→
    82→def _key(name: str) -> str:
    83→    """Build a namespaced Redis key: maya:<name>"""
    84→    return f"{constants.REDIS_KEY_PREFIX}{name}"
    85→
    86→
    87→# ---------------------------------------------------------------------------
    88→# Core cache operations (all fire-and-forget safe)
    89→# ---------------------------------------------------------------------------
    90→
    91→async def cache_get(key: str) -> Optional[str]:
    92→    """Get a value from Redis. Returns None on miss or error."""
    93→    if not _redis_available:
    94→        return None
    95→    try:
    96→        return await _redis_pool.get(_key(key))
    97→    except Exception as e:
    98→        logger.debug(f"REDIS_GET_ERROR: key={key} error={e}")
    99→        return None
   100→


     1→"""Turso database client for storing session metadata."""
     2→
     3→import asyncio
     4→import json
     5→import os
     6→import random
     7→import time
     8→from datetime import datetime, timezone, timedelta
     9→from typing import Optional, Dict, List, Any, Union
    10→from loguru import logger
    11→
    12→from utils import constants
    13→
    14→
    15→DB_SLOW_QUERY_THRESHOLD_MS = 100
    16→DB_CRITICAL_QUERY_THRESHOLD_MS = 500
    17→
    18→
    19→CRITICAL_OPS = {
    20→    "get_today_image_generations_count": "Blocks image generation - user waiting",
    21→    "get_or_create_device_limit": "Blocks image generation - user waiting",
    22→    "is_premium_user": "Called on API request - user waiting",
    23→    "should_show_app_review": "Multiple queries at session end",
    24→}
    25→
    26→
    27→def _log_db_timing(operation: str, duration_ms: float, table: str = None, device_id: str = None, success: bool = True, rows: int = None):
    28→    is_critical = operation in CRITICAL_OPS
    29→    is_slow = duration_ms >= DB_SLOW_QUERY_THRESHOLD_MS
    30→    is_very_slow = duration_ms >= DB_CRITICAL_QUERY_THRESHOLD_MS
    31→
    32→    parts = [f"DB_{operation.upper()}: duration_ms={duration_ms:.2f}"]
    33→    if table:
    34→        parts.append(f"table={table}")
    35→    if device_id:
    36→        parts.append(f"device_id={device_id}")
    37→    if rows is not None:
    38→        parts.append(f"rows={rows}")
    39→    parts.append(f"success={success}")
    40→
    41→    if is_critical:
    42→        parts.append("critical=true")
    43→    if is_slow:
    44→        parts.append(f"slow={'VERY_SLOW' if is_very_slow else 'SLOW'}")
    45→
    46→    message = " ".join(parts)
    47→
    48→    if not success:
    49→        logger.error(message)
    50→    elif is_very_slow:
    51→        logger.warning(f"⚠ {message}")
    52→    elif is_slow and is_critical:
    53→        logger.warning(message)
    54→    elif is_critical:
    55→        logger.info(message)
    56→    else:
    57→        logger.debug(message)
    58→
    59→
    60→def _extract_value(row: Union[list, dict], index_or_key: Union[int, str] = 0, value_type: str = "text") -> Optional[Any]:
    61→    if row is None:
    62→        return None
    63→
    64→    if isinstance(row, list):
    65→        idx = index_or_key if isinstance(index_or_key, int) else 0
    66→        if len(row) <= idx:
    67→            return None
    68→        value_dict = row[idx] if isinstance(row[idx], dict) else None
    69→        if value_dict and value_dict.get("type") == value_type:
    70→            return value_dict.get("value")
    71→        return None
    72→
    73→    elif isinstance(row, dict):
    74→        if isinstance(index_or_key, str):
    75→            return row.get(index_or_key)
    76→        return None
    77→
    78→    return None
    79→
    80→