role: user

type: text

text:
<user_query>
Explore the /home/ubuntu/pipecat project to understand:
1. What is the project structure? List key files and directories.
2. Where is image generation handled? Search for files containing "image", "generate", "generation", "img", "dalle", "stable diffusion", or similar image generation related code.
3. What image generation service/API is currently being used?
4. Find any configuration files, environment variables, or API keys related to image generation.
5. Look for any FastAPI/Flask server files that handle image generation endpoints.

Be thorough - search for patterns like "image_gen", "generate_image", "ImageGen", "banana", "replicate", "flux", "dall-e", "midjourney" etc.

Return:
- The full project directory structure (top 2-3 levels)
- The exact file paths where image generation code lives
- The current image generation implementation details (which API, how it's called)
- Any existing configuration/flag system for switching between services
</user_query>


role: assistant

text:
I'll explore the pipecat project systematically. Let me start with the directory structure and search for image generation related code in parallel.

Let me explore the /home/ubuntu/pipecat project thoroughly to answer all the user's questions.


text: Now let me read the core image generation files in parallel.


text: Now let me read the configuration, constants, and environment files.


text: Now let me read the turso_client for the config/flag functions and the bot/main.py to see how handlers are wired up.


text:
Here is a comprehensive analysis of the pipecat project's image generation system.

---

## 1. Project Directory Structure (Top 3 Levels)

```
/home/ubuntu/pipecat/
├── server.py                          # FastAPI main server
├── requirements.txt
├── .env                               # Environment variables (API keys, configs)
├── README.md
├── bot/
│   ├── __init__.py
│   ├── main.py                        # Bot entrypoint, wires up all handlers
│   ├── config.py                      # BotSessionState, TranscriptManager
│   ├── pipeline.py                    # Pipeline creation (LLM, STT, TTS, tools)
│   ├── session_handlers.py            # Session lifecycle handlers
│   ├── observers.py                   # Pipeline observers/metrics
│   └── image_upload.py                # Image upload handling
├── fnHandlers/                        # ** CORE IMAGE GENERATION LOGIC **
│   ├── __init__.py                    # Exports all handler + function schemas
│   ├── image_handler.py               # Azure OpenAI image generation handler
│   ├── cf_image_handler.py            # CF Flux generation + editing handler (PRIMARY)
│   ├── edit_image_handler.py          # External service image editing
│   ├── azure_edit_image_handler.py    # Azure OpenAI image editing
│   ├── video_handler.py               # Video generation (Replicate)
│   ├── clients/
│   │   ├── __init__.py
│   │   ├── cf_flux_client.py          # Cloudflare Flux API client
│   │   ├── replicate_flux_client.py   # Replicate Flux fallback client
│   │   ├── azure_image_client.py      # Azure OpenAI image client
│   │   ├── edit_service_client.py     # External edit service client
│   │   └── exceptions.py             # Custom exceptions
│   └── utils/
│       ├── __init__.py
│       ├── prompt_writer.py           # Azure GPT-5.2 prompt optimization
│       ├── image_describer.py         # Gemini Vision image description
│       ├── celebrity_reference.py     # Celebrity reference images
│       ├── handler_helpers.py         # Shared handler utilities
│       ├── limit_handler.py           # Rate limit logic
│       └── rtvi_events.py            # RTVI event sender
├── routes/                            # FastAPI route modules
│   ├── bot.py, health.py, characters.py, video.py, explore.py, ...
├── utils/
│   ├── constants.py                   # All config constants & env vars
│   ├── turso_client.py                # Database client + feature flags
│   ├── s3_utils.py, redis_client.py, session_state.py, ...
├── prompts/
│   ├── prompt_global_agent.txt        # Main system prompt
│   ├── image_describer_prompt.txt     # Gemini describer prompt
│   └── prompt_writer_template.txt     # Prompt writer template
├── migrations/                        # SQL migrations
│   ├── 025_add_cf_flux_config.sql     # CF Flux multi-account config
│   └── ...
├── deployments/
│   ├── docker/Dockerfile
│   └── k8s/deployment.yaml, ...
├── references/                        # Celebrity reference images
│   ├── pushpa.png, salmankhan.webp, sharuhkhan.jpg, ...
├── scripts/
│   └── upload_explore_images.py, seed_characters.py
└── cloudflare/                        # Cloudflare status API worker
```

---

## 2. Image Generation File Paths

### Core Handlers (orchestration logic)
| File | Purpose |
|------|---------|
| `fnHandlers/cf_image_handler.py` | **PRIMARY** -- CF Flux generate + edit (the only function registered in `bot/main.py`) |
| `fnHandlers/image_handler.py` | Azure OpenAI `gpt-image-1.5` generation (text/religious images) |
| `fnHandlers/edit_image_handler.py` | External service image editing (kept for reference, not actively used) |
| `fnHandlers/azure_edit_image_handler.py` | Azure OpenAI editing (celebrity, text, religious edits) |

### API Clients (actual API calls)
| File | Purpose |
|------|---------|
| `fnHandlers/clients/cf_flux_client.py` | Cloudflare Workers AI Flux API (model: `flux-2-klein-4b`) |
| `fnHandlers/clients/replicate_flux_client.py` | Replicate fallback (model: `black-forest-labs/flux-2-klein-4b`) |
| `fnHandlers/clients/azure_image_client.py` | Azure OpenAI (model: `gpt-image-1.5-2`) |
| `fnHandlers/clients/edit_service_client.py` | External edit service at `http://154.54.100.103:8080/generate_base64` |

### Utility/Support
| File | Purpose |
|------|---------|
| `fnHandlers/utils/prompt_writer.py` | Azure GPT-5.2 prompt optimization |
| `fnHandlers/utils/image_describer.py` | Gemini Vision image description |
| `fnHandlers/utils/celebrity_reference.py` | Celebrity detection + reference images |
| `bot/image_upload.py` | Handles user image uploads |
| `bot/main.py` | Wires handlers to the LLM (lines 118-125) |
| `bot/pipeline.py` | Registers `cf_image_function` as the tool schema (line 30, 399) |

---

## 3. Current Image Generation Implementation

### Active Architecture (what's actually registered and used)

Only **one** image function is registered with the LLM in `bot/main.py`:

```120:125:bot/main.py
        llm.register_function("cf_image", create_cf_image_handler(
            session_state, rtvi, bot_session, transcript_manager, task, llm, trigger_app_review_if_eligible
        ))
```

The `cf_image` function handles **both** generation and editing via an `operation` parameter (`"generate"` or `"edit"`).

### Service Chain (with fallbacks)

**Image Generation flow:**
1. **Primary: Cloudflare Workers AI** -- `CFFluxClient` calls Cloudflare Workers endpoints using the **Flux 2 Klein 4B** model. Multi-account round-robin from DB config (`cf_flux_config`).
2. **Fallback: Replicate** -- `ReplicateFluxClient` calls `black-forest-labs/flux-2-klein-4b` on Replicate if CF fails.

**Image Editing flow:**
1. **Primary: Cloudflare Workers AI** -- Same `CFFluxClient.edit_image()` with the user's image.
2. **Fallback: Replicate** -- Same `ReplicateFluxClient.edit_image()`.

**Azure handlers** (`generate_image`, `azure_edit_image`) exist but are **gated behind the `is_azure_edit_enabled()` flag** from the database. When disabled, they redirect to `cf_image`. They are NOT registered as LLM tools in `bot/main.py`.

### How Each API is Called

**Cloudflare Flux** (`cf_flux_client.py`):
- HTTP POST to CF Workers URL (e.g., `https://cf-flux-dev-2.dheemanth.workers.dev/api/generate`)
- Sends multipart files: `prompt`, `model` (flux-2-klein-4b), `num_steps` (25), `width/height` (1024), `guidance` (3.5)
- Auth via `api-key` header
- Returns raw PNG image bytes

**Replicate** (`replicate_flux_client.py`):
- Uses `replicate` Python SDK (`replicate.run()` in a thread pool)
- Model: `black-forest-labs/flux-2-klein-4b`
- For edits: passes images as data URIs

**Azure OpenAI** (`azure_image_client.py`):
- Uses `openai` Python SDK (`client.images.generate()` / `client.images.edit()`)
- Endpoint: `https://bhara-mjp8hlry-eastus2.openai.azure.com/openai/v1/`
- Model/deployment: `gpt-image-1.5-2`
- Compresses images to 512px JPEG before editing

---

## 4. Configuration & Feature Flag System

### Environment Variables (`.env` and `constants.py`)

| Variable | Value/Default | Used By |
|----------|---------------|---------|
| `AZURE_OPENAI_ENDPOINT` | `https://bhara-mjp8hlry-eastus2.openai.azure.com/openai/v1/` | Azure image client |
| `AZURE_OPENAI_API_KEY` | Set in `.env`/constants | Azure image client |
| `AZURE_OPENAI_DEPLOYMENT_NAME` | `gpt-image-1.5-2` | Azure image client |
| `REPLICATE_API_TOKEN` | Set in `.env` | Replicate fallback |
| `EXTERNAL_IMAGE_EDIT_SERVICE_URL` | `http://154.54.100.103:8080/generate_base64` | External edit service |
| `OPENROUTER_API_KEY` | Set in `.env` | Prompt writer (OpenRouter) |

### Database-Driven Feature Flags (via `turso_client.py`)

| Flag/Config | Function | Controls |
|-------------|----------|----------|
| `cf_flux_config` | `get_cf_flux_enabled_accounts()` | Multi-account round-robin; each account has `url`, `api_key`, `enabled` |
| `azure_edit_enabled` | `is_azure_edit_enabled()` | Gates Azure image gen/edit; when False, redirects to `cf_image` |
| `prompt_writer_config` | `is_prompt_writer_enabled()` | Toggle prompt optimization via GPT-5.2 |
| `image_describer_config` | `is_image_describer_enabled()` | Toggle Gemini Vision image description |
| `feature_flags` | Various | General feature flags |

The DB config is refreshed periodically by `ConfigSyncManager` (every 300s) started in `server.py`.

### Code-Level Toggle

```16:16:utils/constants.py
UNLIMITED_FOR_ALL = True  # Toggle this to enable/disable unlimited for everyone
```

---

## 5. Server Endpoints

The FastAPI server in `server.py` does **not** have dedicated REST endpoints for image generation. Image generation is handled entirely through the **real-time voice pipeline** (WebRTC via Daily):

1. User connects via WebRTC (`routes/bot.py` creates a Daily room)
2. User speaks to the LLM (Gemini/OpenAI)
3. LLM decides to call the `cf_image` function (registered as an LLM tool)
4. The handler generates/edits the image and sends results back via RTVI events
5. Generated images are uploaded to S3 and delivered via CloudFront CDN

There are no `/api/generate-image` type REST endpoints -- it's all function-calling within the conversational AI pipeline.

---

### Summary

The project uses a **multi-layer image generation architecture** with:
- **Primary**: Cloudflare Workers AI (Flux 2 Klein 4B) with multi-account round-robin
- **Fallback**: Replicate (same Flux model)
- **Optional/Gated**: Azure OpenAI (gpt-image-1.5) for text-on-image and religious/celebrity content
- **Legacy**: External edit service (http endpoint) -- kept for reference but not actively registered

Service switching is controlled through **database-driven feature flags** (`turso_client.py`) that are synced every 5 minutes, allowing runtime toggling without redeployment.