Instruction:
## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  3) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  4) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.


# AGENTS.md instructions for /home/ubuntu/maya3

<INSTRUCTIONS>
## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  3) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  4) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.
</INSTRUCTIONS>


<environment_context>
  <cwd>/home/ubuntu/maya3</cwd>
  <approval_policy>never</approval_policy>
  <sandbox_mode>danger-full-access</sandbox_mode>
  <network_access>enabled</network_access>
  <shell>bash</shell>
</environment_context>


# Context from my IDE setup:

## Open tabs:
- .env: .env

## My request for Codex:
okay so here's the situation. Now, we have 2 buckets in R2.

1. test bucket - where our source truth lies and where our original files are existing in.
2. 1-cleaned-data - where our final cleaned data for each of the data samples we picked from supabase.

now, since we had a different lists of items in different csv files, we generally pushed all raw audio files into the test bucket and synced the csv files with supabase. and had our workers request videoIDs from suapabse and keep track of statusses. and keep working on claimed videos and finish working and push to 1-cleaned-data bucket. Now, my question is there is a descrepency between these 2 buckets with a total of 20k videos. as per the total metadata provided by R2. the test bucket has total of 536.16k items or objects and the 1-cleaned-data bucket has total number of files somewhere at - 502.24k and a few 1000 files yet to come(will come in the next few minutes or seconds)

the point and question is - whatever might be the reason for this descrepency, i'm more concerned about if there is a quickest and fasted way of identifying the missing or descrepency of the difference of 20k-30k videos from both buckets and is there a way for us to identify those IDs without having to search the metadata which takes a hella lot of time and as many classA searches. well i can afford to move ahead as i already have enough 500k and these 20k dont much bother me. but however if there is a quickest way to find that list, i dont wanna lose up on that. so, tell me if there a fastest and cheapest way to find that descrepency or do we have to do it the hard way.


Summary: auto

## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  3) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  4) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.


**Considering Cloudflare R2 options**

I need to respond as a coding assistant about the Cloudflare R2 buckets. The user is looking for a fast and cost-effective way to find missing IDs between the "test" and "1-cleaned-data" buckets without heavy metadata operations. Since they have Codex CLI available, I think we might explore writing scripts to diff object lists. It’s also important to consider using Cloudflare R2 inventory reports, while being mindful of Class A operation costs.


**Exploring Cloudflare R2 Inventory**

I’m considering Cloudflare R2, which has an "R2 Inventory" feature, similar to AWS S3 inventory, that generates daily or weekly CSV/Parquet files listing objects in another bucket. This allows for diffing without needing to list each object. I think they may have announced an "R2 Data Catalog" too, but I need to verify that. The user can also compare cleaned files using a Supabase table if they have one. By checking object keys for video IDs, we can compute the difference between the two buckets.


**Analyzing Class A operations**

I’m realizing that listing 500k objects in each bucket involves a lot of Class A operations. Each call to `ListObjectsV2` returns up to 1000 keys, so that means 500 API calls per bucket. While this isn't a huge number, I understand the user's concern about the time and costs associated with metadata searches. At $0.36 per million Class A operations, the cost for 1000 operations is just $0.00036. The bigger issue might be the time taken and potential timeouts, but using concurrency should help speed things up.


**Considering diff methods for buckets**

I’m thinking that list operations only return keys and not HEAD data for each object. Therefore, the simplest and most cost-effective way to find differences is to list the keys from both buckets and compute the differences. This might take some time, but not excessively. Using `rclone check` between two remotes can be helpful since it lists both sides and computes hashes. Alternatively, if they already have a Supabase list of video IDs, we could compare that directly to identify missing objects. However, we need the keys to pinpoint what’s missing between the buckets.


I'm considering the best way to identify missing objects between two buckets using Cloudflare R2. First, I can suggest using `rclone sync` with `--dry-run` and `--size-only` to see which files would be copied, but listing is still required. A more efficient way could be to extract object keys to deduce video IDs, possibly using patterns like `raw/<videoID>.mp3` versus `cleaned/<videoID>.wav`.

Using `aws s3api list-objects-v2` or `rclone lsf` would allow for listing keys quickly. Despite concerns about Class A operation costs, the expenses are minimal. The real challenge lies in handling the differences in counts between the buckets due to unique or ancillary files. I could sketch a solution to list, deduplicate, and compare video IDs for the user. This could help efficiently find discrepancies.


I'm considering implementing a script in the repo to help identify discrepancies between the buckets. While it's unclear if the user wants this in the repo, a script to output missing IDs could definitely be beneficial. First, I need to check for existing scripts related to R2 operations in the repo. I'll also reference the AGENTS.md instructions provided by the user to ensure compliance with any specified skills.

We can run commands without requiring approval since the environment settings are favorable. However, we must be cautious with R2 credentials stored in the `.env` file to avoid leaking sensitive information while still ensuring the script reads those credentials for local execution. I should guide the user on running the script, suggesting listing keys for both buckets and computing the differences, which is cost-effective and relatively straightforward. This way, they can identify the missing videos without delving into time-consuming metadata searches.


I'm outlining steps for comparing IDs between two buckets. First, we can use the `comm` command to find missing and extra IDs, which is quite straightforward. Using `rclone` is also an option; we just need to configure it with the R2 remote and note that sorting 500k lines might take some time, but it's manageable.

Alternatively, while checking for missing clean outputs, we can list expected IDs from Supabase and confirm their existence using HEAD requests. This isn't as efficient as listing but would still be relatively cheap. However, to reduce memory usage during sorting, we might consider streaming with a Python script that handles the diff process. I'll provide a guide on that, ensuring to use placeholder environment variables to keep credentials safe.


Output:
Exit code: 0
Wall time: 0 seconds
Output:
2026-01-12-this-session-is-being-continued-from-a-previous-co.txt
dashboard-ui
data
discovered_podcasts_te_SUPABASE_UNIQUE_50000.csv
docker-compose.yml
Dockerfile
massive_process.py
metadata.json
new_podcasts.csv
output
pipeline.py
__pycache__
R2_chapters_FINAL.csv
README.md
reports
requirements.txt
scripts
src
tmp
vast_ai.md
vast_ai_sdk.md


Summary: auto


**Exploring R2 interactions**

I'm investigating a Python-based pipeline that might include interactions with R2. I’ll delve into the `scripts` or `src` directories to look for relevant R2 code. I think using ripgrep to search for terms like "r2," "cloudflare," "s3," and "boto3" will help me locate what I need quickly. This feels like a solid plan to ensure I'm covering all my bases in this exploration!


rg -n "\bR2\b|cloudflare|cloudflarestorage|r2\.cloudflarestorage|boto3|s3api|S3" -S . | head


Output:
Exit code: 0
Wall time: 0 seconds
Output:
./requirements.txt:36:boto3
./.env.example:11:# Cloudflare R2 (required for audio source and output)
./.env.example:12:ACCOUNT_ID=your_cloudflare_account_id
./pipeline.py:873:        'source_url': metadata.get('source_url'),  # R2/YouTube URL for re-download
./pipeline.py:1018:                # locate outputs in R2 from Supabase without guessing prefixes.
./pipeline.py:1052:                # Step 1.5: Remove original audio from output (source is preserved in R2)
./pipeline.py:1057:                        logger.debug(f"Removed _original.flac from output (source in R2)")
./pipeline.py:1065:                # Step 3: Upload to R2 (if enabled)
./pipeline.py:1070:                        logger.info(f"R2 upload skipped (exists): {r2.uri(remote_key)}")
./pipeline.py:1074:                            raise RuntimeError(f"R2 upload failed for {r2.uri(remote_key)}")


Output:
Exit code: 0
Wall time: 0 seconds
Output:
audit_completed_r2.py
audit_r2_outputs.py
check_videos.py
clear_supabase_analytics.py
deploy
docker
find_valid_videos.py
fix_rtf.py
monitor
monitor_analytics.py
prepare_benchmark.py
process_specific_videos.py
prune_dead_vast_workers_in_supabase.py
__pycache__
reset_for_production.py
run_analytics_migration.py
run_migration.sh
supabase_migrations.sql
sync_csv_to_supabase.py
sync_discovered_csv_to_supabase.py
test_analytics.py
vast


Output:
Exit code: 0
Wall time: 0 seconds
Output:
analytics.py
audio_buffer.py
boundary_refinement.py
chunk_parallel.py
chunk_reassignment.py
clustering.py
compute.py
config.py
diarization.py
download.py
embeddings.py
export.py
hybrid_music_detection.py
__init__.py
logger.py
models.py
music_detection.py
overlap_detection.py
__pycache__
r2_client.py
spectral_analysis.py
supabase_client.py
unified_embeddings.py
utils.py
vad.py
worker_pools.py
worker.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:
# Maya3 Audio Diarization Pipeline

High-throughput, GPU-accelerated audio diarization and segmentation pipeline. Processes podcast audio from R2 storage, performs speaker diarization, music detection, and outputs clean speaker-segmented data.

## Performance

- **361x real-time** throughput (8x RTX 4090)
- **90+ hours** of audio processed in ~15 minutes
- Distributed processing via Supabase queue

## Architecture

```
R2 (Source Audio) → Supabase Queue → GPU Workers → R2 (Output .tar)
                         ↓
                    Status Updates
```

## Pipeline Stages

| Stage | Description |
|-------|-------------|
| 1. Download | Fetch audio from R2 bucket |
| 2. VAD | Voice activity detection (Silero) |
| 3. Chunking | Split into processable chunks |
| 4. Diarization | Speaker identification (PyAnnote) |
| 5. Embeddings | Speaker embedding extraction |
| 6. Clustering | Speaker clustering & refinement |
| 7. Music Detection | INA + PANNs hybrid detection |
| 8. Finalize | Quality filtering & validation |
| 9. Output | Generate segmented JSON |
| 10. Export | Upload .tar to R2 |

## Quick Start

### Prerequisites

- NVIDIA GPU with CUDA 12.1+
- Docker with NVIDIA Container Toolkit
- Hugging Face token (for PyAnnote models)
- Supabase project (for queue)
- Cloudflare R2 bucket (for storage)

### Setup

1. Clone and configure:
```bash
cp .env.example .env
# Edit .env with your credentials
```

2. Build container:
```bash
docker-compose build
```

3. Run workers:
```bash
docker-compose up -d
```

### Local Docker Smoke Test

On a machine with Docker + NVIDIA Container Toolkit:

```bash
chmod +x scripts/docker/smoke_test.sh
./scripts/docker/smoke_test.sh
```

Optional end-to-end test (process 1 Supabase-queued video on GPU 0 and validate TAR contents):

```bash
E2E=1 GPU_DEVICE=0 ./scripts/docker/smoke_test.sh
```

### Environment Variables

| Variable | Description |
|----------|-------------|


**Considering auditing tools**

I see there are scripts like `audit_r2_outputs.py` that might already serve the purpose of auditing outputs between R2 and Supabase. Our focus is on spotting discrepancies between the two buckets. It's possible there's an existing script for auditing R2 outputs. I'll need to inspect `src/r2_client.py` and related scripts. Since the repo seems centered around an audio pipeline, there might be a suitable R2 client wrapper to help implement a new script or modify something existing quickly.


Output:
Exit code: 0
Wall time: 0 seconds
Output:
#!/usr/bin/env python3
"""
R2 Client Utilities for High-Throughput Pipeline Migration

Manages Cloudflare R2 buckets for:
- Production outputs (1-cleaned-data)
- Test outputs during migration (1-cleaned-data-test)
- Baseline snapshots for comparison (1-cleaned-data-baseline)

Usage:
    from src.r2_client import R2Client

    client = R2Client()
    client.upload_json("path/to/file.json", "video_id/metadata.json")
    client.download_json("video_id/metadata.json", "local/path.json")
"""

import os
import json
import logging
from pathlib import Path
from typing import Dict, Any, Optional, List
from dataclasses import dataclass

import boto3
from botocore.config import Config as BotoConfig

logger = logging.getLogger("R2Client")

@dataclass
class R2Config:
    """R2 bucket configuration."""
    endpoint_url: str
    access_key_id: str
    secret_access_key: str
    bucket: str
    prefix: str = ""

# Bucket configurations
BUCKETS = {
    'production': '1-cleaned-data',
    'test': '1-cleaned-data-test',
    'baseline': '1-cleaned-data-baseline',
    # Source/original videos (DO NOT DELETE). This bucket is expected to contain
    # the original YouTube files re-uploaded to R2 to avoid rate limits.
    'source': 'test',
}

def get_r2_config_from_env() -> R2Config:
    """Load R2 configuration from environment variables."""
    return R2Config(
        # Support a few common aliases used in various deploy templates.
        endpoint_url=os.environ.get('R2_ENDPOINT_URL') or os.environ.get('S3_API') or '',
        access_key_id=os.environ.get('R2_ACCESS_KEY_ID') or os.environ.get('AWS_ACCESS_KEY_ID') or '',
        secret_access_key=os.environ.get('R2_SECRET_ACCESS_KEY') or os.environ.get('AWS_SECRET_ACCESS_KEY') or '',
        bucket=os.environ.get('R2_BUCKET', 'test'),
        prefix=os.environ.get('R2_PREFIX', ''),
    )

class R2Client:
    """
    Cloudflare R2 client for pipeline data management.

    Supports multiple bucket targets for A/B testing during migration.
    """

    def __init__(
        self,
        config: Optional[R2Config] = None,
        bucket_type: str = 'test'
    ):
        """
        Initialize R2 client.

        Args:
            config: R2 configuration (loads from env if None)
            bucket_type: One of 'production', 'test', 'baseline'
        """
        self.config = config or get_r2_config_from_env()

        # Override bucket based on type
        if bucket_type in BUCKETS:
            self.bucket = BUCKETS[bucket_type]
        else:
            self.bucket = self.config.bucket

        # IMPORTANT: The output prefix (R2_PREFIX) should not accidentally apply
        # to the *source* bucket. Vast/RunPod templates often set R2_PREFIX for
        # uploads; for source reads we want R2_SOURCE_PREFIX (or empty).
        if bucket_type == 'source' and config is None:
            self.prefix = os.environ.get('R2_SOURCE_PREFIX', '') or ''
        else:
            self.prefix = self.config.prefix

        # Initialize boto3 client
        self._client = None

    @property
    def client(self):
        """Lazy-initialize boto3 client."""
        if self._client is None:
            if not self.config.endpoint_url:
                raise RuntimeError("R2 not configured: missing R2_ENDPOINT_URL (or S3_API)")
            if not self.config.access_key_id:
                raise RuntimeError("R2 not configured: missing R2_ACCESS_KEY_ID (or AWS_ACCESS_KEY_ID)")
            if not self.config.secret_access_key:
                raise RuntimeError("R2 not configured: missing R2_SECRET_ACCESS_KEY (or AWS_SECRET_ACCESS_KEY)")
            self._client = boto3.client(
                's3',
                endpoint_url=self.config.endpoint_url,
                aws_access_key_id=self.config.access_key_id,
                aws_secret_access_key=self.config.secret_access_key,
                config=BotoConfig(
                    signature_version='s3v4',
                    retries={'max_attempts': 3, 'mode': 'adaptive'},
                    # Prevent long hangs on HEAD/LIST during intermittent network stalls.
                    connect_timeout=int(os.environ.get("R2_CONNECT_TIMEOUT_SEC", "10") or 10),
                    read_timeout=int(os.environ.get("R2_READ_TIMEOUT_SEC", "60") or 60),
                    max_pool_connections=int(os.environ.get("R2_MAX_POOL_CONNECTIONS", "64") or 64),
                )
            )
        return self._client

    def full_key(self, key: str) -> str:
        """Public helper: return fully-qualified key (including configured prefix)."""
        return self._full_key(key)

    def uri(self, key: str) -> str:
        """Return an s3:// URI for a key (including configured prefix)."""
        return f"s3://{self.bucket}/{self._full_key(key)}"

    def _full_key(self, key: str) -> str:
        """Get full S3 key with prefix."""
        if self.prefix:
            return f"{self.prefix}/{key}"
        return key

    def upload_file(self, local_path: str, remote_key: str) -> bool:
        """
        Upload a file to R2.

        Args:
            local_path: Path to local file
            remote_key: Remote key (path in bucket)

        Returns:
            True if successful
        """
        try:
            full_key = self._full_key(remote_key)
            self.client.upload_file(local_path, self.bucket, full_key)
            logger.info(f"Uploaded {local_path} -> s3://{self.bucket}/{full_key}")
            return True
        except Exception as e:
            logger.error(f"Upload failed: {e}")
            return False

    def download_file(self, remote_key: str, local_path: str) -> bool:
        """
        Download a file from R2.

        Args:
            remote_key: Remote key (path in bucket)
            local_path: Path to save locally

        Returns:
            True if successful
        """
        try:
            full_key = self._full_key(remote_key)
            Path(local_path).parent.mkdir(parents=True, exist_ok=True)
            self.client.download_file(self.bucket, full_key, local_path)
            logger.info(f"Downloaded s3://{self.bucket}/{full_key} -> {local_path}")
            return True
        except Exception as e:
            logger.error(f"Download failed: {e}")
            return False

    def upload_json(self, data: Dict[str, Any], remote_key: str) -> bool:
        """Upload JSON data directly to R2."""
        try:
            full_key = self._full_key(remote_key)
            body = json.dumps(data, indent=2).encode('utf-8')
            self.client.put_object(
                Bucket=self.bucket,
                Key=full_key,
                Body=body,
                ContentType='application/json'
            )
            logger.info(f"Uploaded JSON -> s3://{self.bucket}/{full_key}")
            return True
        except Exception as e:
            logger.error(f"JSON upload failed: {e}")
            return False

    def download_json(self, remote_key: str) -> Optional[Dict[str, Any]]:
        """Download JSON data from R2."""
        try:
            full_key = self._full_key(remote_key)
            response = self.client.get_object(Bucket=self.bucket, Key=full_key)
            data = json.loads(response['Body'].read().decode('utf-8'))
            logger.info(f"Downloaded JSON <- s3://{self.bucket}/{full_key}")
            return data
        except Exception as e:
            logger.error(f"JSON download failed: {e}")
            return None

    def head_object(self, remote_key: str) -> Optional[Dict[str, Any]]:
        """Return S3 head_object response for a key, or None if missing/unreadable."""
        try:
            full_key = self._full_key(remote_key)
            return self.client.head_object(Bucket=self.bucket, Key=full_key)
        except Exception:
            return None

    def generate_presigned_get_url(self, remote_key: str, expires_in_sec: int = 3600) -> str:


Output:
Exit code: 0
Wall time: 0 seconds
Output:
    def generate_presigned_get_url(self, remote_key: str, expires_in_sec: int = 3600) -> str:
        """
        Generate a presigned GET URL for an object.

        Note: This is a Class B request pattern (no bucket listing).
        """
        full_key = self._full_key(remote_key)
        return self.client.generate_presigned_url(
            'get_object',
            Params={'Bucket': self.bucket, 'Key': full_key},
            ExpiresIn=expires_in_sec,
        )

    def list_keys(self, prefix: str = "", limit: int = 1000) -> List[str]:
        """
        List keys in bucket with optional prefix.

        Args:
            prefix: Additional prefix to filter by
            limit: Maximum keys to return

        Returns:
            List of keys
        """
        try:
            full_prefix = self._full_key(prefix) if prefix else self.prefix
            response = self.client.list_objects_v2(
                Bucket=self.bucket,
                Prefix=full_prefix,
                MaxKeys=limit
            )

            keys = []
            for obj in response.get('Contents', []):
                key = obj['Key']
                # Remove prefix for cleaner output
                if self.prefix and key.startswith(self.prefix):
                    key = key[len(self.prefix):].lstrip('/')
                keys.append(key)

            return keys
        except Exception as e:
            logger.error(f"List failed: {e}")
            return []

    def exists(self, remote_key: str) -> bool:
        """Check if a key exists in the bucket."""
        try:
            full_key = self._full_key(remote_key)
            self.client.head_object(Bucket=self.bucket, Key=full_key)
            return True
        except:
            return False

    def delete(self, remote_key: str) -> bool:
        """Delete a key from the bucket."""
        try:
            # Safety guard: user explicitly forbids deletions from any bucket other
            # than 1-cleaned-data. Keep this invariant even if callers misuse API.
            if self.bucket != '1-cleaned-data':
                raise PermissionError(
                    f"Refusing to delete from bucket '{self.bucket}'. "
                    f"Only '1-cleaned-data' deletions are permitted."
                )
            full_key = self._full_key(remote_key)
            self.client.delete_object(Bucket=self.bucket, Key=full_key)
            logger.info(f"Deleted s3://{self.bucket}/{full_key}")
            return True
        except Exception as e:
            logger.error(f"Delete failed: {e}")
            return False

def copy_to_baseline(video_ids: List[str], source_type: str = 'production') -> int:
    """
    Copy video outputs from source bucket to baseline bucket.

    Args:
        video_ids: List of video IDs to copy
        source_type: Source bucket type ('production' or 'test')

    Returns:
        Number of successfully copied videos
    """
    source = R2Client(bucket_type=source_type)
    baseline = R2Client(bucket_type='baseline')

    copied = 0
    for video_id in video_ids:
        # Copy metadata.json
        metadata_key = f"{video_id}/metadata.json"
        data = source.download_json(metadata_key)
        if data and baseline.upload_json(data, metadata_key):
            copied += 1
            logger.info(f"Copied {video_id} to baseline")
        else:
            logger.warning(f"Failed to copy {video_id}")

    return copied

if __name__ == "__main__":
    # Quick test
    import sys

    print("R2 Client Test")
    print("-" * 40)

    client = R2Client(bucket_type='test')

    # List some keys
    print(f"Bucket: {client.bucket}")
    print(f"Prefix: {client.prefix}")

    keys = client.list_keys(limit=5)
    print(f"Sample keys: {keys[:5]}")


sed -n '1,260p' scripts/audit_r2_outputs.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:
#!/usr/bin/env python3
"""
Audit Cloudflare R2 output uploads for Maya3.

Prints the configured production bucket + prefix and lists a small sample of keys.

Usage:
  python scripts/audit_r2_outputs.py
  python scripts/audit_r2_outputs.py --limit 50
  python scripts/audit_r2_outputs.py --prefix cleaned/trail/
"""

from __future__ import annotations

import argparse
import os
import sys

sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))

from dotenv import load_dotenv

load_dotenv()

def main() -> int:
    parser = argparse.ArgumentParser(description="Audit R2 outputs")
    parser.add_argument("--limit", type=int, default=25)
    parser.add_argument("--prefix", type=str, default="")
    args = parser.parse_args()

    from src.r2_client import R2Client

    r2 = R2Client(bucket_type="production")
    print(f"bucket: {r2.bucket}")
    print(f"env_prefix (R2_PREFIX): {r2.prefix!r}")

    prefix = (args.prefix or "").strip()
    keys = r2.list_keys(prefix=prefix, limit=args.limit)
    print(f"returned: {len(keys)} keys (limit={args.limit}, prefix={prefix!r})")
    for k in keys[: min(len(keys), 25)]:
        print(f"- {k}")
    return 0

if __name__ == "__main__":
    raise SystemExit(main())


sed -n '1,320p' scripts/audit_completed_r2.py


Output:
Exit code: 0
Wall time: 0 seconds
Total output lines: 320
Output:
#!/usr/bin/env python3
"""
Audit Supabase COMPLETED videos vs Cloudflare R2.

Checks:
1) Every COMPLETED row has a non-empty R2 reference (typically `r2_tar_key`)
2) The referenced object exists in R2 (via S3 HEAD)

Outputs a CSV report (no secrets) with per-video results.
"""

from __future__ import annotations

import argparse
import csv
import os
import sys
import time
from concurrent.futures import ThreadPoolExecutor, as_completed
from dataclasses import dataclass
from datetime import datetime, timezone
from typing import Iterable, List, Optional, Set, Tuple
from urllib.parse import urlparse

import boto3
from botocore.config import Config as BotoConfig
from botocore.exceptions import ClientError, EndpointConnectionError
from dotenv import load_dotenv
from supabase import create_client
from tqdm import tqdm

DEFAULT_BATCH_SIZE = 1000

# Known buckets used by this repo (see `src/r2_client.py`).
KNOWN_BUCKETS = [
    "1-cleaned-data",
    "1-cleaned-data-test",
    "1-cleaned-data-baseline",
    "test",
]

@dataclass(frozen=True)
class ParsedRef:
    raw: str
    bucket_hint: Optional[str]
    key: Optional[str]
    kind: str  # missing | s3 | http | path

def parse_r2_ref(raw: Optional[str], known_buckets: Iterable[str]) -> ParsedRef:
    if raw is None:
        return ParsedRef(raw="", bucket_hint=None, key=None, kind="missing")
    raw = str(raw).strip()
    if not raw:
        return ParsedRef(raw="", bucket_hint=None, key=None, kind="missing")

    buckets = set(b for b in known_buckets if b)

    if raw.startswith("s3://"):
        rest = raw[len("s3://") :]
        if "/" not in rest:
            return ParsedRef(raw=raw, bucket_hint=rest or None, key=None, kind="s3")
        bucket, key = rest.split("/", 1)
        return ParsedRef(raw=raw, bucket_hint=bucket or None, key=key.lstrip("/") or None, kind="s3")

    if raw.startswith("http://") or raw.startswith("https://"):
        parsed = urlparse(raw)
        key = (parsed.path or "").lstrip("/") or None
        host = (parsed.hostname or "").lower()
        bucket_hint = None
        for b in buckets:
            bl = b.lower()
            if host == bl or host.startswith(f"{bl}."):
                bucket_hint = b
                break
        return ParsedRef(raw=raw, bucket_hint=bucket_hint, key=key, kind="http")

    # Plain path or key. Support "bucket/key" where bucket is recognizable.
    candidate = raw.lstrip("/")
    if "/" in candidate:
        first, rest = candidate.split("/", 1)
        if first in buckets:
            return ParsedRef(raw=raw, bucket_hint=first, key=rest.lstrip("/") or None, kind="path")

    return ParsedRef(raw=raw, bucket_hint=None, key=candidate or None, kind="path")

def build_s3_client() -> "boto3.client":
    endpoint_url = os.environ.get("R2_ENDPOINT_URL") or os.environ.get("S3_API") or ""
    access_key_id = os.environ.get("R2_ACCESS_KEY_ID") or os.environ.get("AWS_ACCESS_KEY_ID") or ""
    secret_access_key = os.environ.get("R2_SECRET_ACCESS_KEY") or os.environ.get("AWS_SECRET_ACCESS_KEY") or ""

    if not endpoint_url:
        raise RuntimeError("Missing R2_ENDPOINT_URL (or S3_API) in environment.")
    if not access_key_id:
        raise RuntimeError("Missing R2_ACCESS_KEY_ID (or AWS_ACCESS_KEY_ID) in environment.")
    if not secret_access_key:
        raise RuntimeError("Missing R2_SECRET_ACCESS_KEY (or AWS_SECRET_ACCESS_KEY) in environment.")

    max_pool = int(os.environ.get("R2_MAX_POOL_CONNECTIONS", "128") or 128)
    connect_timeout = int(os.environ.get("R2_CONNECT_TIMEOUT_SEC", "10") or 10)
    read_timeout = int(os.environ.get("R2_READ_TIMEOUT_SEC", "60") or 60)

    return boto3.client(
        "s3",
        endpoint_url=endpoint_url,
        aws_access_key_id=access_key_id,
        aws_secret_access_key=secret_access_key,
        config=BotoConfig(
            signature_version="s3v4",
            retries={"max_attempts": 5, "mode": "adaptive"},
            max_pool_connections=max_pool,
            connect_timeout=connect_timeout,
            read_timeout=read_timeout,
        ),
    )

def head_exists(s3, bucket: str, key: str) -> Tuple[bool, Optional[int], Optional[str]]:
    try:
        s3.head_object(Bucket=bucket, Key=key)
        return True, 200, None
    except ClientError as e:
        status = int(e.response.get("ResponseMetadata", {}).get("HTTPStatusCode") or 0) or None
        code = e.response.get("Error", {}).get("Code")
        if status == 404 or code in {"404", "NoSuchKey", "NotFound"}:
            return False, status or 404, None
        return False, status, f"{code or 'ClientError'}"
    except EndpointConnectionError:
        return False, None, "EndpointConnectionError"
    except Exception as e:
        return False, None, type(e).__name__

def iter_completed_rows(sb, batch_size: int) -> Iterable[dict]:
    # Fetch total count first.
    count_res = sb.table("videos").select("youtube_id", count="exact").eq("status", "COMPLETED").limit(1).execute()
    total = …1334 chars truncated…row in reader:
            vid = (row.get("youtube_id") or "").strip()
            if vid:
                done.add(vid)
    return done

def main() -> int:
    ap = argparse.ArgumentParser(description="Audit COMPLETED videos in Supabase vs object existence in R2.")
    ap.add_argument("--batch-size", type=int, default=DEFAULT_BATCH_SIZE)
    ap.add_argument("--workers", type=int, default=int(os.environ.get("R2_AUDIT_WORKERS", "64") or 64))
    ap.add_argument(
        "--snapshot-completed-before",
        default="",
        help=(
            "Optional ISO timestamp to freeze the COMPLETED set (filters `processing_completed_at` <= timestamp). "
            "Default: now (UTC)."
        ),
    )
    ap.add_argument(
        "--default-bucket",
        default="1-cleaned-data",
        help="Bucket to use when r2_tar_key does not specify one (default: 1-cleaned-data).",
    )
    ap.add_argument(
        "--fallback-buckets",
        default="",
        help="Comma-separated bucket names to try if not found in the default bucket (only when bucket is unspecified).",
    )
    ap.add_argument(
        "--out",
        default="",
        help="Output CSV path (default: reports/r2_audit_completed_<timestamp>.csv).",
    )
    ap.add_argument(
        "--resume",
        action="store_true",
        help="If --out exists, skip youtube_ids already present and append remaining rows.",
    )
    args = ap.parse_args()

    load_dotenv(dotenv_path=os.environ.get("DOTENV_PATH") or ".env", override=False)

    if not os.environ.get("URL") or not os.environ.get("SUPABASE_ADMIN"):
        raise RuntimeError("Missing Supabase env vars: URL and/or SUPABASE_ADMIN.")

    sb = create_client(os.environ["URL"], os.environ["SUPABASE_ADMIN"])
    s3 = build_s3_client()

    snapshot_ts = (args.snapshot_completed_before or "").strip()
    if not snapshot_ts:
        snapshot_ts = datetime.now(timezone.utc).isoformat()

    default_bucket = (args.default_bucket or "").strip()
    env_bucket = (os.environ.get("R2_BUCKET") or "").strip()
    fallback = [b.strip() for b in (args.fallback_buckets or "").split(",") if b.strip()]
    # Reasonable bucket search order when the row doesn't specify bucket.
    bucket_candidates = []
    for b in [default_bucket, env_bucket, *fallback, *KNOWN_BUCKETS]:
        if b and b not in bucket_candidates:
            bucket_candidates.append(b)

    out_path = args.out.strip()
    if not out_path:
        ts = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ")
        os.makedirs("reports", exist_ok=True)
        out_path = os.path.join("reports", f"r2_audit_completed_{ts}.csv")

    # Count upfront for progress bar.
    count_res = (
        sb.table("videos")
        .select("youtube_id", count="exact")
        .eq("status", "COMPLETED")
        .lte("processing_completed_at", snapshot_ts)
        .limit(1)
        .execute()
    )
    total_completed = int(getattr(count_res, "count", 0) or 0)

    done_ids: Set[str] = set()
    if args.resume:
        done_ids = load_done_ids_from_csv(out_path)

    stats = {
        "completed_total": total_completed,
        "snapshot_completed_before": snapshot_ts,
        "missing_r2_ref": 0,
        "bad_r2_ref": 0,
        "exists_in_r2": 0,
        "missing_in_r2": 0,
        "head_errors": 0,
    }
    samples = {
        "missing_r2_ref": [],
        "missing_in_r2": [],
        "head_errors": [],
        "bad_r2_ref": [],
    }

    write_header = True
    file_mode = "w"
    if args.resume and os.path.exists(out_path) and os.path.getsize(out_path) > 0:
        write_header = False
        file_mode = "a"

    def check_one(youtube_id: str, r2_ref_raw: Optional[str]) -> dict:
        parsed = parse_r2_ref(r2_ref_raw, bucket_candidates)
        exists = False
        checked_bucket = ""
        checked_key = ""
        http_status = None
        head_error = None

        if parsed.kind == "missing" or not parsed.key:
            return {
                "youtube_id": youtube_id,
                "status": "COMPLETED",
                "r2_tar_key_raw": "" if r2_ref_raw is None else str(r2_ref_raw),
                "parsed_kind": parsed.kind,
                "parsed_bucket_hint": parsed.bucket_hint or "",
                "parsed_key": parsed.key or "",
                "checked_bucket": "",
                "checked_key": "",
                "exists_in_r2": "false",
                "head_http_status": "",
                "head_error": "",
            }

        buckets_to_try = [parsed.bucket_hint] if parsed.bucket_hint else bucket_candidates

        for b in buckets_to_try:
            ok, st, err = head_exists(s3, b, parsed.key)
            checked_bucket = b
            checked_key = parsed.key
            http_status = st
            head_error = err
            if ok:
                exists = True
                break
            if parsed.bucket_hint is None and err is not None and (st is None or st >= 400) and st != 404:
                break


I need to check the output bucket structure, which seems to be in the form of `video_id/output.tar`. So, I should look closely at the pipeline upload code, especially around `r2_tar_key`. I’ll search in `pipeline.py` focusing on step 3, where the upload to R2 happens. This exploration will help me understand how everything is organized and if there are any issues I need to address. Let's go through that code!


rg -n "r2_tar_key|Upload to R2|1-cleaned-data|bucket_type=\"production\"" pipeline.py | head -n 50


Output:
Exit code: 0
Wall time: 0 seconds
Output:
1065:                # Step 3: Upload to R2 (if enabled)


Output:
Exit code: 0
Wall time: 0 seconds
Output:
    background_export = getattr(config, 'background_export', False)

    if export_enabled:
        log_stage(10, "Export", "START")

        upload_prefix = (getattr(config, 'r2_upload_prefix', '') or '').strip().strip('/')
        remote_key = f"{video_id}.tar" if not upload_prefix else f"{upload_prefix}/{video_id}.tar"
        if getattr(config, 'r2_upload_enabled', False):
            result['r2_upload_key'] = remote_key
        original_audio_path = Path(metadata.get('original_audio_path', ''))
        intro_offset = metadata.get('intro_skipped', 0.0)

        if background_export:
            # === BACKGROUND EXPORT MODE ===
            # Submit job to background manager, don't block GPU
            from src.export import ExportJob, get_export_manager
            from src.r2_client import R2Client

            export_manager = get_export_manager()
            if export_manager:
                job = ExportJob(
                    video_id=video_id,
                    output_dir=output_dir,
                    original_audio_path=original_audio_path,
                    segments=segments,
                    intro_offset=intro_offset,
                    r2_upload_enabled=getattr(config, 'r2_upload_enabled', False),
                    r2_upload_key=remote_key,
                    r2_bucket_type=getattr(config, 'r2_upload_bucket_type', 'production'),
                    include_original_in_tar=getattr(config, 'include_original_in_tar', False),
                    delete_output_dir_after_upload=getattr(config, 'r2_upload_delete_local_output_dir', False),
                )
                export_manager.submit(job)
                log_stage(10, "Export", "QUEUED", f"background job submitted")
                result['export_mode'] = 'background'
                result['r2_upload_key'] = remote_key
                # Record the exact destination (including R2_PREFIX) even though
                # the upload happens asynchronously. This makes it possible to
                # locate outputs in R2 from Supabase without guessing prefixes.
                if getattr(config, 'r2_upload_enabled', False):
                    r2 = R2Client(bucket_type=getattr(config, 'r2_upload_bucket_type', 'production'))
                    result.setdefault('export', {})
                    result['export'].update({
                        'r2_upload_bucket': r2.bucket,
                        'r2_upload_full_key': r2.full_key(remote_key),
                        'r2_upload_uri': r2.uri(remote_key),
                    })
            else:
                logger.warning("Background export manager not initialized, falling back to sync")
                background_export = False  # Fall through to sync mode

        if not background_export:
            # === SYNCHRONOUS EXPORT MODE ===
            from src.r2_client import R2Client
            from src.export import segment_audio_parallel

            with COMPUTE.monitor_stage("Export") as metrics:
                # Step 1: Segment original audio (parallel ffmpeg)
                segments_exported = 0
                if getattr(config, 'segment_audio', True) and original_audio_path.exists():
                    seg_start = time.time()
                    segments_exported = segment_audio_parallel(
                        audio_path=original_audio_path,
                        segments=segments,
                        output_dir=output_dir,
                        intro_offset=intro_offset,
                        max_workers=8,
                        output_format="flac"
                    )
                    seg_time = time.time() - seg_start
                    logger.info(f"📦 Segmented {segments_exported} clips in {seg_time:.1f}s")

                # Step 1.5: Remove original audio from output (source is preserved in R2)
                if not getattr(config, 'include_original_in_tar', False):
                    original_in_output = output_dir / f"{video_id}_original.flac"
                    if original_in_output.exists():
                        original_in_output.unlink()
                        logger.debug(f"Removed _original.flac from output (source in R2)")

                # Step 2: Create tar archive
                tar_path = output_dir.parent / f"{video_id}.tar"
                with tarfile.open(tar_path, "w") as tar:
                    tar.add(output_dir, arcname=video_id)
                tar_size_bytes = tar_path.stat().st_size if tar_path.exists() else 0

                # Step 3: Upload to R2 (if enabled)
                upload_ok = True
                if getattr(config, 'r2_upload_enabled', False):
                    r2 = R2Client(bucket_type=getattr(config, 'r2_upload_bucket_type', 'production'))
                    if getattr(config, 'r2_upload_skip_if_exists', False) and r2.exists(remote_key):
                        logger.info(f"R2 upload skipped (exists): {r2.uri(remote_key)}")
                    else:
                        upload_ok = r2.upload_file(str(tar_path), remote_key)
                        if not upload_ok:
                            raise RuntimeError(f"R2 upload failed for {r2.uri(remote_key)}")

                    # Delete local tar after upload
                    if getattr(config, 'r2_upload_delete_local_tar', True):
                        try:
                            tar_path.unlink()
                        except Exception:
                            pass

                    # Delete per-video output dir after successful upload (prevents disk fill).
                    if getattr(config, 'r2_upload_delete_local_output_dir', True):
                        try:
                            import shutil
                            shutil.rmtree(output_dir, ignore_errors=True)
                            logger.info(f"🧹 Cleaned up output directory after R2 upload")
                        except Exception:
                            pass
                else:
                    # No R2 upload - still cleanup if cleanup_after_processing is enabled
                    if getattr(config, 'cleanup_after_processing', True):
                        try:
                            import shutil
                            shutil.rmtree(output_dir, ignore_errors=True)
                            logger.info(f"🧹 Cleaned up output directory (no R2 upload)")
                        except Exception:
                            pass

            timings['export'] = metrics.duration
            compute_stats['export'] = {'cpu': metrics.cpu_percent_avg, 'gpu': metrics.gpu_percent_avg}
            result['export'] = {
                'segments_exported': segments_exported,
                'tar_size_bytes': int(tar_size_bytes),
                'r2_upload_key': remote_key if getattr(config, 'r2_upload_enabled', False) else None,
                'r2_upload_bucket': r2.bucket if getattr(config, 'r2_upload_enabled', False) else None,
                'r2_upload_full_key': r2.full_key(remote_key) if getattr(config, 'r2_upload_enabled', False) else None,
                'r2_upload_uri': r2.uri(remote_key) if getattr(config, 'r2_upload_enabled', False) else None,
            }

            log_stage(10, "Export", "DONE", f"{timings['export']:.1f}s | {segments_exported} segs | {tar_size_bytes/(1024**2):.1f}MB")

            # Update timing_total
            total_time = time.time() - total_start
            result['timing'] = {k: round(v, 2) for k, v in timings.items()}
            result['timing_total'] = round(total_time, 2)
            result['compute_stats'] = compute_stats

            # Update metadata.json
            try:
                with open(output_dir / "metadata.json", 'w') as f:
                    json.dump(result, f, indent=2)
            except Exception:
                pass

    # Summary (after optional upload so timing_total is end-to-end)
    final_total = result.get('timing_total', round(total_time, 2))
    plog.done(
        video_id=video_id,
        total_time=final_total,
        audio_min=total_duration / 60,
        speakers=result['num_speakers'],
        usable_pct=result['quality_stats']['usable_percentage']
    )

    # === v8.0: FINAL CLEANUP (prevents disk filling) ===
    # Clean up output directory if no export stage ran but cleanup is enabled
    if not export_enabled and getattr(config, 'cleanup_after_processing', True):
        try:


rg -n "R2Client\(bucket_type='source'|bucket_type=\"source\"|source bucket|test\b" src pipeline.py | head -n 80


Output:
Exit code: 0
Wall time: 0 seconds
Output:
src/r2_client.py:7:- Test outputs during migration (1-cleaned-data-test)
src/r2_client.py:44:    'test': '1-cleaned-data-test',
src/r2_client.py:48:    'source': 'test',
src/r2_client.py:59:        bucket=os.environ.get('R2_BUCKET', 'test'),
src/r2_client.py:74:        bucket_type: str = 'test'
src/r2_client.py:81:            bucket_type: One of 'production', 'test', 'baseline'
src/r2_client.py:295:    Copy video outputs from source bucket to baseline bucket.
src/r2_client.py:299:        source_type: Source bucket type ('production' or 'test')
src/r2_client.py:322:    # Quick test
src/r2_client.py:328:    client = R2Client(bucket_type='test')
src/spectral_analysis.py:351:# === Quick test function ===
src/utils.py:22:    Strategy: Pick 3 segments per speaker (longest, median, shortest).
src/utils.py:50:            indices = [0, n // 2, n - 1]  # longest, median, shortest
src/export.py:156:        # (Original is already preserved in R2 source bucket)
src/download.py:396:# Per-process cached R2 client for source/original videos (bucket: "test").
src/download.py:410:        # Avoid accidental prefixing from generic R2_PREFIX when accessing the source bucket.
src/download.py:493:    Resolve a video_id to an object key in the R2 source bucket.
src/download.py:561:        Input: https://xxx.r2.cloudflarestorage.com/test/podcasts/VIDEO.webm?X-Amz-...
src/download.py:562:        Output: test/podcasts/VIDEO.webm
src/download.py:895:        'source_url': r2_path,  # Clean R2 path: bucket/key (e.g., test/podcasts/VIDEO.webm)
src/download.py:1001:    # (bucket_type='source' -> bucket name "test") using HEAD requests only.
src/models.py:53:# - cudnn.benchmark: Auto-select fastest conv algorithm for input sizes
src/config.py:39:    # the R2 source bucket (bucket name: "test") without YouTube rate limits.
src/analytics.py:170:            # Upsert latest heartbeat (only 1 per worker, use most recent)
src/analytics.py:172:                latest = heartbeats[-1]
src/analytics.py:174:                    latest.data,


Output:
Exit code: 0
Wall time: 0 seconds
Output:
            intro_skip = get_dynamic_intro_skip(total_duration)
            duration_threshold = '≤30min' if total_duration <= 1800 else '>30min'
            skip_reason = f"dynamic:{duration_threshold}"
            matched_title = None
            logger.info(f"   ⏱ Dynamic intro skip (no chapters): {intro_skip:.0f}s (video {duration_threshold})")

    return intro_skip, outro_skip, skip_reason, matched_title

def extract_video_id(video_url: str) -> str:
    """Extract video ID from various URL formats (YouTube, R2, etc)."""
    # Bare YouTube video ID (most common in batch files)
    # Example: kIP1SiiNzVA
    if len(video_url) == 11 and all((c.isalnum() or c in "-_") for c in video_url):
        return video_url
    # YouTube formats
    if 'watch?v=' in video_url:
        return video_url.split('watch?v=')[1].split('&')[0]
    elif 'youtu.be/' in video_url:
        return video_url.split('youtu.be/')[1].split('?')[0]
    elif '/shorts/' in video_url:
        return video_url.split('/shorts/')[1].split('?')[0]
    # R2/S3 presigned URL format: extract from path before extension
    # e.g., https://xxx.r2.cloudflarestorage.com/bucket/podcasts/---g_VL7ySo.webm?X-Amz-...
    elif 'r2.cloudflarestorage.com' in video_url or 's3.amazonaws.com' in video_url:
        # Extract path from URL (before query string)
        path = video_url.split('?')[0]
        # Get the filename
        filename = path.split('/')[-1]
        # Remove extension
        video_id = filename.rsplit('.', 1)[0] if '.' in filename else filename
        return video_id
    else:
        return hashlib.md5(video_url.encode()).hexdigest()[:11]

# Per-process cached R2 client for source/original videos (bucket: "test").
# We keep this out of module imports to avoid paying boto3 init cost for
# non-R2 workflows.
_R2_SOURCE_CLIENT = None

def _get_r2_source_client(config) -> "R2Client":
    """Lazy-init the R2 client used for reading source videos (bucket_type='source')."""
    global _R2_SOURCE_CLIENT
    if _R2_SOURCE_CLIENT is None:
        from src.r2_client import R2Client, get_r2_config_from_env, R2Config

        base = get_r2_config_from_env()

        # Avoid accidental prefixing from generic R2_PREFIX when accessing the source bucket.
        # Use R2_SOURCE_PREFIX if provided; otherwise default to empty.
        source_prefix = os.environ.get('R2_SOURCE_PREFIX', '')

        cfg = R2Config(
            endpoint_url=base.endpoint_url,
            access_key_id=base.access_key_id,
            secret_access_key=base.secret_access_key,
            bucket=base.bucket,  # overridden by bucket_type='source'
            prefix=source_prefix,
        )
        _R2_SOURCE_CLIENT = R2Client(config=cfg, bucket_type='source')
    return _R2_SOURCE_CLIENT

def _iter_r2_source_prefixes(config) -> List[str]:
    """
    Build a small set of candidate prefixes for source objects without listing the bucket.
    """
    prefixes: List[str] = []

    # Highest priority: explicit env/config.
    env_prefixes = os.environ.get('R2_SOURCE_PREFIXES', '').strip()
    if env_prefixes:
        prefixes.extend([p.strip() for p in env_prefixes.split(',') if p.strip()])

    cfg_prefix = getattr(config, 'r2_source_prefix', '') or ''
    if cfg_prefix:
        prefixes.append(cfg_prefix)

    # Common defaults seen in earlier code/comments.
    prefixes.extend(['', 'podcasts', 'podcasts/', 'videos', 'videos/'])

    # Normalize to "dir/" or "" and dedupe while preserving order.
    normalized: List[str] = []
    seen = set()
    for p in prefixes:
        p = p.strip()
        if not p:
            p_norm = ''
        else:
            p_norm = p if p.endswith('/') else (p + '/')
        if p_norm not in seen:
            seen.add(p_norm)
            normalized.append(p_norm)
    return normalized

def _iter_r2_source_extensions(config) -> List[str]:
    """Return candidate extensions to try for source objects (small list, no listing)."""
    exts: List[str] = []

    env_exts = os.environ.get('R2_SOURCE_EXTENSIONS', '').strip()
    if env_exts:
        exts.extend([e.strip() for e in env_exts.split(',') if e.strip()])

    cfg_exts = getattr(config, 'r2_source_extensions', None)
    if cfg_exts:
        if isinstance(cfg_exts, str):
            exts.extend([e.strip() for e in cfg_exts.split(',') if e.strip()])
        elif isinstance(cfg_exts, (list, tuple)):
            exts.extend([str(e).strip() for e in cfg_exts if str(e).strip()])

    # Reasonable defaults for YouTube-origin files.
    exts.extend(['', '.webm', '.mp4', '.mkv', '.m4a', '.mp3', '.opus'])

    # Normalize: ensure extensions start with '.' (except empty).
    normalized: List[str] = []
    seen = set()
    for e in exts:
        e = e.strip()
        if not e:
            e_norm = ''
        else:
            e_norm = e if e.startswith('.') else ('.' + e)
        if e_norm not in seen:
            seen.add(e_norm)
            normalized.append(e_norm)
    return normalized

def _resolve_r2_source_object(video_id: str, config) -> Tuple[str, Dict[str, Any]]:
    """
    Resolve a video_id to an object key in the R2 source bucket.

    We avoid bucket listing for performance/cost reasons. Instead we try a small
    set of likely prefixes/extensions via HEAD requests only.

    Returns:
        (remote_key, head_object_response)
    """
    client = _get_r2_source_client(config)

    def _prioritize(items: List[str], preferred: List[str]) -> List[str]:
        """Return items with preferred values moved to the front (stable)."""
        out: List[str] = []
        seen = set()
        for p in preferred:
            if p in items and p not in seen:
                out.append(p)
                seen.add(p)
        for it in items:
            if it not in seen:
                out.append(it)
                seen.add(it)
        return out

    prefixes = _prioritize(
        _iter_r2_source_prefixes(config),
        # Most common layout: podcasts/{id}.webm
        ["podcasts/", "videos/", ""],
    )
    exts = _prioritize(
        _iter_r2_source_extensions(config),
        # Prefer common YouTube container formats first.
        [".webm", ".mp4", ".mkv", ".m4a", ".opus", ".mp3", ""],
    )

    first_key = None
    tried_count = 0
    for prefix in prefixes:
        for ext in exts:
            key = f"{prefix}{video_id}{ext}"
            if first_key is None:
                first_key = key
            tried_count += 1
            head = client.head_object(key)
            if head:
                logger.info(f"R2 source found: s3://{client.bucket}/{key}")
                return key, head

    # Log clearly what was tried - no exception trace needed for missing files
    logger.warning(
        f"R2 source not found for {video_id}: tried {tried_count} keys "
        f"(first: {first_key})"
    )
    raise DownloadError(
        f"R2 source not found: s3://{client.bucket}/{first_key}"
    )

def is_r2_url(url: str) -> bool:
    """Check if URL is an R2/S3 presigned URL."""
    return 'r2.cloudflarestorage.com' in url or ('s3.' in url and 'amazonaws.com' in url)

def extract_r2_path(presigned_url: str) -> str:
    """
    Extract clean R2 bucket/key path from a presigned URL.

    Example:


**Analyzing bucket processing code**

This code suggests they're avoiding bucket listing for resolving source objects, likely to minimize HEAD operations. They check for existence using HEAD instead. For auditing differences between buckets, listing is appropriate. Interestingly, they treat the test bucket as the source. Objects might be stored with prefixes like 'podcasts/' or 'videos/' and various file extensions. To compute video IDs from the source bucket, I’ll need to parse keys. Video IDs could be in the base name before the extension, and I might filter by specific known extensions and patterns.


**Developing a video ID extraction script**

I’ll start by extracting the last segment of each key (the filename), removing its extension. I need to check if the length is 11 and if it contains allowed characters, otherwise I’ll ignore it. Since there are 536k objects, this might include only video IDs. If duplicates exist with different extensions for the same ID, I’ll need to deduplicate them.

Next, I can clean keys to match formats like `<prefix>/<video_id>.tar` and remove `.tar`, checking for the same 11-character pattern. The goal will be to produce a list of missing IDs (perhaps around 20-30k) by comparing those in the source with those in the cleaned bucket. The script will need to efficiently list objects while allowing options for filtering, extract video IDs with regex, write unique sorted lists, and compute the differences for output.


**Evaluating script execution options**

If we have network access and credentials, we could run this script ourselves to generate a list of missing IDs, but I should be careful about handling sensitive information in `.env` files. While the script can load the environment variables, we need to ensure it doesn’t output any secrets. I’m concerned about the script potentially printing bucket names or access keys. I could check if certain environment variables are loaded, but I’ll also consider running the script that audits R2 outputs carefully to avoid leaking sensitive details.


**Planning script implementation**

The user asked a conceptual question, so I could respond without running a script. However, since our environment is within the repo and they might want a working script to generate differences, it would make sense to implement a new script in the `scripts/` folder. I could also extend `src/r2_client.py` to enable paginated listings, making it easier to retrieve missing IDs quickly. While it’s a multi-step task, including the change in plan might be necessary given the complexity. But since the user didn’t explicitly request code modifications, I should focus on what they need.


**Exploring existing scripts**

I want to be careful not to overstep, but adding a small script for comparing buckets is definitely helpful. I’ll take a look at the existing scripts first, like `check_videos.py` and `process_specific_videos.py`. There might already be a script designed to audit differences between buckets. I can search through the scripts for terms like "missing," "diff," or "bucket" to see if anything aligns with what I need to implement. That way, I can build on what's already there!


rg -n "diff|missing|bucket.*(test|1-cleaned-data)|list_objects_v2|paginator" scripts src | head -n 80


Output:
Exit code: 0
Wall time: 0 seconds
Output:
src/r2_client.py:59:        bucket=os.environ.get('R2_BUCKET', 'test'),
src/r2_client.py:68:    Supports multiple bucket targets for A/B testing during migration.
src/r2_client.py:74:        bucket_type: str = 'test'
src/r2_client.py:81:            bucket_type: One of 'production', 'test', 'baseline'
src/r2_client.py:107:                raise RuntimeError("R2 not configured: missing R2_ENDPOINT_URL (or S3_API)")
src/r2_client.py:109:                raise RuntimeError("R2 not configured: missing R2_ACCESS_KEY_ID (or AWS_ACCESS_KEY_ID)")
src/r2_client.py:111:                raise RuntimeError("R2 not configured: missing R2_SECRET_ACCESS_KEY (or AWS_SECRET_ACCESS_KEY)")
src/r2_client.py:213:        """Return S3 head_object response for a key, or None if missing/unreadable."""
src/r2_client.py:246:            response = self.client.list_objects_v2(
src/r2_client.py:279:            if self.bucket != '1-cleaned-data':
src/r2_client.py:299:        source_type: Source bucket type ('production' or 'test')
src/r2_client.py:328:    client = R2Client(bucket_type='test')
src/config.py:39:    # the R2 source bucket (bucket name: "test") without YouTube rate limits.
src/config.py:61:    # and upload it to the R2 production bucket (1-cleaned-data) by default.
src/config.py:63:    r2_upload_bucket_type: str = "production"  # maps to 1-cleaned-data in src/r2_client.py
src/config.py:179:    # Per instructions: 0.75-0.80 to prevent merging different speakers
src/models.py:324:            # missing or partially downloaded, the class count can be wrong
src/analytics.py:435:    Returns NoOpAnalyticsReporter if disabled or credentials missing.
src/clustering.py:16:    Cannot-link constraint: temporal overlap = different speakers.
src/download.py:396:# Per-process cached R2 client for source/original videos (bucket: "test").
src/download.py:541:    # Log clearly what was tried - no exception trace needed for missing files
src/download.py:660:                raise DownloadError("Downloaded file too small or missing")
src/download.py:818:        duration_diff = abs(trimmed_duration - original_duration_check)
src/download.py:820:        if duration_diff > 0.1:  # Allow 0.1s tolerance
src/download.py:822:            raise DownloadError(f"Duration mismatch between outputs: {duration_diff:.2f}s difference")
src/download.py:825:        logger.info(f"   ✅ Duration verified: {trimmed_duration:.2f}s (diff={duration_diff:.3f}s)")
src/download.py:895:        'source_url': r2_path,  # Clean R2 path: bucket/key (e.g., test/podcasts/VIDEO.webm)
src/download.py:1001:    # (bucket_type='source' -> bucket name "test") using HEAD requests only.
src/download.py:1091:            logger.info("⚠ Cached trimmed file exists but original missing, re-downloading")
src/download.py:1217:    encoding each to different formats in one pass.
src/download.py:1368:        duration_diff = abs(trimmed_duration - original_duration_check)
src/download.py:1370:        # Allow up to 0.1s difference (codec/rounding tolerance)
src/download.py:1371:        if duration_diff > 0.1:
src/download.py:1372:            logger.error(f"   ❌ DURATION MISMATCH! trimmed={trimmed_duration:.2f}s, original={original_duration_check:.2f}s (diff={duration_diff:.2f}s)")
src/download.py:1374:            raise DownloadError(f"Duration mismatch between outputs: {duration_diff:.2f}s difference")
src/download.py:1377:        logger.info(f"   ✅ Duration verified: trimmed={trimmed_duration:.2f}s, original={original_duration_check:.2f}s (diff={duration_diff:.3f}s)")
src/unified_embeddings.py:330:                    # Handle different output shapes
src/unified_embeddings.py:517:        # down to `severe_th`, missing real speaker changes.
src/unified_embeddings.py:1301:        # v7.8.1: Now includes "different speaker confirmation" to reduce FPs
src/unified_embeddings.py:1309:            require_different_speaker=True  # Critical for reducing FPs!
src/unified_embeddings.py:1624:    require_different_speaker: bool = True,  # v7.8.1: Critical for reducing FPs
src/unified_embeddings.py:1630:    v7.8.1 FIX: Added "different speaker confirmation" to reduce false positives.
src/unified_embeddings.py:1647:    current_run_embeddings = []  # Collect embeddings for different-speaker check
src/unified_embeddings.py:1664:                    # v7.8.1: Confirm this is ACTUALLY a different speaker
src/unified_embeddings.py:1665:                    if require_different_speaker and other_centroids:
src/unified_embeddings.py:1693:            if require_different_speaker and other_centroids:
scripts/prepare_benchmark.py:40:r2 = R2Client(bucket_type='source')  # 'test' bucket has the podcasts/
scripts/prepare_benchmark.py:57:    response = r2.client.list_objects_v2(**kwargs)
src/chunk_reassignment.py:396:        - Asks "Is chunk[i] different from EVERYTHING after it?"
src/overlap_detection.py:207:            # Check if different speakers overlap
scripts/sync_discovered_csv_to_supabase.py:6:- Inserts missing youtube_ids as `PENDING`
scripts/sync_discovered_csv_to_supabase.py:243:        raise RuntimeError(f"CSV missing required column `video_id`: {csv_path}")
scripts/vast/search_and_deploy.py:253:    missing = [k for k in required if not os.environ.get(k)]
scripts/vast/search_and_deploy.py:254:    if missing:
scripts/vast/search_and_deploy.py:255:        raise SystemExit(f"Missing required env vars: {', '.join(missing)}")
scripts/reset_for_production.py:8:3. Deletes everything in R2 bucket 1-cleaned-data/cleaned/trail/
scripts/reset_for_production.py:177:    """Delete everything in R2 bucket 1-cleaned-data/cleaned/trail/."""
scripts/reset_for_production.py:191:        # Use boto3 paginator for listing all objects
scripts/reset_for_production.py:192:        paginator = client.client.get_paginator('list_objects_v2')
scripts/reset_for_production.py:193:        for page in paginator.paginate(Bucket='1-cleaned-data', Prefix=prefix):
scripts/find_valid_videos.py:20:response = r2.client.list_objects_v2(Bucket=r2.bucket, Prefix='podcasts/', MaxKeys=200)
scripts/vast/deploy_from_offers.py:207:    missing = [k for k in required if not os.environ.get(k)]
scripts/vast/deploy_from_offers.py:208:    if missing:
scripts/vast/deploy_from_offers.py:209:        raise SystemExit(f"Missing required env vars (check .env): {', '.join(missing)}")
scripts/vast/deploy_from_offers.py:321:        missing = [oid for oid in offer_ids if oid not in offers]
scripts/vast/deploy_from_offers.py:322:        if missing:
scripts/vast/deploy_from_offers.py:324:                f"WARN: Some offer IDs were not found in Vast (skipping): {missing}. "
scripts/run_analytics_migration.py:111:# SQL for the view (run separately as it may need different syntax)
scripts/audit_completed_r2.py:49:    kind: str  # missing | s3 | http | path
scripts/audit_completed_r2.py:54:        return ParsedRef(raw="", bucket_hint=None, key=None, kind="missing")
scripts/audit_completed_r2.py:57:        return ParsedRef(raw="", bucket_hint=None, key=None, kind="missing")
scripts/audit_completed_r2.py:266:        "missing_r2_ref": 0,
scripts/audit_completed_r2.py:269:        "missing_in_r2": 0,
scripts/audit_completed_r2.py:273:        "missing_r2_ref": [],
scripts/audit_completed_r2.py:274:        "missing_in_r2": [],
scripts/audit_completed_r2.py:293:        if parsed.kind == "missing" or not parsed.key:
scripts/audit_completed_r2.py:378:                    if parsed_kind == "missing" or not parsed_key:
scripts/audit_completed_r2.py:379:                        stats["missing_r2_ref"] += 1
scripts/audit_completed_r2.py:380:                        if len(samples["missing_r2_ref"]) < 25:
scripts/audit_completed_r2.py:381:                            samples["missing_r2_ref"].append(youtube_id)


**Inspecting paginator usage**

I notice we've already implemented paginator usage in scripts like reset_for_production and prepare_benchmark. I think it would be beneficial to take a closer look at the prepare_benchmark script to see how it's being utilized. It might show how it's handling listings and deduplication, which could help me understand its effectiveness better. This inspection may reveal some useful insights for our current approach!


sed -n '1,160p' scripts/prepare_benchmark.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:
#!/usr/bin/env python3
"""
Prepare 100 videos for benchmark by:
1. Listing R2 files that exist (podcasts/*.webm)
2. Cross-referencing with Supabase
3. Setting them to PENDING status
"""

import os
import sys
import time
import random

sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))

try:
    from dotenv import load_dotenv
    load_dotenv()
except ImportError:
    pass

from supabase import create_client
from src.r2_client import R2Client

# Config
TARGET_COUNT = 100
SUPABASE_URL = os.environ.get('URL') or os.environ.get('SUPABASE_URL')
SUPABASE_KEY = os.environ.get('SUPABASE_ADMIN')

if not SUPABASE_URL or not SUPABASE_KEY:
    print("ERROR: Missing Supabase credentials")
    sys.exit(1)

print("=" * 70)
print("BENCHMARK PREPARATION - 100 Videos")
print("=" * 70)

# Initialize clients
sb = create_client(SUPABASE_URL, SUPABASE_KEY)
r2 = R2Client(bucket_type='source')  # 'test' bucket has the podcasts/

print(f"\n1. Listing R2 files from {r2.bucket}/podcasts/...")

# List all files in podcasts/ prefix
all_r2_files = []
continuation_token = None

while True:
    kwargs = {
        'Bucket': r2.bucket,
        'Prefix': 'podcasts/',
        'MaxKeys': 1000
    }
    if continuation_token:
        kwargs['ContinuationToken'] = continuation_token

    response = r2.client.list_objects_v2(**kwargs)
    files = response.get('Contents', [])
    all_r2_files.extend(files)

    if not response.get('IsTruncated'):
        break
    continuation_token = response.get('NextContinuationToken')

    if len(all_r2_files) >= 5000:  # Limit scan
        break

print(f"   Found {len(all_r2_files)} files in R2")

# Extract video IDs from filenames (podcasts/{video_id}.webm)
r2_video_ids = set()
for f in all_r2_files:
    key = f['Key']  # podcasts/---g_VL7ySo.webm
    if key.endswith('.webm'):
        video_id = key.replace('podcasts/', '').replace('.webm', '')
        r2_video_ids.add(video_id)

print(f"   Extracted {len(r2_video_ids)} unique video IDs")

print(f"\n2. Cross-referencing with Supabase...")

# Query Supabase for these videos
# Do in batches to avoid URL length limits
video_ids_list = list(r2_video_ids)[:500]  # Limit for query

result = sb.table('videos').select(
    'youtube_id, title, source_duration_min, language, status'
).in_('youtube_id', video_ids_list).execute()

videos_in_db = {v['youtube_id']: v for v in (result.data or [])}
print(f"   Found {len(videos_in_db)} videos in Supabase DB")

# Filter to videos that are in both R2 AND Supabase
validated = []
for vid in video_ids_list:
    if vid in videos_in_db:
        v = videos_in_db[vid]
        validated.append(v)
        if len(validated) >= TARGET_COUNT:
            break

print(f"   Validated: {len(validated)} videos (in R2 + Supabase)")

if len(validated) < TARGET_COUNT:
    print(f"\n   WARNING: Only {len(validated)} videos available")
    if len(validated) < 10:
        print("   ERROR: Not enough videos for benchmark")
        sys.exit(1)

# Show status distribution before reset
statuses = {}
for v in validated:
    s = v.get('status', 'unknown')
    statuses[s] = statuses.get(s, 0) + 1
print(f"   Current status: {statuses}")

print(f"\n3. Resetting {len(validated)} videos to PENDING status...")

# Reset to PENDING for benchmark
video_ids = [v['youtube_id'] for v in validated]
batch_size = 50

for i in range(0, len(video_ids), batch_size):
    batch = video_ids[i:i+batch_size]
    sb.table('videos').update({
        'status': 'PENDING',
        'claimed_by': None,
        'claimed_at': None,
        'lease_expires_at': None,
    }).in_('youtube_id', batch).execute()
    print(f"   Reset batch {i//batch_size + 1}/{(len(video_ids) + batch_size - 1)//batch_size}")

print(f"\n4. Summary of benchmark videos:")

# Stats
durations = [v.get('source_duration_min', 0) or 0 for v in validated]
total_duration = sum(durations)
avg_duration = total_duration / len(validated) if validated else 0

print(f"   Total videos: {len(validated)}")
print(f"   Total duration: {total_duration:.1f} min ({total_duration/60:.1f} hours)")
print(f"   Avg duration: {avg_duration:.1f} min")
if durations:
    print(f"   Duration range: {min(durations):.1f} - {max(durations):.1f} min")

# Language distribution
langs = {}
for v in validated:
    lang = v.get('language', 'unknown') or 'unknown'
    langs[lang] = langs.get(lang, 0) + 1
print(f"   Languages: {dict(sorted(langs.items(), key=lambda x: -x[1]))}")

print(f"\n5. Sample videos:")
for v in validated[:5]:
    title = (v.get('title') or 'No title')[:50]
    dur = v.get('source_duration_min', 0) or 0
    print(f"   • {v['youtube_id']} | {dur:.1f}min | {title}")

print(f"\n" + "=" * 70)
print(f"READY: {len(validated)} videos set to PENDING")


Summary: auto


**Inspecting object listing**

I see that this script can list up to 5000 objects and incorporates manual pagination. There’s also `scripts/reset_for_production.py` that uses a paginator, which might also help. I’m thinking it might be worth checking if there’s an existing script that lists all objects in a bucket and performs a diff. Let’s dive into the `scripts/reset_for_production.py` file to see how it handles listing objects. There's definitely something here to explore!


sed -n '150,260p' scripts/reset_for_production.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:
                break
            # Supabase can return UUID objects for `id` columns; normalize to str
            # to avoid "JSON could not be generated" errors.
            pks = [str(r.get(pk)) for r in rows if r.get(pk) is not None and str(r.get(pk))]
            if not pks:
                break
            if dry_run:
                deleted_total += len(pks)
                print(f"   [DRY RUN] Would delete {len(pks)} rows from {table_name} (total so far: {deleted_total})")
                continue
            client.table(table_name).delete().in_(pk, pks).execute()
            deleted_total += len(pks)
        return deleted_total

    for table in tables:
        try:
            if table in ('worker_heartbeats', 'worker_metrics'):
                deleted = delete_in_batches(table, 'worker_id')
            else:
                deleted = delete_in_batches(table, 'id')
            print(f"   ✅ Cleared {table}: {deleted} rows")
        except Exception as e:
            print(f"   ⚠  Could not clear {table}: {e}")

    return True

def delete_r2_folder(dry_run: bool = False):
    """Delete everything in R2 bucket 1-cleaned-data/cleaned/trail/."""
    from src.r2_client import R2Client

    print("\n🪣 R2 Bucket: 1-cleaned-data")
    print("   Prefix to delete: cleaned/trail/")

    try:
        # Initialize client for production bucket with the specific prefix
        client = R2Client(bucket_type='production')

        # List objects in the folder
        objects = []
        prefix = "cleaned/trail/"

        # Use boto3 paginator for listing all objects
        paginator = client.client.get_paginator('list_objects_v2')
        for page in paginator.paginate(Bucket='1-cleaned-data', Prefix=prefix):
            if 'Contents' in page:
                objects.extend([obj['Key'] for obj in page['Contents']])

        print(f"   Found {len(objects)} objects to delete")

        if not objects:
            print("   ℹ  No objects found in cleaned/trail/")
            return True

        if dry_run:
            print(f"\n🔍 [DRY RUN] Would delete {len(objects)} objects:")
            for obj in objects[:10]:
                print(f"      - {obj}")
            if len(objects) > 10:
                print(f"      ... and {len(objects) - 10} more")
            return True

        print(f"\n🗑  Deleting {len(objects)} objects from R2...")

        # Delete in batches of 1000 (S3/R2 limit)
        batch_size = 1000
        deleted = 0
        for i in range(0, len(objects), batch_size):
            batch = objects[i:i+batch_size]
            delete_objects = {'Objects': [{'Key': key} for key in batch]}
            response = client.client.delete_objects(
                Bucket='1-cleaned-data',
                Delete=delete_objects
            )
            deleted += len(response.get('Deleted', []))
            errors = response.get('Errors', [])
            if errors:
                print(f"   ⚠  {len(errors)} errors in batch")

        print(f"   ✅ Deleted {deleted} objects from cleaned/trail/")
        return True

    except Exception as e:
        print(f"   ❌ Error: {e}")
        return False

def main():
    parser = argparse.ArgumentParser(description='Reset Supabase and R2 for fresh production run')
    parser.add_argument('--dry-run', action='store_true', help='Show what would be done without making changes')
    parser.add_argument('--skip-r2', action='store_true', help='Skip deleting R2 cleaned/trail/ prefix')
    args = parser.parse_args()

    print("=" * 60)
    print("PRODUCTION RESET SCRIPT")
    print("=" * 60)

    if args.dry_run:
        print("\n⚠  DRY RUN MODE - No changes will be made\n")
    else:
        print("\n⚠  WARNING: This will reset ALL data!")
        confirm = input("Type 'RESET' to confirm: ")
        if confirm != 'RESET':
            print("Aborted.")
            return

    # 1. Reset videos
    print("\n" + "=" * 60)
    print("STEP 1: Reset Videos to PENDING")
    print("=" * 60)
    reset_videos_to_pending(args.dry_run)

    # 2. Clear analytics


**Evaluating data management options**

The user might set paths like 'cleaned/trail/' in their `1-cleaned-data` bucket, which may contain tar files. We could allow them to specify a prefix for each bucket. Scanning through 536k objects could yield keys, sizes, and ETags, leading to larger responses.

I'm considering three approaches:

1) Using boto3's paginator for listing objects.
2) Checking if Cloudflare's R2 inventory is enabled for easier metadata retrieval.
3) Evaluating Cloudflare's "Super Slurp" for one-time differences or recommending enabling inventory for repeated checks.


**Exploring Cloudflare R2 features**

I want to review the features of Cloudflare R2 as of 2024. They've announced "R2 Bucket Listing," but I'm unsure about the inventory feature. I could suggest enabling R2 bucket inventory for easier management if it's part of their plan, while being careful not to make assumptions about its availability.

They also referred to the challenge of searching metadata through the dashboard. Instead, I can propose using the API for listing and comparison since the UI can be slow. In Cloudflare R2, there’s no free "server-side join" between buckets; we need to list them for comparison.


**Calculating listing operations cost**

Listing 500k keys seems manageable, so let's break down costs. I have a source bucket with 536.16k objects and an output bucket of 502.24k. Each `list_objects_v2` call yields a maximum of 1000 keys, meaning I'll need around 1039 list operations (536 for the source and 503 for the output). At $0.36 per million class A operations, this ends up costing roughly $0.000374, which is less than 0.04 cents—pretty cheap!

Now, if I were to use a HEAD check for existence instead, it would involve 536k head operations, which are class B and cost $0.000019. While class B is cheaper than class A, this approach would take much longer due to the huge number of calls. Therefore, listing is definitely the better option!


**Creating a diff script for R2**

If listing is an issue due to memory or time constraints, I can suggest using commands like `aws s3 sync --dryrun` or `rclone check`. Though these methods rely on listing and may be slower, they can be more convenient.

Since I'm in a coding environment with an R2Client, I’ll create a new script called `scripts/diff_r2_video_ids.py`. This script will accept both source and cleaned bucket names, along with prefixes, file extensions, and handle video ID extraction and validation. I plan to use a paginator for listing and process the pages to write IDs to a temporary file, while considering memory for deduplication.


**Developing a diff script for R2 IDs**

Given the task of managing about 536k keys, we might encounter memory issues, but I think it could be manageable with around 40MB to even 200MB, which isn't too excessive. I’ll write a script that collects video IDs from two buckets—defaulting to 'test' and '1-cleaned-data'—and supports filtering by prefixes and extensions while validating IDs.

I might use AWS S3 APIs to list the objects, but since it's a Linux environment, I can also rely on Unix utilities like `sort`. If memory becomes a concern while computing differences, I could implement an external sort or use Python’s built-in functionality. Setting up progress reporting with `tqdm` seems useful, too.

Ultimately, the script will ensure efficient ID extraction, deduplication, and comparison, aiming to save the output in a clear format.


**Defining a bucket diff procedure**

I’m thinking about how to list and compare video IDs across two buckets. The output bucket likely has processed files while the source bucket might organize them under prefixes. I should mention treating prefixes as base names when listing.

To find differences, listing both buckets seems best: extract IDs, deduplicate, and compare. I might suggest using AWS CLI or boto3 for operations to keep costs low, emphasizing that listing isn't expensive, and it’s better than individually checking each file.

My final message could include step-by-step commands to efficiently extract and compare the IDs, ensuring clarity for the user.


Output:
Exit code: 0
Wall time: 0 seconds
Output:
# Maya3 Pipeline Environment Variables
# Copy this file to .env and fill in your values

# Hugging Face (required for PyAnnote models)
HF_TOKEN=hf_your_token_here

# Supabase (required for queue management)
URL=https://your-project.supabase.co
SUPABASE_ADMIN=your_service_role_key_here

# Cloudflare R2 (required for audio source and output)
ACCOUNT_ID=your_cloudflare_account_id
API_TOKEN=your_r2_api_token

# Optional: RunPod deployment
RUNPOD_KEY=your_runpod_api_key


**Evaluating environment setup**

I'm looking at the `.env.example` file, which uses `ACCOUNT_ID` and `API_TOKEN`, but I'm realizing there’s no mention of S3 keys. It's making me wonder if they rely on a Cloudflare API token for R2 S3 credentials. But the problem arises, because the `src/r2_client.py` file is expecting details like `R2_ENDPOINT_URL` or `S3_API`, along with an access key ID and secret. There's a mismatch here. I might need to check other scripts for how they handle `ACCOUNT_ID`.


rg -n "ACCOUNT_ID|API_TOKEN|R2_ENDPOINT_URL|R2_ACCESS_KEY_ID|R2_SECRET_ACCESS_KEY|S3_API" -S . | head -n 120


Output:
Exit code: 0
Wall time: 0 seconds
Output:
./.env.example:12:ACCOUNT_ID=your_cloudflare_account_id
./.env.example:13:API_TOKEN=your_r2_api_token
./README.md:84:| `ACCOUNT_ID` | Cloudflare account ID |
./README.md:85:| `API_TOKEN` | Cloudflare R2 API token |
./massive_process.py:105:        'R2_ENDPOINT_URL': 'Cloudflare R2 endpoint',
./massive_process.py:106:        'R2_ACCESS_KEY_ID': 'R2 access key',
./massive_process.py:107:        'R2_SECRET_ACCESS_KEY': 'R2 secret key',
./src/r2_client.py:56:        endpoint_url=os.environ.get('R2_ENDPOINT_URL') or os.environ.get('S3_API') or '',
./src/r2_client.py:57:        access_key_id=os.environ.get('R2_ACCESS_KEY_ID') or os.environ.get('AWS_ACCESS_KEY_ID') or '',
./src/r2_client.py:58:        secret_access_key=os.environ.get('R2_SECRET_ACCESS_KEY') or os.environ.get('AWS_SECRET_ACCESS_KEY') or '',
./src/r2_client.py:107:                raise RuntimeError("R2 not configured: missing R2_ENDPOINT_URL (or S3_API)")
./src/r2_client.py:109:                raise RuntimeError("R2 not configured: missing R2_ACCESS_KEY_ID (or AWS_ACCESS_KEY_ID)")
./src/r2_client.py:111:                raise RuntimeError("R2 not configured: missing R2_SECRET_ACCESS_KEY (or AWS_SECRET_ACCESS_KEY)")
./scripts/audit_completed_r2.py:91:    endpoint_url = os.environ.get("R2_ENDPOINT_URL") or os.environ.get("S3_API") or ""
./scripts/audit_completed_r2.py:92:    access_key_id = os.environ.get("R2_ACCESS_KEY_ID") or os.environ.get("AWS_ACCESS_KEY_ID") or ""
./scripts/audit_completed_r2.py:93:    secret_access_key = os.environ.get("R2_SECRET_ACCESS_KEY") or os.environ.get("AWS_SECRET_ACCESS_KEY") or ""
./scripts/audit_completed_r2.py:96:        raise RuntimeError("Missing R2_ENDPOINT_URL (or S3_API) in environment.")
./scripts/audit_completed_r2.py:98:        raise RuntimeError("Missing R2_ACCESS_KEY_ID (or AWS_ACCESS_KEY_ID) in environment.")
./scripts/audit_completed_r2.py:100:        raise RuntimeError("Missing R2_SECRET_ACCESS_KEY (or AWS_SECRET_ACCESS_KEY) in environment.")
./scripts/vast/deploy_from_offers.py:203:        "R2_ENDPOINT_URL",
./scripts/vast/deploy_from_offers.py:204:        "R2_ACCESS_KEY_ID",
./scripts/vast/deploy_from_offers.py:205:        "R2_SECRET_ACCESS_KEY",
./scripts/vast/deploy_from_offers.py:215:        "-e R2_ENDPOINT_URL": os.environ["R2_ENDPOINT_URL"],
./scripts/vast/deploy_from_offers.py:216:        "-e R2_ACCESS_KEY_ID": os.environ["R2_ACCESS_KEY_ID"],
./scripts/vast/deploy_from_offers.py:217:        "-e R2_SECRET_ACCESS_KEY": os.environ["R2_SECRET_ACCESS_KEY"],
./scripts/vast/search_and_deploy.py:249:        "R2_ENDPOINT_URL",
./scripts/vast/search_and_deploy.py:250:        "R2_ACCESS_KEY_ID",
./scripts/vast/search_and_deploy.py:251:        "R2_SECRET_ACCESS_KEY",
./scripts/vast/search_and_deploy.py:261:        "-e R2_ENDPOINT_URL": os.environ["R2_ENDPOINT_URL"],
./scripts/vast/search_and_deploy.py:262:        "-e R2_ACCESS_KEY_ID": os.environ["R2_ACCESS_KEY_ID"],
./scripts/vast/search_and_deploy.py:263:        "-e R2_SECRET_ACCESS_KEY": os.environ["R2_SECRET_ACCESS_KEY"],


**Reviewing R2 credentials setup**

I think the README likely provides guidance on setting up R2 credentials. I should check the relevant section to ensure I understand the instructions clearly. This is important to get it right, as proper setup is crucial for functionality. I want to make sure I’m thorough and capture all the necessary details. Let’s dive into that part of the README and see what it says!


Output:
Exit code: 0
Wall time: 0 seconds
Output:

Optional end-to-end test (process 1 Supabase-queued video on GPU 0 and validate TAR contents):

```bash
E2E=1 GPU_DEVICE=0 ./scripts/docker/smoke_test.sh
```

### Environment Variables

| Variable | Description |
|----------|-------------|
| `HF_TOKEN` | Hugging Face API token |
| `URL` | Supabase project URL |
| `SUPABASE_ADMIN` | Supabase service role key |
| `ACCOUNT_ID` | Cloudflare account ID |
| `API_TOKEN` | Cloudflare R2 API token |

## Local Development

```bash
# Create virtual environment
python -m venv .venv
source .venv/bin/activate

# Install dependencies
pip install -r requirements.txt

# Run with Supabase queue
python massive_process.py --supabase-queue --output ./output
```

## Output Format

Each processed video produces a `.tar` containing:
- `{video_id}_segments.json` - Speaker-segmented data
- Metadata and quality statistics

## Scaling

Workers can be deployed across multiple machines. Each worker:
1. Claims videos from Supabase queue
2. Processes independently
3. Reports completion/failure back to Supabase

## Files

```
├── massive_process.py    # Main entry point
├── pipeline.py           # Core pipeline logic
├── src/
│   ├── config.py         # Configuration


Output:
Exit code: 0
Wall time: 0 seconds
Output:
#!/usr/bin/env python3
"""
MASSIVE COMPUTE - 8x H100 Parallel Processing

Designed for maximum throughput on multi-GPU clusters.
Each GPU runs an independent pipeline worker processing videos in parallel.

Usage:
    # Process videos from file (one URL per line)
    python massive_process.py --input videos.txt

    # Process from YouTube playlist or channel
    python massive_process.py --playlist "PLAYLIST_URL"

    # Benchmark mode with test videos
    python massive_process.py --benchmark --num-videos 16

    # Custom GPU selection
    python massive_process.py --input videos.txt --gpus 0,1,2,3

Architecture:
    Main Process
        │
        ├── GPU Worker 0 (CUDA_VISIBLE_DEVICES=0)
        │     └── pipeline.py (full model stack)
        │
        ├── GPU Worker 1 (CUDA_VISIBLE_DEVICES=1)
        │     └── pipeline.py (full model stack)
        │
        ... (8 workers total)
        │
        └── GPU Worker 7 (CUDA_VISIBLE_DEVICES=7)
              └── pipeline.py (full model stack)

Result: 8 videos processed in parallel → 8x throughput
"""

# CRITICAL: Suppress NNPACK warnings BEFORE any imports
# These warnings are harmless (NNPACK is CPU-only, we use GPU) but spam logs
import os
import warnings
os.environ['NNPACK_DISABLE'] = '1'  # Disable NNPACK entirely
os.environ['TORCH_CPP_LOG_LEVEL'] = 'ERROR'  # Suppress C++ warnings
warnings.filterwarnings('ignore', message='.*NNPACK.*')
warnings.filterwarnings('ignore', category=FutureWarning, module='pynvml')
warnings.filterwarnings('ignore', category=UserWarning, module='torchaudio')

import sys
import time
import json
import logging
import argparse
import signal
import traceback
from pathlib import Path
from datetime import datetime
from multiprocessing import Process, Queue, Value, Manager
from typing import List, Dict, Optional, Tuple
from dataclasses import dataclass, asdict, field

# Supabase client for distributed queue
try:
    from src.supabase_client import SupabaseClient, STATUS_COMPLETED, STATUS_FAILED
    SUPABASE_AVAILABLE = True
except ImportError:
    SUPABASE_AVAILABLE = False

# Optional: load .env for HF/R2/Supabase credentials
try:
    from dotenv import load_dotenv  # type: ignore
except Exception:
    load_dotenv = None

# Clean logging utilities
from src.logger import silence_verbose_loggers

# Analytics reporter for distributed monitoring
from src.analytics import create_reporter, AnalyticsReporter

# Setup logging
logging.basicConfig(
    level=logging.INFO,
    format='%(asctime)s | %(name)-12s | %(levelname)-8s | %(message)s',
    datefmt='%H:%M:%S'
)
logger = logging.getLogger("MassiveCompute")

# Global stop signal for graceful shutdown
_GLOBAL_STOP_SIGNAL = None

def validate_environment() -> bool:
    """
    Validate all required environment variables before starting workers.
    Fails fast on startup instead of runtime errors.
    """
    required = {
        'HF_TOKEN': 'Hugging Face token for model downloads',
        'URL': 'Supabase project URL',
        'SUPABASE_ADMIN': 'Supabase service role key',
    }

    r2_vars = {
        'R2_ENDPOINT_URL': 'Cloudflare R2 endpoint',
        'R2_ACCESS_KEY_ID': 'R2 access key',
        'R2_SECRET_ACCESS_KEY': 'R2 secret key',
    }

    missing = []
    for var, desc in required.items():
        if not os.environ.get(var):
            missing.append(f"  {var}: {desc}")

    # R2 vars only required if --r2-upload is used
    # Will be validated later if needed

    if missing:
        logger.error("Missing required environment variables:")
        for m in missing:
            logger.error(m)
        return False

    return True

def setup_signal_handlers(stop_signal):
    """
    Setup graceful shutdown handlers for SIGTERM and SIGINT.

    This ensures workers complete their current video before exiting,
    preventing lost work on container shutdown.
    """
    global _GLOBAL_STOP_SIGNAL
    _GLOBAL_STOP_SIGNAL = stop_signal

    def graceful_shutdown(signum, frame):
        sig_name = 'SIGTERM' if signum == signal.SIGTERM else 'SIGINT'
        logger.warning(f"Received {sig_name}, initiating graceful shutdown...")
        logger.warning("Workers will complete current videos then exit")

        if _GLOBAL_STOP_SIGNAL is not None:
            _GLOBAL_STOP_SIGNAL.value = True

    signal.signal(signal.SIGTERM, graceful_shutdown)
    signal.signal(signal.SIGINT, graceful_shutdown)

@dataclass
class VideoResult:
    """Result from processing a single video."""
    video_url: str
    video_id: str = ""
    success: bool = False
    error: str = ""
    duration_sec: float = 0.0
    process_time_sec: float = 0.0
    pipeline_time_sec: float = 0.0
    rtf: float = 0.0
    num_speakers: int = 0
    usable_percentage: float = 0.0
    output_path: str = ""
    stage_timings: Dict[str, float] = field(default_factory=dict)
    source_type: str = ""
    r2_upload_key: str = ""
    gpu_id: int = -1
    worker_id: int = -1

@dataclass
class BenchmarkStats:
    """Aggregate statistics from benchmark run."""
    total_videos: int = 0
    successful: int = 0
    failed: int = 0
    total_audio_duration_sec: float = 0.0
    total_audio_hours: float = 0.0
    total_process_time_sec: float = 0.0
    throughput_videos_per_min: float = 0.0
    throughput_hours_audio_per_hour: float = 0.0  # Hours of audio processed per hour of wall time


**Determining minimal updates**

I think we’re okay with the requirements already in place. I might consider updating the README or documentation, but I’m not sure it’s necessary. I want to identify a minimal patch, such as adding `scripts/diff_r2_buckets.py` for usage. We could also update `src/r2_client.py` to include `iter_keys` with pagination, but maybe it’s not needed since the script could build boto3 clients directly. Using `R2Client` seems simpler, but we should accept bucket names explicitly for clarity.