Output:
Exit code: 0
Wall time: 0 seconds
Total output lines: 260
Output:
#!/usr/bin/env python3
"""
Model management - Hot inference with adaptive loading.

Key improvements:
- community-1 FIRST (better per pyannote docs), 3.1 as fallback
- Lazy loading with compute-aware batching
- Better GPU memory management

=== v7.0 OPTIMIZATION: cuDNN Benchmark + TF32 ===
Enables CUDA optimizations for faster inference:
- cudnn.benchmark=True: Auto-tune convolution algorithms
- TF32 precision: 3x faster matmul on Ampere+ GPUs
Expected gain: 5-10% on diarization (conv-heavy model)
"""

# === COMPATIBILITY FIX: torchaudio 2.9+ removed list_audio_backends() ===
# speechbrain calls this old API; we monkey-patch before any imports
import os
import torchaudio
if not hasattr(torchaudio, 'list_audio_backends'):
    torchaudio.list_audio_backends = lambda: ['soundfile', 'sox']
# === END COMPATIBILITY FIX ===

import time
import logging
import gc
import torch

# === COMPATIBILITY FIX: speechbrain expects hf_hub_download(use_auth_token=...) ===
# Newer huggingface_hub versions renamed this to `token`. Rather than pinning a
# fragile dependency set, we shim the old kwarg to the new one.
try:
    import inspect
    import huggingface_hub  # type: ignore

    if hasattr(huggingface_hub, "hf_hub_download"):
        _orig_hf_hub_download = huggingface_hub.hf_hub_download
        _hf_sig = inspect.signature(_orig_hf_hub_download)
        if "use_auth_token" not in _hf_sig.parameters:
            def _patched_hf_hub_download(*args, use_auth_token=None, token=None, **kwargs):
                if token is None and use_auth_token is not None:
                    token = use_auth_token
                return _orig_hf_hub_download(*args, token=token, **kwargs)

            huggingface_hub.hf_hub_download = _patched_hf_hub_download  # type: ignore
except Exception:
    # Best-effort shim; if anything goes wrong we fall back to stock behavior.
    pass

# === v7.0 OPTIMIZATION: Enable CUDA optimizations ===
# These flags enable hardware-specific optimizations:
# - cudnn.benchmark: Auto-select fastest conv algorithm for input sizes
# - TF32: Use TensorFloat-32 for faster matrix ops (Ampere+ GPUs)
# Must be set BEFORE model loading for full effect
if torch.cuda.is_available():
    torch.backends.cudnn.benchmark = True  # Auto-tune convolutions
    torch.backends.cuda.matmul.allow_tf32 = True  # Fast matmul
    torch.backends.cudnn.allow_tf32 = True  # Fast convolutions

# === CRITICAL: Patch torch.load BEFORE any other imports ===
_original_torch_load = torch.load
def _patched_torch_load(*args, **kwargs):
    kwargs['weights_only'] = False
    return _original_torch_load(*args, **kwargs)
torch.load = _patched_torch_load

# Patch PyTorch Lightning too
try:
    import pytorch_lightning.core.saving as pl_saving
    _orig_pl_load = pl_saving.pl_load
    def _patched_pl_load(path_or_url, map_location=None, **kwargs):
        return _original_torch_load(path_or_url, map_location=map_location, weights_only=False)
    pl_saving.pl_load = _patched_pl_load
except ImportError:
    pass

try:
    import lightning_fabric.utilities.cloud_io as cloud_io
    _orig_fabric_load = cloud_io._load
    def _patched_fabric_load(path_or_url, map_location=None):
        return _original_torch_load(path_or_url, map_location=map_location, weights_only=False)
    cloud_io._load = _patched_fabric_load
except (ImportError, AttributeError):
    pass

logger = logging.getLogger("FastPipelineV6.Models")

class ModelManager:
    """
    Singleton model manager - loads all models once, keeps them hot.

    Model priority (per user request):
    1. community-1 (better performance per pyannote docs)
    2. speaker-diarization-3.1 (fallback)

    Key optimization: Load once, inference many times.
    """
    _instance = None

    def __new__(cls):
        if cls._instance is None:
            cls._instance = super().__new__(cls)
            cls._instance._initialized = False
        return cls._instance

    def __init__(self):
        if self._initialized:
            return
        self._initialized = True
        self.device = None
        self.silero_vad = None
        self.silero_utils = None
        self.diarization_pipeline = None
        self.segmentation_model = None
        self.embedding_model = None
        self.panns_model = None  # v6.7: PANNs CNN14 for music detection
        self._loaded = False
        self._panns_loaded = False  # Separate flag for optional PANNs
        self._model_name = None  # Track which model loaded

    def get_device(self) -> torch.device:
        """Get CUDA device or fallback to CPU."""
        if self.device is None:
            self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
        return self.device

    def load_all(self, config):
        """
        Load all models once into GPU memory.

        Order: VAD -> Diarization (community-1 first) -> Segmentation -> Embedding
        """
        if self._loaded:
            logger.info(f"✅ Models alr…585 chars truncated…nnote/segmentation-3.0 (if needed)\n"
            )
        # Update config with token for later use
        config.hf_token = hf_token

        logger.info("=" * 70)
        logger.info("🚀 LOADING ALL MODELS INTO GPU")
        logger.info("=" * 70)
        start = time.time()

        device = self.get_device()
        logger.info(f"Device: {device}")

        # Log GPU memory before
        if torch.cuda.is_available():
            free = torch.cuda.get_device_properties(0).total_memory - torch.cuda.memory_reserved(0)
            logger.info(f"GPU Memory before: {free / 1e9:.2f}GB free")

        # 1. Silero VAD (CPU - fast)
        logger.debug("Loading Silero VAD...")
        model, utils = torch.hub.load(
            repo_or_dir='snakers4/silero-vad',
            model='silero_vad',
            force_reload=False,
            onnx=False,
            verbose=False  # Suppress "Using cache found" messages
        )
        self.silero_vad = model
        self.silero_utils = utils
        logger.debug("Silero VAD loaded")

        # 2. PyAnnote Diarization - COMMUNITY-1 FIRST (per user request)
        from pyannote.audio import Pipeline
        from pyannote.audio.pipelines import SpeakerDiarization
        import inspect

        logger.info("Loading PyAnnote Diarization...")

        # Patch SpeakerDiarization to ignore unsupported params (like 'plda')
        _original_sd_init = SpeakerDiarization.__init__
        _valid_params = set(inspect.signature(_original_sd_init).parameters.keys())

        def _patched_sd_init(self, *args, **kwargs):
            filtered_kwargs = {k: v for k, v in kwargs.items() if k in _valid_params}
            removed = set(kwargs.keys()) - set(filtered_kwargs.keys())
            if removed:
                logger.debug(f"Filtered out unsupported params: {removed}")
            return _original_sd_init(self, *args, **filtered_kwargs)

        SpeakerDiarization.__init__ = _patched_sd_init

        # === PRIORITY: community-1 FIRST (better performance per pyannote docs) ===
        # Repo: https://huggingface.co/pyannote/speaker-diarization-community-1
        # Fallback: speaker-diarization-3.1 if community-1 fails
        models_to_try = [
            ("pyannote/speaker-diarization-community-1", "community-1"),  # FIRST CHOICE
            ("pyannote/speaker-diarization-3.1", "speaker-diarization-3.1"),  # FALLBACK
        ]

        for model_name, short_name in models_to_try:
            try:
                logger.info(f"Trying {short_name}...")
                # Try both 'token' (pyannote 3.3+) and 'use_auth_token' (older versions)
                try:
                    self.diarization_pipeline = Pipeline.from_pretrained(
                        model_name,
                        token=config.hf_token
                    )
                except TypeError:
                    # Fallback for older pyannote.audio versions
                    self.diarization_pipeline = Pipeline.from_pretrained(
                        model_name,
                        use_auth_token=config.hf_token
                    )
                self._model_name = short_name
                logger.info(f"✅ Loaded {short_name} (preferred model)")
                break
            except Exception as e:
                logger.warning(f"Failed to load {short_name}: {e}")
                continue

        if self.diarization_pipeline is None:
            raise RuntimeError("Failed to load any diarization pipeline")

        self.diarization_pipeline.to(device)
        logger.info("✅ PyAnnote Diarization loaded on GPU")

        # 3. Segmentation model - DISABLED (not used in current pipeline)
        # NOTE: segmentation-3.0 was for frame-level OSD but we use community-1 diarization instead
        # The community-1 model handles both diarization AND overlap detection in one pass
        # Keeping this disabled to avoid noisy "gated model" warnings
        self.segmentation_model = None

        # 4. ECAPA-TDNN for embeddings
        logger.info("Loading ECAPA-TDNN...")
        from speechbrain.inference.speaker import EncoderClassifier
        self.embedding_model = EncoderClassifier.from_hparams(
            source="speechbrain/spkrec-ecapa-voxceleb",
            savedir="/ephemeral/models/ecapa",
            run_opts={"device": str(device)}
        )
        logger.info("✅ ECAPA-TDNN loaded")

        self._loaded = True
        elapsed = time.time() - start

        # Log GPU memory after
        if torch.cuda.is_available():
            allocated = torch.cuda.memory_allocated() / 1e9
            reserved = torch.cuda.memory_reserved() / 1e9
            logger.info(f"GPU Memory: {allocated:.2f}GB allocated, {reserved:.2f}GB reserved")

        logger.info(f"✅ All models loaded in {elapsed:.1f}s (STAYING HOT)")
        logger.info("=" * 70)


Output:
Exit code: 0
Wall time: 0 seconds
Total output lines: 331
Output:
#!/usr/bin/env python3
"""
R2 Client Utilities for High-Throughput Pipeline Migration

Manages Cloudflare R2 buckets for:
- Production outputs (1-cleaned-data)
- Test outputs during migration (1-cleaned-data-test)
- Baseline snapshots for comparison (1-cleaned-data-baseline)

Usage:
    from src.r2_client import R2Client

    client = R2Client()
    client.upload_json("path/to/file.json", "video_id/metadata.json")
    client.download_json("video_id/metadata.json", "local/path.json")
"""

import os
import json
import logging
from pathlib import Path
from typing import Dict, Any, Optional, List
from dataclasses import dataclass

import boto3
from botocore.config import Config as BotoConfig

logger = logging.getLogger("R2Client")

@dataclass
class R2Config:
    """R2 bucket configuration."""
    endpoint_url: str
    access_key_id: str
    secret_access_key: str
    bucket: str
    prefix: str = ""

# Bucket configurations
BUCKETS = {
    'production': '1-cleaned-data',
    'test': '1-cleaned-data-test',
    'baseline': '1-cleaned-data-baseline',
    # Source/original videos (DO NOT DELETE). This bucket is expected to contain
    # the original YouTube files re-uploaded to R2 to avoid rate limits.
    'source': 'test',
}

def get_r2_config_from_env() -> R2Config:
    """Load R2 configuration from environment variables."""
    return R2Config(
        # Support a few common aliases used in various deploy templates.
        endpoint_url=os.environ.get('R2_ENDPOINT_URL') or os.environ.get('S3_API') or '',
        access_key_id=os.environ.get('R2_ACCESS_KEY_ID') or os.environ.get('AWS_ACCESS_KEY_ID') or '',
        secret_access_key=os.environ.get('R2_SECRET_ACCESS_KEY') or os.environ.get('AWS_SECRET_ACCESS_KEY') or '',
        bucket=os.environ.get('R2_BUCKET', 'test'),
        prefix=os.environ.get('R2_PREFIX', ''),
    )

class R2Client:
    """
    Cloudflare R2 client for pipeline data management.

    Supports multiple bucket targets for A/B testing during migration.
    """

    def __init__(
        self,
        config: Optional[R2Config] = None,
        bucket_type: str = 'test'
    ):
        """
        Initialize R2 client.

        Args:
            config: R2 configuration (loads from env if None)
            bucket_type: One of 'production', 'test', 'baseline'
        """
        self.config = config or get_r2_config_from_env()

        # Override bucket based on type
        if bucket_type in BUCKETS:
            self.bucket = BUCKETS[bucket_type]
        else:
            self.bucket = self.config.bucket

        # IMPORTANT: The output prefix (R2_PREFIX) should not accidentally apply
        # to the *source* bucket. Vast/RunPod templates often set R2_PREFIX for
        # uploads; for source reads we want R2_SOURCE_PREFIX (or empty).
        if bucket_type == 'source' and config is None:
            self.prefix = os.environ.get('R2_SOURCE_PREFIX', '') or ''
        else:
            self.prefix = self.config.prefix

        # Initialize boto3 client
        self._client = None

    @property
    def client(self):
        """Lazy-initialize boto3 client."""
        if self._client is None:
            if not self.config.endpoint_url:
                raise RuntimeError("R2 not configured: missing R2_ENDPOINT_URL (or S3_API)")
            if not self.config.access_key_id:
                raise RuntimeError("R2 not configured: missing R2_ACCESS_KEY_ID (or AWS_ACCESS_KEY_ID)")
            if not self.config.secret_access_key:
                raise RuntimeError("R2 not configured: missing R2_SECRET_ACCESS_KEY (or AWS_SECRET_ACCESS_KEY)")
            self._client = boto3.client(
                's3',
                endpoint_url=self.config.endpoint_url,
                aws_access_key_id=self.config.access_key_id,
                aws_secret_access_key=self.config.secret_access_key,
                config=BotoConfig(
                    signature_version='s3v4',
                    retries={'max_attempts': 3, 'mode': 'adaptive'}
                )
            )
        return self._client

    def full_key(self, key: str) -> str:
        """Public helper: return fully-qualified key (including configured prefix)."""
        return self._full_key(key)

    def uri(self, key: str) -> str:
        """Return an s3:// URI for a key (including configured prefix)."""
        return f"s3://{self.bucket}/{self._full_key(key)}"

    def _full_key(self, key: str) -> str:
        """Get full S3 key with prefix."""
        if self.prefix:
            return f"{self.prefix}/{key}"
        return key

    def upload_file(self, local_path: str, remote_key: str) -> bool:
        """
        Upload a file to R2.

        Args:
            local_path: Path to local file
            remote_key: Remote key (path in bucket)

        Returns:
            True if successful
        """
        try:
            full_key = self._full_key(remote_key)
            self.client.upload_file(local_path, self.bucket, full_key)
       …1235 chars truncated…                Key=full_key,
                Body=body,
                ContentType='application/json'
            )
            logger.info(f"Uploaded JSON -> s3://{self.bucket}/{full_key}")
            return True
        except Exception as e:
            logger.error(f"JSON upload failed: {e}")
            return False

    def download_json(self, remote_key: str) -> Optional[Dict[str, Any]]:
        """Download JSON data from R2."""
        try:
            full_key = self._full_key(remote_key)
            response = self.client.get_object(Bucket=self.bucket, Key=full_key)
            data = json.loads(response['Body'].read().decode('utf-8'))
            logger.info(f"Downloaded JSON <- s3://{self.bucket}/{full_key}")
            return data
        except Exception as e:
            logger.error(f"JSON download failed: {e}")
            return None

    def head_object(self, remote_key: str) -> Optional[Dict[str, Any]]:
        """Return S3 head_object response for a key, or None if missing/unreadable."""
        try:
            full_key = self._full_key(remote_key)
            return self.client.head_object(Bucket=self.bucket, Key=full_key)
        except Exception:
            return None

    def generate_presigned_get_url(self, remote_key: str, expires_in_sec: int = 3600) -> str:
        """
        Generate a presigned GET URL for an object.

        Note: This is a Class B request pattern (no bucket listing).
        """
        full_key = self._full_key(remote_key)
        return self.client.generate_presigned_url(
            'get_object',
            Params={'Bucket': self.bucket, 'Key': full_key},
            ExpiresIn=expires_in_sec,
        )

    def list_keys(self, prefix: str = "", limit: int = 1000) -> List[str]:
        """
        List keys in bucket with optional prefix.

        Args:
            prefix: Additional prefix to filter by
            limit: Maximum keys to return

        Returns:
            List of keys
        """
        try:
            full_prefix = self._full_key(prefix) if prefix else self.prefix
            response = self.client.list_objects_v2(
                Bucket=self.bucket,
                Prefix=full_prefix,
                MaxKeys=limit
            )

            keys = []
            for obj in response.get('Contents', []):
                key = obj['Key']
                # Remove prefix for cleaner output
                if self.prefix and key.startswith(self.prefix):
                    key = key[len(self.prefix):].lstrip('/')
                keys.append(key)

            return keys
        except Exception as e:
            logger.error(f"List failed: {e}")
            return []

    def exists(self, remote_key: str) -> bool:
        """Check if a key exists in the bucket."""
        try:
            full_key = self._full_key(remote_key)
            self.client.head_object(Bucket=self.bucket, Key=full_key)
            return True
        except:
            return False

    def delete(self, remote_key: str) -> bool:
        """Delete a key from the bucket."""
        try:
            # Safety guard: user explicitly forbids deletions from any bucket other
            # than 1-cleaned-data. Keep this invariant even if callers misuse API.
            if self.bucket != '1-cleaned-data':
                raise PermissionError(
                    f"Refusing to delete from bucket '{self.bucket}'. "
                    f"Only '1-cleaned-data' deletions are permitted."
                )
            full_key = self._full_key(remote_key)
            self.client.delete_object(Bucket=self.bucket, Key=full_key)
            logger.info(f"Deleted s3://{self.bucket}/{full_key}")
            return True
        except Exception as e:
            logger.error(f"Delete failed: {e}")
            return False

def copy_to_baseline(video_ids: List[str], source_type: str = 'production') -> int:
    """
    Copy video outputs from source bucket to baseline bucket.

    Args:
        video_ids: List of video IDs to copy
        source_type: Source bucket type ('production' or 'test')

    Returns:
        Number of successfully copied videos
    """
    source = R2Client(bucket_type=source_type)
    baseline = R2Client(bucket_type='baseline')

    copied = 0
    for video_id in video_ids:
        # Copy metadata.json
        metadata_key = f"{video_id}/metadata.json"
        data = source.download_json(metadata_key)
        if data and baseline.upload_json(data, metadata_key):
            copied += 1
            logger.info(f"Copied {video_id} to baseline")
        else:
            logger.warning(f"Failed to copy {video_id}")

    return copied

if __name__ == "__main__":
    # Quick test
    import sys

    print("R2 Client Test")
    print("-" * 40)

    client = R2Client(bucket_type='test')

    # List some keys
    print(f"Bucket: {client.bucket}")
    print(f"Prefix: {client.prefix}")

    keys = client.list_keys(limit=5)
    print(f"Sample keys: {keys[:5]}")


Output:
Exit code: 0
Wall time: 0 seconds
Output:
"""
Analytics Reporter for Maya3 Pipeline

Lightweight, non-blocking analytics that reports worker status to Supabase.

Design principles:
- Never blocks main processing thread (background queue + thread)
- Batches writes to reduce Supabase calls (~2-3 per video, not 10+)
- Uses upserts for heartbeats (single row per worker)
- Gracefully handles failures (analytics never crashes worker)
- Local counters remain accurate even if Supabase is down
"""

import threading
import queue
import time
import socket
import hashlib
import os
import traceback
from typing import Optional, Dict, Any
from dataclasses import dataclass, field
from datetime import datetime

@dataclass
class AnalyticsEvent:
    """Event to be sent to Supabase."""
    event_type: str  # 'heartbeat', 'done', 'fail'
    data: Dict[str, Any]
    timestamp: datetime = field(default_factory=datetime.utcnow)

def get_machine_id() -> str:
    """
    Generate a unique, readable machine ID.
    Format: hostname-xxxx (where xxxx is a short hash)
    """
    # Prefer cloud/provider instance identifiers when available so analytics can
    # be mapped back to the actual machine/contract in the provider UI.
    vast_id = os.environ.get("CONTAINER_ID")  # Vast.ai predefined env var
    if vast_id:
        return f"vast-{vast_id}"

    runpod_id = os.environ.get("RUNPOD_POD_ID")
    if runpod_id:
        return f"runpod-{runpod_id}"

    hostname = socket.gethostname()[:20]  # Truncate long hostnames
    # Create short hash from hostname + MAC-like identifier
    unique_str = f"{hostname}-{socket.getfqdn()}"
    short_hash = hashlib.md5(unique_str.encode()).hexdigest()[:4]
    return f"{hostname}-{short_hash}"

class AnalyticsReporter:
    """
    Non-blocking analytics reporter with background flushing.

    Usage:
        reporter = AnalyticsReporter(gpu_id=0, supabase_url=..., supabase_key=...)
        reporter.session_start()

        # For each video:
        reporter.pick(video_id)
        reporter.stage_update(video_id, 3, "Diarization")  # Optional stage updates
        reporter.done(video_id, total_time, audio_min, speakers, usable_pct)
        # or
        reporter.fail(video_id, "Download", "NetworkError", "Connection timeout", traceback_str)

        # On shutdown:
        reporter.shutdown()
    """

    def __init__(
        self,
        gpu_id: int,
        supabase_url: str,
        supabase_key: str,
        machine_id: Optional[str] = None,
        flush_interval: float = 5.0,
        batch_size: int = 10
    ):
        self.gpu_id = gpu_id
        self.machine_id = machine_id or get_machine_id()
        self.worker_id = f"{self.machine_id}_gpu{gpu_id}"

        # Background queue for non-blocking writes
        self._queue: queue.Queue = queue.Queue()
        self._stop_event = threading.Event()

        # Configuration
        self._flush_interval = flush_interval
        self._batch_size = batch_size

        # Local counters (always accurate, even if Supabase fails)
        self.session_videos_done = 0
        self.session_videos_failed = 0
        self.session_audio_minutes = 0.0
        self.session_usable_minutes = 0.0
        self.session_start_time = datetime.utcnow()

        # Supabase client (lazy init in background thread)
        self._supabase_url = supabase_url
        self._supabase_key = supabase_key
        self._client = None
        self._client_lock = threading.Lock()

        # Start background flusher
        self._flush_thread = threading.Thread(target=self._flush_loop, daemon=True)
        self._flush_thread.start()

    def _get_client(self):
        """Lazy init Supabase client in background thread (thread-safe)."""
        if self._client is None:
            with self._client_lock:
                if self._client is None:  # Double-check after acquiring lock
                    try:
                        from supabase import create_client
                        self._client = create_client(self._supabase_url, self._supabase_key)
                    except Exception:
                        pass  # Silently fail - analytics should never crash worker
        return self._client

    def _flush_loop(self):
        """Background thread that flushes events to Supabase."""
        events_buffer = []
        last_flush = time.time()

        while not self._stop_event.is_set():
            try:
                # Collect events (non-blocking with timeout)
                try:
                    event = self._queue.get(timeout=1.0)
                    events_buffer.append(event)
                except queue.Empty:
                    pass

                # Flush conditions: batch_size+ events OR flush_interval passed
                should_flush = (
                    len(events_buffer) >= self._batch_size or
                    (events_buffer and time.time() - last_flush >= self._flush_interval)
                )

                if should_flush and events_buffer:
                    self._flush_events(events_buffer)
                    events_buffer = []
                    last_flush = time.time()

            except Exception:
                # Never crash the flush thread
                pass

        # Final flush on shutdown
        if events_buffer:
            self._flush_events(events_buffer)

    def _flush_events(self, events: list):
        """Batch write events to Supabase."""
        client = self._get_client()
        if not client:
            return

        try:
            # Separate by type
            heartbeats = [e for e in events if e.event_type == 'heartbeat']
            done_events = [e for e in events if e.event_type == 'done']
            fail_events = [e for e in events if e.event_type == 'fail']

            # Upsert latest heartbeat (only 1 per worker, use most recent)
            if heartbeats:
                latest = heartbeats[-1]
                client.table('worker_heartbeats').upsert(
                    latest.data,
                    on_conflict='worker_id'
                ).execute()

            # Batch insert done events
            if done_events:
                rows = [e.data for e in done_events]
                client.table('processing_events').insert(rows).execute()

            # Batch insert fail events + error_logs
            if fail_events:
                event_rows = [e.data for e in fail_events]
                client.table('processing_events').insert(event_rows).execute()

                # Also log to error_logs table with full details
                error_rows = [{
                    'worker_id': e.data['worker_id'],
                    'video_id': e.data['video_id'],
                    'stage': e.data.get('stage_name'),
                    'error_type': e.data.get('error_type'),
                    'error_message': e.data.get('error_message'),
                    'stack_trace': e.data.get('stack_trace')
                } for e in fail_events]
                client.table('error_logs').insert(error_rows).execute()

        except Exception:
            # Silently fail - analytics should never crash worker
            pass

    # =========================================================================
    # Public API (all non-blocking)
    # =========================================================================

    def session_start(self):
        """Register worker on startup."""
        self.session_start_time = datetime.utcnow()
        self._queue.put(AnalyticsEvent('heartbeat', {
            'worker_id': self.worker_id,
            'machine_id': self.machine_id,
            'gpu_id': self.gpu_id,
            'status': 'idle',
            'current_video_id': None,
            'current_stage': None,
            'current_stage_name': None,
            'last_heartbeat': datetime.utcnow().isoformat(),
            'session_start': self.session_start_time.isoformat(),
            'session_videos_done': 0,
            'session_videos_failed': 0,
            'session_audio_minutes': 0,
            'session_usable_minutes': 0
        }))

    def pick(self, video_id: str):
        """Worker claimed a video."""
        self._queue.put(AnalyticsEvent('heartbeat', {
            'worker_id': self.worker_id,
            'machine_id': self.machine_id,
            'gpu_id': self.gpu_id,
            'status': 'processing',
            'current_video_id': video_id,
            'current_stage': 1,
            'current_stage_name': 'Download',
            'last_heartbeat': datetime.utcnow().isoformat(),
            'session_start': self.session_start_time.isoformat(),
            'session_videos_done': self.session_videos_done,
            'session_videos_failed': self.session_videos_failed,
            'session_audio_minutes': self.session_audio_minutes,


Output:
Exit code: 0
Wall time: 0 seconds
Output:
#!/usr/bin/env python3
"""
Supabase Client for Distributed Video Processing Queue

Provides:
- Atomic video claiming with lease-based locking
- Status updates after processing
- Bulk upsert for CSV imports
- Queue statistics

Usage:
    from src.supabase_client import SupabaseClient

    client = SupabaseClient()
    video = client.claim_next_video("worker-1")
    if video:
        # Process video...
        client.update_status(video['youtube_id'], 'COMPLETED', results)
"""

import os
import json
import logging
import time
import random
import socket
from datetime import datetime, timezone
from typing import Dict, Any, Optional, List
from urllib.request import Request, urlopen
from urllib.error import HTTPError, URLError
from urllib.parse import urlencode

logger = logging.getLogger("SupabaseClient")

# Status constants
STATUS_PENDING = 'PENDING'
STATUS_CLAIMED = 'CLAIMED'
STATUS_COMPLETED = 'COMPLETED'
STATUS_FAILED = 'FAILED'
STATUS_SKIPPED = 'SKIPPED'

class SupabaseClient:
    """
    Supabase REST API client for video processing queue.

    Uses atomic claims via RPC function for distributed workers.
    """

    _instance = None

    def __new__(cls, *args, **kwargs):
        """Singleton pattern - reuse connection."""
        if cls._instance is None:
            cls._instance = super().__new__(cls)
            cls._instance._initialized = False
        return cls._instance

    def __init__(self, url: str = None, key: str = None):
        """
        Initialize Supabase client.

        Args:
            url: Supabase project URL (loads from URL env var if None)
            key: Service role key (loads from SUPABASE_ADMIN env var if None)
        """
        if self._initialized:
            return

        self.url = url or os.environ.get('URL', '')
        self.key = key or os.environ.get('SUPABASE_ADMIN', '')

        if not self.url or not self.key:
            logger.warning("Supabase credentials not configured. Set URL and SUPABASE_ADMIN env vars.")

        self.rest_url = f"{self.url}/rest/v1"
        self.headers = {
            'apikey': self.key,
            'Authorization': f'Bearer {self.key}',
            'Content-Type': 'application/json',
            'Prefer': 'return=representation'
        }

        self._initialized = True
        logger.info(f"SupabaseClient initialized: {self.url}")

    def _request(
        self,
        method: str,
        endpoint: str,
        data: dict = None,
        params: dict = None,
        headers_override: dict = None
    ) -> Any:
        """Make HTTP request to Supabase REST API."""
        url = f"{self.rest_url}/{endpoint}"
        if params:
            url = f"{url}?{urlencode(params)}"

        headers = {**self.headers, **(headers_override or {})}

        body = json.dumps(data).encode('utf-8') if data else None

        req = Request(url, data=body, headers=headers, method=method)

        timeout_s = float(os.environ.get("SUPABASE_HTTP_TIMEOUT_SEC", "30") or 30)
        max_attempts = int(os.environ.get("SUPABASE_HTTP_MAX_ATTEMPTS", "5") or 5)
        retryable_codes = {408, 409, 425, 429, 500, 502, 503, 504}

        last_err: Exception | None = None
        for attempt in range(1, max_attempts + 1):
            try:
                with urlopen(req, timeout=timeout_s) as resp:
                    content = resp.read().decode("utf-8")
                    return json.loads(content) if content else None
            except HTTPError as e:
                error_body = e.read().decode("utf-8", errors="replace")
                last_err = e
                if e.code in retryable_codes and attempt < max_attempts:
                    sleep_s = min(30.0, (2 ** (attempt - 1)) * (0.5 + random.random()))
                    logger.warning(
                        f"Supabase HTTP {e.code} on {endpoint} (attempt {attempt}/{max_attempts}): "
                        f"{error_body[:200]}. Retrying in {sleep_s:.1f}s..."
                    )
                    time.sleep(sleep_s)
                    continue
                logger.error(f"Supabase error {e.code}: {error_body}")
                raise
            except (URLError, socket.timeout, TimeoutError) as e:
                last_err = e
                if attempt < max_attempts:
                    sleep_s = min(30.0, (2 ** (attempt - 1)) * (0.5 + random.random()))
                    logger.warning(
                        f"Supabase network/timeout on {endpoint} (attempt {attempt}/{max_attempts}): {e}. "
                        f"Retrying in {sleep_s:.1f}s..."
                    )
                    time.sleep(sleep_s)
                    continue
                logger.error(f"Network error: {getattr(e, 'reason', e)}")
                raise

        # Defensive: should have returned or raised.
        if last_err is not None:
            raise last_err
        return None

    def _rpc(self, function_name: str, params: dict = None) -> Any:
        """Call Supabase RPC function."""
        url = f"{self.url}/rest/v1/rpc/{function_name}"

        body = json.dumps(params or {}).encode('utf-8')
        req = Request(url, data=body, headers=self.headers, method='POST')

        timeout_s = float(os.environ.get("SUPABASE_HTTP_TIMEOUT_SEC", "30") or 30)
        max_attempts = int(os.environ.get("SUPABASE_HTTP_MAX_ATTEMPTS", "5") or 5)
        retryable_codes = {408, 409, 425, 429, 500, 502, 503, 504}

        last_err: Exception | None = None
        for attempt in range(1, max_attempts + 1):
            try:
                with urlopen(req, timeout=timeout_s) as resp:
                    content = resp.read().decode("utf-8")
                    return json.loads(content) if content else None
            except HTTPError as e:
                error_body = e.read().decode("utf-8", errors="replace")
                last_err = e
                if e.code in retryable_codes and attempt < max_attempts:
                    sleep_s = min(30.0, (2 ** (attempt - 1)) * (0.5 + random.random()))
                    logger.warning(
                        f"Supabase RPC HTTP {e.code} {function_name} (attempt {attempt}/{max_attempts}): "
                        f"{error_body[:200]}. Retrying in {sleep_s:.1f}s..."
                    )
                    time.sleep(sleep_s)
                    continue
                logger.error(f"RPC error {e.code}: {error_body}")
                raise
            except (URLError, socket.timeout, TimeoutError) as e:
                last_err = e
                if attempt < max_attempts:
                    sleep_s = min(30.0, (2 ** (attempt - 1)) * (0.5 + random.random()))
                    logger.warning(
                        f"Supabase RPC network/timeout {function_name} (attempt {attempt}/{max_attempts}): {e}. "
                        f"Retrying in {sleep_s:.1f}s..."
                    )
                    time.sleep(sleep_s)
                    continue
                logger.error(f"RPC network error: {getattr(e, 'reason', e)}")
                raise

        if last_err is not None:
            raise last_err
        return None

    def claim_next_video(
        self,
        worker_id: str,
        lease_duration_sec: int = 900,
        language: str = None
    ) -> Optional[Dict[str, Any]]:
        """
        Atomically claim the next PENDING video.

        Uses RPC function with FOR UPDATE SKIP LOCKED for safe concurrent access.
        Falls back to REST API claim if RPC not available.

        Args:
            worker_id: Unique identifier for this worker
            lease_duration_sec: How long to hold the claim (default 15 min)
            language: Optional language filter

        Returns:
            Video dict with youtube_id, youtube_url, title, etc. or None if queue empty
        """
        try:
            # Try RPC function first (atomic)
            result = self._rpc('claim_next_video', {
                'p_worker_id': worker_id,
                'p_lease_duration_sec': lease_duration_sec
            })

            if result and len(result) > 0:
                video = result[0]
                logger.info(f"Claimed video: {video.get('youtube_id')} by {worker_id}")
                return video
            return None

        except HTTPError as e:
            if e.code == 404:
                # RPC function not found, fall back to REST API
                logger.warning("claim_next_video RPC not found, using REST fallback")
                return self._claim_video_rest(worker_id, lease_duration_sec, language)
            raise

    def _claim_video_rest(
        self,
        worker_id: str,
        lease_duration_sec: int,
        language: str = None
    ) -> Optional[Dict[str, Any]]:
        """


Summary: auto


rg -n "CONTAINER_ID|VAST|instance_id|INSTANCE_ID" vast_ai.md vast_ai_sdk.md scripts -S | sed -n '1,200p'


Output:
Exit code: 0
Wall time: 0 seconds
Output:
vast_ai_sdk.md:153:| `logs(INSTANCE_ID: int, tail: Optional[str] = None)` | Retrieve logs for an instance.                 |
vast_ai_sdk.md:164:| `attach_ssh(instance_id: int, ssh_key: str)`    | Attach an SSH key to an instance.   |
vast_ai_sdk.md:165:| `detach_ssh(instance_id: int, ssh_key_id: str)` | Detach an SSH key from an instance. |
vast_ai.md:382:vastai attach ssh <instance_id>
vast_ai.md:393:vastai ssh-url <instance_id>
vast_ai.md:396:ssh $(vastai ssh-url <instance_id>)
vast_ai.md:527:| `CONTAINER_ID` | Instance ID |
vast_ai.md:532:| `VAST_TCP_PORT_X` | Mapped external port for internal port X |
vast_ai.md:736:curl "https://console.vast.ai/api/v0/instances/$INSTANCE_ID/" \
vast_ai.md:771:curl -X PUT "https://console.vast.ai/api/v0/instances/$INSTANCE_ID/" \
vast_ai.md:793:curl -X PUT "https://console.vast.ai/api/v0/instances/$INSTANCE_ID/" \
vast_ai.md:812:curl -X PUT "https://console.vast.ai/api/v0/instances/reboot/$INSTANCE_ID/" \
vast_ai.md:829:curl -X PUT "https://console.vast.ai/api/v0/instances/recycle/$INSTANCE_ID/" \
vast_ai.md:849:curl -X DELETE "https://console.vast.ai/api/v0/instances/$INSTANCE_ID/" \
vast_ai.md:864:curl -X PUT "https://console.vast.ai/api/v0/instances/$INSTANCE_ID/" \
vast_ai.md:883:curl -X PUT "https://console.vast.ai/api/v0/instances/bid_price/$INSTANCE_ID/" \
vast_ai.md:900:vastai copy local:./data C.<instance_id>:/workspace/
vast_ai.md:903:vastai copy C.<instance_id>:/workspace/output local:./results/
vast_ai.md:906:vastai copy s3.101:/bucket/data C.<instance_id>:/workspace/
vast_ai.md:907:vastai copy drive:/folder C.<instance_id>:/workspace/
vast_ai.md:926:curl -X PUT "https://console.vast.ai/api/v0/instances/command/$INSTANCE_ID/" \
vast_ai.md:953:  --instance <instance_id> \
vast_ai.md:986:curl -X PUT "https://console.vast.ai/api/v0/instances/request_logs/$INSTANCE_ID" \
vast_ai.md:1030:INSTANCE_ID = 12345
vast_ai.md:1033:    f"https://console.vast.ai/api/v0/instances/{INSTANCE_ID}/",
vast_ai.md:1139:def wait_for_instance(instance_id, timeout=300):
vast_ai.md:1143:            f"{BASE_URL}/instances/{instance_id}/",
vast_ai.md:1160:def get_ssh_info(instance_id):
vast_ai.md:1162:        f"{BASE_URL}/instances/{instance_id}/",
vast_ai.md:1173:def stop_instance(instance_id):
vast_ai.md:1175:        f"{BASE_URL}/instances/{instance_id}/",
vast_ai.md:1182:def destroy_instance(instance_id):
vast_ai.md:1184:        f"{BASE_URL}/instances/{instance_id}/",
vast_ai.md:1209:    instance_id = result.get("new_contract")
vast_ai.md:1210:    print(f"Created instance: {instance_id}")
vast_ai.md:1213:    instance = wait_for_instance(instance_id)
vast_ai.md:1216:    ssh_info = get_ssh_info(instance_id)
vast_ai.md:1222:    # destroy_instance(instance_id)
scripts/vast/vastctl.py:11:  - Set `VAST_API_KEY` (preferred) or `VAST_AI` in the environment.
scripts/vast/vastctl.py:34:    key = explicit or os.environ.get("VAST_API_KEY") or os.environ.get("VAST_AI") or ""
scripts/vast/vastctl.py:37:            "Missing Vast API key. Set `VAST_API_KEY` (preferred) or `VAST_AI` in the environment."
scripts/vast/vastctl.py:96:    for iid in args.instance_ids:
scripts/vast/vastctl.py:107:    for iid in args.instance_ids:
scripts/vast/vastctl.py:124:    resp = _request_json(api_key, "PUT", f"/instances/request_logs/{args.instance_id}", body=body)
scripts/vast/vastctl.py:147:    parser.add_argument("--api-key", default=None, help="Vast API key (or set VAST_API_KEY)")
scripts/vast/vastctl.py:155:    p_destroy.add_argument("instance_ids", nargs="+")
scripts/vast/vastctl.py:160:    p_recycle.add_argument("instance_ids", nargs="+")
scripts/vast/vastctl.py:165:    p_logs.add_argument("instance_id")


Output:
Exit code: 0
Wall time: 0 seconds
Output:

**Common environment variables:**

| Variable | Description |
|----------|-------------|
| `WANDB_API_KEY` | Weights & Biases API key |
| `HF_TOKEN` | Hugging Face token |
| `OPENAI_API_KEY` | OpenAI API key |
| `AWS_ACCESS_KEY_ID` | AWS credentials |
| `AWS_SECRET_ACCESS_KEY` | AWS credentials |

### Port Mapping

```bash
# Map ports
--env '-p 8080:8080 -p 6006:6006 -p 5000:5000'

# Format: -p external:internal
# Ports 70000+ enable identity mapping (external = internal)
```

### Predefined Environment Variables

These are automatically set by Vast.ai:

| Variable | Description |
|----------|-------------|
| `CONTAINER_ID` | Instance ID |
| `CONTAINER_API_KEY` | API key for CLI use inside container |
| `GPU_COUNT` | Number of GPUs |
| `PUBLIC_IPADDR` | Public IP address |
| `SSH_PUBLIC_KEY` | Your SSH public key |
| `VAST_TCP_PORT_X` | Mapped external port for internal port X |

### Onstart Scripts

Commands run after container starts:

```bash
# Inline command
--onstart-cmd 'pip install -r requirements.txt && python setup.py'

# From file
--onstart setup.sh
```

**Example onstart script:**

```bash
#!/bin/bash
# Make environment variables available in SSH sessions
env | grep _ >> /etc/environment

# Install dependencies
pip install torch transformers accelerate
pip install wandb tensorboard

# Clone repo
git clone https://github.com/myrepo/myproject.git /workspace/project

# Start services


sed -n '1,220p' scripts/supabase_migrations.sql


Output:
Exit code: 0
Wall time: 0 seconds
Output:
-- Supabase Migrations for Video Processing Queue
-- Run these in the Supabase SQL Editor or via psql

-- ============================================================================
-- STEP 1: Add chapter columns (if not exist)
-- ============================================================================
ALTER TABLE videos ADD COLUMN IF NOT EXISTS has_chapters BOOLEAN DEFAULT FALSE;
ALTER TABLE videos ADD COLUMN IF NOT EXISTS chapter_count INTEGER DEFAULT 0;
ALTER TABLE videos ADD COLUMN IF NOT EXISTS chapters JSONB;

-- ============================================================================
-- STEP 2: Add extra_meta column for full segment data (if not exist)
-- ============================================================================
ALTER TABLE videos ADD COLUMN IF NOT EXISTS extra_meta JSONB;

-- ============================================================================
-- STEP 3: Create atomic claim function
-- This uses FOR UPDATE SKIP LOCKED for safe concurrent access
-- ============================================================================
CREATE OR REPLACE FUNCTION claim_next_video(
    p_worker_id TEXT,
    p_lease_duration_sec INT DEFAULT 600
) RETURNS TABLE(
    youtube_id TEXT,
    youtube_url TEXT,
    title TEXT,
    source_duration_min DOUBLE PRECISION,
    language TEXT,
    channel TEXT
) AS $$
BEGIN
    RETURN QUERY
    WITH next_video AS (
        SELECT v.id
        FROM videos v
        WHERE v.status = 'PENDING'
          AND (v.lease_expires_at IS NULL OR v.lease_expires_at < NOW())
          AND COALESCE(v.attempt_count, 0) < COALESCE(v.max_attempts, 3)
        ORDER BY v.created_at ASC
        LIMIT 1
        FOR UPDATE SKIP LOCKED
    )
    UPDATE videos v
    SET
        status = 'CLAIMED',
        claimed_by = p_worker_id,
        claimed_at = NOW(),
        lease_expires_at = NOW() + (p_lease_duration_sec || ' seconds')::INTERVAL,
        attempt_count = COALESCE(v.attempt_count, 0) + 1,
        updated_at = NOW()
    FROM next_video
    WHERE v.id = next_video.id
    RETURNING v.youtube_id, v.youtube_url, v.title, v.source_duration_min, v.language, v.channel;
END;
$$ LANGUAGE plpgsql;

-- ============================================================================
-- STEP 4: Create helper function to reclaim expired leases
-- ============================================================================
CREATE OR REPLACE FUNCTION reclaim_expired_leases()
RETURNS INTEGER AS $$
DECLARE
    reclaimed_count INTEGER;
BEGIN
    WITH expired AS (
        UPDATE videos
        SET
            status = 'PENDING',
            claimed_by = NULL,
            lease_expires_at = NULL,
            updated_at = NOW()
        WHERE status = 'CLAIMED'
          AND lease_expires_at < NOW()
        RETURNING id
    )
    SELECT COUNT(*) INTO reclaimed_count FROM expired;

    RETURN reclaimed_count;
END;
$$ LANGUAGE plpgsql;

-- ============================================================================
-- STEP 5: Create queue stats function
-- ============================================================================
CREATE OR REPLACE FUNCTION get_queue_stats()
RETURNS TABLE(
    status TEXT,
    count BIGINT,
    avg_usable_pct DOUBLE PRECISION
) AS $$
BEGIN
    RETURN QUERY
    SELECT
        v.status::TEXT,
        COUNT(*)::BIGINT,
        AVG(v.usable_percentage)::DOUBLE PRECISION
    FROM videos v
    GROUP BY v.status
    ORDER BY v.status;
END;
$$ LANGUAGE plpgsql;

-- ============================================================================
-- STEP 6: Create indexes for efficient queries
-- ============================================================================
CREATE INDEX IF NOT EXISTS idx_videos_status ON videos(status);
CREATE INDEX IF NOT EXISTS idx_videos_status_created ON videos(status, created_at);
CREATE INDEX IF NOT EXISTS idx_videos_lease_expires ON videos(lease_expires_at) WHERE status = 'CLAIMED';
CREATE INDEX IF NOT EXISTS idx_videos_language ON videos(language);

-- ============================================================================
-- STEP 7: Grant execute permissions for RPC functions
-- ============================================================================
GRANT EXECUTE ON FUNCTION claim_next_video(TEXT, INT) TO anon, authenticated, service_role;
GRANT EXECUTE ON FUNCTION reclaim_expired_leases() TO anon, authenticated, service_role;
GRANT EXECUTE ON FUNCTION get_queue_stats() TO anon, authenticated, service_role;

-- ============================================================================
-- ANALYTICS TABLES (for distributed worker monitoring)
-- ============================================================================

-- ============================================================================
-- STEP 8: Worker heartbeats (real-time status, 1 row per worker)
-- ============================================================================
CREATE TABLE IF NOT EXISTS worker_heartbeats (
    worker_id TEXT PRIMARY KEY,
    machine_id TEXT NOT NULL,
    gpu_id INT NOT NULL,
    status TEXT DEFAULT 'idle',
    current_video_id TEXT,
    current_stage INT,
    current_stage_name TEXT,
    last_heartbeat TIMESTAMPTZ DEFAULT NOW(),
    session_start TIMESTAMPTZ DEFAULT NOW(),
    session_videos_done INT DEFAULT 0,
    session_videos_failed INT DEFAULT 0,
    session_audio_minutes FLOAT DEFAULT 0,
    session_usable_minutes FLOAT DEFAULT 0
);

-- ============================================================================
-- STEP 9: Worker lifetime metrics (aggregated stats)
-- ============================================================================
CREATE TABLE IF NOT EXISTS worker_metrics (
    worker_id TEXT PRIMARY KEY,
    machine_id TEXT NOT NULL,
    gpu_id INT NOT NULL,
    total_videos_processed INT DEFAULT 0,
    total_videos_failed INT DEFAULT 0,
    total_audio_hours FLOAT DEFAULT 0,
    total_usable_hours FLOAT DEFAULT 0,
    total_processing_hours FLOAT DEFAULT 0,
    avg_rtf FLOAT DEFAULT 0,
    first_seen TIMESTAMPTZ DEFAULT NOW(),
    last_active TIMESTAMPTZ DEFAULT NOW()
);

-- ============================================================================
-- STEP 10: Processing events (done/fail only, for analysis)
-- ============================================================================
CREATE TABLE IF NOT EXISTS processing_events (
    id UUID DEFAULT gen_random_uuid() PRIMARY KEY,
    worker_id TEXT NOT NULL,
    video_id TEXT NOT NULL,
    event_type TEXT NOT NULL,
    stage_name TEXT,
    duration_seconds FLOAT,
    audio_minutes FLOAT,
    speakers INT,
    usable_pct FLOAT,
    error_type TEXT,
    error_message TEXT,
    created_at TIMESTAMPTZ DEFAULT NOW()
);

CREATE INDEX IF NOT EXISTS idx_events_worker ON processing_events(worker_id, created_at DESC);
CREATE INDEX IF NOT EXISTS idx_events_video ON processing_events(video_id);
CREATE INDEX IF NOT EXISTS idx_events_type ON processing_events(event_type, created_at DESC);

-- ============================================================================
-- STEP 11: Error logs (detailed error tracking)
-- ============================================================================
CREATE TABLE IF NOT EXISTS error_logs (
    id UUID DEFAULT gen_random_uuid() PRIMARY KEY,
    worker_id TEXT NOT NULL,
    video_id TEXT NOT NULL,
    stage TEXT,
    error_type TEXT,
    error_message TEXT,
    stack_trace TEXT,
    created_at TIMESTAMPTZ DEFAULT NOW()
);

CREATE INDEX IF NOT EXISTS idx_errors_worker ON error_logs(worker_id, created_at DESC);
CREATE INDEX IF NOT EXISTS idx_errors_recent ON error_logs(created_at DESC);

-- ============================================================================
-- STEP 12: Cluster overview view (aggregated dashboard)
-- ============================================================================
CREATE OR REPLACE VIEW cluster_overview AS
SELECT
    COUNT(DISTINCT machine_id) as total_machines,
    COUNT(*) as total_workers,
    COUNT(*) FILTER (WHERE status = 'processing') as active_workers,
    COUNT(*) FILTER (WHERE status = 'idle') as idle_workers,
    COUNT(*) FILTER (WHERE last_heartbeat < NOW() - INTERVAL '2 minutes') as stale_workers,
    COALESCE(SUM(session_videos_done), 0) as session_total_videos,
    COALESCE(SUM(session_audio_minutes), 0) as session_total_audio_min,
    COALESCE(SUM(session_usable_minutes), 0) as session_total_usable_min
FROM worker_heartbeats;

-- ============================================================================
-- STEP 13: Function to get cluster stats with RTF calculation
-- ============================================================================
CREATE OR REPLACE FUNCTION get_cluster_stats()
RETURNS TABLE(
    total_machines BIGINT,
    total_workers BIGINT,
    active_workers BIGINT,
    idle_workers BIGINT,


Output:
Exit code: 0
Wall time: 0 seconds
Output:
    background_export = getattr(config, 'background_export', False)

    if export_enabled:
        log_stage(10, "Export", "START")

        upload_prefix = (getattr(config, 'r2_upload_prefix', '') or '').strip().strip('/')
        remote_key = f"{video_id}.tar" if not upload_prefix else f"{upload_prefix}/{video_id}.tar"
        if getattr(config, 'r2_upload_enabled', False):
            result['r2_upload_key'] = remote_key
        original_audio_path = Path(metadata.get('original_audio_path', ''))
        intro_offset = metadata.get('intro_skipped', 0.0)

        if background_export:
            # === BACKGROUND EXPORT MODE ===
            # Submit job to background manager, don't block GPU
            from src.export import ExportJob, get_export_manager

            export_manager = get_export_manager()
            if export_manager:
                job = ExportJob(
                    video_id=video_id,
                    output_dir=output_dir,
                    original_audio_path=original_audio_path,
                    segments=segments,
                    intro_offset=intro_offset,
                    r2_upload_enabled=getattr(config, 'r2_upload_enabled', False),
                    r2_upload_key=remote_key,
                    r2_bucket_type=getattr(config, 'r2_upload_bucket_type', 'production'),
                    include_original_in_tar=getattr(config, 'include_original_in_tar', False),
                    delete_output_dir_after_upload=getattr(config, 'r2_upload_delete_local_output_dir', False),
                )
                export_manager.submit(job)
                log_stage(10, "Export", "QUEUED", f"background job submitted")
                result['export_mode'] = 'background'
                result['r2_upload_key'] = remote_key
            else:
                logger.warning("Background export manager not initialized, falling back to sync")
                background_export = False  # Fall through to sync mode

        if not background_export:
            # === SYNCHRONOUS EXPORT MODE ===
            from src.r2_client import R2Client
            from src.export import segment_audio_parallel

            with COMPUTE.monitor_stage("Export") as metrics:
                # Step 1: Segment original audio (parallel ffmpeg)
                segments_exported = 0
                if getattr(config, 'segment_audio', True) and original_audio_path.exists():
                    seg_start = time.time()
                    segments_exported = segment_audio_parallel(
                        audio_path=original_audio_path,
                        segments=segments,
                        output_dir=output_dir,
                        intro_offset=intro_offset,
                        max_workers=8,
                        output_format="flac"
                    )
                    seg_time = time.time() - seg_start
                    logger.info(f"📦 Segmented {segments_exported} clips in {seg_time:.1f}s")

                # Step 1.5: Remove original audio from output (source is preserved in R2)
                if not getattr(config, 'include_original_in_tar', False):
                    original_in_output = output_dir / f"{video_id}_original.flac"
                    if original_in_output.exists():
                        original_in_output.unlink()
                        logger.debug(f"Removed _original.flac from output (source in R2)")

                # Step 2: Create tar archive
                tar_path = output_dir.parent / f"{video_id}.tar"
                with tarfile.open(tar_path, "w") as tar:
                    tar.add(output_dir, arcname=video_id)
                tar_size_bytes = tar_path.stat().st_size if tar_path.exists() else 0

                # Step 3: Upload to R2 (if enabled)
                upload_ok = True
                if getattr(config, 'r2_upload_enabled', False):
                    r2 = R2Client(bucket_type=getattr(config, 'r2_upload_bucket_type', 'production'))
                    if getattr(config, 'r2_upload_skip_if_exists', False) and r2.exists(remote_key):
                        logger.info(f"R2 upload skipped (exists): {r2.uri(remote_key)}")
                    else:
                        upload_ok = r2.upload_file(str(tar_path), remote_key)
                        if not upload_ok:
                            raise RuntimeError(f"R2 upload failed for {r2.uri(remote_key)}")

                    # Delete local tar after upload
                    if getattr(config, 'r2_upload_delete_local_tar', True):
                        try:
                            tar_path.unlink()
                        except Exception:
                            pass

                    # Delete per-video output dir after successful upload (prevents disk fill).
                    if getattr(config, 'r2_upload_enabled', False) and getattr(config, 'r2_upload_delete_local_output_dir', False):
                        try:
                            import shutil
                            shutil.rmtree(output_dir, ignore_errors=True)
                        except Exception:
                            pass

            timings['export'] = metrics.duration
            compute_stats['export'] = {'cpu': metrics.cpu_percent_avg, 'gpu': metrics.gpu_percent_avg}
            result['export'] = {
                'segments_exported': segments_exported,
                'tar_size_bytes': int(tar_size_bytes),
                'r2_upload_key': remote_key if getattr(config, 'r2_upload_enabled', False) else None,
                'r2_upload_bucket': r2.bucket if getattr(config, 'r2_upload_enabled', False) else None,
                'r2_upload_full_key': r2.full_key(remote_key) if getattr(config, 'r2_upload_enabled', False) else None,
                'r2_upload_uri': r2.uri(remote_key) if getattr(config, 'r2_upload_enabled', False) else None,
            }

            log_stage(10, "Export", "DONE", f"{timings['export']:.1f}s | {segments_exported} segs | {tar_size_bytes/(1024**2):.1f}MB")

            # Update timing_total
            total_time = time.time() - total_start
            result['timing'] = {k: round(v, 2) for k, v in timings.items()}
            result['timing_total'] = round(total_time, 2)
            result['compute_stats'] = compute_stats

            # Update metadata.json
            try:
                with open(output_dir / "metadata.json", 'w') as f:
                    json.dump(result, f, indent=2)
            except Exception:
                pass

    # Summary (after optional upload so timing_total is end-to-end)
    final_total = result.get('timing_total', round(total_time, 2))
    plog.done(
        video_id=video_id,
        total_time=final_total,
        audio_min=total_duration / 60,
        speakers=result['num_speakers'],
        usable_pct=result['quality_stats']['usable_percentage']
    )

    # Clear cache for next video
    MODELS.clear_cache(aggressive=True)
    COMPUTE.refresh_gpu_memory()

    return result


Output:
Exit code: 0
Wall time: 0 seconds
Total output lines: 220
Output:
#!/usr/bin/env python3
"""
Configuration for Fast Pipeline v6.0 - Adaptive Compute-Aware

Key improvements:
- Auto-detects system resources (nproc, vCPUs, GPU vRAM)
- Adapts worker counts and batch sizes to available compute
- Optimized for micro-level speaker detection (0.4s events)
"""

import os
from dataclasses import dataclass, field
from typing import Optional

@dataclass
class Config:
    """
    Pipeline configuration with adaptive compute settings.

    Flow (per instructions.md):
    1. Download (max parallel workers)
    2. Quick VAD (parallel, get speech outline)
    3. Chunk at silence boundaries (VAD-aware)
    4. OSD (mark overlaps as unusable FIRST)
    5. Frame-level segmentation (17ms resolution for micro-changes)
    6. Embeddings + Conservative clustering
    7. Output metadata JSON only
    """

    # === AUTHENTICATION ===
    # NOTE: Must be provided via environment variable HF_TOKEN
    # Never hard-code tokens in repo (HF gated model access + security).
    hf_token: str = os.environ.get('HF_TOKEN', '')

    # === INPUT SOURCE (YouTube vs R2) ===
    # Default behavior remains YouTube URLs. For benchmark/testing you can pass
    # bare 11-char video IDs and set input_source='r2' to stream originals from
    # the R2 source bucket (bucket name: "test") without YouTube rate limits.
    input_source: str = "youtube"  # 'youtube' | 'r2'

    # === R2 SOURCE (input) ===
    # When input_source='r2', download.py will:
    # - Resolve video_id -> object key via HEAD (NO bucket listing)
    # - Generate a presigned GET URL
    # - ffmpeg stream-extract audio
    #
    # Prefix/extension knobs are optional; env overrides exist too:
    # - R2_SOURCE_PREFIX / R2_SOURCE_PREFIXES
    # - R2_SOURCE_EXTENSIONS
    r2_source_prefix: str = ""          # e.g. "podcasts/" (optional)
    r2_source_extensions: str = ""      # e.g. ".webm,.mp4" (optional)
    r2_presign_ttl_sec: int = 3600      # Presigned URL TTL (seconds)

    # Avoid any YouTube metadata calls during R2 runs (rate-limit safe).
    # When False, R2 downloads fall back to duration-based dynamic intro skip.
    fetch_youtube_metadata_for_r2: bool = True

    # === R2 OUTPUT (upload processed artifacts) ===
    # When enabled, pipeline will package each video's output directory into a .tar
    # and upload it to the R2 production bucket (1-cleaned-data) by default.
    r2_upload_enabled: bool = False
    r2_upload_bucket_type: str = "production"  # maps to 1-cleaned-data in src/r2_client.py
    r2_upload_prefix: str = ""                 # e.g. "benchmarks/rtx4090/2026-01-03T16-00Z"
    r2_upload_skip_if_exists: bool = False     # avoid overwriting if True
    r2_upload_delete_local_tar: bool = True    # free local disk after upload

    # === AUDIO SEGMENTATION ===
    # segment_audio: Cut original FLAC into individual speaker segment files
    # Uses parallel ffmpeg (8 workers) - typically 2-3s for 500 segments
    segment_audio: bool = True

    # === BACKGROUND EXPORT ===
    # NOTE: Background export is experimental and not recommended for multiprocessing.
    # In GPU worker subprocesses, background threads die when the process exits.
    # The sync export (5-6s for 60min video) runs AFTER GPU work completes,
    # and overlaps with the NEXT video's download prefetch, so overhead is minimal.
    background_export: bool = False  # Keep False for stability
    background_export_workers: int = 4  # Number of parallel export workers (unused currently)

    # === AUDIO PROCESSING ===
    sample_rate: int = 16000
    intro_skip_seconds: float = 0.0
    outro_skip_seconds: float = 0.0

    # === INTRO SKIP (v6.8 - Dynamic) ===
    # auto_intro_skip: Automatically skip intro based on video duration
    # When True and no chapter-based intro detected:
    #   - Video ≤ 30 min: skip 180s (3 min)
    #   - Video > 30 min: skip 300s (5 min)
    # Set to False to disable automatic intro skipping
    # Manual intro_skip_seconds > 0 always takes priority
    auto_intro_skip: bool = True

    # === HIGH QUALITY AUDIO PRESERVATION (v6.8+) ===
    # preserve_original_audio: Download at highest quality, process at 16kHz
    # When True:
    #   - Downloads original audio format (bestaudio - typically 48kHz)
    #   - Creates 16kHz copy for processing (VAD, diarization, embeddings)
    #   - After processing, deletes 16kHz and keeps only original
    # Final export cuts from original high-quality audio using timestamps
    preserve_original_audio: bool = True
    original_audio_sample_rate: int = 0  # 0 = keep original (highest), or specify (e.g. 48000)

    # === PARALLELISM (Auto-detected, these are fallbacks) ===
    # Actual values computed by COMPUTE.get_optimal_config()
    vad_workers: int = 32          # Will be auto-tuned
    max_workers: int = 32          # Will be auto-tuned
    download_workers: int = 4      # Parallel downloads for batch
    chunk_workers: int = 4         # GPU diarization (VRAM l…985 chars truncated…Window duration (seconds)
    min_segment_duration: float = 0.2   # Min segment (catch 0.4s events)

    # === OVERLAP DETECTION ===
    detect_overlap: bool = True
    overlap_threshold: float = 0.35     # Probability for active speaker
    overlap_min_duration: float = 0.1  # Min overlap to mark
    overlap_padding_ms: int = 100      # Pad overlap boundaries (±100ms)

    # === OVERLAP DENSITY DETECTION (v6.9) ===
    # Mark short segments sandwiched between overlaps as unusable
    # These are "islands" of supposedly clean audio in high-overlap regions
    overlap_density_filter: bool = True
    overlap_density_max_gap: float = 1.0      # Max gap to adjacent overlap (seconds)
    overlap_density_max_duration: float = 3.0  # Max segment duration to mark as unusable

    # === QUALITY FILTERING (v6.1) ===
    filter_by_quality: bool = True     # Enable SNR/quality filtering
    min_snr_db: float = 15.0           # Minimum SNR for TTS usability
    min_quality_score: float = 0.3     # Minimum quality score (0-1)

    # === SEGMENT DURATION FILTERING (v6.1) ===
    # Drop speaker segments shorter than this from the FINAL output.
    #
    # IMPORTANT: we do adjacent same-speaker merging before applying this filter,
    # so short fragments (e.g. 0.4-0.9s) can be absorbed into longer continuous
    # turns when safe to do so.
    min_tts_duration: float = 1.0      # Minimum segment duration for TTS

    # === DIARIZATION ===
    min_speakers: int = 1
    max_speakers: int = 10

    # === EMBEDDINGS (OOM-protected) ===
    embedding_batch_size: int = 8      # Will be auto-tuned
    max_embedding_length: int = 16000 * 10  # Max 10s per segment

    # === CLUSTERING (Conservative - no poison merges) ===
    # Per instructions: 0.75-0.80 to prevent merging different speakers
    cluster_merge_threshold: float = 0.80
    min_segments_for_merge: int = 2

    # === CHUNK REASSIGNMENT (v7.7 - With Validation Gates) ===
    # Detects within-segment speaker changes and surgically splits/reassigns
    #
    # === v7.7 FIX: VALIDATION GATES prevent false positive splits ===
    # Previous issue: 66x speaker fragmentation (6 speakers → 400+)
    # Root cause: Fixed thresholds + create_new_on_ambiguous triggered on natural variation
    #
    # NEW GATES (see unified_embeddings.py):
    # 1. Persistence gate: Require ≥N consecutive low-sim chunks
    # 2. Relative valley gate: Per-segment adaptive threshold (outlier detection)
    # 3. Same-speaker centroid gate: Reject if both sides match original speaker
    # 4. No-new-speaker-unless-proven: Keep original speaker on ambiguous (key fix!)
    enable_chunk_reassignment: bool = True  # RE-ENABLED with validation gates
    chunk_reassignment_threshold: float = 0.30  # Normal threshold (per-segment adaptive overrides)
    chunk_reassignment_severe: float = 0.15  # Severe circuit-breaker (only extreme changes)
    chunk_reassignment_min_speech: float = 0.6  # Min speech ratio for eligibility
    chunk_reassignment_min_portion: float = 1.0  # LOWERED: 1.0s to catch "yeah/ok" (was 2.0s)

    # === v7.8: TUNABLE GATE PARAMETERS for catching small interruptions ===
    # Lower values = more aggressive splitting (catches more but risks false positives)
    chunk_reassignment_min_analysis: float = 2.0  # Min segment duration to analyze (was hardcoded 3.0)
    chunk_reassignment_persistence: int = 1  # Min consecutive low-sim chunks (was hardcoded 2)
    chunk_reassignment_centroid_threshold: float = 0.55  # Match threshold for centroid gate (was hardcoded 0.60)

    # === MUSIC DETECTION (v7.4 - Hybrid INA+PANNs) ===
    # Two-stage detection: Fast INA screening + PANNs verification
    # Stage 1: inaSpeechSegmenter (fast CNN, ~10x realtime)
    # Stage 2: PANNs CNN14 (only on suspicious segments)
    enable_music_detection: bool = True  # Enable music detection
    use_hybrid_detection: bool = False  # DISABLED: Run PANNs on EVERYTHING for strict music detection
    music_chunk_duration: float = 1.5  # Same as embedding chunks
    music_batch_size: int = 128  # Batch size for PANNs inference (v7.5: optimized for A100)

    # === STRICT DETECTION THRESHOLDS FOR TTS QUALITY (v7.1) ===
    # Zero tolerance: ANY contamination → needs_demucs or unusable

    # Per-chunk thresholds (RESTORED to working values)
    music_prob_threshold: float = 0.20  # Chunk "has music" if prob > 20%
    noise_prob_threshold: float = 0.25  # Chunk "has noise" if prob > 25%

    # Per-segment decision thresholds (STRICT but realistic)
    # Based on empirical testing: baseline neural net output is ~0.04-0.06
    music_ratio_clean: float = 0.0      # ANY chunk with music detection → needs_demucs
    music_ratio_demucs: float = 0.15    # 0-15% → demucs, >15% → unusable
    music_mean_clean: float = 0.10      # ADJUSTED: 0.10 accounts for neural net baseline (was 0.05)
    music_mean_demucs: float = 0.25     # Threshold for heavy_contamination


**Investigating upload issues**

I’m looking into the R2 uploads because the user can’t find them. It’s possible that uploads are failing silently. The `r2_client.upload_file` function has try/except for error logging, but I wonder if errors might not appear in the logs due to logger configurations. I’m also considering the logs from the massive process' pipeline, but they may not provide the information I need. I’ll inspect the truncated part of `r2_client.upload_file` and open the complete section to get clarity.


python - <<'PY'
import inspect
import src.r2_client as r
import textwrap
print(inspect.getsource(r.R2Client.upload_file))
PY


Output:
Exit code: 127
Wall time: 0 seconds
Output:
/bin/bash: line 1: python: command not found


**Investigating Python binary**

I notice there seems to be no Python binary available, which is interesting. Maybe it's just the Python 3 version that's installed. I think I'll use sed to examine a specific section of the file. My plan is to open it around where the upload_file command is executed. This should help me understand what's going on! Let’s see what I can find out from there.


rg -n "def upload_file" -n src/r2_client.py && sed -n '90,160p' src/r2_client.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:
138:    def upload_file(self, local_path: str, remote_key: str) -> bool:

        # IMPORTANT: The output prefix (R2_PREFIX) should not accidentally apply
        # to the *source* bucket. Vast/RunPod templates often set R2_PREFIX for
        # uploads; for source reads we want R2_SOURCE_PREFIX (or empty).
        if bucket_type == 'source' and config is None:
            self.prefix = os.environ.get('R2_SOURCE_PREFIX', '') or ''
        else:
            self.prefix = self.config.prefix

        # Initialize boto3 client
        self._client = None

    @property
    def client(self):
        """Lazy-initialize boto3 client."""
        if self._client is None:
            if not self.config.endpoint_url:
                raise RuntimeError("R2 not configured: missing R2_ENDPOINT_URL (or S3_API)")
            if not self.config.access_key_id:
                raise RuntimeError("R2 not configured: missing R2_ACCESS_KEY_ID (or AWS_ACCESS_KEY_ID)")
            if not self.config.secret_access_key:
                raise RuntimeError("R2 not configured: missing R2_SECRET_ACCESS_KEY (or AWS_SECRET_ACCESS_KEY)")
            self._client = boto3.client(
                's3',
                endpoint_url=self.config.endpoint_url,
                aws_access_key_id=self.config.access_key_id,
                aws_secret_access_key=self.config.secret_access_key,
                config=BotoConfig(
                    signature_version='s3v4',
                    retries={'max_attempts': 3, 'mode': 'adaptive'}
                )
            )
        return self._client

    def full_key(self, key: str) -> str:
        """Public helper: return fully-qualified key (including configured prefix)."""
        return self._full_key(key)

    def uri(self, key: str) -> str:
        """Return an s3:// URI for a key (including configured prefix)."""
        return f"s3://{self.bucket}/{self._full_key(key)}"

    def _full_key(self, key: str) -> str:
        """Get full S3 key with prefix."""
        if self.prefix:
            return f"{self.prefix}/{key}"
        return key

    def upload_file(self, local_path: str, remote_key: str) -> bool:
        """
        Upload a file to R2.

        Args:
            local_path: Path to local file
            remote_key: Remote key (path in bucket)

        Returns:
            True if successful
        """
        try:
            full_key = self._full_key(remote_key)
            self.client.upload_file(local_path, self.bucket, full_key)
            logger.info(f"Uploaded {local_path} -> s3://{self.bucket}/{full_key}")
            return True
        except Exception as e:
            logger.error(f"Upload failed: {e}")
            return False

    def download_file(self, remote_key: str, local_path: str) -> bool:
        """
        Download a file from R2.


**Checking upload logs**

I'm trying to upload logs info, but they might not show up due to log levels or configuration issues. If there's a failure, the pipeline should raise an alert. If files are missing, it could be because the r2_upload_enabled is set to false or I'm using a test bucket instead of production. I need to verify if the `--r2-upload` flag in the massive_process CLI is necessary for the r2 upload. Let's confirm that!


rg -n "--r2-upload|r2_upload" massive_process.py | sed -n '1120,1320p'


Output:
Exit code: 0
Wall time: 0 seconds
Output:
rg: unrecognized flag --r2-upload|r2_upload


rg -n "r2-upload|r2_upload" massive_process.py | sed -n '1,200p'


Output:
Exit code: 0
Wall time: 0 seconds
Output:
115:    # R2 vars only required if --r2-upload is used
165:    r2_upload_key: str = ""
386:                        r2_upload_key=(result.get('r2_upload', {}) or {}).get('remote_key', ''),
610:                    'r2_tar_key': ((result.get('export', {}) or {}).get('r2_upload_full_key')
611:                                   or result.get('r2_upload_key', '')),
664:                    r2_upload_key=result.get('r2_upload_key', ''),
1169:    parser.add_argument('--r2-upload', action='store_true',
1171:    parser.add_argument('--r2-upload-prefix', type=str, default='',
1173:    parser.add_argument('--r2-upload-skip-if-exists', action='store_true',
1274:        'r2_upload_enabled': bool(args.r2_upload),
1275:        'r2_upload_bucket_type': 'production',
1276:        'r2_upload_prefix': args.r2_upload_prefix,
1277:        'r2_upload_skip_if_exists': bool(args.r2_upload_skip_if_exists),


Output:
Exit code: 0
Wall time: 0 seconds
Output:
    # Processing options
    parser.add_argument('--gpus', type=str, default=None,
                       help='Comma-separated GPU IDs (default: all available)')
    parser.add_argument('--workers-per-gpu', type=int, default=1,
                       help='Number of workers per GPU (default: 1, try 2-4 for H100)')
    parser.add_argument('--num-videos', type=int, default=8,
                       help='Number of videos for benchmark mode (default: 8)')
    parser.add_argument('--output', '-o', type=str, default='data/massive_output',
                       help='Output directory (default: data/massive_output)')

    # Pipeline config overrides
    parser.add_argument('--merge-threshold', type=float, default=0.80,
                       help='Speaker merge threshold (default: 0.80)')
    parser.add_argument('--min-segment', type=float, default=0.2,
                       help='Minimum segment duration (default: 0.2s)')
    parser.add_argument('--no-music-detection', action='store_true',
                       help='Disable music detection')

    # R2 upload (processed .tar -> 1-cleaned-data)
    parser.add_argument('--r2-upload', action='store_true',
                       help='Upload per-video .tar into R2 bucket 1-cleaned-data (no deletes)')
    parser.add_argument('--r2-upload-prefix', type=str, default='',
                       help="Optional prefix within 1-cleaned-data for uploads (e.g. 'benchmarks/rtx4090/2026-01-03')")
    parser.add_argument('--r2-upload-skip-if-exists', action='store_true',
                       help='Skip upload if remote key already exists')

    # Export options
    parser.add_argument('--segment-audio', action='store_true', default=True,
                       help='Segment original audio into individual speaker clips (default: True)')
    parser.add_argument('--no-segment-audio', action='store_false', dest='segment_audio',
                       help='Disable audio segmentation')
    parser.add_argument('--background-export', action='store_true',
                       help='Run export (segmentation+tar+upload) in background threads')
    parser.add_argument('--export-workers', type=int, default=4,
                       help='Number of background export workers (default: 4)')

    # Supabase queue options
    parser.add_argument('--max-videos', type=int, default=0,
                       help='Max videos per worker in supabase-queue mode (0=unlimited)')
    parser.add_argument('--lease-duration', type=int, default=900,
                       help='Lease duration in seconds for claimed videos (default: 900)')
    parser.add_argument('--poll-interval', type=int, default=10,
                       help='Seconds to wait when queue empty (default: 10)')

    args = parser.parse_args()

    # Load .env once in the parent process so workers inherit env vars when possible.
    if load_dotenv is not None:
        try:
            load_dotenv(dotenv_path=Path("/workspace/maya3_data/.env"), override=False)
        except Exception:
            pass

    # Validate environment before starting workers
    if not validate_environment():
        logger.error("Environment validation failed. Please set required variables.")
        sys.exit(1)

    # Get GPU list
    if args.gpus:
        gpu_ids = [int(x.strip()) for x in args.gpus.split(',')]
    else:
        gpu_ids = get_available_gpus()

    if not gpu_ids:
        logger.error("No GPUs available!")
        sys.exit(1)

    logger.info(f"Detected {len(gpu_ids)} GPUs: {gpu_ids}")

    # Get video URLs (not needed for supabase-queue mode)
    video_urls = None
    supabase_mode = getattr(args, 'supabase_queue', False)

    if supabase_mode:
        if not SUPABASE_AVAILABLE:
            logger.error("Supabase client not available! Install dependencies or check src/supabase_client.py")
            sys.exit(1)
        logger.info("Supabase queue mode: videos will be claimed from database")
    elif args.benchmark:
        video_urls = get_test_videos(args.num_videos)
        logger.info(f"Benchmark mode: {len(video_urls)} test videos")
    elif args.input:
        video_urls = load_video_urls(args.input)
        logger.info(f"Loaded {len(video_urls)} URLs from {args.input}")
    else:
        video_urls = args.urls

    if not supabase_mode and not video_urls:
        logger.error("No videos to process!")
        sys.exit(1)

    # CRITICAL: Capture HF_TOKEN before spawning workers
    # Spawned processes may not inherit environment variables
    hf_token = os.environ.get('HF_TOKEN', '')
    if not hf_token:
        # Try loading from .env file
        env_file = Path('.env')
        if env_file.exists():
            with open(env_file) as f:
                for line in f:
                    if line.startswith('HF_TOKEN='):
                        hf_token = line.strip().split('=', 1)[1].strip('"\'')
                        break

    if not hf_token:
        logger.error("HF_TOKEN not found! Set via environment or .env file")
        sys.exit(1)

    logger.info(f"HF_TOKEN captured (length={len(hf_token)})")

    # Config overrides
    config_overrides = {
        'cluster_merge_threshold': args.merge_threshold,
        'min_segment_duration': args.min_segment,
        'enable_music_detection': not args.no_music_detection,
        'workers_per_gpu': args.workers_per_gpu,
        'hf_token': hf_token,  # Pass token to spawned workers
        # R2 source (always R2, metadata from Supabase)
        'input_source': 'r2',
        'fetch_youtube_metadata_for_r2': False,
        'r2_source_prefix': 'podcasts/',
        'r2_source_extensions': '.webm',
        # R2 output upload
        'r2_upload_enabled': bool(args.r2_upload),
        'r2_upload_bucket_type': 'production',
        'r2_upload_prefix': args.r2_upload_prefix,
        'r2_upload_skip_if_exists': bool(args.r2_upload_skip_if_exists),
        # Export options
        'segment_audio': args.segment_audio,
        'background_export': args.background_export,
        'background_export_workers': args.export_workers,
    }

    # Run!
    if supabase_mode:
        # Supabase distributed queue mode
        stats = run_supabase_queue(
            gpu_ids=gpu_ids,
            output_dir=args.output,
            config_overrides=config_overrides,
            max_videos_per_worker=args.max_videos,
            lease_duration_sec=args.lease_duration,
            poll_interval_sec=args.poll_interval
        )
    else:
        # Traditional file-based mode
        stats = run_massive_process(
            video_urls=video_urls,
            gpu_ids=gpu_ids,
            output_dir=args.output,
            config_overrides=config_overrides
        )

    # NOTE: Background export uses per-worker threadpools; when running with
    # multiprocessing (the default), the main process cannot reliably flush
    # those threadpools. Each worker is responsible for draining its own export
    # queue on shutdown.

    # Exit with appropriate code
    if stats.failed > 0:


Output:
Exit code: 0
Wall time: 0 seconds
Output:
        consecutive_empty = 0
        max_consecutive_empty = 6  # Stop after 1 minute of no videos

        while not stop_signal.value:
            # Check max videos limit
            if max_videos > 0 and videos_processed >= max_videos:
                worker_logger.info(f"Reached max videos limit ({max_videos})")
                break

            # Claim next video from Supabase
            try:
                video = supabase.claim_next_video(
                    worker_id=supabase_worker_id,
                    lease_duration_sec=lease_duration_sec
                )
            except Exception as e:
                worker_logger.warning(f"Supabase claim_next_video failed: {e}")
                time.sleep(poll_interval_sec)
                continue

            if not video:
                consecutive_empty += 1
                if consecutive_empty >= max_consecutive_empty:
                    worker_logger.info("No more pending videos, stopping worker")
                    break
                worker_logger.debug(f"No pending videos, sleeping {poll_interval_sec}s...")
                time.sleep(poll_interval_sec)
                continue

            consecutive_empty = 0
            video_id = video['youtube_id']
            video_url = video['youtube_url']

            # Report video pick to analytics (non-blocking)
            analytics.pick(video_id)

            # Fetch full video record to get chapters
            try:
                full_video = supabase.get_video(video_id)
            except Exception as e:
                worker_logger.warning(f"Supabase get_video failed for {video_id}: {e}")
                full_video = None
            if not full_video:
                full_video = video  # Fallback to claim result

            worker_logger.info(f"Processing: {video_id} - {full_video.get('title', '')[:50]}")
            has_chapters = full_video.get('has_chapters', False)
            if has_chapters:
                worker_logger.info(f"   Chapters: {full_video.get('chapter_count', 0)} from Supabase")
            start_time = time.time()

            try:
                # Process video with Supabase metadata (chapters, title, etc.)
                result = process_single_video(video_url, config, supabase_metadata=full_video)
                process_time = time.time() - start_time

                # Extract results for DB update
                quality_stats = result.get('quality_stats', {}) or {}
                segment_summary = result.get('segment_summary', {}) or {}
                download_meta = result.get('download_meta', {}) or {}

                # Update Supabase with success
                update_data = {
                    'num_speakers': result.get('num_speakers', 0),
                    'total_segments': result.get('total_segments', 0),
                    'usable_percentage': quality_stats.get('usable_percentage', 0),
                    'usable_segments': quality_stats.get('usable_segments', 0),
                    'usable_duration_sec': quality_stats.get('usable_duration', 0),
                    # Store the fully-qualified key (includes R2_PREFIX) so it
                    # can be located in R2 without guessing prefixes.
                    'r2_tar_key': ((result.get('export', {}) or {}).get('r2_upload_full_key')
                                   or result.get('r2_upload_key', '')),
                    'r2_tar_size_bytes': (result.get('export', {}) or {}).get('tar_size_bytes'),
                    'pipeline_version': result.get('pipeline_version', ''),
                    'audio_native_sample_rate': download_meta.get('native_sample_rate'),
                    'audio_channels': download_meta.get('channels'),
                    'audio_duration_sec': result.get('processed_duration', 0),
                    'needs_demucs_count': quality_stats.get('music_detection', {}).get('segments_needs_demucs', 0),
                    'heavy_music_count': quality_stats.get('music_detection', {}).get('segments_heavy_contamination', 0),
                    'processing_meta': {
                        'worker_id': supabase_worker_id,
                        'machine_id': machine_id,
                        'gpu_id': gpu_id,
                        'timing_total': result.get('timing_total', 0),
                        'completed_at': datetime.now().isoformat(),
                        'pipeline_version': result.get('pipeline_version', ''),
                    },
                    'quality_stats': quality_stats,
                    'segment_summary': segment_summary,
                    'download_meta': download_meta,
                }

                supabase.update_status(video_id, STATUS_COMPLETED, update_data)

                worker_logger.info(
                    f"Completed: {video_id} | "
                    f"{process_time:.1f}s | "
                    f"{result.get('num_speakers', 0)} speakers | "
                    f"{quality_stats.get('usable_percentage', 0):.1f}% usable"
                )

                # Report success to analytics (non-blocking)
                analytics.done(
                    video_id=video_id,
                    total_time=result.get('timing_total', process_time),
                    audio_min=result.get('processed_duration', 0) / 60.0,
                    speakers=result.get('num_speakers', 0),
                    usable_pct=quality_stats.get('usable_percentage', 0)
                )

                # Put result in queue for stats collection
                video_result = VideoResult(
                    video_url=video_url,
                    video_id=video_id,
                    success=True,
                    duration_sec=result.get('processed_duration', 0),
                    process_time_sec=process_time,
                    pipeline_time_sec=result.get('timing_total', 0),
                    rtf=(result.get('processed_duration', 0) / result.get('timing_total', 1)) if result.get('timing_total') else 0,
                    num_speakers=result.get('num_speakers', 0),
                    usable_percentage=quality_stats.get('usable_percentage', 0),
                    output_path=str(Path(output_dir) / video_id),
                    stage_timings={k: float(v) for k, v in (result.get('timing', {}) or {}).items()},
                    source_type=result.get('source_type', ''),
                    r2_upload_key=result.get('r2_upload_key', ''),
                    gpu_id=gpu_id,
                    worker_id=worker_id
                )

            except Exception as e:
                process_time = time.time() - start_time
                error_msg = str(e)
                error_type = type(e).__name__
                worker_logger.error(f"Failed: {video_id} - {error_msg}")
                traceback.print_exc()

                # Report failure to analytics (non-blocking)
                analytics.fail(
                    video_id=video_id,
                    stage="processing",
                    error_type=error_type,
                    message=error_msg,
                    stack_trace=traceback.format_exc()
                )

                # Update Supabase with failure
                supabase.update_status(video_id, STATUS_FAILED, {
                    'last_error': error_msg[:1000],  # Truncate long errors
                    'last_error_type': type(e).__name__,
                    'processing_meta': {
                        'worker_id': supabase_worker_id,
                        'gpu_id': gpu_id,
                        'failed_at': datetime.now().isoformat(),
                    },
                })

                video_result = VideoResult(
                    video_url=video_url,
                    video_id=video_id,
                    success=False,
                    error=error_msg,
                    process_time_sec=process_time,
                    gpu_id=gpu_id,
                    worker_id=worker_id
                )

            result_queue.put(video_result)
            videos_processed += 1

            # Clear GPU cache between videos
            MODELS.clear_cache(aggressive=True)

        # Drain background exports (if enabled in this worker)
        if getattr(config, "background_export", False):
            try:
                from src.export import shutdown_export_manager
                worker_logger.info("Waiting for background exports to complete...")
                export_results = shutdown_export_manager(wait=True, timeout=600)
                failed = [r for r in export_results.values() if not getattr(r, "success", False)]
                if failed:
                    worker_logger.warning(f"Background exports failed: {len(failed)}/{len(export_results)}")
                    for r in failed[:5]:
                        worker_logger.warning(f"  {r.video_id}: {r.error}")
            except Exception:
                pass

        # Graceful shutdown - flush remaining analytics events
        analytics.shutdown()
        worker_logger.info(f"Shutting down. Processed {videos_processed} videos.")

    except Exception as e:
        worker_logger.error(f"Worker failed: {e}")
        traceback.print_exc()
        # Try to shutdown analytics even on error (may not be initialized)
        try:
            if analytics is not None:
                analytics.shutdown()
        except Exception:
            pass


**Checking key and R2 details**

I see that r2_tar_key includes a fully qualified key, which is good — it means the user can find it by reading the supabase videos table. They want to check R2 for upload locations, so I can query R2 directly later for that information. But wait, there seems to be a mismatch with PANNs; I should look into where load_panns is in the ModelManager. I'll search for `load_panns` to clarify.


rg -n "def load_panns|load_panns\(" -n src/models.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:
285:    def load_panns(self, device: str = 'cuda'):


Output:
Exit code: 0
Wall time: 0 seconds
Output:
        logger.info("=" * 70)

    def clear_cache(self, aggressive=False):
        """
        Clear GPU cache between operations.

        Args:
            aggressive: If True, also empty reserved memory and run gc
        """
        if torch.cuda.is_available():
            if aggressive:
                gc.collect()
                torch.cuda.empty_cache()
                torch.cuda.reset_peak_memory_stats()
            else:
                torch.cuda.empty_cache()

    def get_vram_free_gb(self) -> float:
        """Get current free VRAM in GB."""
        if torch.cuda.is_available():
            total = torch.cuda.get_device_properties(0).total_memory
            reserved = torch.cuda.memory_reserved(0)
            return (total - reserved) / (1024**3)
        return 0.0

    def load_panns(self, device: str = 'cuda'):
        """
        Load PANNs CNN14 model for music detection (lazy loading).

        === v7.4: Pre-baked model support ===
        If PANNS_MODEL_PATH env var is set, loads from that path (Docker pre-baked).
        Otherwise downloads from default location.

        Args:
            device: 'cuda' or 'cpu'

        Returns:
            Loaded AudioTagging model
        """
        if self._panns_loaded and self.panns_model is not None:
            logger.info("✅ PANNs CNN14 already loaded (hot inference)")
            return self.panns_model

        logger.info("Loading PANNs CNN14 for music detection...")
        start = time.time()

        try:
            # panns-inference builds the model output layer size from the
            # AudioSet label CSV (classes_num=len(labels)). If that CSV is
            # missing or partially downloaded, the class count can be wrong
            # (e.g. 235), causing a shape-mismatch when loading the 527-class
            # CNN14 checkpoint. Ensure the labels CSV is present and valid
            # before importing panns_inference (which otherwise tries to fetch
            # it via `wget` without validation).
            from pathlib import Path
            from urllib.request import urlopen

            labels_path = Path.home() / "panns_data" / "class_labels_indices.csv"
            expected_header = "index,mid,display_name"
            expected_min_lines = 528  # header + 527 classes

            def labels_ok(path: Path) -> bool:
                try:
                    lines = path.read_text(encoding="utf-8", errors="replace").splitlines()
                    return len(lines) >= expected_min_lines and lines[0].strip() == expected_header
                except Exception:
                    return False

            if not labels_ok(labels_path):
                try:
                    if labels_path.exists():
                        labels_path.unlink()
                except Exception:
                    pass

                labels_path.parent.mkdir(parents=True, exist_ok=True)
                labels_url = (
                    "https://storage.googleapis.com/us_audioset/youtube_corpus/v1/csv/"
                    "class_labels_indices.csv"
                )

                last_err: Exception | None = None
                for attempt in range(1, 6):
                    try:
                        with urlopen(labels_url, timeout=30) as resp:
                            body = resp.read()

                        tmp_path = labels_path.with_suffix(f".tmp.{os.getpid()}")
                        tmp_path.write_bytes(body)

                        if not labels_ok(tmp_path):
                            raise ValueError(
                                f"Downloaded AudioSet labels look invalid: {tmp_path}"
                            )

                        tmp_path.replace(labels_path)
                        logger.info(f"   ✅ AudioSet labels ready: {labels_path}")
                        last_err = None
                        break
                    except Exception as e:
                        last_err = e


Output:
Exit code: 0
Wall time: 0 seconds
Output:
                        last_err = e
                        sleep_s = min(30, 2**attempt)
                        logger.warning(
                            f"   ⚠ Failed to fetch AudioSet labels (attempt {attempt}/5): {e}. "
                            f"Retrying in {sleep_s}s..."
                        )
                        time.sleep(sleep_s)

                if last_err is not None:
                    raise RuntimeError(
                        "Unable to download valid AudioSet labels required by panns-inference. "
                        "This would cause CNN14 checkpoint load failures."
                    ) from last_err

            from panns_inference import AudioTagging

            # Determine device
            if device == 'cuda' and not torch.cuda.is_available():
                device = 'cpu'
                logger.warning("CUDA not available, using CPU for PANNs")

            # === v7.4: Check for pre-baked model path (Docker) ===
            checkpoint_path = os.environ.get('PANNS_MODEL_PATH')
            if checkpoint_path and os.path.exists(checkpoint_path):
                logger.info(f"   Using pre-baked model: {checkpoint_path}")
                self.panns_model = AudioTagging(checkpoint_path=checkpoint_path, device=device)
            else:
                # Default: downloads model on first use
                self.panns_model = AudioTagging(device=device)

            self._panns_loaded = True
            elapsed = time.time() - start
            logger.info(f"✅ PANNs CNN14 loaded on {device} in {elapsed:.1f}s")

            return self.panns_model

        except ImportError as e:
            logger.error(f"❌ PANNs not installed. Run: pip install panns-inference")
            raise ImportError("panns-inference package required for music detection. "
                            "Install with: pip install panns-inference") from e
        except Exception as e:
            logger.error(f"❌ Failed to load PANNs: {e}")
            raise

# Global singleton instance
MODELS = ModelManager()


**Checking Python installation**

It seems that Python isn't found, but there's Python 3 in the container. I think I can use Python 3 to inspect the package. I plan to run a command with `python3 -c`, but there’s a chance it could fail if the package isn't installed properly. First, I'll check the Python 3 version with `python3 --version` to see what's available. This could clarify if I can proceed or not!


Output:
Exit code: 0
Wall time: 0 seconds
Output:
Python 3.10.12


**Checking Python environment**

I see that Python 3 is available, which is great! I want to check if panns-inference is installed in the current environment. So, I'll run a command to import it and inspect its signature. I'll specifically look at the `AudioTagging` function to see what it requires. This will help me understand how to proceed, ensuring everything is set up correctly before I dive deeper into using the library. Let's see what it shows!


python3 - <<'PY'
import importlib, inspect
m = importlib.import_module('panns_inference')
print('panns_inference', m.__file__)
from panns_inference import AudioTagging
print('AudioTagging sig:', inspect.signature(AudioTagging))
print('AudioTagging init sig:', inspect.signature(AudioTagging.__init__))
PY


Output:
Exit code: 0
Wall time: 2.1 seconds
Output:
/home/ubuntu/.local/lib/python3.10/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
panns_inference /home/ubuntu/.local/lib/python3.10/site-packages/panns_inference/__init__.py
AudioTagging sig: (model=None, checkpoint_path=None, device='cuda')
AudioTagging init sig: (self, model=None, checkpoint_path=None, device='cuda')