Instruction:
## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  3) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  4) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.


<permissions instructions>Filesystem sandboxing defines which files can be read or written. `sandbox_mode` is `workspace-write`: The sandbox permits reading files, and editing files in `cwd` and `writable_roots`. Editing files in other directories requires approval. Network access is restricted. Approvals are your mechanism to get user consent to run shell commands without the sandbox. `approval_policy` is `on-request`: Commands will be run in the sandbox by default, and you can specify in your tool call if you want to escalate a command to run without sandboxing. If the completing the task requires escalated permissions, Do not let these settings or the sandbox deter you from attempting to accomplish the user's task.

Here are scenarios where you'll need to request approval:
- You need to run a command that writes to a directory that requires it (e.g. running tests that write to /var)
- You need to run a GUI app (e.g., open/xdg-open/osascript) to open browsers or files.
- You are running sandboxed and need to run a command that requires network access (e.g. installing packages)
- If you run a command that is important to solving the user's query, but it fails because of sandboxing, rerun the command with approval. ALWAYS proceed to use the `sandbox_permissions` and `justification` parameters - do not message the user before requesting approval for the command.
- You are about to take a potentially destructive action such as an `rm` or `git reset` that the user did not explicitly ask for.

When requesting approval to execute a command that will require escalated privileges:
  - Provide the `sandbox_permissions` parameter with the value `"require_escalated"`
  - Include a short, 1 sentence explanation for why you need escalated permissions in the justification parameter The writable roots are `/home/ubuntu/maya3_prepare`, `/tmp`.</permissions instructions>


# AGENTS.md instructions for /home/ubuntu/maya3_prepare

<INSTRUCTIONS>
## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  3) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  4) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.
</INSTRUCTIONS>


<environment_context>
  <cwd>/home/ubuntu/maya3_prepare</cwd>
  <shell>bash</shell>
</environment_context>


<permissions instructions>Filesystem sandboxing defines which files can be read or written. `sandbox_mode` is `danger-full-access`: No filesystem sandboxing - all commands are permitted. Network access is enabled. Approvals are your mechanism to get user consent to run shell commands without the sandbox. `approval_policy` is `never`: This is a non-interactive mode where you may NEVER ask the user for approval to run commands. Instead, you must always persist and work around constraints to solve the task for the user. You MUST do your utmost best to finish the task and validate your work before yielding. If this mode is paired with `danger-full-access`, take advantage of it to deliver the best outcome for the user. Further, in this mode, your default testing philosophy is overridden: Even if you don't see local patterns for testing, you may add tests and scripts to validate your work. Just remove them before yielding.</permissions instructions>


# Context from my IDE setup:

## Open tabs:
- optimisations.md: optimisations.md
- Optimisation.md: Optimisation.md
- prepare_shards.py: scripts/prepare_shards.py
- fast_audio_pipeline_v2.py: scripts/fast_audio_pipeline_v2.py
- benchmark_gpu.py: scripts/benchmark_gpu.py

## My request for Codex:

Current performance. - Performance Analysis Complete

  Key Findings
  ┌─────────────────────┬───────────────────────────────────────────┐
  │       Metric        │                   Value                   │
  ├─────────────────────┼───────────────────────────────────────────┤
  │ Measured Throughput │ 26.1 seg/s, 7.56 videos/min               │
  ├─────────────────────┼───────────────────────────────────────────┤
  │ GPU Utilization     │ <5% (only resampling uses GPU)            │
  ├─────────────────────┼───────────────────────────────────────────┤
  │ Primary Bottleneck  │ pyloudnorm LUFS computation (45% of time) │
  ├─────────────────────┼───────────────────────────────────────────┤
  │ GPU Benefit         │ ~2% improvement over CPU-only             │
  └─────────────────────┴───────────────────────────────────────────┘
  Per-Segment Breakdown

  FLAC Decode:     17.9 ms (31.0%)
  Metrics (LUFS):  26.1 ms (45.3%)  ← BOTTLENECK
  GPU Resample:     0.5 ms (0.9%)   ← GPU barely used
  WAV Encode:      13.1 ms (22.8%)

  Optimal A100 Configuration

  --gpu-batch-size 128 --gpu-decode-workers 12 --gpu-encode-workers 12

  This yields ~26-30 seg/s. Larger batch sizes and more workers don't help due to the sequential nature of metrics
   computation.

  Scaling Projections (500k videos)
  ┌────────┬──────────────┬─────────┬───────────┐
  │ Nodes  │   Instance   │  Time   │ Est. Cost │
  ├────────┼──────────────┼─────────┼───────────┤
  │ 1 A100 │ p4d.24xlarge │ 46 days │ ~$3,850   │
  ├────────┼──────────────┼─────────┼───────────┤
  │ 8 CPU  │ c6i.4xlarge  │ 6 days  │ ~$780     │
  └────────┴──────────────┴─────────┴───────────┘
  Recommendation

  The A100 GPU is overkill for this workload. The bottleneck is CPU-bound (pyloudnorm LUFS), and GPU resampling
  provides negligible benefit.

  For cost efficiency:
  - Use c6i.4xlarge (16 vCPUs, $0.68/hr) instead of A100
  - Scale horizontally with 4-8 nodes for faster completion
  - Expected 5× cost reduction with similar or better total throughput

New branch git optimisations. for new approach. create this new optimisations and implement as per the plan or strategy and then do a brnchmark for 100 videos, 500 videos and provide scaling benchmarks for 500k videos with horizontal scaling into new instances suggestions.


Summary: auto

## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  3) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  4) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.


**Preparing to inspect repo for optimizations**


Output:
Exit code: 0
Wall time: 0 seconds
Output:
docs
objective.md
optimisations.md
scripts
strategy.md
Summary.md
tts
X-Codec-2.0


Summary: auto


**Preparing to check git status**


Output:
Exit code: 0
Wall time: 0 seconds
Output:
## optimisations
?? optimisations.md


**Checking duplicate optimisation files**


Output:
Exit code: 0
Wall time: 0 seconds
Output:
inworld_codec.txt
inworld.pdf


**Checking for optimization documentation**


Output:
Exit code: 0
Wall time: 0 seconds
Output:
.
..
.claude
docs
.env
.git
objective.md
optimisations.md
scripts
strategy.md
Summary.md
tts
X-Codec-2.0


**Searching for missing documentation**

I'm looking for the missing Optimisation.md file mentioned in open tabs by searching the entire repo, including subdirectories, to locate it.


Output:
Exit code: 1
Wall time: 0 seconds
Output:


**Opening documentation to understand plan**


Output:
Exit code: 0
Wall time: 0 seconds
Output:
# Audio Pipeline Optimization Strategy

## Target: 500k Videos in <24 Hours

### Current State
```
Performance:     26.1 seg/s, 7.56 videos/min
Bottleneck:      pyloudnorm LUFS computation (45% of time)
GPU Utilization: <5% (only resampling)
Time for 500k:   ~46 days
```

### Target State
```
Required:        500k videos / 24h = 347 videos/min
Speedup needed:  347 / 7.56 = 46x
```

---

## Strategy Overview

We'll achieve this through **3 parallel optimization tracks**:

| Track | Optimization | Expected Speedup | Cost Impact |
|-------|--------------|------------------|-------------|
| **A** | GPU-accelerated LUFS + metrics | 4-6x | Same hardware |
| **B** | Pipeline parallelization | 2-3x | Same hardware |
| **C** | Horizontal scaling | 8-16x | Linear scaling |

**Combined target: 48-288x speedup → 500k in 4-24 hours**

---

## Track A: GPU-Accelerated Audio Metrics

### The Problem with pyloudnorm

pyloudnorm is pure Python and computes LUFS sequentially:
1. K-weighting filter (2x IIR biquad cascades)
2. Mean square computation
3. Gating (400ms blocks with 75% overlap)
4. Integration

**Solution: GPU-native LUFS with torchaudio**

```python
# Current: 26.1ms per segment (sequential, CPU)
# Target:  0.5ms per segment (batched, GPU)
```

### GPU LUFS Implementation

The EBU R128 LUFS algorithm can be fully parallelized:

1. **K-weighting filters** → `torchaudio.functional.biquad` (batched)
2. **Block mean square** → `torch.unfold` + batched mean
3. **Gating** → Vectorized threshold comparison
4. **Integration** → Batched reduction

### Performance Projection

| Operation | CPU (pyloudnorm) | GPU (batched 128) | Speedup |
|-----------|------------------|-------------------|---------|
| LUFS | 26.1ms | ~0.8ms | 32x |
| Spectral | 8ms | ~0.3ms | 26x |
| Resample | 12ms | 0.5ms | 24x |
| **Total** | 57.6ms | ~5ms | **11x** |

---

## Track B: Pipeline Architecture Optimization

### Current Architecture (Sequential)
```
Download → Decode → Metrics → Chunk → Resample → Encode → Write
           [------------ Sequential per segment ------------]
```

### Optimized Architecture (Pipelined + Batched)
```
┌─────────────────────────────────────────────────────────────┐
│                    PRODUCER PROCESSES                        │
│  Download Pool (16 workers) → Decode Pool (8 workers)       │
│            ↓                         ↓                       │
│        [Download Queue]         [Segment Queue]              │
└─────────────────────────────────────────────────────────────┘
                              ↓
┌─────────────────────────────────────────────────────────────┐
│                    GPU BATCH PROCESSOR                       │
│  Batch Accumulator (128 segments) → GPU Kernel              │
│  - K-weight filter (batched)                                 │
│  - LUFS computation (batched)                                │
│  - Spectral analysis (batched FFT)                           │
│  - Resampling (batched)                                      │
└─────────────────────────────────────────────────────────────┘
                              ↓
┌─────────────────────────────────────────────────────────────┐
│                   CONSUMER PROCESSES                         │
│  Encode Pool (12 workers) → Shard Writers (4 workers)       │
│            ↓                         ↓                       │
│       [Output Queue]           [Upload Queue]                │
└─────────────────────────────────────────────────────────────┘
```

### Key Optimizations

1. **Async I/O for downloads** - Don't block GPU on network
2. **Pre-fetch next batch** - GPU never waits for data
3. **Streaming encode** - Encode while GPU processes next batch
4. **Batched shard writes** - Reduce filesystem syscalls

---

## Track C: Horizontal Scaling Strategy

### Compute Requirements

For 500k videos in 24 hours with optimized code (11x speedup):

```
Single node: 7.56 × 11 = 83 videos/min
Required:    347 videos/min
Nodes needed: ceil(347 / 83) = 5 nodes
Safety margin: 8 nodes (2.3x buffer)
```

### Recommended Instance Types

| Provider | Instance | vCPUs | GPU | Memory | Cost/hr |
|----------|----------|-------|-----|--------|---------|
| AWS | g5.4xlarge | 16 | A10G 24GB | 64GB | ~$1.00 |
| AWS | g5.8xlarge | 32 | A10G 24GB | 128GB | ~$2.00 |
| AWS | p4d.24xlarge | 96 | 8×A100 | 1.5TB | ~$32.77 |
| GCP | g2-standard-16 | 16 | L4 24GB | 64GB | ~$0.80 |

**Recommendation: 8× g5.4xlarge ($8/hr total) or 2× g5.8xlarge with better CPU/GPU ratio**

### Partitioning Strategy

```sql
-- Each worker gets deterministic partition
SELECT youtube_id, r2_tar_key
FROM videos
WHERE language = $1
  AND status = 'COMPLETED'
  AND mod(abs(hashtext(youtube_id)), $partition_count) = $partition_index
```

This ensures:
- No overlap between workers
- Deterministic assignment (restart-safe)
- Even distribution

---

## Monitoring & Orchestration

### Progress Tracking Schema (Supabase)

```sql
CREATE TABLE pipeline_runs (
  run_id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  language TEXT NOT NULL,
  total_videos INT NOT NULL,
  started_at TIMESTAMPTZ DEFAULT NOW(),
  completed_at TIMESTAMPTZ,
  status TEXT DEFAULT 'running'
);

CREATE TABLE worker_progress (
  worker_id TEXT PRIMARY KEY,
  run_id UUID REFERENCES pipeline_runs(run_id),
  partition_index INT NOT NULL,
  partition_count INT NOT NULL,
  hostname TEXT,
  instance_type TEXT,

  -- Progress counters
  videos_total INT DEFAULT 0,
  videos_processed INT DEFAULT 0,
  segments_total INT DEFAULT 0,
  segments_processed INT DEFAULT 0,
  segments_dropped INT DEFAULT 0,

  -- Performance metrics
  videos_per_minute FLOAT DEFAULT 0,
  segments_per_second FLOAT DEFAULT 0,
  gpu_utilization FLOAT DEFAULT 0,

  -- Status
  status TEXT DEFAULT 'starting',
  last_heartbeat TIMESTAMPTZ DEFAULT NOW(),
  last_error TEXT,
  error_count INT DEFAULT 0,

  started_at TIMESTAMPTZ DEFAULT NOW(),
  updated_at TIMESTAMPTZ DEFAULT NOW()
);

CREATE TABLE processing_errors (
  id SERIAL PRIMARY KEY,
  worker_id TEXT,
  run_id UUID,
  video_id TEXT,
  error_type TEXT,
  error_message TEXT,
  stack_trace TEXT,
  created_at TIMESTAMPTZ DEFAULT NOW()
);

-- Real-time aggregation view
CREATE VIEW run_progress AS
SELECT
  r.run_id,
  r.language,
  r.total_videos,
  SUM(w.videos_processed) as processed,
  SUM(w.videos_total) as assigned,
  AVG(w.videos_per_minute) as avg_rate,
  SUM(w.videos_per_minute) as total_rate,
  COUNT(*) FILTER (WHERE w.status = 'running') as active_workers,
  COUNT(*) FILTER (WHERE w.status = 'failed') as failed_workers,
  EXTRACT(EPOCH FROM (NOW() - r.started_at)) / 3600 as hours_elapsed,
  (r.total_videos - SUM(w.videos_processed)) / NULLIF(SUM(w.videos_per_minute), 0) / 60 as hours_remaining
FROM pipeline_runs r
LEFT JOIN worker_progress w ON w.run_id = r.run_id
GROUP BY r.run_id;
```

### Worker Heartbeat & Monitoring

```python
class ProgressReporter:
    def __init__(self, supabase_url: str, supabase_key: str, run_id: str, worker_id: str):
        self.client = create_client(supabase_url, supabase_key)
        self.run_id = run_id
        self.worker_id = worker_id
        self.last_report = time.time()
        self.report_interval = 10  # seconds

    def update(self, videos_processed: int, segments_processed: int,
               videos_per_minute: float, segments_per_second: float):
        now = time.time()
        if now - self.last_report < self.report_interval:
            return

        self.client.table('worker_progress').upsert({
            'worker_id': self.worker_id,
            'run_id': self.run_id,
            'videos_processed': videos_processed,
            'segments_processed': segments_processed,
            'videos_per_minute': videos_per_minute,
            'segments_per_second': segments_per_second,
            'last_heartbeat': datetime.utcnow().isoformat(),
            'status': 'running'
        }).execute()
        self.last_report = now
```

---

## Implementation Phases

### Phase 1: GPU-Accelerated Metrics (Day 1)
- [ ] Implement GPU LUFS computation
- [ ] Implement GPU spectral analysis
- [ ] Batch processing with async I/O
- [ ] Benchmark: Target 80+ seg/s single GPU

### Phase 2: Horizontal Scaling Infrastructure (Day 1)
- [ ] Supabase schema setup
- [ ] Worker orchestration script
- [ ] Progress monitoring dashboard
- [ ] Auto-recovery on failure

### Phase 3: Deployment & Execution (Day 2)
- [ ] Deploy to 8 GPU instances
- [ ] Start processing all 12 languages
- [ ] Monitor and adjust
- [ ] Handle failures and retries

---

## Cost Estimate (24-hour run)

| Component | Units | Cost/hr | Hours | Total |
|-----------|-------|---------|-------|-------|
| g5.4xlarge | 8 | $1.00 | 24 | $192 |
| R2 egress | ~5TB | - | - | ~$0 (free) |
| Supabase | Pro | - | - | $25/mo |
| **Total** | | | | **~$220** |

---

## Quick Start

```bash
# 1. Deploy infrastructure
./deploy/setup_workers.sh 8  # Spin up 8 GPU instances

# 2. Start orchestrator
python orchestrator.py --language all --workers 8 --run-id $(uuidgen)

# 3. Monitor progress
python monitor.py --run-id <run_id>  # Real-time dashboard
```

---

## Files to Create

1. `gpu_audio_metrics.py` - GPU-accelerated LUFS and spectral analysis
2. `optimized_pipeline.py` - Pipelined batch processor
3. `orchestrator.py` - Multi-node job orchestration
4. `monitor.py` - Real-time progress dashboard
5. `deploy/` - Infrastructure as code


**Starting optimization implementation review**

I'm identifying relevant existing code in scripts to interpret and implement new optimizations before benchmarking large-scale video processing.


sed -n '1,220p' scripts/prepare_shards.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:
from __future__ import annotations

import argparse
import io
import json
import math
import os
import random
import tarfile
import threading
import time
import zlib
from collections import defaultdict
from contextlib import contextmanager
from dataclasses import dataclass
from pathlib import Path
from typing import Any, Optional

import boto3
import numpy as np
import psycopg
import soundfile as sf
from boto3.s3.transfer import TransferConfig
from concurrent.futures import FIRST_COMPLETED, ThreadPoolExecutor, as_completed, wait
from scipy import signal
from tqdm import tqdm

try:
    import pyloudnorm as pyln
except Exception:  # pragma: no cover - optional dependency
    pyln = None

# GPU pipeline (optional)
try:
    from fast_audio_pipeline import EquivalentBatchProcessor, SegmentInput, OutputSegment
    HAS_GPU_PIPELINE = True
except ImportError:
    HAS_GPU_PIPELINE = False
    EquivalentBatchProcessor = None
    SegmentInput = None
    OutputSegment = None

ENV_PATH = Path(".env")
LANGUAGES = [
    "telugu",
    "malayalam",
    "english",
    "hindi",
    "punjabi",
    "tamil",
    "kannada",
    "gujarati",
    "bengali",
    "odia",
    "marathi",
    "assamese",
]

# Global timing stats (thread-safe)
_timing_lock = threading.Lock()
_timing_stats: dict[str, list[float]] = defaultdict(list)
_timing_enabled = False

@contextmanager
def timer(name: str):
    """Context manager for timing code blocks."""
    if not _timing_enabled:
        yield
        return
    start = time.perf_counter()
    try:
        yield
    finally:
        elapsed = time.perf_counter() - start
        with _timing_lock:
            _timing_stats[name].append(elapsed)

def print_timing_stats():
    """Print aggregated timing statistics."""
    if not _timing_stats:
        return
    print("\n=== Timing Statistics ===")
    total_time = sum(sum(times) for times in _timing_stats.values())
    for name, times in sorted(_timing_stats.items()):
        count = len(times)
        total = sum(times)
        avg = total / count if count > 0 else 0
        percent = (total / total_time * 100) if total_time > 0 else 0
        print(f"{name:35s}: {count:6d} calls, {total:8.2f}s total, {avg*1000:8.2f}ms avg, {percent:5.1f}%")
    print(f"{'TOTAL':35s}:              {total_time:8.2f}s")
    print("=" * 80)

_window_cache: dict[int, np.ndarray] = {}
_freq_cache: dict[tuple[int, int], np.ndarray] = {}
_cache_lock = threading.Lock()
_lufs_local = threading.local()

def get_hann_window(length: int) -> np.ndarray:
    window = _window_cache.get(length)
    if window is not None:
        return window
    with _cache_lock:
        window = _window_cache.get(length)
        if window is None:
            window = np.hanning(length)
            _window_cache[length] = window
    return window

def get_rfftfreq(length: int, sr: int) -> np.ndarray:
    key = (length, sr)
    freqs = _freq_cache.get(key)
    if freqs is not None:
        return freqs
    with _cache_lock:
        freqs = _freq_cache.get(key)
        if freqs is None:
            freqs = np.fft.rfftfreq(length, d=1.0 / sr)
            _freq_cache[key] = freqs
    return freqs

def get_lufs_meter(sr: int) -> Optional[Any]:
    if pyln is None:
        return None
    meters = getattr(_lufs_local, "meters", None)
    if meters is None:
        meters = {}
        _lufs_local.meters = meters
    meter = meters.get(sr)
    if meter is None:
        meter = pyln.Meter(sr)
        meters[sr] = meter
    return meter

@dataclass(frozen=True)
class VideoRecord:
    youtube_id: str
    r2_tar_key: str

@dataclass(frozen=True)
class SegmentOutput:
    sample_rate: int
    audio_bytes: bytes
    metadata: dict[str, Any]

@dataclass(frozen=True)
class ChunkConfig:
    enabled: bool
    min_sec: float
    max_sec: float
    threshold_sec: float
    all_buckets: bool

@dataclass(frozen=True)
class SilenceConfig:
    enabled: bool
    frame_sec: float
    hop_sec: float
    search_sec: float
    percentile: float
    db_drop: float

def load_env(env_path: Path) -> dict[str, str]:
    env: dict[str, str] = {}
    for line in env_path.read_text().splitlines():
        stripped = line.strip()
        if not stripped or stripped.startswith("#") or "=" not in stripped:
            continue
        if stripped.startswith("export "):
            stripped = stripped[len("export ") :]
        key, value = stripped.split("=", 1)
        env[key.strip()] = value.strip()
    return env

def make_r2_client(env: dict[str, str]):
    endpoint = env.get("R2_ENDPOINT_URL")
    access_key = env.get("R2_ACCESS_KEY_ID")
    secret_key = env.get("R2_SECRET_ACCESS_KEY")
    if not endpoint or not access_key or not secret_key:
        raise SystemExit("Missing R2 credentials in .env")
    return boto3.client(
        "s3",
        endpoint_url=endpoint,
        aws_access_key_id=access_key,
        aws_secret_access_key=secret_key,
    )

def resolve_r2_key(
    s3, bucket: str, r2_key: str, prefix: Optional[str]
) -> Optional[str]:
    if not r2_key:
        return None
    key = r2_key.lstrip("/")
    candidates = [key]

    if prefix:
        cleaned_prefix = prefix.strip("/")
        if not key.startswith(cleaned_prefix):
            candidates.append(f"{cleaned_prefix}/{key}")

    base = os.path.basename(key)
    if base != key:
        candidates.append(base)
        if prefix:
            candidates.append(f"{prefix.strip('/')}/{base}")

    for candidate in candidates:
        try:


**Switching to audio pipeline file**


sed -n '1,220p' scripts/fast_audio_pipeline_v2.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:
"""GPU-accelerated audio processing v2 - Fully parallelized.

Key optimization: Parallelize the ENTIRE segment processing chain, not just decode.

Benchmark showed:
- Sequential metrics: 25 seg/s
- Parallel metrics (8-12 workers): 87.5 seg/s

This version parallelizes:
1. FLAC decoding
2. Metrics computation (pyloudnorm LUFS releases GIL)
3. GPU resampling (batched)
4. WAV encoding

Expected throughput: 60-80 seg/s on A100 with 16 vCPUs.
"""

from __future__ import annotations

import io
import math
import random
import threading
from concurrent.futures import ThreadPoolExecutor, as_completed
from dataclasses import dataclass
from typing import Any, Optional

import numpy as np
import soundfile as sf

from prepare_shards import (
    ChunkConfig,
    SilenceConfig,
    SegmentOutput,
    make_chunk_windows,
    compute_audio_metrics,
    compute_quality_score,
    duration_bucket,
    classify_native_bucket,
)

try:
    import torch
    import torchaudio
    HAS_TORCH = True
except ImportError:
    HAS_TORCH = False
    torch = None
    torchaudio = None

@dataclass
class SegmentInput:
    """Input segment for processing."""
    video_id: str
    language: str
    segment_bytes: bytes
    segment_meta: dict[str, Any]
    speaker: str
    index: int
    is_val: bool
    rng_seed: int

@dataclass
class OutputSegment:
    """Final output ready for shard writing."""
    sample_rate: int
    audio_bytes: bytes
    metadata: dict[str, Any]
    is_val: bool

class GPUResampler:
    """Thread-safe GPU resampler with batching."""

    def __init__(self, device: str = "cuda"):
        self.use_gpu = HAS_TORCH and torch.cuda.is_available()
        self.device = torch.device(device if self.use_gpu else "cpu")
        self._resamplers: dict[tuple[int, int], Any] = {}
        self._lock = threading.Lock()

    def resample(self, audio: np.ndarray, src_sr: int, dst_sr: int) -> np.ndarray:
        """Resample audio using GPU if available."""
        if src_sr == dst_sr:
            return audio

        if not self.use_gpu:
            from scipy import signal
            gcd = math.gcd(src_sr, dst_sr)
            up = dst_sr // gcd
            down = src_sr // gcd
            return signal.resample_poly(audio, up, down).astype(np.float32, copy=False)

        resampler = self._get_resampler(src_sr, dst_sr)
        audio_t = torch.from_numpy(audio).unsqueeze(0).to(self.device)
        with torch.no_grad():
            resampled_t = resampler(audio_t)
        return resampled_t.squeeze(0).cpu().numpy()

    def _get_resampler(self, src_sr: int, dst_sr: int):
        key = (src_sr, dst_sr)
        if key not in self._resamplers:
            with self._lock:
                if key not in self._resamplers:
                    self._resamplers[key] = torchaudio.transforms.Resample(
                        src_sr, dst_sr, dtype=torch.float32
                    ).to(self.device)
        return self._resamplers[key]

class ParallelBatchProcessor:
    """Fully parallelized batch processor.

    Unlike EquivalentBatchProcessor which processes segments sequentially after decode,
    this version parallelizes the entire processing chain for maximum throughput.
    """

    def __init__(
        self,
        process_workers: int = 12,
        output_format: str = "wav",
        chunk_cfg: Optional[ChunkConfig] = None,
        silence_cfg: Optional[SilenceConfig] = None,
    ):
        # Use single worker pool for entire processing (decode + metrics + resample + encode)
        self.process_workers = process_workers
        self.output_format = output_format
        self.chunk_cfg = chunk_cfg or ChunkConfig(
            enabled=True, min_sec=6.0, max_sec=10.0, threshold_sec=12.0, all_buckets=False
        )
        self.silence_cfg = silence_cfg or SilenceConfig(
            enabled=True, frame_sec=0.05, hop_sec=0.01,
            search_sec=0.5, percentile=20.0, db_drop=20.0
        )

        # Shared GPU resampler (thread-safe)
        self.resampler = GPUResampler()

        self.stats = {
            "decoded": 0,
            "chunks_emitted": 0,
            "segments_chunked": 0,
            "dropped_too_short": 0,
            "dropped_too_long": 0,
            "dropped_clipped": 0,
        }
        self._stats_lock = threading.Lock()

    def process_segments(self, segments: list[SegmentInput]) -> list[OutputSegment]:
        """Process segments with full parallelization."""
        if not segments:
            return []

        all_outputs: list[OutputSegment] = []

        # Process ALL segments in parallel (decode + metrics + resample + encode)
        with ThreadPoolExecutor(max_workers=self.process_workers) as pool:
            futures = [pool.submit(self._process_single_segment, seg) for seg in segments]

            for future in as_completed(futures):
                try:
                    outputs, local_stats = future.result()
                    all_outputs.extend(outputs)

                    # Update stats thread-safely
                    with self._stats_lock:
                        self.stats["decoded"] += 1
                        for key in local_stats:
                            self.stats[key] += local_stats[key]
                except Exception as e:
                    print(f"Segment processing error: {e}")

        return all_outputs

    def _process_single_segment(
        self, seg: SegmentInput
    ) -> tuple[list[OutputSegment], dict[str, int]]:
        """Process a single segment end-to-end (runs in worker thread)."""
        outputs: list[OutputSegment] = []
        local_stats = {
            "chunks_emitted": 0,
            "segments_chunked": 0,
            "dropped_too_short": 0,
            "dropped_too_long": 0,
            "dropped_clipped": 0,
        }

        # Decode
        try:
            audio, sr = sf.read(io.BytesIO(seg.segment_bytes), dtype="float32")
            if audio.ndim > 1:
                audio = np.mean(audio, axis=1)
            audio = audio.astype(np.float32)
        except Exception:
            return outputs, local_stats

        # Chunk windows
        rng = random.Random(seg.rng_seed)
        windows = make_chunk_windows(audio, sr, self.chunk_cfg, self.silence_cfg, rng)

        music_stats = seg.segment_meta.get("music_stats") or {}
        has_music = bool(music_stats.get("has_music", False))

        sample_id_base = f"{seg.video_id}__{seg.speaker}_{seg.index:04d}"
        use_chunk_suffix = len(windows) > 1
        if use_chunk_suffix:
            local_stats["segments_chunked"] = 1

        segment_start = seg.segment_meta.get("start")
        segment_end = seg.segment_meta.get("end")
        original_start = seg.segment_meta.get("original_start")
        original_end = seg.segment_meta.get("original_end")

        # Process 16k outputs for chunk windows
        for chunk_idx, (offset_sec, chunk_len_sec) in enumerate(windows):
            start_idx = int(round(offset_sec * sr))
            end_idx = int(round((offset_sec + chunk_len_sec) * sr))
            audio_slice = audio[start_idx:end_idx]
            if audio_slice.size == 0:


**Searching for audio metrics usage**

I'm verifying if the pipeline still uses pyloudnorm by checking compute_audio_metrics in prepare_shards and will search for all metric references to confirm.


rg -n "compute_audio_metrics|lufs|pyloudnorm" scripts/prepare_shards.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:
29:    import pyloudnorm as pyln
99:_lufs_local = threading.local()
127:def get_lufs_meter(sr: int) -> Optional[Any]:
130:    meters = getattr(_lufs_local, "meters", None)
133:        _lufs_local.meters = meters
318:def compute_audio_metrics(audio: np.ndarray, sr: int) -> dict[str, float]:
328:    with timer("audio_metrics_lufs"):
329:        loudness_lufs = -100.0
332:                meter = get_lufs_meter(sr)
334:                    loudness_lufs = float(meter.integrated_loudness(audio))
336:                loudness_lufs = -100.0
341:        "loudness_lufs": loudness_lufs,
357:        + 0.2 * normalize(metrics["loudness_lufs"], -40, -14)
648:                "loudness_lufs": metrics["loudness_lufs"],
692:            slice_metrics = compute_audio_metrics(audio_slice, sr)
725:        full_metrics = compute_audio_metrics(audio, sr)
744:                    slice_metrics = compute_audio_metrics(audio_slice, sr)


sed -n '300,430p' scripts/prepare_shards.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:

def classify_native_bucket(effective_nyquist_hz: float) -> str:
    """Classify audio bandwidth based on effective Nyquist frequency.

    These buckets represent actual audio bandwidth, not container sample rates.
    Thresholds are based on observed distribution in the dataset.
    """
    if effective_nyquist_hz < 8000:
        return "<16k"
    if effective_nyquist_hz < 12000:
        return "16-24k"
    if effective_nyquist_hz < 16000:
        return "24-32k"
    if effective_nyquist_hz < 22050:
        return "32-44k"
    return ">=44k"

def compute_audio_metrics(audio: np.ndarray, sr: int) -> dict[str, float]:
    with timer("audio_metrics_basic"):
        audio = audio.astype(np.float32, copy=False)
        rms = float(np.sqrt(np.mean(audio ** 2) + 1e-12))
        loudness_rms_db = 20.0 * math.log10(rms + 1e-12)
        clipping_fraction = float(np.mean(np.abs(audio) > 0.99))
        dc_offset = float(np.mean(audio))
        noise_floor = float(np.percentile(np.abs(audio), 10))
        snr_proxy = 20.0 * math.log10((rms + 1e-9) / (noise_floor + 1e-9))

    with timer("audio_metrics_lufs"):
        loudness_lufs = -100.0
        if pyln is not None:
            try:
                meter = get_lufs_meter(sr)
                if meter is not None:
                    loudness_lufs = float(meter.integrated_loudness(audio))
            except Exception:
                loudness_lufs = -100.0

    rolloff_hz, effective_nyquist_hz, high_freq_ratio = compute_spectral_metrics(audio, sr)

    return {
        "loudness_lufs": loudness_lufs,
        "loudness_rms_db": loudness_rms_db,
        "clipping_fraction": clipping_fraction,
        "dc_offset": dc_offset,
        "snr_proxy": snr_proxy,
        "spectral_rolloff_hz": rolloff_hz,
        "effective_nyquist_hz": effective_nyquist_hz,
        "high_freq_energy_ratio": high_freq_ratio,
    }

def compute_quality_score(metrics: dict[str, float], has_music: bool) -> float:
    score = (
        0.3 * (1.0 - metrics["clipping_fraction"])
        + 0.2 * normalize(metrics["snr_proxy"], 10, 40)
        + 0.2 * (1.0 if not has_music else 0.5)
        + 0.2 * normalize(metrics["loudness_lufs"], -40, -14)
        + 0.1 * (1.0 if abs(metrics["dc_offset"]) < 0.01 else 0.5)
    )
    return max(0.0, min(1.0, score))

def resample_audio(
    audio: np.ndarray,
    src_sr: int,
    dst_sr: int,
) -> np.ndarray:
    if src_sr == dst_sr:
        return audio
    with timer("cpu_resample"):
        gcd = math.gcd(src_sr, dst_sr)
        up = dst_sr // gcd
        down = src_sr // gcd
        return signal.resample_poly(audio, up, down).astype(np.float32, copy=False)

def encode_audio(audio: np.ndarray, sr: int, output_format: str) -> bytes:
    with timer("audio_encode"):
        buf = io.BytesIO()
        if output_format == "wav":
            sf.write(buf, audio, sr, format="WAV", subtype="PCM_16")
        else:
            sf.write(buf, audio, sr, format="FLAC", subtype="PCM_16")
        return buf.getvalue()

def duration_bucket(duration_sec: float) -> Optional[tuple[str, float]]:
    if duration_sec < 3.0:
        return None
    if duration_sec < 6.0:
        return "3-6s", 0.3
    if duration_sec < 12.0:
        return "6-12s", 1.0
    if duration_sec < 24.0:
        return "12-24s", 0.8
    if duration_sec <= 60.0:
        return "24-60s", 0.5
    return None

def compute_frame_rms(
    audio: np.ndarray, sr: int, frame_sec: float, hop_sec: float
) -> tuple[np.ndarray, np.ndarray]:
    frame_len = max(1, int(round(frame_sec * sr)))
    hop_len = max(1, int(round(hop_sec * sr)))
    if audio.size < frame_len:
        return np.array([], dtype=np.float32), np.array([], dtype=np.float32)

    if not audio.flags["C_CONTIGUOUS"]:
        audio = np.ascontiguousarray(audio)

    num_frames = 1 + (audio.size - frame_len) // hop_len
    if num_frames <= 0:
        return np.array([], dtype=np.float32), np.array([], dtype=np.float32)

    stride = audio.strides[0]
    frames = np.lib.stride_tricks.as_strided(
        audio,
        shape=(num_frames, frame_len),
        strides=(hop_len * stride, stride),
    )
    rms = np.sqrt(np.mean(frames * frames, axis=1) + 1e-12)
    times = (np.arange(num_frames, dtype=np.float32) * hop_len + (frame_len / 2)) / sr
    return times.astype(np.float32), rms.astype(np.float32)

def find_silence_boundary(
    times: np.ndarray,
    rms: np.ndarray,
    search_start: float,


**Evaluating audio metric computation options**

I'm noting compute_audio_metrics relies on pyloudnorm, which may not use GPU, and considering optimizing LUFS calculation using numpy vectorization or faster third-party tools; next, I'll inspect compute_spectral_metrics since it might be costly.


rg -n "def compute_spectral_metrics" -n scripts/prepare_shards.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:
276:def compute_spectral_metrics(audio: np.ndarray, sr: int) -> tuple[float, float, float]:


sed -n '240,320p' scripts/prepare_shards.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:
    language: str,
    limit: Optional[int],
    partition_count: int = 1,
    partition_index: int = 0,
) -> list[VideoRecord]:
    records: list[VideoRecord] = []
    with psycopg.connect(db_url) as conn:
        with conn.cursor() as cur:
            query = """
                select youtube_id, r2_tar_key
                from videos
                where lower(language) = %s
                  and r2_tar_key is not null
                  and status = 'COMPLETED'
            """
            params = [language.lower()]
            if partition_count > 1:
                query += " and mod(abs(hashtext(youtube_id)), %s) = %s"
                params.extend([partition_count, partition_index])
            query += " order by random()"
            if limit:
                query += " limit %s"
                params.append(limit)
            cur.execute(query, params)
            for youtube_id, r2_tar_key in cur.fetchall():
                if youtube_id and r2_tar_key:
                    records.append(VideoRecord(youtube_id, r2_tar_key))
    return records

def normalize(value: float, min_val: float, max_val: float) -> float:
    if max_val <= min_val:
        return 0.0
    return max(0.0, min(1.0, (value - min_val) / (max_val - min_val)))

def compute_spectral_metrics(audio: np.ndarray, sr: int) -> tuple[float, float, float]:
    if audio.size == 0:
        return 0.0, 0.0, 0.0
    with timer("cpu_spectral_compute"):
        max_samples = min(audio.size, sr * 5)
        window = get_hann_window(max_samples)
        chunk = audio[:max_samples] * window
        spec = np.fft.rfft(chunk)
        power = np.abs(spec) ** 2
        freqs = get_rfftfreq(max_samples, sr)
        total_energy = float(np.sum(power)) + 1e-12
        cumulative = np.cumsum(power)
        rolloff_idx = int(np.searchsorted(cumulative, 0.85 * cumulative[-1]))
        rolloff_hz = float(freqs[min(rolloff_idx, len(freqs) - 1)])

        power_db = 10.0 * np.log10(power + 1e-12)
        threshold = power_db.max() - 60.0
        above = np.where(power_db >= threshold)[0]
        effective_nyquist_hz = float(freqs[above[-1]]) if above.size > 0 else 0.0

        high_freq_energy = float(np.sum(power[freqs >= 8000.0]))
        high_freq_ratio = high_freq_energy / total_energy
    return rolloff_hz, effective_nyquist_hz, high_freq_ratio

def classify_native_bucket(effective_nyquist_hz: float) -> str:
    """Classify audio bandwidth based on effective Nyquist frequency.

    These buckets represent actual audio bandwidth, not container sample rates.
    Thresholds are based on observed distribution in the dataset.
    """
    if effective_nyquist_hz < 8000:
        return "<16k"
    if effective_nyquist_hz < 12000:
        return "16-24k"
    if effective_nyquist_hz < 16000:
        return "24-32k"
    if effective_nyquist_hz < 22050:
        return "32-44k"
    return ">=44k"

def compute_audio_metrics(audio: np.ndarray, sr: int) -> dict[str, float]:
    with timer("audio_metrics_basic"):
        audio = audio.astype(np.float32, copy=False)


**Identifying audio pipeline options**

Noted pyloudnorm is a bottleneck; now exploring optional fast_audio_pipeline_v2 and fast_audio_pipeline.py to assess alternatives for prepare_shards.


Output:
Exit code: 0
Wall time: 0 seconds
Output:
scripts/fetch_languages.py
scripts/fill_missing_languages.py
scripts/prepare_shards.py
scripts/benchmark_results.json
scripts/supabase_summary.py
scripts/benchmark_gpu.py
scripts/PERFORMANCE_ANALYSIS.md
scripts/fast_audio_pipeline.py
scripts/bulk_download.py
scripts/normalize_languages.py
scripts/fast_audio_pipeline_v2.py
scripts/__pycache__/gpu_audio_processor.cpython-310.pyc
scripts/__pycache__/prepare_shards.cpython-310.pyc
scripts/__pycache__/fast_audio_pipeline.cpython-310.pyc


Summary: auto


**Reviewing audio pipeline code**


sed -n '1,240p' scripts/fast_audio_pipeline.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:
"""GPU-accelerated audio processing - semantically equivalent to CPU path.

This module provides a drop-in replacement for segment processing that:
1. Uses EXACT same chunking logic (make_chunk_windows)
2. Uses EXACT same metrics computation (compute_audio_metrics with pyloudnorm)
3. Only accelerates I/O-bound operations:
   - Parallel FLAC decoding
   - GPU-batched resampling
   - Parallel WAV encoding

The GPU path produces IDENTICAL outputs to the CPU path.
"""

from __future__ import annotations

import io
import math
import random
import threading
from concurrent.futures import ThreadPoolExecutor
from dataclasses import dataclass
from typing import Any, Optional

import numpy as np
import soundfile as sf

# Import exact same functions from prepare_shards for equivalence
from prepare_shards import (
    ChunkConfig,
    SilenceConfig,
    SegmentOutput,
    make_chunk_windows,
    compute_audio_metrics,
    compute_quality_score,
    duration_bucket,
    classify_native_bucket,
)

# GPU acceleration (optional)
try:
    import torch
    import torchaudio
    HAS_TORCH = True
except ImportError:
    HAS_TORCH = False
    torch = None
    torchaudio = None

@dataclass
class SegmentInput:
    """Input segment for processing."""
    video_id: str
    language: str
    segment_bytes: bytes
    segment_meta: dict[str, Any]
    speaker: str
    index: int
    is_val: bool
    rng_seed: int

@dataclass
class OutputSegment:
    """Final output ready for shard writing."""
    sample_rate: int
    audio_bytes: bytes
    metadata: dict[str, Any]
    is_val: bool

class GPUResampler:
    """GPU-accelerated resampler using torchaudio."""

    def __init__(self, device: str = "cuda"):
        self.use_gpu = HAS_TORCH and torch.cuda.is_available()
        self.device = torch.device(device if self.use_gpu else "cpu")
        self._resamplers: dict[tuple[int, int], Any] = {}
        self._lock = threading.Lock()

        if self.use_gpu:
            print(f"GPUResampler: Using {torch.cuda.get_device_name()}")

    def resample(self, audio: np.ndarray, src_sr: int, dst_sr: int) -> np.ndarray:
        """Resample audio using GPU if available."""
        if src_sr == dst_sr:
            return audio

        if not self.use_gpu:
            # CPU fallback using scipy
            from scipy import signal
            gcd = math.gcd(src_sr, dst_sr)
            up = dst_sr // gcd
            down = src_sr // gcd
            return signal.resample_poly(audio, up, down).astype(np.float32, copy=False)

        # GPU path
        resampler = self._get_resampler(src_sr, dst_sr)
        audio_t = torch.from_numpy(audio).unsqueeze(0).to(self.device)
        with torch.no_grad():
            resampled_t = resampler(audio_t)
        return resampled_t.squeeze(0).cpu().numpy()

    def resample_batch(
        self, audios: list[np.ndarray], src_sr: int, dst_sr: int
    ) -> list[np.ndarray]:
        """Resample multiple audio arrays to the same target rate."""
        if src_sr == dst_sr:
            return audios

        if not self.use_gpu or not audios:
            return [self.resample(a, src_sr, dst_sr) for a in audios]

        # Batch on GPU
        resampler = self._get_resampler(src_sr, dst_sr)

        # Pad to max length for batching
        lengths = [a.shape[0] for a in audios]
        max_len = max(lengths)
        batch = np.zeros((len(audios), max_len), dtype=np.float32)
        for i, a in enumerate(audios):
            batch[i, :a.shape[0]] = a

        batch_t = torch.from_numpy(batch).to(self.device)
        with torch.no_grad():
            resampled_t = resampler(batch_t)
        resampled_batch = resampled_t.cpu().numpy()

        # Trim to actual lengths
        ratio = dst_sr / src_sr
        results = []
        for i, orig_len in enumerate(lengths):
            new_len = int(orig_len * ratio)
            results.append(resampled_batch[i, :new_len])

        return results

    def _get_resampler(self, src_sr: int, dst_sr: int):
        """Get or create a resampler for the given rate pair."""
        key = (src_sr, dst_sr)
        if key not in self._resamplers:
            with self._lock:
                if key not in self._resamplers:
                    self._resamplers[key] = torchaudio.transforms.Resample(
                        src_sr, dst_sr, dtype=torch.float32
                    ).to(self.device)
        return self._resamplers[key]

class EquivalentBatchProcessor:
    """Batch processor that produces IDENTICAL outputs to CPU path.

    This processor:
    1. Decodes segments in parallel (CPU ThreadPool)
    2. Applies EXACT same chunking logic as CPU path
    3. Computes metrics using EXACT same functions (including pyloudnorm LUFS)
    4. Resamples using GPU (same algorithm, just faster)
    5. Encodes in parallel (CPU ThreadPool)

    The only difference from CPU is parallelism/hardware, not algorithms.
    """

    def __init__(
        self,
        decode_workers: int = 8,
        encode_workers: int = 8,
        output_format: str = "wav",
        chunk_cfg: Optional[ChunkConfig] = None,
        silence_cfg: Optional[SilenceConfig] = None,
    ):
        self.decode_workers = decode_workers
        self.encode_workers = encode_workers
        self.output_format = output_format
        self.chunk_cfg = chunk_cfg or ChunkConfig(
            enabled=True, min_sec=6.0, max_sec=10.0, threshold_sec=12.0, all_buckets=False
        )
        self.silence_cfg = silence_cfg or SilenceConfig(
            enabled=True, frame_sec=0.05, hop_sec=0.01,
            search_sec=0.5, percentile=20.0, db_drop=20.0
        )

        self.resampler = GPUResampler()

        self.stats = {
            "decoded": 0,
            "chunks_emitted": 0,
            "segments_chunked": 0,
            "dropped_too_short": 0,
            "dropped_too_long": 0,
            "dropped_clipped": 0,
        }

    def process_segments(
        self, segments: list[SegmentInput]
    ) -> list[OutputSegment]:
        """Process segments and return outputs - equivalent to CPU path.

        This method mirrors the exact logic of process_segment() in prepare_shards.py.
        """
        if not segments:
            return []

        # Step 1: Parallel decode
        decoded = self._decode_parallel(segments)

        # Step 2: Process each segment (chunking + metrics + resample + encode)
        all_outputs: list[OutputSegment] = []

        for seg_input, audio, sr in decoded:
            outputs = self._process_single_segment(seg_input, audio, sr)
            all_outputs.extend(outputs)

        return all_outputs

    def _decode_parallel(
        self, segments: list[SegmentInput]
    ) -> list[tuple[SegmentInput, np.ndarray, int]]:
        """Decode segments in parallel."""
        def decode_one(seg: SegmentInput) -> Optional[tuple[SegmentInput, np.ndarray, int]]:
            try:
                audio, sr = sf.read(io.BytesIO(seg.segment_bytes), dtype="float32")
                if audio.ndim > 1:
                    audio = np.mean(audio, axis=1)
                return (seg, audio.astype(np.float32), sr)
            except Exception:
                return None

        with ThreadPoolExecutor(max_workers=self.decode_workers) as pool:
            results = list(pool.map(decode_one, segments))

        decoded = [r for r in results if r is not None]
        self.stats["decoded"] += len(decoded)
        return decoded

    def _process_single_segment(
        self,
        seg: SegmentInput,
        audio: np.ndarray,
        sr: int,
    ) -> list[OutputSegment]:


**Planning optimization branch and benchmarks**

The user expects implementing new optimizations per plan, then benchmarking on 100 and 500 videos, plus scaling estimates for 500k videos with horizontal scaling suggestions; next, I'll confirm branch setup and inspect benchmark scripts.


sed -n '1,240p' scripts/benchmark_gpu.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:
#!/usr/bin/env python3
"""GPU Pipeline Performance Benchmark for A100-80GB.

Tests various configurations to find optimal settings:
- Batch sizes: 16, 32, 64, 128, 256
- Decode workers: 4, 8, 12, 16
- Encode workers: 4, 8, 12, 16

Measures:
- Segments/second throughput
- GPU utilization
- Memory usage
- Bottleneck identification
"""

import argparse
import io
import json
import os
import sys
import tarfile
import time
from concurrent.futures import ThreadPoolExecutor
from dataclasses import dataclass
from pathlib import Path
from typing import Optional

import numpy as np
import soundfile as sf

# Add current dir to path for imports
sys.path.insert(0, str(Path(__file__).parent))

from prepare_shards import (
    ChunkConfig, SilenceConfig, make_segment_seed, parse_segment_filename
)
from fast_audio_pipeline import EquivalentBatchProcessor, SegmentInput, OutputSegment

try:
    import torch
    HAS_TORCH = True
except ImportError:
    HAS_TORCH = False

@dataclass
class BenchmarkResult:
    config_name: str
    batch_size: int
    decode_workers: int
    encode_workers: int
    total_segments: int
    total_outputs: int
    elapsed_sec: float
    segments_per_sec: float
    outputs_per_sec: float
    gpu_memory_mb: float
    stats: dict

def get_gpu_memory_mb() -> float:
    """Get current GPU memory usage in MB."""
    if not HAS_TORCH or not torch.cuda.is_available():
        return 0.0
    return torch.cuda.memory_allocated() / 1024 / 1024

def load_segments_from_tar(tar_path: Path, video_id: str, language: str, seed: int) -> list[SegmentInput]:
    """Load segments from a tar file."""
    segments = []
    try:
        with tarfile.open(tar_path, "r") as tf:
            meta = json.loads(tf.extractfile(f"{video_id}/metadata.json").read())
            usable_segments = [s for s in meta.get("segments", []) if s.get("status") == "usable"]
            segment_map = {
                (seg.get("speaker"), round(seg.get("start", 0.0), 2)): seg
                for seg in usable_segments
            }

            members = [m for m in tf.getmembers() if m.name.endswith(".flac")]

            for member in members:
                parsed = parse_segment_filename(member.name)
                if not parsed:
                    continue
                speaker, start_time = parsed
                seg_meta = segment_map.get((speaker, round(start_time, 2)))
                if not seg_meta:
                    matches = [
                        s for s in usable_segments
                        if s.get("speaker") == speaker
                        and abs(s.get("start", 0.0) - start_time) < 0.1
                    ]
                    if not matches:
                        continue
                    seg_meta = matches[0]

                segment_bytes = tf.extractfile(member).read()
                filename_parts = member.name.split("_")
                try:
                    index = int(filename_parts[-2])
                except (ValueError, IndexError):
                    index = 0
                rng_seed = make_segment_seed(seed, video_id, speaker, index)

                segments.append(SegmentInput(
                    video_id=video_id,
                    language=language,
                    segment_bytes=segment_bytes,
                    segment_meta=seg_meta,
                    speaker=speaker,
                    index=index,
                    is_val=False,
                    rng_seed=rng_seed,
                ))
    except Exception as e:
        print(f"Error loading {tar_path}: {e}")

    return segments

def generate_synthetic_segments(count: int, duration_sec: float = 10.0, sr: int = 48000) -> list[SegmentInput]:
    """Generate synthetic test segments."""
    segments = []
    num_samples = int(sr * duration_sec)
    t = np.linspace(0, duration_sec, num_samples, dtype=np.float32)

    for i in range(count):
        # Generate varied audio
        freq = 200 + (i % 20) * 100
        audio = (
            np.sin(2 * np.pi * freq * t) * 0.3 +
            np.sin(2 * np.pi * 1000 * t) * 0.2 +
            np.sin(2 * np.pi * 5000 * t) * 0.1 +
            np.random.randn(num_samples).astype(np.float32) * 0.02
        ).astype(np.float32)

        buf = io.BytesIO()
        sf.write(buf, audio, sr, format="FLAC")
        flac_bytes = buf.getvalue()

        segments.append(SegmentInput(
            video_id=f"synthetic_{i}",
            language="english",
            segment_bytes=flac_bytes,
            segment_meta={"start": 0.0, "end": duration_sec, "original_start": 0.0, "original_end": duration_sec},
            speaker="SPEAKER_00",
            index=i,
            is_val=False,
            rng_seed=42 + i,
        ))

    return segments

def run_benchmark(
    segments: list[SegmentInput],
    batch_size: int,
    decode_workers: int,
    encode_workers: int,
    output_format: str = "wav",
) -> BenchmarkResult:
    """Run a single benchmark configuration."""
    config_name = f"b{batch_size}_d{decode_workers}_e{encode_workers}"

    chunk_cfg = ChunkConfig(
        enabled=True, min_sec=6.0, max_sec=10.0, threshold_sec=12.0, all_buckets=False
    )
    silence_cfg = SilenceConfig(
        enabled=True, frame_sec=0.05, hop_sec=0.01,
        search_sec=0.5, percentile=20.0, db_drop=20.0
    )

    # Clear GPU cache
    if HAS_TORCH and torch.cuda.is_available():
        torch.cuda.empty_cache()
        torch.cuda.reset_peak_memory_stats()

    processor = EquivalentBatchProcessor(
        decode_workers=decode_workers,
        encode_workers=encode_workers,
        output_format=output_format,
        chunk_cfg=chunk_cfg,
        silence_cfg=silence_cfg,
    )

    total_outputs = 0
    start_time = time.perf_counter()

    # Process in batches
    for i in range(0, len(segments), batch_size):
        batch = segments[i:i + batch_size]
        outputs = processor.process_segments(batch)
        total_outputs += len(outputs)

    elapsed = time.perf_counter() - start_time

    # Get peak GPU memory
    gpu_memory = 0.0
    if HAS_TORCH and torch.cuda.is_available():
        gpu_memory = torch.cuda.max_memory_allocated() / 1024 / 1024

    segments_per_sec = len(segments) / elapsed if elapsed > 0 else 0
    outputs_per_sec = total_outputs / elapsed if elapsed > 0 else 0

    return BenchmarkResult(
        config_name=config_name,
        batch_size=batch_size,
        decode_workers=decode_workers,
        encode_workers=encode_workers,
        total_segments=len(segments),
        total_outputs=total_outputs,
        elapsed_sec=elapsed,
        segments_per_sec=segments_per_sec,
        outputs_per_sec=outputs_per_sec,
        gpu_memory_mb=gpu_memory,
        stats=dict(processor.stats),
    )

def find_tar_files(download_dir: Path, limit: int = 100) -> list[tuple[str, Path]]:
    """Find tar files in download directory."""
    tar_files = []
    for tar_path in sorted(download_dir.glob("*.tar"))[:limit]:
        video_id = tar_path.stem
        tar_files.append((video_id, tar_path))
    return tar_files

def main():
    parser = argparse.ArgumentParser(description="GPU Pipeline Performance Benchmark")
    parser.add_argument("--download-dir", type=Path, help="Directory with downloaded tar files")
    parser.add_argument("--language", default="hindi", help="Language for loading real data")
    parser.add_argument("--synthetic-count", type=int, default=200, help="Number of synthetic segments if no real data")
    parser.add_argument("--synthetic-duration", type=float, default=15.0, help="Duration of synthetic segments (seconds)")
    parser.add_argument("--output-format", choices=["wav", "flac"], default="wav")
    parser.add_argument("--quick", action="store_true", help="Run quick benchmark with fewer configs")
    parser.add_argument("--limit-tars", type=int, default=20, help="Max tar files to load")
    parser.add_argument("--seed", type=int, default=42)
    args = parser.parse_args()


**Reviewing strategy for GPU-accelerated metrics**

I'm verifying the plan calls for implementing GPU-based LUFS and pipeline parallelization to boost metric computation, noting current code still uses CPU-based pyloudnorm; next I'll confirm details in strategy.md and objective files.


Output:
Exit code: 0
Wall time: 0 seconds
Output:
# Codec Training Strategy (Indic, X-Codec-2.0 + Inworld SR)

## Current State

- **Source**: ~507k YouTube podcast videos across 12 Indic languages in Cloudflare R2 (`1-cleaned-data` bucket)
- **Format**: `videoID.tar` files containing `metadata.json` + `segments/*.flac` (original quality from YouTube WebM, compressed as FLAC)
- **Usable hours**: ~149k hours (after VAD/speaker segmentation)
- **Native sample rate**: 99.8% containers at 48kHz, but actual bandwidth varies (see bandwidth distribution below)
- **Note**: Segments are already at original quality; resampling to 16/24/32/44k happens during sharding

### Codebase References
- **X-Codec-2.0**: Full 16kHz codec (encoder + decoder + VQ) with W2V-BERT semantic features
  - Random 6s crops (`min_audio_length = 96000` @ 16kHz)
  - W2V-BERT frozen during training
  - Codebook: 16384 entries, dim=16, vq_dim=2048
- **Inworld TTS**: Decoder-only training from precomputed codes
  - 48kHz SR with `upsample_factors=[3,2]`, `hop_length=160`, `token_rate=50`
  - RMS loss for volume consistency (`lambda_rms=1.0`)
  - Two-stage 48kHz training: ≥32kHz → ≥44.1kHz

### Actual Bandwidth Distribution (from smoke test analysis)

Analysis of effective Nyquist frequency (actual audio bandwidth):
- **24-32k bandwidth**: ~62% of segments (effective_nyquist 12-16 kHz)
- **32-44k bandwidth**: ~30% of segments (effective_nyquist 16-22 kHz)
- **16-24k bandwidth**: ~7% of segments (effective_nyquist 8-12 kHz)
- **<16k bandwidth**: <1% of segments (effective_nyquist <8 kHz)
- **>=44k bandwidth (true wideband)**: Near zero (upsampled or low-passed)

**Key insight**: Container SR (48kHz) ≠ actual bandwidth. Most audio is bandwidth-limited to 12-18 kHz.

### Known Issues to Address
1. ~~**Language missing** from `metadata.json`~~ → FIXED: Fetched from Supabase
2. **Segment duration skew**: ~58% < 6s, ~27% < 3s → HANDLED: Random 6s crops during training
3. ~~**True sample rate untrusted**~~ → FIXED: Spectral analysis computes effective bandwidth
4. ~~**Missing quality metrics**~~ → FIXED: Loudness, clipping, SNR computed during sharding
5. ~~**Native SR buckets missing**~~ → FIXED: Multi-resolution buckets (16/24/32/44k) implemented

---

## 1) Data Validation Pipeline

### 1.1 Fetch Language Metadata from Supabase

Query Supabase using credentials in `.env` (scripts in `scripts/` for reference):

```
videoID → language
```

Store as `language_lookup.json` for fast local access during processing.

### 1.2 Compute Per-Segment Quality Metrics

For each segment in each `videoID.tar`, compute and store:

**Audio Quality Metrics (fast, essential):**
| Metric | Description | Purpose |
|--------|-------------|---------|
| `duration_sec` | Segment length | Already in metadata |
| `loudness_lufs` | Integrated loudness | RMS loss validation |
| `loudness_rms_db` | RMS in dB | Volume normalization |
| `clipping_fraction` | % samples \|x\| > 0.99 | Filter damaged audio |
| `dc_offset` | Mean of waveform | Catch broken pipelines |
| `snr_proxy` | Speech vs noise floor | Quality weighting |

**True Sample Rate Detection (critical for Phase C):**
| Metric | Description | Purpose |
|--------|-------------|---------|
| `spectral_rolloff_hz` | Freq below 85% energy | Detect bandwidth |
| `effective_nyquist_hz` | Detected cutoff frequency | True sample rate |
| `high_freq_energy_ratio` | Energy >8kHz / total | Wideband detection |
| `native_sr_bucket` | `<16k`, `16-24k`, `24-32k`, `32-44k`, `>=44k` | Bandwidth classification |
| `is_truly_wideband` | Boolean: effective_nyquist >= 12kHz | Higher-SR eligibility |

**Preserve Existing Metrics:**
- `music_stats` (already computed via PANN)
- `speaker` label
- `start`, `end`, `original_start`, `original_end`, `duration`

### 1.3 Native SR Classification + Resample Outputs

For each usable segment:
1. Load the segment FLAC from the tar (already at original quality from YouTube WebM).
2. Run spectral detection (rolloff, effective_nyquist, HF ratio).
3. Assign buckets and resample targets based on **actual bandwidth**, not container SR:
   - **16k bucket**: all usable segments (universal). Long segments (>=12s) are split into 6–10s windows.
   - **24k bucket**: `effective_nyquist >= 8000` Hz (narrowband to low-wideband)
   - **32k bucket**: `effective_nyquist >= 12000` Hz (mid-wideband)
   - **44k bucket**: `effective_nyquist >= 16000` Hz (high-wideband, true 44.1k sources)
4. Record `resample_target_sr` for each emitted sample.
5. Wideband flag: `is_truly_wideband = effective_nyquist >= 12000` (relaxed from previous 14k + HF ratio gate)

### 1.3 Composite Quality Score

Derive `quality_score` (0-1) for weighted sampling (store, do not filter):

```python
quality_score = (
    0.3 * (1 - clipping_fraction) +
    0.2 * normalize(snr_proxy, min=10, max=40) +
    0.2 * (1 if not music_stats.has_music else 0.5) +
    0.2 * normalize(loudness_lufs, min=-40, max=-14) +
    0.1 * (1 if abs(dc_offset) < 0.01 else 0.5)
)
```

---

## 2) Sharding Strategy

### 2.1 Segment Handling Decision: No Stitching

**Problem**: ~58% of segments are <6s, but X-Codec requires 6s crops.

**Decision**: Do NOT stitch segments. Reasons:
1. Speaker diarization is not 100% reliable - stitching risks mixing speakers
2. W2V-BERT semantic encoder expects consistent speaker within sample
3. Padding is recoverable (efficiency loss), data corruption is not

**Implemented approach**:
- **Drop segments <3s** (too short, mostly padding anyway - ~27% of segments)
- **Duration-weighted sampling** to favor longer segments
- **Chunk long segments for 16k**: For segments >=12s, split into non-overlapping 6–10s windows to boost sample count (no stitching across speakers).
- **Silence-aware chunking**: Only cut when a low-energy boundary exists near the target window; otherwise keep the full segment.
- **No pre-cropping needed for training**: Random 6s crops still happen during training (X-Codec data_module.py:104)
- **VAD regions**: Store full duration `[[0, duration_sec]]` as placeholder; actual cropping is random at train time

### 2.2 Output Structure

```
processed_data/
├── manifests/
│   ├── telugu_train.jsonl
│   ├── telugu_val.jsonl
│   └── ... (per language)
├── telugu_16/           # 16kHz - universal bucket
│   ├── shard-000000.tar
│   ├── shard-000000.jsonl
│   └── ...
├── telugu_24/           # 24kHz - effective_nyquist >= 8kHz
│   ├── shard-000000.tar
│   ├── shard-000000.jsonl
│   └── ...
├── telugu_32/           # 32kHz - effective_nyquist >= 12kHz
│   ├── shard-000000.tar
│   ├── shard-000000.jsonl
│   └── ...
├── telugu_44/           # 44.1kHz - effective_nyquist >= 16kHz
│   ├── shard-000000.tar
│   ├── shard-000000.jsonl
│   └── ...
└── stats/
    ├── telugu_stats.json
    └── ...
```

Note: Directory naming is `{language}_{sr_khz}` for easier R2 prefix management.

### 2.3 Shard Contents

Each shard (~5GB, multipart upload friendly) contains audio only, plus a single
JSONL sidecar file with per-sample metadata:

```
shard-000000.tar
shard-000000.jsonl
```

`shard-000000.jsonl` is one line per sample with:
```json
{
  "id": "videoID__SPEAKER_00_0001",
  "video_id": "---g_VL7ySo",
  "language": "bengali",
  "duration_sec": 4.11,
  "sample_rate": 16000,
  "resample_target_sr": 16000,
  "native_sr": 48000,
  "effective_nyquist_hz": 24000.0,
  "is_truly_wideband": true,
  "native_sr_bucket": ">=44k",
  "loudness_lufs": -23.5,
  "loudness_rms_db": -18.2,
  "clipping_fraction": 0.0,
  "snr_proxy": 28.5,
  "quality_score": 0.85,
  "music_prob": 0.045,
  "speaker": "SPEAKER_00",
  "vad_speech_regions": [[0.0, 4.11]],
  "duration_bucket": "3-6s",
  "sampling_weight": 0.3
}
```

Metadata storage:
- Keep per-shard JSONL locally for now.
- Also write per-language JSONL files for easy Supabase upsert later.

### 2.4 Filtering Rules (applied during sharding)


**Preparing optimization branch and benchmarks**

I'm setting up a new branch for optimization implementation and planning benchmarks on datasets of 100 and 500 videos, plus scaling estimates for 500k videos with horizontal scaling suggestions. I'll consider synthetic data for benchmarking due to environment constraints.


sed -n '1,240p' scripts/PERFORMANCE_ANALYSIS.md


Output:
Exit code: 0
Wall time: 0 seconds
Output:
# GPU Pipeline Performance Analysis for A100-80GB

## Executive Summary

**Current Performance:** ~26 seg/s, ~7.5 videos/min on A100-80GB with 16 vCPUs

**Key Finding:** The A100 GPU is heavily underutilized. The bottleneck is CPU-bound metrics computation (pyloudnorm LUFS), not GPU operations.

**Recommendation:** For cost optimization, consider using CPU-only instances (e.g., c6i.4xlarge) instead of expensive GPU instances. The GPU provides minimal benefit for this workload.

---

## Benchmark Results

### Per-Operation Timing (Single Segment)

| Operation | Time (ms) | % of Total |
|-----------|-----------|------------|
| FLAC Decode | 17.9 | 31.0% |
| **Metrics (LUFS+Spectral)** | **26.1** | **45.3%** |
| GPU Resample (×4 rates) | 0.5 | 0.9% |
| WAV Encode (×4 rates) | 13.1 | 22.8% |
| **Total** | **57.7** | 100% |

**The GPU is only used for 0.9% of the processing time.**

### Parallelization Effectiveness

| Workers | Metrics-Only | Full Pipeline (GPU) | Full Pipeline (CPU) |
|---------|-------------|---------------------|---------------------|
| 4 | 76.3 seg/s | 51.2 seg/s | 42.3 seg/s |
| 8 | 96.8 seg/s | 50.7 seg/s | 50.1 seg/s |
| 12 | 94.4 seg/s | 50.8 seg/s | 48.6 seg/s |
| 16 | 92.2 seg/s | 49.4 seg/s | 46.8 seg/s |

**Key Insight:** GPU resampling provides only ~2% improvement over CPU resampling. The GPU is bottlenecked by:
1. CPU↔GPU memory transfer overhead
2. Serialized CUDA stream execution
3. The small batch size (single segment at a time)

### Real Data Test (Hindi, 20 videos)

- Segments loaded: 4,142
- Segments/video: 207
- Throughput: **26.1 seg/s**
- Videos/min: **7.56**
- Dropped (too short): 87.7%
- Outputs generated: 7,393

---

## Bottleneck Analysis

### Primary Bottleneck: pyloudnorm LUFS (45% of time)

The `compute_audio_metrics()` function calls `pyloudnorm.Meter.integrated_loudness()`, which is a pure Python implementation of ITU-R BS.1770.

**Options to optimize:**
1. **Skip LUFS for filtering** - If LUFS is only used for quality scoring, consider a simpler RMS-based approximation
2. **Batch LUFS on GPU** - Implement ITU-R BS.1770 K-weighting filters in PyTorch
3. **Use a faster LUFS library** - e.g., `ffmpeg-normalize` or custom C extension

### Secondary Bottleneck: FLAC Decode (31% of time)

Using `soundfile` (libsndfile) for FLAC decoding. Already parallelized.

**Options:**
1. Store audio as WAV in cleaned bucket (eliminates decode entirely)
2. Use `av` library (PyAV) which may be faster for some formats

### Tertiary Bottleneck: WAV Encode (23% of time)

Already parallelized. Limited optimization potential.

---

## Optimal Configuration

### For A100-80GB + 16 vCPUs

```bash
python prepare_shards.py hindi --gpu \
    --gpu-batch-size 128 \
    --gpu-decode-workers 12 \
    --gpu-encode-workers 12 \
    --output-format wav
```

| Parameter | Value | Rationale |
|-----------|-------|-----------|
| gpu-batch-size | 128 | Marginal impact; larger batches don't help |
| gpu-decode-workers | 12 | Optimal for 16 vCPUs (leaves headroom) |
| gpu-encode-workers | 12 | Same as decode |

### Expected Throughput

| Metric | Value |
|--------|-------|
| Segments/sec | 26-30 |
| Videos/min | 7.5-8.5 |
| 16k outputs/min | ~1,500 |

---

## Scaling Projections

### Single A100 Node

| Dataset Size | Time | Days |
|-------------|------|------|
| 10,000 videos | 22 hours | 0.9 |
| 50,000 videos | 110 hours | 4.6 |
| 100,000 videos | 220 hours | 9.2 |
| 500,000 videos | 1,100 hours | 45.8 |

### Horizontal Scaling (500k videos)

| Nodes | Time | Days | Cost Efficiency |
|-------|------|------|-----------------|
| 1 | 1,100h | 46 days | Baseline |
| 2 | 550h | 23 days | 2× |
| 4 | 275h | 11 days | 4× |
| 8 | 138h | 5.7 days | 8× |
| 16 | 69h | 2.9 days | 16× |
| 32 | 34h | 1.4 days | 32× |

**Scaling is linear** - no bottleneck on shared resources (each node processes independent videos).

---

## Cost Optimization Recommendations

### Option 1: Keep A100 (Current)

- **Cost:** ~$3.50/hour (p4d.24xlarge spot) or ~$10/hour (on-demand)
- **Throughput:** 7.5 videos/min
- **GPU utilization:** <5%
- **Verdict:** Inefficient; paying for unused GPU capacity

### Option 2: CPU-Only Instances (Recommended)

Replace A100 with CPU instances:

| Instance | vCPUs | Cost/hr | Est. Throughput | $/video |
|----------|-------|---------|-----------------|---------|
| c6i.4xlarge | 16 | $0.68 | 6.5 vid/min | $0.0017 |
| c6i.8xlarge | 32 | $1.36 | 10 vid/min | $0.0023 |
| c7i.4xlarge | 16 | $0.71 | 6.5 vid/min | $0.0018 |

**Savings:** 80-90% cost reduction vs A100 with similar throughput.

### Option 3: Hybrid Approach

Use A100 nodes only if future optimizations (batch GPU LUFS, batch GPU metrics) are implemented.

---

## Implementation Recommendations

### Immediate (No Code Changes)

1. **Use CPU instances** instead of A100 for cost efficiency
2. **Scale horizontally** with 4-8 CPU nodes for faster completion
3. **Use optimal config:** 12 decode workers, 12 encode workers, batch_size 128

### Short-term (Code Changes)

1. **Parallelize segment processing** - Current code processes segments sequentially after decode; fully parallelize the entire chain
2. **Remove GPU path** - It provides minimal benefit and adds complexity

### Long-term (Significant Changes)

1. **Store cleaned audio as WAV** - Eliminates FLAC decode overhead (31% savings)
2. **Batch LUFS on GPU** - Implement ITU-R BS.1770 in PyTorch for ~10× LUFS speedup
3. **Use NVIDIA DALI** - GPU-accelerated audio pipeline

---

## Conclusion

The A100-80GB GPU is **significantly underutilized** for this workload. The primary bottleneck is CPU-bound LUFS computation (pyloudnorm), which consumes 45% of processing time.

**Recommended action:** Switch to CPU-only instances (c6i.4xlarge or similar) and scale horizontally. This will reduce costs by 80-90% while maintaining similar throughput.

For 500k videos:
- **8 × c6i.4xlarge nodes:** ~6 days, ~$780 total
- **1 × A100 node:** ~46 days, ~$3,850 total

The CPU approach is **5× cheaper** and **8× faster** due to parallelization.


**Clarifying optimization implementation and benchmarking steps**

The user wants to implement a new optimization approach—likely a GPU-accelerated or vectorized LUFS pipeline—in the existing 'optimisations' branch following the documented plan, then benchmark processing at multiple scales including 100, 500, and a projected 500k videos with scaling recommendations. Next, I'll focus on coding this new pipeline and preparing those benchmarks.


**Preparing environment check for GPU-accelerated implementation**

I'm verifying the availability of PyTorch and related GPU libraries needed to implement the planned GPU-accelerated LUFS optimizations before proceeding with coding and benchmarking.


python - <<'PY'
import importlib.util
for mod in ['torch','torchaudio','pyloudnorm']:
    spec = importlib.util.find_spec(mod)
    print(mod, 'ok' if spec else 'missing')
PY


Output:
Exit code: 127
Wall time: 0 seconds
Output:
/bin/bash: line 1: python: command not found


**Checking for python3**


python3 - <<'PY'
import importlib.util
for mod in ['torch','torchaudio','pyloudnorm','numpy','soundfile']:
    spec = importlib.util.find_spec(mod)
    print(mod, 'ok' if spec else 'missing')
PY


Output:
Exit code: 0
Wall time: 0 seconds
Output:
torch ok
torchaudio ok
pyloudnorm ok
numpy ok
soundfile ok


**Evaluating GPU-based LUFS implementation**

I'm assessing if torch/torchaudio can provide a GPU-accelerated LUFS calculation as an alternative to pyloudnorm, aiming for approximate equivalence to maintain quality metrics while improving performance. Next, I'll verify available loudness functions in torchaudio for feasibility.


python3 - <<'PY'
import torchaudio
import inspect
print('torchaudio version', torchaudio.__version__)
print('has functional.loudness', hasattr(torchaudio.functional, 'loudness'))
print('has functional.loudness_weighting', hasattr(torchaudio.functional, 'loudness_weighting'))
print('has functional.loudness', getattr(torchaudio.functional, 'loudness', None))
print('has functional.loudness_weighting', getattr(torchaudio.functional, 'loudness_weighting', None))
PY


Output:
Exit code: 0
Wall time: 1.7 seconds
Output:
/home/ubuntu/.local/lib/python3.10/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
torchaudio version 2.9.0+cu128
has functional.loudness True
has functional.loudness_weighting False
has functional.loudness <function loudness at 0x73014791c280>
has functional.loudness_weighting None


**Checking torchaudio loudness method**


python3 - <<'PY'
import torchaudio, inspect
print(inspect.signature(torchaudio.functional.loudness))
print(torchaudio.functional.loudness.__doc__.split('\n')[0])
PY


Output:
Exit code: 0
Wall time: 1.7 seconds
Output:
/home/ubuntu/.local/lib/python3.10/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
(waveform: torch.Tensor, sample_rate: int)
Measure audio loudness according to the ITU-R BS.1770-4 recommendation.


Summary: auto


**Confirming batched loudness support**

torchaudio's loudness function uses LUFS and likely supports GPU if inputs are on GPU. I'll verify if it accepts batched waveforms or multiple channels to enable vectorized computation.


python3 - <<'PY'
import torchaudio
print(torchaudio.functional.loudness.__doc__)
PY


Output:
Exit code: 0
Wall time: 1.7 seconds
Output:
/home/ubuntu/.local/lib/python3.10/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
Measure audio loudness according to the ITU-R BS.1770-4 recommendation.

    .. devices:: CPU CUDA

    .. properties:: TorchScript

    Args:
        waveform(torch.Tensor): audio waveform of dimension `(..., channels, time)`
        sample_rate (int): sampling rate of the waveform

    Returns:
        Tensor: loudness estimates (LKFS)

    Reference:
        - https://www.itu.int/rec/R-REC-BS.1770-4-201510-I/en


**Planning GPU batched audio metrics**

The current CPU-based compute_audio_metrics uses numpy sequentially, but batch GPU computation is supported by EquivalentBatchProcessor after decode. Next, I'll locate prepare_shards' GPU usage to build a new GPU pipeline for batched metric computation, adding CLI support.


rg -n "gpu|EquivalentBatchProcessor|fast_audio_pipeline" scripts/prepare_shards.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:
35:    from fast_audio_pipeline import EquivalentBatchProcessor, SegmentInput, OutputSegment
39:    EquivalentBatchProcessor = None
1050:        "--gpu",
1055:        "--gpu-batch-size",
1061:        "--gpu-decode-workers",
1067:        "--gpu-encode-workers",
1321:    def handle_gpu_output(out: "OutputSegment") -> None:
1345:    use_gpu = args.gpu and HAS_GPU_PIPELINE
1346:    if use_gpu:
1347:        print(f"Using GPU-accelerated batch processor (batch_size={args.gpu_batch_size})")
1349:        processor = EquivalentBatchProcessor(
1350:            decode_workers=args.gpu_decode_workers,
1351:            encode_workers=args.gpu_encode_workers,
1420:                    if len(batch) >= args.gpu_batch_size:
1421:                        with timer("gpu_batch_process"):
1424:                            handle_gpu_output(out)
1429:            with timer("gpu_batch_process"):
1432:                handle_gpu_output(out)


sed -n '1020,1460p' scripts/prepare_shards.py