# AGENTS.md instructions for /home/ubuntu/maya3_prepare

<INSTRUCTIONS>
## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  3) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  4) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.
</INSTRUCTIONS>

<environment_context>
  <cwd>/home/ubuntu/maya3_prepare</cwd>
  <shell>bash</shell>
</environment_context>

# Context from my IDE setup:

## Open tabs:
- optimisations.md: optimisations.md
- Optimisation.md: Optimisation.md
- prepare_shards.py: scripts/prepare_shards.py
- fast_audio_pipeline_v2.py: scripts/fast_audio_pipeline_v2.py
- benchmark_gpu.py: scripts/benchmark_gpu.py

## My request for Codex:

Current performance. - Performance Analysis Complete

  Key Findings
  ┌─────────────────────┬───────────────────────────────────────────┐
  │       Metric        │                   Value                   │
  ├─────────────────────┼───────────────────────────────────────────┤
  │ Measured Throughput │ 26.1 seg/s, 7.56 videos/min               │
  ├─────────────────────┼───────────────────────────────────────────┤
  │ GPU Utilization     │ <5% (only resampling uses GPU)            │
  ├─────────────────────┼───────────────────────────────────────────┤
  │ Primary Bottleneck  │ pyloudnorm LUFS computation (45% of time) │
  ├─────────────────────┼───────────────────────────────────────────┤
  │ GPU Benefit         │ ~2% improvement over CPU-only             │
  └─────────────────────┴───────────────────────────────────────────┘
  Per-Segment Breakdown

  FLAC Decode:     17.9 ms (31.0%)
  Metrics (LUFS):  26.1 ms (45.3%)  ← BOTTLENECK
  GPU Resample:     0.5 ms (0.9%)   ← GPU barely used
  WAV Encode:      13.1 ms (22.8%)

  Optimal A100 Configuration

  --gpu-batch-size 128 --gpu-decode-workers 12 --gpu-encode-workers 12

  This yields ~26-30 seg/s. Larger batch sizes and more workers don't help due to the sequential nature of metrics
   computation.

  Scaling Projections (500k videos)
  ┌────────┬──────────────┬─────────┬───────────┐
  │ Nodes  │   Instance   │  Time   │ Est. Cost │
  ├────────┼──────────────┼─────────┼───────────┤
  │ 1 A100 │ p4d.24xlarge │ 46 days │ ~$3,850   │
  ├────────┼──────────────┼─────────┼───────────┤
  │ 8 CPU  │ c6i.4xlarge  │ 6 days  │ ~$780     │
  └────────┴──────────────┴─────────┴───────────┘
  Recommendation

  The A100 GPU is overkill for this workload. The bottleneck is CPU-bound (pyloudnorm LUFS), and GPU resampling
  provides negligible benefit.

  For cost efficiency:
  - Use c6i.4xlarge (16 vCPUs, $0.68/hr) instead of A100
  - Scale horizontally with 4-8 nodes for faster completion
  - Expected 5× cost reduction with similar or better total throughput

New branch git optimisations. for new approach. create this new optimisations and implement as per the plan or strategy and then do a brnchmark for 100 videos, 500 videos and provide scaling benchmarks for 500k videos with horizontal scaling into new instances suggestions.

# Context from my IDE setup:

## Open tabs:
- optimisations.md: optimisations.md
- Optimisation.md: Optimisation.md
- prepare_shards.py: scripts/prepare_shards.py
- fast_audio_pipeline_v2.py: scripts/fast_audio_pipeline_v2.py
- benchmark_gpu.py: scripts/benchmark_gpu.py

## My request for Codex:
taking suspiciously huge amount of time.

# Context from my IDE setup:

## Open tabs:
- optimisations.md: optimisations.md
- Optimisation.md: Optimisation.md
- prepare_shards.py: scripts/prepare_shards.py
- fast_audio_pipeline_v2.py: scripts/fast_audio_pipeline_v2.py
- benchmark_gpu.py: scripts/benchmark_gpu.py

## My request for Codex:
Done. Here's the summary:

  Files Created
  ┌───────────────────────────────┬─────────────────────┬─────────┐
  │             File              │       Purpose       │ Speedup │
  ├───────────────────────────────┼─────────────────────┼─────────┤
  │ scripts/gpu_audio_metrics.py  │ GPU LUFS + spectral │ 13.6x   │
  ├───────────────────────────────┼─────────────────────┼─────────┤
  │ scripts/gpu_audio_decode.py   │ DALI batch decode   │ 5.2x    │
  ├───────────────────────────────┼─────────────────────┼─────────┤
  │ scripts/optimized_pipeline.py │ Combined pipeline   │ -       │
  └───────────────────────────────┴─────────────────────┴─────────┘
  DALI Clarification

  DALI audio decode is still CPU-based (GPU decode only works for images/video). The 5.2x speedup comes
  from:
  - Parallel batch decoding (8 threads)
  - Prefetching pipeline
  - Integrated decode→GPU transfer

  True GPU audio decode would require a custom CUDA FLAC decoder, which doesn't exist in standard
  libraries.

  Quick Tests

  # GPU metrics (13.6x speedup)
  python3 scripts/gpu_audio_metrics.py

  # DALI decode (5.2x speedup)
  python3 scripts/gpu_audio_decode.py

  # Verify pipeline
  python3 scripts/optimized_pipeline.py

  Your other agent can now benchmark with real videos using the OptimizedPipeline class. The key
  parameters:

  pipeline = OptimizedPipeline(
      gpu_batch_size=256,    # Bigger = faster for metrics
      use_dali=True,         # 5x decode speedup
      use_multiprocess=True, # Bypass GIL
  )
  outputs = pipeline.process_batch(segments)

✻ Baked for 3m 36s

I need you to test this new pipeline dawg. test it on few 100 videos and give time, ETA for 500k and what if we deploy the same worker-claims-videoID and starts working strategy on 5 machines vs 10 machines and ETA for the same scenario. and also, after confirming the optimisations and validating them. i would like you to do a thorough test on 10 videos, by actually examining everything our pipeline does, and creates and read total counts, deltas, and actual test the audios by reading them and undertstanding if they are actuall what we actually need in the shards. end-to-end reliability test and ensuring max compatibility and reliaibility so i'll be relying on your observation and move to next steps.

# Context from my IDE setup:

## Open tabs:
- optimisations.md: optimisations.md
- Optimisation.md: Optimisation.md
- prepare_shards.py: scripts/prepare_shards.py
- fast_audio_pipeline_v2.py: scripts/fast_audio_pipeline_v2.py
- benchmark_gpu.py: scripts/benchmark_gpu.py

## My request for Codex:
yes, files are from another agent. do bench on this new branch and pull real files from R2. dont use synthetic.

# Context from my IDE setup:

## Open tabs:
- optimisations.md: optimisations.md
- Optimisation.md: Optimisation.md
- prepare_shards.py: scripts/prepare_shards.py
- fast_audio_pipeline_v2.py: scripts/fast_audio_pipeline_v2.py
- benchmark_gpu.py: scripts/benchmark_gpu.py

## My request for Codex:
can you debug why is the mismatch occuring and also, can we go higher with workers, or batches, or anything ? or a certain mechanism, to keep both resources busy all the time ? main task is to debug why the mismatch is happening and making sure that GPU pipeline is absolutely fucking accurate and precise. matching with CPU is not the whole point, but making sure GPU ones are accurate is the point. CPU calculations might not be precise too right ? or is it because of precision calculations ? think and debug and findout. and also see if we can go higher in-terms of batching/utilisation to make it even faster. i was assuming the pipleine went higher becayse the other agent mentioned we had 13x speedup with respect to GPU metrics and 5x speedup with DALI. but these ETAs seeems to be normal what you mentioned on 5machines and 10. what is the real bottleneck now ? and how can we fix it to go brrrr insanely quick and fast.

# Context from my IDE setup:

## Open tabs:
- optimized_pipeline.py: scripts/optimized_pipeline.py
- optimisations.md: optimisations.md
- Optimisation.md: Optimisation.md
- prepare_shards.py: scripts/prepare_shards.py
- fast_audio_pipeline_v2.py: scripts/fast_audio_pipeline_v2.py

## My request for Codex:
I actually had the other agent to rethink the strategy and see if we can be more clever on the pipeline. can you do a quick review on the pipeline and understand if the code is correct and make sure if we are on right track and then do a quick test on 100/500 videos so we get an idea on the ETA for 500k videos with 5 and 10 and 15 machines.

The idea is to be clever. one task doesnt have to wait for the other
  task. tars downloaded -> decode only >3s segments + encode everything
  -> quality metrics -> cut bigger segments into smaller >=6s -> and
  cut the already encoded segments based on our cut regions -> use
  pre-calculated LUFs gating, cache hann windows, avoid re-computation,
  do every fucking thing possible. be clever. where are we doing
  re-work. where are we not being clever.

● Let me analyze the pipeline for inefficiencies and propose a clever
  restructuring.

● Read(optimized_pipeline.py)
  ⎿  Read 200 lines

● Read(gpu_audio_metrics.py)
  ⎿  Read 100 lines

● Read(gpu_audio_metrics.py)
  ⎿  Read 120 lines

● Found the issues. Here's the analysis:

  Current Inefficiencies

  1. LUFS Python Loop (lines 308-337) - THE KILLER

  for i in range(batch_size):  # Python loop = GPU sync per iteration
      valid_blocks = block_ms[i, valid_mask[i]]  # GPU->CPU sync
      ...
      results.append(float(final_lufs.cpu()))  # Another sync
  This kills batching. Every .cpu() call syncs the GPU.

  2. Computing Metrics Twice

  - Full segment metrics for high-rate decisions
  - Per-chunk metrics for 16k outputs
  - LUFS is expensive - compute ONCE

  3. Resample→Encode is Sequential

  - Resample chunk → encode → next chunk
  - Could encode full segment ONCE, then slice bytes

  4. No Caching

  - Hann windows recomputed
  - rfftfreq recomputed
  - K-weight coeffs recomputed

  Let me write the optimized LUFS (fully vectorized) and clever pipeline:

● Write(gpu_audio_metrics_v2.py)
  ⎿  Wrote 455 lines to gpu_audio_metrics_v2.py
      1 """GPU-accelerated audio metrics v2 - FULLY VECTORIZED.
      2 Key optimizations over v1:
      3 1. LUFS gating fully vectorized (no Python loops)
      4 2. Spectral metrics fully vectorized
      5 3. All intermediate tensors cached
      6 4. Single GPU sync at the end of batch
      7 Expected: 30-50x speedup over CPU, 3-5x over v1.
      8 """
     … +445 lines (ctrl+o to expand)

● Bash(python3 gpu_audio_metrics_v2.py 2>&1)
  ⎿ /home/ubuntu/.local/lib/python3.10/site-packages/torch/cuda/__init_
    _.py:63: FutureWarning: The pynvml package is deprecated. Please in
    stall nvidia-ml-py instead. If you did not install pynvml directly,
    … +28 lines (ctrl+o to expand)

● V2 results:
  - Batch 256: 2.84ms/seg (V1: 4.61ms) = 1.6x faster
  - Accuracy: 0.042 dB diff (within tolerance)

  Now let me write the clever "encode once, cut later" pipeline:

● Write(clever_pipeline.py)
  ⎿  Wrote 467 lines to clever_pipeline.py
      1 """Clever Audio Pipeline - Encode Once, Cut Later.
      2 Key optimizations:
      3 1. Compute metrics ONCE on full segment (not per-chunk)
      4 2. Encode full segment to all target rates ONCE
      5 3. Find cut points, then slice WAV bytes directly (no re-enco
        de)
      6 4. Parallel decode/encode overlapped with GPU metrics
      7 5. Skip segments < 3s BEFORE decoding
      8 WAV byte slicing works because:
     … +457 lines (ctrl+o to expand)

● Bash(python3 clever_pipeline.py 2>&1) timeout: 30s
  ⎿ /home/ubuntu/.local/lib/python3.10/site-packages/torch/cuda/__init_
    _.py:63: FutureWarning: The pynvml package is deprecated. Please in
    stall nvidia-ml-py instead. If you did not install pynvml directly,
    … +9 lines (ctrl+o to expand)

● Now let me update the docs with what's been created:

● Update(~/maya3_prepare/optimisations.md)
  ⎿  Added 35 lines, removed 14 lines
       91
       92  ### New Files Created
       93
       94 -1. **`scripts/gpu_audio_metrics.py`** - GPU-accelerated LU
          -FS and spectral analysis
       95 -   - ITU-R BS.1770-4 compliant LUFS on GPU
       96 -   - Batched K-weighting filters via torchaudio
       97 -   - Vectorized spectral analysis (FFT on GPU)
       98 -   - **13.6x speedup** at batch 256
       94 +1. **`scripts/gpu_audio_metrics.py`** - GPU LUFS/spectral
          +(v1, has Python loops)
       95 +   - 13.6x speedup at batch 256
       96 +   - BUT: Python loops in LUFS gating cause GPU sync overh
          +ead
       97
      100 -2. **`scripts/gpu_audio_decode.py`** - NVIDIA DALI batch a
          -udio decode
      101 -   - Optimized parallel batch decoding
      102 -   - Prefetching and integrated decode→GPU transfer
       98 +2. **`scripts/gpu_audio_metrics_v2.py`** - FULLY VECTORIZE
          +D (v2, no Python loops)
       99 +   - **2.84ms/seg** at batch 256 (v1: 4.61ms) = **1.6x fas
          +ter**
      100 +   - All gating logic vectorized with tensor masks
      101 +   - Cached Hann windows, rfftfreq, K-weight coefficients
      102 +   - Single GPU sync at end of batch
      103 +
      104 +3. **`scripts/gpu_audio_decode.py`** - NVIDIA DALI batch d
          +ecode
      105     - **5.2x speedup** at batch 64
      104 -   - Note: Audio decode is still CPU (GPU decode is for im
          -ages/video only)
      106 +   - Note: Audio decode is CPU (GPU decode only for images
          +/video)
      107
      106 -3. **`scripts/optimized_pipeline.py`** - Multiprocess + GP
          -U batch pipeline
      107 -   - DALI batch decode (5x faster)
      108 -   - GPU-batched metrics computation
      109 -   - GPU-batched resampling
      110 -   - Async encode with futures
      108 +4. **`scripts/optimized_pipeline.py`** - Original optimize
          +d pipeline (fixed by other agent)
      109 +   - DALI batch decode
      110 +   - GPU-batched metrics
      111 +   - Matches CPU semantics now
      112
      113 +5. **`scripts/clever_pipeline.py`** - NEW: Encode Once, Cu
          +t Later
      114 +   - **Compute metrics ONCE** on full segment (not per-chu
          +nk)
      115 +   - **Encode full segment ONCE** to all target rates
      116 +   - **Slice WAV bytes** directly for chunks (no re-encode
          +!)
      117 +   - Parallel decode overlapped with GPU metrics
      118 +   - Skip <3s segments BEFORE decoding
      119 +
      120 +### Key Insight: WAV Byte Slicing
      121 +
      122 +WAV PCM16 is uncompressed: 2 bytes/sample. Instead of:
      123 +```
      124 +decode → chunk → resample → encode (per chunk)
      125 +```
      126 +We do:
      127 +```
      128 +decode → resample (full) → encode (full) → slice bytes (in
          +stant!)
      129 +```
      130 +
      131 +This eliminates redundant encoding for chunks.
      132 +
      133  ### Usage
      134
      135  ```bash

● Done. Here's the summary of what I created:

  New Optimizations

  1. gpu_audio_metrics_v2.py - Fully Vectorized LUFS

  - No Python loops in LUFS gating (the killer in v1)
  - Batch 256: 2.84ms/seg (v1: 4.61ms) = 1.6x faster
  - Cached Hann windows, rfftfreq, K-weight coefficients
  - Single GPU sync at end of batch

  2. clever_pipeline.py - Encode Once, Cut Later

  OLD: decode → chunk → resample → encode (per chunk) ← REDUNDANT!
  NEW: decode → resample (full) → encode (full) → slice WAV bytes

  Key optimizations:
  - Metrics computed ONCE on full segment
  - Encode ONCE per target rate
  - Slice WAV bytes for chunks (instant - no re-encode!)
  - Skip <3s segments BEFORE decoding

  Files Summary
  ┌─────────────────────────┬──────────────────────────────────────┐
  │          File           │               Purpose                │
  ├─────────────────────────┼──────────────────────────────────────┤
  │ gpu_audio_metrics_v2.py │ Fully vectorized LUFS (1.6x over v1) │
  ├─────────────────────────┼──────────────────────────────────────┤
  │ clever_pipeline.py      │ Encode once, cut later               │
  ├─────────────────────────┼──────────────────────────────────────┤
  │ optimized_pipeline.py   │ Original (fixed by your other agent) │
  └─────────────────────────┴──────────────────────────────────────┘
  Your other agent can now benchmark CleverPipeline vs OptimizedPipeline on
  real data to measure the improvement.

# Context from my IDE setup:

## Open tabs:
- optimized_pipeline.py: scripts/optimized_pipeline.py
- optimisations.md: optimisations.md
- Optimisation.md: Optimisation.md
- prepare_shards.py: scripts/prepare_shards.py
- fast_audio_pipeline_v2.py: scripts/fast_audio_pipeline_v2.py

## My request for Codex:
why tf is that taking forever to finish. anyways, fuck it. stop it. let me come with new plan for you.

# Context from my IDE setup:

## Open tabs:
- optimized_pipeline.py: scripts/optimized_pipeline.py
- optimisations.md: optimisations.md
- Optimisation.md: Optimisation.md
- prepare_shards.py: scripts/prepare_shards.py
- fast_audio_pipeline_v2.py: scripts/fast_audio_pipeline_v2.py

## My request for Codex:
fine dude. let's stop over optimising shit for diminishing returns. let's scale this horizontally to get max gains. i now want you to go through all our implementations, versions and cleanup/ create a new folder and back them into that folder and make sure that only the best versions of these implementations are noted or pointed somewhere. note the safest and best version only. dont risk over no diminising returns. once we have cleaned this. also now give strategy on how to dockerize this pipeline to act as a worker that will work relentlessly to finish the job.

Now, we need to create a new supabase table with all the videos we have, and a worker claiming few 100 videos-mechanism where a worker is given few 100 videos or whatever count of videos that accumulate to a 5GB shard, set this as a fixed thing because we dont need any dynamic sizing. set this as 100 videos or whatver count that gives 5GB shards. then we do this language by language. we need metrics collection, for each worker to provide heartbeat metrics, processed metrics, and completion and other audio metrics as well. and we can create a small dashboard to monitor later depending on metrics from supabase. so plan to push metrics from each worker. And once each worker has finished their current set of 100 or so videos and we have that single shard packed. let's just push this file to R2 and call it done. and claims next 100 or so videos and pushes that shard to R2 into respective buckets and prefixes and be done. and so on. and if there are final set of only 20-30 videos left and we dont have 100, we can just give it off just like that. so the final shard will be smaller. and update the metrics to supabase again. we need to be sure, to queue the worker requests in such a way that no two workers claim same 100 or duplicate videos in anyway, so keep in mind to process in queue manner, and be really strict on this. i am okay scaling this to 20 workers, but i am not okay accepting duplicates or deadlocks or any mismatches in claiming videos. So, see how you can handle this.

Create a sophisticated and robust plan with all corner cases and i will have my other agent work on this. but you will have to do the cleanup on your own. and then start working on the plan. just dont delete the code files or versions non-important. just back them to a diff folder, so we know which one is updated and latest. also, make sure to pick the right version, cause the latest one took forever to complete. so, i dont wanna trust the latest one is the best. i am now at a point of scaling this pipeline to 20 workers at 20 diff instances or machines.

# Context from my IDE setup:

## Open tabs:
- optimized_pipeline.py: scripts/optimized_pipeline.py
- optimisations.md: optimisations.md
- Optimisation.md: Optimisation.md
- prepare_shards.py: scripts/prepare_shards.py
- fast_audio_pipeline_v2.py: scripts/fast_audio_pipeline_v2.py

## My request for Codex:
Can you create a new folder and name it workers/ and put the new plan in it, and all scripts required for the agent to build the pipeline. so it knows what to add as context and dont waste content.

# Context from my IDE setup:

## Open tabs:
- worker.py: workers/worker.py
- Dockerfile.gpu: workers/Dockerfile.gpu
- optimized_pipeline.py: scripts/optimized_pipeline.py
- optimisations.md: optimisations.md
- Optimisation.md: Optimisation.md

## My request for Codex:
can you validate build, validate this workflow one step at a time.

Bash(ls -la /home/ubuntu/maya3_prepare/workers/)
  ⎿  total 172
     drwxrwxr-x 2 ubuntu ubuntu  4096 Jan 20 04:17 .
     drwxrwxr-x 9 ubuntu ubuntu  4096 Jan 20 04:00 ..
     … +18 lines (ctrl+o to expand)

● Let me verify the Python code syntax and imports work correctly.

● Bash(cd /home/ubuntu/maya3_prepare/workers && python3 -c "
      from config import WorkerConfig, EnvConfig…)
  ⎿ All imports successful!
    WorkerConfig fields: ['worker_id', 'language', 'target_videos', 'lease
    _minutes', 'heartbeat_interval_sec'] ...
    … +2 lines (ctrl+o to expand)

● Bash(python3 -c "
      from worker import ShardWorker, parse_args…)
  ⎿ worker.py imports successful!
    ShardWorker methods: ['claim_videos', 'create_shard_run', 'determinist
    ic_shard_id', 'download_videos', 'process_one_shard', 'process_shard',
     'renew_lease', 'run', 'send_heartbeat', 'start_heartbeat']

● All worker components have been implemented and verified. Here's a summary of
   what was created:

  Implemented Files
  File: worker.py
  Size: 27KB
  Purpose: Main worker orchestration with heartbeat, lease renewal, processing
    pipeline
  ────────────────────────────────────────
  File: worker_processing.py
  Size: 33KB
  Purpose: Audio processing logic adapted from prepare_shards.py
  ────────────────────────────────────────
  File: r2_uploader.py
  Size: 13KB
  Purpose: R2 upload with retry logic, progress tracking
  ────────────────────────────────────────
  File: storage_manager.py
  Size: 12KB
  Purpose: Disk monitoring, cleanup, background monitor thread
  ────────────────────────────────────────
  File: config.py
  Size: 6KB
  Purpose: Configuration management from env vars and .env file
  ────────────────────────────────────────
  File: release_stale_claims.py
  Size: 7KB
  Purpose: Recovery script for crashed worker claims
  ────────────────────────────────────────
  File: entrypoint.sh
  Size: 3KB
  Purpose: Docker entrypoint with validation and signal handling
  ────────────────────────────────────────
  File: Dockerfile.cpu
  Size: 1KB
  Purpose: Docker image for CPU workers
  ────────────────────────────────────────
  File: docker-compose.yml
  Size: 2KB
  Purpose: Local testing with docker-compose
  ────────────────────────────────────────
  File: requirements.txt
  Size: 178B
  Purpose: Python dependencies
  ────────────────────────────────────────
  File: README.md
  Size: 11KB
  Purpose: Comprehensive documentation
  Key Features Implemented

  1. No Duplicates: FOR UPDATE SKIP LOCKED in video claiming
  2. Deterministic Shard IDs: Same videos always produce same shard_id
  (idempotent uploads)
  3. Lease Renewal: Long operations renew leases to prevent timeout
  4. Heartbeat Thread: Background thread updates worker_heartbeats every 15s
  5. Storage Monitor: Background thread checks disk usage, triggers cleanup
  6. Graceful Shutdown: Signal handlers for SIGTERM/SIGINT
  7. Retry Logic: Exponential backoff for R2 uploads
  8. Progress Tracking: Real-time updates to database

  Usage

  # Build Docker image
  docker build -f workers/Dockerfile.cpu -t shard-worker .

  # Run worker
  docker run -d \
      -e LANGUAGE=telugu \
      -e DATABASE_URL="..." \
      -e R2_ENDPOINT_URL="..." \
      -e R2_ACCESS_KEY_ID="..." \
      -e R2_SECRET_ACCESS_KEY="..." \
      -e R2_CLEANED_BUCKET="..." \
      -e R2_BUCKET_SHARDS="..." \
      shard-worker

  # Scale to multiple workers
  docker-compose up -d --scale worker=20

  Your other agent can now build the Docker image and test the end-to-end
  workflow.

Any secrets if you need are in .env, one thing at a time. and remember we dont have to auto-scale directly with docker here.  i will provide infra for that. for now, we need to validate if the docker based workflow + supabase sync + analytics + no-duplicates management is working for one worker and the docker image is built, tested on few 100 videos and double checked with R2's respective bucket and re-downloading the shard and opening it to see if teh data is actually present and we are good to scale this pipeline to different machines one after another. for which i will provide further details. build, test, validate the single worker workflow end to end first from here. all secrets are in .env and you are open to do stuff to supabase. just dont make any writes to previous table. you are free to read or clone previous table of videos and other info from that table and leave new table with other status and other info column empty for this current preparateion pipeline.

# Context from my IDE setup:

## Open tabs:
- worker.py: workers/worker.py
- Dockerfile.gpu: workers/Dockerfile.gpu
- optimized_pipeline.py: scripts/optimized_pipeline.py
- optimisations.md: optimisations.md
- Optimisation.md: Optimisation.md

## My request for Codex:
yes i've had my other agent finish the scaffolding you have created previously and it now has all the end to end implementations. i want you to understand the plan, build, fix any issues if it's not as planned and fix workflow and ensure max robustness with end to end implementations.


Summary: auto

## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  3) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  4) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.


**Confirming working tree state**


Output:
Exit code: 0
Wall time: 0 seconds
Output:
total 84
drwxrwxr-x  9 ubuntu ubuntu  4096 Jan 20 04:00 .
drwxr-x--- 32 ubuntu ubuntu  4096 Jan 20 03:41 ..
drwxrwxr-x  2 ubuntu ubuntu  4096 Jan 19 18:54 .claude
drwxr-xr-x  2 ubuntu ubuntu  4096 Jan 20 04:04 docs
-rw-rw-r--  1 ubuntu ubuntu  2645 Jan 19 04:48 .env
drwxrwxr-x  8 ubuntu ubuntu  4096 Jan 20 04:02 .git
-rw-r--r--  1 ubuntu ubuntu 10978 Jan 19 04:03 objective.md
-rw-rw-r--  1 ubuntu ubuntu  6734 Jan 20 03:49 optimisations.md
drwxr-xr-x  4 ubuntu ubuntu  4096 Jan 20 03:49 scripts
-rw-r--r--  1 ubuntu ubuntu 16701 Jan 19 04:03 strategy.md
-rw-r--r--  1 ubuntu ubuntu  1431 Jan 19 04:03 Summary.md
drwxr-xr-x  8 ubuntu ubuntu  4096 Jan 19 04:03 tts
drwxrwxr-x  3 ubuntu ubuntu  4096 Jan 20 04:17 workers
drwxr-xr-x  9 ubuntu ubuntu  4096 Jan 19 05:21 X-Codec-2.0


Summary: auto


**Inspecting worker.py**


rg -n "class ShardWorker|def process_shard|def upload_shard|def cleanup_shard_data|renew_lease|lease" workers/worker.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:
84:class ShardWorker:
194:                    lease_expires_at = now() + (%s || ' minutes')::interval,
204:                    str(self.config.lease_minutes),
222:                    status, claimed_by, lease_expires_at, shard_prefix, output_format
230:                    lease_expires_at = excluded.lease_expires_at,
239:                    str(self.config.lease_minutes),
287:    def renew_lease(self, conn, shard_id: uuid.UUID, video_ids: list[str]) -> None:
288:        """Renew lease for current shard and videos."""
290:            # Renew shard lease
294:                set lease_expires_at = now() + (%s || ' minutes')::interval,
298:                (str(self.config.lease_minutes), str(shard_id)),
300:            # Renew video leases
304:                set lease_expires_at = now() + (%s || ' minutes')::interval,
308:                (str(self.config.lease_minutes), video_ids),
438:    def process_shard(
503:    def upload_shard(
570:            # Renew lease before processing
572:                self.renew_lease(conn, shard_id, video_ids)
579:            # Renew lease before upload
581:                self.renew_lease(conn, shard_id, video_ids)
696:    parser.add_argument("--lease-minutes", type=int, default=60, help="Lease duration")
739:        lease_minutes=args.lease_minutes,


**Requesting worker.py sections**


Output:
Exit code: 0
Wall time: 0 seconds
Output:
#!/usr/bin/env python3
"""Distributed shard worker for horizontal scaling.

This worker:
1. Claims a batch of videos from Supabase (with row-level locking)
2. Downloads video tars from R2
3. Processes segments to generate shards
4. Uploads shards to R2
5. Updates status in Supabase
6. Cleans up local storage
7. Reports heartbeat and metrics

Key safety features:
- Deterministic shard_id (same videos = same shard = idempotent upload)
- FOR UPDATE SKIP LOCKED prevents duplicate claims
- Lease expiry allows crash recovery
- Storage monitoring prevents disk exhaustion
"""

from __future__ import annotations

import argparse
import hashlib
import json
import logging
import os
import signal
import socket
import sys
import threading
import time
import uuid
from dataclasses import dataclass, field
from pathlib import Path
from typing import Any, Optional

import psycopg
from boto3.s3.transfer import TransferConfig

# Setup logging
logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s [%(levelname)s] %(name)s: %(message)s",
    datefmt="%Y-%m-%d %H:%M:%S",
)
logger = logging.getLogger("worker")

if __package__ in (None, ""):
    sys.path.insert(0, str(Path(__file__).resolve().parents[1]))

# Import worker modules
from workers.config import WorkerConfig, EnvConfig
from workers.storage_manager import StorageManager, StorageStats
from workers.r2_uploader import R2Uploader
from workers.worker_processing import (
    VideoProcessor,
    ChunkConfig,
    SilenceConfig,
    ProcessingStats,
)

@dataclass(frozen=True)
class TaskRow:
    """A video task from the database."""
    video_id: str
    r2_tar_key: str
    language: str

@dataclass
class WorkerState:
    """Current worker state for heartbeat reporting."""
    status: str = "idle"
    shard_id: Optional[str] = None
    videos_assigned: int = 0
    videos_processed: int = 0
    segments_processed: int = 0
    outputs_generated: int = 0
    last_error: Optional[str] = None
    processing_start: Optional[float] = None

class ShardWorker:
    """Main worker class that orchestrates the processing pipeline."""

    def __init__(
        self,
        config: WorkerConfig,
        env_config: EnvConfig,
    ):
        self.config = config
        self.env_config = env_config

        # Generate worker ID if not provided
        self.worker_id = config.worker_id or f"{socket.gethostname()}-{os.getpid()}"
        logger.info(f"Worker ID: {self.worker_id}")

        # Initialize R2 clients
        self.source_uploader = R2Uploader(
            endpoint_url=env_config.r2_endpoint_url,
            access_key_id=env_config.r2_access_key_id,
            secret_access_key=env_config.r2_secret_access_key,
            bucket=env_config.r2_cleaned_bucket,
            max_retries=config.max_retries,
            retry_delay_sec=config.retry_delay_sec,
            max_concurrency=config.download_workers,
        )

        self.shard_uploader = R2Uploader(
            endpoint_url=env_config.r2_endpoint_url,
            access_key_id=env_config.r2_access_key_id,
            secret_access_key=env_config.r2_secret_access_key,
            bucket=env_config.r2_bucket_shards,
            max_retries=config.max_retries,
            retry_delay_sec=config.retry_delay_sec,
            max_concurrency=config.upload_workers,
        )

        # Initialize storage manager
        self.storage_manager = StorageManager(
            work_dir=config.work_dir,
            output_dir=config.output_dir,
            max_usage_gb=config.max_disk_usage_gb,
            cleanup_threshold_gb=config.cleanup_threshold_gb,
            check_interval_sec=config.cleanup_check_interval_sec,
        )

        # Worker state
        self.state = WorkerState()
        self._state_lock = threading.Lock()
        self._stop_event = threading.Event()
        self._heartbeat_thread: Optional[threading.Thread] = None

        # Setup signal handlers
        signal.signal(signal.SIGTERM, self._handle_signal)
        signal.signal(signal.SIGINT, self._handle_signal)

    def _handle_signal(self, signum, frame):
        """Handle shutdown signals gracefully."""
        logger.info(f"Received signal {signum}, initiating graceful shutdown...")
        self._stop_event.set()

    def _update_state(self, **kwargs) -> None:
        """Thread-safe state update."""
        with self._state_lock:
            for key, value in kwargs.items():
                if hasattr(self.state, key):
                    setattr(self.state, key, value)

    def _get_state(self) -> dict:
        """Get current state as dict."""
        with self._state_lock:
            return {
                "status": self.state.status,
                "shard_id": self.state.shard_id,
                "videos_assigned": self.state.videos_assigned,
                "videos_processed": self.state.videos_processed,
                "segments_processed": self.state.segments_processed,
                "outputs_generated": self.state.outputs_generated,
                "last_error": self.state.last_error,
            }

    def deterministic_shard_id(self, language: str, video_ids: list[str]) -> uuid.UUID:
        """Generate deterministic shard ID from language + sorted video IDs.

        This ensures that re-processing the same videos produces the same shard_id,
        making uploads idempotent (overwrites same R2 key instead of creating duplicates).
        """
        joined = f"{language}:" + ",".join(sorted(video_ids))
        digest = hashlib.sha1(joined.encode("utf-8")).hexdigest()
        return uuid.uuid5(uuid.NAMESPACE_URL, digest)

    def claim_videos(self, conn) -> list[TaskRow]:
        """Claim a batch of pending videos with row-level locking.

        Uses FOR UPDATE SKIP LOCKED to prevent duplicate claims across workers.
        """
        with conn.cursor() as cur:
            cur.execute(
                """
                with picked as (
                    select video_id
                    from video_tasks
                    where language = %s
                      and status = 'pending'
                    order by video_id
                    limit %s
                    for update skip locked
                )
                update video_tasks
                set status = 'claimed',
                    claimed_by = %s,
                    lease_expires_at = now() + (%s || ' minutes')::interval,
                    attempts = attempts + 1,
                    updated_at = now()
                where video_id in (select video_id from picked)
                returning video_id, r2_tar_key, language
                """,
                (
                    self.config.language,
                    self.config.target_videos,
                    self.worker_id,
                    str(self.config.lease_minutes),
                ),
            )
            rows = cur.fetchall()
        return [TaskRow(*row) for row in rows]

    def create_shard_run(
        self,
        conn,
        shard_id: uuid.UUID,
        assigned_count: int,
    ) -> None:
        """Create shard_runs entry for tracking."""
        with conn.cursor() as cur:
            cur.execute(
                """
                insert into shard_runs (


Output:
Exit code: 0
Wall time: 0 seconds
Output:
                insert into shard_runs (
                    shard_id, language, target_video_count, assigned_video_count,
                    status, claimed_by, lease_expires_at, shard_prefix, output_format
                ) values (
                    %s, %s, %s, %s,
                    'claimed', %s, now() + (%s || ' minutes')::interval, %s, %s
                )
                on conflict (shard_id) do update set
                    status = 'claimed',
                    claimed_by = excluded.claimed_by,
                    lease_expires_at = excluded.lease_expires_at,
                    updated_at = now()
                """,
                (
                    str(shard_id),
                    self.config.language,
                    self.config.target_videos,
                    assigned_count,
                    self.worker_id,
                    str(self.config.lease_minutes),
                    f"{self.config.language}/{shard_id}",
                    self.config.output_format,
                ),
            )

    def update_task_status(
        self,
        conn,
        video_ids: list[str],
        status: str,
        shard_id: Optional[uuid.UUID] = None,
        error: Optional[str] = None,
    ) -> None:
        """Update status for video tasks."""
        with conn.cursor() as cur:
            cur.execute(
                """
                update video_tasks
                set status = %s,
                    shard_id = %s,
                    last_error = %s,
                    updated_at = now()
                where video_id = any(%s)
                """,
                (status, str(shard_id) if shard_id else None, error, video_ids),
            )

    def update_shard_status(
        self,
        conn,
        shard_id: uuid.UUID,
        status: str,
        metrics: Optional[dict] = None,
    ) -> None:
        """Update shard_runs status."""
        with conn.cursor() as cur:
            cur.execute(
                """
                update shard_runs
                set status = %s,
                    metrics = coalesce(%s, metrics),
                    updated_at = now()
                where shard_id = %s
                """,
                (status, json.dumps(metrics) if metrics else None, str(shard_id)),
            )

    def renew_lease(self, conn, shard_id: uuid.UUID, video_ids: list[str]) -> None:
        """Renew lease for current shard and videos."""
        with conn.cursor() as cur:
            # Renew shard lease
            cur.execute(
                """
                update shard_runs
                set lease_expires_at = now() + (%s || ' minutes')::interval,
                    updated_at = now()
                where shard_id = %s
                """,
                (str(self.config.lease_minutes), str(shard_id)),
            )
            # Renew video leases
            cur.execute(
                """
                update video_tasks
                set lease_expires_at = now() + (%s || ' minutes')::interval,
                    updated_at = now()
                where video_id = any(%s)
                """,
                (str(self.config.lease_minutes), video_ids),
            )

    def send_heartbeat(self, conn) -> None:
        """Send heartbeat to database."""
        state = self._get_state()
        storage = self.storage_manager.get_stats()

        with conn.cursor() as cur:
            cur.execute(
                """
                insert into worker_heartbeats (
                    worker_id, hostname, instance_type, language, status,
                    shard_id, videos_assigned, videos_processed,
                    segments_processed, outputs_generated, last_error, last_heartbeat
                ) values (
                    %s, %s, %s, %s, %s,
                    %s, %s, %s,
                    %s, %s, %s, now()
                )
                on conflict (worker_id) do update set
                    status = excluded.status,
                    shard_id = excluded.shard_id,
                    videos_assigned = excluded.videos_assigned,
                    videos_processed = excluded.videos_processed,
                    segments_processed = excluded.segments_processed,
                    outputs_generated = excluded.outputs_generated,
                    last_error = excluded.last_error,
                    last_heartbeat = now()
                """,
                (
                    self.worker_id,
                    socket.gethostname(),
                    os.environ.get("INSTANCE_TYPE", "unknown"),
                    self.config.language,
                    state["status"],
                    state["shard_id"],
                    state["videos_assigned"],
                    state["videos_processed"],
                    state["segments_processed"],
                    state["outputs_generated"],
                    state["last_error"],
                ),
            )

    def _heartbeat_loop(self) -> None:
        """Background heartbeat loop."""
        while not self._stop_event.is_set():
            try:
                with psycopg.connect(self.env_config.database_url) as conn:
                    self.send_heartbeat(conn)
                    conn.commit()
            except Exception as e:
                logger.warning(f"Heartbeat failed: {e}")

            self._stop_event.wait(self.config.heartbeat_interval_sec)

    def start_heartbeat(self) -> None:
        """Start background heartbeat thread."""
        if self._heartbeat_thread is not None:
            return
        self._heartbeat_thread = threading.Thread(
            target=self._heartbeat_loop,
            name="heartbeat",
            daemon=True,
        )
        self._heartbeat_thread.start()
        logger.info("Heartbeat thread started")

    def stop_heartbeat(self) -> None:
        """Stop heartbeat thread."""
        self._stop_event.set()
        if self._heartbeat_thread:
            self._heartbeat_thread.join(timeout=5)
            self._heartbeat_thread = None

    def download_videos(
        self,
        tasks: list[TaskRow],
        download_dir: Path,
    ) -> dict[str, Path]:
        """Download video tars from R2."""
        download_dir.mkdir(parents=True, exist_ok=True)
        download_paths: dict[str, Path] = {}

        transfer_config = TransferConfig(
            multipart_threshold=8 * 1024 * 1024,
            multipart_chunksize=64 * 1024 * 1024,
            max_concurrency=max(4, self.config.download_workers),
            use_threads=True,
        )

        for i, task in enumerate(tasks):
            if self._stop_event.is_set():
                break

            # Check storage before download
            if self.storage_manager.needs_cleanup():
                logger.warning("Storage threshold exceeded, running cleanup...")
                self.storage_manager.cleanup()

            local_path = download_dir / f"{task.video_id}.tar"
            if local_path.exists():
                download_paths[task.video_id] = local_path
                continue

            # Resolve R2 key
            resolved = self.source_uploader.resolve_key(
                task.r2_tar_key,
                self.env_config.r2_prefix,
            )
            if not resolved:
                logger.warning(f"Could not resolve R2 key: {task.r2_tar_key}")
                continue

            # Download
            try:
                if self.source_uploader.download_file(resolved, local_path):
                    download_paths[task.video_id] = local_path
                    logger.debug(f"Downloaded {task.video_id} ({i+1}/{len(tasks)})")
                else:
                    logger.error(f"Failed to download {task.video_id}")
            except Exception as e:
                logger.error(f"Download error for {task.video_id}: {e}")

            # Update progress
            self._update_state(videos_processed=len(download_paths))

        return download_paths

    def process_shard(
        self,
        shard_id: uuid.UUID,
        download_paths: dict[str, Path],
    ) -> ProcessingStats:
        """Process downloaded videos into shards."""
        self.config.output_dir.mkdir(parents=True, exist_ok=True)

        chunk_cfg = ChunkConfig(
            enabled=self.config.chunk_enabled,
            min_sec=self.config.chunk_min_sec,
            max_sec=self.config.chunk_max_sec,
            threshold_sec=self.config.chunk_threshold_sec,
            all_buckets=self.config.chunk_all_buckets,
        )
        silence_cfg = SilenceConfig(
            enabled=self.config.silence_enabled,
            frame_sec=self.config.silence_frame_ms / 1000.0,
            hop_sec=self.config.silence_hop_ms / 1000.0,
            search_sec=self.config.silence_search_sec,
            percentile=self.config.silence_percentile,
            db_drop=self.config.silence_db_drop,
        )

        # Use shard_id as prefix for deterministic output naming
        shard_prefix = f"shard-{str(shard_id)[:8]}"

        def on_progress(processed: int, total: int):
            self._update_state(videos_processed=processed)
            # Check storage during processing
            if processed % 10 == 0 and self.storage_manager.needs_cleanup():
                logger.warning("Storage threshold during processing, running cleanup...")
                self.storage_manager.cleanup()

        processor = VideoProcessor(
            output_dir=self.config.output_dir,
            language=self.config.language,
            shard_prefix=shard_prefix,
            chunk_cfg=chunk_cfg,
            silence_cfg=silence_cfg,
            output_format=self.config.output_format,
            target_shard_size_mb=self.config.target_shard_size_mb,
            segment_workers=self.config.segment_workers,
            seed=self.config.seed,
            val_ratio=self.config.val_ratio,
            on_progress=on_progress,
        )

        try:
            stats = processor.process_videos(download_paths)
            processor.close()
            processor.save_stats()

            # Update state with processing stats
            total_outputs = sum(stats.segments_kept.values())
            self._update_state(
                segments_processed=stats.segments_total,
                outputs_generated=total_outputs,
            )

            return stats
        except Exception as e:
            processor.close()
            raise

    def upload_shard(
        self,
        shard_id: uuid.UUID,
    ) -> bool:
        """Upload processed shard to R2."""
        shard_prefix = f"shard-{str(shard_id)[:8]}"

        result = self.shard_uploader.upload_shard(
            output_dir=self.config.output_dir,
            language=self.config.language,
            shard_id=str(shard_id)[:8],
            audio_ext="wav" if self.config.output_format == "wav" else "flac",
        )

        if result.success:
            logger.info(
                f"Uploaded shard {shard_id}: {len(result.uploaded_keys)} files, "
                f"{result.total_bytes / (1024**3):.2f}GB"


Output:
Exit code: 0
Wall time: 0 seconds
Output:
                f"{result.total_bytes / (1024**3):.2f}GB"
            )
        else:
            logger.error(f"Upload failed for shard {shard_id}: {result.errors}")

        return result.success

    def process_one_shard(self) -> bool:
        """Process a single shard from claim to upload.

        Returns True if a shard was processed, False if no work available.
        """
        # Claim videos
        self._update_state(status="claiming")
        with psycopg.connect(self.env_config.database_url) as conn:
            conn.execute("begin")
            tasks = self.claim_videos(conn)

            if not tasks:
                conn.execute("commit")
                logger.info("No pending videos available")
                return False

            video_ids = [t.video_id for t in tasks]
            shard_id = self.deterministic_shard_id(self.config.language, video_ids)

            logger.info(f"Claimed {len(tasks)} videos, shard_id={shard_id}")

            # Create shard run and mark videos as processing
            self.create_shard_run(conn, shard_id, len(tasks))
            self.update_task_status(conn, video_ids, "processing", shard_id=shard_id)
            conn.execute("commit")

        self._update_state(
            status="processing",
            shard_id=str(shard_id),
            videos_assigned=len(tasks),
            videos_processed=0,
            processing_start=time.time(),
        )

        try:
            # Download
            self._update_state(status="downloading")
            download_dir = self.config.work_dir / str(shard_id)
            download_paths = self.download_videos(tasks, download_dir)

            if not download_paths:
                raise RuntimeError("No videos downloaded successfully")

            # Renew lease before processing
            with psycopg.connect(self.env_config.database_url) as conn:
                self.renew_lease(conn, shard_id, video_ids)
                conn.commit()

            # Process
            self._update_state(status="processing")
            stats = self.process_shard(shard_id, download_paths)

            # Renew lease before upload
            with psycopg.connect(self.env_config.database_url) as conn:
                self.renew_lease(conn, shard_id, video_ids)
                conn.commit()

            # Upload
            self._update_state(status="uploading")
            if not self.upload_shard(shard_id):
                raise RuntimeError("Upload failed")

            # Mark completed
            with psycopg.connect(self.env_config.database_url) as conn:
                self.update_shard_status(conn, shard_id, "completed", metrics=stats.to_dict())
                self.update_task_status(conn, video_ids, "completed", shard_id=shard_id)
                conn.commit()

            logger.info(f"Completed shard {shard_id} with {len(download_paths)} videos")

            # Cleanup
            self._update_state(status="cleanup")
            self.storage_manager.cleanup_shard_data(str(shard_id), download_dir)
            self.storage_manager.mark_shard_uploaded(str(shard_id))

            self._update_state(status="idle", shard_id=None)
            return True

        except Exception as e:
            logger.exception(f"Failed to process shard {shard_id}")
            self._update_state(status="failed", last_error=str(e))

            # Mark as failed in database
            try:
                with psycopg.connect(self.env_config.database_url) as conn:
                    self.update_shard_status(conn, shard_id, "failed")
                    self.update_task_status(
                        conn, video_ids, "failed", shard_id=shard_id, error=str(e)
                    )
                    conn.commit()
            except Exception as db_err:
                logger.error(f"Failed to update failure status: {db_err}")

            return False

    def run(self) -> None:
        """Main worker loop."""
        logger.info(f"Starting worker for language={self.config.language}")

        # Validate config
        errors = self.env_config.validate()
        if errors:
            for error in errors:
                logger.error(f"Config error: {error}")
            raise SystemExit(1)

        # Start background threads
        self.start_heartbeat()
        self.storage_manager.start_monitor()

        # Set storage callbacks
        def on_storage_critical(stats: StorageStats):
            logger.warning(
                f"Storage critical: {stats.work_dir_gb + stats.output_dir_gb:.1f}GB used, "
                f"{stats.free_gb:.1f}GB free"
            )
            self._update_state(last_error="Storage critical")

        self.storage_manager.set_callbacks(on_storage_critical=on_storage_critical)

        try:
            while not self._stop_event.is_set():
                try:
                    processed = self.process_one_shard()

                    if self.config.once:
                        logger.info("Single shard mode, exiting")
                        break

                    if not processed:
                        # No work available, wait before retry
                        logger.info("No work available, sleeping 30s...")
                        self._stop_event.wait(30)

                except Exception as e:
                    logger.exception(f"Error in main loop: {e}")
                    self._update_state(status="error", last_error=str(e))
                    # Wait before retry
                    self._stop_event.wait(60)

        finally:
            logger.info("Shutting down worker...")
            self.stop_heartbeat()
            self.storage_manager.stop_monitor()

            # Final heartbeat with idle status
            try:
                self._update_state(status="stopped")
                with psycopg.connect(self.env_config.database_url) as conn:
                    self.send_heartbeat(conn)
                    conn.commit()
            except Exception:
                pass

            logger.info("Worker shutdown complete")

def parse_args() -> argparse.Namespace:
    """Parse command line arguments."""
    parser = argparse.ArgumentParser(description="Distributed shard worker")

    # Required
    parser.add_argument("--language", required=True, help="Language to process")

    # Worker identity
    parser.add_argument("--worker-id", default=None, help="Worker ID (auto-generated if not set)")

    # Processing settings
    parser.add_argument("--target-videos", type=int, default=100, help="Videos per shard")
    parser.add_argument("--lease-minutes", type=int, default=60, help="Lease duration")

    # Directories
    parser.add_argument("--work-dir", type=Path, default=Path("/tmp/worker"))
    parser.add_argument("--output-dir", type=Path, default=Path("/tmp/worker_output"))

    # Storage
    parser.add_argument("--max-disk-gb", type=float, default=150.0, help="Max disk usage GB")
    parser.add_argument("--cleanup-threshold-gb", type=float, default=100.0)

    # Parallelism
    parser.add_argument("--download-workers", type=int, default=8)
    parser.add_argument("--segment-workers", type=int, default=8)
    parser.add_argument("--upload-workers", type=int, default=4)

    # Output
    parser.add_argument("--output-format", choices=["flac", "wav"], default="flac")
    parser.add_argument("--shard-size-mb", type=int, default=5120)

    # Run mode
    parser.add_argument("--once", action="store_true", help="Process single shard and exit")
    parser.add_argument("--dry-run", action="store_true", help="Don't actually process")

    # Config file
    parser.add_argument("--env-file", type=Path, default=Path(".env"))

    return parser.parse_args()

def main():
    args = parse_args()

    # Load environment config
    if args.env_file.exists():
        env_config = EnvConfig.from_file(args.env_file)
    else:
        env_config = EnvConfig.from_env_vars()

    # Create worker config from args
    worker_config = WorkerConfig(
        worker_id=args.worker_id or "",
        language=args.language,
        target_videos=args.target_videos,
        lease_minutes=args.lease_minutes,
        work_dir=args.work_dir,
        output_dir=args.output_dir,
        max_disk_usage_gb=args.max_disk_gb,
        cleanup_threshold_gb=args.cleanup_threshold_gb,
        download_workers=args.download_workers,
        segment_workers=args.segment_workers,
        upload_workers=args.upload_workers,
        output_format=args.output_format,
        target_shard_size_mb=args.shard_size_mb,
        once=args.once,
        dry_run=args.dry_run,
    )

    worker = ShardWorker(worker_config, env_config)
    worker.run()

if __name__ == "__main__":
    main()


**Identifying needed modifications and checking worker_processing.py**


rg -n "ShardWriter|target_shard_size" workers/worker_processing.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:
664:class ShardWriter:
738:        target_shard_size_mb: int = 5120,
756:        target_bytes = target_shard_size_mb * 1024 * 1024
759:            16000: ShardWriter(
762:            24000: ShardWriter(
765:            32000: ShardWriter(
768:            44100: ShardWriter(


sed -n '640,820p' workers/worker_processing.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:
                    chunk_idx,
                    chunk_start,
                    chunk_end,
                    chunk_original_start,
                    chunk_original_end,
                    slice_targets,
                )
        else:
            emit_outputs(
                audio,
                full_metrics,
                sample_id_base,
                index,
                None,
                segment_start,
                segment_end,
                original_start,
                original_end,
                targets_high,
            )

    return outputs, stats

class ShardWriter:
    """Write samples to tar + jsonl shards."""

    def __init__(
        self, output_dir: Path, prefix: str, target_size_bytes: int, audio_ext: str
    ) -> None:
        self.output_dir = output_dir
        self.prefix = prefix
        self.target_size_bytes = target_size_bytes
        self.audio_ext = audio_ext
        self.shard_index = 0
        self.current_tar: Optional[tarfile.TarFile] = None
        self.current_jsonl = None
        self.current_size = 0
        self.current_count = 0
        self.current_name = ""
        self.output_dir.mkdir(parents=True, exist_ok=True)

    def _open_new_shard(self) -> None:
        if self.current_tar:
            self.close()
        shard_name = f"{self.prefix}-{self.shard_index:06d}"
        tar_path = self.output_dir / f"{shard_name}.tar"
        jsonl_path = self.output_dir / f"{shard_name}.jsonl"
        self.current_tar = tarfile.open(tar_path, "w")
        self.current_jsonl = jsonl_path.open("w", encoding="utf-8")
        self.current_size = 0
        self.current_count = 0
        self.current_name = shard_name
        self.shard_index += 1

    def add_sample(self, sample_id: str, audio_bytes: bytes, metadata: dict[str, Any]) -> str:
        if self.current_tar is None:
            self._open_new_shard()
        if (
            self.current_count > 0
            and self.current_size + len(audio_bytes) > self.target_size_bytes
        ):
            self._open_new_shard()

        assert self.current_tar is not None
        assert self.current_jsonl is not None

        arcname = f"{sample_id}.{self.audio_ext}"
        info = tarfile.TarInfo(name=arcname)
        info.size = len(audio_bytes)
        info.mtime = int(time.time())
        self.current_tar.addfile(info, io.BytesIO(audio_bytes))
        self.current_jsonl.write(json.dumps(metadata) + "\n")

        self.current_size += len(audio_bytes)
        self.current_count += 1
        return self.current_name

    def close(self) -> None:
        if self.current_tar:
            self.current_tar.close()
        if self.current_jsonl:
            self.current_jsonl.close()
        self.current_tar = None
        self.current_jsonl = None

class VideoProcessor:
    """Process videos and write to shards."""

    def __init__(
        self,
        output_dir: Path,
        language: str,
        shard_prefix: str,
        chunk_cfg: ChunkConfig,
        silence_cfg: SilenceConfig,
        output_format: str = "flac",
        target_shard_size_mb: int = 5120,
        segment_workers: int = 8,
        seed: int = 17,
        val_ratio: float = 0.01,
        on_progress: Optional[Callable[[int, int], None]] = None,
    ):
        self.output_dir = output_dir
        self.language = language
        self.shard_prefix = shard_prefix
        self.chunk_cfg = chunk_cfg
        self.silence_cfg = silence_cfg
        self.output_format = output_format
        self.seed = seed
        self.val_ratio = val_ratio
        self.segment_workers = segment_workers
        self.on_progress = on_progress

        audio_ext = "wav" if output_format == "wav" else "flac"
        target_bytes = target_shard_size_mb * 1024 * 1024

        self.writers = {
            16000: ShardWriter(
                output_dir / f"{language}_16", shard_prefix, target_bytes, audio_ext
            ),
            24000: ShardWriter(
                output_dir / f"{language}_24", shard_prefix, target_bytes, audio_ext
            ),
            32000: ShardWriter(
                output_dir / f"{language}_32", shard_prefix, target_bytes, audio_ext
            ),
            44100: ShardWriter(
                output_dir / f"{language}_44", shard_prefix, target_bytes, audio_ext
            ),
        }

        # Manifests
        manifest_dir = output_dir / "manifests"
        manifest_dir.mkdir(parents=True, exist_ok=True)
        self.train_manifest = (manifest_dir / f"{language}_train_{shard_prefix}.jsonl").open(
            "w", encoding="utf-8"
        )
        self.val_manifest = (manifest_dir / f"{language}_val_{shard_prefix}.jsonl").open(
            "w", encoding="utf-8"
        )

        self.stats = ProcessingStats()
        self.audio_ext = audio_ext

    def process_videos(
        self,
        download_paths: dict[str, Path],
        val_ids: Optional[set[str]] = None,
    ) -> ProcessingStats:
        """Process all videos and write to shards."""
        if val_ids is None:
            # Determine val set
            video_ids = list(download_paths.keys())
            random.seed(self.seed)
            val_count = max(1, int(len(video_ids) * self.val_ratio)) if len(video_ids) > 1 else 0
            val_ids = set(random.sample(video_ids, val_count)) if val_count > 0 else set()

        max_inflight = max(128, self.segment_workers * 2)
        pending: set[Any] = set()
        future_info: dict[Any, bool] = {}

        def handle_future(future) -> None:
            is_val = future_info.pop(future, False)
            outputs, seg_stats = future.result()
            self.stats.dropped_too_short += seg_stats["dropped_too_short"]
            self.stats.dropped_too_long += seg_stats["dropped_too_long"]
            self.stats.dropped_clipped += seg_stats["dropped_clipped"]
            self.stats.segments_chunked += seg_stats.get("segments_chunked", 0)
            self.stats.chunks_emitted += seg_stats.get("chunks_emitted", 0)

            for out in outputs:
                self.stats.segments_kept[out.sample_rate] += 1
                if out.sample_rate == 16000:
                    bucket = out.metadata.get("native_sr_bucket")
                    if bucket:
                        self.stats.native_sr_buckets[bucket] = (
                            self.stats.native_sr_buckets.get(bucket, 0) + 1
                        )
                writer = self.writers[out.sample_rate]


**Adjusting sample method for zero target size**

I'm planning to modify the sample method in ShardWriter to properly handle cases where target_size_bytes is zero or negative, starting by reviewing workers/config.py for related configurations.


Output:
Exit code: 0
Wall time: 0 seconds
Output:
"""Configuration management for workers."""

from __future__ import annotations

import os
from dataclasses import dataclass, field
from pathlib import Path
from typing import Optional

@dataclass
class WorkerConfig:
    """Worker configuration with defaults and environment overrides."""

    # Worker identity
    worker_id: str = ""
    language: str = ""

    # Processing settings
    target_videos: int = 100  # Videos per shard (approx 5GB)
    lease_minutes: int = 60  # Lease duration for claims
    heartbeat_interval_sec: int = 15  # Heartbeat frequency

    # Directory settings
    work_dir: Path = field(default_factory=lambda: Path("/tmp/worker"))
    output_dir: Path = field(default_factory=lambda: Path("/tmp/worker_output"))

    # Storage management
    max_disk_usage_gb: float = 150.0  # Max disk usage before cleanup
    cleanup_threshold_gb: float = 100.0  # Target after cleanup
    cleanup_check_interval_sec: int = 60  # Storage check frequency

    # Parallelism
    download_workers: int = 8
    segment_workers: int = 8
    upload_workers: int = 4

    # Output settings
    output_format: str = "flac"  # flac or wav
    target_shard_size_mb: int = 5120  # 5GB shards

    # Processing options
    chunk_enabled: bool = True
    chunk_min_sec: float = 6.0
    chunk_max_sec: float = 10.0
    chunk_threshold_sec: float = 12.0
    chunk_all_buckets: bool = False

    silence_enabled: bool = True
    silence_frame_ms: float = 50.0
    silence_hop_ms: float = 10.0
    silence_search_sec: float = 0.5
    silence_percentile: float = 20.0
    silence_db_drop: float = 20.0

    val_ratio: float = 0.01
    seed: int = 17

    # Retry settings
    max_retries: int = 3
    retry_delay_sec: float = 5.0

    # Run mode
    once: bool = False  # Process single shard and exit
    dry_run: bool = False  # Don't actually process

    @classmethod
    def from_env(cls) -> "WorkerConfig":
        """Create config from environment variables."""
        return cls(
            worker_id=os.environ.get("WORKER_ID", ""),
            language=os.environ.get("LANGUAGE", ""),
            target_videos=int(os.environ.get("TARGET_VIDEOS", "100")),
            lease_minutes=int(os.environ.get("LEASE_MINUTES", "60")),
            heartbeat_interval_sec=int(os.environ.get("HEARTBEAT_INTERVAL_SEC", "15")),
            work_dir=Path(os.environ.get("WORK_DIR", "/tmp/worker")),
            output_dir=Path(os.environ.get("OUTPUT_DIR", "/tmp/worker_output")),
            max_disk_usage_gb=float(os.environ.get("MAX_DISK_USAGE_GB", "150")),
            cleanup_threshold_gb=float(os.environ.get("CLEANUP_THRESHOLD_GB", "100")),
            cleanup_check_interval_sec=int(os.environ.get("CLEANUP_CHECK_INTERVAL_SEC", "60")),
            download_workers=int(os.environ.get("DOWNLOAD_WORKERS", "8")),
            segment_workers=int(os.environ.get("SEGMENT_WORKERS", "8")),
            upload_workers=int(os.environ.get("UPLOAD_WORKERS", "4")),
            output_format=os.environ.get("OUTPUT_FORMAT", "flac"),
            target_shard_size_mb=int(os.environ.get("TARGET_SHARD_SIZE_MB", "5120")),
            max_retries=int(os.environ.get("MAX_RETRIES", "3")),
            retry_delay_sec=float(os.environ.get("RETRY_DELAY_SEC", "5")),
            once=os.environ.get("ONCE", "").lower() in ("1", "true", "yes"),
            dry_run=os.environ.get("DRY_RUN", "").lower() in ("1", "true", "yes"),
        )

@dataclass
class EnvConfig:
    """Environment configuration from .env file."""

    # Database
    database_url: str = ""

    # R2 Storage
    r2_endpoint_url: str = ""
    r2_access_key_id: str = ""
    r2_secret_access_key: str = ""
    r2_cleaned_bucket: str = ""  # Source bucket with video tars
    r2_bucket_shards: str = ""  # Destination bucket for shards
    r2_prefix: Optional[str] = None

    @classmethod
    def from_file(cls, env_path: Path) -> "EnvConfig":
        """Load from .env file."""
        env: dict[str, str] = {}
        if env_path.exists():
            for line in env_path.read_text().splitlines():
                stripped = line.strip()
                if not stripped or stripped.startswith("#") or "=" not in stripped:
                    continue
                if stripped.startswith("export "):
                    stripped = stripped[len("export "):]
                key, value = stripped.split("=", 1)
                env[key.strip()] = value.strip().strip('"').strip("'")

        return cls(
            database_url=env.get("DATABASE_URL", ""),
            r2_endpoint_url=env.get("R2_ENDPOINT_URL", ""),
            r2_access_key_id=env.get("R2_ACCESS_KEY_ID", ""),
            r2_secret_access_key=env.get("R2_SECRET_ACCESS_KEY", ""),
            r2_cleaned_bucket=env.get("R2_CLEANED_BUCKET") or env.get("R2_ClEANED_BUCKET", ""),
            r2_bucket_shards=env.get("R2_BUCKET_SHARDS", ""),
            r2_prefix=env.get("R2_PREFIX"),
        )

    @classmethod
    def from_env_vars(cls) -> "EnvConfig":
        """Load from environment variables."""
        return cls(
            database_url=os.environ.get("DATABASE_URL", ""),
            r2_endpoint_url=os.environ.get("R2_ENDPOINT_URL", ""),
            r2_access_key_id=os.environ.get("R2_ACCESS_KEY_ID", ""),
            r2_secret_access_key=os.environ.get("R2_SECRET_ACCESS_KEY", ""),
            r2_cleaned_bucket=os.environ.get("R2_CLEANED_BUCKET") or os.environ.get("R2_ClEANED_BUCKET", ""),
            r2_bucket_shards=os.environ.get("R2_BUCKET_SHARDS", ""),
            r2_prefix=os.environ.get("R2_PREFIX"),
        )

    def validate(self) -> list[str]:
        """Validate configuration, return list of errors."""
        errors = []
        if not self.database_url:
            errors.append("DATABASE_URL is required")
        if not self.r2_endpoint_url:
            errors.append("R2_ENDPOINT_URL is required")
        if not self.r2_access_key_id:
            errors.append("R2_ACCESS_KEY_ID is required")
        if not self.r2_secret_access_key:
            errors.append("R2_SECRET_ACCESS_KEY is required")
        if not self.r2_cleaned_bucket:
            errors.append("R2_CLEANED_BUCKET is required")
        if not self.r2_bucket_shards:
            errors.append("R2_BUCKET_SHARDS is required")
        return errors


**Planning fixed shard size default**

I'll update the target shard size default to 0 to disable size-based splitting, matching the user's request for fixed video count shards, and clarify this in the README while ensuring compatibility with related components.


Output:
Exit code: 0
Wall time: 0 seconds
Output:
"""R2 upload module with retry logic and progress tracking."""

from __future__ import annotations

import logging
import os
import time
from dataclasses import dataclass, field
from pathlib import Path
from typing import Any, Callable, Optional

import boto3
from boto3.s3.transfer import TransferConfig
from botocore.exceptions import ClientError

logger = logging.getLogger(__name__)

@dataclass
class UploadResult:
    """Result of an upload operation."""

    success: bool = True
    uploaded_keys: list[str] = field(default_factory=list)
    failed_files: list[str] = field(default_factory=list)
    total_bytes: int = 0
    elapsed_sec: float = 0.0
    errors: list[str] = field(default_factory=list)

class R2Uploader:
    """Handles uploads to R2 with retry logic and progress tracking."""

    def __init__(
        self,
        endpoint_url: str,
        access_key_id: str,
        secret_access_key: str,
        bucket: str,
        max_retries: int = 3,
        retry_delay_sec: float = 5.0,
        max_concurrency: int = 4,
    ):
        self.bucket = bucket
        self.max_retries = max_retries
        self.retry_delay_sec = retry_delay_sec

        self.client = boto3.client(
            "s3",
            endpoint_url=endpoint_url,
            aws_access_key_id=access_key_id,
            aws_secret_access_key=secret_access_key,
        )

        self.transfer_config = TransferConfig(
            multipart_threshold=8 * 1024 * 1024,  # 8MB
            multipart_chunksize=64 * 1024 * 1024,  # 64MB
            max_concurrency=max_concurrency,
            use_threads=True,
        )

        # Progress callback
        self._on_progress: Optional[Callable[[str, int, int], None]] = None

    def set_progress_callback(
        self, callback: Optional[Callable[[str, int, int], None]]
    ) -> None:
        """Set callback for upload progress: callback(filename, bytes_transferred, total_bytes)."""
        self._on_progress = callback

    def upload_file(
        self,
        local_path: Path,
        remote_key: str,
        metadata: Optional[dict[str, str]] = None,
    ) -> bool:
        """Upload a single file with retry logic.

        Args:
            local_path: Path to local file
            remote_key: Remote key in bucket
            metadata: Optional metadata to attach

        Returns:
            True if upload succeeded, False otherwise
        """
        if not local_path.exists():
            logger.error(f"File not found: {local_path}")
            return False

        file_size = local_path.stat().st_size

        for attempt in range(self.max_retries):
            try:
                extra_args = {}
                if metadata:
                    extra_args["Metadata"] = metadata

                # Create progress callback for this file
                callback = None
                if self._on_progress:
                    def make_callback(filename: str, total: int):
                        bytes_transferred = 0

                        def progress(chunk_bytes: int):
                            nonlocal bytes_transferred
                            bytes_transferred += chunk_bytes
                            if self._on_progress:
                                self._on_progress(filename, bytes_transferred, total)

                        return progress

                    callback = make_callback(local_path.name, file_size)

                self.client.upload_file(
                    str(local_path),
                    self.bucket,
                    remote_key,
                    Config=self.transfer_config,
                    ExtraArgs=extra_args if extra_args else None,
                    Callback=callback,
                )

                logger.debug(f"Uploaded {local_path.name} -> {remote_key}")
                return True

            except ClientError as e:
                error_code = e.response.get("Error", {}).get("Code", "Unknown")
                logger.warning(
                    f"Upload attempt {attempt + 1}/{self.max_retries} failed for "
                    f"{local_path.name}: {error_code} - {e}"
                )

                if attempt < self.max_retries - 1:
                    delay = self.retry_delay_sec * (2 ** attempt)  # Exponential backoff
                    logger.info(f"Retrying in {delay:.1f}s...")
                    time.sleep(delay)

            except Exception as e:
                logger.error(f"Unexpected error uploading {local_path.name}: {e}")
                if attempt < self.max_retries - 1:
                    time.sleep(self.retry_delay_sec)

        logger.error(f"Failed to upload {local_path} after {self.max_retries} attempts")
        return False

    def upload_directory(
        self,
        local_dir: Path,
        remote_prefix: str,
        pattern: str = "*",
    ) -> UploadResult:
        """Upload all files from a directory.

        Args:
            local_dir: Local directory to upload from
            remote_prefix: Remote prefix for all files
            pattern: Glob pattern to match files

        Returns:
            UploadResult with details
        """
        result = UploadResult()
        start_time = time.time()

        if not local_dir.exists():
            result.success = False
            result.errors.append(f"Directory not found: {local_dir}")
            return result

        files = list(local_dir.glob(pattern))
        if not files:
            logger.warning(f"No files matching '{pattern}' in {local_dir}")
            return result

        for local_path in sorted(files):
            if not local_path.is_file():
                continue

            remote_key = f"{remote_prefix.rstrip('/')}/{local_path.name}"
            file_size = local_path.stat().st_size

            if self.upload_file(local_path, remote_key):
                result.uploaded_keys.append(remote_key)
                result.total_bytes += file_size
            else:
                result.failed_files.append(str(local_path))

        result.elapsed_sec = time.time() - start_time
        result.success = len(result.failed_files) == 0

        logger.info(
            f"Directory upload complete: {len(result.uploaded_keys)} files, "
            f"{result.total_bytes / (1024**2):.1f}MB in {result.elapsed_sec:.1f}s"
        )

        return result

    def upload_shard(
        self,
        output_dir: Path,
        language: str,
        shard_id: str,
        audio_ext: str = "flac",
    ) -> UploadResult:
        """Upload a complete shard (tar + jsonl for each sample rate).

        This uploads:
        - {language}_16/shard-{shard_id}.tar and .jsonl
        - {language}_24/shard-{shard_id}.tar and .jsonl (if exists)
        - {language}_32/shard-{shard_id}.tar and .jsonl (if exists)
        - {language}_44/shard-{shard_id}.tar and .jsonl (if exists)

        Args:
            output_dir: Base output directory
            language: Language code
            shard_id: Shard identifier
            audio_ext: Audio extension (flac or wav)

        Returns:


sed -n '220,420p' workers/r2_uploader.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:
        Returns:
            UploadResult with details
        """
        result = UploadResult()
        start_time = time.time()

        sr_map = {
            16000: "16",
            24000: "24",
            32000: "32",
            44100: "44",
        }

        for sr, sr_suffix in sr_map.items():
            local_dir = output_dir / f"{language}_{sr_suffix}"
            if not local_dir.exists():
                continue

            # Upload tar file
            tar_pattern = f"*-{shard_id}*.tar"
            for tar_file in local_dir.glob(tar_pattern):
                remote_key = f"{language}_{sr_suffix}/{tar_file.name}"
                if self.upload_file(tar_file, remote_key):
                    result.uploaded_keys.append(remote_key)
                    result.total_bytes += tar_file.stat().st_size
                else:
                    result.failed_files.append(str(tar_file))

            # Upload jsonl file
            jsonl_pattern = f"*-{shard_id}*.jsonl"
            for jsonl_file in local_dir.glob(jsonl_pattern):
                remote_key = f"{language}_{sr_suffix}/{jsonl_file.name}"
                if self.upload_file(jsonl_file, remote_key):
                    result.uploaded_keys.append(remote_key)
                    result.total_bytes += jsonl_file.stat().st_size
                else:
                    result.failed_files.append(str(jsonl_file))

        # Upload manifests
        manifest_dir = output_dir / "manifests"
        if manifest_dir.exists():
            for manifest_file in manifest_dir.glob(f"*{shard_id}*"):
                remote_key = f"manifests/{manifest_file.name}"
                if self.upload_file(manifest_file, remote_key):
                    result.uploaded_keys.append(remote_key)
                    result.total_bytes += manifest_file.stat().st_size
                else:
                    result.failed_files.append(str(manifest_file))

        # Upload stats
        stats_dir = output_dir / "stats"
        if stats_dir.exists():
            for stats_file in stats_dir.glob(f"*{shard_id}*"):
                remote_key = f"stats/{stats_file.name}"
                if self.upload_file(stats_file, remote_key):
                    result.uploaded_keys.append(remote_key)
                    result.total_bytes += stats_file.stat().st_size
                else:
                    result.failed_files.append(str(stats_file))

        result.elapsed_sec = time.time() - start_time
        result.success = len(result.failed_files) == 0

        if result.success:
            logger.info(
                f"Shard {shard_id} uploaded: {len(result.uploaded_keys)} files, "
                f"{result.total_bytes / (1024**2):.1f}MB in {result.elapsed_sec:.1f}s"
            )
        else:
            result.errors.append(
                f"Failed to upload {len(result.failed_files)} files for shard {shard_id}"
            )

        return result

    def verify_upload(self, remote_key: str) -> bool:
        """Verify that a file exists in R2.

        Args:
            remote_key: Remote key to verify

        Returns:
            True if file exists
        """
        try:
            self.client.head_object(Bucket=self.bucket, Key=remote_key)
            return True
        except ClientError as e:
            if e.response.get("Error", {}).get("Code") == "404":
                return False
            raise

    def download_file(
        self,
        remote_key: str,
        local_path: Path,
    ) -> bool:
        """Download a file from R2.

        Args:
            remote_key: Remote key to download
            local_path: Local path to save to

        Returns:
            True if download succeeded
        """
        local_path.parent.mkdir(parents=True, exist_ok=True)

        for attempt in range(self.max_retries):
            try:
                self.client.download_file(
                    self.bucket,
                    remote_key,
                    str(local_path),
                    Config=self.transfer_config,
                )
                return True

            except ClientError as e:
                logger.warning(
                    f"Download attempt {attempt + 1}/{self.max_retries} failed: {e}"
                )
                if attempt < self.max_retries - 1:
                    time.sleep(self.retry_delay_sec * (2 ** attempt))

        return False

    def resolve_key(
        self,
        r2_key: str,
        prefix: Optional[str] = None,
    ) -> Optional[str]:
        """Resolve R2 key, trying multiple path variants.

        Args:
            r2_key: R2 key to resolve
            prefix: Optional prefix to try

        Returns:
            Resolved key or None if not found
        """
        if not r2_key:
            return None

        key = r2_key.lstrip("/")
        candidates = [key]

        if prefix:
            cleaned_prefix = prefix.strip("/")
            if not key.startswith(cleaned_prefix):
                candidates.append(f"{cleaned_prefix}/{key}")

        base = os.path.basename(key)
        if base != key:
            candidates.append(base)
            if prefix:
                candidates.append(f"{prefix.strip('/')}/{base}")

        for candidate in candidates:
            try:
                self.client.head_object(Bucket=self.bucket, Key=candidate)
                return candidate
            except ClientError:
                continue

        return None


**Confirming uploader pattern alignment**

The R2 uploader expects shard IDs to appear as a pattern with the full UUID, so the writer must use the complete UUID as a prefix to ensure matching. Considering verifying the storage_manager's handling of this.


rg -n "cleanup_shard_data|mark_shard_uploaded" workers/storage_manager.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:
119:    def mark_shard_uploaded(self, shard_id: str) -> None:
215:    def cleanup_shard_data(self, shard_id: str, shard_work_dir: Path) -> CleanupResult:


sed -n '100,260p' workers/storage_manager.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:
        try:
            for entry in path.rglob("*"):
                if entry.is_file():
                    try:
                        total += entry.stat().st_size
                    except (OSError, FileNotFoundError):
                        pass
        except Exception as e:
            logger.warning(f"Error calculating dir size for {path}: {e}")
        return total / (1024 ** 3)

    def needs_cleanup(self) -> bool:
        """Check if cleanup is needed based on current usage."""
        stats = self.get_stats()
        return (
            stats.work_dir_gb + stats.output_dir_gb > self.max_usage_gb
            or stats.free_gb < 10.0  # Always clean if less than 10GB free
        )

    def mark_shard_uploaded(self, shard_id: str) -> None:
        """Mark a shard as uploaded (safe to delete local files)."""
        with self._lock:
            self._uploaded_shards.add(shard_id)

    def cleanup(self, force: bool = False) -> CleanupResult:
        """Perform cleanup of old/uploaded data.

        Cleanup priority:
        1. Uploaded shards (output_dir)
        2. Downloaded tars for completed shards (work_dir)
        3. Temporary files
        """
        with self._lock:
            if self._cleanup_in_progress and not force:
                return CleanupResult(success=False, errors=["Cleanup already in progress"])
            self._cleanup_in_progress = True

        if self._on_cleanup_start:
            self._on_cleanup_start()

        result = CleanupResult()
        initial_stats = self.get_stats()
        initial_used = initial_stats.work_dir_gb + initial_stats.output_dir_gb

        try:
            # 1. Clean up uploaded shard data
            for shard_id in list(self._uploaded_shards):
                # Clean shard directory in work_dir
                shard_work_dir = self.work_dir / shard_id
                if shard_work_dir.exists():
                    try:
                        shutil.rmtree(shard_work_dir)
                        result.dirs_removed += 1
                        logger.info(f"Cleaned up work dir for shard {shard_id}")
                    except Exception as e:
                        result.errors.append(f"Failed to clean {shard_work_dir}: {e}")

                # Clean output files for this shard
                for subdir in self.output_dir.iterdir():
                    if subdir.is_dir():
                        for f in subdir.glob(f"*{shard_id}*"):
                            try:
                                f.unlink()
                                result.files_removed += 1
                            except Exception as e:
                                result.errors.append(f"Failed to remove {f}: {e}")

                with self._lock:
                    self._uploaded_shards.discard(shard_id)

            # 2. Clean old temporary files (older than 2 hours)
            cutoff = time.time() - 7200  # 2 hours
            for path in self.work_dir.rglob("*"):
                if path.is_file():
                    try:
                        if path.stat().st_mtime < cutoff:
                            path.unlink()
                            result.files_removed += 1
                    except (OSError, FileNotFoundError):
                        pass

            # 3. Clean empty directories
            for path in sorted(self.work_dir.rglob("*"), reverse=True):
                if path.is_dir():
                    try:
                        if not any(path.iterdir()):
                            path.rmdir()
                            result.dirs_removed += 1
                    except (OSError, FileNotFoundError):
                        pass

            # Calculate freed space
            final_stats = self.get_stats()
            final_used = final_stats.work_dir_gb + final_stats.output_dir_gb
            result.freed_gb = max(0, initial_used - final_used)
            result.success = True

        except Exception as e:
            result.success = False
            result.errors.append(f"Cleanup failed: {e}")
            logger.exception("Cleanup failed")

        finally:
            with self._lock:
                self._cleanup_in_progress = False

        if self._on_cleanup_complete:
            self._on_cleanup_complete(result)

        logger.info(
            f"Cleanup complete: freed {result.freed_gb:.2f}GB, "
            f"removed {result.files_removed} files, {result.dirs_removed} dirs"
        )
        return result

    def cleanup_shard_data(self, shard_id: str, shard_work_dir: Path) -> CleanupResult:
        """Clean up all data for a specific shard after successful upload."""
        result = CleanupResult()

        try:
            # Remove shard work directory (downloads)
            if shard_work_dir.exists():
                shutil.rmtree(shard_work_dir)
                result.dirs_removed += 1
                logger.info(f"Removed shard work dir: {shard_work_dir}")

            # Remove output files for this shard
            for sr_dir in self.output_dir.iterdir():
                if sr_dir.is_dir():
                    for f in sr_dir.glob(f"*-{shard_id}*"):
                        try:
                            f.unlink()
                            result.files_removed += 1
                        except Exception as e:
                            result.errors.append(f"Failed to remove {f}: {e}")

            # Remove manifest entries if they exist
            manifest_dir = self.output_dir / "manifests"
            if manifest_dir.exists():
                for f in manifest_dir.glob(f"*{shard_id}*"):
                    try:
                        f.unlink()
                        result.files_removed += 1
                    except Exception as e:
                        result.errors.append(f"Failed to remove {f}: {e}")

            result.success = True

        except Exception as e:
            result.success = False
            result.errors.append(str(e))
            logger.exception(f"Failed to cleanup shard {shard_id}")

        return result

    def start_monitor(self) -> None:
        """Start background storage monitoring thread."""
        if self._monitor_thread is not None:
            return

        self._stop_event.clear()


**Noting shard_id format impact**

I observed that cleanup uses shard_id as a string in the glob pattern and it's compatible with full UUIDs; I'll update release_stale_claims.py accordingly.


sed -n '1,200p' workers/release_stale_claims.py