Instruction:
## Skills
These skills are discovered at startup from multiple local sources. Each entry includes a name, description, and file path so you can open the source for full instructions.
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
- Discovery: Available skills are listed in project docs and may also appear in a runtime "## Skills" section (name + description + file path). These are the sources of truth; skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  3) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  4) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Description as trigger: The YAML `description` in `SKILL.md` is the primary trigger signal; rely on it to decide applicability. If unsure, ask a brief clarification before proceeding.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deeply nested references; prefer one-hop files explicitly linked from `SKILL.md`.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.


# AGENTS.md instructions for /home/ubuntu/maya3

<INSTRUCTIONS>
## Skills
These skills are discovered at startup from multiple local sources. Each entry includes a name, description, and file path so you can open the source for full instructions.
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
- Discovery: Available skills are listed in project docs and may also appear in a runtime "## Skills" section (name + description + file path). These are the sources of truth; skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  3) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  4) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Description as trigger: The YAML `description` in `SKILL.md` is the primary trigger signal; rely on it to decide applicability. If unsure, ask a brief clarification before proceeding.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deeply nested references; prefer one-hop files explicitly linked from `SKILL.md`.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.
</INSTRUCTIONS>


<environment_context>
  <cwd>/home/ubuntu/maya3</cwd>
  <approval_policy>on-request</approval_policy>
  <sandbox_mode>workspace-write</sandbox_mode>
  <network_access>restricted</network_access>
  <shell>bash</shell>
</environment_context>


<environment_context>
  <approval_policy>never</approval_policy>
  <sandbox_mode>danger-full-access</sandbox_mode>
  <network_access>enabled</network_access>
  <shell>bash</shell>
</environment_context>


# Context from my IDE setup:

## Open tabs:
- .gitignore: .gitignore
- reset_for_production.py: scripts/reset_for_production.py
- requirements.txt: requirements.txt
- Dockerfile: Dockerfile
- models.py: src/models.py

## My request for Codex:
Logs for instance 29758334
Search Filter
ex. Rate Limit Error
Line Count
1000
Thu Jan  8 04:34:19 UTC 2026
Hit:1 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  InRelease
Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease
Hit:3 http://archive.ubuntu.com/ubuntu jammy InRelease
Hit:4 http://archive.ubuntu.com/ubuntu jammy-updates InRelease
Hit:5 http://archive.ubuntu.com/ubuntu jammy-backports InRelease
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
curl is already the newest version (7.81.0-1ubuntu1.21).
less is already the newest version (590-1ubuntu0.22.04.3).
locales is already the newest version (2.35-0ubuntu3.11).
openssh-server is already the newest version (1:8.9p1-3ubuntu0.13).
software-properties-common is already the newest version (0.99.22.9).
sudo is already the newest version (1.9.9-1ubuntu2.5).
tmux is already the newest version (3.2a-4ubuntu0.2).
wget is already the newest version (1.21.2-2ubuntu1.1).
0 upgraded, 0 newly installed, 0 to remove and 65 not upgraded.
Reading package lists...
Building dependency tree...
Reading state information...
git is already the newest version (1:2.34.1-1ubuntu1.15).
rsync is already the newest version (3.2.7-0ubuntu0.22.04.4).
0 upgraded, 0 newly installed, 0 to remove and 65 not upgraded.
rm: cannot remove '/etc/forward_port2': No such file or directory
Code not found, cloning...
Cloning into '/tmp/code'...
Server listening on 0.0.0.0 port 22.
Server listening on :: port 22.
Thu Jan  8 04:34:24 UTC 2026
Warning: Permanently added 'ssh7.vast.ai' (ED25519) to the list of known hosts.
/usr/bin/python: No module named src.worker

And

Logs for instance 29758333
Search Filter
ex. Rate Limit Error
Line Count
1000
Thu Jan  8 04:34:24 UTC 2026
Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease
Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease
Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease
Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease
Hit:5 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  InRelease
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
curl is already the newest version (7.81.0-1ubuntu1.21).
less is already the newest version (590-1ubuntu0.22.04.3).
locales is already the newest version (2.35-0ubuntu3.11).
openssh-server is already the newest version (1:8.9p1-3ubuntu0.13).
software-properties-common is already the newest version (0.99.22.9).
sudo is already the newest version (1.9.9-1ubuntu2.5).
tmux is already the newest version (3.2a-4ubuntu0.2).
wget is already the newest version (1.21.2-2ubuntu1.1).
0 upgraded, 0 newly installed, 0 to remove and 65 not upgraded.
Reading package lists...
Building dependency tree...
Reading state information...
git is already the newest version (1:2.34.1-1ubuntu1.15).
rsync is already the newest version (3.2.7-0ubuntu0.22.04.4).
0 upgraded, 0 newly installed, 0 to remove and 65 not upgraded.
rm: cannot remove '/etc/forward_port2': No such file or directory
Code not found, cloning...
Cloning into '/tmp/code'...
Server listening on 0.0.0.0 port 22.
Server listening on :: port 22.
Thu Jan  8 04:34:29 UTC 2026
Warning: Permanently added 'ssh8.vast.ai' (ED25519) to the list of known hosts.
/usr/bin/python: No module named src.worker

delete the current last 2 instances and Instance ID: 29758333 and Instance ID: 29758334, and create 2 new instances with proper code details as well to avoid the startup error as `/usr/bin/python: No module named src.worker`

[GPU5] 5/9  Embeddings     3.5s  | 303 chunks
Checkpoint path: /root/panns_data/Cnn14_mAP=0.431.pth
04:39:16 | Worker-3     | INFO     | Processing: a7E_Gjk54qo - Sohni Mahiwal Dey Ishq Da Kharrak | Qissa Goi Podc
[GPU1] PICK   a7E_Gjk54qo
[GPU5] 6/9  Clustering     0.5s
[GPU3] 1/9  Download      33.4s
[GPU5] 6/9  MicroContami   0.0s
[GPU0] 1/9  Download      12.7s
04:39:18 | FastPipelineV6.Models | ERROR    | ❌ Failed to load PANNs: Error(s) in loading state_dict for Cnn14:
	size mismatch for fc_audioset.weight: copying a param with shape torch.Size([527, 2048]) from checkpoint, the shape in current model is torch.Size([235, 2048]).
	size mismatch for fc_audioset.bias: copying a param with shape torch.Size([527]) from checkpoint, the shape in current model is torch.Size([235]).
[GPU5] 1/9  Download      20.1s
[GPU2] 5/9  Embeddings     2.8s  | 640 chunks
[GPU2] 6/9  Clustering     0.5s
[GPU7] 1/9  Download      41.2

can you also fix this issue with music model. and also, i've just checked the files are not being uploaded to R2. debug why that is happening as well.

can you check R2 for where the files are being uploaded. are they being uploaded at all ? i cant seem to find files in R2. check and debug vast's logs as well.

and one more issue i found in logs is, although not sure if the worker is retrying after this, will leave it up to you.
[GPU2] 6/9  Clustering     0.0s
04:58:49 | Worker-9     | INFO     | Processing: ZOxuMyEtbWo - What customer crave | PART - 03 | Magic 20 தமிழ்
[GPU4] PICK   ZOxuMyEtbWo
[GPU6] 1/9  Download       2.8s
[GPU2] 6/9  MicroContami   0.0s
04:58:49 | Worker-0     | ERROR    | Worker failed: The read operation timed out
Traceback (most recent call last):
  File "/workspace/maya3_data/massive_process.py", line 536, in gpu_worker_supabase
    full_video = supabase.get_video(video_id)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/maya3_data/src/supabase_client.py", line 380, in get_video
    result = self._request('GET', 'videos', params=params)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/maya3_data/src/supabase_client.py", line 105, in _request
    with urlopen(req, timeout=30) as resp:
         ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/urllib/request.py", line 216, in urlopen
    return opener.open(url, data, timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/urllib/request.py", line 519, in open
    response = self._open(req, data)
               ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/urllib/request.py", line 536, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/urllib/request.py", line 496, in _call_chain
    result = func(*args)
             ^^^^^^^^^^^
  File "/usr/lib/python3.11/urllib/request.py", line 1391, in https_open
    return self.do_open(http.client.HTTPSConnection, req,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/urllib/request.py", line 1352, in do_open
    r = h.getresponse()
        ^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/http/client.py", line 1374, in getresponse
    response.begin()
  File "/usr/lib/python3.11/http/client.py", line 318, in begin
    version, status, reason = self._read_status()
                              ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/http/client.py", line 279, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/socket.py", line 705, in readinto
    return self._sock.recv_into(b)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/ssl.py", line 1278, in recv_into
    return self.read(nbytes, buffer)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/ssl.py", line 1134, in read
    return self._sslobj.read(len, buffer)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TimeoutError: The read operation timed out
[GPU2] 7/9  MusicDetecti   1.0s
[GPU2] 8/9  Finalize       0.0s

Steps.
1. Fix all these issues.
2. delete all the data from supabase from analytics table and reset entire videosIDs to pending so everything starts from beginning.
3. i didnt get any success with recycle command before. i hope there is an option to tell the instance to repull the image and restart, with newer code changes. if not i'd prefer delete the instances and create new ones.
4. quick suggestions for new instances, clusters of 8*3090s, 8*4090s and avoid chinese as their internet is slower, use any region to deploy these as long as they have proper disk, internet, and reliability.
5. deploy into one instance, wait for deployment logs, check running status, and validate if R2 uploads are being done peroperly and everything is alright, then deploy into 9 more totalling 10 instances and 80GPUs running.
6. if any of these fail, i will monitor with UI and will delete them so we can chose a diff instance.


Summary: auto


**Implementing logging and error handling**

I want to focus on implementing logging and error handling. I'll check for any environment variables or configurations since there could be an issue with uploading to the wrong bucket or prefix. I'll consider using the AWS CLI to check the R2 content as well. The user is asking if I can check R2 for uploads, but I might not have the necessary credentials. I also need to look into Supabase timeouts and consider retrying with exponential backoff for errors I encounter.


**Ensuring worker reliability**

I need to ensure the worker continues running since the log shows it failed. The user wonders if the worker is retrying, and I should implement a retry mechanism to handle failures gracefully. They also want to delete all data from the Supabase analytics table and reset video IDs to pending, which might require a script—perhaps `scripts/reset_for_production.py`. Lastly, the user mentioned issues with redeploying instances using the recycle command and wonders if I can suggest pulling a new image and restarting the container.


**Managing instances with Vast.ai**

The user wants to work with 2 instances now and possibly expand to 10 later. As a coding assistant, I could run the Vast.ai CLI, provided the API key is available in the environment. They’ve asked to delete the current last 2 instances and create new ones, but I may not be able to create instances without their Vast account and API key. I'll start by inspecting the repository structure and files to find how the worker is started and review the Supabase and R2 code.


**Creating an operational plan**

I need to be cautious with any destructive operations since the user has asked for changes. Here’s my plan:

First, I’ll reproduce and locate the startup error, ensuring the packaging or entry point correctly identifies `src.worker`. Next, I’ll address the model loading mismatch with PANNs. Then, I’ll check the R2 upload path, add logging, and verify uploads with local tests. I'll enhance retries for Supabase requests to avoid worker crashes. Additionally, I need to provide commands for resetting Supabase analytics and managing Vast instances, after inspecting the repository and running commands.


Output: Plan updated


find .. -maxdepth 4 -name AGENTS.md -print


rg -n "src\.worker|worker\.py|python -m src\.worker|uvicorn|gunicorn|entrypoint" -S .


rg -n "R2|r2|boto3|s3|cloudflare|S3" -S src scripts *.py


rg -n "PANN|Cnn14|fc_audioset|audioset" -S src scripts *.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:
total 384
drwxrwxr-x 13 ubuntu ubuntu   4096 Jan  8 04:16 .
drwxr-x--- 38 ubuntu ubuntu   4096 Jan  8 04:52 ..
drwxrwxr-x  3 ubuntu ubuntu   4096 Jan  8 00:45 .cache
drwx------  2 ubuntu ubuntu   4096 Jan  8 04:44 .claude
drwxrwxr-x  7 ubuntu ubuntu   4096 Jan  8 04:29 dashboard-ui
drwxrwxr-x  3 ubuntu ubuntu   4096 Jan  8 00:32 data
-rw-rw-r--  1 ubuntu ubuntu   1950 Jan  6 22:23 docker-compose.yml
-rw-rw-r--  1 ubuntu ubuntu   4064 Jan  8 01:50 Dockerfile
-rw-rw-r--  1 ubuntu ubuntu    213 Jan  6 22:23 .dockerignore
-rw-rw-r--  1 ubuntu ubuntu   3704 Jan  6 22:23 .env
-rw-rw-r--  1 ubuntu ubuntu    470 Jan  6 22:23 .env.example
drwxrwxr-x  9 ubuntu ubuntu   4096 Jan  8 05:04 .git
-rw-rw-r--  1 ubuntu ubuntu    568 Jan  8 04:16 .gitignore
-rwxrwxr-x  1 ubuntu ubuntu  49067 Jan  8 00:31 massive_process.py
-rw-rw-r--  1 ubuntu ubuntu 131296 Jan  8 01:02 metadata.json
drwxr-xr-x  4 root   root     4096 Jan  8 02:27 output
-rw-rw-r--  1 ubuntu ubuntu  67119 Jan  8 01:06 pipeline.py
drwxrwxr-x  2 ubuntu ubuntu   4096 Jan  7 23:43 __pycache__
-rw-rw-r--  1 ubuntu ubuntu   3654 Jan  8 00:31 README.md
-rw-rw-r--  1 ubuntu ubuntu    623 Jan  8 02:14 requirements.txt
drwxrwxr-x  5 ubuntu ubuntu   4096 Jan  8 02:30 scripts
drwxrwxr-x  3 ubuntu ubuntu   4096 Jan  8 04:57 src
-rw-rw-r--  1 ubuntu ubuntu  30615 Jan  6 22:23 vast_ai.md
-rw-rw-r--  1 ubuntu ubuntu  19445 Jan  6 22:23 vast_ai_sdk.md
drwxrwxr-x  2 ubuntu ubuntu   4096 Jan  8 04:16 .vercel
drwxrwxr-x  2 ubuntu ubuntu   4096 Jan  6 22:31 .xdg_run


Output:
Exit code: 0
Wall time: 0 seconds
Output:


Output:
Exit code: 0
Wall time: 0 seconds
Output:
./docker-compose.yml:62:    entrypoint: ["/workspace/maya3_data/.venv/bin/python"]
./Dockerfile:111:ENTRYPOINT ["/workspace/maya3_data/.venv/bin/python", "massive_process.py"]
./vast_ai.md:231:| `--entrypoint CMD` | Override container entrypoint |
./vast_ai.md:232:| `--args ARGS` | Arguments passed to entrypoint |
./vast_ai.md:1326:| `args` | Custom entrypoint args |
./src/worker_pools.py:12:    from src.worker_pools import GlobalVADPool, DownloadPool
./scripts/docker/smoke_test.sh:23:echo "==> Basic entrypoint check (help output)"
./scripts/docker/smoke_test.sh:28:  --entrypoint /workspace/maya3_data/.venv/bin/python \
./scripts/docker/smoke_test.sh:33:  --entrypoint /workspace/maya3_data/.venv/bin/python \
./scripts/docker/smoke_test.sh:38:  --entrypoint /workspace/maya3_data/.venv/bin/python \


Output:
Exit code: 0
Wall time: 0 seconds
Total output lines: 293
Output:
pipeline.py:170:    total_stages = 10 if getattr(config, 'r2_upload_enabled', False) else 9
pipeline.py:873:        'source_url': metadata.get('source_url'),  # R2/YouTube URL for re-download
pipeline.py:874:        'source_type': metadata.get('source_type', 'youtube'),  # 'r2' or 'youtube'
pipeline.py:979:    export_enabled = getattr(config, 'r2_upload_enabled', False) or getattr(config, 'segment_audio', False)
pipeline.py:985:        upload_prefix = (getattr(config, 'r2_upload_prefix', '') or '').strip().strip('/')
pipeline.py:1003:                    r2_upload_enabled=getattr(config, 'r2_upload_enabled', False),
pipeline.py:1004:                    r2_upload_key=remote_key,
pipeline.py:1005:                    r2_bucket_type=getattr(config, 'r2_upload_bucket_type', 'production'),
pipeline.py:1007:                    delete_output_dir_after_upload=getattr(config, 'r2_upload_delete_local_output_dir', False),
pipeline.py:1012:                result['r2_upload_key'] = remote_key
pipeline.py:1019:            from src.r2_client import R2Client
pipeline.py:1038:                # Step 1.5: Remove original audio from output (source is preserved in R2)
pipeline.py:1043:                        logger.debug(f"Removed _original.flac from output (source in R2)")
pipeline.py:1051:                # Step 3: Upload to R2 (if enabled)
pipeline.py:1053:                if getattr(config, 'r2_upload_enabled', False):
pipeline.py:1054:                    r2 = R2Client(bucket_type=getattr(config, 'r2_upload_bucket_type', 'production'))
pipeline.py:1055:                    if getattr(config, 'r2_upload_skip_if_exists', False) and r2.exists(remote_key):
pipeline.py:1056:                        logger.info(f"R2 upload skipped (exists): s3://{r2.bucket}/{remote_key}")
pipeline.py:1058:                        upload_ok = r2.upload_file(str(tar_path), remote_key)
pipeline.py:1060:                            raise RuntimeError(f"R2 upload failed for s3://{r2.bucket}/{remote_key}")
pipeline.py:1063:                    if getattr(config, 'r2_upload_delete_local_tar', True):
pipeline.py:1070:                    if getattr(config, 'r2_upload_enabled', False) and getattr(config, 'r2_upload_delete_local_output_dir', False):
pipeline.py:1082:                'r2_upload_key': remote_key if getattr(config, 'r2_upload_enabled', False) else None,
massive_process.py:68:# Optional: load .env for HF/R2/Supabase credentials
massive_process.py:104:    r2_vars = {
massive_process.py:105:        'R2_ENDPOINT_URL': 'Cloudflare R2 endpoint',
massive_process.py:106:        'R2_ACCESS_KEY_ID': 'R2 access key',
massive_process.py:107:        'R2_SECRET_ACCESS_KEY': 'R2 secret key',
massive_process.py:115:    # R2 vars only required if --r2-upload is used
massive_process.py:165:    r2_upload_key: str = ""
massive_process.py:376:                        r2_upload_key=(result.get('r2_upload', {}) or {}).get('remote_key', ''),
massive_process.py:563:                    'r2_tar_key': result.get('r2_upload_key', ''),
massive_process.py:614:                    r2_upload_key=result.get('r2_upload_key', ''),
massive_process.py:1104:    # R2 upload (processed .tar -> 1-cleaned-data)
massive_process.py:1105:    parser.add_argument('--r2-upload', action='store_true',
massive_process.py:1106:                       help='Upload per-video .tar into R2 bucket 1-cleaned-data (no deletes)')
massive_process.py:1107:    parser.add_argument('--r2-upload-prefix', type=str, default='',
massive_process.py:1109:    parser.add_argument('--r2-upload-skip-if-exists', action='store_true',
massive_process.py:1204:        # R2 source (always R2, metadata from Supabase)
massive_process.py:1205:        'input_source': 'r2',
massive_process.py:1206:        'fetch_youtube_metadata_for_r2': False,
massive_process.py:1207:        'r2_source_prefix': 'podcasts/',
massive_process.py:1208:        'r2_source_extensions': '.webm',
massive_process.py:1209:        # R2 output upload
massive_process.py:1210:        'r2_upload_enabled': bool(args.r2_upload),
massive_process.py:1211:        'r2_upload_bucket_type': 'production',
massive_process.py:1212:        'r2_upload_prefix': args.r2_upload_prefix,
massive_process.py:1213:        'r2_upload_skip_if_exists': bool(args.r2_upload_skip_if_exists),
massive_process.py:1257:        total_upload_time = sum(r.r2_upload_time_sec for r in export_results.values())
scripts/deploy/deploy_cluster.sh:88:# Test R2 connection
scripts/deploy/deploy_cluster.sh:90:from src.r2_client import R2Client
scripts/deploy/deploy_cluster.sh:91:r2 = R2Client(bucket_type='source')
scripts/deploy/deploy_cluster.sh:92:print(f'R2 OK - Bucket: {r2.bucket}')
scripts/deploy/deploy_cluster.sh:101:echo "  $SSH_CMD 'cd /workspace/maya3_data && .venv/bin/python massive_process.py --supabase-queue --r2-upload'"
scripts/deploy/deploy_all.py:82:    --r2-upload \
scripts/deploy/start_workers.sh:34:    --r2-upload \
scripts/check_videos.py:43:        print(f'  r2_tar_key: {v.get("r2_tar_key", "N/A"…13685 chars truncated…/download.py:1016:        audio_path, metadata = _download_from_r2(
src/download.py:1017:            r2_url=presigned_url,
src/download.py:1027:        metadata['r2_source_bucket'] = client.bucket
src/download.py:1028:        metadata['r2_source_key'] = remote_key
src/download.py:1029:        metadata['r2_source_size_bytes'] = head.get('ContentLength')
src/download.py:1031:        logger.info(f"✅ Download (R2): {elapsed:.1f}s | Duration: {metadata.get('processed_duration', 0):.0f}s")
src/download.py:1034:    # === v2: R2 URL HANDLING ===
src/download.py:1035:    # R2 presigned URLs use ffmpeg directly (not yt-dlp)
src/download.py:1036:    if is_r2_url(video_url):
src/download.py:1037:        logger.info(f"📦 R2 source detected, using ffmpeg direct download")
src/download.py:1038:        audio_path, metadata = _download_from_r2(
src/download.py:1039:            r2_url=video_url,
src/download.py:1048:        logger.info(f"✅ Download (R2 URL): {elapsed:.1f}s | Duration: {metadata.get('processed_duration', 0):.0f}s")
scripts/prepare_benchmark.py:4:1. Listing R2 files that exist (podcasts/*.webm)
scripts/prepare_benchmark.py:23:from src.r2_client import R2Client
scripts/prepare_benchmark.py:40:r2 = R2Client(bucket_type='source')  # 'test' bucket has the podcasts/
scripts/prepare_benchmark.py:42:print(f"\n1. Listing R2 files from {r2.bucket}/podcasts/...")
scripts/prepare_benchmark.py:45:all_r2_files = []
scripts/prepare_benchmark.py:50:        'Bucket': r2.bucket,
scripts/prepare_benchmark.py:57:    response = r2.client.list_objects_v2(**kwargs)
scripts/prepare_benchmark.py:59:    all_r2_files.extend(files)
scripts/prepare_benchmark.py:65:    if len(all_r2_files) >= 5000:  # Limit scan
scripts/prepare_benchmark.py:68:print(f"   Found {len(all_r2_files)} files in R2")
scripts/prepare_benchmark.py:71:r2_video_ids = set()
scripts/prepare_benchmark.py:72:for f in all_r2_files:
scripts/prepare_benchmark.py:76:        r2_video_ids.add(video_id)
scripts/prepare_benchmark.py:78:print(f"   Extracted {len(r2_video_ids)} unique video IDs")
scripts/prepare_benchmark.py:84:video_ids_list = list(r2_video_ids)[:500]  # Limit for query
scripts/prepare_benchmark.py:93:# Filter to videos that are in both R2 AND Supabase
scripts/prepare_benchmark.py:102:print(f"   Validated: {len(validated)} videos (in R2 + Supabase)")
scripts/reset_for_production.py:3:Reset Supabase tables and R2 folder for fresh production run.
scripts/reset_for_production.py:8:3. Deletes everything in R2 bucket 1-cleaned-data/cleaned/trail/
scripts/reset_for_production.py:11:    python scripts/reset_for_production.py [--dry-run] [--skip-r2]
scripts/reset_for_production.py:111:def delete_r2_folder(dry_run: bool = False):
scripts/reset_for_production.py:112:    """Delete everything in R2 bucket 1-cleaned-data/cleaned/trail/."""
scripts/reset_for_production.py:113:    from src.r2_client import R2Client
scripts/reset_for_production.py:115:    print("\n🪣 R2 Bucket: 1-cleaned-data")
scripts/reset_for_production.py:120:        client = R2Client(bucket_type='production')
scripts/reset_for_production.py:126:        # Use boto3 paginator for listing all objects
scripts/reset_for_production.py:146:        print(f"\n🗑  Deleting {len(objects)} objects from R2...")
scripts/reset_for_production.py:148:        # Delete in batches of 1000 (S3/R2 limit)
scripts/reset_for_production.py:171:    parser = argparse.ArgumentParser(description='Reset Supabase and R2 for fresh production run')
scripts/reset_for_production.py:173:    parser.add_argument('--skip-r2', action='store_true', help='Skip deleting R2 cleaned/trail/ prefix')
scripts/reset_for_production.py:201:    # 3. Delete R2 folder
scripts/reset_for_production.py:202:    if args.skip_r2:
scripts/reset_for_production.py:204:        print("STEP 3: Skip R2 deletion (--skip-r2)")
scripts/reset_for_production.py:208:        print("STEP 3: Delete R2 cleaned/trail/ folder")
scripts/reset_for_production.py:210:        delete_r2_folder(args.dry_run)
scripts/reset_for_production.py:219:        print("   1. Run the pipeline WITHOUT --r2-upload-prefix to upload directly to 1-cleaned-data/")
scripts/reset_for_production.py:220:        print("   2. Or use: --r2-upload-prefix '' (empty string)")
scripts/docker/smoke_test.sh:8:R2_UPLOAD="${R2_UPLOAD:-0}"
scripts/docker/smoke_test.sh:36:echo "==> R2 connectivity smoke test"
scripts/docker/smoke_test.sh:39:  "${IMAGE_TAG}" -c "from src.r2_client import R2Client; print('source_bucket', R2Client(bucket_type='source').bucket); print('production_bucket', R2Client(bucket_type='production').bucket)"
scripts/docker/smoke_test.sh:56:  if [ "${R2_UPLOAD}" = "1" ]; then
scripts/docker/smoke_test.sh:58:      --r2-upload
scripts/docker/smoke_test.sh:59:      --r2-upload-prefix "${upload_prefix}"
scripts/docker/smoke_test.sh:60:      --r2-upload-skip-if-exists
scripts/docker/smoke_test.sh:74:    echo "Warning: no .tar found in ${OUTPUT_DIR} (maybe R2 upload deleted it, or export disabled)."


Output:
Exit code: 0
Wall time: 0 seconds
Total output lines: 244
Output:
pipeline.py:125:    supabase_metadata: Optional[Dict] = None,
pipeline.py:134:    === v3: Supabase Metadata Support ===
pipeline.py:135:    If supabase_metadata is provided, it's passed to download_audio for:
pipeline.py:217:        # Standard download (with optional Supabase metadata for chapters)
pipeline.py:219:            audio_path, metadata = download_audio(video_url, config, supabase_metadata=supabase_metadata)
massive_process.py:61:# Supabase client for distributed queue
massive_process.py:63:    from src.supabase_client import SupabaseClient, STATUS_COMPLETED, STATUS_FAILED
massive_process.py:64:    SUPABASE_AVAILABLE = True
massive_process.py:66:    SUPABASE_AVAILABLE = False
massive_process.py:68:# Optional: load .env for HF/R2/Supabase credentials
massive_process.py:100:        'URL': 'Supabase project URL',
massive_process.py:101:        'SUPABASE_ADMIN': 'Supabase service role key',
massive_process.py:423:def gpu_worker_supabase(
massive_process.py:435:    GPU worker that claims videos from Supabase queue.
massive_process.py:438:    1. Claims the next PENDING video from Supabase
massive_process.py:440:    3. Updates Supabase with results (COMPLETED/FAILED)
massive_process.py:452:    supabase_worker_id = f"gpu-{gpu_id}-{pod_id}"
massive_process.py:454:    worker_logger.info(f"Starting worker {supabase_worker_id} on GPU {gpu_id}")
massive_process.py:477:        # Initialize Supabase client
massive_process.py:478:        supabase = SupabaseClient()
massive_process.py:480:        # Initialize analytics reporter (uses same Supabase credentials)
massive_process.py:484:            supabase_url=os.environ.get('URL') or os.environ.get('SUPABASE_URL'),
massive_process.py:485:            supabase_key=os.environ.get('SUPABASE_ADMIN'),
massive_process.py:513:            # Claim next video from Supabase
massive_process.py:514:            video = supabase.claim_next_video(
massive_process.py:515:                worker_id=supabase_worker_id,
massive_process.py:536:            full_video = supabase.get_video(video_id)
massive_process.py:543:                worker_logger.info(f"   Chapters: {full_video.get('chapter_count', 0)} from Supabase")
massive_process.py:547:                # Process video with Supabase metadata (chapters, title, etc.)
massive_process.py:548:                result = process_single_video(video_url, config, supabase_metadata=full_video)
massive_process.py:556:                # Update Supabase with success
massive_process.py:571:                        'worker_id': supabase_worker_id,
massive_process.py:582:                supabase.update_status(video_id, STATUS_COMPLETED, update_data)
massive_process.py:635:                # Update Supabase with failure
massive_process.py:636:                supabase.update_status(video_id, STATUS_FAILED, {
massive_process.py:640:                        'worker_id': supabase_worker_id,
massive_process.py:677:def run_supabase_queue(
massive_process.py:686:    Run distributed processing with Supabase queue.
massive_process.py:688:    Workers claim videos from Supabase, process them, and update status.
massive_process.py:697:    supabase = SupabaseClient()
massive_process.py:698:    initial_stats = supabase.get_stats()
massive_process.py:702:    logger.info("SUPABASE QUEUE - Distributed Processing")
massive_process.py:732:                target=gpu_worker_supabase,
massive_process.py:738:                name=f"GPU{gpu_id}-Supabase-{w}"
massive_process.py:742:            logger.info(f"Started Supabase worker {worker_id} on GPU {gpu_id}")
massive_process.py:807:    final_stats = supabase.get_stats()
massive_process.py:810:    logger.info("SUPABASE QUEUE COMPLETE")
massive_process.py:1083:    input_group.add_argument('--supabase-queue', action='store_true',
massive_process.py:1084:                            help='Use Supabase as distributed video queue (claims PENDING videos from DB)')
massive_process.py:1122:    # Supabase queue options
massive_process.py:1124:                       help='Max videos per worker in supabase-queue mode (0=unlimited)')
massive_process.py:1156:    # Get video URLs (not needed for supabase-queue mode)
massive_process.py:1158:    supabase_mode = getattr(args, 'supabase_queue', False)
massive_process.py:1160:    if supabase_mode:
massive_process.py:1161:        if not SUPABASE_AVAILABLE:
massive_process.py:1162:            logger.error("Supabase client not available! Install dependencies or check src/supabase_client.py")
massive_process.py:1164:        logger.info("Supabase queue mode: videos will be claimed from database")
massive_process.py:1174:    if not supabase_mode and not video_urls:
massive_process.py:1204:        # R2 source (always R2, metadata from Supabase)
massive_process.py:1228:    if supabase_mode:
massive_process.py:1229:        # Supabase distributed queue mode
massive_process.py:1230:        stats = run_supabase_queue(
scripts/sync_csv_to_supabase.py:3:Sync CSVs to Supabase
scripts/sync_csv_to_supabase.py:6:then up…10321 chars truncated…with Supabase...")
scripts/prepare_benchmark.py:82:# Query Supabase for these videos
scripts/prepare_benchmark.py:91:print(f"   Found {len(videos_in_db)} videos in Supabase DB")
scripts/prepare_benchmark.py:93:# Filter to videos that are in both R2 AND Supabase
scripts/prepare_benchmark.py:102:print(f"   Validated: {len(validated)} videos (in R2 + Supabase)")
scripts/prepare_benchmark.py:162:print(f"  .venv/bin/python massive_process.py --supabase-queue --max-videos {len(validated)} --output ./output")
scripts/docker/smoke_test.sh:31:echo "==> Supabase connectivity smoke test"
scripts/docker/smoke_test.sh:34:  "${IMAGE_TAG}" -c "from src.supabase_client import SupabaseClient; print(SupabaseClient().get_stats())"
scripts/docker/smoke_test.sh:47:    --supabase-queue
scripts/docker/smoke_test.sh:64:  echo "==> E2E test: process 1 Supabase-queued video on GPU ${GPU_DEVICE}"
src/analytics.py:4:Lightweight, non-blocking analytics that reports worker status to Supabase.
src/analytics.py:8:- Batches writes to reduce Supabase calls (~2-3 per video, not 10+)
src/analytics.py:11:- Local counters remain accurate even if Supabase is down
src/analytics.py:27:    """Event to be sent to Supabase."""
src/analytics.py:50:        reporter = AnalyticsReporter(gpu_id=0, supabase_url=..., supabase_key=...)
src/analytics.py:67:        supabase_url: str,
src/analytics.py:68:        supabase_key: str,
src/analytics.py:85:        # Local counters (always accurate, even if Supabase fails)
src/analytics.py:92:        # Supabase client (lazy init in background thread)
src/analytics.py:93:        self._supabase_url = supabase_url
src/analytics.py:94:        self._supabase_key = supabase_key
src/analytics.py:103:        """Lazy init Supabase client in background thread (thread-safe)."""
src/analytics.py:108:                        from supabase import create_client
src/analytics.py:109:                        self._client = create_client(self._supabase_url, self._supabase_key)
src/analytics.py:115:        """Background thread that flushes events to Supabase."""
src/analytics.py:148:        """Batch write events to Supabase."""
src/analytics.py:417:    supabase_url: Optional[str] = None,
src/analytics.py:418:    supabase_key: Optional[str] = None,
src/analytics.py:426:    if not enabled or not supabase_url or not supabase_key:
src/analytics.py:431:        supabase_url=supabase_url,
src/analytics.py:432:        supabase_key=supabase_key,
src/logger.py:115:        'SupabaseClient',
src/supabase_client.py:3:Supabase Client for Distributed Video Processing Queue
src/supabase_client.py:12:    from src.supabase_client import SupabaseClient
src/supabase_client.py:14:    client = SupabaseClient()
src/supabase_client.py:31:logger = logging.getLogger("SupabaseClient")
src/supabase_client.py:41:class SupabaseClient:
src/supabase_client.py:43:    Supabase REST API client for video processing queue.
src/supabase_client.py:59:        Initialize Supabase client.
src/supabase_client.py:62:            url: Supabase project URL (loads from URL env var if None)
src/supabase_client.py:63:            key: Service role key (loads from SUPABASE_ADMIN env var if None)
src/supabase_client.py:69:        self.key = key or os.environ.get('SUPABASE_ADMIN', '')
src/supabase_client.py:72:            logger.warning("Supabase credentials not configured. Set URL and SUPABASE_ADMIN env vars.")
src/supabase_client.py:83:        logger.info(f"SupabaseClient initialized: {self.url}")
src/supabase_client.py:93:        """Make HTTP request to Supabase REST API."""
src/supabase_client.py:110:            logger.error(f"Supabase error {e.code}: {error_body}")
src/supabase_client.py:117:        """Call Supabase RPC function."""
src/supabase_client.py:390:        # Use Supabase's count feature
src/supabase_client.py:457:def get_supabase_client() -> SupabaseClient:
src/supabase_client.py:458:    """Get singleton SupabaseClient instance."""
src/supabase_client.py:459:    return SupabaseClient()
scripts/monitor/lease_recovery.py:26:from src.supabase_client import SupabaseClient, STATUS_PENDING, STATUS_CLAIMED
scripts/monitor/lease_recovery.py:36:def reclaim_expired_leases(client: SupabaseClient) -> int:
scripts/monitor/lease_recovery.py:54:def check_stale_workers(client: SupabaseClient, threshold_sec: int = 600) -> list:
scripts/monitor/lease_recovery.py:61:    from supabase import create_client
scripts/monitor/lease_recovery.py:63:    sb = create_client(os.environ['URL'], os.environ['SUPABASE_ADMIN'])
scripts/monitor/lease_recovery.py:84:    client = SupabaseClient()
scripts/monitor/lease_recovery.py:109:    client = SupabaseClient()
scripts/monitor/dashboard.py:29:from supabase import create_client
scripts/monitor/dashboard.py:65:        self.sb = create_client(os.environ['URL'], os.environ['SUPABASE_ADMIN'])
scripts/monitor/retry_failed.py:26:from supabase import create_client
scripts/monitor/retry_failed.py:78:    sb = create_client(os.environ['URL'], os.environ['SUPABASE_ADMIN'])


Output:
Exit code: 0
Wall time: 0 seconds
Output:
pipeline.py:6:- PANNs CNN14 for music/instrument detection in 1.5s chunks
pipeline.py:109:# === v7.4 FEATURE: Hybrid Music Detection (INA + PANNs) ===
pipeline.py:110:# Fast two-stage detection: INA screens everything, PANNs verifies suspicious only
pipeline.py:395:    # On 16GB GPUs (T4), overlapping PANNs model load with embedding inference can spike VRAM and OOM.
pipeline.py:537:    # STAGE 7: MUSIC DETECTION (v7.4 - Hybrid INA+PANNs)
pipeline.py:538:    # Two-stage: Fast INA screening + PANNs verification (suspicious only)
pipeline.py:544:                # === v7.4: HYBRID DETECTION (INA + PANNs) ===
pipeline.py:546:                    logger.info("⚡ Using HYBRID music detection (INA + PANNs)")
pipeline.py:589:                               f"PANNs skipped {hybrid_stats['panns_skip_ratio']*100:.0f}%")
pipeline.py:592:                    # === Original PANNs-only detection ===
pipeline.py:600:                    logger.info(f"   PANNs-only: {num_music_chunks} chunks analyzed")
pipeline.py:624:                    logger.info(f"   PANNs fallback: {num_music_chunks} chunks analyzed")
src/compute.py:236:        # PANNs CNN14 is heavier: ~1GB for batch=64
src/config.py:198:    # === MUSIC DETECTION (v7.4 - Hybrid INA+PANNs) ===
src/config.py:199:    # Two-stage detection: Fast INA screening + PANNs verification
src/config.py:201:    # Stage 2: PANNs CNN14 (only on suspicious segments)
src/config.py:203:    use_hybrid_detection: bool = False  # DISABLED: Run PANNs on EVERYTHING for strict music detection
src/config.py:205:    music_batch_size: int = 128  # Batch size for PANNs inference (v7.5: optimized for A100)
src/config.py:233:    # or enable use_hybrid_detection (INA screening + PANNs on suspicious chunks).
src/music_detection.py:2:"""Music Detection Module - PANNs CNN14 for TTS quality.
src/music_detection.py:23:# Reference: https://github.com/audioset/ontology
src/music_detection.py:153:        logger.info("⚡ MUSIC DETECTION CACHE (PANNs CNN14) - v7.0")
src/music_detection.py:212:        logger.info(f"   [TIMING] PANNs inference: {(time.time()-t0)*1000:.0f}ms")
src/music_detection.py:384:        Run PANNs inference and extract music, noise, and speech probabilities.
src/music_detection.py:412:                logger.error(f"PANNs batch failed: {e}")
src/models.py:118:        self.panns_model = None  # v6.7: PANNs CNN14 for music detection
src/models.py:120:        self._panns_loaded = False  # Separate flag for optional PANNs
src/models.py:287:        Load PANNs CNN14 model for music detection (lazy loading).
src/models.py:290:        If PANNS_MODEL_PATH env var is set, loads from that path (Docker pre-baked).
src/models.py:300:            logger.info("✅ PANNs CNN14 already loaded (hot inference)")
src/models.py:303:        logger.info("Loading PANNs CNN14 for music detection...")
src/models.py:312:                logger.warning("CUDA not available, using CPU for PANNs")
src/models.py:315:            checkpoint_path = os.environ.get('PANNS_MODEL_PATH')
src/models.py:325:            logger.info(f"✅ PANNs CNN14 loaded on {device} in {elapsed:.1f}s")
src/models.py:330:            logger.error(f"❌ PANNs not installed. Run: pip install panns-inference")
src/models.py:334:            logger.error(f"❌ Failed to load PANNs: {e}")
src/hybrid_music_detection.py:13:Stage 2: PANNs CNN14 (DETAILED, only when needed)
src/hybrid_music_detection.py:21:- Before: PANNs on 100% of chunks → 16s
src/hybrid_music_detection.py:22:- After: INA on 100% (fast) + PANNs on 10% (suspicious) → ~4s
src/hybrid_music_detection.py:25:| INA Result | PANNs Run? | Final Decision |
src/hybrid_music_detection.py:29:| music      | YES        | PANNs decides  |
src/hybrid_music_detection.py:30:| noise      | YES        | PANNs decides  |
src/hybrid_music_detection.py:51:    # Stage 2: PANNs settings (only for suspicious segments)
src/hybrid_music_detection.py:57:    panns_music_threshold: float = 0.20  # PANNs music prob threshold
src/hybrid_music_detection.py:58:    panns_noise_threshold: float = 0.25  # PANNs noise prob threshold
src/hybrid_music_detection.py:85:    # Stage 2 (PANNs) results - only populated if INA flagged suspicious
src/hybrid_music_detection.py:99:    Two-stage music detection: Fast INA screening + PANNs verification.
src/hybrid_music_detection.py:159:            Number of suspicious chunks that required PANNs verification
src/hybrid_music_detection.py:168:        logger.info("   Stage 2: PANNs CNN14 (verification, suspicious only)")
src/hybrid_music_detection.py:203:            # Decide if PANNs verification needed
src/hybrid_music_detection.py:207:                # Clean speech or silence - no PANNs needed
src/hybrid_music_detection.py:216:        # === STAGE 2: PANNs verification (only suspicious chunks) ===
src/hybrid_music_detection.py:222:            logger.info(f"   [Stage 2] PANNs verification: {panns_time:.1f}s ({len(suspicious_chunks)} chunks)")
src/hybrid_music_detection.py:224:            logger.warning("   ⚠ PANNs model not provided, suspicious chunks unverified")
src/hybrid_music_detection.py:250:        logger.info(f"   PANNs skipped: {self._stats['panns_skip_ratio']*100:.1f}% of chunks")
src/hybrid_music_detection.py:327:        Run PANNs on suspicious chunks only.
src/hybrid_music_detection.py:337:        # Prepare audio chunks for PANNs (requires 32kHz)
src/hybrid_music_detection.py:364:        # Run PANNs inference in batches
src/hybrid_music_detection.py:399:                        # PANNs cleared it - INA was false positive
src/hybrid_music_detection.py:403:                logger.error(f"PANNs batch failed: {e}")
src/hybrid_music_detection.py:442:        # PANNs results (only from chunks that ran PANNs)
src/hybrid_music_detection.py:565:    logger.info(f"   ⚡ PANNs skipped: {stats['panns_skip_ratio']*100:.1f}% of chunks")