Search thoroughly for a large processed audio dataset (~20 hours) related to "modi" anywhere on this machine. Look for:
1. Large directories with many .wav/.flac/.mp3 files
2. Large .arrow/.parquet/.tar files that could contain hours of audio
3. Any references to "modi" in config files, scripts, or manifests
4. Check /home/ubuntu and subdirectories, also /data, /mnt, /tmp, /opt if they exist
5. Look for manifest files (.json, .jsonl, .csv, .tsv) that reference modi audio paths
6. Check training configs that reference dataset paths with "modi"

Focus on finding where the bulk 20-hour processed dataset lives, not just the small 44MB one we already found.


I'll conduct a thorough search for large processed audio datasets related to "modi" across the machine. Let me start with a systematic search.


Bash

find /home/ubuntu -type d -name "*modi*" 2>/dev/null | head -20


/home/ubuntu/modi
/home/ubuntu/modi_vibevoice_dataset
/home/ubuntu/.npm/_npx/d66c5096c7023bfb/node_modules/@babel/types/lib/modifications
/home/ubuntu/modi_processed
/home/ubuntu/hf_staging/modi_processed
/home/ubuntu/training/asr-benchmark-dashboard/node_modules/@babel/types/lib/modifications
/home/ubuntu/modi_dataset
/home/ubuntu/.local/lib/python3.10/data/modi_hindi_custom


Good! I found several modi-related directories. Let me now check their sizes and contents systematically.


du -sh /home/ubuntu/modi /home/ubuntu/modi_processed /home/ubuntu/modi_vibevoice_dataset /home/ubuntu/modi_dataset /home/ubuntu/hf_staging/modi_processed 2>/dev/null | sort -h


4.0K	/home/ubuntu/hf_staging/modi_processed
12K	/home/ubuntu/modi_dataset
32K	/home/ubuntu/modi
284K	/home/ubuntu/modi_processed
44M	/home/ubuntu/modi_vibevoice_dataset


Let me check /data, /mnt, /opt, and /tmp directories:


find /data -type d -name "*modi*" 2>/dev/null; find /mnt -type d -name "*modi*" 2>/dev/null; find /opt -type d -name "*modi*" 2>/dev/null; find /tmp -type d -name "*modi*" 2>/dev/null


Now let me search for large audio files and archive files with "modi" in their names or paths:


find /home/ubuntu -type f \( -name "*modi*" -o -name "*audio*" \) \( -name "*.tar*" -o -name "*.zip" -o -name "*.arrow" -o -name "*.parquet" \) 2>/dev/null | head -30


Interesting! Let me check for all .wav/.flac/.mp3 files in the modi directories and search for configs that reference "modi":


find /home/ubuntu -type f \( -name "*.wav" -o -name "*.flac" -o -name "*.mp3" \) -path "*modi*" 2>/dev/null | wc -l


Now let me check the contents and sizes of the modi directories more closely:


total 44M
drwxrwxr-x  2 ubuntu ubuntu 4.0K Mar 25 02:05 .
drwxr-x--x 80 ubuntu ubuntu  12K Mar 27 21:40 ..
-rw-rw-r--  1 ubuntu ubuntu  44M Mar 16 06:10 data-00000-of-00001.arrow
-rw-rw-r--  1 ubuntu ubuntu  240 Mar 16 06:14 dataset_info.json
-rw-rw-r--  1 ubuntu ubuntu  601 Mar 16 06:14 state.json


Let me search for manifest and config files that reference "modi" dataset paths:


find /home/ubuntu -type f \( -name "*.json" -o -name "*.jsonl" -o -name "*.yaml" -o -name "*.yml" -o -name "*.csv" -o -name "*.tsv" \) 2>/dev/null | xargs grep -l "modi" 2>/dev/null | head -30


Let me look for actual dataset config and training scripts:


find /home/ubuntu -type f \( -name "*.py" -o -name "*.sh" \) 2>/dev/null | xargs grep -l "modi.*dataset\|modi_processed\|20.*hour" 2>/dev/null | head -20


Let me check the prepare_dataset.py file and look more carefully at what's in each modi directory:


     1→"""
     2→Convert modi_processed/ to HuggingFace Dataset for F5-TTS finetuning.
     3→Format: {audio: {array, sampling_rate}, text: str}
     4→F5-TTS requires audio between 0.3s - 30s.
     5→"""
     6→
     7→import os
     8→import json
     9→import wave
    10→import numpy as np
    11→from datasets import Dataset, Audio
    12→
    13→SEG_DIR = "/home/ubuntu/modi_processed/segments"
    14→TRANS_DIR = "/home/ubuntu/modi_processed/transcripts"
    15→OUTPUT_DIR = "/home/ubuntu/modi_dataset"
    16→
    17→samples = []
    18→skipped = 0
    19→
    20→for vid in sorted(os.listdir(SEG_DIR)):
    21→    vid_seg = os.path.join(SEG_DIR, vid)
    22→    vid_trans = os.path.join(TRANS_DIR, vid)
    23→
    24→    if not os.path.isdir(vid_seg):
    25→        continue
    26→
    27→    for f in sorted(os.listdir(vid_seg)):
    28→        if not f.endswith(".wav"):
    29→            continue
    30→
    31→        wav_path = os.path.join(vid_seg, f)
    32→        json_path = os.path.join(vid_trans, f.replace(".wav", ".json"))
    33→
    34→        if not os.path.exists(json_path):
    35→            skipped += 1
    36→            continue
    37→
    38→        try:
    39→            with wave.open(wav_path) as w:
    40→                dur = w.getnframes() / w.getframerate()
    41→
    42→            if dur < 0.3 or dur > 30:
    43→                skipped += 1
    44→                continue
    45→
    46→            with open(json_path) as fp:
    47→                trans = json.load(fp)
    48→
    49→            text = trans.get("transcription", "").strip()
    50→            if not text or len(text) < 5:
    51→                skipped += 1
    52→                continue
    53→
    54→            samples.append({
    55→                "audio": wav_path,
    56→                "text": text,
    57→            })
    58→        except Exception as e:
    59→            skipped += 1
    60→            continue
    61→
    62→print(f"Valid samples: {len(samples)}")
    63→print(f"Skipped: {skipped}")
    64→
    65→ds = Dataset.from_dict({
    66→    "audio": [s["audio"] for s in samples],
    67→    "text": [s["text"] for s in samples],
    68→})
    69→ds = ds.cast_column("audio", Audio(sampling_rate=24000))
    70→
    71→ds.save_to_disk(OUTPUT_DIR)
    72→print(f"Dataset saved to {OUTPUT_DIR}")
    73→print(f"Columns: {ds.column_names}")
    74→print(f"Example: {ds[0]['text'][:100]}")
    75→print(f"Audio SR: {ds[0]['audio']['sampling_rate']}")
    76→print(f"Audio len: {len(ds[0]['audio']['array'])}")
    77→


Now let me examine the contents of /home/ubuntu/modi_processed:


ls -la /home/ubuntu/modi_processed/ && find /home/ubuntu/modi_processed -type f | wc -l


total 148
drwxrwxr-x  6 ubuntu ubuntu  4096 Mar 25 02:08 .
drwxr-x--x 80 ubuntu ubuntu 12288 Mar 27 21:40 ..
-rw-rw-r--  1 ubuntu ubuntu  6479 Mar 11 20:26 build_combined_dataset.py
-rw-rw-r--  1 ubuntu ubuntu  3147 Mar 12 07:25 build_hf_dataset.py
-rw-rw-r--  1 ubuntu ubuntu    11 Mar 12 08:34 build_pid.txt
-rw-rw-r--  1 ubuntu ubuntu  4816 Mar 11 15:40 debug_inference.py
-rw-rw-r--  1 ubuntu ubuntu  6131 Mar 11 15:45 generate_samples.py
-rw-rw-r--  1 ubuntu ubuntu  6137 Mar 11 17:52 generate_samples_v2.py
-rw-rw-r--  1 ubuntu ubuntu   198 Mar 12 08:31 launch_build.sh
drwxrwxr-x  5 ubuntu ubuntu  4096 Mar 11 15:37 lora_output
drwxrwxr-x  6 ubuntu ubuntu  4096 Mar 25 02:08 lora_output_v2
-rw-rw-r--  1 ubuntu ubuntu 14875 Mar 11 13:36 pipeline.py
drwxrwxr-x  2 ubuntu ubuntu  4096 Mar 11 14:18 __pycache__
-rw-rw-r--  1 ubuntu ubuntu  1355 Mar 11 15:39 run_tests.sh
drwxrwxr-x  2 ubuntu ubuntu  4096 Mar 11 15:38 test_inputs
-rw-rw-r--  1 ubuntu ubuntu  4285 Mar 11 17:57 test_noprefill.py
-rw-rw-r--  1 ubuntu ubuntu   337 Mar 11 15:24 test_script.txt
-rw-rw-r--  1 ubuntu ubuntu  3806 Mar 11 17:56 test_v2_quick.py
-rw-rw-r--  1 ubuntu ubuntu 10053 Mar 11 14:24 transcribe.py
-rw-rw-r--  1 ubuntu ubuntu  6516 Mar 12 06:58 upload_r2_and_build_dataset.py
-rw-rw-r--  1 ubuntu ubuntu  6395 Mar 12 07:02 upload_r2_fast.py
-rw-rw-r--  1 ubuntu ubuntu  7228 Mar 12 07:09 upload_r2_v3.py
30


Let me look for the segments and transcripts directories which are referenced:


find /home/ubuntu/modi_processed -type d -name "segments" -o -type d -name "transcripts" 2>/dev/null


(Bash completed with no output)


Let me search more broadly for any large directories containing audio files or large archives:


find /home/ubuntu -type f -name "*.tar*" -o -name "*.zip" | xargs ls -lh 2>/dev/null | awk '{print $5, $9}' | sort -h | tail -30


3.7M /home/ubuntu/training/api_results.zip
3.7M /home/ubuntu/transcripts/tmp/final_export/validator_actual/pilot-final-export-r2scoped-20260312/shards/pack_000009/audio.tar
4.2M /home/ubuntu/nltk_data/tokenizers/punkt_tab.zip
4.3M /home/ubuntu/.cache/uv/archive-v0/BYRvHqUAAg-kRrj8183_i/tensorboard/webfiles.zip
4.3M /home/ubuntu/Irodori-TTS/.venv/lib/python3.10/site-packages/tensorboard/webfiles.zip
4.3M /home/ubuntu/.local/lib/python3.10/site-packages/tensorboard/webfiles.zip
4.3M /home/ubuntu/maya3_transcribe/venv/lib/python3.10/site-packages/tensorboard/webfiles.zip
4.4M /home/ubuntu/transcripts/preflight/canary_data/kn/o-zn8bAjkBM.tar
4.5M /home/ubuntu/transcripts/tmp/final_export/validator_actual/pilot-final-export-inline-20260312/shards/pack_000010/audio.tar
4.8M /home/ubuntu/transcripts/preflight/canary_data/hi/BFU9eSKO_t4.tar
5.4M /home/ubuntu/transcripts/preflight/canary_data/ml/dlo3pvcyqFY.tar
6.1M /home/ubuntu/transcripts/tmp/final_export/validator_expected/pilot-final-export-inline-20260312/DCxlDOiSrQs/DCxlDOiSrQs.tar
6.1M /home/ubuntu/transcripts/tmp/final_export/validator_expected/pilot-final-export-r2scoped-20260312/DCxlDOiSrQs/DCxlDOiSrQs.tar
6.5M /home/ubuntu/transcripts/preflight/canary_data/te/5MDwLvubhR8.tar
6.7M /home/ubuntu/transcripts/preflight/canary_data/ta/8ozSnoys7JE.tar
7.0M /home/ubuntu/transcripts/preflight/canary_data/gu/X7tqVj0IuOI.tar
7.5M /home/ubuntu/transcripts/preflight/canary_data/en/nM2KMwb86IU.tar
11M /home/ubuntu/transcripts/tmp/final_export/validator_actual/pilot-final-export-docker-20260312/microshards/pack_000005/audio.tar
11M /home/ubuntu/transcripts/tmp/final_export/validator_actual/pilot-final-export-docker-20260312/shards/pack_000005/audio.tar
11M /home/ubuntu/transcripts/tmp/final_export/validator_actual/pilot-final-export-inline-20260312/microshards/pack_000014/audio.tar
15M /home/ubuntu/transcripts/tmp/final_export/validator_actual/pilot-final-export-inline-20260312/shards/pack_000007/audio.tar
16M /home/ubuntu/transcripts/tmp/final_export/validator_expected/pilot-final-export-docker-20260312/CHUbGdCpT70/CHUbGdCpT70.tar
16M /home/ubuntu/transcripts/tmp/final_export/validator_expected/pilot-final-export-inline-20260312/CHUbGdCpT70/CHUbGdCpT70.tar
17M /home/ubuntu/transcripts/tmp/final_export/validator_expected/pilot-final-export-inline-20260312/eyYmTDz01nE/eyYmTDz01nE.tar
17M /home/ubuntu/transcripts/tmp/final_export/validator_expected/pilot-final-export-r2scoped-20260312/eyYmTDz01nE/eyYmTDz01nE.tar
23M /home/ubuntu/transcripts/preflight/canary_data/or/IYKHdR3-w_U.tar
24M /home/ubuntu/transcripts/preflight/canary_data/mr/mqyv2attoI0.tar
30M /home/ubuntu/transcripts/preflight/canary_data/pa/l5eQd1E_Huo.tar
36M /home/ubuntu/.local/lib/python3.10/site-packages/perth/perth_net/pretrained/implicit/perth_net_250000.pth.tar
84M /home/ubuntu/clip00_lalal_pipeline.zip


Let me search for larger directories with audio files and check if there are symlinks or mounted directories:


find /home/ubuntu -type d -size +500M 2>/dev/null | head -20


Let me check what's in each of the main modi directories in detail:


total 44K
drwxrwxr-x  2 ubuntu ubuntu  16K Mar 25 02:08 .
drwxr-x--x 80 ubuntu ubuntu  12K Mar 27 21:40 ..
-rw-rw-r--  1 ubuntu ubuntu  11K Mar 14 10:27 download_all.py
-rw-rw-r--  1 ubuntu ubuntu 3.9K Mar 11 12:15 run_demucs_all.py


Let me check /home/ubuntu/modi_dataset more carefully:


ls -lah /home/ubuntu/modi_dataset/ && find /home/ubuntu/modi_dataset -type f | head -20


total 24K
drwxrwxr-x  2 ubuntu ubuntu 4.0K Mar 25 02:06 .
drwxr-x--x 80 ubuntu ubuntu  12K Mar 27 21:40 ..
-rw-rw-r--  1 ubuntu ubuntu  240 Mar 14 18:47 dataset_info.json
-rw-rw-r--  1 ubuntu ubuntu  601 Mar 14 18:47 state.json
/home/ubuntu/modi_dataset/dataset_info.json
/home/ubuntu/modi_dataset/state.json


Let me look in /home/ubuntu/.local and check if there are any symlinks that might be pointing to large datasets:


find /home/ubuntu -type l -name "*modi*" 2>/dev/null


Let me check what this symlink points to:


ls -lah /home/ubuntu/.local/lib/python3.10/data/modi_hindi_cls_custom && readlink -f /home/ubuntu/.local/lib/python3.10/data/modi_hindi_cls_custom


Error: Exit code 1
lrwxrwxrwx 1 ubuntu ubuntu 48 Mar 14 20:30 /home/ubuntu/.local/lib/python3.10/data/modi_hindi_cls_custom -> /home/ubuntu/sooktam2/data/modi_hindi_cls_custom


Let me check if that directory exists:


du -sh /home/ubuntu/sooktam2/data/modi_hindi_cls_custom 2>/dev/null


Exit code 1


Let me search for any training configs or scripts that might reference a 20-hour dataset:


find /home/ubuntu -type f \( -name "*.py" -o -name "*.sh" -o -name "*.yaml" -o -name "*.yml" \) 2>/dev/null | xargs grep -l "20.*hour\|hours.*audio\|20h\|20H" 2>/dev/null | grep -v venv | head -20


Let me search for any references to "processing" or "dataset" scripts specific to modi:


grep -r "modi" /home/ubuntu/Modi --include="*.py" 2>/dev/null | head -20


Let me check for any audio processing scripts and look at the download script more carefully:


     1→import subprocess
     2→import json
     3→import os
     4→from datetime import datetime, timezone
     5→from pathlib import Path
     6→
     7→OUT = Path("/home/ubuntu/modi")
     8→OUT.mkdir(exist_ok=True)
     9→
    10→URLS = [
    11→    "https://www.youtube.com/watch?v=ERW9i1lwnBw",
    12→    "https://www.youtube.com/watch?v=rPf56BcBTdo",
    13→    "https://www.youtube.com/watch?v=xD64_6cgK64",
    14→    "https://www.youtube.com/watch?v=nnTOz3EHR7w",
    15→    "https://www.youtube.com/watch?v=UE2tlpaxbIA",
    16→    "https://www.youtube.com/watch?v=X5DLhrF6YPA",
    17→    "https://www.youtube.com/watch?v=Snlq1jwyOxg",
    18→    "https://www.youtube.com/watch?v=bcpxVRrI40Y",
    19→    "https://www.youtube.com/watch?v=9p75x0UZUCg",
    20→    "https://www.youtube.com/watch?v=hS7gTBmxMS8",
    21→    "https://www.youtube.com/watch?v=Lk17v9ljYh4",
    22→    "https://www.youtube.com/watch?v=QooiAFBuc9c",
    23→    "https://www.youtube.com/watch?v=ID0AbWp8Oks",
    24→    "https://www.youtube.com/watch?v=HOZ6-QlRak8",
    25→    "https://www.youtube.com/watch?v=DbKL7YAV0Vw",
    26→    "https://www.youtube.com/watch?v=UUgR3U0qhY0",
    27→    "https://www.youtube.com/watch?v=oYPtkzjF9fM",
    28→    "https://www.youtube.com/watch?v=VH2ZIZx29j8",
    29→    "https://www.youtube.com/watch?v=hEZ5uTcH5CI",
    30→    "https://www.youtube.com/watch?v=f3JvoR_-H6M",
    31→    "https://www.youtube.com/watch?v=q4-jfcIox8o",
    32→    "https://www.youtube.com/watch?v=MYms8GfcKsI",
    33→    "https://www.youtube.com/watch?v=Ytzlp3Umct8",
    34→    "https://www.youtube.com/watch?v=u2ZZ4c5rHD8",
    35→    "https://www.youtube.com/watch?v=7mjvqLfckWM",
    36→    "https://www.youtube.com/watch?v=0kMELqcLv4A",
    37→    "https://www.youtube.com/watch?v=GxfX6kZCSRU",
    38→    "https://www.youtube.com/watch?v=JdF1AOEYr-8",
    39→    "https://www.youtube.com/watch?v=JmLhbrr8w7c",
    40→    "https://www.youtube.com/watch?v=YzV_JANRm2M",
    41→    "https://www.youtube.com/watch?v=Fb2v5rlwtNg",
    42→    "https://www.youtube.com/watch?v=ux9hRMHvN-k",
    43→    "https://www.youtube.com/watch?v=hPyE079CAKY",
    44→    "https://www.youtube.com/watch?v=l3348bK9HnE",
    45→    "https://www.youtube.com/watch?v=0AUmYfkJhvo",
    46→    "https://www.youtube.com/watch?v=8PMOaqlfVc8",
    47→    "https://www.youtube.com/watch?v=uju2cVQ5jco",
    48→    "https://www.youtube.com/watch?v=OKDjyJUW_L0",
    49→    "https://www.youtube.com/watch?v=HN5HlLdMy0A",
    50→    "https://www.youtube.com/watch?v=FDUVR-QqYvQ",
    51→    "https://www.youtube.com/watch?v=_B340mkjiiI",
    52→    "https://www.youtube.com/watch?v=kxkLGWBlIIw",
    53→    "https://www.youtube.com/watch?v=Y5_wacGRA2o",
    54→    "https://www.youtube.com/watch?v=e1NmbRqoCnc",
    55→    "https://www.youtube.com/watch?v=O1OmvDy8FKk",
    56→    "https://www.youtube.com/watch?v=qaKVFwk6sU4",
    57→    "https://www.youtube.com/watch?v=WdprihSHmmQ",
    58→    "https://www.youtube.com/watch?v=tRdyASp4zQw",
    59→    "https://www.youtube.com/watch?v=x8aRnmiGMLE",
    60→    "https://www.youtube.com/watch?v=Yf7Krbu6Z8g",
    61→    "https://www.youtube.com/watch?v=reFo3Ex7XA0",
    62→    "https://www.youtube.com/watch?v=WC5TmwjMVXI",
    63→    "https://www.youtube.com/watch?v=YdI8JeVOcYE",
    64→    "https://www.youtube.com/watch?v=BtcqEVW2UNA",
    65→    "https://www.youtube.com/watch?v=44XU-2FIq9s",
    66→    "https://www.youtube.com/watch?v=K8zeo3ID6OY",
    67→    "https://www.youtube.com/watch?v=Nv1CQ2LhlyY",
    68→    "https://www.youtube.com/watch?v=H1ogjsNT23I",
    69→    "https://www.youtube.com/watch?v=UXJRtq2ZYrs",
    70→    "https://www.youtube.com/watch?v=p5ziBc4kT_0",
    71→    "https://www.youtube.com/watch?v=tc0JIvSpVGY",
    72→    "https://www.youtube.com/watch?v=pqozg9Tchc4",
    73→    "https://www.youtube.com/watch?v=qSCxH_pjQos",
    74→    "https://www.youtube.com/watch?v=OgafNvdpGHI",
    75→    "https://www.youtube.com/watch?v=EU6CfwqqEqk",
    76→    "https://www.youtube.com/watch?v=jhJGNRd4ZM4",
    77→    "https://www.youtube.com/watch?v=oEB5RnaDZmE",
    78→    "https://www.youtube.com/watch?v=KfSawaMJLWE",
    79→    "https://www.youtube.com/watch?v=nedl4B4foZ4",
    80→    "https://www.youtube.com/watch?v=lQzo-QhR6VY",
    81→    "https://www.youtube.com/watch?v=FjXoZTkkHNA",
    82→    "https://www.youtube.com/watch?v=gy-DJYDMaGo",
    83→    "https://www.youtube.com/watch?v=OCKQkllsWro",
    84→    "https://www.youtube.com/watch?v=AqQQPttZ8n4",
    85→    "https://www.youtube.com/watch?v=n8THtt-mQ4E",
    86→    "https://www.youtube.com/watch?v=1PLzwByBA-g",
    87→    "https://www.youtube.com/watch?v=LMeIlNfDClw",
    88→    "https://www.youtube.com/watch?v=AbhrpckFsoE",
    89→    "https://www.youtube.com/watch?v=_1LkUMfJzzU",
    90→    "https://www.youtube.com/watch?v=sPa6CoJsbAQ",
    91→    "https://www.youtube.com/watch?v=MQh7n9e6RPs",
    92→    "https://www.youtube.com/watch?v=8fSdjyyfkj4",
    93→    "https://www.youtube.com/watch?v=rzbwxsaMJEY",
    94→    "https://www.youtube.com/watch?v=AJO1iyrPI4M",
    95→    "https://www.youtube.com/watch?v=BF2gzqeGr3c",
    96→    "https://www.youtube.com/watch?v=aDrCj9Fp-a8",
    97→    "https://www.youtube.com/watch?v=SfnxbRL7-W8",
    98→    "https://www.youtube.com/watch?v=z7tNK92DSsU",
    99→    "https://www.youtube.com/watch?v=bBBz2xzKtf8",
   100→    "https://www.youtube.com/watch?v=kEy3gILUK6g",
   101→    "https://www.youtube.com/watch?v=5UJX9OabZk0",
   102→    "https://www.youtube.com/watch?v=yt7HSAzE-Ic",
   103→    "https://www.youtube.com/watch?v=RUZK2yElCts",
   104→    "https://www.youtube.com/watch?v=XDh4q3zKYEY",
   105→    "https://www.youtube.com/watch?v=x9UAD-0cpkI",
   106→    "https://www.youtube.com/watch?v=9jGwJH1H7lw",
   107→    "https://www.youtube.com/watch?v=Nj8OAEt4DKU",
   108→    "https://www.youtube.com/watch?v=dVW722GEb_o",
   109→    "https://www.youtube.com/watch?v=O-onSX28erM",
   110→    "https://www.youtube.com/watch?v=Akp_e0ScP2o",
   111→    "https://www.youtube.com/watch?v=9_WsmodRndQ",
   112→    "https://www.youtube.com/watch?v=3QsHg8Qp02U",
   113→    "https://www.youtube.com/watch?v=iwFDnf2heTo",
   114→    "https://www.youtube.com/watch?v=IS2KBjMUVy4",
   115→    "https://www.youtube.com/watch?v=iagwBs8KvtM",
   116→    "https://www.youtube.com/watch?v=eqsGRZfEZJE",
   117→    "https://www.youtube.com/watch?v=IRY0On1Gh3Y",
   118→    "https://www.youtube.com/watch?v=nNURyUAmcvI",
   119→    "https://www.youtube.com/watch?v=gWCJQBpYDRY",
   120→    "https://www.youtube.com/watch?v=3nZkxCsv-mY",
   121→    "https://www.youtube.com/watch?v=u-yHKClIFcM",
   122→    "https://www.youtube.com/watch?v=W8Fow2P92Tw",
   123→    "https://www.youtube.com/watch?v=7HeMRXIGWtE",
   124→    "https://www.youtube.com/watch?v=5d-21prdt1k",
   125→    "https://www.youtube.com/watch?v=_EXRUAOFCqg",
   126→    "https://www.youtube.com/watch?v=7xEF1hUdagU",
   127→    "https://www.youtube.com/watch?v=_gP1mPkVP6g",
   128→    "https://www.youtube.com/watch?v=crUbQyWPRp0",
   129→    "https://www.youtube.com/watch?v=HtAGgc8E4P8",
   130→    "https://www.youtube.com/watch?v=8_PMruGFw44",
   131→    "https://www.youtube.com/watch?v=ZFHa06SCwkY",
   132→    "https://www.youtube.com/watch?v=kpnFP7cKIKU",
   133→    "https://www.youtube.com/watch?v=3BwZz_X0Cxc",
   134→    "https://www.youtube.com/watch?v=0Lc_11QFxMM",
   135→    "https://www.youtube.com/watch?v=bEcYg8Q4aWs",
   136→    "https://www.youtube.com/watch?v=9g6XsVfL6aQ",
   137→    "https://www.youtube.com/watch?v=9M2Qlh1hbRI",
   138→    "https://www.youtube.com/watch?v=XcuZm4fEM6U",
   139→    "https://www.youtube.com/watch?v=dWIGbpKqMY4",
   140→    "https://www.youtube.com/watch?v=g2bNCHFcSlE",
   141→    "https://www.youtube.com/watch?v=-FuVIwO2PwQ",
   142→]
   143→
   144→def get_ffprobe_info(filepath):
   145→    try:
   146→        result = subprocess.run(
   147→            ["ffprobe", "-v", "quiet", "-print_format", "json", "-show_format", "-show_streams", filepath],
   148→            capture_output=True, text=True, timeout=30
   149→        )
   150→        info = json.loads(result.stdout)
   151→        stream = info.get("streams", [{}])[0]
   152→        fmt = info.get("format", {})
   153→        return {
   154→            "audio_sample_rate": int(stream.get("sample_rate", 0)),
   155→            "audio_channels": int(stream.get("channels", 0)),
   156→            "audio_duration_sec": round(float(fmt.get("duration", 0)), 2),
   157→            "audio_size_bytes": int(fmt.get("size", 0)),
   158→        }
   159→    except Exception as e:
   160→        return {"error": str(e)}
   161→
   162→def download_one(idx, url):
   163→    vid_id = url.split("v=")[1].split("&")[0]
   164→    prefix = f"{idx:03d}_{vid_id}"
   165→    wav_path = OUT / f"{prefix}.wav"
   166→    meta_path = OUT / f"{prefix}_metadata.json"
   167→
   168→    if meta_path.exists():
   169→        print(f"  [{idx}/{len(URLS)}] SKIP (already done): {vid_id}")
   170→        return True
   171→
   172→    # Get YouTube metadata
   173→    try:
   174→        result = subprocess.run(
   175→            ["yt-dlp", "--dump-json", "--no-download", url],
   176→            capture_output=True, text=True, timeout=60
   177→        )
   178→        yt_info = json.loads(result.stdout)
   179→    except Exception as e:
   180→        print(f"  [{idx}/{len(URLS)}] FAIL metadata: {vid_id} - {e}")
   181→        return False
   182→
   183→    # Download audio as wav
   184→    try:
   185→        subprocess.run(
   186→            ["yt-dlp", "-x", "--audio-format", "wav", "-o", str(wav_path), url],
   187→            capture_output=True, text=True, timeout=300
   188→        )
   189→    except Exception as e:
   190→        print(f"  [{idx}/{len(URLS)}] FAIL download: {vid_id} - {e}")
   191→        return False
   192→
   193→    if not wav_path.exists():
   194→        print(f"  [{idx}/{len(URLS)}] FAIL no file: {vid_id}")
   195→        return False
   196→
   197→    # Get audio file properties
   198→    audio_info = get_ffprobe_info(str(wav_path))
   199→
   200→    metadata = {
   201→        "id": yt_info.get("id"),
   202→        "url": url,
   203→        "title": yt_info.get("title"),
   204→        "channel": yt_info.get("channel"),
   205→        "channel_id": yt_info.get("channel_id"),
   206→        "upload_date": yt_info.get("upload_date"),
   207→        "duration_sec": yt_info.get("duration"),
   208→        "description": yt_info.get("description"),
   209→        "view_count": yt_info.get("view_count"),
   210→        "like_count": yt_info.get("like_count"),
   211→        "tags": yt_info.get("tags", []),
   212→        "categories": yt_info.get("categories", []),
   213→        "language": yt_info.get("language"),
   214→        "playlist_index": idx,
   215→        "audio_file": f"{prefix}.wav",
   216→        "audio_format": "wav",
   217→        "audio_sample_rate": audio_info.get("audio_sample_rate"),
   218→        "audio_channels": audio_info.get("audio_channels"),
   219→        "audio_duration_sec": audio_info.get("audio_duration_sec"),
   220→        "audio_size_bytes": audio_info.get("audio_size_bytes"),
   221→        "downloaded_at": datetime.now(timezone.utc).isoformat(),
   222→    }
   223→
   224→    with open(meta_path, "w") as f:
   225→        json.dump(metadata, f, indent=2, ensure_ascii=False)
   226→
   227→    size_mb = wav_path.stat().st_size / 1024 / 1024
   228→    print(f"  [{idx}/{len(URLS)}] OK: {yt_info.get('title', vid_id)[:60]} ({size_mb:.1f}MB, {yt_info.get('duration', '?')}s)")
   229→    return True
   230→
   231→from concurrent.futures import ThreadPoolExecutor, as_completed
   232→
   233→WORKERS = 10
   234→
   235→print(f"=== Downloading {len(URLS)} videos to {OUT} ({WORKERS} parallel) ===\n")
   236→
   237→success = 0
   238→fail = 0
   239→
   240→with ThreadPoolExecutor(max_workers=WORKERS) as pool:
   241→    futures = {pool.submit(download_one, idx, url): idx for idx, url in enumerate(URLS, 1)}
   242→    for future in as_completed(futures):
   243→        if future.result():
   244→            success += 1
   245→        else:
   246→            fail += 1
   247→
   248→print(f"\n=== DONE: {success} ok, {fail} failed out of {len(URLS)} ===")


This is a download script for YouTube videos. Let me check the /home/ubuntu/hf_staging directory:


total 8.0K
drwxrwxr-x 2 ubuntu ubuntu 4.0K Mar 12 06:31 .
drwxrwxr-x 3 ubuntu ubuntu 4.0K Mar 12 06:31 ..
lrwxrwxrwx 1 ubuntu ubuntu   41 Mar 12 06:31 dataset.jsonl -> /home/ubuntu/modi_processed/dataset.jsonl
lrwxrwxrwx 1 ubuntu ubuntu   36 Mar 12 06:31 segments -> /home/ubuntu/modi_processed/segments


So these are symlinks! Let me check if the original segments directory exists:


ls -la /home/ubuntu/modi_processed/segments 2>&1 | head -30


ls: cannot access '/home/ubuntu/modi_processed/segments': No such file or directory


Let me search for any actual segments or processed audio files:


find /home/ubuntu -name "segments" -type d 2>/dev/null


[Request interrupted by user]