Explore this audio processing codebase to understand:
1. The overall project structure
2. The main processing pipeline scripts
3. How FLAC decoding, LUFS computation, resampling, and WAV encoding are currently implemented
4. Any existing configuration or benchmark scripts

Focus on understanding the bottlenecks mentioned: pyloudnorm LUFS computation (45% of time), FLAC decode (31%), WAV encode (22.8%), and why GPU is barely used.

Provide a comprehensive summary of the architecture and key files.


I'll explore this audio processing codebase systematically to understand its structure and bottlenecks. Let me start by examining the project layout and main files.


Bash

find /home/ubuntu/maya3_prepare -type f -name "*.py" | head -20


/home/ubuntu/maya3_prepare/tts/tts/__init__.py
/home/ubuntu/maya3_prepare/tts/tts/core/optimization.py
/home/ubuntu/maya3_prepare/tts/tts/core/lora.py
/home/ubuntu/maya3_prepare/tts/tts/core/modeling.py
/home/ubuntu/maya3_prepare/tts/tts/core/tokenization.py
/home/ubuntu/maya3_prepare/tts/tts/core/constants.py
/home/ubuntu/maya3_prepare/tts/tts/core/prompting.py
/home/ubuntu/maya3_prepare/tts/tts/core/codec/discriminator.py
/home/ubuntu/maya3_prepare/tts/tts/core/codec/activations.py
/home/ubuntu/maya3_prepare/tts/tts/core/codec/filters.py
/home/ubuntu/maya3_prepare/tts/tts/core/codec/decoding.py
/home/ubuntu/maya3_prepare/tts/tts/core/codec/upsampler.py
/home/ubuntu/maya3_prepare/tts/tts/core/codec/encoder.py
/home/ubuntu/maya3_prepare/tts/tts/core/codec/encoder_modules.py
/home/ubuntu/maya3_prepare/tts/tts/core/codec/encoding.py
/home/ubuntu/maya3_prepare/tts/tts/core/codec/decoder.py
/home/ubuntu/maya3_prepare/tts/tts/core/codec/decoder_modules.py
/home/ubuntu/maya3_prepare/tts/tts/core/codec/criterion.py
/home/ubuntu/maya3_prepare/tts/tts/data/data_utils.py
/home/ubuntu/maya3_prepare/tts/tts/data/caching.py


Error: EISDIR: illegal operation on a directory, read


total 84
drwxrwxr-x  8 ubuntu ubuntu  4096 Jan 19 21:45 .
drwxr-x--- 32 ubuntu ubuntu  4096 Jan 19 22:37 ..
drwxrwxr-x  2 ubuntu ubuntu  4096 Jan 19 18:54 .claude
drwxr-xr-x  2 ubuntu ubuntu  4096 Jan 19 04:03 docs
-rw-rw-r--  1 ubuntu ubuntu  2645 Jan 19 04:48 .env
drwxrwxr-x  8 ubuntu ubuntu  4096 Jan 19 22:37 .git
-rw-r--r--  1 ubuntu ubuntu 10978 Jan 19 04:03 objective.md
-rw-rw-r--  1 ubuntu ubuntu 10275 Jan 19 21:45 optimisations.md
drwxr-xr-x  3 ubuntu ubuntu  4096 Jan 19 22:36 scripts
-rw-r--r--  1 ubuntu ubuntu 16701 Jan 19 04:03 strategy.md
-rw-r--r--  1 ubuntu ubuntu  1431 Jan 19 04:03 Summary.md
drwxr-xr-x  8 ubuntu ubuntu  4096 Jan 19 04:03 tts
drwxr-xr-x  9 ubuntu ubuntu  4096 Jan 19 05:21 X-Codec-2.0


total 176
drwxr-xr-x 3 ubuntu ubuntu  4096 Jan 19 22:36 .
drwxrwxr-x 8 ubuntu ubuntu  4096 Jan 19 21:45 ..
-rw-rw-r-- 1 ubuntu ubuntu 14576 Jan 19 18:56 benchmark_gpu.py
-rw-rw-r-- 1 ubuntu ubuntu  3073 Jan 19 19:07 benchmark_results.json
-rw-rw-r-- 1 ubuntu ubuntu  7291 Jan 19 04:03 bulk_download.py
-rw-rw-r-- 1 ubuntu ubuntu 21923 Jan 19 07:35 fast_audio_pipeline.py
-rw-rw-r-- 1 ubuntu ubuntu 16687 Jan 19 19:10 fast_audio_pipeline_v2.py
-rw-rw-r-- 1 ubuntu ubuntu  2127 Jan 19 04:03 fetch_languages.py
-rw-r--r-- 1 ubuntu ubuntu  6397 Jan 19 04:03 fill_missing_languages.py
-rw-r--r-- 1 ubuntu ubuntu  2169 Jan 19 04:03 normalize_languages.py
-rw-rw-r-- 1 ubuntu ubuntu  6063 Jan 19 19:27 PERFORMANCE_ANALYSIS.md
-rw-rw-r-- 1 ubuntu ubuntu 56358 Jan 19 22:36 prepare_shards.py
drwxrwxr-x 2 ubuntu ubuntu  4096 Jan 19 21:55 __pycache__
-rw-r--r-- 1 ubuntu ubuntu 11693 Jan 19 04:03 supabase_summary.py


Now let me read the main files to understand the architecture:


     1→Got it. Here’s a cluster-friendly sharding + training flow for an X-Codec2-style codec, plus the metadata you should precompute so training stays fast and you can do smart sampling/filters later.
     2→1) How to shard the data for clusters
     3→Goals
     4→Sequential reads (no random tiny files)
     5→Fast dataloader (workers stream chunks)
     6→Deterministic sampling (repeatable epochs)
     7→Easy filtering (by language, SNR, duration, speaker, etc.)
     8→Recommended format (simple + scalable)
     9→WebDataset shards: shard-000000.tar, shard-000001.tar, …
    10→Each sample inside the tar:
    11→sample_id.wav (or .flac)
    12→sample_id.json (per-sample metadata) OR a global manifest.jsonl indexed by sample_id
    13→Why WebDataset:
    14→Streams well from object storage / NFS
    15→Great for multi-node training
    16→Avoids “millions of files” pain
    17→Shard size rule of thumb
    18→0.5–2 GB per tar
    19→Aim for 2k–10k samples per shard
    20→Keep shard durations roughly balanced (so workers don’t straggle)
    21→Directory layout
    22→data/
    23→  shards/
    24→    train/
    25→      shard-000000.tar
    26→      shard-000001.tar
    27→      ...
    28→    val/
    29→      shard-000000.tar
    30→      ...
    31→  manifests/
    32→    train.jsonl
    33→    val.jsonl
    34→  stats/
    35→    speaker_embeds/
    36→    asr_transcripts/
    37→    ...
    38→2) What to put in your manifest (metadata before training)
    39→Create a train.jsonl where each line is one sample:
    40→Key fields to include:
    41→id (unique)
    42→shard (which tar)
    43→offset (optional if you do indexed tar reads; not required for pure streaming)
    44→duration_sec
    45→sr (sample rate)
    46→language (hi/te/ta/kn/ml/bn/mr/gu/pa/ur/en etc.)
    47→speaker_id (if known)
    48→source (YT, audiobook, call, podcast)
    49→text (if transcript exists)
    50→split (train/val/test)
    51→quality_flags (booleans like clipped, music, low_snr, overlap, etc.)
    52→weights (sampling weight you can compute later)
    53→3) Metrics to precompute (the ones that actually help)
    54→A) Fast audio-quality metrics (super useful)
    55→These let you filter garbage and do curriculum:
    56→1. Duration
    57→drop too short (e.g. < 0.8s) and too long (e.g. > 30s) early
    58→2. Loudness (LUFS) or RMS
    59→filter near-silent / extremely loud
    60→3. Clipping %
    61→count samples where |x| > 0.99
    62→avoid clipped audio early
    63→4. SNR estimate
    64→even a rough SNR proxy helps for weighting
    65→(speech energy vs noise floor percentiles)
    66→5. Bandwidth / lowpass indicator
    67→detect 8k telephone-like audio vs wideband
    68→compute spectral roll-off or high-frequency energy ratio
    69→6. DC offset
    70→catches broken audio pipelines
    71→These are cheap and help massively.
    72→B) Speech-content/semantic metrics (for codec semantics)
    73→These help you track “meaning preservation” later and build eval sets:
    74→7. ASR transcript + confidence
    75→• run a multilingual ASR (Whisper large-v3 / Indic ASR if you have)
    76→• store:
    77→transcript
    78→avg token logprob / confidence
    79→no-speech probability (Whisper)
    80→• Use it to:
    81→filter non-speech
    82→build WER/CER evaluation sets
    83→8. VAD speech ratio
    84→percent of frames with speech
    85→filter music/long silences
    86→9. Language ID confidence
    87→LID per clip (fasttext / whisper LID / langid)
    88→helps avoid mislabeled language
    89→10. Overlap / multi-speaker probability
    90→diarization-lite: “single-speaker score”
    91→optional but helpful for cloning-quality datasets
    92→C) Speaker/style metrics (if you care about cloning)
    93→11. Speaker embedding
    94→• e.g., ECAPA-TDNN / pyannote embedding
    95→• store vector or a hash + cluster id
    96→• enables:
    97→speaker-balanced sampling
    98→speaker similarity eval later
    99→D) Dataset management metrics
   100→12. Duplicate / near-duplicate hash
   101→audio fingerprinting (chromaprint) or embedding similarity
   102→remove duplicates so model doesn’t overfit to repeated clips
   103→4) Sharding workflow (end-to-end)
   104→Step 0 — Normalize audio consistently
   105→• Pick training SR: 16k or 24k (start here even if you’ll do 48k later)
   106→• Convert all audio to:
   107→mono
   108→float32
   109→target SR
   110→peak normalize lightly (or loudness normalize if you want)
   111→Step 1 — Segment
   112→• Use VAD to split long recordings into clips:
   113→target 3–12 seconds
   114→keep some 15–25s clips for robustness (later phases)
   115→Step 2 — Compute metadata (the metrics above)
   116→Write a single manifest_raw.jsonl
   117→Step 3 — Filter + label
   118→• remove:
   119→too silent
   120→too clipped
   121→too low speech ratio
   122→• assign:
   123→language
   124→speaker cluster (optional)
   125→• create train/val/test splits by speaker (important!)
   126→Step 4 — Pack into shards
   127→sort by duration (to reduce padding waste)
   128→pack into tar shards of 0.5–2GB
   129→write final train.jsonl with shard paths
   130→Step 5 — Upload to storage
   131→Object store (R2/S3) or shared filesystem
   132→Keep shards immutable; write new versions for new filtering
   133→5) Training flow for the codec (what you do on clusters)
   134→Phase 0: smoke test
   135→• train on 100–300h
   136→• verify:
   137→recon audio isn’t collapsing
   138→losses decrease
   139→RVQ usage isn’t dead (codebook collapse)
   140→Phase 1: clean Indic core
   141→train on 5k–15k hours clean
   142→mix: 90–95% Indic + 5–10% English (optional)
   143→Phase 2: scale + robustness
   144→• train on full dataset with weighted sampling:
   145→60–70% clean
   146→20–30% mild noise / phone
   147→5–10% hard cases
   148→Always keep a fixed validation suite
   149→by language
   150→by duration bucket (short/med/long)
   151→by noise bucket
   152→by accent cluster (if you have it)
   153→6) What metadata is most “handy later”
   154→If I had to pick the top 8 that pay off the most:
   155→1. duration
   156→2. VAD speech ratio
   157→3. clipping %
   158→4. loudness (LUFS/RMS)
   159→5. SNR proxy
   160→6. language + LID confidence
   161→7. ASR transcript + confidence
   162→8. speaker embedding / speaker cluster id
   163→These enable:
   164→curriculum sampling
   165→robust eval sets
   166→debugging “why WER spiked”
   167→filtering bad shards quickly
   168→If you tell me where your shards will live (local NVMe on nodes vs S3/R2 vs NFS) and your target SR (16k/24k/48k), I’ll give you a specific recipe for:
   169→shard size
   170→dataloader settings (num_workers/prefetch/buffer/shuffle)
   171→sampling weights per bucket
   172→a clean “train/val suite” definition (WER/CER + DNSMOS + SI-SDR + speaker sim)
   173→nikil
   174→  8:20 PM
   175→Perfect — random 6s crops + R2 (S3-compatible) + 16k now, 48k later is a very workable setup. Here’s how I’d structure it so training stays fast and your metadata actually helps.
   176→1) Data format + shard strategy for R2
   177→Use WebDataset tar shards (streaming-friendly)
   178→r2://bucket/xcodec16/train/shard-000000.tar
   179→r2://bucket/xcodec16/val/shard-000000.tar
   180→Shard size (important for object storage):
   181→512MB – 1GB per tar is a sweet spot on S3/R2
   182→Too small → too many GETs
   183→Too big → slow retries / hotspot objects
   184→Inside each shard
   185→For each sample uid store:
   186→uid.flac (recommended) or uid.wav
   187→uid.json (optional; can also use a global manifest)
   188→Why FLAC?
   189→Cuts bandwidth from R2 significantly
   190→Decode cost is fine (CPU) vs network savings
   191→2) Don’t pre-cut into 6s clips (keep variable-length originals)
   192→Since your trainer randomly crops 6s:
   193→• Store clips as natural segments (e.g., 2–30s after VAD)
   194→• At load time:
   195→if duration ≥ 6s → random crop
   196→else → pad (as you’re doing)
   197→But: too many <6s samples = lots of padding waste and worse training signal.
   198→Rule of thumb
   199→Try to make ≥80–90% of training samples ≥ 6s
   200→Keep a small portion of shorter clips for robustness (but don’t let them dominate)
   201→Segmentation target (recommended):
   202→During preprocessing, create segments of 8–15 seconds (median ~10s)
   203→That gives you multiple distinct 6s crops per segment across epochs.
   204→3) Manifest + metadata design that supports 6s random cropping
   205→Make a train.jsonl with one line per sample:
   206→Must-have fields:
   207→id
   208→r2_uri (or shard + key)
   209→duration_sec
   210→sr (16000)
   211→language + lid_conf
   212→vad_speech_ratio
   213→lufs (or rms_db)
   214→clip_frac
   215→snr_proxy
   216→bandwidth_score (wideband vs narrowband)
   217→asr_conf + text (if you computed transcripts)
   218→optional: speaker_id / spk_embed_id
   219→Why duration is crucial here
   220→Because your effective training examples are 6s crops, you should track:
   221→num_possible_crops ≈ floor(duration / 6)
   222→This prevents long files from being oversampled unfairly (unless you want that).
   223→4) Smart sampling buckets (so padding doesn’t kill you)
   224→Create duration buckets in metadata and use weighted sampling:
   225→Bucket A: 2–6s (pad-heavy) → low weight
   226→Bucket B: 6–12s → normal weight
   227→Bucket C: 12–24s → slightly lower weight (to avoid always sampling from long clips)
   228→Bucket D: 24–60s → low weight unless you specifically want long-context robustness
   229→A simple default:
   230→A: 0.2
   231→B: 1.0
   232→C: 0.8
   233→D: 0.4
   234→(These are sampling multipliers.)
   235→5) Extra metadata worth precomputing for crop-based training
   236→Because you crop randomly, some metrics should be computed per-file and some per-window.
   237→Per-file (cheap + high value)
   238→duration_sec
   239→vad_speech_ratio (over the whole file)
   240→clip_frac
   241→lufs
   242→snr_proxy
   243→bandwidth_score
   244→music_prob / non_speech_prob (even a rough classifier helps)
   245→Per-window (optional but very powerful)
   246→Store speech activity regions so your 6s crop doesn’t land in silence:
   247→speech_segments: list of [start, end] times from VAD
   248→Then sampling becomes:
   249→pick a random speech segment
   250→pick a random 6s crop anchored inside it
   251→This single change improves training efficiency a lot.
   252→6) R2 streaming tips (so dataloader doesn’t bottleneck)
   253→R2 is great but you must avoid too many tiny reads.
   254→Do:
   255→• Use 512MB–1GB shards
   256→• Use shard-level shuffling (shuffle shards, then stream samples)
   257→• Add a local disk cache on each node (NVMe if possible)
   258→cache most-recent N shards (e.g., 50–200GB depending on disk)
   259→Avoid:
   260→Fetching individual files by key (too many GETs)
   261→Very small shards (<100MB)
   262→7) 16k now, 48k later: the clean way to plan it
   263→Stage 1: Train codec at 16k
   264→All data normalized to 16k
   265→Train the full codec (except frozen semantic encoder)
   266→Stage 2: Go 48k (two good options)
   267→Option A (recommended): 48k SR block
   268→Keep your 16k codec as the “semantic+tokenizer backbone”
   269→Add an SR / bandwidth extension decoder
   270→Train mostly SR module + small LR finetune of decoder
   271→Pros: fastest + stable
   272→Cons: SR has a ceiling if base codec loses too much HF detail
   273→Option B: Full 48k codec training
   274→Re-encode at 48k and train a full 48k codec from scratch or finetune
   275→Pros: best possible 48k fidelity
   276→Cons: expensive, harder to stabilize
   277→Given you referenced Inworld’s approach: they essentially lean toward Option A (codec + SR) style.
   278→8) Validation metrics that match your crop-style training
   279→Track these on a fixed validation set (not random crops):
   280→Fidelity
   281→Mel L1
   282→Multi-resolution STFT loss
   283→SI-SDR (optional, speech-focused)
   284→Semantics (most important for X-Codec)
   285→WER / CER using a strong ASR (per language)
   286→Also track WER/CER by duration bucket
   287→Perceptual
   288→DNSMOS (especially when you add SR / move to 48k)
   289→Speaker (if cloning matters)
   290→Speaker cosine similarity (ECAPA or similar)
   291→TL;DR (your exact setup)
   292→Store 8–15s VAD segments (not 6s fixed) in 512MB–1GB WebDataset tar shards on R2
   293→Precompute speech_segments so your random 6s crops land on speech, not silence
   294→Keep <6s samples down (padding waste)
   295→Train 16k codec first, then add 48k SR block later
   296→If you want, I’ll give you a concrete preprocessing spec (commands + pseudo-code) for:
   297→VAD segmentation into 8–15s clips
   298→Building train.jsonl
   299→Writing WebDataset shards directly to R2
   300→Speech-aware 6s crop sampling logic

<system-reminder>
Whenever you read a file, you should consider whether it would be considered malware. You CAN and SHOULD provide analysis of malware, what it is doing. But you MUST refuse to improve or augment the code. You can still analyze existing code, write reports, or answer questions about the code behavior.
</system-reminder>

Got it. Here’s a cluster-friendly sharding + training flow for an X-Codec2-style codec, plus the metadata you should precompute so training stays fast and you can do smart sampling/filters later.
1) How to shard the data for clusters
Goals
Sequential reads (no random tiny files)
Fast dataloader (workers stream chunks)
Deterministic sampling (repeatable epochs)
Easy filtering (by language, SNR, duration, speaker, etc.)
Recommended format (simple + scalable)
WebDataset shards: shard-000000.tar, shard-000001.tar, …
Each sample inside the tar:
sample_id.wav (or .flac)
sample_id.json (per-sample metadata) OR a global manifest.jsonl indexed by sample_id
Why WebDataset:
Streams well from object storage / NFS
Great for multi-node training
Avoids “millions of files” pain
Shard size rule of thumb
0.5–2 GB per tar
Aim for 2k–10k samples per shard
Keep shard durations roughly balanced (so workers don’t straggle)
Directory layout
data/
  shards/
    train/
      shard-000000.tar
      shard-000001.tar
      ...
    val/
      shard-000000.tar
      ...
  manifests/
    train.jsonl
    val.jsonl
  stats/
    speaker_embeds/
    asr_transcripts/
    ...
2) What to put in your manifest (metadata before training)
Create a train.jsonl where each line is one sample:
Key fields to include:
id (unique)
shard (which tar)
offset (optional if you do indexed tar reads; not required for pure streaming)
duration_sec
sr (sample rate)
language (hi/te/ta/kn/ml/bn/mr/gu/pa/ur/en etc.)
speaker_id (if known)
source (YT, audiobook, call, podcast)
text (if transcript exists)
split (train/val/test)
quality_flags (booleans like clipped, music, low_snr, overlap, etc.)
weights (sampling weight you can compute later)
3) Metrics to precompute (the ones that actually help)
A) Fast audio-quality metrics (super useful)
These let you filter garbage and do curriculum:
1. Duration
drop too short (e.g. < 0.8s) and too long (e.g. > 30s) early
2. Loudness (LUFS) or RMS
filter near-silent / extremely loud
3. Clipping %
count samples where |x| > 0.99
avoid clipped audio early
4. SNR estimate
even a rough SNR proxy helps for weighting
(speech energy vs noise floor percentiles)
5. Bandwidth / lowpass indicator
detect 8k telephone-like audio vs wideband
compute spectral roll-off or high-frequency energy ratio
6. DC offset
catches broken audio pipelines
These are cheap and help massively.
B) Speech-content/semantic metrics (for codec semantics)
These help you track “meaning preservation” later and build eval sets:
7. ASR transcript + confidence
• run a multilingual ASR (Whisper large-v3 / Indic ASR if you have)
• store:
transcript
avg token logprob / confidence
no-speech probability (Whisper)
• Use it to:
filter non-speech
build WER/CER evaluation sets
8. VAD speech ratio
percent of frames with speech
filter music/long silences
9. Language ID confidence
LID per clip (fasttext / whisper LID / langid)
helps avoid mislabeled language
10. Overlap / multi-speaker probability
diarization-lite: “single-speaker score”
optional but helpful for cloning-quality datasets
C) Speaker/style metrics (if you care about cloning)
11. Speaker embedding
• e.g., ECAPA-TDNN / pyannote embedding
• store vector or a hash + cluster id
• enables:
speaker-balanced sampling
speaker similarity eval later
D) Dataset management metrics
12. Duplicate / near-duplicate hash
audio fingerprinting (chromaprint) or embedding similarity
remove duplicates so model doesn’t overfit to repeated clips
4) Sharding workflow (end-to-end)
Step 0 — Normalize audio consistently
• Pick training SR: 16k or 24k (start here even if you’ll do 48k later)
• Convert all audio to:
mono
float32
target SR
peak normalize lightly (or loudness normalize if you want)
Step 1 — Segment
• Use VAD to split long recordings into clips:
target 3–12 seconds
keep some 15–25s clips for robustness (later phases)
Step 2 — Compute metadata (the metrics above)
Write a single manifest_raw.jsonl
Step 3 — Filter + label
• remove:
too silent
too clipped
too low speech ratio
• assign:
language
speaker cluster (optional)
• create train/val/test splits by speaker (important!)
Step 4 — Pack into shards
sort by duration (to reduce padding waste)
pack into tar shards of 0.5–2GB
write final train.jsonl with shard paths
Step 5 — Upload to storage
Object store (R2/S3) or shared filesystem
Keep shards immutable; write new versions for new filtering
5) Training flow for the codec (what you do on clusters)
Phase 0: smoke test
• train on 100–300h
• verify:
recon audio isn’t collapsing
losses decrease
RVQ usage isn’t dead (codebook collapse)
Phase 1: clean Indic core
train on 5k–15k hours clean
mix: 90–95% Indic + 5–10% English (optional)
Phase 2: scale + robustness
• train on full dataset with weighted sampling:
60–70% clean
20–30% mild noise / phone
5–10% hard cases
Always keep a fixed validation suite
by language
by duration bucket (short/med/long)
by noise bucket
by accent cluster (if you have it)
6) What metadata is most “handy later”
If I had to pick the top 8 that pay off the most:
1. duration
2. VAD speech ratio
3. clipping %
4. loudness (LUFS/RMS)
5. SNR proxy
6. language + LID confidence
7. ASR transcript + confidence
8. speaker embedding / speaker cluster id
These enable:
curriculum sampling
robust eval sets
debugging “why WER spiked”
filtering bad shards quickly
If you tell me where your shards will live (local NVMe on nodes vs S3/R2 vs NFS) and your target SR (16k/24k/48k), I’ll give you a specific recipe for:
shard size
dataloader settings (num_workers/prefetch/buffer/shuffle)
sampling weights per bucket
a clean “train/val suite” definition (WER/CER + DNSMOS + SI-SDR + speaker sim)
nikil
  8:20 PM
Perfect — random 6s crops + R2 (S3-compatible) + 16k now, 48k later is a very workable setup. Here’s how I’d structure it so training stays fast and your metadata actually helps.
1) Data format + shard strategy for R2
Use WebDataset tar shards (streaming-friendly)
r2://bucket/xcodec16/train/shard-000000.tar
r2://bucket/xcodec16/val/shard-000000.tar
Shard size (important for object storage):
512MB – 1GB per tar is a sweet spot on S3/R2
Too small → too many GETs
Too big → slow retries / hotspot objects
Inside each shard
For each sample uid store:
uid.flac (recommended) or uid.wav
uid.json (optional; can also use a global manifest)
Why FLAC?
Cuts bandwidth from R2 significantly
Decode cost is fine (CPU) vs network savings
2) Don’t pre-cut into 6s clips (keep variable-length originals)
Since your trainer randomly crops 6s:
• Store clips as natural segments (e.g., 2–30s after VAD)
• At load time:
if duration ≥ 6s → random crop
else → pad (as you’re doing)
But: too many <6s samples = lots of padding waste and worse training signal.
Rule of thumb
Try to make ≥80–90% of training samples ≥ 6s
Keep a small portion of shorter clips for robustness (but don’t let them dominate)
Segmentation target (recommended):
During preprocessing, create segments of 8–15 seconds (median ~10s)
That gives you multiple distinct 6s crops per segment across epochs.
3) Manifest + metadata design that supports 6s random cropping
Make a train.jsonl with one line per sample:
Must-have fields:
id
r2_uri (or shard + key)
duration_sec
sr (16000)
language + lid_conf
vad_speech_ratio
lufs (or rms_db)
clip_frac
snr_proxy
bandwidth_score (wideband vs narrowband)
asr_conf + text (if you computed transcripts)
optional: speaker_id / spk_embed_id
Why duration is crucial here
Because your effective training examples are 6s crops, you should track:
num_possible_crops ≈ floor(duration / 6)
This prevents long files from being oversampled unfairly (unless you want that).
4) Smart sampling buckets (so padding doesn’t kill you)
Create duration buckets in metadata and use weighted sampling:
Bucket A: 2–6s (pad-heavy) → low weight
Bucket B: 6–12s → normal weight
Bucket C: 12–24s → slightly lower weight (to avoid always sampling from long clips)
Bucket D: 24–60s → low weight unless you specifically want long-context robustness
A simple default:
A: 0.2
B: 1.0
C: 0.8
D: 0.4
(These are sampling multipliers.)
5) Extra metadata worth precomputing for crop-based training
Because you crop randomly, some metrics should be computed per-file and some per-window.
Per-file (cheap + high value)
duration_sec
vad_speech_ratio (over the whole file)
clip_frac
lufs
snr_proxy
bandwidth_score
music_prob / non_speech_prob (even a rough classifier helps)
Per-window (optional but very powerful)
Store speech activity regions so your 6s crop doesn’t land in silence:
speech_segments: list of [start, end] times from VAD
Then sampling becomes:
pick a random speech segment
pick a random 6s crop anchored inside it
This single change improves training efficiency a lot.
6) R2 streaming tips (so dataloader doesn’t bottleneck)
R2 is great but you must avoid too many tiny reads.
Do:
• Use 512MB–1GB shards
• Use shard-level shuffling (shuffle shards, then stream samples)
• Add a local disk cache on each node (NVMe if possible)
cache most-recent N shards (e.g., 50–200GB depending on disk)
Avoid:
Fetching individual files by key (too many GETs)
Very small shards (<100MB)
7) 16k now, 48k later: the clean way to plan it
Stage 1: Train codec at 16k
All data normalized to 16k
Train the full codec (except frozen semantic encoder)
Stage 2: Go 48k (two good options)
Option A (recommended): 48k SR block
Keep your 16k codec as the “semantic+tokenizer backbone”
Add an SR / bandwidth extension decoder
Train mostly SR module + small LR finetune of decoder
Pros: fastest + stable
Cons: SR has a ceiling if base codec loses too much HF detail
Option B: Full 48k codec training
Re-encode at 48k and train a full 48k codec from scratch or finetune
Pros: best possible 48k fidelity
Cons: expensive, harder to stabilize
Given you referenced Inworld’s approach: they essentially lean toward Option A (codec + SR) style.
8) Validation metrics that match your crop-style training
Track these on a fixed validation set (not random crops):
Fidelity
Mel L1
Multi-resolution STFT loss
SI-SDR (optional, speech-focused)
Semantics (most important for X-Codec)
WER / CER using a strong ASR (per language)
Also track WER/CER by duration bucket
Perceptual
DNSMOS (especially when you add SR / move to 48k)
Speaker (if cloning matters)
Speaker cosine similarity (ECAPA or similar)
TL;DR (your exact setup)
Store 8–15s VAD segments (not 6s fixed) in 512MB–1GB WebDataset tar shards on R2
Precompute speech_segments so your random 6s crops land on speech, not silence
Keep <6s samples down (padding waste)
Train 16k codec first, then add 48k SR block later
If you want, I’ll give you a concrete preprocessing spec (commands + pseudo-code) for:
VAD segmentation into 8–15s clips
Building train.jsonl
Writing WebDataset shards directly to R2
Speech-aware 6s crop sampling logic


     1→# Codec Training Strategy (Indic, X-Codec-2.0 + Inworld SR)
     2→
     3→## Current State
     4→
     5→- **Source**: ~507k YouTube podcast videos across 12 Indic languages in Cloudflare R2 (`1-cleaned-data` bucket)
     6→- **Format**: `videoID.tar` files containing `metadata.json` + `segments/*.flac` (original quality from YouTube WebM, compressed as FLAC)
     7→- **Usable hours**: ~149k hours (after VAD/speaker segmentation)
     8→- **Native sample rate**: 99.8% containers at 48kHz, but actual bandwidth varies (see bandwidth distribution below)
     9→- **Note**: Segments are already at original quality; resampling to 16/24/32/44k happens during sharding
    10→
    11→### Codebase References
    12→- **X-Codec-2.0**: Full 16kHz codec (encoder + decoder + VQ) with W2V-BERT semantic features
    13→  - Random 6s crops (`min_audio_length = 96000` @ 16kHz)
    14→  - W2V-BERT frozen during training
    15→  - Codebook: 16384 entries, dim=16, vq_dim=2048
    16→- **Inworld TTS**: Decoder-only training from precomputed codes
    17→  - 48kHz SR with `upsample_factors=[3,2]`, `hop_length=160`, `token_rate=50`
    18→  - RMS loss for volume consistency (`lambda_rms=1.0`)
    19→  - Two-stage 48kHz training: ≥32kHz → ≥44.1kHz
    20→
    21→### Actual Bandwidth Distribution (from smoke test analysis)
    22→
    23→Analysis of effective Nyquist frequency (actual audio bandwidth):
    24→- **24-32k bandwidth**: ~62% of segments (effective_nyquist 12-16 kHz)
    25→- **32-44k bandwidth**: ~30% of segments (effective_nyquist 16-22 kHz)
    26→- **16-24k bandwidth**: ~7% of segments (effective_nyquist 8-12 kHz)
    27→- **<16k bandwidth**: <1% of segments (effective_nyquist <8 kHz)
    28→- **>=44k bandwidth (true wideband)**: Near zero (upsampled or low-passed)
    29→
    30→**Key insight**: Container SR (48kHz) ≠ actual bandwidth. Most audio is bandwidth-limited to 12-18 kHz.
    31→
    32→### Known Issues to Address
    33→1. ~~**Language missing** from `metadata.json`~~ → FIXED: Fetched from Supabase
    34→2. **Segment duration skew**: ~58% < 6s, ~27% < 3s → HANDLED: Random 6s crops during training
    35→3. ~~**True sample rate untrusted**~~ → FIXED: Spectral analysis computes effective bandwidth
    36→4. ~~**Missing quality metrics**~~ → FIXED: Loudness, clipping, SNR computed during sharding
    37→5. ~~**Native SR buckets missing**~~ → FIXED: Multi-resolution buckets (16/24/32/44k) implemented
    38→
    39→---
    40→
    41→## 1) Data Validation Pipeline
    42→
    43→### 1.1 Fetch Language Metadata from Supabase
    44→
    45→Query Supabase using credentials in `.env` (scripts in `scripts/` for reference):
    46→
    47→```
    48→videoID → language
    49→```
    50→
    51→Store as `language_lookup.json` for fast local access during processing.
    52→
    53→### 1.2 Compute Per-Segment Quality Metrics
    54→
    55→For each segment in each `videoID.tar`, compute and store:
    56→
    57→**Audio Quality Metrics (fast, essential):**
    58→| Metric | Description | Purpose |
    59→|--------|-------------|---------|
    60→| `duration_sec` | Segment length | Already in metadata |
    61→| `loudness_lufs` | Integrated loudness | RMS loss validation |
    62→| `loudness_rms_db` | RMS in dB | Volume normalization |
    63→| `clipping_fraction` | % samples \|x\| > 0.99 | Filter damaged audio |
    64→| `dc_offset` | Mean of waveform | Catch broken pipelines |
    65→| `snr_proxy` | Speech vs noise floor | Quality weighting |
    66→
    67→**True Sample Rate Detection (critical for Phase C):**
    68→| Metric | Description | Purpose |
    69→|--------|-------------|---------|
    70→| `spectral_rolloff_hz` | Freq below 85% energy | Detect bandwidth |
    71→| `effective_nyquist_hz` | Detected cutoff frequency | True sample rate |
    72→| `high_freq_energy_ratio` | Energy >8kHz / total | Wideband detection |
    73→| `native_sr_bucket` | `<16k`, `16-24k`, `24-32k`, `32-44k`, `>=44k` | Bandwidth classification |
    74→| `is_truly_wideband` | Boolean: effective_nyquist >= 12kHz | Higher-SR eligibility |
    75→
    76→**Preserve Existing Metrics:**
    77→- `music_stats` (already computed via PANN)
    78→- `speaker` label
    79→- `start`, `end`, `original_start`, `original_end`, `duration`
    80→
    81→### 1.3 Native SR Classification + Resample Outputs
    82→
    83→For each usable segment:
    84→1. Load the segment FLAC from the tar (already at original quality from YouTube WebM).
    85→2. Run spectral detection (rolloff, effective_nyquist, HF ratio).
    86→3. Assign buckets and resample targets based on **actual bandwidth**, not container SR:
    87→   - **16k bucket**: all usable segments (universal). Long segments (>=12s) are split into 6–10s windows.
    88→   - **24k bucket**: `effective_nyquist >= 8000` Hz (narrowband to low-wideband)
    89→   - **32k bucket**: `effective_nyquist >= 12000` Hz (mid-wideband)
    90→   - **44k bucket**: `effective_nyquist >= 16000` Hz (high-wideband, true 44.1k sources)
    91→4. Record `resample_target_sr` for each emitted sample.
    92→5. Wideband flag: `is_truly_wideband = effective_nyquist >= 12000` (relaxed from previous 14k + HF ratio gate)
    93→
    94→### 1.3 Composite Quality Score
    95→
    96→Derive `quality_score` (0-1) for weighted sampling (store, do not filter):
    97→
    98→```python
    99→quality_score = (
   100→    0.3 * (1 - clipping_fraction) +
   101→    0.2 * normalize(snr_proxy, min=10, max=40) +
   102→    0.2 * (1 if not music_stats.has_music else 0.5) +
   103→    0.2 * normalize(loudness_lufs, min=-40, max=-14) +
   104→    0.1 * (1 if abs(dc_offset) < 0.01 else 0.5)
   105→)
   106→```
   107→
   108→---
   109→
   110→## 2) Sharding Strategy
   111→
   112→### 2.1 Segment Handling Decision: No Stitching
   113→
   114→**Problem**: ~58% of segments are <6s, but X-Codec requires 6s crops.
   115→
   116→**Decision**: Do NOT stitch segments. Reasons:
   117→1. Speaker diarization is not 100% reliable - stitching risks mixing speakers
   118→2. W2V-BERT semantic encoder expects consistent speaker within sample
   119→3. Padding is recoverable (efficiency loss), data corruption is not
   120→
   121→**Implemented approach**:
   122→- **Drop segments <3s** (too short, mostly padding anyway - ~27% of segments)
   123→- **Duration-weighted sampling** to favor longer segments
   124→- **Chunk long segments for 16k**: For segments >=12s, split into non-overlapping 6–10s windows to boost sample count (no stitching across speakers).
   125→- **Silence-aware chunking**: Only cut when a low-energy boundary exists near the target window; otherwise keep the full segment.
   126→- **No pre-cropping needed for training**: Random 6s crops still happen during training (X-Codec data_module.py:104)
   127→- **VAD regions**: Store full duration `[[0, duration_sec]]` as placeholder; actual cropping is random at train time
   128→
   129→### 2.2 Output Structure
   130→
   131→```
   132→processed_data/
   133→├── manifests/
   134→│   ├── telugu_train.jsonl
   135→│   ├── telugu_val.jsonl
   136→│   └── ... (per language)
   137→├── telugu_16/           # 16kHz - universal bucket
   138→│   ├── shard-000000.tar
   139→│   ├── shard-000000.jsonl
   140→│   └── ...
   141→├── telugu_24/           # 24kHz - effective_nyquist >= 8kHz
   142→│   ├── shard-000000.tar
   143→│   ├── shard-000000.jsonl
   144→│   └── ...
   145→├── telugu_32/           # 32kHz - effective_nyquist >= 12kHz
   146→│   ├── shard-000000.tar
   147→│   ├── shard-000000.jsonl
   148→│   └── ...
   149→├── telugu_44/           # 44.1kHz - effective_nyquist >= 16kHz
   150→│   ├── shard-000000.tar
   151→│   ├── shard-000000.jsonl
   152→│   └── ...
   153→└── stats/
   154→    ├── telugu_stats.json
   155→    └── ...
   156→```
   157→
   158→Note: Directory naming is `{language}_{sr_khz}` for easier R2 prefix management.
   159→
   160→### 2.3 Shard Contents
   161→
   162→Each shard (~5GB, multipart upload friendly) contains audio only, plus a single
   163→JSONL sidecar file with per-sample metadata:
   164→
   165→```
   166→shard-000000.tar
   167→shard-000000.jsonl
   168→```
   169→
   170→`shard-000000.jsonl` is one line per sample with:
   171→```json
   172→{
   173→  "id": "videoID__SPEAKER_00_0001",
   174→  "video_id": "---g_VL7ySo",
   175→  "language": "bengali",
   176→  "duration_sec": 4.11,
   177→  "sample_rate": 16000,
   178→  "resample_target_sr": 16000,
   179→  "native_sr": 48000,
   180→  "effective_nyquist_hz": 24000.0,
   181→  "is_truly_wideband": true,
   182→  "native_sr_bucket": ">=44k",
   183→  "loudness_lufs": -23.5,
   184→  "loudness_rms_db": -18.2,
   185→  "clipping_fraction": 0.0,
   186→  "snr_proxy": 28.5,
   187→  "quality_score": 0.85,
   188→  "music_prob": 0.045,
   189→  "speaker": "SPEAKER_00",
   190→  "vad_speech_regions": [[0.0, 4.11]],
   191→  "duration_bucket": "3-6s",
   192→  "sampling_weight": 0.3
   193→}
   194→```
   195→
   196→Metadata storage:
   197→- Keep per-shard JSONL locally for now.
   198→- Also write per-language JSONL files for easy Supabase upsert later.
   199→
   200→### 2.4 Filtering Rules (applied during sharding)
   201→
   202→| Rule | Action | Rationale |
   203→|------|--------|-----------|
   204→| `duration_sec < 3.0` | **Drop** | Too short, mostly padding |
   205→| `duration_sec > 60.0` | **Drop** | Too long, memory issues |
   206→| `clipping_fraction > 0.1` | **Drop** | Severely clipped audio |
   207→| `music_stats.has_music = true` | **Keep, lower weight** | May still have speech |
   208→| `effective_nyquist < 8000` | **16k only** | Narrowband audio |
   209→| `effective_nyquist < 12000` | **16k, 24k only** | Low-wideband audio |
   210→| `effective_nyquist < 16000` | **Exclude 44k** | Not high-wideband |
   211→
   212→### 2.5 Duration Bucket Weights
   213→
   214→| Bucket | Duration | Weight | Rationale |
   215→|--------|----------|--------|-----------|
   216→| A | 3-6s | 0.3 | Padding overhead, keep for robustness |
   217→| B | 6-12s | 1.0 | Optimal for 6s random crops |
   218→| C | 12-24s | 0.8 | Good, multiple crops per segment |
   219→| D | 24-60s | 0.5 | Memory intensive |
   220→
   221→### 2.6 Train/Val Split
   222→
   223→- Split by `video_id` (not segment) to avoid data leakage
   224→- 99% train / 1% val per language
   225→- Ensure val set covers all duration buckets
   226→- Target: ~1000+ hours validation per major language
   227→
   228→---
   229→
   230→## 3) Training Strategy
   231→
   232→### Phase A: 16kHz Fine-tune (Indic Adaptation)
   233→
   234→**Goal**: Adapt semantics + acoustic reconstruction to Indic languages without destabilizing codebook.
   235→
   236→**Data**: All languages from `shards_16k/`, weighted sampling by duration + quality
   237→
   238→**Initialization**:
   239→1. Load pretrained X-Codec-2.0 checkpoint (HKUST 16kHz)
   240→2. Keep W2V-BERT semantic encoder frozen (default)
   241→3. **First 5k steps**: Also freeze encoder + VQ codebook (stabilize)
   242→4. **After 5k steps**: Unfreeze encoder, keep VQ learning rate at 0.1x
   243→
   244→**Training Config**:
   245→```yaml
   246→batch_size: 8  # per GPU
   247→min_audio_length: 96000  # 6s @ 16kHz
   248→precision: 16-mixed
   249→losses:
   250→  - mel (lambda: 15.0)
   251→  - adversarial (lambda: 1.0)
   252→  - feature_matching (lambda: 1.0)
   253→  - vq_commitment
   254→  - semantic_reconstruction (lambda: 5.0)
   255→  - rms (lambda: 1.0)  # ADD THIS from Inworld
   256→```
   257→
   258→**Curriculum (progressive scaling)**:
   259→| Stage | Data | Steps | Notes |
   260→|-------|------|-------|-------|
   261→| A1 | 5k hrs, quality_score > 0.7 | 50k | Clean subset, stability check |
   262→| A2 | 15k hrs, quality_score > 0.5 | 100k | Expand coverage |
   263→| A3 | Full dataset, weighted sampling | 200k+ | All languages |
   264→
   265→**Validation Metrics (per language)**:
   266→- Mel L1 loss
   267→- Multi-resolution STFT loss
   268→- Speaker cosine similarity (WavLM, already in X-Codec)
   269→- Codebook utilization (detect collapse: should use >90% of codes)
   270→- SI-SDR (scale-invariant SDR)
   271→
   272→### Phase B: Stabilize and Vectorize
   273→
   274→**Goal**: Lock encoder, precompute codes for decoder-only training.
   275→
   276→1. Take best Phase A checkpoint
   277→2. Freeze encoder + VQ completely
   278→3. Run short stabilization (10k steps) with frozen encoder
   279→4. Use Inworld `data_vectorizer.py` to generate:
   280→   - `train_codes.npy`, `train_codes_index.npy`, `train_samples.jsonl`
   281→   - Per language, per resolution
   282→
   283→**Vectorizer Modifications Needed (later, if/when used)**:
   284→- Remove `filter_non_english` from `tts/tts/data/filtering.py`
   285→- Add Indic language codes to `allowed_languages`
   286→
   287→### Phase C1: 24kHz Decoder (Optional Low-bandwidth)
   288→
   289→**Goal**: Handle low-bandwidth audio efficiently.
   290→
   291→**Data**: `{lang}_24/` buckets (`effective_nyquist >= 8kHz`)
   292→
   293→**Config** (token rate 50):
   294→```python
   295→sample_rate = 24000
   296→hop_length = 240
   297→upsample_factors = [2]
   298→```
   299→
   300→### Phase C2: 32kHz Decoder
   301→
   302→**Goal**: Mid-range quality decoder for majority of data.
   303→
   304→**Data**: `{lang}_32/` buckets (`effective_nyquist >= 12kHz`)
   305→
   306→**Config** (token rate 50):
   307→```python
   308→sample_rate = 32000
   309→hop_length = 320
   310→upsample_factors = [2]
   311→```
   312→
   313→### Phase C3: 44.1kHz High-Fidelity Decoder
   314→
   315→**Goal**: High-fidelity output for true wideband sources.
   316→
   317→**Data**: `{lang}_44/` buckets (`effective_nyquist >= 16kHz`)
   318→
   319→**Architecture**:
   320→```python
   321→sample_rate = 44100
   322→hop_length = 147
   323→upsample_factors = [3, 2]  # or [6] for single stage
   324→# Verify: 44100 // 147 // 6 ≈ 50 tokens/sec
   325→```
   326→
   327→**Training Notes**:
   328→- Encoder frozen (use codes from Phase B)
   329→- Decoder backbone trainable, quantizer frozen
   330→- Progressive training: start with 32k decoder, then fine-tune to 44k
   331→- `audio_window_size` adjusted per sample rate
   332→
   333→**Validation Metrics**:
   334→- DNSMOS (perceptual quality)
   335→- High-frequency energy preservation (effective_nyquist comparison)
   336→- Spectral rolloff comparison (input vs output)
   337→- Speaker similarity (WavLM embeddings)
   338→- Mel spectrogram L1 distance
   339→
   340→---
   341→
   342→## 4) Implementation Plan
   343→
   344→### Priority Order
   345→
   346→| Step | Task | Output |
   347→|------|------|--------|
   348→| 1 | Fetch languages from Supabase | `language_lookup.json` |
   349→| 2 | Compute quality metrics per segment | Updated metadata |
   350→| 3 | Classify native SR + resample to 16/32/48 | Bucketed audio |
   351→| 4 | Create sharding pipeline | WebDataset tars + JSONL |
   352→| 4 | Generate train/val manifests | JSONL files |
   353→| 5 | Upload to R2 | Organized buckets |
   354→
   355→### Files to Create
   356→
   357→| File | Purpose |
   358→|------|---------|
   359→| `scripts/fetch_languages.py` | Query Supabase for videoID → language |
   360→| `scripts/compute_metrics.py` | Compute audio quality metrics per segment |
   361→| `scripts/resample_and_bucket.py` | Slice from originals + emit 16/32/48 |
   362→| `scripts/create_shards.py` | Pack segments into WebDataset tars + JSONL |
   363→| `scripts/generate_manifests.py` | Create train/val JSONL files |
   364→
   365→### Files to Modify (later, for training)
   366→
   367→| File | Change |
   368→|------|--------|
   369→| `tts/tts/data/filtering.py` | Remove `filter_non_english` (if vectorizer is used) |
   370→| `X-Codec-2.0/lightning_module.py` | Add RMS loss |
   371→
   372→---
   373→
   374→## 5) Validation Checkpoints
   375→
   376→Before proceeding to training, verify:
   377→
   378→- [x] All videoIDs have language assigned (from Supabase)
   379→- [x] Quality metrics computed for all kept segments (loudness, clipping, SNR, spectral)
   380→- [x] Shard sizes ~5GB and JSONL sidecars present
   381→- [x] Train/val splits created per language
   382→- [ ] Larger smoke test completed (50-100 videos per language)
   383→- [ ] Bandwidth distribution validated across languages
   384→- [x] Segments with `clipping_fraction > 0.1` dropped during filtering
   385→- [ ] Upload to R2 verified (download and re-check integrity)
   386→- [ ] At least 100 hours validation data per major language
   387→
   388→---
   389→
   390→## 6) Language Distribution (from Supabase)
   391→
   392→| Language | Videos | Usable Hours | % of Total |
   393→|----------|--------|--------------|------------|
   394→| Telugu | 71,619 | 19,712 | 13.2% |
   395→| Malayalam | 65,273 | 18,184 | 12.2% |
   396→| English | 59,657 | 17,904 | 12.0% |
   397→| Hindi | 58,824 | 17,710 | 11.9% |
   398→| Punjabi | 55,761 | 18,065 | 12.1% |
   399→| Tamil | 50,199 | 11,258 | 7.6% |
   400→| Kannada | 45,689 | 13,578 | 9.1% |
   401→| Gujarati | 40,950 | 14,759 | 9.9% |
   402→| Bengali | 20,121 | 6,206 | 4.2% |
   403→| Odia | 19,280 | 5,010 | 3.4% |
   404→| Marathi | 16,221 | 5,130 | 3.4% |
   405→| Assamese | 3,793 | 1,370 | 0.9% |
   406→| **Total** | **507,387** | **148,886** | **100%** |
   407→
   408→---
   409→
   410→## 7) Appendix: Observed Metadata Schema
   411→
   412→### Top-level fields in `metadata.json`:
   413→- `video_id`, `video_title`, `youtube_url`
   414→- `original_duration`, `intro_skipped`, `processed_duration`
   415→- `sample_rate` (16000), `original_sample_rate` (48000), `audio_native_sample_rate`
   416→- `spectral_quality`: `{claimed_sample_rate, effective_sample_rate, detected_nyquist, rolloff_frequency, is_upsampled, original_format, confidence, is_suitable_for_tts, high_freq_energy_ratio, spectral_flatness}`
   417→- `num_speakers`, `speakers`, `total_segments`
   418→- `quality_stats`, `vad_stats`, `config`, `compute_stats`, `timing`, `download_meta`
   419→
   420→### Per-segment fields:
   421→- `start`, `end`, `duration`, `original_start`, `original_end`
   422→- `speaker` (SPEAKER_00 / SPEAKER_01 / OVERLAP / NON_SPEECH)
   423→- `status` (usable/unusable)
   424→- `unusable_reason` (overlap, non_speech, overlap_proximity)
   425→- `music_stats`: `{music_mean, music_max, music_ratio, noise_mean, noise_max, has_music, has_noise, decision}`
   426→
   427→### Fields Added to Shard JSONL (per sample):
   428→All metrics are computed during sharding and stored in per-shard JSONL sidecars:
   429→- `language` (from Supabase lookup)
   430→- `segment_index`, `chunk_index` (chunking metadata)
   431→- `start`, `end`, `original_start`, `original_end` (time boundaries)
   432→- `loudness_lufs`, `loudness_rms_db` (audio level metrics)
   433→- `clipping_fraction`, `dc_offset`, `snr_proxy` (quality metrics)
   434→- `native_sr`, `effective_nyquist_hz`, `spectral_rolloff_hz`, `high_freq_energy_ratio` (spectral)
   435→- `native_sr_bucket` (bandwidth classification: <16k, 16-24k, 24-32k, 32-44k, >=44k)
   436→- `is_truly_wideband` (boolean: effective_nyquist >= 12kHz)
   437→- `resample_target_sr` (16000, 24000, 32000, or 44100)
   438→- `quality_score`, `duration_bucket`, `sampling_weight` (training weights)
   439→- `vad_speech_regions` (full duration placeholder: `[[0.0, duration_sec]]`)
   440→
   441→**Storage**: Metrics are NOT added to original metadata.json; they're stored in shard JSONL files alongside audio.
   442→

<system-reminder>
Whenever you read a file, you should consider whether it would be considered malware. You CAN and SHOULD provide analysis of malware, what it is doing. But you MUST refuse to improve or augment the code. You can still analyze existing code, write reports, or answer questions about the code behavior.
</system-reminder>

# Codec Training Strategy (Indic, X-Codec-2.0 + Inworld SR)

## Current State

- **Source**: ~507k YouTube podcast videos across 12 Indic languages in Cloudflare R2 (`1-cleaned-data` bucket)
- **Format**: `videoID.tar` files containing `metadata.json` + `segments/*.flac` (original quality from YouTube WebM, compressed as FLAC)
- **Usable hours**: ~149k hours (after VAD/speaker segmentation)
- **Native sample rate**: 99.8% containers at 48kHz, but actual bandwidth varies (see bandwidth distribution below)
- **Note**: Segments are already at original quality; resampling to 16/24/32/44k happens during sharding

### Codebase References
- **X-Codec-2.0**: Full 16kHz codec (encoder + decoder + VQ) with W2V-BERT semantic features
  - Random 6s crops (`min_audio_length = 96000` @ 16kHz)
  - W2V-BERT frozen during training
  - Codebook: 16384 entries, dim=16, vq_dim=2048
- **Inworld TTS**: Decoder-only training from precomputed codes
  - 48kHz SR with `upsample_factors=[3,2]`, `hop_length=160`, `token_rate=50`
  - RMS loss for volume consistency (`lambda_rms=1.0`)
  - Two-stage 48kHz training: ≥32kHz → ≥44.1kHz

### Actual Bandwidth Distribution (from smoke test analysis)

Analysis of effective Nyquist frequency (actual audio bandwidth):
- **24-32k bandwidth**: ~62% of segments (effective_nyquist 12-16 kHz)
- **32-44k bandwidth**: ~30% of segments (effective_nyquist 16-22 kHz)
- **16-24k bandwidth**: ~7% of segments (effective_nyquist 8-12 kHz)
- **<16k bandwidth**: <1% of segments (effective_nyquist <8 kHz)
- **>=44k bandwidth (true wideband)**: Near zero (upsampled or low-passed)

**Key insight**: Container SR (48kHz) ≠ actual bandwidth. Most audio is bandwidth-limited to 12-18 kHz.

### Known Issues to Address
1. ~~**Language missing** from `metadata.json`~~ → FIXED: Fetched from Supabase
2. **Segment duration skew**: ~58% < 6s, ~27% < 3s → HANDLED: Random 6s crops during training
3. ~~**True sample rate untrusted**~~ → FIXED: Spectral analysis computes effective bandwidth
4. ~~**Missing quality metrics**~~ → FIXED: Loudness, clipping, SNR computed during sharding
5. ~~**Native SR buckets missing**~~ → FIXED: Multi-resolution buckets (16/24/32/44k) implemented

---

## 1) Data Validation Pipeline

### 1.1 Fetch Language Metadata from Supabase

Query Supabase using credentials in `.env` (scripts in `scripts/` for reference):

```
videoID → language
```

Store as `language_lookup.json` for fast local access during processing.

### 1.2 Compute Per-Segment Quality Metrics

For each segment in each `videoID.tar`, compute and store:

**Audio Quality Metrics (fast, essential):**
| Metric | Description | Purpose |
|--------|-------------|---------|
| `duration_sec` | Segment length | Already in metadata |
| `loudness_lufs` | Integrated loudness | RMS loss validation |
| `loudness_rms_db` | RMS in dB | Volume normalization |
| `clipping_fraction` | % samples \|x\| > 0.99 | Filter damaged audio |
| `dc_offset` | Mean of waveform | Catch broken pipelines |
| `snr_proxy` | Speech vs noise floor | Quality weighting |

**True Sample Rate Detection (critical for Phase C):**
| Metric | Description | Purpose |
|--------|-------------|---------|
| `spectral_rolloff_hz` | Freq below 85% energy | Detect bandwidth |
| `effective_nyquist_hz` | Detected cutoff frequency | True sample rate |
| `high_freq_energy_ratio` | Energy >8kHz / total | Wideband detection |
| `native_sr_bucket` | `<16k`, `16-24k`, `24-32k`, `32-44k`, `>=44k` | Bandwidth classification |
| `is_truly_wideband` | Boolean: effective_nyquist >= 12kHz | Higher-SR eligibility |

**Preserve Existing Metrics:**
- `music_stats` (already computed via PANN)
- `speaker` label
- `start`, `end`, `original_start`, `original_end`, `duration`

### 1.3 Native SR Classification + Resample Outputs

For each usable segment:
1. Load the segment FLAC from the tar (already at original quality from YouTube WebM).
2. Run spectral detection (rolloff, effective_nyquist, HF ratio).
3. Assign buckets and resample targets based on **actual bandwidth**, not container SR:
   - **16k bucket**: all usable segments (universal). Long segments (>=12s) are split into 6–10s windows.
   - **24k bucket**: `effective_nyquist >= 8000` Hz (narrowband to low-wideband)
   - **32k bucket**: `effective_nyquist >= 12000` Hz (mid-wideband)
   - **44k bucket**: `effective_nyquist >= 16000` Hz (high-wideband, true 44.1k sources)
4. Record `resample_target_sr` for each emitted sample.
5. Wideband flag: `is_truly_wideband = effective_nyquist >= 12000` (relaxed from previous 14k + HF ratio gate)

### 1.3 Composite Quality Score

Derive `quality_score` (0-1) for weighted sampling (store, do not filter):

```python
quality_score = (
    0.3 * (1 - clipping_fraction) +
    0.2 * normalize(snr_proxy, min=10, max=40) +
    0.2 * (1 if not music_stats.has_music else 0.5) +
    0.2 * normalize(loudness_lufs, min=-40, max=-14) +
    0.1 * (1 if abs(dc_offset) < 0.01 else 0.5)
)
```

---

## 2) Sharding Strategy

### 2.1 Segment Handling Decision: No Stitching

**Problem**: ~58% of segments are <6s, but X-Codec requires 6s crops.

**Decision**: Do NOT stitch segments. Reasons:
1. Speaker diarization is not 100% reliable - stitching risks mixing speakers
2. W2V-BERT semantic encoder expects consistent speaker within sample
3. Padding is recoverable (efficiency loss), data corruption is not

**Implemented approach**:
- **Drop segments <3s** (too short, mostly padding anyway - ~27% of segments)
- **Duration-weighted sampling** to favor longer segments
- **Chunk long segments for 16k**: For segments >=12s, split into non-overlapping 6–10s windows to boost sample count (no stitching across speakers).
- **Silence-aware chunking**: Only cut when a low-energy boundary exists near the target window; otherwise keep the full segment.
- **No pre-cropping needed for training**: Random 6s crops still happen during training (X-Codec data_module.py:104)
- **VAD regions**: Store full duration `[[0, duration_sec]]` as placeholder; actual cropping is random at train time

### 2.2 Output Structure

```
processed_data/
├── manifests/
│   ├── telugu_train.jsonl
│   ├── telugu_val.jsonl
│   └── ... (per language)
├── telugu_16/           # 16kHz - universal bucket
│   ├── shard-000000.tar
│   ├── shard-000000.jsonl
│   └── ...
├── telugu_24/           # 24kHz - effective_nyquist >= 8kHz
│   ├── shard-000000.tar
│   ├── shard-000000.jsonl
│   └── ...
├── telugu_32/           # 32kHz - effective_nyquist >= 12kHz
│   ├── shard-000000.tar
│   ├── shard-000000.jsonl
│   └── ...
├── telugu_44/           # 44.1kHz - effective_nyquist >= 16kHz
│   ├── shard-000000.tar
│   ├── shard-000000.jsonl
│   └── ...
└── stats/
    ├── telugu_stats.json
    └── ...
```

Note: Directory naming is `{language}_{sr_khz}` for easier R2 prefix management.

### 2.3 Shard Contents

Each shard (~5GB, multipart upload friendly) contains audio only, plus a single
JSONL sidecar file with per-sample metadata:

```
shard-000000.tar
shard-000000.jsonl
```

`shard-000000.jsonl` is one line per sample with:
```json
{
  "id": "videoID__SPEAKER_00_0001",
  "video_id": "---g_VL7ySo",
  "language": "bengali",
  "duration_sec": 4.11,
  "sample_rate": 16000,
  "resample_target_sr": 16000,
  "native_sr": 48000,
  "effective_nyquist_hz": 24000.0,
  "is_truly_wideband": true,
  "native_sr_bucket": ">=44k",
  "loudness_lufs": -23.5,
  "loudness_rms_db": -18.2,
  "clipping_fraction": 0.0,
  "snr_proxy": 28.5,
  "quality_score": 0.85,
  "music_prob": 0.045,
  "speaker": "SPEAKER_00",
  "vad_speech_regions": [[0.0, 4.11]],
  "duration_bucket": "3-6s",
  "sampling_weight": 0.3
}
```

Metadata storage:
- Keep per-shard JSONL locally for now.
- Also write per-language JSONL files for easy Supabase upsert later.

### 2.4 Filtering Rules (applied during sharding)

| Rule | Action | Rationale |
|------|--------|-----------|
| `duration_sec < 3.0` | **Drop** | Too short, mostly padding |
| `duration_sec > 60.0` | **Drop** | Too long, memory issues |
| `clipping_fraction > 0.1` | **Drop** | Severely clipped audio |
| `music_stats.has_music = true` | **Keep, lower weight** | May still have speech |
| `effective_nyquist < 8000` | **16k only** | Narrowband audio |
| `effective_nyquist < 12000` | **16k, 24k only** | Low-wideband audio |
| `effective_nyquist < 16000` | **Exclude 44k** | Not high-wideband |

### 2.5 Duration Bucket Weights

| Bucket | Duration | Weight | Rationale |
|--------|----------|--------|-----------|
| A | 3-6s | 0.3 | Padding overhead, keep for robustness |
| B | 6-12s | 1.0 | Optimal for 6s random crops |
| C | 12-24s | 0.8 | Good, multiple crops per segment |
| D | 24-60s | 0.5 | Memory intensive |

### 2.6 Train/Val Split

- Split by `video_id` (not segment) to avoid data leakage
- 99% train / 1% val per language
- Ensure val set covers all duration buckets
- Target: ~1000+ hours validation per major language

---

## 3) Training Strategy

### Phase A: 16kHz Fine-tune (Indic Adaptation)

**Goal**: Adapt semantics + acoustic reconstruction to Indic languages without destabilizing codebook.

**Data**: All languages from `shards_16k/`, weighted sampling by duration + quality

**Initialization**:
1. Load pretrained X-Codec-2.0 checkpoint (HKUST 16kHz)
2. Keep W2V-BERT semantic encoder frozen (default)
3. **First 5k steps**: Also freeze encoder + VQ codebook (stabilize)
4. **After 5k steps**: Unfreeze encoder, keep VQ learning rate at 0.1x

**Training Config**:
```yaml
batch_size: 8  # per GPU
min_audio_length: 96000  # 6s @ 16kHz
precision: 16-mixed
losses:
  - mel (lambda: 15.0)
  - adversarial (lambda: 1.0)
  - feature_matching (lambda: 1.0)
  - vq_commitment
  - semantic_reconstruction (lambda: 5.0)
  - rms (lambda: 1.0)  # ADD THIS from Inworld
```

**Curriculum (progressive scaling)**:
| Stage | Data | Steps | Notes |
|-------|------|-------|-------|
| A1 | 5k hrs, quality_score > 0.7 | 50k | Clean subset, stability check |
| A2 | 15k hrs, quality_score > 0.5 | 100k | Expand coverage |
| A3 | Full dataset, weighted sampling | 200k+ | All languages |

**Validation Metrics (per language)**:
- Mel L1 loss
- Multi-resolution STFT loss
- Speaker cosine similarity (WavLM, already in X-Codec)
- Codebook utilization (detect collapse: should use >90% of codes)
- SI-SDR (scale-invariant SDR)

### Phase B: Stabilize and Vectorize

**Goal**: Lock encoder, precompute codes for decoder-only training.

1. Take best Phase A checkpoint
2. Freeze encoder + VQ completely
3. Run short stabilization (10k steps) with frozen encoder
4. Use Inworld `data_vectorizer.py` to generate:
   - `train_codes.npy`, `train_codes_index.npy`, `train_samples.jsonl`
   - Per language, per resolution

**Vectorizer Modifications Needed (later, if/when used)**:
- Remove `filter_non_english` from `tts/tts/data/filtering.py`
- Add Indic language codes to `allowed_languages`

### Phase C1: 24kHz Decoder (Optional Low-bandwidth)

**Goal**: Handle low-bandwidth audio efficiently.

**Data**: `{lang}_24/` buckets (`effective_nyquist >= 8kHz`)

**Config** (token rate 50):
```python
sample_rate = 24000
hop_length = 240
upsample_factors = [2]
```

### Phase C2: 32kHz Decoder

**Goal**: Mid-range quality decoder for majority of data.

**Data**: `{lang}_32/` buckets (`effective_nyquist >= 12kHz`)

**Config** (token rate 50):
```python
sample_rate = 32000
hop_length = 320
upsample_factors = [2]
```

### Phase C3: 44.1kHz High-Fidelity Decoder

**Goal**: High-fidelity output for true wideband sources.

**Data**: `{lang}_44/` buckets (`effective_nyquist >= 16kHz`)

**Architecture**:
```python
sample_rate = 44100
hop_length = 147
upsample_factors = [3, 2]  # or [6] for single stage
# Verify: 44100 // 147 // 6 ≈ 50 tokens/sec
```

**Training Notes**:
- Encoder frozen (use codes from Phase B)
- Decoder backbone trainable, quantizer frozen
- Progressive training: start with 32k decoder, then fine-tune to 44k
- `audio_window_size` adjusted per sample rate

**Validation Metrics**:
- DNSMOS (perceptual quality)
- High-frequency energy preservation (effective_nyquist comparison)
- Spectral rolloff comparison (input vs output)
- Speaker similarity (WavLM embeddings)
- Mel spectrogram L1 distance

---

## 4) Implementation Plan

### Priority Order

| Step | Task | Output |
|------|------|--------|
| 1 | Fetch languages from Supabase | `language_lookup.json` |
| 2 | Compute quality metrics per segment | Updated metadata |
| 3 | Classify native SR + resample to 16/32/48 | Bucketed audio |
| 4 | Create sharding pipeline | WebDataset tars + JSONL |
| 4 | Generate train/val manifests | JSONL files |
| 5 | Upload to R2 | Organized buckets |

### Files to Create

| File | Purpose |
|------|---------|
| `scripts/fetch_languages.py` | Query Supabase for videoID → language |
| `scripts/compute_metrics.py` | Compute audio quality metrics per segment |
| `scripts/resample_and_bucket.py` | Slice from originals + emit 16/32/48 |
| `scripts/create_shards.py` | Pack segments into WebDataset tars + JSONL |
| `scripts/generate_manifests.py` | Create train/val JSONL files |

### Files to Modify (later, for training)

| File | Change |
|------|--------|
| `tts/tts/data/filtering.py` | Remove `filter_non_english` (if vectorizer is used) |
| `X-Codec-2.0/lightning_module.py` | Add RMS loss |

---

## 5) Validation Checkpoints

Before proceeding to training, verify:

- [x] All videoIDs have language assigned (from Supabase)
- [x] Quality metrics computed for all kept segments (loudness, clipping, SNR, spectral)
- [x] Shard sizes ~5GB and JSONL sidecars present
- [x] Train/val splits created per language
- [ ] Larger smoke test completed (50-100 videos per language)
- [ ] Bandwidth distribution validated across languages
- [x] Segments with `clipping_fraction > 0.1` dropped during filtering
- [ ] Upload to R2 verified (download and re-check integrity)
- [ ] At least 100 hours validation data per major language

---

## 6) Language Distribution (from Supabase)

| Language | Videos | Usable Hours | % of Total |
|----------|--------|--------------|------------|
| Telugu | 71,619 | 19,712 | 13.2% |
| Malayalam | 65,273 | 18,184 | 12.2% |
| English | 59,657 | 17,904 | 12.0% |
| Hindi | 58,824 | 17,710 | 11.9% |
| Punjabi | 55,761 | 18,065 | 12.1% |
| Tamil | 50,199 | 11,258 | 7.6% |
| Kannada | 45,689 | 13,578 | 9.1% |
| Gujarati | 40,950 | 14,759 | 9.9% |
| Bengali | 20,121 | 6,206 | 4.2% |
| Odia | 19,280 | 5,010 | 3.4% |
| Marathi | 16,221 | 5,130 | 3.4% |
| Assamese | 3,793 | 1,370 | 0.9% |
| **Total** | **507,387** | **148,886** | **100%** |

---

## 7) Appendix: Observed Metadata Schema

### Top-level fields in `metadata.json`:
- `video_id`, `video_title`, `youtube_url`
- `original_duration`, `intro_skipped`, `processed_duration`
- `sample_rate` (16000), `original_sample_rate` (48000), `audio_native_sample_rate`
- `spectral_quality`: `{claimed_sample_rate, effective_sample_rate, detected_nyquist, rolloff_frequency, is_upsampled, original_format, confidence, is_suitable_for_tts, high_freq_energy_ratio, spectral_flatness}`
- `num_speakers`, `speakers`, `total_segments`
- `quality_stats`, `vad_stats`, `config`, `compute_stats`, `timing`, `download_meta`

### Per-segment fields:
- `start`, `end`, `duration`, `original_start`, `original_end`
- `speaker` (SPEAKER_00 / SPEAKER_01 / OVERLAP / NON_SPEECH)
- `status` (usable/unusable)
- `unusable_reason` (overlap, non_speech, overlap_proximity)
- `music_stats`: `{music_mean, music_max, music_ratio, noise_mean, noise_max, has_music, has_noise, decision}`

### Fields Added to Shard JSONL (per sample):
All metrics are computed during sharding and stored in per-shard JSONL sidecars:
- `language` (from Supabase lookup)
- `segment_index`, `chunk_index` (chunking metadata)
- `start`, `end`, `original_start`, `original_end` (time boundaries)
- `loudness_lufs`, `loudness_rms_db` (audio level metrics)
- `clipping_fraction`, `dc_offset`, `snr_proxy` (quality metrics)
- `native_sr`, `effective_nyquist_hz`, `spectral_rolloff_hz`, `high_freq_energy_ratio` (spectral)
- `native_sr_bucket` (bandwidth classification: <16k, 16-24k, 24-32k, 32-44k, >=44k)
- `is_truly_wideband` (boolean: effective_nyquist >= 12kHz)
- `resample_target_sr` (16000, 24000, 32000, or 44100)
- `quality_score`, `duration_bucket`, `sampling_weight` (training weights)
- `vad_speech_regions` (full duration placeholder: `[[0.0, duration_sec]]`)

**Storage**: Metrics are NOT added to original metadata.json; they're stored in shard JSONL files alongside audio.


Now let me examine the main processing scripts: