The post-pass is running over the already-cleaned parquet and then pruning `final_data`. I’m checking whether it has finished rewriting the final files and the reduced folder contents.

**Inspecting the terminal state**

I need to continue from the previous command and provide an update on the commentary. First off, I'll check the terminal file to see if the process is still running or if it has finished. I have to keep in mind that unexpected changes could have occurred, especially since we made modifications ourselves. So, let's read the terminal file carefully to get the current state.


The post-pass is still in flight because it’s rewriting the large cleaned segment parquet without `[NO_SPEECH]`. Once that finishes, the cleanup step will remove the intermediate files in one go.

**Continuing polling for process**

I need to keep polling until the process is complete. It seems like it might take some time, especially since I'm writing a temporary postpass that’s 5.2G. It's possible that the process is still active, so I think I should wait for maybe 120 seconds before I inspect it. This way, I can ensure everything is going smoothly. Patience is key here!


The post-pass is done.

I updated `scripts/build_final_cleaned_dataset.py` so future runs also drop `[NO_SPEECH]`, then I rewrote the final cleaned outputs and pruned `final_data/` to just the finalized artifacts plus your unrelated prompt JSONs.

## Final State

The `[INAUDIBLE]` / `[NO_SPEECH]` cleanup removed more than the single visibly-found row because there were additional `[NO_SPEECH]` rows in the final cleaned parquet:

- removed `[NO_SPEECH]` segments: `50,634`
- affected videos: `19,951`

Updated final cleaned dataset:
- videos: `395,513`
- total segments: `60,716,547`
- validated: `60,390,887`
- missing: `325,660`
- coverage: `99.463639%`

Important cleanliness metrics now:
- remaining foreign segments: `0`
- remaining `[UNK]` segments: `0`
- remaining `[INAUDIBLE]` segments: `0`
- remaining `[NO_SPEECH]` segments: `0`
- remaining `ta -> te` residual segments: `0`

Other useful metrics:
- golden: `17,486,648`
- redo: `9,392,877`
- dispose: `33,511,362`
- remaining target-language mismatch segments: `6,077,309`
- remaining target-language mismatch pairs: `132`

## `final_data` Cleanup

`final_data/` now contains only:

- `final_data/final_cleaned_segments.parquet`
- `final_data/final_cleaned_video_rollup.parquet`
- `final_data/final_cleaned_keep_videos.csv`
- `final_data/final_cleaned_excluded_videos.csv`
- `final_data/final_cleaned_remaining_target_mismatch_pairs.csv`
- `final_data/final_cleaned_summary.json`

Plus the unrelated prompt experiment files you already had:
- `final_data/transcript_variant_prompt_100x5.json`
- `final_data/transcript_variant_prompt_multikey_smoke.json`
- `final_data/transcript_variant_prompt_postfix_smoke.json`
- `final_data/transcript_variant_prompt_reduced_20x2.json`
- `final_data/transcript_variant_prompt_reduced_smoke.json`
- `final_data/transcript_variant_prompt_smoke.json`

If you want the folder reduced even further, I can remove those prompt JSONs too, but I left them alone because they look unrelated to the dataset finalization work.


<user_query>
which is the final videos CSV.
</user_query>


The final kept-video CSV is `final_data/final_cleaned_keep_videos.csv`.

If you want the excluded side too, that’s `final_data/final_cleaned_excluded_videos.csv`.


<user_query>
can you explain the columns. simplified.
</user_query>


For `final_data/final_cleaned_keep_videos.csv`, the columns are:

- `video_id`: the YouTube video ID.
- `recommended_action`: the old classifier label from your earlier selection step, usually `keep` or `review`.
- `queue_language`: the original language tag from your queue/input pipeline.
- `youtube_audio_language`: YouTube’s detected audio language, if available.
- `youtube_default_language`: YouTube’s default language metadata.
- `dominant_gemini_language`: the language Gemini saw most often across that video’s segments.
- `corrected_language`: the final video language you should trust for this cleaned set.
  - Usually this is `queue_language`
  - but if YouTube audio language is one of your target 12 and matches dominant Gemini, it gets corrected to that instead
- `total_segments`: how many segments from that video survived the video-level filtering stage.
- `distinct_detected_languages`: how many different Gemini languages appeared in that video before segment-level cleanup.
- `foreign_segments`: how many segments in that video were outside your target 12.
- `foreign_share_pct`: percentage of segments in that video that were foreign.

### Simplest way to think about it

The most important columns are:

- `video_id`: which video
- `corrected_language`: what language bucket the video finally belongs to
- `total_segments`: how much usable material it has
- `foreign_share_pct`: how mixed/dirty it was before segment cleanup

If you also want, I can explain the columns in `final_data/final_cleaned_excluded_videos.csv` too, since that one has the boolean reason flags for why a video got dropped.

**Simplifying CSV column explanations**

I need to provide a simplified explanation of the columns in the final_cleaned_keep_videos.csv file. I'll inspect the headers and recall the key columns, such as video_id and recommended_action. I should also consider the excluded CSV, which has similar columns plus boolean reasons. Since the user asked for a concise explanation, I can present the information in bullet points. I might clarify the difference between corrected_language and dominant_gemini_language too. Keeping it straightforward is important!