<user_query>
## Status
I resumed the run after the balance issue by launching a small finisher fleet, and Stage B is now complete.

Exact completion state:
- `876,864 / 876,864` microshards are `compacted`
- final shard rows: `60,711,097`
- final shard count: `4,350`

## Exact Tallies
DB final-shard totals matched Stage A microshard totals exactly for all 12 languages:

- `en`: `17,408,218`
- `hi`: `9,943,819`
- `te`: `8,228,362`
- `ml`: `5,160,601`
- `pa`: `4,821,133`
- `ta`: `4,214,380`
- `kn`: `2,781,057`
- `gu`: `2,564,959`
- `bn`: `2,180,872`
- `mr`: `2,042,838`
- `or`: `946,539`
- `as`: `418,319`

Canonical local total is still `60,711,097` rows across `395,513` videos, and the per-language canonical totals from `gemini_lang` matched the Stage A microshard totals exactly before this final Stage B reconciliation.

## Validation Done
I ran a real post-run audit, not estimates:

- DB reconciliation: `60,711,097 expected` vs `60,711,097 actual`
- Language reconciliation: `0` mismatched languages
- R2 object audit:
  - expected shard objects from DB: `17,400`
  - missing referenced objects: `0`
  - size mismatches vs DB byte counts: `0`
  - there was `1` orphan extra object in R2, and I deleted it
- Sampled shard integrity checks:
  - sampled manifests matched DB segment counts
  - sampled `metadata.parquet` and `audio_index.parquet` row counts aligned
  - sampled segment ordering/alignment was correct
- Sampled row-level canonical checks:
  - initial mismatches were confirmed to be expected Stage A enrichments (`corrected_language` vs `queue_language`, and English native fallback)
  - adjusted apples-to-apples sampled mismatch count: `0`

## Remaining Action Items
Two cleanup items are still open:

- I still owe one direct `COUNT(DISTINCT video_id)` over the final shard metadata itself, so I can state the final-output video count from Stage B artifacts directly, not just from canonical/Stage A.
- I still need to destroy the remaining compactor instances.

I did not mark cleanup complete because local command execution started failing transiently right after the final audit, and I’m not going to pretend those two steps are done when they aren’t. The important part is that the compaction itself is finished and the exact segment/R2 validation is clean.

I want you to verify that no regressions, no errors, no bugs were introduced in this export entire StageA and StageB.
</user_query>


I’m treating this as a real audit, not just a summary check: first I’m mapping the Stage A and Stage B code paths plus the recent export-related diffs, then I’ll run targeted tests and look for concrete failure modes or regressions.

**Verifying exports for issues**

I need to ensure the user’s request to verify that there are no regressions, errors, or bugs in the entire export of Stage A and Stage B is addressed. This feels similar to conducting a code review or audit. I might need to inspect repository changes, run tests, and focus on identifying any potential bugs or risks. It seems essential to understand the context through files and git changes. Given the broad nature of this task, I should also consider a plan for the verification process.
**Investigating user requests**

I might need to switch to a more organized plan since the task seems large, and there are tradeoffs involved. The user has asked me to verify details, so using the current approach seems fine for now, but I do need to investigate further. It's essential to start with a commentary update and check for a virtual environment (venv) before running any application commands. I could use shell commands to check if venv exists and activate it. It's also worth looking for the root README or progress files, as per user instructions, and possibly explore additional tools.
**Exploring code for verification**

I’m considering launching multiple explore agents in parallel to inspect the code for Stage A and Stage B, which aligns with the user's request to verify that there are no regressions, errors, or bugs. Using subagents seems like a good idea to review both stages separately while also running our tests. I should include detailed prompts and may need to check relevant files along with git diffs and logs. It’s also worth validating changed files using tools like rg/Glob/ReadFile for efficiency.


I’ve confirmed the export code is mostly new, not incremental edits, so I’m reviewing it as a fresh pipeline surface: next I’m splitting the audit into Stage A variant generation and Stage B final export/compaction, while also checking what automated coverage actually exists.


The first-pass review already turned up likely correctness bugs in both stages, so I’m now validating those against the actual code and pairing that with runtime checks: targeted tests, import/compile sanity, and a closer read of the queue/idempotency paths.

**Evaluating testing setup**

I need to check the existing terminals for any long-running processes because running pytest shouldn't take too long. Since the user wants the virtual environment (venv) active, I can run `source venv/bin/activate && pytest ...` or even use `python -m pytest`. I might want to consider running stage A/B tests, but I can't access the database right now. It might also help to run static tests or compile new modules to check for any syntax errors. I think I'll compile all relevant new scripts or modules using the virtual environment.
**Considering validation and inspection**

I could create a targeted compile list and maybe run a few validation scripts with `--help` to see if they parse correctly, although that might not be necessary. I really need to inspect the exact code for any flagged issues, so I'll use tools like `rg` and ReadFile. The plan is to search for relevant symbols in the files to identify potential problems. That should help me understand what I need to address!


**Investigating file handling**

I think I need to read the actual files, focusing on line numbers related to my findings. Using ReadFile with appropriate offsets might be key. Additionally, I'm considering inspecting the init_transcript_variant_queue to check for relevant symbols. It might take some time to dig into the details, but I'm determined to get it right. Let's look into it step by step and ensure I’m thorough in my approach!


I’ve finished the runtime sanity layer: the small unit suite passes and all new export files compile, but that does not clear the pipeline. The real issues are in untested queue/idempotency and validation behavior, so I’m pulling exact code references for those now.

**Searching database functions**

I'm thinking I might need to focus more on final_export_db, particularly around insert_microshard and commit. Perhaps I should use rg or check the later sections for relevant quotes. It might also be valuable to look into deploy/run a bit more. I feel I should really inspect the variant_db, especially the insert_pack_manifest and complete_job lines. I could check the lower part and the final_export_db commit method as well. Let's read those targeted sections!


One more thing before I close this out: I’m checking whether either pipeline has stale-claim recovery or duplicate-detection paths elsewhere in the codebase, because that determines whether some of these are real bugs or merely local-looking false alarms.


I’m looking at the final export and noticing that reset_stale_claims was missed by the subagent. I need to inspect the lower section of final_export_db and how the final_export_worker is being called. This could weaken one finding about the compactor wedge. I’ll also remember that stale_claims exists specifically for final_export, not for the variant. So, I need to read the relevant lines carefully to understand what's going on!
**Inspecting reset lines**

I need to take a closer look at the `final_export_db` and the `reset_stale_claims` lines. There might be a variant where a stale reset isn't needed. I'll also check the `compactor` main — does it also call `reset_stale_claims`? I should do a search for `final_export_worker`, but not just the compactor. Maybe it has something to do with the class initialization? I guess it's time to read around the start.


**Inspecting compactor starts**

I need to check how the compactor starts; it seems the `reset_stale_claims` function could be used in its own start. I'm considering if I should also use `rg`, but I'm not quite sure how they interact yet. It’s essential to clarify how these components work together to ensure everything runs smoothly. I guess I'll have to gather more information before proceeding with any changes.


**Investigating stale claims**

I'm noticing something interesting: the video worker is the only component that calls for stale resets, while the compactor doesn’t. That means if only the compactor fleet is running after the video stage, stale claims could linger unless the video worker triggers a reset. The user resumed the compactor, so it's possible that a stale claim reset wasn't executed. I should check the start of the final_export_compactor for clarity on this.


**Clarifying compactor behavior**

The compactor doesn’t reset stale claims but releases them when a worker starts. That could be a risk, but it’s not necessarily related to the wedge. The subagent's claim waits for a stale-claim reset, although it may be called by video workers instead. In stage B, the compactor could get stuck until a manual or video worker restart happens. I need to be precise, so I might inspect the reference store lines around the manifest to check for validation. I should also search for manifest usage.


**Considering test issues**

I need to inspect if there’s no manifest use and think about running tests to check for bugs. I already have several findings. I should consider the severity of these issues and potentially find more. The user asked for the entire StageA and StageB export. However, there’s a failing unrelated test that shows the repo isn’t clean. It might be better to mention this as a test gap, highlighting that due to the test import failure, I can't claim the repo is green.
**Inspecting code for bugs**

The user requested the "entire Stage A and Stage B," which might not need unrelated modules. I should verify some specific code related to the subagent findings to ensure they’re accurate.

For Stage A, there are several identified bugs:

1) A bug in `_run_single_batch` where errors are being counted incorrectly.
2) The `_validate_item` function is mismanaging processing routes despite errors.
3) No reset logic in the Variant DB for stale claims.
4) Issues with rerunning idempotency that could lead to overwriting pack objects without updating the manifest.
**Analyzing pack manifest behavior**

It turns out that using the `insert_pack_manifest DO NOTHING` means the old manifest row stays even if I upload a new pack with the same ID, possibly with a different byte size or row count. Since the `output_key` is deterministically identical for the same `pack_id`, the old object remains at the same key. If the `output_prefix` changes, the `pack_id` might stay the same, but the `output_key` would differ. So, under `DO NOTHING`, the old manifest from the previous prefix persists.


The code audit is enough to say the pipeline isn’t bug-free, but I can tighten the answer for your actual run by checking whether those failure modes left fingerprints in the local artifacts: duplicate canonical keys, carried `variant_validation_errors`, or obviously missing variant payloads.

**Evaluating validation and duplicates**

I'm considering the need for final export validation to detect duplicates. There might be a concern with compactor crashes being non-idempotent. I have to think about how actual duplicates occur when `insert_final_shard` implements a 'DO NOTHING' operation on unique random shard IDs. If a crash happens after an upload, the selected items might be removed from the local buffer, but the old shard could still exist, leading to confirmed duplicates. I should prioritize this and prepare findings for the user, ensuring I document any risks thoroughly.
**Detailing findings and references**

I'm thinking of including the top 6-8 findings across both stages. I'll start with a section detailing "What I verified" followed by "Residual risks/testing." Additionally, I need to mention that unit tests passed and compiled, but the full pytest was blocked due to an unrelated import error. I should provide code references using cited blocks, likely referencing files like `src/variant_worker.py` and `src/final_export_worker.py`. Special requests indicate I can include precise function references and maybe a few code snippets, though precise references are essential in my findings.
**Organizing findings with references**

I'm thinking of using an example from `src/variant_worker.py` where `TranscriptVariantWorker._run_single_batch()` returns `gemini_error` rows for batch exceptions, while `TranscriptVariantWorker._process_job()` counts those rows towards `rows_processed` before calling `VariantPostgresDB.complete_job()` in `src/variant_db.py`. I should include a grouping for Stage A and Stage B findings, most likely ordered by severity across both stages, as the user requested a complete export for both. I could end up with around 7 findings to present.
**Clarifying potential bugs**

I'm considering how the script likely uses the same configuration. If `require_variants` is true and there are missing variants, it could lead to mismatches. This bug only surfaces if reference data has missing variants; otherwise, it doesn't affect the current run. We need to be careful about how we frame it: while a bug could cause a regression on a rerun, we don't have evidence that it specifically affected the user's dataset. Their data audit implies otherwise.
**Evaluating bugs in the export**

I need to clarify that I can't certify "no bugs introduced." While I found several correctness bugs in the code, it seems your run-level reconciliations lessen the chances of them showing up in the finished export. We should consider a separate assessment for run impact. It's interesting to note that the Stage B duplicate final shards may be linked to a compactor crash. The exact match of expected vs. actual row counts suggests no such crash occurred, or duplicates were balanced by missing rows.
**Analyzing potential duplicates and bugs**

I’m thinking that duplicates would increase the total row count unless an equal amount is missing, which seems possible but unlikely. Since the actual count matches the expected count, the specific run likely wasn't affected by duplicates from a compactor crash, though the code bug remains.

I suspect the run-scoping bug might not affect this run if only a unique output prefix was used. The Stage A gemini_error rows raise some questions since the user mentioned that validations matched, but did not clarify if any gemini_error rows exist.
**Evaluating validation errors**

The test harness showed no validation errors, but I noticed the production path might be lacking some protection. The actual run could still be fine, though.

I wonder about the Stage A invalid gemini rows that might be there again. Any analysis might look at `variant_validation_errors` counts. I could also consider using DuckDB or Python to inspect the final dataset or variant outputs, which might help verify if there are any bugs introduced that the user is concerned about. Querying the local folder seems like a solid plan!
**Inspecting final data**

I’ve got the local canonical dataset called final_data, but I’m not sure if it includes the final shard outputs. I think it would be helpful to look through this cleaned dataset for any variant fields, validation errors, or duplicate segment IDs. It’s possible that final_data might also hold some reference data or outputs.

I should list the files, so I can use Glob to find what’s in final_data. Since this command relates to the app, I don’t think I need a virtual environment just for listing files. Let's dig into the final_data files!


I found the canonical Stage A parquet locally, so I’m using that to separate “code has a bug” from “this exact run shows damage.” That should let me tell you whether the risky paths appear to have left obvious bad rows in the dataset you reconciled.

**Running validation queries**

I'm planning to run DuckDB queries against local parquet files to check for variant validation errors or empty variants in the final dataset, especially since the user provided the canonical total. I need to consider potential duplicate keys too. I should inspect the schema to figure out column names for my queries.

I'll check totals for rows, distinct counts on video IDs, and look for rows with empty values in specific text fields. I'll also check for any variant validation errors and other relevant criteria. This will help ensure accuracy in the final cleaned segments.
**Considering Stage B outputs**

I'm thinking about inspecting the outputs from Stage B, although they're not local, just referring to `final_data`. It looks like we might be missing direct counts for any pending actions in the Stage B output artifacts. Since the user already ran audits, I could potentially query the `final_export_reference_data` or a manifest, but I’m not entirely sure about that. Either way, I’ll proceed with running some DuckDB queries to gather the necessary information.


I found a stronger run-specific signal: the canonical parquet that fed Stage B has no duplicate `(video_id, segment_file)` keys and no surviving `gemini_error` rows, but it does contain a large number of rows with non-empty `variant_validation_errors`. I’m breaking those down now, because that’s the clearest evidence of whether bad Stage A variants were actually shipped forward.

**Assessing data quality issues**

This is a concrete run issue and not just about code risk, which is very important. I need to quantify approximately 615,995 invalid variant rows in the exact canonical parquet used for Stage B. That's a significant number! I should note that these aren't necessarily fatal row-count mismatches but rather regressions in content quality. I might also want to query the distinct counts of error categories. Let’s move forward with that!


**Exploring columns and data types**

I need to consider the column names — maybe it's `corrected_language` or `gemini_lang`. I should check using `describe`. It seems like the cleaned segments might not have a segment language. Perhaps I could query the schema or utilize the language field, leaning towards `gemini_lang`, but not absolutely. I might want to use `corrected_language` instead. Rerunning to count by `corrected_language` seems like the best approach.


## Findings
- `High`: Stage A is shipping rows that its own validator already marked invalid. In `src/variant_worker.py`, `TranscriptVariantWorker._validate_item()` records problems, but `_run_single_batch()` still writes those rows as normal `processing_route="gemini"` outputs, and Stage B does not gate on `variant_validation_errors` in `src/final_export_worker.py`. In the actual canonical parquet `final_data/final_cleaned_segments_with_variants_rerouted_repetition_filtered.parquet`, I measured `615,995` rows with non-empty `variant_validation_errors` across `111,642` videos. The biggest buckets are `missing_protected_roman=416,395`, `native_missing_target_script=105,049`, and `roman_not_ascii=94,949`. That means the finished export is count-consistent, but not content-clean.

- `Critical`: Stage B is not safely run-scoped. `src/final_export_db.py` keys `final_export_queue` and `final_export_video_outputs` by `video_id` and `final_export_microshards` by `microshard_id`; `src/final_export_worker.py` builds `microshard_id` as `{video_id}_{language}_{chunk_index}`; and `scripts/deploy_final_export.sh` defaults `FINAL_EXPORT_OUTPUT_PREFIX` to the fixed `final-export/production`. A rerun or concurrent run can overwrite or mix DB state and R2 objects from another run even when `FINAL_EXPORT_RUN_ID` changes.

- `High`: Stage B final-shard publication is non-atomic. In `src/final_export_compactor.py`, `FinalExportCompactor._flush_buffer()` uploads the shard and inserts its DB row before `FinalExportPostgresDB.commit_microshard_consumption()` in `src/final_export_db.py`. If a worker dies in that window, the final shard already exists but the source microshards are still eligible, so a retry can emit the same rows again into a new shard. Your exact row-count reconciliation makes it unlikely this specific finished run hit that path, but the bug is real.

- `High`: Stage A still treats exhausted Gemini failures as successful shard completion. In `src/variant_worker.py`, `_run_single_batch()` converts request failure into `gemini_error` rows and `_process_job()` still counts them as processed before `VariantPostgresDB.complete_job()` in `src/variant_db.py`. I found `47,438` `gemini_error` rows in `final_data/transcript_variants_merged.parquet`. They do not survive into the final canonical parquet, so later recovery/reroute appears to have cleaned them up, but the worker logic is still wrong.

- `Medium`: The shipped Stage B validator cannot prove absence of duplicate logical rows. `scripts/validate_final_export_run.py` loads actual rows into a dict keyed by `(video_id, segment_id)` inside `load_pack_records()`, so later duplicates overwrite earlier ones. A clean run of that validator is useful, but it is not a proof that no duplicate segments exist across packs or final shards.

- `Medium`: Stage A reruns are not idempotent. `scripts/init_transcript_variant_queue.py` reuses a fixed local shard directory and globs every `shard_idx=*` directory it finds; `src/variant_worker.py` uses deterministic pack ids; and `VariantPostgresDB.insert_pack_manifest()` in `src/variant_db.py` does `ON CONFLICT (pack_id) DO NOTHING`. Re-seeding can mix stale shard files, stale manifest rows, and fresh uploads.

- `Medium`: Stage A has no stale-claim recovery for claimed shard jobs. `VariantPostgresDB.claim_job()` in `src/variant_db.py` only claims `status='pending'`, and unlike Stage B there is no reset path in `src/variant_main.py` or the DB layer. A dead worker can strand work indefinitely.

- `Medium`: `src/transcript_variant_prompt.py` promises exact preservation of “file names and literal identifier-like tokens,” but `PROTECTED_SPAN_RE` does not actually match them. That is a prompt/validator contract gap, so the worker can accept outputs that violate its own written rules.

## What I Verified
- Export-focused tests pass: `tests/test_transcript_variant_prompt.py` and `tests/test_final_export_common.py` both passed (`13/13`).
- All new Stage A/Stage B Python modules and scripts compile with `py_compile`; I did not hit syntax/import errors in the export code itself.
- The repo-wide test suite is not green: `pytest` stops at `tests/test_video_tts_classifier.py` because it imports a missing `OpenRouterVideoTtsClassifier`. That looks unrelated to this export path, but it means I cannot claim a fully green repository.
- The final canonical parquet is structurally sane on its primary key surface: `60,711,097` rows and `60,711,097` distinct `(video_id, segment_file)` pairs, with `0` surviving `gemini_error` rows and `0` rows missing both variant fields.
- The strongest run-specific problem is those `615,995` validation-flagged variant rows still present in the exact canonical dataset that fed Stage B.

## Bottom Line
I cannot verify that “no regressions, no errors, no bugs” were introduced. The Stage B row-count and language reconciliations you ran are strong evidence that catastrophic loss/duplication did not occur in this finished run, but the export is not bug-free.

The clearest concrete issue in the actual shipped data is Stage A content correctness: hundreds of thousands of rows were carried forward even though the pipeline itself flagged their variant outputs as invalid. On top of that, Stage B still has real code-level bugs around rerun isolation and compactor atomicity, so I would not certify the pipeline as regression-free.

If you want, I can next do one of two things:
1. patch the concrete Stage A/Stage B bugs and add regression tests, or
2. run a focused forensic query plan for the `615,995` validation-flagged rows so we can estimate whether they are acceptable edge cases or require a repair pass.

**Inspecting artifact validation**

I’m considering if stage A’s final canonical might have duplicate keys while checking for missing variants. It’s good to think about whether we can run specific checks for stage B’s artifacts locally, though scripts or databases may not be accessible. The user has already provided audits. I’ll note that there should be no duplicate rows or errors in the canonical parquet, and row counts matching indicates a reduced likelihood of duplication bugs. But I still need to be thorough in checking for any hidden duplicates.
**Checking for duplicates in Stage B**

I’m looking at the final output count of distinct videos, which is still pending. With the adjusted row-level checks in Stage B, I'm unsure about the absence of logical duplicates without a direct count. Hmm, maybe I can run a local SQL query, but the final shards are on R2/DB, not locally. If the environment has DATABASE_URL in the .env, I could check the DB tables since the user wants verification. It might be best to connect through Python or asyncpg with the necessary credentials to confirm counts or verify duplicates in the final export.
**Verifying output data and errors**

I’m noticing that the final output data isn’t available in individual DB rows. There aren’t direct details for those rows. Maybe I could query the microshards table to check for duplicates in `segment_id_set_sha256`, but that seems insufficient. I could look into `scripts/audit_final_export_run.py`, although the user has already done some of that. It might also be useful to query distinct corrected languages or check for `validation_errors` in `transcript_variants_merged.parquet` to understand if any code bugs caused the 47k gemini_error, which were removed during recovery.
**Evaluating validation errors**

I'm thinking about how `build_export_segment_payload` passes canonical values directly. I could inspect a sample error row in canonical to see if invalid romanization is preserved, but that might not be necessary. I want to check the stage A route counts to confirm that all validation error rows are on `gemini`, indicating no filtering occurred. I also need to query if `variant_validation_errors` are strictly from known exceptions, as some may come from `native_missing_target_script` or `roman_not_ascii`, which aren’t acceptable.
**Evaluating Stage B validation**

I'm considering that while catastrophic row loss or duplication seems unlikely due to matching counts and no duplicates in canonical keys, I'm still not ready to certify Stage B as bug-free. The validator doesn't account for duplicates in the final shards, plus I lack direct counts from final shard metadata. I didn't query the live database due to local command issues, relying instead on prior audit numbers. I also didn't inspect final shard rows, so I may need to ask if the user wants fixes and explore potential next steps.
**Identifying validation errors**

I'm thinking about suggesting a one-off query to isolate validation error rows by language or stage. If needed, I can address the specific issues and add regression tests afterward, but I want to ensure findings are solid per review rules. I might keep the summary brief after those findings and order the severity of the issues:

1. Stage A shipped invalid variant rows.
2. Stage B not run-scoped.
3. Non-atomic publish/consume in Stage B.
4. Stage A failures marked as done.
5. Validator not accounting for duplicates.
6. Stage A stale claim.

I should also include specific file references in the bullets.
**Analyzing validation errors**

In the canonical parquet file `final_data/final_cleaned_segments_with_variants_rerouted_repetition_filtered.parquet`, I found 615,995 rows with non-empty `variant_validation_errors` across 111,642 videos. Specifically, there are issues with 416,395 `missing_protected_roman`, 105,049 `native_missing_target_script`, and 94,949 `roman_not_ascii`. So, even though the counts all reconcile, this indicates that the finished export isn't clean in terms of content, which is potentially a critical problem that needs addressing.
**Evaluating validation findings**

I’m recognizing a spec/validation mismatch, and I should mention tests in my findings. After some analysis, I did not find duplicate canonical keys – there are 60,711,097 rows which means they’re all distinct pairs of `(video_id, segment_file)`. I observed zero `gemini_error` and no missing rows in the final canonical parquet. While the export-focused tests passed, the full repo test suite isn’t green due to a failure that seems unrelated to Stage A/B. So, I can't confirm no regressions or bugs were introduced.
**Summarizing action items**

I think I need to mention the user's remaining action items, like running `COUNT(DISTINCT video_id)` and terminating compactor instances. I should also note that there are still pending elements affecting our final certainty. For residual risks, the final output `COUNT(DISTINCT video_id)` is still pending, and there remains a duplicate blind spot in the final shard validator. I didn’t query the live final shard metadata or the database since it wasn't part of the local audit, but we can consider that for the next steps.
**Structuring risk communication**

I’m thinking I could prefix my bullet points with labels like `Critical`, `High`, or `Medium` to clarify their importance. Also, it might be useful to include file references inline with backticks to make it clearer for the user. I really want to approach this crafting carefully to ensure everything is both informative and easy to follow. Keeping things well-structured will help convey the necessary information more effectively!


<user_query>
The clearest concrete issue in the actual shipped data is Stage A content correctness: hundreds of thousands of rows were carried forward even though the pipeline itself flagged their variant outputs as invalid

what did you mean by invalid. like do they have no rows or empty rows ? why were those 615k rows marked invalid.
</user_query>


I’m checking the exact error labels in the canonical parquet so I can answer precisely whether these were empty/missing rows or rows that failed specific content rules.

**Explaining code validation**

I need to address the user's question clearly, explaining the invalid meaning with evidence from the code and previous data queries. I might need to include additional queries for breaking down reasons with percentages and examples. We've already got counts by error categories and sample rows on hand. I'll be concise but clear. I’ll inspect the relevant code in `src/variant_worker.py` and `src/transcript_variant_prompt.py`, mentioning the specific validation criteria and overlapping categories since the total counts exceed 615k.
**Querying validation errors**

I think it’s important to query the top exact validation strings and error combinations. I could also check if the user wants samples, but I can provide an answer now. It might be helpful to send an intermediary update before proceeding with tool use. Since the user is asking for specifics, I’ll run a query for the top exact error combinations, counting rows with one versus multiple errors. I’ll use DuckDB on canonical parquet, and I might run a couple of queries for strings and JSON parsing.
**Querying validation examples**

I could consider using PyArrow for reading a subset of data and then scan with DuckDB to group by exact strings of variant validation errors. It might not be necessary to check if native and romanized equal the transcription. I should explain specific examples of missing protected roman due to numeral transliteration, like Gujarati numerals `૧૮`. The validator expects exact span preservation, but Gemini normalizes digits instead. I’ll query a few example rows from the canonical parquet to explain each error type clearly, limiting to 3 as the data is large.


`Invalid` here does **not** mean “missing row” or “empty output row.”

It means: the row exists, both variant columns are usually populated, but the Stage A worker’s own post-checks decided the Gemini output violated one or more content rules and wrote those violations into `variant_validation_errors`.

```302:321:src/variant_worker.py
result_map = {item["id"]: item for item in result.items}
output_rows: list[dict[str, Any]] = []
for row in batch_rows:
    actual = result_map.get(row["row_id"])
    if actual is None:
        output_rows.append(self._build_error_row(row, request_id, "missing_from_response"))
        continue
    validation_errors = self._validate_item(row, actual)
    output_rows.append(
        {
            **row,
            "processing_route": "gemini",
            "native_script_text": actual["native_script_text"],
            "romanized_text": actual["romanized_text"],
            "request_id": request_id,
            "request_input_tokens": result.token_usage.input_tokens,
            "request_output_tokens": result.token_usage.output_tokens,
            "request_cached_tokens": result.token_usage.cached_tokens,
            "request_cache_hit": result.token_usage.cache_hit,
            "validation_errors": json.dumps(validation_errors, ensure_ascii=False),
        }
    )
```

```421:446:src/variant_worker.py
def _validate_item(self, expected: dict[str, Any], actual: dict[str, Any]) -> list[str]:
    errors: list[str] = []
    if actual["id"] != expected["row_id"]:
        errors.append("id_mismatch")
    if not actual["native_script_text"].strip():
        errors.append("empty_native_script_text")
    if not actual["romanized_text"].strip():
        errors.append("empty_romanized_text")

    for span in extract_protected_spans(expected["text"]):
        if span not in actual["native_script_text"]:
            errors.append(f"missing_protected_native:{span}")
        if span not in actual["romanized_text"]:
            errors.append(f"missing_protected_roman:{span}")

    if not romanized_text_is_ascii(actual["romanized_text"]):
        errors.append("roman_not_ascii")

    target_script = get_target_script_block(expected["language_code"])
    native_counts = detect_script_counts(actual["native_script_text"])
    roman_counts = detect_script_counts(actual["romanized_text"])
    if target_script != "Latin" and native_counts.get(target_script, 0) == 0:
        errors.append("native_missing_target_script")
    if target_script != "Latin" and roman_counts.get(target_script, 0) > 0:
        errors.append("roman_contains_target_script")
    return errors
```

For your shipped canonical parquet:

- `0` rows were missing both variants.
- Among the `615,995` flagged rows, `0` had `empty_native_script_text`.
- Among the `615,995` flagged rows, `0` had `empty_romanized_text`.
- `0` surviving rows had `variant_route='gemini_error'`.

So the issue is mostly **rule violations**, not blank data.

The main reasons those `615,995` rows were marked invalid were:

- `missing_protected_roman`: `416,395`
- `native_missing_target_script`: `105,049`
- `roman_not_ascii`: `94,949`
- `missing_protected_native`: `55,127`
- `roman_contains_target_script`: `8,395`

These counts overlap, so they do not sum cleanly to `615,995`.

What those mean in practice:

- `missing_protected_roman`
  The prompt says protected spans should be preserved exactly, like numbers, URLs, emails, handles, etc. A common failure is the input containing native-script digits and the romanized output normalizing them to ASCII digits instead of preserving them byte-for-byte.
  Example from your data:
  - input transcription: `... ૧૮ લેક્ચરમાંથી ...`
  - romanized output: `... 18 lecturmathi ...`
  - validator flag: `["missing_protected_roman:૧૮"]`

- `native_missing_target_script`
  The row’s assigned target language expects one script block, but the produced `native_script_text` contains none of that script. This usually means the model emitted the wrong Indic script for that row, or the row’s language assignment and text were mismatched upstream.
  Example in your data: Kannada-looking input, but the produced native text is in Telugu script, so the validator says the expected target script is missing.

- `roman_not_ascii`
  The romanized output is supposed to be plain ASCII after protected spans are stripped. A common real-world trigger is symbols like `₹` surviving in the romanized field.
  Example:
  - romanized output: `... ₹45,000 ...`
  - validator flag: `["roman_not_ascii"]`

Important nuance: not every flagged row is equally bad.

- Some are strict-format violations, like native digits `૧૦૦` becoming ASCII `100` in romanized text. Semantically that may still be understandable.
- Some are more serious, like `native_missing_target_script`, because that suggests the “native” output may be in the wrong script entirely.

Also, one easy confusion point: the canonical parquet has about `17.1M` rows with empty `native_script_text`, but those are **not** this invalid set. Those are the intentional `local_skip_fully_roman` rows, where the worker skips Gemini and keeps only the romanized side because the input was already fully Roman.

If you want, I can next break the `615,995` into:
1. likely harmless strictness violations,
2. probably real bad rows,
3. exact counts by language and route.

**Considering output nuances**

I need to provide some nuance here: it seems the output didn’t include the expected script block based on the assigned language for the row. This doesn’t clearly indicate whether there was a mistake in the model scripting or if the row was mislabeled earlier. It’s a useful observation! Maybe I should query the counts of the exact top error combinations and explain any overlaps. Since I already have the top exact counts, I can dig into that.


<user_query>
1. missing protected roman  - what were in the original transcript version ? we should retain the same. Since we have the transcripts jsonl or shards are seperately, can we just replace or edit the content. Also, just wondering if we can replace them with regex. cause even in the native script, i dont want the numerics to be ૧૮ , i want it to be 18. even in native version. So, now the real issue is how do we change the native version. before that if we know what was in the original transcript, like the mixed version, we will get a different perspective to wheather change only the native ૧૮  to 18, or also the mixed version also has that, so should be changed.

2. native_missing_target_script -- 103,957 rows
These seems like a real issue. Can we fix these without touching the real audio shards. i just checked the structure and it seems we can edit the json files without touching audio tars. But the real question is even if we fix these 103957 rows with gemini again, and then patch update those rows with latest version ? is this feasible ?

3.  roman_not_ascii -- 93,410 rows
What's wrong: The romanized_text contains non-ASCII characters, usually currency symbols like ₹ that Gemini preserved instead of spelling out. -> i dont see anything wrong in this. infact that's the ideal behaviour is what i feel. glad gemini retained it. but does it hurt the ASR or TTS training. that is the intended way of content right ? anything else critical other than this '₹'?

</user_query>


I’m tracing the exact lineage of those bad rows now: checking what the original mixed transcript contained, whether the flagged cases are mostly native digits vs true corruption, and what would actually need to be rewritten if you patch Stage B metadata only.

**Investigating data solutions**

I need to respond to the user and investigate the code and data a bit. The questions involve original transcript versions and the feasibility of patching data. I should probably inspect the canonical parquet schema and the script that builds the final cleaned segments. It's essential to explain data lineage and feasibility, while checking the virtual environment before running app commands. I plan to inspect parquet schema and sample rows for missing data and look for variations.
**Clarifying transcription details**

I need to check if the final cleaned segments contain mixed versions of the original transcription. The user is curious whether the mixed version includes native digits or ASCII. I can query counts to compare the presence of native digits in different transcriptions. It's essential to see if the original transcription in Stage A comes from the relevant columns, such as `transcription` or `tagged`. I also need to inspect the script that creates the input for the init transcript variant queue. Can I patch JSON files without affecting the audio tar files?
**Assessing Stage B artifacts**

I need to inspect the structure of Stage B artifacts, which include metadata.parquet and audio.tar. The user mentioned JSON files, possibly manifests. From what I understand, Stage B artifacts consist of parquet sidecars and tar audio, and it seems they can patch metadata parquets without affecting the audio files. I should explore the feasibility of patching the Stage B output or the canonical dataset, focusing on how patching would only change metadata.parquet, manifest bytes, SHA counts, and database metadata keys.


I’ve confirmed the canonical export already carries the original mixed transcript in the `transcription` column alongside both variant columns, so we can compare and patch text without touching audio bytes. I’m reading the exact variant-source join now to see which parquet is the true Stage A source of record for a repair pass.

**Inspecting source references**

I need to take a closer look at the reference store load of `variants_source`. It's possible that it joins with `variants_source` created from `transcript_variants_clean` or something merged. I’ll focus on lines around 280 and beyond since that could help in determining what patch is needed. I’m curious about how these pieces connect. Let’s see what insights I can gather from that relevant chunk!


The data confirms your intuition on `missing_protected_roman`: almost all of those rows started with non-ASCII native digits in the original mixed transcript, and Gemini normalized them to ASCII in the romanized output. I’m checking the merge step next so I can tell you the safest patch point: Stage A source parquet, canonical parquet, or Stage B shard metadata only.

**Investigating user's questions**

I need to address the user's three questions, and I'm thinking I might need to inspect how `final_cleaned_segments_with_variants` is built from `final_cleaned_segments` and `variants`. It seems like I should search the scripts to see how they merge variants and if that outputs `final_cleaned_segments_with_variants`. Additionally, I want to know if patching the canonical version with stage B metadata is feasible. Time to find that relevant script!


I’ve got the policy implication now: if you want ASCII digits everywhere, that is not just a Stage A native-field tweak. The Stage B export writes the original mixed transcript as `transcription_mixed` from canonical `transcription`, so I’m pulling the exact payload mapping before I answer how broad the patch needs to be.

**Evaluating data patching process**

I need to take a closer look at the script build_gemini_refined_dataset to see if I should merge variants. The user's asking about the feasibility of a patch, so I need to understand the output structure for Stage B. I may want to check the row ID format in variants_source to see how to patch Stage A with data from transcript_variants_clean.parquet. If I'm patching a large batch, using existing data might be helpful. I'll need to confirm if the mixed version also requires changes.
**Inspecting Stage B training setup**

I realize that Stage B training directly uses `transcription_mixed` from `canonical_row['transcription']`, which currently includes mixed transcription with native digits. If we want to standardize to ASCII, changing just the native field won’t cut it. I’ll need to patch `transcription`, `native_script_text`, and potentially leave romanized as is. I should check the code within `build_export_segment_payload` for relevant lines and determine the correct column names for metadata fields. I need to inspect areas around the transcription fields for clarity.


I found a subtlety in the `native_missing_target_script` samples: some rows are flagged, but the final canonical language and the visible native script already seem to line up. I’m quantifying that now, because it may mean part of that 103k is stale validation metadata from an earlier language assignment rather than a real final-data error.

**Considering digits normalization**

I’m thinking that digits normalization is something that can usually be done effectively, possibly using Python’s `unicodedata.digit()` to map any Unicode decimal digits. I want to address the issue labeled as “native_missing_target_script,” which seems significant and seems quite doable without needing to modify the audio itself, since audio isn’t in use here. I notice there are two scopes to focus on: patching the canonical Stage A dataset and just regenerating Stage B metadata, leaving audio untouched.
**Considering database patching**

I'm thinking that if Stage B is complete, a direct patch of the 4,350 shard metadata.parquet might be feasible because only text fields have changed. I would need to update the manifest size and hashes. The audio index and audio.tar seem unchanged, so adjusting the segment_id_set may not be necessary. However, I'd have to rewrite both the metadata.parquet and manifest.json to reflect these changes. Maybe it’s easier to patch the canonical parquet and rerun Stage B, but compaction might be costly. Perhaps I can patch metadata.parquet for final shards offline.
**Evaluating pipeline consistency**

I'm considering pipeline consistency and thinking it might be useful to patch the canonical reference and possibly the microshards, if they're being retained. A direct Stage B patch is feasible without altering the audio. For the 103,957 rows, I could rerun Gemini using `transcription` and `corrected_language` to regenerate just the native or romanized outputs, merging by `(video_id, segment_file)`. I wonder if some issues with `native_missing_target_script` stem from incorrect `corrected_language`, particularly since some flagged examples have discrepancies between target scripts and outputs. I should inspect these carefully to clarify.
**Checking corrected_language issues**

I'm looking at the corrected_language sample, which is only showing `pa`. The output includes Gurmukhi, so I'm puzzled as to why it was flagged for native_missing_target_script. It might be because the validator used the Stage A language_code instead of the corrected_language. Maybe the canonical sample indeed has corrected_language as `pa`, but the input language during Stage A could have been different. I need to clarify these points to avoid confusion moving forward.
**Inspecting language mappings**

I’m considering checking rows that show corrected_language, gemini_lang, and queue_language. It might help to inspect counts of native_missing_target_script to see where corrected_language actually appears in the native text. I could compute approximate counts using the current corrected_language mapping. Maybe I need to use DuckDB regex to check by script ranges. This could clarify whether those 103k entries are still issues or just stale false positives. Also, I should inspect how the Stage A input language_code is chosen during queue initialization.


I confirmed a big part of `native_missing_target_script` is probably stale metadata: `84,184 / 105,049` of those rows have `corrected_language != gemini_lang`, so many were likely validated against an earlier target language and then carried forward after rerouting. I’m doing one more exact count to separate stale flags from rows that are still wrong under the final corrected language.

**Exploring query adjustments**

I’m considering rerunning the first query with literal Unicode ranges. Using `regexp_matches` might help, but I also think checking counts against corrected!=gemini could suffice. I see that out of 105,000 flagged rows, there are 20,865 where corrected is gemini, assuming they're real. It might be easier to compute current-script matches using Python. I guess I can generate specific Unicode characters directly in Python, rather than handling it all in raw SQL. Let's get started!


I’ve separated the `roman_not_ascii` bucket now: it is not just `₹`. There’s a substantial harmless-symbol slice, but also a smaller real-consistency slice with target-script leakage and accented Latin, which matters if you want the Roman channel to stay strictly normalized.

**Evaluating patching feasibility**

I need to provide user insights on how to feasibly patch without affecting audio shards. It might be worth mentioning a metadata-only patch for Stage B. I'd want to quantify the required manifest and database updates and inspect `final_export_db` to check size and stored SHA. It's important to note that if I'm only patching the final shard's `metadata.parquet`, then I also need to update various attributes in the database while the audio components remain unchanged.
**Analyzing training data**

The data indicates there are 56,689 entries with rupee symbols, 2,706 with dollar signs, and others featuring π, accented Latin, and target script leakage. With 38,260 non-rupee rows, it's clear the dataset isn't just about rupees! I need to clarify which characters are harmless and which could impact ASR and TTS training. For TTS, symbols like ₹ and $ could be acceptable, but non-ASCII accents may complicate things if we want a clean Roman text channel.
**Proposing categorization for normalization**

I should clarify that if `romanized_text` is meant for plain ASCII normalization, then identifying `roman_not_ascii` due to the ₹ symbol alone isn't a semantic issue. However, script leakage and accented letters certainly weaken consistency. Given that there are examples of both target-script leakage and accented Latin, I might propose categorizing `roman_not_ascii` into two groups: one for safe symbols like rupee, dollar, and π, and another for those needing review, like target script leakage and accented text. It may be beneficial to query counts for purely currency-like symbols without leakage.
**Evaluating script leakage**

I’m considering excluding certain target scripts, mostly focusing on symbols and currency. I think I might need to compute something, possibly counting rows that have non-ASCII characters in Roman scripts and any leakage from the target script. This involves checking for accents as well as rows with symbols only. Alright, it sounds like I’ll need to query to get more clarity on this issue!


1. `missing_protected_roman`

This bucket is mostly **not empty data** and, based on what you want, it may not even be a real defect.

What I found from the actual canonical parquet:

- `416,395` rows are flagged with `missing_protected_roman`
- `415,680` of those had **non-ASCII/native digits already in the original mixed transcript** (`transcription`)
- `366,776` also still had native digits in `native_script_text`
- `415,762` had **ASCII digits in `romanized_text`**

So the common pattern is:

- original mixed transcript: `... ૧૮ ...`
- native variant: `... ૧૮ ...`
- romanized variant: `... 18 ...`
- validator flags it because it expected the exact byte sequence `૧૮` to survive unchanged

If your desired policy is:

- mixed should become `18`
- native should also become `18`
- romanized should be `18`

then this is **not a Gemini-quality problem first**. It is a **normalization policy mismatch**. The current validator was written to preserve numeric tokens exactly, but your desired output is to normalize all decimal digits to ASCII.

So yes, for this class, a **regex/normalization repair is feasible and probably better than rerunning Gemini**.

Best way:
- patch `transcription`
- patch `native_script_text`
- patch `romanized_text`
- clear or recompute `variant_validation_errors`

I would not use a brittle regex-only approach if we can avoid it. Better is a Unicode digit normalizer:
- for every character, if `unicodedata.decimal()` exists, replace it with ASCII `0-9`
- this handles Devanagari, Gujarati, Bengali, Telugu, etc. cleanly

Also, if you want ASCII digits everywhere, then yes, **changing only native is not enough**. Stage B exports:
- `transcription_mixed` from canonical `transcription`
- `transcription_native` from `native_script_text`
- `transcription_romanized` from `romanized_text`

So mixed/native/romanized should all be updated if that is the new rule.

2. `native_missing_target_script`

This one is more nuanced than it first looked.

There are about `105,049` rows containing that flag, but after checking against the **final corrected language**:

- `84,184` rows have `corrected_language != gemini_lang`
- `80,454` of the flagged rows already **do match the final corrected-language script**
- only about `24,595` rows still appear to be genuinely wrong under the final corrected language

That means a large chunk of this bucket is probably **stale validation metadata** from before language rerouting, not actual final-data corruption.

So yes, this is feasible to fix **without touching audio**.

You do not need to rebuild audio tar members for this kind of repair. The audio is separate. Text repair only means changing metadata.

Practical repair strategy:
1. Recompute validation against `corrected_language`, not the old Stage A target.
2. Split the flagged rows into:
   - `now_ok_under_final_language`
   - `still_bad_under_final_language`
3. Rerun Gemini only for the `still_bad` subset, which is closer to `24.6k`, not `105k`.
4. Merge repaired text back by `(video_id, segment_file)`.

That is very feasible.

Where to patch:
- Best source-of-truth patch point: canonical parquet / Stage A variant parquet
- Then either:
  - regenerate Stage B text metadata, or
  - patch Stage B `metadata.parquet` files in place

If you patch Stage B in place, you can leave `audio.tar` and `audio_index.parquet` alone, but you must also update:
- `metadata.parquet`
- `manifest.json`
- DB size/hash fields for the shard metadata objects

So yes, text-only repair is feasible, but I’d patch upstream first unless you specifically want a minimal in-place final-output hotfix.

3. `roman_not_ascii`

You’re right that this bucket is not automatically “bad”.

Actual breakdown from your data:

- total `roman_not_ascii`: about `94,949`
- approximately `75,121` are symbol-only cases
- `56,689` contain `₹`
- `2,706` contain `$`
- `2,537` contain `π`
- `10,454` contain accented Latin
- `9,379` contain target-script leakage into the Roman field

So this bucket is mixed:

Probably fine / policy choice:
- `₹`
- `$`
- `π`
- other symbols that preserve meaning

More concerning:
- accented Latin in the Roman channel
- actual native-script characters leaking into `romanized_text`

So if your view is “`₹` is ideal and should stay”, I agree that **those rows are not semantic errors**. They only violate the current strict ASCII validator.

Will `₹` hurt ASR/TTS training?
- usually not much if downstream text processing is Unicode-safe
- but it does break a strict “plain ASCII Romanization” contract
- the bigger consistency risk is not `₹`; it is script leakage and accented variants

So I would treat this bucket as:

- keep: symbol-only cases like `₹`, `$`, maybe `π` depending on policy
- review/fix: rows where Roman text still contains native script
- maybe normalize: accented Latin if you want one strict Roman standard

## Net Recommendation

Your real repair set is probably:

- digit normalization policy fix for mixed/native/romanized
- recompute validation against final `corrected_language`
- rerun Gemini only for the approximately `24.6k` rows still missing the final target script
- downgrade or remove `roman_not_ascii` for symbol-only cases like `₹`

That is all feasible without touching audio bytes.

If you want, I can next do one concrete thing:
1. write a precise repair plan with SQL/Python steps only, or
2. start implementing a repair script that:
   - normalizes digits to ASCII
   - recomputes final-language validity
   - exports the true rerun subset for Gemini.

**Assessing script requirements**

I need to check if the `final_cleaned_segments_with_variants` merge script exists. Knowing that the final export uses canonical transcription might be enough. I should mention the importance of canonical parquet and Stage A clean parquet as points of reference.

For clarity, “invalid” indicates that `variant_validation_errors` isn't empty. I must provide the original transcript version from `transcription` in canonical.

When it comes to digits, a normalizing pass using regex/unicode would be better than a complete rerun. I should also consider how to preserve numbers in URLs and emails.
**Considering digit conversion and patching**

I’m thinking about whether converting unicode digits to ASCII digits is safe across different contexts, like emails. It seems safe generally, but I need to be cautious with protected tokens such as phone numbers in native scripts. If the policy allows converting digits, I can replace them in final artifacts during Stage B, editing the `metadata.parquet` files without touching the audio tar.

However, patching the Stage A canonical source first is better. I need to ensure that if I patch final outputs directly, I must update the manifest and database hashes.

For the native missing target script, I can confirm that yes, I can repair text without altering audio shards since they're separate, enabling text-only repairs.
**Analyzing flagged discrepancies**

I'm looking at the stats: out of 105,049 flagged entries, 84,184 have a `corrected_language` that doesn't match `gemini_lang`. Interestingly, 80,454 of these already match the final corrected-language script, which leaves around 24,595 that are still missing it. This means not all 103,957 flagged issues are genuine problems; many are just stale flags after a reroute. I should highlight that the 24,595 could likely represent real issues under the current corrected language.
**Considerations for patching**

I'm thinking that for patching, we can keep the symbol-only cases unchanged while focusing on fixing target-script leakage and accented Latin characters. If the user prefers ASCII numerics but wants to maintain currency symbols, that's a separate consideration. I should probably note that a Stage B metadata patch is the only option available. Additionally, it may be worth mentioning that the audio tars remain untouched through this process.
**Evaluating editing strategy**

I need to mention editing JSON files, but most Stage B artifacts aren't JSON; they're mostly metadata.parquet and manifest.json. Since there isn’t a JSON per row, I can edit content by rewriting parquet rows without altering audio tars.

A concrete plan could be:

1. Normalize digits in `transcription`, `native_script_text`, and `romanized_text` using character mapping for non-ASCII decimal digits.
2. Validate against corrected language.
3. Rerun Gemini on rows still failing.
4. Patch the canonical parquet and update manifests/db.

This approach may resolve issues with stale flags and incorrect language flags.
**Clarifying row counts**

I realize I probably don't need to rerun all 103,957 rows. I want to ensure that numerical counts are consistent. The earlier count for native_missing_target_script was 105,049 in the canonical data, while the user noted 103,957 as the top exact validation errors. However, it seems that this top count refers specifically to the error string count of 103,957, which doesn't include the overall category with additional errors. I need to clarify this distinction in my response.


<user_query>
Agreed.

missing_protected_roman
This bucket is mostly not empty data and, based on what you want, it may not even be a real defect.
What I found from the actual canonical parquet:
416,395 rows are flagged with missing_protected_roman
415,680 of those had non-ASCII/native digits already in the original mixed transcript (transcription)
366,776 also still had native digits in native_script_text
415,762 had ASCII digits in romanized_text
So the common pattern is:
original mixed transcript: ... ૧૮ ...
native variant: ... ૧૮ ...
romanized variant: ... 18 ...
validator flags it because it expected the exact byte sequence ૧૮ to survive unchanged
If your desired policy is:
mixed should become 18
native should also become 18
romanized should be 18
then this is not a Gemini-quality problem first. It is a normalization policy mismatch. The current validator was written to preserve numeric tokens exactly, but your desired output is to normalize all decimal digits to ASCII.
So yes, for this class, a regex/normalization repair is feasible and probably better than rerunning Gemini.
Best way:
patch transcription
patch native_script_text
patch romanized_text
clear or recompute variant_validation_errors
I would not use a brittle regex-only approach if we can avoid it. Better is a Unicode digit normalizer:
for every character, if unicodedata.decimal() exists, replace it with ASCII 0-9
this handles Devanagari, Gujarati, Bengali, Telugu, etc. cleanly
Also, if you want ASCII digits everywhere, then yes, changing only native is not enough. Stage B exports:
transcription_mixed from canonical transcription
transcription_native from native_script_text
transcription_romanized from romanized_text
So mixed/native/romanized should all be updated if that is the new rule.

2.  native_missing_target_script (103,957 rows)
Original Hindi: "पे करते करते चले जाइए जदों वी जाना होएगा" (Devanagari) -> this sentence is supposed to be punjabi. probably should have written in gurmuki in the first place. native script has corrected it. Should we probably remove the original hindi instead ? so it looks like missed and will never enter the training with original sample ? or should we treat it as hindi ? i'm not sure about this.

but then these are specifically wrong.
Hindi segment: native is in Telugu script ఫస్ట్ ఫ్రంట్ దేఖా...
Hindi segment: native is in Odia script ସୋ ଲଗାନ ଇଜ ଦ ଫାର୍ଷ୍ଟ...

i feel like the best approach for these 100k segments is to remove the native form or whichever form is wrong. but how do we identify ?

3. roman_not_ascii - 93410 rows.

 Looking at the full non-ASCII character breakdown from 200 sampled rows:
Char	Count	What it is
₹	194	Indian Rupee symbol -- correct to preserve
ā	48	IAST romanization diacritic
ṭ	31	IAST romanization diacritic
ा (Devanagari)	19	Leaked native script character
ō	17	IAST romanization diacritic
ṇ, ḍ, ṉ, ṟ	~50	IAST romanization diacritics
π	7	Greek pi (math context)

how do you suggest we handle these. ₹,$ i dont have any problem with, but the other's, not sure. Should we remove the IAST and leaked devanagari both ?

suggest first. then we shall act. give updated plan + reasoning.

</user_query>


You’re deciding repair policy now rather than asking me to patch immediately, and there are real tradeoffs around what to preserve vs normalize, so I’m switching to Plan mode to lay out the safest repair strategy before we touch any data.

**Switching to planning mode**

I feel the need to offer an updated plan based on the user's request, so it makes sense to switch modes. Since the user explicitly asked for suggestions first and there are tradeoffs to consider, moving to planning mode should be the best approach. This isn't about coding; it's more about strategy. So, I'll call for a mode switch to 'Plan' with a brief explanation for clarity. Let's get this organized!


## Suggested Policy

I would separate the three text fields conceptually and repair them differently:

- `transcription` = source/mixed form. Keep the original wording/script mixture as provenance, but normalize decimal digits to ASCII if that is your global policy.
- `native_script_text` = clean final text in the `corrected_language` script. This is the field that must be script-correct.
- `romanized_text` = clean Roman fallback. Keep it mostly ASCII, but allow a small whitelist of semantic symbols you explicitly want to preserve.

That means I would **not** treat a Punjabi utterance written in Devanagari as “Hindi” just because the mixed/source transcript is in Devanagari. I would also **not delete the mixed/source field** for those rows. It is still useful provenance. The thing that must be correct is the native field, not the mixed field.

## Per Issue

### 1. `missing_protected_roman`
I would treat this as a **normalization-policy issue**, not a rerun-Gemini issue.

Reasoning:
- Almost all of these came from native digits in the original `transcription`.
- Romanized already normalized them to ASCII.
- Your desired policy is ASCII digits everywhere.

So my recommendation is:

- Normalize all Unicode decimal digits to ASCII in:
  - `transcription`
  - `native_script_text`
  - `romanized_text`
- Recompute `variant_validation_errors` under the new rule.
- Update the validator so numeric spans are compared after digit normalization, not byte-for-byte.

I would not rerun Gemini for this bucket.

### 2. `native_missing_target_script`
I would **not** treat the full `103,957` as real final errors.

From the checks:
- `80,454` flagged rows already match the **final corrected-language script**
- only about `24,595` still look truly wrong under the final `corrected_language`

So the right flow is:

1. Revalidate against `corrected_language`, not stale Stage A target assumptions.
2. Split rows into:
   - `stale_flag_only` -> clear the flag, keep text
   - `true_native_bad` -> native field genuinely wrong
3. For `true_native_bad`, rerun Gemini on the text using the final `corrected_language`.

For these rows, I would repair like this:

- Keep `transcription` as mixed/source text
- Regenerate `native_script_text`
- Regenerate `romanized_text` too if you rerun, because it is cheap and keeps the pair consistent
- If rerun still fails, blank only the bad field(s), rather than keep a wrong native script

I would **not** remove the mixed/source text just because it is in Devanagari for a Punjabi segment. That is a variant issue, not an original-transcript deletion issue.

### 3. `roman_not_ascii`
I would split this bucket into three different treatments.

What the data says:
- about `75,121` are symbol-only-ish cases
- about `56,689` include `₹`
- about `10,454` include IAST-style diacritics
- about `9,379` include leaked Indic script

My recommendation:

- Keep as allowed:
  - `₹`
  - `$`
  - other semantic symbols you explicitly whitelist
- Normalize:
  - IAST/diacritic Roman text to plain ASCII
- Repair:
  - any `romanized_text` that still contains Indic-script characters

So yes:
- `₹` and `$`: keep
- IAST diacritics: normalize away
- leaked Devanagari/Indic script in Roman field: fix

I would **not** “remove” leaked Devanagari characters by deletion. That would corrupt text. Those rows should be regenerated or transliterated safely.

For IAST, I would normalize characters to plain ASCII using Unicode decomposition:
- `ā -> a`
- `ṭ -> t`
- `ṇ -> n`
- `ō -> o`

That is better than deleting them, and better than keeping two Roman standards mixed together.

## Concrete Updated Plan

### Phase 1: Lock the repair contract
Decide and encode these rules:

- ASCII digits everywhere in all three text fields
- `transcription` is allowed to remain mixed-script source text
- `native_script_text` must match `corrected_language` script
- `romanized_text` may keep `₹` and `$`, but should not contain Indic script or IAST diacritics

This is the most important step. Without it, the repair will keep fighting the validator.

### Phase 2: Deterministic repairs first
Apply only non-LLM transforms to the canonical parquet:

- normalize Unicode decimal digits to ASCII in all three fields
- normalize IAST diacritics in `romanized_text` to plain ASCII
- clear stale `native_missing_target_script` flags where native already matches `corrected_language`
- clear `roman_not_ascii` when the only remaining non-ASCII is an allowed symbol like `₹`

This should wipe out most of:
- `missing_protected_roman`
- a lot of `roman_not_ascii`
- most of the fake/stale `native_missing_target_script`

### Phase 3: Build the true rerun subset
After deterministic repair, export only rows still failing real content checks.

Likely rerun set:
- rows where `native_script_text` still lacks corrected-language script
- rows where `romanized_text` still contains Indic script
- any remaining protected-span errors not explained by digit normalization

This is likely much closer to **25k-35k unique rows**, not 100k.

### Phase 4: Gemini repair pass
For those rows only:

- input: mixed/source `transcription`
- target language: `corrected_language`
- output: regenerated `native_script_text` and `romanized_text`

Then revalidate.

### Phase 5: Merge into canonical
Patch the canonical dataset by `(video_id, segment_file)`:

- update text fields
- recompute `variant_route`
- recompute `variant_validation_errors`

### Phase 6: Patch Stage B metadata only
Do **not** touch audio.

Patch only affected:
- `metadata.parquet`
- `manifest.json`
- DB metadata/hash/size records for affected microshards/final shards

That is feasible because Stage B text is independent of the audio tar payloads.

## Why This Plan
This plan minimizes risk because:

- it avoids rerunning Gemini on rows that only need digit normalization
- it avoids deleting useful source transcripts
- it treats stale validation metadata as stale, not as corruption
- it keeps audio immutable
- it makes the expensive rerun subset much smaller

## My Recommendation In One Line
Keep mixed/source text, normalize digits everywhere, keep `₹/$`, strip IAST from Roman, clear stale native-script flags, and rerun Gemini only for the rows that are still truly wrong under `corrected_language`.

If you want, the next step should be to define the exact row-classification matrix before editing anything, for example:

- `mixed_fix_only`
- `native_fix_only`
- `roman_fix_only`
- `rerun_native_and_roman`
- `clear_stale_flag_only`

I can draft that matrix next in a very concrete, implementation-ready form.

**Analyzing transcription types**

I'm considering how adjusting the segment language to "pa" and using Devanagari transcription might confuse the model if mixed scripts are expected. It's important to infer the intended training usage. For ASR/TTS training, there are specific fields to allow downstream choices. I recommend keeping the mixed transcript as it's closest to the source, though we should normalize digits and clean symbols. Removing mixed content would lose critical recognition information, so I need to handle the 100k native_missing_target_script.
**Identifying script errors**

I'm considering how to identify incorrect forms using script detectors. If the native_script_text contains the corrected script, it’s valid, and I can clear the stale flag. But if it's not correct, I wonder if the transcription or romanized versions might be okay? For final training, if native is wrong or absent, I should set `native_script_text=''` and keep mixed or romanized. However, I need to identify the wrong field correctly. It’s tricky, but identifying these inaccuracies is essential!
**Considering transliteration and repair**

I'm thinking about how to handle training with transliteration. Maybe I can convert `ṭ` to `t`, `ṇ` to `n`, and `ō` to `o`. The idea of not doubling base letters makes sense since diacritics often indicate length, but everyday romanization might not capture that fully. Using unidecode for a colloquial approach could be an option, but I’m cautious about adding dependencies. Also, I can't just delete leaked native script; rerunning is a better plan to preserve content.
**Planning symbol normalization**

I need to rethink how to handle the symbol-only `roman_not_ascii` category. It should be labeled as acceptable, and I think a clear validator error will help. I’m considering a phased update that starts with redefining the normalization contract. I’ll ensure that digits remain ASCII across all three columns while preserving currency and math symbols. This setup seems necessary for clarity and consistency in data processing.
**Evaluating transcription symbols**

ASR labels often include symbols reflecting written text, while TTS text normalization may address those. The user prefers preserving `₹`, and now confirms no issues with `₹` or `$`. I think we should stick to this preference. Maybe even keep `π` for math contexts, though it's less common. It’s best to suggest whitelisting such symbols while normalizing other non-ASCII characters. We definitely shouldn't remove the original Hindi mixed version, as it contains valuable information.
**Considering training data options**

In training, it’s crucial for downstream processes to select between `native` or `romanized`. The mixed version doesn't need to align with the corrected script. It’s worth mentioning training exposure here: if there's a desire to avoid Devanagari Punjabi in training data, we should exclude `transcription_mixed` from the language-specific native training. Instead, we could use `transcription_native` or `romanized`. However, if the dataset includes all three fields, we can keep the mixed version, allowing consumers to choose.
I’m considering how to best present information. It seems like using bullet points could work well without needing code citations unless they’re truly helpful. Maybe I’ll skip citations since the user isn't asking for a code breakdown. However, I could mention actual counts or refer to Stage B mapping, but that might not be necessary. Instead, I think I should suggest a policy without delving into switch rejection. Keeping it simple seems best!


<user_query>
All done. Clean run, zero errors across all three phases.

**Final results:**

| Phase | Time | Details |
|---|---|---|
| Download | 146s | 4,350 shards pulled to local disk |
| Patch | 98s | 60.7M rows processed across 120 cores |
| Upload | 147s | 3,643 changed shards pushed back to R2 |
| **Total** | **~6.5 min** | DB updated with patch timestamp |

**Patch stats across 60,711,097 rows:**

| Fix | Rows patched |
|---|---|
| Digit normalization (Unicode → ASCII) | 321,180 |
| Native text nulled (wrong script) | 227,246 |
| Romanized cleaned (IAST/leaked Indic/Arabic) | 41,955 |
| Shards unchanged (no fixes needed) | 707 |

The `native_nulled` count (227,246) is higher than the original 105,049 estimate -- likely because the script checks every row's native text against the expected script, catching additional mismatches the original flagging query missed. The `digits_patched` (321,180) is lower than 416,395 because some flagged rows may have had digits only in fields not being normalized, or the global pass found fewer unique rows needing changes. All local work files remain in `shard_work/` if you want to spot-check.

were these the only issues that were left for us to fix from initial check ?
</user_query>


Not quite. They were the main **artifact-level text/content issues**, but they were not the only issues from the initial audit.

There were two buckets in the original review:

## 1. Shipped data issues
These are the things your patch directly addressed.

- digit normalization / `missing_protected_roman` policy mismatch
- wrong-script or unusable `native_script_text`
- bad `romanized_text` normalization, including IAST and leaked Indic script
- stale/over-strict validation fallout around those fields

So for the **current exported text artifacts**, yes, this patch appears to have covered the core correctness problems I called out.

## 2. Pipeline/code issues
These were also in the initial check, and your metadata patch does **not** automatically fix them.

The important ones still outstanding in code were:

- `Stage B run isolation`
  `final_export_queue`, `final_export_video_outputs`, and `final_export_microshards` are not safely scoped by run, and `deploy_final_export.sh` defaulted to a fixed `FINAL_EXPORT_OUTPUT_PREFIX`. Future reruns can still overwrite or mix runs.

- `Stage B compactor atomicity`
  A final shard is uploaded/inserted before source microshards are marked consumed. If a worker dies in that window, retries can duplicate output rows across shards.

- `Stage A worker completion semantics`
  The worker can still mark shards done even when Gemini batches fail or when rows carry validation errors. Your patch repaired the resulting data, but the worker logic remains risky for future runs.

- `Stage A queue/idempotency`
  The variant queue still lacks stale-claim recovery and reseed/idempotency safeguards.

- `Stage B validator blind spot`
  `validate_final_export_run.py` still cannot reliably detect duplicate logical rows across packs/shards because it keys rows by `(video_id, segment_id)` and overwrites collisions.

- `Reference cache integrity`
  The R2 reference cache trusts byte size, not content hash, so same-size object replacement can go unnoticed.

## Bottom line
If your question is:

- “Were these the only remaining problems in the shipped dataset?”
  Then: **probably yes, these were the main row-content issues.**

- “Were these the only issues from the initial audit overall?”
  Then: **no**. Several future-run / rerun / crash-recovery bugs still exist in the pipeline code.

The good news is that your patch seems to have handled the part that most directly affects the current dataset. The remaining findings are mostly about **future reproducibility and safety**, not necessarily damage already present in this finished export.

If you want, I can turn the remaining code issues into a short, prioritized “must-fix before next export run” list.

**Clarifying user inquiries**

I need to answer the user about whether there are still issues left from the initial check. I have to recall prior findings and answer carefully. It’s important to mention that there were also code-level issues not just resolved by the metadata patch, like Stage B run isolation and stale claim recovery. I should clarify the difference between data issues and pipeline code issues without saying everything is good. I may also need to ask if they want a re-audit after the patch. Overall, I want to be concise yet clear.
**Summarizing validation findings**

I need to mention that the initial check revealed 615,995 validation-flagged rows. The patch addresses major issues, but it might not resolve all residual flags unless those are revalidated. Since the user says there are zero errors across three phases, it raises the question of whether the patch script had phases. I can clarify that while these were the main artifact-level content issues I identified, they’re not the only concerns overall.
**Identifying process issues**

I've identified two main code and process issues. First, Stage B isn’t run-scoped, so reruns could overwrite or mix results due to a fixed output prefix and primary keys not scoped by run. Second, the Stage B compactor upload and DB commit are non-atomic, which might lead to duplicate rows if there’s a crash or retry. Additionally, Stage A treats certain errors as successful completions. I'll consider whether "left for us to fix" refers to the content in the current exported artifacts.
**Clarifying audit findings**

From my initial audit, I can confirm that these were the primary user-visible and shipped-data correctness issues. The remaining findings relate to pipeline robustness bugs, which aren't defects in the current row content. These issues would only be relevant for reruns or future exports, so it's crucial to distinguish between immediate content problems and underlying process issues that could affect future performance. I want to ensure the user understands this distinction clearly.


<user_query>
Correct. None of the pipeline/code issues affected this run's shipped data. Here's why, one by one:

| Code issue | Why it didn't hit us |
|---|---|
| **Stage B run isolation** | Only one run (`production-20260312`) ever executed against this DB. No concurrent or second run existed to collide with. Verified: zero microshards/shards from other run IDs. |
| **Stage B compactor atomicity** | We verified `sum(shard segment_count)` = `sum(microshard row_count)` = `60,711,097` exactly, and sampled 200 shards with zero duplicate `(video_id, segment_id)` pairs. If a crash had caused a double-emit, the shard total would exceed the microshard total. It doesn't. |
| **Stage A worker completion semantics** | The `47,438` gemini_error rows from the variant worker never survived into the canonical parquet. The reroute/recovery pipeline cleaned them before Stage A/B ever saw them. |
| **Stage A queue/idempotency** | Stage A queue was seeded exactly once and ran to completion without re-seeding. No stale claims occurred because all 395,513 videos completed. |
| **Stage B validator blind spot** | Irrelevant because we proved absence of duplicates independently: canonical parquet has `60,711,097` distinct `(video_id, segment_file)` pairs with zero duplicates, and row totals match exactly end to end. |
| **Reference cache integrity** | Reference data was uploaded once and never replaced. No same-size-different-content scenario occurred. |

These are real code-level weaknesses that should be fixed before the next production export run, but they are **future-run risks, not current-data problems**. The dataset you just shipped is clean.
</user_query>


Yes. With the evidence you listed, I’d update the conclusion to:

**This shipped dataset is clean; the remaining findings are latent pipeline defects for future runs, not regressions that affected `production-20260312`.**

Your reasoning holds up.

- `Stage B run isolation`: agreed. A single run in the DB means there was nothing to collide with.
- `Stage B compactor atomicity`: agreed in practice. Exact equality of microshard totals and final shard totals is strong evidence that no double-emit happened in this run. The only theoretical loophole would be a duplicate+drop cancellation, but your shard sampling and row-level audits make that very unlikely.
- `Stage A worker completion semantics`: agreed. Since `gemini_error` rows never survived into the canonical parquet, they did not contaminate Stage B.
- `Stage A queue/idempotency`: agreed. If there was no reseed and no stranded claims, that risk stayed dormant.
- `Stage B validator blind spot`: agreed. Independent duplicate checks supersede the validator limitation for this run.
- `Reference cache integrity`: agreed. No object replacement means that failure mode was never exercised.

So the right final characterization is:

- **Current data status**: clean after your metadata patch.
- **Codebase status**: still has pre-next-run hardening work.

The only caveat I’d keep is wording: I’d say the compactor duplication bug was **not observed and is strongly ruled out by the run evidence**, rather than “mathematically impossible,” because exact totals alone do not prove every possible cancellation pattern. But operationally, your audit is strong enough that I’m comfortable treating this run as clean.

If you want, I can now compress the remaining code issues into a strict `before next production run` priority order.

**Reviewing user claims**

It seems the user is looking for confirmation or perhaps a review of their reasoning. They aren't asking for a detailed review, but they are presenting an argument that I might agree with, with some nuances. It could be beneficial to verify their claims, yet since they provided assertions, I might not need extensive validation unless prompted. I can agree that, under specific conditions and evidence, those pipeline issues could indeed be classified as latent. However, it's worth mentioning that sampling 200 shard duplicates isn’t exhaustive, but the strong evidence supports their conclusion. I’ll keep it concise, maybe even structuring it as bullet points.
**Clarifying potential caveats**

I should highlight a caveat about the duplicate-proofing if the Stage B compactor emitted doubles while omitting others. Theoretically, duplicates could occur if some rows are omitted and others duplicated in equal amounts, but their sampled shards and audits make it unlikely. The validator’s blind spot doesn’t matter here because they have independent proof to confirm their results. I can agree based on the evidence provided. I might suggest preserving evidence and addressing issues before the next run, turning the earlier findings into “latent defects.” If the user wants, I could create a checklist for hardening the next run.
**Summarizing Stage B compactor confidence**

I want to keep this brief. It seems important to mention that the strongest argument for the Stage B compactor’s atomicity proof is the end-to-end equality. The fact that they sampled 200 shards adds some confidence, but it doesn't constitute formal proof. However, since they’ve already removed orphan objects, I could argue that this evidence is “practically sufficient” to support their claims. I’ll make sure to convey this clearly for the user!


<user_query>
Can you query R2 and see if we have all the Shards intact. cause i had the other agent delete the microshards. just making sure the shards are not touched.
</user_query>


I’m going to verify this directly against R2 and the DB for the final shard objects only: count the shard records for `production-20260312`, list the actual R2 objects under those shard keys, and confirm nothing under the final shard paths is missing.

**Considering shard metadata**

I’m thinking I should read patch_shard_metadata.py for details on bucket and prefix, but the config likely lives in .env. I could use a shell script to activate the environment and check things, and maybe inspect the .env file for variable names. I need to confirm an object count of 17,400, but wait, the final shard count says 4,350. So, I should ensure the objects match the database sizes and verify if the shards are intact. I guess a terminal file inspection is only necessary for long processes or duplicates!
**Running shell with Python**

I’m thinking about running a shell command with a Python script that uses boto3 to list keys under the final shard prefix. I’ll need to know the bucket and prefix, possibly from the DB or README files. I might get that information from the .env file too. It seems useful to query the database for distinct output_bucket and key prefixes related to run_id production-20260312. I guess I can do this using psql or Python with asyncpg/psycopg, but I’ll need to activate the virtual environment first!


I’ve got the bucket/run details from your patch script, so I’m now checking every final shard object for `production-20260312` against the DB: all `metadata`, `audio.tar`, `audio_index`, and `manifest` objects, plus size matches.

**Evaluating microshard deletion**

I’m considering whether microshards deleted anything or if they only affected final shard paths. I might need to check if R2 has any deletion or last-modification records; the head should hold the LastModified information. I could confirm that all 17,400 objects exist. The patch script updated manifests for 3,643 changed shards, and while these shards have been recently touched, it seems the final shards remain intact. The user wants reassurance that nothing was deleted or inadvertently affected.


The important integrity check passed: all `4,350` final shard directories are still there and all `17,400` final-shard objects exist under `final-export/production/shards/`. I’m just breaking the size mismatches down by object type now, because your metadata patch intentionally rewrote some shard `metadata.parquet` and `manifest.json` files and that will show up as DB-vs-R2 size drift.

**Interpreting size mismatches**

I need to look into size mismatches due to the patch script not updating the database size fields properly. It seems like the script updated the manifest size but only altered the metadata_json timestamp, not the size fields for final_export_shards. It's expected to see mismatches here. I should verify the integrity of audio and audio_index and count mismatches by kind. A quick summary might help, maybe checking if distinct shards with mismatches equal around 3643*2. It’s a bit convoluted, but I can rerun a lighter summary for clarity.