o 5i\W@sdZddlZddlmZmZddlmZmZmZm Z ddgdddd d gdddd d gdddd d gdddddgdddddgdddddgdddddgdddddgdddddgdddddgddddddgdddd Z e dZ e d Z hd!Zdad"ed#ed$e efd%d&Z ' (dbd)ed"ed#ed$efd*d+Z ' ,dcd)ed"ed-ed$efd.d/ZeGd0d1d1Zdadad"efd2d3Zd4d5Zd6ed7ed$efd8d9Zd)ed$efd:d;Zd)ed"ed$efd  ,ddd?ed@ed"edAedBed-ed$efdCdDZdEdFZ ded@ed"ed$efdGdHZ!e"dIkrddl#Z#e$e#j%dJkr%e&dKe#'dLe#j%dLdMkrVdN(e#j%dOdZ)e!e)Z*e&dPe*dQe*dQsTe&dRe*dSdSdSe#j%dLZ+dN(e#j%dOdZ,e&dTee+e,Z*e e&dUe*j-.e&dVe*j/e&dWe*j0dXe&dYe*j1dZe&d[e*j2dZe&d\e*j3dZe*j4re&d]e*j4dd^e*j5re&d_e*j5D]Z6e&d`e6qdSdSdS)fa Simple Transcription Validator (v4) ==================================== Architecture change: validator now derives its own romanization via uroman instead of trusting Gemini's creative romanization. This makes validation deterministic and reproducible. Four checks: Step 0: Structural sanity (no ML, instant) - unicode, tags, length/duration Step 1: Character validation - catch garbage/alien characters Step 2: Native CTC alignment - verify native script matches audio Step 3: Romanized MMS alignment - verify uroman-derived text matches audio Combined scoring: S = 0.45*N + 0.55*R - 0.10*abs(N-R) N = native CTC score, R = romanized MMS score (using uroman, not Gemini) Stricter thresholds with per-word analysis: ACCEPT: avg >= 0.85 AND min_internal_word >= 0.70 REVIEW: avg in [0.70, 0.85) or boundary issues RETRY: avg in [0.55, 0.70) REJECT: avg < 0.55 or structural failure Usage: from src.validators.simple_validator import validate_transcription result = validate_transcription("audio.flac", native_text, language="te") # result.status: "accept" / "review" / "retry" / "reject" # result.combined_score: 0-1 N) dataclassfield)ListDictOptionalSetTelugu)i i T)namerangesallow_ascii_punct allow_digitsHindi)i i MarathiTamil)i i Kannada)i i Malayalam)i i Bengali)i i AssameseGujarati)i i Punjabi)i i Odia)i i English)AZ)az) tehimrtaknmlbnasgupaorenu$ .,!?;:'"()-–—।॥01234567894abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ> [UNK][sigh][cough][laugh][music][noise][breath] [singing] [applause] [INAUDIBLE] [NO_SPEECH]language allow_englishreturncCs~t}tD] }|t|qt|td}|dD]\}}t||dD]}||q&q|r=tD] }|t|q3|S)z3Get set of valid Unicode codepoints for a language.rr )setALLOWED_COMMONaddord SCRIPT_RANGESgetrange ENGLISH_CHARS)r4r5validc lang_configstartendcprF@/home/ubuntu/maya3_transcribe/src/validators/simple_validator.pyget_valid_charsys rHrFtextc s|r|s dgdddSt||}t|td}g}d}d}t|D]5\}} t| tfdd|dD} | rD|d 7}|d 7}q$|vrY|| |d | tvrY|d 7}q$t |dk|d d t ||dkrq||d Sdd S)z>Check if text contains only valid characters for the language.Frempty)r@ invalid_chars script_ratioreasonrc3,|]\}}|ko|knVqdSNrF.0rCrDrErFrG *z#check_characters..r r7)char codepointpositionN )r@rK invalid_countrL) striprHr<r= enumerater;anyappendr?len) rIr4r5 valid_charsrBrK script_count total_alphairU is_nativerFrRrGcheck_characterss2     rd duration_seccsrg}|r|sddgdS|}|dvrdgdStd|}|D]}|tvr1|d|q#|dkr|td d |D}d }t|td } |D]} t| t fd d | dDrb|d7}qK||} | d kr||| dkr||d|d| |d krt | } | t |d} | dkr|d| dd|dkr| dkr|d| d|ddt |d k|dS)a Fast structural checks before any ML model runs. Catches obvious garbage without wasting GPU cycles. Checks: 1. Unicode block: native text must be in target script (no Latin in non-English) 2. Tag format: only allowed bracketed tags, properly formatted 3. Length vs duration: reject impossibly dense text (>15 words/sec) 4. Empty/whitespace-only: instant reject Returns: {"pass": bool, "reasons": [...]} F empty_text)passreasons)r3r2Tz\[\w+\]z invalid_tag:r'css|] }|tvrdVqdS)r7N)r?rQrArFrFrGrSz*structural_sanity_check..rrc3rNrOrFrPrRrFrGrSrTr r7皙?ztoo_few_native_chars:/zimpossibly_dense:z.1fzw/sg?rXztoo_many_words_for_short_audio:zw/s) rZrefindall VALID_TAGSr]sumr<r=r;r\r^splitmax)rIr4rfristripped found_tagstag latin_chars native_charsrBrAtotal word_count words_per_secrFrRrGstructural_sanity_checks>     r~c@seZdZUdZeed<dZeed<dZe e ed<dZ e ed<dZ e ed <dZe ed <dZe eed <dZe ed <dZe ed <dZe ed<dZe ed<dZe ed<dZe ed<dZeed<dZe eed<dZeed<dZe eed<de fddZdS)ValidationResultz5Result of transcription validation with dual scoring.statusT char_validNrKrerLalignment_scorenative_ctc_scorelow_confidence_wordslow_confidence_ratioroman_mms_scoremms_min_word_scoremms_boundary_word_avgmms_internal_below_thresholdcombined_scorestructural_passstructural_reasonsuroman_romanizedrir6cCsz|j|j|jpgt|jdt|jdt|jdt|jdt|jdt|j dt|j d|j p/gt|j d|j |jp:gdS)N)rrrKrLrrrrrrrrrri)rrrKroundrLrrrrrrrrrri)selfrFrFrGto_dicts        zValidationResult.to_dict)__name__ __module__ __qualname____doc__str__annotations__rboolrKrrrLfloatrrrrrrrrrrrrrirrFrFrFrGrs(             rcCs"tdurddlm}||datS)zLazy load native CTC aligner.Nr7)CTCForcedAligner)r4) _ctc_alignerctc_forced_alignerr)r4rrFrFrG_get_ctc_aligner(s  rcCstdur ddlm}|atS)z Lazy load romanized MMS aligner.Nr7 MMSAligner) _mms_aligner mms_alignerrrrFrFrG_get_mms_aligner0s r native_ctc roman_mmscCs||}}d|d|dt||}|dkr.t||dkr.t||dkr+d}nd}n|d kr5d}n |dkr= 0.75 AND min(N,R) >= 0.50 REVIEW: S in [0.65, 0.75) OR high disagreement RETRY: S in [0.55, 0.65) REJECT: S < 0.55 g?皙?rl???reviewacceptg?retryrejectr)absminr)rrNRSverdictrFrFrGcompute_combined_score9s  rcCs.tdd|}tdd|}tdd|S)zOStrip tags and punctuation for alignment. Keeps native script + Latin + spaces.z \[[\w_]+\]rz![\u0964.,!?;:\'"()\-\u2013\u2014]\s+ )rpsubrZ)rIcleanedrFrFrG_strip_tags_and_punct\srcst|}|dkr |St|td}g}|D]}t|tfdd|dD}|s/|dvr4||qtddd | S) zStrip Latin characters for native CTC alignment on code-mixed text. CTC models are language-specific and choke on English words mixed in. We remove Latin words and only align the native-script portions.r'rc3rNrOrFrPrRrFrGrSnrTz(_strip_to_native_only..r z rrr) rr<r=r;r\r]rprjoinrZ)rIr4rrBresultrUrcrFrRrG_strip_to_native_onlycs  rr audio_path transcriptionromanized_text check_audioc Csg}t|}t||}t|||} | ds(tdd| ddd| dgdSt||dd } | d sAtdd| d | d d gdS| d dkrS|d| d ddddlm} | |} d} g}d}|r|rzt |}| ||}|j } |j }|j }Wn ty}z|dt|ddWYd}~nd}~wwd}d}d}d}|r8| r8zkt}| || }|j }t|dr|jr|j}|rt|}g}t|dkr||dt|dkr||d|rt|t|nd}t|dkr|ddng}|rtdd|D}|t|}Wn!ty7}z|dt|ddWYd}~nd}~ww|rh| dkrh|dkrht| |\}}|d krgt| |d!krg|d"| d#d$|d#nb|r| dkr| }| d%krzd&}n | d'krd(}nd}|dkr| r|d)n8|r|dkr|}|d%krd&}n |d'krd(}nd}|d*n|rd}d }|d+n d}|rd nd&}| d sd}|d(kr|d,|d#d-| d#d.|d#dn|dkr|dkr|d/|d#d-| d#d.|d#dtd=id0|d1| d d | d d | d d2| d3| d4|d5|d6|d7|d8|d9|d:|d;dd<| d|rD|SdS)>a Validate a transcription (v4: uroman-based, stricter, structural checks). Flow: Step 0: Structural sanity checks (instant, no ML) Step 1: Character validation (instant) Step 2: Native CTC forced alignment (language-specific wav2vec2) Step 3: Romanized MMS forced alignment (uroman-derived, NOT Gemini's) Step 4: Weighted combined score with per-word analysis Key v4 change: romanized_text parameter is IGNORED for MMS alignment. Validator derives its own romanization via uroman from the native text. This makes validation deterministic and independent of Gemini's output. Args: audio_path: Path to audio file transcription: Native script text language: Language code (te, hi, ta, etc.) romanized_text: IGNORED in v4 (kept for API compat). Validator uses uroman. check_audio: Whether to run audio alignment checks duration_sec: Audio duration (for structural checks, optional) rhrFrizstructural_fail: z, )rrrriTr5r@rKrLzInvalid/alien characters found)rrrKrLrirToo few native chars (.0%)r)romanize_for_alignmentrezCTC alignment failed: N2 word_scoresr7css|] }|dkrdVqdS)g333333?r7NrF)rQrorFrFrGrSrkz)validate_transcription..zMMS alignment failed: rrzValidator disagreement: CTC=z.2fz MMS=rrrrzMMS failed, using CTC onlyzCTC failed, using MMS onlyzBoth validators returned 0zBelow threshold: combined=z (CTC=z, MMS=zPoor alignment: combined=rrrrrrrrrrrrrrF)rrr~rrrdr]src.romanizationrralignrrr Exceptionrrhasattrrrr^rsrrr=)rrr4rrrfri clean_text native_only structural char_checkr uroman_romanrlow_conf_wordslow_conf_ratioaligner ctc_resulter mms_min_wordmms_boundary_avgmms_internal_belowmms mms_resultscoresboundaryinternalbelowrrrFrFrGvalidate_transcriptionts4       (     (                   rcCs>tdur tdatdurtdaddlm}|dS)z3Release all aligner resources (CTC + MMS + uroman).Nr)cleanup)rrrr) roman_cleanuprFrFrGr0s  rcCs|r|s dddSt||}|dsdd|ddSt||dd}|ds>d d |d d d D}dd|dS|ddkrPdd|ddddSd|ddS)zFQuick character-only + structural validation for native transcription.FzEmpty transcription)r@rMrhz Structural: rirr@cSsg|]}|dqS)rUrFrjrFrFrG Lsz"quick_validate..rKNzInvalid chars: rLrrrrT)r@rL)rZr~rd)rr4rrinvalidrFrFrGquick_validate?s    r__main__ra Simple Validator v4 ==================== Usage: # Full validation (with audio check, uroman-based) python simple_validator.py # Quick validation (character + structural check only) python simple_validator.py --quick r7z--quickrrzValid: r@zReason: rMz Validating (v4: uroman-based)...z Status: zCharacter valid: zScript ratio: z.1%z CTC score: z.4fz MMS score: z Combined: zUroman: Pz Reasons:z - )T)rF)rre)rrTre)r)7rrp dataclassesrrtypingrrrrr<r8r9r?rrrrintrHrdrr~rrrrrtuplerrrrrrrsysr^argvprintexitrrIrrrrupperrrLrrrrrirrFrFrFrGs@K ' ?8  # =      '