o 5¤˜iã@sjdZddlZddlmZdadd„Zdedefdd „Zdedefd d „Zd ed ede fdd„Z dd„Z dS)uL Deterministic Romanization Module ================================== Converts native Indic script text to MMS_FA-compatible Latin romanization using the uroman library. This replaces Gemini-generated romanization with a deterministic, reproducible transform. Why deterministic: - Gemini's "romanized as pronounced" is creative and inconsistent across runs - uroman produces identical output for identical input, every time - MMS_FA was trained with uroman-style romanization, so alignment is optimal MMS_FA normalization: [a-z' ] only, lowercase, collapsed spaces. Per: https://docs.pytorch.org/audio/2.8/tutorials/forced_alignment_for_multilingual_data_tutorial.html Usage: from src.romanization import romanize, romanize_for_alignment # Basic romanization roman = romanize("నాకౠకొనà±à°¨à°¿ యాడà±à°¸à± à°—à±à°°à±à°¤à±à°‚టాయి") # -> "naaku konni yaadds gurtumttaayi" # MMS_FA-ready (stripped to [a-z' ] only) aligned = romanize_for_alignment("నాకౠకొనà±à°¨à°¿, యాడà±à°¸à±!") # -> "naaku konni yaadds" éN)ÚOptionalcCstdur ddl}| ¡atS)z;Lazy load uroman. First call ~3s, subsequent calls instant.Nr)Ú_uroman_instanceÚuromanÚUroman)r©rú1/home/ubuntu/maya3_transcribe/src/romanization.pyÚ _get_uroman#srÚtextÚreturncCs$|r| ¡sdStƒ}| | ¡¡S)a6 Romanize text from any script to Latin using uroman. Deterministic: same input always produces same output. Handles Indic scripts (Telugu, Hindi, Tamil, etc.) + passes Latin through. Args: text: Text in any script (Indic, Latin, mixed) Returns: Romanized Latin text Ú)ÚstriprÚromanize_string)r ÚurrrÚromanize,s rcCs`|r| ¡sdSt dd|¡}t dd|¡}t|ƒ}| ¡}t dd|¡}t dd|¡ ¡}|S)aì Romanize and normalize for MMS_FA forced alignment. MMS_FA expects: lowercase [a-z' ] only. Steps: 1. Strip tags like [laugh], [UNK], [INAUDIBLE] etc. 2. Romanize via uroman 3. Lowercase 4. Strip everything except [a-z' ] 5. Collapse multiple spaces 6. Strip leading/trailing whitespace Args: text: Native script text (may include punctuation, tags) Returns: Normalized romanized text ready for MMS_FA alignment r z \[[\w_]+\]u[।,.!?;:\'"()\-–—…]Ú ú[^a-z' ]ú\s+)r ÚreÚsubrÚlower)r ÚcleanedÚromanrrrÚromanize_for_alignment?s rÚ native_textÚgemini_romanizedc Cst|ƒ}t dd| ¡¡}t dd|¡ ¡}|s|rdSdS|}|}t|ƒ}t|ƒ}|dkr6|dkr4dSdStt|dƒƒ}td|dƒD]>} | gdg|} td|dƒD]*} || d|| dkrfdnd} t| | dd|| d|| d| ƒ| | <qV| }qEtd|||ƒS)a Compute Character Error Rate between uroman-derived and Gemini-provided romanization. Low CER (<3%) = Gemini's romanization is consistent with deterministic version. High CER (>10%) = Gemini hallucinated or used non-standard romanization. Useful for flagging romanization drift without blocking transcriptions. Args: native_text: Native script text (source of truth) gemini_romanized: Gemini's romanized output Returns: CER as float (0.0 = identical, 1.0 = completely different) rrrgð?gré) rrrrr ÚlenÚlistÚrangeÚmin) rrÚ uroman_romanÚ gemini_normÚrefÚhypÚnÚmÚprevÚiÚcurrÚjÚcostrrrÚcompute_romanization_ceris.    ýr+cCsdadS)zRelease uroman resources.N)rrrrrÚcleanupžsr,) Ú__doc__rÚtypingrrrÚstrrrÚfloatr+r,rrrrÚs  *ÿþ ý 5