o i@stdZddlZddlZddlmZmZmZddlmZGdddeZ GdddZ e Z dd e d e d e fd dZdS)u Text Normalization for TTS Pipeline Production-ready text normalization that prepares text for TTS models by: - Expanding numbers, dates, URLs, symbols to words - Language-aware punctuation cleanup (EN, HI, TE) - Preserving meaning while removing distracting elements - Deterministic and idempotent transformations Design principles: - Deterministic: same input → same output - Meaning-preserving: expand before removing - Practical: cover common cases, don't over-complicate - Language-aware: support English, Hindi, Telugu N)DictTupleOptional)Enumc@s eZdZdZdZdZdZdZdS)Languagez%Supported languages for normalizationenhiteunknownN)__name__ __module__ __qualname____doc__ENGLISHHINDITELUGUUNKNOWNrrA/home/ubuntu/veenaModal/veena3modal/processing/text_normalizer.pyrs rc @seZdZdZedZedZdddddd d Zd d d d ddddddddd Z edej Z edej Z gdZ gdZgdZgdZidddd d!d"d#d$d%d&d'd(d)d*d+d,d-d.d/d0d1d2d3d4d5d6d7d8d9d:d;d:dd=iZd?d@ZddBedCedDefdEdFZdBedDeeeeefffdGdHZdBedIeeefdDefdJdKZdBedDefdLdMZdBedDefdNdOZdBedDefdPdQZdBedDefdRdSZdBedDefdTdUZdBedDefdVdWZ dBedDefdXdYZ!dBedDefdZd[Z"dBedDefd\d]Z#dBedDefd^d_Z$dBedDefd`daZ%dBedDefdbdcZ&dBedDefdddeZ'dBedDefdfdgZ(dBedDefdhdiZ)djedDefdkdlZ*dme+dDefdndoZ,dBedDefdpdqZ-dre+dDefdsdtZ.dBedDefdudvZ/dBedDefdwdxZ0dBedyedDefdzd{Z1dBedDefd|d}Z2dBedyedDefd~dZ3dS)TextNormalizera Text normalizer for TTS pipeline. Implements a deterministic, order-dependent normalization pipeline: 1. Input sanitation 2. Language detection 3. Unicode normalization 4. Entity expansion (URLs, emails, currency, etc.) 5. Number expansion (to English words) 6. Date/time expansion 7. Symbol cleanup (language-aware) 8. Whitespace normalization 9. Final validation z[\u0900-\u097F]z[\u0C00-\u0C7F]rupeesdollarseurospoundsyenwon)u₹$u€£¥u₩plusminustimesz divided byequalsz less thanz greater thanzless than or equal tozgreater than or equal toz not equal to approximately) +−–×÷=<>u≤u≥u≠u≈uI\b(\d+(?:\.\d+)?)\s*(km|m|cm|mm|kg|g|mg|ml|l|°C|°F|km/h|mph|mb|gb|tb)\bz\b(\d+)(st|nd|rd|th)\b)zeroonetwothreefourfivesixseveneightnineteneleventwelvethirteenfourteenfifteensixteen seventeeneighteennineteen) r@twentythirtyfortyfiftysixtyseventyeightyninety) januaryfebruarymarchaprilmayjunejulyaugust septemberoctobernovemberdecember) z[angry]z [curious]z [excited][giggle][laughs harder][laughs] [screams][sighs][sings] [whispers]zmr.misterzmrs.missuszms.misszdr.doctorzprof. professorzst.saintzvs.versuszetc.etceterazi.e.zthat isze.g.z for exampleakaz also known asfyizfor your informationdiyzdo it yourselfokokayza.m.za mamzp.m.zp mpmcCsd|_tdtj|_dS)z+Initialize normalizer with default settingsFz](\[(?:angry|curious|excited|giggles?|laughs?(?: harder)?|screams?|sighs?|sings?|whispers?)\])N)verboserecompile IGNORECASE_emotion_tag_patternselfrrr__init__s  zTextNormalizer.__init__FtextrkreturncCs|r|sdS||_||\}}||}||}||}||}||}||}| |}| |}| ||}| |}| ||}|||}|S)z Normalize text for TTS. Args: text: Input text to normalize verbose: If True, return detailed transformation info Returns: Normalized text ready for TTS r@)striprk_protect_emotion_tags_sanitize_input_detect_language_normalize_unicode_expand_entities_expand_dates_times_expand_phones_expand_numbers_expand_abbreviations_cleanup_symbols_normalize_whitespace _final_checks_restore_emotion_tags)rqrsrkprotected_emotionslanguagerrr normalizes"             zTextNormalizer.normalizecsBddddddddid fd d }|j||}|fS) u Protect emotion tags from normalization. Replaces emotion tags with unique placeholders, stores mapping for later restoration. CRITICAL: Must preserve exact emotion tags for TTS model. Normalizes variants to match EXACT INDIC_EMOTION_TAGS format: - [laugh] → [laughs] - [giggles] → [giggle] (NOTE: model uses singular!) - [whisper] → [whispers] etc. Placeholder format: XEMOTIONX0X (no special chars that get removed) rWrUrXrYrZr[rV)z[laugh]z [giggles]z[scream]z[sigh]z[sing]z [whisper]z[laugh harder]rcs:|d}||}dd}||<d7|S)N XEMOTIONXX)grouplowerget)matchtagnormalized_tag placeholdercounteremotion_normalizations protectedrrreplace_emotions   z=TextNormalizer._protect_emotion_tags..replace_emotion)rosub)rqrsrrrrrvs  z$TextNormalizer._protect_emotion_tagsrcCs"|D] \}}|||}q|S)zRestore protected emotion tags)itemsreplace)rqrsrr original_tagrrrrsz$TextNormalizer._restore_emotion_tagscCs:t|tr |jddd}|d}|dddd}|S)zStep 1: Input sanitationzutf-8ignore)errorsuz   ) isinstancebytesdecodelstriprrqrsrrrrws  zTextNormalizer._sanitize_inputcCsRt|j|}t|j|}||kr|dkrtjS||kr&|dkr&tjStjS)zw Step 2: Simple language detection. Returns primary language based on character sets. )lenDEVANAGARI_PATTERNfindallTELUGU_PATTERNrrrr)rqrsdevanagari_count telugu_countrrrrxszTextNormalizer._detect_languagecCsNtd|}tdd|}ddddddd}|D] \}}|||}q|S) z&Step 3: Unicode normalization & safetyNFKCz"[\u200b-\u200f\u202a-\u202e\ufeff]r@"'-z...)rrz: "'", u—r&u…) unicodedatarrlrrr)rqrs replacementsfancyplainrrrrys z!TextNormalizer._normalize_unicodecC6||}||}||}||}||}|S)z Step 4: Expand entities (URLs, emails, currency, etc.) Must happen BEFORE symbol removal to preserve meaning. Order matters: emails before URLs to avoid conflict. )_remove_emojis_expand_emails _expand_urls_expand_social_expand_currencyrrrrrzs     zTextNormalizer._expand_entitiescCs:tjdtjd}|d|}tdtj}|d|}|S)zRemove all emojis and emoticonsuH[😀-🙏🌀-🗿🚀-🛿🇠-🇿✂-➰Ⓜ-🉑🤀-🧿🩰-🫿]+)flagsr@z[:;]-?[()DPO]|<3|XD)rlrmUNICODErrn)rqrs emoji_patternemoticon_patternrrrr8s   zTextNormalizer._remove_emojiscCs"tdtj}dd}|||S)zExpand URLs to readable formzY\b(?:https?://)?(?:www\.)?([a-zA-Z][a-zA-Z0-9-]*(?:\.[a-zA-Z][a-zA-Z0-9-]*)+)(?:/[^\s]*)?cSs|d}|dd}|S)Nr. dot rr)rdomainrrr replace_urlYs  z0TextNormalizer._expand_urls..replace_url)rlrmrnr)rqrs url_patternrrrrrPs  zTextNormalizer._expand_urlscCs0td}dd}|||}tdd|}|S)zExpand email addressesz5\b([a-zA-Z0-9._+-]+)@([a-zA-Z0-9.-]+\.[a-zA-Z]{2,})\bcSs.|d}|ddd}d|d|dS)Nrrr z at r)ruserrrrr replace_emailes z4TextNormalizer._expand_emails..replace_email +rrlrmr)rqrs email_patternrrrrras  zTextNormalizer._expand_emailscCs6tdd|}tdd|}dd}td||}|S)z"Expand social handles and hashtagsz&@\s+(\d{1,2}:\d{2}\s*(?:AM|PM|am|pm)?)zat \1z@([a-zA-Z0-9_]+)cSs&|d}tdd|}d|S)Nrz([a-z])([A-Z])\1 \2zhashtag )rrlrr)rrrrrreplace_hashtagzs z6TextNormalizer._expand_social..replace_hashtagz#([a-zA-Z0-9]+)rlr)rqrsrrrrrps zTextNormalizer._expand_socialcsHjD]\}tt|d}fdd}|||}q|S)zExpand currency amountsz\s*(\d[\d,]*(?:\.\d+)?)cs*|ddd}|}|dS)Nr,r@r)rr_number_to_words)ramount amount_wordsnamerqrrreplace_currencys z9TextNormalizer._expand_currency..replace_currency) CURRENCY_MAPrrlrmescaper)rqrssymbolpatternrrrrrs zTextNormalizer._expand_currencycstdd|}tdd|}jD]\}}|dvrqtdt|d}|d|d |}qtd fd d |}|S) z$Expand mathematical symbols to wordsu(^|\s)([-−–])(\d+)z \1minus \3u(\d+)\s+([-−–])\s+(\d+)z \1 minus \3)rr%r&z(\s|^)z(\s|$)\1z\2z(\d+(?:\.\d+)?)\s*%cs|ddS)Nrz percentrr)mrprrsz5TextNormalizer._expand_math_symbols..)rlr MATH_SYMBOLSrrmr)rqrsrwordrrrpr_expand_math_symbolssz#TextNormalizer._expand_math_symbolscCr)z Step 5: Expand numbers to English words. Per user request: expand to English, NOT localized to language. Order matters: math before ranges, ordinals before units. )r_expand_ordinals _expand_units_expand_ranges_expand_general_numbersrrrrr}s     zTextNormalizer._expand_numbersc siddddddddd d d d d ddddddddddddddddddd d!d"d#d$d%d&d'd(d)d*d+d,d- fd.d/}j||S)0u&Expand ordinal numbers (1st → first)rfirstrsecondthirdfourthrfifthsixthseventheighth ninth tenth eleventh twelfth thirteenth fourteenth fifteenth sixteenth seventeenth eighteenth nineteenth twentieth thirtiethfortiethfiftiethsixtieth seventieth eightieth ninetieth) (2<FPZcst|d}|vr|S|dkrA|dd}|d}|dkr*||dSj|d}|j|}|d|St|dS)Nrdrrthr)intrrTENSONESrstr)rnumtensones tens_word ones_word ordinal_wordsrqrrreplace_ordinals z8TextNormalizer._expand_ordinals..replace_ordinal)ORDINAL_PATTERNr)rqrsr!rrrrsR zTextNormalizer._expand_ordinalscsiddddddddd d d d d ddddddddddddddddddd fd!d"}j||S)#zExpand numbers with unitskm kilometersrmeterscm centimetersmm millimeterskg kilogramsggramsmg milligramsml millilitersllitersu°czdegrees celsiusu°fzdegrees fahrenheitzkm/hzkilometers per hourmphzmiles per hourmb megabytesgb gigabytestb terabytescs<|d}|d}|}||}|d|S)Nrrr)rrrr)rnumberunit number_words unit_wordrqunit_expansionsrr replace_units   z2TextNormalizer._expand_units..replace_unit) UNIT_PATTERNr)rqrsrArr?rrsF      zTextNormalizer._expand_unitscs"td}fdd}|||S)u Expand number ranges (10-12 → ten to twelve) NOTE: Math symbols (minus) are handled separately in _expand_math_symbols This only handles actual ranges (no spaces around hyphen, or clear range context) u%(?.replace_ranger)rqrs range_patternrErrprrs  zTextNormalizer._expand_rangescs0tdd|}td}fdd}|||S)zExpand standalone numbersz(?<=\d),(?=\d)r@z\b(\d+(?:\.\d+)?)\bcs|dS)Nrr)rrprrreplace_number%sz>TextNormalizer._expand_general_numbers..replace_number)rlrrm)rqrsnumber_patternrGrrprrs   z&TextNormalizer._expand_general_numbers number_strcs|}|drd|ddSd|vr>|d\}}|r)t|nd}dfdd |D}|d |St|S) u Convert number string to English words. Handles: - Integers: 123 → "one hundred twenty three" - Decimals: 3.14 → "three point one four" - Negatives: -5 → "minus five" rminus rNrr,rc3|] }t|VqdSN_integer_to_wordsr.0drprr =z2TextNormalizer._number_to_words..z point )ru startswithrsplitrNrjoin)rqrI integer_part decimal_part integer_words decimal_wordsrrprr*s zTextNormalizer._number_to_wordsncsv|dkrdS|dkrd| S|dkrj|S|dkr>|d}|d}|dkr1j|Sj|dj|S|dkrc|d}|d}|dkrVj|d Sj|d |S|d kr|d}|d}|dkr{|d S|d |S|dkr|d }|d }|dkr|dS|d|Sdfddt|DS)z Convert integer to English wordsrr,rJr rrri hundredz hundred i@Bz thousandz thousand iʚ;z millionz million c3s|] }jt|VqdSrL)rrrOrprrrRkrSz3TextNormalizer._integer_to_words..)rNrrrVr)rqr[rrhundreds remainder thousandsmillionsrrprrNCs>  z TextNormalizer._integer_to_wordscsjtd}fdd}|||}td}fdd}|||}td}fdd }|||}|S) z Step 6: Expand dates and times. Handles: - ISO dates: 2025-11-13 - Numeric dates: 13/11/2025, 11/13/2025 - Times: 3:45 PM, 14:30 z\b(\d{4})-(\d{2})-(\d{2})\bcs|d}t|d}t|d}d|krdkr&nnj|dnt|}|}t|}|d|d|S)Nrrrrr)rrMONTHSrrN_year_to_words)ryearmonthday month_nameday_word year_wordrprrreplace_iso_dateys . z.replace_iso_datez\b(\d{1,2})/(\d{1,2})/(\d{4})\bc st|d}t|d}t|d}|dkr||}}n||}}d|kr.dkrKnnj|d}|}|}|d|d|S|dS)Nrrrrrr)rrrarNrb) rrrrcrerdrfrgrhrprr replace_dates     z8TextNormalizer._expand_dates_times..replace_datez&\b(\d{1,2}):(\d{2})\s*(am|pm|AM|PM)?\bcsjt|d}t|d}|d}|}|}|d|}|r3|d|dd7}|S)Nrrrrr)rrrNrr)rhourminuteperiod hour_word minute_wordresultrprr replace_times   z8TextNormalizer._expand_dates_times..replace_timer)rqrs iso_patternri date_patternrj time_patternrqrrprr{ms       z"TextNormalizer._expand_dates_timesrccCsf|dkr.|dkr ||S|d}|d}|dkr!||dS||d||S||S)u9Convert year to words (2025 → two thousand twenty five)irrr\r)rN)rqrccenturyr^rrrrbs  zTextNormalizer._year_to_wordscsHtd}fdd}|||}td}fdd}|||}|S)z Step 7: Expand phone numbers to digit groups. Handles: - US format: (415) 555-2671 - International: +91 98765 43210 z\((\d{3})\)\s*(\d{3})-(\d{4})csndfdd|dD}dfdd|dD}dfdd|dD}|d |d |S) Nrc3rKrLrMrOrprrrRrSzJTextNormalizer._expand_phones..replace_us_phone..rc3rKrLrMrOrprrrRrSrc3rKrLrMrOrprrrRrSr, rVr)rareaprefixlinerprrreplace_us_phonesz7TextNormalizer._expand_phones..replace_us_phonez\+(\d{1,3})\s+(\d+)\s+(\d+)cspdfdd|dD}dfdd|dD}dfdd|dD}d |d |d |S) Nrc3rKrLrMrOrprrrRrSzLTextNormalizer._expand_phones..replace_intl_phone..rc3rKrLrMrOrprrrRrSrc3rKrLrMrOrprrrRrSrzplus rvrw)rcodepart1part2rprrreplace_intl_phonesz9TextNormalizer._expand_phones..replace_intl_phoner)rqrsus_phoner{ intl_phonerrrprr|s     zTextNormalizer._expand_phonescCs@|jD]\}}tdt|dtj}|||}q|S)z"Step 8: Expand known abbreviationsz\b) ABBREVIATIONSrrlrmrrnr)rqrsabbr expansionrrrrr~sz$TextNormalizer._expand_abbreviationsrc Cs"|tjkrhd}tdd|}n|tjkr"ddh}tdd|}n hd}tdd |}|tjtjfvrJ|}d }|D] }||vrG||d }q;|Sg}|D];}d |koYd knpgd|koedkn} |sv|sv||vsv| r|||qN|r|dd kr|d qNd |}|S)ab Step 9: Symbol & punctuation cleanup (language-aware). Keep only allowed punctuation for each language: - EN: , . ? - HI: , | ? - TE: , ? CRITICAL: Must preserve Unicode word integrity for Hindi/Telugu. DO NOT iterate character-by-character as it breaks combining marks. >?|rz [.!;:](\s|$)z|\1rrz,\1>rrr!rz!"#$%&'()*+/:;<=>@[\]^_`{|}~ruऀuॿuఀu౿r@) rrrlrrrisalnumisspaceappendrV) rqrsr allowed_punct allowed_charsascii_punct_to_removepunct cleaned_charscharis_indicrrrrs4    0   zTextNormalizer._cleanup_symbolscCsJtdd|}tdd|}tdd|}tdd|}tdd |}|S) z!Step 10: Whitespace normalizationrrz\n+rz *\n *z \s+([,.|?])rz([,.|?])([^\s\n])rrrrrrr/s z$TextNormalizer._normalize_whitespacecCshdd|dD}d|}tdd|}tdd|}tdd |}td d |}td d |}|S)z%Step 11: Final validation and cleanupcSsg|] }|r|qSr)ru)rPrzrrr Esz0TextNormalizer._final_checks..rz\?+rz,+rz\.+rz\|+rz([.,|?])\s*([.,|?])+r)rUrVrlr)rqrsrlinesrrrrBs zTextNormalizer._final_checksNF)4r r r rrlrmrrrrrnrBr"rrra EMOTION_TAGSrrrrboolrrrrvrrwrrxryrzrrrrrrr}rrrrrrrNr{rbr|r~rrrrrrrrs         "%."  *G  @rFrsrkrtcCstj||dS)z Convenience function for text normalization. Args: text: Input text to normalize verbose: If True, return additional debug info Returns: Normalized text ready for TTS )rk) _normalizerr)rsrkrrrnormalize_textYs rr)rrlrtypingrrrenumrrrrrrrrrrrs >