o wi@sdZgdZddlZddlZddlZidddddd d d d d dddddddddddddddddddddd Zd)d"efd#d$Zd"efd%d&ZGd'd(d(Z dS)*z4OpenAI's non-english basic text normalization module)ADDITIONAL_DIACRITICSremove_symbols_and_diacriticsremove_symbolsBasicTextNormalizerNuœoeuŒOEøoØOæaeÆAEßssuẞSSuđduĐDðÐþthÞułluŁLscs dfddtd|DS)z Replace any other markers, symbols, and punctuations with a space, and drop any diacritics (category 'Mn' and some manual mappings) rc3sX|]'}|vr |n|tvrt|nt|dkrdn t|ddvr&dn|VqdS)MnrrMSP N)r unicodedatacategory.0ckeepU/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/whisper_normalizer/basic.py ,s z0remove_symbols_and_diacritics..NFKDjoinr! normalize)rr'r(r&r)r's rcCsdddtd|DS)z[ Replace any other markers, symbols, punctuations with a space, keeping diacritics rcss*|]}t|ddvrdn|VqdS)rrr N)r!r"r#r(r(r)r*Bs  z!remove_symbols..NFKCr,)rr(r(r)r>s  rc@s6eZdZdZ  d dedefddZdefdd Zd S) raAs per the text normalization/standardization approach mentioned in Appendix Section C pp.21 in the paper [Robust Speech Recognition via Large-Scale Weak Supervision](https://cdn.openai.com/papers/whisper.pdf). The `BasicTextNormalizer` does the following functionality: 1. Remove any phrases between matching brackets ([, ]). 2. Remove any phrases between matching parentheses ((, )). 3. Replace any markers, symbols, and punctuation characters with a space, i.e. when the Unicode category of each character in the NFKC-normalized string starts with M, S, or P. 4. make the text lowercase. 5. replace any successive whitespace characters with a space Note: It's not recommended to use this function for non-english languages because it may removes vowels in languages as identified by [kavya in this tweet](https://twitter.com/kavya_manohar/status/1752574864618365059). Fremove_diacritics split_letterscCs|rtnt|_||_dS)a$ remove_diaciritics - Replace any other markers, symbols, and punctuations with a space and drop any diacritics split_letters - It uses a regular expression \X to find all Unicode graphemes (extended grapheme clusters) in the string s and join them together by space N)rrcleanr1)selfr0r1r(r(r)__init__Us  zBasicTextNormalizer.__init__rcCs`|}tdd|}tdd|}||}|jr'dtd|tj}tdd|}|S)Nz[<\[][^>\]]*[>\]]rz \(([^)]+?)\)r z\Xz\s+) lowerresubr2r1r-regexfindallU)r3rr(r(r)__call__cszBasicTextNormalizer.__call__N)FF)__name__ __module__ __qualname____doc__boolr4strr;r(r(r(r)rHs r)r) r?__all__r6r!r8rrArrrr(r(r(r)sR