o wiZ@sdZgdZddlZddlZddlZddlmZddlmZm Z m Z m Z m Z ddl mZddlmZdd lmZGd d d ZGd d d ZGdddZdS)z,OpenAI's English text standardisation module)EnglishNumberNormalizerEnglishSpellingNormalizerEnglishTextNormalizerN)Fraction)IteratorListMatchOptionalUnion)files)windowed)remove_symbols_and_diacriticscsdeZdZdZfddZdeedeefddZdefd d Z defd d Z defd dZ Z S)rav Convert any spelled-out numbers into arabic numbers, while handling: - remove any commas - keep the suffixes such as: `1960s`, `274th`, `32nd`, etc. - spell out currency symbols after the number. e.g. `$20 million` -> `20000000 dollars` - spell out `one` and `ones` - interpret successive single-digit numbers as nominal: `one oh one` -> `101` c sthd|_ddtgdddD|_dd|jD|_dd d d d d ddd|jD|_i|j|j|_ddddddddd|_ dd|j D|_ dd|j D|_ i|j |j |_ dddddd d!d"d#d$d%d&d' |_ d(d|j D|_d)d|j D|_i|j|j|_h|j|j |j|_d*d*d+d+d,|_d-d-d.d.d/d/d0d0d1|_tt|jt|j|_d2d3id3d4|_hd5|_td6d7|j|j|j|j |j |j |j|j|j|j|jf D|_d8d9h|_dS):N>oohzerocSsi|]\}}||qSr).0inamerrW/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/whisper_normalizer/english.py %sz4EnglishNumberNormalizer.__init__..)onetwothreefourfivesixseveneightnineteneleventwelvethirteenfourteenfifteensixteen seventeeneighteennineteenr )startcSs*i|]\}}|dkr dn|d|dfqS)rsixessrrrvaluerrrr@s)rth)r st)nd)rd)r0) r0)zerothfirstsecondthirdfifthtwelfthcSsDi|]\}}|dkr|dkr|dkr||drdnd|dfqS)r4r6r7thr0)endswithr.rrrrKs (2<FPZ)twentythirtyfortyfiftysixtyseventyeightyninetycS"i|] \}}|dd|dfqS)yiesr-replacer.rrrr]scSrQ)rRiethr0rTr.rrrr`sdi@Biʚ;lJ)lI5lNZol@=7M.clB3v^!<lP~cegl73Me'l(l F3YHqS) hundredthousandmillionbilliontrillion quadrillion quintillion sextillion septillion octillion nonillion decillioncSi|] \}}|d|dfqSr-rr.rrrrtcSre)r0rr.rrrrwrg-+)minusnegativepluspositive£u€$¢)poundpoundseuroeurosdollardollarscentcentsrw%)perpercent>andpointdoubletriplecSsg|] }|D]}|qqSrr)rmappingkeyrrr s z4EnglishNumberNormalizer.__init__..rones)super__init__zeros enumerateritems ones_plural ones_ordinal ones_suffixedtens tens_plural tens_ordinal tens_suffixed multipliersmultipliers_pluralmultipliers_ordinalmultipliers_suffixeddecimalspreceding_prefixersfollowing_prefixerssetlistvaluesprefixes suffixersspecialswords literal_wordsself __class__rrr!s       z EnglishNumberNormalizer.__init__rreturnc#sddd}dtfdd}dtttfffdd }t|dkr$dStdg|dgd D]\}}}|r:d}q/|duoCtd |}|d|jv} | rS|d dn|} td | r|| } | duseJdurttr| d r|tt|q/|V| r|dn| j d kr| j q/| q/||j vrdur|V||Vq/||j vrtpd dq/||jvr'|j|} dur| q/tts||jvr||jvr| dkrddksJddt| q/tt| q/| dkrddkr | 7q/tt| q/ddkr| 7q/tt| q/||jvr|j|\} } durC|t| | VnttsO||jvr||jvrs| dkrsddkscJ|ddt| | VnT|tt| | VnF| dkrddkr|t| | Vn.|tt| | Vn ddkr|t| | Vn |tt| | Vdq/||jvr|j|}dur|q/ttrtt|q/ddkr|7q/tt|q/||jvrR|j|\}} dur|t|| Vq/ttr1|tt|| Vq/ddkrD|t|| Vq/|tt|| Vq/||jvr|j|}dure|q/ttspdkr|} | dur}| |nd}| dur|j d kr|j q/|V|q/dd}d}|||q/||jvr|j|\}} dur|t|| VnSttr|} | dur| |nd}| dur|j d kr|t|j | Vn(|V|t|| Vndd}d}||||t| Vdq/||jvr>dur)|V||j vs2|r8|j|q/||Vq/||jvrZdurT|j||Vq/||Vq/||jvrdur|j|} t| tr|| vr|t| |Vd}q/|V||Vq/|t| Vq/||Vq/||jvr=||j vr|sdur|V||Vq/|dkr||jvr׈dur|V||Vq/|dks|dkr||jvs||j vr|dkrdnd }|j|d} tpd t| |d}q/dur|V||Vq/|dkr6||jvs,|r5tp1d d q/td|td|durP|VdSdS)NFr-cSs zt|WStyYdSwN)r ValueErrorrfrrr to_fractions   z:EnglishNumberNormalizer.process_words..to_fractionresultcs$t|}dur |}dd|Sr)str)rprefixr/rroutputs z5EnglishNumberNormalizer.process_words..outputrr4z ^\d+(\.\d+)?$r .0 rWrXTr|r~rr2r}zUnexpected token: )rr intlenr rematchr isinstancer@ denominator numeratorrrrrrrrrrrrdictrgetrr)rrskiprrprevcurrentnextnext_is_numeric has_prefixcurrent_without_prefixfrsuffixr multiplierpbeforeresidualrepeatsrrr process_wordssJ "                                                                    z%EnglishNumberNormalizer.process_wordsr-cCsg}td|}t|D]=\}}t|dkrq |t|dkr'||q |||jddd}||jvs>||jvrD|dq |dq d |}t d d |}t d d |}t d d|}|S)Nz\band\s+a\s+half\brr r2)maxsplitrz point fivez and a half z([a-z])([0-9])z\1 \2z([0-9])([a-z])z([0-9])\s+(st|nd|rd|th|s)\b\1\2) rsplitrrstripappendrsplitrrjoinsub)rr-resultssegmentsrsegment last_wordrrr preprocesss"      z"EnglishNumberNormalizer.preprocesscCsJdtfdd}dtfdd}td||}td||}tdd |}|S) NmcSsRz|d}|d}t|d}||d|dWSty(|jYSw)Nr r2r4r02d)grouprrstring)rcurrencyintegerrxrrr combine_centss    z:EnglishNumberNormalizer.postprocess..combine_centscSs0z dt|dWSty|jYSw)Nrpr )rrrr)rrrr extract_centss   z:EnglishNumberNormalizer.postprocess..extract_centsu,([€£$])([0-9]+) (?:and )?¢([0-9]{1,2})\bu[€£$]0.([0-9]{1,2})\bz \b1(s?)\bzone\1)rrr)rr-rrrrr postprocesss  z#EnglishNumberNormalizer.postprocesscCs6||}ddd||D}||}|S)Nrcss|] }|dur|VqdSrrrwordrrr sz3EnglishNumberNormalizer.__call__..)rrrrrrr-rrr__call__s  z EnglishNumberNormalizer.__call__) __name__ __module__ __qualname____doc__rrrrrrrr __classcell__rrrrrs `rc@&eZdZdZddZdefddZdS)rz Applies British-American spelling mappings as listed in [1]. [1] https://web.archive.org/web/20230326222449/https://www.tysto.com/uk-us-spelling-list.html cCsJtdd}t|d}t||_WddS1swYdS)Nwhisper_normalizerznormalizers/english.jsonr)r joinpathopenjsonloadr)renglish_json_pathenglish_normalization_dictrrrrs  "z"EnglishSpellingNormalizer.__init__r-csdfdd|DS)Nrc3s|] }j||VqdSr)rrrrrrrsz5EnglishSpellingNormalizer.__call__..)rrrrrrrsz"EnglishSpellingNormalizer.__call__Nrrrrrrrrrrrrsrc@r)rusApplies all the rules for normalizing English text as mentioned in OpenAI whisper paper. As per the text normalization/standardization approach Appendix Section C pp.21 the paper [Robust Speech Recognition via Large-Scale Weak Supervision](https://cdn.openai.com/papers/whisper.pdf). The `EnglishTextNormalizer` does the following functionality: 1. Remove any phrases between matching brackets ([, ]). 2. Remove any phrases between matching parentheses ((, )). 3. Remove any of the following words: hmm, mm, mhm, mmm, uh, um 4. Remove whitespace characters that comes before an apostrophe ’ 5. Convert standard or informal contracted forms of English into the original form. 6. Remove commas (,) between digits 7. Remove periods (.) not followed by numbers 8. Remove symbols as well as diacritics from the text, where symbols are the characters with the Unicode category starting with M, S, or P, except period, percent, and currency symbols that may be detected in the next step. 9. Detect any numeric expressions of numbers and currencies and replace with a form using Arabic numbers, e.g. “Ten thousand dollars” → “$10000”. 10. Convert British spellings into American spellings. 11. Remove remaining symbols that are not part of any numeric expressions. 12. Replace any successive whitespace characters with a space. cCsdd|_idddddddd d d d d dddddddddddddddddddd d!d"id#d$d%d&d'd(d)d*d+d,d-d.d/d0d1d2d3d4d5d6d7d8d9d:d;d<d=d>d?d@dAdBdCdDidEdFdGdHdIdJdKdLdMdNdOdPdQdRdSdTdUdVdWdXdYdZd[d\d]d^d_d`dadbdcddded\dfdgdh|_t|_t|_dS)iNz\b(hmm|mm|mhm|mmm|uh|um)\bz \bwon't\bzwill notz \bcan't\bzcan notz \blet's\bzlet usz \bain't\baintz \by'all\bzyou allz \bwanna\bzwant toz \bkinda\bzkind ofz \bsorta\bzsort ofz \bdunno\bz do not knowz \bgotta\bzgot toz \bgonna\bzgoing toz\bi'ma\bz i am going toz\bimma\bz \bwoulda\bz would havez \bcoulda\bz could havez \bshoulda\bz should havez \bcause\bbecausez \bma'am\bmadamz\bmr\bzmister z\bmrs\bzmissus z\bst\bzsaint z\bdr\bzdoctor z\bprof\bz professor z\bcapt\bzcaptain z\bgov\bz governor z\bald\bz alderman z\bgen\bzgeneral z\bsen\bzsenator z\brep\bzrepresentative z\bpres\bz president z\brev\bz reverend z\bhon\bz honorable z\basst\bz assistant z \bassoc\bz associate z\blt\bz lieutenant z\bcol\bzcolonel z\bjr\bzjunior z\bsr\bzsenior z\besq\bzesquire z 'd been\bz had beenz 's been\bz has beenz 'd gone\bz had gonez 's gone\bz has gonez 'd done\bz had donez's got\bz has gotzn't\bz notz're\bz arez's\bz isz'd\bz wouldz'll\bz willz't\bz havez am)z've\bz'm\b)ignore_patterns replacersrstandardize_numbersrstandardize_spellingsrrrrrs      !"#$%&'()+,-./02345678 ; zEnglishTextNormalizer.__init__r-cCs|}tdd|}tdd|}t|jd|}tdd|}|jD] \}}t|||}q&tdd|}tdd |}t|d d }||}||}td d |}td d|}tdd|}|S)Nz[<\[][^>\]]*[>\]]rz \(([^)]+?)\)z\s+''z (\d),(\d)rz \.([^0-9]|$)z \1u .%$¢€£)keepu[.$¢€£]([^0-9])z ([^0-9])%z\1 z\s+r) lowerrrrrrrrr)rr-pattern replacementrrrr1s    zEnglishTextNormalizer.__call__Nrrrrrrs@r)r__all__rosr fractionsrtypingrrrr r importlib.resourcesr more_itertoolsr basicrrrrrrrrs     9