o i @s&dZddlZddlZddlmZgdZgdZgdZddd d d d d dddd ZddddddddZ gdZ e dZ de defddZde defd d!Zdd#edefd$d%Zd&ede fd'd(Ze d)Ze d*ejZe d+Ze d,Ze d-Ze d.Ze d/Ze d0Ze d1ejZe d2Ze d3Ze d4ejZe d5Z e d6Z!e d7ejZ"e d8Z#e d9Z$e d:Z%e d;Z&e d<Z'de defd=d>Z(d?edefd@dAZ)d?edefdBdCZ*d?edefdDdEZ+d?edefdFdGZ,d?edefdHdIZ-d?edefdJdKZ.d?edefdLdMZ/dd?edOe0defdPdQZ1d?edefdRdSZ2d?edefdTdUZ3d?edefdVdWZ4d?edefdXdYZ5d?edefdZd[Z6d?edefd\d]Z7d?edefd^d_Z8dd?ed`e0defdadbZ9d?edefdcddZ:dd?edfedefdgdhZ;dd?edfedefdidjZdd?edfedefdodpZ?d?edefdqdrZ@d?edefdsdtZAdd?edvedefdwdxZBd?edefdydzZCd?edefd{d|ZDdd?ed}eeEdefd~dZFGdddZGeHdkreGZIgdZJeKdeKdeKdeJD]\ZLZMeKdeLdeKdeMeKdeIeMq eKdeKd!eKddD]ZNeKdeNddeeNq9eKdeKd%eKddD]ZOeKdeOdeeOqXeKdeKdeKdeGdNdZPdD]ZMeKdeMeKdePeMq{dSdS)zU text_preprocessing.py A comprehensive text preprocessing library for NLP pipelines. N)Optional)onetwothreefourfivesixseveneightnineteneleventwelvethirteenfourteenfifteensixteen seventeeneighteennineteen) rrtwentythirtyfortyfiftysixtyseventyeightyninety)rthousandmillionbilliontrillionfirstsecondthirdfourthfifthsixthseventheighthninthtwelfth) rrrrrr r r r rdollareuropoundyenrupeewonbitcoin)$u€£¥u₹u₩u₿) )M)iCM)D)iCD)dC)ZXC)2L)(XL) X) IX)V)IV)Iz<\b(M{0,4})(CM|CD|D?C{0,3})(XC|XL|L?X{0,3})(IX|IV|V?I{0,3})\bnreturncCs|dkrdSg}|d}|d}|r|t|d|dkr*|r)|t|nt|d}t|d}||rA|d|n|d|S) u*Convert a number 0–999 to English words.rrr= hundredrE- )append_ONES_TENSjoin)rOpartshundreds remainder tens_word ones_wordr^H/home/ubuntu/.local/lib/python3.10/site-packages/kittentts/preprocess.py_three_digits_to_words,s   r`cCst|ts t|}|dkrdS|dkrdt| Sd|kr%dkrBnn|ddkrB|ddkrB|d}|dkrBt|dSg}ttD](\}}|d}|rft|}||rc|d |n||d}|dkrpnqHd t |S) u Convert an integer to its English word representation. Examples: 1200 → "twelve hundred" 1000 → "one thousand" 1_000_000 → "one million" -42 → "negative forty-two" 0 → "zero" rzero negative r=i'r7rRrQrT) isinstanceintnumber_to_wordsrV enumerate_SCALEr`rUstriprXreversed)rOrZrYiscalechunk chunk_wordsr^r^r_re?s* 0 repoint decimal_sepc st|tr|n|}|d}|r|dd}d|vrM|dd\}}|r+tt|nd}dgtdddfdd|D}|d|d|}ntt|}|rZd |S|S) uu Convert a float (or numeric string) to words, reading decimal digits individually. Accepts a string to preserve trailing zeros (e.g. "1.50" → "one point five zero"). Examples: 3.14 → "three point one four" -0.5 → "negative zero point five" "3.10" → "three point one zero" 1.007 → "one point zero zero seven" rSrMN.rarTc3s|] }t|VqdSN)rd).0d digit_mapr^r_ zsz!float_to_words..rb)rcstr startswithsplitrerdrVrX) valuerotextnegativeint_partdec_part int_words dec_wordsresultr^rtr_float_to_wordses   rscCsTdddddddd}d }d }t|D]}||}|||kr!|n| 7}|}q|S) z-Convert a Roman numeral string to an integer.rMrIrErAr=r:r7)rNrJrFrBr>r;r8r)riupper)rvalrprevchcurrr^r^r_ roman_to_intsrzhttps?://\S+|www\.\S+z\b[\w.+-]+@[\w-]+\.[a-z]{2,}\bz#\w+z@\w+z<[^>]+>z![^\w\s.,?!;:\-\u2014\u2013\u2026]z\s+z(?._replace)reMatchrw _RE_ORDINALsubr{rr^r^r_expand_ordinalss rcCr)u Expand percentage expressions. Examples: "50% off" → "fifty percent off" "3.5% rate" → "three point five percent rate" "-2% change" → "negative two percent change" rrPcSs:|ddd}d|vrtt|dStt|dS)NrM,rrpz percent)rreplacerfloatrerdrrawr^r^r_rsz$expand_percentages.._replace)rrrw _RE_PERCENTrrr^r^r_expand_percentagess rc2ddddddtjdtffdd }t||S) up Expand currency amounts, including optional scale suffixes. Examples: "$100" → "one hundred dollars" "€1,200.50" → "twelve hundred euros and fifty cents" "£9.99" → "nine pounds and ninety-nine cents" "$85K" → "eighty five thousand dollars" "$2.5M" → "two point five million dollars" rr r!r"Kr8BTrrPcs@|d}|ddd}|d}t|d}|r?|}d|vr't|ntt|}|d|d||r9dndSd|vr|dd\}}t|dd dd } tt|} |rg| d|dn| } | rt| } | d | d | dkrzdnd7} | St|} t| }|r|d|| dkr|rdndn|} | S) NrMrrrrprTr0z and z cent) rr_CURRENCY_SYMBOLSgetrrerdrhryljust)rsymbolr scale_suffixunit scale_wordnumr}r~dec_valrrcentsrwords _scale_mapr^r_rs*   $  *z!expand_currency.._replace)rrrw _RE_CURRENCYrrr^rr_expand_currencys  rcCr)u Expand time expressions. Examples: "3:30pm" → "three thirty pm" "14:00" → "fourteen hundred" "9:05 AM" → "nine oh five am" "12:00pm" → "twelve pm" rrPcSst|d}t|d}|drd|dnd}t|}|dkr8|ds2|d|S||S|dkrG|d t||S|dt||S) NrMrrKrTrrrQrEz oh )rdrlowerre)rrminssuffixh_wordsr^r^r_r<s $zexpand_time.._replace)rrrw_RE_TIMErrr^r^r_ expand_time2s rcCr)u Expand numeric ranges. Examples: "10-20 items" → "ten to twenty items" "pages 100-200" → "pages one hundred to two hundred" "2020-2024" → "twenty twenty to twenty twenty-four" rrPcSs2tt|d}tt|d}|d|S)NrMrz to )rerdr)rlohir^r^r_rSszexpand_ranges.._replace)rrrw _RE_RANGErrr^r^r_ expand_rangesJs rcCstdd|S)u Normalise version/model names that use letter-hyphen-number patterns, so the number is not misread as negative. Examples: "GPT-3" → "GPT 3" "gpt-3.5" → "gpt 3.5" "GPL-3" → "GPL 3" "Python-3.10"→ "Python 3.10" "v2.0" stays as "v2.0" (no hyphen — handled by number replacement) "IPv6" stays as "IPv6" cSs|dd|dS)NrMrTr)rrr^r^r_gsz$expand_model_names..) _RE_MODEL_VERrr{r^r^r_expand_model_namesZs rcsiddddddddd d d d d ddddddddddddddddddd d!d"d#d#d$d$d%d&tjd'tffd(d) }t||S)*u Expand common measurement units glued to numbers. Examples: "100km" → "one hundred kilometers" "50kg" → "fifty kilograms" "25°C" → "twenty-five degrees Celsius" "5GB" → "five gigabytes" km kilometerskg kilogramsmg milligramsml millilitersgb gigabytesmb megabyteskb kilobytestb terabyteshzhertzkhz kilohertzmhz megahertzghz gigahertzmphzmiles per hourkphzkilometers per hourms millisecondsns nanosecondsuµs microsecondszdegrees Celsiuszdegrees Fahrenheit)u°cuc°u°fuf°rrPcsX|d}|d}||d}d|vrtt|ntt|}|d|SNrMrrprT)rrrrrrerd)rrrexpandedr _unit_mapr^r_r~s  zexpand_units.._replace)rrrw_RE_UNITrrr^rr_ expand_unitsjsP  rT context_wordscs4tdtjdtjdtffdd }t|S)ux Expand Roman numerals that appear as standalone tokens (optionally only when preceded by a title-like word to avoid false positives). Examples: "World War II" → "World War two" "Chapter IV" → "Chapter four" "Louis XIV" → "Louis fourteen" "mix I with V" → left unchanged (ambiguous single letters) z\b(war|chapter|part|volume|act|scene|book|section|article|king|queen|pope|louis|henry|edward|george|william|james|phase|round|level|stage|class|type|version|episode|season)\brrPcs|d}|s |St|dkr+|dvr+|}td|d|}|s+|Szt|}|dkr7|WSt|WStyF|YSw)NrrMIVX) rrhrstartmaxsearchrre Exception)rromanr precedingr _TITLE_WORDSr{r^r_rs    z'expand_roman_numerals.._replace)rcompile IGNORECASErrw _RE_ROMANr)r{rrr^rr_expand_roman_numeralss   rcCstdd|}td|S)u Normalise bare leading-decimal floats so the number pipeline handles them. Examples: ".5 teaspoons" → "0.5 teaspoons" "-.25 adjustment" → "-0.25 adjustment" z(?0.\2z0.\1)rr _RE_LEAD_DECrr^r^r_normalize_leading_decimalss rcCr)u Expand scientific-notation numbers to spoken form. Examples: "1e-4" → "one times ten to the negative four" "2.5e10" → "two point five times ten to the ten" "6.022E23"→ "six point zero two two times ten to the twenty three" rrPcSsb|d}t|d}d|vrt|ntt|}tt|}|dkr&dnd}|d||S)NrMrrprrbrz times ten to the )rrdrreabs)r coeff_rawexp coeff_words exp_wordssignr^r^r_rs  z,expand_scientific_notation.._replace)rrrw_RE_SCIrrr^r^r_expand_scientific_notations rcr) uT Expand standalone uppercase scale suffixes attached to numbers. Examples: "7B parameters" → "seven billion parameters" "340M model" → "three hundred forty million model" "1.5K salary" → "one point five thousand salary" "$100K budget" → "$100K budget" (currency handled upstream) rr r!r"rrrPcsJ|d}|d}||}d|vrt|ntt|}|d|Sr)rrrrerd)rrrrr_mapr^r_rs   z'expand_scale_suffixes.._replace)rrrw _RE_SCALErrr^rr_expand_scale_suffixess  rcCr)u Expand simple numeric fractions. Examples: "1/2 cup" → "one half cup" "3/4 mile" → "three quarters mile" "2/3 done" → "two thirds done" "5/8 inch" → "five eighths inch" rrPcSst|d}t|d}|dkr|St|}|dkr'|dkr$dnd}n|dkr4|dkr1dnd}n t|}|dkr@|d 7}|d |S) NrMrrhalfhalvesrKquarterquartersrrT)rdrrer)rrden num_words denom_wordr^r^r_rsz"expand_fractions.._replace)rrrw _RE_FRACTIONrrr^r^r_expand_fractionss rc s>ddddddddd d d d tjd tffdd }t||S)u Expand decade expressions to words. Examples: "the 80s" → "the eighties" "the 1980s" → "the nineteen eighties" "the 2020s" → "the twenty twenties" "'90s music" → "nineties music" rZtenstwentiesthirtiesfortiesfiftiessixties seventieseightiesnineties) rrMrrrKrIrGrrPcsHt|d}|d}|d}|dkr|S|d}t|d|S)NrMrErrT)rdrrre)rr decade_digit decade_word century_part _decade_mapr^r_rs z expand_decades.._replace)rrrw _RE_DECADErrr^r-r_expand_decadess  r0c sVddddddddd d d d td tffdd dtjd tffdd }td||S)u Expand IPv4 addresses to spoken digits per octet. Examples: "192.168.1.1" → "one nine two dot one six eight dot one dot one" "10.0.0.1" → "one zero dot zero dot zero dot one" rarrrrrr r r r r123456789rrPcdfdd|DS)NrTc3|]}|VqdSrqr^rrc_dr^r_rv)z6expand_ip_addresses.._octet..rXrr?r^r__octet(z#expand_ip_addresses.._octetrcsdfdd|DS)Nz dot c3|]}|VqdSrqr^rrgrDr^r_rv,rAz8expand_ip_addresses.._replace..)rXgroupsrrIr^r_r+sz%expand_ip_addresses.._replacez.\b(\d{1,3})\.(\d{1,3})\.(\d{1,3})\.(\d{1,3})\b)rwrrrrr^)r@rDr_expand_ip_addressess  rKc sddddddddd d d d td tffdd d tffdd tdfdd|}tdfdd|}tdfdd|}|S)uR Expand US phone numbers to spoken digits before range expansion claims the hyphens. Examples: "555-1234" → "five five five one two three four" "555-123-4567" → "five five five one two three four five six seven" "1-800-555-0199" → "one eight zero zero five five five zero one nine nine" rarrrrrr r r r r1rrPcr;)NrTc3r<rqr^r=r?r^r_rv>rAz8expand_phone_numbers.._digits..rBrCr?r^r__digits=rEz%expand_phone_numbers.._digitscr;)NrTc3rFrqr^rGrLr^r_rvArAz6expand_phone_numbers.._join..rBrJrMr^r__join@rEz#expand_phone_numbers.._joinz;(?.z1(?._replace)rrrw _RE_NUMBERr)r{rTrr^rWr_replace_numbersTs rYcCs|S)zConvert text to lowercase.rrr^r^r_ to_lowercasejsr[r replacementcCt||S)zRemove URLs from text.)_RE_URLrrhr{r\r^r^r_ remove_urlsor`cCr])z!Remove email addresses from text.) _RE_EMAILrrhr_r^r^r_ remove_emailstrarccC td|S)zStrip HTML tags from text.rT)_RE_HTMLrrr^r^r_remove_html_tagsy rfcC t||S)z&Remove hashtags (e.g. #NLP) from text.) _RE_HASHTAGrr_r^r^r_remove_hashtags~rgrjcCrh)zRemove @mentions from text.) _RE_MENTIONrr_r^r^r_remove_mentionsrgrlcCrd)zXRemove non-prosodic punctuation, keeping marks that affect speech rhythm and intonation.rT) _RE_PUNCTrrr^r^r_remove_punctuationrgrncCstd|S)zKCollapse multiple whitespace characters into a single space and strip ends.rT) _RE_SPACESrrhrr^r^r_remove_extra_whitespacerarpNFCformcCrh)z7Normalize unicode characters (NFC, NFD, NFKC, or NFKD).) unicodedata normalize)r{rrr^r^r_normalize_unicodergrucCs td|}ddd|DS)z3Remove diacritical marks (accents) from characters.NFDrcss"|] }t|dkr|VqdS)MnN)rscategoryr=r^r^r_rvs z!remove_accents..)rsrtrX)r{nfkdr^r^r_remove_accentss rzc CsHddddddddd d d d d }|D]\}}tj|||tjd}q|S)u Expand common English contractions. Examples: "don't" → "do not" "they're" → "they are" "I've" → "I have" cannotzwill notz shall notzis notzlet usz\1 notz\1 arez\1 havez\1 willz\1 wouldz\1 amzit is) z \bcan't\bz \bwon't\bz \bshan't\bz \bain't\bz \blet's\bz \b(\w+)n't\bz \b(\w+)'re\bz \b(\w+)'ve\bz \b(\w+)'ll\bz \b(\w+)'d\bz \b(\w+)'m\bz\bit's\b)flags)rrrr)r{ contractionspatternr\r^r^r_expand_contractionss  r stopwordscs0durhd|}dfdd|DS)z Remove stopwords from text. Args: stopwords: Set of words to remove. Uses a built-in English set if None. N>6arjanatbebydoheinisitmemyofonortoweandarebutdidforhadhasherhimitsmayourshethewasyoubeendoesfromhavethatthemtheythiswerewillwithyourbeingcouldmighttheirthesethosewouldshouldrTc3s |] }|vr|VqdSrqrZ)rrrrr^r_rvsz#remove_stopwords..)ryrX)r{rtokensr^rr_remove_stopwordssrc=@seZdZdZ                              d+dedededed ed ed ed ed ededededededededededededededededededeed ed!ed"efsz-TextPreprocessor.__init__..)localsrconfig _stopwords)rrrYrTrrrrrrrrrrrr0rSrKrrr`rcrrjrlrnrrrurzrpr^r^r___init__s! zTextPreprocessor.__init__r{rPcCs ||Srq)process)rr{r^r^r___call__s zTextPreprocessor.__call__cCs|j}|dr t|}|drt|}|drt|}|dr#t|}|dr+t|}|dr3t|}|dr;t|}|drCt|}|d rKt |}|d rSt |}|d r[t |}|d rct |}|d rkt |}|drst|}|dr{t|}|drt|}|drt|}|drt|}|drt|}|drt|}|drt|}|drt|}|drt||dd}|drt|}|drt|}|drt|}|drt||j}|drt|}|S)Nrurr`rcrjrlrrKrrrrrrrrrr0rSrrrrYrTrWrzrnrrrp)rrurfr`rcrjrlrrKrrrrrrrrrr0rSrrrrYrzrnr[rrrp)rr{cfgr^r^r_rst zTextPreprocessor.process)TTTTTTTTTTTTTTTTTTFTTTFFTFNTFT) __name__ __module__ __qualname____doc__boolrsetrrwrrr^r^r^r_rs      $r__main__)D)z Plain integerz(There are 1200 students and 42 teachers.)z Large numberz/The project costs $1,000,000 and took 365 days.)zNegative numberz,Temperature dropped to -5 degrees overnight.)FloatzPi is approximately 3.14159.)zFloat trailing zerozThe voltage is 1.50 volts.)zLeading decimalz-Add .5 teaspoons of salt and .25 cup of milk.)zNegative leading decimalzA -.05 correction was applied.)Zeroz%There were 0 errors and 0.0 warnings.)zComma thousandsz The population is 7,900,000,000.)zScientific e-notationz)Learning rate is 1e-4, weight decay 1e-5.)zScientific capital EzAvogadro's number is 6.022E23.)zScientific large expzThe signal is 2.5e10 Hz.)zModel params Bz2We trained a 7B parameter model and a 13B variant.)zModel params Mz#The 340M model beat the 7B on MMLU.)zScale suffix KzThe salary was $85K per year.)z Dollar amountzA coffee costs $4.99 here.)z Euro amountuRent is €1,200 per month.)zPound with centsuThe book is £9.99.) Percentagez$Inflation rose by 3.5% last quarter.)zNegative percentagezStocks fell -2% today.)zOrdinals 1st/2nd/3rdz)She finished 1st, he came 2nd, I was 3rd.)z Ordinal 21stz0It's the 21st century and the 100th anniversary.)z Ordinal 42ndzHe ran his 42nd marathon.)z Ordinal 33rdzOn the 33rd floor.)HalfzCut the recipe in 1/2.)Quartersz.Add 3/4 cup of sugar and 1/4 teaspoon of salt.)Thirdsz&The team completed 2/3 of the project.)Eighthsz!The pipe is 5/8 inch in diameter.)z 12-hour timezThe meeting starts at 3:30pm.)z 24-hour timezDeparture at 14:00.)z Time with ohzAlarm set for 9:05 AM.)MidnightzThe server restarts at 0:00.)z Bare decadezThe 80s music scene was iconic.)z Full decadez&She grew up listening to 1990s grunge.)2000szThe 2000s brought social media.)2020szAI took off in the 2020s.)zApostrophe decadez&Born in the '90s, raised on 2000s pop.)z Numeric rangezRead pages 10-20 for homework.)z Year rangezThe war lasted from 2020-2024.)zTemperature rangezStore between 5-10 degrees.)zGPT-3zgpt-3 is pretty sick.)zGPT-3.5z$They upgraded to GPT-3.5 last month.)z GPL-3 licensez%This project is licensed under GPL-3.)zPython versionzRequires Python-3.10 or higher.)zMultiple versionsz'Both CUDA-11 and CUDA-12 are supported.)DistancezThe trail is 42km long.)WeightzEach package weighs 500kg.)uTemperature °CuWater boils at 100°C.)z Data size GBzDownload the 2.5GB model file.)z Frequency GHzzThe CPU runs at 3.6GHz.)z Latency mszAverage latency is 12ms.)z HTML tagsz%Hello World! It's a great day.)z URL and emailz5Visit https://example.com or email hello@example.com.)zHashtags and mentionsz#NLP @user great post!) Contractionsz3I don't know, won't you help? They've already left.)z Ain't / let'sz(Ain't no mountain high enough. Let's go!)z Score / ratiozThe final score was 3:0.)z Aspect ratiozThe display is 16:9.)z IP addressz.Connect to server at 192.168.1.1 on port 8080.)z Phone numberz&Call us at 555-1234 or 1-800-555-0199.)zNegative vs. hyphenz(On a scale of -10 to 10, she rated it 8.)EllipsiszHe paused... then spoke.)zEm dash numberu)The result — 42 — surprised everyone.)zResearch abstractzEWe trained a 7B parameter model for 100 epochs at 1e-4 learning rate.)z GPT benchmarku>GPT-4 scored 90% on the benchmark — 15% better than GPT-3.5.)z News headlinez.Fed raises rates by 0.25%, S&P 500 drops 1.2%.)z Startup pitchzEWe raised $2.5M in seed funding and are growing 20% month-over-month.)z Tech speczBThe M3 chip runs at 4.05GHz with a 40M transistor GPU and 8GB RAM.zF======================================================================zTextPreprocessor Demoz []z IN : z OUT: zG ======================================================================) rrM rRcr=r7ii:i@Biiɚ;z z>15,u → )gQ @gg)\Ò@g@g&1?gMbP?zexpand_roman_numerals (opt-in))r)zWorld War II ended in 1945.zChapter IV begins here.zLouis XIV was king.)rn)T)r)rqrq)QrrrstypingrrVrWrgrr_ROMANrrrdrwr`rerrr^rrbrirkrermrorXrrrrrrrrrrr/rrrrrrrrrrrrrrrr0rKrSrYr[r`rcrfrjrlrnrprurzrrrrrppcasesprintlabelr{rOfpp_romanr^r^r^r_s &                *( # W  r