o Xεiu@svdZddlZddlZddlmZmZmZmZddlm Z ddddZ Gdddej Z d d Z ed kr9edSdS) Tests for TextProcessorN)Sentence TextProcessorTextProcessorSettingsWord print_graphF) explicit_langphonemesposc@s0eZdZdZddZddZddZdd Zd d Zd d Z ddZ ddZ ddZ ddZ ddZddZddZddZddZd d!Zd"d#Zd$d%Zd&d'Zd(d)Zd*d+Zd,d-Zd.d/Zd0d1Zd2d3Zd4d5Zd6d7Zd8d9Zd:d;Z dd?Z"d@dAZ#dBdCZ$dDdEZ%dFdGZ&dHdIZ'dJS)KTextProcessorTestCaserc Csrt}|d\}}t|j||fit}||tdddddtdddddtd dd d dtd dd ddgdS)zText whitespace preservationThis is a test rThisThis idxsent_idxtext text_with_wsiszis aza testztest Nrlistwords WORDS_KWARGS assertEqualrself processorgraphrootrr%M/home/ubuntu/.local/lib/python3.10/site-packages/tests/test_text_processor.pytest_whitespaces z%TextProcessorTestCase.test_whitespacec Csvtdd}|d\}}t|j||fit}||tdddddtdddddtd dd d dtd dd d dgd S)z)Test disabling of whitespace preservationF)keep_whitespacer rrrrrrrrrNrr r%r%r&test_no_whitespace s  z(TextProcessorTestCase.test_no_whitespacecCstddhddhdhdhd}|d\}}t|j||fit}||tddd d d td dddd dtddddd tddddd dtddddd dtddddd tddddd dtddddd tddddd dtddddd dg dS)z/Test splitting of punctuation from around words"«»,.)begin_punctuationsend_punctuations minor_breaks major_breaksuThis «is», a "test".rrrrrTrrrris_punctuationrrrz, rrrris_minor_breakra r rrrris_major_breakNrr r%r%r&test_punctuation1sD       z&TextProcessorTestCase.test_punctuationcCstdhdhdhd}|d\}}t|j||fit}||tdddddtd dddd d td dd d dtddddd dtddddd d tdd dddtd d dddtd d ddd dgdS)z Test break inside of punctuationr*r.)r/r0r2zTest "one." Test two.rTestzTest rrTr3ronerr>r5z" twoNrr r%r%r&!test_punctuation_with_inner_breakYs2      z7TextProcessorTestCase.test_punctuation_with_inner_breakcCstdhdhgdd}|d\}}t|j||fit}||tdddddtd ddd d d td ddddtdddddtdddddtdddddtddddd dgdS)z+Test regex replacements during tokenizationr-r.))z\B'r*)z'\Br*)z[\<\>\(\)\[\]"]+)r1r2 replacementsz#"This," [is] (test) 'sentence.'rrrr, Tr6rris rrr9r5rtest r8sentencer:r>Nrr r%r%r&test_replacementszs,   z'TextProcessorTestCase.test_replacementscCs tdhddhddddd}|d \}}t|j||fit}||td d d d d td d ddddtd d ddd td d ddd tdd ddd tdd ddd tdd ddddtdd ddd td d d!d"d td#d d$d%d td&d d'd(d td)d d*d*d td+d ddddg d,S)-z8Test expansion of abbreviations (with case preservation)r-r.?z\1octorz\1isterz\1treet)z ^([dD])r\.z ^([mM])r\.z ^([sS])t\.)r1r2 abbreviationsz*Mr.? I'm just a dr., on this St. at least.rMisterrrz? Tr>zI'mzI'm justzjust rrr9rdoctorr5rGr6r8onzon r:thiszthis r;StreetzStreet r<atzat r=least Nrr r%r%r&test_abbreviationssB    z(TextProcessorTestCase.test_abbreviationscCs6tddhd}|d\}}t|j||fit}||tdddddtd dd d dtd ddd d dtdd dddtd d d d dtd d ddd dgt|j||fit}||tddddtdddddtd dd d dtd ddd d dgdtd dddtdd dddtd d d d dtd d ddd dgdgdS)zTest sentence breakr.!r2z"First sentence. Second sentence! rFirstzFirst rrrJr. Tr>SecondSecond z! zFirst sentence.zFirst sentence. zFirst sentence)rrr text_spokenrzSecond sentence!zSecond sentence! zSecond sentenceN)rrrrrr sentencesr)r!r"r#r$rr_r%r%r&test_multiple_sentencesst   z-TextProcessorTestCase.test_multiple_sentencesc Cs~t}|ddd\}}t|j||fit}||tddddddtdddd d dtdddd d dtdddd d dgd S) zTest paragraph indexz<

First paragraph

Second paragraph

TssmlrrZFirst )rrpar_idxrrr paragraphr\r]Nrr r%r%r&test_multiple_paragraphss6  z.TextProcessorTestCase.test_multiple_paragraphscCstddhd}|ddd\}}t|j||fit}||tdddd d td dd d d td dddddtddddd tddd d d tddddddgdS)z,Test in SSML for avoiding sentence breakr.rXrYz'First sentence. Second sentence!TrarrZrcrrrJrr[r>rr\r]r5r8Nrr r%r%r&test_explicit_sentence(s"  z,TextProcessorTestCase.test_explicit_sentencec Cstdhd}|d\}}t|j||fit}||tdddddtddddd d td dd d dtdddddtdddddgdS)zTest minor (phrase) breakr-)r1zthis, is a testrrRrrrGTr6rrrHrrr9r5rNrr r%r%r&test_minor_breaks?s   z'TextProcessorTestCase.test_minor_breaksc Cs\tdhd}|d\}}t|j||fit}||tdddddtddd d dgd S) zTest inner-word break-) word_breaksz ninety-ninerninetyninety rrnineNrr r%r%r&test_word_breaksSs  z&TextProcessorTestCase.test_word_breakscCstdd}|ddd\}}t|j||fit}t||||tddddd td dd d d td dddd tddddd tddddd tddddd tddddd gdS)z%Test interpret-as="spell-out" in SSMLen_US default_langz1test123Trartzt rreze rszs rr5rBone r8rCtwo r:threeN)rrrrrrrr r%r%r&test_spell_outbs"   z$TextProcessorTestCase.test_spell_outc Cstdhddtd}|d\}}t|j||fit}||tddddd td dddd td dd d d td dddddgdS)zTest initialism spell outr.cSs|o|SN)isalphaisupper)rtr%r%r&~z8TextProcessorTestCase.test_initialisms..)r2 is_initialismsplit_initialismzTTS.rTzT rrrSrTr>Nrr r%r%r&test_initialismszs"  z&TextProcessorTestCase.test_initialismsc Cshtdd}|d\}}t|j||fit}||tdddddtddd d dtd dd d dgd S)z+Test number verbalization (single language)rorpz1 2 3rrBrurrrCrvrrwNrr r%r%r&test_numbers_one_languages  z/TextProcessorTestCase.test_numbers_one_languagec Csntdd}|ddd\}}t|j||dd}||tdddd d d td d dddd tdddddd gdS)z4Test number verbalization (SSML, multiple languages)rorpz-1 2 3TraF)r rrBrulangrrrres_ESrdoszdos de_DErdreiNrrrrrr r%r%r&test_numbers_multiple_languagess  z5TextProcessorTestCase.test_numbers_multiple_languagesc Cs\tdd}|d\}}t|j||ddd}||tdddddd tdd dd d d gd S) z-Test currency verbalization (single language)rorpz$10Fr r rtenten rrdollarsNrr r%r%r&test_currency_one_languages  z0TextProcessorTestCase.test_currency_one_languagecCstdd}|dddd\}}t|j||ddd}||tdddd d d tdd dd dd tdddddd tdddddd tdddddd tdddd d d gdS)z6Test currency verbalization (SSML, multiple languages)rorpu9€10 €10 €10TFrb phonemizerrrrrreurozeuro fr_FRrdixdix reuroszeuros nl_NLr5tienztien r8Nrr r%r%r&test_currency_multiple_languages.     z5TextProcessorTestCase.test_currency_multiple_languagec Csbtddd}|ddd\}}t|j||ddd}||tdd d d d d tdd d ddd gdS)zKTest default currency use when no currency symbol (interpret-as="currency")roUSD)rqdefault_currencyz+10TraFrrrrrrrNrr r%r%r&test_currency_defaults"  z+TextProcessorTestCase.test_currency_defaultcCstdd}|d\}}t|j||ddd}||tdddddd tdd dd d d tdd dddd tdddddd tdddddd gdS)z!Test time verbalization (English)rorpz 4:01pmFrrfourz four rrohzoh rrBrurPzP r5MNrr r%r%r& test_times   zTextProcessorTestCase.test_timec Csltdd}|d\}}t|j||ddd}||tdddddd tdd dd d d tdd dddd gdS)z1Test time verbalization without a colon (English)rorp10amFrrrrrrAA rrNrr r%r%r&test_time_no_colons  z(TextProcessorTestCase.test_time_no_coloncCstddhd}|d\}}t|j||ddd}||tddddd d tdd dd d d tdd dddddtdddddd tdddddd tdddddd gdS)z)Test date verbalization (single language)rorirqrjz4/1/1999FrrAprilApril rrfirstrr-rGTrrrrrr7rnineteen nineteen r5rkrlr8rmNrr r%r%r&test_date_one_language sH   z,TextProcessorTestCase.test_date_one_languagecCshtddhd}|dddd\}}t|j||ddd}||tdd d d d d tdd d ddd tddd ddddtddd ddd tddd ddd tddd ddd tdd d ddd tdd d ddd tddd d d!d tddd d"d#d tddd d$d%d tddd d&d'd tdd(d d)d*d tdd+d d,d-d tdd.d d"d"d td/d dd0d1d td/d dd d d td/ddd2d2d gd3S)4z2Test date verbalization (SSML, multiple languages)rorirzY4/1/1999 4/1/199901.04.1999TFrrrrrrrrrr-rGrrrrr5rkrlr8rmru quatrièmeu quatrième janvierzjanvier millezmille neufzneuf centzcent quatrezquatre r:vingtzvingt r;rrr<rerstezerste neunzehnhundertneunundneunzigNrr r%r%r&test_date_multiple_languagesJs       z2TextProcessorTestCase.test_date_multiple_languagesc Cs`tdd}|ddd\}}t|j||ddd}||tdddd d d tdd dd d d gdS)z"Test date format in SSML (ordinal)rorpz44/1TraFrrrrrrrBNrr r%r%r&test_date_format_ordinals   z.TextProcessorTestCase.test_date_format_ordinalc Cstdd}|ddd\}}t|j||ddd}||tdddd d d tdd dd dd tdddddd tdddddd gdS)z#Test date format in SSML (cardinal)rorpz:4/1/2000TraFrrrBrurrrrrrCrvrthousandNrr r%r%r&test_date_format_cardinals*   z/TextProcessorTestCase.test_date_format_cardinalc Csddd}t|d}|d\}}t|j||ddd}||tdddd d d td dd d dd gdS)zTest part-of-speech taggingc_sdd|DS)NcSsg|]}|qSr%)upper).0wr%r%r& szbTextProcessorTestCase.test_part_of_speech_tagging..get_parts_of_speech..r%)rargskwargsr%r%r&get_parts_of_speechszNTextProcessorTestCase.test_part_of_speech_tagging..get_parts_of_speech)rza testF)r r rrr9r)rrrrr rrTESTNr)r!rr"r#r$rr%r%r&test_part_of_speech_taggings z1TextProcessorTestCase.test_part_of_speech_taggingc Cs^dtfdd}t|d}|d\}}t|j||ddd}||tddddgd d gd S) z!Test phonemizer (single language)wordc_t|Sryrrrrr%r%r&lookup_phonemeszJTextProcessorTestCase.test_phonemize_one_language..lookup_phonemesrrFr r rrrrsrtrrrrrrr N)strrrrrrr!rr"r#r$rr%r%r&test_phonemize_one_languages" z1TextProcessorTestCase.test_phonemize_one_languagec Cstddd}t|d}|dddd\}}t|j||ddd }||td d d d gd dtdd d d gddgdS)z+Test phonemizer (SSML, multiple word roles)Nc[s|st|St|Sryrr)rrolerr%r%r&rszYTextProcessorTestCase.test_phonemize_one_language_multiple_roles..lookup_phonemesrz0test testTF)rbr rrrrIrrrrErrryrrr%r%r&*test_phonemize_one_language_multiple_roless4  z@TextProcessorTestCase.test_phonemize_one_language_multiple_rolesc Csdtfdd}dtfdd}td|dtd|did }|d d d \}}t|||}||tdd d ddgddtddd ddgddgdS)z*Test phonemizer (SSML, multiple languages)rc_rryrrr%r%r&en_lookup_phonemes;rzSTextProcessorTestCase.test_phonemize_multiple_languages..en_lookup_phonemesc_s t|Sryrrr%r%r&de_lookup_phonemes>s zSTextProcessorTestCase.test_phonemize_multiple_languages..de_lookup_phonemesror)rr)rqrsettingsz,test testTrarrrIr)rrrrrr rrN)rrrrrrr)r!rrr"r#r$rr%r%r&!test_phonemize_multiple_languages8sD  z7TextProcessorTestCase.test_phonemize_multiple_languagesc Csztdd}|ddd\}}t|j||fit}||tddddd td dd d d td dddd tddddd gdS)zTest SSML substitutionrorpz?W3CTrarWorldzWorld rrWidezWide rWebzWeb r ConsortiumNrr r%r%r&test_subfs  zTextProcessorTestCase.test_subcCsxtddd}|ddd\}}t|j||fit}||tddddd d tddd d d d d tdddddgdgdS)zTest SSML break tagroFrqr(a2

Break here

Trarz Break herei i*Breaki)rrrrpause_before_mspause_after_msrherer)rrrr^rrrNrrr_rrrrr!r"r#r$r_r%r%r& test_breakys8  z TextProcessorTestCase.test_breakcCstddd}|ddd\}}t|j||fit}||tddddd d gd d gtddd d dgdgdtdddddgdgdS)zTest SSML mark tagroFra%

Mark here

Trarz Mark hererbrsfMarkcd)rrrr marks_before marks_afterrrr)rrrr^rrrNrrr%r%r& test_marks8  zTextProcessorTestCase.test_markc CsZt}|ddd\}}t|j||fit}||tdddddtdddddgd S) z"Test SSML with missing tagzhelloworldTrarhellorrworldNrr r%r%r&test_missing_speaksz(TextProcessorTestCase.test_missing_speakc Csbt}|ddd\}}t|j||fddit}||tdddddd tdd d d d d gd S)z$Test SSML with adjacent tagsz<hello.world.Trar2Frrr)rrvoicerrrrrNrr r%r%r&test_adjacent_voices z)TextProcessorTestCase.test_adjacent_voicec Csvt}|d\}}t|j||fit}||tddddtddddtd d d dtd d ddtddddgdS)z9Test sentence that needs multiple passes to fully resolvezABCD-10rrrrrrrBzB rCzC rDzD r5rNrr r%r%r&test_multiple_passess      z*TextProcessorTestCase.test_multiple_passesc CsRt}|d\}}t|j||fit}||tddddtddddgdS) zTest sentence with nan or infznan infrnanznan rrinfNrr r%r%r&test_number_nonfinite s   z+TextProcessorTestCase.test_number_nonfinitec Cst}|d\}}t|j||fit}||tddddtddddtd dddtd d d dtd dddtddddg|ddd\}}t|j||fit}||tddddgdS)zBTest use of inline lexicon pronunciation to override an initialismROOFUSrRzR rrOzO rrFzF r5UzU r8ru ROOFUS ɹ ˈu f ə s ROOFUS TraNrr r%r%r&test_override_initialisms*          z.TextProcessorTestCase.test_override_initialismN)(__name__ __module__ __qualname____doc__r'r)r@rDrKrWr`rfrgrhrnrxrrrrrrrrrrrrrrrrrrrrrrrrr%r%r%r&r sL(! (E!*i&.00 r cCst||ddddS)zPrint graph to stderrcWst|dtjiS)Nfile)printsysstderr)pr%r%r&r|Dr}z$print_graph_stderr..) print_funcNr)r#r$r%r%r&print_graph_stderrBsr __main__)rrunittestgruut.text_processorrrrr gruut.utilsrrTestCaser r rmainr%r%r%r&s&  >