o i2@svddlZejdddZejdddZejdejd d d gd d ZejdejdddgddZejdejdddgddZejdejdgdddZ ejdejdgdd d!Z ejd"ejdd#d#gd$d%Z ejd"ejdd&d'gd(d)Z ejd*ejdd+d,gd-d.Z ejd/ejdd0gd1d2Zejd3ejdd4gd5d6Zejd7ejjd8d9Zejdd:d;gejd<d=d>Zejd?d@dAZejdBdCdDZejdEdFdGZejdHgdIejdJdKdLZejdMejddNdOgdPdQZdS)RNi_cCsD|d}|djdks Jt|ddksJ|djdks JdS)Nz This is a cat.r)idxlen en_tokenizerdocr V/home/ubuntu/.local/lib/python3.10/site-packages/spacy/tests/lang/en/test_tokenizer.py test_issue351sr ihcCs|d}t|dks JdS)z!Test tokenization of big ellipsisz$45...............AskingNrrtokensr r r test_issue360 riz text,number)7am7)z11p.m.11cCs.||}t|dks J|dj|ksJdS)z`Test that times like "7am" are tokenized correctly and that numbers are converted to string.r rNrtext)rrnumberrr r r test_issue736rirz3/4/2012z 01/12/1900cC||}t|dks JdS)zTest that dates are not split and kept as one token. This behaviour is currently inconsistent, since dates separated by hyphens are still split. This will be hard to prevent without causing clashes with numeric ranges.rNr rrrr r r test_issue740srizWe were scaredzWe Were ScaredcCs2||}t|dks J|djdksJdS)zqTest that 'were' and 'Were' are excluded from the contractions generated by the English tokenizer exceptions.rrwereN)rrlowerrr r r test_issue744'sriz text,is_num))oneT)tenT) tenelevenFcCs||}|dj|ks JdS)Nr)like_num)rris_numrr r r test_issue7591sr%i)ShellshellShedshedcCs.||}t|dks J|dj|ksJdS)zsTest that 'Shell' and 'shell' are excluded from the contractions generated by the English tokenizer exceptions.rrNrrr r r test_issue775:rr*izThis is a string cC(||}ddd|D|ksJdS)zGTest for Issue #792: Trailing whitespace is removed after tokenization.cSg|]}|jqSr  text_with_ws.0tokenr r r Iz!test_issue792..Njoinrrrr r r test_issue792D r8zThis is a stringzThis is a string cCr+)z6Test base case for Issue #792: Non-trailing whitespacer,cSr-r r.r0r r r r3Qr4z)test_control_issue792..Nr5r7r r r test_control_issue792Lr9r:i[zaaabbb@ccc.com Thank you!zaaabbb@ccc.com Thank you!cCs||}|j|ks JdS)z5Test that no extra space is added in doc.text method.N)rr7r r r test_issue859Tsr;ivzDatum:2014-06-02 Dokument:76467cCsF||}|D]}t|jt|jksJ||j|jdks JqdS)zLTest that token.idx matches the original text index for texts with newlines.rN)rrr/r)rrrr2r r r test_issue886^s r<i{z want/needcCs.||}t|dks J|djdksJdS)z(Test that / infixes are split correctly.rr/Nrrr r r test_issue891hsr>icCsLtddD]}d}tddD] }||t|7}q||}|s#JqdS)zTest that spaCy doesn't hang on many punctuation characters. If this test hangs, check (new) regular expressions for conflicting greedy operators pytest_timeout) .,'":?!;-0rdN)pytest importorskiprangestr)rpunctstringirr r r test_issue957qs  rRztest@example.comzjohn.doe@example.co.ukicCs*||}t|dks J|djrJdS)z6Test that doc doesn't identify email-addresses as URLsrrN)rlike_urlr7r r r test_issue1698srTicCs|d}t|dks JdS)zDTest that "would've" is handled by the English tokenizer exceptions.zwould'ver Nr rr r r test_issue1758rrUicCs0|d}|djdkr|djdksJdSdS)zyTest that spaces don't receive a POS but no TAG. This is the root cause of the serialization issue reported in #1773. rSPACEr,N)pos_tag_rr r r test_issue1773s rZi cCsR|d}t|dks J|djdksJ|djdksJ|djdks'JdS) z2Test that hyphens are split correctly as prefixes.uA—Yo me llamo... –murmuró el niño– Emilio Sánchez Pérez.ru—u– Nr) es_tokenizerrr r r test_issue3277s r_word)zdon'tudon’tzI'duI’di cCs||d}|js JdS)Nr)is_stop)rr`tokr r r test_issue3521s rci)thesesthisrecCr)zuTest that 'theses' and 'thisre' are excluded from the contractions generated by the English tokenizer exceptions.rNr rr r r test_issue10699srf)rKmarkissuer r parametrizerrrr%r*r8r:r;r<r>slowrRrTrUrZr_rcrfr r r r sr