o i* @sddlZddlZddlmZmZddlmZejdddZ ddZ d d Z d d Z d dZ ddZddZddZddZejdddiigddiigddiigddiddigdddddigddigddZdd Zd!d"Zd#d$ZdS)%N)DocToken)Vocabi c Cs&gd}tjddgddgddgdd gd d gd d ggdd}t||d}||_gd}dd|D|ks5Jgd}t|D] \}}|||_q=dd|D|ksRJdd|D}t|t|kscJ|+} |ddf|dg} ddgddgddgd} | j|dddg| | dWdn1swYgd}d d|D|ksJgd}d!d|D|ksJd"d|D} t| t|ksJ|d# | d# ksJ|d | d ksJ|d | d ksJ|d$ | d% ksJ|d% | d& ksJdS)'N)IliveinNewYorkrightnowg?g?g@g@g@g@g@gffffff@g@gffffff@g@gffffff@f)dtypewordscSg|]}|jqStext.0tokenrrY/home/ubuntu/.local/lib/python3.10/site-packages/spacy/tests/doc/test_retokenize_split.py z"test_issue3540..cSrrlemma_rrrrrrcSrrvectorrrrrrrPROPNNewYorkpobjcompound)POSLEMMADEP)headsattrs)rrrr!r"r r cSrrrrrrrr$rcSrrrrrrrr&rcSrrrrrrrr'rr) numpyasarrayrtensor enumeraterlen retokenizesplittolist) en_vocabrr/doc gold_text gold_lemmailemma vectors_1 retokenizerr(r) vectors_2rrrtest_issue3540sD&    "r>c Csgd}gd}dgt|}t||||d}t|dksJtt|dks)J|djjdks3J|d jjd ks=J|-}|j|dd d g|dd f|d gd gdd d gdgddgdddWdn1sqwYt|dks~J|djd ksJ|djjd ksJ|djdksJt|djdksJ|d jdksJ|d jd ksJ|d jjdksJt|d jdksJ|djdksJ|djjd ksJ|djd ksJ|djjd ksJtt|dksJdS)N LosAngelesstart.rrrdeprr(depsrrrArrBLosAngelesNNPrGPEz Number=Sing)tagr:ent_typemorphr)r*) r1rstrheadrr2r3idxrN)r5rr(rFr6r<rrrtest_doc_retokenize_split0sD  rScCsTgd}gd}dgt|}t||||d}|}||dddg|ddf|dgWdn1s9wY|djd ksGJ|djd ksPJgd}gd}dgt|}t||||d}|D]}d |_qi|}||dddg|ddf|dgWdn1swY|djdksJ|djdksJdS) Nr?rCrDrErrHrIra)r1rr2r3r)r5rr(rFr6r<trrr test_doc_retokenize_split_lemmasUs8  rWcCst|gdd}|jjd}|jjd}|!}|j|dddg|ddf|dgd ||gid Wdn1s>wY|dj|ksLJ|dj|ksUJdS) Nr?r amodsubjectrrHrIrrDrO)rvocabstringsaddr2r3rD)r5r6dep1dep2r<rrr&test_doc_retokenize_split_dependenciesus  r_c Cst|gdd}tt*|}||dddg|dgWdn1s+wYWdn1s:wYtt9|}||dddg|d|d|dgWdn1shwYWddSWddS1swYdS)Nr?r rrHrIrrpytestraises ValueErrorr2r3r5r6r<rrr%test_doc_retokenize_split_heads_errors    *"rec Csgd}tt|d}|jjdddfg|_|djdks J|djdks)J| }||dgd |ddf|ddf|dgWdn1sPwY|djdks^J|djdksgJ|djdkspJ|d jdksyJdS) N)abcder zent-abcdrrBrr)rUbcr) rrrZr[r\entsent_iob_r2r3)rr6r<rrr*test_doc_retokenize_spans_entity_split_iobs 2rnc Cs gd}gd}gd}t||||d}t|j\}}t|}t|}|:} | j|dddg|ddf|dgd d d gid | j|d ddg|d df|dgd d dgid Wdn1sdwYt|j\}}t||dkszJt||dksJdS)N) StewartLeeisrUstandupcomedianrBHelivesrEnglandandloves JoePasqualerB)rrrr+rrrrzrz rzrz )nsubjROOTdetrXprtattrpunctr~rprepr#ccconjr$rrErStewartLeerrDr$r~rO JoePasqualer}dobj)rlistsentsr1r2r3) r5rr(rFr6sent1sent2init_len init_len2r<rrr5test_doc_retokenize_spans_sentence_update_after_splits0    rc Cst|gdd}tt:|}||dddg|ddf|ddfgWdn1s2wYWddSWddS1sJwYdS)alTest that the regular retokenizer.split raises an error if the orths don't match the original token text. There might still be a method that allows this, but for the default use cases, merging and splitting should always conform with spaCy's non-destructive tokenization policy. Otherwise, it can lead to very confusing and unexpected results. r?r rLANr`rdrrr(test_doc_retokenize_split_orths_mismatchs  ,"rcCstjddddtjddddt|ddgd }|/}|d d f|d g}dd d ddig}ddg|d}|j|d ddg||dWdn1sNwY|d jdks\J|d jjdusfJ|d jjd kspJ|d jdksyJ|d jjdusJ|d jjdksJdS)NrUFTdefaultforcerjnothingr@rAr rr1)rUrj2losangeles)r:_rHrIrO) r set_extensionrr2r3rrrUrj)r5r6r<r( underscorer)rrr)test_doc_retokenize_split_extension_attrss runderscore_attrsrUxrjrk)rUrc Cstjddddtjdddddtjd d ddd t|d d gd}d|i}tt<|}|ddf|dg}|j|dddg||dWdn1sTwYWddSWddS1slwYdS)NrFTrrUcS|SNrrrrrzCtest_doc_retokenize_split_extension_attrs_invalid..)getterrrjcSrrrrrrrrr)methodrr@rAr rrrrHrIrO)rrrrarbrcr2r3)r5rr6r)r<r(rrr1test_doc_retokenize_split_extension_attrs_invalids   "rcCst|dgddjr Jt|dgddjrJt|ddgd}|djr'J|%}ddd gi}|dd f|d g}|j|dddg||d Wd n1sSwY|djs_J|d jrfJd S) aTest that retokenization also sets attributes on the lexeme if they're lexical attributes. For example, if a user sets IS_STOP, it should mean that "all tokens with that lexeme" are marked as a stop word, so the ambiguity here is acceptable. Also see #2390. rHr rrIr@rAis_stopTFrrON)rrr2r3)r5r6r<r)r(rrr$test_doc_retokenizer_split_lex_attrss  rcCsd}t||ddd}|#}|d}|dfgt|}|j||jt|j|dWdn1s8wYt||d}|$}|d}|dfgt|}|j||jt|j|dWddS1spwYdS)zB#4604: realloc correctly when new tokens outnumber original tokenszOHyperglycemic adverse events following antipsychotic drug administration in theNr rr()rr3r2r1r9rr)r5rr6r<rr(rrrtest_doc_retokenizer_reallocs  "rcsd}t||d}d|d_|d|}|jgdfddtdDd Wd n1s5wY|d jd ksCJ|d jdksLJ|djd ksUJ|djd ks^Jd S)z#6060: reset norm in splitz6The quick brownfoxjumpsoverthe lazy dog w/ white spotsr withr+r)brownfoxjumpsoverthecsg|]}|fqSrr)rrRrrrr"sz3test_doc_retokenizer_split_norm..rNr{zw/r)rr3norm_r2ranger)r5rr6r<rrrtest_doc_retokenizer_split_norms  r)r-ra spacy.tokensrr spacy.vocabrmarkissuer>rSrWr_rernrrr parametrizerrrrrrrrs8   '%