o i<@snddlZddlZddlmZddlZzddlmZWney)ddlmZYnwddl m Z m Z m Z m Z mZmZmZmZddlmZmZddlmZmZmZmZddlmZddlmZdd lm Z dd l!m"Z"dd l#m$Z$m%Z%dd l&m'Z'm(Z(m)Z)dd l*m+Z+ddl,m-Z-m.Z.m/Z/m0Z0m1Z1ddlm2Z2m3Z3ej4ddZ5ej67dddZ8ej67dddZ9ej6:dddgddZ;ej6:ddd gd!d"Zd-d.Z?d/d0Z@d1d2ZAd3d4ZBd5d6ZCej6:d7eed8fed9ed8fd:d;dd?d@g dAdBZDej6:dCgdDdEdFZEej6:dGgdHdIdJZFej6:dKd8d8dLdMd8dNd8idOifdPdQdRdSdPdTdQidUifdVdWidXdYdZdWiiifgd[d\ZGej6:dKd8d8dLdMd8dNd8idOifdPdQdRdSdPdTdQidUifd]dZdWiidXdYdZdWiiifgd^d_ZHd`daZIej6:dbgdcd)gfgddd'gfgded)d*gfgdfd(gfgdgd(gfgdhd)d)gfgdid)d)gfgdjgdkfgdld)gfgdmd'gfgdnd*d*gfgdod*d*gfg dpdqZJej6:dbgdrdd*gfgdsdd'gfgdgdd(gfgdtgdufgdvdd*gfgdwgdxfgdydzZKd{d|ZLd}d~ZMddZNddZOddZPddZQdS)N)Path)ValidationError)ConfigConfigValidationErrorCupyOpsMPSOpsNumpyOps Optimizerget_current_opsset_current_ops) has_cupy_gpuhas_torch_mps_gpu) prefer_gpu require_cpu require_gpuutil) __version__)English)Dutch)DEFAULT_CONFIG_PATH)PrecomputableAffine&_backprop_precomputable_affine_padding)ConfigSchemaTraining TokenPatternTokenPatternSchema)minibatch_by_words)SimpleFrozenList dot_to_objectfind_available_port import_fileto_ternary_int)get_random_doc make_tempdircCs8z tdk}W|Stytjjdk}Y|Sw)z/Determine if the tests are run as admin or not.r)osgetuidAttributeErrorctypeswindllshell32 IsUserAnAdmin)adminr,I/home/ubuntu/.local/lib/python3.10/site-packages/spacy/tests/test_misc.pyis_admin.s r.i?cCsd|d}|dd}|dd}|dd}t|||f}||vs$J||vs*J||vs0JdS)Nz zero one two three four five six)r filter_spans) en_tokenizerdocs1s2s3resultr,r,r-test_issue62079s     r;ircCsHttgdtttgdWddS1swYdS)z=Test that the non-empty constraint pattern field is respected)patternN)rrpytestraisesrr,r,r,r-test_issue6258Hs  "r?textz hello/worldz hello worldcCst|}t|ts JdSN)r ensure_path isinstancer)r@pathr,r,r-test_util_ensure_path_succeedsSs rEzpackage,result)numpyT)sfkodskfosdkfpsdpofkspdofFcCst||us JdS)zHTest that an installed package via pip is recognised by util.is_package.N)r is_package)packager:r,r,r-test_util_is_packageYsrJrIthinccCst|}t|ts JdS)z7Test that a Path object is returned for a package name.N)rget_package_pathrCr)rIrDr,r,r-test_util_get_package_pathas rMr/r2r0c CsVt||||d}|dj||||fksJ|jd|f}||\}}|j|jdd|||fks7J|jd||f}|jd|f} d| d<d|d<|d rWJt||| } | d d kseJ| d | d d|d<d| d<d| d <d| d<d|d<d| d<d|d<t||| } | ddksJ| ddksJ| d dksJdS)N)nOnInFnPW rr!)r!rNpad)rrNrr?)r!r!)r!r)rNrr2rN)rrrrr1)rr!rr) r initialize get_paramshapeopsalloc begin_updatehas_gradrfill) rOrPrQrRmodeltensorYget_dXdYidsd_padr,r,r-test_PrecomputableAffinehs2     ricCs\t}trts JtttsJntr#tsJttts"Jntr(Jt|dSrA)r r rrCrr rr  current_opsr,r,r-test_prefer_gpus    rlcCsHt}trttttsJn trttttsJt|dSrA)r r rrCrr rr rjr,r,r-test_require_gpus rmcCstt}ttttsJzddl}ttttsJWn ty(Ynwtttts4Jt|dS)Nr) r rrCrcupyrr ImportErrorr )rkrnr,r,r-test_require_cpus  rpcCs>ttjj}|dD]}tdd|jDsJ|jq dS)z{Test that all filenames in the project are ASCII. See: https://twitter.com/_inesmontani/status/1177941471632211968 z**/*css|] }t|dkVqdS)N)ord).0cr,r,r- sz'test_ascii_filenames..N)r__file__parentgloballname)rootrDr,r,r-test_ascii_filenamess  r|cCstd}|jdks J|jgksJtt tdWdn1s(wYtttdWddS1sCwYdS)z`Test that using a model name like "blank:en" works as a shortcut for spacy.blank("en"). zblank:enenz blank:zxxNzblank:fjsfijsdof)r load_modellangpipeliner=r>ro)nlpr,r,r-test_load_model_blank_shortcuts     "rzversion,constraint,compatibleTz>=)3.0.0z2.0.0F)3.2.1z>=2.0.0T)z2.2.10a1z>=1.0.0,<2.1.1F)z 3.0.0.dev3>=1.2.3,<4.5.6T)n/arN)z1.2.3rN)rrNcCst|||us JdSrA)ris_compatible_version)version constraint compatibler,r,r-test_is_compatible_versionsrzconstraint,expected) )rF)z==3.0.0F)z>=2.3.0T)z>2.0.0T)z<=2.0.0T)z >2.0.0,<3.0.0F)z>=2.0.0,<3.0.0F)z!=1.1,>=1.0,~=1.0T)rNcCst||us JdSrA)ris_unconstrained_version)rexpectedr,r,r-test_is_unconstrained_versionsrza1,a2,b1,b2,is_match))r3.0z3.0.1rT)z3.1.0z3.1rz3.2F)xxxNz 1.2.3.dev0z1.2FcCsPt||ks Jt||ksJt|||usJt|||us&JdSrA)rget_minor_versionis_minor_version_match)a1a2b1b2is_matchr,r,r-test_minor_versions rzdot_notation,expected)z token.posz token._.xyztokenxyz)pos_rqg{Gz?)ztraining.batch_sizeztraining.optimizer.learn_ratetraining learn_rate) batch_size optimizerzattribute_ruler.scorer.@scorerszspacy.tagger_scorer.v1attribute_rulerscorerz@scorerscCs,t|}||ks Jt||ksJdSrAr dot_to_dict dict_to_dot dot_notationrr:r,r,r-test_dot_to_dicts  rzattribute_ruler.scorercCs0t|}||ks Jtj|dd|ksJdS)NT) for_overridesrrr,r,r-test_dot_to_dict_overridess  rcCsrdddiddddiid}ttt|d d Wdn1s%wYttt|d d Wdn1sAwYttt|d d Wdn1s]wYt|d d |ddd kssJt|dddi|dddddksJ|ddddksJt|dd|ddksJt|ddt|dddksJdS)Nr!xybarbazabrt)footestz foo.bar.bazdz hello.worldz test.a.b.czfoo.barrrz foo.baz.xhelloworldrr{)r=r>KeyErrorrset_dot_to_objectdict)configr,r,r-test_set_dot_to_object's&   rzdoc_sizes, expected_batches)r)rrrr0)rrrr0)rrrr0r!)rrrr0r!i)rrrr0r!r)rrrr0r!)rrrr0r!rr)r0rNr!r!)r!rNr)r!rNrr!)r!rrr!)r!rrr!cCstdd|D}d}d}tt|||dd}dd|D|ks J|||}|D]}tdd|D|ks7Jq(dS) NcSg|]}t|qSr,r"rsdoc_sizer,r,r- Lz'test_util_minibatch..皙?Tsize tolerancediscard_oversizecSrr,lenrsbatchr,r,r-rRrcSrr,r)rsr6r,r,r-rVr)listrsum) doc_sizesexpected_batchesdocstolrbatchesmax_sizerr,r,r-test_util_minibatch:s r)rir)rrr r)rrrrrr)r!r!r0rN)r!rNi')rr!rr!r!r!r)r!r!r!r/cCsDdd|D}d}d}tt|||dd}dd|D|ks JdS) z=Test that oversized documents are returned in their own batchcSrr,rrr,r,r-rfrz0test_util_minibatch_oversize..rrFrcSrr,rrr,r,r-rlrN)rr)rrrrrrr,r,r-test_util_minibatch_oversizeYs rcCsdd}t|}tj|dd}tt}d|dd<tj|dd}t|ts)Jt|ts0J|j gks7J|j dgks?J| dj j dd usLJ|j dd dgksXJ|j dd gkscJttt|j d Wdn1sywYttt|j d Wdn1swYtjj|j d td}ttd |idtsJdS)Na) [nlp] lang = "en" pipeline = ["textcat"] [components] [components.textcat] factory = "textcat" [components.textcat.model] @architectures = "spacy.TextCatBOW.v3" exclusive_classes = true length = 262144 ngram_size = 1 no_output_layer = false T) auto_fillnlrrtextcat multi_labelFrznlp.pipeline.taggerznlp.unknownattributer)schematraining.optimizer)rfrom_strrload_model_from_config from_diskrrCrr pipe_namesget_piperbattrsrr=r>rrregistryresolverr ) cfg_string nlp_configen_nlpdefault_confignl_nlpTr,r,r-test_util_dot_sectionos*     rcCsFtddg}|ddgksJ|ddksJtt |dWdn1s,wYtt |Wdn1sEwYtt|dgWdn1s`wYtt |Wdn1sywYtddgdd}tt|dWddS1swYdS)Nrrr!rzError!)error) rindexr=r>NotImplementedErrorappendsortextendpop)tr,r,r-test_simple_frozen_lists&          "rcCsdddiidddd}t|dg}t|dtsJtt}t|ddgWdn1s4wY|jj}t |d ksEJ|dd d d gksQJdS) Nrz @optimizerszAdam.v1rz training.xyzr)rrrr!locrr) rresolve_dot_namesrCr r=r>rvalueerrorsr)rr:err,r,r-test_resolve_dot_namess  rc Csd}tG}tj|d}t|d }||Wdn1s"wYtd|ddddd iiii}t|}| d| WddS1sOwYdS) NaG from spacy import Language class DummyComponent: def __init__(self, vocab, name): pass def initialize(self, get_examples, *, nlp, dummy_param: int): pass @Language.factory( "dummy_component", ) def make_dummy_component( nlp: Language, name: str ): return DummyComponent(nlp.vocab, name) zcode.pyw python_coderZ componentsdummy_component dummy_paramr!) r#r$rDjoinopenwriterr from_configadd_piperZ)code_strtemp_dir code_pathfilehrrr,r,r-test_import_codes      "rcCstddksJtddksJtddksJtddks Jtddks(Jtddks0Jtddks8Jtddks@JtddksHJtd dksPJtd dksXJtdd gdksbJdS) NTr!rFrVrXrYr2istring)r r,r,r,r-test_to_ternary_intsrc Csd}d}t|||ksJdddlm}m}||||1}tjtddt||dd }Wdn1s7wY||d ksFJd WddS1sQwYdS) Nz0.0.0.0izPort 5001 isn't freer)demo_app make_serverzalready in use)matchT) auto_selectr!zDidn't find next port)rwsgiref.simple_serverrrr=warns UserWarning)hostportrrhttpd found_portr,r,r-test_find_available_ports"r)r/r2r0rN)Rr'r$pathlibrr= pydantic.v1rropydantic thinc.apirrrrrr r r thinc.compatr r spacyrrrr spacy.aboutr spacy_version spacy.lang.enr spacy.lang.nlrspacy.languagerspacy.ml._precomputable_affinerr spacy.schemasrrrspacy.training.batchersr spacy.utilrrrrr r"r#fixturer.markissuer;r? parametrizerErJrMrirlrmrpr|rrrrrrrrrrrrrrrr,r,r,r-s  (                                *