o iM@s<ddlZz ddlmZmZWneyddlmZmZYnwddlmZmZm Z ddl Z ddl m Z ddl mZddlmZddlmZddlmZdd lmZmZmZd d lmZejd d dZddZddZddZ ddZ!ddZ"ddZ#ddZ$ddZ%dd Z&d!d"Z'd#d$Z(ej)d%d&d'id(d'id)d'igid*d*d*d+fd&d'id(d,id)d-igid.d*d/d+fd0d1d2d)d'id3d3d4gid5d6d*d7d7d8fd-d9d2d,d,d:gid;dd2fd3d?d2gid'd?d2fd3d3d2d(d'igd&d?id?d'd2fd?d?d2d)d?igid?d?d?d+fd?d?d2d)d'igid?d?d'd+fd?d?d2d)d?igd)d;id?d?d'd+fd3d3d'd'd=gd?d?d2d?d?d3d3d=fd3d3d'd'd=gd?d?d?d@d?d?d3d3d?dAfg dBdCZ*dDdEZ+dFdGZ,dHdIZ-dJdKZ.dLdMZ/GdNdOdOZ0ej)dPddQdRdSdRfdTe0e0ddfgdUdVZ1dWdXZ2dS)YN) StrictInt StrictStr)ConfigValidationErrorLinearModel)German)English)Language)DEFAULT_TOK2VEC_MODEL)Doc)SimpleFrozenDictcombine_score_weightsregistry) make_tempdiricsd}dt|Gfddd}t}|j|d}|jdks#Jt'}||ddd iii}tj||d }| jd ksEJWddS1sPwYdS) Ntest_issue5137 my_componentcs2eZdZdfddZddZddZdd Zd S) z#test_issue5137..MyComponentall_categoriescSs||_||_||_dSN)nlp categoriesname)selfrrrr\/home/ubuntu/.local/lib/python3.10/site-packages/spacy/tests/pipeline/test_pipe_factories.py__init__s z,test_issue5137..MyComponent.__init__cSdSrrrdocrrr__call__!z,test_issue5137..MyComponent.__call__c[rrr)rpathkwargsrrrto_disk$r z+test_issue5137..MyComponent.to_diskc[rrr)rr!cfgrrr from_disk'r z-test_issue5137..MyComponent.from_diskN)__name__ __module__ __qualname__rrr#r%r pipe_namerr MyComponents  r+rr componentsr my_categoriesconfig) r factoryradd_piperrr#spacyloadget_pipe) factory_namer+rrtmpdir overridesnlp2rr)rrs "rcCsd}t|dtdtfdd}|tjvsJt}tt ||Wdn1s/wY||||j vs@J|j ||ksIJt |sPJ| |sWJ| |}||ksbJ||}||ksmJdS)Ntest_componentrreturncS|Srrrrrr component7sz/test_pipe_function_component..component)r r>r r factoriespytestraises ValueErrorr2 pipe_namespipe_factoriesget_factory_meta get_pipe_metar5 create_pipe)rr>rpiperrrtest_pipe_function_component4s"      rIc sPd}d}t|Gddd}Gdddt||fdtffdd }t}||f|ffD]s\}}|tjvs=Jtt||||Wdn1sUwY||||jvsfJ|j ||ksoJt |svJ| |s}J| |}t ||sJt |jtsJ||}t ||sJt |jtsJq2dS) Ntest_class_component1test_class_component2c@0eZdZdedefddZdedefddZd S) z2test_pipe_class_component_init..Component1rrcS ||_dSrrrrrrrrrP z;test_pipe_class_component_init..Component1.__init__rr;cS|SrrrrrrrSr z;test_pipe_class_component_init..Component1.__call__Nr&r'r(r strrr rrrrr Component1NsrTc@rL) z2test_pipe_class_component_init..Component2rrcSrMrrNrOrrrrWrPz;test_pipe_class_component_init..Component2.__init__rr;cSrQrrrrrrrZr z;test_pipe_class_component_init..Component2.__call__NrRrrrr Component2VsrUrcs ||SrrrrrUrrr1] z/test_pipe_class_component_init..factory)r r1rr?r@rArBr2rCrDrErFr5 isinstancerrG)name1name2rTr1rr ComponentrHrrWrtest_pipe_class_component_initJs2    r]cCsd}t|Gddd}t|Gddd}t}tt ||Wdn1s2wYtt|j|dddd Wdn1sQwYtt|j|d dd d d Wdn1sqwY| |}t |j tsJ|j d ksJ|j dksJ|jd usJ|j|ksJt}tt|j|dddd Wdn1swY|j|d ddd | |}t |j tsJ|j d ksJ|j dksJ|jdusJdS)Ntest_class_component_configc@8eZdZdedededefddZdedefd d Z d S) z3test_pipe_class_component_config..Componentrrvalue1value2cSs"||_||_||_d|_||_dS)NT)rr`rais_baserrrrr`rarrrrxs  z.Component.__init__rr;cSrQrrrrrrrr z.Component.__call__N r&r'r(r rSrrrr rrrrrr\vs  r\c@r_) z5test_pipe_class_component_config..ComponentENrrr`racSs||_||_||_d|_dS)NF)rr`rarbrcrrrrs z>test_pipe_class_component_config..ComponentEN.__init__rr;cSrQrrrrrrrr z>test_pipe_class_component_config..ComponentEN.__call__Nrdrrrr ComponentENs re10hello)r`rar/ wrong_name)r`rarTF)r r1rr@rArr2warns UserWarningr5rYrr`rarbr)rr\rerrHnlp_enrrr test_pipe_class_component_configssB         rmcCsfd}t|Gddd}t}||||}t|jts#J|jdks*J|jdks1JdS)Ntest_class_component_defaultsc @sFeZdZededfdedededefddZd ed efd d Z d S)z5test_pipe_class_component_defaults..Componentrhrgrrr`racSs||_||_||_dSr)rr`rarcrrrrs z>test_pipe_class_component_defaults..Component.__init__rr;cSrQrrrrrrrr z>test_pipe_class_component_defaults..Component.__call__N) r&r'r(rrr rSrr rrrrrr\s  r\rhrg)r r1r2r5rYrr`ra)rr\rrHrrr"test_pipe_class_component_defaultss  rocCsd}dtdddddddd }tj||d Gd d d }t}||||}t|jts3J|jdks:Jt|jt sBJdS) Ntest_class_component_modelzspacy.TextCatEnsemble.v2zspacy.TextCatBOW.v3F)@architecturesexclusive_classes ngram_sizeno_output_layer)rrtok2vec linear_modelrh)modelr`default_configc@r_) z2test_pipe_class_component_model..Componentrrxrr`cS||_||_||_||_dSrrrxr`rrrrxrr`rrrrs z;test_pipe_class_component_model..Component.__init__rr;cSrQrrrrrrrr z;test_pipe_class_component_model..Component.__call__N) r&r'r(r rrSrrr rrrrrr\sr\) r r r1r2r5rYrr`rxr)rrzr\rrHrrrtest_pipe_class_component_models&    r~cCsfd}|d}d|dddd}tj||dGdd d }t|d td tfd d }t}d|dddd}|j||d||}t|jtsLJ|j dksSJt|j t s[J|j j dkscJt}t td|dddd}|j||dWdn1swYt td|dddd}|j||dWddS1swYdS)N!test_class_component_model_customz.archrqr)rrnOnI)r`rxryc @s@eZdZedfdedededefddZded efd d Z d S) z9test_pipe_class_component_model_custom..Componentrhrrxrr`cSr{rr|r}rrrrs zBtest_pipe_class_component_model_custom..Component.__init__rr;cSrQrrrrrrrr zBtest_pipe_class_component_model_custom..Component.__call__N) r&r'r(rr rrSrr rrrrrr\s  r\rrcSs t||Sr)r)rrrrrmake_custom_archrXz@test_pipe_class_component_model_custom..make_custom_archrr/linear20?g@)r r1r architecturesrr2r5rYrr`rxrrr@rAr)rarchrzr\rrr0rHrrr&test_pipe_class_component_model_customs0     "rcCstttjdtdtfdd}Wdn1swYtttjdtdtfdd}Wdn1s=wYtttddtdtfdd }WddS1sawYdS) NfoobarcSrrrrrrrrr>r z4test_pipe_factories_wrong_formats..componentcSrrrrrrrfactory1r z3test_pipe_factories_wrong_formats..factory1 test_pipe_factories_missing_argscSrrrrrrrfactory2#r z3test_pipe_factories_wrong_formats..factory2)r@rArBr r>intrSr1)r>rrrrr!test_pipe_factories_wrong_formatss   "rcCs\t}|jddd|d|dsJ|dsJ|ds$J|ds+J|ds2J|ds9J|dd|dsFJ|dsMJtt | dWdn1sbwY| dd|j vssJd|j vszJtt| ddWdn1swY| dd|dsJ|dj dksJdS)zTest that component-specific meta and config entries are represented correctly and cleaned up when pipes are removed, replaced or renamed.ner ner_componentr,textcattcNparser)r r2rErFget_pipe_config rename_piper@rArB remove_pipe _pipe_meta _pipe_configs replace_piper1rNrrr%test_pipe_factory_meta_config_cleanup's0       rcCs@d}tj|diiddtdtdtfdd}t}||dS) zeTest that default config values can be empty dicts and that no config validation error is raised.&test_pipe_factories_empty_dict_defaultrryrrcSrrr)rrrrrrr1Hr z7test_pipe_factories_empty_dict_default..factoryN)r r1rSdictrG)rr1rrrrrBs rcCs`d}d}tj|dddtj|dddtj|dddt|s&Jt|r-Jt|s4Jt|r;Jt|sBJt|sIJt}||dksVJtt ||Wd n1skwYt}||d ks}Jtt ||Wd n1swYt}||dksJ||d ksJd S) zfTest that language sub-classes can have their own factories, with fallbacks to the base factories.specific_component1specific_component2cSdS)NbaserrrrrTz7test_pipe_factories_language_specific..funccSr)NenrrrrrrUrcSr)NderrrrrrVrrNrr) r r>rr has_factoryrGr@rArB)rZr[rrlnlp_derrr%test_pipe_factories_language_specificOs0    rcCsttjtsJtt dtjd<Wdn1swYt}t|jts-Jt|js4Jttd|jd<WddS1sJwYdS)z`Test that assigning directly to Language.factories is now invalid and raises a custom error.rrN)rYr r?r r@rANotImplementedErrorlenrNrrrtest_language_factories_invalidls    "rzweights,override,expectedarbcgQ?)rrr2dg{Gz?Q?ffffff?g333333?)rr?)degq= ףp?皙?(\?)rrrrri,)rr皙?g333333?)rrrrgq= ףp?)rrf)rrrrrcCs.t||}t|dvsJ||ksJdS)N)gGz?rr)r sumvalues)weightsoverrideexpectedresultrrr-test_language_factories_combine_score_weightsys &rc Csd}dd}ddd}dddd }tj|d ||d tj|d ||d t|d }|j|ks6Jt|d }|j|ksEJt}i|jd d<||d ||d |jd }dddddd}|d|kstJ|j} d| d dd<d| d dd<t | }|jd d} dddddd} | | ksJ|j} d| d dd<t | }|jd d} dddddd} | | ksJdS)Ntest_language_factories_scorescSddS)NcSr<rrr=rrrrrzBtest_language_factories_scores....rrVrrrrz0test_language_factories_scores..r)a1a2rrr)b1b2b31)default_score_weightsr2training score_weightsg?gffffff?g?)rrrrrrrg?rgQ?rg?r) r r1rEr_configr2r0copyr from_config) rrweights1weights2meta1meta2rr$expected_weightsr0rrrrrrs<        rcCst}|jdddt}tt|jdddWdn1s$wY|jd|dd|jvs7Jtt|jd|dWddS1sOwYdS)z+Test adding components from a source model.tagger my_taggerr,en_core_web_smsourceNcustom)rr2r@rArBrCKeyError) source_nlprrrrtest_pipe_factories_from_sources  "rcsGdddtjtdGfdddt}t}|d|}|jd|dd|jvs1Jt}|jd|dd|jvsBJt}|jj d|jjj d gd d t t |jd|dWddS1sowYdS) Nc@seZdZeddgZdS)zPtest_pipe_factories_from_source_language_subclass..CustomEnglishDefaultsrstopN)r&r'r(set stop_wordsrrrrCustomEnglishDefaultssr custom_encseZdZdZZdS)zHtest_pipe_factories_from_source_language_subclass..CustomEnglishrN)r&r'r(langDefaultsrrrr CustomEnglishsrrr)rqcat)rqrr)vector)rrr languagesr2rCrvocabvectorsresizeaddr@rjrk)rrrrrr1test_pipe_factories_from_source_language_subclasss"  "rcCsd}tj|ddiddtfdd}t}|d|j|ddid t}|j||d ||jvs4J||jddks@J|jd |}|d |ksOJ|ddksWJd S)zBTest adding components from a source model with custom components.&test_pipe_factories_from_source_customargrgrycSr)NcSr<rrr=rrrrrzNtest_pipe_factories_from_source_custom..test_factory..rrrrrrr test_factoryz.test_factoryrworldr/rr-r1N) r r1rSrr2rCrFrzr0)rrrrr0rrrrs rc Cs4d}tj|ddiddtfdd}t}|d|j|ddd id d d d gd}t#}||dd it|ddd}||d}t|}Wdn1sTwY|jd d gksbJ|j d |dkslJ| d }|j|ksxJ|j ddksJ|j dd } | d|ksJ| dd ksJdS)N&test_pipe_factories_from_source_configrrgrycSr)NcSr<rrr=rrrr rzNtest_pipe_factories_from_source_config..test_factory..rrrrrr rz.test_factoryryolor)rr0rrrrpipeliner1)rr>)rrrr-r-) r r1rSrr2rr#rrCrDrFrzr0) rrr dest_nlp_cfgtempdirdest_components_cfg dest_configrmetar0rrrrs.      rc@seZdZddZddZdS)PipeFactoriesIdempotentcCrrrrOrrrr$rz PipeFactoriesIdempotent.__init__cCrrrrrrrr&rz PipeFactoriesIdempotent.__call__N)r&r'r(rrrrrrr#s rz i,func,func2cCr)NcSr<rrr=rrrr,rz..rrVrrrr,rrcCr<rrr=rrrr,rrqcCsd|}tdD] }tj||dq t}||tj||d|d}tdD] }tj||dq+t}||tj||ddS)zCheck that decorator can be run multiple times if the function is the same. This is especially relevant for live reloading because we don't want spaCy to raise an error if a module registering components is reloaded. )test_pipe_factories_decorator_idempotent_rrN)ranger r1r2r>)irfunc2rrr[rrr(test_pipe_factories_decorator_idempotent)s     r cCs~d}dd}tj||dd|gd|d|iid}t|}|j|gks'J||}|d|ik|j|d|iks=Jd S) z~Test that the extra values we temporarily add to component config blocks/functions are removed and not copied around. 'test_pipe_factories_config_excludes_nlpcSr)NcSr<rrr=rrrrJrzKtest_pipe_factories_config_excludes_nlp....rrVrrrrJrz9test_pipe_factories_config_excludes_nlp..rrrr1rN)r r1rrrCrr)rrr0rpipe_cfgrrrrEs     r)3r@ pydantic.v1rr ImportErrorpydantic thinc.apirrrr3 spacy.lang.der spacy.lang.enrspacy.languager spacy.pipeline.tok2vecr spacy.tokensr spacy.utilr r rutilrmarkissuerrIr]rmror~rrrrrr parametrizerrrrrrrr rrrrrs         )9#,  ""      %#