o iT@s ddlZddlZddlmZddlmZddlmZmZm Z m Z m Z m Z m Z mZmZmZmZz ddlmZmZmZmZmZmZmZmZmZmZddlmZWn'eyuddl mZmZmZmZmZmZmZmZmZmZddl!mZYnwddl"m#Z#m$Z$m%Z%ddl&m'Z'd d l(m)Z)d d l*m+Z+d d l,m-Z-d d l.m/Z/erd dl0m1Z1d dl2m3Z3d dl4m5Z5edZ6ee e e6ge e e6fe'fZ7ee de8ge dfe'fZ9ee dgee e e8efgdfe ffe'fZ:deede e8efde e8fddZ;GdddZdddd e d!e e8d"e8d#e?def d$d%Z@dd&d'd(d e d)e e8efd*e e8d"e8d!e e8de e8eff d+d,ZAdeBde e8fd-d.ZCGd/d0d0eZDGd1d2d2eZEGd3d4d4e8eZFGd5d6d6eZGeeFeGfZHeeDefZIeeEeefZJeeDeEe8eKeLeBe?fZMe+d7ZNGd8d9d9eZOGd:d;d;eZPGdd?d?eZRGd@dAdAeZSGdBdCdCeZTGdDdEdEeZUGdFdGdGeZVGdHdIdIeZWeSeReUeVdJZXGdKdLdLeZYGdMdNdNeZZGdOdPdPeZ[GdQdRdReZ\dS)SN) defaultdict)Enum) TYPE_CHECKINGAnyCallableDictIterableListOptionalTupleTypeTypeVarUnion) BaseModelConstrainedStrField StrictBool StrictFloat StrictInt StrictStrValidationError create_model validator)ModelMetaclass)ConfigValidationErrorModel Optimizer)Promise)NAMES)Literal)Lookups)is_cython_func)Language)Example)VocabItemTr#r$schemaobjreturnc Csz |di|gWStyJ}z4|}tt}|D]}ddd|dgD}|||dqdd|DWYd}~Sd}~ww) zValidate data against a given pydantic schema. obj (Dict[str, Any]): JSON-serializable data to validate. schema (pydantic.BaseModel): The schema to validate against. RETURNS (List[str]): A list of error messages, if available. z -> cSsg|]}t|qS)str).0pr*r*A/home/ubuntu/.local/lib/python3.10/site-packages/spacy/schemas.py Tszvalidate..locmsgcSs&g|]\}}d|dd|qS)[z] z, )join)r,r0r1r*r*r.r/Vs&Nr*)rerrorsrlistr3getappenditems)r'r(er4dataerrorerr_locr*r*r.validateFsr=c@eZdZdZdZdS)ArgSchemaConfigforbidTN__name__ __module__ __qualname__extraarbitrary_types_allowedr*r*r*r.r?\r?c@r>)ArgSchemaConfigExtrar@TNrAr*r*r*r.rHarGrHArgModelT)excludenamestrictfuncrJrKrLc Csi}zt|}Wntyt|tdYSwd}|jD]6}|j|vr(q |j|j kr1d}q |j |j kr:|j nt }t |rBdnd} |j|j krM|jn| } || f||j<q |o[| } | r`tnt|d<t|fi|S)aGenerate a pydantic model for function arguments. func (Callable): The function to generate the schema for. exclude (Iterable[str]): Parameter names to ignore. name (str): Name of created model class. strict (bool): Don't allow extra arguments if no variable keyword arguments are allowed on the function. RETURNS (ModelMetaclass): A pydantic model. ) __config__FTN.rN)inspect signature ValueErrorrrH parametersvaluesrKkind VAR_KEYWORD annotationemptyrr"defaultr?) rMrJrKrLsig_argssig has_variableparamrV default_emptyrX is_strictr*r*r. get_arg_modelfs(    r_) get_examplesnlp)sectionrKrJsettingsrcc Cspt||dd}z |d i|WSty7}z|sdnd|}d|d}t||||ddd}~ww) aValidate initialization settings against the expected arguments in the method signature. Will parse values if possible (e.g. int to string) and return the updated settings dict. Will raise a ConfigValidationError if types don't match or required values are missing. func (Callable): The initialize method of a given component etc. settings (Dict[str, Any]): The settings from the respective [initialize] block. section (str): Initialize section, for error message. name (str): Name of the block in the section. exclude (Iterable[str]): Parameter names to exclude from schema. RETURNS (Dict[str, Any]): The validated settings. InitArgModel)rJrK initializez initialize.z-Error validating initialization settings in [])titler4configparentNr*)r_dictrrr4) rMrdrcrKrJr'r9blockrhr*r*r.validate_init_settingss  rmcs\ddt|tr'g}|D]}t|trfdd|D}||q |}ttd|iS)NcSs"t|tr|ttkrt|S|SN) isinstanceintlenr)kr*r*r.s"z(validate_token_pattern..csi|] \}}||qSr*r*)r,rrvget_keyr*r. sz*validate_token_pattern..pattern)ror5rkr8r7r=TokenPatternSchema)r( convertedrxr*rur.validate_token_patterns   r{c@s eZdZUedddZeeedfed<edddZ ee eed<edddZ ee eed<edd dZ ee eed <edd dZ ee eed <edd dZee eed<edddZeeedfed<edddZeeedfed<edddZeeedfed<edddZeeedfed<edddZeeedfed<edddZeeedfed<edddZeeedfed<edddZeeedfed<edddZeeedfed <edd!dZeeedfed"<Gd#d$d$Zed%d&d&d&d'd(d)ZdS)*TokenPatternStringNregexaliasREGEXinINnot_inNOT_IN is_subset IS_SUBSET is_superset IS_SUPERSET intersects INTERSECTSfuzzyFUZZYfuzzy1FUZZY1fuzzy2FUZZY2fuzzy3FUZZY3fuzzy4FUZZY4fuzzy5FUZZY5fuzzy6FUZZY6fuzzy7FUZZY7fuzzy8FUZZY8fuzzy9FUZZY9c@r>)zTokenPatternString.Configr@TNrBrCrDrEallow_population_by_field_namer*r*r*r.ConfigrGr*Tpre each_item allow_reusecC|durtd|SNzNone / null is not allowedrQclsrtr*r*r.raise_for_nonez!TokenPatternString.raise_for_none)rBrCrDrrr rr__annotations__rr rrrrrrrrrrrrrrrrrr*r*r*r.r|sL  r|c@seZdZUedddZeeed<edddZee e ed<edddZ ee e ed<edd dZ ee e ed <edd dZ ee e ed <edd dZee e ed<edddZeee efed<edddZeee efed<edddZeee efed<edddZeee efed<edddZeee efed<edddZeee efed<GdddZedddddd d!ZdS)"TokenPatternNumberNr}r~rrrrrrrrrrrz==EQz!=NEQz>=GEQz<=LEQ>GT<LTc@r>)zTokenPatternNumber.Configr@TNrr*r*r*r.rrGrrTrcCrrrrr*r*r.rrz!TokenPatternNumber.raise_for_none)rBrCrDrrr rrrr rrrrrrrrrrrrrrrrr*r*r*r.rs       rc@sNeZdZUedZeed<edZeed<edZeed<edZeed<d S) TokenPatternOperatorSimple+plusrstar?question! exclamationN) rBrCrDrrrrrrr*r*r*r.r s rc@seZdZedZdS)TokenPatternOperatorMinMaxz^({\d+}|{\d+,\d*}|{\d*,\d+})$N)rBrCrDrecompiler}r*r*r*r.rsr)r`IOBrrc@seZdZUdZeeed<dZeeed<dZeeed<dZ eeed<dZ eeed<dZ eeed<dZ eeed<dZ eeed <dZeeed <dZeeed <dZeeed <dZeeed <dZeeed<dZeeed<dZeeed<dZeeed<dZeeed<dZeeed<dZeeed<dZeeed<dZeeed<dZeeed<dZeeed<dZ eeed<dZ!eeed<dZ"eeed<dZ#eeed<dZ$eeed<dZ%eeed<dZ&eeed<dZ'eeed <dZ(eeed!<dZ)eeed"<dZ*eeed#<dZ+eeed$<dZ,ee-ed%<e.dd&d'Z/ee0e1e2fed(<Gd)d*d*Z3e4d+d,d,d-d.d/Z5dS)0 TokenPatternNorthtextlowerpostagmorphdeplemmashapeent_typeent_iobent_id ent_kb_idnormlengthspacyis_alphais_asciiis_digitis_loweris_upperis_titleis_punctis_space is_bracketis_quote is_left_punctis_right_punct is_currencyis_stop is_sent_start sent_startlike_numlike_url like_emailop_r~ underscorec@seZdZdZdZddZdS)zTokenPattern.Configr@TcCs|Srn)upper)valuer*r*r.rsIszTokenPattern.Config.N)rBrCrDrEralias_generatorr*r*r*r.rFs rrT)rrcCrrrrr*r*r.rKrzTokenPattern.raise_for_none)6rBrCrDrr StringValuerrrrrrrrrrrIobValuerrrr NumberValuerrrrrrrrrrrrrrrrrrrrrrTokenPatternOperatorrrrrUnderscoreValuerrrr*r*r*r.rsR   rc@s4eZdZUedddZeeed<GdddZdS)ry.r) min_itemsrxc@eZdZdZdS)zTokenPatternSchema.Configr@NrBrCrDrEr*r*r*r.rUrN) rBrCrDrrxr rrrr*r*r*r.ryRs ryc@seZdZUedddZeed<edddZeed<edddZeed<ed d dZ eed <ed d dZ eed<egddZ e eed<egddZ e eed<ed ddZeed<ed ddZeed<ed ddZeed<ed ddZeed<ed ddZeed<edddZeee ee eeeffed<eid dZeeefed!<eid"dZeee efed#<eid$dZeeefed%<ed d&dZeed'<dS)(ModelMetaSchema.z#Two-letter language code, e.g. 'en'rhlangz Model namerKz Model versionversionr`z#Compatible spaCy version identifier spacy_versionrz9Name of parent spaCy package, e.g. spacy or spacy-nightlyparent_packagezIAdditional Python package dependencies, used for the Python package setup requirementszNames of pipeline componentspipelinezModel description descriptionz Model licenselicensezModel author nameauthorzModel author emailemailzModel author URLurlNzTraining data sourcessourceszIncluded word vectorsvectorsz)Component labels, keyed by component namelabelszAccuracy and speed numbers performancezCommit of spaCy version usedspacy_git_version)rBrCrDrrrrrKrrrrr rrrrrrrr rrr+r rr r r r*r*r*r.r\s$ 0 rc@seZdZUedddZeed<edddZeed<edddZe ed<edd dZ e ed <edd dZ e ed <edd dZe ed<edddZe ed<edddZe ed<edddZee ed<edddZeeed<edddZe ed<edddZeeeee e ffed<edddZeed<edddZeed<edddZeeed <edd!dZeeed"<edd#dZ ee!d$gd$fed%<edd&dZ"ee!d$eee#fgd'fed(<Gd)d*d*Z$d'S)+ConfigSchemaTraining.z"Path in the config to the dev datar dev_corpus'Path in the config to the training data train_corpusBatcher for the training databatcher Dropout ratedropoutzBHow many steps to continue without improvement in evaluation scorepatience%Maximum number of epochs to train for max_epochsz+Maximum number of update steps to train for max_stepsz-How often to evaluate during training (steps)eval_frequencyz Random seedseedz$Memory allocator when running on GPU gpu_allocatorz,Whether to divide the batch up into substepsaccumulate_gradientz)zConfigSchemaTraining.Configr@TNrAr*r*r*r.rrGr)%rBrCrDrrrrrrBatcherrrrrrrrrr rrrrrrrr Loggerr!r r+r"r#rr$rrr*r*r*r.r zs( (",r c@seZdZUedddZeed<edddZeeed<edddZ eeed<edd dZ e ed <edd dZ e e ed ged fed <edddZe e d gd fed<edddZe e d gd fed<edddZe eed<edddZe ed<GdddZdS)ConfigSchemaNlp.zThe base language to userrz%The pipeline component names in orderrz)Pipeline components to disable by defaultdisabledzThe tokenizer to use tokenizerz@Optional callback to modify Language class before initializationr#before_creationz\Optional callback to modify nlp object after creation and before the pipeline is constructedafter_creationzHOptional callback to modify nlp object after the pipeline is constructedafter_pipeline_creationzDefault batch size batch_sizezVectors implementationr c@r>)zConfigSchemaNlp.Configr@TNrAr*r*r*r.rrGrN)rBrCrDrrrrrr r(r)rr*r r r+r,r-rpr rr*r*r*r.r's *""r'c@seZdZGdddZdS)ConfigSchemaPretrainEmptyc@r)z ConfigSchemaPretrainEmpty.Configr@Nrr*r*r*r.rrrN)rBrCrDrr*r*r*r.r.sr.c@seZdZUedddZeed<edddZeed<edddZ e eed<edd dZ e eed <edd dZ e ed <edd dZeed<edddZeed<edddZeed<edddZeed<edddZedegefed<GdddZdS)ConfigSchemaPretrain.rrrrrzASaving additional temporary model after n batches within an epoch n_save_everyz Saving model after every n epoch n_save_epochrrrcorpusrrz'Component to find the layer to pretrain componentz(Layer to pretrain. Whole model if empty.layerz2A function that creates the pretraining objective.r% objectivec@r>)zConfigSchemaPretrain.Configr@TNrAr*r*r*r.rrGrN)rBrCrDrrrrrrr0r r1rrr2rrr%r3r+r4r5rrrr*r*r*r.r/s  r/c@seZdZUedddZeeed<edddZee ed<edddZ eeed<edd dZ eeed <edd d Z e eefed <eddd Ze ee eeffed<edddZeedgdfed<edddZeedgdfed<GdddZdS)ConfigSchemaInit.z&Path to JSON-formatted vocabulary filer vocab_dataz-Vocabulary lookups, e.g. lexeme normalizationlookupszPath to vectorsr z"Path to pretrained tok2vec weights init_tok2vecz0Arguments to be passed into Tokenizer.initialize)helpr)zYArguments for TrainablePipe.initialize methods of pipeline components, keyed by component componentsz)zConfigSchemaInit.Configr@TNrAr*r*r*r.rrGrN)rBrCrDrr7r rrr8r!r r9r)rrr;r<rr=rr*r*r*r.r6s $""r6c@speZdZUeed<eed<iZeee fed<e e e e e ffed<e e e fed<eed<GdddZd S) ConfigSchematrainingrb pretrainingr;corporarfc@r>)zConfigSchema.ConfigallowTNrAr*r*r*r.rrGrN)rBrCrDr rr'r@rr/r.rr+rReaderr6rr*r*r*r.r>s r>)rbr?r@rfc@seZdZUeed<eed<dS)RecommendationTrfItemrK size_factorN)rBrCrDr+rrpr*r*r*r.rD  rDc@seZdZUeed<eed<dS)RecommendationTrf efficiencyaccuracyN)rBrCrDrDrr*r*r*r.rGrFrGc@s:eZdZUdZeeed<dZeeed<dZ e ed<dS)RecommendationSchemaN word_vectors transformerT has_letters) rBrCrDrKr r+rrLrGrMboolr*r*r*r.rJs rJc @sleZdZUdZedddZeeee fe d<edddZ ee eee eeffe d<edddZee eeefe d <ed d dZee d <edd dZeeee eee eefffe d<ed ddZe eee eeffe d<eddddZeeeefe d<edddZeeee eeeffe d<edddZeeee eeeffe d<dS) DocJSONSchemazB JSON/dict format for JSON representation of Doc objects. Nz+Categories with corresponding probabilitiesrcatszInformation on entitiesentsz+Indices of sentences' start and end indicessents.z Document textrz2Span information - end/start indices, label, KB IDspansz*Token information - ID, start, annotationstokensz4Any custom data stored in the document's _ attributer)rhrunderscore_docz1Any custom data stored in the token's _ attributeunderscore_tokenz0Any custom data stored in the span's _ attributeunderscore_span)rBrCrD__doc__rrPr rrrrrQr rrrRrrSrTrUrrVrWr*r*r*r.rOs< & "&*rO)]rOr collectionsrenumrtypingrrrrrr r r r r r pydantic.v1rrrrrrrrrrpydantic.v1.mainr ImportErrorpydantic pydantic.main thinc.apirrr thinc.configrattrsrcompatr r8r!utilr"languager#r?r$vocabr%r&r%r+rCr&r=r?rHtuplerNr_rmr5r{r|rrrrrrrpfloatrrrryrr r'r.r/r6r>CONFIG_SCHEMASrDrGrJrOr*r*r*r.s  40 0         "0& 2   " /  3