o i/@s&ddlZddlZddlZddlmZddlmZmZmZm Z m Z m Z m Z m Z ddlmZddlmZddlmZmZddlmZdd lmZmZdd lmZdd lmZmZdd lm Z dd lm!Z!m"Z"m#Z#ddl$m%Z%ddl&m'Z'de e dee(effddZ)ddZ*Gddde'Z+ddZ,dS)N)Path)AnyCallableDictIterableListOptionalTupleUnion)Model)util)ErrorsWarnings)Language)Lookups load_lookups)Scorer)DocToken)Example)SimpleFrozenListloggerregistry)Vocab)PipeexamplesreturncKstj|dfi|S)Nlemma)rscore_token_attr)rkwargsr"M/home/ubuntu/.local/lib/python3.10/site-packages/spacy/pipeline/lemmatizer.pylemmatizer_scoresr$cCstSN)r$r"r"r"r#make_lemmatizer_scorersr&c@seZdZdZededeeeeeffddZ d3dde d d e d e e d eded e de eddfddZeddZdedefddZ d4dddde egeefde ede efddZejfdeddfddZd edeefd!d"Zd edeefd#d$Zd ede fd%d&Ze d'd(e!ee"fd)eefd*d+Z#e d'd(e!ee"fd)eeddfd,d-Z$e d'd)eede%fd.d/Z&e d'd0e%d)eeddfd1d2Z'dS)5 Lemmatizerz The Lemmatizer supports simple part-of-speech-sensitive suffix rules and lookup tables. DOCS: https://spacy.io/api/lemmatizer modercCs0|dkr dggfS|dkrdgddgfSggfS)a Returns the lookups configuration settings for a given mode for use in Lemmatizer.load_lookups. mode (str): The lemmatizer mode. RETURNS (Tuple[List[str], List[str]]): The required and optional lookup tables for this mode. lookup lemma_lookuprule lemma_rules lemma_exc lemma_indexr")clsr(r"r"r#get_lookups_config%s  zLemmatizer.get_lookups_config lemmatizerr)F)r( overwritescorervocabmodelnamer2r3NcCs||_||_||_||_t|_||_d|_|jdkr |j |_ n$|jdkr*|j |_ n|jd}t ||s>t tjj|dt|||_ i|_||_dS)a&Initialize a Lemmatizer. vocab (Vocab): The vocab. model (Model): A model (not yet implemented). name (str): The component name. Defaults to "lemmatizer". mode (str): The lemmatizer mode: "lookup", "rule". Defaults to "lookup". overwrite (bool): Whether to overwrite existing lemmas. Defaults to `False`. scorer (Optional[Callable]): The scoring method. Defaults to Scorer.score_token_attr for the attribute "lemma". DOCS: https://spacy.io/api/lemmatizer#init Fr)r+ _lemmatize)r(N)r4r5r6_moderlookupsr2 _validatedr(lookup_lemmatize lemmatizerule_lemmatizehasattr ValueErrorrE1003formatgetattrcacher3)selfr4r5r6r(r2r3 mode_attrr"r"r#__init__4s"        zLemmatizer.__init__cCs|jSr%)r8rDr"r"r#r(^szLemmatizer.modedocc Cs|js |tj|}z|D]}|js|jdkr"||d|_q|WSt yA}z||j ||g|WYd}~dSd}~ww)zApply the lemmatizer to one document. doc (Doc): The Doc to process. RETURNS (Doc): The processed Doc. DOCS: https://spacy.io/api/lemmatizer#call rN) r:_validate_tablesrE1004get_error_handlerr2rr<lemma_ Exceptionr6)rDrH error_handlertokener"r"r#__call__bs  zLemmatizer.__call__)nlpr9 get_examplesrRr9cCsx||j\}}|dur1tdt|jj|d}t|jj|dd}|jD] }||| |q%||_ | t j dS)aInitialize the lemmatizer and load in data. get_examples (Callable[[], Iterable[Example]]): Function that returns a representative sample of gold-standard Example objects. nlp (Language): The current nlp object the component is part of. lookups (Lookups): The lookups object containing the (optional) tables such as "lemma_rules", "lemma_index", "lemma_exc" and "lemma_lookup". Defaults to None. Nz2Lemmatizer: loading tables from spacy-lookups-data)langtablesF)rTrUstrict)r0r(rdebugrr4rTrU set_table get_tabler9rIrrJ)rDrSrRr9required_tablesoptional_tablesoptional_lookupstabler"r"r# initializeus   zLemmatizer.initialize error_messagecCsH||j\}}|D]}||jvrt|j|j||jjdq d|_dS)z8Check that the lookups are correct for the current mode.)r(rUfoundTN)r0r(r9r?rArUr:)rDr_rZr[r]r"r"r#rIs  zLemmatizer._validate_tablesrOcCs2|jdi}||j|j}t|tr|g}|S)zLemmatize using a lookup-based approach. token (Token): The token to lemmatize. RETURNS (list): The available lemmas for the string. DOCS: https://spacy.io/api/lemmatizer#lookup_lemmatize r*)r9rYgettext isinstancestr)rDrO lookup_tableresultr"r"r#r;s  zLemmatizer.lookup_lemmatizecCs|j|j|jjf}||jvr|j|S|j}|j}|dvr.|dkr)t t j |gS| |r8|gS|j di}|j di}|j di}t||||||fsi|dkrd|gS|gS||i}||i} ||i} |} |}g} g} | D]8\}}||r|dt|t||}|sq||vs|s||vr| d|q| |q| |qtt| } | |gD] }|| vr| d|q| s| | | s| | | |j|<| S) zLemmatize using a rule-based approach. token (Token): The token to lemmatize. RETURNS (list): The available lemmas for the string. DOCS: https://spacy.io/api/lemmatizer#rule_lemmatize )eolspacergr.r-r,propnNr)orthposmorphkeyrCrbpos_lowerwarningswarnrW108 is_base_formr9rYanyraendswithlenisalphainsertappendlistdictfromkeysextend)rDrO cache_keystringuniv_pos index_table exc_table rules_tableindex exceptionsrulesorigforms oov_formsoldnewformr"r"r#r=sh                   zLemmatizer.rule_lemmatizecCsdS)aCheck whether the token is a base form that does not need further analysis for lemmatization. token (Token): The token. RETURNS (bool): Whether the token is a base form. DOCS: https://spacy.io/api/lemmatizer#is_base_form Fr")rDrOr"r"r#rts zLemmatizer.is_base_formexcludepathrcs8i}fdd|d<fdd|d<t||dS)zSerialize the pipe to disk. path (str / Path): Path to a directory. exclude (Iterable[str]): String names of serialization fields to exclude. DOCS: https://spacy.io/api/lemmatizer#to_disk cjj|dSNr)r4to_diskprrDr"r# z$Lemmatizer.to_disk..r4c j|Sr%)r9rrrGr"r#r  r9N)r r)rDrr serializer"rr#rs zLemmatizer.to_diskc@i}fdd|d<fdd|d<t||S)aHLoad the pipe from disk. Modifies the object in place and returns it. path (str / Path): Path to a directory. exclude (Iterable[str]): String names of serialization fields to exclude. RETURNS (Lemmatizer): The modified Lemmatizer object. DOCS: https://spacy.io/api/lemmatizer#from_disk crr)r4 from_diskrrr"r#rrz&Lemmatizer.from_disk..r4crr%)r9rrrGr"r#rrr9)r rrI)rDrr deserializer"rr#r   zLemmatizer.from_diskcs.i}fdd|d<jj|d<t|S)zSerialize the pipe to a bytestring. exclude (Iterable[str]): String names of serialization fields to exclude. RETURNS (bytes): The serialized object. DOCS: https://spacy.io/api/lemmatizer#to_bytes csjjdSr)r4to_bytesr"rr"r#r(sz%Lemmatizer.to_bytes..r4r9)r9rr )rDrrr"rr#rs  zLemmatizer.to_bytes bytes_datacr)aLoad the pipe from a bytestring. bytes_data (bytes): The serialized pipe. exclude (Iterable[str]): String names of serialization fields to exclude. RETURNS (Lemmatizer): The loaded Lemmatizer. DOCS: https://spacy.io/api/lemmatizer#from_bytes crr)r4 from_bytesbrr"r#r8rz'Lemmatizer.from_bytes..r4crr%)r9rrrGr"r#r9rr9)r rrI)rDrrrr"rr#r,rzLemmatizer.from_bytes)r1r%)(__name__ __module__ __qualname____doc__ classmethodrdr rr0r$rrr boolrrFpropertyr(rrQrrrrr^rE912rIrr;r=rtrr rrrbytesrrr"r"r"r#r's$ *  F     r'cCs,|dkr td}|jStdtd|)Nmake_lemmatizerzspacy.pipeline.factorieszmodule z has no attribute ) importlib import_modulerAttributeErrorr)r6moduler"r"r# __getattr__@s r)-rsysrqpathlibrtypingrrrrrrr r thinc.apir rgr errorsrrlanguagerr9rrr3rtokensrrtrainingrrrrr4rpiperrdr$r&r'rr"r"r"r#s, (        %