o i!5 @sddlZddlZddlmZddlmZmZmZmZm Z m Z m Z m Z ddl Z ddlmZddlmZddlmZddlmZdd lmZdd lmZdd lmZmZdd lmZmZdd l m!Z!ddlm"Z"m#Z#ddl$m%Z%ddl&m'Z'e ee e(e)fefZ*ee)e e*ee(ffZ+ee)ee e(e)fe e(e)fffZ,ee)ee)ee e(e)fe e(e)ffffZ-dee!dee)effddZ.ddZ/Gddde'Z0de1de e1e1ffddZ2ddZ3dS)N)Path)AnyCallableDictIterableListOptionalTupleUnion)util)Errors)Language)Matcher)Scorer)IDS)DocSpan)normalize_token_attrsset_token_attrs)Example)SimpleFrozenListregistry)Vocab)PipeexamplesreturncKsdd}i}|tj|dfi||tj|dfi||tj|dfd|i||tj|dfd|i||tj|dfi||S)NcSs t||jSN)getattrkey)tokenattrr#Q/home/ubuntu/.local/lib/python3.10/site-packages/spacy/pipeline/attributeruler.pymorph_key_getters z/attribute_ruler_score..morph_key_gettertagposmorphgetterlemma)updaterscore_token_attrscore_token_attr_per_feat)rkwargsr%resultsr#r#r$attribute_ruler_scores$r0cCstSr)r0r#r#r#r$make_attribute_ruler_scorer.sr1c @seZdZdZ d6deddedededee d d f d d Z d7d dZ d d d d ddee ge e fdeedee edeedeed d f ddZded efddZdefddZddZdeeeeeefeeefffd d fddZdeeeeeeeefeeeffffd d fd d!Z "d8de ed#ed$ed d fd%d&Zde ed d fd'd(Zed e efd)d*Z!e"fd+e ed e#fd,d-Z$e"fd.e#d+e ed dfd/d0Z%e"fd1ee&efd+e ed d fd2d3Z'e"fd1ee&efd+e ed dfd4d5Z(d S)9AttributeRulerzSet token-level attributes for tokens matched by Matcher patterns. Additionally supports importing patterns from tag maps and morph rules. DOCS: https://spacy.io/api/attributeruler attribute_rulerF)validatescorervocabnamer4r5rNcCs>||_||_t|j|d|_||_g|_g|_g|_||_dS)aCreate the AttributeRuler. After creation, you can add patterns with the `.initialize()` or `.add_patterns()` methods, or load patterns with `.from_bytes()` or `.from_disk()`. Loading patterns will remove any patterns you've added previously. vocab (Vocab): The vocab. name (str): The pipe name. Defaults to "attribute_ruler". scorer (Optional[Callable]): The scoring method. Defaults to Scorer.score_token_attr for the attributes "tag", "pos", "morph" and "lemma" and Scorer.score_token_attr_per_feat for the attribute "morph". RETURNS (AttributeRuler): The AttributeRuler component. DOCS: https://spacy.io/api/attributeruler#init r4N) r7r6rmatcherr4attrs_attrs_unnormedindicesr5)selfr6r7r4r5r#r#r$__init__9s zAttributeRuler.__init__cCs(t|j|jd|_g|_g|_g|_dS)zReset all patterns.r8N)rr6r4r9r:r;r<r=r#r#r$clearZs zAttributeRuler.clear)nlppatternstag_map morph_rules get_examplesrArBrCrDcCs:||r |||r|||r||dSdS)aInitialize the attribute ruler by adding zero or more patterns. Rules can be specified as a sequence of dicts using the `patterns` keyword argument. You can also provide rules using the "tag map" or "morph rules" formats supported by spaCy prior to v3. N)r@ add_patternsload_from_tag_mapload_from_morph_rules)r=rErArBrCrDr#r#r$ initializeas  zAttributeRuler.initializedocc Cs\|}z||}||||WSty-}z||j||g|WYd}~Sd}~ww)zApply the AttributeRuler to a Doc and set all attribute exceptions. doc (Doc): The document to process. RETURNS (Doc): The processed Doc. DOCS: https://spacy.io/api/attributeruler#call N)get_error_handlermatchset_annotations Exceptionr7)r=rJ error_handlermatcheser#r#r$__call__xs  zAttributeRuler.__call__cs.j|ddd}fdd|D}||S)NTF) allow_missingas_spanscs*g|]\}}}tjj||||fqSr#)intr6strings).0m_idsrQr?r#r$ sz(AttributeRuler.match..)r9sort)r=rJrPr#r?r$rLs  zAttributeRuler.matchc Cs|D]C\}}}}t||||d}|j|}|j|} z|| } Wnty=ttjj|j |j dd|D| ddwt || |qdS)zModify the document in place)labelcSsg|]}|jqSr#)text)rWtr#r#r$rZsz2AttributeRuler.set_annotations..)rBspanindexN) rr:r< IndexError ValueErrorr E1001formatr9getr\r) r=rJrPattr_idmatch_idstartendr_r:r`r!r#r#r$rMs&       zAttributeRuler.set_annotationscCs|D];\}}d|ig}t|\}}d|vr'|jj|}|jj||d<n|jj|d}|jj||d<||g|qdS)zLoad attribute ruler patterns from a tag map. tag_map (dict): The tag map that maps fine-grained tags to coarse-grained tags and morphological features. DOCS: https://spacy.io/api/attributeruler#load_from_morph_rules TAGMORPHN)items_split_morph_attrsr6 morphologyaddrV)r=rCr&r:pattern morph_attrsr(r#r#r$rGs  z AttributeRuler.load_from_tag_mapcCs|D]I}||D]B}||dg}|||}t|\}}d|vr2|jj|d}|jj||d<n|rC|jj|}|jj||d<||g|qqdS)a+Load attribute ruler patterns from morph rules. morph_rules (dict): The morph rules that map token text and fine-grained tags to coarse-grained tags, lemmas and morphological features. DOCS: https://spacy.io/api/attributeruler#load_from_morph_rules )ORTHrjrkN)rmr6rnrorV)r=rDr&wordrpr:rqr(r#r#r$rHs    z$AttributeRuler.load_from_morph_rulesrr:r`cCsZtt|j}|j|jj|||j|t |j|}|j||j |dS)aBAdd Matcher patterns for tokens that should be modified with the provided attributes. The token at the specified index within the matched span will be assigned the attributes. patterns (Iterable[List[Dict]]): A list of Matcher patterns. attrs (Dict): The attributes to assign to the target token in the matched span. index (int): The index of the token in the matched span to modify. May be negative to index from the end of the span. Defaults to 0. DOCS: https://spacy.io/api/attributeruler#add N) strlenr:r9ror6rVr;appendrr<)r=rBr:r`r r#r#r$ros    zAttributeRuler.addcCs|D] }|jdi|qdS)acAdd patterns from a list of pattern dicts with the keys as the arguments to AttributeRuler.add. patterns (Iterable[dict]): A list of pattern dicts with the keys as the arguments to AttributeRuler.add (patterns/attrs/index) to add as patterns. DOCS: https://spacy.io/api/attributeruler#add_patterns Nr#)ro)r=rBpr#r#r$rFs zAttributeRuler.add_patternscCs^g}tt|jD]#}i}|jt|d|d<|j||d<|j||d<||q |S)zAll the added patterns.rrBr:r`) rangerur:r9rertr;r<rv)r= all_patternsirwr#r#r$rBs zAttributeRuler.patternsexcludecs2i}fdd|d<fdd|d<t|S)zSerialize the AttributeRuler to a bytestring. exclude (Iterable[str]): String names of serialization fields to exclude. RETURNS (bytes): The serialized object. DOCS: https://spacy.io/api/attributeruler#to_bytes csjjdSN)r{)r6to_bytesr#r{r=r#r$ z)AttributeRuler.to_bytes..r6cs tjSr)srsly msgpack_dumpsrBr#r?r#r$rs rB)r r})r=r{ serializer#r~r$r}s zAttributeRuler.to_bytes bytes_datac2fdd}fdd|d}t||S)a'Load the AttributeRuler from a bytestring. bytes_data (bytes): The data to load. exclude (Iterable[str]): String names of serialization fields to exclude. returns (AttributeRuler): The loaded object. DOCS: https://spacy.io/api/attributeruler#from_bytes ct|dSr)rFr msgpack_loadsbr?r#r$ load_patternsz0AttributeRuler.from_bytes..load_patternscjj|dSr|)r6 from_bytesrr~r#r$r!z+AttributeRuler.from_bytes..r6rB)r r)r=rr{r deserializer#r~r$r zAttributeRuler.from_bytespathcs.fddfddd}t||dS)zSerialize the AttributeRuler to disk. path (Union[Path, str]): A path to a directory. exclude (Iterable[str]): String names of serialization fields to exclude. DOCS: https://spacy.io/api/attributeruler#to_disk crr|)r6to_diskrwr~r#r$r2rz(AttributeRuler.to_disk..cst|jSr)r write_msgpackrBrr?r#r$r3rrN)r r)r=rr{rr#r~r$r's zAttributeRuler.to_diskcr)a(Load the AttributeRuler from disk. path (Union[Path, str]): A path to a directory. exclude (Iterable[str]): String names of serialization fields to exclude. RETURNS (AttributeRuler): The loaded object. DOCS: https://spacy.io/api/attributeruler#from_disk crr)rFr read_msgpackrr?r#r$rCrz/AttributeRuler.from_disk..load_patternscrr|)r6 from_diskrr~r#r$rGrz*AttributeRuler.from_disk..r)r r)r=rr{rrr#r~r$r7rzAttributeRuler.from_disk)r3)rN)r))__name__ __module__ __qualname____doc__r0rrtboolrrr>r@rrrAttributeRulerPatternType TagMapTypeMorphRulesTyperIrrRrLrMrr rUrGrHMatcherPatternTyperorFpropertyrrBrbytesr}rrrrr#r#r#r$r22s  !    " *        r2r:cCsTi}i}|D]\}}|dvs|tvs|tvr!|||<q|||<q||fS)zSplit entries from a tag map or morph rules dict into to two dicts, one with the token-level features (POS, LEMMA) and one with the remaining features, which are presumed to be individual MORPH features._)rlrkeysvalues)r: other_attrsrqkvr#r#r$rmNs   rmcCs,|dkr td}|jStdtd|)Nmake_attribute_rulerzspacy.pipeline.factorieszmodule z has no attribute ) importlib import_modulerAttributeErrorr)r7moduler#r#r$ __getattr__]s r)4rsyspathlibrtypingrrrrrrr r rr errorsr languagerr9rr5rsymbolsrtokensrrtokens._retokenizerrtrainingrrrr6rpiperrUrtrrrrr0r1r2dictrmrr#r#r#r$s8 (         $,