o i1J@sddlZddlZddlZddlmZddlmZddlmZm Z m Z m Z m Z m Z mZmZmZmZmZddlZddlmZddlmZmZddlmZdd lmZmZdd lmZdd l m!Z!dd l"m#Z#m$Z$dd l%m&Z&ddlm'Z'm(Z(m)Z)ddl*m+Z+e e,ee,e e e,efffZ-dZ.de e$de e$de e$fddZ/ddZ0de e$de e$de e$fddZ1ddZ2e.dde e&de e,effdd Z3e.fd!e,fd"d#Z4Gd$d%d%e+Z5d&d'Z6dS)(N)partial)Path) AnyCallableDictIterableListOptionalSequenceSetTupleUnioncast)util)ErrorsWarnings)Language)Matcher PhraseMatcher)levenshtein_compare)Scorer)DocSpan)Example)SimpleFrozenList ensure_pathregistry)Piperulerentitiesspansreturncsdd}t||dd}t|}g}t|D]*}|j|jtfdd|Dr@||fdd|D}tq||S) aMerge entities and spans into one list without overlaps by allowing spans to overwrite any entities that they overlap with. Intended to replicate the overwrite_ents=True behavior from the EntityRuler. entities (Iterable[Span]): The entities, already filtered for overlaps. spans (Iterable[Span]): The spans to merge, may contain overlaps. RETURNS (List[Span]): Filtered list of non-overlapping spans. cS|j|j|j fSNendstartspanr+M/home/ubuntu/.local/lib/python3.10/site-packages/spacy/pipeline/span_ruler.py0z,prioritize_new_ents_filter..Tkeyreversec3|]}|jvVqdSr%i.0token seen_tokensr+r, 8z-prioritize_new_ents_filter..cs$g|]}|jkr|jks|qSr+)r(r')r6er&r+r, :s$z.prioritize_new_ents_filter..) sortedlistsetr(r'allappendupdaterange)r!r" get_sort_key new_entitiesr*r+)r'r9r(r,prioritize_new_ents_filter%s  rGcCtSr%)rGr+r+r+r,make_prioritize_new_ents_filter?rIcsdd}t||dd}t|}g}tjdd|D|D] }|j}|j}tfdd|Dr@||t||q ||S)aMerge entities and spans into one list without overlaps by prioritizing existing entities. Intended to replicate the overwrite_ents=False behavior from the EntityRuler. entities (Iterable[Span]): The entities, already filtered for overlaps. spans (Iterable[Span]): The spans to merge, may contain overlaps. RETURNS (List[Span]): Filtered list of non-overlapping spans. cSr$r%r&r)r+r+r,r-Nr.z1prioritize_existing_ents_filter..Tr/css|] }t|j|jVqdSr%)rDr(r')r6entr+r+r,r:Ssz2prioritize_existing_ents_filter..c3r2r%r3r5r8r+r,r:Wr;) r>r?r@rCr(r'rArBrD)r!r"rErFr*r(r'r+r8r,prioritize_existing_ents_filterCs  rLcCrHr%)rLr+r+r+r,"make_preserve_existing_ents_filter]rJrM spans_keyexamplesc srt|}d|d|dd|dd|dfdd|d fd dtj|fi|S) Nspans_attr allow_overlapTlabeledgettercs|j|tdgSr%)r"getlen)docr0) attr_prefixr+r,r-jz1overlapping_labeled_spans_score..has_annotationcs |jvSr%)r")rXrNr+r,r-ls )dict setdefaultr score_spans)rPrOkwargsr+)rYrOr,overlapping_labeled_spans_scoreas   r`rOcCs tt|dS)NrN)rr`rNr+r+r,%make_overlapping_labeled_spans_scorerps rac@sveZdZdZ dFeddejdeddee edd de de d e e d e e eeeegeefd ed e eeeegeefd e eee fde dedede e ddfddZdefddZde defddZede e fddZdedefddZdefdd Zd!d"Zedee d#ffd$d%Zedee d#ffd&d'Zddd(d)e geefde e d*e e e!fd+d,Z"ede#e!fd-d.Z$d*e#e!ddfd/d0Z%dGd1d2Z&de ddfd3d4Z'd5e ddfd6d7Z(dGd8d9Z)e*d:d;e+dZ,e*d:d.) _require_patternswarningscatch_warningsfilterwarningsrrr intr?matcherphrase_matcherr@r>)rprXrdeduplicated_matchesr+rr,r{s  zSpanRuler.matchcCs|jr(g}|j|jvr|js|j|j}||jr|||n|||j|j<|jrPg}|js5t|j}|||}zt ||_WdSt yOt t j wdS)zModify the document in placeN) r0r"rjextendrdrer?entsrfr> ValueErrorrE854)rprXrr"r+r+r,r|s&      zSpanRuler.set_annotations.cCstttdd|jDS)zAll labels present in the match patterns. RETURNS (set): The string labels. DOCS: https://spacy.io/api/spanruler#labels cSsg|] }tt|dqSru)rstrr6pr+r+r,r=rZz$SpanRuler.labels..tupler>r@rrrsr+r+r,labelsszSpanRuler.labelscCs&tttdd|jDtdgS)zAll IDs present in the match patterns. RETURNS (set): The string IDs. DOCS: https://spacy.io/api/spanruler#ids cSsg|] }tt|dqSr)rrrVrr+r+r,r="z!SpanRuler.ids..Nrrsr+r+r,idss z SpanRuler.ids)rlpatterns get_examplesrcCs||r ||dSdS)aInitialize the pipe for training. get_examples (Callable[[], Iterable[Example]]): Function that returns a representative sample of gold-standard Example objects. nlp (Language): The current nlp object the component is part of. patterns (Optional[Iterable[PatternType]]): The list of patterns. DOCS: https://spacy.io/api/spanruler#initialize N)ro add_patterns)rprrlrr+r+r, initialize%szSpanRuler.initializecCry)zGet all patterns that were added to the span ruler. RETURNS (list): The original patterns, one dictionary per pattern. DOCS: https://spacy.io/api/spanruler#patterns )rrrsr+r+r,r9szSpanRuler.patternscCsz&d}t|jjD]\}\}}||kr|}nq dd|jj|dD}Wn ty1g}Ynw|jj|dg}g}|D]]} tt| d} tt| dd} t | | f} | | d |j |jj j | <t| d trz|| || d nt| d tr|j| | d gn ttjj| d d |j| q@t||j|D] \} } |j| | gqWddS1swYdS) aAdd patterns to the span ruler. A pattern can either be a token pattern (list of dicts) or a phrase pattern (string). For example: {'label': 'ORG', 'pattern': 'Apple'} {'label': 'ORG', 'pattern': 'Apple', 'id': 'apple'} {'label': 'GPE', 'pattern': [{'lower': 'san'}, {'lower': 'francisco'}]} patterns (list): The patterns to add. DOCS: https://spacy.io/api/spanruler#add_patterns cSsg|]}|qSr+r+)r6piper+r+r,r=Wz*SpanRuler.add_patterns..N)disablerur)rurpattern)r) enumeraterlpipeline pipe_namesr select_pipesrrrVreprrnvocabstringsas_int isinstancerBr?raddrE097formatrrziprr)rpr current_indexr4rmrsubsequent_pipesphrase_pattern_labelsphrase_pattern_textsentryp_labelp_idrurr+r+r,rCsF     "zSpanRuler.add_patternscCs:g|_t|jj|j|jd|_t|jj|j|jd|_ dS)zfReset all patterns. RETURNS: None DOCS: https://spacy.io/api/spanruler#clear )ri fuzzy_compare)rRriN) rrrrlrrirhrrrgrrsr+r+r,ross zSpanRuler.clearcs|vrttjjd|jdfdd|jD|_|jD])}|j|dkrG|jjj |}||j vr<|j |||j vrG|j |qdS)zRemove a pattern by its label. label (str): Label of the pattern to be removed. RETURNS: None DOCS: https://spacy.io/api/spanruler#remove ru attr_typeru componentcsg|] }|dkr|qSrr+rrr+r,r=rz$SpanRuler.remove..N)rrE1024rrmrrrnrlrr as_stringrremover)rprum_label m_label_strr+rr,rs     zSpanRuler.remove pattern_idcst|}fdd|jD|_|t|kr!ttjjd|jd|jD])}|j|dkrM|jj j |}||j vrB|j |||jvrM|j |q$dS)zRemove a pattern by its pattern ID. pattern_id (str): ID of the pattern to be removed. RETURNS: None DOCS: https://spacy.io/api/spanruler#remove_by_id csg|] }|dkr|qSr)rVrrr+r,r=sz*SpanRuler.remove_by_id..IDrrN)rWrrrrrrrmrnrlrrrrrr)rprorig_lenrrr+rr, remove_by_ids"      zSpanRuler.remove_by_idcCs*t|dkrttjj|jddSdS)z:Raise a warning if this component has no patterns defined.r)rmN)rWrwarnrW036rrmrsr+r+r,rs zSpanRuler._require_patterns)exclude bytes_datarcs*dfddi}t|||S)zLoad the span ruler from a bytestring. bytes_data (bytes): The bytestring to load. RETURNS (SpanRuler): The loaded span ruler. DOCS: https://spacy.io/api/spanruler#from_bytes rct|Sr%)rsrsly json_loads)brsr+r,r-rz&SpanRuler.from_bytes..)ror from_bytes)rprr deserializersr+rsr,rs  zSpanRuler.from_bytescsdfddi}t||S)zSerialize the span ruler to a bytestring. RETURNS (bytes): The serialized patterns. DOCS: https://spacy.io/api/spanruler#to_bytes rcs tjSr%)r json_dumpsrr+rsr+r,r-s z$SpanRuler.to_bytes..)rto_bytes)rpr serializersr+rsr,rs  zSpanRuler.to_bytespathcs2t|}dfddi}t||iS)zLoad the span ruler from a directory. path (Union[str, Path]): A path to a directory. RETURNS (SpanRuler): The loaded span ruler. DOCS: https://spacy.io/api/spanruler#from_disk rcrr%)rr read_jsonlrrsr+r,r-rz%SpanRuler.from_disk..)rorr from_disk)rprrrr+rsr,rs   zSpanRuler.from_diskcs*t|}dfddi}t||idS)zSave the span ruler patterns to a directory. path (Union[str, Path]): A path to a directory. DOCS: https://spacy.io/api/spanruler#to_disk rcst|jSr%)r write_jsonlrrrsr+r,r-sz#SpanRuler.to_disk..N)rrto_disk)rprrrr+rsr,rs zSpanRuler.to_disk)rc)r#N)1__name__ __module__ __qualname____doc__DEFAULT_SPANS_KEYrfilter_chain_spansrrr`rrr rrrboolr rrqrtrxpropertyr0rrr{r|r rrrr PatternTyperrrrrorrrrbytesrrrrrr+r+r+r,rbts     H    0      rbcCsD|dkr td}|jS|dkrtd}|jStdtd|)Nmake_span_rulerzspacy.pipeline.factoriesmake_entity_rulerzmodule z has no attribute ) importlib import_modulermake_future_entity_rulerAttributeErrorr)rmmoduler+r+r, __getattr__s  r)7rsysr functoolsrpathlibrtypingrrrrrr r r r r rrrrerrorsrrlanguagerrrrmatcher.levenshteinrrkrtokensrrtrainingrrrrrrrrrrGrIrLrMr`rarbrr+r+r+r,s^  4