o ia @s6ddlZddlZddlmZddlmZddlmZmZm Z m Z m Z m Z m Z mZmZddlZddlmZmZmZmZmZmZddlmZmZmZmZddlmZmZdd l m!Z!dd l"m#Z#dd l$m%Z%dd l&m'Z'm(Z(m)Z)dd l*m+Z+m,Z,ddl-m.Z.ddl/m0Z0ddl1m2Z2dZ3dZ4dZ5e6e3dZ7e6e4dZ8eGdddeZ9ddde e'de e:de edefddZ;ddde e'ded$e:d%e:de9fd&d'Z?derr<rF) r+rRr*rHrIrJrLspanrOrPr/r/r1preset_spans_suggesterss"    rWcC tt|dS)zSuggest all spans of the given lengths. Spans are returned as a ragged array of integers. The array has two columns, indicating the start and end position.r6)rrQrYr/r/r1build_ngram_suggesters rZmin_sizemax_sizecCstt||d}t|S)zSuggest all spans of the given lengths between a given min and max value - both inclusive. Spans are returned as a ragged array of integers. The array has two columns, indicating the start and end position.r$)listrangerZ)r[r\r6r/r/r1build_ngram_range_suggestersr_cCrX)zSuggest all spans that are already stored in doc.spans[spans_key]. This is useful when an upstream component is used to set the spans on the Doc such as a SpanRuler or SpanFinder.rR)rrWr`r/r/r1build_preset_spans_suggesters raexamplesc snt|}d|d|d|dd|dfdd|d fd dtj|fi|S) Nspans_rRattr allow_overlapTgettercs|j|tdgSr.)rHgetr>)rJkey) attr_prefixr/r1szspancat_score..has_annotationcs |jvSr.)rH)rJ)rhr/r1rjs )dict setdefaultr score_spans)rbkwargsr/)rirhr1 spancat_scores  rpcCstSr.)rpr/r/r/r1make_spancat_scorersrqc@s(eZdZdZddZddZddZdS) _Intervalsz: Helper class to avoid storing overlapping spans. cCs t|_dSr.)setrangesr0r/r/r1__init__s z_Intervals.__init__cCs t||D]}|j|qdSr.)r^rtadd)r0r7jer/r/r1rwsz_Intervals.addcCs,|\}}t||D] }||jvrdSq dS)NTF)r^rt)r0rangr7rxryr/r/r1 __contains__s  z_Intervals.__contains__N)r3r4r5__doc__rvrwr{r/r/r/r1rrs  rrc@steZdZdZ dPdddddded d ed eeee e fe fd e d e dede deedeedeedeedeeddfddZede fddZdQddZde defddZedee fdd Zedee fd!d"Zedee effd#d$Zedefd%d&Zedeedffd'd(Zd)ee fd*d+Z d,d-d)ee d.e ddfd/d0Z!d)ee ddfd1d2Z"d3ddd4d5ee#d6ed7ee$d8eee efdee eff d9d:Z%d5ee#d;ee e fdeeeffdd?egee#fd@ee'dAeee ddfdBdCZ(d5ee#fdDdEZ)dFe#fdGdHZ*dIe dJe+dKe de,fdLdMZ- dRdIe dJe+dKe dede,f dNdOZ.dS)SSpanCategorizerz_Pipeline component to label spans of text. DOCS: https://spacy.io/api/spancategorizer spancatFrH?TNg?)add_negative_labelrRnegative_weightre max_positive thresholdscorervocabr' suggesternamerrRrrerrrr,c Csjg|| | ||d|_||_||_||_||_| |_||_|s/| dur1| dkr3ttj j | ddSdSdS)a}Initialize the multi-label or multi-class span categorizer. vocab (Vocab): The shared vocabulary. model (thinc.api.Model): The Thinc Model powering the pipeline component. For multi-class classification (single label per span) we recommend using a Softmax classifier as a the final layer, while for multi-label classification (multiple possible labels per span) we recommend Logistic. suggester (Callable[[Iterable[Doc], Optional[Ops]], Ragged]): A function that suggests spans. Spans are returned as a ragged array with two integer columns, for the start and end positions. name (str): The component instance name, used to add entries to the losses during training. spans_key (str): Key of the Doc.spans dict to save the spans under. During initialization and training, the component will look for spans on the reference document under the same key. Defaults to `"spans"`. add_negative_label (bool): Learn to predict a special 'negative_label' when a Span is not annotated. threshold (Optional[float]): Minimum probability to consider a prediction positive. Defaults to 0.5. Spans with a positive prediction will be saved on the Doc. max_positive (Optional[int]): Maximum number of labels to consider positive per span. Defaults to None, indicating no limit. negative_weight (float): Multiplier for the loss terms. Can be used to downweight the negative samples if there are too many when add_negative_label is True. Otherwise its unused. allow_overlap (bool): If True the data is assumed to contain overlapping spans. Otherwise it produces non-overlapping spans greedily prioritizing higher assigned label scores. Only used when max_positive is 1. scorer (Optional[Callable]): The scoring method. Defaults to Scorer.score_spans for the Doc.spans[spans_key] with overlapping spans allowed. DOCS: https://spacy.io/api/spancategorizer#init )labelsrRrrrreNr$)r) cfgrrr'rrr ValueErrorrE1051format) r0rr'rrrrRrrerrrr/r/r1rvs 3 zSpanCategorizer.__init__cCt|jdS)zKey of the doc.spans dict to save the spans under. During initialization and training, the component will look for spans on the reference document under the same key. rR)strrrur/r/r1rhzSpanCategorizer.keycCsd}|jdr|jd}n|jdr'|jddr'|jdd}|durB||jkrD|jsFttj j |j |jdddSdSdS)zKsz.SpanCategorizer._label_map..) enumeraterrur/r/r1 _label_mapHszSpanCategorizer._label_mapcCs|jr t|jdSt|jS)z RETURNS (int): Number of labels.r$)rr>rrur/r/r1rMs zSpanCategorizer._n_labelscCs|jrt|jSdS)z8RETURNS (Union[int, None]): Index of the negative label.N)rr>rrur/r/r1_negative_label_iUs z!SpanCategorizer._negative_label_ir+cCsP|j||jjd}|jdkr|jjdd}||fS|j||f}||fS)zApply the pipeline's model to a batch of docs, without modifying them. docs (Iterable[Doc]): The documents to predict. RETURNS: The models prediction for each document. DOCS: https://spacy.io/api/spancategorizer#predict r)r)rr'r*rIsumalloc2fpredict)r0r+indicesscoresr/r/r1r]s zSpanCategorizer.predict candidates)candidates_keyrcCs`|j||jjd}t||D]\}}g|j|<|jD]}|j|||d|dqqdS)aoUse the spancat suggester to add a list of span candidates to a list of docs. This method is intended to be used for debugging purposes. docs (Iterable[Doc]): The documents to modify. candidates_key (str): Key of the Doc.spans dict to save the candidate spans under. DOCS: https://spacy.io/api/spancategorizer#set_candidates r)rr$N)rr'r*ziprHrGr@)r0r+rsuggester_outputrrJindexr/r/r1set_candidatesls   "zSpanCategorizer.set_candidatesc Cs|\}}d}t|D]I\}}||j}tt|jd} |jddkr8|||||||j|| |j|j<n| ||||||j||j|j<||j|7}q dS)aModify a batch of Doc objects, using pre-computed scores. docs (Iterable[Doc]): The documents to modify. scores: The scores to set, produced by SpanCategorizer.predict. DOCS: https://spacy.io/api/spancategorizer#set_annotations rrerr$N) rrGr boolr_make_span_group_singlelabelrIrHrh_make_span_group_multilabel) r0r+indices_scoresrroffsetr7rJ indices_irer/r/r1set_annotations~s&  zSpanCategorizer.set_annotations)dropsgdlossesrbrrrc Cs|duri}||jdt|d||tdd|Ds"|Sdd|D}|j||jjd}|j d kr;|St |j||j ||f\}}| |||f\} } || |durb| |||j| 7<|S) a1Learn from a batch of documents and gold-standard information, updating the pipe's model. Delegates to predict and get_loss. examples (Iterable[Example]): A batch of Example objects. drop (float): The dropout rate. sgd (thinc.api.Optimizer): The optimizer. losses (Dict[str, float]): Optional record of the loss during training. Updated using the component name as the key. RETURNS (Dict[str, float]): The updated losses dictionary. DOCS: https://spacy.io/api/spancategorizer#update NrSpanCategorizer.updatecss$|] }|jr t|jndVqdS)rN) predictedr>regr/r/r1 s"z)SpanCategorizer.update..cSg|]}|jqSr/)rrr/r/r1 z*SpanCategorizer.update..r)r)rmrr!_validate_categoriesanyrr'r*rIrr begin_updateget_loss finish_update) r0rbrrrr+rHrbackprop_scoreslossd_scoresr/r/r1updates&    r spans_scorescCs|\}}t|jj|j|jj|j}tj|j|j d}|j r)t |jd}d}|j }t |D]\\} } i} || j} t|j| D]} t| | df}t| | df}|| | ||f<qD|| D]"}|j|jf}|| vr| |}||j}d|||f<|j rd||<qd||j| 7}q2|jjj|dd}|j rt|d}d|||jf<||}|j rtt|jd}|dkr|||9<t|d}||fS) akFind the loss and gradient of loss for the batch of documents and their predicted scores. examples (Iterable[Examples]): The batch of examples. spans_scores: Scores representing the model's predictions. RETURNS (Tuple[float, float]): The loss and the gradient. DOCS: https://spacy.io/api/spancategorizer#get_loss r8rr$rrfrr)rr'r*to_numpydatarInumpyrFrBr9ronesrrrGr^int_get_aligned_spansrSrTlabel_rUnonzerorr floatrr)r0rbrrHrtargetnegative_spansr label_mapr7r spans_indexspans_irxrSrT gold_spanrhrowknegative_samplesr neg_weightrr/r/r1rsJ     zSpanCategorizer.get_loss)nlpr get_examplesrrc Csg}|dur|D]}||q|D]#}|dur+|jj|jgD]}||jq"t|dkr6||q||redd|D}t dgd|} |j j | j jd|j} |j j|| f| ddS|j dS) a|Initialize the pipe for training, using a representative set of data examples. get_examples (Callable[[], Iterable[Example]]): Function that returns a representative sample of gold-standard Example objects. nlp (Optional[Language]): The current nlp object the component is part of. labels (Optional[List[str]]): The labels to add to the component, typically generated by the `init labels` command. If no labels are provided, the get_examples callback is used to extract the labels from the data. DOCS: https://spacy.io/api/spancategorizer#initialize N cSrr/)xrr/r/r1r rz.SpanCategorizer.initialize..r$rYr)XY)r referencerHrgrhrr>r@_require_labelsrZr'r*rrGrBr initialize) r0rrrsubbatchrrrVr+rHrr/r/r1rs$    zSpanCategorizer.initializecCr-r.r/)r0rbr/r/r1r'sz$SpanCategorizer._validate_categoriesrcCs|j|jj|jgddS)NT)re)get_aligned_spans_y2xrrHrgrh)r0rr/r/r1r+sz"SpanCategorizer._get_aligned_spansrJrrc Cst||jd}|jdkr|S|jj|}|jj|}|jd}|jd}||k}|dur~t|ts5J|j r_t |dd|j f}t j |dd|j f<|d} ||dd|j f<n|d} | dd|df} t| D] \} } d|| | f<qsg} t|jdD]7} || df}|| df}t|| D]"\}}|r||j kr|t||||j|d | || |fqqt | |jd <|S) z5Find the top-k labels for each span (k=max_positive).rrrrNr:Fr$rr)rrhrMr'r*rrrrrrcopyrinfargsortrr^rBr@rrarrayattrs)r0rJrrrHrrkeepsnegative_scoresranked span_filterr7r attrs_scoresrSrTrxkeepr/r/r1r0s@        z+SpanCategorizer._make_span_group_multilabelc Cs|jdkr t||jdS|jj|}|jj|}|jdd}tj|t |ddd}tj |j t d}|j r@t|||jk}|jd}|durSt|||k}|sm|d} || }|| }|| }|| }t} t||jd} g} t|j dD];} || sq|| }|| df}|| df}|s||f| vrq| ||| || | t||||j|d qt| | jd <| S) z$Find the argmax label for each span.rrr$)axisr8rNr:rr)rMrrhr'r*rargmaxrtake_along_axis expand_dimsrrBrr logical_andrrsqueezerrrr^rwr@rrrr)r0rJrrrer argmax_scoresrrsort_idxseenrHrr7rrSrTr/r/r1rYsJ       z,SpanCategorizer._make_span_group_singlelabel)r~)r,N)T)/r3r4r5r|rpr#rr rrrrr(rrr rrrrvpropertyrhrrrrrrrr rrrrrr rrrrrrrrrrrr/r/r/r1r}s       C     )   B  & .r}cCsD|dkr td}|jS|dkrtd}|jStdtd|)N make_spancatzspacy.pipeline.factoriesmake_spancat_singlelabelzmodule z has no attribute ) importlib import_modulerrAttributeErrorr3)rmoduler/r/r1 __getattr__s  r )Frsys dataclassesr functoolsrtypingrrrrrr r r r r thinc.apir rrrrr thinc.typesrrrrcompatrrerrorsrlanguagerrrtokensrrrtrainingr r!utilr"rr#trainable_piper%spancat_default_config"spancat_singlelabel_default_configDEFAULT_SPANS_KEYfrom_strDEFAULT_SPANCAT_MODEL!DEFAULT_SPANCAT_SINGLELABEL_MODELr(rrQrrWrZr_rarprqrrr}r r/r/r/r1sx  ,           N