o iS)@s,ddlZddlZddlmZmZmZmZmZmZm Z ddl m Z m Z m Z mZddlmZddlmZddlmZddlmZdd lmZmZdd lmZdd lmZd d lmZd dl m!Z!dZ"e #e"dZ$ddZ%deedee&effddZ'dede e(e(ffddZ)Gddde!Z*ddZ+dS)N)AnyCallableDictIterableListOptionalTuple)ConfigModel Optimizerset_dropout_rate)Floats2d)Errors)Language)Scorer)DocSpan)Example)registry)DEFAULT_SPANS_KEY) TrainablePipea [model] @architectures = "spacy.SpanFinder.v1" [model.scorer] @layers = "spacy.LinearLogistic.v1" nO = 2 [model.tok2vec] @architectures = "spacy.Tok2Vec.v2" [model.tok2vec.embed] @architectures = "spacy.MultiHashEmbed.v2" width = 96 rows = [5000, 1000, 2500, 1000] attrs = ["NORM", "PREFIX", "SUFFIX", "SHAPE"] include_static_vectors = false [model.tok2vec.encode] @architectures = "spacy.MaxoutWindowEncoder.v2" width = ${model.tok2vec.embed.width} window_size = 1 maxout_pieces = 3 depth = 4 modelcCstSN)span_finder_scorerrN/home/ubuntu/.local/lib/python3.10/site-packages/spacy/pipeline/span_finder.pymake_span_finder_scorer.srexamplesreturnc st|}d|d|d|dfdd|dfdd|d d |d d tj|fi|}||dd d|S)Nspans_ spans_keyattrgettercs|j|tdgSr)spansgetlen)dockey) attr_prefixrr8sz#span_finder_score..has_annotationcs |jvSr)r%)r()r)rrr+:s allow_overlapTlabeledF _per_type)dict setdefaultr score_spanspop)rkwargsscoresr)r*r)rr2s   rspancCs(|dj}|djt|d}||fS)Nr)idxr')r6startendrrr _char_indicesBs r;c@s0eZdZdZ d'edddeddedeee e fde d e d e d e ed e ed e eddfddZdee fddZdee de ddfddZdddddeede de ede ee e fdee e ff ddZdee e ffdd Zdee e ffd!d"Zdd#d$egeefde eddfd%d&ZdS)( SpanFinderzUPipeline that learns span boundaries. DOCS: https://spacy.io/api/spanfinder span_finderg?N)r" threshold max_length min_lengthscorernlprnamer"r>r?r@rAr c Csb|j|_|dur |dks|dur|dkrttjj||d||_||_||_||||d|_dS)aInitialize the span finder. model (thinc.api.Model): The Thinc Model powering the pipeline component. name (str): The component instance name, used to add entries to the losses during training. threshold (float): Minimum probability to consider a prediction positive. scorer (Optional[Callable]): The scoring method. spans_key (str): Key of the doc.spans dict to save the spans under. During initialization and training, the component will look for spans on the reference document under the same key. max_length (Optional[int]): Maximum length of the produced spans, defaults to None meaning unlimited length. min_length (Optional[int]): Minimum length of the produced spans, defaults to None meaning shortest span length is 1. DOCS: https://spacy.io/api/spanfinder#init Nr)r@r?)r@r?r>r") vocab ValueErrorrE1053formatrrCrAcfg) selfrBrrCr"r>r?r@rArrr__init__Ns zSpanFinder.__init__docscCs|j|}|S)aApply the pipeline's model to a batch of docs, without modifying them. docs (Iterable[Doc]): The documents to predict. RETURNS: The models prediction for each document. DOCS: https://spacy.io/api/spanfinder#predict )rpredict)rIrKr5rrrrL}s zSpanFinder.predictr5c Cs&d}t|D]\}}g|j|jd<g}g}|||t|}t||D]"\} } | d|jdkr8|| j| d|jdkrG|| jq%|D]?} |D]:} | d| } | dkr[qN|jddusi|jd| kr|jddusw| |jdkr|j|jd|| | dqNqJ|t|7}qdS)aModify a batch of Doc objects, using pre-computed scores. docs (Iterable[Doc]): The documents to modify. scores: The scores to set, produced by SpanFinder predict method. DOCS: https://spacy.io/api/spanfinder#set_annotations rr"r>rr@Nr?) enumerater%rHr'zipappendi)rIrKr5offsetrPr(startsends doc_scorestoken token_scorer9r: span_lengthrrrset_annotationss4   " zSpanFinder.set_annotations)dropsgdlossesrrZr[r\c Cs|duri}||jddd|D}t|j||j|\}}|||\}} || |dur7||||j|7<|S)a?Learn from a batch of documents and gold-standard information, updating the pipe's model. Delegates to predict and get_loss. examples (Iterable[Example]): A batch of Example objects. drop (float): The dropout rate. sgd (Optional[thinc.api.Optimizer]): The optimizer. losses (Optional[Dict[str, float]]): Optional record of the loss during training. Updated using the component name as the key. RETURNS (Dict[str, float]): The updated losses dictionary. DOCS: https://spacy.io/api/spanfinder#update NrYcSg|]}|jqSr) predicted.0egrrr z%SpanFinder.update..)r1rCr r begin_updateget_loss finish_update) rIrrZr[r\r^r5backprop_scoreslossd_scoresrrrupdates  zSpanFinder.updatecCsF|||jj\}}||jj|}||9}t|d}||fS)abFind the loss and gradient of loss for the batch of documents and their predicted scores. examples (Iterable[Examples]): The batch of examples. scores: Scores representing the model's predictions. RETURNS (Tuple[float, Floats2d]): The loss and the gradient. DOCS: https://spacy.io/api/spanfinder#get_loss r)_get_aligned_truth_scoresrops asarray2ffloatsum)rIrr5truthsmasksrirhrrrres  zSpanFinder.get_losscCsZg}g}|D]}|jj|jjkrttjjddt|j}|j j |dfdd}|j j |dfdd}|j d|j jvr|j j|j dD]I} t| \} } |jj| | dd} t| \} }| | k}|| k}|rod || d jd f<n d || d jd f<|rd || d jd f<qDd || d jd f<qD||||q|j j|d d }|j j|d d }||fS) z\Align scores of the predictions to the references for calculating the loss. r=) componentrfloat32)dtyper"expand)alignment_moderrr7)axis)xtextyrErE1054rGr'r^xpzerosonesrH referencer%r; char_spanrPrO concatenate)rIrrlrprqran_tokenstruthmaskr6ref_start_char ref_end_char pred_spanpred_start_char pred_end_char start_match end_matchrrrrks8     z$SpanFinder._get_aligned_truth_scores)rB get_examplescCsng}|D] }t|dkr||q|r0dd|D}|||jj\}}|jj||ddS|jdS)aInitialize the pipe for training, using a representative set of data examples. get_examples (Callable[[], Iterable[Example]]): Function that returns a representative sample of gold-standard Example objects. nlp (Optional[Language]): The current nlp object the component is part of. DOCS: https://spacy.io/api/spanfinder#initialize cSr]r)rr_rrrrbrcz)SpanFinder.initialize..)XYN)r'rOrkrrl initialize)rIrrBsubbatchrarKr_rrrrs   zSpanFinder.initialize)r=)__name__ __module__ __qualname____doc__rrrr rrr strrnrintrrJrLrXrr rrjrrerkrrrrrr<Hsl    / '   'r<cCs,|dkr td}|jStdtd|)Nmake_span_finderzspacy.pipeline.factorieszmodule z has no attribute ) importlib import_modulerAttributeErrorr)rCmodulerrr __getattr__s r),rsystypingrrrrrrr thinc.apir r r r thinc.typesr errorsrlanguagerrArtokensrrtrainingrutilrspancatrtrainable_piperspan_finder_default_configfrom_strDEFAULT_SPAN_FINDER_MODELrrrrr;r<rrrrrs*$         U