o iȲ@s(ddlmZddlmZmZmZmZmZmZm Z m Z m Z ddl Z ddlmZddlmZddlmZmZmZddlmZdd lmZmZerLdd lmZd Zegd ZGd ddZ GdddZ!GdddZ"deedee#effddZ$d'ddZ%ddZ&ddZ'dd Z(d(d#d$Z)d%d&Z*dS))) defaultdict) TYPE_CHECKINGAnyCallableDictIterableListOptionalSetTupleN)Errors) Morphology)DocSpanToken)Example)SimpleFrozenListget_lang_class)Language)sentertagger morphologizerparsernertextcat)Nrc @seZdZdZdddddedededdfd d Zdefd d Zd dZddZde de ddfddZ e de fddZ e de fddZe de fddZdeee ffddZdS)PRFScorezA precision / recall / F score.rtpfpfnrr r!returnNcCs||_||_||_dSNr)selfrr r!r%@/home/ubuntu/.local/lib/python3.10/site-packages/spacy/scorer.py__init__"s zPRFScore.__init__cCs|j|j|jSr#rr$r%r%r&__len__-zPRFScore.__len__cCs4|j|j7_|j|j7_|j|j7_|Sr#rr$otherr%r%r&__iadd__0szPRFScore.__iadd__cCs&t|j|j|j|j|j|jdS)Nr)rrr r!r+r%r%r&__add__6szPRFScore.__add__candgoldcCsH|jt||7_|jt||7_|jt||7_dSr#)rlen intersectionr r!r$r/r0r%r%r& score_set;szPRFScore.score_setcC|j|j|jdSN0.++)rr r(r%r%r& precision@zPRFScore.precisioncCr5r6)rr!r(r%r%r&recallDr9zPRFScore.recallcCs$|j}|j}d||||dS)Nr7)r8r:)r$prr%r%r&fscoreHszPRFScore.fscorecCs|j|j|jdS)N)r<r=f)r8r:r>r(r%r%r&to_dictNr*zPRFScore.to_dict)__name__ __module__ __qualname____doc__intr'r)r-r.setr4propertyfloatr8r:r>rstrr@r%r%r%r&rs2  rc@s8eZdZdZd ddZd ddZdd Zed d ZdS) ROCAUCScorezAn AUC ROC score. This is only defined for binary classification. Use the method is_binary before calculating the score, otherwise it may throw an error.r"NcCsg|_g|_d|_d|_dS)Nr)goldscands saved_scoresaved_score_at_lenr(r%r%r&r'Ws zROCAUCScore.__init__cCs|j||j|dSr#)rMappendrLr3r%r%r&r4]s zROCAUCScore.score_setcCstt|jdkSNr;)r1npuniquerLr(r%r%r& is_binaryaszROCAUCScore.is_binarycCsX|sttjjt|jdt|j|jkr|j St |j|j |_ t|j|_|j S)Nlabel) rT ValueErrorr E165formatrFrLr1rOrN_roc_auc_scorerMr(r%r%r&scoreds zROCAUCScore.score)r"N) rArBrCrDr'r4rTrGr[r%r%r%r&rJRs  rJc@sfeZdZdZddefdeddedeeddfd d Zd d d ee de de ee ffddZ ed ee de ee ffddZeeedd ee dedeeege fdee de ee ff ddZeeedd ee dedeeege fdee de ee ff ddZeeddd dd ee dedeeegeefdeeege fde de de ee ffd d!Zeeedddd"d ee dedeeege fd#eed$e d%eed&eede ee ffd'd(Zed ee d)eede ee ffd*d+Zeed,eeed-d ee dedeeege fd.ed/eeegefd0eedee de ee ffd1d2ZdS)3ScorerzCompute evaluation scores.Nxxnlpr default_langdefault_pipeliner"cKs<||_|r ||_dSt|}|D]}||q||_dS)zOInitialize the Scorer. DOCS: https://spacy.io/api/scorer#init N)cfgr^radd_pipe)r$r^r_r`rapiper%r%r&r'rs     zScorer.__init__F) per_componentexamplesrdcCsi}t|jjdr)|r|jjj|fi|j|d<n||jjj|fi|j|jjD]%\}}t|drR|rE|j|fi|j||<q-||j|fi|jq-|S)aQEvaluate a list of Examples. examples (Iterable[Example]): The predicted annotations + correct annotations. per_component (bool): Whether to return the scores keyed by component name. Defaults to False. RETURNS (Dict): A dictionary of scores. DOCS: https://spacy.io/api/scorer#score r[ tokenizer)hasattrr^rfr[raupdatepipeline)r$rerdscoresname componentr%r%r&r[s  z Scorer.scorec Kst}t}|D]c}|j}|j}|jrq|j}t}t} |D]} | jr'q|| j | j t | fq|D]-} | jr?q7| | j | j t | f|j j | j dkr]|jd7_q7|jd7_q7|| |qt |dkr}|j|j|j|jdSdddddS)a~Returns accuracy and PRF scores for tokenization. * token_acc: # correct tokens / # gold tokens * token_p/r/f: PRF for token character spans examples (Iterable[Example]): Examples to score RETURNS (Dict[str, Any]): A dictionary containing the scores token_acc/p/r/f. DOCS: https://spacy.io/api/scorer#score_tokenization r r) token_acctoken_ptoken_rtoken_fN)r reference predictedhas_unknown_spaces alignmentrForth_isspaceaddidxr1x2ylengthsir rr4r8r:r>) rera acc_score prf_scoreexamplegold_docpred_docalign gold_spans pred_spanstokenr%r%r&score_tokenizationsB    zScorer.score_tokenization)gettermissing_valuesattrrrcKst}|D]f}|j}|j}|j} t} t} t|D]\} } || |}||vr2| | || |fq| | qt}|D]'} | jrEq=| j j | j dkrd| j | j d} | | vrd|| || |fq=| || q|d}t |dkr{|diS||jiS)aReturns an accuracy score for a token-level attribute. examples (Iterable[Example]): Examples to score attr (str): The attribute to score. getter (Callable[[Token, str], Any]): Defaults to getattr. If provided, getter(token, attr) should return the value of the attribute for an individual token. missing_values (Set[Any]): Attribute values to treat as missing annotation in the reference annotation. RETURNS (Dict[str, Any]): A dictionary containing the accuracy score under the key attr_acc. DOCS: https://spacy.io/api/scorer#score_token_attr r r_accN)rrqrrrtrF enumeraterwrurvryrzr{r4r1r>)rerrrra tag_scorer~rrr gold_tagsmissing_indicesgold_irvalue pred_tags score_keyr%r%r&score_token_attrs4      zScorer.score_token_attrc Ks~t}i}|D]}|j}|j} |j} i} t} t| D]J\} }|||}| jj|}||vr`|tj kr`| tj D]%}| tj \}}||vrLt||<|| vrUt| |<| | | |fq9q| | qi}|D]]}|jrrqj| jj|jdkr| j|jd} | | vr|||}| jj|}||vr|tj kr| tj D]%}| tj \}}||vrt||<||vrt||<|| | |fqqj|D]$}|||t| |t||||t| |tqqi}t|dkr!|j||d<|j||d<|j||d<dd|D||d<|Sd ||d<d ||d<d ||d<d ||d<|S) aReturn micro PRF and PRF scores per feat for a token attribute in UFEATS format. examples (Iterable[Example]): Examples to score attr (str): The attribute to score. getter (Callable[[Token, str], Any]): Defaults to getattr. If provided, getter(token, attr) should return the value of the attribute for an individual token. missing_values (Set[Any]): Attribute values to treat as missing annotation in the reference annotation. RETURNS (dict): A dictionary containing the micro PRF scores under the key attr_micro_p/r/f and the per-feat PRF scores under attr_per_feat. r r_micro_p_micro_r_micro_fcSi|] \}}||qSr%r@.0kvr%r%r& Qz4Scorer.score_token_attr_per_feat.. _per_featN)rrrrqrtrFrvocabstringsr EMPTY_MORPHsplit FEATURE_SEP FIELD_SEPrwrurvryrzr{r4getr1r8r:r>items)rerrrra micro_scoreper_featr~rrr gold_per_featrrrrmorphfeatfieldvalues pred_per_featresultr%r%r&score_token_attr_per_featsx           z Scorer.score_token_attr_per_featT)rhas_annotationlabeled allow_overlaprrrcKsbt}t}|D]} | j} | j} |dur|| sqtdd|| |D} |dus1|dur?|| r?| tdd|| |DO} dd| D} | D] }||vrSt||<qHt}t}|| |D]$}|rn|j|j|jdf}n|j|jdf}||| |j|q_dd| D}|dus|dur|| r| || ||D]$}|r|j|j|jdf}n|j|jdf}||||j|q|r| D]\}}||vr| ||| |q| ||q|d d|d d|d di}|rd||d <t |d kr/|j ||d <|j||d <|j||d <|r/dd| D||d <|S)abReturns PRF scores for labeled spans. examples (Iterable[Example]): Examples to score attr (str): The attribute to score. getter (Callable[[Doc, str], Iterable[Span]]): Defaults to getattr. If provided, getter(doc, attr) should return the spans for the individual doc. has_annotation (Optional[Callable[[Doc], bool]]) should return whether a `Doc` has annotation for this `attr`. Docs without annotation are skipped for scoring purposes. labeled (bool): Whether or not to include label information in the evaluation. If set to 'False', two spans will be considered equal if their start and end match, irrespective of their label. allow_overlap (bool): Whether or not to allow overlapping spans. If set to 'False', the alignment will automatically resolve conflicts. RETURNS (Dict[str, Any]): A dictionary containing the PRF scores under the keys attr_p/r/f and the per-type PRF scores under attr_per_type. DOCS: https://spacy.io/api/scorer#score_spans NcSg|]}|jqSr%label_rrr%r%r& z&Scorer.score_spans..cSrr%rrr%r%r&rrcSi|]}|tqSr%rFrrVr%r%r&rz&Scorer.score_spans..r cSrr%rrr%r%r&rr_p_r_f _per_typercSrr%rrr%r%r&r)rdictrrrqrFrstartendrwget_aligned_spans_x2yrr4r1r8r:r>)rerrrrrrar[score_per_typer~rrlabels gold_per_typerVrrspan gold_span pred_per_type pred_spanrr final_scoresr%r%r& score_spansYsp       zScorer.score_spans)rr multi_labelpositive_label thresholdrrrrc sN|dur |rdnd}|sd}ddD}ddD} t|D]} || j|} fdd| D} || j|} fdd| D} D]} | | d}| | }|s\|s\d}|durh| | ||qI|rD]F} | | d}| | }|dur||kr|d kr|| jd 7_qm||kr|d kr|| jd 7_qm||kr|d kr|| jd 7_qmq"| r| rt | d d d \}}t | dd d \}}||kr||jd 7_q"||jd 7_||jd 7_q"| rt | dd d \}}|d kr ||jd 7_q"| r&t | dd d \}}||jd 7_q"t }| D]}|j|j7_|j|j7_|j|j7_q.t |d}t dd| D|}t dd| D|}t dd| D|}t dd| D|}|dd|dd|d|j|d|j|d|j|d||d||d||d||d d!d|D|d"d#d| Di }t d$kr|s|r||d |d%}|||d<d&|d'||d<|S|s||d||d<d(||d<|S||d||d<d)||d<|S)*aReturns PRF and ROC AUC scores for a doc-level attribute with a dict with scores for each label like Doc.cats. The reported overall score depends on the scorer settings. examples (Iterable[Example]): Examples to score attr (str): The attribute to score. getter (Callable[[Doc, str], Any]): Defaults to getattr. If provided, getter(doc, attr) should return the values for the individual doc. labels (Iterable[str]): The set of possible labels. Defaults to []. multi_label (bool): Whether the attribute allows multiple labels. Defaults to True. When set to False (exclusive labels), missing gold labels are interpreted as 0.0 and the threshold is set to 0.0. positive_label (str): The positive label for a binary task with exclusive classes. Defaults to None. threshold (float): Cutoff to consider a prediction "positive". Defaults to 0.5 for multi-label, and 0.0 (i.e. whatever's highest scoring) otherwise. RETURNS (Dict[str, Any]): A dictionary containing the scores, with inapplicable scores as None: for all: attr_score (one of attr_micro_f / attr_macro_f / attr_macro_auc), attr_score_desc (text description of the overall score), attr_micro_p, attr_micro_r, attr_micro_f, attr_macro_p, attr_macro_r, attr_macro_f, attr_macro_auc, attr_f_per_type, attr_auc_per_type DOCS: https://spacy.io/api/scorer#score_cats Ng?rKcSrr%)rrr%r%r&rrz%Scorer.score_cats..cSrr%)rJrr%r%r&rrci|] \}}|vr||qSr%r%rrr%r&rcrr%r%rrr%r&rrrr cS|dSNr r%itr%r%r&z#Scorer.score_cats..)keycSrrr%rr%r%r&rrcSrrr%rr%r%r&rrcSrrr%rr%r%r&rrr7cs|]}|jVqdSr#r8rprfr%r%r& "z$Scorer.score_cats..csrr#r:rr%r%r&r#rcsrr#r>rr%r%r&r$rcss"|] }|r |jndVqdS)rKNrTr[)raucr%r%r&r(s _score _score_descrrr_macro_p_macro_r_macro_f _macro_auc _f_per_typecSrr%rrr%r%r&r5r _auc_per_typecSs$i|]\}}||r|jndqSr#rrr%r%r&r6sr;r?zF ()zmacro Fz macro AUC)rFrrrrqrr4rr r!maxrrr1sumr8r:r>)rerrrrrrra f_per_type auc_per_typer~ pred_cats gold_catsrV pred_score gold_score pred_label gold_label micro_prf label_prfn_catsmacro_pmacro_rmacro_f macro_aucresultspositive_label_fr%rr& score_catss.                     zScorer.score_catsnegative_labelscKsi}|D]}i}|jjD] }|||j|jf<q |jjD]k}||j|jfd}|dur|j} | |vr7t|| <|j} | dur|j} | |vrJ| |vrJq| | krX|| j d7_ q| |vrf|| j d7_ q| |vrt|| j d7_ q|| j d7_ || j d7_ qqt} | D]} | j | j 7_ | j | j 7_ | j | j 7_ qt |d}tdd| D|}tdd| D|}tdd| D|}d| jd d d | jd | jd | jd|d|d|ddd|Di }|S)aReturns PRF for predicted links on the entity level. To disentangle the performance of the NEL from the NER, this method only evaluates NEL links for entities that overlap between the gold reference and the predictions. examples (Iterable[Example]): Examples to score negative_labels (Iterable[str]): The string values that refer to no annotation (e.g. "NIL") RETURNS (Dict[str, Any]): A dictionary containing the scores. DOCS: https://spacy.io/api/scorer#score_links Nr r7csrr#rrr%r%r&r{rz%Scorer.score_links..csrr#rrr%r%r&r|rcsrr#rrr%r%r&r}r nel_scorenel_score_desczmicro F nel_micro_p nel_micro_r nel_micro_f nel_macro_p nel_macro_r nel_macro_fnel_f_per_typecSrr%rrr%r%r&rrz&Scorer.score_links..)rqents start_charend_charrrrrrkb_id_rr r!rr1rr>r8r:r)rerrarr~gold_ent_by_offsetgold_entpred_entrrVr0predrrn_labelsrrrrr%r%r& score_linksFs`       zScorer.score_linkshead)r head_attr head_getter ignore_labelsrrrrc Kst}t} t} t} |D]} | j} | j}| j}t}i}t| D]B\}}|||}|||}||vr`||vr_|||j|f|| vrKt| |<||vrTt||<||||j|fq#| |q#t}i}|D]}|j ruqm|j j |jdkrd}n|j |jd}|| vr|||}|||}||vr|j r|j j |jdkr|j |jd}nd}|dus|dur|jd7_| jd7_qm||||f|| vrt| |<||vrt||<|||||fqm| ||| D]}| |||t||tq|tdd|Dtdd|Dqt|dkrC|d|j|d| j|dd d | DiS|dd|dd|ddiS) a2Returns the UAS, LAS, and LAS per type scores for dependency parses. examples (Iterable[Example]): Examples to score attr (str): The attribute containing the dependency label. getter (Callable[[Token, str], Any]): Defaults to getattr. If provided, getter(token, attr) should return the value of the attribute for an individual token. head_attr (str): The attribute containing the head token. Defaults to 'head'. head_getter (Callable[[Token, str], Token]): Defaults to getattr. If provided, head_getter(token, attr) should return the value of the head for an individual token. ignore_labels (Tuple): Labels to ignore while scoring (e.g., punct). missing_values (Set[Any]): Attribute values to treat as missing annotation in the reference annotation. RETURNS (Dict[str, Any]): A dictionary containing the scores: attr_uas, attr_las, and attr_las_per_type. DOCS: https://spacy.io/api/scorer#score_deps r Nrcss|] }|ddVqdSrQr%)ritemr%r%r&rsz$Scorer.score_deps.._uas_las _las_per_typecSrr%rrr%r%r&rrz%Scorer.score_deps..)rrrFrqrrrtrrwr{rurvryrzstripr r4rr1r>r)rerrrrrrra unlabelledlabelledlabelled_per_deprr~rrr gold_depsgold_deps_per_deprrdepr pred_depspred_deps_per_dep gold_headr%r%r& score_depss!                 zScorer.score_deps)rArBrCrDDEFAULT_PIPELINEr rIrr'rboolrrr[ staticmethodrgetattrMISSING_VALUESrrr rrrrrrrHrrr*r%r%r%r&r\os    1  2  S d   D r\rer"cKsptt}|D]}|jdsqdd|jjD}|jj}|jjD]Y}|j|vr-t||j<||j |j }t |ry|j|d|dd}t dd|Dry|j|d|ddf} | |vro||jj d7_ || q ||jjd7_q |D]\} } } || jd7_q|qt} |D]}| |7} qt | dkr| j| j| jd d |Dd Sd d d d d S) zGCompute micro-PRF and per-entity PRF scores for a sequence of examples.ENT_IOBcSsh|] }|j|j|jfqSr%)rrr)rer%r%r& szget_ner_prf..rr css|]}|jdkVqdS)rN)ent_iob)rrr%r%r&r szget_ner_prf..cSrr%rrr%r%r&rrzget_ner_prf..)ents_pents_rents_f ents_per_typeN)rryrrrtryxrrrr1allrremover r!rr8r:r>r)rekwargsregrL align_x2yrindicesg_spanrrVrrtotalsrr%r%r& get_ner_prfsH        rC?r3c Cs(t|}|dur |}n)t|}|jdkr.t|}dg|j}|jd||<||}ntj||d}|j}tdg|}tdg|}tdd||<tdd||<z||t||t|d|} W| St yt |}t |}tj ||t||t|d|} Y| Sw)a Integrate along the given axis using the composite trapezoidal rule. If `x` is provided, the integration happens in sequence along its elements - they are not sorted. Integrate `y` (`x`) along each 1d slice on the given axis, compute :math:`\int y(x) dx`. When `x` is specified, this integrates along the parametric curve, computing :math:`\int_t y(t) dt = \int_t y(t) \left.\frac{dx}{dt}\right|_{x=x(t)} dt`. Parameters ---------- y : array_like Input array to integrate. x : array_like, optional The sample points corresponding to the `y` values. If `x` is None, the sample points are assumed to be evenly spaced `dx` apart. The default is None. dx : scalar, optional The spacing between sample points when `x` is None. The default is 1. axis : int, optional The axis along which to integrate. Returns ------- trapezoid : float or ndarray Definite integral of `y` = n-dimensional array as approximated along a single axis by the trapezoidal rule. If `y` is a 1-dimensional array, then the result is a float. If `n` is greater than 1, then the result is an `n`-1 dimensional array. See Also -------- cumulative_trapezoid, simpson, romb Notes ----- Image [2]_ illustrates trapezoidal rule -- y-axis locations of points will be taken from `y` array, by default x-axis distances between points will be 1.0, alternatively they can be provided with `x` array or with `dx` scalar. Return value will be equal to combined area under the red lines. References ---------- .. [1] Wikipedia page: https://en.wikipedia.org/wiki/Trapezoidal_rule .. [2] Illustration image: https://en.wikipedia.org/wiki/File:Composite_trapezoidal_rule_illustration.png Examples -------- Use the trapezoidal rule on evenly spaced points: >>> import numpy as np >>> from scipy import integrate >>> integrate.trapezoid([1, 2, 3]) 4.0 The spacing between sample points can be selected by either the ``x`` or ``dx`` arguments: >>> integrate.trapezoid([1, 2, 3], x=[4, 6, 8]) 8.0 >>> integrate.trapezoid([1, 2, 3], dx=2) 8.0 Using a decreasing ``x`` corresponds to integrating in reverse: >>> integrate.trapezoid([1, 2, 3], x=[8, 6, 4]) -8.0 More generally ``x`` is used to integrate along a parametric curve. We can estimate the integral :math:`\int_0^1 x^2 = 1/3` using: >>> x = np.linspace(0, 1, num=50) >>> y = x**2 >>> integrate.trapezoid(y, x) 0.33340274885464394 Or estimate the area of a circle, noting we repeat the sample which closes the curve: >>> theta = np.linspace(0, 2 * np.pi, num=1000, endpoint=True) >>> integrate.trapezoid(np.cos(theta), x=np.sin(theta)) 3.141571941375841 ``trapezoid`` can be applied along a specified axis to do multiple computations in one call: >>> a = np.arange(6).reshape(2, 3) >>> a array([[0, 1, 2], [3, 4, 5]]) >>> integrate.trapezoid(a, axis=0) array([1.5, 2.5, 3.5]) >>> integrate.trapezoid(a, axis=1) array([2., 8.]) Nr raxisr3g@) rR asanyarrayndimdiffshapereshapeslicetuplerrWasarrayrwreduce) r9r:dxrFdrJndslice1slice2retr%r%r& trapezoid)s0 f     (  ,rVcCsDtt|dkrttjjt|dt||\}}}t||S)aCompute Area Under the Receiver Operating Characteristic Curve (ROC AUC) from prediction scores. Note: this implementation is restricted to the binary classification task Parameters ---------- y_true : array, shape = [n_samples] or [n_samples, n_classes] True binary labels or binary label indicators. The multiclass case expects shape = [n_samples] and labels with values in ``range(n_classes)``. y_score : array, shape = [n_samples] or [n_samples, n_classes] Target scores, can either be probability estimates of the positive class, confidence values, or non-thresholded measure of decisions (as returned by "decision_function" on some classifiers). For binary y_true, y_score is supposed to be the score of the class with greater label. The multiclass case expects shape = [n_samples, n_classes] where the scores correspond to probability estimates. Returns ------- auc : float References ---------- .. [1] `Wikipedia entry for the Receiver operating characteristic `_ .. [2] Fawcett T. An introduction to ROC analysis[J]. Pattern Recognition Letters, 2006, 27(8):861-874. .. [3] `Analyzing a portion of the ROC curve. McClish, 1989 `_ r;rU) r1rRrSrWr rXrY _roc_curve_auc)y_truey_scorefprtpr_r%r%r&rZs$ rZcCst||\}}}tjd|f}tjd|f}tj|dd|f}|ddkr0ttj|j}n||d}|ddkrEttj|j}n||d}|||fS)aCompute Receiver operating characteristic (ROC) Note: this implementation is restricted to the binary classification task. Parameters ---------- y_true : array, shape = [n_samples] True binary labels. If labels are not either {-1, 1} or {0, 1}, then pos_label should be explicitly given. y_score : array, shape = [n_samples] Target scores, can either be probability estimates of the positive class, confidence values, or non-thresholded measure of decisions (as returned by "decision_function" on some classifiers). Returns ------- fpr : array, shape = [>2] Increasing false positive rates such that element i is the false positive rate of predictions with score >= thresholds[i]. tpr : array, shape = [>2] Increasing true positive rates such that element i is the true positive rate of predictions with score >= thresholds[i]. thresholds : array, shape = [n_thresholds] Decreasing thresholds on the decision function used to compute fpr and tpr. `thresholds[0]` represents no instances being predicted and is arbitrarily set to `max(y_score) + 1`. Notes ----- Since the thresholds are sorted from low to high values, they are reversed upon returning them to ensure they correspond to both ``fpr`` and ``tpr``, which are sorted in reversed order during their calculation. References ---------- .. [1] `Wikipedia entry for the Receiver operating characteristic `_ .. [2] Fawcett T. An introduction to ROC analysis[J]. Pattern Recognition Letters, 2006, 27(8):861-874. rr r3)_binary_clf_curverRr_repeatnanrJ)rYrZfpstps thresholdsr[r\r%r%r&rWs.     rWc Csd}t|}t|}||k}tj|ddddd}||}||}d}tt|d}tj||jdf}t|||}d||}||||fS)aCalculate true and false positives per binary classification threshold. Parameters ---------- y_true : array, shape = [n_samples] True targets of binary classification y_score : array, shape = [n_samples] Estimated probabilities or decision function Returns ------- fps : array, shape = [n_thresholds] A count of false positives, at index i being the number of negative samples assigned a score >= thresholds[i]. The total number of negative samples is equal to fps[-1] (thus true negatives are given by fps[-1] - fps). tps : array, shape = [n_thresholds <= len(np.unique(y_score))] An increasing count of true positives, at index i being the number of positive samples assigned a score >= thresholds[i]. The total number of positive samples is equal to tps[-1] (thus false negatives are given by tps[-1] - tps). thresholds : array, shape = [n_thresholds] Decreasing score values. rD mergesort)kindNr3rr )rRravelargsortwhererIr_size_stable_cumsum) rYrZ pos_labeldesc_score_indicesweightdistinct_value_indicesthreshold_idxsrcrbr%r%r&r^s   r^h㈵>:0yE>c CsVtj||tjd}tj||tjd}ttj|jd|d|||dds)ttj |S)aUse high precision for cumsum and check that final value matches sum Parameters ---------- arr : array-like To be cumulatively summed as flat axis : int, optional Axis along which the cumulative sum is computed. The default (None) is to compute the cumsum over the flattened array. rtol : float Relative tolerance, see ``np.allclose`` atol : float Absolute tolerance, see ``np.allclose`` )rFdtyper3rET)rtolatol equal_nan) rRcumsumfloat64rr;isclosetakerWr E163)arrrFrtruoutexpectedr%r%r&rkRs rkcCst|}t|}d}t|}t|dkr+t|dkr"d}n ttjj|d|t ||}t |tj r>|j |}|S)aCompute Area Under the Curve (AUC) using the trapezoidal rule This is a general function, given points on a curve. For computing the area under the ROC-curve, see :func:`roc_auc_score`. Parameters ---------- x : array, shape = [n] x coordinates. These must be either monotonic increasing or monotonic decreasing. y : array, shape = [n] y coordinates. Returns ------- auc : float r rr3)r:)rRrgrIanyr;rWr E164rYrV isinstancememmaprstype)r:r9 directionrParear%r%r&rXls     rX)NrDr3)Nrqrr)+ collectionsrtypingrrrrrrr r r numpyrRerrorsr morphologyrtokensrrrtrainingrutilrrlanguagerr+ frozensetr/rrJr\rIrCrVrZrWr^rkrXr%r%r%r&s8 ,     3 1*C 6