o pio4@sddlmZmZmZmZmZddlZddlZddl Z ddl Z ddl mZmZddlmZmZGdddZdZdZd ZGd d d eZd Zd Zd ZGdddeZdded edefddZdS))ListUnionOptionalSetTupleN) AnnotationTimeline)DetailsMetricComponentsc s2eZdZdZedefddZedefddZfddZ d d Z d d Z e d dZ d,deeefdeeefdedeefddZd-dedejfddZddZddZdedeeeffdd Zd!d"Zdeeefdeeefdefd#d$Zd%efd&d'Zd.d)edeeeeefffd*d+Z Z!S)/ BaseMetricz :class:`BaseMetric` is the base class for most pyannote evaluation metrics. Attributes ---------- name : str Human-readable name of the metric (eg. 'diarization error rate') returncCt|jd)Nz\ is missing a 'metric_name' class method. It should return the name of the metric as string.NotImplementedError__name__clsrI/home/ubuntu/.local/lib/python3.10/site-packages/pyannote/metrics/base.py metric_name1zBaseMetric.metric_namecCr )Nzh is missing a 'metric_components' class method. It should return the list of names of metric components.rrrrrmetric_components8rzBaseMetric.metric_componentsc s6tt||j|_t|j|_| dSN) superr __init__ __class__r metric_name_setr components_reset)selfkwargsrrrr?s  zBaseMetric.__init__cCsdd|jDS)NcSsi|]}|dqS)r).0valuerrr Fsz.BaseMetric.init_components..)rr rrrinit_componentsEszBaseMetric.init_componentscCs*t|_t|_|jD]}d|j|<q dS)z.Reset accumulated components and metric valuesr#N)dict accumulated_listresults_r)r r%rrrrHs   zBaseMetric.resetcCs|S)z Metric name.)rr'rrrnameOszBaseMetric.nameFN reference hypothesisdetaileduricKsx|j||fi|}||||j<|pt|dd}|j||f|jD] }|j|||7<q%|r7|S||jS)aCompute metric value and accumulate components Parameters ---------- reference : type depends on the metric Manual `reference` hypothesis : type depends on the metric Evaluated `hypothesis` uri : optional Override uri. detailed : bool, optional By default (False), return metric value only. Set `detailed` to True to return dictionary where keys are components names and values are component values Returns ------- value : float (if `detailed` is False) Metric value components : dict (if `detailed` is True) `components` updated with metric value r1NA)compute_componentscompute_metricrgetattrr,appendrr*)r r.r/r0r1r! componentsr-rrr__call__Ws  zBaseMetric.__call__displayc Csg}g}d|v}|jD]V\}}i}|r|d}|D];\} } | |jkr/d| || df<q| dkr:| || df<q| || df<|rX|dkrQd| ||| df<qtj|| df<q||||q i}|j}|ro|d}|D];\} } | |jkrd| || df<qs| dkr| || df<qs| || df<|r|dkrd| ||| df<qstj|| df<qsdt|||jdf<|||dt |} || d<| d} t j | j| _| |jg|} |rt| jdd d d d d | S)akEvaluation report Parameters ---------- display : bool, optional Set to True to print the report to stdout. Returns ------- report : pandas.DataFrame Dataframe with one column per metric component, one row per evaluated item, and one final row for accumulated results. totald%rTOTALitemTFrightcS d|SNz{0:.2f}formatfrrr z#BaseMetric.report..)indexsparsifyjustify float_format)rr,itemsr-npnanr6r*abspd DataFrame set_index MultiIndex from_tuplescolumnsprint to_string) r r9reporturispercentr1r7rowr:keyr%dfrrrrYsj            zBaseMetric.reportcCs|jdd}|jddddS)NF)r9cSrArBrCrErrrrGrHz$BaseMetric.__str__..)rJrL)rYrX)r rYrrr__str__s zBaseMetric.__str__cCs ||jS)z0Compute metric value from accumulated components)r4r*r'rrr__abs__s zBaseMetric.__abs__ componentcCs$|tdddkr t|jS|j|S)aGet value of accumulated `component`. Parameters ---------- component : str Name of `component` Returns ------- value : type depends on the metric Value of accumulated `component` N)slicer)r*)r rarrr __getitem__s  zBaseMetric.__getitem__ccs |jD] \}}||fVqdS)z*Iterator over the accumulated (uri, value)N)r,)r r1rarrr__iter__s zBaseMetric.__iter__cKt|jjd)aCompute metric components Parameters ---------- reference : type depends on the metric Manual `reference` hypothesis : same as `reference` Evaluated `hypothesis` Returns ------- components : dict Dictionary where keys are component names and values are component values z is missing a 'compute_components' method.It should return a dictionary where keys are component names and values are component values.rrr)r r.r/r!rrrr3s zBaseMetric.compute_componentsr7cCre)aACompute metric value from computed `components` Parameters ---------- components : dict Dictionary where keys are components names and values are component values Returns ------- value : type depends on the metric Metric value z is missing a 'compute_metric' method. It should return the actual value of the metric based on the precomputed component dictionary given as input.rf)r r7rrrr4s zBaseMetric.compute_metric?alphacsnfddjD}t|dkrtdt|dkr-td|d}}}|||ffStjj||ddS)aPCompute confidence interval on accumulated metric values Parameters ---------- alpha : float, optional Probability that the returned confidence interval contains the true metric value. Returns ------- (center, (lower, upper)) with center the mean of the conditional pdf of the metric value and (lower, upper) is a confidence interval centered on the median, containing the estimate to a probability alpha. See Also: --------- scipy.stats.bayes_mvs csg|] \}}|jqSr)r)r$_rr'rr ?sz2BaseMetric.confidence_interval..rzFPlease evaluate a bunch of files before computing confidence interval.zCCannot compute a reliable confidence interval out of just one file.)rh)r,len ValueErrorwarningswarnscipystats bayes_mvs)r rhvaluescenterlowerupperrr'rconfidence_interval(s    zBaseMetric.confidence_interval)FN)F)rg)"r __module__ __qualname____doc__ classmethodstrrr rrr(rpropertyr-rrrboolrr8rQrRrYr_r`floatr rcrdr3r4rrx __classcell__rrr"rr 'sF      -T   r precisionz # retrievedz# relevant retrievedc@@eZdZdZeddZedefddZdede fdd Z d S) PrecisionaU :class:`Precision` is a base class for precision-like evaluation metrics. It defines two components '# retrieved' and '# relevant retrieved' and the compute_metric() method to compute the actual precision: Precision = # retrieved / # relevant retrieved Inheriting classes must implement compute_components(). cCtSr)PRECISION_NAMErrrrr^zPrecision.metric_namer cCttgSr)PRECISION_RETRIEVEDPRECISION_RELEVANT_RETRIEVEDrrrrrbzPrecision.metric_componentsr7cC4|t}|t}|dkr|dkrdStd||S)z#Compute precision from `components`r#r?r=)rrrnr r7 numerator denominatorrrrr4fzPrecision.compute_metricN rryrzr{r|rr rr rr4rrrrrR rrecallz # relevantc@r) RecallaG :class:`Recall` is a base class for recall-like evaluation metrics. It defines two components '# relevant' and '# relevant retrieved' and the compute_metric() method to compute the actual recall: Recall = # relevant retrieved / # relevant Inheriting classes must implement compute_components(). cCrr) RECALL_NAMErrrrrrzRecall.metric_namer cCrr)RECALL_RELEVANTRECALL_RELEVANT_RETRIEVEDrrrrrrzRecall.metric_componentsr7cCr)z Compute recall from `components`r#rrr=)rrrnrrrrr4rzRecall.compute_metricNrrrrrrxrrrr cCs4||dkrdSd||||||||S)uCompute f-measure f-measure is defined as follows: F(P, R, b) = (1+b²).P.R / (b².P + R) where P is `precision`, R is `recall` and b is `beta` r#rrlr)rrbetarrr f_measures $r)r)typingrrrrrronumpyrNpandasrQ scipy.statsrq pyannote.corerrpyannote.metrics.typesr r r rrrrrrrrrrrrrrs&(!!