o piV@s:dZddlmZmZmZddlZddlmZm Z ddl m Z ddl m Z mZddlmZdd lmZdd lmZdd lmZmZdd lmZerI d ZGdddeZGdddeZdZdZdZGdddeZdZ dZ!dZ"Gdddee Z#dZ$Gddde#Z%dZ&d Z'd!Z(Gd"d#d#ee Z)d$Z*Gd%d&d&e)Z+dS)'zMetrics for diarization)OptionalDict TYPE_CHECKINGN) AnnotationTimeline)Label) BaseMetric f_measure)IdentificationErrorRate) GreedyMapper)HungarianMapper)DetailsMetricComponents)UEMSupportMixinzdiarization error ratec seZdZdZedefddZddedeffd d Z dd e d e de e de eeffddZ dd e d e de e deffdd ZZS)DiarizationErrorRateaLDiarization error rate First, the optimal mapping between reference and hypothesis labels is obtained using the Hungarian algorithm. Then, the actual diarization error rate is computed as the identification error rate with each hypothesis label translated into the corresponding reference label. Parameters ---------- collar : float, optional Duration (in seconds) of collars removed from evaluation around boundaries of reference segments. skip_overlap : bool, optional Set to True to not evaluate overlap regions. Defaults to False (i.e. keep overlap regions). Examples -------- * Diarization error rate between `reference` and `hypothesis` annotations >>> metric = DiarizationErrorRate() >>> reference = Annotation(...) # doctest: +SKIP >>> hypothesis = Annotation(...) # doctest: +SKIP >>> value = metric(reference, hypothesis) # doctest: +SKIP * Compute global diarization error rate and confidence interval over multiple documents >>> for reference, hypothesis in ... # doctest: +SKIP ... metric(reference, hypothesis) # doctest: +SKIP >>> global_value = abs(metric) # doctest: +SKIP >>> mean, (lower, upper) = metric.confidence_interval() # doctest: +SKIP * Get diarization error rate detailed components >>> components = metric(reference, hypothesis, detailed=True) #doctest +SKIP * Get accumulated components >>> components = metric[:] # doctest: +SKIP >>> metric['confusion'] # doctest: +SKIP See Also -------- :class:`pyannote.metric.base.BaseMetric`: details on accumulation :class:`pyannote.metric.identification.IdentificationErrorRate`: identification error rate returncCtSNDER_NAMEclsrP/home/ubuntu/.local/lib/python3.10/site-packages/pyannote/metrics/diarization.py metric_nameez DiarizationErrorRate.metric_nameFcollar skip_overlapc $tjd||d|t|_dSN)rrrsuper__init__r mapper_selfrrkwargs __class__rrr$i zDiarizationErrorRate.__init__N reference hypothesisuemcC$|r |j|||d\}}|||S)aiOptimal label mapping Parameters ---------- reference : Annotation hypothesis : Annotation Reference and hypothesis diarization uem : Timeline Evaluation map Returns ------- mapping : dict Mapping between hypothesis (key) and reference (value) labels r.uemifyr%r'r,r-r.rrroptimal_mappingms z$DiarizationErrorRate.optimal_mappingc r|j||||j|jdd\}}}|jdd}|jdd}|||}|j|d}tt|j||f|ddd |S NTr.rr returns_uemstring generatorint)mappingrFr.rr)r2rr rename_labelsr4r#rcompute_componentsr'r,r-r.r(r=mappedr)rrr@&     z'DiarizationErrorRate.compute_componentsrFr)__name__ __module__ __qualname____doc__ classmethodstrrfloatboolr$rrrrrr4rr@ __classcell__rrr)rr3s21  #rc seZdZdZeddZddedeffdd Z dd e d e d e e de e e ffddZ dd e d e d e e deffdd ZZS)GreedyDiarizationErrorRateaGreedy diarization error rate First, the greedy mapping between reference and hypothesis labels is obtained. Then, the actual diarization error rate is computed as the identification error rate with each hypothesis label translated into the corresponding reference label. Parameters ---------- collar : float, optional Duration (in seconds) of collars removed from evaluation around boundaries of reference segments. skip_overlap : bool, optional Set to True to not evaluate overlap regions. Defaults to False (i.e. keep overlap regions). Examples -------- * Greedy diarization error rate between `reference` and `hypothesis` annotations >>> metric = GreedyDiarizationErrorRate() >>> reference = Annotation(...) # doctest: +SKIP >>> hypothesis = Annotation(...) # doctest: +SKIP >>> value = metric(reference, hypothesis) # doctest: +SKIP * Compute global greedy diarization error rate and confidence interval over multiple documents >>> for reference, hypothesis in ... # doctest: +SKIP ... metric(reference, hypothesis) # doctest: +SKIP >>> global_value = abs(metric) # doctest: +SKIP >>> mean, (lower, upper) = metric.confidence_interval() # doctest: +SKIP * Get greedy diarization error rate detailed components >>> components = metric(reference, hypothesis, detailed=True) #doctest +SKIP * Get accumulated components >>> components = metric[:] # doctest: +SKIP >>> metric['confusion'] # doctest: +SKIP See Also -------- :class:`pyannote.metric.base.BaseMetric`: details on accumulation cCrrrrrrrrrz&GreedyDiarizationErrorRate.metric_namerFrrc s(tt|jd||d|t|_dSr!)r#rNr$r r%r&r)rrr$s  z#GreedyDiarizationErrorRate.__init__Nr,r-r.rcCr/)ahGreedy label mapping Parameters ---------- reference : Annotation hypothesis : Annotation Reference and hypothesis diarization uem : Timeline Evaluation map Returns ------- mapping : dict Mapping between hypothesis (key) and reference (value) labels r0r1r3rrrgreedy_mappings z)GreedyDiarizationErrorRate.greedy_mappingc r5r6)r2rrr?rOr#rNr@rAr)rrr@rCz-GreedyDiarizationErrorRate.compute_componentsrDr)rErFrGrHrIrrKrLr$rrrrrrOrr@rMrrr)rrNs20    rNzjaccard error ratez speaker errorz speaker countc steZdZdZeddZedefddZdfd d Z dd e d e de e de fddZ de defddZZS)JaccardErrorRateui Jaccard error rate Parameters ---------- collar : float, optional Duration (in seconds) of collars removed from evaluation around boundaries of reference segments. skip_overlap : bool, optional Set to True to not evaluate overlap regions. Defaults to False (i.e. keep overlap regions). Examples -------- >>> metric = JaccardErrorRate() >>> reference = Annotation(...) # doctest: +SKIP >>> hypothesis = Annotation(...) # doctest: +SKIP >>> jer = metric(reference, hypothesis) # doctest: +SKIP Reference --------- Second DIHARD Challenge Evaluation Plan. Version 1.1 N. Ryant, K. Church, C. Cieri, A. Cristia, J. Du, S. Ganapathy, M. Liberman https://coml.lscp.ens.fr/dihard/2019/second_dihard_eval_plan_v1.1.pdf "The Jaccard error rate is based on the Jaccard index, a similarity measure used to evaluate the output of image segmentation systems. An optimal mapping between reference and system speakers is determined and for each pair the Jaccard index is computed. The Jaccard error rate is then defined as 1 minus the average of these scores. While similar to DER, it weights every speaker’s contribution equally, regardless of how much speech they actually produced. More concretely, assume we have N reference speakers and M system speakers. An optimal mapping between speakers is determined using the Hungarian algorithm so that each reference speaker is paired with at most one system speaker and each system speaker with at most one reference speaker. Then, for each reference speaker ref the speaker-specific Jaccard error rate JERref is computed as JERref = (FA + MISS) / TOTAL where - TOTAL is the duration of the union of reference and system speaker segments; if the reference speaker was not paired with a system speaker, it is the duration of all reference speaker segments - FA is the total system speaker time not attributed to the reference speaker; if the reference speaker was not paired with a system speaker, it is 0 - MISS is the total reference speaker time not attributed to the system speaker; if the reference speaker was not paired with a system speaker, it is equal to TOTAL The Jaccard error rate then is the average of the speaker specific Jaccard error rates. JER and DER are highly correlated with JER typically being higher, especially in recordings where one or more speakers is particularly dominant. Where it tends to track DER is in outliers where the diarization is especially bad, resulting in one or more unmapped system speakers whose speech is not then penalized. In these cases, where DER can easily exceed 500%, JER will never exceed 100% and may be far lower if the reference speakers are handled correctly." cCrr)JER_NAMErrrrrurzJaccardErrorRate.metric_namercCttgSr)JER_SPEAKER_COUNTJER_SPEAKER_ERRORrrrrmetric_componentsysz"JaccardErrorRate.metric_componentsrFc r r!r"r&r)rrr$r+zJaccardErrorRate.__init__Nr,r-r.cKs|j||||j|jdd\}}}|jdd}|jdd}|||}|}|D]N}||d}|dur9d} n/||} ||} | |  } | | |  } | | |  }| || } |t d7<|t| 7<q*|S)NTr7r9r:r<?r)r2rrr?r4init_componentslabelsgetlabel_timelineunionsupportdurationcroprSrT)r'r,r-r.r(r=detail ref_speaker hyp_speakerjerrhtotalfamissrrrr@s2        z#JaccardErrorRate.compute_componentsr_cCs|t|tSr)rTrSr'r_rrrcompute_metricszJaccardErrorRate.compute_metricrDr)rErFrGrHrIrrrUr$rrrrr@rKrirMrrr)rrP3s$A  FrPpurityrecorrectc @s\eZdZdZeddZeddZ ddeded ee d e fd d Z d e d e fddZ dS)DiarizationPurityzCluster purity A hypothesized annotation has perfect purity if all of its labels overlap only segments which are members of a single reference label. cCrr) PURITY_NAMErrrrrrzDiarizationPurity.metric_namecCrRr PURITY_TOTALPURITY_CORRECTrrrrrUz#DiarizationPurity.metric_componentsNr,r-r.rc Ksn|}d}d}|D] }||} || 7}|| } || s%dn| dd7}q ||t<||t<|S)Nrr)rWrXrZr]r^chartrorp) r'r,r-r.r(r_r]largestlabelr\rrrrrr@s   z$DiarizationPurity.compute_componentsr_cCs |tdkr|t|tSdS)NrrVrnrhrrrris z DiarizationPurity.compute_metricr)rErFrGrHrIrrUrrrrr@rKrirrrrrls"   rlcoveragec FeZdZdZeddZ d dededeede ffd d Z Z S) DiarizationCoveragezCluster coverage A hypothesized annotation has perfect coverage if all segments from a given reference label are assigned to the same cluster. cCrr) COVERAGE_NAMErrrrr rzDiarizationCoverage.metric_nameNr,r-r.rc tt|j||fd|i|SNr.)r#rwr@r'r,r-r.r(r)rrr@  z&DiarizationCoverage.compute_componentsr rErFrGrHrIrrrrrr@rMrrr)rrws rw homogeneityentropyz cross-entropyc sneZdZdZeddZeddZdded effd d Z dd e de de e de fddZddZZS)DiarizationHomogeneityaUCluster homogeneity Parameters ---------- collar : float, optional Duration (in seconds) of collars removed from evaluation around boundaries of reference segments. skip_overlap : bool, optional Set to True to not evaluate overlap regions. Defaults to False (i.e. keep overlap regions). cCrr)HOMOGENEITY_NAMErrrrr,rz"DiarizationHomogeneity.metric_namecCrRr)HOMOGENEITY_ENTROPYHOMOGENEITY_CROSS_ENTROPYrrrrrU0rqz(DiarizationHomogeneity.metric_componentsrFrrc s&tt|jdi|||_||_dS)Nr)r#rr$rrr&r)rrr$4s zDiarizationHomogeneity.__init__Nr,r-r.rc Ks|}|j||||j|jd\}}||}t|}tj|dd}tj|dd} tj||d} t| tj | d |t <tj||d} tj|| d} t| tj | d |t <|S)Nr>r)axisrr) rWr2rrnpsummadividefilledlogrr) r'r,r-r.r(r_matrixr] rduration hdurationratiohratiorrrr@9s(  " z)DiarizationHomogeneity.compute_componentscCs<d|t}d|t}|dkr|dkrdSdSd||S)NrVrr)rr)r'r_ numerator denominatorrrrri^s   z%DiarizationHomogeneity.compute_metricrDr)rErFrGrHrIrrUrKrLr$rrrrr@rirMrrr)rrs$    %r completenessc rv) DiarizationCompletenessaVCluster completeness Parameters ---------- collar : float, optional Duration (in seconds) of collars removed from evaluation around boundaries of reference segments. skip_overlap : bool, optional Set to True to not evaluate overlap regions. Defaults to False (i.e. keep overlap regions). cCrr)COMPLETENESS_NAMErrrrr{rz#DiarizationCompleteness.metric_nameNr,r-r.rc ryrz)r#rr@r{r)rrr@r|z*DiarizationCompleteness.compute_componentsrr}rrr)rrms r),rHtypingrrrnumpyr pyannote.corerrpyannote.core.utils.typesrbaser r identificationr matcherr r typesrrutilsrrrrNrQrTrSrPrmrorprlrxrwrrrrrrrrrrsD     {,L