o wiWQ@sddlZddlmZddlmZmZmZmZddlZddl Z ddl Z ddl m Z mZddlmZddlmZddlmZddlmZgd Zd eed eed eefd dZ d5dedededededeeefded eeeefeeefffddZd6ddZ    d7deded eeed!ed"ed#ed eeeeffd$d%Z d8d'd(Z!d)eed*eed eeeeffd+d,Z" -d9d)eed*eed.ed eeeeffd/d0Z#d1eeed2eeed eeeeeeeffd3d4Z$dS):N) permutations)DictListOptionalTuple)SegmentTimeline)DiarizationErrorRate)word_error_rate)linear_sum_assignment)logging) score_labelscalculate_session_cpWER"calculate_session_cpWER_bruteforceconcat_perm_word_error_rate pred_labels ref_labelsreturnc Cst|dkrgSt|dkrd}n tdd|D}g}|D]4}|\}}}t|t|}}||krIt||}|d|d|}||q||krR||q|S)a_ For evaluation of online diarization performance, generate partial reference labels from the last prediction time. Args: pred_labels (list[str]): list of partial prediction labels ref_labels (list[str]): list of full reference labels Returns: ref_labels_out (list[str]): list of partial reference labels rcSsg|] }t|dqS))floatsplit).0labelsr]/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/nemo/collections/asr/metrics/der.py :sz*get_partial_ref_labels.. )lenmaxrrminappend) rrlast_pred_timeref_labels_outlabelstartendspeakerend_timerrrget_partial_ref_labels%s"     r(DERCERFAMISSdiar_eval_count der_stat_dictdecicCstd||td||td||td||d}|d|7<|d|7<td|d|||d<td|d|||d<tt|d|d||d<tt|d |d ||d <||fS) a For evaluation of online diarization performance, add cumulative, average, and maximum DER/CER. Args: DER (float): Diarization Error Rate from the start to the current point CER (float): Confusion Error Rate from the start to the current point FA (float): False Alarm from the start to the current point MISS (float): Miss rate from the start to the current point diar_eval_count (int): Number of evaluation sessions der_stat_dict (dict): Dictionary containing cumulative, average, and maximum DER/CER deci (int): Number of decimal places to round Returns: der_dict (dict): Dictionary containing DER, CER, FA, and MISS der_stat_dict (dict): Dictionary containing cumulative, average, and maximum DER/CER d)r*r+r,r-cum_DERcum_CERavg_DERavg_CERr*max_DERr+max_CER)roundr)r*r+r,r-r.r/r0der_dictrrrget_online_DER_statsJs    r:c Cs|t|d}t|d*}|}|D]}|}|\}}}}|tt|t|qWd|S1s7wY|S)z Generate pyannote timeline segments for uem file file format UNIQ_SPEAKER_ID CHANNEL START_TIME END_TIME )urirN)ropen readlinesstripraddrr) uem_file uniq_nametimelineflinesline_ start_timer'rrruem_timeline_from_filers   rJ?T all_referenceall_hypothesisall_uemcollarignore_overlapverbosecCsXd}t|t|kr"td||d}id}} tt||D]\} \} } | \} }| \}}t|t|kr=| d7} |ret|t|kretd| dddt|d t|d}|durv||||| d d n(|| d ddur|| d d}t|| d }||||d d n|||d d| |||| <q| t|}t |}|ddkrt d|d|d}|d|d}|d|d}||||f}|rt ddt ddt ddt ddtd|td|d|d|dd|dd|dd|dd |dd|||fS|r*td!dS)"a Calculate DER, CER, FA and MISS rate from hypotheses and references. Hypothesis results are coming from Pyannote-formatted speaker diarization results and References are coming from Pyannote-formatted RTTM data. Args: AUDIO_RTTM_MAP (dict): Dictionary containing information provided from manifestpath all_reference (list[uniq_name,Annotation]): reference annotations for score calculation all_hypothesis (list[uniq_name,Annotation]): hypothesis annotations for score calculation all_uem (list[list[float]]): List of UEM segments for each audio file. If UEM file is not provided, it will be read from manifestpath collar (float): Length of collar (in seconds) for diarization error rate calculation ignore_overlap (bool): If True, overlapping segments in reference and hypothesis will be ignored verbose (bool): If True, warning messages will be printed Returns: metric (pyannote.DiarizationErrorRate): Pyannote Diarization Error Rate metric object. This object contains detailed scores of each audiofile. mapping (dict): Mapping dict containing the mapping speaker label for each audio input itemized_errors (tuple): Tuple containing (DER, CER, FA, MISS) for each audio file. - DER: Diarization Error Rate, which is sum of all three errors, CER + FA + MISS. - CER: Confusion Error Rate, which is sum of all errors - FA: False Alarm Rate, which is the number of false alarm segments - MISS: Missed Detection Rate, which is the number of missed detection segments < Caveat > Unlike md-eval.pl, "no score" collar in pyannote.metrics is the maximum length of "no score" collar from left to right. Therefore, if 0.25s is applied for "no score" collar in md-eval.pl, 0.5s should be applied for pyannote.metrics. N)rO skip_overlaprrz!Wrong Spk. Count with uniq_id:...iz, Ref: z, Hyp: T)uemdetailed uem_filepath)rBrC)rUtotalz"Total evaluation time is 0. Abort. confusionz false alarmzmissed detectionzdisplay.max_rowszdisplay.max_columnsz display.widthzdisplay.max_colwidth zCumulative Results for collar z sec and ignore_overlap z : | FA: z.4fz | MISS: z | CER: z | DER: z | Spk. Count Acc. z|Check if each ground truth RTTMs were present in the provided manifest file. Skipping calculation of Diariazation Error Rate)rr enumerateziprr infogetrJoptimal_mappingabs ValueErrorpd set_optionreportwarning)AUDIO_RTTM_MAPrLrMrNrOrPrQmetric mapping_dictcorrect_spk_countidx reference hypothesisref_keyrrH hyp_labelsuem_objrB spk_count_accr*r+r,r-itemized_errorsrrrr sv'            r allcCsrg}|dkr dg}n|dkrdg}n|dkrdg}n |dkr#gd}ntd |D] \}}t|||||d }q)|S) a Evaluate with a selected diarization evaluation scheme AUDIO_RTTM_MAP (dict): Dictionary containing information provided from manifestpath all_reference (list[uniq_name,annotation]): reference annotations for score calculation all_hypothesis (list[uniq_name,annotation]): hypothesis annotations for score calculation diar_eval_mode (str): Diarization evaluation modes diar_eval_mode == "full": DIHARD challenge style evaluation, the most strict way of evaluating diarization (collar, ignore_overlap) = (0.0, False) diar_eval_mode == "fair": Evaluation setup used in VoxSRC challenge (collar, ignore_overlap) = (0.25, False) diar_eval_mode == "forgiving": Traditional evaluation setup (collar, ignore_overlap) = (0.25, True) diar_eval_mode == "all": Compute all three modes (default) fullgFfairrKF forgivingrKTrq)rsrurwz7`diar_eval_mode` variable contains an unsupported value)rerLrMrOrP)r`r )audio_rttm_map_dictrLrMdiar_eval_mode eval_settingsrOrP diar_scorerrr evaluate_ders&  r|spk_hypothesis spk_referencecCsgg}}g}t|D] \}}||q d|}t|D]}d|} || t| g|gd} || qt|} || } || } | | |fS)aM Calculate cpWER with actual permutations in brute-force way when LSA algorithm cannot deliver the correct result. Args: spk_hypothesis (list): List containing the hypothesis transcript for each speaker. A list containing the sequence of words is assigned for each speaker. Example: >>> spk_hypothesis = ["hey how are you we that's nice", "i'm good yes hi is your sister"] spk_reference (list): List containing the reference transcript for each speaker. A list containing the sequence of words is assigned for each speaker. Example: >>> spk_reference = ["hi how are you well that's nice", "i'm good yeah how is your sister"] Returns: cpWER (float): cpWER value for the given session. min_perm_hyp_trans (str): Hypothesis transcript containing the permutation that minimizes WER. Words are separated by spaces. ref_trans (str): Reference transcript in an arbitrary permutation. Words are separated by spaces. r hypotheses references)rZr joinrr npargmin)r}r~ p_wer_listpermed_hyp_lists ref_word_listspk_id word_list ref_trans hyp_word_list hyp_transp_wer argmin_idxmin_perm_hyp_transcpWERrrrrs         rF use_lsa_onlycs|g}ttj|}tt|}}|s#||kr#t|\}}} nHg} |D]\} } t| g| gd} | | q't|  tt|g}t |\}}fddt |D}d |}d |} t|g| gd}||| fS)a Calculate a session-level concatenated minimum-permutation word error rate (cpWER) value. cpWER is a scoring method that can evaluate speaker diarization and speech recognition performance at the same time. cpWER is calculated by going through the following steps. 1. Concatenate all utterances of each speaker for both reference and hypothesis files. 2. Compute the WER between the reference and all possible speaker permutations of the hypothesis. 3. Pick the lowest WER among them (this is assumed to be the best permutation: `min_perm_hyp_trans`). cpWER was proposed in the following article: CHiME-6 Challenge: Tackling Multispeaker Speech Recognition for Unsegmented Recordings https://arxiv.org/pdf/2004.09249.pdf Implementation: - Brute force permutation method for calculating cpWER has a time complexity of `O(n!)`. - To reduce the computational burden, linear sum assignment (LSA) algorithm is applied (also known as Hungarian algorithm) to find the permutation that leads to the lowest WER. - In this implementation, instead of calculating all WER values for all permutation of hypotheses, we only calculate WER values of (estimated number of speakers) x (reference number of speakers) combinations with `O(n^2)`) time complexity and then select the permutation that yields the lowest WER based on LSA algorithm. - LSA algorithm has `O(n^3)` time complexity in the worst case. - We cannot use LSA algorithm to find the best permutation when there are more hypothesis speakers than reference speakers. In this case, we use the brute-force permutation method instead. Example: >>> transcript_A = ['a', 'b', 'c', 'd', 'e', 'f'] # 6 speakers >>> transcript_B = ['a c b d', 'e f'] # 2 speakers [case1] hypothesis is transcript_A, reference is transcript_B [case2] hypothesis is transcript_B, reference is transcript_A LSA algorithm based cpWER is: [case1] 4/6 (4 deletion) [case2] 2/6 (2 substitution) brute force permutation based cpWER is: [case1] 0 [case2] 2/6 (2 substitution) Args: spk_hypothesis (list): List containing the hypothesis transcript for each speaker. A list containing the sequence of words is assigned for each speaker. Example: >>> spk_hypothesis = ["hey how are you we that's nice", "i'm good yes hi is your sister"] spk_reference (list): List containing the reference transcript for each speaker. A list containing the sequence of words is assigned for each speaker. Example: >>> spk_reference = ["hi how are you well that's nice", "i'm good yeah how is your sister"] Returns: cpWER (float): cpWER value for the given session. min_perm_hyp_trans (str): Hypothesis transcript containing the permutation that minimizes WER. Words are separated by spaces. ref_trans (str): Reference transcript in an arbitrary permutation. Words are separated by spaces. rcsg|]}|qSrr)rkr}rrrsz+calculate_session_cpWER..r)list itertoolsproductrrr r torchtensorreshaper rargsortr)r}r~r hyp_ref_pair all_pairs num_hyp_spks num_ref_spksrrr lsa_wer_list spk_hyp_trans spk_ref_transspk_wercost_wer row_hyp_ind col_ref_ind hyp_permedrrrrGs B       rspk_hypothesesspk_referencesc Cst|t|krtdt|dt|dggg}}}t||D]\}}t||\}}} |||||| q$|||fS)a Launcher function for `calculate_session_cpWER`. Calculate session-level cpWER and average cpWER. For detailed information about cpWER, see docstrings of `calculate_session_cpWER` function. As opposed to `cpWER`, `WER` is the regular WER value where the hypothesis transcript contains words in temporal order regardless of the speakers. `WER` value can be different from cpWER value, depending on the speaker diarization results. Args: spk_hypotheses (list): List containing the lists of speaker-separated hypothesis transcripts. spk_references (list): List containing the lists of speaker-separated reference transcripts. Returns: cpWER (float): List containing cpWER values for each session min_perm_hyp_trans (list): List containing transcripts that lead to the minimum WER in string format ref_trans (list): List containing concatenated reference transcripts zIn concatenated-minimum permutation word error rate calculation, hypotheses and reference lists must have the same number of elements. But got arguments:z and z correspondingly)rr`r[rr ) rr cpWER_valueshyps_spkrefs_spkr}r~rmin_hypothesisconcat_referencerrrrs    r)r))r;)NrKTT)rq)F)%rrtypingrrrrnumpyrpandasrar pyannote.corerrpyannote.metrics.diarizationr nemo.collections.asr.metrics.werr 3nemo.collections.asr.parts.utils.optimization_utilsr nemo.utilsr __all__strr(rintr:rJrboolr r|rrrrrrrs     ",  (  `(04  d