o %ݫij@sdZddlZddlmZdddddZd ed efd d Zeefd eeegeffddZ efd eeegeffddZ efd eeegeffddZ ddZ ddZ ddZdefd eeegeffddZddefd eeegeffddZd d!Zd"d#Zd*d%d&Zd+d(d)ZdS),zVEdit distance and WER computation. Authors * Aku Rouhe 2020 * Salima Mdhaffar 2021 N)Callable=IDS)eqinsdelsubabcCs||kSN)r r rrS/home/ubuntu/.local/lib/python3.10/site-packages/speechbrain/utils/edit_distance.py _str_equalssrequality_comparatorcCs`|t|||d}|ddkrtd|d<|St|d|d|dg}d ||d|d<|S) aComputes word error rate and the related counts for a batch. Can also be used to accumulate the counts over many batches, by passing the output back to the function in the call for the next batch. Arguments --------- refs : iterable Batch of reference sequences. hyps : iterable Batch of hypothesis sequences. stats : collections.Counter The running statistics. Pass the output of this function back as this parameter to accumulate the counts. It may be cleanest to initialize the stats yourself; then an empty collections.Counter() should be used. equality_comparator : Callable[[str, str], bool] The function used to check whether two words are equal. Returns ------- collections.Counter The updated running statistics, with keys: * "WER" - word error rate * "insertions" - number of insertions * "deletions" - number of deletions * "substitutions" - number of substitutions * "num_ref_tokens" - number of reference tokens Example ------- >>> import collections >>> batches = [[[[1,2,3],[4,5,6]], [[1,2,4],[5,6]]], ... [[[7,8], [9]], [[7,8], [10]]]] >>> stats = collections.Counter() >>> for batch in batches: ... refs, hyps = batch ... stats = accumulatable_wer_stats(refs, hyps, stats) >>> print("%WER {WER:.2f}, {num_ref_tokens} ref tokens".format(**stats)) %WER 33.33, 9 ref tokens rnum_ref_tokensrnanWER insertions deletions substitutionsY@) _batch_statsfloatsum)refshypsstatsr updated_stats num_editsrrraccumulatable_wer_statss1   r"cCsjt|t|kr tdt}t||D]\}}t|||d}t|}||7}|dt|7<q|S)aInternal function which actually computes the counts. Used by accumulatable_wer_stats Arguments --------- refs : iterable Batch of reference sequences. hyps : iterable Batch of hypothesis sequences. equality_comparator : Callable[[str, str], bool] The function used to check whether two words are equal. Returns ------- collections.Counter Edit statistics over the batch, with keys: * "insertions" - number of insertions * "deletions" - number of deletions * "substitutions" - number of substitutions * "num_ref_tokens" - number of reference tokens Example ------- >>> from speechbrain.utils.edit_distance import _batch_stats >>> batch = [[[1,2,3],[4,5,6]], [[1,2,4],[5,6]]] >>> refs, hyps = batch >>> print(_batch_stats(refs, hyps)) Counter({'num_ref_tokens': 6, 'substitutions': 1, 'deletions': 1}) z=The reference and hypothesis batches are not of the same sizerr)len ValueError collectionsCounterzipop_table count_ops)rrrr ref_tokens hyp_tokenstableeditsrrrr_s"rcsddttdD}dgtd}fddtt|dD}tt|dD] }td||d<q-ttdD] }td|d|<q@td|dd<t|dd D]t\}}|dd7<tdd D]W\}} ||dd} ||d} ||| rdnd} ||d| } | | kr| | kr| ||<| rtd |||<qk| | kr| ||<td|||<qk| ||<td|||<qk|d d |d d <qY|S) a5Table of edit operations between a and b. Solves for the table of edit operations, which is mainly used to compute word error rate. The table is of size ``[|a|+1, |b|+1]``, and each point ``(i, j)`` in the table has an edit operation. The edit operations can be deterministically followed backwards to find the shortest edit path to from ``a[:i-1] to b[:j-1]``. Indexes of zero (``i=0`` or ``j=0``) correspond to an empty sequence. The algorithm itself is well known, see `Levenshtein distance `_ Note that in some cases there are multiple valid edit operation paths which lead to the same edit distance minimum. Arguments --------- a : iterable Sequence for which the edit operations are solved. b : iterable Sequence for which the edit operations are solved. equality_comparator : Callable[[str, str], bool] The function used to check whether two words are equal. Returns ------- list List of lists, Matrix, Table of edit operations. Example ------- >>> ref = [1,2,3] >>> hyp = [1,2,4] >>> for row in op_table(ref, hyp): ... print(row) ['=', 'I', 'I', 'I'] ['D', '=', 'I', 'I'] ['D', 'D', '=', 'I'] ['D', 'D', 'D', 'S'] cSsg|]}|qSrr.0jrrr szop_table..rcs&g|]}ddttdDqS)cSsg|]}tdqS)r) EDIT_SYMBOLSr.rrrr1sz'op_table...r2)ranger#)r/ir rrr1sr rr)startr N)r4r#r3 enumerate)r r rprev_rowcurr_rowr,r5r0a_tokenb_tokeninsertion_cost deletion_cost substitutionsubstitution_costrr6rr(s</  r(cCs^g}t|d}t|dd}|dkr|dks|dkr,|d8}|dtdd|fny|dkr@|d8}|dtd|dfne|||tdkrZ|d8}|dtdd|fnK|||tdkrt|d8}|dtd|dfn1|||tdkr|d8}|d8}|dtd||fn|d8}|d8}|dtd||f|dkr|dkr|S)a0Get the edit distance alignment from an edit op table. Walks back an edit operations table, produced by calling ``table(a, b)``, and collects the edit distance alignment of a to b. The alignment shows which token in a corresponds to which token in b. Note that the alignment is monotonic, one-to-zero-or-one. Arguments --------- table : list Edit operations table from ``op_table(a, b)``. Returns ------- list Schema: ``[(str , int-or-None , int-or-None ),]`` List of edit operations, and the corresponding indices to a and b. See the EDIT_SYMBOLS dict for the edit-ops. The i indexes a, j indexes b, and the indices can be None, which means aligning to nothing. Example ------- >>> # table for a=[1,2,3], b=[1,2,4]: >>> table = [['I', 'I', 'I', 'I'], ... ['D', '=', 'I', 'I'], ... ['D', 'D', '=', 'I'], ... ['D', 'D', 'D', 'S']] >>> print(alignment(table)) [('=', 0, 0), ('=', 1, 1), ('S', 2, 2)] r2rrNr r r)r#insertr3)r, alignmentr5r0rrrrBs2" rBcCst}t|d}t|dd}|dkr|dks|dkr+|dd7<|d8}nY|dkr<|dd7<|d8}nH|||tdkrS|dd7<|d8}n1|||tdkrj|dd7<|d8}n|||tdkr||dd7<|d8}|d8}|dkr|dkr|S) aCount the edit operations in the shortest edit path in edit op table. Walks back an edit operations table produced by table(a, b) and counts the number of insertions, deletions, and substitutions in the shortest edit path. This information is typically used in speech recognition to report the number of different error types separately. Arguments --------- table : list Edit operations table from ``op_table(a, b)``. Returns ------- collections.Counter The counts of the edit operations, with keys: * "insertions" * "deletions" * "substitutions" NOTE: not all of the keys might appear explicitly in the output, but for the missing keys collections. The counter will return 0. Example ------- >>> table = [['I', 'I', 'I', 'I'], ... ['D', '=', 'I', 'I'], ... ['D', 'D', '=', 'I'], ... ['D', 'D', 'D', 'S']] >>> print(count_ops(table)) Counter({'substitutions': 1}) r2rrrrr r r)r%r&r#r3)r,r-r5r0rrrr)-s,"     r)cCstt||Sr )dictr')idsseqsrrr_batch_to_dict_formatisrFFcCs&t||}t||}t|||d|dS)a&Convenient batch interface for ``wer_details_by_utterance``. ``wer_details_by_utterance`` can handle missing hypotheses, but sometimes (e.g. CTC training with greedy decoding) they are not needed, and this is a convenient interface in that case. Arguments --------- ids : list, torch.tensor Utterance ids for the batch. refs : list, torch.tensor Reference sequences. hyps : list, torch.tensor Hypothesis sequences. compute_alignments : bool, optional Whether to compute alignments or not. If computed, the details will also store the refs and hyps. (default: False) equality_comparator : Callable[[str, str], bool] The function used to check whether two words are equal. Returns ------- list See ``wer_details_by_utterance`` Example ------- >>> ids = [['utt1'], ['utt2']] >>> refs = [[['a','b','c']], [['d','e']]] >>> hyps = [[['a','b','d']], [['d','e']]] >>> wer_details = [] >>> for ids_batch, refs_batch, hyps_batch in zip(ids, refs, hyps): ... details = wer_details_for_batch(ids_batch, refs_batch, hyps_batch) ... wer_details.extend(details) >>> print(wer_details[0]['key'], ":", ... "{:.2f}".format(wer_details[0]['WER'])) utt1 : 33.33 strict)compute_alignments scoring_moder)rFwer_details_by_utterance)rDrrrHrrrrwer_details_for_batchns - rKrGc Csg}|D]\}}|ddddt|ddddd|r|nddd }||vr0|ddi||} n1|dkr>|ddig} n#|dkrO|ddi||q|dkr[td |d td |t|| |d } t| } t|d ksx|d dkr{d } nt|} |dt| d krdndt| | dt| t d| | d| d| d|rt | nd|r|nd|r| ndd ||q|S)a; Computes a wealth WER info about each single utterance. This info can then be used to compute summary details (WER, SER). Arguments --------- ref_dict : dict Should be indexable by utterance ids, and return the reference tokens for each utterance id as iterable hyp_dict : dict Should be indexable by utterance ids, and return the hypothesis tokens for each utterance id as iterable compute_alignments : bool Whether alignments should also be saved. This also saves the tokens themselves, as they are probably required for printing the alignments. scoring_mode : {'strict', 'all', 'present'} How to deal with missing hypotheses (reference utterance id not found in hyp_dict). * 'strict': Raise error for missing hypotheses. * 'all': Score missing hypotheses as empty. * 'present': Only score existing hypotheses. equality_comparator : Callable[[str, str], bool] The function used to check whether two words are equal. Returns ------- list A list with one entry for every reference utterance. Each entry is a dict with keys: * "key": utterance id * "scored": (bool) Whether utterance was scored. * "hyp_absent": (bool) True if a hypothesis was NOT found. * "hyp_empty": (bool) True if hypothesis was considered empty (either because it was empty, or not found and mode 'all'). * "num_edits": (int) Number of edits in total. * "num_ref_tokens": (int) Number of tokens in the reference. * "WER": (float) Word error rate of the utterance. * "insertions": (int) Number of insertions. * "deletions": (int) Number of deletions. * "substitutions": (int) Number of substitutions. * "alignment": If compute_alignments is True, alignment as list, see ``speechbrain.utils.edit_distance.alignment``. If compute_alignments is False, this is None. * "ref_tokens": (iterable) The reference tokens only saved if alignments were computed, else None. * "hyp_tokens": (iterable) the hypothesis tokens, only saved if alignments were computed, else None. Raises ------ KeyError If scoring mode is 'strict' and a hypothesis is not found. FN) keyscored hyp_absent hyp_emptyr!rrrrrrBr*r+rNallTpresentrGzKey z; in reference but missing in hypothesis and strict mode on.zInvalid scoring mode: rrrr2rrr) rMrOr!rrrrrrBr*r+) itemsr#updateappendKeyErrorr$r(r)rvaluesmaxrB) ref_dicthyp_dictrHrIrdetails_by_utterancerLr*utterance_detailsr+r,opsrrrrrJst?        rJc Csd}}}d}}}}}} |D]>} | d7} | drJ|d7}|| d7}|| d7}|| d7}|| d7}|| d7}| ddkrJ|d7}| d rR|d7}q|dkr^d ||} nd } | d |||||||| |||d } | S) aD Computes summary stats from the output of details_by_utterance Summary stats like WER Arguments --------- details_by_utterance : list See the output of wer_details_by_utterance Returns ------- dict Dictionary with keys: * "WER": (float) Word Error Rate. * "SER": (float) Sentence Error Rate (percentage of utterances which had at least one error). * "num_edits": (int) Total number of edits. * "num_scored_tokens": (int) Total number of tokens in scored reference utterances (a missing hypothesis might still have been scored with 'all' scoring mode). * "num_erroneous_sents": (int) Total number of utterances which had at least one error. * "num_scored_sents": (int) Total number of utterances which were scored. * "num_absent_sents": (int) Hypotheses which were not found. * "num_ref_sents": (int) Number of all reference utterances. * "insertions": (int) Total number of insertions. * "deletions": (int) Total number of deletions. * "substitutions": (int) Total number of substitutions. NOTE: Some cases lead to ambiguity over number of insertions, deletions and substitutions. We aim to replicate Kaldi compute_wer numbers. rr2rMrrrrr!rNrg) rSERr!num_scored_tokensnum_erroneous_sentsnum_scored_sentsnum_absent_sents num_ref_sentsrrrr) r[rdelssubsr_rar!r`rbrcdetsr wer_detailsrrr wer_summary)sF &        rhcCs2i}|D]W}||d}||t|dddddddddd }t}|dr/|ddi|drV|d|d|d |d |d |d d |d dkrV|ddi||qg}|D]4\}}||d<|ddkrd|d |d|d<d|d|d|d<nd|d<d|d<||qb|S)aCompute word error rate and another salient info grouping by speakers. Arguments --------- details_by_utterance : list See the output of wer_details_by_utterance utt2spk : dict Map from utterance id to speaker id Returns ------- dict Maps speaker id to a dictionary of the statistics, with keys: * "speaker": Speaker id, * "num_edits": (int) Number of edits in total by this speaker. * "insertions": (int) Number insertions by this speaker. * "dels": (int) Number of deletions by this speaker. * "subs": (int) Number of substitutions by this speaker. * "num_scored_tokens": (int) Number of scored reference tokens by this speaker (a missing hypothesis might still have been scored with 'all' scoring mode). * "num_scored_sents": (int) number of scored utterances by this speaker. * "num_erroneous_sents": (int) number of utterance with at least one error, by this speaker. * "num_absent_sents": (int) number of utterances for which no hypotheses was found, by this speaker. * "num_ref_sents": (int) number of utterances by this speaker in total. rLr) speakerrrdrer_rar!r`rbrcrNrbr2rMrrrrr!)rar_rrdrer!r`rirarr_rr^N) setdefaultr%r&rTrSrU)r[utt2spkdetails_by_speakerrfrispk_dets utt_statsdetails_by_speaker_dictsrrrwer_details_by_speakertsf"     rpcCsdd|D}t|dddd}g}g}|rTt||ks"t||krT|d}|dr7t||kr7||n|dsFt||krF|||rTt||ks"t||ks"||fS) a Finds the k utterances with highest word error rates. Useful for diagnostic purposes, to see where the system is making the most mistakes. Returns results utterances which were not empty i.e. had to have been present in the hypotheses, with output produced Arguments --------- details_by_utterance : list See output of wer_details_by_utterance. top_k : int Number of utterances to return. Returns ------- list List of at most K utterances, with the highest word error rates, which were not empty. The utterance dict has the same keys as details_by_utterance. cSsg|]}|dr|qS)rMrr/rfrrrr1s z top_wer_utts..cS|dSNrrdrrrztop_wer_utts..TrLreverserrO)sortedr#poprU)r[top_kscored_utterances utts_by_wer top_non_empty top_emptyuttrrr top_wer_uttss$    r cCs<dd|D}t|dddd}t||kr|d|S|S)a Finds the K speakers with the highest word error rates. Useful for diagnostic purposes. Arguments --------- details_by_speaker : list See output of wer_details_by_speaker. top_k : int Number of speakers to return. Returns ------- list List of at most K dicts (with the same keys as details_by_speaker) of speakers sorted by WER. cSsg|] }|ddkr|qS)rarrrrrrrr1sz top_wer_spks..cSrsrtrrurrrrwrxztop_wer_spks..TryN)r{r#)rlr}scored_speakers spks_by_werrrr top_wer_spkss  r)rq)r)__doc__r%typingrr3strrr&boolr"rr(rBr)rFrKrJrhrprrrrrrsR   E 2 ^?<  ; K ]+