o 7wiV@shdZddlZddlZddlmZmZmZddlmZeGdddej j Z ddZ d d Z d d ZdS) zI Alignment code Authors * Elena Rastorgueva 2020 * Loren Lugosch 2020 N)mark_as_loader mark_as_saverregister_checkpoint_hooks) undo_paddingcseZdZdZ       d1fdd Zd d Zd2d dZddZddZddZ ddZ ddZ ddZ d3ddZ ddZdd Zd!d"Zd#d$Zd%d&Zd'd(Zd3d)d*Zd+d,Zed-d.Zed4d/d0ZZS)5 HMMAlignerasThis class calculates Viterbi alignments in the forward method. It also records alignments and creates batches of them for use in Viterbi training. Arguments --------- states_per_phoneme : int Number of hidden states to use per phoneme. output_folder : str It is the folder that the alignments will be stored in when saved to disk. Not yet implemented. neg_inf : float The float used to represent a negative infinite log probability. Using `-float("Inf")` tends to give numerical instability. A number more negative than -1e5 also sometimes gave errors when the `genbmm` library was used (currently not in use). (default: -1e5) batch_reduction : string One of "none", "sum" or "mean". What kind of batch-level reduction to apply to the loss calculated in the forward method. input_len_norm : bool Whether to normalize the loss in the forward method by the length of the inputs. target_len_norm : bool Whether to normalize the loss in the forward method by the length of the targets. lexicon_path : string The location of the lexicon. Example ------- >>> log_posteriors = torch.tensor([[[ -1., -10., -10.], ... [-10., -1., -10.], ... [-10., -10., -1.]], ... ... [[ -1., -10., -10.], ... [-10., -1., -10.], ... [-10., -10., -10.]]]) >>> lens = torch.tensor([1., 0.66]) >>> phns = torch.tensor([[0, 1, 2], ... [0, 1, 0]]) >>> phn_lens = torch.tensor([1., 0.66]) >>> aligner = HMMAligner() >>> forward_scores = aligner( ... log_posteriors, lens, phns, phn_lens, 'forward' ... ) >>> forward_scores.shape torch.Size([2]) >>> viterbi_scores, alignments = aligner( ... log_posteriors, lens, phns, phn_lens, 'viterbi' ... ) >>> alignments [[0, 1, 2], [0, 1]] >>> viterbi_scores.shape torch.Size([2]) jnoneFNcst||_||_||_||_||_||_i|_||_ |j durt |j ddd }| } Wdn1s9wYt | D]\} } | ddkrP| } nqBi} t }t| t| D]L} | | } | d}| dd}dd d |D}|d D]}||qd |vr|d d}|| vrt| |}|| ||<q]d|i| |<q]| |_t|}|d dt |D|_ddt |D|_d|jd<d|jd<dSdS)Nrzutf-8)encodingr;/rrcSsg|]}|s|qS)isdigit).0prrZ/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/speechbrain/alignment/aligner.py wsz'HMMAligner.__init__.. ~cSsi|] \}}||dqSrrrirrrr z'HMMAligner.__init__..cSsi|] \}}|d|qSrrrrrrrrsil)super__init__states_per_phoneme output_folderneg_infbatch_reductioninput_len_normtarget_len_norm align_dict lexicon_pathopen readlines enumeratesetrangelensplitjoinaddlexiconlistsort lex_lab2ind lex_ind2lab)selfrr r!r"r#r$r&flinesrline start_indexr0lexicon_phoneswordphonesr!number_of_existing_pronunciations __class__rrrQsR        zHMMAligner.__init__c sldgd}g}|D]}|dks|dur@|fddtjDgdg|fddtjD7}j7|d7}|gdg|rXtj|dkrXtj|ttj|D]E}j||}|} dg| D]*|fddtjD7}d|fd dtjD7<j7qw|rnqa|d7}q |fd dtjDgdg|fd dtjD7}j7|d7}d t } g} D]d} d } | tdk}fddttdD}dD]}tt|dD]}||}||d}d | ||f<qq |rd| s6|s;| dn| dfddttdD}|D]}|D] }d | ||f<qWqSn| |7} | s| d}|ddd}|D] }d | ||f<q{q| d}ddddg}|fddttddD7}t |}|||| fS)aDo processing using the lexicon to return a sequence of the possible phonemes, the transition/pi probabilities, and the possible final states. Inputs correspond to a single utterance, not a whole batch. Arguments --------- words : list List of the words in the transcript. interword_sils : bool If True, optional silences will be inserted between every word. If False, optional silences will only be placed at the beginning and end of each utterance. sample_pron : bool If True, it will sample a single possible sequence of phonemes. If False, it will return statistics for all possible sequences of phonemes. Returns ------- poss_phns : torch.Tensor (phoneme) The phonemes that are thought to be in each utterance. log_transition_matrix : torch.Tensor (batch, from, to) Tensor containing transition (log) probabilities. start_states : list of ints A list of the possible starting states in each utterance. final_states : list of ints A list of the possible final states for each utterance. rTcg|]}|qSrrrrnumber_of_statesrrrz+HMMAligner._use_lexicon..cg|] }jj|qSr silence_indexrrAr5rrrrFcs g|] }jj|qSr)r3rrA)rr5rrrcr@rrrArBrrrrDcr@rrrArBrrrcrErrFrArHrrrrIg?csg|] }d|dqS)rrLrrA) word_primerrrscs g|] }d|dqSrrrrA) next_word_idx words_primerrrrJcs g|] }dd|dqSrOrrA)rQrrr#s) r+rappendr,r0randomshuffler-torcheyelog log_softmaxtensor)r5wordsinterword_sils sample_pron word_indexphoneme_indicesr;pron_idx pronunciationphonemestransition_matrix final_statesword_idxis_optional_silencenext_word_existsthis_word_last_states state_idxstate next_statenext_word_starting_statesthis_word_last_statenext_word_starting_statenext_silence_idxnext_silence_starting_statelog_transition_matrix start_states poss_phnsr)rPrCrr5rNrQr _use_lexicons                    zHMMAligner._use_lexiconTcCsn|jd|_g}g}g}g}|D]!}||||\} } } } || || || || qdd|D} t| }t|}t|D]-}|t||}tjj j ||d|fdd||<tjj j ||d|d|f|j d||<qEt |}t |}|j ||t d k<|j t||g}|D] }d|dd|f<qtjj j|dd }t|  |} || |||fS) aDo processing using the lexicon to return a sequence of the possible phonemes, the transition/pi probabilities, and the possible final states. Does processing on an utterance-by-utterance basis. Each utterance in the batch is processed by a helper method `_use_lexicon`. Arguments --------- words : list List of the words in the transcript interword_sils : bool If True, optional silences will be inserted between every word. If False, optional silences will only be placed at the beginning and end of each utterance. sample_pron: bool If True, it will sample a single possible sequence of phonemes. If False, it will return statistics for all possible sequences of phonemes. Returns ------- poss_phns: torch.Tensor (batch, phoneme in possible phn sequence) The phonemes that are thought to be in each utterance. poss_phn_lens: torch.Tensor (batch) The relative length of each possible phoneme sequence in the batch. trans_prob: torch.Tensor (batch, from, to) Tensor containing transition (log) probabilities. pi_prob: torch.Tensor (batch, state) Tensor containing initial (log) probabilities. final_state: list of lists of ints A list of lists of possible final states for each utterance. Example ------- >>> aligner = HMMAligner() >>> aligner.lexicon = { ... "a": {0: "a"}, ... "b": {0: "b", 1: "c"} ... } >>> words = [["a", "b"]] >>> aligner.lex_lab2ind = { ... "sil": 0, ... "a": 1, ... "b": 2, ... "c": 3, ... } >>> poss_phns, poss_phn_lens, trans_prob, pi_prob, final_states = aligner.use_lexicon( ... words, ... interword_sils = True ... ) >>> poss_phns tensor([[0, 1, 0, 2, 3, 0]]) >>> poss_phn_lens tensor([1.]) >>> trans_prob tensor([[[-6.9315e-01, -6.9315e-01, -1.0000e+05, -1.0000e+05, -1.0000e+05, -1.0000e+05], [-1.0000e+05, -1.3863e+00, -1.3863e+00, -1.3863e+00, -1.3863e+00, -1.0000e+05], [-1.0000e+05, -1.0000e+05, -1.0986e+00, -1.0986e+00, -1.0986e+00, -1.0000e+05], [-1.0000e+05, -1.0000e+05, -1.0000e+05, -6.9315e-01, -1.0000e+05, -6.9315e-01], [-1.0000e+05, -1.0000e+05, -1.0000e+05, -1.0000e+05, -6.9315e-01, -6.9315e-01], [-1.0000e+05, -1.0000e+05, -1.0000e+05, -1.0000e+05, -1.0000e+05, 0.0000e+00]]]) >>> pi_prob tensor([[-6.9315e-01, -6.9315e-01, -1.0000e+05, -1.0000e+05, -1.0000e+05, -1.0000e+05]]) >>> final_states [[3, 4, 5]] >>> # With no optional silences between words >>> poss_phns_, _, trans_prob_, pi_prob_, final_states_ = aligner.use_lexicon( ... words, ... interword_sils = False ... ) >>> poss_phns_ tensor([[0, 1, 2, 3, 0]]) >>> trans_prob_ tensor([[[-6.9315e-01, -6.9315e-01, -1.0000e+05, -1.0000e+05, -1.0000e+05], [-1.0000e+05, -1.0986e+00, -1.0986e+00, -1.0986e+00, -1.0000e+05], [-1.0000e+05, -1.0000e+05, -6.9315e-01, -1.0000e+05, -6.9315e-01], [-1.0000e+05, -1.0000e+05, -1.0000e+05, -6.9315e-01, -6.9315e-01], [-1.0000e+05, -1.0000e+05, -1.0000e+05, -1.0000e+05, 0.0000e+00]]]) >>> pi_prob_ tensor([[-6.9315e-01, -6.9315e-01, -1.0000e+05, -1.0000e+05, -1.0000e+05]]) >>> final_states_ [[2, 3, 4]] >>> # With sampling of a single possible pronunciation >>> import random >>> random.seed(0) >>> poss_phns_, _, trans_prob_, pi_prob_, final_states_ = aligner.use_lexicon( ... words, ... sample_pron = True ... ) >>> poss_phns_ tensor([[0, 1, 0, 2, 0]]) >>> trans_prob_ tensor([[[-6.9315e-01, -6.9315e-01, -1.0000e+05, -1.0000e+05, -1.0000e+05], [-1.0000e+05, -1.0986e+00, -1.0986e+00, -1.0986e+00, -1.0000e+05], [-1.0000e+05, -1.0000e+05, -6.9315e-01, -6.9315e-01, -1.0000e+05], [-1.0000e+05, -1.0000e+05, -1.0000e+05, -6.9315e-01, -6.9315e-01], [-1.0000e+05, -1.0000e+05, -1.0000e+05, -1.0000e+05, 0.0000e+00]]]) rcSg|]}t|qSrr,)r poss_phns_rrrrrKz*HMMAligner.use_lexicon..rvalueInfrNdim)r3rGrsrRmaxr,r+rUnn functionalpadr!stackfloatonesrXrY)r5rZr[r\rr trans_probrqrcwords_rv trans_prob_ start_states_ final_states_ poss_phn_lensU_max batch_sizeindexphn_pad_lengthpi_prob start_staterrr use_lexicon+sL j          zHMMAligner.use_lexiconcCs<t|}t|}|jt||g}d|dddf<|S)awCreates tensor of initial (log) probabilities (known as 'pi'). Assigns all probability mass to the first phoneme in the sequence. Arguments --------- phn_lens_abs : torch.Tensor (batch) The absolute length of each phoneme sequence in the batch. Returns ------- pi_prob : torch.Tensor (batch, phn) rN)r,intr|r!rUr)r5 phn_lens_absrrrrrr _make_pi_probs  zHMMAligner._make_pi_probc CsRt|}t|}|j}t|d}t|ddg}td|g}t||fd}t||fd}t|}||} | d|| |dd |} tj ||ddddf|dddfk} | d} | dd|} | ddd} | | | @} t| dk| tjtd |d} tjjj| dd} |j| | | k<|j| | td k<| S) aCreates tensor of transition (log) probabilities. Only allows transitions to the same phoneme (self-loop) or the next phoneme in the phn sequence Arguments --------- phn_lens_abs : torch.Tensor (batch) The absolute length of each phoneme sequence in the batch. Returns ------- trans_prob : torch.Tensor (batch, from, to) rrdeviceNrMrLryrz)r,rr|rrUrVzeroscatreshaperepeattoarange unsqueezeexpandpermuterwhererYr}r~rXr!) r5rrrrtrans_prob_off_diag zero_side zero_bottomtrans_prob_main_diagrmask_amask_brrr_make_trans_probs:    ( zHMMAligner._make_trans_probc Cst|}t|}|j}t||dddf|dddfk}t|dddddf|tjdg|d} ||}| d d|d} t | d| } t||dddf|dddfk} t| dddddf| tj|j g|d} | ddd} | S)aECreates a 'useful' form of the posterior probabilities, rearranged into the order of phoneme appearance in phns. Arguments --------- emission_pred : torch.Tensor (batch, time, phoneme in vocabulary) posterior probabilities from our acoustic model lens_abs : torch.Tensor (batch) The absolute length of each input to the acoustic model, i.e., the number of frames. phn_lens_abs : torch.Tensor (batch) The absolute length of each phoneme sequence in the batch. phns : torch.Tensor (batch, phoneme in phn sequence) The phonemes that are known/thought to be in each utterance. Returns ------- emiss_pred_useful : torch.Tensor Tensor shape (batch, phoneme in phn sequence, time). NrrrLrMr)rr|itemrrUrrrrYrrgatherr!r) r5 emission_predlens_absrphnsr fb_max_lengthr mask_lensemiss_pred_acc_lens phns_copiedemiss_pred_useful mask_phn_lensrrr_make_emiss_pred_usefuls,* *z"HMMAligner._make_emiss_pred_usefulcCs`t|}|}|} |j} || }|| }|jtj||| g| d} ||dddddf| dddddf<td| D][} || k} d| vrq| }|jt|||}d|ddt |t |f<|| }||| <t | ddd| dddd| df}||dddd| f| dddd| f<qAtj | t |dddfdd }|S) aDoes forward dynamic programming algorithm. Arguments --------- pi_prob : torch.Tensor (batch, phn) Tensor containing initial (log) probabilities. trans_prob : torch.Tensor (batch, from, to) Tensor containing transition (log) probabilities. emiss_pred_useful : torch.Tensor (batch, phoneme in phn sequence, time) A 'useful' form of the posterior probabilities, rearranged into the order of phoneme appearance in phns. lens_abs : torch.Tensor (batch) The absolute length of each input to the acoustic model, i.e., the number of frames. phn_lens_abs : torch.Tensor (batch) The absolute length of each phoneme sequence in the batch. phns : torch.Tensor (batch, phoneme in phn sequence) The phonemes that are known/thought to be in each utterance. Returns ------- sum_alpha_T : torch.Tensor (batch) The (log) likelihood of each utterance in the batch. rNrrTrrMrLrz) r,r|rrr!rUrr+sumrbatch_log_matvecmulr logsumexp)r5rrrrrrrrrr alpha_matrixtutt_lens_passedn_passedI_tensoralpha_times_trans sum_alpha_Trrr _dp_forwardYs6"   , $zHMMAligner._dp_forwardc Cs6t|}|} |} |j} || }|| }|jtj|| | g| d} dtj|| | g| d} ||dddddf| dddddf<td| D]>}t| ddd| dddd|df\}}||dddd|f| dddd|f<| tj | dddd|f<qMg}g}t|D]s}||}|dur||}| |||df}t | }||}n || d}|g}|||df g}t|ddD]'}|d}| |||df }|||f }|d||d|q||||q| t||d|df}|||fS)aCalculates Viterbi alignment using dynamic programming. Arguments --------- pi_prob : torch.Tensor (batch, phn) Tensor containing initial (log) probabilities. trans_prob : torch.Tensor (batch, from, to) Tensor containing transition (log) probabilities. emiss_pred_useful : torch.Tensor (batch, phoneme in phn sequence, time) A 'useful' form of the posterior probabilities, rearranged into the order of phoneme appearance in phns. lens_abs : torch.Tensor (batch) The absolute length of each input to the acoustic model, i.e., the number of frames. phn_lens_abs : torch.Tensor (batch) The absolute length of each phoneme sequence in the batch. phns : torch.Tensor (batch, phoneme in phn sequence) The phonemes that are known/thought to be in each utterance. final_states : list List of final states Returns ------- z_stars : list of lists of int Viterbi alignments for the files in the batch. z_stars_loc : list of lists of int The locations of the Viterbi alignments for the files in the batch. e.g., for a batch with a single utterance with 5 phonemes, `z_stars_loc` will look like: [[0, 0, 0, 1, 1, 2, 3, 3, 3, 4, 4]]. viterbi_scores : torch.Tensor (batch) The (log) likelihood of the Viterbi path for each utterance. riNrrrMrL)r,r|rrr!rUrr+batch_log_maxvecmulrtype FloatTensorargmaxrlonginsertrRr) r5rrrrrrrcrrrrv_matrix backpointersrxrz_stars z_stars_locutterance_in_batchlen_absfinal_states_utterviterbi_finalsfinal_state_chosenU z_star_i_locz_star_i time_stepcurrent_best_locearlier_best_locearlier_z_starviterbi_scoresrrr _dp_viterbisl-    ,$,         zHMMAligner._dp_viterbicCsp|jdur t||}|jdurt||}|jdkr |S|jdkr)|}|S|jdkr4|}|Std)a Applies reduction to loss as specified during object initialization. Arguments --------- loss : torch.Tensor (batch) The loss tensor to be reduced. input_lens : torch.Tensor (batch) The absolute durations of the inputs. target_lens : torch.Tensor (batch) The absolute durations of the targets. Returns ------- loss : torch.Tensor (batch, or scalar) The loss with reduction applied if it is specified. Tr rmeanzB`batch_reduction` parameter must be one of 'none', 'sum' or 'mean')r#rUdivr$r"rr ValueError)r5loss input_lens target_lensrrr_loss_reductions       zHMMAligner._loss_reductionc Cst|jd|}t|jd|}|}|dur-||} ||} d} nd|vrFd|vrFd|vrF|d} |d} |d} ntd|||||} |dkri|| | | |||} | | ||} | S|dkr| | | | |||| \}}}| |||}||fStd ) aPrepares relevant (log) probability tensors and does dynamic programming: either the forward or the Viterbi algorithm. Applies reduction as specified during object initialization. Arguments --------- emission_pred : torch.Tensor (batch, time, phoneme in vocabulary) Posterior probabilities from our acoustic model. lens : torch.Tensor (batch) The relative duration of each utterance sound file. phns : torch.Tensor (batch, phoneme in phn sequence) The phonemes that are known/thought to be in each utterance phn_lens : torch.Tensor (batch) The relative length of each phoneme sequence in the batch. dp_algorithm : string Either "forward" or "viterbi". prob_matrices : dict (Optional) Must contain keys 'trans_prob', 'pi_prob' and 'final_states'. Used to override the default forward and viterbi operations which force traversal over all of the states in the `phns` sequence. Returns ------- tensor (1) if dp_algorithm == "forward". ``forward_scores`` : torch.Tensor (batch, or scalar) The (log) likelihood of each utterance in the batch, with reduction applied if specified. (OR) (2) if dp_algorithm == "viterbi". ``viterbi_scores`` : torch.Tensor (batch, or scalar) The (log) likelihood of the Viterbi path for each utterance, with reduction applied if specified. ``alignments`` : list of lists of int Viterbi alignments for the files in the batch. rNrrrcz``prob_matrices` must contain the keys `pi_prob`, `trans_prob` and `final_states`forwardviterbiz8dp_algorithm input must be either 'forward' or 'viterbi') rUroundshaperrrrrrrr)r5rlensrphn_lens dp_algorithm prob_matricesrrrrrcrforward_scores alignments_rrrrr:sb5      zHMMAligner.forwardcst|jd|jdj}||j}t||}t|D]'\}}g}|D]|fddtjD7}q&t |||dt |f<q|S)a"Expands each phoneme in the phn sequence by the number of hidden states per phoneme defined in the HMM. Arguments --------- phns : torch.Tensor (batch, phoneme in phn sequence) The phonemes that are known/thought to be in each utterance. phn_lens : torch.Tensor (batch) The relative length of each phoneme sequence in the batch. Returns ------- expanded_phns : torch.Tensor (batch, phoneme in expanded phn sequence) Example ------- >>> phns = torch.tensor([[0., 3., 5., 0.], ... [0., 2., 0., 0.]]) >>> phn_lens = torch.tensor([1., 0.75]) >>> aligner = HMMAligner(states_per_phoneme = 3) >>> expanded_phns = aligner.expand_phns_by_states_per_phoneme( ... phns, phn_lens ... ) >>> expanded_phns tensor([[ 0., 1., 2., 9., 10., 11., 15., 16., 17., 0., 1., 2.], [ 0., 1., 2., 6., 7., 8., 0., 1., 2., 0., 0., 0.]]) rrcsg|] }j|qSrr)ri_ phoneme_indexr5rrrs z@HMMAligner.expand_phns_by_states_per_phoneme..N) rUrrrrrrr)r+rYr,)r5rr expanded_phnsrphns_uttexpanded_phns_uttrrr!expand_phns_by_states_per_phonemes   z,HMMAligner.expand_phns_by_states_per_phonemecCs<t|D]\}}||}tj|tjd}||j|<qdS)aRecords Viterbi alignments in `self.align_dict`. Arguments --------- ids : list of str IDs of the files in the batch. alignments : list of lists of int Viterbi alignments for the files in the batch. Without padding. Example ------- >>> aligner = HMMAligner() >>> ids = ['id1', 'id2'] >>> alignments = [[0, 2, 4], [1, 2, 3, 4]] >>> aligner.store_alignments(ids, alignments) >>> aligner.align_dict.keys() dict_keys(['id1', 'id2']) >>> aligner.align_dict['id1'] tensor([0, 2, 4], dtype=torch.int16) )dtypeN)r)rUrYint16cpur%)r5idsrrid alignment_irrrstore_alignmentss  zHMMAligner.store_alignmentsc Cs|}t|}t|}tj|||jd}t|D]M}||}|d||}t||t|} | dkr>> ids = ['id1', 'id2'] >>> emission_pred = torch.tensor([[[ -1., -10., -10.], ... [-10., -1., -10.], ... [-10., -10., -1.]], ... ... [[ -1., -10., -10.], ... [-10., -1., -10.], ... [-10., -10., -10.]]]) >>> lens = torch.tensor([1., 0.66]) >>> phns = torch.tensor([[0, 1, 2], ... [0, 1, 0]]) >>> phn_lens = torch.tensor([1., 0.66]) >>> aligner = HMMAligner() >>> alignment_batch = aligner.get_prev_alignments( ... ids, emission_pred, lens, phns, phn_lens ... ) >>> alignment_batch tensor([[0, 1, 2], [0, 1, 0]]) rr)rUrrrr%rr)r5rrrrrrrrrrget_prev_alignmentsZs 1 zHMMAligner.get_prev_alignmentsc sdgddDfddtdtD}g}tt|D] }|||g||7}q t|}tttt|t|}t|}||}|dt|}t|t|krmtjj |dt|t|f}||k d} | S)aCalculates the accuracy between predicted alignments and ground truth alignments for a single sentence/utterance. Arguments --------- alignments_ : list of ints The predicted alignments for the utterance. ends_ : list of ints A list of the sample indices where each ground truth phoneme ends, according to the transcription. phns_ : list of ints The unpadded list of ground truth phonemes in the utterance. Returns ------- mean_acc : float The mean percentage of times that the upsampled predicted alignment matches the ground truth alignment. rcSrtr)r)rendrrrrrKz2HMMAligner._calc_accuracy_sent..cs g|] }||dqSrrrAends_rrrs rNd) r+r,rUrYrrrr}r~rrrr) r5 alignments_rphns_true_durationstrue_alignmentsrupsample_factoralignments_upsampledaccuracyrrr_calc_accuracy_sents*   zHMMAligner._calc_accuracy_sentc sg}jdkrfdd|D}|durt|||\}}t|||D]\}}}|||} || q"t|}|} | dS)aqCalculates mean accuracy between predicted alignments and ground truth alignments. Ground truth alignments are derived from ground truth phns and their ends in the audio sample. Arguments --------- alignments : list of lists of ints/floats The predicted alignments for each utterance in the batch. ends : list of lists of ints A list of lists of sample indices where each ground truth phoneme ends, according to the transcription. Note: current implementation assumes that 'ends' mark the index where the next phoneme begins. phns : list of lists of ints/floats The unpadded list of lists of ground truth phonemes in the batch. ind2labs : tuple (Optional) Contains the original index-to-label dicts for the first and second sequence of phonemes. Returns ------- mean_acc : float The mean percentage of times that the upsampled predicted alignment matches the ground truth alignment. Example ------- >>> aligner = HMMAligner() >>> alignments = [[0., 0., 0., 1.]] >>> phns = [[0., 1.]] >>> ends = [[2, 4]] >>> mean_acc = aligner.calc_accuracy(alignments, ends, phns) >>> mean_acc.item() 75.0 rcg|] }fdd|DqS)cg|]}|jqSrrrArHrrrz7HMMAligner.calc_accuracy...rruttrHrrrsz,HMMAligner.calc_accuracy..Nr) rmap_inds_to_intersectzipr rRrUrYrr) r5rendsrind2labsacc_histrrraccmean_accrrHr calc_accuracys%     zHMMAligner.calc_accuracycs>fddtD}fdd|D}fdd|D}|S)aT Converts alignments to 1 state per phoneme style. Arguments --------- alignments : list of ints Predicted alignments for a single utterance. Returns ------- sequence : list of ints The predicted alignments converted to a 1 state per phoneme style. Example ------- >>> aligner = HMMAligner(states_per_phoneme = 3) >>> alignments = [0, 1, 2, 3, 4, 5, 3, 4, 5, 0, 1, 2] >>> sequence = aligner.collapse_alignments(alignments) >>> sequence [0, 1, 1, 0] cs,g|]\}}|dks||dkr|qS)rrr)rrv)rrrrs z2HMMAligner.collapse_alignments..csg|] }|jdkr|qS)rrrrrHrrr!crrrrrHrrr$r)r))r5rsequencer)rr5rcollapse_alignmentss zHMMAligner.collapse_alignmentscCst|j|dSN)rUsaver%)r5pathrrr_save(szHMMAligner._savecCs~t||_dSr)rUloadr%)r5r! end_of_epochrrr_load,szHMMAligner._load)rrr r FFN)TFr)F)__name__ __module__ __qualname____doc__rrsrrrrrrrrrrrrrr rrrr"rr% __classcell__rrr>rrsD<> B:Ht, w0:#9 4;& rc s|\tt}}||}||}||}ddt|Dfddt|Dfddt|Dfdd|D}fdd|D} fdd|D} fd d| D} | | fS) aConverts 2 lists containing indices for phonemes from different phoneme sets to a single phoneme so that comparing the equality of the indices of the resulting lists will yield the correct accuracy. Arguments --------- lists1 : list of lists of ints Contains the indices of the first sequence of phonemes. lists2 : list of lists of ints Contains the indices of the second sequence of phonemes. ind2labs : tuple (dict, dict) Contains the original index-to-label dicts for the first and second sequence of phonemes. Returns ------- lists1_new : list of lists of ints Contains the indices of the first sequence of phonemes, mapped to the new phoneme set. lists2_new : list of lists of ints Contains the indices of the second sequence of phonemes, mapped to the new phoneme set. Example ------- >>> lists1 = [[0, 1]] >>> lists2 = [[0, 1]] >>> ind2lab1 = { ... 0: "a", ... 1: "b", ... } >>> ind2lab2 = { ... 0: "a", ... 1: "c", ... } >>> ind2labs = (ind2lab1, ind2lab2) >>> out1, out2 = map_inds_to_intersect(lists1, lists2, ind2labs) >>> out1 [[0, 1]] >>> out2 [[0, 2]] cSsi|]\}}||qSrrrrlabrrrrjrz)map_inds_to_intersect..ci|] \}}|t|qSrrur+ new_lab2indrrrlrcr-rrur+r.rrrorcr )cg|]}|qSrrrindind2lab1rrrsrK4map_inds_to_intersect...rrr3rrrsrz)map_inds_to_intersect..cr )cr0rrr1ind2lab2rrrtrKr5rrr6rrrtrcr )cr0rrrr,r.rrrvrKr5rrr.rrrvrcr )cr0rrr8r.rrrwrKr5rrr.rrrwr)r*values intersection differencer)update) lists1lists2rset1set2 intersect set1_only set2_only lists1_lab lists2_lab lists1_new lists2_newr)r4r7r/rr2s",   rcCs |d}tj||dd}|S)aKFor each 'matrix' and 'vector' pair in the batch, do matrix-vector multiplication in the log domain, i.e., logsumexp instead of add, add instead of multiply. Arguments --------- A : torch.Tensor (batch, dim1, dim2) Tensor b : torch.Tensor (batch, dim1) Tensor. Returns ------- x : torch.Tensor (batch, dim1) Example ------- >>> A = torch.tensor([[[ 0., 0.], ... [ -1e5, 0.]]]) >>> b = torch.tensor([[0., 0.,]]) >>> x = batch_log_matvecmul(A, b) >>> x tensor([[0.6931, 0.0000]]) >>> >>> # non-log domain equivalent without batching functionality >>> A_ = torch.tensor([[1., 1.], ... [0., 1.]]) >>> b_ = torch.tensor([1., 1.,]) >>> x_ = torch.matmul(A_, b_) >>> x_ tensor([2., 1.]) rrMrz)rrUr)Abrrrrr|s !rcCs(|d}tj||dd\}}||fS)aSimilar to batch_log_matvecmul, but takes a maximum instead of logsumexp. Returns both the max and the argmax. Arguments --------- A : torch.Tensor (batch, dim1, dim2) Tensor. b : torch.Tensor (batch, dim1) Tensor Returns ------- x : torch.Tensor (batch, dim1) Tensor. argmax : torch.Tensor (batch, dim1) Tensor. Example ------- >>> A = torch.tensor([[[ 0., -1.], ... [ -1e5, 0.]]]) >>> b = torch.tensor([[0., 0.,]]) >>> x, argmax = batch_log_maxvecmul(A, b) >>> x tensor([[0., 0.]]) >>> argmax tensor([[0, 1]]) rrMrz)rrUr|)rHrIrrrrrrs r)r)rSrUspeechbrain.utils.checkpointsrrrspeechbrain.utils.data_utilsrr}Modulerrrrrrrrs( &J '