o ei}#@sdZddlmZddlZddlmZddlmZmZm Z ddl m Z Gdddej j ZGd d d ej j ZGd d d eZGd ddeZGdddeZGdddeZGdddeZGdddeZGdddeZGdddeZGdddeZdS)zDecoding methods for seq2seq autoregressive model. Authors * Adel Moumen 2022, 2023, 2024 * Ju-Chieh Chou 2020 * Peter Plantinga 2020 * Mirco Ravanelli 2020 * Sung-Lin Yeh 2020 )cached_propertyN) Categorical) _update_meminflate_tensormask_by_condition) undo_paddingc0eZdZdZfddZddZddZZS)AlivedHypothesesaxThis class handle the data for the hypotheses during the decoding. Arguments --------- alived_seq : torch.Tensor The sequence of tokens for each hypothesis. alived_log_probs : torch.Tensor The log probabilities of each token for each hypothesis. sequence_scores : torch.Tensor The sum of log probabilities for each hypothesis. cs t||_||_||_dSN)super__init__ alived_seqalived_log_probssequence_scores)selfr rr __class__Z/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/speechbrain/decoders/seq2seq.pyr %s  zAlivedHypotheses.__init__cCs|j||j||j|fSr r rr)rindexrrr __getitem__+szAlivedHypotheses.__getitem__cCsd|jd|jd|jdS)NzAlivedHypotheses(alived_seq=z, alived_log_probs=z, sequence_scores=)rrrrr__str__2szAlivedHypotheses.__str__)__name__ __module__ __qualname____doc__r rr __classcell__rrrrr s  r csPeZdZdZfddZddZddZdd Zd d Zd d Z ddZ Z S)S2SBaseSearcheraS2SBaseSearcher class to be inherited by other decoding approaches for seq2seq model. Arguments --------- bos_index : int The index of the beginning-of-sequence (bos) token. eos_index : int The index of end-of-sequence (eos) token. min_decode_ratio : float The ratio of minimum decoding steps to the length of encoder states. max_decode_ratio : float The ratio of maximum decoding steps to the length of encoder states. cs&t||_||_||_||_dSr )r r bos_index eos_indexmin_decode_ratiomax_decode_ratio)rr!r"r#r$rrrr Fs  zS2SBaseSearcher.__init__cCt)aThis method should implement the forward algorithm of decoding method. Arguments --------- enc_states : torch.Tensor The precomputed encoder states to be used when decoding. (ex. the encoded speech representation to be attended). wav_len : torch.Tensor The speechbrain-style relative length. Returns ------- hyps The predicted tokens, as a list of lists or, if return_topk is True, a Tensor of shape (batch, topk, max length of token_id sequences). top_lengths The length of each topk sequence in the batch. top_scores This final scores of topk hypotheses. top_log_probs The log probabilities of each hypotheses. NotImplementedError)r enc_stateswav_lenrrrforwardOszS2SBaseSearcher.forwardcCr%)aIThis method should implement one step of forwarding operation in the autoregressive model. Arguments --------- inp_tokens : torch.Tensor The input tensor of the current step. memory : No limit The memory variables input for this step. (ex. RNN hidden states). enc_states : torch.Tensor The encoder states to be attended. enc_lens : torch.Tensor The actual length of each enc_states sequence. Returns ------- log_probs : torch.Tensor Log-probabilities of the current step output. memory : No limit The memory variables generated in this step. (ex. RNN hidden states). attn : torch.Tensor The attention weight for doing penalty. r&)r inp_tokensmemoryr(enc_lensrrr forward_stepiszS2SBaseSearcher.forward_stepcCr%)aThis method should implement the resetting of memory variables for the seq2seq model. E.g., initializing zero vector as initial hidden states. Arguments --------- batch_size : int The size of the batch. device : torch.device The device to put the initial variables. Return ------ memory : No limit The initial memory variable. r&r batch_sizedevicerrr reset_memszS2SBaseSearcher.reset_memcCs||fS)zzCS2SWhisperGreedySearcher.get_tokens_to_suppress..,rBcSg|]}|dkr|qSrrrrrrrNrsuppress_tokens must be a listr isinstancestrsplitextendr~non_speech_tokensrolist transcribe translaterbos_prevbos_lmtuplesortedsetrrrrrget_tokens_to_suppress,  z/S2SWhisperGreedySearcher.get_tokens_to_suppresscC|jjj}|j}|j}|r9t|tr|jjjd|ddn|}|j dur5|j d|j }|| d}||}|rbt|trM|jjjd|ddn|}|jj g||j dd d|}t |Sz2Get the initial tokens to be used during decoding. Fadd_special_tokensNrr@ r~ tokenizer prefix_tokensrrrrencodestriprrrrrtokensrrrmax_prefix_len prompt_tokensrrrr8    z,S2SWhisperGreedySearcher._get_initial_tokenscCt|jrd|_tjg||_|jdd}t|g||}|jdur8|j|dd|j |j j df<d|_ |SzJThis method set the first tokens to be decoder_input_tokens during search.NrBr@ rrrFnanrrrqtorrr~r lang_tokenrr0r1 memory_tokensmemrrrr2s z"S2SWhisperGreedySearcher.reset_memc Cst||}|jj|||jd\}}}|jd|jkr;|dd|j|jjf j dd} | dd|jj f |_ |dddf}|jrI||_|jri|jd|jkritj |dd|jjjddd|jgf<|jr|jjjdurw|j} n|jj} tj |ddt| f<|||fS 0Performs a step in the implemented beamsearcher.)past_key_valuesr@NrBrCrFr)rr~forward_decoderrrHrrrrrUr no_speechtolistrrrrFrWrrr"rrrget_suppress_tokensr) rr+r,r(r-rrErkv probs_at_bostokens_to_suppressrrrr. sD   z%S2SWhisperGreedySearcher.forward_stepcCs|jd|j|jkSz0This method checks if the max length is reached.r@)rHrrr<rrrr=2sz-S2SWhisperGreedySearcher._check_end_condition)rzTTrNNN)rrrrr rrrrrrr2r.r=rrrrrr`s&!' !(rcrx) S2SRNNGreedySearchera This class implements the greedy decoding for AttentionalRNNDecoder (speechbrain/nnet/RNN.py). See also S2SBaseSearcher() and S2SGreedySearcher(). Arguments --------- embedding : torch.nn.Module An embedding layer. decoder : torch.nn.Module Attentional RNN decoder. linear : torch.nn.Module A linear output layer. temperature : float The temperature to use during decoding. **kwargs see S2SBaseSearcher, arguments are directly passed. Example ------- >>> import speechbrain as sb >>> from speechbrain.decoders import S2SRNNGreedySearcher >>> emb = torch.nn.Embedding(5, 3) >>> dec = sb.nnet.RNN.AttentionalRNNDecoder( ... "gru", "content", 3, 3, 1, enc_dim=7, input_size=3 ... ) >>> lin = sb.nnet.linear.Linear(n_neurons=5, input_size=3) >>> searcher = S2SRNNGreedySearcher( ... embedding=emb, ... decoder=dec, ... linear=lin, ... bos_index=0, ... eos_index=1, ... min_decode_ratio=0, ... max_decode_ratio=1, ... ) >>> batch_size = 2 >>> enc = torch.rand([batch_size, 6, 7]) >>> wav_len = torch.ones([batch_size]) >>> top_hyps, top_lengths, _, _ = searcher(enc, wav_len) rzc s>tjdi|||_||_||_||_tjjdd|_ dSNrBrCr) r r embdecr6rOrFrRrrr embeddingrlinearrOrrrrr bs zS2SRNNGreedySearcher.__init__cC,d}|jjtj||jj|d}||fS)zcWhen doing greedy search, keep hidden state (hs) and context vector (c) as memory. NrArrresetrFzerosattn_dimrr0r1hscrrrr2js zS2SRNNGreedySearcher.reset_memc CsF|\}}||}|j|||||\}}}} ||} | ||f| fS)r)rrr.r6) rr+r,r(r-rredec_outr7rErrrr.ss   z!S2SRNNGreedySearcher.forward_steprrrrrrr7s * rcseZdZdZ         d;fd d Zd d Zd dZddZddZddZ ddZ ddZ ddZ ddZ ddZdd Zd!d"Zd#d$Zd%d&Zd'd(Zd)d*Zd+d,Zd-d.Zd/d0Zd1d2Zd3d4Zd5d6Zd7d8Zd9d:ZZS)<S2SBeamSearchera3This class implements the beam-search algorithm for the seq2seq model. See also S2SBaseSearcher(). Arguments --------- bos_index : int The index of beginning-of-sequence token. eos_index : int The index of end-of-sequence token. min_decode_ratio : float The ratio of minimum decoding steps to length of encoder states. max_decode_ratio : float The ratio of maximum decoding steps to length of encoder states. beam_size : int The width of beam. scorer: speechbrain.decoders.scorers.ScorerBuilder Scorer instance. Default: None. return_topk : bool Whether to return topk hypotheses. The topk hypotheses will be padded to the same length. Default: False. topk : int If return_topk is True, then return topk hypotheses. Default: 1. using_eos_threshold : bool Whether to use eos threshold. Default: True. eos_threshold : float The threshold coefficient for eos token. Default: 1.5. See 3.1.2 in reference: https://arxiv.org/abs/1904.02619 length_normalization : bool Whether to divide the scores by the length. Default: True. using_max_attn_shift: bool Whether using the max_attn_shift constraint. Default: False. max_attn_shift: int Beam search will block the beams that attention shift more than max_attn_shift. Default: 60. Reference: https://arxiv.org/abs/1904.02619 minus_inf : float The value of minus infinity to block some path of the search. Default: -1e20. NFr@T?<@xcst||||||_||_||_||_| |_| |_| |_| |_ | |_ d|_ d|_ ||_ |jdurs| r@|jjddkr@td|jjddkrui|jj|jj}|dj}t|||hdkrdtd|jjd|_ d|j |_ dSdSdS)N?rzlengthz=Length normalization is not compatible with length rewarding.ctczMSet blank, eos and bos to different indexes for joint ATT/CTC or CTC decoding)r r beam_sizescorer return_topktopklength_normalizationusing_eos_threshold eos_thresholdusing_max_attn_shiftmax_attn_shift attn_weight ctc_weight minus_infweights ValueError full_scorerspartial_scorers blank_indexro)rr!r"r#r$rrrrrrrrrr all_scorersrrrrr sF  zS2SBeamSearcher.__init__cs0dd|D}fddtt|D}||kS)aVThis method checks whether hyps has been full. Arguments --------- hyps : List This list contains batch_size number. Each inside list contains a list stores all the hypothesis for this sentence. Returns ------- bool Whether the hyps has been full. cSrrro)rlstrrrrrz5S2SBeamSearcher._check_full_beams..csg|]}jqSr)rrr_rrrrs)rNro)rrhhyps_len beams_sizerrr_check_full_beamssz!S2SBeamSearcher._check_full_beamscCsDtj|dd\}}|||jk}|||jk}||d}||fS)aThis method checks whether attention shift is more than attn_shift. Arguments --------- attn : torch.Tensor The attention to be checked. prev_attn_peak : torch.Tensor The previous attention peak place. Returns ------- cond : torch.BoolTensor Each element represents whether the beam is within the max_shift range. attn_peak : torch.Tensor The peak of the attn tensor. r@rC)rFrZrrr)rrprev_attn_peakr_ attn_peaklt_condmt_condcondrrr_check_attn_shifts z!S2SBeamSearcher._check_attn_shiftcCs6tj|dd\}}|dd|jf}||j|k}|S)aIThis method checks whether eos log-probabilities exceed threshold. Arguments --------- log_probs : torch.Tensor The log-probabilities. Returns ------- cond : torch.BoolTensor Each element represents whether the eos log-probabilities will be kept. rBrCN)rFrZr"r)rr` max_probsr_ eos_probsr*rrr_check_eos_threshold s z$S2SBeamSearcher._check_eos_thresholdcCsVttj|jd|jdtj|jd|jdtj|j|jdtdd|j ddS)zThis method initializes the AlivedHypotheses object. Returns ------- AlivedHypotheses The alived hypotheses filled with the initial values. rrA-infrzr) r rFemptyn_bhr1rLrKrU index_fill_ beam_offsetrrrrinit_hypothesess  zS2SBeamSearcher.init_hypothesescCs4|jdkr|||||\}}}|j|}|||fS)aThis method computes a forward_step if attn_weight is superior to 0. Arguments --------- inp_tokens : torch.Tensor The input tensor of the current step. memory : No limit The memory variables input for this step. (ex. RNN hidden states). enc_states : torch.Tensor The encoder states to be attended. enc_lens : torch.Tensor The actual length of each enc_states sequence. attn : torch.Tensor The attention weight. log_probs : torch.Tensor The log-probabilities of the current step output. Returns ------- log_probs : torch.Tensor Log-probabilities of the current step output. memory : No limit The memory variables generated in this step. (ex. RNN hidden states). attn : torch.Tensor The attention weight. r)rr.)rr+r,r(r-rr`rrr_attn_weight_step,s   z!S2SBeamSearcher._attn_weight_stepcCs.|jr|||\}}t|||jd}||fS)aHThis method will block the beams that attention shift more than max_attn_shift. Arguments --------- attn : torch.Tensor The attention weight. prev_attn_peak : torch.Tensor The previous attention peak place. log_probs : torch.Tensor The log-probabilities of the current step output. Returns ------- log_probs : torch.Tensor Log-probabilities of the current step output. prev_attn_peak : torch.Tensor The previous attention peak place.  fill_value)rr+rr)rrr&r`r*rrr_max_attn_shift_stepRs z$S2SBeamSearcher._max_attn_shift_stepcCs,|jdur|j|||||j\}}||fS)aThis method call the scorers if scorer is not None. Arguments --------- inp_tokens : torch.Tensor The input tensor of the current step. scorer_memory : No limit The memory variables input for this step. (ex. RNN hidden states). attn : torch.Tensor The attention weight. log_probs : torch.Tensor The log-probabilities of the current step output. Returns ------- log_probs : torch.Tensor Log-probabilities of the current step output. scorer_memory : No limit The memory variables generated in this step. N)rscorer)rr+ scorer_memoryrr`rrr _scorer_stepms  zS2SBeamSearcher._scorer_stepcCs ||kr|j|dd|jf<|S)aThis method set the log_probs of eos to minus infinity if the step is less than min_decode_steps. Arguments --------- log_probs : torch.Tensor The log-probabilities of the current step output. step : int The current decoding step. min_decode_steps : int The minimum decoding steps. Returns ------- log_probs : torch.Tensor Log-probabilities of the current step output. N)rr")rr`r^r3rrr_set_eos_minus_inf_stepsz'S2SBeamSearcher._set_eos_minus_inf_stepcCs@|jr||}t|dd|jf||jd|dd|jf<|S)axThis method set the log_probs of eos to minus infinity if the eos log-probabilities is less than eos_threshold. Arguments --------- log_probs : torch.Tensor The log-probabilities of the current step output. Returns ------- log_probs : torch.Tensor Log-probabilities of the current step output. Nr6)rr.rr"r)rr`r*rrr_eos_threshold_steps  z#S2SBeamSearcher._eos_threshold_stepcCs|jdkr |j||d}|S)aThis method permute the memory if attn_weight is superior to 0. Arguments --------- memory : No limit The memory variables input for this step. (ex. RNN hidden states). predecessors : torch.Tensor The index of which beam the current top-K output came from in (t-1) steps. Returns ------- memory : No limit The memory variables generated in this step. (ex. RNN hidden states). r)r)r permute_mem)rr, predecessorsrrr _attn_weight_permute_memory_steps z0S2SBeamSearcher._attn_weight_permute_memory_stepcCs |jdur|jj|||d}|S)a1This method permute the scorer_memory if scorer is not None. Arguments --------- scorer_memory : No limit The memory variables input for this step. (ex. RNN hidden states). predecessors : torch.Tensor The index of which beam the current top-K output came from in (t-1) steps. candidates : torch.Tensor The index of the current top-K output. Returns ------- scorer_memory : No limit The memory variables generated in this step. N)r candidates)rpermute_scorer_mem)rr:r?rArrr_scorer_permute_memory_steps z+S2SBeamSearcher._scorer_permute_memory_stepcCs|jr tj|d|d}|S)aThis method permute the prev_attn_peak if using_max_attn_shift is True. Arguments --------- prev_attn_peak : torch.Tensor The previous attention peak place. predecessors : torch.Tensor The index of which beam the current top-K output came from in (t-1) steps. Returns ------- prev_attn_peak : torch.Tensor The previous attention peak place. rrDr)rrF index_select)rr&r?rrr#_max_attn_shift_permute_memory_steps z3S2SBeamSearcher._max_attn_shift_permute_memory_stepcCs6|j|j|jd}d}|jdur|j||}||fS)aCall reset memory for each module. Arguments --------- enc_states : torch.Tensor The encoder states to be attended. enc_lens : torch.Tensor The actual length of each enc_states sequence. Returns ------- memory : No limit The memory variables generated in this step. scorer_memory : No limit The memory variables generated in this step. rAN)r2r1r1rreset_scorer_mem)rr(r-r,r:rrr_update_reset_memorys  z$S2SBeamSearcher._update_reset_memorycCs0|||}||||}|||}|||fS)aCall permute memory for each module. It allows us to synchronize the memory with the output. Arguments --------- memory : No limit The memory variables input for this step. (ex. RNN hidden states). scorer_memory : No limit The memory variables input for this step. (ex. RNN hidden states). predecessors : torch.Tensor The index of which beam the current top-K output came from in (t-1) steps. candidates : torch.Tensor The index of the current top-K output. prev_attn_peak : torch.Tensor The previous attention peak place. Returns ------- memory : No limit The memory variables generated in this step. scorer_memory : No limit The memory variables generated in this step. prev_attn_peak : torch.Tensor The previous attention peak place. )r@rCrF)rr,r:r?rAr&rrr_update_permute_memory s  z&S2SBeamSearcher._update_permute_memorycCsvtjtj|jd|d|dgdd|_|t|jd|f|j}tjtj|j d|d|dgdd|_ |S)aThis method update sequences and log probabilities by adding the new inp_tokens. Arguments --------- log_probs : torch.Tensor The log-probabilities of the current step output. inp_tokens : torch.Tensor The input tensor of the current step. predecessors : torch.Tensor The index of which beam the current top-K output came from in (t-1) steps. candidates : torch.Tensor The index of the current top-K output. alived_hyps : AlivedHypotheses The alived hypotheses. Returns ------- alived_hyps : AlivedHypotheses The alived hypotheses. rrDr@rBrC) rFcatrEr rraranger0reshaper1r)rr`r+r?rA alived_hypsbeam_log_probsrrr_update_sequences_and_log_probs7s,  z/S2SBeamSearcher._update_sequences_and_log_probscCs|jdd|j}||}|jr||d}||jdj|jdd\}}||j|j }||j }||_|jrD|j|d|_t j ||jdd|j d ||j }|||||fS)aCompute scores and next input tokens. Arguments --------- alived_hyps : AlivedHypotheses The alived hypotheses. log_probs : torch.Tensor The log-probabilities of the current step output. step : int The current decoding step. Returns ------- scores : torch.Tensor The scores of the current step output. candidates : torch.Tensor The index of the current top-K output. predecessors : torch.Tensor The index of which beam the current top-K output came from in (t-1) steps. inp_tokens : torch.Tensor The input tensor of the current step. alived_hyps : AlivedHypotheses The alived hypotheses. r@rBrCfloor rounding_mode)rrrexpandn_outrviewr0rrr1rFdivr3 expand_as)rrMr`r^rarAr+r?rrr#_compute_scores_and_next_inp_tokensks2   z3S2SBeamSearcher._compute_scores_and_next_inp_tokensc Cs~t|jd|}|j|_|jd|_|j|j|_||_ | ||\}}t ||jdd}t ||jdd}tj |j|jd |j}tj|j|jd|j|_tj|j|jd |j}|d|jdddt|jD}t|jd|j|_t|jd|j|_||j|j\|_|_tj |j|jd} d} tj|j|j fd|jd} |} | || |||| | ||f S) aInitialize the beam search data. Arguments --------- enc_states : torch.Tensor The encoder states to be attended. wav_len : torch.Tensor The actual length of each enc_states sequence. Returns ------- alived_hyps : AlivedHypotheses The alived hypotheses. inp_tokens : torch.Tensor The input tensor of the current step. log_probs : torch.Tensor The log-probabilities of the current step output. eos_hyps_and_log_probs_scores : list Generated hypotheses (the ones that have reached eos) and log probs scores. memory : No limit The memory variables generated in this step. scorer_memory : No limit The memory variables generated in this step. attn : torch.Tensor The attention weight. prev_attn_peak : torch.Tensor The previous attention peak place. enc_states : torch.Tensor The encoder states to be attended. enc_lens : torch.Tensor The actual length of each enc_states sequence. r@r)timesrDrArzcSsg|]}gqSrrr"rrrrsz9S2SBeamSearcher.init_beam_search_data..N)rFrGrHrIr1r0rr1r9rTrHrrrKr!rLrKr3r0rr2rNr#r3r$r4r5fullr4) rr(r)r-r,r:r+reos_hyps_and_log_probs_scoresr&rr`rMrrrinit_beam_search_datasT!  z%S2SBeamSearcher.init_beam_search_datac Cs||j}tj|dd\}|jddkrS|D];}|}tj||jdd}t|||jkr0q|j |ddf} |j |ddf} || } || | | | fq|S)asThis method will update hyps and scores if inp_tokens are eos. Arguments --------- inp_tokens : torch.Tensor The current output. alived_hyps : AlivedHypotheses alived_seq : torch.Tensor alived_log_probs : torch.Tensor eos_hyps_and_log_probs_scores : list Generated hypotheses (the ones that have reached eos) and log probs scores. scores : torch.Tensor Scores at the current step. Returns ------- is_eos : torch.BoolTensor Each element represents whether the token is eos. TrirrPrQN) eqr"rFrnrHrprVrror rclonerV) rr+rMr[rais_eos eos_indicesrbatch_idhypr` final_scoresrrr$_update_hyps_and_scores_if_eos_tokens&   z4S2SBeamSearcher._update_hyps_and_scores_if_eos_tokencCslggggf\}}}}t|}tt|D] }t||\}} } ||7}|| 7}|| 7}|dd|D7}qtjjjj|ddd}tjjjj|ddd}tj|tj |j d}tj |dd |d}|d | d }|j|jdd\} } | |jd  ||j} tj|d| d } | ||jd} tj|d| d }| ||j}tj|d| d }| ||jd}| || |fS) a"This method sorts the scores and return corresponding hypothesis and log probs. Arguments --------- eos_hyps_and_log_probs_scores : list Generated hypotheses (the ones that have reached eos) and log probs scores. Returns ------- topk_hyps : torch.Tensor (batch, topk, max length of token_id sequences) This tensor stores the topk predicted hypothesis. topk_lengths : torch.Tensor (batch, topk) This tensor contains the final scores of topk hypotheses. topk_scores : torch.Tensor (batch, topk) The length of each topk sequence in the batch. topk_log_probs : torch.Tensor (batch, topk, max length of token_id sequences) The log probabilities of each hypotheses. cSrrr )rrbrrrr]rz8S2SBeamSearcher._get_topk_prediction..Tr) batch_first padding_valuerkrCrBr@rD)rorNziprFrRutilsrnn pad_sequencerqrUr1rYrUrmrr3rrrE)rr[rdrgrfrer0irhr`ra topk_scoresindices topk_hyps topk_lengthstopk_log_probsrrr_get_topk_predictionAs<    z$S2SBeamSearcher._get_topk_predictionc  Cs|||| | ||\}}}||jd} ||||\}}||| |j}||}|||||\}}| ||| \} }}}}| |||||\}}}| | ||||}| |||| }|j |td||||||||| f S)a|A search step for the next most likely tokens. Arguments --------- alived_hyps : AlivedHypotheses The alived hypotheses. inp_tokens : torch.Tensor The input tensor of the current step. log_probs : torch.Tensor The log-probabilities of the current step output. eos_hyps_and_log_probs_scores : list Generated hypotheses (the ones that have reached eos) and log probs scores. memory : No limit The memory variables input for this step. (ex. RNN hidden states). scorer_memory : No limit The memory variables input for this step. (ex. RNN hidden states). attn : torch.Tensor The attention weight. prev_attn_peak : torch.Tensor The previous attention peak place. enc_states : torch.Tensor The encoder states to be attended. enc_lens : torch.Tensor The actual length of each enc_states sequence. step : int The current decoding step. Returns ------- alived_hyps : AlivedHypotheses The alived hypotheses. inp_tokens : torch.Tensor The input tensor of the current step. log_probs : torch.Tensor The log-probabilities of the current step output. eos_hyps_and_log_probs_scores : list Generated hypotheses (the ones that have reached eos) and log probs scores. memory : No limit The memory variables generated in this step. scorer_memory : No limit The memory variables generated in this step. attn : torch.Tensor The attention weight. prev_attn_peak : torch.Tensor The previous attention peak place. scores : torch.Tensor The scores of the current step output. rBr/)r5r^rLr0r8r<r3r=r;rXrIrOrdr masked_fill_rU)rrMr+r`r[r,r:rr&r(r-r^log_probs_clonerarAr?r_rrr search_step}sV@       zS2SBeamSearcher.search_stepcCs<||stj|j|jd|j}||||||S)a,Fill the alived_hyps that have not reached eos with eos. Arguments --------- alived_hyps : AlivedHypotheses The alived hypotheses. eos_hyps_and_log_probs_scores : list Generated hypotheses (the ones that have reached eos) and log probs scores. scores : torch.Tensor The scores of the current step output. Returns ------- eos_hyps_and_log_probs_scores : list Generated hypotheses (the ones that have reached eos) and log probs scores. rA) r%rFrr1r1rKr"rLrd)rrMr[rar+rrr _fill_alived_hyps_with_eos_tokens z0S2SBeamSearcher._fill_alived_hyps_with_eos_tokencCs|||\ }}}}}}} } }} t|jD](} ||rn |||||||| | || | \ }}}}}}} } } ||r=nq|||| }||\}}}}|jrW||||fS|dddddf}|dddf}|dddf}|dddddf}t ||}||||fS)alApplies beamsearch and returns the predicted tokens. Arguments --------- enc_states : torch.Tensor The encoder states to be attended. wav_len : torch.Tensor The actual length of each enc_states sequence. Returns ------- hyps : list The predicted tokens. best_lens : torch.Tensor The length of each predicted tokens. best_scores : torch.Tensor The scores of each predicted tokens. best_log_probs : torch.Tensor The log probabilities of each predicted tokens. Nr) r\rNr4r%rtr=rurqrr)rr(r)rMr+r`r[r,r:rr&r-r^ra finals_hyps_and_log_probs_scoresrnrorlrp best_hyps best_lens best_scoresbest_log_probsrhrrrr*sv        zS2SBeamSearcher.forwardcCr:r;rrrMrrrr=tr>z$S2SBeamSearcher._check_end_conditioncCr%)aThis method permutes the seq2seq model memory to synchronize the memory index with the current output. Arguments --------- memory : No limit The memory variable to be permuted. index : torch.Tensor The index of the previous path. Returns ------- The variable of the memory being permuted. r&rr,rrrrr>|szS2SBeamSearcher.permute_mem) NFr@TrTFrr )rrrrr r%r+r.r4r5r8r;r<r=r@rCrFrHrIrOrXr\rdrqrtrur*r=r>rrrrrr~sH/6&*4?h/<z ]rc:eZdZdZd fdd ZddZddZd d ZZS) S2SRNNBeamSearcheraf This class implements the beam search decoding for AttentionalRNNDecoder (speechbrain/nnet/RNN.py). See also S2SBaseSearcher(), S2SBeamSearcher(). Arguments --------- embedding : torch.nn.Module An embedding layer. decoder : torch.nn.Module Attentional RNN decoder. linear : torch.nn.Module A linear output layer. temperature : float Temperature factor applied to softmax. It changes the probability distribution, being softer when T>1 and sharper with T<1. **kwargs see S2SBeamSearcher, arguments are directly passed. Example ------- >>> import speechbrain as sb >>> vocab_size = 5 >>> emb = torch.nn.Embedding(vocab_size, 3) >>> dec = sb.nnet.RNN.AttentionalRNNDecoder( ... "gru", "content", 3, 3, 1, enc_dim=7, input_size=3 ... ) >>> lin = sb.nnet.linear.Linear(n_neurons=vocab_size, input_size=3) >>> coverage_scorer = sb.decoders.scorer.CoverageScorer(vocab_size) >>> scorer = sb.decoders.scorer.ScorerBuilder( ... full_scorers = [coverage_scorer], ... partial_scorers = [], ... weights= dict(coverage=1.5) ... ) >>> searcher = S2SRNNBeamSearcher( ... embedding=emb, ... decoder=dec, ... linear=lin, ... bos_index=4, ... eos_index=4, ... min_decode_ratio=0, ... max_decode_ratio=1, ... beam_size=2, ... scorer=scorer, ... ) >>> batch_size = 2 >>> enc = torch.rand([batch_size, 6, 7]) >>> wav_len = torch.ones([batch_size]) >>> hyps, _, _, _ = searcher(enc, wav_len) r c s>tjdi|||_||_||_tjjdd|_||_ dSr) r r rrr6rFrRrrrOrrrrr s  zS2SRNNBeamSearcher.__init__cCr)-Needed to reset the memory during beamsearch.NrArrrrrr2s zS2SRNNBeamSearcher.reset_memc Cst7|\}}||}|j|||||\}}}} ||||j} |jjdkr4tj | dd} Wdn1s>wY| ||f| fS)rmultiheadlocationr@rCN) rFrwrrr.rr6rO attn_typemean) rr+r,r(r-rrrrr7r`rrrr.s      zS2SRNNBeamSearcher.forward_stepcCs|\}}t|tr"tj|dd|d}tj|dd|d}||f}ntj|d|d}tj|d|d}|jjdkrFtj|jjjd|d|jj_||fS)%Memory permutation during beamsearch.rr@rDlocation)rrrFrErrr prev_attn)rr,rrrhs_0hs_1rrrr>s     zS2SRNNBeamSearcher.permute_memr ) rrrrr r2r.r>rrrrrr~s 3r~cr}) S2STransformerBeamSearcheraThis class implements the beam search decoding for Transformer. See also S2SBaseSearcher(), S2SBeamSearcher(). Arguments --------- modules : list with the following one: model : torch.nn.Module A Transformer model. seq_lin : torch.nn.Module A linear output layer. temperature : float Temperature factor applied to softmax. It changes the probability distribution, being softer when T>1 and sharper with T<1. **kwargs Arguments to pass to S2SBeamSearcher Example ------- >>> from speechbrain.nnet.linear import Linear >>> from speechbrain.lobes.models.transformer.TransformerASR import TransformerASR >>> from speechbrain.decoders import S2STransformerBeamSearcher >>> batch_size=8 >>> n_channels=6 >>> input_size=40 >>> d_model=128 >>> tgt_vocab=140 >>> src = torch.rand([batch_size, n_channels, input_size]) >>> tgt = torch.randint(0, tgt_vocab, [batch_size, n_channels]) >>> net = TransformerASR( ... tgt_vocab, input_size, d_model, 8, 1, 1, 1024, activation=torch.nn.GELU ... ) >>> ctc_lin = Linear(input_shape=(1, 40, d_model), n_neurons=tgt_vocab) >>> lin = Linear(input_shape=(1, 40, d_model), n_neurons=tgt_vocab) >>> searcher = S2STransformerBeamSearcher( ... modules=[net, lin], ... bos_index=1, ... eos_index=2, ... min_decode_ratio=0.0, ... max_decode_ratio=1.0, ... using_eos_threshold=False, ... beam_size=7, ... temperature=1.15, ... ) >>> enc, dec = net.forward(src, tgt) >>> hyps, _, _, _ = searcher(enc, torch.ones(batch_size)) r c r{r|r}rrrrr %rz#S2STransformerBeamSearcher.__init__cCr:)rNrr/rrrr2.rz$S2STransformerBeamSearcher.reset_memcCstj|d|d}|S)rrrD)rFrEr|rrrr>2sz&S2STransformerBeamSearcher.permute_memcCsPt||}|j|||\}}||||j}|dddddf||fSrNrB)rr~rrr6rOrr+r,r(r-rur prob_distrrrr.7s z'S2STransformerBeamSearcher.forward_stepr) rrrrr r2r>r.rrrrrrs 0 rcseZdZdZ       dfdd Zdd Zd d Zd d ZeddZ ddZ ddZ ddZ ddZ ddZddZddZZS)S2SWhisperBeamSearcheraThis class implements the beam search decoding for Whisper neural nets made by OpenAI in https://cdn.openai.com/papers/whisper.pdf. The beam search is stateful, meaning that some variables are stored in the searcher. If you want to reuse the searcher in different contexts, you should make sure that the variables are updated accordingly. Arguments --------- module : list with the following one: model : torch.nn.Module A whisper model. It should have a decode() method. temperature: float The temperature to use during decoding. use_kv_cache: bool (default: True) Whether to use key-value cache. suppress_blank: bool (default: True) This will suppress blank outputs. suppress_tokens: str or list (default: "-1") list of tokens ids (or comma-separated token ids) to suppress "-1" will suppress a set of symbols as defined in `model.non_speech_tokens()` sample_len: int (default: None) Maximum number of tokens to sample. prefix: str or list (default: None) Prefix to add to the input tokens. See: https://github.com/openai/whisper/discussions/117#discussioncomment-3727051 prompt: str or list (default: None) Prompt to add to the input tokens. See: https://github.com/openai/whisper/discussions/117#discussioncomment-3727051 **kwargs see S2SBeamSearcher, arguments are directly passed. r TrNc  stjd|dj|djd| |d|_||_||_d|_||_||_ ||_ ||_ |jjj j j|_|p:|jd|_||_t|j|_|jj|_|jd|_d|_d|_dS)NrrrrBrr) rmodulerOrrrrrrrrrrr cs,       zS2SWhisperBeamSearcher.__init__cCrrrrrrrrrz&S2SWhisperBeamSearcher.set_lang_tokenscCrrrrrrrrrzS2SWhisperBeamSearcher.set_taskcCrrrrrrrrrz!S2SWhisperBeamSearcher.set_promptcCr) rcSrrrrrrrrrzAS2SWhisperBeamSearcher.get_tokens_to_suppress..rrBcSrrrrrrrrrNrrrrrrrrrz-S2SWhisperBeamSearcher.get_tokens_to_suppresscCrrrrrrrrrz*S2SWhisperBeamSearcher._get_initial_tokenscCrrrrrrrr2s z S2SWhisperBeamSearcher.reset_memcCs*tj|d|d}|jr||j||_|S)zPermutes the memory.rrD)rFrEr_reorder_cacherr|rrrr>sz"S2SWhisperBeamSearcher.permute_memcs.d}|D]}|tfdd|Df7}q|S)aReorder the key-value cache. Arguments --------- past_key_values : tuple The key-value cache. beam_idx : torch.Tensor The index of the previous path. Returns ------- The reordered key-value cache. rc3s|] }|dVqdS)rN)rE)r past_statebeam_idxrr s  z8S2SWhisperBeamSearcher._reorder_cache..)r)rrrreordered_past layer_pastrrrrs z%S2SWhisperBeamSearcher._reorder_cachecCs|jjjjjjdS) set the number of output tokens.r)r~r embed_tokensweightrHrrrrr9sz S2SWhisperBeamSearcher.set_n_outc Cs4t||}|jj|||jd\}}}|jd|jkr;|dd|j|jjf j dd} | dd|jj f |_ |dddf}|jrI||_|jri|jd|jkritj |dd|jjjddd|jgf<|jr|jjjdurw|j} n|jj} tj |ddt| f<tjjj| dd|j} | ||fSr)rr~rrrHrrrrrUrrrrrrrFrWrrr"rrrrrrRrSrTrO) rr+r,r(r-rrErrrrr`rrrr.sL   z#S2SWhisperBeamSearcher.forward_stepcCs|jjd|j|jkSr)r rHrrr{rrrr=@s  z+S2SWhisperBeamSearcher._check_end_condition)r TTrNNN)rrrrr rrrrrrr2r>rr9r.r=rrrrrr?s,&' !-rcr)S2SHFTextBasedBeamSearcheraThis class implements the beam search decoding for the text-based HF seq2seq models, such as mBART or NLLB. It is NOT significantly different from S2STransformerBeamSearcher. This is why it inherits S2STransformerBeamSearcher. The main difference might arise when one wishes to use directly the lm_head of the text-based HF model rather than making a new projection layer (self.fc = None). Arguments --------- modules : list with the following one: model : torch.nn.Module A Transformer model. seq_lin : torch.nn.Module A linear output layer. Normally set to None for this usecase. vocab_size : int The dimension of the lm_head. **kwargs Arguments to pass to S2SBeamSearcher c stj|fi|||_dSr )r r vocab_size)rrrrrrrr _s z#S2SHFTextBasedBeamSearcher.__init__cCs^t||}|j|||\}}|jdur||}|||j}|dddddf||fSr)rr~rr6rrOrrrrr.cs   z'S2SHFTextBasedBeamSearcher.forward_stepcCs|jS)r)rrrrrr9lsz$S2SHFTextBasedBeamSearcher.set_n_out)rrrrr r.r9rrrrrrHs   r)r functoolsrrFtorch.distributionsrspeechbrain.decoders.utilsrrrspeechbrain.utils.data_utilsrrRModuler r r?ryrrrr~rrrrrrrs:   x &XGeK