o %ݫi@sdZddlZddlZddlZddlmZGdddZGdddeZ Gdd d eZ Gd d d eZ Gd d d eZ GdddeZ GdddeZGdddZGdddeZGdddeZGdddeZGdddeZGdddZdS)zg Token scorer abstraction and specifications. Authors: * Adel Moumen 2022, 2023 * Sung-Lin Yeh 2021 N)CTCPrefixScorec@s(eZdZdZddZddZddZdS) BaseScorerInterfacea A scorer abstraction to be inherited by other scoring approaches for beam search. A scorer is a module that scores tokens in vocabulary based on the current timestep input and the previous scorer states. It can be used to score on full vocabulary set (i.e., full scorers) or a pruned set of tokens (i.e. partial scorers) to prevent computation overhead. In the latter case, the partial scorers will be called after the full scorers. It will only scores the top-k candidates (i.e., pruned set of tokens) extracted from the full scorers. The top-k candidates are extracted based on the beam size and the scorer_beam_scale such that the number of candidates is int(beam_size * scorer_beam_scale). It can be very useful when the full scorers are computationally expensive (e.g., KenLM scorer). Inherit this class to implement your own scorer compatible with speechbrain.decoders.seq2seq.S2SBeamSearcher(). See: - speechbrain.decoders.scorer.CTCPrefixScorer - speechbrain.decoders.scorer.RNNLMScorer - speechbrain.decoders.scorer.TransformerLMScorer - speechbrain.decoders.scorer.KenLMScorer - speechbrain.decoders.scorer.CoverageScorer - speechbrain.decoders.scorer.LengthScorer cCt)aThis method scores the new beams based on the information of the current timestep. A score is a tensor of shape (batch_size x beam_size, vocab_size). It is the log probability of the next token given the current timestep input and the previous scorer states. It can be used to score on pruned top-k candidates to prevent computation overhead, or on full vocabulary set when candidates is None. Arguments --------- inp_tokens : torch.Tensor The input tensor of the current timestep. memory : No limit The scorer states for this timestep. candidates : torch.Tensor (batch_size x beam_size, scorer_beam_size). The top-k candidates to be scored after the full scorers. If None, scorers will score on full vocabulary set. attn : torch.Tensor The attention weight to be used in CoverageScorer or CTCScorer. Returns ------- torch.Tensor (batch_size x beam_size, vocab_size), Scores for the next tokens. memory : No limit The memory variables input for this timestep. NotImplementedErrorself inp_tokensmemory candidatesattnr O/home/ubuntu/.local/lib/python3.10/site-packages/speechbrain/decoders/scorer.pyscore,s zBaseScorerInterface.scorecCdS)asThis method permutes the scorer memory to synchronize the memory index with the current output and perform batched beam search. Arguments --------- memory : No limit The memory variables input for this timestep. index : torch.Tensor (batch_size, beam_size). The index of the previous path. Nr rr indexr r r permute_memO zBaseScorerInterface.permute_memcCr)ayThis method should implement the resetting of memory variables for the scorer. Arguments --------- x : torch.Tensor The precomputed encoder states to be used when decoding. (ex. the encoded speech representation to be attended). enc_lens : torch.Tensor The speechbrain-style relative length. Nr rxenc_lensr r r reset_mem]rzBaseScorerInterface.reset_memN)__name__ __module__ __qualname____doc__rrrr r r rrs # rc@2eZdZdZd ddZddZddZd d Zd S) CTCScoreraA wrapper of CTCPrefixScore based on the BaseScorerInterface. This Scorer is used to provides the CTC label-synchronous scores of the next input tokens. The implementation is based on https://www.merl.com/publications/docs/TR2017-190.pdf. See: - speechbrain.decoders.scorer.CTCPrefixScore Arguments --------- ctc_fc : torch.nn.Module A output linear layer for ctc. blank_index : int The index of the blank token. eos_index : int The index of the end-of-sequence (eos) token. ctc_window_size : int Compute the ctc scores over the time frames using windowing based on attention peaks. If 0, no windowing applied. (default: 0) Example ------- >>> import torch >>> from speechbrain.nnet.linear import Linear >>> from speechbrain.lobes.models.transformer.TransformerASR import TransformerASR >>> from speechbrain.decoders import S2STransformerBeamSearcher, CTCScorer, ScorerBuilder >>> batch_size=8 >>> n_channels=6 >>> input_size=40 >>> d_model=128 >>> tgt_vocab=140 >>> src = torch.rand([batch_size, n_channels, input_size]) >>> tgt = torch.randint(0, tgt_vocab, [batch_size, n_channels]) >>> net = TransformerASR( ... tgt_vocab, input_size, d_model, 8, 1, 1, 1024, activation=torch.nn.GELU ... ) >>> ctc_lin = Linear(input_shape=(1, 40, d_model), n_neurons=tgt_vocab) >>> lin = Linear(input_shape=(1, 40, d_model), n_neurons=tgt_vocab) >>> eos_index = 2 >>> ctc_scorer = CTCScorer( ... ctc_fc=ctc_lin, ... blank_index=0, ... eos_index=eos_index, ... ) >>> scorer = ScorerBuilder( ... full_scorers=[ctc_scorer], ... weights={'ctc': 1.0} ... ) >>> searcher = S2STransformerBeamSearcher( ... modules=[net, lin], ... bos_index=1, ... eos_index=eos_index, ... min_decode_ratio=0.0, ... max_decode_ratio=1.0, ... using_eos_threshold=False, ... beam_size=7, ... temperature=1.15, ... scorer=scorer ... ) >>> enc, dec = net.forward(src, tgt) >>> hyps, _, _, _ = searcher(enc, torch.ones(batch_size)) rcCs.||_||_||_||_tjjjdd|_dSNT) apply_log) ctc_fc blank_index eos_indexctc_window_sizesbnnet activationsSoftmaxsoftmax)rr!r"r#r$r r r__init__s zCTCScorer.__init__cCs|j||||\}}||fS)a This method scores the new beams based on the CTC scores computed over the time frames. See: - speechbrain.decoders.scorer.CTCPrefixScore Arguments --------- inp_tokens : torch.Tensor The input tensor of the current timestep. memory : No limit The scorer states for this timestep. candidates : torch.Tensor (batch_size x beam_size, scorer_beam_size). The top-k candidates to be scored after the full scorers. If None, scorers will score on full vocabulary set. attn : torch.Tensor The attention weight to be used in CoverageScorer or CTCScorer. Returns ------- scores : torch.Tensor memory ) ctc_score forward_step)rr r r r scoresr r rrszCTCScorer.scorecCs|j||\}}||fS)aThis method permutes the scorer memory to synchronize the memory index with the current output and perform batched CTC beam search. Arguments --------- memory : No limit The memory variables input for this timestep. index : torch.Tensor (batch_size, beam_size). The index of the previous path. Returns ------- r, psi : see ``ctc_score.permute_mem`` )r+r)rr rrpsir r rrszCTCScorer.permute_memcCs0||}||}t|||j|j|j|_dS)avThis method implement the resetting of memory variables for the CTC scorer. Arguments --------- x : torch.Tensor The precomputed encoder states to be used when decoding. (ex. the encoded speech representation to be attended). enc_lens : torch.Tensor The speechbrain-style relative length. N)r!r)rr"r#r$r+)rrrlogitsr r rrs  zCTCScorer.reset_memNrrrrrr*rrrr r r rrls  @ rc@r) RNNLMScoreraJ A wrapper of RNNLM based on BaseScorerInterface. The RNNLMScorer is used to provide the RNNLM scores of the next input tokens based on the current timestep input and the previous scorer states. Arguments --------- language_model : torch.nn.Module A RNN-based language model. temperature : float Temperature factor applied to softmax. It changes the probability distribution, being softer when T>1 and sharper with T<1. (default: 1.0) Example ------- >>> from speechbrain.nnet.linear import Linear >>> from speechbrain.lobes.models.RNNLM import RNNLM >>> from speechbrain.nnet.RNN import AttentionalRNNDecoder >>> from speechbrain.decoders import S2SRNNBeamSearcher, RNNLMScorer, ScorerBuilder >>> input_size=17 >>> vocab_size=11 >>> emb = torch.nn.Embedding( ... embedding_dim=input_size, ... num_embeddings=vocab_size, ... ) >>> d_model=7 >>> dec = AttentionalRNNDecoder( ... rnn_type="gru", ... attn_type="content", ... hidden_size=3, ... attn_dim=3, ... num_layers=1, ... enc_dim=d_model, ... input_size=input_size, ... ) >>> n_channels=3 >>> seq_lin = Linear(input_shape=[d_model, n_channels], n_neurons=vocab_size) >>> lm_weight = 0.4 >>> lm_model = RNNLM( ... embedding_dim=d_model, ... output_neurons=vocab_size, ... dropout=0.0, ... rnn_neurons=128, ... dnn_neurons=64, ... return_hidden=True, ... ) >>> rnnlm_scorer = RNNLMScorer( ... language_model=lm_model, ... temperature=1.25, ... ) >>> scorer = ScorerBuilder( ... full_scorers=[rnnlm_scorer], ... weights={'rnnlm': lm_weight} ... ) >>> beam_size=5 >>> searcher = S2SRNNBeamSearcher( ... embedding=emb, ... decoder=dec, ... linear=seq_lin, ... bos_index=1, ... eos_index=2, ... min_decode_ratio=0.0, ... max_decode_ratio=1.0, ... topk=2, ... using_eos_threshold=False, ... beam_size=beam_size, ... temperature=1.25, ... scorer=scorer ... ) >>> batch_size=2 >>> enc = torch.rand([batch_size, n_channels, d_model]) >>> wav_len = torch.ones([batch_size]) >>> hyps, _, _, _ = searcher(enc, wav_len) ?cC,||_|j||_tjjjdd|_dSrlmeval temperaturer%r&r'r(r)rlanguage_modelr9r r rr*D zRNNLMScorer.__init__cCsXt|j||d\}}|||j}Wd||fS1s#wY||fS)aThis method scores the new beams based on the RNNLM scores computed over the previous tokens. Arguments --------- inp_tokens : torch.Tensor The input tensor of the current timestep. memory : No limit The scorer states for this timestep. candidates : torch.Tensor (batch_size x beam_size, scorer_beam_size). The top-k candidates to be scored after the full scorers. If None, scorers will score on full vocabulary set. attn : torch.Tensor The attention weight to be used in CoverageScorer or CTCScorer. Returns ------- log_probs : torch.Tensor Output probabilities. hs : torch.Tensor LM hidden states. )hxN)torchno_gradr7r)r9)rr r r r r0hs log_probsr r rrJs  zRNNLMScorer.scorecCsRt|trtj|dd|d}tj|dd|d}||f}|Stj|d|d}|S)This method permutes the scorer memory to synchronize the memory index with the current output and perform batched beam search. Arguments --------- memory : No limit The memory variables input for this timestep. index : torch.Tensor (batch_size, beam_size). The index of the previous path. Returns ------- memory rdimr) isinstancetupler> index_select)rr rmemory_0memory_1r r rrgs zRNNLMScorer.permute_memcCrxThis method implement the resetting of memory variables for the RNNLM scorer. Arguments --------- x : torch.Tensor The precomputed encoder states to be used when decoding. (ex. the encoded speech representation to be attended). enc_lens : torch.Tensor The speechbrain-style relative length. Nr rr r rrrzRNNLMScorer.reset_memNr4r2r r r rr3s  K r3c@r) TransformerLMScorera A wrapper of TransformerLM based on BaseScorerInterface. The TransformerLMScorer is used to provide the TransformerLM scores of the next input tokens based on the current timestep input and the previous scorer states. Arguments --------- language_model : torch.nn.Module A Transformer-based language model. temperature : float Temperature factor applied to softmax. It changes the probability distribution, being softer when T>1 and sharper with T<1. (default: 1.0) Example ------- >>> from speechbrain.nnet.linear import Linear >>> from speechbrain.lobes.models.transformer.TransformerASR import TransformerASR >>> from speechbrain.lobes.models.transformer.TransformerLM import TransformerLM >>> from speechbrain.decoders import S2STransformerBeamSearcher, TransformerLMScorer, CTCScorer, ScorerBuilder >>> input_size=17 >>> vocab_size=11 >>> d_model=128 >>> net = TransformerASR( ... tgt_vocab=vocab_size, ... input_size=input_size, ... d_model=d_model, ... nhead=8, ... num_encoder_layers=1, ... num_decoder_layers=1, ... d_ffn=256, ... activation=torch.nn.GELU ... ) >>> lm_model = TransformerLM( ... vocab=vocab_size, ... d_model=d_model, ... nhead=8, ... num_encoder_layers=1, ... num_decoder_layers=0, ... d_ffn=256, ... activation=torch.nn.GELU, ... ) >>> n_channels=6 >>> ctc_lin = Linear(input_size=d_model, n_neurons=vocab_size) >>> seq_lin = Linear(input_size=d_model, n_neurons=vocab_size) >>> eos_index = 2 >>> ctc_scorer = CTCScorer( ... ctc_fc=ctc_lin, ... blank_index=0, ... eos_index=eos_index, ... ) >>> transformerlm_scorer = TransformerLMScorer( ... language_model=lm_model, ... temperature=1.15, ... ) >>> ctc_weight_decode=0.4 >>> lm_weight=0.6 >>> scorer = ScorerBuilder( ... full_scorers=[transformerlm_scorer, ctc_scorer], ... weights={'transformerlm': lm_weight, 'ctc': ctc_weight_decode} ... ) >>> beam_size=5 >>> searcher = S2STransformerBeamSearcher( ... modules=[net, seq_lin], ... bos_index=1, ... eos_index=eos_index, ... min_decode_ratio=0.0, ... max_decode_ratio=1.0, ... using_eos_threshold=False, ... beam_size=beam_size, ... temperature=1.15, ... scorer=scorer ... ) >>> batch_size=2 >>> wav_len = torch.ones([batch_size]) >>> src = torch.rand([batch_size, n_channels, input_size]) >>> tgt = torch.randint(0, vocab_size, [batch_size, n_channels]) >>> enc, dec = net.forward(src, tgt) >>> hyps, _, _, _ = searcher(enc, wav_len) r4cCr5rr6r:r r rr*r<zTransformerLMScorer.__init__cCst@|durtj|dd|jd}tj||dgdd}t|j j s0|j |j||}| ||j }Wdn1sGwY|dddddf|fS)aThis method scores the new beams based on the TransformerLM scores computed over the previous tokens. Arguments --------- inp_tokens : torch.Tensor The input tensor of the current timestep. memory : No limit The scorer states for this timestep. candidates : torch.Tensor (batch_size x beam_size, scorer_beam_size). The top-k candidates to be scored after the full scorers. If None, scorers will score on full vocabulary set. attn : torch.Tensor The attention weight to be used in CoverageScorer or CTCScorer. Returns ------- log_probs : torch.Tensor memory NrdevicerCrE)r>r?emptysizerPcat unsqueezenextr7 parametersis_cudator)r9)rr r r r r0rAr r rrs   zTransformerLMScorer.scorecCtj|d|d}|S)rBrrDr>rHrr r rr szTransformerLMScorer.permute_memcCrrKr rr r rrrzTransformerLMScorer.reset_memNrMr2r r r rrNs  Q# rNc@s0eZdZdZddZddZddZdd Zd S) KenLMScoreraI KenLM N-gram scorer. This scorer is based on KenLM, which is a fast and efficient N-gram language model toolkit. It is used to provide the n-gram scores of the next input tokens. This scorer is dependent on the KenLM package. It can be installed with the following command: > pip install https://github.com/kpu/kenlm/archive/master.zip Note: The KenLM scorer is computationally expensive. It is recommended to use it as a partial scorer to score on the top-k candidates instead of the full vocabulary set. Arguments --------- lm_path : str The path of ngram model. vocab_size: int The total number of tokens. token_list : list The tokens set. Example ------- # >>> from speechbrain.nnet.linear import Linear # >>> from speechbrain.nnet.RNN import AttentionalRNNDecoder # >>> from speechbrain.decoders import S2SRNNBeamSearcher, KenLMScorer, ScorerBuilder # >>> input_size=17 # >>> vocab_size=11 # >>> lm_path='path/to/kenlm_model.arpa' # or .bin # >>> token_list=['', '', '', 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i'] # >>> emb = torch.nn.Embedding( # ... embedding_dim=input_size, # ... num_embeddings=vocab_size, # ... ) # >>> d_model=7 # >>> dec = AttentionalRNNDecoder( # ... rnn_type="gru", # ... attn_type="content", # ... hidden_size=3, # ... attn_dim=3, # ... num_layers=1, # ... enc_dim=d_model, # ... input_size=input_size, # ... ) # >>> n_channels=3 # >>> seq_lin = Linear(input_shape=[d_model, n_channels], n_neurons=vocab_size) # >>> kenlm_weight = 0.4 # >>> kenlm_model = KenLMScorer( # ... lm_path=lm_path, # ... vocab_size=vocab_size, # ... token_list=token_list, # ... ) # >>> scorer = ScorerBuilder( # ... full_scorers=[kenlm_model], # ... weights={'kenlm': kenlm_weight} # ... ) # >>> beam_size=5 # >>> searcher = S2SRNNBeamSearcher( # ... embedding=emb, # ... decoder=dec, # ... linear=seq_lin, # ... bos_index=1, # ... eos_index=2, # ... min_decode_ratio=0.0, # ... max_decode_ratio=1.0, # ... topk=2, # ... using_eos_threshold=False, # ... beam_size=beam_size, # ... temperature=1.25, # ... scorer=scorer # ... ) # >>> batch_size=2 # >>> enc = torch.rand([batch_size, n_channels, d_model]) # >>> wav_len = torch.ones([batch_size]) # >>> hyps, _, _, _ = searcher(enc, wav_len) cCsxz ddl}||_Wn tyd}t|w|j||_||_t|j|_d|_t ||kr7d}t |||_ dS)NrzCouldn't import KenLM It is an optional dependency; it is not installed with SpeechBrain by default. Install it with: > pip install https://github.com/kpu/kenlm/archive/master.zip g@xz:The size of the token_list and vocab_size are not matched.) kenlm ImportErrorModelr7 vocab_sizenparangefull_candidates minus_inflen ValueErrorid2char)rlm_pathra token_listr^MSGr r rr*{s    zKenLMScorer.__init__cCsF|d}dttj}|dur$|j}t|g|}t|}n|\}}|dur2|jg|}t||j f|j } tj ||j ft d} t||j fd} t |D]<} || dkr^qU|| } || D]*}|j|}|j}||j| ||}|| | |f<|| | |f<d| | |f<qfqUt| |j} | | | ffS)aThis method scores the new beams based on the n-gram scores. Arguments --------- inp_tokens : torch.Tensor The input tensor of the current timestep. memory : No limit The scorer states for this timestep. candidates : torch.Tensor (batch_size x beam_size, scorer_beam_size). The top-k candidates to be scored after the full scorers. If None, scorers will score on full vocabulary set. attn : torch.Tensor The attention weight to be used in CoverageScorer or CTCScorer. Returns ------- scores : torch.Tensor (new_memory, new_scoring_table) : tuple rr4N)dtyperQrC)rTrblog10er^Statearrayonesrdrarezerosobjectrangerhitemr7 BaseScorer> from_numpyfloatrZrP)rr r r r n_bhscalestate scoring_tabler- new_memorynew_scoring_tablei parent_statetoken_idchar out_staterr r rrs4           zKenLMScorer.scorecCs~|\}}|}|jd}|j|}|tt|d|j|j}|d}|d}||}|d}||}||fS)aThis method permutes the scorer memory to synchronize the memory index with the current output and perform batched beam search. Arguments --------- memory : No limit The memory variables input for this timestep. index : torch.Tensor (batch_size, beam_size). The index of the previous path. Returns ------- state : torch.Tensor scoring_table : torch.Tensor rCrQ) cpunumpyshape batch_indexrb broadcast_to expand_dimsrareshape)rr rr{r| beam_size beam_offset hyp_indexr r rrs       zKenLMScorer.permute_memcCs,|j}|j|t|d|_dS)axThis method implement the resetting of memory variables for the KenLM scorer. Arguments --------- x : torch.Tensor The precomputed encoder states to be used when decoding. (ex. the encoded speech representation to be attended). enc_lens : torch.Tensor The speechbrain-style relative length. rN)r^ror7NullContextWriterbrcrTr)rrrr{r r rrs zKenLMScorer.reset_memNr2r r r rr]+s O7 $r]c@r) CoverageScorera A coverage penalty scorer to prevent looping of hyps, where ```coverage``` is the cumulative attention probability vector. Reference: https://arxiv.org/pdf/1612.02695.pdf, https://arxiv.org/pdf/1808.10792.pdf Arguments --------- vocab_size: int The total number of tokens. threshold: float The penalty increases when the coverage of a frame is more than given threshold. (default: 0.5) Example ------- >>> from speechbrain.nnet.linear import Linear >>> from speechbrain.lobes.models.RNNLM import RNNLM >>> from speechbrain.nnet.RNN import AttentionalRNNDecoder >>> from speechbrain.decoders import S2SRNNBeamSearcher, RNNLMScorer, CoverageScorer, ScorerBuilder >>> input_size=17 >>> vocab_size=11 >>> emb = torch.nn.Embedding( ... num_embeddings=vocab_size, ... embedding_dim=input_size ... ) >>> d_model=7 >>> dec = AttentionalRNNDecoder( ... rnn_type="gru", ... attn_type="content", ... hidden_size=3, ... attn_dim=3, ... num_layers=1, ... enc_dim=d_model, ... input_size=input_size, ... ) >>> n_channels=3 >>> seq_lin = Linear(input_shape=[d_model, n_channels], n_neurons=vocab_size) >>> lm_weight = 0.4 >>> coverage_penalty = 1.0 >>> lm_model = RNNLM( ... embedding_dim=d_model, ... output_neurons=vocab_size, ... dropout=0.0, ... rnn_neurons=128, ... dnn_neurons=64, ... return_hidden=True, ... ) >>> rnnlm_scorer = RNNLMScorer( ... language_model=lm_model, ... temperature=1.25, ... ) >>> coverage_scorer = CoverageScorer(vocab_size=vocab_size) >>> scorer = ScorerBuilder( ... full_scorers=[rnnlm_scorer, coverage_scorer], ... weights={'rnnlm': lm_weight, 'coverage': coverage_penalty} ... ) >>> beam_size=5 >>> searcher = S2SRNNBeamSearcher( ... embedding=emb, ... decoder=dec, ... linear=seq_lin, ... bos_index=1, ... eos_index=2, ... min_decode_ratio=0.0, ... max_decode_ratio=1.0, ... topk=2, ... using_eos_threshold=False, ... beam_size=beam_size, ... temperature=1.25, ... scorer=scorer ... ) >>> batch_size=2 >>> enc = torch.rand([batch_size, n_channels, d_model]) >>> wav_len = torch.ones([batch_size]) >>> hyps, _, _, _ = searcher(enc, wav_len) ?cCs||_||_d|_dS)Nr)ra threshold time_step)rrarr r rr*Js zCoverageScorer.__init__cCs|d}|jd7_|durtj||jd}t|dkr(tj|dd}n||}t|| |j d}||d|j }| | d d|j}d||j|fS)aThis method scores the new beams based on the Coverage scorer. Arguments --------- inp_tokens : torch.Tensor The input tensor of the current timestep. coverage : No limit The scorer states for this timestep. candidates : torch.Tensor (batch_size x beam_size, scorer_beam_size). The top-k candidates to be scored after the full scorers. If None, scorers will score on full vocabulary set. attn : torch.Tensor The attention weight to be used in CoverageScorer or CTCScorer. Returns ------- score : torch.Tensor coverage rrCNrOrRrQ)rTrr> zeros_likerPrfsummaxclonefill_rviewrVexpandra)rr coverager r rypenaltyr r rrPs zCoverageScorer.scorecCr[)aThis method permutes the scorer memory to synchronize the memory index with the current output and perform batched beam search. Arguments --------- coverage : No limit The memory variables input for this timestep. index : torch.Tensor (batch_size, beam_size). The index of the previous path. Returns ------- coverage rrDr\)rrrr r rr{szCoverageScorer.permute_memcCs d|_dS)rLrN)rrr r rrs zCoverageScorer.reset_memN)rr2r r r rrs  M+ rc@s eZdZdZddZddZdS) LengthScorera A length rewarding scorer. The LengthScorer is used to provide the length rewarding scores. It is used to prevent the beam search from favoring short hypotheses. Note: length_normalization is not compatible with this scorer. Make sure to set is to False when using LengthScorer. Arguments --------- vocab_size: int The total number of tokens. Example ------- >>> from speechbrain.nnet.linear import Linear >>> from speechbrain.lobes.models.RNNLM import RNNLM >>> from speechbrain.nnet.RNN import AttentionalRNNDecoder >>> from speechbrain.decoders import S2SRNNBeamSearcher, RNNLMScorer, CoverageScorer, ScorerBuilder >>> input_size=17 >>> vocab_size=11 >>> emb = torch.nn.Embedding( ... num_embeddings=vocab_size, ... embedding_dim=input_size ... ) >>> d_model=7 >>> dec = AttentionalRNNDecoder( ... rnn_type="gru", ... attn_type="content", ... hidden_size=3, ... attn_dim=3, ... num_layers=1, ... enc_dim=d_model, ... input_size=input_size, ... ) >>> n_channels=3 >>> seq_lin = Linear(input_shape=[d_model, n_channels], n_neurons=vocab_size) >>> lm_weight = 0.4 >>> length_weight = 1.0 >>> lm_model = RNNLM( ... embedding_dim=d_model, ... output_neurons=vocab_size, ... dropout=0.0, ... rnn_neurons=128, ... dnn_neurons=64, ... return_hidden=True, ... ) >>> rnnlm_scorer = RNNLMScorer( ... language_model=lm_model, ... temperature=1.25, ... ) >>> length_scorer = LengthScorer(vocab_size=vocab_size) >>> scorer = ScorerBuilder( ... full_scorers=[rnnlm_scorer, length_scorer], ... weights={'rnnlm': lm_weight, 'length': length_weight} ... ) >>> beam_size=5 >>> searcher = S2SRNNBeamSearcher( ... embedding=emb, ... decoder=dec, ... linear=seq_lin, ... bos_index=1, ... eos_index=2, ... min_decode_ratio=0.0, ... max_decode_ratio=1.0, ... topk=2, ... using_eos_threshold=False, ... beam_size=beam_size, ... temperature=1.25, ... length_normalization=False, ... scorer=scorer ... ) >>> batch_size=2 >>> enc = torch.rand([batch_size, n_channels, d_model]) >>> wav_len = torch.ones([batch_size]) >>> hyps, _, _, _ = searcher(enc, wav_len) cCs ||_dSN)ra)rrar r rr*s zLengthScorer.__init__cCs*tjdg|j|jd|d|jdfS)aThis method scores the new beams based on the Length scorer. Arguments --------- inp_tokens : torch.Tensor The input tensor of the current timestep. memory : No limit The scorer states for this timestep. candidates : torch.Tensor (batch_size x beam_size, scorer_beam_size). The top-k candidates to be scored after the full scorers. If None, scorers will score on full vocabulary set. attn : torch.Tensor The attention weight to be used in CoverageScorer or CTCScorer. Returns ------- torch.Tensor Scores None r4)rPrlrN)r>tensorrPrlrrTrarr r rrs  zLengthScorer.scoreN)rrrrr*rr r r rrsN rc@sHeZdZdZeeedfddZddZddZd d Z d d Z d S) ScorerBuildera Builds scorer instance for beamsearch. The ScorerBuilder class is responsible for building a scorer instance for beam search. It takes weights for full and partial scorers, as well as instances of full and partial scorer classes. It combines the scorers based on the weights specified and provides methods for scoring tokens, permuting scorer memory, and resetting scorer memory. This is the class to be used for building scorer instances for beam search. See speechbrain.decoders.seq2seq.S2SBeamSearcher() Arguments --------- weights : dict Weights of full/partial scorers specified. full_scorers : list Scorers that score on full vocabulary set. partial_scorers : list Scorers that score on pruned tokens to prevent computation overhead. Partial scoring is performed after full scorers. scorer_beam_scale : float The scale decides the number of pruned tokens for partial scorers: int(beam_size * scorer_beam_scale). Example ------- >>> from speechbrain.nnet.linear import Linear >>> from speechbrain.lobes.models.transformer.TransformerASR import TransformerASR >>> from speechbrain.lobes.models.transformer.TransformerLM import TransformerLM >>> from speechbrain.decoders import S2STransformerBeamSearcher, TransformerLMScorer, CoverageScorer, CTCScorer, ScorerBuilder >>> input_size=17 >>> vocab_size=11 >>> d_model=128 >>> net = TransformerASR( ... tgt_vocab=vocab_size, ... input_size=input_size, ... d_model=d_model, ... nhead=8, ... num_encoder_layers=1, ... num_decoder_layers=1, ... d_ffn=256, ... activation=torch.nn.GELU ... ) >>> lm_model = TransformerLM( ... vocab=vocab_size, ... d_model=d_model, ... nhead=8, ... num_encoder_layers=1, ... num_decoder_layers=0, ... d_ffn=256, ... activation=torch.nn.GELU, ... ) >>> n_channels=6 >>> ctc_lin = Linear(input_size=d_model, n_neurons=vocab_size) >>> seq_lin = Linear(input_size=d_model, n_neurons=vocab_size) >>> eos_index = 2 >>> ctc_scorer = CTCScorer( ... ctc_fc=ctc_lin, ... blank_index=0, ... eos_index=eos_index, ... ) >>> transformerlm_scorer = TransformerLMScorer( ... language_model=lm_model, ... temperature=1.15, ... ) >>> coverage_scorer = CoverageScorer(vocab_size=vocab_size) >>> ctc_weight_decode=0.4 >>> lm_weight=0.6 >>> coverage_penalty = 1.0 >>> scorer = ScorerBuilder( ... full_scorers=[transformerlm_scorer, coverage_scorer], ... partial_scorers=[ctc_scorer], ... weights={'transformerlm': lm_weight, 'ctc': ctc_weight_decode, 'coverage': coverage_penalty} ... ) >>> beam_size=5 >>> searcher = S2STransformerBeamSearcher( ... modules=[net, seq_lin], ... bos_index=1, ... eos_index=eos_index, ... min_decode_ratio=0.0, ... max_decode_ratio=1.0, ... using_eos_threshold=False, ... beam_size=beam_size, ... topk=3, ... temperature=1.15, ... scorer=scorer ... ) >>> batch_size=2 >>> wav_len = torch.ones([batch_size]) >>> src = torch.rand([batch_size, n_channels, input_size]) >>> tgt = torch.randint(0, vocab_size, [batch_size, n_channels]) >>> enc, dec = net.forward(src, tgt) >>> hyps, _, _, _ = searcher(enc, wav_len) rc Cst|t|t|ksJd||_ddtD}dd|D}dd|D}dd|D}i|||_tt|||_tt|||_| |dS)Nz$Weights and scorers are not matched.cS(g|]}|dr|ddqS)Scorerscorerrendswithlowersplit.0kr r r | z*ScorerBuilder.__init__..cS"g|] }|jjddqSrr __class__rrrrimplr r rrcSrrrrr r rrrcSi|]}|dqSr rr r r z*ScorerBuilder.__init__..) rfscorer_beam_scaleglobalskeysweightsdictzip full_scorerspartial_scorers_validate_scorer) rrrrrall_scorer_namesfull_scorer_namespartial_scorer_names init_weightsr r rr*ps( zScorerBuilder.__init__c Cst}|jD]*\}}|dkr|jj|dd|jf<||||d|\} ||<|| |j|7}q|jt ||j dd\} } |j D]\}}||||| |\} ||<|| |j|7}qF||fS)aBThis method scores tokens in vocabulary based on defined full scorers and partial scorers. Scores will be added to the log probs for beamsearch. Arguments --------- inp_tokens : torch.Tensor See BaseScorerInterface(). memory : dict[str, scorer memory] The states of scorers for this timestep. attn : torch.Tensor See BaseScorerInterface(). log_probs : torch.Tensor (batch_size x beam_size, vocab_size). The log probs at this timestep. beam_size : int The beam size. Returns ------- log_probs : torch.Tensor (batch_size x beam_size, vocab_size). Log probs updated by scorers. new_memory : dict[str, scorer memory] The updated states of scorers. ctcNrQrR) rritemsr+rer"rrtopkintrr) rr r r rArr}rrr_r r r rrs   zScorerBuilder.scorecCsz|jD]!\}}|dks|dkr||||||<q||||||<q|jD]\}}||||||<q,|S)aUpdate memory variables of scorers to synchronize the memory index with the current output and perform batched beam search. Arguments --------- memory : dict[str, scorer memory] The states of scorers for this timestep. index : torch.Tensor (batch_size x beam_size). The index of the previous path. candidates : torch.Tensor (batch_size, beam_size). The index of the topk candidates. Returns ------- memory : dict rr^)rrrr)rr rr rrr r rpermute_scorer_memsz ScorerBuilder.permute_scorer_memcCs8t}i|j|jD] \}}|||||<q |S)aReset memory variables for scorers. Arguments --------- x : torch.Tensor See BaseScorerInterface(). enc_lens : torch.Tensor See BaseScorerInterface(). Returns ------- memory : dict )rrrrr)rrrr rrr r rreset_scorer_memszScorerBuilder.reset_scorer_memcCst|jt|krtd|d|jdkr dks%tdtd|jddkrBd|jvr7td|jddkrDtdd Sd S) zThese error messages indicate scorers are not properly set. Arguments --------- scorer_names : list Prefix of scorers defined in speechbrain.decoders.scorer. )The keys of weights should be named in {}rrr4z%ctc_weight should not > 1.0 and < 0.0z:CTC scorer should be a full scorer when it's weight is 1.0rzBPure CTC scorer doesn't have attention weights for coverage scorerN)rfrrgformatrr)r scorer_namesr r rrs&zScorerBuilder._validate_scorerN) rrrrrlistr*rrrrr r r rrsb #0 rc@s2eZdZdZddZddZddZd d d ZdS) BaseRescorerInterfaceaA scorer abstraction intended for inheritance by other scoring approaches used in beam search. In this approach, a neural network is employed to assign scores to potential text transcripts. The beam search decoding process produces a collection of the top K hypotheses. These candidates are subsequently sent to a language model (LM) for ranking. The ranking is carried out by the LM, which assigns a score to each candidate. The score is computed as follows: score = beam_search_score + lm_weight * rescorer_score See: - speechbrain.decoders.scorer.RNNLMRescorer - speechbrain.decoders.scorer.TransformerLMRescorer - speechbrain.decoders.scorer.HuggingFaceLMRescorer cC|S)zThis method should implement the normalization of the text before scoring. Arguments --------- text : list of str The text to be normalized. Returns ------- Normalized text r rtextr r rnormalize_textrz$BaseRescorerInterface.normalize_textcCr)zThis method should implement the preprocessing of the hypotheses before scoring. Arguments --------- hyps : list of str The hypotheses to be preprocessed. rrhypsr r rpreprocess_func-z%BaseRescorerInterface.preprocess_funccCr)zThis method should implement the rescoring of the hypotheses. Arguments --------- hyps : list of str The hypotheses to be rescored. rrr r r rescore_hyps7rz"BaseRescorerInterface.rescore_hypsNcCr)a%This method should implement the moving of the scorer to a device. If device is None, the scorer should be moved to the default device provided in the constructor. Arguments --------- device : str The device to move the scorer to. rrrPr r r to_deviceAs zBaseRescorerInterface.to_devicer)rrrrrrrrr r r rr s   rc@NeZdZdZ     dddZddZdd d Zd d Ze ddZ d S) RNNLMRescorera A wrapper of RNNLM based on the BaseRescorerInterface. Arguments --------- language_model : torch.nn.Module A RNN-based language model. tokenizer : SentencePieceProcessor A SentencePiece tokenizer. device : str The device to move the scorer to. temperature : float Temperature factor applied to softmax. It changes the probability distribution, being softer when T>1 and sharper with T<1. (default: 1.0) bos_index : int The index of the beginning-of-sequence (bos) token. eos_index : int The index of the end-of-sequence (eos) token. pad_index : int The index of the padding token. Note ---- This class is intended to be used with a pretrained TransformerLM model. Please see: https://huggingface.co/speechbrain/asr-crdnn-rnnlm-librispeech By default, this model is using SentencePiece tokenizer. Example ------- >>> import torch >>> from sentencepiece import SentencePieceProcessor >>> from speechbrain.lobes.models.RNNLM import RNNLM >>> from speechbrain.utils.parameter_transfer import Pretrainer >>> source = "speechbrain/asr-crdnn-rnnlm-librispeech" >>> lm_model_path = source + "/lm.ckpt" >>> tokenizer_path = source + "/tokenizer.ckpt" >>> # define your tokenizer and RNNLM from the HF hub >>> tokenizer = SentencePieceProcessor() >>> lm_model = RNNLM( ... output_neurons = 1000, ... embedding_dim = 128, ... activation = torch.nn.LeakyReLU, ... dropout = 0.0, ... rnn_layers = 2, ... rnn_neurons = 2048, ... dnn_blocks = 1, ... dnn_neurons = 512, ... return_hidden = True, ... ) >>> pretrainer = Pretrainer( ... collect_in = getfixture("tmp_path"), ... loadables = { ... "lm" : lm_model, ... "tokenizer" : tokenizer, ... }, ... paths = { ... "lm" : lm_model_path, ... "tokenizer" : tokenizer_path, ... }) >>> _ = pretrainer.collect_files() >>> pretrainer.load_collected() >>> from speechbrain.decoders.scorer import RNNLMRescorer, RescorerBuilder >>> rnnlm_rescorer = RNNLMRescorer( ... language_model = lm_model, ... tokenizer = tokenizer, ... temperature = 1.0, ... bos_index = 0, ... eos_index = 0, ... pad_index = 0, ... ) >>> # Define a rescorer builder >>> rescorer = RescorerBuilder( ... rescorers=[rnnlm_rescorer], ... weights={"rnnlm":1.0} ... ) >>> # topk hyps >>> topk_hyps = [["HELLO", "HE LLO", "H E L L O"]] >>> topk_scores = [[-2, -2, -2]] >>> rescored_hyps, rescored_scores = rescorer.rescore(topk_hyps, topk_scores) >>> # NOTE: the returned hypotheses are already sorted by score. >>> rescored_hyps # doctest: +SKIP [['HELLO', 'H E L L O', 'HE LLO']] >>> # NOTE: as we are returning log-probs, the more it is closer to 0, the better. >>> rescored_scores # doctest: +SKIP [[-17.863974571228027, -25.12890625, -26.075977325439453]] cudar4rcCJ||_|j||_||_tjjjdd|_||_ ||_ ||_ ||_ dSr r7r8 tokenizerr9r%r&r'r(r)rP bos_indexr# pad_indexrr;rrPr9rr#rr r rr*s  zRNNLMRescorer.__init__cC|S)aThis method should implement the normalization of the text before scoring. Default to uppercasing the text because the (current) language models are trained on LibriSpeech which is all uppercase. Arguments --------- text : str The text to be normalized. Returns ------- str The normalized text. upperrr r rrzRNNLMRescorer.normalize_textNcC*|dur |j|jdS|j|dS)aThis method moves the scorer to a device. If device is None, the scorer is moved to the default device provided in the constructor. Arguments --------- device : str The device to move the scorer to. Nr7rZrPrr r rrs zRNNLMRescorer.to_devicec Cg}|D]}|D] }|||qqg}|D]}|t|jg|j||jgqdd|D}tjj j j |d|j d |jj}||fS)tThis method preprocesses the hypotheses before scoring. Arguments --------- topk_hyps : list of list of str The hypotheses to be preprocessed. Returns ------- padded_hyps : torch.Tensor The padded hypotheses. enc_hyps_length : list of int The length of each hypothesis. cSg|]}|jdqSr1rrenc_seqr r rrz1RNNLMRescorer.preprocess_func..T batch_first padding_valueappendrr>rrr encode_as_idsr#nnutilsrnn pad_sequencerrZr7rX__next__rPr topk_hyps decoded_seqbatchseqenc_hypsenc_hyps_length padded_hypsr r rr.  zRNNLMRescorer.preprocess_funcc s||\}fddD}tj|tj|jd}t|jjs)|j |j||\}}| ||j }|ddddf d|ddddf dd}tj||ddddfdd} | S) 8This method implement the rescoring of the hypotheses. Arguments --------- topk_hyps : list of list of str The hypotheses to be rescored. Returns ------- log_probs_scores : torch.Tensor[B * Topk, 1] The rescored hypotheses scores c&g|]fddttDqS)cg|] }|kr dndqSrCrr rrlengthr rrz9RNNLMRescorer.rescore_hyps...rtrrrr rrz.RNNLMRescorer.rescore_hyps..rlrPNrQrrCrR)rr>rboolrPrWr7rXrYrZr)r9gatherrVsqueezenansum) rrr bool_maskbool_mask_tensorr0rrAtarget_log_probslog_probs_scoresr rrrs&  zRNNLMRescorer.rescore_hypsrr4rrrr rrrrr*rrrr>r?rr r r rrOs[  )rc@r)TransformerLMRescorera A wrapper of TransformerLM based on the BaseRescorerInterface. Arguments --------- language_model : torch.nn.Module A Transformer-based language model. tokenizer : SentencePieceProcessor A SentencePiece tokenizer. device : str The device to move the scorer to. temperature : float Temperature factor applied to softmax. It changes the probability distribution, being softer when T>1 and sharper with T<1. (default: 1.0) bos_index : int The index of the beginning-of-sequence (bos) token. eos_index : int The index of the end-of-sequence (eos) token. pad_index : int The index of the padding token. Note ---- This class is intended to be used with a pretrained TransformerLM model. Please see: https://huggingface.co/speechbrain/asr-transformer-transformerlm-librispeech By default, this model is using SentencePiece tokenizer. Example ------- >>> import torch >>> from sentencepiece import SentencePieceProcessor >>> from speechbrain.lobes.models.transformer.TransformerLM import TransformerLM >>> from speechbrain.utils.parameter_transfer import Pretrainer >>> source = "speechbrain/asr-transformer-transformerlm-librispeech" >>> lm_model_path = source + "/lm.ckpt" >>> tokenizer_path = source + "/tokenizer.ckpt" >>> tokenizer = SentencePieceProcessor() >>> lm_model = TransformerLM( ... vocab=5000, ... d_model=768, ... nhead=12, ... num_encoder_layers=12, ... num_decoder_layers=0, ... d_ffn=3072, ... dropout=0.0, ... activation=torch.nn.GELU, ... normalize_before=False, ... ) >>> pretrainer = Pretrainer( ... collect_in = getfixture("tmp_path"), ... loadables={ ... "lm": lm_model, ... "tokenizer": tokenizer, ... }, ... paths={ ... "lm": lm_model_path, ... "tokenizer": tokenizer_path, ... } ... ) >>> _ = pretrainer.collect_files() >>> pretrainer.load_collected() >>> from speechbrain.decoders.scorer import TransformerLMRescorer, RescorerBuilder >>> transformerlm_rescorer = TransformerLMRescorer( ... language_model=lm_model, ... tokenizer=tokenizer, ... temperature=1.0, ... bos_index=1, ... eos_index=2, ... pad_index=0, ... ) >>> rescorer = RescorerBuilder( ... rescorers=[transformerlm_rescorer], ... weights={"transformerlm": 1.0} ... ) >>> topk_hyps = [["HELLO", "HE LLO", "H E L L O"]] >>> topk_scores = [[-2, -2, -2]] >>> rescored_hyps, rescored_scores = rescorer.rescore(topk_hyps, topk_scores) >>> # NOTE: the returned hypotheses are already sorted by score. >>> rescored_hyps # doctest: +SKIP [["HELLO", "HE L L O", "HE LLO"]] >>> # NOTE: as we are returning log-probs, the more it is closer to 0, the better. >>> rescored_scores # doctest: +SKIP [[-17.863974571228027, -25.12890625, -26.075977325439453]] rr4rcCrrrrr r rr*s  zTransformerLMRescorer.__init__cCr)aiThis method should implement the normalization of the text before scoring. Default to uppercasing the text because the language models are trained on LibriSpeech. Arguments --------- text : str The text to be normalized. Returns ------- str The normalized text. rrr r rrrz$TransformerLMRescorer.normalize_textNcCrahThis method moves the scorer to a device. If device is None, the scorer is moved to the default device provided in the constructor. This method is dynamically called in the recipes when the stage is equal to TEST. Arguments --------- device : str The device to move the scorer to. Nrrr r rrzTransformerLMRescorer.to_devicec Cr)rcSrr1rrr r rrrz9TransformerLMRescorer.preprocess_func..Trrrr r rrrz%TransformerLMRescorer.preprocess_funcc s||\}fddD}tj|tj|jd}t|jjs)|j |j||}| ||j }t d|dddd|j f<|ddddfd|ddddfdd}||ddddfjdd }tj||ddddfdd }|S) rcr)cr r r r r r rrrzATransformerLMRescorer.rescore_hyps...rrrr rrrz6TransformerLMRescorer.rescore_hyps..r-infNrQrrCrR)rr>rrrPrWr7rXrYrZr)r9rxrrrVr logsumexpr) rrrrrr0rArrr rrrs.   z"TransformerLMRescorer.rescore_hypsrrrr r r rr5sY  )rc@sNeZdZdZ dddZdddZdd Zd d Zd d Ze ddZ dS)HuggingFaceLMRescoreraA wrapper of HuggingFace's TransformerLM based on the BaseRescorerInterface. Arguments --------- model_name : str The name of the model to be loaded. device : str The device to be used for scoring. (default: "cuda") Example ------- >>> from speechbrain.decoders.scorer import HuggingFaceLMRescorer, RescorerBuilder >>> source = "gpt2-medium" >>> huggingfacelm_rescorer = HuggingFaceLMRescorer( ... model_name=source, ... ) >>> rescorer = RescorerBuilder( ... rescorers=[huggingfacelm_rescorer], ... weights={"huggingfacelm": 1.0} ... ) >>> topk_hyps = [["Hello everyone.", "Hell o every one.", "Hello every one"]] >>> topk_scores = [[-2, -2, -2]] >>> rescored_hyps, rescored_scores = rescorer.rescore(topk_hyps, topk_scores) >>> # NOTE: the returned hypotheses are already sorted by score. >>> rescored_hyps # doctest: +SKIP [['Hello everyone.', 'Hello every one', 'Hell o every one.']] >>> # NOTE: as we are returning log-probs, the more it is closer to 0, the better. >>> rescored_scores # doctest: +SKIP [[-20.03631591796875, -27.615638732910156, -42.662353515625]] rcCs||_||_z ddlm}m}Wn tytdw|j|jdd|_|j|jdd|_ |j j durPd|j _ |j d|j j gi|jj t |j d d |j j|_|j j|_dS) Nr)AutoModelForCausalLM AutoTokenizerz:Please install transformers with: pip install transformersT) is_decoder)use_fastz<|pad|>additional_special_tokens )pad_to_multiple_of) model_namerP transformersr$r%r_from_pretrainedr8r7r pad_tokenadd_special_tokensresize_token_embeddingsrf bos_token eos_token)rr+rPr$r%r r rr*Bs6     zHuggingFaceLMRescorer.__init__NcCrrrrr r rrer zHuggingFaceLMRescorer.to_devicecCr)aThis method should implement the normalization of the text before scoring. Arguments --------- text : str The text to be normalized. Returns ------- normalized_text : str The normalized text. In this case we do not apply any normalization. However, this method can be overridden to apply any normalization. r rr r rrxsz$HuggingFaceLMRescorer.normalize_textcCs|j||jS)zThis method adds the special tokens to the text. Arguments --------- text : str The text to be augmented. Returns ------- augmented_text : str The augmented text. )r1r2rr r r_add_special_tokenss z)HuggingFaceLMRescorer._add_special_tokenscCsNg}|D]}|D] }|||qqtt|j|}|jj|ddd}|S)aThis method preprocesses the hypotheses before scoring. Arguments --------- topk_hyps : list of str The hypotheses to be preprocessed. Returns ------- encoding : tensor The encoding of the hypotheses. ptT)return_tensorspadding)rrrmapr3rbatch_encode_plus)rrnormalized_hypsrrtext_augmented_with_tokensencodingr r rrs z%HuggingFaceLMRescorer.preprocess_funccCs||}|d|jj}|d|jj}|j||dd}td|dddd|jjdf<|ddddfd|ddd dfd d}||ddddfj dd }t j ||ddd dfdd }|S) r input_idsattention_mask)r=rr!NrQrrCrR) rrZr7rPrxr pad_token_idrrVrr"r>r)rrr;idsr=r0rrr r rrs "6 z"HuggingFaceLMRescorer.rescore_hyps)rr) rrrrr*rrr3rr>r?rr r r rr#"s" #r#c@s<eZdZdZeefddZddZddZd d d Z dS) RescorerBuildera Builds rescorer instance for beamsearch. The RescorerBuilder class is responsible for building a scorer instance for beam search. It takes weights and rescorers classes. It combines the scorers based on the weights specified and provides methods for rescoring text. This is the class to be used for building rescorer instances for beam search. Arguments --------- weights : dict Weights of rescorers specified. rescorers : list Rescorers that re-ranks topk hypotheses. cCszt|t|ks Jd||_ddtD}dd|D}dd|D}i|||_tt|||_||dS)Nz&Weights and rescorers are not matched.cSr)Rescorerrescorerrrrr r rrrz,RescorerBuilder.__init__..cSr)rBrrrr r rrrcSrrr rr r rrrz,RescorerBuilder.__init__..)rfrrrrr rescorersr)rrrCall_rescorer_namesfull_rescorer_namesrr r rr*s zRescorerBuilder.__init__c Cs|}|jD]6\}}||}d}tt|D]$}tt||D]} ||| |j|||7<|d7}q$qq tddt ||Dg} g} g} | D]} | D]}|\}}| t|| t|qWqS| | fS)aThis method rescores the topk candidates. Arguments --------- topk_candidates : list of list of str The topk candidates to be rescored. topk_scores : list of list of float The scores of the topk candidates. Returns ------- output_candidates : list of list of str The rescored candidates. output_scores : list of list of float The rescored scores. rrCcss0|]\}}ttt||ddddVqdS)cSs|dS)NrCr )rr r r$sz3RescorerBuilder.rescore...T)keyreverseN)rsorted)rsublistrr r r !s z*RescorerBuilder.rescore..) copyrCrrrtrfrrurrr)rtopk_candidates topk_scores new_scoresrrr- index_scoresrjsorted_candidatesoutput_candidates output_scoresrJrutextsr r rrescores2    zRescorerBuilder.rescorecCs$t|jt|krtd|dS)zThese error messages indicate rescorers are not properly set. Arguments --------- rescorer_names : list Prefix of rescorers defined in speechbrain.decoders.scorer. rN)rfrrgr)rrescorer_namesr r rr5sz RescorerBuilder._validate_scorerNcCs"|jD] \}}||qdS)zMoves rescorers to device. Useful to avoid having on GPU rescorers while being on TRAIN and VALID Stages. Arguments --------- device : str The device to be used for scoring. (default: None) N)rCrr)rrPrrr r rmove_rescorers_to_deviceDs z(RescorerBuilder.move_rescorers_to_devicer) rrrrrrr*rVrrXr r r rr@s 3r@)rrrbr> speechbrainr%speechbrain.decoders.ctcrrrr3rNr]rrrrrrr#r@r r r rs6 \ R#qBgn4