o }oiӤ@s:ddlZddlZddlZddlmZddlmZmZmZddl m Z m Z m Z m Z mZmZddlZddlZddlmZddlmZmZmZddlmZmZddlmZmZdd lm Z m!Z!dd l"m#Z#dd l$m%Z%dd l&m'Z'm(Z(Gd ddeZ)Gddde)Z*Gddde)Z+eGdddZ,eGddde,Z-dS)N)abstractmethod) dataclassfield is_dataclass)CallableDictListOptionalSetUnion) OmegaConf)rnnt_beam_decodingrnnt_greedy_decodingtdt_beam_decoding)ConfidenceConfigConfidenceMixin)BlankLMScoreMode PruningMode) HypothesisNBestHypotheses)AggregateTokenizer) TokenizerSpec)logging logging_modecs4eZdZdZd:dedeeffdd Z  d;dej d ej d e d ee e d e e e e e e ff d dZde e d e e e effddZde e d e e fddZede eee eeffde eee eeffdedeed e eee eefff ddZede ed efddZede ed e efddZede ed efddZede ed e efd d!Zde ed efd"d#Zd$d%ZdAbstractRNNTDecodingu/ Used for performing RNN-T auto-regressive decoding of the Decoder+Joint network given the encoder state. Args: decoding_cfg: A dict-like object which contains the following key-value pairs. strategy: str value which represents the type of decoding that can occur. Possible values are : - greedy, greedy_batch (for greedy decoding). - beam, tsd, alsd (for beam search decoding). compute_hypothesis_token_set: A bool flag, which determines whether to compute a list of decoded tokens as well as the decoded string. Default is False in order to avoid double decoding unless required. preserve_alignments: Bool flag which preserves the history of logprobs generated during decoding (sample / batched). When set to true, the Hypothesis will contain the non-null value for `alignments` in it. Here, `alignments` is a List of List of Tuple(Tensor (of length V + 1), Tensor(scalar, label after argmax)). In order to obtain this hypothesis, please utilize `rnnt_decoder_predictions_tensor` function with the `return_hypotheses` flag set to True. The length of the list corresponds to the Acoustic Length (T). Each value in the list (Ti) is a torch.Tensor (U), representing 1 or more targets from a vocabulary. U is the number of target tokens for the current timestep Ti. tdt_include_token_duration: Bool flag, which determines whether predicted durations for each token need to be included in the Hypothesis object. Defaults to False. compute_timestamps: A bool flag, which determines whether to compute the character/subword, or word based timestamp mapping the output log-probabilities to discrete intervals of timestamps. The timestamps will be available in the returned Hypothesis.timestep as a dictionary. rnnt_timestamp_type: A str value, which represents the types of timestamps that should be calculated. Can take the following values - "char" for character/subword time stamps, "word" for word level time stamps, "segment" for segment level time stamps and "all" (default), for character, word and segment level time stamps. word_seperator: Str token representing the seperator between words. segment_seperators: List containing tokens representing the seperator(s) between segments. segment_gap_threshold: The threshold (in frames) that caps the gap between two words necessary for forming the segments. preserve_frame_confidence: Bool flag which preserves the history of per-frame confidence scores generated during decoding (sample / batched). When set to true, the Hypothesis will contain the non-null value for `frame_confidence` in it. Here, `alignments` is a List of List of ints. confidence_cfg: A dict-like object which contains the following key-value pairs related to confidence scores. In order to obtain hypotheses with confidence scores, please utilize `rnnt_decoder_predictions_tensor` function with the `preserve_frame_confidence` flag set to True. preserve_frame_confidence: Bool flag which preserves the history of per-frame confidence scores generated during decoding (sample / batched). When set to true, the Hypothesis will contain the non-null value for `frame_confidence` in it. Here, `alignments` is a List of List of floats. The length of the list corresponds to the Acoustic Length (T). Each value in the list (Ti) is a torch.Tensor (U), representing 1 or more confidence scores. U is the number of target tokens for the current timestep Ti. preserve_token_confidence: Bool flag which preserves the history of per-token confidence scores generated during greedy decoding (sample / batched). When set to true, the Hypothesis will contain the non-null value for `token_confidence` in it. Here, `token_confidence` is a List of floats. The length of the list corresponds to the number of recognized tokens. preserve_word_confidence: Bool flag which preserves the history of per-word confidence scores generated during greedy decoding (sample / batched). When set to true, the Hypothesis will contain the non-null value for `word_confidence` in it. Here, `word_confidence` is a List of floats. The length of the list corresponds to the number of recognized words. exclude_blank: Bool flag indicating that blank token confidence scores are to be excluded from the `token_confidence`. aggregation: Which aggregation type to use for collapsing per-token confidence into per-word confidence. Valid options are `mean`, `min`, `max`, `prod`. tdt_include_duration: Bool flag indicating that the duration confidence scores are to be calculated and attached to the regular frame confidence, making TDT frame confidence element a pair: (`prediction_confidence`, `duration_confidence`). method_cfg: A dict-like object which contains the method name and settings to compute per-frame confidence scores. name: The method name (str). Supported values: - 'max_prob' for using the maximum token probability as a confidence. - 'entropy' for using a normalized entropy of a log-likelihood vector. entropy_type: Which type of entropy to use (str). Used if confidence_method_cfg.name is set to `entropy`. Supported values: - 'gibbs' for the (standard) Gibbs entropy. If the alpha (α) is provided, the formula is the following: H_α = -sum_i((p^α_i)*log(p^α_i)). Note that for this entropy, the alpha should comply the following inequality: (log(V)+2-sqrt(log^2(V)+4))/(2*log(V)) <= α <= (1+log(V-1))/log(V-1) where V is the model vocabulary size. - 'tsallis' for the Tsallis entropy with the Boltzmann constant one. Tsallis entropy formula is the following: H_α = 1/(α-1)*(1-sum_i(p^α_i)), where α is a parameter. When α == 1, it works like the Gibbs entropy. More: https://en.wikipedia.org/wiki/Tsallis_entropy - 'renyi' for the Rényi entropy. Rényi entropy formula is the following: H_α = 1/(1-α)*log_2(sum_i(p^α_i)), where α is a parameter. When α == 1, it works like the Gibbs entropy. More: https://en.wikipedia.org/wiki/R%C3%A9nyi_entropy alpha: Power scale for logsoftmax (α for entropies). Here we restrict it to be > 0. When the alpha equals one, scaling is not applied to 'max_prob', and any entropy type behaves like the Shannon entropy: H = -sum_i(p_i*log(p_i)) entropy_norm: A mapping of the entropy value to the interval [0,1]. Supported values: - 'lin' for using the linear mapping. - 'exp' for using exponential mapping with linear shift. The config may further contain the following sub-dictionaries: "greedy": max_symbols: int, describing the maximum number of target tokens to decode per timestep during greedy decoding. Setting to larger values allows longer sentences to be decoded, at the cost of increased execution time. preserve_frame_confidence: Same as above, overrides above value. confidence_method_cfg: Same as above, overrides confidence_cfg.method_cfg. "beam": beam_size: int, defining the beam size for beam search. Must be >= 1. If beam_size == 1, will perform cached greedy search. This might be slightly different results compared to the greedy search above. score_norm: optional bool, whether to normalize the returned beam score in the hypotheses. Set to True by default. return_best_hypothesis: optional bool, whether to return just the best hypothesis or all of the hypotheses after beam search has concluded. This flag is set by default. tsd_max_sym_exp: optional int, determines number of symmetric expansions of the target symbols per timestep of the acoustic model. Larger values will allow longer sentences to be decoded, at increased cost to execution time. alsd_max_target_len: optional int or float, determines the potential maximum target sequence length. If an integer is provided, it can decode sequences of that particular maximum length. If a float is provided, it can decode sequences of int(alsd_max_target_len * seq_len), where seq_len is the length of the acoustic model output (T). NOTE: If a float is provided, it can be greater than 1! By default, a float of 2.0 is used so that a target sequence can be at most twice as long as the acoustic model output length T. maes_num_steps: Number of adaptive steps to take. From the paper, 2 steps is generally sufficient, and can be reduced to 1 to improve decoding speed while sacrificing some accuracy. int > 0. maes_prefix_alpha: Maximum prefix length in prefix search. Must be an integer, and is advised to keep this as 1 in order to reduce expensive beam search cost later. int >= 0. maes_expansion_beta: Maximum number of prefix expansions allowed, in addition to the beam size. Effectively, the number of hypothesis = beam_size + maes_expansion_beta. Must be an int >= 0, and affects the speed of inference since large values will perform large beam search in the next step. maes_expansion_gamma: Float pruning threshold used in the prune-by-value step when computing the expansions. The default (2.3) is selected from the paper. It performs a comparison (max_log_prob - gamma <= log_prob[v]) where v is all vocabulary indices in the Vocab set and max_log_prob is the "most" likely token to be predicted. Gamma therefore provides a margin of additional tokens which can be potential candidates for expansion apart from the "most likely" candidate. Lower values will reduce the number of expansions (by increasing pruning-by-value, thereby improving speed but hurting accuracy). Higher values will increase the number of expansions (by reducing pruning-by-value, thereby reducing speed but potentially improving accuracy). This is a hyper parameter to be experimentally tuned on a validation set. softmax_temperature: Scales the logits of the joint prior to computing log_softmax. decoder: The Decoder/Prediction network module. joint: The Joint network module. blank_id: The id of the RNNT blank token. supported_punctuation: Set of punctuation marks in the vocabulary Nblank_idsupported_punctuationcs tt|t|rt|}||_||_||_|j |_ |j dd|_ |j dd|_ |j dd|_ | dd|_|j dd|_|j dd|_|j dd|_|j d d|_|j d d |_|j d gd |_|j dd|_|j duo|j gk|_|jr|dkrtd|j dur|j gkrtd|jjdvrtd|j dur|j gkr|dkrtd|jjdvrtdgd}|jj|vrtd||jdur|jjdvr|jj dd|_n|jjdvr|jj dd|_|jdur|jjdvr |jj dd|_n|jjdvr|jj dd|_|jr5ddd|jD}td|d|_|jsI|jdurI|jdurItd ||j d!d|jrw|j durg|jdurgtd"|jpm|j|_|j!|_"|j#|_$|j s|jjdvr|jj d#drt%d$|jjd%|jjd&kr|j dus|j gkr|jst&j'|||j|jj d'dp|jj d(d|j|j |j(d)|_)nt&j*|||j|j |jj d'dp|jj d(d|j|j |j|j+|j(d* |_)nt&j,|||j|j |jj d'dp|jj d(d|j|j |j(d+|_)n|jjd,kr|j dus-|j gkr|jsot&j-|||j|jj d'dpG|jj d(d|j|j |j(|jj d-d|jj d.d|jj d/d|jj d0dd1 |_)n_t&j.|||j|j |jj d'dp|jj d(d|j|j |j|j+|j(|jj d.d|jj d/d|jj d0dd2 |_)n"t&j/|||j|j |jj d'dp|jj d(d|j|j |j(d+|_)n|jjd3kr3|j dus|j gkr1|js t0j1|||jjj2|j d4dd5|jj d6d|jj d7d8|jd9|_)nt3j4|||j |jjj2|j d4dd5|jj d6d|jj d7d8|jd: |_)n|jjd;krft0j1|||jjj2|j d4dd;|jj d6d|jj d |_)nh|jjd?krt0j1|||jjj2|j d4dd?|jj d6d|jj d@dA|jj d7d8|jdB |_)n5|jjdCkr|j dus|j gkr}|js.t0j1d^idD|dE|dF|jjj2d4|j d4ddGdCd6|jj d6ddH|jj dHdAdI|jj dIdJdK|jj dKdLdM|jj dMdNd7|jj d7d8d|jd/|jj d/dd0|jj d0dOdP|jj dPddQ|jj dQdO|_)nt3j4|||j |jjj2|j d4ddC|jj d6d|jj dHdA|jj dIdJ|jj dKdL|jj dMdN|jj d7d8|j|jj d/d|jj d0dRdS|_)nO|jjdTkr0|j dus|j gkr/|jst0j5|||j|jjj2dT|jj d'd=|j|jj d/d|jj d0dO|jj dUt6j7|jj dVt8j9|jj d6d|jj dWd|jj d4ddX|_)nt3j:|||j|j |jjj2dT|jj d'd=|j|jj d/d|jj d0dO|jj dUt6j7|jj dVt8j9|jj d6d|jj dWd|jj d4ddY|_)n|jjdZkr|j dusC|j gkr|jst0j5d^idD|dE|d[|jdF|jjj2dGdZdH|jj dHdAdM|jj dMdAdK|jj dKdLd|jd/|jj d/dd0|jj d0dOdU|jj dUt6j7dV|jj dVt8j9d6|jj d6ddW|jj dWdd4|jj d4d|_)n td\|d]|jj|;dS)_Nbig_blank_durations durationscompute_hypothesis_token_setF compute_langspreserve_alignmentsfused_batch_sizecompute_timestampstdt_include_token_durationword_seperator segment_seperators).?!segment_gap_thresholdrz8blank_id must equal len(non_blank_vocabs) for TDT modelsz7duration and big_blank_durations can't both be not None)greedy greedy_batchbeammaes malsd_batchzXcurrently only greedy, greedy_batch, beam and maes inference is supported for TDT modelszsz1AbstractRNNTDecoding.__init__..z(\s)()TzUIf `compute_timesteps` flag is set, then `preserve_alignments` flag must also be set.confidence_cfgz]If `preserve_frame_confidence` flag is set, then `preserve_alignments` flag must also be set.preserve_frame_confidencez6Confidence calculation is not supported for strategy ``r, max_symbolsmax_symbols_per_step) decoder_model joint_model blank_indexrBr!r?confidence_method_cfg) rCrDrErrBr!r?include_durationinclude_duration_confidencerF)rCrDrErrBr!r?rFr- loop_labelsuse_cuda_graph_decoderngram_lm_modelngram_lm_alpha) rCrDrErBr!r?rFrIrJrKrL) rCrDrErrBr!r?rGrHrFrJrKrLr.return_best_hypothesisdefault score_normsoftmax_temperature?)rCrD beam_sizerM search_typerOrPr!) rCrDrrRrMrSrOrPr!r1tsd_max_sym_exp ) rCrDrRrMrSrOtsd_max_sym_exp_per_steprPr!r2alsd_max_target_len) rCrDrRrMrSrOrWrPr!r/rCrDrRrSmaes_num_stepsmaes_prefix_alphamaes_expansion_gammagffffff@maes_expansion_betag@ghat_subtract_ilmhat_ilm_weightg333333?)rCrDrrRrMrSrOrYrZr\r]rPr!rKrLr0blank_lm_score_mode pruning_modeallow_cuda_graphs)rCrDrErRrSrBr!rKrLr`rarOrbrM)rCrDrErrRrSrBr!rKrLr`rarOrbrMr3rEz5Incorrect decoding strategy supplied. Must be one of z but was provided r5)<superr__init__rr structuredcfgrrnum_extra_outputsgetrrrr r!joint_fused_batch_sizer#r$r%r'r+_is_tdt ValueErrorstrategyr,r.joinr6compilespace_before_punct_pattern_init_confidencer?_compute_offsets_tdt_compute_offsets_refine_timestamps_tdt_refine_timestampsNotImplementedErrorrGreedyRNNTInferrFdecodingGreedyTDTInfertdt_include_duration_confidenceGreedyMultiblankRNNTInferGreedyBatchedRNNTInferGreedyBatchedTDTInfer GreedyBatchedMultiblankRNNTInferr BeamRNNTInferrRr BeamTDTInferBeamBatchedRNNTInferrLM_WEIGHTED_FULLrLATEBeamBatchedTDTInferupdate_joint_fused_batch_size)self decoding_cfgdecoderjointrrpossible_strategies punct_pattern __class__r5r;rds                                                zAbstractRNNTDecoding.__init__Fencoder_outputencoded_lengthsreturn_hypothesespartial_hypothesesreturncCs^t|j|||d}|d}Wdn1swY|}t|dtrrg}g}|D]5} | j} || } |jdurX|j dd} t t | D] } | | | | | | <qK| | d| | q/|ri|Sdd|D}|S||}|jdur|j dd} t t |D] } | || | || <q|r|jr|js|jr||}|Sd d|DS) a Decode an encoder output by autoregressive decoding of the Decoder+Joint networks. Args: encoder_output: torch.Tensor of shape [B, D, T]. encoded_lengths: torch.Tensor containing lengths of the padded encoder outputs. Shape [B]. return_hypotheses: bool. If set to True it will return list of Hypothesis or NBestHypotheses Returns: If `return_all_hypothesis` is set: A list[list[Hypothesis]]. Look at rnnt_utils.Hypothesis for more information. If `return_all_hypothesis` is not set: A list[Hypothesis]. List of best hypotheses Look at rnnt_utils.Hypothesis for more information. )rrrrNTrnnt_timestamp_typeallcSsg|] }dd|DqS)cSg|] }t|j|j|jqSr5rscore y_sequencetextr9hr5r5r;r<ozSAbstractRNNTDecoding.rnnt_decoder_predictions_tensor...r5)r9hhr5r5r;r<ozHAbstractRNNTDecoding.rnnt_decoder_predictions_tensor..cSrr5rrr5r5r;r<r)torchinference_moderw isinstancern_best_hypothesesdecode_hypothesisr#rfrhrangelencompute_rnnt_timestampsappendr?preserve_word_confidencepreserve_token_confidencecompute_confidence)rrrrrhypotheses_listprediction_list hypothesesall_hypotheses nbest_hypn_hyps decoded_hypstimestamp_typehyp_idxall_hypr5r5r;rnnt_decoder_predictions_tensor7sJ        z4AbstractRNNTDecoding.rnnt_decoder_predictions_tensorrcstt|D]~}||j}t|tkr|}jdur1jgkr1tjfdd|D}njr>fdd|D}n fdd|D}jdurUjrU|ddf}n*jdurot ||j }dgt|}|||f}n |}j r|||_|||_q|S)z Decode a list of hypotheses into a list of strings. Args: hypotheses_list: List of Hypothesis. Returns: A list of strings. Ncsg|] }|jkr|qSr5rr8rgrr5r;r<rz:AbstractRNNTDecoding.decode_hypothesis..csg|] }|jkr|qSr5rr8rr5r;r<rcg|] }|jkr|qSr5rr8rr5r;r<rTr[)rrrtypelisttolistrrjr#copydeepcopy alignments+decode_tokens_to_str_with_strip_punctuationrdecode_ids_to_tokenstokensr)rrind prediction hypothesisrtoken_repetitionsr5rr;rs*        z&AbstractRNNTDecoding.decode_hypothesisc s|jrjr fddndd}|D]}g}g}t|jddd|jdddD]\}}t|dkrrttt|D]5\}} | dj krSj sR| |||q;|s_| |||q;| |||g|g}q;q)d|d}} | dj krj s| |||q)|s| |||q)| |||g|g}q)|ddd}||_ qnuj r|D]}|j|_ qnh|D]e}t|jtjr|jn|j} d} g}t| dkr(t| | ddt|jgD]6\} } | | kr| |j| | gdd |j| d| Dd} q| |j| | | d7} q||_ qƈjr<|D] }||_q2|S) aI Computes high-level (per-token and/or per-word) confidence scores for a list of hypotheses. Assumes that `frame_confidence` is present in the hypotheses. Args: hypotheses_list: List of Hypothesis. Returns: A list of hypotheses with high-level confidence scores. cs |SN)_aggregate_confidencexrr5r;s z9AbstractRNNTDecoding.compute_confidence..cSs|Srr5rr5r5r;rsNr[rcSsg|]}|dqS)rr5)r9fcr5r5r;r<sz;AbstractRNNTDecoding.compute_confidence..)rjryzipframe_confidencerrreversedr enumeraterexclude_blank_from_confidencerrtoken_confidencenon_blank_frame_confidencer timestamprTensorrr_aggregate_token_confidenceword_confidence)rrmaybe_pre_aggregatehyprsubsequent_blank_confidencerfaiatimestepoffsettster5rr;rs| * $ &  z'AbstractRNNTDecoding.compute_confidence char_offsetsencoded_char_offsetsword_delimiter_charcCt)zL Implemented by subclass in order to get the words offsets. ru)rrrrrr5r5r;get_words_offsets z&AbstractRNNTDecoding.get_words_offsetsrcCr)z Implemented by subclass in order to decoder a token id list into a string. Args: tokens: List of int representing the token ids. Returns: A decoded string. rrrr5r5r;decode_tokens_to_strrz)AbstractRNNTDecoding.decode_tokens_to_strcCr& Implemented by subclass in order to decode a token id list into a token list. A token list is the string representation of each token id. Args: tokens: List of int representing the token ids. Returns: A list of decoded tokens. rrr5r5r;r z)AbstractRNNTDecoding.decode_ids_to_tokenscCr)z Implemented by subclass in order to compute the most likely language ID (LID) string given the tokens. Args: tokens: List of int representing the token ids. Returns: A decoded LID string. rrr5r5r;decode_tokens_to_lang-rz*AbstractRNNTDecoding.decode_tokens_to_langcCr)z Implemented by subclass in order to decode a token id list into language ID (LID) list. Args: tokens: List of int representing the token ids. Returns: A list of decoded LIDS. rrr5r5r;decode_ids_to_langs;rz(AbstractRNNTDecoding.decode_ids_to_langscCs"||}|jr|jd|}|S)zn Decodes a list of tokens to a string and removes a space before supported punctuation marks. z\2)rrrosub)rrrr5r5r;rIs z@AbstractRNNTDecoding.decode_tokens_to_str_with_strip_punctuationcCs|jdurdSt|jjdstddSt|jjds#tddS|jdkr2|jj|jdStd|jjddS) a " Updates the fused batch size for the joint module if applicable. If `joint_fused_batch_size` is set, verifies that the joint module has the required `set_fused_batch_size` and `set_fuse_loss_wer` functions. If present, updates the batch size; otherwise, logs a warning. If `joint_fused_batch_size` is <= 0, disables fused batch processing. Nset_fused_batch_sizez{The joint module does not have `set_fused_batch_size(int)` as a setter function. Ignoring update of joint fused batch size.set_fuse_loss_werzThe joint module does not have `set_fuse_loss_wer(bool, RNNTLoss, RNNTWER)` as a setter function. Ignoring update of joint fused batch size.rzTJoint fused batch size <= 0; Will temporarily disable fused batch step in the Joint.F) rihasattrrwrrwarningrinforrr5r5r;rRs   z2AbstractRNNTDecoding.update_joint_fused_batch_sizerrrc s"|dvsJ|j\}}}d}}||j}||_d}tt|D]} |tfdd|| dD7}q#|t|jkrRtd|d|jd |d t|jt|} t|D]#\} } g} | dD]}|jkrw| t |gqe| || d<q[ | |j \} }g}|D]}|d}td d |D}| |qd}|d vrj|| jj d}d}|dvrňj|jj jd}t|jdkr|j}ng}d|i|_|dur|dvr||jd<|dur|dvr||jd<|dur|dvr||jd<|j|_jr||_|S)a Computes character, word, and segment timestamps for an RNN-T hypothesis. This function generates timestamps for characters, words, and segments within a hypothesis sequence. The type of timestamps computed depends on `timestamp_type`, which can be 'char', 'word', 'segment', or 'all'. Args: hypothesis (Hypothesis): Hypothesis. timestamp_type (str): Type of timestamps to compute. Options are 'char', 'word', 'segment', or 'all'. Defaults to 'all'. Returns: Hypothesis: The updated hypothesis with computed timestamps for characters, words, and/or segments. )charwordsegmentrNrcrr5rr9crr5r;r<rz@AbstractRNNTDecoding.compute_rnnt_timestamps..rz`char_offsets`: z and `processed_tokens`: z9 have to be of the same length, but are: `len(offsets)`: z and `len(processed_tokens)`: css|]}t|VqdSr)rrr5r5r; sz?AbstractRNNTDecoding.compute_rnnt_timestamps..)rrr)rrrr)rr)segment_delimiter_tokensrr+r)rr)rrrr)rrrrrrrkrrrrrintrtrmaxrr%_get_segment_offsetsr'r+rrrrr)rrrdecoded_predictionrrr word_offsetsnum_flattened_tokenstrroffsets decoded_charsrlensvrmax_lensegment_offsets timestep_infor5rr;rus|  $         z,AbstractRNNTDecoding.compute_rnnt_timestampsr rnnt_tokenc sd}|jdur%t|jdkr%|jd}t|tr|n|}td|d}t|}t |g|ddf}dd|j dD}t t|D]}t t||D]} ||| d||| <qRqHddt |||D} t tfdd | } | S) a Utility method that calculates the indidual time indices where a token starts and ends. Args: hypothesis: A Hypothesis object that contains `text` field that holds the character / subword token emitted at every time step after rnnt collapse. token_repetitions: A list of ints representing the number of repetitions of each emitted token. rnnt_token: The integer of the rnnt blank token used during rnnt collapse. Returns: rNr[rcSsg|]}|qSr5r5)r9al_logits_labelsr5r5r;r<z9AbstractRNNTDecoding._compute_offsets..cSsg|] \}}}|||dqS)r start_offset end_offsetr5)r9rser5r5r;r< s cs|ddkS)Nrrr5)rr r5r;rr z7AbstractRNNTDecoding._compute_offsets..)rrrritemrnpasarraycumsum concatenaterrrrfilter) rrr  start_indexfirst_timestep end_indices start_indicesalignment_labelsrurr5rr;rrs"  z%AbstractRNNTDecoding._compute_offsetscGs`t|jtjr|j|_t|jtjr|j|_ddt|jd|j|jD}|S)ac Utility method that calculates the indidual time indices where a token starts and ends. Args: hypothesis: A Hypothesis object that contains `text` field that holds the character / subword token emitted at a specific time step considering predicted durations of the previous tokens. Returns: cSs$g|]\}}}|g|||dqSrr5)r9rrdr5r5r;r<*sz=AbstractRNNTDecoding._compute_offsets_tdt..r) rrrrtoken_durationcpurrr)rargsrr5r5r;rqs z)AbstractRNNTDecoding._compute_offsets_tdtcCs||fSrr5)rrrr5r5r;rt0s z'AbstractRNNTDecoding._refine_timestampscCsv|s||fSt|D],\}}|dd|vr6|dkr6||dd||d<|d<|d||d<|d<q ||fS)Nrrr[rr)r)rrrrrr5r5r;rs;s z+AbstractRNNTDecoding._refine_timestamps_tdtrrr+c CsT|rt||s|stjd|dtjdg}g}d}t|D]j\}}|d} |rZ|rZ|d||dd} | |krY|d |||d||ddd | g}|}q!n,| r| d |vsf| |vr|| |r|d |||d|dd g}|d}q!|| q!|r||d} |d || |d dd | |S) a Utility method which constructs segment time stamps out of word time stamps. Args: offsets: A list of dictionaries, each containing "word", "start_offset" and "end_offset". segments_delimiter_tokens: List containing tokens representing the seperator(s) between segments. supported_punctuation: Set containing punctuation marks in the vocabulary. segment_gap_threshold: Number of frames between 2 consecutive words necessary to form segments out of plain text. Returns: A list of dictionaries containing the segment offsets. Each item contains "segment", "start_offset" and "end_offset". z>Specified segment seperators are not in supported punctuation z. If the seperators are not punctuation marks, ignore this warning. Otherwise, specify 'segment_gap_threshold' parameter in decoding config to form segments.)moderrrr[rr&)rrrr) set intersectionrrrONCErrrmclear) rrrr+r  segment_wordsprevious_word_indexrrrgap_between_wordsrr5r5r;rQsf        z)AbstractRNNTDecoding._get_segment_offsetsr)FN)r)NN)%__name__ __module__ __qualname____doc__rr r rdrrboolrrr rrrrrrstrfloatrrrrrrrr staticmethodrrrqrtrsr __classcell__r5r5rr;r#s.l  "N0P       #u ,(   rcseZdZdZfddZdedeefddZdee de fd d Z dee dee fd d Z dee de fd dZ dee dee fddZe  ddeee ee effdeee ee effde deedeee ee efff ddZZS) RNNTDecodingu + Used for performing RNN-T auto-regressive decoding of the Decoder+Joint network given the encoder state. Args: decoding_cfg: A dict-like object which contains the following key-value pairs. strategy: str value which represents the type of decoding that can occur. Possible values are : - greedy, greedy_batch (for greedy decoding). - beam, tsd, alsd (for beam search decoding). compute_hypothesis_token_set: A bool flag, which determines whether to compute a list of decoded tokens as well as the decoded string. Default is False in order to avoid double decoding unless required. preserve_alignments: Bool flag which preserves the history of logprobs generated during decoding (sample / batched). When set to true, the Hypothesis will contain the non-null value for `logprobs` in it. Here, `alignments` is a List of List of Tuple(Tensor (of length V + 1), Tensor(scalar, label after argmax)). In order to obtain this hypothesis, please utilize `rnnt_decoder_predictions_tensor` function with the `return_hypotheses` flag set to True. The length of the list corresponds to the Acoustic Length (T). Each value in the list (Ti) is a torch.Tensor (U), representing 1 or more targets from a vocabulary. U is the number of target tokens for the current timestep Ti. confidence_cfg: A dict-like object which contains the following key-value pairs related to confidence scores. In order to obtain hypotheses with confidence scores, please utilize `rnnt_decoder_predictions_tensor` function with the `preserve_frame_confidence` flag set to True. preserve_frame_confidence: Bool flag which preserves the history of per-frame confidence scores generated during decoding (sample / batched). When set to true, the Hypothesis will contain the non-null value for `frame_confidence` in it. Here, `alignments` is a List of List of floats. The length of the list corresponds to the Acoustic Length (T). Each value in the list (Ti) is a torch.Tensor (U), representing 1 or more confidence scores. U is the number of target tokens for the current timestep Ti. preserve_token_confidence: Bool flag which preserves the history of per-token confidence scores generated during greedy decoding (sample / batched). When set to true, the Hypothesis will contain the non-null value for `token_confidence` in it. Here, `token_confidence` is a List of floats. The length of the list corresponds to the number of recognized tokens. preserve_word_confidence: Bool flag which preserves the history of per-word confidence scores generated during greedy decoding (sample / batched). When set to true, the Hypothesis will contain the non-null value for `word_confidence` in it. Here, `word_confidence` is a List of floats. The length of the list corresponds to the number of recognized words. exclude_blank: Bool flag indicating that blank token confidence scores are to be excluded from the `token_confidence`. aggregation: Which aggregation type to use for collapsing per-token confidence into per-word confidence. Valid options are `mean`, `min`, `max`, `prod`. tdt_include_duration: Bool flag indicating that the duration confidence scores are to be calculated and attached to the regular frame confidence, making TDT frame confidence element a pair: (`prediction_confidence`, `duration_confidence`). method_cfg: A dict-like object which contains the method name and settings to compute per-frame confidence scores. name: The method name (str). Supported values: - 'max_prob' for using the maximum token probability as a confidence. - 'entropy' for using a normalized entropy of a log-likelihood vector. entropy_type: Which type of entropy to use (str). Used if confidence_method_cfg.name is set to `entropy`. Supported values: - 'gibbs' for the (standard) Gibbs entropy. If the alpha (α) is provided, the formula is the following: H_α = -sum_i((p^α_i)*log(p^α_i)). Note that for this entropy, the alpha should comply the following inequality: (log(V)+2-sqrt(log^2(V)+4))/(2*log(V)) <= α <= (1+log(V-1))/log(V-1) where V is the model vocabulary size. - 'tsallis' for the Tsallis entropy with the Boltzmann constant one. Tsallis entropy formula is the following: H_α = 1/(α-1)*(1-sum_i(p^α_i)), where α is a parameter. When α == 1, it works like the Gibbs entropy. More: https://en.wikipedia.org/wiki/Tsallis_entropy - 'renyi' for the Rényi entropy. Rényi entropy formula is the following: H_α = 1/(1-α)*log_2(sum_i(p^α_i)), where α is a parameter. When α == 1, it works like the Gibbs entropy. More: https://en.wikipedia.org/wiki/R%C3%A9nyi_entropy alpha: Power scale for logsoftmax (α for entropies). Here we restrict it to be > 0. When the alpha equals one, scaling is not applied to 'max_prob', and any entropy type behaves like the Shannon entropy: H = -sum_i(p_i*log(p_i)) entropy_norm: A mapping of the entropy value to the interval [0,1]. Supported values: - 'lin' for using the linear mapping. - 'exp' for using exponential mapping with linear shift. The config may further contain the following sub-dictionaries: "greedy": max_symbols: int, describing the maximum number of target tokens to decode per timestep during greedy decoding. Setting to larger values allows longer sentences to be decoded, at the cost of increased execution time. preserve_frame_confidence: Same as above, overrides above value. confidence_method_cfg: Same as above, overrides confidence_cfg.method_cfg. "beam": beam_size: int, defining the beam size for beam search. Must be >= 1. If beam_size == 1, will perform cached greedy search. This might be slightly different results compared to the greedy search above. score_norm: optional bool, whether to normalize the returned beam score in the hypotheses. Set to True by default. return_best_hypothesis: optional bool, whether to return just the best hypothesis or all of the hypotheses after beam search has concluded. This flag is set by default. tsd_max_sym_exp: optional int, determines number of symmetric expansions of the target symbols per timestep of the acoustic model. Larger values will allow longer sentences to be decoded, at increased cost to execution time. alsd_max_target_len: optional int or float, determines the potential maximum target sequence length. If an integer is provided, it can decode sequences of that particular maximum length. If a float is provided, it can decode sequences of int(alsd_max_target_len * seq_len), where seq_len is the length of the acoustic model output (T). NOTE: If a float is provided, it can be greater than 1! By default, a float of 2.0 is used so that a target sequence can be at most twice as long as the acoustic model output length T. maes_num_steps: Number of adaptive steps to take. From the paper, 2 steps is generally sufficient, and can be reduced to 1 to improve decoding speed while sacrificing some accuracy. int > 0. maes_prefix_alpha: Maximum prefix length in prefix search. Must be an integer, and is advised to keep this as 1 in order to reduce expensive beam search cost later. int >= 0. maes_expansion_beta: Maximum number of prefix expansions allowed, in addition to the beam size. Effectively, the number of hypothesis = beam_size + maes_expansion_beta. Must be an int >= 0, and affects the speed of inference since large values will perform large beam search in the next step. maes_expansion_gamma: Float pruning threshold used in the prune-by-value step when computing the expansions. The default (2.3) is selected from the paper. It performs a comparison (max_log_prob - gamma <= log_prob[v]) where v is all vocabulary indices in the Vocab set and max_log_prob is the "most" likely token to be predicted. Gamma therefore provides a margin of additional tokens which can be potential candidates for expansion apart from the "most likely" candidate. Lower values will reduce the number of expansions (by increasing pruning-by-value, thereby improving speed but hurting accuracy). Higher values will increase the number of expansions (by reducing pruning-by-value, thereby reducing speed but potentially improving accuracy). This is a hyper parameter to be experimentally tuned on a validation set. softmax_temperature: Scales the logits of the joint prior to computing log_softmax. decoder: The Decoder/Prediction network module. joint: The Joint network module. vocabulary: The vocabulary (excluding the RNNT blank token) which will be used for decoding. cst|j}ddD}t|dr|jdkrt}tfddttD|_tt|j |||||dt |j t j sGt |j tjrO|j ddSdS) NcS*h|]}|D] }t|dr|qqSP unicodedatacategory startswithr9tokenrr5r5r; Zz(RNNTDecoding.__init__.. model_typetdtcsg|]}||fqSr5r5)r9r vocabularyr5r;r<asz)RNNTDecoding.__init__..rrrrrr)rrgrrAdictr labels_maprcr5rdrrwr r~rrset_decoding_type)rrrrrDrrrrCr;rdQs&  zRNNTDecoding.__init__rrcCs||j|jS)z Implemented by subclass in order to aggregate token confidence to a word-level confidence. Args: hypothesis: Hypothesis Returns: A list of word-level confidence scores. )!_aggregate_token_confidence_charswordsrrrr5r5r;rps z(RNNTDecoding._aggregate_token_confidencercCsd||}|S) Implemented by subclass in order to decoder a token list into a string. Args: tokens: List of int representing the token ids. Returns: A decoded string. )rmrrrrr5r5r;r|s z!RNNTDecoding.decode_tokens_to_strcsfdd|D}|S)rcs&g|]}|jjkrj|qSr5)rrgrGrrr5r;r<s&z5RNNTDecoding.decode_ids_to_tokens..r5rr token_listr5rr;rs z!RNNTDecoding.decode_ids_to_tokenscC|j|}|Sz Compute the most likely language ID (LID) string given the tokens. Args: tokens: List of int representing the token ids. Returns: A decoded LID string.  tokenizer ids_to_langrrlangr5r5r;r z"RNNTDecoding.decode_tokens_to_langcCrQz Decode a token id list into language ID (LID) list. Args: tokens: List of int representing the token ids. Returns: A list of decoded LIDS. rTids_to_text_and_langsrr lang_listr5r5r;rrXz RNNTDecoding.decode_ids_to_langsr&NrrrrcCsHg}d}d}d}d}|D]} | dD]} | |krdnd} |o#| |v} | rd| dkrd|r?|ddkr8|ddn|}|| 7}n$|d} | d | d <| d ddkr[| d dd| d <| d | 7<q| |kru| dkru| d }|| 7}n| dkr|r||||d d}n | d }| d }| }| }qq |dkr||||d |S) a Utility method which constructs word time stamps out of character time stamps. References: This code is a port of the Hugging Face code for word time stamp construction. Args: char_offsets: A list of dictionaries, each containing "char", "start_offset" and "end_offset", where "char" is decoded with the tokenizer. encoded_char_offsets: A list of dictionaries, each containing "char", "start_offset" and "end_offset", where "char" is the original id/ids from the hypotheses (not decoded with the tokenizer). As we are working with char-based models here, we are using the `char_offsets` to get the word offsets. `encoded_char_offsets` is passed for keeping the consistency with `AbstractRNNTDecoding`'s abstract method. word_delimiter_char: Character token that represents the word delimiter. By default, " ". supported_punctuation: Set containing punctuation marks in the vocabulary. Returns: A list of dictionaries containing the word offsets. Each item contains "word", "start_offset" and "end_offset". DELIMITERrMrrWORDrr&Nrrrrrr)stripr)rrrrr last_staterrrrrstatecurr_punctuationlast_built_wordr5r5r;rsB$      *zRNNTDecoding.get_words_offsetsr&N)r,r-r.r/rdrrr2rrr1rrrrr3rr r r rr4r5r5rr;r5sB )         r5csfeZdZdZdeffdd ZedeedefddZ ed ed ede eege ffd d Z d e deefddZdeedefddZdeedeefddZdeedefddZdeedeefddZdee deee efffdd Z  d#deeeeeeffdeeeeeeffd ed eedeeeeeefff d!d"ZZS)$RNNTBPEDecodingu0 Used for performing RNN-T auto-regressive decoding of the Decoder+Joint network given the encoder state. Args: decoding_cfg: A dict-like object which contains the following key-value pairs. strategy: str value which represents the type of decoding that can occur. Possible values are : - greedy, greedy_batch (for greedy decoding). - beam, tsd, alsd (for beam search decoding). compute_hypothesis_token_set: A bool flag, which determines whether to compute a list of decoded tokens as well as the decoded string. Default is False in order to avoid double decoding unless required. preserve_alignments: Bool flag which preserves the history of logprobs generated during decoding (sample / batched). When set to true, the Hypothesis will contain the non-null value for `alignments` in it. Here, `alignments` is a List of List of Tuple(Tensor (of length V + 1), Tensor(scalar, label after argmax)). In order to obtain this hypothesis, please utilize `rnnt_decoder_predictions_tensor` function with the `return_hypotheses` flag set to True. The length of the list corresponds to the Acoustic Length (T). Each value in the list (Ti) is a torch.Tensor (U), representing 1 or more targets from a vocabulary. U is the number of target tokens for the current timestep Ti. compute_timestamps: A bool flag, which determines whether to compute the character/subword, or word based timestamp mapping the output log-probabilities to discrete intervals of timestamps. The timestamps will be available in the returned Hypothesis.timestep as a dictionary. compute_langs: a bool flag, which allows to compute language id (LID) information per token, word, and the entire sample (most likely language id). The LIDS will be available in the returned Hypothesis object as a dictionary rnnt_timestamp_type: A str value, which represents the types of timestamps that should be calculated. Can take the following values - "char" for character/subword time stamps, "word" for word level time stamps and "all" (default), for both character level and word level time stamps. word_seperator: Str token representing the seperator between words. segment_seperators: List containing tokens representing the seperator(s) between segments. segment_gap_threshold: The threshold (in frames) that caps the gap between two words necessary for forming the segments. preserve_frame_confidence: Bool flag which preserves the history of per-frame confidence scores generated during decoding (sample / batched). When set to true, the Hypothesis will contain the non-null value for `frame_confidence` in it. Here, `alignments` is a List of List of ints. confidence_cfg: A dict-like object which contains the following key-value pairs related to confidence scores. In order to obtain hypotheses with confidence scores, please utilize `rnnt_decoder_predictions_tensor` function with the `preserve_frame_confidence` flag set to True. preserve_frame_confidence: Bool flag which preserves the history of per-frame confidence scores generated during decoding (sample / batched). When set to true, the Hypothesis will contain the non-null value for `frame_confidence` in it. Here, `alignments` is a List of List of floats. The length of the list corresponds to the Acoustic Length (T). Each value in the list (Ti) is a torch.Tensor (U), representing 1 or more confidence scores. U is the number of target tokens for the current timestep Ti. preserve_token_confidence: Bool flag which preserves the history of per-token confidence scores generated during greedy decoding (sample / batched). When set to true, the Hypothesis will contain the non-null value for `token_confidence` in it. Here, `token_confidence` is a List of floats. The length of the list corresponds to the number of recognized tokens. preserve_word_confidence: Bool flag which preserves the history of per-word confidence scores generated during greedy decoding (sample / batched). When set to true, the Hypothesis will contain the non-null value for `word_confidence` in it. Here, `word_confidence` is a List of floats. The length of the list corresponds to the number of recognized words. exclude_blank: Bool flag indicating that blank token confidence scores are to be excluded from the `token_confidence`. aggregation: Which aggregation type to use for collapsing per-token confidence into per-word confidence. Valid options are `mean`, `min`, `max`, `prod`. tdt_include_duration: Bool flag indicating that the duration confidence scores are to be calculated and attached to the regular frame confidence, making TDT frame confidence element a pair: (`prediction_confidence`, `duration_confidence`). method_cfg: A dict-like object which contains the method name and settings to compute per-frame confidence scores. name: The method name (str). Supported values: - 'max_prob' for using the maximum token probability as a confidence. - 'entropy' for using a normalized entropy of a log-likelihood vector. entropy_type: Which type of entropy to use (str). Used if confidence_method_cfg.name is set to `entropy`. Supported values: - 'gibbs' for the (standard) Gibbs entropy. If the alpha (α) is provided, the formula is the following: H_α = -sum_i((p^α_i)*log(p^α_i)). Note that for this entropy, the alpha should comply the following inequality: (log(V)+2-sqrt(log^2(V)+4))/(2*log(V)) <= α <= (1+log(V-1))/log(V-1) where V is the model vocabulary size. - 'tsallis' for the Tsallis entropy with the Boltzmann constant one. Tsallis entropy formula is the following: H_α = 1/(α-1)*(1-sum_i(p^α_i)), where α is a parameter. When α == 1, it works like the Gibbs entropy. More: https://en.wikipedia.org/wiki/Tsallis_entropy - 'renyi' for the Rényi entropy. Rényi entropy formula is the following: H_α = 1/(1-α)*log_2(sum_i(p^α_i)), where α is a parameter. When α == 1, it works like the Gibbs entropy. More: https://en.wikipedia.org/wiki/R%C3%A9nyi_entropy alpha: Power scale for logsoftmax (α for entropies). Here we restrict it to be > 0. When the alpha equals one, scaling is not applied to 'max_prob', and any entropy type behaves like the Shannon entropy: H = -sum_i(p_i*log(p_i)) entropy_norm: A mapping of the entropy value to the interval [0,1]. Supported values: - 'lin' for using the linear mapping. - 'exp' for using exponential mapping with linear shift. The config may further contain the following sub-dictionaries: "greedy": max_symbols: int, describing the maximum number of target tokens to decode per timestep during greedy decoding. Setting to larger values allows longer sentences to be decoded, at the cost of increased execution time. preserve_frame_confidence: Same as above, overrides above value. confidence_method_cfg: Same as above, overrides confidence_cfg.method_cfg. "beam": beam_size: int, defining the beam size for beam search. Must be >= 1. If beam_size == 1, will perform cached greedy search. This might be slightly different results compared to the greedy search above. score_norm: optional bool, whether to normalize the returned beam score in the hypotheses. Set to True by default. return_best_hypothesis: optional bool, whether to return just the best hypothesis or all of the hypotheses after beam search has concluded. tsd_max_sym_exp: optional int, determines number of symmetric expansions of the target symbols per timestep of the acoustic model. Larger values will allow longer sentences to be decoded, at increased cost to execution time. alsd_max_target_len: optional int or float, determines the potential maximum target sequence length.If an integer is provided, it can decode sequences of that particular maximum length. If a float is provided, it can decode sequences of int(alsd_max_target_len * seq_len), where seq_len is the length of the acoustic model output (T). NOTE: If a float is provided, it can be greater than 1! By default, a float of 2.0 is used so that a target sequence can be at most twice as long as the acoustic model output length T. maes_num_steps: Number of adaptive steps to take. From the paper, 2 steps is generally sufficient, and can be reduced to 1 to improve decoding speed while sacrificing some accuracy. int > 0. maes_prefix_alpha: Maximum prefix length in prefix search. Must be an integer, and is advised to keep this as 1 in order to reduce expensive beam search cost later. int >= 0. maes_expansion_beta: Maximum number of prefix expansions allowed, in addition to the beam size. Effectively, the number of hypothesis = beam_size + maes_expansion_beta. Must be an int >= 0, and affects the speed of inference since large values will perform large beam search in the next step. maes_expansion_gamma: Float pruning threshold used in the prune-by-value step when computing the expansions. The default (2.3) is selected from the paper. It performs a comparison (max_log_prob - gamma <= log_prob[v]) where v is all vocabulary indices in the Vocab set and max_log_prob is the "most" likely token to be predicted. Gamma therefore provides a margin of additional tokens which can be potential candidates for expansion apart from the "most likely" candidate. Lower values will reduce the number of expansions (by increasing pruning-by-value, thereby improving speed but hurting accuracy). Higher values will increase the number of expansions (by reducing pruning-by-value, thereby reducing speed but potentially improving accuracy). This is a hyper parameter to be experimentally tuned on a validation set. softmax_temperature: Scales the logits of the joint prior to computing log_softmax. decoder: The Decoder/Prediction network module. joint: The Joint network module. tokenizer: The tokenizer which will be used for decoding. rTcs|jj}t|dr |j}ndd|jD}t|dr&|jdkr&|jj|j}||_||j|_t t |j |||||dt |j tjsKt |j tjrS|j ddSdS)NrcSr6r7r9r=r5r5r;r?r@z+RNNTBPEDecoding.__init__..rA multiblankrEsubword)rT vocab_sizerrvocabrArgdefine_tokenizer_typetokenizer_typercrgrdrrwr r~rrrH)rrrrrTrrrr5r;rds,  zRNNTBPEDecoding.__init__rDrcCstdd|Dr dSdS)zD Define the tokenizer type based on the vocabulary. css|]}|dVqdS)##Nr<)r9r>r5r5r;rsz8RNNTBPEDecoding.define_tokenizer_type..wpebpe)anyrCr5r5r;rlsz%RNNTBPEDecoding.define_tokenizer_typermrcs,dkr|dkr ddSddSfddS)zk Define the word start condition based on the tokenizer type and word delimiter character. r&rpcSs|o|d S)Nrnror> token_textr5r5r;rr z=RNNTBPEDecoding.define_word_start_condition..cSs||kSrr5rsr5r5r;rcs|kSrr5rsrr5r;rrur5)rmrr5rvr;define_word_start_conditions  z+RNNTBPEDecoding.define_word_start_conditionrcCs||j|j|jS)a" Implemented by subclass in order to reduce token confidence to a word-level confidence. **Note**: Only supports Sentencepiece based tokenizers! Args: hypothesis: Hypothesis Returns: A list of word-level confidence scores. )2_aggregate_token_confidence_subwords_sentencepiecerJrrrKr5r5r;rs z+RNNTBPEDecoding._aggregate_token_confidencercCrQ)rL)rT ids_to_textrNr5r5r;r rXz$RNNTBPEDecoding.decode_tokens_to_strcCrQr)rT ids_to_tokensrOr5r5r;rs z$RNNTBPEDecoding.decode_ids_to_tokenscCrQrRrSrVr5r5r;r&rXz%RNNTBPEDecoding.decode_tokens_to_langcCrQrYrZr\r5r5r;r3rXz#RNNTBPEDecoding.decode_ids_to_langsrcst|}jrGtjtrBtt|D]*}||j}t |t kr&| }fdd|D} |||_ |||_q|Std|S)z Decode a list of hypotheses into a list of strings. Overrides the super() method optionally adding lang information Args: hypotheses_list: List of Hypothesis. Returns: A list of strings. crr5rr8rr5r;r<Wrz5RNNTBPEDecoding.decode_hypothesis..zzIgnoring request for lang output in hypotheses since the model does not use an aggregate tokenizer)rcrr rrTrrrrrrrrlangsr langs_charsrr)rrrrrrrr;r@s    z!RNNTBPEDecoding.decode_hypothesisr&Nrrrc Cs|}g}d}g}||j|}t|D]\} } | dD]} | |jkr%qt| } || gd} || g} |o>| |v}|| | rr|sr|rb||}|rb| |||d|| ddd| | |krq| | | }q|r|s|d}| d|d<|ddd kr|dd d|d<|d| 7<q|s| }| | qqt |dkr|r||}|r| ||dd|ddd|S|dd|dd<|r||}|r| |||d|ddd|S) a Utility method which constructs word time stamps out of sub-word time stamps. **Note**: Only supports Sentencepiece based tokenizers ! Args: char_offsets: A list of dictionaries, each containing "char", "start_offset" and "end_offset", where "char" is decoded with the tokenizer. encoded_char_offsets: A list of dictionaries, each containing "char", "start_offset" and "end_offset", where "char" is the original id/ids from the hypotheses (not decoded with the tokenizer). This is needed for subword tokenization models. word_delimiter_char: Character token that represents the word delimiter. By default, " ". supported_punctuation: Set containing punctuation marks in the vocabulary. Returns: A list of dictionaries containing the word offsets. Each item contains "word", "start_offset" and "end_offset". rrrr[rr`rrr&N) rrwrmrrrrrrarr(r)rrrrrrprevious_token_index built_tokenscondition_for_word_startrrrr>rtrd built_wordrer5r5r;rcsx          8      z!RNNTBPEDecoding.get_words_offsetsrf)r,r-r.r/rrdr3rr1rlrr0rwrr2rrrrrrr rrrr r rr4r5r5rr;rg s6<$   & 'rgc@sLeZdZUdZdZeed<dZeed<dZe ed<dZ e e ed <dZ e e ed <e d d d Zeed<dZe eed<dZe e ed<dZe ed<dZeed<e dd d Ze eeed<dZe eed<dZeed<e ejd Zejed<e dd d Zejed<dZe ed<e e!d Z"e eeed<e e!d Z#e eeed<dS) RNNTDecodingConfigz RNNT Decoding config rnntrAr-rlFrNr!r$cCstSr)rr5r5r5r;rszRNNTDecodingConfig.)default_factoryr>r"r#r r&r%cCsgdS)N)r(r*r)r5r5r5r5r;rrur'r+rrr,cCs tjddS)N)rR)r BeamRNNTInferConfigr5r5r5r;rs r.rQ temperaturerr)$r,r-r.r/rAr1__annotations__rlrr0r!r r$rr>rr"rr#r r%r'rr+rrGreedyBatchedRNNTInferConfigr,r.r rrr2rrrr5r5r5r;rs0        rc@seZdZdZdS)RNNTBPEDecodingConfigz" RNNT BPE Decoding Config N)r,r-r.r/r5r5r5r;rsr).rr6r:abcr dataclassesrrrtypingrrrr r r numpyrr omegaconfr %nemo.collections.asr.parts.submodulesr rr5nemo.collections.asr.parts.utils.asr_confidence_utilsrrsJ     fX<