o }oio2@s2ddlZddlZddlZddlmZddlmZmZmZddl m Z m Z m Z m Z mZmZddlZddlZddlmZmZddlmZmZddlmZmZddlmZmZdd lm Z dd l!m"Z"dd l#m$Z$m%Z%d d Z&GdddeZ'Gddde'Z(Gddde'Z)eGdddZ*eGddde*Z+dS)N)abstractmethod) dataclassfield is_dataclass)CallableDictListOptionalSetUnion) DictConfig OmegaConf)ctc_beam_decodingctc_greedy_decoding)ConfidenceConfigConfidenceMixin) HypothesisNBestHypotheses)DummyTokenizer) TokenizerSpec)logging logging_modecCs6tt|j}|j|g|d|||ddS)N)listrangendimpermute)tensor dim_indexall_dimsr f/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/asr/parts/submodules/ctc_decoding.pymove_dimension_to_the_front"s(r"cseZdZdZd:dedeeffdd Z   d;d ej d ej d e d e d e e e e e e ff ddZde e d e d e e e effddZde e d e e fddZede eee eeffde eee eeffdedeed e eee eefff ddZede ed efddZede ed e efddZde ed efdd ZdAbstractCTCDecodingu" Used for performing CTC auto-regressive / non-auto-regressive decoding of the logprobs. Args: decoding_cfg: A dict-like object which contains the following key-value pairs. strategy: str value which represents the type of decoding that can occur. Possible values are : greedy (for greedy decoding). beam (for DeepSpeed KenLM based decoding). compute_timestamps: A bool flag, which determines whether to compute the character/subword, or word based timestamp mapping the output log-probabilities to discrite intervals of timestamps. The timestamps will be available in the returned Hypothesis.timestep as a dictionary. ctc_timestamp_type: A str value, which represents the types of timestamps that should be calculated. Can take the following values - "char" for character/subword time stamps, "word" for word level time stamps and "all" (default), for both character level and word level time stamps. word_seperator: Str token representing the seperator between words. segment_seperators: List containing tokens representing the seperator(s) between segments. segment_gap_threshold: The threshold (in frames) that caps the gap between two words necessary for forming the segments. preserve_alignments: Bool flag which preserves the history of logprobs generated during decoding (sample / batched). When set to true, the Hypothesis will contain the non-null value for `logprobs` in it. Here, `logprobs` is a torch.Tensors. confidence_cfg: A dict-like object which contains the following key-value pairs related to confidence scores. In order to obtain hypotheses with confidence scores, please utilize `ctc_decoder_predictions_tensor` function with the `preserve_frame_confidence` flag set to True. preserve_frame_confidence: Bool flag which preserves the history of per-frame confidence scores generated during decoding. When set to true, the Hypothesis will contain the non-null value for `frame_confidence` in it. Here, `frame_confidence` is a List of floats. preserve_token_confidence: Bool flag which preserves the history of per-token confidence scores generated during greedy decoding (sample / batched). When set to true, the Hypothesis will contain the non-null value for `token_confidence` in it. Here, `token_confidence` is a List of floats. The length of the list corresponds to the number of recognized tokens. preserve_word_confidence: Bool flag which preserves the history of per-word confidence scores generated during greedy decoding (sample / batched). When set to true, the Hypothesis will contain the non-null value for `word_confidence` in it. Here, `word_confidence` is a List of floats. The length of the list corresponds to the number of recognized words. exclude_blank: Bool flag indicating that blank token confidence scores are to be excluded from the `token_confidence`. aggregation: Which aggregation type to use for collapsing per-token confidence into per-word confidence. Valid options are `mean`, `min`, `max`, `prod`. tdt_include_duration: Bool flag indicating that the duration confidence scores are to be calculated and attached to the regular frame confidence, making TDT frame confidence element a pair: (`prediction_confidence`, `duration_confidence`). method_cfg: A dict-like object which contains the method name and settings to compute per-frame confidence scores. name: The method name (str). Supported values: 'max_prob' for using the maximum token probability as a confidence. 'entropy' for using a normalized entropy of a log-likelihood vector. entropy_type: Which type of entropy to use (str). Used if confidence_method_cfg.name is set to `entropy`. Supported values: - 'gibbs' for the (standard) Gibbs entropy. If the alpha (α) is provided, the formula is the following: H_α = -sum_i((p^α_i)*log(p^α_i)). Note that for this entropy, the alpha should comply the following inequality: (log(V)+2-sqrt(log^2(V)+4))/(2*log(V)) <= α <= (1+log(V-1))/log(V-1) where V is the model vocabulary size. - 'tsallis' for the Tsallis entropy with the Boltzmann constant one. Tsallis entropy formula is the following: H_α = 1/(α-1)*(1-sum_i(p^α_i)), where α is a parameter. When α == 1, it works like the Gibbs entropy. More: https://en.wikipedia.org/wiki/Tsallis_entropy - 'renyi' for the Rényi entropy. Rényi entropy formula is the following: H_α = 1/(1-α)*log_2(sum_i(p^α_i)), where α is a parameter. When α == 1, it works like the Gibbs entropy. More: https://en.wikipedia.org/wiki/R%C3%A9nyi_entropy alpha: Power scale for logsoftmax (α for entropies). Here we restrict it to be > 0. When the alpha equals one, scaling is not applied to 'max_prob', and any entropy type behaves like the Shannon entropy: H = -sum_i(p_i*log(p_i)) entropy_norm: A mapping of the entropy value to the interval [0,1]. Supported values: - 'lin' for using the linear mapping. - 'exp' for using exponential mapping with linear shift. batch_dim_index: Index of the batch dimension of ``targets`` and ``predictions`` parameters of ``ctc_decoder_predictions_tensor`` methods. Can be either 0 or 1. The config may further contain the following sub-dictionaries: "greedy": preserve_alignments: Same as above, overrides above value. compute_timestamps: Same as above, overrides above value. preserve_frame_confidence: Same as above, overrides above value. confidence_method_cfg: Same as above, overrides confidence_cfg.method_cfg. "beam": beam_size: int, defining the beam size for beam search. Must be >= 1. If beam_size == 1, will perform cached greedy search. This might be slightly different results compared to the greedy search above. return_best_hypothesis: optional bool, whether to return just the best hypothesis or all of the hypotheses after beam search has concluded. This flag is set by default. ngram_lm_alpha: float, the strength of the Language model on the final score of a token. final_score = acoustic_score + ngram_lm_alpha * lm_score + beam_beta * seq_length. beam_beta: float, the strength of the sequence length penalty on the final score of a token. final_score = acoustic_score + ngram_lm_alpha * lm_score + beam_beta * seq_length. ngram_lm_model: str, path to a KenLM ARPA or .binary file (depending on the strategy chosen). If the path is invalid (file is not found at path), will raise a deferred error at the moment of calculation of beam search, so that users may update / change the decoding strategy to point to the correct file. blank_id: The id of the RNNT blank token. supported_punctuation: Set of punctuation marks in the vocabulary. Nblank_idsupported_punctuationcstt|rt|}t|tst|}t|ddg}|D] }||vr0ti||<q#||_ ||_ ||_ |j dd|_ |j dd|_|j dd|_|j dd|_|j d gd |_|j d d|_gd }|j j|vrtd |d|j j|j dur|j jdvr|j j dd|_ n |j j dd|_ |jdur|j jdvr|j j dd|_n|j jdvr|j j dd|_|j rddd|j D}td|d|_||j dd|js|j jdvr|j j ddrtd|j jd|jdur|j|jO_|j jdvrC|j jjdur.t d|j jj|j j_!|j jj"durCt d|j jj"|j j_#|j jdkr\t$j%|j |j |j|j|j&d|_'dS|j jdkrt$j(|j |j |j|j|j&|j j dd|j j d d!|j j d"d#d$|_'dS|j jd%krt)j*||j j d&d'd(|j j d)d#|j |j|j j d d*|j j d+d!|j j ddd, |_'d|j'_+dS|j jd-kr t)j*||j j d&d'd-|j j d)d#|j |j|j j d d*|j j d+d!|j j dd|j j d.dd/ |_'d|j'_+dS|j jd0krLt)j*||j j d&d'd0|j j d)d#|j |j|j j d d*|j j d+d!|j j dd|j j d1dd2 |_'d|j'_+dS|j jd3krt)j,||j j- d&d'|j j- d4d5|j j- d)d#|j |j|j j- d6d7|j j- d8d|j j- d9d:|j j- d;d*|j j- dd|j j- d?d|j j- d@d|j j- dAddB|_'d|j'_+dS|j jdCkrt)j.||j j d&d'|j j d)d#|j |j|j j d d*|j j d+d!|j j dDdE|j j dd|j j d"d#dF |_'d|j'_+dStdG|dH|j j)INFgreedypreserve_alignmentscompute_timestampsbatch_dim_indexrword_seperator segment_seperators).?!segment_gap_threshold)r& greedy_batchbeam pyctcdecode flashlightwfst beam_batchz!Decoding strategy must be one of z. Given )r&r1)r2|cSsg|]}t|qSr )reescape).0pr r r! sz0AbstractCTCDecoding.__init__..z(\s)()confidence_cfgpreserve_frame_confidencez6Confidence calculation is not supported for strategy ``)r4r5r6r3r2zh`beam_alpha` is deprecated and will be removed in a future release. Please use `ngram_lm_alpha` instead.zh`kenlm_path` is deprecated and will be removed in a future release. Please use `ngram_lm_model` instead.)r$r'r(r?confidence_method_cfgr1ngram_lm_modelngram_lm_alphagallow_cuda_graphsT)r$r'r(r?rArBrCrDr2 beam_sizerdefaultreturn_best_hypothesis? beam_beta) r$rE search_typerGr'r(rCrIrBr3pyctcdecode_cfg) r$rErJrGr'r(rCrIrBrKr4flashlight_cfg) r$rErJrGr'r(rCrIrBrLr5rJriva decoding_modenbestopen_vocabulary_decoding beam_widthg$@ lm_weightdevicecuda arpa_lm_path wfst_lm_pathriva_decoding_cfgk2_decoding_cfg)r$rErJrGr'r(rNrPrQrRrSrUrVrWrXr6beam_thresholdg4@) blank_indexrErGr'r(rCrIrYrBrDz5Incorrect decoding strategy supplied. Must be one of z but was provided )/super__init__rr structured isinstancer create set_structcfgr$r%getr'r(r)r*r,r0strategy ValueErrorr&r2joinr8compilespace_before_punct_pattern_init_confidencer?NotImplementedError beam_alpharwarningrC kenlm_pathrBrGreedyCTCInferrAdecodingGreedyBatchedCTCInferr BeamCTCInferoverride_fold_consecutive_value WfstCTCInferr5BeamBatchedCTCInfer)self decoding_cfgr$r% minimal_cfgitempossible_strategies punct_pattern __class__r r!r\s2                    zAbstractCTCDecoding.__init__TFdecoder_outputsdecoder_lengthsfold_consecutivereturn_hypothesesreturncCst|tjr t||j}t|jdr,|jjdur,tj d|d|jjt j d|jj}t |j||d}|d}Wdn1sFwYt|dt r|jjdkr`d d |D}n4g}|D]/}|j}|||} |jd ur|jd d } tt| D] } || | | | | <q|| qd|r|Sdd |D} | S|jjdkr|} n@|||} |jd ur|r|js|jr|| } n | D] }|jdd|_q|jd d } tt| D] } || | | | | <q|r| Sdd | DS)a Decodes a sequence of labels to words Args: decoder_outputs: An integer torch.Tensor of shape [Batch, Time, {Vocabulary}] (if ``batch_index_dim == 0``) or [Time, Batch] (if ``batch_index_dim == 1``) of integer indices that correspond to the index of some character in the label set. decoder_lengths: Optional tensor of length `Batch` which contains the integer lengths of the sequence in the padded `predictions` tensor. fold_consecutive: Bool, determine whether to perform "ctc collapse", folding consecutive tokens into a single token. return_hypotheses: Bool flag whether to return just the decoding predictions of the model or a Hypothesis object that holds information such as the decoded `text`, the `alignment` of emited by the CTC Model, and the `length` of the sequence (if available). May also contain the log-probabilities of the decoder (if this method is called via transcribe()) Returns: A list of Hypothesis objects containing additional information. rqNztBeam search requires that consecutive ctc tokens are not folded. Overriding provided value of `fold_consecutive` = z to mode)decoder_outputr}rr5cSsg|]}|jqSr )n_best_hypotheses)r:hypr r r!r<szFAbstractCTCDecoding.ctc_decoder_predictions_tensor..Tctc_timestamp_typeallcSsg|] }dd|DqS)cSg|] }t|j|j|jqSr rscore y_sequencetextr:hr r r!r<zQAbstractCTCDecoding.ctc_decoder_predictions_tensor...r )r:hhr r r!r<scSrr rrr r r!r<r)r^torchTensorr"r)hasattrrnrqrinforONCEinference_moderrarcrdecode_hypothesisr(rbrlencompute_ctc_timestampsappendpreserve_word_confidencepreserve_token_confidencecompute_confidencer)rtr|r}r~rhypotheses_listall_hypotheses nbest_hypn_hyps decoded_hypstimestamp_typehyp_idxall_hyp hypothesesrr r r!ctc_decoder_predictions_tensorsl            z2AbstractCTCDecoding.ctc_decoder_predictions_tensorrcCstt|D]}||}|j}|jdkr|jnd}|rt|tkr'|}|dur1|d|}g}g}g} |j} d} d} t |D]6\} }|| ksO| |jkri||jkri| || | | | } | | d} || krv| |jkrv| d7} |} qBt| dkr| dd| g} n!|dur|d|}|||jk}dgt|}dgt|} |j dur||| f}n| |}|||_ q|S)a Decode a list of hypotheses into a list of strings. Args: hypotheses_list: List of Hypothesis. fold_consecutive: Whether to collapse the ctc blank tokens or not. Returns: A list of strings. rNrT)rrrlengthtypernumpytolistr$ enumeraterr(+decode_tokens_to_str_with_strip_punctuationr)rtrr~indr predictionpredictions_lendecoded_prediction token_lengthstoken_repetitionsprevious last_lengthlast_repetitionpidxr; hypothesisr r r!rsL            z%AbstractCTCDecoding.decode_hypothesisc Cs|D]v}t|jtrt|jdkrtd|jd}|jdd|_g}|jrD|j}d}|D]}||}||||||}q.n1|jd} t| dkru| d} | ddt|j gD]} |||j | | | | | 7} q_||_ q|j r|D]}| ||_ q~|S)aI Computes high-level (per-token and/or per-word) confidence scores for a list of hypotheses. Assumes that `frame_confidence` is present in the hypotheses. Args: hypotheses_list: List of Hypothesis. Returns: A list of hypotheses with high-level confidence scores. zWrong format of the `text` attribute of a hypothesis. Expected: (decoded_prediction, token_repetitions) The method invocation is expected between .decode_hypothesis() and .compute_ctc_timestamps()rNrr)r^rtuplerrdexclude_blank_from_confidencenon_blank_frame_confidencer_aggregate_confidenceframe_confidencetoken_confidencer_aggregate_token_confidenceword_confidence) rtrrrrritrjrtstlr r r!rAs6     z&AbstractCTCDecoding.compute_confidence char_offsetsencoded_char_offsetsword_delimiter_charcCt)zL Implemented by subclass in order to get the words offsets. ri)rtrrrr%r r r!get_words_offsetsm z%AbstractCTCDecoding.get_words_offsetstokenscCr)z Implemented by subclass in order to decoder a token id list into a string. Args: tokens: List of int representing the token ids. Returns: A decoded string. rrtrr r r!decode_tokens_to_strzrz(AbstractCTCDecoding.decode_tokens_to_strcCr& Implemented by subclass in order to decode a token id list into a token list. A token list is the string representation of each token id. Args: tokens: List of int representing the token ids. Returns: A list of decoded tokens. rrr r r!decode_ids_to_tokenss z(AbstractCTCDecoding.decode_ids_to_tokenscCs"||}|jr|jd|}|S)zn Decodes a list of tokens to a string and removes a space before supported punctuation marks. z\2)rr%rgsub)rtrrr r r!rs z?AbstractCTCDecoding.decode_tokens_to_str_with_strip_punctuationrrrc Cs|dvsJ|j\}}||_d}}||||j}t|t|jkr9td|d|jdt|dt|jt|}t|jD]\}} || g||d<qC|j |||j d\}}d}|d vrn|j |||j |j d }d} |d vr|j ||j|j |jd } } t|jd kr|j} ng} d| i|_|dur|dvr||jd<|dur|dvr||jd<| dur|d vr| |jd<||j|_|S)a Method to compute time stamps at char/subword, and word level given some hypothesis. Requires the input hypothesis to contain a `text` field that is the tuple. The tuple contains - the ctc collapsed integer ids, and the number of repetitions of each token. Args: hypothesis: A Hypothesis object, with a wrapped `text` field. The `text` field must contain a tuple with two values - The ctc collapsed integer ids A list of integers that represents the number of repetitions per token. timestamp_type: A str value that represents the type of time stamp calculated. Can be one of "char", "word" "segment" or "all" Returns: A Hypothesis object with a modified `timestep` value, which is now a dictionary containing the time stamp information. )charwordsegmentrNz`char_offsets`: z and `processed_tokens`: z9 have to be of the same length, but are: `len(offsets)`: z and `len(processed_tokens)`: r)rrr%)rrr)rrrr%)rr)segment_delimiter_tokensr%r0rtimestep)rr)rrrr)r_compute_offsetsr$rrdcopydeepcopyrr_refine_timestampsr%rr*_get_segment_offsetsr,r0 timestampr) rtrrrrr word_offsetsrrrsegment_offsets timestep_infor r r!rsb         z*AbstractCTCDecoding.compute_ctc_timestampsr ctc_tokencsd}|jdurt|jdkrtd|jdd}t|}t|g|ddf}ddt|j||D}t t fdd|}|S) a Utility method that calculates the indidual time indices where a token starts and ends. Args: hypothesis: A Hypothesis object that contains `text` field that holds the character / subword token emitted at every time step after ctc collapse. token_lengths: A list of ints representing the lengths of each emitted token. ctc_token: The integer of the ctc blank token used during ctc collapse. Returns: rNrcSsg|] \}}}|||dqS))r start_offset end_offsetr )r:tser r r!r<s z8AbstractCTCDecoding._compute_offsets..cs |dkS)Nrr )offsetsrr r! z6AbstractCTCDecoding._compute_offsets..) rrmaxnpasarraycumsum concatenateziprrfilter)rrr start_index end_indices start_indicesrr rr!rs z$AbstractCTCDecoding._compute_offsetscCs^|s||fSt|D] \}}|dr*|dd|vr*|dkr*|d||d<|d<q ||fS)Nrrrr)r)rrr%roffsetr r r!r"s z&AbstractCTCDecoding._refine_timestampsrrr0c CsT|rt||s|stjd|dtjdg}g}d}t|D]j\}}|d} |rZ|rZ|d||dd} | |krY|d |||d||ddd | g}|}q!n,| r| d |vsf| |vr|| |r|d |||d|dd g}|d}q!|| q!|r||d} |d || |d dd | |S) a Utility method which constructs segment time stamps out of word time stamps. Args: offsets: A list of dictionaries, each containing "word", "start_offset" and "end_offset". segments_delimiter_tokens: List containing tokens representing the seperator(s) between segments. supported_punctuation: Set containing punctuation marks in the vocabulary. segment_gap_threshold: Number of frames between 2 consecutive words necessary to form segments out of plain text. Returns: A list of dictionaries containing the segment offsets. Each item contains "segment", "start_offset" and "end_offset". z>Specified segment seperators are not in supported punctuation z. If the seperators are not punctuation marks, ignore this warning. Otherwise, specify 'segment_gap_threshold' parameter in decoding config to form segments.rrrrrrr+)rrrr) set intersectionrrkrrrrreclear) rrr%r0r segment_wordsprevious_word_indexrrrgap_between_wordsrr r r!r6sf        z(AbstractCTCDecoding._get_segment_offsetscC|jSN)_preserve_alignmentsrtr r r!r'z'AbstractCTCDecoding.preserve_alignmentscC ||_t|dr||j_dSdSNrn)rrrnr'rtvaluer r r!r'  cCrr)_compute_timestampsrr r r!r(rz&AbstractCTCDecoding.compute_timestampscCrr)r rrnr(rr r r!r(r cCrr)_preserve_frame_confidencerr r r!r?rz-AbstractCTCDecoding.preserve_frame_confidencecCrr)r rrnr?rr r r!r?r r)NTF)r)NN)%__name__ __module__ __qualname____doc__intr r r\rrboolr rrrrrrrrstrfloatrrrrr staticmethodrrrpropertyr'setterr(r? __classcell__r r rzr!r#'s"J i I,     ^ $  V     r#cseZdZdZfddZdedeefddZdee de fd d Z dee dee fd d Z e ddeee ee effdeee ee effde deedeee ee efff ddZZS) CTCDecodingu" Used for performing CTC auto-regressive / non-auto-regressive decoding of the logprobs for character based models. Args: decoding_cfg: A dict-like object which contains the following key-value pairs. strategy: str value which represents the type of decoding that can occur. Possible values are : - greedy (for greedy decoding). - beam (for DeepSpeed KenLM based decoding). compute_timestamps: A bool flag, which determines whether to compute the character/subword, or word based timestamp mapping the output log-probabilities to discrite intervals of timestamps. The timestamps will be available in the returned Hypothesis.timestep as a dictionary. ctc_timestamp_type: A str value, which represents the types of timestamps that should be calculated. Can take the following values - "char" for character/subword time stamps, "word" for word level time stamps and "all" (default), for both character level and word level time stamps. word_seperator: Str token representing the seperator between words. segment_seperators: List containing tokens representing the seperator(s) between segments. segment_gap_threshold: The threshold (in frames) that caps the gap between two words necessary for forming the segments. preserve_alignments: Bool flag which preserves the history of logprobs generated during decoding (sample / batched). When set to true, the Hypothesis will contain the non-null value for `logprobs` in it. Here, `logprobs` is a torch.Tensors. confidence_cfg: A dict-like object which contains the following key-value pairs related to confidence scores. In order to obtain hypotheses with confidence scores, please utilize `ctc_decoder_predictions_tensor` function with the `preserve_frame_confidence` flag set to True. preserve_frame_confidence: Bool flag which preserves the history of per-frame confidence scores generated during decoding. When set to true, the Hypothesis will contain the non-null value for `frame_confidence` in it. Here, `frame_confidence` is a List of floats. preserve_token_confidence: Bool flag which preserves the history of per-token confidence scores generated during greedy decoding (sample / batched). When set to true, the Hypothesis will contain the non-null value for `token_confidence` in it. Here, `token_confidence` is a List of floats. The length of the list corresponds to the number of recognized tokens. preserve_word_confidence: Bool flag which preserves the history of per-word confidence scores generated during greedy decoding (sample / batched). When set to true, the Hypothesis will contain the non-null value for `word_confidence` in it. Here, `word_confidence` is a List of floats. The length of the list corresponds to the number of recognized words. exclude_blank: Bool flag indicating that blank token confidence scores are to be excluded from the `token_confidence`. aggregation: Which aggregation type to use for collapsing per-token confidence into per-word confidence. Valid options are `mean`, `min`, `max`, `prod`. tdt_include_duration: Bool flag indicating that the duration confidence scores are to be calculated and attached to the regular frame confidence, making TDT frame confidence element a pair: (`prediction_confidence`, `duration_confidence`). method_cfg: A dict-like object which contains the method name and settings to compute per-frame confidence scores. name: The method name (str). Supported values: - 'max_prob' for using the maximum token probability as a confidence. - 'entropy' for using a normalized entropy of a log-likelihood vector. entropy_type: Which type of entropy to use (str). Used if confidence_method_cfg.name is set to `entropy`. Supported values: - 'gibbs' for the (standard) Gibbs entropy. If the alpha (α) is provided, the formula is the following: H_α = -sum_i((p^α_i)*log(p^α_i)). Note that for this entropy, the alpha should comply the following inequality: (log(V)+2-sqrt(log^2(V)+4))/(2*log(V)) <= α <= (1+log(V-1))/log(V-1) where V is the model vocabulary size. - 'tsallis' for the Tsallis entropy with the Boltzmann constant one. Tsallis entropy formula is the following: H_α = 1/(α-1)*(1-sum_i(p^α_i)), where α is a parameter. When α == 1, it works like the Gibbs entropy. More: https://en.wikipedia.org/wiki/Tsallis_entropy - 'renyi' for the Rényi entropy. Rényi entropy formula is the following: H_α = 1/(1-α)*log_2(sum_i(p^α_i)), where α is a parameter. When α == 1, it works like the Gibbs entropy. More: https://en.wikipedia.org/wiki/R%C3%A9nyi_entropy alpha: Power scale for logsoftmax (α for entropies). Here we restrict it to be > 0. When the alpha equals one, scaling is not applied to 'max_prob', and any entropy type behaves like the Shannon entropy: H = -sum_i(p_i*log(p_i)) entropy_norm: A mapping of the entropy value to the interval [0,1]. Supported values: - 'lin' for using the linear mapping. - 'exp' for using exponential mapping with linear shift. batch_dim_index: Index of the batch dimension of ``targets`` and ``predictions`` parameters of ``ctc_decoder_predictions_tensor`` methods. Can be either 0 or 1. The config may further contain the following sub-dictionaries: "greedy": preserve_alignments: Same as above, overrides above value. compute_timestamps: Same as above, overrides above value. preserve_frame_confidence: Same as above, overrides above value. confidence_method_cfg: Same as above, overrides confidence_cfg.method_cfg. "beam": beam_size: int, defining the beam size for beam search. Must be >= 1. If beam_size == 1, will perform cached greedy search. This might be slightly different results compared to the greedy search above. return_best_hypothesis: optional bool, whether to return just the best hypothesis or all of the hypotheses after beam search has concluded. This flag is set by default. ngram_lm_alpha: float, the strength of the Language model on the final score of a token. final_score = acoustic_score + ngram_lm_alpha * lm_score + beam_beta * seq_length. beam_beta: float, the strength of the sequence length penalty on the final score of a token. final_score = acoustic_score + ngram_lm_alpha * lm_score + beam_beta * seq_length. ngram_lm_model: str, path to a KenLM ARPA or .binary file (depending on the strategy chosen). If the path is invalid (file is not found at path), will raise a deferred error at the moment of calculation of beam search, so that users may update / change the decoding strategy to point to the correct file. blank_id: The id of the RNNT blank token. cs~t}|_tfddttD|_ddD}tj|||dt|jt j r=|j |j|j ddSdS)Ncsg|]}||fqSr r )r:r vocabularyr r!r<Wsz(CTCDecoding.__init__..cS*h|]}|D] }t|dr|qqSP unicodedatacategory startswithr:tokenrr r r! Yz'CTCDecoding.__init__..rur$r%r) rrdictr labels_mapr[r\r^rnrAbstractBeamCTCInferset_vocabularyset_decoding_type)rtrurr$r%rzrr!r\Ps zCTCDecoding.__init__rrcCs|||jd|jS)z Implemented by subclass in order to aggregate token confidence to a word-level confidence. Args: hypothesis: Hypothesis Returns: A list of word-level confidence scores. r)!_aggregate_token_confidence_charsrrsplitrrtrr r r!rds z'CTCDecoding._aggregate_token_confidencercCsd||}|S) Implemented by subclass in order to decoder a token list into a string. Args: tokens: List of int representing the token ids. Returns: A decoded string. )rerrtrrr r r!rrs z CTCDecoding.decode_tokens_to_strcsfdd|D}|S)rcs g|] }|jkrj|qSr )r$r()r:crr r!r<s z4CTCDecoding.decode_ids_to_tokens..r rtr token_listr rr!rs z CTCDecoding.decode_ids_to_tokensr+Nrrrr%cCsg}d}d}d}d}t|D]]\} } | d} | |krdnd} | t|dkr.|| ddnd} | r;| |vo9| |k}nd}| d krD|rDq| |krQ| d }|| 7}n| dkr_||||d n | d }| d }| }| }q|dkry||||d |S) a Utility method which constructs word time stamps out of character time stamps. References: This code is a port of the Hugging Face code for word time stamp construction. Args: char_offsets: A list of dictionaries, each containing "char", "start_offset" and "end_offset", where "char" is decoded with the tokenizer. encoded_char_offsets: A list of dictionaries, each containing "char", "start_offset" and "end_offset", where "char" is the original id/ids from the hypotheses (not decoded with the tokenizer). As we are working with char-based models here, we are using the `char_offsets` to get the word offsets. `encoded_char_offsets` is passed for keeping the consistency with `AbstractRNNTDecoding`'s abstract method. word_delimiter_char: Character token that represents the word delimiter. By default, " ". supported_punctuation: Set containing punctuation marks in the vocabulary. Returns: A list of dictionaries containing the word offsets. Each item contains "word", "start_offset" and "end_offset". DELIMITERr0rrWORDrNFr+rrrrr)rrr)rrrr%r last_staterrrrrrstate next_charnext_punctuationr r r!rs4$  zCTCDecoding.get_words_offsetsr+N)r r rrr\rrrrrrrrrrr r r rrr r rzr!rs* ! rcseZdZdZdeffdd ZedeedefddZ ed ed ede eege ffd d Z d e deefddZdeedefddZdeedeefddZ  ddeeeeeeffdeeeeeeffd edeedeeeeeefff ddZZS)CTCBPEDecodingu! Used for performing CTC auto-regressive / non-auto-regressive decoding of the logprobs for subword based models. Args: decoding_cfg: A dict-like object which contains the following key-value pairs. strategy: str value which represents the type of decoding that can occur. Possible values are : - greedy (for greedy decoding). - beam (for DeepSpeed KenLM based decoding). compute_timestamps: A bool flag, which determines whether to compute the character/subword, or word based timestamp mapping the output log-probabilities to discrite intervals of timestamps. The timestamps will be available in the returned Hypothesis.timestep as a dictionary. ctc_timestamp_type: A str value, which represents the types of timestamps that should be calculated. Can take the following values - "char" for character/subword time stamps, "word" for word level time stamps and "all" (default), for both character level and word level time stamps. word_seperator: Str token representing the seperator between words. preserve_alignments: Bool flag which preserves the history of logprobs generated during decoding (sample / batched). When set to true, the Hypothesis will contain the non-null value for `logprobs` in it. Here, `logprobs` is a torch.Tensors. confidence_cfg: A dict-like object which contains the following key-value pairs related to confidence scores. In order to obtain hypotheses with confidence scores, please utilize `ctc_decoder_predictions_tensor` function with the `preserve_frame_confidence` flag set to True. preserve_frame_confidence: Bool flag which preserves the history of per-frame confidence scores generated during decoding. When set to true, the Hypothesis will contain the non-null value for `frame_confidence` in it. Here, `frame_confidence` is a List of floats. preserve_token_confidence: Bool flag which preserves the history of per-token confidence scores generated during greedy decoding (sample / batched). When set to true, the Hypothesis will contain the non-null value for `token_confidence` in it. Here, `token_confidence` is a List of floats. The length of the list corresponds to the number of recognized tokens. preserve_word_confidence: Bool flag which preserves the history of per-word confidence scores generated during greedy decoding (sample / batched). When set to true, the Hypothesis will contain the non-null value for `word_confidence` in it. Here, `word_confidence` is a List of floats. The length of the list corresponds to the number of recognized words. exclude_blank: Bool flag indicating that blank token confidence scores are to be excluded from the `token_confidence`. aggregation: Which aggregation type to use for collapsing per-token confidence into per-word confidence. Valid options are `mean`, `min`, `max`, `prod`. tdt_include_duration: Bool flag indicating that the duration confidence scores are to be calculated and attached to the regular frame confidence, making TDT frame confidence element a pair: (`prediction_confidence`, `duration_confidence`). method_cfg: A dict-like object which contains the method name and settings to compute per-frame confidence scores. name: The method name (str). Supported values: - 'max_prob' for using the maximum token probability as a confidence. - 'entropy' for using a normalized entropy of a log-likelihood vector. entropy_type: Which type of entropy to use (str). Used if confidence_method_cfg.name is set to `entropy`. Supported values: - 'gibbs' for the (standard) Gibbs entropy. If the alpha (α) is provided, the formula is the following: H_α = -sum_i((p^α_i)*log(p^α_i)). Note that for this entropy, the alpha should comply the following inequality: (log(V)+2-sqrt(log^2(V)+4))/(2*log(V)) <= α <= (1+log(V-1))/log(V-1) where V is the model vocabulary size. - 'tsallis' for the Tsallis entropy with the Boltzmann constant one. Tsallis entropy formula is the following: H_α = 1/(α-1)*(1-sum_i(p^α_i)), where α is a parameter. When α == 1, it works like the Gibbs entropy. More: https://en.wikipedia.org/wiki/Tsallis_entropy - 'renyi' for the Rényi entropy. Rényi entropy formula is the following: H_α = 1/(1-α)*log_2(sum_i(p^α_i)), where α is a parameter. When α == 1, it works like the Gibbs entropy. More: https://en.wikipedia.org/wiki/R%C3%A9nyi_entropy alpha: Power scale for logsoftmax (α for entropies). Here we restrict it to be > 0. When the alpha equals one, scaling is not applied to 'max_prob', and any entropy type behaves like the Shannon entropy: H = -sum_i(p_i*log(p_i)) entropy_norm: A mapping of the entropy value to the interval [0,1]. Supported values: - 'lin' for using the linear mapping. - 'exp' for using exponential mapping with linear shift. batch_dim_index: Index of the batch dimension of ``targets`` and ``predictions`` parameters of ``ctc_decoder_predictions_tensor`` methods. Can be either 0 or 1. The config may further contain the following sub-dictionaries: "greedy": preserve_alignments: Same as above, overrides above value. compute_timestamps: Same as above, overrides above value. preserve_frame_confidence: Same as above, overrides above value. confidence_method_cfg: Same as above, overrides confidence_cfg.method_cfg. "beam": beam_size: int, defining the beam size for beam search. Must be >= 1. If beam_size == 1, will perform cached greedy search. This might be slightly different results compared to the greedy search above. return_best_hypothesis: optional bool, whether to return just the best hypothesis or all of the hypotheses after beam search has concluded. This flag is set by default. ngram_lm_alpha: float, the strength of the Language model on the final score of a token. final_score = acoustic_score + ngram_lm_alpha * lm_score + beam_beta * seq_length. beam_beta: float, the strength of the sequence length penalty on the final score of a token. final_score = acoustic_score + ngram_lm_alpha * lm_score + beam_beta * seq_length. ngram_lm_model: str, path to a KenLM ARPA or .binary file (depending on the strategy chosen). If the path is invalid (file is not found at path), will raise a deferred error at the moment of calculation of beam search, so that users may update / change the decoding strategy to point to the correct file. tokenizer: NeMo tokenizer object, which inherits from TokenizerSpec. tokenizercs|jj}||_||j|_t|dr|j}ndd|jD}tj|||dt |j t j rft|jjdrY|jj }t |jjtrF|}nt|}|j ||j |ntd|j ddSdS)Nr%cSrrrr"r r r!r$yr%z*CTCBPEDecoding.__init__..r& get_vocabz3Could not resolve the vocabulary of the tokenizer !subword)r> vocab_sizedefine_tokenizer_typevocabtokenizer_typerr%r[r\r^rnrr)r?rrkeysr* set_tokenizerrrkr+)rtrur>r$r% vocab_dictrCrzr r!r\qs(     zCTCBPEDecoding.__init__rrcCstdd|Dr dSdS)zD Define the tokenizer type based on the vocabulary. css|]}|dVqdS)##Nr!)r:r#r r r! sz7CTCBPEDecoding.define_tokenizer_type..wpebpe)anyrr r r!rBsz$CTCBPEDecoding.define_tokenizer_typerDrcs,dkr|dkr ddSddSfddS)zk Define the word start condition based on the tokenizer type and word delimiter character. r+rKcSs|o|d S)NrHrIr# token_textr r r!rsz.cSs||kSrr rNr r r!rcs|kSrr rNrr r!rrPr )rDrr rQr!define_word_start_conditions  z*CTCBPEDecoding.define_word_start_conditionrcCs&|||jd|j|jdS)a% Implemented by subclass in order to aggregate token confidence to a word-level confidence. **Note**: Only supports Sentencepiece based tokenizers! Args: hypothesis: Hypothesis Returns: A list of word-level confidence scores. r)2_aggregate_token_confidence_subwords_sentencepiecerrr-rr.r r r!rs z*CTCBPEDecoding._aggregate_token_confidencercC|j|}|S)r/)r> ids_to_textr1r r r!rs z#CTCBPEDecoding.decode_tokens_to_strcCrTr)r> ids_to_tokensr3r r r!rs z#CTCBPEDecoding.decode_ids_to_tokensr+Nrrr%cCs|}g}d}g}||j|}t|D]\} } | d} || gd} || g} |o2| |v}|| | rf|sf|rV||}|rV||||d|| ddd|| |kre|| | }q|r|s|d}| d|d<|ddd kr|dd d|d<|d| 7<q|s| }|| qt |dkr|r||}|r|||dd|ddd|S|dd|dd<|r||}|r||||d|ddd|S) a Utility method which constructs word time stamps out of sub-word time stamps. **Note**: Only supports Sentencepiece based tokenizers ! Args: char_offsets: A list of dictionaries, each containing "char", "start_offset" and "end_offset", where "char" is decoded with the tokenizer. encoded_char_offsets: A list of dictionaries, each containing "char", "start_offset" and "end_offset", where "char" is the original id/ids from the hypotheses (not decoded with the tokenizer). This is needed for subword tokenization models. word_delimiter_char: Character token that represents the word delimiter. By default, " ". supported_punctuation: Set containing punctuation marks in the vocabulary. Returns: A list of dictionaries containing the word offsets. Each item contains "word", "start_offset" and "end_offset". rrrrrr7rrr+N) rrRrDrrrstriprrr)rtrrrr%rprevious_token_index built_tokenscondition_for_word_startrrrr#rOcurr_punctuation built_wordlast_built_wordr r r!rsp              z CTCBPEDecoding.get_words_offsetsr<)r r rrrr\rrrrBrrrRrrrrrrrr r r rrr r rzr!r=s0$  r=c@seZdZUdZeed<dZeeed<dZ eeed<dZ eed<e dd d Z ee eed <dZeeed <d Zeed<dZeed<e dd d Zejed<e dd d Zejed<e dd d Zejed<e dd d Zeed<dZeed<dS)CTCDecodingConfigr1rcNr'r(r+r*cCsgdS)N)r-r/r.r r r r r!rWrPzCTCDecodingConfig.)default_factoryr,r0rrrr)cCstSr)rGreedyCTCInferConfigr r r r!rdrPr&cC tjddSN)rE)rBeamCTCInferConfigr r r r!rirr2cCrarb)rWfstCTCInferConfigr r r r!rnrr5cCstSr)rr r r r!rrsr>rH temperature)r r rrcr__annotations__r'r rr(r*rr,rr0rrr)r&rr`r2rrdr5rer>rrfrr r r r!r^Is(    r^c@s eZdZdS)CTCBPEDecodingConfigN)r r rr r r r!rhxsrh),rr8rabcr dataclassesrrrtypingrrrr r r rrr omegaconfr r %nemo.collections.asr.parts.submodulesrr5nemo.collections.asr.parts.utils.asr_confidence_utilsrr+nemo.collections.asr.parts.utils.rnnt_utilsrr6nemo.collections.common.tokenizers.aggregate_tokenizerr1nemo.collections.common.tokenizers.tokenizer_specr nemo.utilsrrr"r#rr=r^rhr r r r!sD    )u.