o }oiq@sddlmZddlmZmZmZddlmZmZm Z ddl Z ddl m Z ddl mZmZmZddlmZmZddlmZmZdd lmZmZdd lmZdd lmZdd lmZGd ddeZ Gddde Z!eGdddZ"dS))abstractmethod) dataclassfield is_dataclass)ListOptionalUnionN) OmegaConf) AEDBeamInferAEDBeamInferConfigTransformerAEDBeamInfer)AEDGreedyInferConfigTransformerAEDGreedyInfer)ConfidenceConfigConfidenceMixin) HypothesisNBestHypotheses)AggregateTokenizer) TokenizerSpec)loggingcsZeZdZdZdejjdejjdeffdd Zde ddfd d Z d%d ej dej de ej de de eedeeeeeeff ddZdeedeeeeffddZdeedeefddZdedeefddZedeede fddZedeedee fdd Zedeede fd!d"Zedeedee fd#d$ZZS)&AbstractMultiTaskDecodinguo Used for performing AED auto-regressive decoding of the Multi task model given the encoder state. Args: decoding_cfg: A dict-like object which contains the following key-value pairs. strategy: str value which represents the type of decoding that can occur. Possible values are : - greedy, greedy_batch (for greedy decoding). - beam, tsd, alsd (for beam search decoding). compute_langs: a bool flag, which allows to compute language id (LID) information per token, word, and the entire sample (most likely language id). The LIDS will be available in the returned Hypothesis object as a dictionary compute_hypothesis_token_set: A bool flag, which determines whether to compute a list of decoded tokens as well as the decoded string. Default is False in order to avoid double decoding unless required. preserve_alignments: Bool flag which preserves the history of logprobs generated during decoding (sample / batched). When set to true, the Hypothesis will contain the non-null value for `alignments` in it. Here, `alignments` is a List of List of Tuple(Tensor (of length V + 1), Tensor(scalar, label after argmax)). In order to obtain this hypothesis, please utilize `rnnt_decoder_predictions_tensor` function with the `return_hypotheses` flag set to True. confidence_cfg: A dict-like object which contains the following key-value pairs related to confidence scores. preserve_frame_confidence: Bool flag which preserves the history of per-frame confidence scores generated during greedy decoding. When set to true, the Hypothesis will contain the non-null value for `frame_confidence` in it. Here, `frame_confidence` is a List of tensor floats. preserve_token_confidence: Bool flag which preserves the history of per-token confidence scores generated during greedy decoding. When set to true, the Hypothesis will contain the non-null value for `token_confidence` in it. Here, `token_confidence` is a List of tensor floats. The length of the list corresponds to the number of recognized tokens. preserve_word_confidence: Bool flag which preserves the history of per-word confidence scores generated during greedy decoding. When set to true, the Hypothesis will contain the non-null value for `word_confidence` in it. Here, `word_confidence` is a List of tensor floats. The length of the list corresponds to the number of recognized words. aggregation: Which aggregation type to use for collapsing per-token confidence into per-word confidence. Valid options are `mean`, `min`, `max`, `prod`. method_cfg: A dict-like object which contains settings to compute per-frame confidence scores. name: The method name (str). Supported values: - 'max_prob' for using the maximum token probability as a confidence. - 'entropy' for using a normalized entropy of a log-likelihood vector. entropy_type: Which type of entropy to use (str). Used if confidence_method_cfg.name is set to `entropy`. Supported values: - 'gibbs' for the (standard) Gibbs entropy. If the alpha (α) is provided, the formula is the following: H_α = -sum_i((p^α_i)*log(p^α_i)). Note that for this entropy, the alpha should comply the following inequality: (log(V)+2-sqrt(log^2(V)+4))/(2*log(V)) <= α <= (1+log(V-1))/log(V-1) where V is the model vocabulary size. - 'tsallis' for the Tsallis entropy with the Boltzmann constant one. Tsallis entropy formula is the following: H_α = 1/(α-1)*(1-sum_i(p^α_i)), where α is a parameter. When α == 1, it works like the Gibbs entropy. More: https://en.wikipedia.org/wiki/Tsallis_entropy - 'renyi' for the Rényi entropy. Rényi entropy formula is the following: H_α = 1/(1-α)*log_2(sum_i(p^α_i)), where α is a parameter. When α == 1, it works like the Gibbs entropy. More: https://en.wikipedia.org/wiki/R%C3%A9nyi_entropy alpha: Power scale for logsoftmax (α for entropies). Here we restrict it to be > 0. When the alpha equals one, scaling is not applied to 'max_prob', and any entropy type behaves like the Shannon entropy: H = -sum_i(p_i*log(p_i)) entropy_norm: A mapping of the entropy value to the interval [0,1]. Supported values: - 'lin' for using the linear mapping. - 'exp' for using exponential mapping with linear shift. The config may further contain the following sub-dictionaries: "greedy": temperature: None (disabled) or float, specifying this enables temperature sampling instead of greedy decoding. max_generation_delta: int = -1 # -1 means up to the max length of the decoder preserve_alignments: bool = False (unsupported) "beam": beam_size: int, defining the beam size for beam search. Must be >= 1. If beam_size == 1, will perform cached greedy search. This might be slightly different results compared to the greedy search above. length_penalty: float, length penalty for beam search decoding. Must be >= 0.0. max_generation_delta: int,in case of encoder-decoder generation (e.g. NMT), forbids generated sequences to be longer than the length of source sequences plus max_generation_delta return_best_hypothesis: optional bool, whether to return just the best hypothesis or all of the hypotheses after beam search has concluded. This flag is set by default. transformer_decoder: Transformer decoder module. log_softmax_module: Log Softmax projection module to the vocab size. tokenizer: Aggregate Tokenizer. transformer_decoderlog_softmax_module tokenizercstt|rt|}||_|jdd|_|jdd|_|jdd|_ ||_ ||_ ||_ | |jdd||jjdS)Npreserve_alignments compute_langsFcompute_hypothesis_token_setconfidence_cfg)super__init__rr structuredcfggetrrrrrr_init_confidencechange_strategystrategyself decoding_cfgrrr __class__l/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/asr/parts/submodules/multitask_decoding.pyrs  z"AbstractMultiTaskDecoding.__init__r%returncCsfgd}||vrtd|d||jdur2|dvr%|jjdd|_n |dvr2|jjdd|_|dvrh|jjddurCtd t|j|j|j |jjd d |j|j pX|j |j |jjj |jjjd |_dS|d krt|j|j|j |jjdd|jjj|jjdd|jjd d |jjdd|j|jjdd|jjddd |_dStd|d|)N)greedy greedy_batchbeamz!Decoding strategy must be one of zbut was provided )r.r/rF)r0ngram_lm_modelzZGreedy strategy cannot be used with ngram_lm_model. Use beam strategy with beam=1 instead.max_generation_delta) rrrr2rpreserve_token_confidenceconfidence_method_cfg temperature n_samplesr0 search_typedefaultlength_penaltygreturn_best_hypothesisTngram_lm_alpha) rrrr8 beam_sizer:r2r;rr1r<z5Incorrect decoding strategy provided. Must be one of z but was provided ) ValueErrorrr!r.r"r0rrrrr4preserve_frame_confidencer5r6r7decodingr r=)r'r%possible_strategiesr+r+r,r$sV     z)AbstractMultiTaskDecoding.change_strategyNFencoder_hidden_statesencoder_input_maskdecoder_input_idsreturn_hypothesespartial_hypothesescCst|j||||d}|d}Wdn1swY|}t|dtrTg}g} |D]} | j} || } || d| | q0|rK| Sdd| D} | S||}|rk|jri|j sd|j ri| |}|Sdd|DS)a Decode an encoder output by autoregressive decoding of the Decoder+Joint networks. Args: encoder_output: torch.Tensor of shape [B, D, T]. encoded_lengths: torch.Tensor containing lengths of the padded encoder outputs. Shape [B]. return_hypotheses: bool. If set to True it will return list of Hypothesis or NBestHypotheses Returns: If `return_all_hypothesis` is set: A list[list[Hypothesis]]. Look at rnnt_utils.Hypothesis for more information. If `return_all_hypothesis` is not set: A list[Hypothesis]. List of best hypotheses Look at rnnt_utils.Hypothesis for more information. )rBrCrDrFrNcSsg|] }dd|DqS)cSg|] }t|j|j|jqSr+rscore y_sequencetext.0hr+r+r, zRAbstractMultiTaskDecoding.decode_predictions_tensor...r+)rMhhr+r+r,rOszGAbstractMultiTaskDecoding.decode_predictions_tensor..cSrGr+rHrLr+r+r,rOrP) torchinference_moder@ isinstancern_best_hypothesesdecode_hypothesisappendr?preserve_word_confidencer4compute_confidence)r'rBrCrDrErFhypotheses_listprediction_list hypothesesall_hypotheses nbest_hypn_hyps decoded_hypsall_hypr+r+r,decode_predictions_tensors@       z3AbstractMultiTaskDecoding.decode_predictions_tensorrZcCs^tt|D]&}||j}t|tkr|}||}|jr'||||_ |||_ q|S)z Decode a list of hypotheses into a list of strings. Args: hypotheses_list: List of Hypothesis. Returns: A list of strings. ) rangelenrJtypelisttolistdecode_tokens_to_strrdecode_ids_to_tokenstokensrK)r'rZind prediction hypothesisr+r+r,rVs    z+AbstractMultiTaskDecoding.decode_hypothesiscCs |jr|D]}|||_q|S)aH Compute high-level (per-token and/or per-word) confidence scores for a list of hypotheses. Assumes that `token_confidence` is present in the hypotheses. Args: hypotheses_list: List of Hypothesis. Returns: A list of hypotheses with high-level confidence scores. )rX_aggregate_token_confidenceword_confidence)r'rZhypr+r+r,rY9s z,AbstractMultiTaskDecoding.compute_confidencermcCs||j|j|jS)a" Implemented by subclass in order to reduce token confidence to a word-level confidence. **Note**: Only supports Sentencepiece based tokenizers! Args: hypothesis: Hypothesis Returns: A list of word-level confidence scores. )2_aggregate_token_confidence_subwords_sentencepiecewordstoken_confidencerJ)r'rmr+r+r,rnIs z5AbstractMultiTaskDecoding._aggregate_token_confidencerjcCt)z Implemented by subclass in order to decoder a token id list into a string. Args: tokens: List of int representing the token ids. Returns: A decoded string. NotImplementedErrorr'rjr+r+r,rhYs z.AbstractMultiTaskDecoding.decode_tokens_to_strcCrta& Implemented by subclass in order to decode a token id list into a token list. A token list is the string representation of each token id. Args: tokens: List of int representing the token ids. Returns: A list of decoded tokens. rurwr+r+r,rif z.AbstractMultiTaskDecoding.decode_ids_to_tokenscCrt)z Implemented by subclass in order to compute the most likely language ID (LID) string given the tokens. Args: tokens: List of int representing the token ids. Returns: A decoded LID string. rurwr+r+r,decode_tokens_to_langtryz/AbstractMultiTaskDecoding.decode_tokens_to_langcCrt)z Implemented by subclass in order to decode a token id list into language ID (LID) list. Args: tokens: List of int representing the token ids. Returns: A list of decoded LIDS. rurwr+r+r,decode_ids_to_langsryz-AbstractMultiTaskDecoding.decode_ids_to_langs)NFN)__name__ __module__ __qualname____doc__rRnnModulerrstrr$TensorrboolrrrrbrrVrYfloatrnrintrhrirzr{ __classcell__r+r+r)r,r&sJa9  "F   $rcseZdZdZdejjdejjdeffdd Zde e de fd d Z de e de e fd d Z de e de fd dZde e de e fddZde ede eeefffdd ZZS)MultiTaskDecodingu Used for performing AED auto-regressive decoding of the Multi task model given the encoder state. Args: decoding_cfg: A dict-like object which contains the following key-value pairs. strategy: str value which represents the type of decoding that can occur. Possible values are : - greedy, greedy_batch (for greedy decoding). - beam, tsd, alsd (for beam search decoding). compute_langs: a bool flag, which allows to compute language id (LID) information per token, word, and the entire sample (most likely language id). The LIDS will be available in the returned Hypothesis object as a dictionary compute_hypothesis_token_set: A bool flag, which determines whether to compute a list of decoded tokens as well as the decoded string. Default is False in order to avoid double decoding unless required. preserve_alignments: Bool flag which preserves the history of logprobs generated during decoding (sample / batched). When set to true, the Hypothesis will contain the non-null value for `alignments` in it. Here, `alignments` is a List of List of Tuple(Tensor (of length V + 1), Tensor(scalar, label after argmax)). In order to obtain this hypothesis, please utilize `rnnt_decoder_predictions_tensor` function with the `return_hypotheses` flag set to True. confidence_cfg: A dict-like object which contains the following key-value pairs related to confidence scores. preserve_frame_confidence: Bool flag which preserves the history of per-frame confidence scores generated during greedy decoding. When set to true, the Hypothesis will contain the non-null value for `frame_confidence` in it. Here, `frame_confidence` is a List of tensor floats. preserve_token_confidence: Bool flag which preserves the history of per-token confidence scores generated during greedy decoding. When set to true, the Hypothesis will contain the non-null value for `token_confidence` in it. Here, `token_confidence` is a List of tensor floats. The length of the list corresponds to the number of recognized tokens. preserve_word_confidence: Bool flag which preserves the history of per-word confidence scores generated during greedy decoding. When set to true, the Hypothesis will contain the non-null value for `word_confidence` in it. Here, `word_confidence` is a List of tensor floats. The length of the list corresponds to the number of recognized words. aggregation: Which aggregation type to use for collapsing per-token confidence into per-word confidence. Valid options are `mean`, `min`, `max`, `prod`. method_cfg: A dict-like object which contains settings to compute per-frame confidence scores. name: The method name (str). Supported values: - 'max_prob' for using the maximum token probability as a confidence. - 'entropy' for using a normalized entropy of a log-likelihood vector. entropy_type: Which type of entropy to use (str). Used if confidence_method_cfg.name is set to `entropy`. Supported values: - 'gibbs' for the (standard) Gibbs entropy. If the alpha (α) is provided, the formula is the following: H_α = -sum_i((p^α_i)*log(p^α_i)). Note that for this entropy, the alpha should comply the following inequality: (log(V)+2-sqrt(log^2(V)+4))/(2*log(V)) <= α <= (1+log(V-1))/log(V-1) where V is the model vocabulary size. - 'tsallis' for the Tsallis entropy with the Boltzmann constant one. Tsallis entropy formula is the following: H_α = 1/(α-1)*(1-sum_i(p^α_i)), where α is a parameter. When α == 1, it works like the Gibbs entropy. More: https://en.wikipedia.org/wiki/Tsallis_entropy - 'renyi' for the Rényi entropy. Rényi entropy formula is the following: H_α = 1/(1-α)*log_2(sum_i(p^α_i)), where α is a parameter. When α == 1, it works like the Gibbs entropy. More: https://en.wikipedia.org/wiki/R%C3%A9nyi_entropy alpha: Power scale for logsoftmax (α for entropies). Here we restrict it to be > 0. When the alpha equals one, scaling is not applied to 'max_prob', and any entropy type behaves like the Shannon entropy: H = -sum_i(p_i*log(p_i)) entropy_norm: A mapping of the entropy value to the interval [0,1]. Supported values: - 'lin' for using the linear mapping. - 'exp' for using exponential mapping with linear shift. The config may further contain the following sub-dictionaries: "greedy": temperature: None (disabled) or float, specifying this enables temperature sampling instead of greedy decoding. max_generation_delta: int = -1 # -1 means up to the max length of the decoder preserve_alignments: bool = False (unsupported) "beam": beam_size: int, defining the beam size for beam search. Must be >= 1. If beam_size == 1, will perform cached greedy search. This might be slightly different results compared to the greedy search above. length_penalty: float, length penalty for beam search decoding. Must be >= 0.0. max_generation_delta: int, maximum number of additional target tokens to generate return_best_hypothesis: optional bool, whether to return just the best hypothesis or all of the hypotheses after beam search has concluded. This flag is set by default. transformer_decoder: Transformer decoder module. log_softmax_module: Log Softmax projection module to the vocab size. tokenizer: TokenizerSpec. rrrcs:||_tj||||dt|jtr|jddSdS)N)r(rrrsubword)rrrrTr@r set_decoding_typer&r)r+r,rs zMultiTaskDecoding.__init__rjr-cC|j|}|S)z Implemented by subclass in order to decoder a token list into a string. Args: tokens: List of int representing the token ids. Returns: A decoded string. )r ids_to_text)r'rjrmr+r+r,rh z&MultiTaskDecoding.decode_tokens_to_strcCrrx)r ids_to_tokens)r'rj token_listr+r+r,ris z&MultiTaskDecoding.decode_ids_to_tokenscCr)z Compute the most likely language ID (LID) string given the tokens. Args: tokens: List of int representing the token ids. Returns: A decoded LID string. )r ids_to_lang)r'rjlangr+r+r,rz"rz'MultiTaskDecoding.decode_tokens_to_langcCr)z Decode a token id list into language ID (LID) list. Args: tokens: List of int representing the token ids. Returns: A list of decoded LIDS. )rids_to_text_and_langs)r'rj lang_listr+r+r,r{/rz%MultiTaskDecoding.decode_ids_to_langsrZcst|}|jr>t|jtr9tt|D]!}||j}t |t kr&| }| |||_ ||||_q|Std|S)z Decode a list of hypotheses into a list of strings. Overrides the super() method optionally adding lang information Args: hypotheses_list: List of Hypothesis. Returns: A list of strings. zbIgnoring request for lang output in hypotheses since the model does not use an aggregate tokenizer)rrVrrTrrrcrdrJrerfrgrzlangsr{ langs_charsrwarning)r'rZr\rkrlr)r+r,rV<s    z#MultiTaskDecoding.decode_hypothesis)r|r}r~rrRrrrrrrrrhrirzr{rrrrVrr+r+r)r,rsb  . rc@seZdZUdZeed<dZeed<dZe eed<e ddd Z e ed <dZ eed <e ed Zeed <e d dd Zeed<dZeed<dS)MultiTaskDecodingConfigr0r%FrNrcCstS)N)rr+r+r+r,esz MultiTaskDecodingConfig.)default_factoryrrr.cCs tddS)N)r=)r r+r+r+r,rns g?r6)r|r}r~r%r__annotations__rrrrrrrrr r.r0r r6rr+r+r+r,r[s   r)#abcr dataclassesrrrtypingrrrrR omegaconfr =nemo.collections.asr.parts.submodules.multitask_beam_decodingr r r ?nemo.collections.asr.parts.submodules.multitask_greedy_decodingr r5nemo.collections.asr.parts.utils.asr_confidence_utilsrr+nemo.collections.asr.parts.utils.rnnt_utilsrr6nemo.collections.common.tokenizers.aggregate_tokenizerr1nemo.collections.common.tokenizers.tokenizer_specr nemo.utilsrrrrr+r+r+r,s&     mK