o irN@sddlmZddlZddlmZddlmZddlm Z m Z m Z m Z m Z mZddlZddlmZmZmZmZmZmZmZmZ m!Z"m#Z$ddl%m&Z'm(Z)m*Z+ddl,m-Z-zdd l.m/Z0Wne1ywzdd lm/Z0Wn e1ytdZ0YnwYnwgd Z2ed gd Z3d dZ4ddZ5Gddde Z6GdddeZ7GdddeZ8GdddZ9dddddddde:dddddd fdBd:d;Z;dCd>d?Z;sz#_construct_trie..) index_size_Triestartitemsr&scoreinsertsmear _SmearingModeMAX)r* word_dictr"r$silence vocab_sizetrie start_stateword spellingsword_idx_r1spelling spelling_idxr%r)r+_construct_trie2s    rAcsvd}|dur t|}|r|durt|}|S|s9|dur9t|tkr9fddtD}|gg||<t|}|S)Ncs"i|] }||ggqSr%) get_entry)r'ir)r%r+ Is"z"_get_word_dict..) _Dictionary_create_word_dicttypestrranger-)r"r$lm_dictr*unk_wordr6dr%r)r+_get_word_dictAs  rMc@s8eZdZUdZded< ded< ded< ded <d S) rzORepresents hypothesis generated by CTC beam search decoder :class:`CTCDecoder`.torch.LongTensorr#z List[str]wordsfloatr1torch.IntTensor timestepsN)__name__ __module__ __qualname____doc____annotations__r%r%r%r+rPs rcs>eZdZdZed fdd Zdfdd Zdd d ZZS)rzLanguage model state.returnDict[int, CTCDecoderLMState]cstjS)zMap of indices to LM states)superchildrenself __class__r%r+r[hszCTCDecoderLMState.children usr_indexintcs t|S)a!Returns child corresponding to usr_index, or creates and returns a new state if input index is not found. Args: usr_index (int): index corresponding to child state Returns: CTCDecoderLMState: child state corresponding to usr_index )rZchild)r]r`r^r%r+rbms zCTCDecoderLMState.childstatecCsdS)zCompare two language model states. Args: state (CTCDecoderLMState): LM state to compare against Returns: int: 0 if the states are the same, -1 if self is less, +1 if self is greater. Nr%r]rcr%r%r+compareys zCTCDecoderLMState.compare)rXrY)r`rarXr)rcrrXr) rSrTrUrVpropertyr[rbre __classcell__r%r%r^r+res  rc@s:eZdZdZedddZedd d ZedddZdS)rzVLanguage model base class for creating custom language models to use with the decoder.start_with_nothingboolrXrcCt)zInitialize or reset the language model. Args: start_with_nothing (bool): whether or not to start sentence with sil token. Returns: CTCDecoderLMState: starting state NotImplementedError)r]rhr%r%r+r/s zCTCDecoderLM.startrc usr_token_idxraTuple[CTCDecoderLMState, float]cCrj)axEvaluate the language model based on the current LM state and new word. Args: state (CTCDecoderLMState): current LM state usr_token_idx (int): index of the word Returns: (CTCDecoderLMState, float) CTCDecoderLMState: new LM state float: score rk)r]rcrmr%r%r+r1szCTCDecoderLM.scorecCrj)a8Evaluate end for language model based on current LM state. Args: state (CTCDecoderLMState): current LM state Returns: (CTCDecoderLMState, float) CTCDecoderLMState: new LM state float: score rkrdr%r%r+finishszCTCDecoderLM.finishN)rhrirXr)rcrrmrarXrn)rcrrXrn)rSrTrUrVrr/r1ror%r%r%r+rs  rc@steZdZdZd2ddZd3ddZd4ddZddZddZd5d"d#Z d6d%d&Z d6d'd(Z )d7d8d-d.Z d9d0d1Z d)S):rzCTC beam search decoder from *Flashlight* :cite:`kahn2022flashlight`. .. devices:: CPU Note: To build the decoder, please use the factory function :func:`ctc_decoder`. nbestrar"Optional[Dict]r6rEr*r$rdecoder_options9Union[_LexiconDecoderOptions, _LexiconFreeDecoderOptions] blank_tokenrH sil_tokenrKrXNonec  Cs||_||_||_|j||_|j|} g} |r7t||||| } || } d} t|| || |j| | | |_n t||| |j| |_||_ dS)a Args: nbest (int): number of best decodings to return lexicon (Dict or None): lexicon mapping of words to spellings, or None for lexicon-free decoder word_dict (_Dictionary): dictionary of words tokens_dict (_Dictionary): dictionary of tokens lm (CTCDecoderLM): language model. If using a lexicon, only word level LMs are currently supported decoder_options (_LexiconDecoderOptions or _LexiconFreeDecoderOptions): parameters used for beam search decoding blank_token (str): token corresopnding to blank sil_token (str): token corresponding to silence unk_word (str): word corresponding to unknown FN) rpr6r*r&blankrA_LexiconDecoderdecoder_LexiconFreeDecoderr$)r]rpr"r6r*r$rrrtrurKr7 transitionsr9token_lmr%r%r+__init__s,   zCTCDecoder.__init__idxsrQrNcs4ddt|D}tfdd|}tt|S)Ncss|]}|dVqdS)rNr%)r'gr%r%r+ sz)CTCDecoder._get_tokens..cs |jkSN)rw)xr\r%r+s z(CTCDecoder._get_tokens..)itgroupbyfiltertorch LongTensorlistr]r~r%r\r+ _get_tokensszCTCDecoder._get_tokenscCsNg}t|D]\}}||jkrq|dks|||dkr!||qt|S)z8Returns frame numbers corresponding to non-blank tokens.r) enumeraterwappendr IntTensor)r]r~rRrCidxr%r%r+_get_timestepss   zCTCDecoder._get_timestepscC|jdS)aInitialize the internal state of the decoder. See :py:meth:`decode_step` for the usage. .. note:: This method is required only when performing online decoding. It is not necessary when performing batch decoding with :py:meth:`__call__`. N)ry decode_beginr\r%r%r+r zCTCDecoder.decode_begincCr)aFinalize the internal state of the decoder. See :py:meth:`decode_step` for the usage. .. note:: This method is required only when performing online decoding. It is not necessary when performing batch decoding with :py:meth:`__call__`. N)ry decode_endr\r%r%r+rrzCTCDecoder.decode_end emissionstorch.FloatTensorcCsp|jtjkr td|jstd|std|jdkr&td|j| \}}|j | ||dS)aPerform incremental decoding on top of the curent internal state. .. note:: This method is required only when performing online decoding. It is not necessary when performing batch decoding with :py:meth:`__call__`. Args: emissions (torch.FloatTensor): CPU tensor of shape `(frame, num_tokens)` storing sequences of probability distribution over labels; output of acoustic model. Example: >>> decoder = torchaudio.models.decoder.ctc_decoder(...) >>> decoder.decode_begin() >>> decoder.decode_step(emission1) >>> decoder.decode_step(emission2) >>> decoder.decode_end() >>> result = decoder.get_final_hypothesis() emissions must be float32.emissions must be a CPU tensor.emissions must be contiguous.zemissions must be 2D. Found N) dtyperfloat32 ValueErroris_cpu RuntimeError is_contiguousndimshapesizery decode_stepdata_ptr)r]rTNr%r%r+r s   zCTCDecoder.decode_stepList[CTCHypothesis]cfdd|DS)Nc s>g|]}t|jfdd|jD|j|jdqS)cs g|] }|dkrj|qS)r)r6rB)r'rr\r%r+r,Gs z2CTCDecoder._to_hypo...)r#rOr1rR)rrr#rOr1r)r'resultr\r%r+r,Ds  z'CTCDecoder._to_hypo..r%r]resultsr%r\r+_to_hypoCs zCTCDecoder._to_hypocCs|j}||d|jS)a9Get the final hypothesis Returns: List[CTCHypothesis]: List of sorted best hypotheses. .. note:: This method is required only when performing online decoding. It is not necessary when performing batch decoding with :py:meth:`__call__`. N)ryget_all_final_hypothesisrrprr%r%r+get_final_hypothesisNs zCTCDecoder.get_final_hypothesisNlengthsOptional[torch.Tensor]List[List[CTCHypothesis]]c Cs|jtjkr td|jstd|std|jdkr&td|j|dur1|js1td| \}}}|durCt |f|}d}g}t |D]&}| ||| d } |j| |||} ||| d|jqK|S) a Performs batched offline decoding. .. note:: This method performs offline decoding in one go. To perform incremental decoding, please refer to :py:meth:`decode_step`. Args: emissions (torch.FloatTensor): CPU tensor of shape `(batch, frame, num_tokens)` storing sequences of probability distribution over labels; output of acoustic model. lengths (Tensor or None, optional): CPU tensor of shape `(batch, )` storing the valid length of in time axis of the output Tensor in each batch. Returns: List[List[CTCHypothesis]]: List of sorted best hypotheses for each audio sequence in the batch. rrrzemissions must be 3D. Found Nzlengths must be a CPU tensor.r)rrrrrrrrrrfullrIrstriderydecoderrrp) r]rrBrr float_byteshyposb emissions_ptrrr%r%r+__call__]s(   zCTCDecoder.__call__rcr)z Map raw token IDs into corresponding tokens Args: idxs (LongTensor): raw token IDs generated from decoder Returns: List: tokens corresponding to the input IDs csg|] }j|qSr%)r*rBitem)r'rr\r%r+r,sz-CTCDecoder.idxs_to_tokens..r%rr%r\r+idxs_to_tokenss zCTCDecoder.idxs_to_tokens)rprar"rqr6rEr*rEr$rrrrsrtrHrurHrKrHrXrv)r~rQrXrN)r~rQrXrQ)rr)rXrr)rrrrrXr)r~rNrXr)rSrTrUrVr}rrrrrrrrrr%r%r%r+rs  9   #  2rr2rz-infF-|zr" Optional[str]r#Union[str, List[str]]r$Union[str, CTCDecoderLM]rJrpra beam_sizebeam_size_token Optional[int]beam_thresholdrP lm_weight word_score unk_score sil_scorelog_addrirtrHrurKrXc Cs|durt|turtdt|}|r+t|}t||p|||| | | | tjd }nt ||p2|||| | tjd}t |||||}t|tkrWt durQt dt ||}n|dur^t }t||||||| ||d S)aY Builds an instance of :class:`CTCDecoder`. Args: lexicon (str or None): lexicon file containing the possible words and corresponding spellings. Each line consists of a word and its space separated spelling. If `None`, uses lexicon-free decoding. tokens (str or List[str]): file or list containing valid tokens. If using a file, the expected format is for tokens mapping to the same index to be on the same line lm (str, CTCDecoderLM, or None, optional): either a path containing KenLM language model, custom language model of type `CTCDecoderLM`, or `None` if not using a language model lm_dict (str or None, optional): file consisting of the dictionary used for the LM, with a word per line sorted by LM index. If decoding with a lexicon, entries in lm_dict must also occur in the lexicon file. If `None`, dictionary for LM is constructed using the lexicon file. (Default: None) nbest (int, optional): number of best decodings to return (Default: 1) beam_size (int, optional): max number of hypos to hold after each decode step (Default: 50) beam_size_token (int, optional): max number of tokens to consider at each decode step. If `None`, it is set to the total number of tokens (Default: None) beam_threshold (float, optional): threshold for pruning hypothesis (Default: 50) lm_weight (float, optional): weight of language model (Default: 2) word_score (float, optional): word insertion score (Default: 0) unk_score (float, optional): unknown word insertion score (Default: -inf) sil_score (float, optional): silence insertion score (Default: 0) log_add (bool, optional): whether or not to use logadd when merging hypotheses (Default: False) blank_token (str, optional): token corresponding to blank (Default: "-") sil_token (str, optional): token corresponding to silence (Default: "|") unk_word (str, optional): word corresponding to unknown (Default: "") Returns: CTCDecoder: decoder Example >>> decoder = ctc_decoder( >>> lexicon="lexicon.txt", >>> tokens="tokens.txt", >>> lm="kenlm.bin", >>> ) >>> results = decoder(emissions) # List of shape (B, nbest) of Hypotheses Nz!lm_dict must be None or str type.) rrrrrrrrcriterion_type)rrrrrrrzflashlight-text is installed, but KenLM is not installed. Please refer to https://github.com/kpu/kenlm#python-module for how to install it.) rpr"r6r*r$rrrtrurK)rGrHrrE _load_words_LexiconDecoderOptionsr-_CriterionTypeCTC_LexiconFreeDecoderOptionsrM_KenLMr_ZeroLMr)r"r#r$rJrprrrrrrrrrtrurKr*rrr6r%r%r+rsZ9     rmodel_PretrainedFilescCsP|dvr t|dd|}t|d|d|dkr$|ddSddS) N) librispeechzlibrispeech-3-gramzlibrispeech-4-gramzZ not supported. Must be one of ['librispeech-3-gram', 'librispeech-4-gram', 'librispeech']zdecoder-assets/z /lexicon.txtz /tokens.txtrz/lm.binr!)rr)rprefixr%r%r+_get_filenames s rcCsDt|}t|j}t|j}|jdurt|j}nd}t|||dS)aM Retrieves pretrained data files used for :func:`ctc_decoder`. Args: model (str): pretrained language model to download. Valid values are: ``"librispeech-3-gram"``, ``"librispeech-4-gram"`` and ``"librispeech"``. Returns: Object with the following attributes * ``lm``: path corresponding to downloaded language model, or ``None`` if the model is not associated with an lm * ``lexicon``: path corresponding to downloaded lexicon file * ``tokens``: path corresponding to downloaded tokens file Nr!)rrr"r#r$r)rfiles lexicon_file tokens_filelm_filer%r%r+rs    r)"r"rr#rr$rrJrrprarrarrrrPrrPrrPrrPrrPrrirtrHrurHrKrHrXr)rrHrXr)> __future__r itertoolsrabcr collectionsrtypingrrrrr r rflashlight.lib.text.decoderr rr rxr rrrzrrr_LMr_LMStaterr4rr.rrflashlight.lib.text.dictionaryrrFrrErrtorchaudio.utilsr!flashlight.lib.text.decoder.kenlmrr Exception__all__rrArMrrrrrPrrrr%r%r%r+sZ    0     1j q