o }oi3@sddlZddlZddlZddlZddlZddlmZmZddlm Z ddl Z ddl m Z m Z mZddl mZddlmmmZddlmZddlmZddlmZdd lmZdd lmZdd lm Z m!Z!Gd d d eZ"GdddeZ#GdddeZ$Gddde$Z%dS)N)ABCabstractmethod)List) DictConfig OmegaConf open_dict)Tensor)ASRAdapterModelMixin)StreamingEncoder)asr_module_utils) Hypothesis) tokenizers) app_stateloggingc@seZdZdZdZdefddZdefddZdefdd Zddefd d Z d dZ ddZ de fddZ dede fddZddZd S) ASRBPEMixina9ASR BPE Mixin class that sets up a Tokenizer via a config This mixin class adds the method `_setup_tokenizer(...)`, which can be used by ASR models which depend on subword tokenization. The setup_tokenizer method adds the following parameters to the class - - tokenizer_cfg: The resolved config supplied to the tokenizer (with `dir` and `type` arguments). - tokenizer_dir: The directory path to the tokenizer vocabulary + additional metadata. - tokenizer_type: The type of the tokenizer. Currently supports `bpe` and `wpe`, as well as `agg`. - vocab_path: Resolved path to the vocabulary text file. In addition to these variables, the method will also instantiate and preserve a tokenizer (subclass of TokenizerSpec) if successful, and assign it to self.tokenizer. The mixin also supports aggregate tokenizers, which consist of ordinary, monolingual tokenizers. If a conversion between a monolongual and an aggregate tokenizer (or vice versa) is detected, all registered artifacts will be cleaned up. langs tokenizer_cfgcCsH|d}|dur td|dkr||n|||dS)Ntypez`tokenizer.type` cannot be Noneagg)get ValueErrorlower_setup_aggregate_tokenizer_setup_monolingual_tokenizer_derive_tokenizer_properties)selfrtokenizer_typer\/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/asr/parts/mixins/mixins.py_setup_tokenizer;s     zASRBPEMixin._setup_tokenizerc sddtjvr dtjd<tj|dd|_|jd|_|jd|_|jdi|_ | t |drfd |j vrf|j|j j _|j|j j _d|vrft|j j |d|j j _Wdn1sawY|jd vrotd |jd kr?d |jvr|jd }ntj|jd}|d|}||_d|jvr|jd}|durtdd|jvr||dd|d|_ ntj|d|_ d|jvr|jd}ntj|jd}|d|}||_zd|jvr|jd}ntj|jd}|d|}||_Wn tyd|_Ynwit|j jD]}|j |g}|d}|d|<qfdd}t!|j j _||j j _"|j j#|j j _$ncd|jvrL|jd}ntj|jd}|d|}||_d|jvri|jdtj%d |j|j d!d|j d"d|j d#d|j d$d|j d%d|j d&d|j d'd|j d(d)d* |_ t&'d+(|j j)j*|j jdS),NTOKENIZERS_PARALLELISMfalseTresolvedirr hf_kwargscfg tokenizerbpewpezc`tokenizer.type` must be either `bpe` for SentencePiece tokenizer or `wpe` for BERT based tokenizerr) model_pathtokenizer.modeltokenizer.model_pathspecial_tokensL`special_tokens` are no longer supported for SentencePiece based tokenizers.custom_tokenizer_target_)r1r+r+ vocab_path vocab.txttokenizer.vocab_pathspe_tokenizer_vocabtokenizer.vocabtokenizer.spe_tokenizer_vocabrcSNrr vocabularyrr get_vocabz;ASRBPEMixin._setup_monolingual_tokenizer..get_vocabbert-base-cased mask_token bos_token eos_token pad_token sep_token cls_token unk_tokenuse_fastF pretrained_model_name vocab_filerArBrCrDrErFrGrH'Tokenizer {} initialized with {} tokens)+osenvironr to_containerrpop tokenizer_dirrrhf_tokenizer_kwargs1_cleanup_aggregate_config_and_artifacts_if_neededhasattrr&r'r$rrrr%rpathjoinregister_artifactr+from_config_dictr SentencePieceTokenizerr3spe_vocab_pathFileNotFoundErrorrange vocab_size ids_to_tokenslenr>special_token_to_idall_special_tokens AutoTokenizerrinfoformat __class____name__) rrr+r.r3rZipiecer>rr<rrFs                                z(ASRBPEMixin._setup_monolingual_tokenizerc Csdtjvr dtjd<tj|dd|_d|_|jdd|jd|_i|_ |jdit d| t |d rHd |jvrH|j|jj_i}|j|jD]S\}}|||\}}}}|||<t |d rt|jj.|j|j|d|jj|j|d<|j|j|d|jj|j|d<Wdn1swYqRd |vr||d d |d |_dSt||_dS)Nr r!Tr"r$rr%z1_setup_tokenizer: detected an aggregate tokenizerr&r'r0r1)r1r )rMrNrrOrrQrPrrrRrrcA_cleanup_monolingual_and_aggregate_config_and_artifacts_if_neededrTr&r'r AGGREGATE_TOKENIZERS_DICT_PREFIXitems_make_tokenizerrrXr AggregateTokenizer) rrtokenizers_dictlangtokenizer_configr'r+r3rZrrrrsX       z&ASRBPEMixin._setup_aggregate_tokenizerNcs|d}|d}|dvrtdd}d}d}|dkrd|vr(|d}ntj|d}|d|jd |d |}d |vrN|d }|durNtd tj |d } d|vr^|d}ntj|d}|d|jd |d|}z"d|vr|d}ntj|d}|d|jd |d|}Wn t yd}Ynwit | j D]} | | g} | d} | d| <qfdd} t| j_ | | j_| j| j_nbd|vr|d}ntj|d}|d|jd |d|}d|vr|d|di} tj| dd|| dd| dd| dd| dd| dd| d d| d!d| d"d#d$ } td%| jj| j | |||fS)&Nrr$r(zb`tokenizer.type` must be either `bpe` for SentencePiece tokenizer or`wpe` for BERT based tokenizerr)r+r, tokenizer..z .model_pathr.r/r2r3r4z .vocab_pathr6r7z.spe_tokenizer_vocabrr9cr:r;rrr<rrr>2r?z.ASRBPEMixin._make_tokenizer..get_vocabr%rJr@rArBrCrDrErFrGrHFrIrL)rrrrMrUrVrWrjr rYr[r\r]r^r_r'r>r`rarPrbrrcrdrerf)rrrorrQr+r3rZr.r'rgrhr>rRrr<rrls                        zASRBPEMixin._make_tokenizercCst|drCt|jj0|jjdd|jjdd|jjdd|jjdd|jjddWdn1s>wYt|drw|jd d|jd d|jd dt|jD]}|d |j d rv|j|qddSdS)a Clean ups any monolingual and some aggregate config items and artifacts. We need to do this when we switch from a monolingual tokenizer to an aggregate one or go between aggregate tokenizers which could have a different number of languages r&r$Nr+r3r6r% artifactsr-r5r8rqrr) rTrr&r'rPrslistkeys startswithrjrakeyrrrri^s$   zMASRBPEMixin._cleanup_monolingual_and_aggregate_config_and_artifacts_if_neededcCst|dr$t|jj|jj|jdWdn1swYt|drCt|jD]}| d|jdrB|j|q0dSdS)z Clean ups any aggregate config items and artifacts. We need to do this when we switch from an aggregate tokenizer to a monolingual one r&Nrsrqrr) rTrr&r'rPrjrtrsrurvrwrrrrSxs   z=ASRBPEMixin._cleanup_aggregate_config_and_artifacts_if_needed directorycCslt|ds td|jdkr,|jjD]}|jjj|}tj ||}| ||qdS| |jj|dS)z Save the model tokenizer(s) to the specified directory. Args: directory: The directory to save the tokenizer(s) to. r&z}The model has not been initialized with a tokenizer yet. Please call the model's __init__ and _setup_tokenizer methods first.rN) rT RuntimeErrorrr'rr&rrMrUrV_extract_tokenizer_from_config)rryro subconfignew_dirrrrsave_tokenizerss   zASRBPEMixin.save_tokenizersr$c stj|s tj|ddg}|D]5\}}t|tr3tj|r3t||}t d|d|t|trH| drH|dd| qt |dkrt d |d |t|d sctd t}||jj}|durvtd d} z t|| } | Wn tjyd} Ynwt|| } |D]Ufdd| D} | D]C} | | || jddd}t |dkrd|}n|d}ttj|| jtj||t ddtj||qqdSdS)a Extracts the tokenizer from the config and write the objects to dir. The file may be from a local path (new model init) or from a .nemo file (restored model). If its from a newly initialized model, the file is copied to dir. If its from a restored model, the file is extracted from the .nemo file and copied to dir. Args: tokenizer_cfg: The tokenizer config to extract the tokenizer from. dir: The directory to write the tokenizer objects to. T)exist_okzSaved z at znemo:Nrz+Copying the following nemo file objects to z: model_guidzqThe model does not have a model_guid attribute. Please ensure that the model has been restored from a .nemo file.zsThe model has not been restored from a .nemo file. Cannot extract the tokenizer as the nemo file cannot be located.zr:zr:gzcsg|] }|jvr|qSr)name).0xnemo_object_namerr sz>ASRBPEMixin._extract_tokenizer_from_config.._r9) rMrUexistsmakedirsrk isinstancestrshutilcopy2rrcrvappendr_debugrTrrAppStateget_model_metadata_from_guidrrestoration_pathtarfileopenclose ReadError getmembersextractrsplitrVrename)rrr$nemo_file_objectskvlocappstate restore_path tar_headertar_testtarmembersmembernew_namerrrr{sX            ""z*ASRBPEMixin._extract_tokenizer_from_configcCs@|jj}dd|D}t||j_dd|D}||j_dS)NcSs&h|]}tdd|Dr|qS)css|]}|VqdSr;)isupper)rcharrrr szEASRBPEMixin._derive_tokenizer_properties...)anystrip)rtokenrrr s&z;ASRBPEMixin._derive_tokenizer_properties..cSs*h|]}|D] }t|dr|qqS)P) unicodedatacategoryrv)rrrrrrrs*)r'r>boolsupports_capitalizationsupported_punctuation)rvocabcapitalized_tokens punctuationrrrrs   z(ASRBPEMixin._derive_tokenizer_propertiesr;)rf __module__ __qualname____doc__rjrrrrrlrirSrr~r{rrrrrr$s u6m Frc@seZdZdZd'dedefddZ d(ded eedefd d Z d'd edefd dZ          d)de de de de de dedee de dededefddZ e    d*deeded ed!ed"ef d#d$Z d+deedefd%d&ZdS),ASRModuleMixinaQ ASRModuleMixin is a mixin class added to ASR models in order to add methods that are specific to a particular instantiation of a module inside of an ASRModel. Each method should first check that the module is present within the subclass, and support additional functionality if the corresponding module is present. Tcontext_window update_configcCstj|||ddS)a Update the context window of the SqueezeExcitation module if the provided model contains an `encoder` which is an instance of `ConvASREncoder`. Args: context_window: An integer representing the number of input timeframes that will be used to compute the context. Each timeframe corresponds to a single window stride of the STFT features. Say the window_stride = 0.01s, then a context window of 128 represents 128 * 0.01 s of context to compute the Squeeze step. update_config: Whether to update the config or not with the new context window. )rrN)r !change_conv_asr_se_context_window)rrrrrrrs z0ASRModuleMixin.change_conv_asr_se_context_windowNself_attention_modelatt_context_sizecCs|dur |dur dSt|dstddSt|jds#tddS|j||||j|rQt|j||jj_||jj_ WddS1sJwYdSdS)a Update the self_attention_model if function is available in encoder. Args: self_attention_model (str): type of the attention layer and positional encoding 'rel_pos': relative positional embedding and Transformer-XL 'rel_pos_local_attn': relative positional embedding and Transformer-XL with local attention using overlapping windows. Attention context is determined by att_context_size parameter. 'abs_pos': absolute positional embedding and Transformer If None is provided, the self_attention_model isn't changed. Defauts to None. att_context_size (List[int]): List of 2 ints corresponding to left and right attention context sizes, or None to keep as it is. Defauts to None. update_config (bool): Whether to update the config or not with the new attention model. Defaults to True. NencoderzCould not change the self_attention_model in encoder since the model provided does not contain an `encoder` module in its config.change_attention_modelz;Model encoder doesn't have a change_attention_model method ) rTrrcrrdevicerr&rr)rrrrrrrrs"      "z%ASRModuleMixin.change_attention_model subsampling_conv_chunking_factorcCst|ds tddSt|jdstddS|j||r>t|j||jj_WddS1s7wYdSdS)a Update the conv_chunking_factor (int) if function is available in encoder. Default is 1 (auto) Set it to -1 (disabled) or to a specific value (power of 2) if you OOM in the conv subsampling layers Args: conv_chunking_factor (int) rzCould not call the change_subsampling_conv_chunking_factor method in encoder since the model provided does not contain an `encoder` module in its config.N'change_subsampling_conv_chunking_factorzLModel encoder doesn't have a change_subsampling_conv_chunking_factor method )rTrrcrrrr&r)rrrrrrr4s      "z6ASRModuleMixin.change_subsampling_conv_chunking_factorFprocessed_signalprocessed_signal_lengthcache_last_channelcache_last_timecache_last_channel_lenkeep_all_outputsprevious_hypothesesprevious_pred_outdrop_extra_pre_encodedreturn_transcriptionreturn_log_probsc  Cs t|tjst|tjstdt|dt|jts tdt|tjr/| dur/t dt|tjs>| dur>t d|jj ||||||| d\} } }}}t|tjsat|tj r|j d krt |d rm|j}|j}n|j}|j}|| d }|jd dd }g}| rg}nd}t|D]T\}}| dur||}n ||d| |f}|durtj|||fd d}| |t||7<n|}||| r|j|d| ||ddd}||dqd}n|jj| | d|d}dd|D}|}||||||g}| r |||| t|S)a6 It simulates a forward step with caching for streaming purposes. It supports the ASR models where their encoder supports streaming like Conformer. Args: processed_signal: the input audio signals processed_signal_length: the length of the audios cache_last_channel: the cache tensor for last channel layers like MHA cache_last_channel_len: lengths for cache_last_channel cache_last_time: the cache tensor for last time layers like convolutions keep_all_outputs: if set to True, would not drop the extra outputs specified by encoder.streaming_cfg.valid_out_len previous_hypotheses: the hypotheses from the previous step for RNNT models previous_pred_out: the predicted outputs from the previous step for CTC models drop_extra_pre_encoded: number of steps to drop from the beginning of the outputs after the downsampling module. This can be used if extra paddings are added on the left side of the input. return_transcription: whether to decode and return the transcriptions. It can not get disabled for Transducer models. return_log_probs: whether to return the log probs, only valid for ctc model Returns: greedy_predictions: the greedy predictions from the decoder all_hyp_or_transcribed_texts: the decoder hypotheses for Transducer models and the transcriptions for CTC models cache_last_channel_next: the updated tensor cache for last channel layers to be used for next streaming step cache_last_time_next: the updated tensor cache for last time layers to be used for next streaming step cache_last_channel_next_len: the updated lengths for cache_last_channel best_hyp: the best hypotheses for the Transducer models log_probs: the logits tensor of current streaming chunk, only returned when return_log_probs=True encoded_len: the length of the output log_probs + history chunk log_probs, only returned when return_log_probs=True zstream_step does not support !1Encoder of this model does not support streaming!Fzfreturn_transcription can not be False for Transducer models as decoder returns the transcriptions too.Tz1return_log_probs can only be True for CTC models.)rrrrrrrctc ctc_decoder)encoder_output)dimkeepdimN)rrr9)decoder_outputsdecoder_lengthsreturn_hypotheses)rencoded_lengthsrpartial_hypothesescSsg|]}|jqSr) y_sequence)rhyprrrrsz8ASRModuleMixin.conformer_stream_step..)r asr_modelsEncDecRNNTModelEncDecCTCModelNotImplementedErrorrrr rrccache_aware_stream_stepEncDecHybridRNNTCTCModel cur_decoderrT ctc_decodingrdecodingdecoderargmax enumeratetorchcatr_rctc_decoder_predictions_tensor unsqueezernnt_decoder_predictions_tensortuple)rrrrrrrrrrrrencoded encoded_lencache_last_channel_nextcache_last_time_nextcache_last_channel_next_lenrr log_probspredictions_tensorgreedy_predictionsall_hyp_or_transcribed_texts preds_idxpreds preds_curgreedy_predictions_concat decoded_outbest_hypresultrrrconformer_stream_stepPs(          z$ASRModuleMixin.conformer_stream_steppaths2audio_files batch_sizelogprobsronline_normalizationcCs |dus t|dkr iS|r|rtdt|tjs$tdt|dt|jts.td| |||}g}g}|D]} t | } t| j }|jj |d\} } } d}d}d}d}g}t | D]W\}\}}|dkrm|jjjnd}t:|j||| | | | |||d|p|d }|s|r|\}}} } } }}}||n|\}}} } } }Wdn1swYq^|s|rtj|d d }t||D] \}}||d|q|dur|d g|7}q;||7}q;|r|S|s|Sg}t||D]\}}|t||d ddq|S)a Args: paths2audio_files: (a list) of paths to audio files. batch_size: (int) batch size to use during inference. Bigger will result in better throughput performance but would use more memory. logprobs: (bool) pass True to get log probabilities instead of transcripts. return_hypotheses: (bool) Either return hypotheses or text With hypotheses can do some postprocessing like getting timestamp or rescoring online_normalization: (bool) Perform normalization on the run per chunk. Returns: A list of transcriptions (or raw log probabilities if logprobs is True) in the same order as paths2audio_files NrzuEither `return_hypotheses` or `logprobs` can be True at any given time.Returned hypotheses will contain the logprobs.z$simulate streaming does not support rr)rT) rrrrrrrrrrrr9)axis)rtextscore dec_state)r_rrrrrrrr &_setup_streaming_transcribe_dataloaderiterstreams_lengthget_initial_cache_stater streaming_cfgrrinference_moderis_buffer_emptyrcpurzipr )rrrrrr data_loadertotal_log_probs total_textsstreaming_bufferstreaming_buffer_iterrrrrpred_out_streamrtranscribed_textsbatch_log_probsstep_num chunk_audio chunk_lengthsrrcur_chunk_log_probsr log_prob_lenhypsr rrr)transcribe_simulate_cache_aware_streamings        $ z8ASRModuleMixin.transcribe_simulate_cache_aware_streamingc csddlm}|||d}t|D]?\}}|j|dd\}} } td||d|dks7|t|dkrPtd|t|dd |d |V|qd S) a Setup function for a temporary data loader which wraps the provided audio file. Args: paths2audio_files: (a list) of paths to audio files. batch_size: (int) batch size to use during inference. Bigger will result in better throughput performance but would use more memory. online_normalization: whether to do online normalization Returns: a new batch streaming buffer r)CacheAwareStreamingAudioBuffer)modelrr) stream_idz!Added this sample to the buffer: r9zStarting to stream samples z to z...N)0nemo.collections.asr.parts.utils.streaming_utilsr&rappend_audio_filerrcr_ reset_buffer) rrrrr&r sample_idxsamplerrr(rrrrJs    $z5ASRModuleMixin._setup_streaming_transcribe_dataloader)T)NNT) NNNNTNNNTF)rFFF)F)rfrrrintrrrrrrrr rrno_gradr%rrrrrrs .       qrc@seZdZeddZdS)VerificationMixincCsjt|ddd$}|D]}|}|ddddd}|t|d q WddS1s.wYdS) a Takes paths to audio files and manifest filepath and creates manifest file with the audios Args: paths2audio_files: paths to audio fragment to be verified manifest_filepath: path to manifest file to bre created wzutf-8)encodingr N-infer)audio_filepathoffsetdurationr label )rrwritejsondumps)rmanifest_filepathfp audio_fileentryrrrpath2audio_files_to_manifestgs"z.VerificationMixin.path2audio_files_to_manifestN)rfrr staticmethodrArrrrr0fsr0c @s0eZdZeddeededeefddZdS) DiarizationMixinr9rrreturncCsdS)z Takes paths to audio files and returns speaker labels Args: paths2audio_files: paths to audio fragment to be transcribed Returns: Speaker labels Nr)rrrrrrdiarizews zDiarizationMixin.diarizeN)r9)rfrrrrrr.rErrrrrCvs&rC)&r;rMrrrabcrrtypingrr omegaconfrrrrnemo.collections.asr.models collectionsasrmodelsr4nemo.collections.asr.parts.mixins.asr_adapter_mixinsr +nemo.collections.asr.parts.mixins.streamingr nemo.collections.asr.parts.utilsr +nemo.collections.asr.parts.utils.rnnt_utilsr nemo.collections.commonr nemo.utilsrrrrr0rCrrrrs4       K|