o }oiwm@sFddlZddlmZddlmZmZmZmZmZddl Z ddl m Z m Z m Z ddlmZddlmZddlmZddlmZdd lmZmZdd lmZmZmZmZmZdd lm Z dd l!m"Z"m#Z#dd l$m%Z%m&Z&m'Z'ddl(m)Z)ddl*m+Z+m,Z,ddl-m.Z.m/Z/m0Z0m1Z1m2Z2m3Z3ddl4m5Z5m6Z6dgZ7GdddeeeeeZ8dS)N)ceil)AnyDictListOptionalUnion) DictConfig OmegaConf open_dict) DataLoader)audio_to_text_dataset) DALIOutputs)WER)ASRModelExportableEncDecModel) ASRBPEMixinASRModuleMixinASRTranscriptionMixinTranscribeConfigTranscriptionReturnType)process_augmentations)CTCBPEDecodingCTCBPEDecodingConfig)SequenceGeneratorSequenceGeneratorConfig get_seq_mask)SmoothedNLLLoss)PretrainedModelInfo typecheck) AudioSignal LabelsType LengthsType LogprobsType NeuralTypeSpectrogramType)logging model_utilsSLUIntentSlotBPEModelcseZdZdZd@deffdd Zedeee e ffddZ edeee e ffd d Z de fd d Ze      dAd dZddZ dBdee fddZdCddZdCddZdCddZddZdeefddZdeeeeffd d!Zd"eeeeffd#d$Zd%eeeeffd&d'Zdedd(fd)d*Ze + ,  -dDd.eee e fd/e!d0e"d1e!d2e"de#f fd3d4 Z$ d5e%d6e&fd7d8Z'd6e&dee fd9d:Z(e)dee*fd;d<Z+ed=d>Z,e,j-d?d>Z,Z.S)Er'ztModel for end-to-end speech intent classification and slot filling, which is formulated as a speech-to-sequence taskNcfgcst|}t|}d|vrtd||jtj||d||j j |_ ||j j |_ ||j j |_ t |jdrM|jjdurM||jj|_nd|_|||jj|_t|j}||j jd<||j j|_||j jd<||j j|_t|j jjd|_t|j j|j|j |j|jd|_|j d d}|durtt }t!|j  ||j _"Wdn1swYt#|j j"|jd |_"t$|j"|jd d d |jdd d d|_%dS)N tokenizerz:`cfg` must have `tokenizer` config to create a tokenizer !)r(trainer spec_augment vocab_size num_classes)label_smoothing)r( embeddingdecoder log_softmaxr)decoding)r)use_cerFTlog_prediction)r2r3dist_sync_on_stepr4fold_consecutive)&r&#convert_model_config_to_dict_configmaybe_update_config_version ValueError_setup_tokenizerr)super__init__from_config_dictr( preprocessorencoderr0hasattr_cfgr+spec_augmentationsetup_optimization_flagssetup_adapters get_vocab vocabularylenr/ classifierrlossr.rsequence_generatorgetr structuredrr r2rrwer)selfr(r*r, decoding_cfg __class__Z/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/asr/models/slu_models.pyr<1sT             zSLUIntentSlotBPEModel.__init__returnc Cst|jdrt|jjd}nt}td|ddttdtddtd|ddttdtddtdtddttdtddttdtdddS) N _sample_rate)freqBTToptionalrX)rXDrY) input_signalinput_signal_lengthtarget_semanticstarget_semantics_lengthprocessed_signalprocessed_signal_length sample_id)r@r>rrUr#tupler!r$)rNinput_signal_eltyperRrRrS input_typesvs   z!SLUIntentSlotBPEModel.input_typescCs4tdtddttdtddtdtdddS)N)rXrYr\TrZrXrW) log_probslengthsgreedy_predictions)r#r"rdr!r rNrRrRrS output_typessz"SLUIntentSlotBPEModel.output_typescCs*|jjj|_t||j|j|j|j|_dSN) rJ generatormax_seq_lengthmax_sequence_lengthrr/r0rHr))rNr(rRrRrSset_decoding_strategys z+SLUIntentSlotBPEModel.set_decoding_strategycCs |duo|du}|duo|du}||Adkrt|d|s(|j||d\}}|jdur7|jr7|j||d}|j||d\} } | dd} t| | } |durZ|| | } dd| fS|dddd f} || }t||d}|j | | ||d }| |}|j d dd } |j | }||| fS) a Forward pass of the model. Params: input_signal: Tensor that represents a batch of raw audio signals, of shape [B, T]. T here represents timesteps, with 1 second of audio represented as `self.sample_rate` number of floating point values. input_signal_length: Vector of length B, that contains the individual lengths of the audio sequences. target_semantics: Tensor that represents a batch of semantic tokens, of shape [B, L]. target_semantics_length: Vector of length B, that contains the individual lengths of the semantic sequences. processed_signal: Tensor that represents a batch of processed audio signals, of shape (B, D, T) that has undergone processing via some DALI preprocessor. processed_signal_length: Vector of length B, that contains the individual lengths of the processed audio sequences. Returns: A tuple of 3 elements - 1) The log probabilities tensor of shape [B, T, D]. 2) The lengths of the output sequence after decoder, of shape [B]. 3) The token predictions of the model of shape [B, T]. NF Arguments ``input_signal`` and ``input_signal_length`` are mutually exclusive with ``processed_signal`` and ``processed_signal_len`` arguments.r]length input_specrs audio_signalrs)encoder_states encoder_maskdecoder_states decoder_mask)dimkeepdim) r9r>rBtrainingr? transposerrJr/r0rHargmaxget_seq_length)rNr]r^r_r`rarbhas_input_signalhas_processed_signalencoded encoded_len encoded_mask predictionsbos_semantics_tokens bos_semanticsbos_semantics_maskdecodedrgpred_lenrRrRrSforwards@#          zSLUIntentSlotBPEModel.forwardcCs>t|dkr |\}}}}n|\}}}}}|j||||d\}} } |ddddf} |d} |j|| | d} d| i}t|jjdkrO|jjdd|d<nt|jjD] \}}|d|d |<qUt|d rr|jdurr|jj }nd}|d|dkr|j j | | | | d |j \}}}|j | d |i| |d S)Nr]r^r_r`rxrglabelsrh train_lossrlr learning_ratelearning_rate_g_trainerrtargetspredictions_lengthstargets_lengthstraining_batch_wer)rIlog)rGrrIitem _optimizer param_groups enumerater@rlog_every_n_stepsrMupdatecomputereset)rNbatchbatch_nbsignal signal_len semantics semantics_lenrcrgrr eos_semanticseos_semantics_len loss_valuetensorboard_logsigrouprrM_rRrRrS training_steps>      z#SLUIntentSlotBPEModel.training_steprc Cs|duo|du}|duo|du}||Adkrt|d|s(|j||d\}}|jdur7|jr7|j||d}|j||d\}} |dd}t|| } ||| } |j| } | S)NFrqrrrtrvrxry) r9r>rBrr?rrrJdecode_semantics_from_tokens) rNr]r^rarbdataloader_idxrrrrr pred_tokensrrRrRrSpredict s&      zSLUIntentSlotBPEModel.predictcCst|dkr |\}}}}n|\}}}}}t|tr)|jr)|j||||d\} } } n |j||||d\} } } |ddddf} |d} |j| | | d}|jj| | | | d|j\}}}|j ||||dS)Nr)rarbr_r`rrxrr)val_loss val_wer_num val_wer_denomval_wer) rG isinstancer rrrIrMrrr)rNr batch_idxrrrrrrcrgrrrrrrMwer_num wer_denomrRrRrSvalidation_pass's@   z%SLUIntentSlotBPEModel.validation_passcCsR||||}t|jjtkr!t|jjdkr!|j|||S|j||S)Nrx)rtyper*val_dataloaderslistrGvalidation_step_outputsappend)rNrrrmetricsrRrRrSvalidation_stepQs   z%SLUIntentSlotBPEModel.validation_stepcCsf|j|||d}dd|D}t|jjtkr+t|jjdkr+|j|||S|j||S)N)rcSsi|] \}}|dd|qS)val_test_)replace).0namevaluerRrRrS [sz3SLUIntentSlotBPEModel.test_step..rx) ritemsrr*test_dataloadersrrGtest_step_outputsr)rNrrrlogs test_logsrRrRrS test_stepYs  zSLUIntentSlotBPEModel.test_stepcCs|jdurg|_|jSrl)_test_dlrjrRrRrStest_dataloaderbs z%SLUIntentSlotBPEModel.test_dataloaderconfigc Csd|vr t|d}nd}|d}tjrdnd}|ddr<|dkr'|jnd}tj||j|||j |j |j j d}|S|ddrd |vrL|d dusVd |vr`|d dur`t d |dS|rl|d d |dnd}tj||j||j |j |d}d}nd |vr|d durt d|dStj||j|d}t|dr|j}nt|jddr|jdj}n |jdjdj}tjjj||d||dd||dd|dddS)N augmentorshufflegpucpuuse_daliF)rr)r device_id global_rank world_sizepreprocessor_cfg is_tarredtarred_audio_filepathsmanifest_filepathznCould not load dataset as `manifest_filepath` was None or `tarred_audio_filepaths` is None. Provided config : shuffle_nr batch_sizer)rr)rrrrzJCould not load dataset as `manifest_filepath` was None. Provided config : )rr)r collate_fn drop_last num_workers pin_memory)datasetrrrrrr)rtorchcuda is_availablerK local_rankr get_dali_bpe_datasetr)rrrAr>r%warningget_tarred_datasetget_bpe_datasetr@rdatasetsutilsdatar ) rNrrrdevicerrrrrRrRrS_setup_dataloader_from_configisp      z3SLUIntentSlotBPEModel._setup_dataloader_from_configtrain_data_configcCsd|vrd|d<|jd|d|j|d|_|jdur\t|jdr^t|jjtjjj r`|j durPt|j j t rPt |j j tt|jj|j|d|j _ dS|j durbtd dSdSdSdSdS) a Sets up the training data loader via a Dict-like object. Args: train_data_config: A config that contains the information regarding construction of an ASR Training dataset. Supported Datasets: - :class:`~nemo.collections.asr.data.audio_to_text.AudioToCharDataset` - :class:`~nemo.collections.asr.data.audio_to_text.AudioToBPEDataset` - :class:`~nemo.collections.asr.data.audio_to_text.TarredAudioToCharDataset` - :class:`~nemo.collections.asr.data.audio_to_text.TarredAudioToBPEDataset` - :class:`~nemo.collections.asr.data.audio_to_text_dali.AudioToCharDALIDataset` rTtrain dataset_namerrNrrzModel Trainer was not set before constructing the dataset, incorrect number of training batches will be used. Please set the trainer and rebuild the dataset.)_update_dataset_configr _train_dlr@rrrrrIterableDatasetrlimit_train_batchesfloatintrrGrr%r)rNrrRrRrSsetup_training_datas*     z)SLUIntentSlotBPEModel.setup_training_dataval_data_configcC0d|vrd|d<|jd|d|j|d|_dS)a Sets up the validation data loader via a Dict-like object. Args: val_data_config: A config that contains the information regarding construction of an ASR Training dataset. Supported Datasets: - :class:`~nemo.collections.asr.data.audio_to_text.AudioToCharDataset` - :class:`~nemo.collections.asr.data.audio_to_text.AudioToBPEDataset` - :class:`~nemo.collections.asr.data.audio_to_text.TarredAudioToCharDataset` - :class:`~nemo.collections.asr.data.audio_to_text.TarredAudioToBPEDataset` - :class:`~nemo.collections.asr.data.audio_to_text_dali.AudioToCharDALIDataset` rF validationrrN)rr_validation_dl)rNrrRrRrSsetup_validation_dataz+SLUIntentSlotBPEModel.setup_validation_datatest_data_configcCr)a Sets up the test data loader via a Dict-like object. Args: test_data_config: A config that contains the information regarding construction of an ASR Training dataset. Supported Datasets: - :class:`~nemo.collections.asr.data.audio_to_text.AudioToCharDataset` - :class:`~nemo.collections.asr.data.audio_to_text.AudioToBPEDataset` - :class:`~nemo.collections.asr.data.audio_to_text.TarredAudioToCharDataset` - :class:`~nemo.collections.asr.data.audio_to_text.TarredAudioToBPEDataset` - :class:`~nemo.collections.asr.data.audio_to_text_dali.AudioToCharDALIDataset` rFtestrrN)rrr)rNr rRrRrSsetup_test_datar z%SLUIntentSlotBPEModel.setup_test_dataztorch.utils.data.DataLoaderc Csd|vr |d}|d}ntj|dd}t|dt|d}||jj|d|dt|tdd |j j d dd }|j t |d }|S) a Setup function for a temporary data loader which wraps the provided audio file. Args: config: A python dictionary which contains the following keys: paths2audio_files: (a list) of paths to audio files. The files should be relatively short fragments. Recommended length per file is between 5 and 25 seconds. batch_size: (int) batch size to use during inference. Bigger will result in better throughput performance but would use more memory. temp_dir: (str) A temporary directory where the audio manifest is temporarily stored. num_workers: (int) number of workers. Depends of the batch_size and machine. 0 - only the main process will load batches, 1 - one worker (not main process) Returns: A pytorch DataLoader for the given audio file(s). rrtemp_dirz manifest.jsonpaths2audio_filesFrrxTuse_start_end_token)r sample_raterrrrrr) ospathjoinminrGr>rUrK cpu_countr( validation_dsrr)rNrrr dl_configtemporary_datalayerrRrRrS_setup_transcribe_dataloader s  z2SLUIntentSlotBPEModel._setup_transcribe_dataloaderrFTaudiorreturn_hypothesesrverbosecstj|||||dS)a` Uses greedy decoding to transcribe audio files into SLU semantics. Use this method for debugging and prototyping. Args: audio: (a single or list) of paths to audio files or a np.ndarray audio array. Can also be a dataloader object that provides values that can be consumed by the model. Recommended length per file is between 5 and 25 seconds. But it is possible to pass a few hours long file if enough GPU memory is available. batch_size: (int) batch size to use during inference. Bigger will result in better throughput performance but would use more memory. return_hypotheses: (bool) Either return hypotheses or text With hypotheses can do some postprocessing like getting timestamp or rescoring num_workers: (int) number of workers for DataLoader verbose: (bool) whether to display tqdm progress bar Returns: A list of transcriptions (or raw log probabilities if logprobs is True) in the same order as paths2audio_files )rrrrr)r; transcribe)rNrrrrrrPrRrSr0sz SLUIntentSlotBPEModel.transcribertrcfgcCs"|j|d|dd}d|i}|S)Nrrx)r]r^r)r)rNrr routputrRrRrS_transcribe_forwardVsz)SLUIntentSlotBPEModel._transcribe_forwardcCs|d}|S)Nr)pop)rNoutputsr  hypothesesrRrRrS_transcribe_output_processing[s z3SLUIntentSlotBPEModel._transcribe_output_processingcCs g}tdddd}||dS)z This method returns a list of pre-trained model which can be instantiated directly from NVIDIA's NGC cloud. Returns: List of available pre-trained models. %slu_conformer_transformer_large_slurpzFor details about this model, please visit https://ngc.nvidia.com/catalog/models/nvidia:nemo:slu_conformer_transformer_large_slurpzhttps://api.ngc.nvidia.com/v2/models/nvidia/nemo/slu_conformer_transformer_large_slurp/versions/1.13.0/files/slu_conformer_transformer_large_slurp.nemo)pretrained_model_name descriptionlocationN)rr)clsresultsmodelrRrRrSlist_available_models_sz+SLUIntentSlotBPEModel.list_available_modelscCs|jSrl_werrjrRrRrSrMpszSLUIntentSlotBPEModel.wercCs ||_dSrlr/)rNrMrRrRrSrMts rl)NNNNNN)NNr)r)rFrT)/__name__ __module__ __qualname____doc__rr<propertyrrstrr#rfrkrrprrrrrrrrrrrrr rrrno_gradr rboolrrrrr"r& classmethodrr.rMsetter __classcell__rRrRrPrSr'.srE M,  *  E-'# )9rmathrtypingrrrrrr omegaconfrr r torch.utils.datar nemo.collections.asr.datar ,nemo.collections.asr.data.audio_to_text_dalir nemo.collections.asr.metrics.werr%nemo.collections.asr.models.asr_modelrr!nemo.collections.asr.parts.mixinsrrrrr0nemo.collections.asr.parts.preprocessing.perturbr2nemo.collections.asr.parts.submodules.ctc_decodingrr*nemo.collections.asr.parts.utils.slu_utilsrrrnemo.collections.common.lossesrnemo.core.classes.commonrrnemo.core.neural_typesrr r!r"r#r$ nemo.utilsr%r&__all__r'rRrRrRrSs(