o }oi7@sddlZddlmZddlmZmZddlZddlmZddl m Z ddl m Z ddl mZddlmZdd lmZmZmZmZdd lmZdd lmZmZdd lmZd gZGdd d e eZGdddeZdS)N)ABC)ListOptional)WithOptionalCudaGraphs)ModelPT)PretrainedModelInfo) Exportable) AccessMixin) AudioSignal LabelsType LengthsType NeuralType) get_io_names)logging model_utils)cast_allASRModelcseZdZd#defddZd#defddZed$d d Zd%d ej d e dej fddZ ddZ fddZ d&ddZd&ddZd&ddZdeeeeeej ffffdd Zd&ddZd&dd Zedefd!d"ZZS)'rrdataloader_idxcCs4i}i}d|dvrtdd|D}d|i}||d|dvrHtdd|D}tdd|D}d||i}||d |dvrtd d|D} td d|D} td d|Djdd } tdd|Djdd } d|j| | | | i} || i|d|iS)Nval_lossrcSg|]}|dqS)r.0xrrY/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/asr/models/asr_model.py 'z7ASRModel.multi_validation_epoch_end.. val_wer_numcSr)rrrrrrr-rcSr) val_wer_denomrrrrrr.rval_wer val_bleu_numcSr)val_bleu_pred_lenrrrrrr4rcSr)val_bleu_target_lenrrrrrr5rcSr)r rrrrrr6r)dimcSr)val_bleu_denomrrrrrr7rval_bleulog)torchstackmeanupdatesumbleu _compute_bleuselfoutputsrrtensorboard_logs val_loss_meanwer_num wer_denomr bleu_pred_lenbleu_target_lenbleu_num bleu_denomr%rrrmulti_validation_epoch_end"s&       z#ASRModel.multi_validation_epoch_endcCs,i}i}d|dvrtdd|D}d|i}||d|dvrHtdd|D}tdd|D}d||i}||d |dvrtd d|D} td d|D} td d|D} td d|D} d|j| | | | i} || i|d|iS)N test_lossrcSr)r:rrrrrrCrz1ASRModel.multi_test_epoch_end.. test_wer_numcSr)r;rrrrrrIrcSr)test_wer_denomrrrrrrJrtest_wer test_bleu_numcSr)test_bleu_pred_lenrrrrrrPrcSr)test_bleu_target_lenrrrrrrQrcSr)r>rrrrrrRrcSr)test_bleu_denomrrrrrrSr test_bleur&)r'r(r)r*r+werr-r.rrrmulti_test_epoch_end>s&       zASRModel.multi_test_epoch_endreturnList[PretrainedModelInfo]cCst|}|S)z This method returns a list of pre-trained model which can be instantiated directly from NVIDIA's NGC cloud. Returns: List of available pre-trained models. )r&resolve_subclass_pretrained_model_info)clslist_of_modelsrrrlist_available_modelsZs zASRModel.list_available_modelsFlossreset_registryc Cst|jrEt|}i}|D](\}}d|vr9|d}t|}||7}|d} d| } d| } ||| <qt |dkrE| ||rLt ||S)al Utility method to enable calculation of auxiliary losses for ASR training. Args: loss: The output loss value prior to addition with auxiliary losses. reset_registry: Bool, whether to reset the AccessMixin registry after adding auxiliary losses. Returns: Loss tensor used for back propagation. adapter_loss./z adapter_loss/r) r is_access_enabled model_guidget_module_registryitemsr+splitjoindetachlenlog_dictrL) r/rKrLregistryrXloss_key loss_registry loss_list loss_valuekeyskeyrrradd_auxiliary_losseses$       zASRModel.add_auxiliary_lossescCs2d|_d|jvr|jdr|jd|_dSdSdS)aT Utility method that must be explicitly called by the subclass in order to support optional optimization flags. This method is the only valid place to access self.cfg prior to DDP training occurs. The subclass may chose not to support this method, therefore all variables here must be checked via hasattr() F skip_nan_gradN)_skip_nan_grad_cfgr/rrrsetup_optimization_flagssz!ASRModel.setup_optimization_flagscstt|drd|jrft|j}tjdg|tj d}| D]"\}}|j durDt |j  p:t|j  }|sD|d}nq"tjrUtjj|tjjjd|dkrhtd|dSdSdSdS)zH zero-out the gradients which any of them is NAN or INF rb)devicedtypeNr)opzCdetected inf or nan values in gradients! Setting gradients to zero.)superon_after_backwardhasattrrbnext parametersrgr'tensorfloat32named_parametersgradisnananyisinf distributedis_initialized all_reduceReduceOpMINrwarning zero_grad)r/rgvalid_gradients param_nameparamis_not_nan_or_inf __class__rrrks$  "   zASRModel.on_after_backwardNcCtj|dddS)z Decoder with CUDA graphs does not release memory, thus we disable it for training epoch. EncDecRNNTModel.decoding.decoding is the inference class with CUDA graphs decoding.decodingattribute_pathN)rdisable_cuda_graphs_recursiverdrrron_train_epoch_startzASRModel.on_train_epoch_startcCr)z After training, we can enable the decoder with CUDA graphs. EncDecRNNTModel.decoding.decoding is the inference class with CUDA graphs rrNrenable_cuda_graphs_recursiverdrrron_train_epoch_endrzASRModel.on_train_epoch_endcCr)z For validation, we enable CUDA graphs to speedup validation. EncDecRNNTModel.decoding.decoding is the inference class with CUDA graphs. rrNrrdrrron_validation_epoch_startrz"ASRModel.on_validation_epoch_startcstj|ddtjddS)z After validation, we disable CUDA graphs, since `validation` can be called in training loop, and training will continue after validation EncDecRNNTModel.decoding.decoding is the inference class with CUDA graphs. rrT) sync_metrics)rrrjon_validation_epoch_endrdrrrrsz ASRModel.on_validation_epoch_endcCr)a  For testing, we enable CUDA graphs to speedup validation. We do not need to disable CUDA graphs after testing, since `test` cannot be called in training loop. EncDecRNNTModel.decoding.decoding is the inference class with CUDA graphs. rrNrrdrrron_test_epoch_startzASRModel.on_test_epoch_startcCr)a For predicting, we enable CUDA graphs to speedup validation. We do not need to disable CUDA graphs after predicting, since `predict` cannot be called in training loop. EncDecRNNTModel.decoding.decoding is the inference class with CUDA graphs rrNrrdrrron_predict_epoch_startrzASRModel.on_predict_epoch_startcCsPttdtddtdtddtdtd|jjdtdtddgdS)z Return a typing schema for optimal batch size calibration for various sequence lengths using OOMptimizer. )BTinput)type seq_length)routput)rr vocab_size)rHinputs)tupler r r r tokenizerrrdrrroomptimizer_schemas zASRModel.oomptimizer_schema)r)rErF)F)rEN)__name__ __module__ __qualname__intr9rD classmethodrJr'Tensorboolr`rerkrrrrdictstrrrrpropertyr __classcell__rrrrr!s   &   ( csheZdZdZeddZeddZeddZ dd d Zed d Z ed dZ fddZ Z S)ExportableEncDecModelz Simple utiliy mix-in to export models that consist of encoder/decoder pair plus pre/post processor, but have to be exported as encoder/decoder pair only (covers most ASR classes) cC|jSN)encoderrdrrr input_modulez"ExportableEncDecModel.input_modulecCrr)decoderrdrrr output_modulerz#ExportableEncDecModel.output_modulecCsn|jj}t|jddr1|jj}ddt|ddD}t|ddD]\}}|||<q(t||jS)Nexport_cache_supportFcSsi|]\}}||qSrr)rntrrr sz6ExportableEncDecModel.output_names..rf)r output_typesgetattrrlistrSr disabled_deployment_output_names)r/otypesin_typesrrrrr output_namess  z"ExportableEncDecModel.output_namesNc Cst|jd|jj}|dur|||d}t|tr|d}n||||||d\}}}}}t|jd|jj}||d} t| trB| d} |durM| ||||f} t| tjtj dS)ad This forward is used when we need to export the model to ONNX format. Inputs cache_last_channel and cache_last_time are needed to be passed for exporting streaming models. Args: input: Tensor that represents a batch of raw audio signals of shape [B, T]. T here represents timesteps. length: Vector of length B, that contains the individual lengths of the audio sequences. cache_last_channel: Tensor of shape [N, B, T, H] which contains the cache for last channel layers cache_last_time: Tensor of shape [N, B, H, T] which contains the cache for last time layers N is the number of such layers which need caching, B is batch size, H is the hidden size of activations, and T is the length of the cache Returns: the output of the model forward_for_exportN) audio_signallengthr)rrcache_last_channelcache_last_timecache_last_channel_len)encoder_output) from_dtypeto_dtype) rrforward isinstancerrrr'float16rp) r/rrrrrenc_funrdec_funretrrrr s(    z(ExportableEncDecModel.forward_for_exportcC|jjSr)rdisabled_deployment_input_namesrdrrrr4z5ExportableEncDecModel.disabled_deployment_input_namescCrr)rrrdrrrr8rz6ExportableEncDecModel.disabled_deployment_output_namescsFd|vrt|d}||j_td||jt|dS)N cache_supportzCaching support enabled: )rrrrinfosetup_streaming_paramsrjset_export_config)r/argsenablerrrr<s   z'ExportableEncDecModel.set_export_config)NNNN) rrr__doc__rrrrrrrrrrrrrrs    (  r)rabcrtypingrrr'2nemo.collections.common.parts.optional_cuda_graphsrnemo.core.classesrnemo.core.classes.commonrnemo.core.classes.exportablernemo.core.classes.mixinsr nemo.core.neural_typesr r r r !nemo.core.utils.neural_type_utilsr nemo.utilsrnemo.utils.cast_utilsr__all__rrrrrrs"        S