o }oi8@sJddlmZmZmZmZddlZddlmZddlm Z GdddZ dS))DictListOptionalTupleN) AccessMixin)loggingc@seZdZdZdeedeefddZddZdd Z d d Z d d Z de fddZ de fddZdedeedefddZdeeejejffddZ   d(dejdejd ejd!e d"e d#e d$edeeejeffd%d&Zd'S)) InterCTCMixinaAdds utilities for computing interCTC loss from https://arxiv.org/abs/2102.03216. To use, make sure encoder accesses ``interctc['capture_layers']`` property in the AccessMixin and registers ``interctc/layer_output_X`` and ``interctc/layer_length_X`` for all layers that we want to get loss from. Additionally, specify the following config parameters to set up loss:: interctc: # can use different values loss_weights: [0.3] apply_at_layers: [8] Then call * ``self.setup_interctc(ctc_decoder_name, ctc_loss_name, ctc_wer_name)`` in the init method * ``self.add_interctc_losses`` after computing regular loss. * ``self.finalize_interctc_metrics(metrics, outputs, prefix="val_")`` in the `multi_validation_epoch_end` method. * ``self.finalize_interctc_metrics(metrics, outputs, prefix="test_")`` in the `multi_test_epoch_end` method. loss_weightsapply_at_layerscCs|d||d||ddt||ddkr!td|dt|dkt|t|kr7td tjd d |iit|d dd t|drQ| dSt ddS)Nintermediate_loss_weightsr main_loss_weightg?gzMake sure that sum of intermediate loss weights is < 1.0. Note that we don't do any normalization and assign remaining weight to the regular model loss. E.g., if interctc.loss_weights = [0.1, 0.3], regular loss will have weight of 0.6enabledrzELength of interctc.apply_at_layers has to match interctc.loss_weightsinterctccapture_layers model_guid)guidpropagate_model_guidzsNot able to propagate model_guid to the submodules. Make sure to call self.propagate_model_guid() in ModelPT class.) set_interctc_paramsumget_interctc_param ValueErrorlenrupdate_access_cfggetattrhasattrrrwarning)selfr r rd/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/asr/parts/mixins/interctc_mixin.py_process_config_values.s$    z$InterCTCMixin._process_config_valuescCs\i|_|jd}|dur&|jdi|||jd<||jd<||jd<dS|dddS) aSets up all interctc-specific parameters and checks config consistency. Caller has to specify names of attributes to perform CTC-specific WER, decoder and loss computation. They will be looked up in the class state with ``getattr``. The reason we get the names and look up object later is because those objects might change without re-calling the setup of this class. So we always want to look up the most up-to-date object instead of "caching" it here. rN decoder_name loss_namewer_namer Fr)_interctc_paramscfggetrr)rr r!r"interctc_configrrrsetup_interctcKs   zInterCTCMixin.setup_interctccCs&|dvrt||j|dS|j|S)zzEither directly get parameter from ``self._interctc_params`` or call getattr with the corresponding name. decoderlosswer_name)rr#)r param_namerrrrfs z InterCTCMixin.get_interctc_paramcCs|dvrtd||j|<dS)zSetting the parameter to the ``self._interctc_params`` dictionary. Raises an error if trying to set decoder, loss or wer as those should always come from the main class. r(zgCannot set "decoder", "loss" or "wer" as parameters. They are always looked up in the main class state.N)rr#)rr- param_valuerrrrns z InterCTCMixin.set_interctc_paramcCst|ds tddS)z3Can be used to verify if setup_interctc was called.r#zuself.setup_interctc(ctc_decoder_name, ctc_loss_name, ctc_wer_name) has to be called before InterCTC loss can be used!N)r RuntimeErrorrrrr_verify_setup_was_called{s z&InterCTCMixin._verify_setup_was_calledreturncCs||dS)z)Returns whether interCTC loss is enabled.r )r1rr0rrris_interctc_enableds z!InterCTCMixin.is_interctc_enabledr cCsZ||r%t|ddkrtdt|dt|dkr%td|d|dS)z0Can be used to enable/disable InterCTC manually.r rzWInterCTC cannot be enabled since interctc.loss_weights was not specified in the config.r z\InterCTC cannot be enabled, since length of "loss_weights" does not match "apply_at_layers".r N)r1rrr/r)rr rrrset_interctc_enabledsz"InterCTCMixin.set_interctc_enabledmetricsoutputsprefixcs|r|dD]]d|dvr0tfdd|D}||dd<d|dvrftfdd|D}tfd d|D}|||dd <q d |dvrtfd d|D|dd <d Sd Sd S)a)Finalizes InterCTC WER and loss metrics for logging purposes. Should be called inside ``multi_validation_epoch_end`` (with ``prefix="val_"``) or ``multi_test_epoch_end`` (with ``prefix="test_"``). Note that ``metrics`` dictionary is going to be updated in-place. r inter_ctc_loss_lrcg|] }|dqS)r8r.0x layer_idxr7rr z;InterCTCMixin.finalize_interctc_metrics..loginter_wer_num_lcr9)rBrr:r=rrr?r@cr9)inter_wer_denom_lrr:r=rrr?r@ inter_wer_l final_losscsg|] }|dqS)rErr:)r7rrr?sN)r3rtorchstackmeanr)rr5r6r7r*wer_num wer_denomrr=rfinalize_interctc_metricss. z'InterCTCMixin.finalize_interctc_metricsc Cs|sgSi}t|jD]}|D]}|dr'||vr'td|dq||qg}|dD]C}z|d|}|d|}Wnt yVtd|dwt |d ksct |d krgtd | |d |d d |d fq5|S)zReturns a list of captured tensors from encoder: tuples of (output, length). Will additionally apply ``ctc_decoder`` to the outputs. z interctc/zlayer z has been logged multiple times!r zinterctc/layer_output_zinterctc/layer_length_zIntermediate layer z was not captured! Check if length of model.encoder.captured_layer_outputs matches length of model.intermediate_loss_weights properties.zVMake sure encoder.forward is called exactly one time before interCTC loss is computed.r)r)encoder_output) r3rget_module_registryencodervalues startswithr/updaterKeyErrorrappend)rtotal_registrymodule_registrykeycaptured_tensorsr> layer_outputs layer_lengthsrrrget_captured_interctc_tensorss6   z+InterCTCMixin.get_captured_interctc_tensorsTF loss_value transcripttranscript_len compute_wer compute_losslog_wer_num_denom log_prefixcCsP|r tt|dds|ifSi}|r|||d<nd}|} |r,||d9}t|d| |dD]k\} } } |r_|d| d||| d d } | ||d | <|| | 7}|r|d j| d||| d d |d  \}}}|d  ||d| |i|r||d| ||d| |iq8||fS)aAdding interCTC losses if required. Will also register loss/wer metrics in the returned dictionary. Args: loss_value (torch.Tensor): regular loss tensor (will add interCTC loss to it). transcript (torch.Tensor): current utterance transcript. transcript_len (torch.Tensor): current utterance transcript length. compute_wer (bool): whether to compute WER for the current utterance. Should typically be True for validation/test and only True for training if current batch WER should be logged. compute_loss (bool): whether to compute loss for the current utterance. Should always be True in training and almost always True in validation, unless all other losses are disabled as well. Defaults to True. log_wer_num_denom (bool): if True, will additionally log WER num/denom in the returned metrics dictionary. Should always be True for validation/test to allow correct metrics aggregation. Should always be False for training. Defaults to False. log_prefix (str): prefix added to all log values. Should be ``""`` for training and ``"val_"`` for validation. Defaults to "". Returns: tuple[Optional[torch.Tensor], Dict]: tuple of new loss tensor and dictionary with logged metrics. rNrEr r r r*rrL) log_probstargetstarget_lengths input_lengthsr8r+) predictionsretargets_lengthspredictions_lengthsrDrBrC) r3ris_access_enabledrr[rzipdetachrRcomputereset)rr]r^r_r`rarbrcr5rXr>intermediate_result loss_weightinter_loss_valuer+rIrJrrradd_interctc_lossessR#  z!InterCTCMixin.add_interctc_lossesN)TFr\)__name__ __module__ __qualname____doc__rfloatintrr'rrr1boolr3r4rstrrKrrFTensorr[rrsrrrrrs> - r) typingrrrrrFnemo.core.classes.mixinsr nemo.utilsrrrrrrs