o wiO@sddlZddlZddlmZddlmZmZmZmZm Z m Z ddl Z ddl m Z mZddlmZmZmZddlmZmZddlmZmZmZmZmZddlmZdd lmZdd l m!Z!dd l"m#Z#m$Z$m%Z%zddl&Z'd Z(Wn e)e*fy{d Z(Ynwzddl+m,Z,m-Z-m.Z.d Z/Wn e)e*fyd Z/Ynwzddl0m1Z1ddl2m3Z3d Z4Wn e)e*fyd Z4YnwdZ5eGdddZ6e6dde(e5d de6ddde/e!d de6dddd dd de6ddde/e!d de6dddd d d de6d!d"e4ed de6d#d"e4ed de6d$dde/e!d%e6d&ddd d'd%d( Z7e7de7d)<d*d+Z8 d;d,e9d-e ee9efd.ed/e e e9d0ee9eff d1d2Z:d0e9fd3d4Z;d;d,e9d5ej?fd7d8Z@Gd9d:d:eZAdS)<N) dataclass)AnyCallableDictListOptionalSet) DictConfig OmegaConf)MultiblankRNNTLossPytorchRNNTLossPytorchTDTLossPytorch)Loss typecheck) LabelsType LengthsType LogprobsTypeLossType NeuralType) numba_utils)K2_INSTALLATION_MESSAGE)NUMBA_INSTALLATION_MESSAGE)logging logging_mode model_utilsTF)MultiblankRNNTLossNumba RNNTLossNumba TDTLossNumba) GraphRnntLoss)GraphWTransducerLosszCould not import `warprnnt_pytorch`. Please visit https://github.com/HawkAaron/warp-transducer and follow the steps in the readme to build and install the pytorch bindings for RNNT Loss, or use the provided docker container that supports RNN-T loss.c@sReZdZUeed<eed<dZeed<dZeed<dZe eed<d Z eed <dS) RNNTLossConfig loss_namelib_nameF is_availableinstallation_msgN min_versionT force_float32) __name__ __module__ __qualname__str__annotations__r#boolr%r&rr'r.r.]/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/nemo/collections/asr/losses/rnnt.pyr Ms   r warprnntwarprnnt_pytorch)r!r"r#r%r'warprnnt_numbanumbaz0.53.0)r!r"r&r#r%r'pytorchtorchz0.0zPPure Pytorch implementation of RNN-T loss. Slow and for debugging purposes only.multiblank_rnntz[Pure Pytorch implementation of Multiblank RNN-T loss. Slow and for debugging purposes only.graph_w_transducerk2 graph_rnnttdt)r!r"r&r#r% tdt_pytorchzNPure Pytorch implementation of TDT loss. Slow and for debugging purposes only.) r0r2r4r6multiblank_rnnt_pytorchr7r9r:r;defaultcCs*t|dkrtd|d|dSdS)NrzLoss function `z\` was provided with following additional kwargs, however they were ignored as it is unused. )lenrwarning)r!kwargsr.r.r/_warn_unused_additional_kwargss rAr!r@ init_method ignore_paramsreturnc Cs|siStt|jdh}|dur||8}t}t}|D]\}}||vr0|||<q#|||<q#t|dkr@t|||S)af Cleans kwargs for the given loss function. Warn if there are unused kwargs. Args: loss_name: name of the loss function kwargs: kwargs to clean init_method: LossClass.__init__ method ignore_params: set of argument names for init_method to ignore Returns: only used kwargs for the given `init_method` selfNr) setinspect signature parameterskeysdictitemsr>rA) r!r@rBrC init_params unused_kwargs used_kwargskeyvaluer.r.r/ _clean_kwargss    rRcCs tdjS)Nr=)RNNT_LOSS_RESOLVERr!r.r.r.r/resolve_rnnt_default_loss_names rT blank_idx loss_kwargsc Cs6tt}||vrtd|d|ddtD}t|}|js5dt|d|j}t||jdurWt j |j |jt j d\}}|durW|d |j}t||dur]in|}t|trktj|d d }|d krr|j} |d krtj|dd}t|||S|dkrt |_|dd} |dd} t|d| | d}t|||S|dkrt|dd}t|||S|dkr|dd} |dd} |dd} |dd} t|| d| | | d}t|||S|dkr|dd} |dd} t|| d| d}t|||S|dkr=|dd} |dd} |dd} |dd} |dd}t|| d| | | |d }t|||S|d!kr]|dd} |dd} t || d| d"}t|||S|d#krwt!||t"j#d$hd%}t"d)d$|i|}|S|d&krt!||t$j#d$hd%}t$d)d$|i|}|Std'|d(|)*NzProvided `loss_name` z' not in list of available RNNT losses cSsi|] \}}|jr||qSr.)r#).0nameconfigr.r.r/ sz%resolve_rnnt_loss..zInstalled RNNT losses are : z. **************************************************************** To install the selected loss function, please follow the steps below: )checked_versionoperatorFz **************************************************************** To update the selected loss function, please follow the steps below: T)resolver=r0none)blank reductionr2fastemit_lambdagclampg)r_r`rarbr4r6big_blank_durationssigma)r_rcr`rarbrdr<)r_rcr`rdr: durationsomega)r_rer`rarbrdrfr;)r_rer`rdr9r_)rCr7zInvalid value of `loss_name`: z. Allowed loss names are :r.)%listrSrJ ValueErrorrLr#r% ImportErrorr&rcheck_lib_versionr"r\ge RuntimeError isinstancer r to_containerr!r0RNNTLossrAris_numba_cuda_fp16_supportedr'poprr rr rr rRr__init__r)r!rUrVloss_function_namesall_available_losses loss_configmsg ver_matched loss_funcrarbrcrdrerfr.r.r/resolve_rnnt_losss      H    ?  ;     ,   $           rycsVeZdZeddZeddZdded effd d Zd d Ze ddZ Z S)rocCs6tdttdtttdtttdtdS)z-Input types definitions for CTCLoss. )BTr{D)rzr{rz) log_probstargets input_lengthstarget_lengths)rrrtuplerrEr.r.r/ input_typesNs  zRNNTLoss.input_typescCsdttdiS)zYOutput types definitions for CTCLoss. loss: NeuralType(None) loss) elements_type)rrrr.r.r/ output_typesYszRNNTLoss.output_types mean_batchr=Nr`r!csRtt||dvrtd||_||_t||j|d|_t|j |_ d|_ dS)ab RNN-T Loss function based on https://github.com/HawkAaron/warp-transducer. Optionally, can utilize a numba implementation of the same loss without having to compile the loss, albiet there is a small speed penalty for JIT numba compile. Note: Requires Numba 0.53.0 or later to be installed to use this loss function. Losses can be selected via the config, and optionally be passed keyword arguments as follows. Examples: .. code-block:: yaml model: # RNNT Model config ... loss: loss_name: "warprnnt_numba" warprnnt_numba_kwargs: fastemit_lambda: 0.0 Warning: In the case that GPU memory is exhausted in order to compute RNNTLoss, it might cause a core dump at the cuda level with the following error message. ``` ... costs = costs.to(acts.device) RuntimeError: CUDA error: an illegal memory access was encountered terminate called after throwing an instance of 'c10::Error' ``` Please kill all remaining python processes after this point, and use a smaller batch size for train, validation and test sets so that CUDA memory is not exhausted. Args: num_classes: Number of target classes for the joint network to predict. In all cases (conventional RNNT, multi-blank RNNT, and TDT model), this equals the token-id for the standard "blank" symbol. In particular, say V is the number of non-blank tokens in the vocabulary, then in the case of, standard RNNT: num_classes = V multiblank RNNT: num_classes = V + number-big-blanks (since we store big-blanks before standard blank, and the standard blank is the last symbol in the vocab) TDT: num_classes = V. Note, V here does not include any of the "duration outputs". reduction: Type of reduction to perform on loss. Possible values are `mean_batch`, 'mean_volume`, `mean`, `sum` or None. `None` will return a torch vector comprising the individual loss values of the batch. `mean_batch` will average the losses in the batch `mean` will divide each loss by the target length and then average `mean_volume` will add up all the losses and divide by sum of target lengths loss_name: String that is resolved into an RNNT loss function. Available list of losses is ininitialized in `RNNT_LOSS_RESOLVER` dictionary. loss_kwargs: Optional Dict of (str, value) pairs that are passed to the instantiated loss function. )Nmeansumr mean_volumez?`reduction` must be one of [mean, sum, mean_batch, mean_volume])rUrVFN) superrorrrh_blankr`ry_lossrSr'_force_float32_fp16_compat_checked)rE num_classesr`r!rV __class__r.r/rras:  zRNNTLoss.__init__cCst|trt|d}t|d}|jdkr|}|S|jdkr+t||}|S|jdkr6|}|S|jdkrC||}|S)Nrrrrr)rmrr5catr`rdivr)rElossesrr.r.r/reduces       zRNNTLoss.reducec Cs>|}|}|}|}|}|jstrn3|jrO|jtjkrO|jtjkrH|j sHtjdd\}}t j d|jd|t j dd|_ |} |}~ |jd|kr`|jdd|d}|sh|}|jd|kry|jdd|d}|jj} d|j_|j||||d } | |j_|jdur|| |} ~~~~| S) NT) return_reasonz'Provided RNNT Joint tensor is of dtype zy, but RNNT loss could not be calculated in fp16 due to following reason stated below. Loss will be calculated in fp32. )moder)dimstartlength)actslabelsact_lens label_lens)longmaxrrrpdtyper5float32float16rrr?rONCEfloatshapenarrow contiguous is_contiguousrr`r) rEr}r~rr max_logit_lenmax_targets_len_reason logits_origloss_reductionrr.r.r/forwardsJ   zRNNTLoss.forward)rr=N) r(r)r*propertyrrr+rrrrr __classcell__r.r.rr/roMs  Ero)N)BrGr\ dataclassesrtypingrrrrrrr5 omegaconfr r (nemo.collections.asr.losses.rnnt_pytorchr r r nemo.core.classesrrnemo.core.neural_typesrrrrrnemo.core.utilsrnemo.core.utils.k2_utilsrnemo.core.utils.numba_utilsr nemo.utilsrrrr1r0WARP_RNNT_AVAILABLEriModuleNotFoundError*nemo.collections.asr.parts.numba.rnnt_lossrrrNUMBA_RNNT_AVAILABLE.nemo.collections.asr.parts.k2.graph_transducerr*nemo.collections.asr.parts.k2.w_transducerr K2_AVAILABLEWARP_RNNT_INSTALLATION_MESSAGEr rSrAr+rRrTintrKnnModuleryror.r.r.r/s          F