o ei8:@sndZddlZddlZddlZddlZddlmZddlmZddl m Z dZ e e Z z-ddlmZe sQe dedZeejdd lmZejd ed ne d Wn5eyd Zed7Zed7Zed7Zed7Zed7Zed7Zed7Zed7Zed7Zed7ZeeweddZeddZeddZGdddeZGd d!d!eZdS)"zn Transducer loss implementation (depends on numba) Authors * Abdelwahab Heba 2020 * Titouan Parcollet 2023 N)Function)Module) get_logger)cudazCNumba verbose is deactivated. To enable it, set NUMBA_VERBOSE to 1.numba)NumbaPerformanceWarningignore)categoryzCNumba verbose is enabled. To deactivate it, set NUMBA_VERBOSE to 0.z;The optional dependency Numba is needed to use this module z,Cannot import numba. To use Transducer loss z%Please follow the instructions below ============================= If you use your localhost: pip install numba =export NUMBAPRO_LIBDEVICE='/usr/local/cuda/nvvm/libdevice/' >export NUMBAPRO_NVVM='/usr/local/cuda/nvvm/lib64/libnvvm.so' "================================ If you use conda: zconda install numba cudatoolkitc Cstjj}tjj} d} | ||kr | ||kr| dkrH| dkr7||| ddf||| dd|f||| df<tj||| dfd| d7} ntj||| fddkr| dkrw||d| df||d| d||| dff||d| f<nC||| | df||| | d||| dff} ||| d| f||| d| |f} t| | tt t | |  ||| | f<| ||krtj||| dfdtj||| fd| d7} | ||ks| ||kr ||||d||f||||d|||f||||<dSdSdS)a Compute forward pass for the forward-backward algorithm using Numba cuda kernel. Sequence Transduction with naive implementation : https://arxiv.org/pdf/1211.3711.pdf Arguments --------- log_probs : torch.Tensor 4D Tensor of (batch x TimeLength x LabelLength x outputDim) from the Transducer network. labels : torch.Tensor 2D Tensor of (batch x MaxSeqLabelLength) containing targets of the batch with zero padding. alpha : torch.Tensor 3D Tensor of (batch x TimeLength x LabelLength) for forward computation. log_p : torch.Tensor 1D Tensor of (batch) for forward cost computation. T : torch.Tensor 1D Tensor of (batch) containing TimeLength of each target. U : torch.Tensor 1D Tensor of (batch) containing LabelLength of each target. blank : int Blank index. lock : torch.Tensor 2D Tensor of (batch x LabelLength) containing bool(1-0) lock for parallel computation. rN rblockIdxx threadIdxatomicaddmaxmathlog1pexpabs) log_probslabelsalphalog_pTUblanklockbutemitno_emitr+c/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/speechbrain/nnet/loss/transducer_loss.pycu_kernel_forward:sJ $  $   !4 &r-c Cstjj}tjj} ||d} | ||kr| dkr| ||kr[| ||dkr4||| | |f||| | f<n||| d| f||| | |f||| | f<tj||| dfd| d8} ntj||| fddkr| ||dkr||| | df||| | ||| ff||| | f<n=||| | df||| | ||| ff} ||| d| f||| | |f} t| | tt t | |  ||| | f<| dkrtj||| dfdtj||| fd| d8} | dks| dkr||ddf||||<dSdS)a Compute backward pass for the forward-backward algorithm using Numba cuda kernel. Sequence Transduction with naive implementation : https://arxiv.org/pdf/1211.3711.pdf Arguments --------- log_probs : torch.Tensor 4D Tensor of (batch x TimeLength x LabelLength x outputDim) from the Transducer network. labels : torch.Tensor 2D Tensor of (batch x MaxSeqLabelLength) containing targets of the batch with zero padding. beta : torch.Tensor 3D Tensor of (batch x TimeLength x LabelLength) for backward computation. log_p : torch.Tensor 1D Tensor of (batch) for backward cost computation. T : torch.Tensor 1D Tensor of (batch) containing TimeLength of each target. U : torch.Tensor 1D Tensor of (batch) containing LabelLength of each target. blank : int Blank index. lock : torch.Tensor 2D Tensor of (batch x LabelLength) containing bool(1-0) lock for parallel computation. rrrNr) rrbetar!r"r#r$r%r&r'r(r)r*r+r+r,cu_kernel_backwards<      (("  r/c Cstjj}tjj} ||| kr|dkrDt|| || d|| f|| || d|| |f|| ddf || || d|| |f<||| dkrt|| dD]9} || || f|| |d| f|| || |f<t|| || |f|| || |f|| ddf || || |f<qTt|| D]C\} } | || kr|| || f|| || df|| || | f<t|| || | f|| || | f|| ddf || || | f<qdSdS)a Compute gradient for the forward-backward algorithm using Numba cuda kernel. Sequence Transduction with naive implementation : https://arxiv.org/pdf/1211.3711.pdf Arguments --------- log_probs : torch.Tensor 4D Tensor of (batch x TimeLength x LabelLength x outputDim) from the Transducer network. labels : torch.Tensor 2D Tensor of (batch x MaxSeqLabelLength) containing targets of the batch with zero padding. alpha : torch.Tensor 3D Tensor of (batch x TimeLength x LabelLength) for backward computation. beta : torch.Tensor 3D Tensor of (batch x TimeLength x LabelLength) for backward computation. grads : torch.Tensor Grads for backward computation. T : torch.Tensor 1D Tensor of (batch) containing TimeLength of each target. U : torch.Tensor 1D Tensor of (batch) containing LabelLength of each target. blank : int Blank index. rrN)rrrrrrrange enumerate) rrr r.gradsr"r#r$r(r&r'lr+r+r,cu_kernel_compute_grads<  ,  ,,r4c@s(eZdZdZeddZeddZdS) Transducera This class implements the Transducer loss computation with forward-backward algorithm Sequence Transduction with naive implementation : https://arxiv.org/pdf/1211.3711.pdf This class use torch.autograd.Function. In fact of using the forward-backward algorithm, we need to compute the gradient manually. This class can't be instantiated, please refer to TransducerLoss class It is also possible to use this class directly by using Transducer.apply c Csl|}|j\}}} } tj||| | f|j|jd} tj||| f|j|jd} tj||| f|j|jd} tj|| ftj|jd}tj|f|j|jd}tj|f|j|jd}t|| f||| ||||||d}t|| f||| |||||t ||f||| | | |||| |_ ~ ~ ~~~~~~tj |dkr| S|dkrt| S|dkr| Std|)Computes the transducer loss.)dtypedevice)r8r7rmeansumnonezUnexpected reduction {})detachshapetorchzerosr7r8int32r-r/r4r2r empty_cacher9r: Exceptionformat)ctxrrr"r#r$ reductionBmaxTmaxUAr2r r.r% log_p_alpha log_p_betar+r+r,forwardsP        zTransducer.forwardcCs2|dddd|j}|j|ddddddfS)z.Backward computations for the transducer loss.rrN)viewtor2mul_)rD grad_outputr+r+r,backward>szTransducer.backwardN)__name__ __module__ __qualname____doc__ staticmethodrLrQr+r+r+r,r5s  ,r5cs*eZdZdZdfdd ZddZZS) TransducerLossa This class implements the Transduce loss computation with forward-backward algorithm. Sequence Transduction with naive implementation : https://arxiv.org/pdf/1211.3711.pdf The TransducerLoss(nn.Module) use Transducer(autograd.Function) to compute the forward-backward loss and gradients. Input tensors must be on a cuda device. Arguments --------- blank : int Token to use as blank token. reduction : str Type of reduction to use, default "mean" Example ------- >>> import torch >>> loss = TransducerLoss(blank=0) >>> logits = torch.randn((1,2,3,5)).cuda().requires_grad_() >>> labels = torch.Tensor([[1,2]]).cuda().int() >>> act_length = torch.Tensor([2]).cuda().int() >>> # U = label_length+1 >>> label_length = torch.Tensor([2]).cuda().int() >>> l = loss(logits, labels, act_length, label_length) >>> l.backward() rr9cst||_||_tj|_ztjWdSt yBd}|d7}|d7}|d7}|d7}|d7}|d7}|d7}|d 7}t |w) Nz,cannot import numba. To use Transducer loss r r r r rrrzDconda install numba cudatoolkit=XX (XX is your cuda toolkit version)) super__init__r$rEr5applylossr cuda_paths ImportError)selfr$rEerr_msg __class__r+r,rYcs(   zTransducerLoss.__init__cCs\tdd||||fDr|d}||||||j|jStd|j|j|j|jgd)r6css|]}|jVqdS)N)is_cuda).0r(r+r+r, {sz)TransducerLoss.forward..rzFound inputs tensors to be on zB while needed to be on a 'cuda' device to use the transducer loss.)all log_softmaxr[r$rE ValueErrorr8)r^logitsrr"r#rr+r+r,rLxs zTransducerLoss.forward)rr9)rRrSrTrUrYrL __classcell__r+r+r`r,rWEsrW) rUloggingrwarningsr>torch.autogradrtorch.nnrspeechbrain.utils.loggerr NUMBA_VERBOSErRloggerrrinfo getLogger nb_loggersetLevelERRORnumba.core.errorsr simplefilterr]r_jitr-r/r4r5rWr+r+r+r,s\         K G 5A