o eiW@shdZddlZddlZddlmZddlmZddlZddl Z ddl m Z ddl m m ZddlmZddlmZddlmZddlmZeeZ dVd d ZGd dde jZ dWddZ dXddZ dXddZ dXddZ     dYddZ     dZddZ!     d[ddZ"   d\d!d"Z#d#d$Z$d]d%d&Z%   ' d^d(d)Z&d_d+d,Z'   d`d-d.Z(d/d0Z)d1d2Z*d3d4Z+d5d6Z,d7d8Z-Gd9d:d:e jZ.Gd;d<dd>e jZ0d?d@Z1dAdBZ2dCdDZ3GdEdFdFe jZ4GdGdHdHe jZ5GdIdJdJe jZ6dKdLZ7edMgdNZ8edOdPdQgZ9GdRdSdSe jZ:GdTdUdUe jZ;dS)az Losses for training neural networks. Authors * Mirco Ravanelli 2020 * Samuele Cornell 2020 * Hwidong Na 2020 * Yan Gao 2020 * Titouan Parcollet 2020 N) namedtuple) permutations)length_to_mask)filter_ctc_output) unsqueeze_as) get_loggermeanTc Cs||jd}||jd}|rJzddlm}Wnty=d}|d7}|d7}|d7}|d7}t|w|||||||d Sdd lm} |d } | | |||||S) a?Transducer loss, see `speechbrain/nnet/loss/transducer_loss.py`. Arguments --------- logits : torch.Tensor Predicted tensor, of shape [batch, maxT, maxU, num_labels]. targets : torch.Tensor Target tensor, without any blanks, of shape [batch, target_len]. input_lens : torch.Tensor Length of each utterance. target_lens : torch.Tensor Length of each target sequence. blank_index : int The location of the blank symbol among the label indices. reduction : str Specifies the reduction to apply to the output: 'mean' | 'batchmean' | 'sum'. use_torchaudio: bool If True, use Transducer loss implementation from torchaudio, otherwise, use Speechbrain Numba implementation. Returns ------- The computed transducer loss. r) rnnt_losszEThe dependency torchaudio >= 0.10.0 is needed to use Transducer Loss z/Cannot import torchaudio.functional.rnnt_loss. z/To use it, please install torchaudio >= 0.10.0 z================== zMOtherwise, you can use our numba implementation, set `use_torchaudio=False`. )blank reduction) Transducer) shaperoundinttorchaudio.functionalr ImportError%speechbrain.nnet.loss.transducer_lossr log_softmaxapply) logitstargets input_lens target_lens blank_indexr use_torchaudior err_msgr log_probsrU/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/speechbrain/nnet/losses.pytransducer_losss4!   r!cs@eZdZdZfddZddZddZdd Zd d ZZ S) PitWrappera Permutation Invariant Wrapper to allow Permutation Invariant Training (PIT) with existing losses. Permutation invariance is calculated over the sources/classes axis which is assumed to be the rightmost dimension: predictions and targets tensors are assumed to have shape [batch, ..., channels, sources]. Arguments --------- base_loss : function Base loss function, e.g. torch.nn.MSELoss. It is assumed that it takes two arguments: predictions and targets and no reduction is performed. (if a pytorch loss is used, the user must specify reduction="none"). Example ------- >>> pit_mse = PitWrapper(nn.MSELoss(reduction="none")) >>> targets = torch.rand((2, 32, 4)) >>> p = (3, 0, 2, 1) >>> predictions = targets[..., p] >>> loss, opt_p = pit_mse(predictions, targets) >>> loss tensor([0., 0.]) ct||_dSN)super__init__ base_loss)selfr' __class__rr r&{  zPitWrapper.__init__cCsZd}d}tt|jdD]}|t|jd|f}|dus$||kr(|}|}q ||fS)a Arguments --------- loss_mat : torch.Tensor Tensor of shape [sources, source] containing loss values for each possible permutation of predictions. Returns ------- loss : torch.Tensor Permutation invariant loss for the current batch, tensor of shape [1] assigned_perm : tuple Indexes for optimal permutation of the input over sources which minimizes the loss. Nr)rrangerr)r(loss_matloss assigned_permpc_lossrrr _fast_pitszPitWrapper._fast_pitcCs|d}|djgddtt|jdD|dR}|djdgddtt|jdD|R}|||}t|jdksKJdd dtt|jD}|j|d dd }||S) a2 Arguments --------- pred : torch.Tensor Network prediction for the current example, tensor of shape [..., sources]. target : torch.Tensor Target for the current example, tensor of shape [..., sources]. Returns ------- loss : torch.Tensor Permutation invariant loss for the current example, tensor of shape [1] assigned_perm : tuple Indexes for optimal permutation of the input over sources which minimizes the loss. rcSg|]}dqSr r.0xrrr z-PitWrapper._opt_perm_loss..r cSr4r5rr6rrr r9r:z4Base loss should not perform any reduction operationcSsg|]}|qSrrr6rrr r9r:Ndim) size unsqueezerepeatr,lenrr'rr2)r(predtarget n_sourcesr- mean_overrrr _opt_perm_losss,     zPitWrapper._opt_perm_losscCsDtj||jd}t|jdD]}||d||f||<q|S)a Arguments --------- tensor : torch.Tensor torch.Tensor to reorder given the optimal permutation, of shape [batch, ..., sources]. p : list of tuples List of optimal permutations, e.g. for batch=2 and n_sources=3 [(0, 1, 2), (0, 2, 1]. Returns ------- reordered : torch.Tensor Reordered tensor given permutation p. devicer.)torch zeros_likerHr,rclone)r(tensorr0 reorderedbrrr reorder_tensorszPitWrapper.reorder_tensorc CsRg}g}t||D]\}}|||\}}||||q t|}||fS)a| Arguments --------- preds : torch.Tensor Network predictions tensor, of shape [batch, channels, ..., sources]. targets : torch.Tensor Target tensor, of shape [batch, channels, ..., sources]. Returns ------- loss : torch.Tensor Permutation invariant loss for current examples, tensor of shape [batch] perms : list List of indexes for optimal permutation of the inputs over sources. e.g., [(0, 1, 2), (2, 1, 0)] for three sources and 2 examples per batch. )ziprFappendrIstack) r(predsrlossespermsrBlabelr.r0rrr forwards   zPitWrapper.forward) __name__ __module__ __qualname____doc__r&r2rFrOrW __classcell__rrr)r r"_s %r"c Cs||jd}||jd}|dd}|dkr#d}n |dkr*d}n|}tjjj|||||d|d}|dkrE||jdS|dkr`|d}| |d  d| |d  dS|S) a CTC loss. Arguments --------- log_probs : torch.Tensor Predicted tensor, of shape [batch, time, chars]. targets : torch.Tensor Target tensor, without any blanks, of shape [batch, target_len] input_lens : torch.Tensor Length of each utterance. target_lens : torch.Tensor Length of each target sequence. blank_index : int The location of the blank symbol among the character indexes. reduction : str What reduction to apply to the output. 'mean', 'sum', 'batch', 'batchmean', 'none'. See pytorch for 'mean', 'sum', 'none'. The 'batch' option returns one loss per item in the batch, 'batchmean' returns sum / batch size. Returns ------- The computed CTC loss. r r batchmeansumbatchnoneT) zero_infinityr r) rrr transposerInn functionalctc_lossr>viewr^) rrrrrr reduction_lossr.Nrrr res.   $recC6t|||\}}tjtjjjdd}t|||||dS)aCompute the true l1 loss, accounting for length differences. Arguments --------- predictions : torch.Tensor Predicted tensor, of shape ``[batch, time, *]``. targets : torch.Tensor Target tensor with the same size as predicted tensor. length : torch.Tensor Length of each utterance for computing true error with a mask. allowed_len_diff : int Length difference that will be tolerated before raising an exception. reduction : str Options are 'mean', 'batch', 'batchmean', 'sum'. See pytorch for 'mean', 'sum'. The 'batch' option returns one loss per item in the batch, 'batchmean' returns sum / batch size. Returns ------- The computed L1 loss. Example ------- >>> probs = torch.tensor([[0.9, 0.1, 0.1, 0.9]]) >>> l1_loss(probs, torch.tensor([[1., 0., 0., 1.]])) tensor(0.1000) r`r )truncate functoolspartialrIrcrdl1_losscompute_masked_loss predictionsrlengthallowed_len_diffr r.rrr ro+  rocCrj)aCompute the true mean squared error, accounting for length differences. Arguments --------- predictions : torch.Tensor Predicted tensor, of shape ``[batch, time, *]``. targets : torch.Tensor Target tensor with the same size as predicted tensor. length : torch.Tensor Length of each utterance for computing true error with a mask. allowed_len_diff : int Length difference that will be tolerated before raising an exception. reduction : str Options are 'mean', 'batch', 'batchmean', 'sum'. See pytorch for 'mean', 'sum'. The 'batch' option returns one loss per item in the batch, 'batchmean' returns sum / batch size. Returns ------- The computed MSE loss. Example ------- >>> probs = torch.tensor([[0.9, 0.1, 0.1, 0.9]]) >>> mse_loss(probs, torch.tensor([[1., 0., 0., 1.]])) tensor(0.0100) r`rk)rlrmrnrIrcrdmse_lossrprqrrr rvPrurvcsNtjdkrt|jdkrt||\}fdd}t||||dS)aComputes the classification error at frame or batch level. Arguments --------- probabilities : torch.Tensor The posterior probabilities of shape [batch, prob] or [batch, frames, prob] targets : torch.Tensor The targets, of shape [batch] or [batch, frames] length : torch.Tensor Length of each utterance, if frame-level loss is desired. allowed_len_diff : int Length difference that will be tolerated before raising an exception. reduction : str Options are 'mean', 'batch', 'batchmean', 'sum'. See pytorch for 'mean', 'sum'. The 'batch' option returns one loss per item in the batch, 'batchmean' returns sum / batch size. Returns ------- The computed classification error. Example ------- >>> probs = torch.tensor([[[0.9, 0.1], [0.1, 0.9]]]) >>> classification_error(probs, torch.tensor([1, 1])) tensor(0.5000) rir;cstjdd}||kS)z"Computes the classification error.rr<)rIargmaxfloat)rrr probabilitiesrr errors z#classification_error..errorrk)rArrlrplong)rzrrsrtr r{rryr classification_errorus r}cCsXt|jdkrt|||\}}|dd}tjtjjj |dd}t ||| |||dS)aComputes negative log likelihood loss. Arguments --------- log_probabilities : torch.Tensor The probabilities after log has been applied. Format is [batch, log_p] or [batch, frames, log_p]. targets : torch.Tensor The targets, of shape [batch] or [batch, frames]. length : torch.Tensor Length of each utterance, if frame-level loss is desired. label_smoothing : float The amount of smoothing to apply to labels (default 0.0, no smoothing) allowed_len_diff : int Length difference that will be tolerated before raising an exception. weight: torch.Tensor A manual rescaling weight given to each class. If given, has to be a Tensor of size C. reduction : str Options are 'mean', 'batch', 'batchmean', 'sum'. See pytorch for 'mean', 'sum'. The 'batch' option returns one loss per item in the batch, 'batchmean' returns sum / batch size. Returns ------- The computed NLL loss. Example ------- >>> probs = torch.tensor([[0.9, 0.1], [0.1, 0.9]]) >>> nll_loss(torch.log(probs), torch.tensor([1, 1])) tensor(1.2040) rir rr`)weightr label_smoothingr ) rArrlrbrmrnrIrcrdnll_lossrpr|)log_probabilitiesrrsrrtrr r.rrr rs *  rc Cst|jt|jdkr|d}t|jdkr!t|||\}}n|dur)td|d|d}}|durC|dkrC|d}tjt j j j ||dd}t ||||||dS) uComputes binary cross-entropy (BCE) loss. It also applies the sigmoid function directly (this improves the numerical stability). Arguments --------- inputs : torch.Tensor The output before applying the final softmax Format is [batch[, 1]?] or [batch, frames[, 1]?]. (Works with or without a singleton dimension at the end). targets : torch.Tensor The targets, of shape [batch] or [batch, frames]. length : torch.Tensor Length of each utterance, if frame-level loss is desired. weight : torch.Tensor A manual rescaling weight if provided it’s repeated to match input tensor shape. pos_weight : torch.Tensor A weight of positive examples. Must be a vector with length equal to the number of classes. reduction: str Options are 'mean', 'batch', 'batchmean', 'sum'. See pytorch for 'mean', 'sum'. The 'batch' option returns one loss per item in the batch, 'batchmean' returns sum / batch size. allowed_len_diff : int Length difference that will be tolerated before raising an exception. label_smoothing : float The amount of smoothing to apply to labels (default 0.0, no smoothing) Returns ------- The computed BCE loss. Example ------- >>> inputs = torch.tensor([10.0, -6.0]) >>> targets = torch.tensor([1, 0]) >>> bce_loss(inputs, targets) tensor(0.0013) r rr;Nz+length can be passed only for >= 2D inputs.r`)r pos_weightr r)rArsqueezerl ValueErrorr?r=rmrnrIrcrd binary_cross_entropy_with_logitsrprx) inputsrrsrrr rtrr.rrr bce_losss.2  rcCsN|dkr|dkr|d}|j\}}} |}d|} |d| }|d}t)|} | || d||k} | | d}| d|d| Wdn1sZwYtj j j|| dd} | | dd} |dkr|| S|d kr| |S|d kr| |dd|S|d kr| S| St||||dS) aComputes the KL-divergence error at the batch level. This loss applies label smoothing directly to the targets Arguments --------- log_probabilities : torch.Tensor The posterior probabilities of shape [batch, prob] or [batch, frames, prob]. targets : torch.Tensor The targets, of shape [batch] or [batch, frames]. length : torch.Tensor Length of each utterance, if frame-level loss is desired. label_smoothing : float The amount of smoothing to apply to labels (default 0.0, no smoothing) allowed_len_diff : int Length difference that will be tolerated before raising an exception. pad_idx : int Entries of this value are considered padding. reduction : str Options are 'mean', 'batch', 'batchmean', 'sum'. See pytorch for 'mean', 'sum'. The 'batch' option returns one loss per item in the batch, 'batchmean' returns sum / batch size. Returns ------- The computed kldiv loss. Example ------- >>> probs = torch.tensor([[0.9, 0.1], [0.1, 0.9]]) >>> kldiv_loss(torch.log(probs), torch.tensor([1, 1])) tensor(1.2040) rr;r rNr`rkrr]r_r^)r=r?rr|detachrfrIno_gradrKfill_ masked_fillscatter_rcrdkl_divr^rr)rrrsrrtpad_idxr bztimen_class confidencetrue_distributionignorer.rrr kldiv_loss5s:*          r?Y@cCs ttjt||d||||ddS)aA loss function that can be used in cases where a model outputs an arbitrary probability distribution for a discrete variable on an interval scale, such as the length of a sequence, and the ground truth is the precise values of the variable from a data sample. The loss is defined as loss_i = p_i * exp(beta * |i - y|) - 1. The loss can also be used where outputs aren't probabilities, so long as high values close to the ground truth position and low values away from it are desired Arguments --------- predictions: torch.Tensor a (batch x max_len) tensor in which each element is a probability, weight or some other value at that position targets: torch.Tensor a 1-D tensor in which each element is thr ground truth length: torch.Tensor lengths (for masking in padded batches) beta: torch.Tensor a hyperparameter controlling the penalties. With a higher beta, penalties will increase faster max_weight: torch.Tensor the maximum distance weight (for numerical stability in long sequences) reduction: str Options are 'mean', 'batch', 'batchmean', 'sum'. See pytorch for 'mean', 'sum'. The 'batch' option returns one loss per item in the batch, 'batchmean' returns sum / batch size Returns ------- The masked loss. Example ------- >>> predictions = torch.tensor( ... [[0.25, 0.5, 0.25, 0.0], ... [0.05, 0.05, 0.9, 0.0], ... [8.0, 0.10, 0.05, 0.05]] ... ) >>> targets = torch.tensor([2., 3., 1.]) >>> length = torch.tensor([.75, .75, 1.]) >>> loss = distance_diff_loss(predictions, targets, length) >>> loss tensor(0.2967) )beta max_weightr.)rrrrsr mask_shape)rprmrn_distance_diff_loss)rrrrsrrr rrr distance_diff_losss8rc Csb|j\}}t|d|d|j}||d}||dj |d}||dS)atComputes the raw (unreduced) distance difference loss Arguments --------- predictions: torch.Tensor a (batch x max_len) tensor in which each element is a probability, weight or some other value at that position targets: torch.Tensor a 1-D tensor in which each element is thr ground truth beta: torch.Tensor a hyperparameter controlling the penalties. With a higher beta, penalties will increase faster max_weight: torch.Tensor the maximum distance weight (for numerical stability in long sequences) Returns ------- The raw distance loss. rr r?)max) rrIaranger?r@torHabsexpclamp) rrrrr batch_sizemax_len pos_range diff_range loss_weightsrrr rs rcCs|jd|jd}|dkr||fSt||kr&td|jd|jdf|dkr9||ddd|jdffS|ddd|jdf|fS)aEnsure that predictions and targets are the same length. Arguments --------- predictions : torch.Tensor First tensor for checking length. targets : torch.Tensor Second tensor for checking length. allowed_len_diff : int Length difference that will be tolerated before raising an exception. Returns ------- predictions : torch.Tensor targets : torch.Tensor Same as inputs, but with the same shape. r rzNPredictions and targets should be same length, but got %s and %s respectively.N)rrr)rrrrtlen_diffrrr rls rlrc Csf|||}|dkr |}n|dkr|}n|dkr|}ntd|t||} || 9}t|| ||||S)aCompute the true average loss of a set of waveforms of unequal length. Arguments --------- loss_fn : function A function for computing the loss taking just predictions and targets. Should return all the losses, not a reduction (e.g. reduction="none"). predictions : torch.Tensor First argument to loss function. targets : torch.Tensor Second argument to loss function. length : torch.Tensor Length of each utterance to compute mask. If None, global average is computed and returned. label_smoothing: float The proportion of label smoothing. Should only be used for NLL loss. Ref: Regularizing Neural Networks by Penalizing Confident Output Distributions. https://arxiv.org/abs/1701.06548 mask_shape: torch.Tensor the shape of the mask The default is "targets", which will cause the mask to be the same shape as the targets Other options include "predictions" and "loss", which will use the shape of the predictions and the unreduced loss, respectively. These are useful for loss functions that whose output does not match the shape of the targets reduction : str One of 'mean', 'batch', 'batchmean', 'none' where 'mean' returns a single value and 'batch' returns one per item in the batch and 'batchmean' is sum / batch_size and 'none' returns all. Returns ------- The masked loss. rrrr.zInvalid mask_shape value )rcompute_length_mask reduce_loss) loss_fnrrrrsrrr r. mask_datamaskrrr rps /  rpr cCst|}|dur>t||j||j|d}t|jt|jkr0|d}t|jt|jks!||jd|}||9}|S)aComputes a length mask for the specified data shape Arguments --------- data: torch.Tensor the data shape length: torch.Tensor the length of the corresponding data samples len_dim: int the length dimension (defaults to 1) Returns ------- mask: torch.Tensor the mask Example ------- >>> data = torch.arange(5)[None, :, None].repeat(3, 1, 2) >>> data += torch.arange(1, 4)[:, None, None] >>> data *= torch.arange(1, 3)[None, None, :] >>> data tensor([[[ 1, 2], [ 2, 4], [ 3, 6], [ 4, 8], [ 5, 10]], [[ 2, 4], [ 3, 6], [ 4, 8], [ 5, 10], [ 6, 12]], [[ 3, 6], [ 4, 8], [ 5, 10], [ 6, 12], [ 7, 14]]]) >>> compute_length_mask(data, torch.tensor([1., .4, .8])) tensor([[[1, 1], [1, 1], [1, 1], [1, 1], [1, 1]], [[1, 1], [1, 1], [0, 0], [0, 0], [0, 0]], [[1, 1], [1, 1], [1, 1], [1, 1], [0, 0]]]) >>> compute_length_mask(data, torch.tensor([.5, 1., .5]), len_dim=2) tensor([[[1, 0], [1, 0], [1, 0], [1, 0], [1, 0]], [[1, 1], [1, 1], [1, 1], [1, 1], [1, 1]], [[1, 0], [1, 0], [1, 0], [1, 0], [1, 0]]]) N)rrr ) rI ones_likerrrAr?typedtyperb)datarslen_dimr length_maskrrr rHs M  rcCs|d}|dkr|t|}n!|dkr||}n|dkr4||dd||dd}|dkr:|Stj|dd|}|dkrRt|t|}n|dkrat||jd}n|dkro|d|d}| |d||S)a*Performs the specified reduction of the raw loss value Arguments --------- loss : function A function for computing the loss taking just predictions and targets. Should return all the losses, not a reduction (e.g. reduction="none"). mask : torch.Tensor Mask to apply before computing loss. reduction : str One of 'mean', 'batch', 'batchmean', 'none' where 'mean' returns a single value and 'batch' returns one per item in the batch and 'batchmean' is sum / batch_size and 'none' returns all. label_smoothing: float The proportion of label smoothing. Should only be used for NLL loss. Ref: Regularizing Neural Networks by Penalizing Confident Output Distributions. https://arxiv.org/abs/1701.06548 predictions : torch.Tensor First argument to loss function. Required only if label smoothing is used. targets : torch.Tensor Second argument to loss function. Required only if label smoothing is used. Returns ------- Reduced loss. rrr]r_rr r<)r>r^rIreshaperr)r.rr rrrrrhloss_regrrr rs" "$rcCtt}|||\}}|S)aThis function wraps si_snr calculation with the speechbrain pit-wrapper. Arguments --------- source: torch.Tensor Shape is [B, T, C], Where B is the batch size, T is the length of the sources, C is the number of sources the ordering is made so that this loss is compatible with the class PitWrapper. estimate_source: torch.Tensor The estimated source, of shape [B, T, C] Returns ------- loss: torch.Tensor The computed SNR Example ------- >>> x = torch.arange(600).reshape(3, 100, 2) >>> xhat = x[:, :, (1, 0)] >>> si_snr = -get_si_snr_with_pitwrapper(x, xhat) >>> print(si_snr) tensor([135.2284, 135.2284, 135.2284]) )r" cal_si_snr)sourceestimate_source pit_si_snrr.rUrrr get_si_snr_with_pitwrappersrcCr)a This function wraps snr calculation with the speechbrain pit-wrapper. Arguments --------- source: torch.Tensor Shape is [B, T, E, C], Where B is the batch size, T is the length of the sources, E is binaural channels, C is the number of sources the ordering is made so that this loss is compatible with the class PitWrapper. estimate_source: torch.Tensor The estimated source, of shape [B, T, E, C] Returns ------- loss: torch.Tensor The computed SNR )r"cal_snr)rrpit_snrr.rUrrr get_snr_with_pitwrappersrcCs>d}||ks J|jj}tj|jdg|jd|d}t||}||9}|ddd }tj |ddd|}tj |ddd|}||} ||} | |9} | |9} | } | } tj | | ddd} tj | d ddd|}| | |}| |}tj |d dd tj |d dd |}d t ||}| d S) aCalculate SI-SNR. Arguments --------- source: torch.Tensor Shape is [T, B, C], Where B is batch size, T is the length of the sources, C is the number of sources the ordering is made so that this loss is compatible with the class PitWrapper. estimate_source: torch.Tensor The estimated source, of shape [T, B, C] Returns ------- The calculated SI-SNR. Example: --------- >>> import numpy as np >>> x = torch.Tensor([[1, 0], [123, 45], [34, 5], [2312, 421]]) >>> xhat = x[:, (1, 0)] >>> x = x.unsqueeze(-1).repeat(1, 1, 2) >>> xhat = xhat.unsqueeze(1).repeat(1, 2, 1) >>> si_snr = -cal_si_snr(x, xhat) >>> print(si_snr) tensor([[[ 25.2142, 144.1789], [130.9283, 25.2142]]]) :0yE>rr3rGr rTr=keepdimr;r< r>rHrrIrLrget_mask contiguousrrxr^log10r?)rrEPSrHsource_lengthsr num_samples mean_target mean_estimatezero_mean_targetzero_mean_estimates_target s_estimatedots_target_energyproje_noisesi_snr_beforelogsi_snrrrr rs:   rcCsd}||ks J|jj}tj|jdg|jd|d}t||}||9}|ddd }tj |ddd|}tj |ddd|}||} ||} | |9} | |9} | } | } tj | d dd tj | | d dd |} d t | |}| d S) aCalculate binaural channel SNR. Arguments --------- source: torch.Tensor Shape is [T, E, B, C] Where B is batch size, T is the length of the sources, E is binaural channels, C is the number of sources the ordering is made so that this loss is compatible with the class PitWrapper. estimate_source: torch.Tensor The estimated source, of shape [T, E, B, C] Returns ------- Binaural channel SNR rrr3rGr rTrr;r<rr)rrrrHrrrrrrrrr snr_beforelogsnrrrr r]s0  rcCs\||ddddd}|d}t|D] }d|||d|f<q|ddS)a Arguments --------- source : torch.Tensor Shape [T, B, C] source_lengths : torch.Tensor Shape [B] Returns ------- mask : torch.Tensor Shape [T, B, 1] Example ------- >>> source = torch.randn(4, 3, 2) >>> source_lengths = torch.Tensor([2, 1, 4]).int() >>> mask = get_mask(source, source_lengths) >>> print(mask) tensor([[[1.], [1.], [1.]], [[1.], [0.], [1.]], [[0.], [0.], [1.]], [[0.], [0.], [1.]]]) Nrr r3r)new_onesr>r?rbr,)rrrBirrr rs $$   rcs*eZdZdZdfdd ZddZZS) AngularMargina An implementation of Angular Margin (AM) proposed in the following paper: '''Margin Matters: Towards More Discriminative Deep Neural Network Embeddings for Speaker Recognition''' (https://arxiv.org/abs/1906.07317) Arguments --------- margin : float The margin for cosine similarity scale : float The scale for cosine similarity Example ------- >>> pred = AngularMargin() >>> outputs = torch.tensor([ [1., -1.], [-1., 1.], [0.9, 0.1], [0.1, 0.9] ]) >>> targets = torch.tensor([ [1., 0.], [0., 1.], [ 1., 0.], [0., 1.] ]) >>> predictions = pred(outputs, targets) >>> predictions[:,0] > predictions[:,1] tensor([ True, False, True, False]) r~rcst||_||_dSr$)r%r&marginscale)r(rrr)rr r&s  zAngularMargin.__init__cCs||j|}|j|S)a`Compute AM between two tensors Arguments --------- outputs : torch.Tensor The outputs of shape [N, C], cosine similarity is required. targets : torch.Tensor The targets of shape [N, C], where the margin is applied for. Returns ------- predictions : torch.Tensor )rr)r(outputsrrrr rWs zAngularMargin.forward)r~rrXrYrZr[r&rWr\rrr)r rsrcs*eZdZdZd fdd ZddZZS) AdditiveAngularMargina An implementation of Additive Angular Margin (AAM) proposed in the following paper: '''Margin Matters: Towards More Discriminative Deep Neural Network Embeddings for Speaker Recognition''' (https://arxiv.org/abs/1906.07317) Arguments --------- margin : float The margin for cosine similarity. scale : float The scale for cosine similarity. easy_margin : bool Example ------- >>> outputs = torch.tensor([ [1., -1.], [-1., 1.], [0.9, 0.1], [0.1, 0.9] ]) >>> targets = torch.tensor([ [1., 0.], [0., 1.], [ 1., 0.], [0., 1.] ]) >>> pred = AdditiveAngularMargin() >>> predictions = pred(outputs, targets) >>> predictions[:,0] > predictions[:,1] tensor([ True, False, True, False]) r~rFcsbt||||_t|j|_t|j|_ttj |j|_ ttj |j|j|_ dSr$) r%r& easy_marginmathcosrcos_msinsin_mpithmm)r(rrrr)rr r&s zAdditiveAngularMargin.__init__cCs|}t|dd}tdt|d}||j||j}|jr-t|dk||}n t||j k|||j }||d||}|j |S)aj Compute AAM between two tensors Arguments --------- outputs : torch.Tensor The outputs of shape [N, C], cosine similarity is required. targets : torch.Tensor The targets of shape [N, C], where the margin is applied for. Returns ------- predictions : torch.Tensor gPgP?rr;r) rxrIrsqrtpowrrrwhererrr)r(rrcosinesinephirrr rW s zAdditiveAngularMargin.forward)r~rFrrrr)r rs rcs*eZdZdZfddZdddZZS)LogSoftmaxWrappera= Arguments --------- loss_fn : Callable The LogSoftmax function to wrap. Example ------- >>> outputs = torch.tensor([ [1., -1.], [-1., 1.], [0.9, 0.1], [0.1, 0.9] ]) >>> outputs = outputs.unsqueeze(1) >>> targets = torch.tensor([ [0], [1], [0], [1] ]) >>> log_prob = LogSoftmaxWrapper(nn.Identity()) >>> loss = log_prob(outputs, targets) >>> 0 <= loss < 1 tensor(True) >>> log_prob = LogSoftmaxWrapper(AngularMargin(margin=0.2, scale=32)) >>> loss = log_prob(outputs, targets) >>> 0 <= loss < 1 tensor(True) >>> outputs = torch.tensor([ [1., -1.], [-1., 1.], [0.9, 0.1], [0.1, 0.9] ]) >>> log_prob = LogSoftmaxWrapper(AdditiveAngularMargin(margin=0.3, scale=32)) >>> loss = log_prob(outputs, targets) >>> 0 <= loss < 1 tensor(True) cs$t||_tjjdd|_dS)Nr^rk)r%r&rrIrc KLDivLoss criterion)r(rr)rr r&Cs zLogSoftmaxWrapper.__init__NcCs|d}|d}t||jd}z|||}Wnty-||}Ynwtj|dd}| ||| }|S)a Arguments --------- outputs : torch.Tensor Network output tensor, of shape [batch, 1, outdim]. targets : torch.Tensor Target tensor, of shape [batch, 1]. length : torch.Tensor The lengths of the corresponding inputs. Returns ------- loss: torch.Tensor Loss for current examples. r r<) rFone_hotr|rrxr TypeErrorrrr^)r(rrrsrrr.rrr rWHs   zLogSoftmaxWrapper.forwardr$rrrr)r r(s rcCsPtj|dd\}}g}g}t|jdD]3} || } || |jd} | d| } tt|  |d} t | } | | | | qt|} t|jdD]} | || }t|D] }||  dq_qSt t |}|||}t t |}||||jd}|dd}tjjj|||||ddS)aKnowledge distillation for CTC loss. Reference --------- Distilling Knowledge from Ensembles of Acoustic Models for Joint CTC-Attention End-to-End Speech Recognition. https://arxiv.org/abs/2005.09310 Arguments --------- log_probs : torch.Tensor Predicted tensor from student model, of shape [batch, time, chars]. targets : torch.Tensor Predicted tensor from single teacher model, of shape [batch, time, chars]. input_lens : torch.Tensor Length of each utterance. blank_index : int The location of the blank symbol among the character indexes. device : str Device for computing. Returns ------- The computed CTC loss. rr<rr )blank_idT)ra)rIrr,rrrrlistcpunumpyrArQ from_numpynparrayrrbrcrdre)rrrrrHscoresrr pred_list pred_len_listj current_pred actual_sizecurrent_pred_len max_pred_lendiffnfake_labfake_lab_lengthsrrr ctc_loss_kdfsD        r cCs| |dS)aiSimple version of distillation for cross-entropy loss. Arguments --------- inp : torch.Tensor The probabilities from student model, of shape [batch_size * length, feature] target : torch.Tensor The probabilities from teacher model, of shape [batch_size * length, feature] Returns ------- The distilled outputs. r )r^)inprCrrr ce_kdsrc Cs|jd}|jd}t||jd}||||jd}t||tj|jd}||||jd}t||} t | |||t |} | S)aKnowledge distillation for negative log-likelihood loss. Reference --------- Distilling Knowledge from Ensembles of Acoustic Models for Joint CTC-Attention End-to-End Speech Recognition. https://arxiv.org/abs/2005.09310 Arguments --------- probabilities : torch.Tensor The predicted probabilities from the student model. Format is [batch, frames, p] targets : torch.Tensor The target probabilities from the teacher model. Format is [batch, frames, p] rel_lab_lengths : torch.Tensor Length of each utterance, if the frame-level loss is desired. Returns ------- Computed NLL KD loss. Example ------- >>> probabilities = torch.tensor([[[0.8, 0.2], [0.2, 0.8]]]) >>> targets = torch.tensor([[[0.9, 0.1], [0.1, 0.9]]]) >>> rel_lab_lengths = torch.tensor([1.]) >>> nll_loss_kd(probabilities, targets, rel_lab_lengths) tensor(-0.7400) rr r)rrrH) rrIrrrrrxrHrr^) rzrrel_lab_lengthsN_sntr lab_lengths prob_currrlab_currr.rrr nll_loss_kds    rcs(eZdZdZfddZddZZS)ContrastiveLossa&Contrastive loss as used in wav2vec2. Reference --------- wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations https://arxiv.org/abs/2006.11477 Arguments --------- logit_temp : torch.Float A temperature to divide the logits. cr#r$)r%r& logit_temp)r(rr)rr r&r+zContrastiveLoss.__init__c Cs||kd}|d}tj||gdd}tj||dd|}|r2td|dd|<|dd d| d}tj | dtj |j d}tj||j|d d }t|ddk|| d} || fS) aCompute contrastive loss. Arguments --------- x : torch.Tensor Encoded embeddings with shape (B, T, C). y : torch.Tensor Feature extractor target embeddings with shape (B, T, C). negs : torch.Tensor Negative embeddings from feature extractor with shape (N, B, T, C) where N is number of negatives. Can be obtained with our sample_negatives function (check in lobes/wav2vec2). Returns ------- loss : torch.Tensor The computed loss accuracy : torch.Tensor The computed accuracy rrr<z-infr Nr;)rrHr^rk)allr?rIcatcosine_similarityrxtype_asanyrbrr>zerosr|rHr cross_entropyrr^rwnumel) r(r8ynegs neg_is_postarget_and_negativesrrr.accuracyrrr rWs*  zContrastiveLoss.forwardrrrr)r rs rcsFeZdZdZdfdd Zddd Zdd d Zd d ZddZZ S)VariationalAutoencoderLossa"The Variational Autoencoder loss, with support for length masking From Autoencoding Variational Bayes: https://arxiv.org/pdf/1312.6114.pdf Arguments --------- rec_loss: callable a function or module to compute the reconstruction loss len_dim: int the dimension to be used for the length, if encoding sequences of variable length dist_loss_weight: float the relative weight of the distribution loss (K-L divergence) Example ------- >>> from speechbrain.nnet.autoencoders import VariationalAutoencoderOutput >>> vae_loss = VariationalAutoencoderLoss(dist_loss_weight=0.5) >>> predictions = VariationalAutoencoderOutput( ... rec=torch.tensor( ... [[0.8, 1.0], ... [1.2, 0.6], ... [0.4, 1.4]] ... ), ... mean=torch.tensor( ... [[0.5, 1.0], ... [1.5, 1.0], ... [1.0, 1.4]], ... ), ... log_var=torch.tensor( ... [[0.0, -0.2], ... [2.0, -2.0], ... [0.2, 0.4]], ... ), ... latent=torch.randn(3, 1), ... latent_sample=torch.randn(3, 1), ... latent_length=torch.tensor([1., 1., 1.]), ... ) >>> targets = torch.tensor( ... [[0.9, 1.1], ... [1.4, 0.6], ... [0.2, 1.4]] ... ) >>> loss = vae_loss(predictions, targets) >>> loss tensor(1.1264) >>> details = vae_loss.details(predictions, targets) >>> details #doctest: +NORMALIZE_WHITESPACE VariationalAutoencoderLossDetails(loss=tensor(1.1264), rec_loss=tensor(0.0333), dist_loss=tensor(2.1861), weighted_dist_loss=tensor(1.0930)) Nr MbP?cs,t|dur t}||_||_||_dSr$)r%r&rvrec_lossdist_loss_weightr)r(r&rr'r)rr r&is  z#VariationalAutoencoderLoss.__init__r]cCs|||||jS)a?Computes the forward pass Arguments --------- predictions: speechbrain.nnet.autoencoders.VariationalAutoencoderOutput the variational autoencoder output targets: torch.Tensor the reconstruction targets length : torch.Tensor Length of each sample for computing true error with a mask. reduction: str The type of reduction to apply, default "batchmean" Returns ------- loss: torch.Tensor the VAE loss (reconstruction + K-L divergence) )detailsr.)r(rrrrsr rrr rWqsz"VariationalAutoencoderLoss.forwardc Cs`|dur t|d}|||\}}t|||}t|||}|j|}||}t||||S)a/Gets detailed information about the loss (useful for plotting, logs, etc.) Arguments --------- predictions: speechbrain.nnet.autoencoders.VariationalAutoencoderOutput the variational autoencoder output (or a tuple of rec, mean, log_var) targets: torch.Tensor targets for the reconstruction loss length : torch.Tensor Length of each sample for computing true error with a mask. reduction: str The type of reduction to apply, default "batchmean" Returns ------- details: VAELossDetails a namedtuple with the following parameters loss: torch.Tensor the combined loss rec_loss: torch.Tensor the reconstruction loss dist_loss: torch.Tensor the distribution loss (K-L divergence), raw value weighted_dist_loss: torch.Tensor the weighted value of the distribution loss, as used in the combined loss Nr)rIonesr>_compute_components_reduce_autoencoder_lossr'!VariationalAutoencoderLossDetails) r(rrrrsr r& dist_lossweighted_dist_lossr.rrr r(s   z"VariationalAutoencoderLoss.detailsc CsP|\}}}}}}||j||dd}|dd||d|}||fS)Nrkgr r;)_align_length_axisr&r) r(rrrrec_rlog_varr&r-rrr r*sz.VariationalAutoencoderLoss._compute_componentscC||jdSNr moveaxisrr(rLrrr r/z-VariationalAutoencoderLoss._align_length_axis)Nr r%Nr]) rXrYrZr[r&rWr(r*r/r\rrr)r r$2s6  * r$cs>eZdZdZd fdd ZdddZdd d Zd d ZZS)AutoencoderLossaAn implementation of a standard (non-variational) autoencoder loss Arguments --------- rec_loss: callable the callable to compute the reconstruction loss len_dim: int the dimension index to be used for length Example ------- >>> from speechbrain.nnet.autoencoders import AutoencoderOutput >>> ae_loss = AutoencoderLoss() >>> rec = torch.tensor( ... [[0.8, 1.0], ... [1.2, 0.6], ... [0.4, 1.4]] ... ) >>> predictions = AutoencoderOutput( ... rec=rec, ... latent=torch.randn(3, 1), ... latent_length=torch.tensor([1., 1.]) ... ) >>> targets = torch.tensor( ... [[0.9, 1.1], ... [1.4, 0.6], ... [0.2, 1.4]] ... ) >>> ae_loss(predictions, targets) tensor(0.0333) >>> ae_loss.details(predictions, targets) AutoencoderLossDetails(loss=tensor(0.0333), rec_loss=tensor(0.0333)) Nr cs&t|dur t}||_||_dSr$)r%r&rvr&r)r(r&rr)rr r&s  zAutoencoderLoss.__init__r]cCs$||j||jdd}t|||S)aComputes the autoencoder loss Arguments --------- predictions: speechbrain.nnet.autoencoders.AutoencoderOutput the autoencoder output targets: torch.Tensor targets for the reconstruction loss length: torch.Tensor Length of each sample for computing true error with a mask reduction: str The type of reduction to apply, default "batchmean" Returns ------- The computed loss. Nrk)r/r&r0r+)r(rrrrsr r&rrr rWs zAutoencoderLoss.forwardcCs|||||}t||S)aGets detailed information about the loss (useful for plotting, logs, etc.) This is provided mainly to make the loss interchangeable with more complex autoencoder loses, such as the VAE loss. Arguments --------- predictions: speechbrain.nnet.autoencoders.AutoencoderOutput the autoencoder output targets: torch.Tensor targets for the reconstruction loss length : torch.Tensor Length of each sample for computing true error with a mask. reduction: str The type of reduction to apply, default "batchmean" Returns ------- details: AutoencoderLossDetails a namedtuple with the following parameters loss: torch.Tensor the combined loss rec_loss: torch.Tensor the reconstruction loss )AutoencoderLossDetails)r(rrrrsr r.rrr r(s zAutoencoderLoss.detailscCr3r4r5r7rrr r/r8z"AutoencoderLoss._align_length_axisr4r9) rXrYrZr[r&rWr(r/r\rrr)r r:s #  r:cCsR|d}|durt|||}t|||}nt|}t||||d}|S)Nr rk)r>rr expand_asrIrr)r.rsr rr reduced_lossrrr r+"s  r+r,)r.r&r-r.r;r.r&cs6eZdZdZejffdd ZddZddZZ S) LaplacianaAComputes the Laplacian for image-like data Arguments --------- kernel_size: int the size of the Laplacian kernel dtype: torch.dtype the data type (optional) Example ------- >>> lap = Laplacian(3) >>> lap.get_kernel() tensor([[[[-1., -1., -1.], [-1., 8., -1.], [-1., -1., -1.]]]]) >>> data = torch.eye(6) + torch.eye(6).flip(0) >>> data tensor([[1., 0., 0., 0., 0., 1.], [0., 1., 0., 0., 1., 0.], [0., 0., 1., 1., 0., 0.], [0., 0., 1., 1., 0., 0.], [0., 1., 0., 0., 1., 0.], [1., 0., 0., 0., 0., 1.]]) >>> lap(data.unsqueeze(0)) tensor([[[ 6., -3., -3., 6.], [-3., 4., 4., -3.], [-3., 4., 4., -3.], [ 6., -3., -3., 6.]]]) cs.t||_||_|}|d|dS)Nkernel)r%r& kernel_sizer get_kernelregister_buffer)r(r@rr?r)rr r&Ws zLaplacian.__init__cCsPtj|j|j|jd }|jd}|jdd}||||f<|dd}|S)zComputes the Laplacian kernel)rr;rr)rIr)r@rr?)r(r? mid_position mid_valuerrr rA^s   zLaplacian.get_kernelcCst||jS)aComputes the Laplacian of image-like data Arguments --------- data: torch.Tensor a (B x C x W x H) or (B x C x H x W) tensor with image-like data Returns ------- The transformed outputs. )rconv2dr?)r(rrrr rWis zLaplacian.forward) rXrYrZr[rIfloat32r&rArWr\rrr)r r>7s  r>cs,eZdZdZd fdd Zd ddZZS) LaplacianVarianceLossa~The Laplacian variance loss - used to penalize blurriness in image-like data, such as spectrograms. The loss value will be the negative variance because the higher the variance, the sharper the image. Arguments --------- kernel_size: int the Laplacian kernel size len_dim: int the dimension to be used as the length Example ------- >>> lap_loss = LaplacianVarianceLoss(3) >>> data = torch.ones(6, 6).unsqueeze(0) >>> data tensor([[[1., 1., 1., 1., 1., 1.], [1., 1., 1., 1., 1., 1.], [1., 1., 1., 1., 1., 1.], [1., 1., 1., 1., 1., 1.], [1., 1., 1., 1., 1., 1.], [1., 1., 1., 1., 1., 1.]]]) >>> lap_loss(data) tensor(-0.) >>> data = ( ... torch.eye(6) + torch.eye(6).flip(0) ... ).unsqueeze(0) >>> data tensor([[[1., 0., 0., 0., 0., 1.], [0., 1., 0., 0., 1., 0.], [0., 0., 1., 1., 0., 0.], [0., 0., 1., 1., 0., 0.], [0., 1., 0., 0., 1., 0.], [1., 0., 0., 0., 0., 1.]]]) >>> lap_loss(data) tensor(-17.6000) rir cs t||_t|d|_dS)N)r@)r%r&rr> laplacian)r(r@rr)rr r&s zLaplacianVarianceLoss.__init__NcCsb||}||jd}t||}|dkr'tddt||D}| S|| }| S)a~Computes the Laplacian loss Arguments --------- predictions: torch.Tensor a (B x C x W x H) or (B x C x H x W) tensor length: torch.Tensor The length of the corresponding inputs. reduction: str "batch" or None Returns ------- loss: torch.Tensor the loss value r r_cSsg|] \}}||qSr) masked_selectvar)r7item item_maskrrr r9s z1LaplacianVarianceLoss.forward..) rHr6rrboolrIrRrPrIrJ)r(rrrsr rHrr.rrr rWs zLaplacianVarianceLoss.forward)rir )NNrrrr)r rGxs)rG)rT)r)Nrir)Nr~riNr)NNNrrir~)Nr~rirr)Nrrr)ri)Nr~rrr4)rr~NN)rGrrrr s          A 9 & & 1 A W S D $ B_ 8!H5+.=>C9? d A