o %ݫit@sdZddlZddlZddlmmZddlmZddlm Z ddl m Z m Z m Z mZddlmZmZddlmZddlmZdd lmZGd d d ejZGd d d ejZGdddejZGdddejZGdddejZddZd6ddZGdddZ GdddejZ!ddZ"d7d"d#Z#Gd$d%d%ejjZ$Gd&d'd'e Z%Gd(d)d)Z&d*d+Z'Gd,d-d-ejjZ(Gd.d/d/ejZ)Gd0d1d1ejZ*Gd2d3d3ejZ+Gd4d5d5ejZ,dS)8z Neural network modules for the FastSpeech 2: Fast and High-Quality End-to-End Text to Speech synthesis model Authors * Sathvik Udupa 2022 * Pradnya Kandarkar 2023 * Yingzhi Wang 2023 N)nn)_Loss)PositionalEncodingTransformerEncoderget_key_padding_maskget_mask_from_lengths)CNNlinear) Embedding)bce_loss) LayerNormc*eZdZdZdfdd ZddZZS) EncoderPreNeta?Embedding layer for tokens Arguments --------- n_vocab: int size of the dictionary of embeddings blank_id: int padding index out_channels: int the size of each embedding vector Example ------- >>> from speechbrain.nnet.embedding import Embedding >>> from speechbrain.lobes.models.FastSpeech2 import EncoderPreNet >>> encoder_prenet_layer = EncoderPreNet(n_vocab=40, blank_id=0, out_channels=384) >>> x = torch.rand(3, 5) >>> y = encoder_prenet_layer(x) >>> y.shape torch.Size([3, 5, 384]) cstt|||d|_dS)N)num_embeddings embedding_dimblank_id)super__init__r token_embedding)selfn_vocabr out_channels __class__X/home/ubuntu/.local/lib/python3.10/site-packages/speechbrain/lobes/models/FastSpeech2.pyr3s  zEncoderPreNet.__init__cCs|j|j|_||}|S)zComputes the forward pass Arguments --------- x: torch.Tensor a (batch, tokens) input tensor Returns ------- output: torch.Tensor the embedding layer output )rtodevice)rxrrrforward;s zEncoderPreNet.forward)r__name__ __module__ __qualname____doc__rr __classcell__rrrrrsrcs4eZdZdZ     d fdd Zdd ZZS) PostNeta FastSpeech2 Conv Postnet Arguments --------- n_mel_channels: int input feature dimension for convolution layers postnet_embedding_dim: int output feature dimension for convolution layers postnet_kernel_size: int postnet convolution kernel size postnet_n_convolutions: int number of convolution layers postnet_dropout: float dropout probability for postnet Pr?c stt|tj|||dd|_t|_t d|dD]}|j tj|||ddqtj|||dd|_ t |_ t||_t||_t||_t||_t||_t||_dS)Nsame in_channelsr kernel_sizepadding)rr'rrConv1dconv_prer ModuleListconvs_intermediaterangeappend conv_postTanhtanhr ln1ln2ln3Dropoutdropout1dropout2dropout3)rn_mel_channelspostnet_embedding_dimpostnet_kernel_sizepostnet_n_convolutionspostnet_dropoutirrrr^s<        zPostNet.__init__cCs||}|||j}||}||}tt|jD] }|j||}q| ||j}||}| |}| |}| ||j}| |}|S)zComputes the forward pass Arguments --------- x: torch.Tensor a (batch, time_steps, features) input tensor Returns ------- output: torch.Tensor the spectrogram predicted )r2r:rdtyper9r>r5lenr4r;r?r7r<r@)rrrFrrrr s       zPostNet.forward)r(rr)r)r*r!rrrrr'Ms*r'cs,eZdZdZ dfdd ZddZZS) DurationPredictoraDuration predictor layer Arguments --------- in_channels: int input feature dimension for convolution layers out_channels: int output feature dimension for convolution layers kernel_size: int duration predictor convolution kernel size dropout: float dropout probability, 0 by default n_units: int Example ------- >>> from speechbrain.lobes.models.FastSpeech2 import FastSpeech2 >>> duration_predictor_layer = DurationPredictor(in_channels=384, out_channels=384, kernel_size=3) >>> x = torch.randn(3, 400, 384) >>> mask = torch.ones(3, 400, 384) >>> y = duration_predictor_layer(x, mask) >>> y.shape torch.Size([3, 400, 1]) r0cs|ttj|||dd|_tj|||dd|_tj||d|_t||_ t||_ t |_ t ||_t ||_dS)Nr+r, n_neurons input_size)rrrr1conv1conv2r Linearr r:r;rReLUrelur=r>r?)rr-rr.dropoutn_unitsrrrrs&     zDurationPredictor.__init__cCsn||||}|||j}||}||||}|||j}||}| ||S)a?Computes the forward pass Arguments --------- x: torch.Tensor a (batch, time_steps, features) input tensor x_mask: torch.Tensor mask of input tensor Returns ------- output: torch.Tensor the duration predictor outputs ) rRrNr:rrGr>rOr;r?r )rrx_maskrrrr s  zDurationPredictor.forward)rJr0r!rrrrrIs rIcs0eZdZdZfddZddZddZZS) SPNPredictora< This module for the silent phoneme predictor. It receives phoneme sequences without any silent phoneme token as input and predicts whether a silent phoneme should be inserted after a position. This is to avoid the issue of fast pace at inference time due to having no silent phoneme tokens in the input sequence. Arguments --------- enc_num_layers: int number of transformer layers (TransformerEncoderLayer) in encoder enc_num_head: int number of multi-head-attention (MHA) heads in encoder transformer layers enc_d_model: int the number of expected features in the encoder enc_ffn_dim: int the dimension of the feedforward network model enc_k_dim: int the dimension of the key enc_v_dim: int the dimension of the value enc_dropout: float Dropout for the encoder normalize_before: bool whether normalization should be applied before or after MHA or FFN in Transformer layers. ffn_type: str whether to use convolutional layers instead of feed forward network inside transformer layer ffn_cnn_kernel_size_list: list of int conv kernel size of 2 1d-convs if ffn_type is 1dcnn n_char: int the number of symbols for the token embedding padding_idx: int the index for padding c sft||_| |_t| | |d|_t||_t|||||||t j || | d |_ t j d|d|_dS)Nr num_layersnheadd_ffnd_modelkdimvdimrS activationnormalize_beforeffn_typeffn_cnn_kernel_size_listr0rK)rr enc_num_head padding_idxr encPreNetr#sinusoidal_positional_embed_encoderrrrQ spn_encoderr rP spn_linear) renc_num_layersrc enc_d_model enc_ffn_dim enc_k_dim enc_v_dim enc_dropoutr`rarbn_charrdrrrrs. zSPNPredictor.__init__c Cs||}t|ddd|jd}||}t||jd}|d}||}t|||}tj tj |jd|jd|j ddd |j |jddd}|j|||d\}} ||d} | S) aforward pass for the module Arguments --------- tokens: torch.Tensor input tokens without silent phonemes last_phonemes: torch.Tensor indicates if a phoneme at an index is the last phoneme of a word or not Returns ------- spn_decision: torch.Tensor indicates if a silent phoneme should be inserted after a phoneme r0pad_idxr)diagonalrsrc_masksrc_key_padding_mask)retorch unsqueezerepeatshaperrdrfaddtriuonesrboolrcrgrhsqueeze) rtokens last_phonemes token_featssrcmasksrcmask_invertedposspn_maskspn_token_feats_ spn_decisionrrrr ?s2       zSPNPredictor.forwardcCs|||}t|dk}|S)ainference function Arguments --------- tokens: torch.Tensor input tokens without silent phonemes last_phonemes: torch.Tensor indicates if a phoneme at an index is the last phoneme of a word or not Returns ------- spn_decision: torch.Tensor indicates if a silent phoneme should be inserted after a phoneme g?)r rysigmoid)rrrrrrrinferns zSPNPredictor.infer)r"r#r$r%rr rr&rrrrrVs  !+/rVcs6eZdZdZfddZ      dddZZS) FastSpeech2aThe FastSpeech2 text-to-speech model. This class is the main entry point for the model, which is responsible for instantiating all submodules, which, in turn, manage the individual neural network layers Simplified STRUCTURE: input->token embedding ->encoder ->duration/pitch/energy predictor ->duration upsampler -> decoder -> output During training, teacher forcing is used (ground truth durations are used for upsampling) Arguments --------- enc_num_layers: int number of transformer layers (TransformerEncoderLayer) in encoder enc_num_head: int number of multi-head-attention (MHA) heads in encoder transformer layers enc_d_model: int the number of expected features in the encoder enc_ffn_dim: int the dimension of the feedforward network model enc_k_dim: int the dimension of the key enc_v_dim: int the dimension of the value enc_dropout: float Dropout for the encoder dec_num_layers: int number of transformer layers (TransformerEncoderLayer) in decoder dec_num_head: int number of multi-head-attention (MHA) heads in decoder transformer layers dec_d_model: int the number of expected features in the decoder dec_ffn_dim: int the dimension of the feedforward network model dec_k_dim: int the dimension of the key dec_v_dim: int the dimension of the value dec_dropout: float dropout for the decoder normalize_before: bool whether normalization should be applied before or after MHA or FFN in Transformer layers. ffn_type: str whether to use convolutional layers instead of feed forward network inside transformer layer. ffn_cnn_kernel_size_list: list of int conv kernel size of 2 1d-convs if ffn_type is 1dcnn n_char: int the number of symbols for the token embedding n_mels: int number of bins in mel spectrogram postnet_embedding_dim: int output feature dimension for convolution layers postnet_kernel_size: int postnet convolution kernel size postnet_n_convolutions: int number of convolution layers postnet_dropout: float dropout probability for postnet padding_idx: int the index for padding dur_pred_kernel_size: int the convolution kernel size in duration predictor pitch_pred_kernel_size: int kernel size for pitch prediction. energy_pred_kernel_size: int kernel size for energy prediction. variance_predictor_dropout: float dropout probability for variance predictor (duration/pitch/energy) Example ------- >>> import torch >>> from speechbrain.lobes.models.FastSpeech2 import FastSpeech2 >>> model = FastSpeech2( ... enc_num_layers=6, ... enc_num_head=2, ... enc_d_model=384, ... enc_ffn_dim=1536, ... enc_k_dim=384, ... enc_v_dim=384, ... enc_dropout=0.1, ... dec_num_layers=6, ... dec_num_head=2, ... dec_d_model=384, ... dec_ffn_dim=1536, ... dec_k_dim=384, ... dec_v_dim=384, ... dec_dropout=0.1, ... normalize_before=False, ... ffn_type='1dcnn', ... ffn_cnn_kernel_size_list=[9, 1], ... n_char=40, ... n_mels=80, ... postnet_embedding_dim=512, ... postnet_kernel_size=5, ... postnet_n_convolutions=5, ... postnet_dropout=0.5, ... padding_idx=0, ... dur_pred_kernel_size=3, ... pitch_pred_kernel_size=3, ... energy_pred_kernel_size=3, ... variance_predictor_dropout=0.5) >>> inputs = torch.tensor([ ... [13, 12, 31, 14, 19], ... [31, 16, 30, 31, 0], ... ]) >>> input_lengths = torch.tensor([5, 4]) >>> durations = torch.tensor([ ... [2, 4, 1, 5, 3], ... [1, 2, 4, 3, 0], ... ]) >>> mel_post, postnet_output, predict_durations, predict_pitch, avg_pitch, predict_energy, avg_energy, mel_lens = model(inputs, durations=durations) >>> mel_post.shape, predict_durations.shape (torch.Size([2, 15, 80]), torch.Size([2, 5])) >>> predict_pitch.shape, predict_energy.shape (torch.Size([2, 5, 1]), torch.Size([2, 5, 1])) c st||_| |_||_t||_t| |_t|||d|_ t ||||d|_ t ||||d|_ t ||||d|_ tjd||ddd|_tjd||ddd|_t|||||||tj|||d |_t|| | | | | |tj|||d |_tj|| d|_t|||||d |_dS) NrWr-rr.rSr0r+Tr-rr.r/skip_transposerXrKrArBrCrDrE)rrrc dec_num_headrdrrf#sinusoidal_positional_embed_decoderrrerIdurPred pitchPred energyPredrr1 pitchEmbed energyEmbedrrrQencoderdecoderr rPr'postnet)rrircrjrkrlrmrndec_num_layersr dec_d_model dec_ffn_dim dec_k_dim dec_v_dim dec_dropoutr`rarbron_melsrBrCrDrErddur_pred_kernel_sizepitch_pred_kernel_sizeenergy_pred_kernel_sizevariance_predictor_dropoutrrrrs ! zFastSpeech2.__init__N?c Cst||jd}|d} ||} || } t| | | } |d|jd| j d ddd } |j | | |d\} } | | } | | | d}|dkrV|d}|durdttj|d}d}|| | }||}|durt|d|}||}| ddd}n || ddd}| ddd}| |} d}|| | }||}|durt|d|}||}| ddd}n || ddd}| ddd}| |} t| |dur|n||d\}}tt|}||j}|d} |d|jd|j d ddd } ||} t|| | }|j|| |d^}}} | || }|!||}|||||||t|fS) aforward pass for training and inference Arguments --------- tokens: torch.Tensor batch of input tokens durations: torch.Tensor batch of durations for each token. If it is None, the model will infer on predicted durations pitch: torch.Tensor batch of pitch for each frame. If it is None, the model will infer on predicted pitches energy: torch.Tensor batch of energy for each frame. If it is None, the model will infer on predicted energies pace: float scaling factor for durations pitch_rate: float scaling factor for pitches energy_rate: float scaling factor for energies Returns ------- mel_post: torch.Tensor mel outputs from the decoder postnet_output: torch.Tensor mel outputs from the postnet predict_durations: torch.Tensor predicted durations of each token predict_pitch: torch.Tensor predicted pitches of each token avg_pitch: torch.Tensor target pitches for each token if input pitch is not None None if input pitch is None predict_energy: torch.Tensor predicted energies of each token avg_energy: torch.Tensor target energies for each token if input energy is not None None if input energy is None mel_length: predicted lengths of mel spectrograms rqrsr0rrprvNpace)"rrdrzrerfryr}r{rcr|permuterrrrdimclampspecialexpm1raverage_over_durationsrrrupsamplertensorrrrrrr r)rr durationspitchenergyr pitch_rate energy_raterrrr attn_maskrpredict_durationsdur_pred_reverse_log avg_pitch predict_pitch avg_energypredict_energy spec_featsmel_lensoutput_mel_featsmemorymel_postpostnet_outputrrrr ls2                     zFastSpeech2.forwardNNNrrrr!rrrrrs txrcCstj|dd}tjj|ddddfd}tjjtj|dkddd}tjjtj|ddd}|\}}|d}|dddddf|||} |dddddf|||} t|d| t|d|  } t|d| t|d|  } t | dk| | | } | S)zAverage values over durations. Arguments --------- values: torch.Tensor shape: [B, 1, T_de] durs: torch.Tensor shape: [B, T_en] Returns ------- avg: torch.Tensor shape: [B, 1, T_en] r0rNrsr0rrJrp) rycumsumlongr functionalpadsizeexpandgatherfloatwhere)valuesdursdurs_cums_endsdurs_cums_startsvalues_nonzero_cums values_cumsbslength n_formantsdcsdce values_sums values_nelemsavgrrrrs,       rrrJcsJfddttD}dd|D}tjjjj|d|d}||fS)aupsample encoder output according to durations Arguments --------- feats: torch.Tensor batch of input tokens durs: torch.Tensor durations to be used to upsample pace: float scaling factor for durations padding_value: int padding index Returns ------- mel_post: torch.Tensor mel outputs from the decoder predict_durations: torch.Tensor predicted durations for each token cs,g|]}tj||ddqS)rr)ryrepeat_interleaver).0rFrfeatsrrr =szupsample..cSsg|]}|jdqSr)r|)rmelrrrrBsT) batch_first padding_value)r5rHryrutilsrnn pad_sequence)rrrrupsampled_melsrpadded_upsampled_melsrrrr(s  rc@eZdZdZddZdS)TextMelCollatezEZero-pads model inputs and targets based on number of frames per stepc! CsZt|}tt|D] }||d||<q tjtdd|Dddd\}}|d}tjtdd|Dddd\}}|d} tt||} tt|| } tt|| } tt||} tt|| }| | | | |tt|D]g}|||d}|||d}t|||d }|||d }t|||d }|| |d |df<|| |d |df<|| |d |df<|| |d |df<|||d |df<q|dd d}t dd|D}tt|||}|tt||}|tt||}|tt|}gg}}tt|D]V}||}||d }||d}||d}|||d d d |d f<|||d |df<|||d |df<|d ||<| ||d| ||dq4dd|D} t | } | dd d }| | |||||| ||| || f S)aCollate's training batch from normalized text and mel-spectrogram Arguments --------- batch: list [text_normalized, mel_normalized] Returns ------- text_padded: torch.Tensor dur_padded: torch.Tensor input_lengths: torch.Tensor mel_padded: torch.Tensor pitch_padded: torch.Tensor energy_padded: torch.Tensor output_lengths: torch.Tensor len_x: torch.Tensor labels: torch.Tensor wavs: torch.Tensor no_spn_seq_padded: torch.Tensor spn_labels_padded: torch.Tensor last_phonemes_padded: torch.Tensor mel_text_paircSg|]}t|dqSrrHrrrrrroz+TextMelCollate.__call__..rTr descendingcSr)rrrrrrurrr0rsNrpcSsg|] }|ddqS)rpr0rrrrrrlabelwavcSsg|]}|dqS)r)rrrrrrs) listr5rHrysort LongTensor FloatTensorzero_rmaxr6Tensorr)!rbatch raw_batchrF input_lengthsids_sorted_decreasing max_input_lenno_spn_seq_lengthsno_spn_ids_sorted_decreasingmax_no_spn_seq_len text_paddedno_spn_seq_paddedlast_phonemes_padded dur_paddedspn_labels_paddedtext no_spn_seqrdur spn_labelsnum_melsmax_target_len mel_padded pitch_padded energy_paddedoutput_lengthslabelswavsidxrrrlen_xrrr__call__Ns       zTextMelCollate.__call__Nr"r#r$r%r rrrrrJs rcs.eZdZdZ  dfdd ZddZZS) LossasLoss Computation Arguments --------- log_scale_durations: bool applies logarithm to target durations ssim_loss_weight: float weight for ssim loss duration_loss_weight: float weight for the duration loss pitch_loss_weight: float weight for the pitch loss energy_loss_weight: float weight for the energy loss mel_loss_weight: float weight for the mel loss postnet_mel_loss_weight: float weight for the postnet mel loss spn_loss_weight: float weight for spn loss spn_loss_max_epochs: int Max number of epochs rc s~tt|_t|_t|_t|_t|_ t|_ ||_ ||_ ||_ ||_||_||_||_||_| |_dSN)rrSSIMLoss ssim_lossrMSELossmel_losspostnet_mel_lossdur_loss pitch_loss energy_losslog_scale_durationsssim_loss_weightmel_loss_weightpostnet_mel_loss_weightduration_loss_weightpitch_loss_weightenergy_loss_weightspn_loss_weightspn_loss_max_epochs) rr-r.r1r2r3r/r0r4r5rrrrs       z Loss.__init__c Cs|\}}}}}} } t|jdksJ|\ } } } }}}}}}|d}|d}|d}|d}| d} |jr@t|}t|jdD]}|dkr|| |d||ddf||d||ddf}| | |d||ddf||d||ddf}| | |d| |f||d| |f tj }| ||d||f||d||f tj }|||d||f||d||f tj }qG||| |d||ddf||d||ddf}|| | |d||ddf||d||ddf}|| | |d| |f||d| |f tj }|| ||d||f||d||f tj }||||d||f||d||f tj }qG|| ||}t|t|}t|t|}t|t|}t|t|}t|t|}t|| }||jkrd|_||j||j||j||j||j||j||j}|||j||j||j||j||j||j||jd}|S)asComputes the value of the loss function and updates stats Arguments --------- predictions: tuple model predictions targets: tuple ground truth data current_epoch: int The count of the current epoch. Returns ------- loss: torch.Tensor the loss value rrsrN) total_lossr&r(r)r*r+r,spn_loss)rHr|rr-rylog1prr5r(r)r*rfloat32r+r,r&divr r5r4r.r/r0r1r2r3)r predictionstargets current_epoch mel_targettarget_durations target_pitch target_energy mel_lengthphon_lenrmel_outpostnet_mel_out log_durationspredicted_pitch average_pitchpredicted_energyaverage_energyr spn_predslog_target_durationsrFr(r)r*r+r,r&r7r6lossrrrr s          z Loss.forward)rr#r!rrrrr"s !r"c Csddlm}|j|||||d| j}|j||dd|||| | d| j} || }| |}|dks8J|jd|ksAJtj |dd}| r[|t |t |t |}| rat |}||fS)acalculates MelSpectrogram for a raw audio signal Arguments --------- sample_rate : int Sample rate of audio signal. hop_length : int Length of hop between STFT windows. win_length : int Window size. n_fft : int Size of FFT. n_mels : int Number of mel filterbanks. f_min : float Minimum frequency. f_max : float Maximum frequency. power : float Exponent for the magnitude spectrogram. normalized : bool Whether to normalize by magnitude after stft. min_max_energy_norm : bool Whether to normalize by min-max norm : str or None If "slaney", divide the triangular mel weights by the width of the mel band mel_scale : str Scale to use: "htk" or "slaney". compression : bool whether to do dynamic range compression audio : torch.Tensor input audio signal Returns ------- mel : torch.Tensor rmse : torch.Tensor r) transforms) hop_length win_lengthn_fftpower normalizedrpr0) sample_raten_stftrf_minf_maxnorm mel_scaler) torchaudiorN SpectrogramrrMelScalerr|ryrXminrdynamic_range_compression)rTrOrPrQrrVrWrRrSmin_max_energy_normrXrY compressionaudiorN audio_to_melspecrrmserrrmel_spectogramys> 6  "rer0h㈵>cCsttj||d|S)z+Dynamic range compression for audio signalsr])rylogr)rCclip_valrrrr^sr^csHeZdZdZfddZd ddZdejdejfd d Zd d Z Z S)r%zkSSIM loss as (1 - SSIM) SSIM is explained here https://en.wikipedia.org/wiki/Structural_similarity cstt|_dSr$)rr _SSIMLoss loss_funcrrrrrs  zSSIMLoss.__init__NcCs>|dur |j}tj||j|jd}|d|dk}|S)a:Create a sequence mask for filtering padding in a sequence tensor. Arguments --------- sequence_length: torch.Tensor Sequence lengths. max_len: int Maximum sequence length. Defaults to None. Returns ------- mask: [B, T_max] N)rGrrr0)datarryarangerGrrz)rsequence_lengthmax_len seq_rangemaskrrr sequence_masks  zSSIMLoss.sequence_maskrrscCsHtj||dddd}tj||dddd}||||dS)aMin-Max normalize tensor through first dimension Arguments --------- x: torch.Tensor input tensor [B, D1, D2] mask: torch.Tensor input mask [B, D1, 1] Returns ------- Normalized tensor r)r0rpT)rkeepdimgꌠ9Y>)F:0yE>)ryamax masked_fillamin)rrrsmaximumminimumrrrsample_wise_min_maxs zSSIMLoss.sample_wise_min_maxcCs|j||ddd}|||}|||}|||d||d}|dkrAtd|dtjd|j d}|dkrYtd|d tjd|j d}|S) at Arguments --------- y_hat: torch.Tensor model prediction values [B, T, D]. y: torch.Tensor target values [B, T, D]. length: torch.Tensor length of each sample in a batch for masking. Returns ------- loss: Average loss value in range [0, 1] masked by the length. r0)rprqrprz > SSIM loss is out-of-range z, setting it 1.0rtrJz, setting it 0.0) rtrrzr|rlitemprintryrr)ry_hatyrrsy_norm y_hat_normr&rrrr s*     zSSIMLoss.forwardr$) r"r#r$r%rrtryrr|r r&rrrrr%s   r%cseZdZdZgdZ       dfd d Zdd d Z   d ddZddZd!ddZ d!ddZ       d"ddZ ddZ Z S)#rkaECreates a criterion that measures the structural similarity index error between each element in the input x and target y. Equation link: https://en.wikipedia.org/wiki/Structural_similarity x and y are tensors of arbitrary shapes with a total of n elements each. The sum operation still operates over all the elements, and divides by n. The division by n can be avoided if one sets reduction = sum. In case of 5D input tensors, complex value is returned as a tensor of size 2. Arguments --------- kernel_size: int By default, the mean and covariance of a pixel is obtained by convolution with given filter_size. kernel_sigma: float Standard deviation for Gaussian kernel. k1: float Coefficient related to c1 (see equation in the link above). k2: float Coefficient related to c2 (see equation in the link above). downsample: bool Perform average pool before SSIM computation (Default: True). reduction: str Specifies the reduction type data_range: Union[int, float] Maximum value range of images (usually 1.0 or 255). Example ------- >>> loss = _SSIMLoss() >>> x = torch.rand(3, 3, 256, 256, requires_grad=True) >>> y = torch.rand(3, 3, 256, 256) >>> output = loss(x, y) >>> output.backward() )r.k1k2sigmakernel reduction ?{Gz?Q?TmeanrcsTt||_||_|ddksJd|d||_||_||_||_||_dS)Nrpr0Kernel size must be odd, got []) rrrr. kernel_sigmarr downsample data_range)rr.rrrrrrrrrrYs   z_SSIMLoss.__init__cCs<|dkr|S|dkr|jddS|dkr|jddStd)a?Reduce input in batch dimension if needed. Arguments --------- x: torch.Tensor Tensor with shape (B, *). reduction: str Specifies the reduction type: none | mean | sum (Default: mean) Returns ------- Reduced outputs. nonerrrsumz:Unknown reduction. Expected one of {'none', 'mean', 'sum'})rr ValueError)rrrrrr_reducevs  z_SSIMLoss._reducerrsrJgNc Cs |d}|D]}t|sJdt||j|jks)Jd|jd|j|durD||ksCJd|d|n&||d|d ||d|d ksjJd |d||d|d kr||dksJd |dd|n,|d|d kr|d|kr|d ksnJd |dd|d d||d|d kr|d|ksJd |dd|||d ksJd|d d|qdS)aCheck if the input satisfies the requirements Arguments --------- tensors: torch.Tensor torch.Tensors to check dim_range: Tuple[int, int] Allowed number of dimensions. (min, max) data_range: Tuple[float, float] Allowed range of values in tensors. (min, max) size_range: Tuple[int, int] Dimensions to include in size comparison. (start_dim, end_dim + 1) Returns ------- None FNrzExpected torch.Tensor, got zExpected tensors to be on z, got z%Expected tensors with same size, got z and r0z9Expected tensors with same size at given dimensions, got z$Expected number of dimensions to be z,Expected number of dimensions to be between z*Expected values to be greater or equal to z(Expected values to be lower or equal to )ry is_tensortyperrrr]r)rtensors dim_ranger size_rangertrrr_validate_inputsB$ z_SSIMLoss._validate_inputcCsdtj|tjd}||dd8}|d}|d|d d|d}||}|dS)a7Returns 2D Gaussian kernel N(0,sigma^2) Arguments --------- kernel_size: int Size of the kernel sigma: float Std of the distribution Returns ------- gaussian_kernel: torch.Tensor [1, kernel_size, kernel_size] rGr0@rpr)ryror9rzexpr)rr.rcoordsgrrrgaussian_filters &  z_SSIMLoss.gaussian_filtercCsF|d|dks|d|dkr"td|d||d}|d}|d}tj||dd|d} tj||dd|d} | d} | d} | | } tj|d|dd|d| }tj|d|dd|d| }tj|||dd|d| }d |||||}d | || | ||}|jd d }|jd d }||fS) aCalculate Structural Similarity (SSIM) index for X and Y per channel. Arguments --------- x: torch.Tensor An input tensor (N, C, H, W). y: torch.Tensor A target tensor (N, C, H, W). kernel: torch.Tensor 2D Gaussian kernel. k1: float Algorithm parameter (see equation in the link above). k2: float Algorithm parameter (see equation in the link above). Try a larger K2 constant (e.g. 0.4) if you get a negative or NaN results. Returns ------- Full Value of Structural Similarity (SSIM) index. rsrAKernel size can't be greater than actual input size. Input size: . Kernel size: rpr0rweightstrider/groupsr)rsrr)rrFconv2dr)rrrrrrc1c2 n_channelsmu_xmu_ymu_xxmu_yymu_xysigma_xxsigma_yysigma_xycsssssim_valrrr_ssim_per_channelsR(      z_SSIMLoss._ssim_per_channelc$Csf|d}|d|dks|d|dkr'td|d||d}|d}|d} |d } |d} |d } tj| |dd |d } tj| |dd |d }tj| |dd |d }tj| |dd |d }| d|d}|d|d}| |||}| |||}d }| d| d}| d| d}| | | | }| | | | }tj||dd |d |}tj||dd |d |}tj||dd |d |}tj||dd |d |}tj||fdd }tj||fdd }|d|||d|d||} |d|||d|d||}!|!| }!|!jdd }"| jdd }#|"|#fS)aCalculate Structural Similarity (SSIM) index for Complex X and Y per channel. Arguments --------- x: torch.Tensor An input tensor (N, C, H, W, 2). y: torch.Tensor A target tensor (N, C, H, W, 2). kernel: torch.Tensor 2-D gauss kernel. k1: float Algorithm parameter (see equation in the link above). k2: float Algorithm parameter (see equation in the link above). Try a larger K2 constant (e.g. 0.4) if you get a negative or NaN results. Returns ------- Full Value of Complex Structural Similarity (SSIM) index. r0rrsrrrrp).r).r0rrrr)rr) rrrrpowrystackrzr)$rrrrrrrrrx_realx_imagy_realy_imagmu1_realmu1_imagmu2_realmu2_imagmu1_sqmu2_sq mu1_mu2_real mu1_mu2_imag compensationx_sqy_sqx_y_realx_y_imag sigma1_sq sigma2_sq sigma12_real sigma12_imagsigma12mu1_mu2cs_mapssim_maprrrrr_ssim_per_channel_complex,s (           z#_SSIMLoss._ssim_per_channel_complexFc Cs$|ddksJd|d|j||gdd|fd|t|}|t|}tdtt|dd d } | dkrK|rKtj|| d }tj|| d }||| |dddd |} | d krf|j n|j } | ||| | | d \}}|d}|d}|||}|||}|r||gS|S)a^Interface of Structural Similarity (SSIM) index. Inputs supposed to be in range [0, data_range]. To match performance with skimage and tensorflow set downsample = True. Arguments --------- x: torch.Tensor An input tensor (N, C, H, W) or (N, C, H, W, 2). y: torch.Tensor A target tensor (N, C, H, W) or (N, C, H, W, 2). kernel_size: int The side-length of the sliding window used in comparison. Must be an odd value. kernel_sigma: float Sigma of normal distribution. data_range: Union[int, float] Maximum value range of images (usually 1.0 or 255). reduction: str Specifies the reduction type: none | mean | sum. Default:mean full: bool Return cs map or not. downsample: bool Perform average pool before SSIM computation. Default: True k1: float Algorithm parameter (see equation in the link above). k2: float Algorithm parameter (see equation in the link above). Try a larger K2 constant (e.g. 0.4) if you get a negative or NaN results. Returns ------- Value of Structural Similarity (SSIM) index. In case of 5D input tensors, complex value is returned as a tensor of size 2. rpr0rr)rr)r)rrrN)r.r))rrrrr)rrrroundr]rr avg_pool2drr{rrrrrr)rrrr.rrrfullrrrfr_compute_ssim_per_channelrrrrrrrssims<0   "        z_SSIMLoss.ssimc Cs:|j|||j|j|j|j|jd|j|jd }t ||S)aComputation of Structural Similarity (SSIM) index as a loss function. Arguments --------- x: torch.Tensor An input tensor (N, C, H, W) or (N, C, H, W, 2). y: torch.Tensor A target tensor (N, C, H, W) or (N, C, H, W, 2). Returns ------- Value of SSIM loss to be minimized, i.e 1 - ssim in [0, 1] range. In case of 5D input tensors, complex value is returned as a tensor of size 2. F) rrr.rrrrrrr) rr.rrrrrrry ones_like)rrrscorerrrr s z_SSIMLoss.forward)rrrrTrr)r)rrN)rr)rrrrFTrr)r"r#r$r% __constants__rrrrrrrr r&rrrrrk3s:#  ?  Fk Wrkc@r)TextMelCollateWithAlignmenta`Zero-pads model inputs and targets based on number of frames per step result: tuple a tuple of tensors to be used as inputs/targets ( text_padded, dur_padded, input_lengths, mel_padded, output_lengths, len_x, labels, wavs ) c Cst|}tt|D] }||d||<q tjtdd|Dddd\}}|d}tt||}|tt|D]}|||d}|||d|df<q=|ddd} td d|D} t t|| | } | t t|| } | t t|| } | tt|}gg}}tt|D]U}||}||d}||d }||d }|| |ddd|df<|| |d|df<|| |d|df<|d||<| ||d | ||d q| dd d} ||| | | |||fS)aCollate's training batch from normalized text and mel-spectrogram Arguments --------- batch: list [text_normalized, mel_normalized] Returns ------- phoneme_padded: torch.Tensor input_lengths: torch.Tensor mel_padded: torch.Tensor pitch_padded: torch.Tensor energy_padded: torch.Tensor output_lengths: torch.Tensor labels: torch.Tensor wavs: torch.Tensor rcSrrrrrrrr6rz8TextMelCollateWithAlignment.__call__..rTrNr0cSsg|] }|ddqS)r0rrrrrrDrrprrr) rr5rHryrrrrrrr6r)rrrrFrrr phoneme_paddedphonemerrrrrrrrrrrrrrrr sZ     z$TextMelCollateWithAlignment.__call__Nr!rrrrr s rc Cstj }||}|j}|j}|}|tj}|j \}}}tj |j tj d}tj ||ftj d} tj |tj ddd} t|D]G} tj| ddgddggd|dddddf} | } | | k}t|| | }||dddd| f<| | k}t|||dddd| f|} qKt||d}tj |j tj d}|dddddfdtj d}t |}tt|D]} d|||| f<||||| fd}q||tj }t|j||d}|S) a< Monotonic alignment search algorithm, numpy works faster than the torch implementation. Arguments --------- value: torch.Tensor input alignment values [b, t_x, t_y] mask: torch.Tensor input alignment mask [b, t_x, t_y] Returns ------- path: torch.Tensor Example ------- >>> import torch >>> from speechbrain.lobes.models.FastSpeech2 import maximum_path_numpy >>> alignment = torch.rand(2, 5, 100) >>> mask = torch.ones(2, 5, 100) >>> hard_alignments = maximum_path_numpy(alignment, mask) rr0rsrconstant)modeconstant_valuesN)rrG)npinfrrGcpudetachnumpyastypebool_r|zerosint64r9roreshaper5rrrreversedry from_numpyr)valuers max_neg_valrrGbt_xt_y directionvx_rangejv0v1max_maskv_max index_maskpathindex index_rangerrrmaximum_path_numpyhs@  &( rcs2eZdZdZ    d fdd ZddZZS) AlignmentNetworkaLearns the alignment between the input text and the spectrogram with Gaussian Attention. query -> conv1d -> relu -> conv1d -> relu -> conv1d -> L2_dist -> softmax -> alignment key -> conv1d -> relu -> conv1d - - - - - - - - - - - -^ Arguments --------- in_query_channels: int Number of channels in the query network. Defaults to 80. in_key_channels: int Number of channels in the key network. Defaults to 512. attn_channels: int Number of inner channels in the attention layers. Defaults to 80. temperature: float Temperature for the softmax. Defaults to 0.0005. Example ------- >>> import torch >>> from speechbrain.lobes.models.FastSpeech2 import AlignmentNetwork >>> aligner = AlignmentNetwork( ... in_query_channels=80, ... in_key_channels=512, ... attn_channels=80, ... temperature=0.0005, ... ) >>> phoneme_feats = torch.rand(2, 512, 20) >>> mels = torch.rand(2, 80, 100) >>> alignment_soft, alignment_logprob = aligner(mels, phoneme_feats, None, None) >>> alignment_soft.shape, alignment_logprob.shape (torch.Size([2, 1, 100, 20]), torch.Size([2, 1, 100, 20])) r(rMb@?cst||_tjjdd|_tjjdd|_t t j ||ddddddtj t j |d|ddddd|_ t t j ||ddddddtj t j |d|dddddtj t j ||ddddd|_dS)Nrrrpr+T)r-rr.r/biasrr0)rr temperatureryrSoftmaxsoftmax LogSoftmax log_softmax Sequentialrr1rQ key_layer query_layer)rin_query_channelsin_key_channels attn_channelsr rrrrsf  zAlignmentNetwork.__init__c Cs||}||}|dddddddf|dddddfd}|j |jddd}|durF||t|dddfd}|durZ|j| dt d | |} | |fS)a9Forward pass of the aligner encoder. Arguments --------- queries: torch.Tensor the query tensor [B, C, T_de] keys: torch.Tensor the query tensor [B, C_emb, T_en] mask: torch.Tensor the query mask[B, T_de] attn_prior: torch.Tensor the prior attention tensor [B, 1, T_en, T_de] Returns ------- attn: torch.Tensor soft attention [B, 1, T_en, T_de] attn_logp: torch.Tensor log probabilities [B, 1, T_en , T_de] Nrpr0T)rurvr) rrr rr ryrhrn masked_fill_rrzrr ) rquerieskeysrs attn_priorkey_out query_out attn_factor attn_logpattnrrrr s  6  zAlignmentNetwork.forward)r(rr(rr!rrrrrs$=rcs>eZdZdZfddZddZ      d dd ZZS) FastSpeech2WithAlignmentaThe FastSpeech2 text-to-speech model with internal alignment. This class is the main entry point for the model, which is responsible for instantiating all submodules, which, in turn, manage the individual neural network layers. Certain parts are adopted from the following implementation: https://github.com/coqui-ai/TTS/blob/dev/TTS/tts/models/forward_tts.py Simplified STRUCTURE: input -> token embedding -> encoder -> aligner -> duration/pitch/energy -> upsampler -> decoder -> output Arguments --------- enc_num_layers: int number of transformer layers (TransformerEncoderLayer) in encoder enc_num_head: int number of multi-head-attention (MHA) heads in encoder transformer layers enc_d_model: int the number of expected features in the encoder enc_ffn_dim: int the dimension of the feedforward network model enc_k_dim: int the dimension of the key enc_v_dim: int the dimension of the value enc_dropout: float Dropout for the encoder in_query_channels: int Number of channels in the query network. in_key_channels: int Number of channels in the key network. attn_channels: int Number of inner channels in the attention layers. temperature: float Temperature for the softmax. dec_num_layers: int number of transformer layers (TransformerEncoderLayer) in decoder dec_num_head: int number of multi-head-attention (MHA) heads in decoder transformer layers dec_d_model: int the number of expected features in the decoder dec_ffn_dim: int the dimension of the feedforward network model dec_k_dim: int the dimension of the key dec_v_dim: int the dimension of the value dec_dropout: float dropout for the decoder normalize_before: bool whether normalization should be applied before or after MHA or FFN in Transformer layers. ffn_type: str whether to use convolutional layers instead of feed forward network inside transformer layer. ffn_cnn_kernel_size_list: list of int conv kernel size of 2 1d-convs if ffn_type is 1dcnn n_char: int the number of symbols for the token embedding n_mels: int number of bins in mel spectrogram postnet_embedding_dim: int output feature dimension for convolution layers postnet_kernel_size: int postnet convolution kernel size postnet_n_convolutions: int number of convolution layers postnet_dropout: float dropout probability for postnet padding_idx: int the index for padding dur_pred_kernel_size: int the convolution kernel size in duration predictor pitch_pred_kernel_size: int kernel size for pitch prediction. energy_pred_kernel_size: int kernel size for energy prediction. variance_predictor_dropout: float dropout probability for variance predictor (duration/pitch/energy) Example ------- >>> import torch >>> from speechbrain.lobes.models.FastSpeech2 import FastSpeech2WithAlignment >>> model = FastSpeech2WithAlignment( ... enc_num_layers=6, ... enc_num_head=2, ... enc_d_model=384, ... enc_ffn_dim=1536, ... enc_k_dim=384, ... enc_v_dim=384, ... enc_dropout=0.1, ... in_query_channels=80, ... in_key_channels=384, ... attn_channels=80, ... temperature=0.0005, ... dec_num_layers=6, ... dec_num_head=2, ... dec_d_model=384, ... dec_ffn_dim=1536, ... dec_k_dim=384, ... dec_v_dim=384, ... dec_dropout=0.1, ... normalize_before=False, ... ffn_type='1dcnn', ... ffn_cnn_kernel_size_list=[9, 1], ... n_char=40, ... n_mels=80, ... postnet_embedding_dim=512, ... postnet_kernel_size=5, ... postnet_n_convolutions=5, ... postnet_dropout=0.5, ... padding_idx=0, ... dur_pred_kernel_size=3, ... pitch_pred_kernel_size=3, ... energy_pred_kernel_size=3, ... variance_predictor_dropout=0.5) >>> inputs = torch.tensor([ ... [13, 12, 31, 14, 19], ... [31, 16, 30, 31, 0], ... ]) >>> mels = torch.rand(2, 100, 80) >>> mel_post, postnet_output, durations, predict_pitch, avg_pitch, predict_energy, avg_energy, mel_lens, alignment_durations, alignment_soft, alignment_logprob, alignment_mas = model(inputs, mels) >>> mel_post.shape, durations.shape (torch.Size([2, 100, 80]), torch.Size([2, 5])) >>> predict_pitch.shape, predict_energy.shape (torch.Size([2, 5, 1]), torch.Size([2, 5, 1])) >>> alignment_soft.shape, alignment_mas.shape (torch.Size([2, 100, 5]), torch.Size([2, 100, 5])) c!! s t||_| |_||_t||_t||_t|||d|_ t |||| d|_ t |||| d|_ t |||| d|_ tjd||ddd|_tjd||ddd|_t|||||||tj|||d |_t| | |||||tj|||d |_tj||d|_t|||||d |_t|| | | d |_dS) NrWrr0r+TrrXrKr)rrrr )rrrcrrdrrfrrrerIrrrrr1rrrrrQrrr rPr'rraligner)!rrircrjrkrlrmrnrrrr rrrrrrrr`rarbrorrBrCrDrErdrrrrrrrrs & z!FastSpeech2WithAlignment.__init__c Cst|dt|d}||dd|dd|d\}}t|ddd|d}t|d} |ddd}| |||fS)aAligner forward pass. 1. Compute a mask to apply to the attention map. 2. Run the alignment network. 3. Apply MAS (Monotonic alignment search) to compute the hard alignment map. 4. Compute the durations from the hard alignment map. Arguments --------- x: torch.Tensor Input sequence [B, T_en, C_en]. y: torch.Tensor Output sequence [B, T_de, C_de]. x_mask: torch.Tensor Input sequence mask [B, 1, T_en]. y_mask: torch.Tensor Output sequence mask [B, 1, T_de]. Returns ------- durations: torch.Tensor Durations from the hard alignment map [B, T_en]. alignment_soft: torch.Tensor soft alignment potentials [B, T_en, T_de]. alignment_logprob: torch.Tensor log scale alignment potentials [B, 1, T_de, T_en]. alignment_mas: torch.Tensor hard alignment map [B, T_en, T_de]. rsrpr0N) ryrzr transposerr contiguousrint) rrrrUy_maskralignment_softalignment_logprob alignment_masrrrr_forward_aligner( s  z)FastSpeech2WithAlignment._forward_alignerNrc Cst||jd}|d} ||} || } t| | | } |d|jd| j d ddd } |j | | |d\} } | | } d}d}d}d}|dur{t||jd}|d}| | || dd|dd\}}}}|dd}|dd}|| | }|dkr|d}ttj|d}d}|| | }||}|durt|d|}||}| ddd}n || ddd}| ddd}| |} d}|| | }||}|durt|d|}||}| ddd}n || ddd}| ddd}| |} t| |dur|n||d\}}tt|}||j}|d} |d|jd|j d ddd } | |} t|| | }|j!|| |d^}}} |"|| }|#||}|||||||t|||||f S) aforward pass for training and inference Arguments --------- tokens: torch.Tensor batch of input tokens mel_spectograms: torch.Tensor batch of mel_spectograms (used only for training) pitch: torch.Tensor batch of pitch for each frame. If it is None, the model will infer on predicted pitches energy: torch.Tensor batch of energy for each frame. If it is None, the model will infer on predicted energies pace: float scaling factor for durations pitch_rate: float scaling factor for pitches energy_rate: float scaling factor for energies Returns ------- mel_post: torch.Tensor mel outputs from the decoder postnet_output: torch.Tensor mel outputs from the postnet predict_durations: torch.Tensor predicted durations of each token predict_pitch: torch.Tensor predicted pitches of each token avg_pitch: torch.Tensor target pitches for each token if input pitch is not None None if input pitch is None predict_energy: torch.Tensor predicted energies of each token avg_energy: torch.Tensor target energies for each token if input energy is not None None if input energy is None mel_length: predicted lengths of mel spectrograms alignment_durations: durations from the hard alignment map alignment_soft: torch.Tensor soft alignment potentials alignment_logprob: torch.Tensor log scale alignment potentials alignment_mas: torch.Tensor hard alignment map rqrsr0rrprvNr)$rrdrzrerfryr}r{rcr|rrrr&rrrrrrrrrrrrrrrrrrrrr r) rrmel_spectogramsrrrrrrrrrrralignment_durationsr#r$r%r"y_mask_invertedrpredict_durations_reverse_logrrrrrrrrrrrrrr Q s:                            z FastSpeech2WithAlignment.forwardr)r"r#r$r%rr&r r&rrrrr(s ,rc(eZdZdZfddZddZZS)LossWithAlignmentaLoss computation including internal aligner Arguments --------- log_scale_durations: bool applies logarithm to target durations ssim_loss_weight: float weight for the ssim loss duration_loss_weight: float weight for the duration loss pitch_loss_weight: float weight for the pitch loss energy_loss_weight: float weight for the energy loss mel_loss_weight: float weight for the mel loss postnet_mel_loss_weight: float weight for the postnet mel loss aligner_loss_weight: float weight for the alignment loss binary_alignment_loss_weight: float weight for the postnet mel loss binary_alignment_loss_warmup_epochs: int Number of epochs to gradually increase the impact of binary loss. binary_alignment_loss_max_epochs: int From this epoch on the impact of binary loss is ignored. c stt|_t|_t|_t|_t|_ t|_ t |_ t |_||_||_||_||_||_||_||_||_| |_| |_| |_dSr$)rrr%r&rr'r(r)r*r+r,ForwardSumLoss aligner_lossBinaryAlignmentLossbinary_alignment_lossr-r.r/r0r1r2r3aligner_loss_weightbinary_alignment_loss_weight#binary_alignment_loss_warmup_epochs binary_alignment_loss_max_epochs) rr-r.r1r2r3r/r0r1r2r3r4rrrr. s*       zLossWithAlignment.__init__c" Cs|\}}}}}t|jdksJ|\ } } } } } }}}}}}}| d} |d}| d}|d}| d} |jrAt|}t|jdD]}|dkr|| |d||ddf||d||ddf}| | |d||ddf||d||ddf}| | |d||f||d||f tj }| | |d||f||d||f tj }|||d||f||d||f tj }qH||| |d||ddf||d||ddf}|| | |d||ddf||d||ddf}|| | |d||f||d||f tj }|| | |d||f||d||f tj }||||d||f||d||f tj }qHd}i}|| ||}||j|d<t|t|}||j|d<t|t|}||j|d<t|t|}||j|d<t|t|}||j|d <t|t|}||j|d <|dur||||}||j|d <|dur|dur||jkrd} n t||jd d } |||}!|!|j| |d <t|}||d<|S)aComputes the value of the loss function and updates stats Arguments --------- predictions: tuple model predictions targets: tuple ground truth data current_epoch: int used to determinate the start/end of the binary alignment loss Returns ------- loss: torch.Tensor the loss value rrsrNr&r(r)r*r+r,r.rr0r6) rHr|rr-ryr8rr5r(r)r*rr9r+r,r&r.r:r/r0r1r2r3r.r1r4r]r3r0r2rr)"rr;r<r=r>r@rArBrCrDrErFrGrHrIrJrr(r#r$alignment_hardrLrFr(r)r*r+r,r6rMr&r.binary_loss_warmup_weightr0rrrr T s         zLossWithAlignment.forwardr!rrrrr, s &r,cr )r-aCTC alignment loss Arguments --------- blank_logprob: pad value Example ------- >>> import torch >>> from speechbrain.lobes.models.FastSpeech2 import ForwardSumLoss >>> loss_func = ForwardSumLoss() >>> attn_logprob = torch.rand(2, 1, 100, 5) >>> key_lens = torch.tensor([5, 5]) >>> query_lens = torch.tensor([100, 100]) >>> loss = loss_func(attn_logprob, key_lens, query_lens) rscs4ttjjdd|_tjjdd|_||_dS)NrrT) zero_infinity) rrryrr r CTCLossctc_loss blank_logprob)rr:rrrr s  zForwardSumLoss.__init__c Cstjjj|d|jd}d}t|jdD]K}td||dd}|| dddd||ddd||df}| |dd}|j |||||d|||dd} || }q||jd}|S) aN Arguments --------- attn_logprob: torch.Tensor log scale alignment potentials [B, 1, query_lens, key_lens] key_lens: torch.Tensor mel lengths query_lens: torch.Tensor phoneme lengths Returns ------- total_loss: torch.Tensor r)inputrrrJrr0rpN)rtarget_lengths) ryrrrr:r5r|rorzrr r9) r attn_logprobkey_lens query_lensattn_logprob_paddedr6bid target_seq curr_logprobrMrrrr  s&  zForwardSumLoss.forward)rsr!rrrrr- sr-cr+)r/aBinary loss that forces soft alignments to match the hard alignments as explained in `https://arxiv.org/pdf/2108.10447.pdf`. Example ------- >>> import torch >>> from speechbrain.lobes.models.FastSpeech2 import BinaryAlignmentLoss >>> loss_func = BinaryAlignmentLoss() >>> alignment_hard = torch.randint(0, 2, (2, 100, 5)) >>> alignment_soft = torch.rand(2, 100, 5) >>> loss = loss_func(alignment_hard, alignment_soft) cstdSr$)rrrmrrrr7 szBinaryAlignmentLoss.__init__cCs.ttj||dkdd}| |S)z alignment_hard: torch.Tensor hard alignment map [B, mel_lens, phoneme_lens] alignment_soft: torch.Tensor soft alignment potentials [B, mel_lens, phoneme_lens] r0g-q=rg)ryrhrr)rr5r#log_sumrrrr : s zBinaryAlignmentLoss.forwardr!rrrrr/* s r/)rrJ)r0rf)-r%rrrytorch.nn.functionalrrrtorch.nn.modules.lossr0speechbrain.lobes.models.transformer.Transformerrrrrspeechbrain.nnetrr speechbrain.nnet.embeddingr speechbrain.nnet.lossesr speechbrain.nnet.normalizationr Modulerr'rIrVrrrrr"rer^r%rkrrrrr,r-r/rrrrsV      1ZK )"y7 X]Y_;l[?