o ½e¦i•tã@sÂdZddlZddlZddlmmZddlmZddlm Z ddl m Z m Z m Z mZddlmZmZddlmZddlmZdd lmZGd d „d ejƒZGd d „d ejƒZGdd„dejƒZGdd„dejƒZGdd„dejƒZdd„Zd6dd„ZGdd„dƒZ Gdd„dejƒZ!dd„Z"d7d"d#„Z#Gd$d%„d%ejjƒZ$Gd&d'„d'e ƒZ%Gd(d)„d)ƒZ&d*d+„Z'Gd,d-„d-ejjƒZ(Gd.d/„d/ejƒZ)Gd0d1„d1ejƒZ*Gd2d3„d3ejƒZ+Gd4d5„d5ejƒZ,dS)8z¸ Neural network modules for the FastSpeech 2: Fast and High-Quality End-to-End Text to Speech synthesis model Authors * Sathvik Udupa 2022 * Pradnya Kandarkar 2023 * Yingzhi Wang 2023 éN)Únn)Ú_Loss)ÚPositionalEncodingÚTransformerEncoderÚget_key_padding_maskÚget_mask_from_lengths)ÚCNNÚlinear)Ú Embedding)Úbce_loss)Ú LayerNormcó*eZdZdZd‡fdd„ Zdd„Z‡ZS)Ú EncoderPreNeta?Embedding layer for tokens Arguments --------- n_vocab: int size of the dictionary of embeddings blank_id: int padding index out_channels: int the size of each embedding vector Example ------- >>> from speechbrain.nnet.embedding import Embedding >>> from speechbrain.lobes.models.FastSpeech2 import EncoderPreNet >>> encoder_prenet_layer = EncoderPreNet(n_vocab=40, blank_id=0, out_channels=384) >>> x = torch.rand(3, 5) >>> y = encoder_prenet_layer(x) >>> y.shape torch.Size([3, 5, 384]) écstƒ ¡t|||d|_dS)N)Únum_embeddingsÚ embedding_dimÚblank_id)ÚsuperÚ__init__r Útoken_embedding)ÚselfÚn_vocabrÚ out_channels©Ú __class__©úb/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/speechbrain/lobes/models/FastSpeech2.pyr3s  ýzEncoderPreNet.__init__cCs|j |j¡|_| |¡}|S)zïComputes the forward pass Arguments --------- x: torch.Tensor a (batch, tokens) input tensor Returns ------- output: torch.Tensor the embedding layer output )rÚtoÚdevice)rÚxrrrÚforward;s zEncoderPreNet.forward)r©Ú__name__Ú __module__Ú __qualname__Ú__doc__rr Ú __classcell__rrrrrsrcs4eZdZdZ     d ‡fdd„ Zdd „Z‡ZS) ÚPostNeta­ FastSpeech2 Conv Postnet Arguments --------- n_mel_channels: int input feature dimension for convolution layers postnet_embedding_dim: int output feature dimension for convolution layers postnet_kernel_size: int postnet convolution kernel size postnet_n_convolutions: int number of convolution layers postnet_dropout: float dropout probability for postnet éPréçà?c sÄtt|ƒ ¡tj|||dd|_t ¡|_t d|dƒD]}|j  tj|||dd¡qtj|||dd|_ t  ¡|_ t |¡|_t |¡|_t |¡|_t |¡|_t |¡|_t |¡|_dS)NÚsame©Ú in_channelsrÚ kernel_sizeÚpaddingé)rr'rrÚConv1dÚconv_prerÚ ModuleListÚconvs_intermediateÚrangeÚappendÚ conv_postÚTanhÚtanhr Úln1Úln2Úln3ÚDropoutÚdropout1Údropout2Údropout3)rÚn_mel_channelsÚpostnet_embedding_dimÚpostnet_kernel_sizeÚpostnet_n_convolutionsÚpostnet_dropoutÚirrrr^s<ü üÿ ü      zPostNet.__init__cCs¢| |¡}| |¡ |j¡}| |¡}| |¡}tt|jƒƒD] }|j||ƒ}q|  |¡ |j¡}| |¡}|  |¡}|  |¡}|  |¡ |j¡}|  |¡}|S)züComputes the forward pass Arguments --------- x: torch.Tensor a (batch, time_steps, features) input tensor Returns ------- output: torch.Tensor the spectrogram predicted )r2r:rÚdtyper9r>r5Úlenr4r;r?r7r<r@)rrrFrrrr ˆs       zPostNet.forward)r(rr)r)r*r!rrrrr'Msú*r'cs,eZdZdZ d‡fdd„ Zdd„Z‡ZS) ÚDurationPredictoraÑDuration predictor layer Arguments --------- in_channels: int input feature dimension for convolution layers out_channels: int output feature dimension for convolution layers kernel_size: int duration predictor convolution kernel size dropout: float dropout probability, 0 by default n_units: int Example ------- >>> from speechbrain.lobes.models.FastSpeech2 import FastSpeech2 >>> duration_predictor_layer = DurationPredictor(in_channels=384, out_channels=384, kernel_size=3) >>> x = torch.randn(3, 400, 384) >>> mask = torch.ones(3, 400, 384) >>> y = duration_predictor_layer(x, mask) >>> y.shape torch.Size([3, 400, 1]) çr0cs|tƒ ¡tj|||dd|_tj|||dd|_tj||d|_t|ƒ|_ t|ƒ|_ t   ¡|_ t  |¡|_t  |¡|_dS)Nr+r,©Ú n_neuronsÚ input_size)rrrr1Úconv1Úconv2r ÚLinearr r:r;rÚReLUÚrelur=r>r?)rr-rr.ÚdropoutÚn_unitsrrrrÁs& üü    zDurationPredictor.__init__cCsn| | ||¡¡}| |¡ |j¡}| |¡}| | ||¡¡}| |¡ |j¡}| |¡}|  ||¡S)a?Computes the forward pass Arguments --------- x: torch.Tensor a (batch, time_steps, features) input tensor x_mask: torch.Tensor mask of input tensor Returns ------- output: torch.Tensor the duration predictor outputs ) rRrNr:rrGr>rOr;r?r )rrÚx_maskrrrr Øs  zDurationPredictor.forward)rJr0r!rrrrrI§s ÿrIcs0eZdZdZ‡fdd„Zdd„Zdd„Z‡ZS)Ú SPNPredictora< This module for the silent phoneme predictor. It receives phoneme sequences without any silent phoneme token as input and predicts whether a silent phoneme should be inserted after a position. This is to avoid the issue of fast pace at inference time due to having no silent phoneme tokens in the input sequence. Arguments --------- enc_num_layers: int number of transformer layers (TransformerEncoderLayer) in encoder enc_num_head: int number of multi-head-attention (MHA) heads in encoder transformer layers enc_d_model: int the number of expected features in the encoder enc_ffn_dim: int the dimension of the feedforward network model enc_k_dim: int the dimension of the key enc_v_dim: int the dimension of the value enc_dropout: float Dropout for the encoder normalize_before: bool whether normalization should be applied before or after MHA or FFN in Transformer layers. ffn_type: str whether to use convolutional layers instead of feed forward network inside transformer layer ffn_cnn_kernel_size_list: list of int conv kernel size of 2 1d-convs if ffn_type is 1dcnn n_char: int the number of symbols for the token embedding padding_idx: int the index for padding c sftƒ ¡||_| |_t| | |d|_t|ƒ|_t|||||||t j || | d |_ t j d|d|_dS)N©r© Ú num_layersÚnheadÚd_ffnÚd_modelÚkdimÚvdimrSÚ activationÚnormalize_beforeÚffn_typeÚffn_cnn_kernel_size_listr0rK)rrÚ enc_num_headÚ padding_idxrÚ encPreNetrÚ#sinusoidal_positional_embed_encoderrrrQÚ spn_encoderr rPÚ spn_linear) rÚenc_num_layersrcÚ enc_d_modelÚ enc_ffn_dimÚ enc_k_dimÚ enc_v_dimÚ enc_dropoutr`rarbÚn_charrdrrrrs. ÿÿõzSPNPredictor.__init__c CsÌ| |¡}t |d¡ dd|jd¡}||}t||jd}| d¡}| |¡}t ||¡|}tj tj |jd|jd|j ddd  ¡ |j |jddd¡}|j|||d\}} | |¡ d¡} | S) a forward pass for the module Arguments --------- tokens: torch.Tensor input tokens without silent phonemes last_phonemes: torch.Tensor indicates if a phoneme at an index is the last phoneme of a word or not Returns ------- spn_decision: torch.Tensor indicates if a silent phoneme should be inserted after a phoneme ér0©Úpad_idxéÿÿÿÿ©r)Údiagonalr©Úsrc_maskÚsrc_key_padding_mask)reÚtorchÚ unsqueezeÚrepeatÚshaperrdrfÚaddÚtriuÚonesrÚboolrcrgrhÚsqueeze) rÚtokensÚ last_phonemesÚ token_featsÚsrcmaskÚsrcmask_invertedÚposÚspn_maskÚspn_token_featsÚ_Ú spn_decisionrrrr ?s2   ÿ  ýúö  ÿzSPNPredictor.forwardcCs| ||¡}t |¡dk}|S)a—inference function Arguments --------- tokens: torch.Tensor input tokens without silent phonemes last_phonemes: torch.Tensor indicates if a phoneme at an index is the last phoneme of a word or not Returns ------- spn_decision: torch.Tensor indicates if a silent phoneme should be inserted after a phoneme gš™™™™™é?)r ryÚsigmoid)rr‚rƒr‹rrrÚinferns zSPNPredictor.infer)r"r#r$r%rr rr&rrrrrVòs  !+/rVcs6eZdZdZ‡fdd„Z      ddd„Z‡ZS) Ú FastSpeech2aÀThe FastSpeech2 text-to-speech model. This class is the main entry point for the model, which is responsible for instantiating all submodules, which, in turn, manage the individual neural network layers Simplified STRUCTURE: input->token embedding ->encoder ->duration/pitch/energy predictor ->duration upsampler -> decoder -> output During training, teacher forcing is used (ground truth durations are used for upsampling) Arguments --------- enc_num_layers: int number of transformer layers (TransformerEncoderLayer) in encoder enc_num_head: int number of multi-head-attention (MHA) heads in encoder transformer layers enc_d_model: int the number of expected features in the encoder enc_ffn_dim: int the dimension of the feedforward network model enc_k_dim: int the dimension of the key enc_v_dim: int the dimension of the value enc_dropout: float Dropout for the encoder dec_num_layers: int number of transformer layers (TransformerEncoderLayer) in decoder dec_num_head: int number of multi-head-attention (MHA) heads in decoder transformer layers dec_d_model: int the number of expected features in the decoder dec_ffn_dim: int the dimension of the feedforward network model dec_k_dim: int the dimension of the key dec_v_dim: int the dimension of the value dec_dropout: float dropout for the decoder normalize_before: bool whether normalization should be applied before or after MHA or FFN in Transformer layers. ffn_type: str whether to use convolutional layers instead of feed forward network inside transformer layer. ffn_cnn_kernel_size_list: list of int conv kernel size of 2 1d-convs if ffn_type is 1dcnn n_char: int the number of symbols for the token embedding n_mels: int number of bins in mel spectrogram postnet_embedding_dim: int output feature dimension for convolution layers postnet_kernel_size: int postnet convolution kernel size postnet_n_convolutions: int number of convolution layers postnet_dropout: float dropout probability for postnet padding_idx: int the index for padding dur_pred_kernel_size: int the convolution kernel size in duration predictor pitch_pred_kernel_size: int kernel size for pitch prediction. energy_pred_kernel_size: int kernel size for energy prediction. variance_predictor_dropout: float dropout probability for variance predictor (duration/pitch/energy) Example ------- >>> import torch >>> from speechbrain.lobes.models.FastSpeech2 import FastSpeech2 >>> model = FastSpeech2( ... enc_num_layers=6, ... enc_num_head=2, ... enc_d_model=384, ... enc_ffn_dim=1536, ... enc_k_dim=384, ... enc_v_dim=384, ... enc_dropout=0.1, ... dec_num_layers=6, ... dec_num_head=2, ... dec_d_model=384, ... dec_ffn_dim=1536, ... dec_k_dim=384, ... dec_v_dim=384, ... dec_dropout=0.1, ... normalize_before=False, ... ffn_type='1dcnn', ... ffn_cnn_kernel_size_list=[9, 1], ... n_char=40, ... n_mels=80, ... postnet_embedding_dim=512, ... postnet_kernel_size=5, ... postnet_n_convolutions=5, ... postnet_dropout=0.5, ... padding_idx=0, ... dur_pred_kernel_size=3, ... pitch_pred_kernel_size=3, ... energy_pred_kernel_size=3, ... variance_predictor_dropout=0.5) >>> inputs = torch.tensor([ ... [13, 12, 31, 14, 19], ... [31, 16, 30, 31, 0], ... ]) >>> input_lengths = torch.tensor([5, 4]) >>> durations = torch.tensor([ ... [2, 4, 1, 5, 3], ... [1, 2, 4, 3, 0], ... ]) >>> mel_post, postnet_output, predict_durations, predict_pitch, avg_pitch, predict_energy, avg_energy, mel_lens = model(inputs, durations=durations) >>> mel_post.shape, predict_durations.shape (torch.Size([2, 15, 80]), torch.Size([2, 5])) >>> predict_pitch.shape, predict_energy.shape (torch.Size([2, 5, 1]), torch.Size([2, 5, 1])) c stƒ ¡||_| |_||_t|ƒ|_t| ƒ|_t|||d|_ t ||||d|_ t ||||d|_ t ||||d|_ tjd||ddd|_tjd||ddd|_t|||||||tj|||d |_t|| | | | | |tj|||d |_tj|| d|_t|||||d |_dS) NrW©r-rr.rSr0r+T©r-rr.r/Úskip_transposerXrK©rArBrCrDrE)rrrcÚ dec_num_headrdrrfÚ#sinusoidal_positional_embed_decoderrrerIÚdurPredÚ pitchPredÚ energyPredrr1Ú pitchEmbedÚ energyEmbedrrrQÚencoderÚdecoderr rPr'Úpostnet)rrircrjrkrlrmrnÚdec_num_layersr“Ú dec_d_modelÚ dec_ffn_dimÚ dec_k_dimÚ dec_v_dimÚ dec_dropoutr`rarbroÚn_melsrBrCrDrErdÚdur_pred_kernel_sizeÚpitch_pred_kernel_sizeÚenergy_pred_kernel_sizeÚvariance_predictor_dropoutrrrr÷sž !ÿÿÿüüüûûõõ ûzFastSpeech2.__init__Nçð?c Csˆt||jd}| d¡} | |¡} | | ¡} t | | ¡| } | d¡ |jd| j d¡  ddd¡  ¡} |j | | |d\} } | | } |  | | ¡ d¡}| ¡dkrV| d¡}|durdt tj |¡d¡}d}| | | ¡}||}|dur‰t| d¡|ƒ}| |¡}|  ddd¡}n | |  ddd¡¡}|  ddd¡}|  |¡} d}| | | ¡}||}|durÄt| d¡|ƒ}| |¡}|  ddd¡}n | |  ddd¡¡}|  ddd¡}|  |¡} t| |durâ|n||d\}}tt |¡ƒ}| |j¡}| d¡} | d¡ |jd|j d¡  ddd¡  ¡} | |¡} t || ¡| }|j|| |d^}}} |  |¡| }| !|¡|}|||||||t |¡fS) aõforward pass for training and inference Arguments --------- tokens: torch.Tensor batch of input tokens durations: torch.Tensor batch of durations for each token. If it is None, the model will infer on predicted durations pitch: torch.Tensor batch of pitch for each frame. If it is None, the model will infer on predicted pitches energy: torch.Tensor batch of energy for each frame. If it is None, the model will infer on predicted energies pace: float scaling factor for durations pitch_rate: float scaling factor for pitches energy_rate: float scaling factor for energies Returns ------- mel_post: torch.Tensor mel outputs from the decoder postnet_output: torch.Tensor mel outputs from the postnet predict_durations: torch.Tensor predicted durations of each token predict_pitch: torch.Tensor predicted pitches of each token avg_pitch: torch.Tensor target pitches for each token if input pitch is not None None if input pitch is None predict_energy: torch.Tensor predicted energies of each token avg_energy: torch.Tensor target energies for each token if input energy is not None None if input energy is None mel_length: predicted lengths of mel spectrograms rqrsr0rrprvN©Úpace)"rrdrzrerfryr}r{rcr|Úpermuter€ršr•rÚdimÚclampÚspecialÚexpm1r–Úaverage_over_durationsr˜r—r™ÚupsamplerÚtensorrrr“r”r›r rœ)rr‚Ú durationsÚpitchÚenergyrªÚ pitch_rateÚ energy_rater…r†r„r‡Ú attn_maskrŠÚpredict_durationsÚdur_pred_reverse_logÚ avg_pitchÚ predict_pitchÚ avg_energyÚpredict_energyÚ spec_featsÚmel_lensÚoutput_mel_featsÚmemoryÚmel_postÚpostnet_outputrrrr ls2    ü ÿ ÿ   ÿ       ý   ü  ÿøzFastSpeech2.forward©NNNr¨r¨r¨r!rrrrrŽ‚s txørŽcCstj|dd ¡}tjj |dd…dd…fd¡}tjj tj|dkddd¡}tjj tj|ddd¡}| ¡\}}| d¡}|dd…ddd…f |||¡} |dd…ddd…f |||¡} t |d| ¡t |d| ¡  ¡} t |d| ¡t |d| ¡  ¡} t  | dk| | | ¡} | S)zðAverage values over durations. Arguments --------- values: torch.Tensor shape: [B, 1, T_de] durs: torch.Tensor shape: [B, T_en] Returns ------- avg: torch.Tensor shape: [B, 1, T_en] r0©r¬Nrs©r0rrJrp) ryÚcumsumÚlongrÚ functionalÚpadÚsizeÚexpandÚgatherÚfloatÚwhere)ÚvaluesÚdursÚdurs_cums_endsÚdurs_cums_startsÚvalues_nonzero_cumsÚ values_cumsÚbsÚlengthÚ n_formantsÚdcsÚdceÚ values_sumsÚ values_nelemsÚavgrrrr°ÿs, ÿ    þ  ÿýÿr°r¨rJcsJ‡‡‡fdd„ttˆƒƒDƒ}dd„|Dƒ}tjjjj|d|d}||fS)aÐupsample encoder output according to durations Arguments --------- feats: torch.Tensor batch of input tokens durs: torch.Tensor durations to be used to upsample pace: float scaling factor for durations padding_value: int padding index Returns ------- mel_post: torch.Tensor mel outputs from the decoder predict_durations: torch.Tensor predicted durations for each token cs,g|]}tjˆ|ˆˆ| ¡dd‘qS)rrÆ)ryÚrepeat_interleaverÉ)Ú.0rF©rÒÚfeatsrªrrÚ =sÿÿzupsample..cSsg|]}|jd‘qS©r)r|)ràÚmelrrrrãBsT)Ú batch_firstÚ padding_value)r5rHryrÚutilsÚrnnÚ pad_sequence)rârÒrªrçÚupsampled_melsrÀÚpadded_upsampled_melsrrárr±(s þ ÿr±c@óeZdZdZdd„ZdS)ÚTextMelCollatezEZero-pads model inputs and targets based on number of frames per stepc! CsZt|ƒ}tt|ƒƒD] }||d||<q tjt dd„|Dƒ¡ddd\}}|d}tjt dd„|Dƒ¡ddd\}}|d} t t|ƒ|¡} t t|ƒ| ¡} t t|ƒ| ¡} t t|ƒ|¡} t t|ƒ| ¡}|  ¡|  ¡|  ¡|  ¡| ¡tt|ƒƒD]g}|||d}|||d}t |||d ¡}|||d }t |||d ¡}|| |d | d¡…f<|| |d | d¡…f<|| |d | d¡…f<|| |d | d¡…f<|||d | d¡…f<qƒ|dd  d¡}t dd„|Dƒƒ}t t|ƒ||¡}| ¡t t|ƒ|¡}| ¡t t|ƒ|¡}| ¡t t|ƒ¡}gg}}tt|ƒƒD]V}||}||d }||d}||d}|||d d …d | d ¡…f<|||d | d¡…f<|||d | d¡…f<| d ¡||<|  ||d¡|  ||d¡q4dd„|Dƒ} t  | ¡} |  dd d ¡}| | |||||| ||| || f S)a•Collate's training batch from normalized text and mel-spectrogram Arguments --------- batch: list [text_normalized, mel_normalized] Returns ------- text_padded: torch.Tensor dur_padded: torch.Tensor input_lengths: torch.Tensor mel_padded: torch.Tensor pitch_padded: torch.Tensor energy_padded: torch.Tensor output_lengths: torch.Tensor len_x: torch.Tensor labels: torch.Tensor wavs: torch.Tensor no_spn_seq_padded: torch.Tensor spn_labels_padded: torch.Tensor last_phonemes_padded: torch.Tensor Ú mel_text_paircSóg|]}t|dƒ‘qSrä©rH©ràrrrrrãoóz+TextMelCollate.__call__..rT©r¬Ú descendingcSrð)éþÿÿÿrñròrrrrãuróröéýÿÿÿr0rsNrpcSsg|] }|d d¡‘qS)rpr0©rÌròrrrrã—óééÚlabelÚwavcSsg|]}|d‘qS)r)rròrrrrã®s) Úlistr5rHryÚsortÚ LongTensorÚ FloatTensorÚzero_rÌÚmaxr6ÚTensorr«)!rÚbatchÚ raw_batchrFÚ input_lengthsÚids_sorted_decreasingÚ max_input_lenÚno_spn_seq_lengthsÚno_spn_ids_sorted_decreasingÚmax_no_spn_seq_lenÚ text_paddedÚno_spn_seq_paddedÚlast_phonemes_paddedÚ dur_paddedÚspn_labels_paddedÚtextÚ no_spn_seqrƒÚdurÚ spn_labelsÚnum_melsÚmax_target_lenÚ mel_paddedÚ pitch_paddedÚ energy_paddedÚoutput_lengthsÚlabelsÚwavsÚidxrår´rµÚlen_xrrrÚ__call__Ns˜ÿ ÿ ýÿ     ózTextMelCollate.__call__N©r"r#r$r%r rrrrrîJs rîcs.eZdZdZ  d‡fdd„ Zdd„Z‡ZS) ÚLossasLoss Computation Arguments --------- log_scale_durations: bool applies logarithm to target durations ssim_loss_weight: float weight for ssim loss duration_loss_weight: float weight for the duration loss pitch_loss_weight: float weight for the pitch loss energy_loss_weight: float weight for the energy loss mel_loss_weight: float weight for the mel loss postnet_mel_loss_weight: float weight for the postnet mel loss spn_loss_weight: float weight for spn loss spn_loss_max_epochs: int Max number of epochs r¨éc s~tƒ ¡tƒ|_t ¡|_t ¡|_t ¡|_t ¡|_ t ¡|_ ||_ ||_ ||_ ||_||_||_||_||_| |_dS©N)rrÚSSIMLossÚ ssim_lossrÚMSELossÚmel_lossÚpostnet_mel_lossÚdur_lossÚ pitch_lossÚ energy_lossÚlog_scale_durationsÚssim_loss_weightÚmel_loss_weightÚpostnet_mel_loss_weightÚduration_loss_weightÚpitch_loss_weightÚenergy_loss_weightÚspn_loss_weightÚspn_loss_max_epochs) rr-r.r1r2r3r/r0r4r5rrrrÜs       z Loss.__init__c CsÐ|\}}}}}} } t|jƒdksJ‚|\ } } } }}}}}}| d¡}| d¡}| d¡}| d¡}|  d¡} |jr@t | ¡¡}t|jdƒD]}|dkrÕ| | |d||…dd…f||d||…dd…f¡}|  | |d||…dd…f||d||…dd…f¡}|  | |d| |…f||d| |…f  tj ¡¡}|  ||d||…f||d||…f  tj ¡¡}| ||d||…f||d||…f  tj ¡¡}qG|| | |d||…dd…f||d||…dd…f¡}||  | |d||…dd…f||d||…dd…f¡}||  | |d| |…f||d| |…f  tj ¡¡}||  ||d||…f||d||…f  tj ¡¡}|| ||d||…f||d||…f  tj ¡¡}qG| | ||¡}t |t|ƒ¡}t |t|ƒ¡}t |t|ƒ¡}t |t|ƒ¡}t |t|ƒ¡}t|| ƒ}||jkr£d|_||j||j||j||j||j||j||j}|||j||j||j||j||j||j||jdœ}|S)asComputes the value of the loss function and updates stats Arguments --------- predictions: tuple model predictions targets: tuple ground truth data current_epoch: int The count of the current epoch. Returns ------- loss: torch.Tensor the loss value rúrsrN)Ú total_lossr&r(r)r*r+r,Úspn_loss)rHr|rr-ryÚlog1prÏr5r(r)r*rÚfloat32r+r,r&Údivr r5r4r.r/r0r1r2r3)rÚ predictionsÚtargetsÚ current_epochÚ mel_targetÚtarget_durationsÚ target_pitchÚ target_energyÚ mel_lengthÚphon_lenrÚmel_outÚpostnet_mel_outÚ log_durationsÚpredicted_pitchÚ average_pitchÚpredicted_energyÚaverage_energyrÀÚ spn_predsÚlog_target_durationsrFr(r)r*r+r,r&r7r6Úlossrrrr úsÎø ö     þþþþþþþþþþ  ÿþýüûúÿ ø z Loss.forward)r¨r#r!rrrrr"Ãs !ör"c CsÊddlm}|j|||||d | j¡}|j||dd|||| | d | j¡} || ƒ}| |ƒ}| ¡dks8J‚|jd|ksAJ‚tj |dd}| r[|t  |¡t  |¡t  |¡}| rat |ƒ}||fS)aëcalculates MelSpectrogram for a raw audio signal Arguments --------- sample_rate : int Sample rate of audio signal. hop_length : int Length of hop between STFT windows. win_length : int Window size. n_fft : int Size of FFT. n_mels : int Number of mel filterbanks. f_min : float Minimum frequency. f_max : float Maximum frequency. power : float Exponent for the magnitude spectrogram. normalized : bool Whether to normalize by magnitude after stft. min_max_energy_norm : bool Whether to normalize by min-max norm : str or None If "slaney", divide the triangular mel weights by the width of the mel band mel_scale : str Scale to use: "htk" or "slaney". compression : bool whether to do dynamic range compression audio : torch.Tensor input audio signal Returns ------- mel : torch.Tensor rmse : torch.Tensor r)Ú transforms)Ú hop_lengthÚ win_lengthÚn_fftÚpowerÚ normalizedrpr0)Ú sample_rateÚn_stftr£Úf_minÚf_maxÚnormÚ mel_scalerÆ) Ú torchaudiorNÚ SpectrogramrrÚMelScaler¬r|ryrXÚminrÚdynamic_range_compression)rTrOrPrQr£rVrWrRrSÚmin_max_energy_normrXrYÚ compressionÚaudiorNÚ audio_to_melÚspecråÚrmserrrÚmel_spectogramys> 6ûú ùø "rer0çñh㈵øä>cCst tj||d|¡S)z+Dynamic range compression for audio signals©r])ryÚlogr­)rÚCÚclip_valrrrr^Ñsr^csHeZdZdZ‡fdd„Zd dd„Zdejdejfd d „Zd d „Z ‡Z S)r%zkSSIM loss as (1 - SSIM) SSIM is explained here https://en.wikipedia.org/wiki/Structural_similarity cstƒ ¡tƒ|_dSr$)rrÚ _SSIMLossÚ loss_func©rrrrrÛs  zSSIMLoss.__init__NcCs>|dur |j ¡}tj||j|jd}| d¡| d¡k}|S)a:Create a sequence mask for filtering padding in a sequence tensor. Arguments --------- sequence_length: torch.Tensor Sequence lengths. max_len: int Maximum sequence length. Defaults to None. Returns ------- mask: [B, T_max] N)rGrrr0)ÚdatarryÚarangerGrrz)rÚsequence_lengthÚmax_lenÚ seq_rangeÚmaskrrrÚ sequence_maskàs  ÿzSSIMLoss.sequence_maskrrscCsHtj| |d¡ddd}tj| |d¡ddd}||||dS)aMin-Max normalize tensor through first dimension Arguments --------- x: torch.Tensor input tensor [B, D1, D2] mask: torch.Tensor input mask [B, D1, 1] Returns ------- Normalized tensor r)r0rpT)r¬Úkeepdimgꌠ9Y>)Fç:Œ0âŽyE>)ryÚamaxÚ masked_fillÚamin)rrrsÚmaximumÚminimumrrrÚsample_wise_min_max÷s ÿzSSIMLoss.sample_wise_min_maxcCs¶|j|| d¡d d¡}| ||¡}| ||¡}| || d¡|| d¡¡}| ¡dkrAtd| ¡›dƒtjd|j d}| ¡dkrYtd| ¡›d ƒtjd|j d}|S) at Arguments --------- y_hat: torch.Tensor model prediction values [B, T, D]. y: torch.Tensor target values [B, T, D]. length: torch.Tensor length of each sample in a batch for masking. Returns ------- loss: Average loss value in range [0, 1] masked by the length. r0)rprqrpr¨z > SSIM loss is out-of-range z, setting it 1.0rtrJz, setting it 0.0) rtrÌrzr|rlÚitemÚprintryr²r)rÚy_hatÚyrØrsÚy_normÚ y_hat_normr&rrrr s* ÿþ  ÿ ÿ ÿzSSIMLoss.forwardr$) r"r#r$r%rrtryrr|r r&rrrrr%Ös   r%cseZdZdZgd¢Z       d‡fd d „ Zdd d „Z   d dd„Zdd„Zd!dd„Z d!dd„Z       d"dd„Z dd„Z ‡Z S)#rkaECreates a criterion that measures the structural similarity index error between each element in the input x and target y. Equation link: https://en.wikipedia.org/wiki/Structural_similarity x and y are tensors of arbitrary shapes with a total of n elements each. The sum operation still operates over all the elements, and divides by n. The division by n can be avoided if one sets reduction = sum. In case of 5D input tensors, complex value is returned as a tensor of size 2. Arguments --------- kernel_size: int By default, the mean and covariance of a pixel is obtained by convolution with given filter_size. kernel_sigma: float Standard deviation for Gaussian kernel. k1: float Coefficient related to c1 (see equation in the link above). k2: float Coefficient related to c2 (see equation in the link above). downsample: bool Perform average pool before SSIM computation (Default: True). reduction: str Specifies the reduction type data_range: Union[int, float] Maximum value range of images (usually 1.0 or 255). Example ------- >>> loss = _SSIMLoss() >>> x = torch.rand(3, 3, 256, 256, requires_grad=True) >>> y = torch.rand(3, 3, 256, 256) >>> output = loss(x, y) >>> output.backward() )r.Úk1Úk2ÚsigmaÚkernelÚ reductioné çø?ç{®Gáz„?縅ëQ¸ž?TÚmeanr¨csTtƒ ¡||_||_|ddksJd|›dƒ‚||_||_||_||_||_dS)Nrpr0úKernel size must be odd, got [ú]) rrr‡r.Ú kernel_sigmarƒr„Ú downsampleÚ data_range)rr.rrƒr„rr‡r‘rrrrYs  ÿ z_SSIMLoss.__init__cCs<|dkr|S|dkr|jddS|dkr|jddStdƒ‚)a?Reduce input in batch dimension if needed. Arguments --------- x: torch.Tensor Tensor with shape (B, *). reduction: str Specifies the reduction type: none | mean | sum (Default: mean) Returns ------- Reduced outputs. ÚnonerŒrrÆÚsumz:Unknown reduction. Expected one of {'none', 'mean', 'sum'})rŒr“Ú ValueError)rrr‡rrrÚ_reducevs  ÿz_SSIMLoss._reduce©rrs©rJgð¿Nc CsØ |d}|D]â}t |¡sJdt|ƒ›ƒ‚|j|jks)Jd|j›d|j›ƒ‚|durD| ¡| ¡ksCJd| ¡›d| ¡›ƒ‚n&| ¡|d|d …| ¡|d|d …ksjJd | ¡›d| ¡›ƒ‚|d|d kr‰| ¡|dksˆJd |d›d| ¡›ƒ‚n,|d|d krµ|d| ¡kr¡|d ksµnJd |d›d|d ›d| ¡›ƒ‚|d|d kré|d| ¡ksÓJd |d›d| ¡›ƒ‚| ¡|d kséJd|d ›d| ¡›ƒ‚qdS)aøCheck if the input satisfies the requirements Arguments --------- tensors: torch.Tensor torch.Tensors to check dim_range: Tuple[int, int] Allowed number of dimensions. (min, max) data_range: Tuple[float, float] Allowed range of values in tensors. (min, max) size_range: Tuple[int, int] Dimensions to include in size comparison. (start_dim, end_dim + 1) Returns ------- None FNrzExpected torch.Tensor, got zExpected tensors to be on z, got z%Expected tensors with same size, got z and r0z9Expected tensors with same size at given dimensions, got z$Expected number of dimensions to be z,Expected number of dimensions to be between z*Expected values to be greater or equal to z(Expected values to be lower or equal to )ryÚ is_tensorÚtyperrÌr¬r]r)rÚtensorsÚ dim_ranger‘Ú size_rangerÚtrrrÚ_validate_inputsBÿÿÿþÿ$ ÿÿÿ€âz_SSIMLoss._validate_inputcCsdtj|tjd}||dd8}|d}| d¡| d¡ d|d ¡}|| ¡}| d¡S)a7Returns 2D Gaussian kernel N(0,sigma^2) Arguments --------- kernel_size: int Size of the kernel sigma: float Std of the distribution Returns ------- gaussian_kernel: torch.Tensor [1, kernel_size, kernel_size] ©rGr0ç@rpr)ryror9rzÚexpr“)rr.r…ÚcoordsÚgrrrÚgaussian_filterÎs &  z_SSIMLoss.gaussian_filtercCsF| d¡| d¡ks| d¡| d¡kr"td| ¡›d| ¡›ƒ‚|d}|d}| d¡}tj||dd|d} tj||dd|d} | d} | d} | | } tj|d|dd|d| }tj|d|dd|d| }tj|||dd|d| }d |||||}d | || | ||}|jd d }|jd d }||fS) aCalculate Structural Similarity (SSIM) index for X and Y per channel. Arguments --------- x: torch.Tensor An input tensor (N, C, H, W). y: torch.Tensor A target tensor (N, C, H, W). kernel: torch.Tensor 2D Gaussian kernel. k1: float Algorithm parameter (see equation in the link above). k2: float Algorithm parameter (see equation in the link above). Try a larger K2 constant (e.g. 0.4) if you get a negative or NaN results. Returns ------- Full Value of Structural Similarity (SSIM) index. rsröúAKernel size can't be greater than actual input size. Input size: ú. Kernel size: rpr0r©ÚweightÚstrider/Úgroupsr )rsrörÆ)rÌr”ÚFÚconv2drŒ)rrr€r†rƒr„Úc1Úc2Ú n_channelsÚmu_xÚmu_yÚmu_xxÚmu_yyÚmu_xyÚsigma_xxÚsigma_yyÚsigma_xyÚcsÚssÚssim_valrrrÚ_ssim_per_channelæsR( ÿÿ  ÿ ÿÿýÿÿýÿÿýÿ  z_SSIMLoss._ssim_per_channelc$Csf| d¡}| d¡| d¡ks| d¡| d¡kr'td| ¡›d| ¡›ƒ‚|d}|d}|d} |d } |d} |d } tj| |dd |d } tj| |dd |d }tj| |dd |d }tj| |dd |d }|  d¡| d¡}| d¡| d¡}| |||}| |||}d }|  d¡|  d¡}|  d¡|  d¡}| | | | }| | | | }tj||dd |d |}tj||dd |d |}tj||dd |d |}tj||dd |d |}tj||fdd }tj||fdd }|d||| d¡| d¡||} |d||| d¡| d¡||}!|!| }!|!jdd }"| jdd }#|"|#fS)a¡Calculate Structural Similarity (SSIM) index for Complex X and Y per channel. Arguments --------- x: torch.Tensor An input tensor (N, C, H, W, 2). y: torch.Tensor A target tensor (N, C, H, W, 2). kernel: torch.Tensor 2-D gauss kernel. k1: float Algorithm parameter (see equation in the link above). k2: float Algorithm parameter (see equation in the link above). Try a larger K2 constant (e.g. 0.4) if you get a negative or NaN results. Returns ------- Full Value of Complex Structural Similarity (SSIM) index. r0rörsr÷r¥r¦rp).r).r0rr§r¨rÆ)rör÷) rÌr”r«r¬ÚpowryÚstackrzrŒ)$rrr€r†rƒr„r¯r­r®Úx_realÚx_imagÚy_realÚy_imagÚmu1_realÚmu1_imagÚmu2_realÚmu2_imagÚmu1_sqÚmu2_sqÚ mu1_mu2_realÚ mu1_mu2_imagÚ compensationÚx_sqÚy_sqÚx_y_realÚx_y_imagÚ sigma1_sqÚ sigma2_sqÚ sigma12_realÚ sigma12_imagÚsigma12Úmu1_mu2Úcs_mapÚssim_maprºr¸rrrÚ_ssim_per_channel_complex,s” ( ÿÿ ÿ ÿ ÿ ÿ ÿýÿ ÿýÿ ÿýÿ ÿýÿÿþÿÿ  z#_SSIMLoss._ssim_per_channel_complexFc Cs$|ddksJd|›dƒ‚|j||gdd|fd|t|ƒ}|t|ƒ}tdtt| ¡dd …ƒd ƒƒ} | dkrK|rKtj|| d }tj|| d }| ||¡  | d¡ddd¡  |¡} |  ¡d krf|j n|j } | ||| | | d \}}| d¡}| d¡}| ||¡}| ||¡}|r||gS|S)a^Interface of Structural Similarity (SSIM) index. Inputs supposed to be in range [0, data_range]. To match performance with skimage and tensorflow set downsample = True. Arguments --------- x: torch.Tensor An input tensor (N, C, H, W) or (N, C, H, W, 2). y: torch.Tensor A target tensor (N, C, H, W) or (N, C, H, W, 2). kernel_size: int The side-length of the sliding window used in comparison. Must be an odd value. kernel_sigma: float Sigma of normal distribution. data_range: Union[int, float] Maximum value range of images (usually 1.0 or 255). reduction: str Specifies the reduction type: none | mean | sum. Default:mean full: bool Return cs map or not. downsample: bool Perform average pool before SSIM computation. Default: True k1: float Algorithm parameter (see equation in the link above). k2: float Algorithm parameter (see equation in the link above). Try a larger K2 constant (e.g. 0.4) if you get a negative or NaN results. Returns ------- Value of Structural Similarity (SSIM) index. In case of 5D input tensors, complex value is returned as a tensor of size 2. rpr0rrŽ)rûr)r)r›r‘röNé)r.r))rr€r†rƒr„)ržrÏrÚroundr]rÌr«Ú avg_pool2dr¤r{rr¬r×r»rŒr•)rrr€r.rr‘r‡Úfullrrƒr„Úfr†Ú_compute_ssim_per_channelrÖrÕrºr¸rrrÚssim“s<0 ÿÿ  "  ý ÿý  ÿ    z_SSIMLoss.ssimc Cs:|j|||j|j|j|j|jd|j|jd }t  |¡|S)aÞComputation of Structural Similarity (SSIM) index as a loss function. Arguments --------- x: torch.Tensor An input tensor (N, C, H, W) or (N, C, H, W, 2). y: torch.Tensor A target tensor (N, C, H, W) or (N, C, H, W, 2). Returns ------- Value of SSIM loss to be minimized, i.e 1 - ssim in [0, 1] range. In case of 5D input tensors, complex value is returned as a tensor of size 2. F) rr€r.rrr‘r‡rÛrƒr„) rÞr.rrr‘r‡rƒr„ryÚ ones_like)rrr€Úscorerrrr êsö z_SSIMLoss.forward)rˆr‰rŠr‹TrŒr¨)rŒ)r–r—N)rŠr‹)rˆr‰r¨rŒFTrŠr‹)r"r#r$r%Ú __constants__rr•ržr¤r»r×rÞr r&rrrrrk3s:#ø  û?  Fk õWrkc@rí)ÚTextMelCollateWithAlignmenta`Zero-pads model inputs and targets based on number of frames per step result: tuple a tuple of tensors to be used as inputs/targets ( text_padded, dur_padded, input_lengths, mel_padded, output_lengths, len_x, labels, wavs ) c Cst|ƒ}tt|ƒƒD] }||d||<q tjt dd„|Dƒ¡ddd\}}|d}t t|ƒ|¡}| ¡tt|ƒƒD]}|||d}|||d| d¡…f<q=|dd d¡} td d„|Dƒƒ} t  t|ƒ| | ¡} |  ¡t  t|ƒ| ¡} |  ¡t  t|ƒ| ¡} |  ¡t t|ƒ¡}gg}}tt|ƒƒD]U}||}||d}||d }||d }|| |dd…d| d¡…f<|| |d| d¡…f<|| |d| d¡…f<| d¡||<|  ||d ¡|  ||d ¡qœ|   dd d¡} ||| | | |||fS)aàCollate's training batch from normalized text and mel-spectrogram Arguments --------- batch: list [text_normalized, mel_normalized] Returns ------- phoneme_padded: torch.Tensor input_lengths: torch.Tensor mel_padded: torch.Tensor pitch_padded: torch.Tensor energy_padded: torch.Tensor output_lengths: torch.Tensor labels: torch.Tensor wavs: torch.Tensor rïcSrðrärñròrrrrã6róz8TextMelCollateWithAlignment.__call__..rTrôNr0cSsg|] }|d d¡‘qS)r0røròrrrrãDrùrprúrürý) rþr5rHryrÿrrrÌrrr6r«)rrrrFrrr Úphoneme_paddedÚphonemerrrrrrrrrrår´rµrrrr sZÿ ÿ    øz$TextMelCollateWithAlignment.__call__Nr!rrrrrâ s râc CsÖtj }||}|j}|j}| ¡ ¡ ¡}| ¡ ¡ ¡ tj¡}|j \}}}tj |j tj d}tj ||ftj d} tj |tj d dd¡} t|ƒD]G} tj| ddgddggd|ddd…dd…f} | } | | k}t || | ¡}||dd…dd…| f<| | k}t |||dd…dd…| f|¡} qKt ||d¡}tj |j tj d}|dd…dd…df d¡ tj ¡d}t  |¡}tt|ƒƒD]} d|||| f<||||| fd}qÂ|| tj ¡}t |¡j||d}|S) a< Monotonic alignment search algorithm, numpy works faster than the torch implementation. Arguments --------- value: torch.Tensor input alignment values [b, t_x, t_y] mask: torch.Tensor input alignment mask [b, t_x, t_y] Returns ------- path: torch.Tensor Example ------- >>> import torch >>> from speechbrain.lobes.models.FastSpeech2 import maximum_path_numpy >>> alignment = torch.rand(2, 5, 100) >>> mask = torch.ones(2, 5, 100) >>> hard_alignments = maximum_path_numpy(alignment, mask) rŸr0rsrÚconstant)ÚmodeÚconstant_valuesN)rrG)ÚnpÚinfrrGÚcpuÚdetachÚnumpyÚastypeÚbool_r|ÚzerosÚint64r9roÚreshaper5rËrÐr“ÚreversedryÚ from_numpyr)ÚvaluersÚ max_neg_valrrGÚbÚt_xÚt_yÚ directionÚvÚx_rangeÚjÚv0Úv1Úmax_maskÚv_maxÚ index_maskÚpathÚindexÚ index_rangerrrÚmaximum_path_numpyhs@  ÿþ&( rcs2eZdZdZ    d ‡fdd„ Zdd„Z‡ZS) ÚAlignmentNetworkaÅLearns the alignment between the input text and the spectrogram with Gaussian Attention. query -> conv1d -> relu -> conv1d -> relu -> conv1d -> L2_dist -> softmax -> alignment key -> conv1d -> relu -> conv1d - - - - - - - - - - - -^ Arguments --------- in_query_channels: int Number of channels in the query network. Defaults to 80. in_key_channels: int Number of channels in the key network. Defaults to 512. attn_channels: int Number of inner channels in the attention layers. Defaults to 80. temperature: float Temperature for the softmax. Defaults to 0.0005. Example ------- >>> import torch >>> from speechbrain.lobes.models.FastSpeech2 import AlignmentNetwork >>> aligner = AlignmentNetwork( ... in_query_channels=80, ... in_key_channels=512, ... attn_channels=80, ... temperature=0.0005, ... ) >>> phoneme_feats = torch.rand(2, 512, 20) >>> mels = torch.rand(2, 80, 100) >>> alignment_soft, alignment_logprob = aligner(mels, phoneme_feats, None, None) >>> alignment_soft.shape, alignment_logprob.shape (torch.Size([2, 1, 100, 20]), torch.Size([2, 1, 100, 20])) r(rçü©ñÒMb@?csÔtƒ ¡||_tjjdd|_tjjdd|_t  t j ||ddddddtj  ¡t j |d|ddddd¡|_ t  t j ||ddddddtj  ¡t j |d|dddddtj  ¡t j ||ddddd¡|_dS)NrúrÆrpr+T)r-rr.r/Úbiasr‘r0)rrÚ temperatureryrÚSoftmaxÚsoftmaxÚ LogSoftmaxÚ log_softmaxÚ Sequentialrr1rQÚ key_layerÚ query_layer)rÚin_query_channelsÚin_key_channelsÚ attn_channelsr rrrrÆsf úúöúúú ízAlignmentNetwork.__init__c CsÆ| |¡}| |¡}|dd…dd…dd…df|dd…dd…dfd}|j |jddd}|durF| |¡t |dd…dfd¡}|durZ|j |  ¡  d¡t dƒ ¡|  |¡} | |fS)a9Forward pass of the aligner encoder. Arguments --------- queries: torch.Tensor the query tensor [B, C, T_de] keys: torch.Tensor the query tensor [B, C_emb, T_en] mask: torch.Tensor the query mask[B, T_de] attn_prior: torch.Tensor the prior attention tensor [B, 1, T_en, T_de] Returns ------- attn: torch.Tensor soft attention [B, 1, T_en, T_de] attn_logp: torch.Tensor log probabilities [B, 1, T_en , T_de] Nrpr0T)rurvré) rrr r“r ryrhrnÚ masked_fill_r€rzrÏr ) rÚqueriesÚkeysrsÚ attn_priorÚkey_outÚ query_outÚ attn_factorÚ attn_logpÚattnrrrr s  6 ÿÿ zAlignmentNetwork.forward)r(rr(rr!rrrrr£s$û=rcs>eZdZdZ‡fdd„Zdd„Z      d dd „Z‡ZS) ÚFastSpeech2WithAlignmentaßThe FastSpeech2 text-to-speech model with internal alignment. This class is the main entry point for the model, which is responsible for instantiating all submodules, which, in turn, manage the individual neural network layers. Certain parts are adopted from the following implementation: https://github.com/coqui-ai/TTS/blob/dev/TTS/tts/models/forward_tts.py Simplified STRUCTURE: input -> token embedding -> encoder -> aligner -> duration/pitch/energy -> upsampler -> decoder -> output Arguments --------- enc_num_layers: int number of transformer layers (TransformerEncoderLayer) in encoder enc_num_head: int number of multi-head-attention (MHA) heads in encoder transformer layers enc_d_model: int the number of expected features in the encoder enc_ffn_dim: int the dimension of the feedforward network model enc_k_dim: int the dimension of the key enc_v_dim: int the dimension of the value enc_dropout: float Dropout for the encoder in_query_channels: int Number of channels in the query network. in_key_channels: int Number of channels in the key network. attn_channels: int Number of inner channels in the attention layers. temperature: float Temperature for the softmax. dec_num_layers: int number of transformer layers (TransformerEncoderLayer) in decoder dec_num_head: int number of multi-head-attention (MHA) heads in decoder transformer layers dec_d_model: int the number of expected features in the decoder dec_ffn_dim: int the dimension of the feedforward network model dec_k_dim: int the dimension of the key dec_v_dim: int the dimension of the value dec_dropout: float dropout for the decoder normalize_before: bool whether normalization should be applied before or after MHA or FFN in Transformer layers. ffn_type: str whether to use convolutional layers instead of feed forward network inside transformer layer. ffn_cnn_kernel_size_list: list of int conv kernel size of 2 1d-convs if ffn_type is 1dcnn n_char: int the number of symbols for the token embedding n_mels: int number of bins in mel spectrogram postnet_embedding_dim: int output feature dimension for convolution layers postnet_kernel_size: int postnet convolution kernel size postnet_n_convolutions: int number of convolution layers postnet_dropout: float dropout probability for postnet padding_idx: int the index for padding dur_pred_kernel_size: int the convolution kernel size in duration predictor pitch_pred_kernel_size: int kernel size for pitch prediction. energy_pred_kernel_size: int kernel size for energy prediction. variance_predictor_dropout: float dropout probability for variance predictor (duration/pitch/energy) Example ------- >>> import torch >>> from speechbrain.lobes.models.FastSpeech2 import FastSpeech2WithAlignment >>> model = FastSpeech2WithAlignment( ... enc_num_layers=6, ... enc_num_head=2, ... enc_d_model=384, ... enc_ffn_dim=1536, ... enc_k_dim=384, ... enc_v_dim=384, ... enc_dropout=0.1, ... in_query_channels=80, ... in_key_channels=384, ... attn_channels=80, ... temperature=0.0005, ... dec_num_layers=6, ... dec_num_head=2, ... dec_d_model=384, ... dec_ffn_dim=1536, ... dec_k_dim=384, ... dec_v_dim=384, ... dec_dropout=0.1, ... normalize_before=False, ... ffn_type='1dcnn', ... ffn_cnn_kernel_size_list=[9, 1], ... n_char=40, ... n_mels=80, ... postnet_embedding_dim=512, ... postnet_kernel_size=5, ... postnet_n_convolutions=5, ... postnet_dropout=0.5, ... padding_idx=0, ... dur_pred_kernel_size=3, ... pitch_pred_kernel_size=3, ... energy_pred_kernel_size=3, ... variance_predictor_dropout=0.5) >>> inputs = torch.tensor([ ... [13, 12, 31, 14, 19], ... [31, 16, 30, 31, 0], ... ]) >>> mels = torch.rand(2, 100, 80) >>> mel_post, postnet_output, durations, predict_pitch, avg_pitch, predict_energy, avg_energy, mel_lens, alignment_durations, alignment_soft, alignment_logprob, alignment_mas = model(inputs, mels) >>> mel_post.shape, durations.shape (torch.Size([2, 100, 80]), torch.Size([2, 5])) >>> predict_pitch.shape, predict_energy.shape (torch.Size([2, 5, 1]), torch.Size([2, 5, 1])) >>> alignment_soft.shape, alignment_mas.shape (torch.Size([2, 100, 5]), torch.Size([2, 100, 5])) c!! s tƒ ¡||_| |_||_t|ƒ|_t|ƒ|_t|||d|_ t |||| d|_ t |||| d|_ t |||| d|_ tjd||ddd|_tjd||ddd|_t|||||||tj|||d |_t| | |||||tj|||d |_tj||d|_t|||||d |_t|| | | d |_dS) NrWrr0r+TrrXrKr’)rrrr )rrrcr“rdrrfr”rrerIr•r–r—rr1r˜r™rrrQršr›r rPr'rœrÚaligner)!rrircrjrkrlrmrnrrrr rr“ržrŸr r¡r¢r`rarbror£rBrCrDrErdr¤r¥r¦r§rrrr¨sª &ÿÿÿüüüûûõõû üz!FastSpeech2WithAlignment.__init__c Cst |d¡t |d¡}| | dd¡| dd¡|d¡\}}t| d¡ dd¡ ¡| d¡ ¡ƒ}t |d¡ ¡} | d¡ dd¡}| |||fS)aêAligner forward pass. 1. Compute a mask to apply to the attention map. 2. Run the alignment network. 3. Apply MAS (Monotonic alignment search) to compute the hard alignment map. 4. Compute the durations from the hard alignment map. Arguments --------- x: torch.Tensor Input sequence [B, T_en, C_en]. y: torch.Tensor Output sequence [B, T_de, C_de]. x_mask: torch.Tensor Input sequence mask [B, 1, T_en]. y_mask: torch.Tensor Output sequence mask [B, 1, T_de]. Returns ------- durations: torch.Tensor Durations from the hard alignment map [B, T_en]. alignment_soft: torch.Tensor soft alignment potentials [B, T_en, T_de]. alignment_logprob: torch.Tensor log scale alignment potentials [B, 1, T_de, T_en]. alignment_mas: torch.Tensor hard alignment map [B, T_en, T_de]. rsrpr0N) ryrzrÚ transposerrÚ contiguousr“Úint) rrr€rUÚy_maskr¸Úalignment_softÚalignment_logprobÚ alignment_masr³rrrÚ_forward_aligner( sÿ þ z)FastSpeech2WithAlignment._forward_alignerNr¨c Csút||jd}| d¡} | |¡} | | ¡} t | | ¡| } | d¡ |jd| j d¡  ddd¡  ¡} |j | | |d\} } | | } d}d}d}d}|dur{t||jd}| d¡}|  | ||  dd¡| dd¡¡\}}}}| dd¡}| dd¡}| | | ¡ ¡}| ¡dkrŽ| d¡}t tj |¡d¡}d}| | | ¡}||}|dur½t| d¡|ƒ}| |¡}|  ddd¡}n | |  ddd¡¡}|  ddd¡}|  |¡} d}| | | ¡}||}|durøt| d¡|ƒ}| |¡}|  ddd¡}n | |  ddd¡¡}|  ddd¡}|  |¡} t| |dur|n||d\}}tt |¡ƒ}| |j¡}| d¡} | d¡ |jd|j d¡  ddd¡  ¡} |  |¡} t || ¡| }|j!|| |d^}}} | "|¡| }| #|¡|}|||||||t |¡||||f S) aÿforward pass for training and inference Arguments --------- tokens: torch.Tensor batch of input tokens mel_spectograms: torch.Tensor batch of mel_spectograms (used only for training) pitch: torch.Tensor batch of pitch for each frame. If it is None, the model will infer on predicted pitches energy: torch.Tensor batch of energy for each frame. If it is None, the model will infer on predicted energies pace: float scaling factor for durations pitch_rate: float scaling factor for pitches energy_rate: float scaling factor for energies Returns ------- mel_post: torch.Tensor mel outputs from the decoder postnet_output: torch.Tensor mel outputs from the postnet predict_durations: torch.Tensor predicted durations of each token predict_pitch: torch.Tensor predicted pitches of each token avg_pitch: torch.Tensor target pitches for each token if input pitch is not None None if input pitch is None predict_energy: torch.Tensor predicted energies of each token avg_energy: torch.Tensor target energies for each token if input energy is not None None if input energy is None mel_length: predicted lengths of mel spectrograms alignment_durations: durations from the hard alignment map alignment_soft: torch.Tensor soft alignment potentials alignment_logprob: torch.Tensor log scale alignment potentials alignment_mas: torch.Tensor hard alignment map rqrsr0rrprvNr©)$rrdrzrerfryr}r{rcr|r«r€ršr&rr•rr¬r­r®r¯r–r°r˜r—r™r±rr²rrr“r”r›r rœ) rr‚Úmel_spectogramsr´rµrªr¶r·r…r†r„r‡r¸rŠÚalignment_durationsr#r$r%r"Úy_mask_invertedr¹Úpredict_durations_reverse_logr»r¼r½r¾r¿rÀrÁrÂrÃrÄrrrr Q sÒ:    ü ÿÿ   üû  ÿþ   ÿ  ÿ    ÿ   ÿ ù   ü  ÿôz FastSpeech2WithAlignment.forwardrÅ)r"r#r$r%rr&r r&rrrrr(s ,ørcó(eZdZdZ‡fdd„Zdd„Z‡ZS)ÚLossWithAlignmentaLoss computation including internal aligner Arguments --------- log_scale_durations: bool applies logarithm to target durations ssim_loss_weight: float weight for the ssim loss duration_loss_weight: float weight for the duration loss pitch_loss_weight: float weight for the pitch loss energy_loss_weight: float weight for the energy loss mel_loss_weight: float weight for the mel loss postnet_mel_loss_weight: float weight for the postnet mel loss aligner_loss_weight: float weight for the alignment loss binary_alignment_loss_weight: float weight for the postnet mel loss binary_alignment_loss_warmup_epochs: int Number of epochs to gradually increase the impact of binary loss. binary_alignment_loss_max_epochs: int From this epoch on the impact of binary loss is ignored. c sštƒ ¡tƒ|_t ¡|_t ¡|_t ¡|_t ¡|_ t ¡|_ t ƒ|_ t ƒ|_||_||_||_||_||_||_||_||_| |_| |_| |_dSr$)rrr%r&rr'r(r)r*r+r,ÚForwardSumLossÚ aligner_lossÚBinaryAlignmentLossÚbinary_alignment_lossr-r.r/r0r1r2r3Úaligner_loss_weightÚbinary_alignment_loss_weightÚ#binary_alignment_loss_warmup_epochsÚ binary_alignment_loss_max_epochs) rr-r.r1r2r3r/r0r1r2r3r4rrrr. s*      ÿ zLossWithAlignment.__init__c" Cs|\}}}}}t|jƒdksJ‚|\ } } } } } }}}}}}}|  d¡} | d¡}|  d¡}| d¡}|  d¡} |jrAt | ¡¡}t|jdƒD]}|dkrÖ| | |d||…dd…f||d||…dd…f¡}|  | |d||…dd…f||d||…dd…f¡}|  | |d||…f||d||…f  tj ¡¡}|  | |d||…f||d||…f  tj ¡¡}| ||d||…f||d||…f  tj ¡¡}qH|| | |d||…dd…f||d||…dd…f¡}||  | |d||…dd…f||d||…dd…f¡}||  | |d||…f||d||…f  tj ¡¡}||  | |d||…f||d||…f  tj ¡¡}|| ||d||…f||d||…f  tj ¡¡}qHd}i}| | ||¡}||j|d<t |t|ƒ¡}||j|d<t |t|ƒ¡}||j|d<t |t|ƒ¡}||j|d<t |t|ƒ¡}||j|d <t |t|ƒ¡}||j|d <|dur×| |||¡}||j|d <|dur|dur||jkrêd} n t||jd ƒd } | ||¡}!|!|j| |d <t| ¡ƒ}||d<|S)a’Computes the value of the loss function and updates stats Arguments --------- predictions: tuple model predictions targets: tuple ground truth data current_epoch: int used to determinate the start/end of the binary alignment loss Returns ------- loss: torch.Tensor the loss value rúrsrNr&r(r)r*r+r,r.r¨r0r6) rHr|rr-ryr8rÏr5r(r)r*rr9r+r,r&r.r:r/r0r1r2r3r.r1r4r]r3r0r2r“rÑ)"rr;r<r=r>r@rArBrCrDrErFrGrHrIrJrÀr(r#r$Úalignment_hardrLrFr(r)r*r+r,r6rMr&r.Úbinary_loss_warmup_weightr0rrrr T sæúó     þþþþþþþþþþÿ ÿ ÿýûÿ ÿÿþÿ zLossWithAlignment.forwardr!rrrrr, s &r,cr )r-a¯CTC alignment loss Arguments --------- blank_logprob: pad value Example ------- >>> import torch >>> from speechbrain.lobes.models.FastSpeech2 import ForwardSumLoss >>> loss_func = ForwardSumLoss() >>> attn_logprob = torch.rand(2, 1, 100, 5) >>> key_lens = torch.tensor([5, 5]) >>> query_lens = torch.tensor([100, 100]) >>> loss = loss_func(attn_logprob, key_lens, query_lens) rscs4tƒ ¡tjjdd|_tjjdd|_||_dS)NrúrÆT)Ú zero_infinity) rrryrr r ÚCTCLossÚctc_lossÚ blank_logprob)rr:rrrrý s  zForwardSumLoss.__init__c CsÒtjjj|d|jd}d}t|jdƒD]K}t d||d¡ d¡}||  ddd¡d||…dd…d||d…f}|  |d¡d}|j |||||d…|||d…d} || }q||jd}|S) aN Arguments --------- attn_logprob: torch.Tensor log scale alignment potentials [B, 1, query_lens, key_lens] key_lens: torch.Tensor mel lengths query_lens: torch.Tensor phoneme lengths Returns ------- total_loss: torch.Tensor rÇ)ÚinputrËrôrJrr0rpN)rÚtarget_lengths) ryrrÊrËr:r5r|rorzr«r r9) rÚ attn_logprobÚkey_lensÚ query_lensÚattn_logprob_paddedr6ÚbidÚ target_seqÚ curr_logprobrMrrrr  s&ÿ ÿü zForwardSumLoss.forward)rsr!rrrrr-ë sr-cr+)r/aËBinary loss that forces soft alignments to match the hard alignments as explained in `https://arxiv.org/pdf/2108.10447.pdf`. Example ------- >>> import torch >>> from speechbrain.lobes.models.FastSpeech2 import BinaryAlignmentLoss >>> loss_func = BinaryAlignmentLoss() >>> alignment_hard = torch.randint(0, 2, (2, 100, 5)) >>> alignment_soft = torch.rand(2, 100, 5) >>> loss = loss_func(alignment_hard, alignment_soft) cstƒ ¡dSr$)rrrmrrrr7 szBinaryAlignmentLoss.__init__cCs.t tj||dkdd¡ ¡}| | ¡S)zÐ alignment_hard: torch.Tensor hard alignment map [B, mel_lens, phoneme_lens] alignment_soft: torch.Tensor soft alignment potentials [B, mel_lens, phoneme_lens] r0gê-™—q=rg)ryrhr­r“)rr5r#Úlog_sumrrrr : s ÿþzBinaryAlignmentLoss.forwardr!rrrrr/* s r/)r¨rJ)r0rf)-r%rìrèryÚtorch.nn.functionalrrÊr«Útorch.nn.modules.lossrÚ0speechbrain.lobes.models.transformer.TransformerrrrrÚspeechbrain.nnetrr Úspeechbrain.nnet.embeddingr Úspeechbrain.nnet.lossesr Úspeechbrain.nnet.normalizationr ÚModulerr'rIrVrŽr°r±rîr"rer^r%rkrârrrr,r-r/rrrrÚsV      1ZK )"y7 X]Y_;l[?