o ei@sdZddlZddlmZddlmmZddlmZddl Z ddl m Z m Z mZdZd9dd Zd d Zd d ZGdddejjZGdddejjZGdddejjZGdddejZGdddeZGdddejjZGdddejjZGdddejjZGdddejjZGd d!d!ejZd:d#d$ZGd%d&d&ejZGd'd(d(ejjZ Gd)d*d*ejZ!Gd+d,d,ejZ"Gd-d.d.ejZ#Gd/d0d0ejZ$d1d2Z%d3d4Z&Gd5d6d6ejZ'Gd7d8d8ejZ(dS);a Neural network modules for the HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis For more details: https://arxiv.org/pdf/2010.05646.pdf, https://arxiv.org/abs/2406.10735 Authors * Jarod Duret 2021 * Yingzhi WANG 2022 N) transforms)Conv1dConv2dConvTranspose1dg?h㈵>cCsttj||d|S)z-Dynamique range compression for audio signalsmin)torchlogclamp)xCclip_valr^/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/speechbrain/lobes/models/HifiGAN.pydynamic_range_compression.src  Cs@tj|||||||||| | d | j} | | }| rt|}|S)acalculates MelSpectrogram for a raw audio signal Arguments --------- sample_rate : int Sample rate of audio signal. hop_length : int Length of hop between STFT windows. win_length : int Window size. n_fft : int Size of FFT. n_mels : int Number of mel filterbanks. f_min : float Minimum frequency. f_max : float Maximum frequency. power : float Exponent for the magnitude spectrogram. normalized : bool Whether to normalize by magnitude after stft. norm : str or None If "slaney", divide the triangular mel weights by the width of the mel band mel_scale : str Scale to use: "htk" or "slaney". compression : bool whether to do dynamic range compression audio : torch.tensor input audio signal Returns ------- Mel spectrogram ) sample_rate hop_length win_lengthn_fftn_melsf_minf_maxpower normalizednorm mel_scale)rMelSpectrogramtodevicer)rrrrrrrrrrr compressionaudio audio_to_melmelrrrmel_spectogram3s&3 r%c Csbg}g}t|dD]K}tj||ddfdd\}}t|dkr6||dd|jdddd }n |||jddd}||||ddfd|dq t|}t d d |D}|d t|||ddf} t | t|d} t |D]"\}} | | |d| df<| |ddf| dk| |ddf<q| | |fS) a Process a given batch of code to extract consecutive unique elements and their associated features. Arguments --------- code : torch.Tensor (batch, time) Tensor of code indices. code_feat : torch.Tensor (batch, time, channel) Tensor of code features. Returns ------- uniq_code_feat_filtered : torch.Tensor (batch, time) Features of consecutive unique codes. mask : torch.Tensor (batch, time) Padding mask for the unique codes. uniq_code_count : torch.Tensor (n) Count of unique codes. Example ------- >>> code = torch.IntTensor([[40, 18, 18, 10]]) >>> code_feat = torch.rand([1, 4, 128]) >>> out_tensor, mask, uniq_code = process_duration(code, code_feat) >>> out_tensor.shape torch.Size([1, 1, 128]) >>> mask.shape torch.Size([1, 1]) >>> uniq_code.shape torch.Size([1]) rNT) return_countsrdimcss|]}|dVqdS)rN)size).0featrrr sz#process_duration..)ranger,r unique_consecutivelenappendcumsumviewcatmax new_zerosarangerepeat enumerateboolfloat) code code_featuniq_code_countuniq_code_feati_count uniq_code_idxmax_lenuniq_code_feat_filteredmaskvrrrprocess_duration|s,    (rJc2eZdZdZd fdd ZddZdd ZZS) ResBlock1ak Residual Block Type 1, which has 3 convolutional layers in each convolution block. Arguments --------- channels : int number of hidden channels for the convolutional layers. kernel_size : int size of the convolution filter in each layer. dilation : list list of dilation value for each conv layer in a block. rrMcsttt|||d|dddddt|||d|dddddt|||d|dddddg|_tt|||ddddddt|||ddddddt|||ddddddg|_dS)NrrsameT in_channels out_channels kernel_sizestridedilationpaddingskip_transpose weight_normr')super__init__nn ModuleListrconvs1convs2selfchannelsrTrV __class__rrr[s   #   zResBlock1.__init__cCsLt|j|jD]\}}t|t}||}t|t}||}||}q|SzReturns the output of ResBlock1 Arguments --------- x : torch.Tensor (batch, channel, time) input tensor. Returns ------- The ResBlock outputs )zipr^r_F leaky_relu LRELU_SLOPE)rar c1c2xtrrrforwards   zResBlock1.forwardcCs,|jD]}|q|jD]}|q dSz=This functions removes weight normalization during inference.N)r^remove_weight_normr_ralayerrrrro)s    zResBlock1.remove_weight_norm)rMrN__name__ __module__ __qualname____doc__r[rmro __classcell__rrrcrrLs  HrLcrK) ResBlock2ak Residual Block Type 2, which has 2 convolutional layers in each convolution block. Arguments --------- channels : int number of hidden channels for the convolutional layers. kernel_size : int size of the convolution filter in each layer. dilation : list list of dilation value for each conv layer in a block. rMrrMc sNttt|||d|dddddt|||d|dddddg|_dS)NrrrPTrQ)rZr[r\r]rconvsr`rcrrr[?s0   zResBlock2.__init__cCs,|jD]}t|t}||}||}q|Sre)rzrgrhri)rar crlrrrrmZs  zResBlock2.forwardcCs|jD]}|qdSrn)rzrorprrrroms  zResBlock2.remove_weight_norm)rMryrrrrrcrrx1s  rxcsLeZdZdZ   dfdd Zddd Zd d Zedd d Z Z S)HifiganGeneratoraHiFiGAN Generator with Multi-Receptive Field Fusion (MRF) Arguments --------- in_channels : int number of input tensor channels. out_channels : int number of output tensor channels. resblock_type : str type of the `ResBlock`. '1' or '2'. resblock_dilation_sizes : List[List[int]] list of dilation values in each layer of a `ResBlock`. resblock_kernel_sizes : List[int] list of kernel sizes for each `ResBlock`. upsample_kernel_sizes : List[int] list of kernel sizes for each transposed convolution. upsample_initial_channel : int number of channels for the first upsampling layer. This is divided by 2 for each consecutive upsampling layer. upsample_factors : List[int] upsampling factors (stride) for each upsampling layer. inference_padding : int constant padding applied to the input at inference time. Defaults to 5. cond_channels : int If provided, adds a conv layer to the beginning of the forward. conv_post_bias : bool Whether to add a bias term to the final conv. Example ------- >>> inp_tensor = torch.rand([4, 80, 33]) >>> hifigan_generator= HifiganGenerator( ... in_channels = 80, ... out_channels = 1, ... resblock_type = "1", ... resblock_dilation_sizes = [[1, 3, 5], [1, 3, 5], [1, 3, 5]], ... resblock_kernel_sizes = [3, 7, 11], ... upsample_kernel_sizes = [16, 16, 4, 4], ... upsample_initial_channel = 512, ... upsample_factors = [8, 8, 2, 2], ... ) >>> out_tensor = hifigan_generator(inp_tensor) >>> out_tensor.shape torch.Size([4, 1, 8448]) rOrTc  sDt| |_t||_t||_t||dddddd|_|dkr$tnt } t |_ t t||D]#\} \}}|j t|d| |d| d||||ddddq2t |_tt|j D]"} |d| d}t t||D]\}\}}|j| |||qsqbt|ddddd| dd|_| d krt| |dd |_dSdS) NrrPTrRrSrTrUrWrXrY1r')rRrSrTrUrWrXbiasrYr)rRrSrT)rZr[inference_paddingr2 num_kernels num_upsamplesrconv_prerLrxr\r]upsr;rfr3r resblocksr0 conv_post cond_layer)rarRrS resblock_typeresblock_dilation_sizesresblock_kernel_sizesupsample_kernel_sizesupsample_initial_channelupsample_factorsr cond_channelsconv_post_biasresblockrBukchrCdrcrrr[sl         zHifiganGenerator.__init__NcCs||}t|dr|||}t|jD]=}t|t}|j||}d}t|j D]!}|dur?|j ||j ||}q,||j ||j ||7}q,||j }qt|}| |}t |}|S)a Arguments --------- x : torch.Tensor (batch, channel, time) feature input tensor. g : torch.Tensor (batch, 1, time) global conditioning input tensor. Returns ------- The generator outputs rN)rhasattrrr0rrgrhrirrrrr tanh)rar gorBz_sumjrrrrms       zHifiganGenerator.forwardcCs@|jD]}|q|jD]}|q |j|jdSrn)rrorrrrprrrro s     z#HifiganGenerator.remove_weight_normcCs(|rtjj||j|jfd}||S)aNThe inference function performs a padding and runs the forward method. Arguments --------- c : torch.Tensor (batch, channel, time) feature input tensor. padding : bool Whether to pad tensor before forward. Returns ------- The generator outputs replicate)r r\ functionalpadrrm)rar{rWrrr inferences  zHifiganGenerator.inference)rOrTN)T) rsrtrurvr[rmror no_gradrrwrrrcrr|ss8 G  r|c(eZdZdZfddZddZZS)VariancePredictoraVariance predictor inspired from FastSpeech2 Arguments --------- encoder_embed_dim : int number of input tensor channels. var_pred_hidden_dim : int size of hidden channels for the convolutional layers. var_pred_kernel_size : int size of the convolution filter in each layer. var_pred_dropout : float dropout probability of each layer. Example ------- >>> inp_tensor = torch.rand([4, 80, 128]) >>> duration_predictor = VariancePredictor( ... encoder_embed_dim = 128, ... var_pred_hidden_dim = 128, ... var_pred_kernel_size = 3, ... var_pred_dropout = 0.5, ... ) >>> out_tensor = duration_predictor (inp_tensor) >>> out_tensor.shape torch.Size([4, 80]) c sfttt|||ddddt|_||_tt|||ddddt|_t |d|_ dS)NrPT)rRrSrTrWrXrYr) rZr[r\ SequentialrReLUconv1dropoutconv2Linearproj)raencoder_embed_dimvar_pred_hidden_dimvar_pred_kernel_sizevar_pred_dropoutrcrrr[Es2   zVariancePredictor.__init__cCsn||dddd}tj||j|jd}||dddd}tj||j|jd}||jddS)z Arguments --------- x : torch.Tensor (batch, channel, time) feature input tensor. Returns ------- Variance predictor output rr')ptrainingr))r transposergrrrrsqueeze)rar rrrrmfs  zVariancePredictor.forwardrsrtrurvr[rmrwrrrcrr)s !rcsneZdZdZ             dfd d Zed dZdfdd Ze dfdd Z Z S)UnitHifiganGeneratora The UnitHiFiGAN generator takes discrete speech tokens as input. The generator is adapted to support bitrate scalability training. For more details, refer to: https://arxiv.org/abs/2406.10735. Arguments --------- in_channels : int number of input tensor channels. out_channels : int number of output tensor channels. resblock_type : str type of the `ResBlock`. '1' or '2'. resblock_dilation_sizes : List[List[int]] list of dilation values in each layer of a `ResBlock`. resblock_kernel_sizes : List[int] list of kernel sizes for each `ResBlock`. upsample_kernel_sizes : List[int] list of kernel sizes for each transposed convolution. upsample_initial_channel : int number of channels for the first upsampling layer. This is divided by 2 for each consecutive upsampling layer. upsample_factors : List[int] upsampling factors (stride) for each upsampling layer. inference_padding : int constant padding applied to the input at inference time. Defaults to 5. cond_channels : int Whether to add a conv to the front conv_post_bias : bool Whether to add a bias to the last conv vocab_size : int size of the dictionary of embeddings. embedding_dim : int size of each embedding vector. attn_dim : int size of attention dimension. duration_predictor : bool enable duration predictor module. var_pred_hidden_dim : int size of hidden channels for the convolutional layers of the duration predictor. var_pred_kernel_size : int size of the convolution filter in each layer of the duration predictor. var_pred_dropout : float dropout probability of each layer in the duration predictor. multi_speaker : bool enable multi speaker training. normalize_speaker_embeddings: bool enable normalization of speaker embeddings. skip_token_embedding: bool Whether to skip the embedding layer in the case of continuous input. pooling_type: str, optional The type of pooling to use. Must be one of ["attention", "sum", "none"]. Defaults to "attention" for scalable vocoder. Example ------- >>> inp_tensor = torch.randint(0, 100, (4, 10, 1)) >>> unit_hifigan_generator= UnitHifiganGenerator( ... in_channels = 128, ... out_channels = 1, ... resblock_type = "1", ... resblock_dilation_sizes = [[1, 3, 5], [1, 3, 5], [1, 3, 5]], ... resblock_kernel_sizes = [3, 7, 11], ... upsample_kernel_sizes = [11, 8, 8, 4, 4], ... upsample_initial_channel = 512, ... upsample_factors = [5, 4, 4, 2, 2], ... vocab_size = 100, ... embedding_dim = 128, ... duration_predictor = True, ... var_pred_hidden_dim = 128, ... var_pred_kernel_size = 3, ... var_pred_dropout = 0.5, ... ) >>> out_tensor, _ = unit_hifigan_generator(inp_tensor) >>> out_tensor.shape torch.Size([4, 1, 3200]) rOrTdFrM? attentionc st||||||||| | | tj| | |_||_|dkr7tjtj| |tj tjj|ddd|_ ||_ |rDt | ||||_ ||_||_||_dS)NrrF)r)rZr[r r\ Embeddingunit_embedding pooling_typerrr attn_poolingduration_predictorr var_predictor multi_speakernormalize_speaker_embeddingsskip_token_embedding)rarRrSrrrrrrrrr vocab_size embedding_dimattn_dimrrrrrrrrrcrrr[s@   zUnitHifiganGenerator.__init__cCs:|\}}}|dddd||}||||}|S)zO Upsamples the input tensor to match the specified max_frames. rMr)r, unsqueezer:r5)r max_framesbatch hidden_dim cond_lengthrrr _upsampleszUnitHifiganGenerator._upsampleNcsT|jr|}n||}|j\}}}}|||||} |jdkr8|| } tj| dd} | | } tj | dd} n|jdkrEtj | dd} n|jdkrL| } | |||}| dd}d}d}|j r|t || dd\}}}| |}||}t|d}|jr|jrtjj|}|d}|||jd}tj||gdd}t|||ffS) aK Arguments --------- x : torch.Tensor (batch, time, channel) feature input tensor. g : torch.Tensor (batch, 1, time) global conditioning input tensor. spk : torch.Tensor Speaker embeddings Returns ------- Generator output rrr)sumnoner'Nr()rrshaper5rrrgsoftmaxr rrrrJrr rrr\r normalizerrr6rZrm)rar rspkr batch_sizetimechannelemb_sizeu_ attn_scores attn_weights u_weightedu_pooledlog_dur log_dur_predrAuniq_code_maskdurrcrrrm s@         zUnitHifiganGenerator.forwardcsf|js||}|j\}}}}|||||}|jdkr5||}tj|dd} || } tj | dd} n|jdkrBtj |dd} n|jdkrI|} | |||}| dd}|j r| ddksdJd| | dd} tjtt| ddd } tj|| d dd}|jr|jrtjj|}|d }|||jd }tj||gdd}t|S) aDThe inference function performs duration prediction and runs the forward method. Arguments --------- x : torch.Tensor (batch, time, channel) feature input tensor. spk : torch.Tensor Speaker embeddings Returns ------- Generator output rrr)rrr'rz-only support single sample batch in inferencerr()rrrr5rrrgrr rrrr,rr roundexplongrepeat_interleaverrr\rrrrr6rZrm)rar rrrrrx_rr x_weightedx_pooledrdur_outrcrrrCs>        zUnitHifiganGenerator.inference)rOrTrrrFrrMrFFFr)NNr) rsrtrurvr[ staticmethodrrmr rrrwrrrcrrxs,W;  8rcs*eZdZdZdfdd ZddZZS) DiscriminatorPaHiFiGAN Periodic Discriminator Takes every Pth value from the input waveform and applies a stack of convolutions. Note: if period is 2 waveform = [1, 2, 3, 4, 5, 6 ...] --> [1, 3, 5 ... ] --> convs -> score, feat Arguments --------- period : int Take every a new value every `period` kernel_size : int Size of 1-d kernel for conv stack stride : int Stride of conv stack rOrMcst||_ttdd|df|dfddddtdd|df|dfddddtdd|df|dfddddtdd|df|dfddddtdd|dfdddddg|_tddd ddddd|_dS) Nr rPTr~r)rMr)rZr[periodr\r]rrzr)rarrTrUrcrrr[sv     1 zDiscriminatorP.__init__cCsg}|j\}}}||jdkr$|j||j}t|d|fd}||}|||||j|j}|jD]}||}t|t}||q3| |}||t |dd}||fS) Arguments --------- x : torch.Tensor (batch, 1, time) input waveform. Returns ------- Scores and features rreflectrr() rrrgrr5rzrhrir3rr flatten)rar r.br{tn_padrqrrrrms      zDiscriminatorP.forward)rOrMrrrrcrrs?rcr)MultiPeriodDiscriminatorzHiFiGAN Multi-Period Discriminator (MPD) Wrapper for the `PeriodDiscriminator` to apply it in different periods. Periods are suggested to be prime numbers to reduce the overlap between each discriminator. cs8tttdtdtdtdtdg|_dS)Nr'rMrOr} )rZr[r\r]rdiscriminatorsrarcrrr[s  z!MultiPeriodDiscriminator.__init__cCsDg}g}t|jD]\}}||\}}||||q ||fS)zReturns Multi-Period Discriminator scores and features Arguments --------- x : torch.Tensor (batch, 1, time) input waveform. Returns ------- Scores and features )r;rr3)rar scoresfeatsrCrscorer.rrrrms    z MultiPeriodDiscriminator.forwardrrrrcrrs  rc*eZdZdZdfdd ZddZZS)DiscriminatorSaXHiFiGAN Scale Discriminator. It is similar to `MelganDiscriminator` but with a specific architecture explained in the paper. SpeechBrain CNN wrappers are not used here because spectral_norm is not often used Arguments --------- use_spectral_norm : bool if `True` switch to spectral norm instead of weight norm. Fcst|r tjjntjj}t|tjdddddd|tjdddddd d |tjdd ddd d d |tjd d ddd d d |tjd dddd d d |tjddddd d d |tjddddddg|_|tjdddddd|_ dS)Nrrr}rW)r')groupsrWrrrOrM) rZr[r\utils spectral_normrYr]rrzr)rause_spectral_normnorm_frcrrr[#s   zDiscriminatorS.__init__cCsXg}|jD]}||}t|t}||q||}||t|dd}||fS)rrr()rzrgrhrir3rr r)rar r.rqrrrrm7s     zDiscriminatorS.forward)Frrrrcrrs rcr)MultiScaleDiscriminatorzHiFiGAN Multi-Scale Discriminator. Similar to MultiScaleMelganDiscriminator but specially tailored for HiFiGAN as in the paper. csRtttddttg|_ttjddddtjddddg|_dS)NT)rrr'r)rZr[r\r]rr AvgPool1d meanpoolsrrcrrr[Ss  z MultiScaleDiscriminator.__init__cCs^g}g}t|jD]!\}}|dkr|j|d|}||\}}||||q ||fS)rrr)r;rr r3)rar rrrBrrr.rrrrm`s    zMultiScaleDiscriminator.forwardrrrrcrrNs  rcr)HifiganDiscriminatoraHiFiGAN discriminator wrapping MPD and MSD. Example ------- >>> inp_tensor = torch.rand([4, 1, 8192]) >>> hifigan_discriminator= HifiganDiscriminator() >>> scores, feats = hifigan_discriminator(inp_tensor) >>> len(scores) 8 >>> len(feats) 8 cstt|_t|_dSr)rZr[rmpdrmsdrrcrrr[s  zHifiganDiscriminator.__init__cCs,||\}}||\}}||||fS)zReturns list of list of features from each layer of each discriminator. Arguments --------- x : torch.Tensor input waveform. Returns ------- Features from each discriminator layer )r r )rar rrscores_feats_rrrrms zHifiganDiscriminator.forwardrrrrcrr ws r  hann_windowc Csrt|d|||}|dddddddf}|dddddddf}ttj|d|ddd}|S)zHcomputes the Fourier transform of short overlapping windows of the inputrNrr'g:0yE>r)r stftrsqrtr ) r rrr window_fnrMPSrrrrs rcr)STFTLossaSTFT loss. Input generate and real waveforms are converted to spectrograms compared with L1 and Spectral convergence losses. It is from ParallelWaveGAN paper https://arxiv.org/pdf/1910.11480.pdf Arguments --------- n_fft : int size of Fourier transform. hop_length : int the distance between neighboring sliding window frames. win_length : int the size of window frame and STFT filter. cs t||_||_||_dSr)rZr[rrr)rarrrrcrrr[s  zSTFTLoss.__init__cCsht||j|j|j}t||j|j|j}tt|t|}tj||ddtj|dd}||fS)a5Returns magnitude loss and spectral convergence loss Arguments --------- y_hat : torch.tensor generated waveform tensor y : torch.tensor real waveform tensor Returns ------- Magnitude loss and spectral convergence loss fro)r) rrrrrgl1_lossr r r)ray_hatyy_hat_My_Mloss_magloss_scrrrrms  zSTFTLoss.forwardrrrrcrrs rcs0eZdZdZ   d fdd ZddZZS) MultiScaleSTFTLosszMulti-scale STFT loss. Input generate and real waveforms are converted to spectrograms compared with L1 and Spectral convergence losses. It is from ParallelWaveGAN paper https://arxiv.org/pdf/1910.11480.pdfrirx2iXir#csFttj|_t|||D]\}}}|jt|||qdSr) rZr[r r\r] loss_funcsrfr3r)ran_ffts hop_lengths win_lengthsrrrrcrrr[s  zMultiScaleSTFTLoss.__init__c CsTt|j}d}d}|jD]}|||\}}||7}||7}q ||}||}||fS)aAReturns multi-scale magnitude loss and spectral convergence loss Arguments --------- y_hat : torch.tensor generated waveform tensor y : torch.tensor real waveform tensor Returns ------- Magnitude loss and spectral convergence loss r)r2r&) rarrNrrflmlscrrrrms   zMultiScaleSTFTLoss.forward)r r!r%rrrrcrrs rcsDeZdZdZ        dfdd ZddZZS) L1SpecLossaL1 Loss over Spectrograms as described in HiFiGAN paper https://arxiv.org/pdf/2010.05646.pdf Note : L1 loss helps leaning details compared with L2 loss Arguments --------- sample_rate : int Sample rate of audio signal. hop_length : int Length of hop between STFT windows. win_length : int Window size. n_mel_channels : int Number of mel filterbanks. n_fft : int Size of FFT. n_stft : int Size of STFT. mel_fmin : float Minimum frequency. mel_fmax : float Maximum frequency. mel_normalized : bool Whether to normalize by magnitude after stft. power : float Exponent for the magnitude spectrogram. norm : str or None If "slaney", divide the triangular mel weights by the width of the mel band mel_scale : str Scale to use: "htk" or "slaney". dynamic_range_compression : bool whether to do dynamic range compression "VrPr@@F?slaneyTcsdt||_||_||_||_||_|dd|_||_||_ | |_ | |_ | |_ | |_ | |_dS)Nr'r)rZr[rrrn_mel_channelsrn_stftmel_fminmel_fmaxmel_normalizedrrrr)rarrrr7rr8r9r:r;rrrrrcrrr[,s  zL1SpecLoss.__init__cCst|j|j|j|j|j|j|j|j|j |j |j |j | }t|j|j|j|j|j|j|j|j|j |j |j |j | }t ||}|S)zReturns L1 Loss over Spectrograms Arguments --------- y_hat : torch.tensor generated waveform tensor y : torch.tensor real waveform tensor Returns ------- L1 loss )r%rrrrr7r9r:rr;rrrrgr)rarrrrrrrrrmLs@ zL1SpecLoss.forward) r/rr0r1rr2r3r4Fr5r6r6Trrrrcrr. s"# r.c@seZdZdZddZdS)MSEGLosszMean Squared Generator Loss The generator is trained to fake the discriminator by updating the sample quality to be classified to a value almost equal to 1. cCst|||j}|S)zReturns Generator GAN loss Arguments --------- score_fake : list discriminator scores of generated waveforms D(G(s)) Returns ------- Generator loss )rgmse_lossnew_onesr)ra score_fake loss_fakerrrrms zMSEGLoss.forwardN)rsrtrurvrmrrrrr<s r<cr)MelganFeatureLosszCalculates the feature matching loss, which is a learned similarity metric measured by the difference in features of the discriminator between a ground truth sample and a generated sample (Larsen et al., 2016, Kumar et al., 2019). ctt|_dSr)rZr[r\L1Loss loss_funcrrcrrr[ zMelganFeatureLoss.__init__c CsZd}d}t|D]\}}t||||D]\}}||||7}|d7}qq||}|S)a4Returns feature matching loss Arguments --------- fake_feats : list discriminator features of generated waveforms real_feats : list discriminator features of groundtruth waveforms Returns ------- Feature matching loss rr)r;rfrD) ra fake_feats real_feats loss_feats num_featsidxrC fake_feat real_featrrrrms zMelganFeatureLoss.forwardrrrrcrrAs rAcr)MSEDLosszMean Squared Discriminator Loss The discriminator is trained to classify ground truth samples to 1, and the samples synthesized from the generator to 0. crBr)rZr[r\MSELossrDrrcrrr[rEzMSEDLoss.__init__cCs:||||j}||||j}||}|||fS)a2Returns Discriminator GAN losses Arguments --------- score_fake : list discriminator scores of generated waveforms score_real : list discriminator scores of groundtruth waveforms Returns ------- Discriminator losses )rDr>rr8)rar? score_real loss_realr@loss_drrrrms   zMSEDLoss.forwardrrrrcrrMs rMcCs<d}t|tr|D] }||}||7}q |S||}|}|S)aCompute Generator adversarial loss function and normalize values Arguments --------- scores_fake : list discriminator scores of generated waveforms loss_func : object object of target generator loss Returns ------- Generator loss r) isinstancelist) scores_fakerDadv_lossr? fake_lossrrr_apply_G_adv_losss  rWc Cstd}d}d}t|tr+t||D]\}}|||d\}}}||7}||7}||7}qn |||\}}}|}|||fS)aeCompute Discriminator losses and normalize loss values Arguments --------- scores_fake : list discriminator scores of generated waveforms scores_real : list discriminator scores of groundtruth waveforms loss_func : object object of target discriminator loss Returns ------- Discriminator losses r)r?rO)rRrSrf) rT scores_realrDloss real_lossrVr?rO total_lossrrr _apply_D_loss s     r\csNeZdZdZ          dfdd Z       d ddZZS) GeneratorLossaCreates a summary of generator losses and applies weights for different losses Arguments --------- stft_loss : object object of stft loss stft_loss_weight : float weight of STFT loss mseg_loss : object object of mseg loss mseg_loss_weight : float weight of mseg loss feat_match_loss : object object of feature match loss feat_match_loss_weight : float weight of feature match loss l1_spec_loss : object object of L1 spectrogram loss l1_spec_loss_weight : float weight of L1 spectrogram loss mseg_dur_loss : object object of mseg duration loss mseg_dur_loss_weight : float weight of mseg duration loss Nrc sJt||_||_||_||_||_||_||_||_ | |_ | |_ dSr) rZr[ stft_lossstft_loss_weight mseg_lossmseg_loss_weightfeat_match_lossfeat_match_loss_weight l1_spec_lossl1_spec_loss_weight mseg_dur_lossmseg_dur_loss_weight) rar^r_r`rarbrcrdrerfrgrcrrr[Ss  zGeneratorLoss.__init__c  CsPd} d} d} i} |jr8||ddddd|dfd|d\} }| | d<|| d<| |j| |} |jrL|||}|| d<| |j|} |jrd|durdt||j}|| d<| |j|} |j r||dur|| ||}|| d <| |j |} |j r|t j jkrtj||d d } | | d <| |j9} | | | | d <| | d<| | d<| S)aReturns a dictionary of generator losses and applies weights Arguments --------- stage : speechbrain.Stage training, validation or testing y_hat : torch.tensor generated waveform tensor y : torch.tensor real waveform tensor scores_fake : list discriminator scores of generated waveforms feats_fake : list discriminator features of generated waveforms feats_real : list discriminator features of groundtruth waveforms log_dur_pred : torch.Tensor Predicted duration for duration loss log_dur : torch.Tensor Real duration for duration loss Returns ------- Dictionary of generator losses rNr'rG_stft_loss_mgG_stft_loss_scG_l1_spec_lossG_mse_fake_lossG_feat_match_lossmean) reduction G_dur_lossG_loss G_gen_loss G_adv_loss)r^r,rr_rdrer`rWrarbrcrfsbStageTRAINrgr=rg)rastagerrrT feats_fake feats_realrrgen_lossrUdur_lossrY stft_loss_mg stft_loss_scrd mse_fake_lossrbrrrrmlsB%,    zGeneratorLoss.forward) NrNrNrNrNr)NNNNNNNrrrrcrr]7s*r]cr)DiscriminatorLosszCreates a summary of discriminator losses Arguments --------- msed_loss : object object of MSE discriminator loss Ncst||_dSr)rZr[ msed_loss)rarrcrrr[s  zDiscriminatorLoss.__init__cCsPd}i}|jr"t|||jd\}}}||d<||d<||d<||7}||d<|S)aNReturns a dictionary of discriminator losses Arguments --------- scores_fake : list discriminator scores of generated waveforms scores_real : list discriminator scores of groundtruth waveforms Returns ------- Dictionary of discriminator losses r)rTrXrDD_mse_gan_lossD_mse_gan_real_lossD_mse_gan_fake_lossD_loss)rr\)rarTrX disc_lossrY mse_D_lossmse_D_real_lossmse_D_fake_lossrrrrms zDiscriminatorLoss.forwardrrrrrcrr~sr~)rr)r))rvr torch.nnr\torch.nn.functionalrrg torchaudior speechbrainrsspeechbrain.nnet.CNNrrrrirr%rJModulerLrxr|rrrrrrr rrrr.r<rArMrWr\r]r~rrrrsH"   IBsB7O q(6) +--v+*-