o i6^@sdZddlmZmZmZddlZddlmZddlm Z ddl m Z m Z m Z mZmZddlmZmZmZmZddlmZdd lmZmZdd lmZdd lmZd eiZeee e e d ZGddde Z dS)zJETS module for GAN-TTS task.)AnyDictOptionalN)check_argument_types) AbsGANTTS)HiFiGANMultiPeriodDiscriminatorHiFiGANMultiScaleDiscriminator)HiFiGANMultiScaleMultiPeriodDiscriminatorHiFiGANPeriodDiscriminatorHiFiGANScaleDiscriminator)DiscriminatorAdversarialLossFeatureMatchLossGeneratorAdversarialLossMelSpectrogramLoss) JETSGenerator)ForwardSumLoss VarianceLoss) get_segments)force_gatherablejets_generator)hifigan_period_discriminatorhifigan_scale_discriminator"hifigan_multi_period_discriminator!hifigan_multi_scale_discriminator.hifigan_multi_scale_multi_period_discriminatorc#seZdZdZddidddddd d d d d d d ddddddddddddddddddddddiddd dd!dd"dd#dd$dd%d&d'd(d)d*d+d,d-dd.dd/dd0d1d2d3d4dd5d6id7d8d9dd:dd;d6dd?dd@d>dAddBdCdDd6dEdCdFd>dGddHd>dIddJdidKdLdMdNdOd1dPgdQdRgdSdTgdUdVgdWgdWgdWgdXddYddZd[d\d]did^dd_d`dadNdbdNdcdddedfddgd dhgdid8dddjdkdldldddmdnddod dddpddgdqdjd drdgdsd[d]didddt dgdudddCd8gdvgdwd dd[d]didddx dyddzd{ddzd{dddd|dd dddd}d~dddddd dldddlddfdededededeeefdedeeefdeeefdeeefdeeefdeeefdededededede f"fdd Z e ddZ e ddZ d d d ddejdejdejdejdejdejdeejdeejdeejde deeeffddZ d d dddejdejdejdejdejdejdeejdeejdeejdeeeffddZ d d dddejdejdejdejdejdejdeejdeejdeejdeeeffddZ d d d ddejdeejdeejdeejde deeejff ddZZS)JETSa3JETS module (generator + discriminator). This is a module of JETS described in `JETS: Jointly Training FastSpeech2 and HiFi-GAN for End to End Text to Speech'_. .. _`JETS: Jointly Training FastSpeech2 and HiFi-GAN for End to End Text to Speech` : https://arxiv.org/abs/2203.16852 i"Vradimaheadselayerseunitsidlayersdunitspositionwise_layer_typeconv1dpositionwise_conv_kernel_sizeuse_scaled_pos_encTuse_batch_normencoder_normalize_beforedecoder_normalize_beforeencoder_concat_afterFdecoder_concat_afterreduction_factor encoder_type transformer decoder_typetransformer_enc_dropout_rateg?'transformer_enc_positional_dropout_rate!transformer_enc_attn_dropout_ratetransformer_dec_dropout_rate'transformer_dec_positional_dropout_rate!transformer_dec_attn_dropout_rateconformer_rel_pos_typelatestconformer_pos_enc_layer_typerel_posconformer_self_attn_layer_type rel_selfattnconformer_activation_typeswishuse_macaron_style_in_conformeruse_cnn_in_conformer zero_triuconformer_enc_kernel_sizeconformer_dec_kernel_sizeduration_predictor_layersduration_predictor_chansiduration_predictor_kernel_sizeduration_predictor_dropout_rateenergy_predictor_layersenergy_predictor_chansenergy_predictor_kernel_sizeenergy_predictor_dropoutg?energy_embed_kernel_sizeenergy_embed_dropout#stop_gradient_from_energy_predictorpitch_predictor_layerspitch_predictor_chanspitch_predictor_kernel_sizepitch_predictor_dropoutpitch_embed_kernel_sizepitch_embed_dropout"stop_gradient_from_pitch_predictorgenerator_out_channelsgenerator_channelsigenerator_global_channelsgenerator_kernel_sizegenerator_upsample_scales)rbrrgenerator_upsample_kernel_sizes)rdr!r!generator_resblock_kernel_sizes)rKrE generator_resblock_dilations)r(rKrUgenerator_use_additional_convsgenerator_biasgenerator_nonlinear_activation LeakyReLU%generator_nonlinear_activation_paramsnegative_slopegenerator_use_weight_norm segment_size@spkslangs spk_embed_dimNspk_embed_integration_typeadd ) rxrprpryryxavier_uniformg?)use_gst gst_tokens gst_headsgst_conv_layersgst_conv_chans_listgst_conv_kernel_sizegst_conv_stridegst_gru_layers gst_gru_units init_typeinit_enc_alphainit_dec_alpha use_maskinguse_weighted_maskingr AvgPool1d) kernel_sizestridepadding))rUrKrd)rrr!r!r() in_channels out_channels kernel_sizeschannelsmax_downsample_channels max_groupsbiasdownsample_scalesnonlinear_activationnonlinear_activation_paramsuse_weight_normuse_spectral_norm)rrKrUrErfrx)rKrKrKrKr() rrrrrrrrrrr)scalesscale_downsample_poolingscale_downsample_pooling_paramsscale_discriminator_paramsfollow_official_normperiodsperiod_discriminator_paramsmse)average_by_discriminators loss_type)raverage_by_layersinclude_final_outputshannPr) fsn_fft hop_length win_lengthwindown_melsfminfmaxlog_basegF@g@idimodim sampling_rategenerator_typegenerator_paramsdiscriminator_typediscriminator_paramsgenerator_adv_loss_paramsdiscriminator_adv_loss_paramsfeat_match_loss_paramsmel_loss_params lambda_adv lambda_mellambda_feat_match lambda_var lambda_aligncache_generator_outputscstsJtt|}|j||d|di||_t|}|di||_tdi||_ t di| |_ t di| |_ tdi| |_t|_t|_| |_| |_||_||_||_||_d|_||_|jj|_|jj|_|jj|_dS)aInitialize JETS module. Args: idim (int): Input vocabrary size. odim (int): Acoustic feature dimension. The actual output channels will be 1 since JETS is the end-to-end text-to-wave model but for the compatibility odim is used to indicate the acoustic feature dimension. sampling_rate (int): Sampling rate, not used for the training but it will be referred in saving waveform during the inference. generator_type (str): Generator type. generator_params (Dict[str, Any]): Parameter dict for generator. discriminator_type (str): Discriminator type. discriminator_params (Dict[str, Any]): Parameter dict for discriminator. generator_adv_loss_params (Dict[str, Any]): Parameter dict for generator adversarial loss. discriminator_adv_loss_params (Dict[str, Any]): Parameter dict for discriminator adversarial loss. feat_match_loss_params (Dict[str, Any]): Parameter dict for feat match loss. mel_loss_params (Dict[str, Any]): Parameter dict for mel loss. lambda_adv (float): Loss scaling coefficient for adversarial loss. lambda_mel (float): Loss scaling coefficient for mel spectrogram loss. lambda_feat_match (float): Loss scaling coefficient for feat match loss. lambda_var (float): Loss scaling coefficient for variance loss. lambda_align (float): Loss scaling coefficient for alignment loss. cache_generator_outputs (bool): Whether to cache generator outputs. )rrN)rsuper__init__AVAILABLE_GENERATERSupdate generatorAVAILABLE_DISCRIMINATORS discriminatorrgenerator_adv_lossr discriminator_adv_lossr feat_match_lossrmel_lossrvar_lossrforwardsum_lossrrrrrr_cacherrqrrrs)selfrrrrrrrrrrrrrrrrrgenerator_classdiscriminator_class __class__rM/home/ubuntu/.local/lib/python3.10/site-packages/espnet2/gan_tts/jets/jets.pyr5sJ ?   z JETS.__init__cCdS)z)Return whether or not speech is required.Trrrrrrequire_raw_speech#zJETS.require_raw_speechcCr)z*Return whether or not vocoder is required.Frrrrrrequire_vocoder(rzJETS.require_vocodertext text_lengthsfeats feats_lengthsspeechspeech_lengthssidsspembslidsforward_generatorreturnc KsL| r|jd||||||||| d | S|jd||||||||| d | S)aPerform generator forward. Args: text (Tensor): Text index tensor (B, T_text). text_lengths (Tensor): Text length tensor (B,). feats (Tensor): Feature tensor (B, T_feats, aux_channels). feats_lengths (Tensor): Feature length tensor (B,). speech (Tensor): Speech waveform tensor (B, T_wav). speech_lengths (Tensor): Speech length tensor (B,). sids (Optional[Tensor]): Speaker index tensor (B,) or (B, 1). spembs (Optional[Tensor]): Speaker embedding tensor (B, spk_embed_dim). lids (Optional[Tensor]): Language index tensor (B,) or (B, 1). forward_generator (bool): Whether to forward generator. Returns: Dict[str, Any]: - loss (Tensor): Loss scalar tensor. - stats (Dict[str, float]): Statistics to be monitored. - weight (Tensor): Weight tensor to summarize losses. - optim_idx (int): Optimizer index (0 for G and 1 for D). ) rrrrrrrrrNr)_forward_generator_forward_discrminator) rrrrrrrrrrrkwargsrrrforward-s6$   z JETS.forwardc (Ks|d} |d}d} |jr|jdur'd} |jd ||||||| d| } n|j} |jr5|jr5| s5| |_| \ }}}}}}}}}}t|||jj|jj|jjd}| |}t | |}Wdn1skwY| ||}| |}|||}||||||||\}}} ||||}!||j}||j}||j}|||}"||| |j}#|!||j}$|"|#|$}%t|%|"|#|$|||||| |!|d }&t|%|&| f|%j\}%}&}'| s|jsd|_|%|&|'dd S) aPerform generator forward. Args: text (Tensor): Text index tensor (B, T_text). text_lengths (Tensor): Text length tensor (B,). feats (Tensor): Feature tensor (B, T_feats, aux_channels). feats_lengths (Tensor): Feature length tensor (B,). speech (Tensor): Speech waveform tensor (B, T_wav). speech_lengths (Tensor): Speech length tensor (B,). sids (Optional[Tensor]): Speaker index tensor (B,) or (B, 1). spembs (Optional[Tensor]): Speaker embedding tensor (B, spk_embed_dim). lids (Optional[Tensor]): Language index tensor (B,) or (B, 1). Returns: Dict[str, Any]: * loss (Tensor): Loss scalar tensor. * stats (Dict[str, float]): Statistics to be monitored. * weight (Tensor): Weight tensor to summarize losses. * optim_idx (int): Optimizer index (0 for G and 1 for D). rr(TNFrrrrrrrx start_idxsro) generator_lossgenerator_g_lossgenerator_var_lossgenerator_align_lossgenerator_g_mel_lossgenerator_g_adv_lossgenerator_g_feat_match_lossgenerator_var_dur_lossgenerator_var_pitch_lossgenerator_var_energy_lossgenerator_align_forwardsum_lossgenerator_align_bin_losslossstatsweight optim_idxr)size unsqueezerrrtrainingrupsample_factorrortorchno_gradrrrrrrrrrrdictitemrdevice)(rrrrrrrrrrr batch_size reuse_cacheouts speech_hat_bin_loss log_p_attnrd_outsdsp_outspse_outsesspeech_p_hatpradv_lossrdur_loss pitch_loss energy_lossrg_lossr align_lossrrrrrrrls #                zJETS._forward_generatorc  Ks|d} |d}d} |jr|jdur'd} |jd ||||||| d| } n|j} |jr2| s2| |_| ^}}}}}t|||jj|jj|jjd}|| }||}| ||\}}||}t | | | d}t ||| f|j\}}}| s|jsd|_|||dd S) aPerform discriminator forward. Args: text (Tensor): Text index tensor (B, T_text). text_lengths (Tensor): Text length tensor (B,). feats (Tensor): Feature tensor (B, T_feats, aux_channels). feats_lengths (Tensor): Feature length tensor (B,). speech (Tensor): Speech waveform tensor (B, T_wav). speech_lengths (Tensor): Speech length tensor (B,). sids (Optional[Tensor]): Speaker index tensor (B,) or (B, 1). spembs (Optional[Tensor]): Speaker embedding tensor (B, spk_embed_dim). lids (Optional[Tensor]): Language index tensor (B,) or (B, 1). Returns: Dict[str, Any]: * loss (Tensor): Loss scalar tensor. * stats (Dict[str, float]): Statistics to be monitored. * weight (Tensor): Weight tensor to summarize losses. * optim_idx (int): Optimizer index (0 for G and 1 for D). rr(TNFrr)discriminator_lossdiscriminator_real_lossdiscriminator_fake_lossrr)rrrrrrrrordetachrrrrr r)rrrrrrrrrrrr r r r _rrrr real_loss fake_lossrrrrrrrsT #     zJETS._forward_discrminatorpitchenergyuse_teacher_forcingc Ks|d}tj|dgtj|jd}d|vr|dd|d<|r`|dus&J|d}tj|dgtj|jd}|dus>J|d}|dusHJ|d}|jjd |||||||d|\} } n|jjd ||d|\} } t| d| dd S) a Run inference. Args: text (Tensor): Input text index tensor (T_text,). feats (Tensor): Feature tensor (T_feats, aux_channels). pitch (Tensor): Pitch tensor (T_feats, 1). energy (Tensor): Energy tensor (T_feats, 1). use_teacher_forcing (bool): Whether to use teacher forcing. Returns: Dict[str, Tensor]: * wav (Tensor): Generated waveform tensor (T_wav,). * duration (Tensor): Predicted duration tensor (T_text,). Nr()dtyper r)rrrrr&r'r()rrr_r)wavdurationr) rtensorrlongr r inferencerview) rrrr&r'r(rrrr*durrrrr.KsL        zJETS.inference)NNNT)NNN)NNNF)__name__ __module__ __qualname____doc__intstrrrfloatboolrpropertyrrrTensorrrrrr. __classcell__rrrrr*s      !"#$%&'()*+,-./0123456789:;<=>?@ABCDEU(  \ ]     !"o       G       ` r)!r4typingrrrr typeguardrespnet2.gan_tts.abs_gan_ttsrespnet2.gan_tts.hifiganrrr r r espnet2.gan_tts.hifigan.lossr r rrespnet2.gan_tts.jets.generatorrespnet2.gan_tts.jets.lossrrespnet2.gan_tts.utilsr espnet2.torch_utils.device_funcsrrrrrrrrs(