o i] @s:dZddlmZmZddlZddlmZddlmZddl m Z m Z m Z m Z mZmZddlmZmZmZmZddlmZmZdd lmZdd lmZmZdd lmZmZdd l m!Z!m"Z"dd l#m$Z$ddl%m&Z&ddl'm(Z(ddl)m*Z*ddl+m,Z,e*e,e&e(dZ-e eeedZ.eee e e eeedZ/GdddeZ0dS)z1Joint text-to-wav module for end-to-end training.)AnyDictN)check_argument_types) AbsGANTTS)HiFiGANGeneratorHiFiGANMultiPeriodDiscriminatorHiFiGANMultiScaleDiscriminator)HiFiGANMultiScaleMultiPeriodDiscriminatorHiFiGANPeriodDiscriminatorHiFiGANScaleDiscriminator)DiscriminatorAdversarialLossFeatureMatchLossGeneratorAdversarialLossMelSpectrogramLoss)MelGANGeneratorMelGANMultiScaleDiscriminator)PQMF)ParallelWaveGANDiscriminatorParallelWaveGANGenerator)StyleMelGANDiscriminatorStyleMelGANGenerator)get_random_segments get_segments)force_gatherable) FastSpeech) FastSpeech2) Tacotron2) Transformer) tacotron2 transformer fastspeech fastspeech2)hifigan_generatormelgan_generatorparallel_wavegan_generatorstyle_melgan_generator)hifigan_period_discriminatorhifigan_scale_discriminator"hifigan_multi_period_discriminator!hifigan_multi_scale_discriminator.hifigan_multi_scale_multi_period_discriminator melgan_multi_scale_discriminatorparallel_wavegan_discriminatorstyle_melgan_discriminatorc/sbeZdZdZdddiddddd d d d d d dd dddddddddddddddddddddd id!d d"dd#d$d%d$d&d'd(d'd)d'd*d'd+d'd,d'd-d.d/d0d1d2d3d4d5dd6dd7d id8d9d:d;dd?d@d'dAddBddCd?dDddEddFddGd dHddIddJddKdidLddMddNddOdPdQdPdRdSdTdUdVd dWdXdYd dZd[d\gd]d^d?d_dd`ddadbdcdddeded d dfdgdddPd9gdhgdigdjgdkgdkgdkgdddldmd'iddn d d dodpdqdrdsddtd ddduddgdvdbdwdxdgdydldmd'idd dz d gd{dddd?gdgd|dwddldmd'idd d} d~d ddd dddd d dddddwddSddddSdSd dededdd fdedededededeeefdedeeefdedeeefdedeeefdeeefdeeefdedeeefdedeeefde de de de def.fdd Z e ddZ e ddZ ddejdejdejdejdejdejdedeeeffddZdejdejdejdejdejdejdeeeffddZdejdejdejdejdejdejdeeeffddZdejdeeejffddZZS) JointText2Wavz:General class to jointly train text2mel and vocoder parts. i"Vr!adimiaheadselayerseunitsidlayersdunitspostnet_layers postnet_chansi postnet_filtspostnet_dropout_rateg?positionwise_layer_typeconv1dpositionwise_conv_kernel_sizeuse_scaled_pos_encTuse_batch_normencoder_normalize_beforedecoder_normalize_beforeencoder_concat_afterFdecoder_concat_afterreduction_factor encoder_type conformer decoder_typetransformer_enc_dropout_rateg?'transformer_enc_positional_dropout_rate!transformer_enc_attn_dropout_ratetransformer_dec_dropout_rate'transformer_dec_positional_dropout_rate!transformer_dec_attn_dropout_rateconformer_rel_pos_typelatestconformer_pos_enc_layer_typerel_posconformer_self_attn_layer_type rel_selfattnconformer_activation_typeswishuse_macaron_style_in_conformeruse_cnn_in_conformer zero_triuconformer_enc_kernel_sizeconformer_dec_kernel_sizeduration_predictor_layersduration_predictor_chansduration_predictor_kernel_sizeduration_predictor_dropout_rateenergy_predictor_layersenergy_predictor_chansenergy_predictor_kernel_sizeenergy_predictor_dropoutenergy_embed_kernel_sizeenergy_embed_dropout#stop_gradient_from_energy_predictorpitch_predictor_layerspitch_predictor_chanspitch_predictor_kernel_sizepitch_predictor_dropoutpitch_embed_kernel_sizepitch_embed_dropout"stop_gradient_from_pitch_predictorspkslangs spk_embed_dimNspk_embed_integration_typeadduse_gst gst_tokens gst_headsgst_conv_layersgst_conv_chans_list)r/r/@rrgst_conv_kernel_sizegst_conv_stridegst_gru_layers gst_gru_unitsr init_typexavier_uniformg?)init_enc_alphainit_dec_alpha use_maskinguse_weighted_maskingr")rr2r2)rr4r4)rcr] )r@rcr9 LeakyReLUnegative_slope) out_channelschannelsglobal_channels kernel_sizeupsample_scalesupsample_kernel_sizesresblock_kernel_sizesresblock_dilationsuse_additional_convsbiasnonlinear_activationnonlinear_activation_paramsuse_weight_norm>gV-?g"@)subbandstaps cutoff_ratiobetar* AvgPool1d)rstridepadding))r9rcir)r2r2r4r4r@) in_channelsr kernel_sizesrmax_downsample_channels max_groupsrdownsample_scalesrrruse_spectral_norm)r2rcr9r]r)rcrcrcrcr@) rrrrrrrrrrr)scalesscale_downsample_poolingscale_downsample_pooling_paramsscale_discriminator_paramsfollow_official_normperiodsperiod_discriminator_paramsmse)average_by_discriminators loss_type)raverage_by_layersinclude_final_outputshannPr) fsn_fft hop_length win_lengthwindown_melsfminfmaxlog_baseg@gF@idimodim segment_size sampling_rate text2mel_typetext2mel_params vocoder_typevocoder_paramsuse_pqmf pqmf_paramsdiscriminator_typediscriminator_paramsgenerator_adv_loss_paramsdiscriminator_adv_loss_paramsuse_feat_match_lossfeat_match_loss_params use_mel_lossmel_loss_paramslambda_text2mel lambda_advlambda_feat_match lambda_melcache_generator_outputscstsJt||_| |_tj|_t |}|j ||d|d i||jd<t |}|dvr:|j |dn |dvrD|j |d|d i||jd<|jrYt d i| |_ t| }|d i| |_td i| |_td i||_||_|jrtd i||_||_|jrtd i||_||_||_|jr||_|jr||_||_d|_||_ |jdj!|_!|jdj"|_"|jdj#|_#dS) a.Initialize JointText2Wav module. Args: idim (int): Input vocabrary size. odim (int): Acoustic feature dimension. The actual output channels will be 1 since the model is the end-to-end text-to-wave model but for the compatibility odim is used to indicate the acoustic feature dimension. segment_size (int): Segment size for random windowed inputs. sampling_rate (int): Sampling rate, not used for the training but it will be referred in saving waveform during the inference. text2mel_type (str): The text2mel model type. text2mel_params (Dict[str, Any]): Parameter dict for text2mel model. use_pqmf (bool): Whether to use PQMF for multi-band vocoder. pqmf_params (Dict[str, Any]): Parameter dict for PQMF module. vocoder_type (str): The vocoder model type. vocoder_params (Dict[str, Any]): Parameter dict for vocoder model. discriminator_type (str): Discriminator type. discriminator_params (Dict[str, Any]): Parameter dict for discriminator. generator_adv_loss_params (Dict[str, Any]): Parameter dict for generator adversarial loss. discriminator_adv_loss_params (Dict[str, Any]): Parameter dict for discriminator adversarial loss. use_feat_match_loss (bool): Whether to use feat match loss. feat_match_loss_params (Dict[str, Any]): Parameter dict for feat match loss. use_mel_loss (bool): Whether to use mel loss. mel_loss_params (Dict[str, Any]): Parameter dict for mel loss. lambda_text2mel (float): Loss scaling coefficient for text2mel model loss. lambda_adv (float): Loss scaling coefficient for adversarial loss. lambda_feat_match (float): Loss scaling coefficient for feat match loss. lambda_mel (float): Loss scaling coefficient for mel loss. cache_generator_outputs (bool): Whether to cache generator outputs. )rrtext2mel)r"r#)r)r$r%) aux_channelsvocoderN)$rsuper__init__rrtorchnn ModuleDict generatorAVAILABLE_TEXT2MELupdateAVAILABLE_VOCODERrpqmfAVAILABLE_DISCRIMINATORS discriminatorrgenerator_adv_lossr discriminator_adv_lossrr feat_match_lossrrmel_lossrrrrr_cacherrsrurv)selfrrrrrrrrrrrrrrrrrrrrrrrtext2mel_class vocoder_classdiscriminator_class __class__rX/home/ubuntu/.local/lib/python3.10/site-packages/espnet2/gan_tts/joint/joint_text2wav.pyrCsj T     zJointText2Wav.__init__cCdS)z)Return whether or not speech is required.Trrrrrrequire_raw_speechVz JointText2Wav.require_raw_speechcCr)z*Return whether or not vocoder is required.Frrrrrrequire_vocoder[rzJointText2Wav.require_vocodertext text_lengthsfeats feats_lengthsspeechspeech_lengthsforward_generatorreturnc Ks@|r|jd||||||d|S|jd||||||d|S)a/Perform generator forward. Args: text (Tensor): Text index tensor (B, T_text). text_lengths (Tensor): Text length tensor (B,). feats (Tensor): Feature tensor (B, T_feats, aux_channels). feats_lengths (Tensor): Feature length tensor (B,). speech (Tensor): Speech waveform tensor (B, T_wav). speech_lengths (Tensor): Speech length tensor (B,). forward_generator (bool): Whether to forward generator. Returns: Dict[str, Any]: - loss (Tensor): Loss scalar tensor. - stats (Dict[str, float]): Statistics to be monitored. - weight (Tensor): Weight tensor to summarize losses. - optim_idx (int): Optimizer index (0 for G and 1 for D). )rrrrrrNr)_forward_generator_forward_discrminator) rrrrrrrrkwargsrrrforward`s* zJointText2Wav.forwardcKs|d}|d}d} |jr|jdurHd} |jdd||||dd|\} } } t| dd||jd \} }|jd | }|jrG|j |}n|j\} } }}|j r^|jr^| s^| | ||f|_t |||jd j |j|jd j d }||}t ||}Wdn1swY||}||j}| |j} || }|jr|||}||j}||}| j|d |jr|||}|j|}||}| j|d | j|| |dt|| |f|j\}} }| s|j sd|_|| |ddS)aPerform generator forward. Args: text (Tensor): Text index tensor (B, T_text). text_lengths (Tensor): Text length tensor (B,). feats (Tensor): Feature tensor (B, T_feats, aux_channels). feats_lengths (Tensor): Feature length tensor (B,). speech (Tensor): Speech waveform tensor (B, T_wav). speech_lengths (Tensor): Speech length tensor (B,). Returns: Dict[str, Any]: * loss (Tensor): Loss scalar tensor. * stats (Dict[str, float]): Statistics to be monitored. * weight (Tensor): Weight tensor to summarize losses. * optim_idx (int): Optimizer index (0 for G and 1 for D). rr@TNFrrrrrjoint_trainingr2x x_lengthsrrr  start_idxsr)r)r)adv_loss text2mel_losslossrstatsweight optim_idxr)size unsqueezerrrr transposerrr synthesistrainingrupsample_factorrrno_gradrrrrrrritemrrrrdevice)rrrrrrrr batch_size reuse_cacherr feats_gen feats_gen_r speech_hat_speech_p_hatprrrrrrrrrs|                  z JointText2Wav._forward_generatorcKs^|d}|d}d} |jr|jdurHd} |jdd||||dd|\} } } t| dd||jd \} }|jd | }|jrG|j |}n|j\}}}}|jr[| s[| | ||f|_t |||jd j |j|jd j d }| |}| |}|||\}}||}t|||d } t|| |f|j\}} }| s|jsd|_|| |dd S)aPerform discriminator forward. Args: text (Tensor): Text index tensor (B, T_text). text_lengths (Tensor): Text length tensor (B,). feats (Tensor): Feature tensor (B, T_feats, aux_channels). feats_lengths (Tensor): Feature length tensor (B,). speech (Tensor): Speech waveform tensor (B, T_wav). speech_lengths (Tensor): Speech length tensor (B,). Returns: Dict[str, Any]: * loss (Tensor): Loss scalar tensor. * stats (Dict[str, float]): Statistics to be monitored. * weight (Tensor): Weight tensor to summarize losses. * optim_idx (int): Optimizer index (0 for G and 1 for D). rr@TNFrr r2r rr)discriminator_loss real_loss fake_lossrr)rrrrrrrrrrrrrrdetachrdictrrr r)rrrrrrrrr!r"rrr#r$rr%_r&r'r(r*r+rrrrrrs`          z#JointText2Wav._forward_discrminatorcKsp|jdjd d|i|}|jd|d}|jr0|j|ddd}|ddd}|j|d|S) a%Run inference. Args: text (Tensor): Input text index tensor (T_text,). Returns: Dict[str, Tensor]: * wav (Tensor): Generated waveform tensor (T_wav,). * feat_gan (Tensor): Generated feature tensor (T_text, C). rrrfeat_genrr@r2)wavNr) r inferencerrrrrsqueezer)rrr output_dictr0rrrr1_s zJointText2Wav.inference)T)__name__ __module__ __qualname____doc__intstrrrboolfloatrpropertyrrrTensorr rrr1 __classcell__rrrrr.@s      !"#$%&'()*+,-./0123456789:;<=>?@ABCDEJ(  R Sb cj k   ! "-./01    3 m _ r.)1r7typingrrr typeguardrespnet2.gan_tts.abs_gan_ttsrespnet2.gan_tts.hifiganrrrr r r espnet2.gan_tts.hifigan.lossr r rrespnet2.gan_tts.melganrrespnet2.gan_tts.melgan.pqmfr espnet2.gan_tts.parallel_waveganrrespnet2.gan_tts.style_melganrrespnet2.gan_tts.utilsrr espnet2.torch_utils.device_funcsrespnet2.tts.fastspeechrespnet2.tts.fastspeech2respnet2.tts.tacotron2respnet2.tts.transformerrrrrr.rrrrsJ