o i:@sdZddlZddlmZmZmZmZmZmZddl Z ddl Z ddl m mZddlmZddlmZmZmZddlmZddlmZddlmZdd lmZdd lmZdd l m!Z"dd l#m$Z$dd l%m&Z&m'Z'ddl(m)Z)m*Z*dd l+m!Z,Gddde j j-Z.dS)Generator module in JETS.N)AnyDictListOptionalSequenceTuple)HiFiGANGenerator)AlignmentModuleaverage_by_durationviterbi_decode)GaussianUpsampling)get_random_segments) initialize)VariancePredictor) StyleEncoder)Encoder)DurationPredictor)make_non_pad_mask make_pad_mask)PositionalEncodingScaledPositionalEncodingcseZdZdZddddddddddddd d dd d d d d d d d d d ddddd dddddd ddddddd ddddddd ddddd dddddddddddd d dddd dgd!gd"gd#gd$gd$gd$gddd%d&d idfRd'ed(ed)ed*ed+ed,ed-ed.ed/ed0ed1ed2ed3ed4ed5ed6ed7ed8ed9ed:ed;eded?ed@edAedBedCedDedEedFedGedHedIedJedKedLedMedNedOedPedQedRedSedTedUedVedWedXedYedZed[eed\eed]eed^ed_ed`edaedbedce eddedeedfedgedhediedjedkedledmednedoedpedqedre edse edte edue e edvedwedxedye ee fdzeffd{d| Z   dd}ejd~ejdejdejdejdejdejdejdeejdeejdeejdeejejejejejejejejejejf fddZ        dd}ejd~ejdeejdeejdeejdeejdeejdeejdeejdedeejejejffddZdejdejdejfddZdejdejfddZdhediedjefddZZS) JETSGeneratorriconv1dTF transformerg?legacyrel_pos rel_selfattnswishig? Nadd ) r*@r+r,r,xavier_uniformg?r+i)r/rr)r0rr)r%r# )rr% LeakyReLUnegative_slopeidimodimadimaheadselayerseunitsdlayersdunitspositionwise_layer_typepositionwise_conv_kernel_sizeuse_scaled_pos_encuse_batch_normencoder_normalize_beforedecoder_normalize_beforeencoder_concat_afterdecoder_concat_afterreduction_factor encoder_type decoder_typetransformer_enc_dropout_rate'transformer_enc_positional_dropout_rate!transformer_enc_attn_dropout_ratetransformer_dec_dropout_rate'transformer_dec_positional_dropout_rate!transformer_dec_attn_dropout_rateconformer_rel_pos_typeconformer_pos_enc_layer_typeconformer_self_attn_layer_typeconformer_activation_typeuse_macaron_style_in_conformeruse_cnn_in_conformer zero_triuconformer_enc_kernel_sizeconformer_dec_kernel_sizeduration_predictor_layersduration_predictor_chansduration_predictor_kernel_sizeduration_predictor_dropout_rateenergy_predictor_layersenergy_predictor_chansenergy_predictor_kernel_sizeenergy_predictor_dropoutenergy_embed_kernel_sizeenergy_embed_dropout#stop_gradient_from_energy_predictorpitch_predictor_layerspitch_predictor_chanspitch_predictor_kernel_sizepitch_predictor_dropoutpitch_embed_kernel_sizepitch_embed_dropout"stop_gradient_from_pitch_predictorspkslangs spk_embed_dimspk_embed_integration_typeuse_gst gst_tokens gst_headsgst_conv_layersgst_conv_chans_listgst_conv_kernel_sizegst_conv_stridegst_gru_layers gst_gru_units init_typeinit_enc_alphainit_dec_alpha use_maskinguse_weighted_masking segment_sizegenerator_out_channelsgenerator_channelsgenerator_global_channelsgenerator_kernel_sizegenerator_upsample_scalesgenerator_upsample_kernel_sizesgenerator_resblock_kernel_sizesgenerator_resblock_dilationsgenerator_use_additional_convsgenerator_biasgenerator_nonlinear_activation%generator_nonlinear_activation_paramsgenerator_use_weight_normcUWs\t|G|_tt|L|_||_||_||_ ||_ ||_ |4|_ |-|_ | |_|9|_d|_|jr3tnt}Ud||fvrn|dkrV|dkrJd}td|dkrUd}td n|d krg|dks`J|dksfJntd |tjj|||jd }V|d krt||||||V||||U| || | d|_nN|dkrtd-id|d|d|d|d|d|Vd|d|d|d| d|d| d| d|d|d|d|d |d!|!d"| |_nt|d#|jrt||:||;|<|=|>|?|@|Ad$ |_d%|_|5d%ur|5d&kr|5|_tj|5||_d%|_ |6d%ur |6d&kr |6|_ tj|6||_!d%|_"|7d%ur3|7dkr3|7|_"|8|_#|j"d%urT|j#d'krItj$|j"||_%n tj$||j"||_%t&||#|$|%|&d(|_'t(||.|/|0|1d(|_)tj*tjj+d&||2|2d&d)d*tj,|3|_-t(||'|(|)|*d(|_.tj*tjj+d&||+|+d&d)d*tj,|,|_/t0|||_1t2|_3|d krtd||||d%||||U||| | d|_4nL|dkr td-iddd|d|d|d|dd%d|d|d|d|d|d| d| d|d|d|d|d |d!|"|_4nt|d#t5||H|I|J|K|L|M|N|O|P|Q|R|S|Td+|_6|j7|B|C|Dd,d%S).aInitialize JETS generator module. Args: idim (int): Dimension of the inputs. odim (int): Dimension of the outputs. elayers (int): Number of encoder layers. eunits (int): Number of encoder hidden units. dlayers (int): Number of decoder layers. dunits (int): Number of decoder hidden units. use_scaled_pos_enc (bool): Whether to use trainable scaled pos encoding. use_batch_norm (bool): Whether to use batch normalization in encoder prenet. encoder_normalize_before (bool): Whether to apply layernorm layer before encoder block. decoder_normalize_before (bool): Whether to apply layernorm layer before decoder block. encoder_concat_after (bool): Whether to concatenate attention layer's input and output in encoder. decoder_concat_after (bool): Whether to concatenate attention layer's input and output in decoder. reduction_factor (int): Reduction factor. encoder_type (str): Encoder type ("transformer" or "conformer"). decoder_type (str): Decoder type ("transformer" or "conformer"). transformer_enc_dropout_rate (float): Dropout rate in encoder except attention and positional encoding. transformer_enc_positional_dropout_rate (float): Dropout rate after encoder positional encoding. transformer_enc_attn_dropout_rate (float): Dropout rate in encoder self-attention module. transformer_dec_dropout_rate (float): Dropout rate in decoder except attention & positional encoding. transformer_dec_positional_dropout_rate (float): Dropout rate after decoder positional encoding. transformer_dec_attn_dropout_rate (float): Dropout rate in decoder self-attention module. conformer_rel_pos_type (str): Relative pos encoding type in conformer. conformer_pos_enc_layer_type (str): Pos encoding layer type in conformer. conformer_self_attn_layer_type (str): Self-attention layer type in conformer conformer_activation_type (str): Activation function type in conformer. use_macaron_style_in_conformer: Whether to use macaron style FFN. use_cnn_in_conformer: Whether to use CNN in conformer. zero_triu: Whether to use zero triu in relative self-attention module. conformer_enc_kernel_size: Kernel size of encoder conformer. conformer_dec_kernel_size: Kernel size of decoder conformer. duration_predictor_layers (int): Number of duration predictor layers. duration_predictor_chans (int): Number of duration predictor channels. duration_predictor_kernel_size (int): Kernel size of duration predictor. duration_predictor_dropout_rate (float): Dropout rate in duration predictor. pitch_predictor_layers (int): Number of pitch predictor layers. pitch_predictor_chans (int): Number of pitch predictor channels. pitch_predictor_kernel_size (int): Kernel size of pitch predictor. pitch_predictor_dropout_rate (float): Dropout rate in pitch predictor. pitch_embed_kernel_size (float): Kernel size of pitch embedding. pitch_embed_dropout_rate (float): Dropout rate for pitch embedding. stop_gradient_from_pitch_predictor: Whether to stop gradient from pitch predictor to encoder. energy_predictor_layers (int): Number of energy predictor layers. energy_predictor_chans (int): Number of energy predictor channels. energy_predictor_kernel_size (int): Kernel size of energy predictor. energy_predictor_dropout_rate (float): Dropout rate in energy predictor. energy_embed_kernel_size (float): Kernel size of energy embedding. energy_embed_dropout_rate (float): Dropout rate for energy embedding. stop_gradient_from_energy_predictor: Whether to stop gradient from energy predictor to encoder. spks (Optional[int]): Number of speakers. If set to > 1, assume that the sids will be provided as the input and use sid embedding layer. langs (Optional[int]): Number of languages. If set to > 1, assume that the lids will be provided as the input and use sid embedding layer. spk_embed_dim (Optional[int]): Speaker embedding dimension. If set to > 0, assume that spembs will be provided as the input. spk_embed_integration_type: How to integrate speaker embedding. use_gst (str): Whether to use global style token. gst_tokens (int): The number of GST embeddings. gst_heads (int): The number of heads in GST multihead attention. gst_conv_layers (int): The number of conv layers in GST. gst_conv_chans_list: (Sequence[int]): List of the number of channels of conv layers in GST. gst_conv_kernel_size (int): Kernel size of conv layers in GST. gst_conv_stride (int): Stride size of conv layers in GST. gst_gru_layers (int): The number of GRU layers in GST. gst_gru_units (int): The number of GRU units in GST. init_type (str): How to initialize transformer parameters. init_enc_alpha (float): Initial value of alpha in scaled pos encoding of the encoder. init_dec_alpha (float): Initial value of alpha in scaled pos encoding of the decoder. use_masking (bool): Whether to apply masking for padded part in loss calculation. use_weighted_masking (bool): Whether to apply weighted masking in loss calculation. segment_size (int): Segment size for random windowed discriminator generator_out_channels (int): Number of output channels. generator_channels (int): Number of hidden representation channels. generator_global_channels (int): Number of global conditioning channels. generator_kernel_size (int): Kernel size of initial and final conv layer. generator_upsample_scales (List[int]): List of upsampling scales. generator_upsample_kernel_sizes (List[int]): List of kernel sizes for upsample layers. generator_resblock_kernel_sizes (List[int]): List of kernel sizes for residual blocks. generator_resblock_dilations (List[List[int]]): List of list of dilations for residual blocks. generator_use_additional_convs (bool): Whether to use additional conv layers in residual blocks. generator_bias (bool): Whether to add bias parameter in convolution layers. generator_nonlinear_activation (str): Activation function module name. generator_nonlinear_activation_params (Dict[str, Any]): Hyperparameters for activation function. generator_use_weight_norm (bool): Whether to use weight norm. If set to true, it will be applied to all of the conv layers. r conformerrr legacy_rel_poszFallback to conformer_pos_enc_layer_type = 'legacy_rel_pos' due to the compatibility. If you want to use the new one, please use conformer_pos_enc_layer_type = 'latest'.r!legacy_rel_selfattnzFallback to conformer_self_attn_layer_type = 'legacy_rel_selfattn' due to the compatibility. If you want to use the new one, please use conformer_pos_enc_layer_type = 'latest'.latestzUnknown rel_pos_type: )num_embeddings embedding_dim padding_idxr)r5 attention_dimattention_heads linear_units num_blocks input_layer dropout_ratepositional_dropout_rateattention_dropout_rate pos_enc_classnormalize_before concat_afterr=r>r5rrrrrrrrrrr=r> macaron_stylepos_enc_layer_typeselfattention_layer_typeactivation_typeuse_cnn_modulecnn_module_kernelrTz is not supported.) r5rn gst_token_dimro conv_layersconv_chans_listconv_kernel_size conv_stride gru_layers gru_unitsNrr')r5n_layersn_chans kernel_sizerr) in_channels out_channelsrpadding)rrchannelsglobal_channelsrupsample_scalesupsample_kernel_sizesresblock_kernel_sizesresblock_dilationsuse_additional_convsbiasnonlinear_activationnonlinear_activation_paramsuse_weight_norm)rvrwrx)8super__init__r{intnpprodupsample_factorr5r6rErFrGrhrar?rmrrrloggingwarning ValueErrortorchnn EmbeddingTransformerEncoderencoderConformerEncoderrgstrisid_embrjlid_embrkrlLinear projectionrduration_predictorrpitch_predictor SequentialConv1dDropout pitch_embedenergy_predictor energy_embedr alignment_moduler length_regulatordecoderr generator_reset_parameters)Wselfr5r6r7r8r9r:r;r<r=r>r?r@rArBrCrDrErFrGrHrIrJrKrLrMrNrOrPrQrRrSrTrUrVrWrXrYrZr[r\r]r^r_r`rarbrcrdrerfrgrhrirjrkrlrmrnrorprqrrrsrtrurvrwrxryrzr{r|r}r~rrrrrrrrrrrencoder_input_layer __class__rR/home/ubuntu/.local/lib/python3.10/site-packages/espnet2/gan_tts/jets/generator.pyr's R                          zJETSGenerator.__init__text text_lengthsfeats feats_lengthspitch pitch_lengthsenergyenergy_lengthssidsspembslidsreturnc " Cs|ddd|f}|ddd|f}|ddd|f}|ddd|f}||} ||| \} }|jrL||}| |d} |jdur`|| d}| |d} |j durt| | d}| |d} |j dur| | | } t || j}|| ||}t|||\}}t||d||d}t||d||d}|jr|| |d}n || |d}|jr|| |d}n || |d}|| |}||dddd}||dddd}| ||} t|| j}t||j}|| |||} ||}|| |\}}t |dd||j!\}} |"|}!|!||| ||||||f S)aCalculate forward propagation. Args: text (Tensor): Text index tensor (B, T_text). text_lengths (Tensor): Text length tensor (B,). feats (Tensor): Feature tensor (B, T_feats, aux_channels). feats_lengths (Tensor): Feature length tensor (B,). pitch (Tensor): Batch of padded token-averaged pitch (B, T_text, 1). pitch_lengths (LongTensor): Batch of pitch lengths (B, T_text). energy (Tensor): Batch of padded token-averaged energy (B, T_text, 1). energy_lengths (LongTensor): Batch of energy lengths (B, T_text). sids (Optional[Tensor]): Speaker index tensor (B,) or (B, 1). spembs (Optional[Tensor]): Speaker embedding tensor (B, spk_embed_dim). lids (Optional[Tensor]): Language index tensor (B,) or (B, 1). Returns: Tensor: Waveform tensor (B, 1, segment_size * upsample_factor). Tensor: Binarization loss (). Tensor: Log probability attention matrix (B, T_feats, T_text). Tensor: Segments start index tensor (B,). Tensor: predicted duration (B, T_text). Tensor: ground-truth duration obtained from an alignment module (B, T_text). Tensor: predicted pitch (B, T_text,1). Tensor: ground-truth averaged pitch (B, T_text, 1). Tensor: predicted energy (B, T_text, 1). Tensor: ground-truth averaged energy (B, T_text, 1). Nrr.r)#max _source_maskrrmr unsqueezerirviewrjrrk_integrate_with_spk_embedrtodevicerr r squeezerhrdetachrarrr transposerrrrrr{r)"rrrrrrrrrrrrx_maskshs_ style_embssid_embslid_embsh_masks log_p_attndsbin_losspsesp_outse_outsd_outsp_embse_embsd_maskszs z_segments z_start_idxswavrrrforwards|5           zJETSGenerator.forwarduse_teacher_forcingc Cs||} ||| \} } |jr||}| |d} |jdur0||d}| |d} |jdurD| | d}| |d} |j durO| | |} t | | j}| r|| ||}t|||\}} t||d||d}t||d||d}n|| |d}|| |d}|j| |}||dddd}||dddd}| ||} |durt| | j}nd}t| |j}|| |||} |dur||}nd}|| |\}} ||dd}|d|fS)aMRun inference. Args: text (Tensor): Input text index tensor (B, T_text,). text_lengths (Tensor): Text length tensor (B,). feats (Tensor): Feature tensor (B, T_feats, aux_channels). feats_lengths (Tensor): Feature length tensor (B,). pitch (Tensor): Pitch tensor (B, T_feats, 1) energy (Tensor): Energy tensor (B, T_feats, 1) sids (Optional[Tensor]): Speaker index tensor (B,) or (B, 1). spembs (Optional[Tensor]): Speaker embedding tensor (B, spk_embed_dim). lids (Optional[Tensor]): Language index tensor (B,) or (B, 1). use_teacher_forcing (bool): Whether to use teacher forcing. Returns: Tensor: Generated waveform tensor (B, T_wav). Tensor: Duration tensor (B, T_text). rNr.r)rrrmrrrirrrjrrkrrrrrr r rrrr inferencerrrrrrr)rrrrrrrrrrr rrrrrrrrrrrrrrrr rrrr|sX !       zJETSGenerator.inferencercCsz|jdkr|t|}||d}|S|jdkr9t|dd|dd}|tj||gdd}|St d)aEIntegrate speaker embedding with hidden states. Args: hs (Tensor): Batch of hidden state sequences (B, T_text, adim). spembs (Tensor): Batch of speaker embeddings (B, spk_embed_dim). Returns: Tensor: Batch of integrated hidden state sequences (B, T_text, adim). r'rconcatr.)dimzsupport only add or concat.) rlrF normalizerexpandsizercatNotImplementedError)rrrrrrrs  z'JETSGenerator._integrate_with_spk_embedilenscCs"t|t|j}|dS)aMake masks for self-attention. Args: ilens (LongTensor): Batch of lengths (B,). Returns: Tensor: Mask tensor for self-attention. dtype=torch.uint8 in PyTorch 1.2- dtype=torch.bool in PyTorch 1.2+ (including 1.2) Examples: >>> ilens = [5, 3] >>> self._source_mask(ilens) tensor([[[1, 1, 1, 1, 1], [1, 1, 1, 0, 0]]], dtype=torch.uint8) )rrnext parametersrr)rrrrrrrs zJETSGenerator._source_maskcCsj|dkr t|||jdkr|jrt||jjdj_|j dkr1|jr3t||j jdj_dSdSdS)Npytorchrr.) rrFr?rtensorrembedalphadatarGr)rrvrwrxrrrr s zJETSGenerator._reset_parameters)NNN)NNNNNNNF)__name__ __module__ __qualname____doc__rstrboolfloatrrrrrrrTensorrr rrrr __classcell__rrrrr$s      !"#$&'()+,-./013456789;<=>?@ABCDEFGIJKLMNPQRSTUV W\]^ _`Y          ^ r)/r#rtypingrrrrrrnumpyrrtorch.nn.functionalr functionalrespnet2.gan_tts.hifiganr espnet2.gan_tts.jets.alignmentsr r r %espnet2.gan_tts.jets.length_regulatorr espnet2.gan_tts.utilsrespnet2.torch_utils.initializer*espnet2.tts.fastspeech2.variance_predictorrespnet2.tts.gst.style_encoderr-espnet.nets.pytorch_backend.conformer.encoderrr9espnet.nets.pytorch_backend.fastspeech.duration_predictorr&espnet.nets.pytorch_backend.nets_utilsrr1espnet.nets.pytorch_backend.transformer.embeddingrr/espnet.nets.pytorch_backend.transformer.encoderrModulerrrrrs&