o iR@sdZddlZddlmZmZmZmZddlZddlm m Z ddl m Z ddlmZddlmZddlmZddlmZmZdd lmZdd lmZmZmZdd lmZdd lm Z Gd ddeZ!dS)z'Tacotron 2 related modules for ESPnet2.N)DictOptionalSequenceTuple)check_argument_types)force_gatherable)AbsTTS) StyleEncoder)GuidedAttentionLoss Tacotron2Loss) make_pad_mask) AttForward AttForwardTAAttLoc)Decoder)Encoderc_seZdZdZ                               dkdededededed ed!ed"ed#ed$ed%ed&ed'ed(ed)ed*ed+ed,ed-ed.ed/ed0ed1ed2ed3ed4eed5eed6eed7ed8ed9ed:ed;eded?ed@edAe dBe dCedDedEe dFedGedHe dIe f^fdJdK Z dldLe j dMe j dNe j dOe j dPee j dQee j dRee j dSedTe e j eee j fe j ffdUdVZdWe j dXe j dYe j dZe j dPe j dQe j dRe j dTe e j e j e j ffd[d\Z  ] ^    dmdLe j dNee j dPee j dQee j dRee j d_e d`e dae dbedceddedeedTeee j ffdfdgZdhe j dPe j dTe j fdidjZZS)n Tacotron2aTacotron2 module for end-to-end text-to-speech. This is a module of Spectrogram prediction network in Tacotron2 described in `Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions`_, which converts the sequence of characters into the sequence of Mel-filterbanks. .. _`Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions`: https://arxiv.org/abs/1712.05884 location TNFconcat rr@r"r#r#?皙?@L1+L2皙??idimodim embed_dimelayerseunits econv_layers econv_chans econv_filtsatypeadim aconv_chans aconv_filtscumulate_att_wdlayersdunits prenet_layers prenet_unitspostnet_layers postnet_chans postnet_filtsoutput_activationuse_batch_norm use_concate use_residualreduction_factorspkslangs spk_embed_dimspk_embed_integration_typeuse_gst gst_tokens gst_headsgst_conv_layersgst_conv_chans_listgst_conv_kernel_sizegst_conv_stridegst_gru_layers gst_gru_units dropout_rate zoneout_rate use_maskinguse_weighted_maskingbce_pos_weight loss_typeuse_guided_attn_lossguided_attn_loss_sigmaguided_attn_loss_lambdac03 stsJt||_||_|d|_| |_||_||_|-|_ |,|_ |dur,d|_ nt t |r8tt ||_ ntd|dd}0|0|_t||||||||||'|0d |_|jrgt|||| |!|"|#|$|%|&d |_d|_|dur}|dkr}||_tj|||_d|_|dur|dkr||_tj|||_d|_|dur|dkr||_||_|jdur|}1n"|jdkr||}1n|jd kr|}1tj|j||_nt|d | d krt |1|| | | }2n7| d krt!|1|| | | }2|jrt"#d d|_n| dkrt$|1|| | | |}2|jr t"#d d|_nt%dt&d$id|1d|d|2d|d|d|d|d|d|d|d|j d|jd|d|d|'d |(d!||_'t(|)|*|+d"|_)|j rbt*|.|/d#|_+dSdS)%a Initialize Tacotron2 module. Args: idim (int): Dimension of the inputs. odim: (int) Dimension of the outputs. embed_dim (int): Dimension of the token embedding. elayers (int): Number of encoder blstm layers. eunits (int): Number of encoder blstm units. econv_layers (int): Number of encoder conv layers. econv_filts (int): Number of encoder conv filter size. econv_chans (int): Number of encoder conv filter channels. dlayers (int): Number of decoder lstm layers. dunits (int): Number of decoder lstm units. prenet_layers (int): Number of prenet layers. prenet_units (int): Number of prenet units. postnet_layers (int): Number of postnet layers. postnet_filts (int): Number of postnet filter size. postnet_chans (int): Number of postnet filter channels. output_activation (str): Name of activation function for outputs. adim (int): Number of dimension of mlp in attention. aconv_chans (int): Number of attention conv filter channels. aconv_filts (int): Number of attention conv filter size. cumulate_att_w (bool): Whether to cumulate previous attention weight. use_batch_norm (bool): Whether to use batch normalization. use_concate (bool): Whether to concat enc outputs w/ dec lstm outputs. reduction_factor (int): Reduction factor. spks (Optional[int]): Number of speakers. If set to > 1, assume that the sids will be provided as the input and use sid embedding layer. langs (Optional[int]): Number of languages. If set to > 1, assume that the lids will be provided as the input and use sid embedding layer. spk_embed_dim (Optional[int]): Speaker embedding dimension. If set to > 0, assume that spembs will be provided as the input. spk_embed_integration_type (str): How to integrate speaker embedding. use_gst (str): Whether to use global style token. gst_tokens (int): Number of GST embeddings. gst_heads (int): Number of heads in GST multihead attention. gst_conv_layers (int): Number of conv layers in GST. gst_conv_chans_list: (Sequence[int]): List of the number of channels of conv layers in GST. gst_conv_kernel_size (int): Kernel size of conv layers in GST. gst_conv_stride (int): Stride size of conv layers in GST. gst_gru_layers (int): Number of GRU layers in GST. gst_gru_units (int): Number of GRU units in GST. dropout_rate (float): Dropout rate. zoneout_rate (float): Zoneout rate. use_masking (bool): Whether to mask padded part in loss calculation. use_weighted_masking (bool): Whether to apply weighted masking in loss calculation. bce_pos_weight (float): Weight of positive sample of stop token (only for use_masking=True). loss_type (str): Loss function type ("L1", "L2", or "L1+L2"). use_guided_attn_loss (bool): Whether to use guided attention loss. guided_attn_loss_sigma (float): Sigma in guided attention loss. guided_attn_loss_lambda (float): Lambda in guided attention loss. rNz*there is no such an activation function. ()r) r*r,r-r.r/r0r1r?rArP padding_idx) r*rH gst_token_dimrI conv_layersconv_chans_listconv_kernel_size conv_stride gru_layers gru_unitsraddz is not supported.rforwardzAcumulation of attention weights is disabled in forward attention.F forward_taz Support only location or forwardr*r+attr7r8r9r:r;r<r=output_activation_fnr6r?r@rPrQrB)rRrSrT)sigmaalpha),rsuper__init__r*r+eosr6rBrGrVrUrfhasattrFgetattr ValueErrorrZrencr gstrCtorchnn Embeddingsid_embrDlid_embrErFLinear projectionrr loggingwarningrNotImplementedErrorrdecr taco2_lossr attn_loss)3selfr*r+r,r-r.r/r0r1r2r3r4r5r6r7r8r9r:r;r<r=r>r?r@rArBrCrDrErFrGrHrIrJrKrLrMrNrOrPrQrRrSrTrUrVrWrXrZdec_idimre __class__riS/home/ubuntu/.local/lib/python3.10/site-packages/espnet2/tts/tacotron2/tacotron2.pyrk&s m                zTacotron2.__init__text text_lengthsfeats feats_lengthsspembssidslidsjoint_trainingreturnc  sn|ddd|f}|ddd|f}|d} t|ddgdj} t|D] \} } j| | | f<q,|d} |}|}t|d|j |j }t|ddgdd}j | | |||||d\}}}}j dkr| j swJd|fdd |D}t|}|ddd|f}|ddd|f}t|d|ddd}||||||\}}}jd kr|||}njd kr||}njd kr||}ntd jt|||d}jrj dkr|fdd |D}n|}|| |}||}|j|d|s2|j|dt||| f|j \}}}|||fS|||fS)aVCalculate forward propagation. Args: text (LongTensor): Batch of padded character ids (B, T_text). text_lengths (LongTensor): Batch of lengths of each input batch (B,). feats (Tensor): Batch of padded target features (B, T_feats, odim). feats_lengths (LongTensor): Batch of the lengths of each target (B,). spembs (Optional[Tensor]): Batch of speaker embeddings (B, spk_embed_dim). sids (Optional[Tensor]): Batch of speaker IDs (B, 1). lids (Optional[Tensor]): Batch of language IDs (B, 1). joint_training (bool): Whether to perform joint training with vocoder. Returns: Tensor: Loss scalar value. Dict: Statistics to be monitored. Tensor: Weight value if not joint training else model outputs. Nrrconstantr)xsilensysolensrrrz@Output length must be greater than or equal to reduction factor.csg|] }||jqSrirB.0olenrrir Ssz%Tacotron2.forward..r'L1L2zunknown --loss-type )l1_lossmse_lossbce_losscsg|]}|jqSrirrrrirrss)r)loss)maxsizernpadrZ enumeraterlr todevicedtype_forwardrBgeallnewrsscatter unsqueezer~rUrpdictitemrVrupdater)rrrrrrrrr batch_sizerilrrrlabels after_outs before_outslogitsatt_wsmax_outrrrrstatsolens_inrweightrirrrcs            zTacotron2.forwardrrrrc Cs|||\}} |jr||} || d}|jdur+||d} || d}|jdur?||d} || d}|j durJ| ||}| || |S)Nr) rqrGrrrrCrvviewrDrwrE_integrate_with_spk_embedr}) rrrrrrrrhshlens style_embssid_embslid_embsririrrs      zTacotron2._forward$@ threshold minlenratio maxlenratiouse_att_constraintbackward_windowforward_windowuse_teacher_forcingc  Cs|} |}|}t| ddgd|j} | ra|dusJd| d|d}}|dur,dn|d}| |dg}||dg}|j|||||||d\}}}}t|d|ddS|j | }|j rv| |d}||}|j dur||d}||}|jdur||d}||}|jdur|d|d}}|||d}|jj ||||| | | d \}}}t|||d S) aGenerate the sequence of features given the sequences of characters. Args: text (LongTensor): Input sequence of characters (T_text,). feats (Optional[Tensor]): Feature sequence to extract style (N, idim). spembs (Optional[Tensor]): Speaker embedding (spk_embed_dim,). sids (Optional[Tensor]): Speaker ID (1,). lids (Optional[Tensor]): Language ID (1,). threshold (float): Threshold in inference. minlenratio (float): Minimum length ratio in inference. maxlenratio (float): Maximum length ratio in inference. use_att_constraint (bool): Whether to apply attention constraint. backward_window (int): Backward window in attention constraint. forward_window (int): Forward window in attention constraint. use_teacher_forcing (bool): Whether to use teacher forcing. Returns: Dict[str, Tensor]: Output dict including the following items: * feat_gen (Tensor): Output sequence of features (T_feats, odim). * prob (Tensor): Output sequence of stop probabilities (T_feats,). * att_w (Tensor): Attention weights (T_feats, T). rrrNz,feats must be provided with teacher forcing.r)feat_genatt_wr)rrrrrr)rprobr)rnrrlr new_tensorrlongrrrq inferencerGrrrCrvrrDrwrErr})rrrrrrrrrrrrrxyspembrrrrouts_rh style_embrvrwroutrrririrrsV&      zTacotron2.inferencercCst|jdkr|t|}||d}|S|jdkr6t|dd|dd}tj||gdd}|St d)aIntegrate speaker embedding with hidden states. Args: hs (Tensor): Batch of hidden state sequences (B, Tmax, eunits). spembs (Tensor): Batch of speaker embeddings (B, spk_embed_dim). Returns: Tensor: Batch of integrated hidden state sequences (B, Tmax, eunits) if integration_type is "add" else (B, Tmax, eunits + spk_embed_dim). rbrrr)dimzsupport only add or concat.) rFryrn normalizerexpandrrscatr|)rrrririrrs  z#Tacotron2._integrate_with_spk_embed)-rrrrrrrrrrTrrrrrrrNTTFrNNNrFrrr r!rrrr#r$r%TFr&r'Tr(r))NNNF) NNNNr$rrFrrF)__name__ __module__ __qualname____doc__intstrboolrrfloatrkrsTensorrrrcrrr __classcell__ririrrrs      !"#$%&'()+,-./0123u  o        [r)"rrztypingrrrrrstorch.nn.functionalrt functionalrn typeguardr espnet2.torch_utils.device_funcsrespnet2.tts.abs_ttsrespnet2.tts.gst.style_encoderr -espnet.nets.pytorch_backend.e2e_tts_tacotron2r r &espnet.nets.pytorch_backend.nets_utilsr *espnet.nets.pytorch_backend.rnn.attentionsr rr-espnet.nets.pytorch_backend.tacotron2.decoderr-espnet.nets.pytorch_backend.tacotron2.encoderrrriririrs