o ¡¿¯i°_ã@sŽdZddlZddlZddlmmZddlmZdd„Z Gdd„dejj ƒZ Gdd „d ejj ƒZ Gd d „d ejj ƒZ Gd d „d ejj ƒZdS)z"Tacotron2 decoder related modules.éN)Ú AttForwardTAcCs2t|tjjƒrtjj |jtjj d¡¡dSdS)zInitialize decoder parameters.ÚtanhN)Ú isinstanceÚtorchÚnnÚConv1dÚinitÚxavier_uniform_ÚweightÚcalculate_gain)Úm©r úa/home/ubuntu/.local/lib/python3.10/site-packages/espnet/nets/pytorch_backend/tacotron2/decoder.pyÚ decoder_inits ÿrcs2eZdZdZd ‡fdd„ Zdd„Zdd„Z‡ZS) Ú ZoneOutCellaZoneOut Cell module. This is a module of zoneout described in `Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations`_. This code is modified from `eladhoffer/seq2seq.pytorch`_. Examples: >>> lstm = torch.nn.LSTMCell(16, 32) >>> lstm = ZoneOutCell(lstm, 0.5) .. _`Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations`: https://arxiv.org/abs/1606.01305 .. _`eladhoffer/seq2seq.pytorch`: https://github.com/eladhoffer/seq2seq.pytorch çš™™™™™¹?cs>tt|ƒ ¡||_|j|_||_|dks|dkrtdƒ‚dS)aInitialize zone out cell module. Args: cell (torch.nn.Module): Pytorch recurrent cell module e.g. `torch.nn.Module.LSTMCell`. zoneout_rate (float, optional): Probability of zoneout from 0.0 to 1.0. gð?çz9zoneout probability must be in the range from 0.0 to 1.0.N)ÚsuperrÚ__init__ÚcellÚ hidden_sizeÚ zoneout_rateÚ ValueError)Úselfrr©Ú __class__r rr)s ÿÿzZoneOutCell.__init__cCs | ||¡}| |||j¡}|S)aÞCalculate forward propagation. Args: inputs (Tensor): Batch of input tensor (B, input_size). hidden (tuple): - Tensor: Batch of initial hidden states (B, hidden_size). - Tensor: Batch of initial cell states (B, hidden_size). Returns: tuple: - Tensor: Batch of next hidden states (B, hidden_size). - Tensor: Batch of next cell states (B, hidden_size). )rÚ_zoneoutr)rÚinputsÚhiddenÚ next_hiddenr r rÚforward;s zZoneOutCell.forwardcsŒtˆtƒr%tˆƒ}tˆtƒstˆg|ƒ‰t‡‡‡‡fdd„t|ƒDƒƒSˆjr<ˆjˆ ¡Ž ˆ¡}|ˆd|ˆSˆˆdˆˆS)Ncs&g|]}ˆ ˆ|ˆ|ˆ|¡‘qSr )r)Ú.0Úi©ÚhÚnext_hÚprobrr rÚ Us&z(ZoneOutCell._zoneout..é)rÚtupleÚlenÚrangeÚtrainingÚnewÚsizeÚ bernoulli_)rr$r%r&Únum_hÚmaskr r#rrNs  ÿzZoneOutCell._zoneout)r)Ú__name__Ú __module__Ú __qualname__Ú__doc__rr rÚ __classcell__r r rrrs rcs*eZdZdZd ‡fdd„ Zdd„Z‡ZS) ÚPrenetaÌPrenet module for decoder of Spectrogram prediction network. This is a module of Prenet in the decoder of Spectrogram prediction network, which described in `Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions`_. The Prenet preforms nonlinear conversion of inputs before input to auto-regressive lstm, which helps to learn diagonal attentions. Note: This module alway applies dropout even in evaluation. See the detail in `Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions`_. .. _`Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions`: https://arxiv.org/abs/1712.05884 ééçà?c sptt|ƒ ¡||_tj ¡|_tj   |¡D]}|dkr|n|}|jtj  tj  ||¡tj  ¡¡g7_qdS)aInitialize prenet module. Args: idim (int): Dimension of the inputs. odim (int): Dimension of the outputs. n_layers (int, optional): The number of prenet layers. n_units (int, optional): The number of prenet units. rN)rr7rÚ dropout_raterrÚ ModuleListÚprenetÚsixÚmovesr+Ú SequentialÚLinearÚReLU)rÚidimÚn_layersÚn_unitsr;ÚlayerÚn_inputsrr rrss   ÿþzPrenet.__init__cCs4tj t|jƒ¡D]}t |j||ƒ|j¡}q |S)zÄCalculate forward propagation. Args: x (Tensor): Batch of input tensors (B, ..., idim). Returns: Tensor: Batch of output tensors (B, ..., odim). )r>r?r+r*r=ÚFÚdropoutr;)rÚxr"r r rr †s zPrenet.forward)r8r9r:©r2r3r4r5rr r6r r rrr7_sr7cs4eZdZdZ     d ‡fdd„ Zdd „Z‡ZS) ÚPostnetaþPostnet module for Spectrogram prediction network. This is a module of Postnet in Spectrogram prediction network, which described in `Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions`_. The Postnet predicts refines the predicted Mel-filterbank of the decoder, which helps to compensate the detail structure of spectrogram. .. _`Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions`: https://arxiv.org/abs/1712.05884 éér:Tc s’tt|ƒ ¡tj ¡|_tj  |d¡D]b}|dkr|n|} ||dkr'|n|} |rT|jtj  tjj | | |d|ddddtj  | ¡tj  ¡tj |¡¡g7_q|jtj  tjj | | |d|ddddtj  ¡tj |¡¡g7_q|dkr~|n|} |r¨|jtj  tjj | ||d|ddddtj  |¡tj |¡¡g7_dS|jtj  tjj | ||d|ddddtj |¡¡g7_dS)aãInitialize postnet module. Args: idim (int): Dimension of the inputs. odim (int): Dimension of the outputs. n_layers (int, optional): The number of layers. n_filts (int, optional): The number of filter size. n_units (int, optional): The number of filter channels. use_batch_norm (bool, optional): Whether to use batch normalization.. dropout_rate (float, optional): Dropout rate.. r(rr8F)ÚstrideÚpaddingÚbiasN)rrLrrrr<Úpostnetr>r?r+r@rÚ BatchNorm1dÚTanhÚDropout) rrCÚodimrDÚn_chansÚn_filtsr;Úuse_batch_normrFÚichansÚochansrr rr¥s€  ú  õ ÿ ú ö ÿ ú  ö ÿ ú ÷ ÿzPostnet.__init__cCs*tj t|jƒ¡D] }|j||ƒ}q |S)zæCalculate forward propagation. Args: xs (Tensor): Batch of the sequences of padded input tensors (B, idim, Tmax). Returns: Tensor: Batch of padded output tensor. (B, odim, Tmax). )r>r?r+r*rR)rÚxsr"r r rr þs zPostnet.forward)rMrNrMr:TrKr r rrrL–søYrLcsleZdZdZ            d‡fd d „ Zdd„Zdd„Z     ddd„Zdd„Z‡Z S)ÚDecoderaÔDecoder module of Spectrogram prediction network. This is a module of decoder of Spectrogram prediction network in Tacotron2, which described in `Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions`_. The decoder generates the sequence of features from the sequence of the hidden states. .. _`Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions`: https://arxiv.org/abs/1712.05884 r8ér9rMrNNTr:rr(c sTtt|ƒ ¡||_||_||_| |_| |_||_||_ t |jt ƒr&d|_ nd|_ |dkr/|n|}t j ¡|_tj |¡D]$}|dkrG||n|}t j ||¡}|dkrYt||ƒ}|j|g7_q=|dkrpt||||d|_nd|_|dkr„t|||| | | |d|_nd|_|r||n|}t jj|||dd|_t j ||¡|_| t¡dS) aDInitialize Tacotron2 decoder module. Args: idim (int): Dimension of the inputs. odim (int): Dimension of the outputs. att (torch.nn.Module): Instance of attention class. dlayers (int, optional): The number of decoder lstm layers. dunits (int, optional): The number of decoder lstm units. prenet_layers (int, optional): The number of prenet layers. prenet_units (int, optional): The number of prenet units. postnet_layers (int, optional): The number of postnet layers. postnet_filts (int, optional): The number of postnet filter size. postnet_chans (int, optional): The number of postnet filter channels. output_activation_fn (torch.nn.Module, optional): Activation function for outputs. cumulate_att_w (bool, optional): Whether to cumulate previous attention weight. use_batch_norm (bool, optional): Whether to use batch normalization. use_concate (bool, optional): Whether to concatenate encoder embedding with decoder lstm outputs. dropout_rate (float, optional): Dropout rate. zoneout_rate (float, optional): Zoneout rate. reduction_factor (int, optional): Reduction factor. TFrr)rCrDrEr;N)rCrVrDrWrXrYr;)rQ)rr]rrCrVÚattÚoutput_activation_fnÚcumulate_att_wÚ use_concateÚreduction_factorrrÚuse_att_extra_inputsrrr<Úlstmr>r?r+ÚLSTMCellrr7r=rLrRrAÚfeat_outÚprob_outÚapplyr)rrCrVr_ÚdlayersÚdunitsÚ prenet_layersÚ prenet_unitsÚpostnet_layersÚ postnet_chansÚ postnet_filtsr`rarYrbr;rrcrFÚiunitsrerr rrsT-    ü ù zDecoder.__init__cCs| | d¡|jdj¡}|S)Nr)Ú new_zerosr.rer)rÚhsÚinit_hsr r rÚ _zero_state„szDecoder._zero_statecCsÚ|jdkr|dd…|jdd|j…f}ttt|ƒƒ}| |¡g}| |¡g}tj dt|j ƒ¡D]}|| |¡g7}|| |¡g7}q1|  |  d¡|j ¡}d}|j  ¡ggg} } } | dd¡D]¯} |jrv|  |||d||¡\} }n |  |||d|¡\} }|jdurŒ| |¡n|}tj| |gdd}|j d||d|dfƒ\|d<|d<tj dt|j ƒ¡D]}|j |||d||||fƒ\||<||<qµ|jrÞtj|d| gddn|d}| | |¡ |  d¡|j d¡g7} | | |¡g7} | |g7} | }|jr|dur||}qc|}qctj| dd} tj| dd}tj| dd} |jdkr9| |  d¡|j d¡}|jdurG|| |¡}n|}| dd¡}| dd¡}| } |jdurg| |¡}| |¡}||| | fS)aÅCalculate forward propagation. Args: hs (Tensor): Batch of the sequences of padded hidden states (B, Tmax, idim). hlens (LongTensor): Batch of lengths of each input batch (B,). ys (Tensor): Batch of the sequences of padded target features (B, Lmax, odim). Returns: Tensor: Batch of output tensors after postnet (B, Lmax, odim). Tensor: Batch of output tensors before postnet (B, Lmax, odim). Tensor: Batch of logits of stop prediction (B, Lmax). Tensor: Batch of attention weights (B, Lmax, Tmax). Note: This computation is performed in teacher-forcing manner. r(Nr©Údiméÿÿÿÿr8)rcÚlistÚmapÚintrur>r?r+r*rerrr.rVr_ÚresetÚ transposerdr=rÚcatrbrgÚviewrhraÚstackrRr`)rrsÚhlensÚysÚc_listÚz_listÚ_Úprev_outÚ prev_att_wÚoutsÚlogitsÚatt_wsÚyÚatt_cÚatt_wÚ prenet_outr\r"ÚzcsÚ before_outsÚ after_outsr r rr ˆsf    (ÿÿý"   ÿ       zDecoder.forwardrç$@Fc CsXt| ¡ƒdks J‚| d¡}| d¡g} t| d¡|ƒ} t| d¡|ƒ} | |¡g} | |¡g} tj dt|jƒ¡D]}| | |¡g7} | | |¡g7} q=|  d|j ¡}d}|j   ¡|rcd}nd}d}ggg}}} ||j 7}|jrŠ|j || | d|||||d\}}n|j || | d||||d\}}||g7}|jdur©| |¡n|}tj||gdd}|jd|| d| dfƒ\| d<| d<tj dt|jƒ¡D]}|j|| |d| || |fƒ\| |<| |<qÒ|jrûtj| d|gddn| d}|| |¡ d|j d¡g7}|t | |¡¡dg7}|jdur1| |ddd…dd…df¡}n |ddd…dd…df}|jrL|durL||}n|}|rWt| ¡ƒ}tt|d|kƒƒdksi|| kr›|| kroqotj|dd}|jdurƒ|| |¡}| dd¡ d¡}tj|dd}tj|dd}nqp|jdur§| |¡}|||fS) aØGenerate the sequence of features given the sequences of characters. Args: h (Tensor): Input sequence of encoder hidden states (T, C). threshold (float, optional): Threshold to stop generation. minlenratio (float, optional): Minimum length ratio. If set to 1.0 and the length of input is 10, the minimum length of outputs will be 10 * 1 = 10. minlenratio (float, optional): Minimum length ratio. If set to 10 and the length of input is 10, the maximum length of outputs will be 10 * 10 = 100. use_att_constraint (bool): Whether to apply attention constraint introduced in `Deep Voice 3`_. backward_window (int): Backward window size in attention constraint. forward_window (int): Forward window size in attention constraint. Returns: Tensor: Output sequence of features (L, odim). Tensor: Output sequence of stop probabilities (L,). Tensor: Attention weights (L, T). Note: This computation is performed in auto-regressive manner. .. _`Deep Voice 3`: https://arxiv.org/abs/1710.07654 r8rr(NT)Úlast_attended_idxÚbackward_windowÚforward_windowrvrx)r*r.Ú unsqueezer{rur>r?r+rerrrVr_r|rcrdr=rr~rbrgrÚsigmoidrhr`raÚargmaxÚsumrRr}Úsqueeze)rr$Ú thresholdÚ minlenratioÚ maxlenratioÚuse_att_constraintr”r•rsÚilensÚmaxlenÚminlenrƒr„r…r†r‡r“ÚidxrˆrŠÚprobsrŒrrŽr\r"rr r rÚ inferenceâs˜&       ø  ù (ÿÿý "  $  À B  zDecoder.inferencecCsÒ|jdkr|dd…|jdd|j…f}ttt|ƒƒ}| |¡g}| |¡g}tj dt|j ƒ¡D]}|| |¡g7}|| |¡g7}q1|  |  d¡|j ¡}d}|j  ¡g} | dd¡D]‚} |jrp|  |||d||¡\} } n |  |||d|¡\} } | | g7} |jdur‹| |¡n|} tj| | gdd}|j d||d|dfƒ\|d<|d<tj dt|j ƒ¡D]}|j |||d||||fƒ\||<||<q´| }|jrÝ|durÝ|| }q]| }q]tj| dd} | S)aüCalculate all of the attention weights. Args: hs (Tensor): Batch of the sequences of padded hidden states (B, Tmax, idim). hlens (LongTensor): Batch of lengths of each input batch (B,). ys (Tensor): Batch of the sequences of padded target features (B, Lmax, odim). Returns: numpy.ndarray: Batch of attention weights (B, Lmax, Tmax). Note: This computation is performed in teacher-forcing manner. r(Nrrv)rcryrzr{rur>r?r+r*rerrr.rVr_r|r}rdr=rr~rar€)rrsrr‚rƒr„r…r†r‡rŠr‹rŒrrŽr\r"r r rÚcalculate_all_attentionsjs<     (ÿ z Decoder.calculate_all_attentions)r8r^r8r9rMrNrMNTTTr:rr()r:rr’FNN) r2r3r4r5rrur r¤r¥r6r r rrr] s8îi] ø r])r5r>rÚtorch.nn.functionalrÚ functionalrHÚ*espnet.nets.pytorch_backend.rnn.attentionsrrÚModulerr7rLr]r r r rÚs I7w