o iu'@sdZddlmZddlZddlmZddlmZGdddej j Z Gdd d ej j Z Gd d d ej j Z Gd d d eZdS)zStyle encoder of GST-Tacotron.)SequenceN)check_argument_types)MultiHeadedAttentioncszeZdZdZ        dd ed ededededeededededeffdd ZdejdejfddZ Z S) StyleEncoderaStyle encoder. This module is style encoder introduced in `Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis`. .. _`Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis`: https://arxiv.org/abs/1803.09017 Args: idim (int, optional): Dimension of the input mel-spectrogram. gst_tokens (int, optional): The number of GST embeddings. gst_token_dim (int, optional): Dimension of each GST embedding. gst_heads (int, optional): The number of heads in GST multihead attention. conv_layers (int, optional): The number of conv layers in the reference encoder. conv_chans_list: (Sequence[int], optional): List of the number of channels of conv layers in the referece encoder. conv_kernel_size (int, optional): Kernel size of conv layers in the reference encoder. conv_stride (int, optional): Stride size of conv layers in the reference encoder. gru_layers (int, optional): The number of GRU layers in the reference encoder. gru_units (int, optional): The number of GRU units in the reference encoder. Todo: * Support manual weight specification in inference. P  r @r rridim gst_tokens gst_token_dim gst_heads conv_layersconv_chans_listconv_kernel_size conv_stride gru_layers gru_unitsc sFtsJtt|t|||||| | d|_t| |||d|_dS)z&Initilize global style encoder module.)rrrrrrr) ref_embed_dimrrrN)rsuperr__init__ReferenceEncoderref_encStyleTokenLayerstl) selfrrrrrrrrrr __class__Q/home/ubuntu/.local/lib/python3.10/site-packages/espnet2/tts/gst/style_encoder.pyr-s"   zStyleEncoder.__init__speechreturncCs||}||}|S)zCalculate forward propagation. Args: speech (Tensor): Batch of padded target features (B, Lmax, odim). Returns: Tensor: Style token embeddings (B, token_dim). )r r")r#r(ref_embs style_embsr&r&r'forwardNs zStyleEncoder.forward) rrrr r r rrrr __name__ __module__ __qualname____doc__intrrtorchTensorr, __classcell__r&r&r$r'rsD   !rc sdeZdZdZ       dd ed eed ed ed edef fdd ZdejdejfddZ Z S)raReference encoder module. This module is reference encoder introduced in `Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis`. .. _`Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis`: https://arxiv.org/abs/1803.09017 Args: idim (int, optional): Dimension of the input mel-spectrogram. conv_layers (int, optional): The number of conv layers in the reference encoder. conv_chans_list: (Sequence[int], optional): List of the number of channels of conv layers in the referece encoder. conv_kernel_size (int, optional): Kernel size of conv layers in the reference encoder. conv_stride (int, optional): Stride size of conv layers in the reference encoder. gru_layers (int, optional): The number of GRU layers in the reference encoder. gru_units (int, optional): The number of GRU units in the reference encoder. rr r rrrrrrrrrrc s$tsJtt||ddksJdt||ks Jdg}|dd} t|D],} | dkr4dn|| d} || } |tjj| | ||| ddtj | tjj dd g7}q,tjj ||_ ||_ ||_||_| |_|} t|D]} | |d| |d} qr| | 9} tjj| ||dd |_d S) z#Initilize reference encoder module.rrzkernel size must be odd.zGthe number of conv layers and length of channels list must be the same.rF) kernel_sizestridepaddingbiasT)inplace) batch_firstN)rrrrlenranger3nnConv2d BatchNorm2dReLU Sequentialconvsrr6r7r8GRUgru)r#rrrrrrrrCr8i conv_in_chansconv_out_chans gru_in_unitsr$r&r'rusJ      zReferenceEncoder.__init__r(r)cCsf|d}|d}||dd}|d}|||d}|j||\}}|d}|S)zCalculate forward propagation. Args: speech (Tensor): Batch of padded target features (B, Lmax, idim). Returns: Tensor: Reference embedding (B, gru_units) rrr)size unsqueezerC transpose contiguousviewrEflatten_parameters)r#r( batch_sizexshs time_length_r*r&r&r'r,s   zReferenceEncoder.forward)rr r rrrrr-r&r&r$r'r^s.6rc sXeZdZdZ     ddeded ed ed ef fd d ZdejdejfddZ Z S)r!aStyle token layer module. This module is style token layer introduced in `Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis`. .. _`Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis`: https://arxiv.org/abs/1803.09017 Args: ref_embed_dim (int, optional): Dimension of the input reference embedding. gst_tokens (int, optional): The number of GST embeddings. gst_token_dim (int, optional): Dimension of each GST embedding. gst_heads (int, optional): The number of heads in GST multihead attention. dropout_rate (float, optional): Dropout rate in multi-head attention. rrrr rrrr dropout_ratecs^tsJtt|t|||}|dtj|t ||||||||d|_ dS)z#Initilize style token layer module.gst_embs)q_dimk_dimv_dimn_headn_featrWN) rrr!rr3randnregister_parameterr> Parameterrmha)r#rrrrrWrXr$r&r'rs  zStyleTokenLayer.__init__r*r)cCsJ|d}t|jd|dd}|d}||||d}|dS)zCalculate forward propagation. Args: ref_embs (Tensor): Reference embeddings (B, ref_embed_dim). Returns: Tensor: Style token embeddings (B, gst_token_dim). rrJrN)rKr3tanhrXrLexpandrasqueeze)r#r*rQrXr+r&r&r'r,s   zStyleTokenLayer.forward)rrrr rV) r.r/r0r1r2floatrr3r4r,r5r&r&r$r'r!s&r!c@seZdZdZdddZdS)rz;Multi head attention module with different input dimension.rVcCstjj|||dksJ|||_||_tj|||_tj|||_tj|||_ tj|||_ d|_ tjj |d|_ dS)z'Initialize multi head attention module.rN)p)r3r>Modulerd_khLinearlinear_qlinear_klinear_v linear_outattnDropoutdropout)r#rYrZr[r\r]rWr&r&r'rs zMultiHeadedAttention.__init__N)rV)r.r/r0r1rr&r&r&r'rsr)r1typingrr3 typeguardr1espnet.nets.pytorch_backend.transformer.attentionrBaseMultiHeadedAttentionr>rgrrr!r&r&r&r's   Nd=