o %ݫi"@sBdZddlmZddlmZddlZddlmZddlmZ ddl m Z ddl m Z Gd d d ejjZGd d d ejjZGd ddejZGdddejZGdddejZGdddejZGdddejZGdddejZGdddejZddZeddZGdd d ejZGd!d"d"Zd)d%d&Zd'd(ZdS)*z Neural network modules for the Tacotron2 end-to-end neural Text-to-Speech (TTS) model Authors * Georges Abous-Rjeili 2021 * Artem Ploujnikov 2021 ) namedtuple)sqrtN)nn) functional)get_mask_from_lengths)GuidedAttentionLosscs*eZdZdZdfdd ZddZZS) LinearNorma:A linear layer with Xavier initialization Arguments --------- in_dim: int the input dimension out_dim: int the output dimension bias: bool whether or not to use a bias w_init_gain: linear the weight initialization gain type (see torch.nn.init.calculate_gain) Example ------- >>> import torch >>> from speechbrain.lobes.models.Tacotron2 import LinearNorm >>> layer = LinearNorm(in_dim=5, out_dim=3) >>> x = torch.randn(3, 5) >>> y = layer(x) >>> y.shape torch.Size([3, 3]) TlinearcsBttjj|||d|_tjjj|jjtjj |ddS)Nbiasgain) super__init__torchrLinear linear_layerinitxavier_uniform_weightcalculate_gain)selfin_dimout_dimr w_init_gain __class__V/home/ubuntu/.local/lib/python3.10/site-packages/speechbrain/lobes/models/Tacotron2.pyrOs   zLinearNorm.__init__cC ||S)zComputes the forward pass Arguments --------- x: torch.Tensor a (batch, features) input tensor Returns ------- output: torch.Tensor the linear layer output )r)rxrrrforwardXs zLinearNorm.forward)Tr __name__ __module__ __qualname____doc__rr! __classcell__rrrrr6s rcs6eZdZdZ      d fdd Zdd ZZS) ConvNormaA 1D convolution layer with Xavier initialization Arguments --------- in_channels: int the number of input channels out_channels: int the number of output channels kernel_size: int the kernel size stride: int the convolutional stride padding: int the amount of padding to include. If not provided, it will be calculated as dilation * (kernel_size - 1) / 2 dilation: int the dilation of the convolution bias: bool whether or not to use a bias w_init_gain: linear the weight initialization gain type (see torch.nn.init.calculate_gain) Example ------- >>> import torch >>> from speechbrain.lobes.models.Tacotron2 import ConvNorm >>> layer = ConvNorm(in_channels=10, out_channels=5, kernel_size=3) >>> x = torch.randn(3, 10, 5) >>> y = layer(x) >>> y.shape torch.Size([3, 5, 5]) NTr c svt|dur|ddksJt||dd}tjj|||||||d|_tjjj|jj tjj |ddS)Nr)) kernel_sizestridepaddingdilationr r ) rrintrrConv1dconvrrrr) r in_channels out_channelsr+r,r-r.r rrrrrs   zConvNorm.__init__cCr)zComputes the forward pass Arguments --------- signal: torch.Tensor the input to the convolutional layer Returns ------- output: torch.Tensor the output )r1)rsignalrrrr!s zConvNorm.forward)r)r)Nr)Tr r"rrrrr(js%r(cs0eZdZdZ   d fdd ZddZZS) LocationLayeraA location-based attention layer consisting of a Xavier-initialized convolutional layer followed by a dense layer Arguments --------- attention_n_filters: int the number of filters used in attention attention_kernel_size: int the kernel size of the attention layer attention_dim: int the dimension of linear attention layers Example ------- >>> import torch >>> from speechbrain.lobes.models.Tacotron2 import LocationLayer >>> layer = LocationLayer() >>> attention_weights_cat = torch.randn(3, 2, 64) >>> processed_attention = layer(attention_weights_cat) >>> processed_attention.shape torch.Size([3, 64, 128]) c sHtt|dd}td|||dddd|_t||ddd|_dS)Nr)r*F)r+r-r r,r.tanhr r)rrr/r( location_convrlocation_dense)rattention_n_filtersattention_kernel_size attention_dimr-rrrrs   zLocationLayer.__init__cCs$||}|dd}||}|S)a.Performs the forward pass for the attention layer Arguments --------- attention_weights_cat: torch.Tensor the concatenating attention weights Returns ------- processed_attention: torch.Tensor the attention layer output r)r*)r; transposer<)rattention_weights_catprocessed_attentionrrrr!s   zLocationLayer.forward)r6r7r8r"rrrrr5sr5cs<eZdZdZ     d fdd Zd d Zd d ZZS) AttentionaCThe Tacotron attention layer. Location-based attention is used. Arguments --------- attention_rnn_dim: int the dimension of the RNN to which the attention layer is applied embedding_dim: int the embedding dimension attention_dim: int the dimension of the memory cell attention_location_n_filters: int the number of location filters attention_location_kernel_size: int the kernel size of the location layer Example ------- >>> import torch >>> from speechbrain.lobes.models.Tacotron2 import ( ... Attention) >>> from speechbrain.lobes.models.transformer.Transformer import ( ... get_mask_from_lengths) >>> layer = Attention() >>> attention_hidden_state = torch.randn(2, 1024) >>> memory = torch.randn(2, 173, 512) >>> processed_memory = torch.randn(2, 173, 128) >>> attention_weights_cat = torch.randn(2, 2, 173) >>> memory_lengths = torch.tensor([173, 91]) >>> mask = get_mask_from_lengths(memory_lengths) >>> attention_context, attention_weights = layer( ... attention_hidden_state, ... memory, ... processed_memory, ... attention_weights_cat, ... mask ... ) >>> attention_context.shape, attention_weights.shape (torch.Size([2, 512]), torch.Size([2, 173])) r8r6r7cs\tt||ddd|_t||ddd|_t|ddd|_t||||_td |_ dS)NFr9r:r)r inf) rrr query_layer memory_layervr5location_layerfloatscore_mask_value)rattention_rnn_dim embedding_dimr?attention_location_n_filtersattention_location_kernel_sizerrrr)s zAttention.__init__cCs@||d}||}|t|||}|d}|S)aComputes the alignment energies Arguments --------- query: torch.Tensor decoder output (batch, n_mel_channels * n_frames_per_step) processed_memory: torch.Tensor processed encoder outputs (B, T_in, attention_dim) attention_weights_cat: torch.Tensor cumulative and prev. att weights (B, 2, max_time) Returns ------- alignment : torch.Tensor (batch, max_time) r)r*)rG unsqueezerJrIrr9squeeze)rqueryprocessed_memoryrAprocessed_queryprocessed_attention_weightsenergiesrrrget_alignment_energies@s   z Attention.get_alignment_energiesc CsN||||}|||j}tj|dd}t|d|}|d}||fS)a7Computes the forward pass Arguments --------- attention_hidden_state: torch.Tensor attention rnn last output memory: torch.Tensor encoder outputs processed_memory: torch.Tensor processed encoder outputs attention_weights_cat: torch.Tensor previous and cumulative attention weights mask: torch.Tensor binary mask for padded data Returns ------- result: tuple a (attention_context, attention_weights) tuple r)dim) rX masked_fillrLFsoftmaxrbmmrQrR) rattention_hidden_statememoryrTrAmask alignmentattention_weightsattention_contextrrrr!_s zAttention.forward)rDrEr8r6r7)r#r$r%r&rrXr!r'rrrrrCs+rCcs4eZdZdZdddgdffdd ZddZZS) Preneta*The Tacotron pre-net module consisting of a specified number of normalized (Xavier-initialized) linear layers Arguments --------- in_dim: int the input dimensions sizes: int the dimension of the hidden layers/output dropout: float the dropout probability Example ------- >>> import torch >>> from speechbrain.lobes.models.Tacotron2 import Prenet >>> layer = Prenet() >>> x = torch.randn(862, 2, 80) >>> output = layer(x) >>> output.shape torch.Size([862, 2, 256]) P?csBt|g|dd}tddt||D|_||_dS)NcSsg|] \}}t||ddqS)Fr )r).0in_sizeout_sizerrr s z#Prenet.__init__..)rrr ModuleListziplayersdropout)rrsizesrqin_sizesrrrrs  zPrenet.__init__cCs,|jD]}tjt|||jdd}q|S)zComputes the forward pass for the prenet Arguments --------- x: torch.Tensor the prenet inputs Returns ------- output: torch.Tensor the output T)ptraining)rpr\rqrelu)rr r rrrr!s zPrenet.forwardr"rrrrres recs2eZdZdZ    d fdd ZddZZS) PostnetahThe Tacotron postnet consists of a number of 1-d convolutional layers with Xavier initialization and a tanh activation, with batch normalization. Depending on configuration, the postnet may either refine the MEL spectrogram or upsample it to a linear spectrogram Arguments --------- n_mel_channels: int the number of MEL spectrogram channels postnet_embedding_dim: int the postnet embedding dimension postnet_kernel_size: int the kernel size of the convolutions within the decoders postnet_n_convolutions: int the number of convolutions in the postnet Example ------- >>> import torch >>> from speechbrain.lobes.models.Tacotron2 import Postnet >>> layer = Postnet() >>> x = torch.randn(2, 80, 861) >>> output = layer(x) >>> output.shape torch.Size([2, 80, 861]) rfrEcstt|_|jtt|||dt|dddddt |t d|dD]}|jtt|||dt|dddddt |q-|jtt|||dt|dddddt |t |j|_ dS)Nr)r*r9r+r,r-r.rr ) rrrrn convolutionsappend Sequentialr(r/ BatchNorm1drangelenn_convs)rn_mel_channelspostnet_embedding_dimpostnet_kernel_sizepostnet_n_convolutionsirrrrs\     zPostnet.__init__cCs^d}|jD]'}||jdkrtjt||d|jd}n tj||d|jd}|d7}q|S)azComputes the forward pass of the postnet Arguments --------- x: torch.Tensor the postnet input (usually a MEL spectrogram) Returns ------- output: torch.Tensor the postnet output (a refined MEL spectrogram or a linear spectrogram depending on how the model is configured) rr)rh)ru)rzrr\rqrr9ru)rr rr1rrrr!s  zPostnet.forward)rfrErxrxr"rrrrrws9rwcsHeZdZdZ   d fdd ZejjddZejj d d Z Z S) EncoderaThe Tacotron2 encoder module, consisting of a sequence of 1-d convolution banks (3 by default) and a bidirectional LSTM Arguments --------- encoder_n_convolutions: int the number of encoder convolutions encoder_embedding_dim: int the dimension of the encoder embedding encoder_kernel_size: int the kernel size of the 1-D convolutional layers within the encoder Example ------- >>> import torch >>> from speechbrain.lobes.models.Tacotron2 import Encoder >>> layer = Encoder() >>> x = torch.randn(2, 512, 128) >>> input_lengths = torch.tensor([128, 83]) >>> outputs = layer(x, input_lengths) >>> outputs.shape torch.Size([2, 128, 512]) rErxc stg}t|D]}tt|||dt|dddddt|}||q t ||_ tj |t|ddddd|_ dS)Nr)r*rvryT) batch_first bidirectional) rrr~rr|r(r/r}r{rnrzLSTMlstm)rencoder_n_convolutionsencoder_embedding_dimencoder_kernel_sizerz_ conv_layerrrrrGs0      zEncoder.__init__cCs|jD]}tt||d|j}q|dd}|}tj j j ||dd}|j | |\}}tj j j|dd\}}|S)aDComputes the encoder forward pass Arguments --------- x: torch.Tensor a batch of inputs (sequence embeddings) input_lengths: torch.Tensor a tensor of input lengths Returns ------- outputs: torch.Tensor the encoder output rhr)r*Tr)rzr\rqrvrur@cpunumpyrutilsrnnpack_padded_sequencerflatten_parameterspad_packed_sequence)rr input_lengthsr1outputsrrrrr!hs    zEncoder.forwardc Cs|j}|jD]}tt|||d|j}q|dd}|}t j j j ||dd}| |\}}t j j j|dd\}}|S)aSPerforms a forward step in the inference context Arguments --------- x: torch.Tensor a batch of inputs (sequence embeddings) input_lengths: torch.Tensor a tensor of input lengths Returns ------- outputs: torch.Tensor the encoder output rhr)r*Tr)devicerzr\rqrvtorur@rrrrrrr)rr rrr1rrrrrinfers " z Encoder.infer)rrErx) r#r$r%r&rrjitignorer!exportrr'rrrrr,s! "rcseZdZdZ         dfdd ZddZddZddZddZddZ e j j ddZ e j jddZZS)DecoderaThe Tacotron decoder Arguments --------- n_mel_channels: int the number of channels in the MEL spectrogram n_frames_per_step: int the number of frames in the spectrogram for each time step of the decoder encoder_embedding_dim: int the dimension of the encoder embedding attention_dim: int Size of attention vector attention_location_n_filters: int the number of filters in location-based attention attention_location_kernel_size: int the kernel size of location-based attention attention_rnn_dim: int RNN dimension for the attention layer decoder_rnn_dim: int the encoder RNN dimension prenet_dim: int the dimension of the prenet (inner and output layers) max_decoder_steps: int the maximum number of decoder steps for the longest utterance expected for the model gate_threshold: float the fixed threshold to which the outputs of the decoders will be compared p_attention_dropout: float dropout probability for attention layers p_decoder_dropout: float dropout probability for decoder layers early_stopping: bool Whether to stop training early. Example ------- >>> import torch >>> from speechbrain.lobes.models.Tacotron2 import Decoder >>> layer = Decoder() >>> memory = torch.randn(2, 173, 512) >>> decoder_inputs = torch.randn(2, 80, 173) >>> memory_lengths = torch.tensor([173, 91]) >>> mel_outputs, gate_outputs, alignments = layer( ... memory, decoder_inputs, memory_lengths) >>> mel_outputs.shape, gate_outputs.shape, alignments.shape (torch.Size([2, 80, 173]), torch.Size([2, 173]), torch.Size([2, 173, 173])) rfr)rEr8r6r7rDrgrh皙?Tcst||_||_||_||_||_| |_| |_| |_ | |_ | |_ ||_ t ||| | g|_t| |||_t||||||_t|||d|_t|||||_t||dddd|_dS)Nr)Tsigmoidr:)rrrn_frames_per_steprrMdecoder_rnn_dim prenet_dimmax_decoder_stepsgate_thresholdp_attention_dropoutp_decoder_dropoutearly_stoppingreprenetrLSTMCell attention_rnnrCattention_layer decoder_rnnrlinear_projection gate_layer)rrrrr?rOrPrMrrrrrrrrrrrsL    zDecoder.__init__cCs4|d}|j}|j}tj||j|j||d}|S)zGets all zeros frames to use as first decoder input Arguments --------- memory: torch.Tensor decoder outputs Returns ------- decoder_input: torch.Tensor all zeros frames rdtyper)sizerrrzerosrr)rr`Brr decoder_inputrrr get_go_frames  zDecoder.get_go_framecCs|d}|d}|j}|j}tj||j||d}tj||j||d}tj||j||d}tj||j||d} tj||||d} tj||||d} tj||j||d} |j |} |||| | | | | fS)arInitializes attention rnn states, decoder rnn states, attention weights, attention cumulative weights, attention context, stores memory and stores processed memory Arguments --------- memory: torch.Tensor Encoder outputs Returns ------- attention_hidden: torch.Tensor attention_cell: torch.Tensor decoder_hidden: torch.Tensor decoder_cell: torch.Tensor attention_weights: torch.Tensor attention_weights_cum: torch.Tensor attention_context: torch.Tensor processed_memory: torch.Tensor rr)r) rrrrrrMrrrrH)rr`rMAX_TIMErrattention_hiddenattention_celldecoder_hidden decoder_cellrcattention_weights_cumrdrTrrrinitialize_decoder_states5sB        z!Decoder.initialize_decoder_statescCs@|dd}||dt|d|jd}|dd}|S)a,Prepares decoder inputs, i.e. mel outputs Arguments --------- decoder_inputs: torch.Tensor inputs used for teacher-forced training, i.e. mel-specs Returns ------- decoder_inputs: torch.Tensor processed decoder inputs r)r*rri)r@viewrr/r)rdecoder_inputsrrrparse_decoder_inputsrs  zDecoder.parse_decoder_inputscCsz|dd}|dkr|d}n|dd}|dd}|jdd|jf}|j|}|dd}|||fS)aPrepares decoder outputs for output Arguments --------- mel_outputs: torch.Tensor MEL-scale spectrogram outputs gate_outputs: torch.Tensor gate output energies alignments: torch.Tensor the alignment tensor Returns ------- mel_outputs: torch.Tensor MEL-scale spectrogram outputs gate_outputs: torch.Tensor gate output energies alignments: torch.Tensor the alignment tensor rr)rir*)r@ contiguousrZrQshaperr)r mel_outputs gate_outputs alignmentsrrrrparse_decoder_outputss     zDecoder.parse_decoder_outputsc  Cst||fd} || ||f\}}t||j|j}tj|d|dfdd} ||| | | | \}}||7}t||fd}| |||f\}}t||j |j}tj||fdd}| |}| |}|||||||||f S)aaDecoder step using stored states, attention and memory Arguments --------- decoder_input: torch.Tensor previous mel output attention_hidden: torch.Tensor the hidden state of the attention module attention_cell: torch.Tensor the attention cell state decoder_hidden: torch.Tensor the decoder hidden state decoder_cell: torch.Tensor the decoder cell state attention_weights: torch.Tensor the attention weights attention_weights_cum: torch.Tensor cumulative attention weights attention_context: torch.Tensor the attention context tensor memory: torch.Tensor the memory tensor processed_memory: torch.Tensor the processed memory tensor mask: torch.Tensor Returns ------- mel_output: torch.Tensor the MEL-scale outputs gate_output: torch.Tensor gate output energies attention_weights: torch.Tensor attention weights rir)rY) rcatrr\rqrrurQrrrrr)rrrrrrrcrrdr`rTra cell_inputrA decoder_hidden_attention_contextdecoder_outputgate_predictionrrrdecodesZ2   zDecoder.decodec Cs(||d}||}tj||fdd}||}t|}||\}}}} } } } } ggg}}}t|| ddkr||t|}| ||||| | | | || | \ }}}}}} } } } || dg7}|| g7}|| g7}t|| ddks?| t |t |t |\}}}|||fS)a`Decoder forward pass for training Arguments --------- memory: torch.Tensor Encoder outputs decoder_inputs: torch.Tensor Decoder inputs for teacher forcing. i.e. mel-specs memory_lengths: torch.Tensor Encoder output lengths for attention masking. Returns ------- mel_outputs: torch.Tensor mel outputs from the decoder gate_outputs: torch.Tensor gate outputs from the decoder alignments: torch.Tensor sequence of attention weights from the decoder rrYr))rrQrrrrrrrrrrRrstack)rr`rmemory_lengthsrrarrrrrcrrdrTrrr mel_output gate_outputrrrr!sf       zDecoder.forwardc Cs||}t|}||\}}}}} } } } tj|dgtj|jd} tj|dgtj|jd}tdtdtd}}}d} | |}| |||||| | | || | \ }}}}}}} } } |rq| d}|}| }d}ntj || dfdd}tj ||fdd}tj || fdd}t t||jtjd}||}| |7} |jrt|dkrn t||jkrn|}qF||||\}}}|||| fS)a=Decoder inference Arguments --------- memory: torch.Tensor Encoder outputs memory_lengths: torch.Tensor The corresponding relative lengths of the inputs. Returns ------- mel_outputs: torch.Tensor mel outputs from the decoder gate_outputs: torch.Tensor gate outputs from the decoder alignments: torch.Tensor sequence of attention weights from the decoder mel_lengths: torch.Tensor the length of MEL spectrograms rrr)TFrY)rrrrrrint32ronesrrrQrlerrrrRrsumrrr)rr`rrrarrrrrcrrdrT mel_lengths not_finishedrrr first_iterrrdecrrrris      5 z Decoder.infer)rfr)rEr8r6r7rDrDrgrrhrrT)r#r$r%r&rrrrrrrrrr!rrr'rrrrrs43>=&i NrcsjeZdZdZ                 dfdd ZdddZdddZddZZS) Tacotron2aThe Tactron2 text-to-speech model, based on the NVIDIA implementation. This class is the main entry point for the model, which is responsible for instantiating all submodules, which, in turn, manage the individual neural network layers Simplified STRUCTURE: input->word embedding ->encoder ->attention ->decoder(+prenet) -> postnet ->output prenet(input is decoder previous time step) output is input to decoder concatenated with the attention output Arguments --------- mask_padding: bool whether or not to mask pad-outputs of tacotron n_mel_channels: int number of mel channels for constructing spectrogram n_symbols: int=128 number of accepted char symbols defined in textToSequence symbols_embedding_dim: int number of embedding dimension for symbols fed to nn.Embedding encoder_kernel_size: int size of kernel processing the embeddings encoder_n_convolutions: int number of convolution layers in encoder encoder_embedding_dim: int number of kernels in encoder, this is also the dimension of the bidirectional LSTM in the encoder attention_rnn_dim: int input dimension attention_dim: int number of hidden representation in attention attention_location_n_filters: int number of 1-D convolution filters in attention attention_location_kernel_size: int length of the 1-D convolution filters n_frames_per_step: int=1 only 1 generated mel-frame per step is supported for the decoder as of now. decoder_rnn_dim: int number of 2 unidirectional stacked LSTM units prenet_dim: int dimension of linear prenet layers max_decoder_steps: int maximum number of steps/frames the decoder generates before stopping gate_threshold: int cut off level any output probability above that is considered complete and stops generation so we have variable length outputs p_attention_dropout: float attention drop out probability p_decoder_dropout: float decoder drop out probability postnet_embedding_dim: int number os postnet dfilters postnet_kernel_size: int 1d size of posnet kernel postnet_n_convolutions: int number of convolution layers in postnet decoder_no_early_stopping: bool determines early stopping of decoder along with gate_threshold . The logical inverse of this is fed to the decoder Example ------- >>> import torch >>> _ = torch.manual_seed(213312) >>> from speechbrain.lobes.models.Tacotron2 import Tacotron2 >>> model = Tacotron2( ... mask_padding=True, ... n_mel_channels=80, ... n_symbols=148, ... symbols_embedding_dim=512, ... encoder_kernel_size=5, ... encoder_n_convolutions=3, ... encoder_embedding_dim=512, ... attention_rnn_dim=1024, ... attention_dim=128, ... attention_location_n_filters=32, ... attention_location_kernel_size=31, ... n_frames_per_step=1, ... decoder_rnn_dim=1024, ... prenet_dim=256, ... max_decoder_steps=32, ... gate_threshold=0.5, ... p_attention_dropout=0.1, ... p_decoder_dropout=0.1, ... postnet_embedding_dim=512, ... postnet_kernel_size=5, ... postnet_n_convolutions=5, ... decoder_no_early_stopping=False ... ) >>> _ = model.eval() >>> inputs = torch.tensor([ ... [13, 12, 31, 14, 19], ... [31, 16, 30, 31, 0], ... ]) >>> input_lengths = torch.tensor([5, 4]) >>> outputs, output_lengths, alignments = model.infer(inputs, input_lengths) >>> outputs.shape, output_lengths.shape, alignments.shape (torch.Size([2, 80, 1]), torch.Size([2]), torch.Size([2, 1, 5])) TrfrErxrrDr8r6r7r)rgrrhrFcst||_||_| |_t|||_td||}td|}|jj j | |t ||||_ t|| || | | || |||||| |_t|||||_dS)Ng@g@)rr mask_paddingrrr Embedding embeddingrrdatauniform_rencoderrdecoderrwpostnet)rrr n_symbolssymbols_embedding_dimrrrrMr?rOrPrrrrrrrrrrdecoder_no_early_stoppingstdvalrrrr=sB   zTacotron2.__init__Nc Cs|\}}}}|jrH|durHt||dd}||j|d|d}|ddd}||d||d||dddddfd|durYt |d||df}||||fS) a Masks the padded part of output Arguments --------- outputs: list a list of tensors - raw outputs output_lengths: torch.Tensor a tensor representing the lengths of all outputs alignments_dim: int the desired dimension of the alignments along the last axis Optional but needed for data-parallel training Returns ------- mel_outputs: torch.Tensor mel_outputs_postnet: torch.Tensor gate_outputs: torch.Tensor alignments: torch.Tensor the original outputs - with the mask applied Nri)max_lenrr)r*g@@) rrrexpandrpermuteclone masked_fill_r\pad) rroutput_lengthsalignments_dimrmel_outputs_postnetrrrarrr parse_outputs    zTacotron2.parse_outputc Csx|\}}}}}|j|j}}||dd}|||}|j|||d\} } } || } | | } || | | | g||S)aDecoder forward pass for training Arguments --------- inputs: tuple batch object alignments_dim: int the desired dimension of the alignments along the last axis Optional but needed for data-parallel training Returns ------- mel_outputs: torch.Tensor mel outputs from the decoder mel_outputs_postnet: torch.Tensor mel outputs from postnet gate_outputs: torch.Tensor gate outputs from the decoder alignments: torch.Tensor sequence of attention weights from the decoder output_lengths: torch.Tensor length of the output without padding r)r*)r)rrr@rrrr) rinputsrrtargetsrrembedded_inputsencoder_outputsrrrrrrrr!s   zTacotron2.forwardc Csr||dd}|j||}|j||\}}}}||} || } | d} |d| | dd}| ||fS)aProduces outputs Arguments --------- inputs: torch.tensor text or phonemes converted input_lengths: torch.tensor the lengths of input parameters Returns ------- mel_outputs_postnet: torch.Tensor final mel output of tacotron 2 mel_lengths: torch.Tensor length of mels alignments: torch.Tensor sequence of attention weights r)r*r)rr@rrrrrunfold) rrrrrrrrrrBSrrrrs   zTacotron2.infer)TrfrrErxrrErDr8r6r7r)rDrgrrhrrrErxrxFN) r#r$r%r&rrr!rr'rrrrrs8h B )-rcCs |||S)a} An inference hook for pretrained synthesizers Arguments --------- model: Tacotron2 the tacotron model text_sequences: torch.Tensor encoded text sequences input_lengths: torch.Tensor input lengths Returns ------- result: tuple (mel_outputs_postnet, mel_lengths, alignments) - the exact model output )r)modeltext_sequencesrrrrrs r TacotronLossz-loss mel_loss gate_loss attn_loss attn_weightcs<eZdZdZ     d fdd ZddZdd ZZS) LossaThe Tacotron loss implementation The loss consists of an MSE loss on the spectrogram, a BCE gate loss and a guided attention loss (if enabled) that attempts to make the attention matrix diagonal The output of the module is a LossStats tuple, which includes both the total loss Arguments --------- guided_attention_sigma: float The guided attention sigma factor, controlling the "width" of the mask gate_loss_weight: float The constant by which the hate loss will be multiplied guided_attention_weight: float The weight for the guided attention guided_attention_scheduler: callable The scheduler class for the guided attention loss guided_attention_hard_stop: int The number of epochs after which guided attention will be completely turned off Example ------- >>> import torch >>> _ = torch.manual_seed(42) >>> from speechbrain.lobes.models.Tacotron2 import Loss >>> loss = Loss(guided_attention_sigma=0.2) >>> mel_target = torch.randn(2, 80, 861) >>> gate_target = torch.randn(1722, 1) >>> mel_out = torch.randn(2, 80, 861) >>> mel_out_postnet = torch.randn(2, 80, 861) >>> gate_out = torch.randn(2, 861) >>> alignments = torch.randn(2, 861, 173) >>> targets = mel_target, gate_target >>> model_outputs = mel_out, mel_out_postnet, gate_out, alignments >>> input_lengths = torch.tensor([173, 91]) >>> target_lengths = torch.tensor([861, 438]) >>> loss(model_outputs, targets, input_lengths, target_lengths, 1) TacotronLoss(loss=tensor(4.8566), mel_loss=tensor(4.0097), gate_loss=tensor(0.8460), attn_loss=tensor(0.0010), attn_weight=tensor(1.)) N?csXt|dkr d}||_t|_t|_t|d|_ ||_ ||_||_ ||_ dS)Nr)sigma) rrguided_attention_weightrMSELossmse_lossBCEWithLogitsLossbce_lossrguided_attention_lossgate_loss_weightguided_attention_schedulerguided_attention_hard_stop)rguided_attention_sigmar rr r rrrrBs    z Loss.__init__cCs|d|d}}d|_d|_|dd}|\}} } } | dd} ||||| |} |j|| |} || |||\}}| | |}t|| | ||S)aComputes the loss Arguments --------- model_output: tuple the output of the model's forward(): (mel_outputs, mel_outputs_postnet, gate_outputs, alignments) targets: tuple the targets input_lengths: torch.Tensor a (batch, length) tensor of input lengths target_lengths: torch.Tensor a (batch, length) tensor of target (spectrogram) lengths epoch: int the current epoch number (used for the scheduling of the guided attention loss) A StepScheduler is typically used Returns ------- result: LossStats the total loss - and individual losses (mel and gate) rr)Fri) requires_gradrrr rget_attention_loss LossStats)r model_outputrrtarget_lengthsepoch mel_target gate_targetmel_outmel_out_postnetgate_outrmel_loss gate_loss attn_loss attn_weight total_lossrrrr!Xs"     z Loss.forwardc Cstjd|jd}|jdus|jdkr||}}||fS|jduo$||jk}|r-||}}n|j}|jdur<||\} }tj||jd}|||||}||fS)aaComputes the attention loss Arguments --------- alignments: torch.Tensor the alignment matrix from the model input_lengths: torch.Tensor a (batch, length) tensor of input lengths target_lengths: torch.Tensor a (batch, length) tensor of target (spectrogram) lengths epoch: int the current epoch number (used for the scheduling of the guided attention loss) A StepScheduler is typically used Returns ------- attn_loss: torch.Tensor the attention loss value r)rNr)rtensorrrr r r ) rrrrr zero_tensorrrhard_stop_reachedrrrrrs$      zLoss.get_attention_loss)NrrNN)r#r$r%r&rr!rr'rrrrrs..rc@s"eZdZdZdddZddZdS) TextMelCollatezZero-pads model inputs and targets based on number of frames per step Arguments --------- n_frames_per_step: int the number of output frames per step r)cCs ||_dSr)r)rrrrrrs zTextMelCollate.__init__c Cst|}tt|D] }||d||<q tjtdd|Dddd\}}|d}tt||}|tt|D]}|||d}|||d|df<q=|ddd} td d|D} | |j dkr| |j | |j 7} | |j dksJt t|| | } | t t|| } | tt|} gg}}tt|D]@}||}||d}|| |ddd|df<d| ||dddf<|d| |<| ||d | ||d qd d|D}t |}||| | | |||fS) aCollate's training batch from normalized text and mel-spectrogram Arguments --------- batch: list [text_normalized, mel_normalized] Returns ------- text_padded: torch.Tensor input_lengths: torch.Tensor mel_padded: torch.Tensor gate_padded: torch.Tensor output_lengths: torch.Tensor len_x: torch.Tensor labels: torch.Tensor wavs: torch.Tensor mel_text_paircSsg|]}t|dqS)r)rrjr rrrrmsz+TextMelCollate.__call__..rT)rZ descendingNr)cSsg|] }|ddqSr))rr#rrrrmslabelwavcSsg|]}|dqS)r*rr#rrrrms) listr~rrsort LongTensorzero_rmaxr FloatTensorr{Tensor)rbatch raw_batchrrids_sorted_decreasing max_input_len text_paddedtextnum_melsmax_target_len mel_padded gate_paddedrlabelswavsidxmellen_xrrr__call__s\    zTextMelCollate.__call__Nr%)r#r$r%r&rr>rrrrr!s  r!r)h㈵>cCsttj||d|S)z+Dynamic range compression for audio signals)min)rlogclamp)r Cclip_valrrrdynamic_range_compressionsrEc  CsLddlm} | j|||||||||| | d | j}|| }| r$t|}|S)acalculates MelSpectrogram for a raw audio signal Arguments --------- sample_rate : int Sample rate of audio signal. hop_length : int Length of hop between STFT windows. win_length : int Window size. n_fft : int Size of FFT. n_mels : int Number of mel filterbanks. f_min : float Minimum frequency. f_max : float Maximum frequency. power : float Exponent for the magnitude spectrogram. normalized : bool Whether to normalize by magnitude after stft. norm : str or None If "slaney", divide the triangular mel weights by the width of the mel band mel_scale : str Scale to use: "htk" or "slaney". compression : bool whether to do dynamic range compression audio : torch.Tensor input audio signal Returns ------- mel : torch.Tensor The computed mel spectrogram features. r) transforms) sample_rate hop_length win_lengthn_fftn_melsf_minf_maxpower normalizednorm mel_scale) torchaudiorFMelSpectrogramrrrE)rGrHrIrJrKrLrMrNrOrPrQ compressionaudiorF audio_to_melr<rrrmel_spectograms( 3 rW)r)r?)r& collectionsrmathrrrtorch.nnrr\0speechbrain.lobes.models.transformer.Transformerr%speechbrain.nnet.loss.guidedattn_lossrModulerr(r5rCrerwrrrrrrr!rErWrrrrsD (     4PE 5o-& \