o ½e¦iÙjã@s¬dZddlZddlmZddlmZddlZddlmZddlm Z ddl m Z m Z mZmZmZddlmZGd d „d ejƒZed d ƒZGd d„dejƒZGdd„dƒZdS)zÅ Neural network modules for the Zero-Shot Multi-Speaker Tacotron2 end-to-end neural Text-to-Speech (TTS) model Authors * Georges Abous-Rjeili 2021 * Artem Ploujnikov 2021 * Pradnya Kandarkar 2023 éN)Ú namedtuple)Úsqrt)Únn)Ú functional)ÚDecoderÚEncoderÚ LinearNormÚPostnetÚget_mask_from_lengths)ÚGuidedAttentionLosscsjeZdZdZ                 d‡fdd„ Zddd„Zddd„Zdd„Z‡ZS)Ú Tacotron2axThe Tactron2 text-to-speech model, based on the NVIDIA implementation. This class is the main entry point for the model, which is responsible for instantiating all submodules, which, in turn, manage the individual neural network layers Simplified STRUCTURE: phoneme input->token embedding ->encoder -> (encoder output + speaker embedding) ->attention ->decoder(+prenet) -> postnet ->output prenet(input is decoder previous time step) output is input to decoder concatenated with the attention output Arguments --------- spk_emb_size: int Speaker embedding size mask_padding: bool whether or not to mask pad-outputs of tacotron n_mel_channels: int number of mel channels for constructing spectrogram n_symbols: int=128 number of accepted char symbols defined in textToSequence symbols_embedding_dim: int number of embedding dimension for symbols fed to nn.Embedding encoder_kernel_size: int size of kernel processing the embeddings encoder_n_convolutions: int number of convolution layers in encoder encoder_embedding_dim: int number of kernels in encoder, this is also the dimension of the bidirectional LSTM in the encoder attention_rnn_dim: int input dimension attention_dim: int number of hidden representation in attention attention_location_n_filters: int number of 1-D convolution filters in attention attention_location_kernel_size: int length of the 1-D convolution filters n_frames_per_step: int=1 only 1 generated mel-frame per step is supported for the decoder as of now. decoder_rnn_dim: int number of 2 unidirectional stacked LSTM units prenet_dim: int dimension of linear prenet layers max_decoder_steps: int maximum number of steps/frames the decoder generates before stopping gate_threshold: int cut off level any output probability above that is considered complete and stops generation so we have variable length outputs p_attention_dropout: float attention drop out probability p_decoder_dropout: float decoder drop out probability postnet_embedding_dim: int number os postnet dfilters postnet_kernel_size: int 1d size of posnet kernel postnet_n_convolutions: int number of convolution layers in postnet decoder_no_early_stopping: bool determines early stopping of decoder along with gate_threshold . The logical inverse of this is fed to the decoder Example ------- >>> import torch >>> _ = torch.manual_seed(213312) >>> from speechbrain.lobes.models.Tacotron2 import Tacotron2 >>> model = Tacotron2( ... mask_padding=True, ... n_mel_channels=80, ... n_symbols=148, ... symbols_embedding_dim=512, ... encoder_kernel_size=5, ... encoder_n_convolutions=3, ... encoder_embedding_dim=512, ... attention_rnn_dim=1024, ... attention_dim=128, ... attention_location_n_filters=32, ... attention_location_kernel_size=31, ... n_frames_per_step=1, ... decoder_rnn_dim=1024, ... prenet_dim=256, ... max_decoder_steps=32, ... gate_threshold=0.5, ... p_attention_dropout=0.1, ... p_decoder_dropout=0.1, ... postnet_embedding_dim=512, ... postnet_kernel_size=5, ... postnet_n_convolutions=5, ... decoder_no_early_stopping=False ... ) >>> _ = model.eval() >>> inputs = torch.tensor([ ... [13, 12, 31, 14, 19], ... [31, 16, 30, 31, 0], ... ]) >>> input_lengths = torch.tensor([5, 4]) >>> outputs, output_lengths, alignments = model.infer(inputs, input_lengths) >>> outputs.shape, output_lengths.shape, alignments.shape (torch.Size([2, 80, 1]), torch.Size([2]), torch.Size([2, 1, 5])) TéPé”ééééé€é ééééèçà?çš™™™™™¹?FcsÞtƒ ¡||_||_| |_t ||¡|_td||ƒ}tdƒ|}|jj j   | |¡t |||ƒ|_ t|| || | | | ||||||| ƒ|_t||||ƒ|_t||dƒ|_t||jƒ|_t|j|ƒ|_t|j|ƒ|_dS)Ng@g@é)ÚsuperÚ__init__Ú mask_paddingÚn_mel_channelsÚn_frames_per_steprÚ EmbeddingÚ embeddingrÚweightÚdataÚuniform_rÚencoderrÚdecoderr ÚpostnetÚintÚms_film_hidden_sizerÚms_film_hiddenÚ ms_film_hÚ ms_film_g)ÚselfÚ spk_emb_sizerrÚ n_symbolsÚsymbols_embedding_dimÚencoder_kernel_sizeÚencoder_n_convolutionsÚencoder_embedding_dimÚattention_rnn_dimÚ attention_dimÚattention_location_n_filtersÚattention_location_kernel_sizer Údecoder_rnn_dimÚ prenet_dimÚmax_decoder_stepsÚgate_thresholdÚp_attention_dropoutÚp_decoder_dropoutÚpostnet_embedding_dimÚpostnet_kernel_sizeÚpostnet_n_convolutionsÚdecoder_no_early_stoppingÚstdÚval©Ú __class__©úb/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/speechbrain/lobes/models/MSTacotron2.pyr¥sV ! ÿòü ÿÿ ÿzTacotron2.__init__Nc CsÀ|\}}}}|jrH|durHt|| d¡d}| |j| d¡| d¡¡}| ddd¡}| ¡ |d¡| |d¡| |dd…ddd…fd¡|durYt  |d|| d¡f¡}|||||fS) a¨ Masks the padded part of output Arguments --------- outputs: list a list of tensors - raw outputs output_lengths: torch.Tensor a tensor representing the lengths of all outputs alignments_dim: int the desired dimension of the alignments along the last axis Optional but needed for data-parallel training Returns ------- mel_outputs: torch.Tensor mel_outputs_postnet: torch.Tensor gate_outputs: torch.Tensor alignments: torch.Tensor output_lengths: torch.Tensor the original outputs - with the mask applied Néÿÿÿÿ)Úmax_lenrrrçg@@) rr ÚsizeÚexpandrÚpermuteÚcloneÚ masked_fill_ÚFÚpad) r.ÚoutputsÚoutput_lengthsÚalignments_dimÚ mel_outputsÚmel_outputs_postnetÚ gate_outputsÚ alignmentsÚmaskrGrGrHÚ parse_outputõs(  ÿ ÿûzTacotron2.parse_outputcCsä|\}}}}}|j|j}}| |¡ dd¡}| ||¡} t | |¡¡} | | ¡} t  | d¡  d| j dd¡} | | } |  | ¡} t  | d¡  d| j dd¡} | | } |j | ||d\} }}| | ¡}| |}| | |||g||¡S)a1Decoder forward pass for training Arguments --------- inputs: tuple batch object spk_embs: torch.Tensor Speaker embeddings corresponding to the inputs alignments_dim: int the desired dimension of the alignments along the last axis Optional but needed for data-parallel training Returns ------- mel_outputs: torch.Tensor mel outputs from the decoder mel_outputs_postnet: torch.Tensor mel outputs from postnet gate_outputs: torch.Tensor gate outputs from the decoder alignments: torch.Tensor sequence of attention weights from the decoder output_lengths: torch.Tensor length of the output without padding rr)Úmemory_lengths)r$r"Ú transposer&rQÚrelur+r,ÚtorchÚ unsqueezeÚrepeatÚshaper-r'r(r[)r.ÚinputsÚspk_embsrUÚ input_lengthsÚtargetsrJrTÚembedded_inputsÚencoder_outputsÚspk_embs_sharedÚ spk_embs_hÚ spk_embs_grVrXrYrWrGrGrHÚforward$s2    ÿ   ÿ ÿ  ýzTacotron2.forwardcCsÞ| |¡ dd¡}|j ||¡}t | |¡¡}| |¡}t  |d¡  d|j dd¡}||}|  |¡}t  |d¡  d|j dd¡}||}|j  ||¡\} } } } | | ¡} | | } |  d¡}|  d||¡ dd¡} | | | fS)a*Produces outputs Arguments --------- inputs: torch.tensor text or phonemes converted spk_embs: torch.Tensor Speaker embeddings corresponding to the inputs input_lengths: torch.tensor the lengths of input parameters Returns ------- mel_outputs_postnet: torch.Tensor final mel output of tacotron 2 mel_lengths: torch.Tensor length of mels alignments: torch.Tensor sequence of attention weights rrr)r"r]r&ÚinferrQr^r+r,r_r`rarbr-r'r(rLÚunfold)r.rcrdrergrhrirjrkrVrXrYÚ mel_lengthsrWÚBSrGrGrHrmas*   ÿ   ÿ ÿ   zTacotron2.infer)Tr rrrrrrrrrrrrrrrrrrrF©N) Ú__name__Ú __module__Ú __qualname__Ú__doc__rr[rlrmÚ __classcell__rGrGrErHr <s8ká P /=r Ú TacotronLossz:loss mel_loss spk_emb_loss gate_loss attn_loss attn_weightcsBeZdZdZ        d ‡fdd„ Zdd„Zdd „Z‡ZS) ÚLossa|The Tacotron loss implementation The loss consists of an MSE loss on the spectrogram, a BCE gate loss and a guided attention loss (if enabled) that attempts to make the attention matrix diagonal The output of the module is a LossStats tuple, which includes both the total loss Arguments --------- guided_attention_sigma: float The guided attention sigma factor, controlling the "width" of the mask gate_loss_weight: float The constant by which the gate loss will be multiplied mel_loss_weight: float The constant by which the mel loss will be multiplied spk_emb_loss_weight: float The constant by which the speaker embedding loss will be multiplied - placeholder for future work spk_emb_loss_type: str Type of the speaker embedding loss - placeholder for future work guided_attention_weight: float The weight for the guided attention guided_attention_scheduler: callable The scheduler class for the guided attention loss guided_attention_hard_stop: int The number of epochs after which guided attention will be completely turned off Example ------- >>> import torch >>> _ = torch.manual_seed(42) >>> from speechbrain.lobes.models.MSTacotron2 import Loss >>> loss = Loss(guided_attention_sigma=0.2) >>> mel_target = torch.randn(2, 80, 861) >>> gate_target = torch.randn(1722, 1) >>> mel_out = torch.randn(2, 80, 861) >>> mel_out_postnet = torch.randn(2, 80, 861) >>> gate_out = torch.randn(2, 861) >>> alignments = torch.randn(2, 861, 173) >>> pred_mel_lens = torch.randn(2) >>> targets = mel_target, gate_target >>> model_outputs = mel_out, mel_out_postnet, gate_out, alignments, pred_mel_lens >>> input_lengths = torch.tensor([173, 91]) >>> target_lengths = torch.tensor([861, 438]) >>> spk_embs = None >>> loss(model_outputs, targets, input_lengths, target_lengths, spk_embs, 1) TacotronLoss(loss=tensor([4.8566]), mel_loss=tensor(4.0097), spk_emb_loss=tensor([0.]), gate_loss=tensor(0.8460), attn_loss=tensor(0.0010), attn_weight=tensor(1.)) Nçð?c sŒtƒ ¡|dkr d}||_||_||_||_||_t ¡|_ t  ¡|_ t |d|_ t ¡|_tjjdd„d|_t ¡|_||_||_dS)Nr)ÚsigmacSsdt ||¡S)Nry)rQÚcosine_similarity)ÚxÚyrGrGrHÚêszLoss.__init__..)Údistance_function)rrÚguided_attention_weightÚgate_loss_weightÚmel_loss_weightÚspk_emb_loss_weightÚspk_emb_loss_typerÚMSELossÚmse_lossÚBCEWithLogitsLossÚbce_lossr Úguided_attention_lossÚCosineSimilarityÚcos_simr_ÚTripletMarginWithDistanceLossÚ triplet_lossÚCosineEmbeddingLossÚ cos_emb_lossÚguided_attention_schedulerÚguided_attention_hard_stop) r.Úguided_attention_sigmarr‚rƒr„r€rr‘rErGrHrÏs(   ÿ ÿ  z Loss.__init__cCs^|d|d}}d|_d|_| dd¡}|\} } } } } |  dd¡} | | |¡| | |¡}|j|}|j| | |¡}| | |||¡\}}t dg¡  |j ¡}|j dkrk|\}}|  ||¡}t  t |¡t|ƒ¡ }|j dkr„|\}}| ||t t|ƒ¡  |j ¡¡}|j dkr™|\}}}|dur™| |||¡}|j|}||||}t||||||ƒS) aZComputes the loss Arguments --------- model_output: tuple the output of the model's forward(): (mel_outputs, mel_outputs_postnet, gate_outputs, alignments) targets: tuple the targets input_lengths: torch.Tensor a (batch, length) tensor of input lengths target_lengths: torch.Tensor a (batch, length) tensor of target (spectrogram) lengths spk_embs: torch.Tensor Speaker embedding input for the loss computation - placeholder for future work epoch: int the current epoch number (used for the scheduling of the guided attention loss) A StepScheduler is typically used Returns ------- result: LossStats the total loss - and individual losses (mel and gate) rrFrIÚscl_lossrrN)Ú requires_gradÚviewr†r‚rrˆÚget_attention_lossr_ÚTensorÚtoÚdevicer„r‹ÚdivÚsumÚlenrÚonesrrƒÚ LossStats)r.Ú model_outputrfreÚtarget_lengthsrdÚepochÚ mel_targetÚ gate_targetÚmel_outÚmel_out_postnetÚgate_outrYÚ pred_mel_lensÚmel_lossÚ gate_lossÚ attn_lossÚ attn_weightÚ spk_emb_lossÚtarget_spk_embsÚpreds_spk_embsÚcos_sim_scoresÚanchor_spk_embsÚ pos_spk_embsÚ neg_spk_embsÚ total_lossrGrGrHrlñsd ú ÿ ÿ  ÿ ý  ÿ úz Loss.forwardc Cs¢tjd|jd}|jdus|jdkr||}}||fS|jduo$||jk}|r-||}}n|j}|jdur<| |¡\} }tj||jd}|| |||¡}||fS)a_Computes the attention loss Arguments --------- alignments: torch.Tensor the alignment matrix from the model input_lengths: torch.Tensor a (batch, length) tensor of input lengths target_lengths: torch.Tensor a (batch, length) tensor of target (spectrogram) lengths epoch: int the current epoch number (used for the scheduling of the guided attention loss) A StepScheduler is typically used Returns ------- attn_loss: torch.Tensor the attention loss value rK)r™Nr)r_Útensorr™r€r‘rr‰) r.rYrer r¡Ú zero_tensorr«rªÚhard_stop_reachedÚ_rGrGrHr–Os$    óþ  ÿzLoss.get_attention_loss)NryryryNryNN)rrrsrtrurrlr–rvrGrGrErHrxœs4÷"^rxc@s$eZdZdZ ddd„Zdd„ZdS) ÚTextMelCollatea Zero-pads model inputs and targets based on number of frames per step Arguments --------- speaker_embeddings_pickle : str Path to the file containing speaker embeddings n_frames_per_step: int The number of output frames per step rcCs||_||_dSrq)r Úspeaker_embeddings_pickle)r.r¹r rGrGrHr†s zTextMelCollate.__init__c Cs†t|ƒ}tt|ƒƒD] }||d||<q tjt dd„|Dƒ¡ddd\}}|d}t t|ƒ|¡}| ¡tt|ƒƒD]}|||d}|||d| d¡…f<q=|dd d¡} td d„|Dƒƒ} | |j dkr| |j | |j 7} | |j dksJ‚t  t|ƒ| | ¡} |  ¡t  t|ƒ| ¡} |  ¡t t|ƒ¡} ggggf\}}}}t |j d ƒ }t  |¡}Wdƒn1s¿wYtt|ƒƒD][}||}||d}|| |dd…d| d¡…f<d| || d¡dd…f<| d¡| |<| ||d ¡| ||d ¡|||d }| |¡| ||d  d¡d¡qÊt |¡}dd„|Dƒ}t |¡}||| | | |||||f S)aCollate's training batch from normalized text and mel-spectrogram Arguments --------- batch: list [text_normalized, mel_normalized] Returns ------- text_padded: torch.Tensor input_lengths: torch.Tensor mel_padded: torch.Tensor gate_padded: torch.Tensor output_lengths: torch.Tensor len_x: torch.Tensor labels: torch.Tensor wavs: torch.Tensor spk_embs: torch.Tensor spk_ids: torch.Tensor Ú mel_text_paircSsg|]}t|dƒ‘qS)r)rœ©Ú.0r|rGrGrHÚ ¯sz+TextMelCollate.__call__..rT)ÚdimÚ descendingNrcSsg|] }|d d¡‘qS©r)rLr»rGrGrHr½»sÚrbÚlabelÚwavÚuttidr·cSsg|]}|d‘qS)rrGr»rGrGrHr½ßs)ÚlistÚrangerœr_ÚsortÚ LongTensorÚzero_rLÚmaxr Ú FloatTensorÚopenr¹ÚpickleÚloadÚappendÚsplitÚstackr—)r.ÚbatchÚ raw_batchÚireÚids_sorted_decreasingÚ max_input_lenÚ text_paddedÚtextÚnum_melsÚmax_target_lenÚ mel_paddedÚ gate_paddedrTÚlabelsÚwavsÚ spk_embs_listÚspk_idsÚspeaker_embeddings_fileÚspeaker_embeddingsÚidxÚmelÚspk_embrdÚlen_xrGrGrHÚ__call__stÿ ÿÿÿ ý    özTextMelCollate.__call__NrÀ)rrrsrtrurrçrGrGrGrHr¸{s  ý r¸)rurÍÚ collectionsrÚmathrr_rÚtorch.nnrrQÚ"speechbrain.lobes.models.Tacotron2rrrr r Ú%speechbrain.nnet.loss.guidedattn_lossr ÚModuler ržrxr¸rGrGrGrHÚs$)     ]ÿ`