o eiG@sdZddlZddlmZddlmZmZmZddlmZddl m Z Gdddej Z Gd d d ej Z Gd d d eZd dZddZddZdS)zlThe Attentional RNN model for Grapheme-to-Phoneme Authors * Mirco Ravanelli 2021 * Artem Ploujnikov 2021 N)nn)TransformerInterfaceget_key_padding_maskget_lookahead_mask) normalization)Linearcs:eZdZdZ   d fdd Zd ddZd d ZZS) AttentionSeq2Seqa The Attentional RNN encoder-decoder model Arguments --------- enc: torch.nn.Module the encoder module encoder_emb: torch.nn.Module the encoder_embedding_module emb: torch.nn.Module the embedding module dec: torch.nn.Module the decoder module lin: torch.nn.Module the linear module out: torch.nn.Module the output layer (typically log_softmax) bos_token: int the index of the Beginning-of-Sentence token use_word_emb: bool whether or not to use word embedding word_emb_enc: nn.Module a module to encode word embeddings rFNc sRt||_||_||_||_||_||_||_||_ |r$| |_ dSd|_ dS)N) super__init__enc encoder_embembdeclinout bos_token use_word_emb word_emb_enc) selfr r r rrrrrr __class__`/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/speechbrain/lobes/models/g2p/model.pyr .s zAttentionSeq2Seq.__init__cCs|\}}|durt|d|j}n|\}}||}|jr%t|j||}||\} }||} | | | |\} } | | } | | }||| | fS)aZComputes the forward pass Arguments --------- grapheme_encoded: torch.Tensor graphemes encoded as a Torch tensor phn_encoded: torch.Tensor the encoded phonemes word_emb: torch.Tensor word embeddings (optional) Returns ------- p_seq: torch.Tensor a (batch x position x token) tensor of token probabilities in each position char_lens: torch.Tensor a tensor of character sequence lengths encoder_out: the raw output of the encoder Nr) get_dummy_phonemessizedevicer r_apply_word_embrr r rrr)rgrapheme_encoded phn_encodedword_embchars char_lensphn_bos_emb_char encoder_oute_inhwlogitsp_seqrrrforwardEs     zAttentionSeq2Seq.forwardcCs*|jdur ||n|}tj||gddS)aConcatenate character embeddings with word embeddings, possibly encoding the word embeddings if an encoder is provided Arguments --------- emb_char: torch.Tensor the character embedding tensor word_emb: torch.Tensor the word embedding tensor Returns ------- result: torch.Tensor the concatenation of the tensorNdim)rtorchcat)rr$rrrrrrns  z AttentionSeq2Seq._apply_word_emb)rFNNN)__name__ __module__ __qualname____doc__r r+r __classcell__rrrrrs! )rcsFeZdZdZ d fdd ZddZddZejej ej d Z Z S) WordEmbeddingEncoderaA small encoder module that reduces the dimensionality and normalizes word embeddings Arguments --------- word_emb_dim: int the dimension of the original word embeddings word_emb_enc_dim: int the dimension of the encoded word embeddings norm: torch.nn.Module the normalization to be used ( e.g. speechbrain.nnet.normalization.LayerNorm) norm_type: str the type of normalization to be used NcsLt||_||_|r||||_n||_t||d|_t |_ dS)N) n_neurons input_size) r r word_emb_dimword_emb_enc_dim _get_normnormrrrTanh activation)rr:r;r= norm_typerrrr s zWordEmbeddingEncoder.__init__cCs(|j|}|std|||dS)a"Determines the type of normalizer Arguments --------- norm: str the normalization type: "batch", "layer" or "instance dim: int the dimensionality of the inputs Returns ------- The normalized outputs. zInvalid norm: )r9)NORMSget ValueError)rr=r.norm_clsrrrr<s  zWordEmbeddingEncoder._get_normcCs,|jdur ||}||}||}|S)zComputes the forward pass of the embedding Arguments --------- emb: torch.Tensor the original word embeddings Returns ------- emb_enc: torch.Tensor encoded word embeddings N)r=rr?)rr xrrrr+s   zWordEmbeddingEncoder.forward)batchlayerinstancer1) r2r3r4r5r r<r+r BatchNorm1d LayerNormInstanceNorm1drAr6rrrrr7s r7csxeZdZdZddddddejdddd d d d d d ddddddddffdd ZdddZddZdddZ ddZ Z S)TransformerG2Pu] A Transformer-based Grapheme-to-Phoneme model Arguments ---------- emb: torch.nn.Module the embedding module encoder_emb: torch.nn.Module the encoder embedding module char_lin: torch.nn.Module a linear module connecting the inputs to the transformer phn_lin: torch.nn.Module a linear module connecting the outputs to the transformer out: torch.nn.Module the decoder module (usually Softmax) lin: torch.nn.Module the linear module for outputs d_model: int The number of expected features in the encoder/decoder inputs (default=512). nhead: int The number of heads in the multi-head attention models (default=8). num_encoder_layers: int, optional The number of encoder layers in1ì the encoder. num_decoder_layers: int, optional The number of decoder layers in the decoder. dim_ffn: int, optional The dimension of the feedforward network model hidden layer. dropout: int, optional The dropout value. activation: torch.nn.Module, optional The activation function for Feed-Forward Network layer, e.g., relu or gelu or swish. custom_src_module: torch.nn.Module, optional Module that processes the src features to expected feature dim. custom_tgt_module: torch.nn.Module, optional Module that processes the src features to expected feature dim. positional_encoding: str, optional Type of positional encoding used. e.g. 'fixed_abs_sine' for fixed absolute positional encodings. normalize_before: bool, optional Whether normalization should be applied before or after MHA or FFN in Transformer layers. Defaults to True as this was shown to lead to better performance and training stability. kernel_size: int, optional Kernel size in convolutional layers when Conformer is used. bias: bool, optional Whether to use bias in Conformer convolutional layers. encoder_module: str, optional Choose between Conformer and Transformer for the encoder. The decoder is fixed to be a Transformer. conformer_activation: torch.nn.Module, optional Activation module used after Conformer convolutional layers. E.g. Swish, ReLU etc. it has to be a torch Module. attention_type: str, optional Type of attention layer used in all Transformer or Conformer layers. e.g. regularMHA or RelPosMHA. max_length: int, optional Max length for the target and source sequence in input. Used for positional encodings. causal: bool, optional Whether the encoder should be causal or not (the decoder is always causal). If causal the Conformer convolutional layer is causal. pad_idx: int the padding index (for masks) encoder_kdim: int, optional Dimension of the key for the encoder. encoder_vdim: int, optional Dimension of the value for the encoder. decoder_kdim: int, optional Dimension of the key for the decoder. decoder_vdim: int, optional Dimension of the value for the decoder. iig?Nfixed_abs_sineT transformer regularMHAi Frcstjdid|d|d| d| d| d| d| d|d |d |d |d |d |d|d|d|d|d|d|d|d|||_||_||_||_||_||_||_||_ ||_ | dS)Nd_modelnheadnum_encoder_layersnum_decoder_layersd_ffndropoutr?custom_src_modulecustom_tgt_modulepositional_encodingnormalize_before kernel_sizebiasencoder_moduleattention_type max_lengthcausal encoder_kdim encoder_vdim decoder_kdim decoder_vdimr) r r r r char_linphn_linrrpad_idxrr _reset_params)rr r rgrhrrrSrTrUrVrWrXr?rYrZr[r\r]r^r_r`rarbrircrdrerfrrrrrr sj !      zTransformerG2P.__init__c Cs`|\}}|durt|d|j}n|\}}||}|jr%t|j||}||} ||} | | } |j | | ||j d\} } } }d}|j dkrO| | }n|jdkr]| | | } d}|j| | | |d\}}|j dkr| || } | || } d}d}n|jdkr| | | } d}d}|j| || || | ||d\}}}||}||}||||fS)auComputes the forward pass Arguments --------- grapheme_encoded: torch.Tensor graphemes encoded as a Torch tensor phn_encoded: torch.Tensor the encoded phonemes word_emb: torch.Tensor word embeddings (if applicable) Returns ------- p_seq: torch.Tensor the log-probabilities of individual tokens i a sequence char_lens: torch.Tensor the character length syntax encoder_out: torch.Tensor the encoder state attention: torch.Tensor the attention state Nrri RelPosMHAXLrO)srcsrc_masksrc_key_padding_maskpos_embs)tgtmemory memory_masktgt_masktgt_key_padding_maskmemory_key_padding_mask pos_embs_tgt pos_embs_src)rrrr rrrrgr rh make_masksrir`r[positional_encoding_typeencoderpositional_encoding_decoderdecoderrr)rrrrr r!phnr#r$rmrqrorurnrtpos_embs_encoderr%pos_embs_target decoder_out attentionr)r*rrrr+^sd           zTransformerG2P.forwardcCs,|D]}|dkrtjj|qdS)z"Resets the parameters of the modelN) parametersr.r/rinitxavier_normal_)rprrrrjs  zTransformerG2P._reset_paramsc Csr|dur't||jd}t|jddddf||dddfk}t||d}d}t|} |||| fS)aThis method generates the masks for training the transformer model. Arguments --------- src : torch.Tensor The sequence to the encoder (required). tgt : torch.Tensor The sequence to the decoder (required). src_len : torch.Tensor Lengths corresponding to the src tensor. pad_idx : int The index for token (default=0). Returns ------- src_key_padding_mask: torch.Tensor the source key padding mask tgt_key_padding_mask: torch.Tensor the target key padding masks src_mask: torch.Tensor the source mask tgt_mask: torch.Tensor the target mask Nrrk)r/roundshapearangetorr) rrmrqsrc_lenriabs_lenrorurnrtrrrrys   zTransformerG2P.make_masksc Cst|}||}||}|jdkr"|||}|||}n |jdkr.|||}|j|||ddd\}}}|d}||fS)aXThis method implements a decoding step for the transformer model. Arguments --------- tgt : torch.Tensor The sequence to the decoder. encoder_out : torch.Tensor Hidden output of the encoder. enc_lens : torch.Tensor The corresponding lengths of the encoder inputs. Returns ------- prediction: torch.Tensor the predicted sequence attention: torch.Tensor the attention matrix corresponding to the last attention head (useful for plotting attention) rlrON)rtrwrxr,)rr rhr`r|rzr[r}) rrqr%enc_lensrt prediction self_attnsmultihead_attnsrrrrdecodes&     zTransformerG2P.decoder1)Nr) r2r3r4r5rReLUr r+rjryrr6rrrrrLs>P DU &rLcCs |||S)abComputes the input dimension (intended for hparam files) Arguments --------- use_word_emb: bool whether to use word embeddings embedding_dim: int the embedding dimension word_emb_enc_dim: int the dimension of encoded word embeddings Returns ------- input_dim: int the input dimension r)r embedding_dimr;rrr input_dim s rcCs*|dur ||jn|j}tj||gddS)a Concatenates character and word embeddings together, possibly applying a custom encoding/transformation Arguments --------- word_emb_enc: callable an encoder to apply (typically, speechbrain.lobes.models.g2p.model.WordEmbeddingEncoder) emb_char: torch.Tensor character embeddings word_emb: char word embeddings Returns ------- result: torch.Tensor the resulting (concatenated) tensor Nr,r-)datar/r0)rr$rrrrrs  rcCstjdg|d|dS)z Creates a dummy phoneme sequence Arguments --------- batch_size: int the batch size device: str the target device Returns ------- result: torch.Tensor r)rr)r/tensorexpand) batch_sizerrrrr8sr)r5r/r0speechbrain.lobes.models.transformer.Transformerrrrspeechbrain.nnetrspeechbrain.nnet.linearrModulerr7rLrrrrrrrs   rK: