o %ݫiu:@sdZddlmZddlZddlmZddlmZddlmZm Z m Z m Z m Z ddl mZddlmZdd lmZdd lmZeeZGd d d eZdS) zLTransformer for ST in the SpeechBrain style. Authors * YAO FEI, CHENG 2021 )OptionalN)nn)ConformerEncoder)NormalizedEmbeddingTransformerDecoderTransformerEncoderget_key_padding_maskget_lookahead_mask)TransformerASR)Swish) ModuleList) get_loggercseZdZdZddddddejddd d d ed d d dddddfdeedee dee deej dee deedee de de de dedeffdd Z d(ddZd(d d!Zd(d"d#Zd$d%Zd(d&d'ZZS)) TransformerSTa This is an implementation of transformer model for ST. The architecture is based on the paper "Attention Is All You Need": https://arxiv.org/pdf/1706.03762.pdf Arguments --------- tgt_vocab: int Size of vocabulary. input_size: int Input feature size. d_model : int, optional Embedding dimension size. (default=512). nhead : int, optional The number of heads in the multi-head attention models (default=8). num_encoder_layers : int, optional The number of sub-encoder-layers in the encoder (default=6). num_decoder_layers : int, optional The number of sub-decoder-layers in the decoder (default=6). d_ffn : int, optional The dimension of the feedforward network model (default=2048). dropout : int, optional The dropout value (default=0.1). activation : torch.nn.Module, optional The activation function of FFN layers. Recommended: relu or gelu (default=relu). positional_encoding: str, optional Type of positional encoding used. e.g. 'fixed_abs_sine' for fixed absolute positional encodings. normalize_before: bool, optional Whether normalization should be applied before or after MHA or FFN in Transformer layers. Defaults to True as this was shown to lead to better performance and training stability. kernel_size: int, optional Kernel size in convolutional layers when Conformer is used. bias: bool, optional Whether to use bias in Conformer convolutional layers. encoder_module: str, optional Choose between Conformer and Transformer for the encoder. The decoder is fixed to be a Transformer. conformer_activation: torch.nn.Module, optional Activation module used after Conformer convolutional layers. E.g. Swish, ReLU etc. it has to be a torch Module. attention_type: str, optional Type of attention layer used in all Transformer or Conformer layers. e.g. regularMHA or RelPosMHA. max_length: int, optional Max length for the target and source sequence in input. Used for positional encodings. causal: bool, optional Whether the encoder should be causal or not (the decoder is always causal). If causal the Conformer convolutional layer is causal. ctc_weight: float The weight of ctc for asr task asr_weight: float The weight of asr task for calculating loss mt_weight: float The weight of mt task for calculating loss asr_tgt_vocab: int The size of the asr target language mt_src_vocab: int The size of the mt source language Example ------- >>> src = torch.rand([8, 120, 512]) >>> tgt = torch.randint(0, 720, [8, 120]) >>> net = TransformerST( ... 720, 512, 512, 8, 1, 1, 1024, activation=torch.nn.GELU, ... ctc_weight=1, asr_weight=0.3, ... ) >>> enc_out, dec_out = net.forward(src, tgt) >>> enc_out.shape torch.Size([8, 120, 512]) >>> dec_out.shape torch.Size([8, 120, 512]) iig?fixed_abs_sineFT transformer regularMHAi gr kernel_sizebiasencoder_moduleconformer_activationattention_type max_lengthcausal ctc_weight asr_weight mt_weight asr_tgt_vocab mt_src_vocabc sJtjdid|d|d|d|d|d|d|d|d | d | d | d | d | d|d|d|d|d||dkr[|dkr[t|||||| | ddd |_tt|||_|dkrtt|||_|dkr|t|||||| | |j |j d |_ n#|dkrt ||||||| | |j |j d |_ | sJd|dusJd| dS)N tgt_vocab input_sized_modelnheadnum_encoder_layersnum_decoder_layersd_ffndropout activationpositional_encodingnormalize_beforerrrrrrrrTr) num_layersr$r'r#r(r)r+rrr) r$r-r'r#r(r)r+rr conformer) r$r-r'r#r(r)rrrrz+normalize_before must be True for Conformerz%conformer_activation must not be None)super__init__r asr_decoderr rcustom_asr_tgt_modulecustom_mt_src_modulerrr mt_encoderr _init_params)selfr!r"r#r$r%r&r'r(r)r*r+rrrrrrrrrrrr  __class__r/f/home/ubuntu/.local/lib/python3.10/site-packages/speechbrain/lobes/models/transformer/TransformerST.pyr1hs            zTransformerST.__init__cCs|dkr|j\}}}} ||||| }|j||||d\} } } } ||}|jdkr5|||}n |jdkrA|||}|j||| | | | d\}}}|S)adThis method implements a decoding step for asr task Arguments --------- encoder_out : torch.Tensor The representation of the encoder (required). src : torch.Tensor Input sequence (required). tgt : torch.Tensor The sequence to the decoder (transcription) (required). wav_len : torch.Tensor Length of input tensors (required). pad_idx : int The index for token (default=0). Returns ------- asr_decoder_out : torch.Tensor One step of asr decoder. pad_idx RelPosMHAXLrtgtmemory memory_masktgt_masktgt_key_padding_maskmemory_key_padding_mask) dimshapereshape make_masksr3rpositional_encoding_decoderr*r2)r7 encoder_outsrcr@wav_lenr=bztch1ch2src_key_padding_maskrDsrc_maskrC transcriptionasr_decoder_out_r/r/r: forward_asrs6      zTransformerST.forward_asrc Cs|j|||d\}}}}||}|jdkr||}n|jdkr*|||}d}|j||||d\} } ||}|jdkrN|||}|||}n |jdkrZ|||}|j|| ||||d\} } } | | fS)aThis method implements a forward step for mt task Arguments --------- src : torch.Tensor The sequence to the encoder (transcription) (required). tgt : torch.Tensor The sequence to the decoder (translation) (required). pad_idx : int The index for token (default=0). Returns ------- encoder_out : torch.Tensor Output of encoder decoder_out : torch.Tensor Output of decoder r<r>rN)rLrSrRpos_embsr?) make_masks_for_mtr4rr*positional_encoding_typer5custom_tgt_modulerJdecoder) r7rLr@r=rRrDrSrCpos_embs_encoderrKrV decoder_outr/r/r: forward_mtsB         zTransformerST.forward_mtc Cst|j|||d\}}}}||}|jdkr|||}n |jdkr*|||}|j||||||d\}} } |S)aThis method implements a forward step for mt task using a wav2vec encoder (same than above, but without the encoder stack) Arguments ---------- src (transcription): torch.Tensor output features from the w2v2 encoder tgt (translation): torch.Tensor The sequence to the decoder (required). pad_idx : int The index for token (default=0). r<r>rr?)rYr[rrJrZr*r\) r7rLr@r=rRrDrSrCr^rV multiheadr/r/r:forward_mt_decoder_onlyDs(     z%TransformerST.forward_mt_decoder_onlycCstt|}||}|jdkr|||}|||}n |jdkr)|||}|j|||d\}}}||dfS)aThis method implements a decoding step for the transformer model. Arguments --------- tgt : torch.Tensor The sequence to the decoder. encoder_out : torch.Tensor Hidden output of the encoder. Returns ------- prediction : torch.Tensor The predicted outputs. multihead_attns : torch.Tensor The last step of attention. r>r)rC)r r[rrJrZr*r2)r7r@rKrC predictionrVmultihead_attnsr/r/r: decode_asrls    zTransformerST.decode_asrcCs:d}|jr t||d}t||d}d}t|}||||fS)aThis method generates the masks for training the transformer model. Arguments --------- src : torch.Tensor The sequence to the encoder (required). tgt : torch.Tensor The sequence to the decoder (required). pad_idx : int The index for token (default=0). Returns ------- src_key_padding_mask : torch.Tensor Timesteps to mask due to padding tgt_key_padding_mask : torch.Tensor Timesteps to mask due to padding src_mask : torch.Tensor Timesteps to mask for causality tgt_mask : torch.Tensor Timesteps to mask for causality Nr<)trainingrr )r7rLr@r=rRrDrSrCr/r/r:rYs   zTransformerST.make_masks_for_mt)r)__name__ __module__ __qualname____doc__rReLUr rintboolstrModulefloatr1rWr_rarerY __classcell__r/r/r8r:rsjO  g 7 >("r)rjtypingrtorchr.speechbrain.lobes.models.transformer.Conformerr0speechbrain.lobes.models.transformer.Transformerrrrrr 3speechbrain.lobes.models.transformer.TransformerASRr speechbrain.nnet.activationsr speechbrain.nnet.containersr speechbrain.utils.loggerr rgloggerrr/r/r/r:s