o p’×iž|ã@sèdZddlZddlmZddlZddlZddlmZddl mm Z ddl m Z ddlmZeeƒZGdd„dejƒZGdd „d ejƒZGd d „d ejƒZGd d „d ejƒZGdd„dejƒZGdd„dejƒZGdd„dejƒZdS)zŒLibrary implementing attention modules. Authors * Ju-Chieh Chou 2020 * Jianyuan Zhong 2020 * Loren Lugosch 2020 * Samuele Cornell 2020 éN)ÚOptional)Úlength_to_mask)Ú get_loggercs2eZdZdZd ‡fdd„ Zdd„Zdd„Z‡ZS) ÚContentBasedAttentiona”This class implements content-based attention module for seq2seq learning. Reference: NEURAL MACHINE TRANSLATION BY JOINTLY LEARNING TO ALIGN AND TRANSLATE, Bahdanau et.al. https://arxiv.org/pdf/1409.0473.pdf Arguments --------- enc_dim : int Size of encoder layer. dec_dim : int Size of decoder layer. attn_dim : int Size of the attention feature. output_dim : int Size of the output context vector. scaling : float The factor controls the sharpening degree (default: 1.0). Example ------- >>> enc_tensor = torch.rand([4, 10, 20]) >>> enc_len = torch.ones([4]) * 10 >>> dec_tensor = torch.rand([4, 25]) >>> net = ContentBasedAttention(enc_dim=20, dec_dim=25, attn_dim=30, output_dim=5) >>> out_tensor, out_weight = net(enc_tensor, enc_len, dec_tensor) >>> out_tensor.shape torch.Size([4, 5]) çð?csftƒ ¡t ||¡|_t ||¡|_tj|ddd|_t ||¡|_||_tj dd|_ |  ¡dS)NéF©Úbiaséÿÿÿÿ©Údim) ÚsuperÚ__init__ÚnnÚLinearÚmlp_encÚmlp_decÚmlp_attnÚmlp_outÚscalingÚSoftmaxÚsoftmaxÚreset)ÚselfÚenc_dimÚdec_dimÚattn_dimÚ output_dimr©Ú __class__©úX/home/ubuntu/SoloSpeech/.venv/lib/python3.10/site-packages/speechbrain/nnet/attention.pyr7s  zContentBasedAttention.__init__cCód|_d|_d|_dS©z)Reset the memory in the attention module.N)Úenc_lenÚprecomputed_enc_hÚmask©rr r r!rFó zContentBasedAttention.resetcCsª|jdur| |¡|_t|| d¡|jd|_| | d¡¡}| t   |j|¡¡  d¡}|  |jdkt j ¡}| ||j¡}t  | d¡|¡  d¡}| |¡}||fS)áReturns the output of the attention module. Arguments --------- enc_states : torch.Tensor The tensor to be attended. enc_len : torch.Tensor The real length (without padding) of enc_states for each sentence. dec_states : torch.Tensor The query tensor. Returns ------- The output of the attention module. Nr©Úmax_lenÚdevicer r)r%rrÚsizer,r&rÚ unsqueezerÚtorchÚtanhÚsqueezeÚ masked_fillÚnpÚinfrrÚbmmr)rÚ enc_statesr$Ú dec_statesÚdec_hÚattnÚcontextr r r!ÚforwardLs  ÿÿþ zContentBasedAttention.forward©r©Ú__name__Ú __module__Ú __qualname__Ú__doc__rrr;Ú __classcell__r r rr!rs rcsDeZdZUdZeejed< d ‡fdd„ Zdd„Z dd „Z ‡Z S) ÚLocationAwareAttentiona{This class implements location-aware attention module for seq2seq learning. Reference: Attention-Based Models for Speech Recognition, Chorowski et.al. https://arxiv.org/pdf/1506.07503.pdf Arguments --------- enc_dim : int Size of encoder. dec_dim : int Size of decoder. attn_dim : int Size of the attention feature. output_dim : int Size of the output context vector. conv_channels : int Number of channel for location feature. kernel_size : int Kernel size of convolutional layer for location feature. scaling : float The factor controls the sharpening degree (default: 1.0). Example ------- >>> enc_tensor = torch.rand([4, 10, 20]) >>> enc_len = torch.ones([4]) * 10 >>> dec_tensor = torch.rand([4, 25]) >>> net = LocationAwareAttention( ... enc_dim=20, ... dec_dim=25, ... attn_dim=30, ... output_dim=5, ... conv_channels=10, ... kernel_size=100) >>> out_tensor, out_weight = net(enc_tensor, enc_len, dec_tensor) >>> out_tensor.shape torch.Size([4, 5]) r%rcs¤tƒ ¡t ||¡|_t ||¡|_tj|ddd|_tjd|d|d|dd|_t ||¡|_ tj|ddd|_t ||¡|_ ||_ tj dd|_ | ¡dS)NrFré)Ú kernel_sizeÚpaddingr r r )r rrrrrrÚConv1dÚconv_locÚmlp_locrrrrr)rrrrrÚ conv_channelsrErrr r!ržs"  û zLocationAwareAttention.__init__cCsd|_d|_d|_d|_dS)z%Reset the memory in attention module.N)r$r%r&Ú prev_attnr'r r r!r¿s zLocationAwareAttention.resetcCsö|jdur$| |¡|_t|| d¡|jd|_|jd| ¡ d¡|_|  |j d¡¡}|  |  dd¡¡}|  | d¡¡}|  t |j||¡¡ d¡}| |jdktj ¡}| ||j¡}| ¡|_t | d¡|¡ d¡}| |¡}||fS)r)Nrr*rDr r)r%rrr-r,r&Úfloatr.rKrHrIÚ transposerrr/r0r1r2r3r4rrÚdetachr5r)rr6r$r7Ú attn_convr8r9r:r r r!r;Æs(  ÿÿþ  zLocationAwareAttention.forwardr<) r>r?r@rArr/ÚTensorÚ__annotations__rrr;rBr r rr!rCts ' ø!rCcs0eZdZdZ‡fdd„Zdd„Zdd„Z‡ZS)ÚKeyValueAttentionaeThis class implements a single-headed key-value attention module for seq2seq learning. Reference: "Attention Is All You Need" by Vaswani et al., sec. 3.2.1 Arguments --------- enc_dim : int Size of the encoder feature vectors from which keys and values are computed. dec_dim : int Size of the decoder feature vectors from which queries are computed. attn_dim : int Size of the attention feature. output_dim : int Size of the output context vector. Example ------- >>> enc_tensor = torch.rand([4, 10, 20]) >>> enc_len = torch.ones([4]) * 10 >>> dec_tensor = torch.rand([4, 25]) >>> net = KeyValueAttention(enc_dim=20, dec_dim=25, attn_dim=30, output_dim=5) >>> out_tensor, out_weight = net(enc_tensor, enc_len, dec_tensor) >>> out_tensor.shape torch.Size([4, 5]) csVtƒ ¡t ||¡|_t ||¡|_t ||¡|_t t  |¡  ¡¡|_ |  ¡dS)N) r rrrÚ key_linearÚ query_linearÚ value_linearr/ÚsqrtÚtensorrLrr)rrrrrrr r!rs  zKeyValueAttention.__init__cCr"r#)ÚvaluesÚkeysr&r'r r r!r r(zKeyValueAttention.resetcCs¨|jdur | |¡|_| |¡|_t|| d¡|jd d¡|_|  |¡ d¡}t   |j|¡|j }|  |jdktj ¡}| d¡ dd¡}t   ||j¡ d¡}||fS)r)Nrr*rDr)rYrSrUrXrr-r,r.r&rTr/Úmatmulrr2r3r4rrMr1)rr6r$r7ÚqueryÚscoresÚnormalized_scoresÚoutr r r!r;&s   ÿþzKeyValueAttention.forwardr=r r rr!rRùs   rRcsXeZdZdZejfdedejf‡fdd„ Ze  ¡defdd„ƒZ d ej fd d „Z ‡Z S) Ú RelPosEncXLaÕRelative positional encoding for the :class:`~RelPosMHAXL`. Arguments --------- emb_dim : int Size of the embedding, which controls the size of the last dimension of the positional embedding as well dtype : torch.dtype, optional If unspecified, defaults to `torch.float32`. Controls the data type of the output embedding (but does not affect the precision of the computations, which remain `torch.float32`). Úemb_dimÚdtypecsTtƒ ¡||_t tjd|jdtjdt d¡|j ¡}|  d|¡||_ dS)NrrD)ragˆÃ@Úinv_freq) r rr`r/ÚexpÚarangeÚfloat32ÚmathÚlogÚregister_bufferÚ emb_dtype)rr`rarbrr r!rTs ÿÿ  zRelPosEncXL.__init__Úseq_lenc CsB|j}|jj}t ¡Œtjd||jftj|d}|d}|d}tjd|tj|d  d¡}t  ||j¡}||dd…ddd…f<t  ||j¡|dd…ddd…f<||dd…ddd…f<t  | |j¡|dd…ddd…f<t  |d¡  d¡}|dd…  d¡}tj ||gdd} |  |¡} Wdƒ| S1sšwY| S) ab Builds the positional embedding tensor for a given sequence length. Arguments --------- seq_len : int The length of the sequence to create the position embedding for. Returns ------- torch.Tensor Positional embedding tensor of shape `[1, 2*seq_len-1, embed_dim]` rD)rar,rrr N)rr )rirbr,r/Úno_gradÚemptyr`rerdr.ÚsinÚcosÚflipÚcatÚto) rrjrir,Útot_peÚpe_pastÚ pe_futureÚ positionsÚ sinusoidsÚper r r!Úmake_pe`s>  ýüû"$  äâzRelPosEncXL.make_peÚxcCs|j| d¡dS)aº Builds the positional embedding tensor. Similar to :meth:`~RelPosEncXL.make_pe` but uses the shape information from the provided tensor. Arguments --------- x : torch.Tensor input tensor with shape batch_size, seq_len, embed_dim Returns ------- pos_emb : torch.Tensor Positional embedding tensor of shape `[1, 2*seq_len-1, embed_dim]` r)rj)rxr-©rryr r r!r;“szRelPosEncXL.forward)r>r?r@rAr/reÚintrarrkrxrPr;rBr r rr!r_Fs   2r_csJeZdZdZ    d‡fdd„ Zdd„Zd d „Z   dd d „Z‡ZS)Ú RelPosMHAXLa÷This class implements the relative multihead implementation similar to that in Transformer XL https://arxiv.org/pdf/1901.02860.pdf Arguments --------- embed_dim : int Size of the encoder feature vectors from which keys and values are computed. num_heads: int Number of attention heads. dropout : float, optional Dropout rate. vbias: bool, optional Whether to use bias for computing value. vdim: int, optional Size for value. Default is embed_dim (Note each head is embed_dim // num_heads). mask_pos_future: bool, optional Whether to mask future positional encodings values. Must be true for causal applications e.g. decoder. Example ------- >>> inputs = torch.rand([6, 60, 512]) >>> pos_emb = torch.rand([1, 2*60-1, 512]) >>> net = RelPosMHAXL(num_heads=8, embed_dim=inputs.shape[-1]) >>> outputs, attn = net(inputs, inputs, inputs, pos_emb) >>> outputs.shape torch.Size([6, 60, 512]) çFNcs¢tƒ ¡||_|dur|n||_|j|k|_||_||_||_||_|||_ |j||_ |j ||jks:Jdƒ‚|j ||jksFJdƒ‚|jdurct   t  d||¡¡|_t   t  |j|¡¡|_n t   t  d||¡¡|_|r|t   t  |j¡¡|_nd|_t  |¡|_t  |j|¡|_t j||dd|_t   t  |j |j¡¡|_t   t  |j |j¡¡|_t| ¡ƒjt jkr¼d|_ntdƒ |_|  ¡d t! "|j¡|_#dS) Nz(embed_dim must be divisible by num_headsz#vdim must be divisible by num_headsFrDériÿÿr4r)$r rÚ embed_dimÚvdimÚ_qkv_same_embed_dimÚmask_pos_futureÚvbiasÚ num_headsÚdropoutÚhead_dimÚ vhead_dimrÚ Parameterr/rlÚqk_proj_weightÚ v_proj_weightÚin_proj_weightÚvalue_bias_weightÚDropoutÚ dropout_attrÚout_projÚ linear_posÚ pos_bias_uÚ pos_bias_vÚnextÚ parametersraÚfloat16Úattn_fill_valuerLÚ_reset_parametersrfrVÚscale)rrr„r…rƒr€r‚rr r!rÅsR    ÿÿ ÿÿ ÿÿ zRelPosMHAXL.__init__cCsx|jr tjj |j¡ntjj |j¡tjj |j¡|jdur*tjj  |j d¡tjj |j ¡tjj |j ¡dS)Nr}) rr/rÚinitÚxavier_uniform_r‹r‰rŠrƒÚ constant_rŒr‘r’r'r r r!r—s zRelPosMHAXL._reset_parameterscCsÊ| ¡\}}}}tjjj|dd}| ||d|¡}|dd…dd…dd…f ||||¡}|jrYtj| d¡| d¡f|jd}|t  || d¡| d¡¡dddd…dd…f}|d d|dd…fS) zRelative shift implementation.)rr)Úpadr NrrDr~)r,.) r-r/rÚ functionalrœÚviewr‚Úonesr,Útril)rryÚbÚhÚqlenÚpos_lenrŸr r r!Ú rel_shifts& 4zRelPosMHAXL.rel_shiftTc Cs|jd}|jd} |jd} |jrz||ust ||¡rA||us&t ||¡rAtj ||j¡ |d|j |j d¡j ddd\}}}n;|jj ddd\} } } tj || ¡ |d|j |j ¡}tj || ¡ |d|j |j ¡}tj || ¡ |d|j |j ¡}nt ‚|jdurŽ||j dd|j |j¡}| |¡ dd|j |j ¡}||j dd|j |j ¡ dd¡}||j dd|j |j ¡ dd¡}t ||j| dddd¡¡}t ||j| dddd¡¡}| |¡}||}|dur|jdkr÷| dd| | ¡}n | d|j | | ¡}|jtjkr| ||j¡}n||7}|dur%| | |dd| ¡|j¡}tj |dtj!d}| "|¡}|durG|jtjkrF| |d ¡}n |durX| | |dd| ¡d ¡}t || dd¡¡}| dd¡ #¡ |d|j|j ¡}| $|¡}|r||fS|S) aX Compute attention. Arguments --------- query : torch.Tensor (B, L, E) where L is the target sequence length, B is the batch size, E is the embedding dimension. key : torch.Tensor (B, S, E) where S is the source sequence length, B is the batch size, E is the embedding dimension. value : torch.Tensor (B, S, E) where S is the source sequence length, B is the batch size, E is the embedding dimension. pos_embs : torch.Tensor bidirectional sinusoidal positional embedding tensor (1, 2*S-1, E) where S is the max length between source and target sequence lengths, and E is the embedding dimension. key_padding_mask : torch.Tensor (B, S) where B is the batch size, S is the source sequence length. If a ByteTensor is provided, the non-zero positions will be ignored while the position with the zero positions will be unchanged. If a BoolTensor is provided, the positions with the value of True will be ignored while the position with the value of False will be unchanged. attn_mask : torch.Tensor 2D mask (L, S) where L is the target sequence length, S is the source sequence length. 3D mask (N*num_heads, L, S) where N is the batch size, L is the target sequence length, S is the source sequence length. attn_mask ensure that position i is allowed to attend the unmasked positions. If a ByteTensor is provided, the non-zero positions are not allowed to attend while the zero positions will be unchanged. If a BoolTensor is provided, positions with True is not allowed to attend while False values will be unchanged. If a FloatTensor is provided, it will be added to the attention weight. return_attn_weights : bool Whether to additionally return the attention weights. Returns ------- out : torch.Tensor (B, L, E) where L is the target sequence length, B is the batch size, E is the embedding dimension. attn_score : torch.Tensor (B, L, S) where B is the batch size, L is the target sequence length, S is the source sequence length. rrr r~r rDN)r rar})%Úshaperr/ÚequalrrÚlinearr‹ržr„r†ÚchunkÚNotImplementedErrorr‰rŠr‡rƒrŒrr‘rMr’rZr˜Úpermuter¥ÚndimraÚboolr2r–ÚFrrerŽÚ contiguousr)rr[ÚkeyÚvalueÚpos_embsÚkey_padding_maskÚ attn_maskÚreturn_attn_weightsÚbszÚklenr£ÚqweightÚkweightÚvweightÚp_kÚ q_with_bias_uÚ q_with_bias_vÚ matrix_acÚ matrix_bdÚ attn_scoreryr^r r r!r;$sž :  þ ÿ ÿ ÿ ÿ  ÿ  ÿþþ ÿÿ   ÿ þ   þ ÿ ý zRelPosMHAXL.forward)r}FNF)NNT) r>r?r@rArr—r¥r;rBr r rr!r|§s!ù?ør|c sdeZdZdZ      d‡fdd„ Z    ddeejd eejd ed eejfd d „Z ‡Z S)ÚMultiheadAttentionaßThe class is a wrapper of MultiHead Attention for torch.nn.MultiHeadAttention. Reference: https://pytorch.org/docs/stable/nn.html Arguments --------- nhead : int parallel attention heads. d_model : int The size of the model layers. dropout : float a Dropout layer on attn_output_weights (default: 0.0). bias : bool add bias as module parameter (default: True). add_bias_kv : bool add bias to the key and value sequences at dim=0. add_zero_attn : bool add a new batch of zeros to the key and value sequences at dim=1. kdim : int total number of features in key (default: None). vdim : int total number of features in value (default: None). Example ------- >>> inputs = torch.rand([8, 60, 512]) >>> net = MultiheadAttention(nhead=8, d_model=inputs.shape[-1]) >>> outputs, attn = net(inputs, inputs, inputs) >>> outputs.shape torch.Size([8, 60, 512]) r}TFNc s*tƒ ¡tj||||||||d|_dS)N)rr„r…r Ú add_bias_kvÚ add_zero_attnÚkdimr€)r rrrÁÚatt) rÚnheadÚd_modelr…r rÂrÃrÄr€rr r!rs  øzMultiheadAttention.__init__r´r³rµr²c Cs€| ddd¡}| ddd¡}| ddd¡}|dur$|dur"||7}n|}|j||||||d\}} | ddd¡}|r>|| fS|S)a Compute attention. Arguments --------- query : torch.Tensor (B, L, E) where L is the target sequence length, B is the batch size, E is the embedding dimension. key : torch.Tensor (B, S, E) where S is the source sequence length, B is the batch size, E is the embedding dimension. value : torch.Tensor (B, S, E) where S is the source sequence length, B is the batch size, E is the embedding dimension. attn_mask : torch.Tensor, optional 2D mask (L, S) where L is the target sequence length, S is the source sequence length. 3D mask (N*num_heads, L, S) where N is the batch size, L is the target sequence length, S is the source sequence length. attn_mask ensure that position i is allowed to attend the unmasked positions. If a ByteTensor is provided, the non-zero positions are not allowed to attend while the zero positions will be unchanged. If a BoolTensor is provided, positions with True is not allowed to attend while False values will be unchanged. If a FloatTensor is provided, it will be added to the attention weight. key_padding_mask : torch.Tensor, optional (B, S) where B is the batch size, S is the source sequence length. If a ByteTensor is provided, the non-zero positions will be ignored while the position with the zero positions will be unchanged. If a BoolTensor is provided, the positions with the value of True will be ignored while the position with the value of False will be unchanged. return_attn_weights : bool, optional True to additionally return the attention weights, False otherwise. pos_embs : torch.Tensor, optional Positional embeddings added to the attention map of shape (L, S, E) or (L, S, 1). Returns ------- attn_output : torch.Tensor (B, L, E) where L is the target sequence length, B is the batch size, E is the embedding dimension. attn_output_weights : torch.Tensor (B, L, S) where B is the batch size, L is the target sequence length, S is the source sequence length. This is returned only if `return_attn_weights=True` (True by default). rrrDN)r´r³Ú need_weights)r«rÅ) rr[r°r±r´r³rµr²ÚoutputÚattention_weightsr r r!r;s&9  ú zMultiheadAttention.forward)r}TFFNN)NNTN) r>r?r@rArrr/rPr­r;rBr r rr!rÁâs,$÷øûúùørÁcs4eZdZdZdddejf‡fdd„ Zdd„Z‡ZS)ÚPositionalwiseFeedForwarduThe class implements the positional-wise feed forward module in “Attention Is All You Needâ€. Arguments --------- d_ffn: int Hidden layer size. input_shape : tuple, optional Expected shape of the input. Alternatively use ``input_size``. input_size : int, optional Expected size of the input. Alternatively use ``input_shape``. dropout: float, optional Dropout rate. activation: torch.nn.Module, optional activation functions to be applied (Recommendation: ReLU, GELU). Example ------- >>> inputs = torch.rand([8, 60, 512]) >>> net = PositionalwiseFeedForward(256, input_size=inputs.shape[-1]) >>> outputs = net(inputs) >>> outputs.shape torch.Size([8, 60, 512]) Nr}c s`tƒ ¡|dur|durtdƒ‚|dur|d}t t ||¡|ƒt |¡t ||¡¡|_dS)Nz)Expected one of input_shape or input_sizer )r rÚ ValueErrorrÚ SequentialrrÚffn)rÚd_ffnÚ input_shapeÚ input_sizer…Ú activationrr r!rŒs    üz"PositionalwiseFeedForward.__init__cCs*| ddd¡}| |¡}| ddd¡}|S)z8Applies PositionalwiseFeedForward to the input tensor x.rrrD)r«rÎrzr r r!r;£s z!PositionalwiseFeedForward.forward) r>r?r@rArÚReLUrr;rBr r rr!rËrsúrË)rArfÚtypingrÚnumpyr3r/Útorch.nnrÚtorch.nn.functionalrr®Úspeechbrain.dataio.dataiorÚspeechbrain.utils.loggerrr>ÚloggerÚModulerrCrRr_r|rÁrËr r r r!Ús*    \Ma=