o %ݫi @sdZddlZddlmZmZmZmZmZddlZ ddl Z ddl m Z ddl m mZddlmZddlmZeeZGddde jZGdd d e jZGd d d e jZGd d d e jZGddde jZGddde jZGddde jZGddde jZGdddZdeegefdeegeffddZ e ddde!de!d e j"d!e j#def d"d#Z$d$d%Z%Gd&d'd'e jZ&d(d)Z'dS)*zLibrary implementing attention modules. Authors * Ju-Chieh Chou 2020 * Jianyuan Zhong 2020 * Loren Lugosch 2020 * Samuele Cornell 2020 * Shucong Zhang 2024 N)AnyCallableDictOptionalTuple)length_to_mask) get_loggercs2eZdZdZd fdd ZddZddZZS) ContentBasedAttentionaThis class implements content-based attention module for seq2seq learning. Reference: NEURAL MACHINE TRANSLATION BY JOINTLY LEARNING TO ALIGN AND TRANSLATE, Bahdanau et.al. https://arxiv.org/pdf/1409.0473.pdf Arguments --------- enc_dim : int Size of encoder layer. dec_dim : int Size of decoder layer. attn_dim : int Size of the attention feature. output_dim : int Size of the output context vector. scaling : float The factor controls the sharpening degree (default: 1.0). Example ------- >>> enc_tensor = torch.rand([4, 10, 20]) >>> enc_len = torch.ones([4]) * 10 >>> dec_tensor = torch.rand([4, 25]) >>> net = ContentBasedAttention(enc_dim=20, dec_dim=25, attn_dim=30, output_dim=5) >>> out_tensor, out_weight = net(enc_tensor, enc_len, dec_tensor) >>> out_tensor.shape torch.Size([4, 5]) ?csftt|||_t|||_tj|ddd|_t|||_||_tj dd|_ | dS)NFbiasdim) super__init__nnLinearmlp_encmlp_decmlp_attnmlp_outscalingSoftmaxsoftmaxreset)selfenc_dimdec_dimattn_dim output_dimr __class__N/home/ubuntu/.local/lib/python3.10/site-packages/speechbrain/nnet/attention.pyr8s  zContentBasedAttention.__init__cCd|_d|_d|_dSz)Reset the memory in the attention module.N)enc_lenprecomputed_enc_hmaskrr$r$r%rG zContentBasedAttention.resetcCs|jdur|||_t||d|jd|_||d}|t |j| d}| |jdkt j }|||j}t |d| d}||}||fS)Returns the output of the attention module. Arguments --------- enc_states : torch.Tensor The tensor to be attended. enc_len : torch.Tensor The real length (without padding) of enc_states for each sentence. dec_states : torch.Tensor The query tensor. Returns ------- The output of the attention module. Nr max_lendevicerr)r)rrsizer0r*r unsqueezertorchtanhsqueeze masked_fillnpinfrrbmmr)r enc_statesr( dec_statesdec_hattncontextr$r$r%forwardMs   zContentBasedAttention.forwardr __name__ __module__ __qualname____doc__rrr? __classcell__r$r$r"r%r s r csDeZdZUdZeejed< d fdd ZddZ dd Z Z S) LocationAwareAttentiona{This class implements location-aware attention module for seq2seq learning. Reference: Attention-Based Models for Speech Recognition, Chorowski et.al. https://arxiv.org/pdf/1506.07503.pdf Arguments --------- enc_dim : int Size of encoder. dec_dim : int Size of decoder. attn_dim : int Size of the attention feature. output_dim : int Size of the output context vector. conv_channels : int Number of channel for location feature. kernel_size : int Kernel size of convolutional layer for location feature. scaling : float The factor controls the sharpening degree (default: 1.0). Example ------- >>> enc_tensor = torch.rand([4, 10, 20]) >>> enc_len = torch.ones([4]) * 10 >>> dec_tensor = torch.rand([4, 25]) >>> net = LocationAwareAttention( ... enc_dim=20, ... dec_dim=25, ... attn_dim=30, ... output_dim=5, ... conv_channels=10, ... kernel_size=100) >>> out_tensor, out_weight = net(enc_tensor, enc_len, dec_tensor) >>> out_tensor.shape torch.Size([4, 5]) r)r cstt|||_t|||_tj|ddd|_tjd|d|d|dd|_t|||_ tj|ddd|_t|||_ ||_ tj dd|_ |dS)Nr Fr ) kernel_sizepaddingr rr)rrrrrrrConv1dconv_locmlp_locrrrrr)rrrr r! conv_channelsrIrr"r$r%rs"   zLocationAwareAttention.__init__cCsd|_d|_d|_d|_dS)z%Reset the memory in attention module.N)r(r)r* prev_attnr+r$r$r%rs zLocationAwareAttention.resetcCs|jdur$|||_t||d|jd|_|jd|d|_| |jd}| | dd}| |d}| t|j||d}||jdktj }|||j}||_t|d|d}||}||fS)r-Nr r.rHrr)r)rrr1r0r*floatr2rOrLrM transposerrr3r4r5r6r7r8rrdetachr9r)rr:r(r; attn_convr<r=r>r$r$r%r?s(    zLocationAwareAttention.forwardr@) rBrCrDrErr3Tensor__annotations__rrr?rFr$r$r"r%rGus ' !rGcs0eZdZdZfddZddZddZZS)KeyValueAttentionaeThis class implements a single-headed key-value attention module for seq2seq learning. Reference: "Attention Is All You Need" by Vaswani et al., sec. 3.2.1 Arguments --------- enc_dim : int Size of the encoder feature vectors from which keys and values are computed. dec_dim : int Size of the decoder feature vectors from which queries are computed. attn_dim : int Size of the attention feature. output_dim : int Size of the output context vector. Example ------- >>> enc_tensor = torch.rand([4, 10, 20]) >>> enc_len = torch.ones([4]) * 10 >>> dec_tensor = torch.rand([4, 25]) >>> net = KeyValueAttention(enc_dim=20, dec_dim=25, attn_dim=30, output_dim=5) >>> out_tensor, out_weight = net(enc_tensor, enc_len, dec_tensor) >>> out_tensor.shape torch.Size([4, 5]) csVtt|||_t|||_t|||_tt | |_ | dSN) rrrr key_linear query_linear value_linearr3sqrttensorrPrr)rrrr r!r"r$r%rs  zKeyValueAttention.__init__cCr&r')valueskeysr*r+r$r$r%r!r,zKeyValueAttention.resetcCs|jdur |||_|||_t||d|jdd|_| |d}t |j||j }| |jdktj }|ddd}t ||jd}||fS)r-Nr r.rHr)r^rXrZr]rr1r0r2r*rYr3matmulrr6r7r8rrQr5)rr:r(r;queryscoresnormalized_scoresoutr$r$r%r?'s   zKeyValueAttention.forwardrAr$r$r"r%rVs   rVcsXeZdZdZejfdedejffdd Ze defddZ d ej fd d Z Z S) RelPosEncXLaRelative positional encoding for the :class:`~RelPosMHAXL`. Arguments --------- emb_dim : int Size of the embedding, which controls the size of the last dimension of the positional embedding as well dtype : torch.dtype, optional If unspecified, defaults to `torch.float32`. Controls the data type of the output embedding (but does not affect the precision of the computations, which remain `torch.float32`). emb_dimdtypecsTt||_ttjd|jdtjdtd|j }| d|||_ dS)NrrH)rf@inv_freq) rrrer3exparangefloat32mathlogregister_buffer emb_dtype)rrerfrhr"r$r%rUs   zRelPosEncXL.__init__seq_lenc CsB|j}|jj}ttjd||jftj|d}|d}|d}tjd|tj|d d}t ||j}||dddddf<t ||j|dddddf<||dddddf<t | |j|dddddf<t |d d}|dd d}tj ||gdd} | |} Wd| S1swY| S) ab Builds the positional embedding tensor for a given sequence length. Arguments --------- seq_len : int The length of the sequence to create the position embedding for. Returns ------- torch.Tensor Positional embedding tensor of shape `[1, 2*seq_len-1, embed_dim]` rHrfr0rr rN)rr)rorhr0r3no_grademptyrerkrjr2sincosflipcatto) rrpror0tot_pepe_past pe_future positions sinusoidsper$r$r%make_peas>  "$  zRelPosEncXL.make_pexcCs|j|ddS)a Builds the positional embedding tensor. Similar to :meth:`~RelPosEncXL.make_pe` but uses the shape information from the provided tensor. Arguments --------- x : torch.Tensor input tensor with shape batch_size, seq_len, embed_dim Returns ------- pos_emb : torch.Tensor Positional embedding tensor of shape `[1, 2*seq_len-1, embed_dim]` r )rp)rr1rrr$r$r%r?szRelPosEncXL.forward)rBrCrDrEr3rkintrfrrrrrTr?rFr$r$r"r%rdGs   2rdcsJeZdZdZ    dfdd ZddZd d Z   dd d ZZS) RelPosMHAXLaThis class implements the relative multihead implementation similar to that in Transformer XL https://arxiv.org/pdf/1901.02860.pdf Arguments --------- embed_dim : int Size of the encoder feature vectors from which keys and values are computed. num_heads: int Number of attention heads. dropout : float, optional Dropout rate. vbias: bool, optional Whether to use bias for computing value. vdim: int, optional Size for value. Default is embed_dim (Note each head is embed_dim // num_heads). mask_pos_future: bool, optional Whether to mask future positional encodings values. Must be true for causal applications e.g. decoder. Example ------- >>> inputs = torch.rand([6, 60, 512]) >>> pos_emb = torch.rand([1, 2*60-1, 512]) >>> net = RelPosMHAXL(num_heads=8, embed_dim=inputs.shape[-1]) >>> outputs, attn = net(inputs, inputs, inputs, pos_emb) >>> outputs.shape torch.Size([6, 60, 512]) FNcst||_|dur|n||_|j|k|_||_||_||_||_|||_ |j||_ |j ||jks:Jd|j ||jksFJd|jdurct t d|||_t t |j||_n t t d|||_|r|t t |j|_nd|_t ||_t |j||_t j||dd|_t t |j |j|_t t |j |j|_t|jt jkrd|_ntd |_| d t!"|j|_#dS) N(embed_dim must be divisible by num_heads#vdim must be divisible by num_headsFrHr r8r )$rr embed_dimvdim_qkv_same_embed_dimmask_pos_futurevbias num_headsdropouthead_dim vhead_dimr Parameterr3rsqk_proj_weight v_proj_weightin_proj_weightvalue_bias_weightDropout dropout_attrout_proj linear_pos pos_bias_u pos_bias_vnext parametersrffloat16attn_fill_valuerP_reset_parametersrlr[scale)rrrrrrrr"r$r%rsR       zRelPosMHAXL.__init__cCsx|jr tjj|jntjj|jtjj|j|jdur*tjj |j dtjj|j tjj|j dSNr) rr3rinitxavier_uniform_rrrr constant_rrrr+r$r$r%rs zRelPosMHAXL._reset_parameterscCs|\}}}}tjjj|dd}|||d|}|ddddddf||||}|jrYtj|d|df|jd}|t ||d|dddddddf}|d d|ddfS) zRelative shift implementation.)r r)padrNr rHrr0.) r1r3r functionalrviewronesr0tril)rrbhqlenpos_lenrr$r$r% rel_shifts& 4zRelPosMHAXL.rel_shiftTc Cs|jd}|jd} |jd} |jrz||ust||rA||us&t||rAtj||j|d|j |j dj ddd\}}}n;|jj ddd\} } } tj|| |d|j |j }tj|| |d|j |j }tj|| |d|j |j }nt |jdur||jdd|j |j}||dd|j |j }||jdd|j |j dd}||jdd|j |j dd}t||j|dddd}t||j|dddd}||}||}|dur|jdkr|dd| | }n |d|j | | }|jtjkr|||j}n||7}|dur%|||dd| |j}tj |dtj!d}|"|}|durG|jtjkrF||d }n |durX|||dd| d }t||dd}|dd#|d|j|j }|$|}|r||fS|S) aX Compute attention. Arguments --------- query : torch.Tensor (B, L, E) where L is the target sequence length, B is the batch size, E is the embedding dimension. key : torch.Tensor (B, S, E) where S is the source sequence length, B is the batch size, E is the embedding dimension. value : torch.Tensor (B, S, E) where S is the source sequence length, B is the batch size, E is the embedding dimension. pos_embs : torch.Tensor bidirectional sinusoidal positional embedding tensor (1, 2*S-1, E) where S is the max length between source and target sequence lengths, and E is the embedding dimension. key_padding_mask : torch.Tensor (B, S) where B is the batch size, S is the source sequence length. If a ByteTensor is provided, the non-zero positions will be ignored while the position with the zero positions will be unchanged. If a BoolTensor is provided, the positions with the value of True will be ignored while the position with the value of False will be unchanged. attn_mask : torch.Tensor 2D mask (L, S) where L is the target sequence length, S is the source sequence length. 3D mask (N*num_heads, L, S) where N is the batch size, L is the target sequence length, S is the source sequence length. attn_mask ensure that position i is allowed to attend the unmasked positions. If a ByteTensor is provided, the non-zero positions are not allowed to attend while the zero positions will be unchanged. If a BoolTensor is provided, positions with True is not allowed to attend while False values will be unchanged. If a FloatTensor is provided, it will be added to the attention weight. return_attn_weights : bool Whether to additionally return the attention weights. Returns ------- out : torch.Tensor (B, L, E) where L is the target sequence length, B is the batch size, E is the embedding dimension. attn_score : torch.Tensor (B, L, S) where B is the batch size, L is the target sequence length, S is the source sequence length. rr rrrrHN)rrfr)%shaperr3equalrrlinearrrrrchunkNotImplementedErrorrrrrrrrrQrr_rpermuterndimrfboolr6rFrrkr contiguousr)rr`keyvaluepos_embskey_padding_mask attn_maskreturn_attn_weightsbszklenrqweightkweightvweightp_k q_with_bias_u q_with_bias_v matrix_ac matrix_bd attn_scorerrcr$r$r%r?%s :                     zRelPosMHAXL.forward)rFNF)NNT) rBrCrDrErrrr?rFr$r$r"r%rs!?rc sdeZdZdZ      dfdd Z    ddeejd eejd ed eejfd d Z Z S)MultiheadAttentionaThe class is a wrapper of MultiHead Attention for torch.nn.MultiHeadAttention. Reference: https://pytorch.org/docs/stable/nn.html Arguments --------- nhead : int parallel attention heads. d_model : int The size of the model layers. dropout : float a Dropout layer on attn_output_weights (default: 0.0). bias : bool add bias as module parameter (default: True). add_bias_kv : bool add bias to the key and value sequences at dim=0. add_zero_attn : bool add a new batch of zeros to the key and value sequences at dim=1. kdim : int total number of features in key (default: None). vdim : int total number of features in value (default: None). Example ------- >>> inputs = torch.rand([8, 60, 512]) >>> net = MultiheadAttention(nhead=8, d_model=inputs.shape[-1]) >>> outputs, attn = net(inputs, inputs, inputs) >>> outputs.shape torch.Size([8, 60, 512]) rTFNc s*ttj||||||||d|_dS)N)rrrr add_bias_kv add_zero_attnkdimr)rrrratt) rnheadd_modelrr rrrrr"r$r%rs  zMultiheadAttention.__init__rrrrc Cs|ddd}|ddd}|ddd}|dur$|dur"||7}n|}|j||||||d\}} |ddd}|r>|| fS|S)a Compute attention. Arguments --------- query : torch.Tensor (B, L, E) where L is the target sequence length, B is the batch size, E is the embedding dimension. key : torch.Tensor (B, S, E) where S is the source sequence length, B is the batch size, E is the embedding dimension. value : torch.Tensor (B, S, E) where S is the source sequence length, B is the batch size, E is the embedding dimension. attn_mask : torch.Tensor, optional 2D mask (L, S) where L is the target sequence length, S is the source sequence length. 3D mask (N*num_heads, L, S) where N is the batch size, L is the target sequence length, S is the source sequence length. attn_mask ensure that position i is allowed to attend the unmasked positions. If a ByteTensor is provided, the non-zero positions are not allowed to attend while the zero positions will be unchanged. If a BoolTensor is provided, positions with True is not allowed to attend while False values will be unchanged. If a FloatTensor is provided, it will be added to the attention weight. key_padding_mask : torch.Tensor, optional (B, S) where B is the batch size, S is the source sequence length. If a ByteTensor is provided, the non-zero positions will be ignored while the position with the zero positions will be unchanged. If a BoolTensor is provided, the positions with the value of True will be ignored while the position with the value of False will be unchanged. return_attn_weights : bool, optional True to additionally return the attention weights, False otherwise. pos_embs : torch.Tensor, optional Positional embeddings added to the attention map of shape (L, S, E) or (L, S, 1). Returns ------- attn_output : torch.Tensor (B, L, E) where L is the target sequence length, B is the batch size, E is the embedding dimension. attn_output_weights : torch.Tensor (B, L, S) where B is the batch size, L is the target sequence length, S is the source sequence length. This is returned only if `return_attn_weights=True` (True by default). r rrHN)rr need_weights)rr) rr`rrrrrroutputattention_weightsr$r$r%r?s&9   zMultiheadAttention.forward)rTFFNN)NNTN) rBrCrDrErrr3rTrr?rFr$r$r"r%rs,$rcs4eZdZdZdddejffdd ZddZZS)PositionalwiseFeedForwarduThe class implements the positional-wise feed forward module in “Attention Is All You Need”. Arguments --------- d_ffn: int Hidden layer size. input_shape : tuple, optional Expected shape of the input. Alternatively use ``input_size``. input_size : int, optional Expected size of the input. Alternatively use ``input_shape``. dropout: float, optional Dropout rate. activation: torch.nn.Module, optional activation functions to be applied (Recommendation: ReLU, GELU). Example ------- >>> inputs = torch.rand([8, 60, 512]) >>> net = PositionalwiseFeedForward(256, input_size=inputs.shape[-1]) >>> outputs = net(inputs) >>> outputs.shape torch.Size([8, 60, 512]) Nrc s`t|dur|durtd|dur|d}tt|||t|t|||_dS)Nz)Expected one of input_shape or input_sizer)rr ValueErrorr Sequentialrrffn)rd_ffn input_shape input_sizer activationr"r$r%rs    z"PositionalwiseFeedForward.__init__cCs*|ddd}||}|ddd}|S)z8Applies PositionalwiseFeedForward to the input tensor x.r rrH)rrrr$r$r%r?s z!PositionalwiseFeedForward.forward) rBrCrDrErReLUrr?rFr$r$r"r%rssrcs6eZdZdZdededejdejffdd ZZ S)PrecomputedRoPESinusoidsa  A cache for the sines and cosines needed to rotate the vectors for rotary position embeddings (RoPE). This stores the nonzero entries from eq(15) from https://arxiv.org/pdf/2104.09864 Arguments --------- max_length : int The allowed max length of the input sequence. For a fixed setting of the other arguments, the computation takes O(max_length) time. input_size : int Size of each vector in the input sequence, i.e. the dimension of each attention head. dtype : torch.dtype The dtype of the tensors. device : torch.device The Torch device to put the tensors on. Example ------- >>> precomputed = PrecomputedRoPESinusoids(3, 8, torch.float32, torch.device('cpu')) >>> precomputed.cosines.shape torch.Size([3, 8]) >>> precomputed.sines.shape == precomputed.cosines.shape True >>> precomputed.cosines tensor([[ 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000], [ 0.5403, 0.5403, 0.9950, 0.9950, 0.9999, 0.9999, 1.0000, 1.0000], [-0.4161, -0.4161, 0.9801, 0.9801, 0.9998, 0.9998, 1.0000, 1.0000]]) >>> precomputed.sines tensor([[-0.0000, 0.0000, -0.0000, 0.0000, -0.0000, 0.0000, -0.0000, 0.0000], [-0.8415, 0.8415, -0.0998, 0.0998, -0.0100, 0.0100, -0.0010, 0.0010], [-0.9093, 0.9093, -0.1987, 0.1987, -0.0200, 0.0200, -0.0020, 0.0020]]) >>> precomputed.index_swap tensor([1, 0, 3, 2, 5, 4, 7, 6]) max_lengthrrfr0c sNt|tjkr tjntj}|ddksJ||_ttjd|d||dt d| }tj||d}tjd|||d}t ||} t | } tj | | gdd ||} t| } tj | | gdd ||} dtj|||d| } tj |ddd|dddgdd d}|d | ||d | ||d |dS) NrHrrqrgrrrr cosinessines index_swap)rrr3float64rkrrirjrlrmouterrustackreshapertrnrx)rrrrfr0internal_dtypeangles dimensionstimes times_anglesrunsigned_sinesunsigned_repeated_sinesrrr"r$r%rsH     z!PrecomputedRoPESinusoids.__init__) rBrCrDrErr3rfr0rrFr$r$r"r%rs'rc@s:eZdZdZdedeegeffddZdefddZd S) MemoiseAtLeastSizea Memoises a function which has as its first argument a value that indicates a minimum value to call the underlying function with. Arguments --------- function: Callable The function to call. round_up: Callable[[Any], Any] A function that rounds up. The fewer values this rounds up to, the less likely it is that the function will be called repeatedly. functionround_upcCs||_||_i|_dSrW)rrmemo)rrrr$r$r%r)s zMemoiseAtLeastSize.__init__r1cGs\||jvs|j|d|kr'||}||krJ||j|g|Rf|j|<|j|dS)Nrr )rrr)rr1args rounded_sizer$r$r%__call__1s   zMemoiseAtLeastSize.__call__N)rBrCrDrErrrrr$r$r$r%rsrrreturncsdtdtffdd }|S)a Decorator that memoises a function which has as its first argument a value that indicates a minimum value to call the underlying function with. If the memo has stored the result from a matching previous function call, The stored result will be returned instead of calling the function again. Arguments --------- round_up: Callable[[Any], Any] A function that rounds up. This will be called with the first argument passed in. The underlying function will receive, instead of this first argument, the rounded-up version. The fewer values this rounds up to, the less likely it is that the function will be called repeatedly. Returns ------- The passed function but with MemoiseAtLeastSize capability. rrcs t|S)z2 Set the function to be memoised. )r)rrr$r% with_functionQs z'memoise_at_least..with_function)rr)rrr$rr%memoise_at_least9srcCsdttt|S)NrH)rrlceillog2)lengthr$r$r%Zsrrrrfr0cCs0ttt|}|d|ksJt||||S)a) Return an object of type PrecomputedRoPESinusoids that is valid for the length, input_size, dtype and device. Consider a single (input_size, dtype, device), which are usually fixed for one model. The sinusoids will be recomputed only if they are not yet available for such a long length (because of the decorator applied to the function). Each time they are precomputed, the length is rounded up to the next power of two. As a consequence, the total number of calls during one program run is upper-bounded by ceil(log2(max_length)) where max_length is the highest length that is seen in the program run. On realistic lengths, the total number of calls is likely only a few. The total number of time steps for which sinusoids are precomputed during the program run is O(max_length). Arguments --------- length : int The length of the input sequence. input_size : int Size of each vector in the input sequence, i.e. the dimension of each attention head. dtype : torch.dtype The dtype of the tensors. device : torch.device The Torch device to put the tensors on. Return ------ An object of type PrecomputedRoPESinusoids that is valid for the length, input_size, dtype and device. rH)rroundrlrr)rrrfr0 length_powerr$r$r%_get_precomputed_valuesZs(rc Csz|j\}}}}|ddksJt|||j|j}|jd|}|jd|}tj|d|jd}|| d|| dS)z~ Perform the rotation for RoPE on each of the vectors in x. Details about RoPE: https://arxiv.org/pdf/2104.09864. rHrNr)rindexr ) rrrfr0rrr3 index_selectrr2) r _batch_sizer _num_headsr precomputedrr swapped_pairsr$r$r% _rope_rotatesrcsBeZdZdZ   d fdd ZddZ    d d d ZZS)RoPEMHAaThis is an implementation of multihead self-attention with RoPE positional embeddings. As it relies on Torch for self-attention, it is significantly faster than RelPosMHAXL while offering the same or better levels of accuracy. Details about RoPE: https://arxiv.org/pdf/2104.09864. Arguments --------- embed_dim : int Size of the encoder feature vectors from which keys and values are computed. num_heads: int Number of attention heads. dropout : float, optional Dropout rate. vbias: bool, optional Whether to use bias for computing value. vdim: int, optional Size for value. Default is embed_dim (Note each head is embed_dim // num_heads). Example ------- >>> max_len = 64 >>> inputs = torch.rand([6, 60, 512]) >>> num_heads = 8 >>> net = RoPEMHA(num_heads=num_heads, embed_dim=inputs.shape[-1]) >>> outputs, attn = net(inputs, inputs, inputs) >>> outputs.shape torch.Size([6, 60, 512]) rFNcsZt||_|dur|n||_|j|k|_||_||_||_|||_|j||_ |j||jks7Jd|j ||jksCJd|jdur`t t d|||_t t |j||_n t t d|||_|ryt t |j|_nd|_t ||_t |j||_t|jt jkrd|_ntd |_|dt|j|_dS) NrrFrHrrr8r ) rrrrrrrrrrrrr3rsrrrrrrrrrrrfrrrPrrlr[r)rrrrrrr"r$r%rsB       zRoPEMHA.__init__cCs\|jr tjj|jntjj|jtjj|j|jdur,tjj |j ddSdSr) rr3rrrrrrrrrr+r$r$r%rs zRoPEMHA._reset_parametersTc Cs|dusJd|jd}|jd} |jr}||ust||rD||us)t||rDtj||j|d|j |j dj ddd\}}}n;|jj ddd\} } } tj|| |d|j |j }tj|| |d|j |j }tj|| |d|j |j }nt |j dur||jdd|j |j}t|} t|}t|| |j ||}tj| dddd|dddd|dddd||j|jd }|dd|d|j|j }||}|r|dfS|S) a Compute attention through Pytorch attention. Arguments --------- query : torch.Tensor (B, L, E) where L is the target sequence length, B is the batch size, E is the embedding dimension. key : torch.Tensor (B, S, E) where S is the source sequence length, B is the batch size, E is the embedding dimension. value : torch.Tensor (B, S, E) where S is the source sequence length, B is the batch size, E is the embedding dimension. key_padding_mask : torch.Tensor (B, S) where B is the batch size, S is the source sequence length. If a ByteTensor is provided, the non-zero positions will be ignored while the position with the zero positions will be unchanged. If a BoolTensor is provided, the positions with the value of True will be ignored while the position with the value of False will be unchanged. attn_mask : torch.BoolTensor 2D mask (L, S) where L is the target sequence length, S is the source sequence length. The positions with the value of True will be ignored while the position with the value of False will be unchanged. pos_embs : torch.Tensor Not used by this class. It is kept for compliance. return_attn_weights : bool Whether to additionally return the attention weights. Returns ------- out : torch.Tensor (B, L, E) where L is the target sequence length, B is the batch size, E is the embedding dimension. attn_score : torch.Tensor (B, L, S) where B is the batch size, L is the target sequence length, S is the source sequence length. Nzpos_embs is not supportedrr rrrrH)r`rrr dropout_pr)rrr3rrrrrrrrrrrrrr masks_unionrscaled_dot_product_attentionrrrrQrr)rr`rrrrrrrrrrr q_rotated k_rotated final_masksrrcr$r$r%r?s^0           zRoPEMHA.forward)rFN)NNNT)rBrCrDrErrr?rFr$r$r"r%r s"5r cCsd}|dur||dd|||||}|}|dur*|dd||||||}|}|dur8|dur8t||}|durAt|}|S)aThis is an utility function combining standard key_padding_mask and attn_mask from SpeechBrain into a single one for scaled_dot_product_attention. This function does not support weighting of the attn_score. Hence, if one wish to use float values as masks, they should not use this function. Arguments --------- bsz : int Batch size dimension. klen : int Time dimension of the key tensor. (Sequence length). num_heads : int Number of heads of the attention module using these masks. attn_mask : torch.BoolTensor 2D mask (L, S) where L is the target sequence length, S is the source sequence length. The positions with the value of True will be ignored while the position with the value of False will be unchanged. key_padding_mask : torch.BoolTensor (B, S) where B is the batch size, S is the source sequence length. The positions with the value of True will be ignored while the position with the value of False will be unchanged. Returns ------- out : torch.BoolTensor (bsz, num_heads, klen, klen) where False values are masked and True are unmasked (opposite of the input tensors). Nr )rexpandr3 logical_or logical_not)rrrrr final_maskr$r$r%r ls   r )(rErltypingrrrrrnumpyr7r3torch.nnrtorch.nn.functionalrrspeechbrain.dataio.dataiorspeechbrain.utils.loggerrrBloggerModuler rGrVrdrrrrrrrrfr0rrr r r$r$r$r%sX    \Ma==j   ! , N