o ½e¦i’Âã @sŽdZddlZddlmZmZmZmZmZddlZ ddl Z ddl m Z ddl m mZddlmZddlmZeeƒZGdd„de jƒZGdd „d e jƒZGd d „d e jƒZGd d „d e jƒZGdd„de jƒZGdd„de jƒZGdd„de jƒZGdd„de jƒZGdd„dƒZdeegefdeegeffdd„Z e dd„ƒde!de!d e j"d!e j#def d"d#„ƒZ$d$d%„Z%Gd&d'„d'e jƒZ&d(d)„Z'dS)*z¢Library implementing attention modules. Authors * Ju-Chieh Chou 2020 * Jianyuan Zhong 2020 * Loren Lugosch 2020 * Samuele Cornell 2020 * Shucong Zhang 2024 éN)ÚAnyÚCallableÚDictÚOptionalÚTuple)Úlength_to_mask)Ú get_loggercs2eZdZdZd ‡fdd„ Zdd„Zdd„Z‡ZS) ÚContentBasedAttentiona”This class implements content-based attention module for seq2seq learning. Reference: NEURAL MACHINE TRANSLATION BY JOINTLY LEARNING TO ALIGN AND TRANSLATE, Bahdanau et.al. https://arxiv.org/pdf/1409.0473.pdf Arguments --------- enc_dim : int Size of encoder layer. dec_dim : int Size of decoder layer. attn_dim : int Size of the attention feature. output_dim : int Size of the output context vector. scaling : float The factor controls the sharpening degree (default: 1.0). Example ------- >>> enc_tensor = torch.rand([4, 10, 20]) >>> enc_len = torch.ones([4]) * 10 >>> dec_tensor = torch.rand([4, 25]) >>> net = ContentBasedAttention(enc_dim=20, dec_dim=25, attn_dim=30, output_dim=5) >>> out_tensor, out_weight = net(enc_tensor, enc_len, dec_tensor) >>> out_tensor.shape torch.Size([4, 5]) çð?csftƒ ¡t ||¡|_t ||¡|_tj|ddd|_t ||¡|_||_tj dd|_ |  ¡dS)NéF©Úbiaséÿÿÿÿ©Údim) ÚsuperÚ__init__ÚnnÚLinearÚmlp_encÚmlp_decÚmlp_attnÚmlp_outÚscalingÚSoftmaxÚsoftmaxÚreset)ÚselfÚenc_dimÚdec_dimÚattn_dimÚ output_dimr©Ú __class__©úX/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/speechbrain/nnet/attention.pyr8s  zContentBasedAttention.__init__cCód|_d|_d|_dS©z)Reset the memory in the attention module.N)Úenc_lenÚprecomputed_enc_hÚmask©rr$r$r%rGó zContentBasedAttention.resetcCsª|jdur| |¡|_t|| d¡|jd|_| | d¡¡}| t   |j|¡¡  d¡}|  |jdkt j ¡}| ||j¡}t  | d¡|¡  d¡}| |¡}||fS)áReturns the output of the attention module. Arguments --------- enc_states : torch.Tensor The tensor to be attended. enc_len : torch.Tensor The real length (without padding) of enc_states for each sentence. dec_states : torch.Tensor The query tensor. Returns ------- The output of the attention module. Nr ©Úmax_lenÚdevicerr)r)rrÚsizer0r*rÚ unsqueezerÚtorchÚtanhÚsqueezeÚ masked_fillÚnpÚinfrrÚbmmr)rÚ enc_statesr(Ú dec_statesÚdec_hÚattnÚcontextr$r$r%ÚforwardMs  ÿÿþ zContentBasedAttention.forward©r ©Ú__name__Ú __module__Ú __qualname__Ú__doc__rrr?Ú __classcell__r$r$r"r%r s r csDeZdZUdZeejed< d ‡fdd„ Zdd„Z dd „Z ‡Z S) ÚLocationAwareAttentiona{This class implements location-aware attention module for seq2seq learning. Reference: Attention-Based Models for Speech Recognition, Chorowski et.al. https://arxiv.org/pdf/1506.07503.pdf Arguments --------- enc_dim : int Size of encoder. dec_dim : int Size of decoder. attn_dim : int Size of the attention feature. output_dim : int Size of the output context vector. conv_channels : int Number of channel for location feature. kernel_size : int Kernel size of convolutional layer for location feature. scaling : float The factor controls the sharpening degree (default: 1.0). Example ------- >>> enc_tensor = torch.rand([4, 10, 20]) >>> enc_len = torch.ones([4]) * 10 >>> dec_tensor = torch.rand([4, 25]) >>> net = LocationAwareAttention( ... enc_dim=20, ... dec_dim=25, ... attn_dim=30, ... output_dim=5, ... conv_channels=10, ... kernel_size=100) >>> out_tensor, out_weight = net(enc_tensor, enc_len, dec_tensor) >>> out_tensor.shape torch.Size([4, 5]) r)r cs¤tƒ ¡t ||¡|_t ||¡|_tj|ddd|_tjd|d|d|dd|_t ||¡|_ tj|ddd|_t ||¡|_ ||_ tj dd|_ | ¡dS)Nr Fr é)Ú kernel_sizeÚpaddingr rr)rrrrrrrÚConv1dÚconv_locÚmlp_locrrrrr)rrrr r!Ú conv_channelsrIrr"r$r%rŸs"  û zLocationAwareAttention.__init__cCsd|_d|_d|_d|_dS)z%Reset the memory in attention module.N)r(r)r*Ú prev_attnr+r$r$r%rÀs zLocationAwareAttention.resetcCsö|jdur$| |¡|_t|| d¡|jd|_|jd| ¡ d¡|_|  |j d¡¡}|  |  dd¡¡}|  | d¡¡}|  t |j||¡¡ d¡}| |jdktj ¡}| ||j¡}| ¡|_t | d¡|¡ d¡}| |¡}||fS)r-Nr r.rHrr)r)rrr1r0r*Úfloatr2rOrLrMÚ transposerrr3r4r5r6r7r8rrÚdetachr9r)rr:r(r;Ú attn_convr<r=r>r$r$r%r?Çs(  ÿÿþ  zLocationAwareAttention.forwardr@) rBrCrDrErr3ÚTensorÚ__annotations__rrr?rFr$r$r"r%rGus ' ø!rGcs0eZdZdZ‡fdd„Zdd„Zdd„Z‡ZS)ÚKeyValueAttentionaeThis class implements a single-headed key-value attention module for seq2seq learning. Reference: "Attention Is All You Need" by Vaswani et al., sec. 3.2.1 Arguments --------- enc_dim : int Size of the encoder feature vectors from which keys and values are computed. dec_dim : int Size of the decoder feature vectors from which queries are computed. attn_dim : int Size of the attention feature. output_dim : int Size of the output context vector. Example ------- >>> enc_tensor = torch.rand([4, 10, 20]) >>> enc_len = torch.ones([4]) * 10 >>> dec_tensor = torch.rand([4, 25]) >>> net = KeyValueAttention(enc_dim=20, dec_dim=25, attn_dim=30, output_dim=5) >>> out_tensor, out_weight = net(enc_tensor, enc_len, dec_tensor) >>> out_tensor.shape torch.Size([4, 5]) csVtƒ ¡t ||¡|_t ||¡|_t ||¡|_t t  |¡  ¡¡|_ |  ¡dS©N) rrrrÚ key_linearÚ query_linearÚ value_linearr3ÚsqrtÚtensorrPrr)rrrr r!r"r$r%rs  zKeyValueAttention.__init__cCr&r')ÚvaluesÚkeysr*r+r$r$r%r!r,zKeyValueAttention.resetcCs¨|jdur | |¡|_| |¡|_t|| d¡|jd d¡|_|  |¡ d¡}t   |j|¡|j }|  |jdktj ¡}| d¡ dd¡}t   ||j¡ d¡}||fS)r-Nr r.rHr)r^rXrZr]rr1r0r2r*rYr3Úmatmulrr6r7r8rrQr5)rr:r(r;ÚqueryÚscoresÚnormalized_scoresÚoutr$r$r%r?'s   ÿþzKeyValueAttention.forwardrAr$r$r"r%rVús   rVcsXeZdZdZejfdedejf‡fdd„ Ze  ¡defdd„ƒZ d ej fd d „Z ‡Z S) Ú RelPosEncXLaÕRelative positional encoding for the :class:`~RelPosMHAXL`. Arguments --------- emb_dim : int Size of the embedding, which controls the size of the last dimension of the positional embedding as well dtype : torch.dtype, optional If unspecified, defaults to `torch.float32`. Controls the data type of the output embedding (but does not affect the precision of the computations, which remain `torch.float32`). Úemb_dimÚdtypecsTtƒ ¡||_t tjd|jdtjdt d¡|j ¡}|  d|¡||_ dS)NrrH)rfçˆÃ@Úinv_freq) rrrer3ÚexpÚarangeÚfloat32ÚmathÚlogÚregister_bufferÚ emb_dtype)rrerfrhr"r$r%rUs ÿÿ  zRelPosEncXL.__init__Úseq_lenc CsB|j}|jj}t ¡Œtjd||jftj|d}|d}|d}tjd|tj|d  d¡}t  ||j¡}||dd…ddd…f<t  ||j¡|dd…ddd…f<||dd…ddd…f<t  | |j¡|dd…ddd…f<t  |d¡  d¡}|dd…  d¡}tj ||gdd} |  |¡} Wdƒ| S1sšwY| S) ab Builds the positional embedding tensor for a given sequence length. Arguments --------- seq_len : int The length of the sequence to create the position embedding for. Returns ------- torch.Tensor Positional embedding tensor of shape `[1, 2*seq_len-1, embed_dim]` rH©rfr0rr rN)rr)rorhr0r3Úno_gradÚemptyrerkrjr2ÚsinÚcosÚflipÚcatÚto) rrpror0Útot_peÚpe_pastÚ pe_futureÚ positionsÚ sinusoidsÚper$r$r%Úmake_peas>  ýüû"$  äâzRelPosEncXL.make_peÚxcCs|j| d¡dS)aº Builds the positional embedding tensor. Similar to :meth:`~RelPosEncXL.make_pe` but uses the shape information from the provided tensor. Arguments --------- x : torch.Tensor input tensor with shape batch_size, seq_len, embed_dim Returns ------- pos_emb : torch.Tensor Positional embedding tensor of shape `[1, 2*seq_len-1, embed_dim]` r )rp)rr1©rr€r$r$r%r?”szRelPosEncXL.forward)rBrCrDrEr3rkÚintrfrrrrrTr?rFr$r$r"r%rdGs   2rdcsJeZdZdZ    d‡fdd„ Zdd„Zd d „Z   dd d „Z‡ZS)Ú RelPosMHAXLa÷This class implements the relative multihead implementation similar to that in Transformer XL https://arxiv.org/pdf/1901.02860.pdf Arguments --------- embed_dim : int Size of the encoder feature vectors from which keys and values are computed. num_heads: int Number of attention heads. dropout : float, optional Dropout rate. vbias: bool, optional Whether to use bias for computing value. vdim: int, optional Size for value. Default is embed_dim (Note each head is embed_dim // num_heads). mask_pos_future: bool, optional Whether to mask future positional encodings values. Must be true for causal applications e.g. decoder. Example ------- >>> inputs = torch.rand([6, 60, 512]) >>> pos_emb = torch.rand([1, 2*60-1, 512]) >>> net = RelPosMHAXL(num_heads=8, embed_dim=inputs.shape[-1]) >>> outputs, attn = net(inputs, inputs, inputs, pos_emb) >>> outputs.shape torch.Size([6, 60, 512]) çFNcs¢tƒ ¡||_|dur|n||_|j|k|_||_||_||_||_|||_ |j||_ |j ||jks:Jdƒ‚|j ||jksFJdƒ‚|jdurct   t  d||¡¡|_t   t  |j|¡¡|_n t   t  d||¡¡|_|r|t   t  |j¡¡|_nd|_t  |¡|_t  |j|¡|_t j||dd|_t   t  |j |j¡¡|_t   t  |j |j¡¡|_t| ¡ƒjt jkr¼d|_ntdƒ |_|  ¡d t! "|j¡|_#dS) Nú(embed_dim must be divisible by num_headsú#vdim must be divisible by num_headsFrHér éÿÿr8r )$rrÚ embed_dimÚvdimÚ_qkv_same_embed_dimÚmask_pos_futureÚvbiasÚ num_headsÚdropoutÚhead_dimÚ vhead_dimrÚ Parameterr3rsÚqk_proj_weightÚ v_proj_weightÚin_proj_weightÚvalue_bias_weightÚDropoutÚ dropout_attrÚout_projÚ linear_posÚ pos_bias_uÚ pos_bias_vÚnextÚ parametersrfÚfloat16Úattn_fill_valuerPÚ_reset_parametersrlr[Úscale)rr‰rŽrrrŠrŒr"r$r%rÆsR    ÿÿ ÿÿ ÿÿ zRelPosMHAXL.__init__cCsx|jr tjj |j¡ntjj |j¡tjj |j¡|jdur*tjj  |j d¡tjj |j ¡tjj |j ¡dS©Nr„) r‹r3rÚinitÚxavier_uniform_r•r“r”rÚ constant_r–r›rœr+r$r$r%r¡s zRelPosMHAXL._reset_parameterscCsÊ| ¡\}}}}tjjj|dd}| ||d|¡}|dd…dd…dd…f ||||¡}|jrYtj| d¡| d¡f|jd}|t  || d¡| d¡¡dddd…dd…f}|d d|dd…fS) zRelative shift implementation.)r r)ÚpadrNr rHr‡©r0.) r1r3rÚ functionalr§ÚviewrŒÚonesr0Útril)rr€ÚbÚhÚqlenÚpos_lenr«r$r$r%Ú rel_shifts& 4zRelPosMHAXL.rel_shiftTc Cs|jd}|jd} |jd} |jrz||ust ||¡rA||us&t ||¡rAtj ||j¡ |d|j |j d¡j ddd\}}}n;|jj ddd\} } } tj || ¡ |d|j |j ¡}tj || ¡ |d|j |j ¡}tj || ¡ |d|j |j ¡}nt ‚|jdurŽ||j dd|j |j¡}| |¡ dd|j |j ¡}||j dd|j |j ¡ dd¡}||j dd|j |j ¡ dd¡}t ||j| dddd¡¡}t ||j| dddd¡¡}| |¡}||}|dur|jdkr÷| dd| | ¡}n | d|j | | ¡}|jtjkr| ||j¡}n||7}|dur%| | |dd| ¡|j¡}tj |dtj!d}| "|¡}|durG|jtjkrF| |d ¡}n |durX| | |dd| ¡d ¡}t || dd¡¡}| dd¡ #¡ |d|j|j ¡}| $|¡}|r||fS|S) aX Compute attention. Arguments --------- query : torch.Tensor (B, L, E) where L is the target sequence length, B is the batch size, E is the embedding dimension. key : torch.Tensor (B, S, E) where S is the source sequence length, B is the batch size, E is the embedding dimension. value : torch.Tensor (B, S, E) where S is the source sequence length, B is the batch size, E is the embedding dimension. pos_embs : torch.Tensor bidirectional sinusoidal positional embedding tensor (1, 2*S-1, E) where S is the max length between source and target sequence lengths, and E is the embedding dimension. key_padding_mask : torch.Tensor (B, S) where B is the batch size, S is the source sequence length. If a ByteTensor is provided, the non-zero positions will be ignored while the position with the zero positions will be unchanged. If a BoolTensor is provided, the positions with the value of True will be ignored while the position with the value of False will be unchanged. attn_mask : torch.Tensor 2D mask (L, S) where L is the target sequence length, S is the source sequence length. 3D mask (N*num_heads, L, S) where N is the batch size, L is the target sequence length, S is the source sequence length. attn_mask ensure that position i is allowed to attend the unmasked positions. If a ByteTensor is provided, the non-zero positions are not allowed to attend while the zero positions will be unchanged. If a BoolTensor is provided, positions with True is not allowed to attend while False values will be unchanged. If a FloatTensor is provided, it will be added to the attention weight. return_attn_weights : bool Whether to additionally return the attention weights. Returns ------- out : torch.Tensor (B, L, E) where L is the target sequence length, B is the batch size, E is the embedding dimension. attn_score : torch.Tensor (B, L, S) where B is the batch size, L is the target sequence length, S is the source sequence length. rr rr‡rrHN)rrfr„)%Úshaper‹r3Úequalrr©Úlinearr•rªrŽrÚchunkÚNotImplementedErrorr“r”r‘rr–ršr›rQrœr_r¢Úpermuter±ÚndimrfÚboolr6r ÚFrrkr˜Ú contiguousr™)rr`ÚkeyÚvalueÚpos_embsÚkey_padding_maskÚ attn_maskÚreturn_attn_weightsÚbszÚklenr¯ÚqweightÚkweightÚvweightÚp_kÚ q_with_bias_uÚ q_with_bias_vÚ matrix_acÚ matrix_bdÚ attn_scorer€rcr$r$r%r?%sž :  þ ÿ ÿ ÿ ÿ  ÿ  ÿþþ ÿÿ   ÿ þ   þ ÿ ý zRelPosMHAXL.forward)r„FNF)NNT) rBrCrDrErr¡r±r?rFr$r$r"r%rƒ¨s!ù?ørƒc sdeZdZdZ      d‡fdd„ Z    ddeejd eejd ed eejfd d „Z ‡Z S)ÚMultiheadAttentionaßThe class is a wrapper of MultiHead Attention for torch.nn.MultiHeadAttention. Reference: https://pytorch.org/docs/stable/nn.html Arguments --------- nhead : int parallel attention heads. d_model : int The size of the model layers. dropout : float a Dropout layer on attn_output_weights (default: 0.0). bias : bool add bias as module parameter (default: True). add_bias_kv : bool add bias to the key and value sequences at dim=0. add_zero_attn : bool add a new batch of zeros to the key and value sequences at dim=1. kdim : int total number of features in key (default: None). vdim : int total number of features in value (default: None). Example ------- >>> inputs = torch.rand([8, 60, 512]) >>> net = MultiheadAttention(nhead=8, d_model=inputs.shape[-1]) >>> outputs, attn = net(inputs, inputs, inputs) >>> outputs.shape torch.Size([8, 60, 512]) r„TFNc s*tƒ ¡tj||||||||d|_dS)N)r‰rŽrr Ú add_bias_kvÚ add_zero_attnÚkdimrŠ)rrrrÍÚatt) rÚnheadÚd_modelrr rÎrÏrÐrŠr"r$r%rs  øzMultiheadAttention.__init__rÀr¿rÁr¾c Cs€| ddd¡}| ddd¡}| ddd¡}|dur$|dur"||7}n|}|j||||||d\}} | ddd¡}|r>|| fS|S)a Compute attention. Arguments --------- query : torch.Tensor (B, L, E) where L is the target sequence length, B is the batch size, E is the embedding dimension. key : torch.Tensor (B, S, E) where S is the source sequence length, B is the batch size, E is the embedding dimension. value : torch.Tensor (B, S, E) where S is the source sequence length, B is the batch size, E is the embedding dimension. attn_mask : torch.Tensor, optional 2D mask (L, S) where L is the target sequence length, S is the source sequence length. 3D mask (N*num_heads, L, S) where N is the batch size, L is the target sequence length, S is the source sequence length. attn_mask ensure that position i is allowed to attend the unmasked positions. If a ByteTensor is provided, the non-zero positions are not allowed to attend while the zero positions will be unchanged. If a BoolTensor is provided, positions with True is not allowed to attend while False values will be unchanged. If a FloatTensor is provided, it will be added to the attention weight. key_padding_mask : torch.Tensor, optional (B, S) where B is the batch size, S is the source sequence length. If a ByteTensor is provided, the non-zero positions will be ignored while the position with the zero positions will be unchanged. If a BoolTensor is provided, the positions with the value of True will be ignored while the position with the value of False will be unchanged. return_attn_weights : bool, optional True to additionally return the attention weights, False otherwise. pos_embs : torch.Tensor, optional Positional embeddings added to the attention map of shape (L, S, E) or (L, S, 1). Returns ------- attn_output : torch.Tensor (B, L, E) where L is the target sequence length, B is the batch size, E is the embedding dimension. attn_output_weights : torch.Tensor (B, L, S) where B is the batch size, L is the target sequence length, S is the source sequence length. This is returned only if `return_attn_weights=True` (True by default). r rrHN)rÀr¿Ú need_weights)r·rÑ) rr`r¼r½rÀr¿rÁr¾ÚoutputÚattention_weightsr$r$r%r?s&9  ú zMultiheadAttention.forward)r„TFFNN)NNTN) rBrCrDrErrr3rTr¹r?rFr$r$r"r%rÍãs,$÷øûúùørÍcs4eZdZdZdddejf‡fdd„ Zdd„Z‡ZS)ÚPositionalwiseFeedForwarduThe class implements the positional-wise feed forward module in “Attention Is All You Needâ€. Arguments --------- d_ffn: int Hidden layer size. input_shape : tuple, optional Expected shape of the input. Alternatively use ``input_size``. input_size : int, optional Expected size of the input. Alternatively use ``input_shape``. dropout: float, optional Dropout rate. activation: torch.nn.Module, optional activation functions to be applied (Recommendation: ReLU, GELU). Example ------- >>> inputs = torch.rand([8, 60, 512]) >>> net = PositionalwiseFeedForward(256, input_size=inputs.shape[-1]) >>> outputs = net(inputs) >>> outputs.shape torch.Size([8, 60, 512]) Nr„c s`tƒ ¡|dur|durtdƒ‚|dur|d}t t ||¡|ƒt |¡t ||¡¡|_dS)Nz)Expected one of input_shape or input_sizer)rrÚ ValueErrorrÚ Sequentialrr—Úffn)rÚd_ffnÚ input_shapeÚ input_sizerÚ activationr"r$r%rs    üz"PositionalwiseFeedForward.__init__cCs*| ddd¡}| |¡}| ddd¡}|S)z8Applies PositionalwiseFeedForward to the input tensor x.r rrH)r·rÚrr$r$r%r?¤s z!PositionalwiseFeedForward.forward) rBrCrDrErÚReLUrr?rFr$r$r"r%r×ssúr×cs6eZdZdZdededejdejf‡fdd„ Z‡Z S)ÚPrecomputedRoPESinusoidsa  A cache for the sines and cosines needed to rotate the vectors for rotary position embeddings (RoPE). This stores the nonzero entries from eq(15) from https://arxiv.org/pdf/2104.09864 Arguments --------- max_length : int The allowed max length of the input sequence. For a fixed setting of the other arguments, the computation takes O(max_length) time. input_size : int Size of each vector in the input sequence, i.e. the dimension of each attention head. dtype : torch.dtype The dtype of the tensors. device : torch.device The Torch device to put the tensors on. Example ------- >>> precomputed = PrecomputedRoPESinusoids(3, 8, torch.float32, torch.device('cpu')) >>> precomputed.cosines.shape torch.Size([3, 8]) >>> precomputed.sines.shape == precomputed.cosines.shape True >>> precomputed.cosines tensor([[ 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000], [ 0.5403, 0.5403, 0.9950, 0.9950, 0.9999, 0.9999, 1.0000, 1.0000], [-0.4161, -0.4161, 0.9801, 0.9801, 0.9998, 0.9998, 1.0000, 1.0000]]) >>> precomputed.sines tensor([[-0.0000, 0.0000, -0.0000, 0.0000, -0.0000, 0.0000, -0.0000, 0.0000], [-0.8415, 0.8415, -0.0998, 0.0998, -0.0100, 0.0100, -0.0010, 0.0010], [-0.9093, 0.9093, -0.1987, 0.1987, -0.0200, 0.0200, -0.0020, 0.0020]]) >>> precomputed.index_swap tensor([1, 0, 3, 2, 5, 4, 7, 6]) Ú max_lengthrÝrfr0c sNtƒ ¡|tjkr tjntj}|ddksJ‚||_t tjd|d||dt  d¡| ¡}tj||d}tjd|||d}t  ||¡} t  | ¡} tj | | gdd  ||¡} t | ¡} tj | | gdd  ||¡} dtj|||d| } tj |ddd…|ddd…gdd  d¡}| d |  |¡¡| d |  |¡¡| d |¡dS) NrHrrqrgr¨rrr ÚcosinesÚsinesÚ index_swap)rrr3Úfloat64rkrárirjrlrmÚouterruÚstackÚreshapertrnrx)rrárÝrfr0Úinternal_dtypeÚanglesÚ dimensionsÚtimesÚ times_anglesrâÚunsigned_sinesÚunsigned_repeated_sinesrãrär"r$r%rØsH ÿÿÿ  ÿ ÿþÿý ÿþz!PrecomputedRoPESinusoids.__init__) rBrCrDrEr‚r3rfr0rrFr$r$r"r%rà°s'þýüûràc@s:eZdZdZdedeegeffdd„Zdefdd„Zd S) ÚMemoiseAtLeastSizea Memoises a function which has as its first argument a value that indicates a minimum value to call the underlying function with. Arguments --------- function: Callable The function to call. round_up: Callable[[Any], Any] A function that rounds up. The fewer values this rounds up to, the less likely it is that the function will be called repeatedly. ÚfunctionÚround_upcCs||_||_i|_dSrW)rñròÚmemo)rrñròr$r$r%r)s zMemoiseAtLeastSize.__init__r1cGs\||jvs|j|d|kr'| |¡}||krJ‚||j|g|¢RŽf|j|<|j|dS)Nrr )róròrñ)rr1ÚargsÚ rounded_sizer$r$r%Ú__call__1s   zMemoiseAtLeastSize.__call__N)rBrCrDrErrrrör$r$r$r%rðsrðròÚreturncsdtdtf‡fdd„ }|S)a Decorator that memoises a function which has as its first argument a value that indicates a minimum value to call the underlying function with. If the memo has stored the result from a matching previous function call, The stored result will be returned instead of calling the function again. Arguments --------- round_up: Callable[[Any], Any] A function that rounds up. This will be called with the first argument passed in. The underlying function will receive, instead of this first argument, the rounded-up version. The fewer values this rounds up to, the less likely it is that the function will be called repeatedly. Returns ------- The passed function but with MemoiseAtLeastSize capability. rñr÷cs t|ˆƒS)z2 Set the function to be memoised. )rð)rñ©ròr$r%Ú with_functionQs z'memoise_at_least..with_function)rrð)ròrùr$rør%Úmemoise_at_least9srúcCsdtt t |¡¡ƒS)NrH)r‚rlÚceilÚlog2)Úlengthr$r$r%ÚZsrþrýrÝrfr0cCs0ttt |¡ƒƒ}|d|ksJ‚t||||ƒS)a) Return an object of type PrecomputedRoPESinusoids that is valid for the length, input_size, dtype and device. Consider a single (input_size, dtype, device), which are usually fixed for one model. The sinusoids will be recomputed only if they are not yet available for such a long length (because of the decorator applied to the function). Each time they are precomputed, the length is rounded up to the next power of two. As a consequence, the total number of calls during one program run is upper-bounded by ceil(log2(max_length)) where max_length is the highest length that is seen in the program run. On realistic lengths, the total number of calls is likely only a few. The total number of time steps for which sinusoids are precomputed during the program run is O(max_length). Arguments --------- length : int The length of the input sequence. input_size : int Size of each vector in the input sequence, i.e. the dimension of each attention head. dtype : torch.dtype The dtype of the tensors. device : torch.device The Torch device to put the tensors on. Return ------ An object of type PrecomputedRoPESinusoids that is valid for the length, input_size, dtype and device. rH)r‚Úroundrlrürà)rýrÝrfr0Ú length_powerr$r$r%Ú_get_precomputed_valuesZs(rc Csz|j\}}}}|ddksJ‚t|||j|jƒ}|jd|…}|jd|…}tj|d|jd}||  d¡||  d¡S)z~ Perform the rotation for RoPE on each of the vectors in x. Details about RoPE: https://arxiv.org/pdf/2104.09864. rHrNr)rÚindexr ) r²rrfr0rârãr3Ú index_selecträr2) r€Ú _batch_sizerýÚ _num_headsrÚ precomputedrârãÚ swapped_pairsr$r$r%Ú _rope_rotate‡srcsBeZdZdZ   d ‡fdd„ Zdd„Z    d d d „Z‡ZS)ÚRoPEMHAaïThis is an implementation of multihead self-attention with RoPE positional embeddings. As it relies on Torch for self-attention, it is significantly faster than RelPosMHAXL while offering the same or better levels of accuracy. Details about RoPE: https://arxiv.org/pdf/2104.09864. Arguments --------- embed_dim : int Size of the encoder feature vectors from which keys and values are computed. num_heads: int Number of attention heads. dropout : float, optional Dropout rate. vbias: bool, optional Whether to use bias for computing value. vdim: int, optional Size for value. Default is embed_dim (Note each head is embed_dim // num_heads). Example ------- >>> max_len = 64 >>> inputs = torch.rand([6, 60, 512]) >>> num_heads = 8 >>> net = RoPEMHA(num_heads=num_heads, embed_dim=inputs.shape[-1]) >>> outputs, attn = net(inputs, inputs, inputs) >>> outputs.shape torch.Size([6, 60, 512]) r„FNcsZtƒ ¡||_|dur|n||_|j|k|_||_||_||_|||_|j||_ |j||jks7Jdƒ‚|j ||jksCJdƒ‚|jdur`t   t   d||¡¡|_t   t   |j|¡¡|_n t   t   d||¡¡|_|ryt   t   |j¡¡|_nd|_t  |¡|_t  |j|¡|_t| ¡ƒjt jkr˜d|_ntdƒ |_| ¡dt |j¡|_dS) Nr…r†FrHr‡rˆr8r ) rrr‰rŠr‹rrŽrrr‘rr’r3rsr“r”r•r–r—r˜rr™rržrfrŸr rPr¡rlr[r¢)rr‰rŽrrrŠr"r$r%r¾sB    ÿÿ ÿÿ  zRoPEMHA.__init__cCs\|jr tjj |j¡ntjj |j¡tjj |j¡|jdur,tjj  |j d¡dSdSr£) r‹r3rr¤r¥r•r“r”rr¦r–r+r$r$r%r¡ós ÿzRoPEMHA._reset_parametersTc Cs¾|dusJdƒ‚|jd}|jd} |jr}||ust ||¡rD||us)t ||¡rDtj ||j¡ |d|j |j d¡j ddd\}}}n;|jj ddd\} } } tj || ¡ |d|j |j ¡}tj || ¡ |d|j |j ¡}tj || ¡ |d|j |j ¡}nt ‚|j dur‘||j dd|j |j¡}t|ƒ} t|ƒ}t|| |j ||ƒ}tj|  dddd¡| dddd¡| dddd¡||j|jd }| dd¡ ¡ |d|j|j ¡}| |¡}|rÝ|dfS|S) a Compute attention through Pytorch attention. Arguments --------- query : torch.Tensor (B, L, E) where L is the target sequence length, B is the batch size, E is the embedding dimension. key : torch.Tensor (B, S, E) where S is the source sequence length, B is the batch size, E is the embedding dimension. value : torch.Tensor (B, S, E) where S is the source sequence length, B is the batch size, E is the embedding dimension. key_padding_mask : torch.Tensor (B, S) where B is the batch size, S is the source sequence length. If a ByteTensor is provided, the non-zero positions will be ignored while the position with the zero positions will be unchanged. If a BoolTensor is provided, the positions with the value of True will be ignored while the position with the value of False will be unchanged. attn_mask : torch.BoolTensor 2D mask (L, S) where L is the target sequence length, S is the source sequence length. The positions with the value of True will be ignored while the position with the value of False will be unchanged. pos_embs : torch.Tensor Not used by this class. It is kept for compliance. return_attn_weights : bool Whether to additionally return the attention weights. Returns ------- out : torch.Tensor (B, L, E) where L is the target sequence length, B is the batch size, E is the embedding dimension. attn_score : torch.Tensor (B, L, S) where B is the batch size, L is the target sequence length, S is the source sequence length. Nzpos_embs is not supportedrr rr‡rrH)r`r¼r½rÀÚ dropout_pr¢)r²r‹r3r³rr©r´r•rªrŽrrµr¶rr–r‘rÚ masks_unionrºÚscaled_dot_product_attentionr·rr¢rQr»r™)rr`r¼r½r¿rÀr¾rÁrÂrÃrÄrÅrÆÚ q_rotatedÚ k_rotatedÚ final_masksr€rcr$r$r%r?ýs^0  þ ÿ ÿ ÿ ÿ  ÿ ÿú ý zRoPEMHA.forward)r„FN)NNNT)rBrCrDrErr¡r?rFr$r$r"r%r Ÿs"ú5ør cCs†d}|dur| |dd|¡ ||||¡}|}|dur*| dd||¡ ||||¡}|}|dur8|dur8t ||¡}|durAt |¡}|S)a¯This is an utility function combining standard key_padding_mask and attn_mask from SpeechBrain into a single one for scaled_dot_product_attention. This function does not support weighting of the attn_score. Hence, if one wish to use float values as masks, they should not use this function. Arguments --------- bsz : int Batch size dimension. klen : int Time dimension of the key tensor. (Sequence length). num_heads : int Number of heads of the attention module using these masks. attn_mask : torch.BoolTensor 2D mask (L, S) where L is the target sequence length, S is the source sequence length. The positions with the value of True will be ignored while the position with the value of False will be unchanged. key_padding_mask : torch.BoolTensor (B, S) where B is the batch size, S is the source sequence length. The positions with the value of True will be ignored while the position with the value of False will be unchanged. Returns ------- out : torch.BoolTensor (bsz, num_heads, klen, klen) where False values are masked and True are unmasked (opposite of the input tensors). Nr )rªÚexpandr3Ú logical_orÚ logical_not)rÂrÃrŽrÀr¿Ú final_maskr$r$r%r ls ÿÿ  r )(rErlÚtypingrrrrrÚnumpyr7r3Útorch.nnrÚtorch.nn.functionalr©rºÚspeechbrain.dataio.dataiorÚspeechbrain.utils.loggerrrBÚloggerÚModuler rGrVrdrƒrÍr×ràrðrúr‚rfr0rrr r r$r$r$r%ÚsX    \Ma==j ÿ  þ !ÿÿÿÿ þ, N