o ´©iì/ã@s¦dZddlZddlZddlZddlmZddlmZmZddlmm Z ddl m Z ddl mmmZGdd„dejƒZGdd „d eƒZGd d „d ejƒZdS) z&Multi-Head Attention layer definition.éN)Únn)ÚOptionalÚTuple)Ú make_pad_maskcs8eZdZdZ‡fdd„Zdd„Zdd„Zdd „Z‡ZS) ÚMultiHeadedAttentionú±Multi-Head Attention layer. Args: n_head (int): The number of heads. n_feat (int): The number of features. dropout_rate (float): Dropout rate. cs~tt|ƒ ¡||dksJ‚|||_||_t ||¡|_t ||¡|_t ||¡|_ t ||¡|_ d|_ tj |d|_ dS)ú)Construct an MultiHeadedAttention object.rN©Úp)ÚsuperrÚ__init__Úd_kÚhrÚLinearÚlinear_qÚlinear_kÚlinear_vÚ linear_outÚattnÚDropoutÚdropout)ÚselfÚn_headÚn_featÚ dropout_rate©Ú __class__©úP/home/ubuntu/.local/lib/python3.10/site-packages/funasr/models/sond/attention.pyr s zMultiHeadedAttention.__init__cCs†| d¡}| |¡ |d|j|j¡}| |¡ |d|j|j¡}| |¡ |d|j|j¡}| dd¡}| dd¡}| dd¡}|||fS)á Transform query, key and value. Args: query (torch.Tensor): Query tensor (#batch, time1, size). key (torch.Tensor): Key tensor (#batch, time2, size). value (torch.Tensor): Value tensor (#batch, time2, size). Returns: torch.Tensor: Transformed query tensor (#batch, n_head, time1, d_k). torch.Tensor: Transformed key tensor (#batch, n_head, time2, d_k). torch.Tensor: Transformed value tensor (#batch, n_head, time2, d_k). réÿÿÿÿéé)ÚsizerÚviewrr rrÚ transpose)rÚqueryÚkeyÚvalueÚn_batchÚqÚkÚvrrrÚ forward_qkv-s     z MultiHeadedAttention.forward_qkvc Cs¸| d¡}|dur4| d¡ d¡}tt tjd|jd ¡j¡j ƒ}|  ||¡}tj |dd  |d¡}ntj |dd}|  |¡}t  ||¡}| dd¡ ¡ |d|j|j¡}| |¡S© aÒCompute attention context vector. Args: value (torch.Tensor): Transformed value (#batch, n_head, time2, d_k). scores (torch.Tensor): Attention score (#batch, n_head, time1, time2). mask (torch.Tensor): Mask (#batch, 1, time2) or (#batch, time1, time2). Returns: torch.Tensor: Transformed value (#batch, time1, d_model) weighted by the attention score (#batch, time1, time2). rNr!)Údtyper ©Údimgr"©r#Ú unsqueezeÚeqÚfloatÚnumpyÚfinfoÚtorchÚtensorr/ÚminÚ masked_fillÚsoftmaxrÚmatmulr%Ú contiguousr$rr r) rr(ÚscoresÚmaskr)Ú min_valuerÚp_attnÚxrrrÚforward_attentionEs " ÿ   ÿ z&MultiHeadedAttention.forward_attentionc CsB| |||¡\}}}t || dd¡¡t |j¡}| |||¡S)áËCompute scaled dot product attention. Args: query (torch.Tensor): Query tensor (#batch, time1, size). key (torch.Tensor): Key tensor (#batch, time2, size). value (torch.Tensor): Value tensor (#batch, time2, size). mask (torch.Tensor): Mask tensor (#batch, 1, time2) or (#batch, time1, time2). Returns: torch.Tensor: Output tensor (#batch, time1, d_model). éþÿÿÿr )r-r8r=r%ÚmathÚsqrtr rD) rr&r'r(r@r*r+r,r?rrrÚforwardes zMultiHeadedAttention.forward© Ú__name__Ú __module__Ú __qualname__Ú__doc__r r-rDrIÚ __classcell__rrrrrs   rcs2eZdZdZd ‡fdd„ Zdd„Zdd„Z‡ZS) ÚRelPositionMultiHeadedAttentionaµMulti-Head Attention layer with relative position encoding (new implementation). Details can be found in https://github.com/espnet/espnet/pull/2816. Paper: https://arxiv.org/abs/1901.02860 Args: n_head (int): The number of heads. n_feat (int): The number of features. dropout_rate (float): Dropout rate. zero_triu (bool): Whether to zero the upper triangular part of attention matrix. Fcs|tƒ |||¡||_tj||dd|_t t |j |j ¡¡|_ t t |j |j ¡¡|_ tjj  |j ¡tjj  |j ¡dS)z4Construct an RelPositionMultiHeadedAttention object.F)ÚbiasN)r r Ú zero_triurrÚ linear_posÚ Parameterr8ÚTensorrr Ú pos_bias_uÚ pos_bias_vÚinitÚxavier_uniform_)rrrrrRrrrr ‡sz(RelPositionMultiHeadedAttention.__init__cCstjg| ¡dd…¢d‘R|j|jd}tj||gdd}|jg| ¡dd…¢| d¡d‘| d¡‘RŽ}|dd…dd…dd…f |¡dd…dd…dd…d| d¡dd…f}|jrŠtj | d¡| d¡f|jd}|t  || d¡| d¡¡dddd…dd…f}|S) zùCompute relative positional encoding. Args: x (torch.Tensor): Input tensor (batch, head, time1, 2*time1-1). time1 means the length of query vector. Returns: torch.Tensor: Output tensor. Nér!)Údevicer/r r0r")r[) r8Úzerosr#r[r/Úcatr$Úview_asrRÚonesÚtril)rrCÚzero_padÚx_paddedr_rrrÚ rel_shift”s* 4(ÿ 4z)RelPositionMultiHeadedAttention.rel_shiftcCsÈ| |||¡\}}}| dd¡}| d¡} | |¡ | d|j|j¡} |  dd¡} ||j dd¡} ||j dd¡} t   | | dd¡¡} t   | |  dd¡¡}|  |¡}| |t   |j¡}| |||¡S)aVCompute 'Scaled Dot Product Attention' with rel. positional encoding. Args: query (torch.Tensor): Query tensor (#batch, time1, size). key (torch.Tensor): Key tensor (#batch, time2, size). value (torch.Tensor): Value tensor (#batch, time2, size). pos_emb (torch.Tensor): Positional embedding tensor (#batch, 2*time1-1, size). mask (torch.Tensor): Mask tensor (#batch, 1, time2) or (#batch, time1, time2). Returns: torch.Tensor: Output tensor (#batch, time1, d_model). r!r"rr rF)r-r%r#rSr$rr rVrWr8r=rcrGrHrD)rr&r'r(Úpos_embr@r*r+r,Ú n_batch_posr Ú q_with_bias_uÚ q_with_bias_vÚ matrix_acÚ matrix_bdr?rrrrI­s    z'RelPositionMultiHeadedAttention.forward)F)rKrLrMrNr rcrIrOrrrrrPxs  rPcs<eZdZdZ‡fdd„Zdd„Zd dd„Zd d d „Z‡ZS) ÚMultiHeadSelfAttentionrcsftt|ƒ ¡||dksJ‚|||_||_t ||¡|_t ||d¡|_d|_ tj |d|_ dS)rrrZNr ) r rjr r rrrrÚ linear_q_k_vrrr)rrÚin_featrrrrrr ãs zMultiHeadSelfAttention.__init__c Cs¦| ¡\}}}| |¡}tj|t|j|jƒdd\}}}t ||||j|jf¡ dd¡} t ||||j|jf¡ dd¡} t ||||j|jf¡ dd¡} | | | |fS)rr r0r!r") r#rkr8ÚsplitÚintrr Úreshaper%) rrCÚbÚtÚdÚq_k_vr*r+r,Úq_hÚk_hÚv_hrrrr-ïs "ÿÿÿ z"MultiHeadSelfAttention.forward_qkvNc CsÈ| d¡}|dur<|dur||}| d¡ d¡}tt tjd|jd ¡j¡j ƒ}|  ||¡}tj |dd  |d¡}ntj |dd}|  |¡}t  ||¡} |  dd¡ ¡ |d|j|j¡} | | ¡Sr.r2) rr(r?r@Úmask_att_chunk_encoderr)rArrBrCrrrrD s " ÿ   ÿ z(MultiHeadSelfAttention.forward_attentionc CsH| |¡\}}}}||jd}t || dd¡¡}| ||||¡} | S)rEgà¿rFr )r-r r8r=r%rD) rrCr@rwrtrurvr,r?Úatt_outsrrrrI0s zMultiHeadSelfAttention.forward)NrJrrrrrjÙs   $rj)rNrGr6r8rÚtypingrrÚtorch.nn.functionalÚ functionalÚFÚ*funasr.models.transformer.utils.nets_utilsrÚfunasr.models.lora.layersÚmodelsÚloraÚlayersÚModulerrPrjrrrrÚs  ca