o ´©i8[ã@sÚdZddlZddlZddlZddlmZddlmZmZddlmm Z ddl m Z ddl mmmZGdd„dejƒZGdd „d ejƒZGd d „d eƒZGd d „d eƒZGdd„deƒZGdd„dejjƒZdS)z&Multi-Head Attention layer definition.éN)Únn)ÚOptionalÚTuple)Ú make_pad_maskcs8eZdZdZ‡fdd„Zdd„Zdd„Zdd „Z‡ZS) ÚMultiHeadedAttentionz±Multi-Head Attention layer. Args: n_head (int): The number of heads. n_feat (int): The number of features. dropout_rate (float): Dropout rate. cs~tt|ƒ ¡||dksJ‚|||_||_t ||¡|_t ||¡|_t ||¡|_ t ||¡|_ d|_ tj |d|_ dS)ú)Construct an MultiHeadedAttention object.rN©Úp)ÚsuperrÚ__init__Úd_kÚhrÚLinearÚlinear_qÚlinear_kÚlinear_vÚ linear_outÚattnÚDropoutÚdropout)ÚselfÚn_headÚn_featÚ dropout_rate©Ú __class__©úW/home/ubuntu/.local/lib/python3.10/site-packages/funasr/models/transformer/attention.pyr s zMultiHeadedAttention.__init__cCs†| d¡}| |¡ |d|j|j¡}| |¡ |d|j|j¡}| |¡ |d|j|j¡}| dd¡}| dd¡}| dd¡}|||fS)a Transform query, key and value. Args: query (torch.Tensor): Query tensor (#batch, time1, size). key (torch.Tensor): Key tensor (#batch, time2, size). value (torch.Tensor): Value tensor (#batch, time2, size). Returns: torch.Tensor: Transformed query tensor (#batch, n_head, time1, d_k). torch.Tensor: Transformed key tensor (#batch, n_head, time2, d_k). torch.Tensor: Transformed value tensor (#batch, n_head, time2, d_k). réÿÿÿÿéé)ÚsizerÚviewr r rrÚ transpose©rÚqueryÚkeyÚvalueÚn_batchÚqÚkÚvrrrÚ forward_qkv-s     z MultiHeadedAttention.forward_qkvc Cs | d¡}|dur(| d¡ d¡}tdƒ }| ||¡}tj|dd |d¡}ntj|dd}| |¡}t ||¡}|  dd¡  ¡  |d|j |j ¡}| |¡S) aÒCompute attention context vector. Args: value (torch.Tensor): Transformed value (#batch, n_head, time2, d_k). scores (torch.Tensor): Attention score (#batch, n_head, time1, time2). mask (torch.Tensor): Mask (#batch, 1, time2) or (#batch, time1, time2). Returns: torch.Tensor: Transformed value (#batch, time1, d_model) weighted by the attention score (#batch, time1, time2). rNrÚinfr©Údimçr )r!Ú unsqueezeÚeqÚfloatÚ masked_fillÚtorchÚsoftmaxrÚmatmulr#Ú contiguousr"r r r) rr'ÚscoresÚmaskr(Ú min_valuerÚp_attnÚxrrrÚforward_attentionEs ÿ ÿ   ÿ z&MultiHeadedAttention.forward_attentionc CóB| |||¡\}}}t || dd¡¡t |j¡}| |||¡S)aËCompute scaled dot product attention. Args: query (torch.Tensor): Query tensor (#batch, time1, size). key (torch.Tensor): Key tensor (#batch, time2, size). value (torch.Tensor): Value tensor (#batch, time2, size). mask (torch.Tensor): Mask tensor (#batch, 1, time2) or (#batch, time1, time2). Returns: torch.Tensor: Output tensor (#batch, time1, d_model). éþÿÿÿr©r,r5r7r#ÚmathÚsqrtr r>© rr%r&r'r:r)r*r+r9rrrÚforwardhs zMultiHeadedAttention.forward) Ú__name__Ú __module__Ú __qualname__Ú__doc__r r,r>rEÚ __classcell__rrrrrs  #rcsJeZdZ‡fdd„Zdd„Zdejdejfdd„Zd d „Zd d „Z ‡Z S) ÚMultiHeadedAttentionExportcsRtƒ ¡|j|_|j|_|j|_|j|_|j|_|j|_d|_|j|j|_ dS©N) r r r r rrrrrÚ all_head_size©rÚmodelrrrr |s z#MultiHeadedAttentionExport.__init__c Cr?)Nr@rrArDrrrrE‡s z"MultiHeadedAttentionExport.forwardr=ÚreturncCs6| ¡dd…|j|jf}| |¡}| dddd¡S)Nrrr ré)r!r r r"Úpermute)rr=Ú new_x_shaperrrÚtranspose_for_scoresŒs z/MultiHeadedAttentionExport.transpose_for_scorescCsF| |¡}| |¡}| |¡}| |¡}| |¡}| |¡}|||fSrL)rrrrT)rr%r&r'r)r*r+rrrr,‘s       z&MultiHeadedAttentionExport.forward_qkvcCób||}tj|dd}t ||¡}| dddd¡ ¡}| ¡dd…|jf}| |¡}| |¡S©Nrr.rr rrQr@© r5r6r7rRr8r!rMr"r©rr'r9r:rÚ context_layerÚnew_context_layer_shaperrrr>šó   z,MultiHeadedAttentionExport.forward_attention) rFrGrHr rEr5ÚTensorrTr,r>rJrrrrrK{s    rKcs4eZdZ‡fdd„Zdd„Zdd„Zdd„Z‡ZS) Ú RelPosMultiHeadedAttentionExportcs(tƒ |¡|j|_|j|_|j|_dSrL)r r Ú linear_posÚ pos_bias_uÚ pos_bias_vrNrrrr §s  z)RelPosMultiHeadedAttentionExport.__init__cCs¨| |||¡\}}}| dd¡}| | |¡¡} ||j dd¡} ||j dd¡} t | | dd¡¡} t | |  dd¡¡} | | ¡} | | t   |j ¡}|  |||¡S)Nrr r@r) r,r#rTr^r_r`r5r7Ú rel_shiftrBrCr r>)rr%r&r'Úpos_embr:r)r*r+r Ú q_with_bias_uÚ q_with_bias_vÚ matrix_acÚ matrix_bdr9rrrrE­s  z(RelPosMultiHeadedAttentionExport.forwardcCs¾tjg| ¡dd…¢d‘R|j|jd}tj||gdd}|jg| ¡dd…¢| d¡d‘| d¡‘RŽ}|dd…dd…dd…f |¡dd…dd…dd…d| d¡dd…f}|S)NrQr©ÚdeviceÚdtyperr.r )r5Úzerosr!rhriÚcatr"Úview_as)rr=Úzero_padÚx_paddedrrrraÇs*4(ÿz*RelPosMultiHeadedAttentionExport.rel_shiftcCrUrVrWrXrrrr>Ñr[z2RelPosMultiHeadedAttentionExport.forward_attention)rFrGrHr rErar>rJrrrrr]¦s   r]có2eZdZdZd ‡fdd„ Zdd„Zdd„Z‡ZS) Ú%LegacyRelPositionMultiHeadedAttentiona®Multi-Head Attention layer with relative position encoding (old version). Details can be found in https://github.com/espnet/espnet/pull/2816. Paper: https://arxiv.org/abs/1901.02860 Args: n_head (int): The number of heads. n_feat (int): The number of features. dropout_rate (float): Dropout rate. zero_triu (bool): Whether to zero the upper triangular part of attention matrix. Fcó|tƒ |||¡||_tj||dd|_t t |j |j ¡¡|_ t t |j |j ¡¡|_ tjj  |j ¡tjj  |j ¡dS©z4Construct an RelPositionMultiHeadedAttention object.F©ÚbiasN©r r Ú zero_triurrr^Ú Parameterr5r\r r r_r`ÚinitÚxavier_uniform_©rrrrrvrrrr ìóz.LegacyRelPositionMultiHeadedAttention.__init__cCsètjg| ¡dd…¢d‘R|j|jd}tj||gdd}|jg| ¡dd…¢| d¡d‘| d¡‘RŽ}|dd…dd…dd…f |¡}|jrrt  | d¡| d¡f¡}|t  || d¡| d¡¡dddd…dd…f}|S)zÁCompute relative positional encoding. Args: x (torch.Tensor): Input tensor (batch, head, time1, time2). Returns: torch.Tensor: Output tensor. NrQrrgrr.r © r5rjr!rhrirkr"rlrvÚonesÚtril©rr=rmrnr}rrrraùs* 4 4z/LegacyRelPositionMultiHeadedAttention.rel_shiftcCóÈ| |||¡\}}}| dd¡}| d¡} | |¡ | d|j|j¡} |  dd¡} ||j dd¡} ||j dd¡} t   | | dd¡¡} t   | |  dd¡¡}|  |¡}| |t   |j¡}| |||¡S)aBCompute 'Scaled Dot Product Attention' with rel. positional encoding. Args: query (torch.Tensor): Query tensor (#batch, time1, size). key (torch.Tensor): Key tensor (#batch, time2, size). value (torch.Tensor): Value tensor (#batch, time2, size). pos_emb (torch.Tensor): Positional embedding tensor (#batch, time1, size). mask (torch.Tensor): Mask tensor (#batch, 1, time2) or (#batch, time1, time2). Returns: torch.Tensor: Output tensor (#batch, time1, d_model). rr rrr@©r,r#r!r^r"r r r_r`r5r7rarBrCr>©rr%r&r'rbr:r)r*r+Ú n_batch_posr rcrdrerfr9rrrrEs    z-LegacyRelPositionMultiHeadedAttention.forward©F©rFrGrHrIr rarErJrrrrrpÝs  rpcro) ÚRelPositionMultiHeadedAttentionaµMulti-Head Attention layer with relative position encoding (new implementation). Details can be found in https://github.com/espnet/espnet/pull/2816. Paper: https://arxiv.org/abs/1901.02860 Args: n_head (int): The number of heads. n_feat (int): The number of features. dropout_rate (float): Dropout rate. zero_triu (bool): Whether to zero the upper triangular part of attention matrix. Fcrqrrrurzrrrr Ir{z(RelPositionMultiHeadedAttention.__init__cCstjg| ¡dd…¢d‘R|j|jd}tj||gdd}|jg| ¡dd…¢| d¡d‘| d¡‘RŽ}|dd…dd…dd…f |¡dd…dd…dd…d| d¡dd…f}|jrŠtj | d¡| d¡f|jd}|t  || d¡| d¡¡dddd…dd…f}|S) zùCompute relative positional encoding. Args: x (torch.Tensor): Input tensor (batch, head, time1, 2*time1-1). time1 means the length of query vector. Returns: torch.Tensor: Output tensor. NrQrrgrr.r )rhr|rrrrraVs* 4(ÿ 4z)RelPositionMultiHeadedAttention.rel_shiftcCr€)aVCompute 'Scaled Dot Product Attention' with rel. positional encoding. Args: query (torch.Tensor): Query tensor (#batch, time1, size). key (torch.Tensor): Key tensor (#batch, time2, size). value (torch.Tensor): Value tensor (#batch, time2, size). pos_emb (torch.Tensor): Positional embedding tensor (#batch, 2*time1-1, size). mask (torch.Tensor): Mask tensor (#batch, 1, time2) or (#batch, time1, time2). Returns: torch.Tensor: Output tensor (#batch, time1, d_model). rr rrr@rr‚rrrrEos    z'RelPositionMultiHeadedAttention.forwardr„r…rrrrr†:s  r†csNeZdZdZ  d"dededededd f ‡fd d „ Zd#d ej dedej fdd„Z d#dej dej dej dedej f dd„Z d#dej dej dej dedej f dd„Z dej dej dej de ej ej ej ffdd„Z d$dej dej dej deej dej f dd„Z d%dej dej dej dej dej deej dedej fd d!„Z‡ZS)&Ú$RelPositionMultiHeadedAttentionChunkz°RelPositionMultiHeadedAttention definition. Args: num_heads: Number of attention heads. embed_size: Embedding size. dropout_rate: Dropout rate. r0FÚ num_headsÚ embed_sizerÚsimplified_attention_scorerPNcstƒ ¡|||_||_|j||ksJd||ffƒ‚tj ||¡|_tj ||¡|_tj ||¡|_ tj ||¡|_ |rKtj ||¡|_ |j |_ n6tjj||dd|_ tj t ||j¡¡|_tj t ||j¡¡|_tjj |j¡tjj |j¡|j|_ tjj|d|_d|_dS)rz3embed_size (%d) must be divisible by num_heads (%d)FrsrN)r r r rˆr5rrrrrrr^Ú"compute_simplified_attention_scoreÚcompute_att_scorerwr\r_r`rxryÚcompute_attention_scorerrr)rrˆr‰rrŠrrrr £s,  þ  z-RelPositionMultiHeadedAttentionChunk.__init__rr=Ú left_contextc CsR|j\}}}}||}| ¡\}} } } |j||||f|| | | | f| |ddS)zòCompute relative positional encoding. Args: x: Input sequence. (B, H, T_1, 2 * T_1 - 1) left_context: Number of frames in left context. Returns: x: Output sequence. (B, H, T_1, T_2) r)Ústorage_offset)ÚshapeÚstrideÚ as_strided) rr=rŽÚ batch_sizeÚn_headsÚtime1ÚnÚtime2Ú batch_strideÚn_heads_strideÚ time1_strideÚn_striderrrraÌs  ýz.RelPositionMultiHeadedAttentionChunk.rel_shiftr%r&Úpos_enccCs`| |¡}t || dd¡¡}|j| dd¡ d¡ dd| d¡d¡|d}||t  |j ¡S)aºSimplified attention score computation. Reference: https://github.com/k2-fsa/icefall/pull/458 Args: query: Transformed query tensor. (B, H, T_1, d_k) key: Transformed key tensor. (B, H, T_2, d_k) pos_enc: Positional embedding tensor. (B, 2 * T_1 - 1, size) left_context: Number of frames in left context. Returns: : Attention score. (B, H, T_1, T_2) r rQr©rŽ) r^r5r7r#rar1Úrepeatr!rBrCr )rr%r&rœrŽrerfrrrr‹ßs "þzGRelPositionMultiHeadedAttentionChunk.compute_simplified_attention_scorec Csž| |¡ | d¡d|j|j¡}| dd¡}||j dd¡}||j dd¡}t  || dd¡¡}t  ||  dddd¡¡} |j | |d} || t   |j¡S)aqAttention score computation. Args: query: Transformed query tensor. (B, H, T_1, d_k) key: Transformed key tensor. (B, H, T_2, d_k) pos_enc: Positional embedding tensor. (B, 2 * T_1 - 1, size) left_context: Number of frames in left context. Returns: : Attention score. (B, H, T_1, T_2) rrrr r@rQr)r^r"r!rˆr r#r_r`r5r7rRrarBrC) rr%r&rœrŽr rcrdrerfrrrrûs  z-s    ÿz6RelPositionMultiHeadedAttentionChunk.forward_attentionc Cs8| |||¡\}} } |j|| ||d} |j| | ||dS)aCompute scaled dot product attention with rel. positional encoding. Args: query: Query tensor. (B, T_1, size) key: Key tensor. (B, T_2, size) value: Value tensor. (B, T_2, size) pos_enc: Positional embedding tensor. (B, 2 * T_1 - 1, size) mask: Source mask. (B, T_2) chunk_mask: Chunk mask. (T_1, T_1) left_context: Number of frames in left context. Returns: : Output tensor. (B, T_1, H * d_k) r)rŸ)r,rŒr>) rr%r&r'rœr:rŸrŽr)r*r+r9rrrrEMsz,RelPositionMultiHeadedAttentionChunk.forward)r0F)rrL)Nr)rFrGrHrIÚintr3Úboolr r5r\rar‹rrr,rr>rErJrrrrr‡›s  ûþýüûú)ûþýüû ú!ûþýüû úÿÿÿ þûþýüû ú'øþýüûúùø ÷r‡)rIrBÚnumpyr5rÚtypingrrÚtorch.nn.functionalÚ functionalÚFÚ*funasr.models.transformer.utils.nets_utilsrÚfunasr.models.lora.layersÚmodelsÚloraÚlayersÚModulerrKr]rpr†r‡rrrrÚs  f+7]a