o ´©i¤aã@s’ddlZddlZddlmZmZmZmZddlZddlm m Z ddlm Z m Z ddl mZddlmZGdd„de jƒZGdd „d e jƒZdS) éN)ÚDictÚListÚOptionalÚTuple)ÚTensorÚnn)Ú Parameter)Ú quant_noisec sReZdZd ‡fdd„ Zddefdd„Z  dded ed eeefd d „Z ‡Z S)ÚFairseqDropoutNcs tƒ ¡||_||_d|_dS)NF)ÚsuperÚ__init__ÚpÚ module_nameÚapply_during_inference)Úselfr r©Ú __class__©ú^/home/ubuntu/.local/lib/python3.10/site-packages/funasr/models/data2vec/multihead_attention.pyr s  zFairseqDropout.__init__FÚinplacecCs.|jdkr|js |jrtj||jd|dS|S)NrT)r Útrainingr)r rrÚFÚdropout)rÚxrrrrÚforwardszFairseqDropout.forwardÚnameÚretain_dropoutÚretain_dropout_modulescKsn|r5|dur|jdurt d |¡¡dS|dus|j|vr+t d |¡¡d|_dSt d |¡¡dSdS)NzTCannot enable dropout during inference for module {} because module_name was not setz0Enabling dropout during inference for module: {}Tz Disabling dropout for module: {})rÚloggingÚwarningÚformatÚinfor)rrrrÚkwargsrrrÚmake_generation_fast_sþ  óz$FairseqDropout.make_generation_fast_©N)F)FN) Ú__name__Ú __module__Ú __qualname__r ÚboolrÚstrrrr#Ú __classcell__rrrrr s üþý ür csÒeZdZdZ          d4‡fdd„ Zd d „Zd d „Zd efdd„Zde e eeffdd„Z dd„Z       d5de ede ede ede eeeee effdedede ededede ee effdd „Zede ed!e ed"ed#edede ef d$d%„ƒZejjdeeeee effd&efd'd(„ƒZde eeeee effdeee effd)d*„Zdeeeee effd+eee effd,d-„Zd.ed#ed/efd0d1„Zd2d3„Z‡ZS)6ÚMultiheadAttentionzSMulti-headed attention. See "Attention Is All You Need" for more details. NçTFéc sttƒ ¡||_|dur|n||_|dur|n||_|j|ko#|j|k|_||_t||jj d|_ |||_ |j ||jksBJdƒ‚|j d|_ | |_ | |_|j rX|jsXJdƒ‚ttj|j||d| | ƒ|_ttj|j||d| | ƒ|_ttj|||d| | ƒ|_ttj|||d| | ƒ|_|r¥tt dd|¡ƒ|_tt dd|¡ƒ|_nd|_|_||_| ¡d|_d|_dS)N)rz(embed_dim must be divisible by num_headsgà¿zCSelf-attention requires query, key and value to be of the same size)ÚbiaséF)r r Ú embed_dimÚkdimÚvdimÚ qkv_same_dimÚ num_headsr rr%Údropout_moduleÚhead_dimÚscalingÚself_attentionÚencoder_decoder_attentionr rÚLinearÚk_projÚv_projÚq_projÚout_projrÚtorchrÚbias_kÚbias_vÚ add_zero_attnÚreset_parametersÚ onnx_traceÚskip_embed_dim_check) rr0r4r1r2rr.Ú add_bias_kvrBr8r9Úq_noiseÚ qn_block_sizerrrr <sJ  ÿ ÿÿÿÿÿ  zMultiheadAttention.__init__cCó d|_dS©NT)rD©rrrrÚprepare_for_onnx_export_|ó z+MultiheadAttention.prepare_for_onnx_export_cCsø|jr1tjj|jjdt d¡dtjj|jjdt d¡dtjj|j jdt d¡dntj |jj¡tj |jj¡tj |j j¡tj |j j¡|j j dur`tj  |j j d¡|j durltj |j ¡|jdurztj |j¡dSdS)Nr/é)Úgainr,)r3rÚinitÚxavier_uniform_r;ÚweightÚmathÚsqrtr<r=r>r.Ú constant_r@Úxavier_normal_rArKrrrrCs    ÿz#MultiheadAttention.reset_parametersÚnum_heads_to_keepc sžg}g}g}t|jƒD]z}||j}|d|j}| t t |jj||…f¡¡  ¡t t |jj ||…¡¡  ¡¡| t t |j j||…f¡¡  ¡t t |j j ||…¡¡  ¡¡| t t |j j||…f¡¡  ¡t t |j j ||…¡¡  ¡¡q g‰t|jƒD]}ˆ ||||||¡qt t|jƒ‡fdd„dd}g} t|ƒD]}|||j} ||d|j} |  | | f¡q³| S)Nr/csˆ|Sr$r)Úk©Ú heads_normrrÚ¬sz.T)ÚkeyÚreverse)Úranger4r6Úappendr?ÚsumÚabsr;rRÚtolistr.r=r<Úsorted) rrWÚk_proj_heads_normÚq_proj_heads_normÚv_proj_heads_normÚiÚ start_idxÚend_idxÚsorted_head_indexÚreserve_head_indexÚstartÚendrrYrÚ_get_reserve_head_index“s>  ÿÿ ÿÿ ÿÿ  z*MultiheadAttention._get_reserve_head_indexrkc Csg}g}g}g}g}g}g}|D]Z} | \} } | |jj| | …f¡| |jj| | …¡| |jj| | …f¡| |jj| | …¡| |jj| | …f¡| |jj| | …¡| |jjdd…| | …f¡qt |¡  ¡}t |¡  ¡}t |¡  ¡}tj|dd  ¡}d|_ d|_ d|_ d|_ t |¡  ¡}d|_ t |¡  ¡}d|_ t |¡  ¡}d|_ tj   |¡|j_tj   |¡|j_tj   |¡|j_tj   |¡|j_tj   |¡|j_tj   |¡|j_tj   |¡|j_t |ƒ|_|j|j|_|j|j_|j|j_|j|j_dS)Néÿÿÿÿ©ÚdimT)r_r=rRr.r;r<r>r?ÚcatÚdetachÚ requires_gradrrÚlenr4r6r0Ú out_features) rrkÚ new_q_weightÚ new_q_biasÚ new_k_weightÚ new_k_biasÚ new_v_weightÚ new_v_biasÚnew_out_proj_weightÚelerhrirrrÚ_adaptive_prune_heads´sT    z(MultiheadAttention._adaptive_prune_headscCrIrJ)rErKrrrÚ_set_skip_embed_dim_checkîrMz,MultiheadAttention._set_skip_embed_dim_checkr\ÚvalueÚkey_padding_maskÚincremental_stateÚ need_weightsÚ static_kvÚ attn_maskÚbefore_softmaxÚneed_head_weightsÚreturnc CsH| rd}|jjdk} | ¡\} } }| }|js&||jks&Jd|›d|j›ƒ‚t| ¡ƒ| | |gks3J‚|dur\| ¡\}}}tj ¡s\|| ksIJ‚|dusOJ‚|s\J| |j dd…kƒ‚|j s¹| s¹|dur¹|s¹tj ¡s¹|js¹|durw|dusyJ‚t j ||||j|j t dg¡t |jj|jj|jjf¡|j|j|j|jj|jj|jj|jp©|jj|||d|jj|jj|jjdS|durÙ| |¡}|durØd |vrØ|rØ|jrÒ|jrÔJ‚d}}nd}|jrî| |¡}| |¡}| |¡}n@|jr| |¡}|dur|dusJ‚d}}n&| |¡}| |¡}n|dur|dusJ‚| |¡}| |¡}| |¡}||j 9}|jdur…|jdusAJ‚t ||j !d | d ¡g¡}t ||j !d | d ¡g¡}|durptj|| "| d¡d ¡gd d }|dur…tj|| "| d¡d ¡gd d }| #¡ $| | |j |j%¡ &dd ¡}|dur¬| #¡ $d | |j |j%¡ &dd ¡}|durÂ| #¡ $d | |j |j%¡ &dd ¡}|dur{d |vrý|d }|dus×J‚| $| |j d |j%¡}|rè|}n|dusïJ‚tj||gd d }| d ¡}d |vr.|d }|dus J‚| $| |j d |j%¡}|r|}n|dus%J‚tj||gd d }d}d|vr9|d}|durC|dusEJ‚t'j(||| | d ¡|d}| $| |j d |j%¡|d <| $| |j d |j%¡|d <||d<|dusuJ‚| )||¡}|dus‚J‚| d ¡|ksŒJ‚|durš| *¡dkršd}|dur³| d¡| ks©J‚| d ¡|ks³J‚|jr!|dus¾J‚|d 7}tj|| "| d¡d f| ¡dd…¡gd d }tj|| "| d¡d f| ¡dd…¡gd d }|dur tj|| "| d¡d ¡gd d }|dur!tj|t +| d¡d ¡ ,|¡gd d }t -|| &d d¡¡}| .|| || ¡}t| ¡ƒ| |j | |gksDJ‚|dur`| /d¡}|j r\| !| d¡d d ¡}||7}|dur¢| $| |j | |¡}| s„| 0| /d ¡ /d¡ 1tj2¡t3dƒ¡}n| &dd¡}| 0|t3dƒ¡}| &dd¡}| $| |j | |¡}| r©||fSt j4|d tj5d}| ,|¡}| |¡}|dusÃJ‚t -||¡}t| ¡ƒ| |j | |j%gksÛJ‚|j rò| d ¡d krò| #¡ $| | |j¡}n| &dd ¡ #¡ $| | |j¡}| |¡}d}|r | $| |j | |¡ &d d¡}| s |j6dd }||fS)a¥Input shape: Time x Batch x Channel Args: key_padding_mask (ByteTensor, optional): mask to exclude keys that are pads, of shape `(batch, src_len)`, where padding elements are indicated by 1s. need_weights (bool, optional): return the attention weights, averaged over heads (default: False). attn_mask (ByteTensor, optional): typically used to implement causal attention, where the mask prevents the attention from looking forward in time (default: None). before_softmax (bool, optional): return the raw attention weights and values before the attention softmax. need_head_weights (bool, optional): return the attention weights for each head. Implies *need_weights*. Default: return the average attention weights over all heads. TÚxlaz query dim z != NrNr)Úuse_separate_proj_weightÚ q_proj_weightÚ k_proj_weightÚ v_proj_weightÚprev_keyr/rproÚ prev_valueÚprev_key_padding_mask)r‚r‘Ú batch_sizeÚsrc_lenr…z-inf)rqÚdtype)7ÚdeviceÚtypeÚsizerEr0Úlistr?ÚjitÚ is_scriptingÚshaperDrÚmulti_head_attention_forwardr4Úemptyrrr=r.r;r<r@rArBr5r r>rRrrÚ_get_input_bufferr9r8r7ÚrepeatÚ new_zerosÚ contiguousÚviewr6Ú transposer+Ú_append_prev_key_padding_maskÚ_set_input_bufferrqÚzerosÚtype_asÚbmmÚapply_sparse_maskÚ unsqueezeÚ masked_fillÚtor(ÚfloatÚsoftmaxÚfloat32Úmean)rÚqueryr\rr‚rƒr„r…r†r‡rˆÚis_tpuÚtgt_lenÚbszr0r“Úkey_bszÚ_Ú saved_stateÚqrXÚvÚ _prev_keyrÚ _prev_valuerr‘Ú attn_weightsÚattn_weights_floatÚ attn_probsÚattnrrrrñsr     ÿþÿü ÷   ë €                þû" " "     û  22   þû"   þ     $ ÿ zMultiheadAttention.forwardr‘r’r“cCs|dur |r |}|S|dur!|dur!tj| ¡| ¡gdd}|S|durP|| d¡krJtj||| d¡f|jd}tj| ¡| ¡gdd}|S| ¡}|S|dur|| d¡krytj||| d¡f|jd}tj| ¡| ¡gdd}|S| ¡}|S|}|S)Nr/rp)r•)r?rrr­r—r¦r•)r‚r‘r’r“r…Únew_key_padding_maskÚfillerrrrr¤ôs@ ãÿêþÿô õþýÿz0MultiheadAttention._append_prev_key_padding_maskÚ new_ordercCsn| |¡}|dur5| ¡D]!}||}|dur.|jr&| d¡| d¡kr&n | d|¡||<q | ||¡}|S)z=Reorder buffered internal state (for incremental generation).Nr)ržÚkeysr9r—Ú index_selectr¥)rrƒrÂÚ input_bufferrXÚinput_buffer_krrrÚreorder_incremental_states  ÿ€ z,MultiheadAttention.reorder_incremental_statecCs | |d¡}|dur |Si}|S©NÚ attn_state)Úget_incremental_state)rrƒÚresultÚ empty_resultrrrrž2s z$MultiheadAttention._get_input_bufferÚbuffercCs| |d|¡SrÈ)Úset_incremental_state)rrƒrÍrrrr¥<sz$MultiheadAttention._set_input_bufferr³r´cCs|Sr$r)rr¼r³r“r´rrrr©Csz$MultiheadAttention.apply_sparse_maskc Cs^|dkr|dnd}i}g}| ¡D]…}| |d¡r—t||jddƒ}||d|…||d<|||d|…||d<||d|d…||d <| |¡|d }|| ¡vr—t||jddƒ}||d|…||d <|||d|…||d <||d|d…||d <| |d ¡q|D]}||=qš| ¡D]\} } | || <q¤dS)NÚÚ.Úin_proj_weightréz q_proj.weightrNz k_proj.weightz v_proj.weightÚ in_proj_biasz q_proj.biasz k_proj.biasz v_proj.bias)rÃÚendswithÚintr›r_Úitems) rÚ state_dictrÚprefixÚ items_to_addÚkeys_to_removerXrqÚk_biasr\rrrrÚupgrade_state_dict_namedFs.   € ÿz+MultiheadAttention.upgrade_state_dict_named) NNr,TFFFFr,r-)NNTFNFF)r%r&r'Ú__doc__r rLrCrÕrnrrrr€rrrr)r(rÚ staticmethodr¤r?r™ÚexportrÇržr¥r©rÜr*rrrrr+6s  ó@!:õýüûúùø ÷ ö õ ôÿþýüû ú)þ ýÿ þ þ ýr+)rrSÚtypingrrrrr?Útorch.nn.functionalrÚ functionalrrÚtorch.nnrÚ"funasr.models.data2vec.quant_noiser ÚModuler r+rrrrÚs  $