o i@sddlZddlZddlmZddlmmZddlmZGdddej Z Gdddej Z Gdddej ZGd d d ej Zdd d ZddZdejdejfddZddZdefddZddZdS)N)MultiheadAttentionc$eZdZfddZddZZS) Fp32LayerNormctj|i|dSNsuper__init__selfargskwargs __class__P/home/ubuntu/.local/lib/python3.10/site-packages/funasr/models/data2vec/utils.pyr zFp32LayerNorm.__init__cCLt||j|jdur|jnd|jdur|jnd|j}||Sr)F layer_normfloatnormalized_shapeweightbiasepstype_asr inputoutputrrrforward zFp32LayerNorm.forward__name__ __module__ __qualname__r r __classcell__rrrrr rcr) Fp32GroupNormcrrrr rrrr rzFp32GroupNorm.__init__cCrr)r group_normr num_groupsrrrrrrrrr"r zFp32GroupNorm.forwardr!rrrrr'r&r'c&eZdZdfdd ZddZZS) TransposeLastNcst||_dSr)rr deconstruct_idx)r r,rrrr .s  zTransposeLast.__init__cCs |jdur ||j}|ddS)N)r, transposer xrrrr2s   zTransposeLast.forwardrr!rrrrr+-sr+cr*)SamePadFcs6t|r|d|_dS|ddkrdnd|_dS)Nr)rr remove)r kernel_sizecausalrrrr 9s zSamePad.__init__cCs,|jdkr|ddddd|j f}|SNr)r5r0rrrr@s zSamePad.forward)Fr!rrrrr28sr2r.cCsr|durdS||}||}t|||}|r |dfSdd|d}tj|g|d|R|d|fS)Nr8r)rr.r4)value)sizemathceil is_integerrpad)r1multipledimr9tszm remainder pad_offsetrrrpad_to_multipleFs "rEc CsHttdstdtjt_d|dttj|dt|dS)N_ar4g?r3gHm?) hasattr gelu_accurater;sqrtpirFtorchtanhpowr1rrrrITs ,rIr1returncCstjj||Sr)rLnn functionalgelurrrOrrrrSZsrScCsgdS)N)relurS gelu_fastrIrMlinearrrrrrget_available_activation_fns^srW activationcCsb|dkrtjS|dkr tS|dkrtS|dkrtjS|dkr"ddS|dkr*tjjStd |) z=Returns the activation function corresponding to `activation`rTrSrIrMrVcSs|SrrrOrrrusz#get_activation_fn..swishz --activation-fn {} not supported) rrTrSrIrLrMrQSiLU RuntimeErrorformat)rXrrrget_activation_fnisr^cCsdd}t|tjr||jj|jdur|jjt|tjr5||jj|jdur5|jj|jt|t rQ||j jj||j jj||j jjdSdS)a Initialize the weights specific to the BERT Model. This overrides the default initializations depending on the specified arguments. 1. If normal_init_linear_weights is set then weights of linear layer will be initialized using the normal distribution and bais will be set to the specified value. 2. If normal_init_embed_weights is set then weights of embedding layer will be initialized using the normal distribution. 3. If normal_init_proj_weights is set then weights of in_project_weight for MultiHeadAttention initialized using the normal distribution (to be validated). cSs$||jddd|jdS)Ngg{Gz?)meanstd)copy_cpunormal_todevice)datarrrrcs$z!init_bert_params..normal_N) isinstancerQLinearrrfrzero_ Embedding padding_idxrq_projk_projv_proj)modulercrrrinit_bert_params|s        rp)r.r)r;rLtorch.nnrQtorch.nn.functionalrRr*funasr.models.data2vec.multihead_attentionr LayerNormr GroupNormr'Moduler+r2rErITensorrSrWstrr^rprrrrs