o il@sdZddlZddlmZmZmZmZmZddlZddlm Z ddl m Z ddl m Z mZmZmZddlmZmZmZmZmZddlmZdd lmZdd lmZdd lmZdd lmZdd lmZm Z m!Z!m"Z"ddl#m$Z$ddl%m&Z&m'Z'ddl(m)Z)ddl(m*Z*ddl(m+Z+ddl(m,Z,ddl(mZddl(m Z ddl(m-Z-ddl(m.Z.ddl/m0Z0ddl1Z1Gddde j2Z3Gddde j2Z4e05ddGddde j2Z6Gd d!d!ej j2Z7Gd"d#d#ej j2Z8e05dd$Gd%d&d&ej j2Z9dS)'zConformer encoder definition.N)UnionDictListTupleOptional)nn)CTC)MultiHeadedAttentionRelPositionMultiHeadedAttention%LegacyRelPositionMultiHeadedAttention$RelPositionMultiHeadedAttentionChunk)PositionalEncodingScaledPositionalEncodingRelPositionalEncodingLegacyRelPositionalEncodingStreamingRelPositionalEncoding) LayerNorm) Conv1dLinear)MultiLayeredConv1d)get_activation) make_pad_mask)TooShortUttErrorcheck_short_uttmake_chunk_maskmake_source_mask)PositionwiseFeedForward)repeat MultiBlocks)Conv2dSubsampling)Conv2dSubsampling2)Conv2dSubsampling6)Conv2dSubsampling8)r)r)Conv2dSubsamplingPad)StreamingConvInput)tablescs2eZdZdZedffdd ZddZZS)ConvolutionModulezConvolutionModule in Conformer model. Args: channels (int): The number of channels of conv layers. kernel_size (int): Kernerl size of conv layers. Tc stt||dddksJtj|d|ddd|d|_tj|||d|dd||d|_t||_tj||ddd|d|_ ||_ dS)z&Construct an ConvolutionModule object.r) kernel_sizestridepaddingbias)r)r*groupsr+N) superr%__init__rConv1dpointwise_conv1depthwise_conv BatchNorm1dnormpointwise_conv2 activation)selfchannelsr(r5r+ __class__S/home/ubuntu/.local/lib/python3.10/site-packages/funasr/models/conformer/encoder.pyr.>s:   zConvolutionModule.__init__cCsV|dd}||}tjj|dd}||}|||}||}|ddS)zCompute convolution module. Args: x (torch.Tensor): Input tensor (#batch, time, channels). Returns: torch.Tensor: Output tensor (#batch, time, channels). r&r'dim) transposer0r functionalglur1r5r3r4)r6xr:r:r;forward`s    zConvolutionModule.forward) __name__ __module__ __qualname____doc__rReLUr.rB __classcell__r:r:r8r;r%5s"r%cs2eZdZdZ   d fdd Zd dd ZZS) EncoderLayeraEncoder layer module. Args: size (int): Input dimension. self_attn (torch.nn.Module): Self-attention module instance. `MultiHeadedAttention` or `RelPositionMultiHeadedAttention` instance can be used as the argument. feed_forward (torch.nn.Module): Feed-forward module instance. `PositionwiseFeedForward`, `MultiLayeredConv1d`, or `Conv1dLinear` instance can be used as the argument. feed_forward_macaron (torch.nn.Module): Additional feed-forward module instance. `PositionwiseFeedForward`, `MultiLayeredConv1d`, or `Conv1dLinear` instance can be used as the argument. conv_module (torch.nn.Module): Convolution module instance. `ConvlutionModule` instance can be used as the argument. dropout_rate (float): Dropout rate. normalize_before (bool): Whether to use layer_norm before the first block. concat_after (bool): Whether to concat attention layer's input and output. if True, additional linear will be applied. i.e. x -> x + linear(concat(x, att(x))) if False, no additional linear will be applied. i.e. x -> x + att(x) stochastic_depth_rate (float): Proability to skip this layer. During training, the layer may skip residual computation and return input as-is with given probability. TFc stt|||_||_||_||_t||_t||_ |dur*t||_ d|_ nd|_ |jdur x + linear(concat(x, att(x))) If False, no additional linear will be applied. i.e. x -> x + att(x) positionwise_layer_type (str): "linear", "conv1d", or "conv1d-linear". positionwise_conv_kernel_size (int): Kernel size of positionwise conv1d layer. rel_pos_type (str): Whether to use the latest relative positional encoding or the legacy one. The legacy relative positional encoding will be deprecated in the future. More Details can be found in https://github.com/espnet/espnet/pull/2816. encoder_pos_enc_layer_type (str): Encoder positional encoding layer type. encoder_attn_layer_type (str): Encoder attention layer type. activation_type (str): Encoder activation function type. macaron_style (bool): Whether to use macaron style for positionwise layer. use_cnn_module (bool): Whether to use convolution module. zero_triu (bool): Whether to zero the upper triangular part of attention matrix. cnn_module_kernel (int): Kernerl size of convolution module. padding_idx (int): Padding idx for input_layer=embed. 皙?rJconv2dTFlinearlegacyrel_pos rel_selfattnswishr` input_size output_sizeattention_heads linear_units num_blocksr_positional_dropout_rateattention_dropout_rate input_layerrZr[positionwise_layer_typepositionwise_conv_kernel_size macaron_style rel_pos_typepos_enc_layer_typeselfattention_layer_typeactivation_typeuse_cnn_module zero_triucnn_module_kernel padding_idxinterctc_layer_idxinterctc_use_conditioningr^cst|_|dkr|dkrd}|dkrd}n|dkr*|dks#J|dks)Jntd|t|}|dkr;t}n,|d krBt}n%|dkrO|dksLJt}n|dkra|dksYJt}t d ntd || d krt j t j |t j t j |||_n| d krt||||_n| dkrt||||_nq| dkrt||||_na| dkrt||||_nQ| dkrt||||_nA| dkrt j t j j||d|||_n*t| t j jrt j | |||_n| durt j |||_ntd| |_| d kr(t ||f n | dkr6t || f n| dkrDt || f ntd|dkrUt||fn4|dkrn|dksaJt ||ft dn|dkr|dkszJt!|||fntd|t"||ft t#r g| t$ |krtdt$ d|dt%| f dd|_&|jrt|_'||_(t$|d krd t)|krt*||ksJ||_+d|_,dS)!Nr~rlegacy_rel_posrlegacy_rel_selfattnlatestzunknown rel_pos_type: abs_posscaled_abs_posz=Using legacy_rel_pos and it will be deprecated in the future.zunknown pos_enc_layer: r|r{ conv2dpadconv2d2conv2d6conv2d8embed)rzunknown input_layer: conv1dz conv1d-linearzSupport only linear or conv1d.selfattnzBUsing legacy_rel_selfattn and it will be deprecated in the future.zunknown encoder_attn_layer: z!Length of stochastic_depth_rate (z!) should be equal to num_blocks ()c s<t r nd rnd | Srs)rI)lnum r[convolution_layerconvolution_layer_argsr_encoder_selfattn_layerencoder_selfattn_layer_argsrrZrpositionwise_layerpositionwise_layer_argsr^rr:r;sz+ConformerEncoder.__init__..r)-r-r. _output_size ValueErrorrr rrrloggingwarningrdr Sequentialr\rrWrrr"rr r! EmbeddingraModulerZrrrNotImplementedErrorr r r r%floatlenrencoders after_normrminmaxrconditioning_layer)r6rrrrrr_rrrrZr[rrrrrrrrrrrrrr^r5 pos_enc_classr8rr;r.Bs&                        "   zConformerEncoder.__init__returncC|jSrsrr6r:r:r;rzConformerEncoder.output_sizeNxs_padilens prev_statesctccCst|dddddf|j}t|jts0t|jts0t|jts0t|jts0t|jt r\t |j| d\}}|rSt d| ddd|d| d||||\}}n||}g}t |jdkrs|||\}}nXt|jD]R\} } | ||\}}| d|jvr|} t| tr| d} |jr|| } || d| f|jr|| } t|tr|\} }| || } | |f}qx||| }qxt|tr|d}|jr||}|dd}t |dkr||f|dfS||dfS)aCalculate forward propagation. Args: xs_pad (torch.Tensor): Input tensor (#batch, L, input_size). ilens (torch.Tensor): Input length (#batch). prev_states (torch.Tensor): Not to be used now. Returns: torch.Tensor: Output tensor (#batch, L, output_size). torch.Tensor: Output length (#batch). torch.Tensor: Not to be used now. Nr&has ) frames and is too short for subsampling (it needs more than  frames), return empty resultsr)rtodevicerarrrr r!r"rrYrrrr enumeraterbrZrappendrsoftmaxrsqueezesum)r6rrrrmasks short_status limit_sizeintermediate_outs layer_idx encoder_layer encoder_outctc_outrArlolensr:r:r;rBsd$                zConformerEncoder.forward)NN)rCrDrErFintrstrboolrrr.rrdTensorrrrrBrHr:r:r8r;rus%     Mc seZdZdZejidfdededejjde de dd f fd d Z dd ej de ej dedeej ej ffddZZS)CausalConvolutionaFConformerConvolution module definition. Args: channels: The number of channels. kernel_size: Size of the convolving kernel. activation: Type of activation function. norm_args: Normalization module arguments. causal: Whether to use causal convolution (set to True if streaming). Fr7r(r5 norm_argscausalrNcst|dddksJ||_tjj|d|dddd|_|r*|d|_d}n d|_|dd}tjj|||d||d|_tjj |fi||_ tjj||dddd|_ ||_ dS)z)Construct an ConformerConvolution object.r&r'r)r(r)r*)r)r*r,N) r-r.r(rdrr/r0lorderr1r2r3r4r5)r6r7r(r5rrr*r8r:r;r.ks@    zCausalConvolution.__init__rrArk right_contextcCs||dd}tjjj|dd}|jdkrX|dur)tjj||jdfdd}n/tj||gdd}|dkrI|dddd|j| | f}n|dddd|j df}| |}| | |}| |dd}||fS)aCompute convolution module. Args: x: ConformerConvolution input sequences. (B, T, D_hidden) cache: ConformerConvolution input cache. (1, conv_kernel, D_hidden) right_context: Number of frames in right context. Returns: x: ConformerConvolution output sequences. (B, T, D_hidden) cache: ConformerConvolution output cache. (1, conv_kernel, D_hidden) r&r'r<rNconstantrJ) r0r>rdrr?r@rpadrgr1r5r3r4)r6rArkrr:r:r;rBs & zCausalConvolution.forward)Nr)rCrDrErFrdrrGrrrrr.rrrrBrHr:r:r8r;ras: 4rcseZdZdZeidfdedejjdejjdejjdejjdejjd e d e d d ffd d Z dedej d d fddZ ddejdejdejdeejd eejejejff ddZ   d dejdejdejdededed eejejffddZZS)!ChunkEncoderLayeraChunk Conformer module definition. Args: block_size: Input/output size. self_att: Self-attention module instance. feed_forward: Feed-forward module instance. feed_forward_macaron: Feed-forward module instance for macaron network. conv_mod: Convolution module instance. norm_class: Normalization module class. norm_args: Normalization module arguments. dropout_rate: Dropout rate. rJ block_sizeself_attrNrOconv_mod norm_classrr_rNc st||_||_||_d|_||_||fi||_||fi||_||fi||_ ||fi||_ ||fi||_ t j ||_||_d|_dS)zConstruct a Conformer object.rKN)r-r.rrNrOfeed_forward_scalernorm_feed_forward norm_self_att norm_macaronrUrVrdrrWrXrrk) r6rrrNrOrrrr_r8r:r;r.s  zChunkEncoderLayer.__init__ left_contextrcCs:tjd||jf|dtjd|j|jjdf|dg|_dS)zInitialize/Reset self-attention and convolution modules cache for streaming. Args: left_context: Number of left frames during chunk-by-chunk inference. device: Device to use for cache tensor. r&rN)rdzerosrrr(rkr6rrr:r:r;reset_streaming_caches   z'ChunkEncoderLayer.reset_streaming_cacherApos_encrj chunk_maskc Cs|}||}||j|||}|}||}|}|||j||||||d}|}||}||\}}|||}|}||}||j|| |}| |}|||fS)aEncode input sequences. Args: x: Conformer input sequences. (B, T, D_block) pos_enc: Positional embedding sequences. (B, 2 * (T - 1), D_block) mask: Source mask. (B, T) chunk_mask: Chunk mask. (T_2, T_2) Returns: x: Conformer output sequences. (B, T, D_block) mask: Source mask. (B, T) pos_enc: Positional embedding sequences. (B, 2 * (T - 1), D_block) r) rrrXrOrrrUrrrNrV)r6rArrjrrorp_r:r:r;rBs2      zChunkEncoderLayer.forwardr chunk_sizerc Cs*|}||}||j||}|}||}|dkr)tj|jd|gdd}n|}|} |dkrC|dd|| | ddf} n|dd| dddf} ||j||| |||d}|}||}|j ||jd|d\}} ||}|}| |}||j| |}| |}| | g|_||fS)aEncode chunk of input sequence. Args: x: Conformer input sequences. (B, T, D_block) pos_enc: Positional embedding sequences. (B, 2 * (T - 1), D_block) mask: Source mask. (B, T_2) left_context: Number of frames in left context. right_context: Number of frames in right context. Returns: x: Conformer output sequences. (B, T, D_block) pos_enc: Positional embedding sequences. (B, 2 * (T - 1), D_block) rr&r<Nr)rkr) rrrOrrdrgrkrrUrrrNrV) r6rArrjrrrrokeyval att_cache conv_cacher:r:r; chunk_forward7s<  $    zChunkEncoderLayer.chunk_forwardrs)rrr)rCrDrErFrrrdrrrrr.rrrrrrBrrHr:r:r8r;rsn  " 7rChunkConformerEncodercGseZdZdZ                           dXdedededededededed ed!ed"ed#ed$ed%ed&ed'ed(ed)ed*ed+ed,ed-ed.ed/ed0ed1ed2ed3ed4ed5ed6ed7ed8ed9ed:d;fFfdd?Z d@edAed:efdBdCZ d@ed:efdDdEZ dFedGe j d:d;fdHdIZdJe jdKe jd:ee je jffdLdMZdJe jdKe jd:ee je jffdNdOZ  P dYdJe jdKe jdQedFedRed:e jf dSdTZ  P dYdJe jdKe jdUe jdQedFedRed:e jfdVdWZZS)ZConformerChunkEncoderzEncoder module definition. Args: input_size: Input size. body_conf: Encoder body configuration. input_conf: Encoder input configuration. main_conf: Encoder main configuration. rvrwrxryrzrJFTr|r}r~rrr layer_normrh㈵>?rr&rrrrrrr_rrembed_vgg_likerZr[rrrrrrrrr norm_typerconv_mod_norm_epsconv_mod_norm_momentumsimplified_att_scoredynamic_chunk_trainingshort_chunk_thresholdshort_chunk_sizeleft_chunk_sizetime_reduction_factorunified_model_trainingdefault_chunk_size jitter_rangesubsampling_factorrNc#(stt||"| d|_t||_t|}#|||#f||d}$||#|$|p+|f|||fg}%t|D]}&fdd}'|%|'q9t dd|%D|_ |_ ||_ ||_ ||_||_||_| |_|!|_||_dS)zConstruct an Encoder object.)r conv_sizervgg_liker)epsmomentumcs$tttttdS)N)r_)rr rrr: conv_mod_argsr_ mult_att_argsr pos_wise_argsr:r;rsz0ConformerChunkEncoder.__init__..cSsg|]}|qSr:r:).0fnr:r:r; sz2ConformerChunkEncoder.__init__..N)r-r.r#rrrrrangerrrrr r r rrrrr)(r6rrrrrr_rrrrZr[rrrrrrrrrrrrr r r r r rrrrrrr5conv_mod_norm_args fn_modulesrmoduler8rr;r.~sb &     zConformerChunkEncoder.__init__cCrrsrrr:r:r;rrz!ConformerChunkEncoder.output_sizerY hop_lengthcCs|j||S)aLReturn the corresponding number of sample for a given chunk size, in frames. Where size is the number of features frames after applying subsampling. Args: size: Number of frames after subsampling. hop_length: Frontend's hop length Returns: : Number of raw samples rget_size_before_subsampling)r6rYr#r:r:r;get_encoder_input_raw_sizes z0ConformerChunkEncoder.get_encoder_input_raw_sizecCs |j|S)aReturn the corresponding number of sample for a given chunk size, in frames. Where size is the number of features frames after applying subsampling. Args: size: Number of frames after subsampling. Returns: : Number of raw samples r$)r6rYr:r:r;get_encoder_input_sizes z,ConformerChunkEncoder.get_encoder_input_sizerrcCs|j||S)zInitialize/Reset encoder streaming cache. Args: left_context: Number of frames in left context. device: Device ID. )rrrr:r:r;rsz+ConformerChunkEncoder.reset_streaming_cacherAx_lenc Cstt|jj|d\}}|r$td|ddd|d|d|t||j}|jr|j rD|j t |j |j dd}n|j }||||\}}||}t|d||j|jd}|j|||dd } |j||||d } |d d} |jdkr| dddd|jddf} | dddd|jddf} t | d|jd} | | | fS|jr|d} |j rt d| d}|| |jkr| }n ||jd}n|j }||||\}}||}t|d||j|jd}n|||d\}}||}d}|j||||d }|d d} |jdkr5|dddd|jddf}t | d|jd} || dfS) Encode input sequences. Args: x: Encoder input features. (B, T_in, F) x_len: Encoder input features lengths. (B,) Returns: x: Encoder outputs. (B, T_out, D_enc) x_len: Encoder outputs lenghts. (B,) r&rrrr)r&rrNrr)rrrrYrrrrrrcrrdrandintrrfrrrreqrr floor_divider r r ) r6rAr(rrrjrrrx_uttx_chunkrmax_lenr:r:r;rBs          zConformerChunkEncoder.forwardcCst|jj|d\}}|r$td|ddd|d|d|t||j}|||d\}}||}|j |||dd}|j dkrW|dddd|j ddf}|S)r)r&rrrrNr) rrrrYrrrrrrr)r6rAr(rrrjrr.r:r:r;full_utt_forwardss*    z&ConformerChunkEncoder.full_utt_forward rrc Cst|jj|d\}}|r$td|ddd|d|d|t|}||||\}}||} t|d||j|j d} |j || || d}| d d} |j dkrh|dddd|j ddf}|S) Nr&rrrrr*rr)rrrrYrrrrrrrr,rr) r6rAr(rrrrrrjrrrr:r:r;simu_chunk_forwards8   z(ConformerChunkEncoder.simu_chunk_forwardprocessed_framesc Cst|}|||d\}}|dkr-tj||jdd|d}||k}tj||gdd}|j||d} |j j || ||||d}|dkrR|ddd| ddf}|j dkrf|dddd|j ddf}|S)aEncode input sequences as chunks. Args: x: Encoder input features. (1, T_in, F) x_len: Encoder input features lengths. (1,) processed_frames: Number of frames already seen. left_context: Number of frames in left context. right_context: Number of frames in right context. Returns: x: Encoder outputs. (B, T_out, D_enc) Nrrr&r<r)rrr) rrrdarangerviewfliprgrrrr) r6rAr(r4rrrrjprocessed_maskrr:r:r;rs*  z#ConformerChunkEncoder.chunk_forward)!rvrwrxryrzrzrJFTFr|r}Fr~rrrTFrrrrzFFrrrr&Frrwr&)rr2r)rCrDrErFrrrrr.rr&r'rdrrrrrBr1r3tensorrrHr:r:r8r;rts<       !"#$p   e ) .r):rFrtypingrrrrrrdrfunasr.models.ctc.ctcr#funasr.models.transformer.attentionr r r r #funasr.models.transformer.embeddingr rrrr$funasr.models.transformer.layer_normr0funasr.models.transformer.utils.multi_layer_convrr*funasr.models.transformer.utils.nets_utilsrrrrrr3funasr.models.transformer.positionwise_feed_forwardr&funasr.models.transformer.utils.repeatrr+funasr.models.transformer.utils.subsamplingrrr r!r"r#funasr.registerr$pdbrr%rIregisterrurrrr:r:r:r;sL                 E %D` 4