o iG@sDdZddlmZddlmZddlmZddlZddlmZddlZddlm Z ddl m Z dd l m Z dd lmZdd lmZdd lmZdd lmZddlmZddlmZddlmZddlmZddlmZddlm Z ddlm!Z!ddlm"Z"ddlm#Z#ddlm$Z$ddlm%Z%Gdddej&Z'Gdddej&Z(dS)zTransformer encoder definition.)List)Optional)TupleN)nn)MultiHeadedAttention)PositionalEncoding) LayerNorm) Conv1dLinear)MultiLayeredConv1d) make_pad_mask)PositionwiseFeedForward)repeat)DynamicConvolution)DynamicConvolution2D)LightweightConvolution)LightweightConvolution2D)Conv2dSubsampling)Conv2dSubsampling2)Conv2dSubsampling6)Conv2dSubsampling8)TooShortUttError)check_short_uttcs2eZdZdZ   d fdd Zd dd ZZS) EncoderLayera%Encoder layer module. Args: size (int): Input dimension. self_attn (torch.nn.Module): Self-attention module instance. `MultiHeadedAttention` or `RelPositionMultiHeadedAttention` instance can be used as the argument. feed_forward (torch.nn.Module): Feed-forward module instance. `PositionwiseFeedForward`, `MultiLayeredConv1d`, or `Conv1dLinear` instance can be used as the argument. dropout_rate (float): Dropout rate. normalize_before (bool): Whether to use layer_norm before the first block. concat_after (bool): Whether to concat attention layer's input and output. if True, additional linear will be applied. i.e. x -> x + linear(concat(x, att(x))) if False, no additional linear will be applied. i.e. x -> x + att(x) stochastic_depth_rate (float): Proability to skip this layer. During training, the layer may skip residual computation and return input as-is with given probability. TFcsntt|||_||_t||_t||_t ||_ ||_ ||_ ||_ |j r2t||||_||_dS)z!Construct an EncoderLayer object.N)superr__init__ self_attn feed_forwardrnorm1norm2rDropoutdropoutsizenormalize_before concat_afterLinear concat_linearstochastic_depth_rate)selfr"rr dropout_rater#r$r' __class__d/home/ubuntu/.local/lib/python3.10/site-packages/funasr/models/language_model/transformer_encoder.pyr:s     zEncoderLayer.__init__Nc Csd}d}|jr|jdkrtd|jk}dd|j}|r0|dur,tj||gdd}||fS|}|jr:||}|durA|}n@|j|jd|jdd|j fksTJ|ddddddf}|ddddddf}|durtdn |ddddddf}|j rtj|| ||||fdd}||| |}n||| | ||||}|js||}|}|jr||}||| ||}|js||}|durtj||gdd}||fS)aCompute encoded features. Args: x_input (torch.Tensor): Input tensor (#batch, time, size). mask (torch.Tensor): Mask tensor for the input (#batch, time). cache (torch.Tensor): Cache tensor of the input (#batch, time - 1, size). Returns: torch.Tensor: Output tensor (#batch, time, size). torch.Tensor: Mask tensor (#batch, time). Fg?rN)dim)trainingr'torchranditemcatr#rshaper"r$rr&r!rr) r(xmaskcache skip_layerstoch_layer_coeffresidualx_qx_concatr,r,r-forwardRsB  &&   zEncoderLayer.forward)TFrN)__name__ __module__ __qualname____doc__rr? __classcell__r,r,r*r-r$srcsteZdZdZdddddddddd d ed dd d ddd dddffdd Z    dddZddZdddZZ S)TransformerEncoder_lmaTransformer encoder module. Args: idim (int): Input dimension. attention_dim (int): Dimension of attention. attention_heads (int): The number of heads of multi head attention. conv_wshare (int): The number of kernel of convolution. Only used in selfattention_layer_type == "lightconv*" or "dynamiconv*". conv_kernel_length (Union[int, str]): Kernel size str of convolution (e.g. 71_71_71_71_71_71). Only used in selfattention_layer_type == "lightconv*" or "dynamiconv*". conv_usebias (bool): Whether to use bias in convolution. Only used in selfattention_layer_type == "lightconv*" or "dynamiconv*". linear_units (int): The number of units of position-wise feed forward. num_blocks (int): The number of decoder blocks. dropout_rate (float): Dropout rate. positional_dropout_rate (float): Dropout rate after adding positional encoding. attention_dropout_rate (float): Dropout rate in attention. input_layer (Union[str, torch.nn.Module]): Input layer type. pos_enc_class (torch.nn.Module): Positional encoding module class. `PositionalEncoding `or `ScaledPositionalEncoding` normalize_before (bool): Whether to use layer_norm before the first block. concat_after (bool): Whether to concat attention layer's input and output. if True, additional linear will be applied. i.e. x -> x + linear(concat(x, att(x))) if False, no additional linear will be applied. i.e. x -> x + att(x) positionwise_layer_type (str): "linear", "conv1d", or "conv1d-linear". positionwise_conv_kernel_size (int): Kernel size of positionwise conv1d layer. selfattention_layer_type (str): Encoder attention layer type. padding_idx (int): Padding idx for input_layer=embed. stochastic_depth_rate (float): Maximum probability to skip the encoder layer. intermediate_layers (Union[List[int], None]): indices of intermediate CTC layer. indices start from 1. if not None, intermediate outputs are returned (which changes return type signature.) 11F皙?rconv2dTlinearr.selfattnr0Nc std|_| dkr+tjtj|tjtjtj | | |_ n| dkr:t ||_ d|_np| dkrMt || | |_ d|_n]| dkr\t ||_ d|_nN| dkrkt ||_ d |_n?| d krtjtjj||d | | |_ n(t| tjjrtj| | | |_ n| d urtj| | |_ ntd |  |_||||\ |dvrtdt|fg ns|dkrtdtfddt DnX|dkrtdtfddt Dn<|dkrtdtfddt Dn |dkr;tdtfddt Dnt|t  f dd|_|jr[t|_||_ |d urednd |_!|j!ry||_"tj||_#d Sd S)!zConstruct an Encoder object.r.rNrMrHzconv2d-scaled-pos-encconv2d6rKconv2d8embed) padding_idxNzunknown input_layer: )rO rel_selfattnlegacy_rel_selfattnz2encoder self-attention layer type = self-attention lightconvz;encoder self-attention layer type = lightweight convolutionc *g|]}td|dfqS_Fintsplit.0lnum attention_dimattention_dropout_rateconv_kernel_length conv_usebias conv_wsharer,r-  z2TransformerEncoder_lm.__init__.. lightconv2dzIencoder self-attention layer type = lightweight convolution 2-dimensionalc rXrYr[r^rar,r-rg"rh dynamicconvz7encoder self-attention layer type = dynamic convolutionc rXrYr[r^rar,r-rg0rh dynamicconv2dzEencoder self-attention layer type = dynamic convolution 2-dimensionalc rXrYr[r^rar,r-rg>rhc s0t| td|S)Nr.)rfloat)r`) rbr$r)encoder_selfattn_layerencoder_selfattn_layer_argsr# num_blockspositionwise_layerpositionwise_layer_argsr'r,r-Ns z0TransformerEncoder_lm.__init__..TF)$rrconv_subsampling_factorr2r Sequentialr%rr ReLUrSrrr Embedding isinstanceModule ValueErrorr#get_positionwise_layerlogginginforrrangerrrNotImplementedErrorr encoders after_normintermediate_layersuse_conditioning ctc_softmaxconditioning_layer)r(idimrbattention_headsrfrdre linear_unitsror)positional_dropout_raterc input_layer pos_enc_classr#r$positionwise_layer_typepositionwise_conv_kernel_sizeselfattention_layer_typerTr'rrconditioning_layer_dimr*)rbrcr$rdrerfr)rmrnr#rorprqr'r-rs                zTransformerEncoder_lm.__init__cCsf|dkrt}|||f}||fS|dkrt}||||f}||fS|dkr/t}||||f}||fStd)zDefine positionwise layer.rNconv1dz conv1d-linearzSupport only linear or conv1d.)r r r r~)r(rrbrr)rrprqr,r,r-rzas*  z,TransformerEncoder_lm.get_positionwise_layercCst|jtttfr|||\}}n||}|jdur%|||\}}n=g}t|jD]5\}}|||\}}|jdura|d|jvra|}|jrM| |}| ||j ra| |}|| |}q,|jrj| |}|jdurt|||fS||fS)a>Encode input sequence. Args: xs (torch.Tensor): Input tensor (#batch, time, idim). masks (torch.Tensor): Mask tensor (#batch, time). Returns: torch.Tensor: Output tensor (#batch, time, attention_dim). torch.Tensor: Mask tensor (#batch, time). Nr.)rwrSrrrrr enumerater#rappendrrr)r(xsmasksintermediate_outputs layer_idx encoder_layerencoder_outputintermediate_resultr,r,r-r?s4          zTransformerEncoder_lm.forwardcCst|jtr|||\}}n||}|dur$ddtt|jD}g}t||jD]\}}||||d\}}||q,|jrG| |}|||fS)adEncode input frame. Args: xs (torch.Tensor): Input tensor. masks (torch.Tensor): Mask tensor. cache (List[torch.Tensor]): List of cache tensors. Returns: torch.Tensor: Output tensor. torch.Tensor: Mask tensor. List[torch.Tensor]: List of new cache tensors. NcSsg|]}dqSr@r,)r_rZr,r,r-rgsz:TransformerEncoder_lm.forward_one_step..)r9) rwrSrr}lenrziprr#r)r(rrr9 new_cachecer,r,r-forward_one_steps     z&TransformerEncoder_lm.forward_one_step)rNrGrJrLr.r@) rArBrCrDrrrzr?rrEr,r,r*r-rFsD)0  0rF))rDtypingrrrr2rr{#funasr.models.transformer.attentionr#funasr.models.transformer.embeddingr$funasr.models.transformer.layer_normr0funasr.models.transformer.utils.multi_layer_convr r *funasr.models.transformer.utils.nets_utilsr 3funasr.models.transformer.positionwise_feed_forwardr &funasr.models.transformer.utils.repeatr ,funasr.models.transformer.utils.dynamic_convr.funasr.models.transformer.utils.dynamic_conv2dr)funasr.models.transformer.utils.lightconvr+funasr.models.transformer.utils.lightconv2dr+funasr.models.transformer.utils.subsamplingrrrrrrrxrrFr,r,r,r-s6                      i