o i9@sdZddlmZddlmZddlmZddlZddlmZddlZddlm Z ddl m Z dd l m Z dd lmZdd lmZdd lmZdd lmZddlmZGdddejZeddGdddejZeddGdddejZeddGdddejZdS)zTransformer encoder definition.)List)Optional)TupleN)nn)MultiHeadedAttention) MultiHeadedAttentionReturnWeight)PositionalEncoding) LayerNorm) make_pad_mask)PositionwiseFeedForward)repeat)tablescs2eZdZdZ   d fdd Zd dd ZZS) EncoderLayera%Encoder layer module. Args: size (int): Input dimension. self_attn (torch.nn.Module): Self-attention module instance. `MultiHeadedAttention` or `RelPositionMultiHeadedAttention` instance can be used as the argument. feed_forward (torch.nn.Module): Feed-forward module instance. `PositionwiseFeedForward`, `MultiLayeredConv1d`, or `Conv1dLinear` instance can be used as the argument. dropout_rate (float): Dropout rate. normalize_before (bool): Whether to use layer_norm before the first block. concat_after (bool): Whether to concat attention layer's input and output. if True, additional linear will be applied. i.e. x -> x + linear(concat(x, att(x))) if False, no additional linear will be applied. i.e. x -> x + att(x) stochastic_depth_rate (float): Proability to skip this layer. During training, the layer may skip residual computation and return input as-is with given probability. TFcsntt|||_||_t||_t||_t ||_ ||_ ||_ ||_ |j r2t||||_||_dS)z!Construct an EncoderLayer object.N)superr__init__ self_attn feed_forwardr norm1norm2rDropoutdropoutsizenormalize_before concat_afterLinear concat_linearstochastic_depth_rate)selfrrr dropout_raterrr __class__P/home/ubuntu/.local/lib/python3.10/site-packages/funasr/models/lcbnet/encoder.pyr/s     zEncoderLayer.__init__Nc Csd}d}|jr|jdkrtd|jk}dd|j}|r0|dur,tj||gdd}||fS|}|jr:||}|durA|}n@|j|jd|jdd|j fksTJ|ddddddf}|ddddddf}|durtdn |ddddddf}|j rtj|| ||||fdd}||| |}n||| | ||||}|js||}|}|jr||}||| ||}|js||}|durtj||gdd}||fS)aCompute encoded features. Args: x_input (torch.Tensor): Input tensor (#batch, time, size). mask (torch.Tensor): Mask tensor for the input (#batch, time). cache (torch.Tensor): Cache tensor of the input (#batch, time - 1, size). Returns: torch.Tensor: Output tensor (#batch, time, size). torch.Tensor: Mask tensor (#batch, time). Fg?rNdim)trainingrtorchranditemcatrrshaperrrrrrr) rxmaskcache skip_layerstoch_layer_coeffresidualx_qx_concatr"r"r#forwardGsB  &&   zEncoderLayer.forward)TFrN__name__ __module__ __qualname____doc__rr6 __classcell__r"r"r r#rsrencoder_classesTransformerTextEncodercseZdZdZdddddddedd f d ed ed ed ededededededeffdd ZdefddZ de j de j de e j e j e e j ffddZZS)r?ahTransformer text encoder module. Args: input_size: input dim output_size: dimension of attention attention_heads: the number of heads of multi head attention linear_units: the number of units of position-wise feed forward num_blocks: the number of decoder blocks dropout_rate: dropout rate attention_dropout_rate: dropout rate in attention positional_dropout_rate: dropout rate after adding positional encoding input_layer: input layer type pos_enc_class: PositionalEncoding or ScaledPositionalEncoding normalize_before: whether to use layer_norm before the first block concat_after: whether to concat attention layer's input and output if True, additional linear will be applied. i.e. x -> x + linear(concat(x, att(x))) if False, no additional linear will be applied. i.e. x -> x + att(x) positionwise_layer_type: linear of conv1d positionwise_conv_kernel_size: kernel size of positionwise conv1d layer padding_idx: padding_idx for input_layer=embed 皙?rTF input_size output_sizeattention_heads linear_units num_blocksrpositional_dropout_rateattention_dropout_raterrc s~t|_tjtj|| ||_|_t |ft |fdd|_ |jr=t |_ dSdS)NcsttSr7)rr)lnumrKrGrrrrFpositionwise_layerpositionwise_layer_argsr"r#s z1TransformerTextEncoder.__init__..)rr _output_sizer)r Sequential Embeddingembedrr r encodersr after_norm) rrErFrGrHrIrrJrK pos_enc_classrrr rMr#rs&   zTransformerTextEncoder.__init__returncCs|jSr7)rQ)rr"r"r#rFsz"TransformerTextEncoder.output_sizexs_padilenscCsht|dddddf|j}||}|||\}}|jr'||}|dd}||dfS)zEmbed positions in tensor. Args: xs_pad: input tensor (B, L, D) ilens: input length (B) Returns: position embedded tensor and mask Nr$) r todevicerTrUrrVsqueezesum)rrYrZmasksolensr"r"r#r6s$   zTransformerTextEncoder.forward)r9r:r;r<rintfloatboolrrFr)Tensorrrr6r=r"r"r r#r?sR   ,FusionSANEncodercs0eZdZdZ  d fdd Zd ddZZS) SelfSrcAttentionaSingle decoder layer module. Args: size (int): Input dimension. self_attn (torch.nn.Module): Self-attention module instance. `MultiHeadedAttention` instance can be used as the argument. src_attn (torch.nn.Module): Self-attention module instance. `MultiHeadedAttention` instance can be used as the argument. feed_forward (torch.nn.Module): Feed-forward module instance. `PositionwiseFeedForward`, `MultiLayeredConv1d`, or `Conv1dLinear` instance can be used as the argument. dropout_rate (float): Dropout rate. normalize_before (bool): Whether to use layer_norm before the first block. concat_after (bool): Whether to concat attention layer's input and output. if True, additional linear will be applied. i.e. x -> x + linear(concat(x, att(x))) if False, no additional linear will be applied. i.e. x -> x + att(x) TFc stt|||_t||||_t||||_t||||_ t ||_ t ||_ t ||_ t||_| |_| |_|jrQt||||_t||||_dSdS)z%Construct an SelfSrcAttention object.N)rrfrrrrrsrc_attnr rr rrnorm3rrrrrrconcat_linear1concat_linear2) rrrG attention_dimrHself_attention_dropout_ratesrc_attention_dropout_raterJrrrr r"r#rs*    zSelfSrcAttention.__init__Nc Cs|}|jr ||}|dur|}|}nT|j|jd|jdd|jfks:J|jd|jd|jdd|jf|ddddddf}|ddddddf}d}|durg|ddddddf}|jrtj||||||fdd} ||| } n || |||||} |js|| } | }|jr| | } |jrtj| | | |||fdd} || | } n| | |||\} } || | } |js| | } | }|jr| | } || || } |js| | } |durtj|| gdd} | |||fS)a"Compute decoded features. Args: tgt (torch.Tensor): Input tensor (#batch, maxlen_out, size). tgt_mask (torch.Tensor): Mask for input tensor (#batch, maxlen_out). memory (torch.Tensor): Encoded memory, float32 (#batch, maxlen_in, size). memory_mask (torch.Tensor): Encoded memory mask (#batch, maxlen_in). cache (List[torch.Tensor]): List of cached tensors. Each tensor shape should be (#batch, maxlen_out - 1, size). Returns: torch.Tensor: Output tensor(#batch, maxlen_out, size). torch.Tensor: Mask for output tensor (#batch, maxlen_out). torch.Tensor: Encoded memory (#batch, maxlen_in, size). torch.Tensor: Encoded memory mask (#batch, maxlen_in). Nrr$z == r'r%)rrr-rrr)r,rrirrrgrjrhr) rtgttgt_maskmemory memory_maskr0r3tgt_q tgt_q_mask tgt_concatr.r5scorer"r"r#r6sV  &      zSelfSrcAttention.forward)TFr7r8r"r"r r#rfs #rfConvBiasPredictorcs2eZdZ      d fdd Zdd ZZS) ConvPredictorr@rArDrBcsztt||||_t||_t||||_t||_t ||fd|_ t j ||||d|d|_ t |d|_dS)Nrr$)groups)rrrattenr rr rrr ConstantPad1dpadConv1dconv1dr output_linear)rrl_orderr_orderrGrKrHr r"r#rgs   zConvPredictor.__init__cCs|}|||||d}|}||}|||}||}|dd}||}||}||}|dd}t|}| |}| dkrN| d}|S)Nr$rx) rzrrr transposer|r~r)relurr&r])rtext_encasr_encr3contextqueriesrpoutputr"r"r#r6ys           zConvPredictor.forward)r@rxrxrArDrB)r9r:r;rr6r=r"r"r r#rwesrw)r<typingrrrr)rlogging#funasr.models.transformer.attentionrfunasr.models.lcbnet.attentionr#funasr.models.transformer.embeddingr$funasr.models.transformer.layer_normr *funasr.models.transformer.utils.nets_utilsr 3funasr.models.transformer.positionwise_feed_forwardr &funasr.models.transformer.utils.repeatr funasr.registerr Modulerregisterr?rfrwr"r"r"r#s.             i a