o ic2@s8dZddlmZddlmZddlmZddlZddlmZddlZddlm Z ddl m Z dd l m Z dd lmZdd lmZdd lmZdd lmZddlmZddlmZddlmZddlmZddlmZddlmZddlmZddlmZddl m!Z!Gdddej"Z#e!$ddGdddej"Z%dS)zTransformer encoder definition.)List)Optional)TupleN)nn)MultiHeadedAttention)PositionalEncoding) LayerNorm) Conv1dLinear)MultiLayeredConv1d) make_pad_mask)PositionwiseFeedForward)repeat)CTC)Conv2dSubsampling)Conv2dSubsampling2)Conv2dSubsampling6)Conv2dSubsampling8)TooShortUttError)check_short_utt)tablescs2eZdZdZ   d fdd Zd dd ZZS) EncoderLayera%Encoder layer module. Args: size (int): Input dimension. self_attn (torch.nn.Module): Self-attention module instance. `MultiHeadedAttention` or `RelPositionMultiHeadedAttention` instance can be used as the argument. feed_forward (torch.nn.Module): Feed-forward module instance. `PositionwiseFeedForward`, `MultiLayeredConv1d`, or `Conv1dLinear` instance can be used as the argument. dropout_rate (float): Dropout rate. normalize_before (bool): Whether to use layer_norm before the first block. concat_after (bool): Whether to concat attention layer's input and output. if True, additional linear will be applied. i.e. x -> x + linear(concat(x, att(x))) if False, no additional linear will be applied. i.e. x -> x + att(x) stochastic_depth_rate (float): Proability to skip this layer. During training, the layer may skip residual computation and return input as-is with given probability. TFcsjt||_||_t||_t||_t||_ ||_ ||_ ||_ |j r0t ||||_||_dS)z!Construct an EncoderLayer object.N)super__init__ self_attn feed_forwardrnorm1norm2rDropoutdropoutsizenormalize_before concat_afterLinear concat_linearstochastic_depth_rate)selfr rr dropout_rater!r"r% __class__U/home/ubuntu/.local/lib/python3.10/site-packages/funasr/models/transformer/encoder.pyr8s     zEncoderLayer.__init__Nc Csd}d}|jr|jdkrtd|jk}dd|j}|r0|dur,tj||gdd}||fS|}|jr:||}|durA|}n@|j|jd|jdd|j fksTJ|ddddddf}|ddddddf}|durtdn |ddddddf}|j rtj|| ||||fdd}||| |}n||| | ||||}|js||}|}|jr||}||| ||}|js||}|durtj||gdd}||fS)aCompute encoded features. Args: x_input (torch.Tensor): Input tensor (#batch, time, size). mask (torch.Tensor): Mask tensor for the input (#batch, time). cache (torch.Tensor): Cache tensor of the input (#batch, time - 1, size). Returns: torch.Tensor: Output tensor (#batch, time, size). torch.Tensor: Mask tensor (#batch, time). Fg?rN)dim)trainingr%torchranditemcatr!rshaper r"rr$rrr) r&xmaskcache skip_layerstoch_layer_coeffresidualx_qx_concatr*r*r+forwardPsB  &&   zEncoderLayer.forward)TFrN)__name__ __module__ __qualname____doc__rr= __classcell__r*r*r(r+r"srencoder_classesTransformerEncoderc!seZdZdZdddddddded d d d d gd fdededededededededeede de dededede ede f fdd Z d efd!d"Z # #d*d$e jd%e jd&e jd'ed ee je jee jff d(d)ZZS)+rEacTransformer encoder module. Args: input_size: input dim output_size: dimension of attention attention_heads: the number of heads of multi head attention linear_units: the number of units of position-wise feed forward num_blocks: the number of decoder blocks dropout_rate: dropout rate attention_dropout_rate: dropout rate in attention positional_dropout_rate: dropout rate after adding positional encoding input_layer: input layer type pos_enc_class: PositionalEncoding or ScaledPositionalEncoding normalize_before: whether to use layer_norm before the first block concat_after: whether to concat attention layer's input and output if True, additional linear will be applied. i.e. x -> x + linear(concat(x, att(x))) if False, no additional linear will be applied. i.e. x -> x + att(x) positionwise_layer_type: linear of conv1d positionwise_conv_kernel_size: kernel size of positionwise conv1d layer padding_idx: padding_idx for input_layer=embed ig?rconv2dTFlinearr,r. input_size output_sizeattention_heads linear_units num_blocksr'positional_dropout_rateattention_dropout_rate input_layerr!r"positionwise_layer_typepositionwise_conv_kernel_size padding_idxinterctc_layer_idxinterctc_use_conditioningc st|_| dkr+tjtj|tjtjtj | ||_ nb| dkr7t ||_ nV| dkrCt ||_ nJ| dkrOt ||_ n>| dkr[t||_ n2| dkrrtjtjj||d| ||_ n| dur|kr~d|_ ntj||_ ntd| |_| dkrt|fn| d krt||fn| d krt||fntd t|fd d |_|jrt|_||_t|dkrdt|krt||ksJ||_d|_dS)NrJrIconv2d2conv2d6conv2d8embed)rUzunknown input_layer: conv1dz conv1d-linearzSupport only linear or conv1d.csttSr>)rr)lnumrQrMr"r'r!rLpositionwise_layerpositionwise_layer_argsr*r+s z-TransformerEncoder.__init__..r)rr _output_sizer0r Sequentialr#rrReLUr[rrrr Embedding ValueErrorr!r r r NotImplementedErrorr encoders after_normrVlenminmaxrWconditioning_layer)r&rKrLrMrNrOr'rPrQrR pos_enc_classr!r"rSrTrUrVrWr(r^r+rsz        zTransformerEncoder.__init__returncCs|jSr>)rb)r&r*r*r+rLszTransformerEncoder.output_sizeNxs_padilens prev_statesctccCst|dddddf|j}|jdur|}nIt|jts2t|jts2t|jts2t|jtr^t |j| d\}}|rUt d| ddd|d| d||||\}}n||}g}t |j dkru|||\}}n:t|jD]4\} } | ||\}}| d|j vr|} |jr|| } || d| f|jr|| } ||| }qz|jr||}|dd} t |dkr||f| dfS|| dfS)zEmbed positions in tensor. Args: xs_pad: input tensor (B, L, D) ilens: input length (B) prev_states: Not to be used now. Returns: position embedded tensor and mask Nr,zhas z) frames and is too short for subsampling z(it needs more than z frames), return empty resultsr)r todevicer[ isinstancerrrrrr rrjrVrh enumerater!riappendrWsoftmaxrmsqueezesum)r&rprqrrrsmasks short_status limit_sizeintermediate_outs layer_idx encoder_layer encoder_outctc_outolensr*r*r+r= sT$            zTransformerEncoder.forward)NN)r?r@rArBrintfloatrstrboolrrrLr0Tensorrrr=rCr*r*r(r+rEs    a)&rBtypingrrrr0rlogging#funasr.models.transformer.attentionr#funasr.models.transformer.embeddingr$funasr.models.transformer.layer_normr0funasr.models.transformer.utils.multi_layer_convr r *funasr.models.transformer.utils.nets_utilsr 3funasr.models.transformer.positionwise_feed_forwardr &funasr.models.transformer.utils.repeatr funasr.models.ctc.ctcr+funasr.models.transformer.utils.subsamplingrrrrrrfunasr.registerrModulerregisterrEr*r*r*r+s4                     i