o iV`@sdZddlmZddlmZddlmZddlmZddlZddlmZddlm Z dd l m Z dd l m Z dd lmZdd lmZdd lmZddlmZddlmZddlmZddlmZddlmZddlmZddl m!Z!Gdddej"Z#Gdddej"Z$Gdddej"eZ%e!&ddGddde%Z'e!&ddGddde%Z(e!&dd Gd!d d e%Z)e!&dd"Gd#d"d"e%Z*e!&dd$Gd%d$d$e%Z+dS)&zDecoder definition.)Any)List)Sequence)TupleN)nn)MultiHeadedAttention)DynamicConvolution)DynamicConvolution2D)PositionalEncoding) LayerNorm)LightweightConvolution)LightweightConvolution2D)subsequent_mask) make_pad_mask)PositionwiseFeedForward)repeat)BatchScorerInterface)tablescs0eZdZdZ  d fdd Zd ddZZS) DecoderLayeraSingle decoder layer module. Args: size (int): Input dimension. self_attn (torch.nn.Module): Self-attention module instance. `MultiHeadedAttention` instance can be used as the argument. src_attn (torch.nn.Module): Self-attention module instance. `MultiHeadedAttention` instance can be used as the argument. feed_forward (torch.nn.Module): Feed-forward module instance. `PositionwiseFeedForward`, `MultiLayeredConv1d`, or `Conv1dLinear` instance can be used as the argument. dropout_rate (float): Dropout rate. normalize_before (bool): Whether to use layer_norm before the first block. concat_after (bool): Whether to concat attention layer's input and output. if True, additional linear will be applied. i.e. x -> x + linear(concat(x, att(x))) if False, no additional linear will be applied. i.e. x -> x + att(x) TFcstt|||_||_||_||_t||_t||_ t||_ t ||_ ||_||_|jrEt ||||_t ||||_dSdS)z!Construct an DecoderLayer object.N)superr__init__size self_attnsrc_attn feed_forwardr norm1norm2norm3rDropoutdropoutnormalize_before concat_afterLinearconcat_linear1concat_linear2)selfrrrr dropout_rater r! __class__U/home/ubuntu/.local/lib/python3.10/site-packages/funasr/models/transformer/decoder.pyr6s     zDecoderLayer.__init__Nc Cs|}|jr ||}|dur|}|}nT|j|jd|jdd|jfks:J|jd|jd|jdd|jf|ddddddf}|ddddddf}d}|durg|ddddddf}|jrtj||||||fdd} ||| } n || |||||} |js|| } | }|jr| | } |jrtj| | | |||fdd} || | } n || | | |||} |js| | } | }|jr| | } || || } |js| | } |durtj|| gdd} | |||fS)a"Compute decoded features. Args: tgt (torch.Tensor): Input tensor (#batch, maxlen_out, size). tgt_mask (torch.Tensor): Mask for input tensor (#batch, maxlen_out). memory (torch.Tensor): Encoded memory, float32 (#batch, maxlen_in, size). memory_mask (torch.Tensor): Encoded memory mask (#batch, maxlen_in). cache (List[torch.Tensor]): List of cached tensors. Each tensor shape should be (#batch, maxlen_out - 1, size). Returns: torch.Tensor: Output tensor(#batch, maxlen_out, size). torch.Tensor: Mask for output tensor (#batch, maxlen_out). torch.Tensor: Encoded memory (#batch, maxlen_in, size). torch.Tensor: Encoded memory mask (#batch, maxlen_in). Nrz == dim)r rshaperr!torchcatrr#rrrr$rr) r%tgttgt_maskmemory memory_maskcacheresidualtgt_q tgt_q_mask tgt_concatxx_concatr)r)r*forwardPsT  &      zDecoderLayer.forward)TFN)__name__ __module__ __qualname____doc__rr= __classcell__r)r)r'r*r s rcs&eZdZfddZdddZZS)DecoderLayerExportcs>t|j|_|j|_|j|_|j|_|j|_|j|_dSr>)rrrrrrrr)r%modelr'r)r*rs  zDecoderLayerExport.__init__Nc Cst|}||}|}|}||||||} | }|| } ||| |||} | }|| } ||| } | |||fSr>)rrrrrr) r%r2r3r4r5r6r7r8r9r;r)r)r*r=s    zDecoderLayerExport.forwardr>)r?r@rArr=rCr)r)r'r*rDs  rDcseZdZdZddddedfdedededed ed ed effd d Z de j de j de j de j de e j e j ff ddZ d#de j de j de j dee j de e j ee j ff ddZddZde j deed e j de e j eeffd!d"ZZS)$BaseTransformerDecoderaBase class of Transfomer decoder module. Args: vocab_size: output dim encoder_output_size: dimension of attention attention_heads: the number of heads of multi head attention linear_units: the number of units of position-wise feed forward num_blocks: the number of decoder blocks dropout_rate: dropout rate self_attention_dropout_rate: dropout rate for attention input_layer: input layer type use_output_layer: whether to use output layer pos_enc_class: PositionalEncoding or ScaledPositionalEncoding normalize_before: whether to use layer_norm before the first block concat_after: whether to concat attention layer's input and output if True, additional linear will be applied. i.e. x -> x + linear(concat(x, att(x))) if False, no additional linear will be applied. i.e. x -> x + att(x) 皙?embedT vocab_sizeencoder_output_sizer&positional_dropout_rate input_layeruse_output_layerr c st|} |dkrtjtj|| || ||_n*|dkr?tjtj|| tj| tj |tj || ||_nt d|||_ |j rQt| |_ |r\tj| ||_nd|_d|_dS)NrHlinearz'only 'embed' or 'linear' is supported: )rrr0r Sequential EmbeddingrHr"r rReLU ValueErrorr after_norm output_layerdecoders) r%rIrJr&rKrLrM pos_enc_classr attention_dimr'r)r*rs.       zBaseTransformerDecoder.__init__hs_padhlens ys_in_pad ys_in_lensreturnc Cs|}t|dddddf|j}t|d|jdd}||@}|}t||dddddddf|j} | jd|jdkr^|jd| jd} tjj | d| fdd} | |} | | ||| \} }}} |j rw|| } |jdur|| } |d} | | fS) a@Forward decoder. Args: hs_pad: encoded memory, float32 (batch, maxlen_in, feat) hlens: (batch) ys_in_pad: input token ids, int64 (batch, maxlen_out) if input_layer == "embed" input tensor (batch, maxlen_out, #mels) in the other cases ys_in_lens: (batch) Returns: (tuple): tuple containing: x: decoded token score before softmax (batch, maxlen_out, token) if use_output_layer is True, olens: (batch, ) Nr,devicerr+)maxlenconstantF)rtor^rr unsqueezer/r0r functionalpadrHrUr rSrTsum) r%rXrYrZr[r2r3mr4r5padlenr;olensr)r)r*r=s"$.     zBaseTransformerDecoder.forwardNr2r3r4r6c Cs||}|durdgt|j}g}t||jD]\}}||||d|d\}}}} ||q|jr?||dddf} n|dddf} |jdurVtj || dd} | |fS)a5Forward one step. Args: tgt: input token ids, int64 (batch, maxlen_out) tgt_mask: input token mask, (batch, maxlen_out) dtype=torch.uint8 in PyTorch 1.2- dtype=torch.bool in PyTorch 1.2+ (include 1.2) memory: encoded memory, float32 (batch, maxlen_in, feat) cache: cached output list of (batch, max_time_out-1, size) Returns: y, cache: NN output value and cache per `self.decoders`. y.shape` is (batch, maxlen_out, token) Nr6r,r-) rHlenrUzipappendr rSrTr0 log_softmax) r%r2r3r4r6r; new_cachecdecoderr5yr)r)r*forward_one_step#s   z'BaseTransformerDecoder.forward_one_stepcCsHtt||jdd}|j|d||d|d\}}|d|fS)zScore.r]rri)rrjr^rbrrsqueeze)r%ysstater;ys_masklogpr)r)r*scoreHs"zBaseTransformerDecoder.scorertstatesxscst|t|jddurd}n fddtD}t|d|jdd}|j||||d\}fddtD}||fS) aScore new token batch. Args: ys (torch.Tensor): torch.int64 prefix tokens (n_batch, ylen). states (List[Any]): Scorer states for prefix tokens. xs (torch.Tensor): The encoder feature that generates ys (n_batch, xlen, n_feat). Returns: tuple[torch.Tensor, List[Any]]: Tuple of batchfied scores for next token with shape of `(n_batch, n_vocab)` and next state list for ys. rNcs*g|]tfddtDqS)cg|]}|qSr)r)).0b)iryr)r* gABaseTransformerDecoder.batch_score...)r0stackranger|)n_batchry)r~r*rfsz6BaseTransformerDecoder.batch_score..r,r]rics$g|]fddtDqS)cr{r)r))r|r~)r}ryr)r*rorr)rr)n_layersry)r}r*ros$)rjrUrrrr^rbrr)r%rtryrz batch_statervrw state_listr))rrryr* batch_scoreNs   z"BaseTransformerDecoder.batch_scorer>)r?r@rArBr intfloatstrboolrr0Tensorrr=rrrrxrrrCr)r)r'r*rFsp ) 6 %rFdecoder_classesTransformerDecodercsleZdZdddddddddeddf d ed ed ed ed ededededededededeffdd ZZ S)rrGrHTFrIrJattention_heads linear_units num_blocksr&rKself_attention_dropout_ratesrc_attention_dropout_raterLrMr r!c sFtj|||| | | d|t|fdd|_dS)NrIrJr&rKrLrMrVr cs,ttttSr>)rrrlnumrWrr!r&rr rrr)r*s   z-TransformerDecoder.__init__..)rrrrU)r%rIrJrrrr&rKrrrLrMrVr r!r'rr*rus  zTransformerDecoder.__init__) r?r@rAr rrrrrrCr)r)r'r*rssP    (LightweightConvolutionTransformerDecoderc!eZdZdddddddddedddd dfd ed ed ed edededededededededededeedef fdd Z Z S)rrrrrGrrHTF rrrrrrIrJrrrr&rKrrrLrMr r! conv_wshareconv_kernel_length conv_usebiasc pt|krtdtd|tj|||| | | d|t| f dd|_dS)NCconv_kernel_length must have equal number of values to num_blocks:  != rc 8tt |ddt tSNT)wsharen_featr& kernel_sizeuse_kernel_maskuse_bias)rr rrr rWrr!rrrr&rr rrr)r*r   zCLightweightConvolutionTransformerDecoder.__init__..rjrRrrrrUr%rIrJrrrr&rKrrrLrMrVr r!rrrr'rr*r.   z1LightweightConvolutionTransformerDecoder.__init__ r?r@rAr rrrrrrrCr)r)r'r*rb    *LightweightConvolution2DTransformerDecoderc!r)rrrrrGrrHTFrrIrJrrrr&rKrrrLrMr r!rrrc r)Nrrrc rr)rr rrrrr)r*rrzELightweightConvolution2DTransformerDecoder.__init__..rrr'rr*rrz3LightweightConvolution2DTransformerDecoder.__init__rr)r)r'r*rr$DynamicConvolutionTransformerDecoderc!r)rrrrrGrrHTFrrIrJrrrr&rKrrrLrMr r!rrrc r)Nrrrc rr)rrrrrrr)r*rArz?DynamicConvolutionTransformerDecoder.__init__..rrr'rr*r.   z-DynamicConvolutionTransformerDecoder.__init__rr)r)r'r*rr&DynamicConvolution2DTransformerDecoderc!r)rrrrrGrrHTFrrIrJrrrr&rKrrrLrMr r!rrrc r)Nrrrc rr)rr rrrrr)r*r}rzADynamicConvolution2DTransformerDecoder.__init__..rrr'rr*rVrz/DynamicConvolution2DTransformerDecoder.__init__rr)r)r'r*rTr),rBtypingrrrrr0r#funasr.models.transformer.attentionr,funasr.models.transformer.utils.dynamic_convr.funasr.models.transformer.utils.dynamic_conv2dr #funasr.models.transformer.embeddingr $funasr.models.transformer.layer_normr )funasr.models.transformer.utils.lightconvr +funasr.models.transformer.utils.lightconv2dr $funasr.models.transformer.utils.maskr*funasr.models.transformer.utils.nets_utilsr3funasr.models.transformer.positionwise_feed_forwardr&funasr.models.transformer.utils.repeatr2funasr.models.transformer.scorers.scorer_interfacerfunasr.registerrModulerrDrFregisterrrrrrr)r)r)r*sD                  v A , ; ; ;