o i8L@sHdZddlmZddlmZddlmZddlmZddlZddlmZddlm Z dd l m Z dd l m Z dd lmZdd lmZdd lmZddlmZddlmZddlmZddlmZddlmZddlmZddl m!Z!ddl"m#Z#GdddejZGdddej$Z%Gdddej$eZ&e#'ddGddde&Z(dS)zDecoder definition.)Any)List)Sequence)TupleN)nn)MultiHeadedAttention)DynamicConvolution)DynamicConvolution2D)PositionalEncoding) LayerNorm)LightweightConvolution)LightweightConvolution2D)subsequent_mask) make_pad_mask)PositionwiseFeedForward)repeat)BatchScorerInterface) OmegaConf)tablescseZdZfddZZS)r cst||jSN)superforwardfloattypedtype)selfx __class__X/home/ubuntu/.local/lib/python3.10/site-packages/funasr/models/conformer_rwkv/decoder.pyr!szLayerNorm.forward)__name__ __module__ __qualname__r __classcell__rrrr r sr cs4eZdZdZdddiffdd Zd ddZZS) DecoderLayeraSingle decoder layer module. Args: size (int): Input dimension. self_attn (torch.nn.Module): Self-attention module instance. `MultiHeadedAttention` instance can be used as the argument. src_attn (torch.nn.Module): Self-attention module instance. `MultiHeadedAttention` instance can be used as the argument. feed_forward (torch.nn.Module): Feed-forward module instance. `PositionwiseFeedForward`, `MultiLayeredConv1d`, or `Conv1dLinear` instance can be used as the argument. dropout_rate (float): Dropout rate. normalize_before (bool): Whether to use layer_norm before the first block. concat_after (bool): Whether to concat attention layer's input and output. if True, additional linear will be applied. i.e. x -> x + linear(concat(x, att(x))) if False, no additional linear will be applied. i.e. x -> x + att(x) TFNc  sRtt|||_||_||_t||_t||_t||_ t ||_ ||_ ||_|jr@t ||||_t ||||_||_|dddkrXddlm} ddlm} n!|dddkrmddlm} ddlm} n ddlm} ddlm} | ||d |_||_d |_|jdkr|d d st|j|_|d d rtd d}d||dd} t j !|jj"| |d d rtd d||dd} t j !|jj"| |d d rtd t j j#|jj$j"ddt j j#|jj%j"ddt j j#|jj&j"ddt j j#|jj'j"ddt j (|jj)j"|dddkr'|j*t+j,d Sd S)z!Construct an DecoderLayer object.versionv4r) RWKVLayer) RWKV_TimeMixv5)RWKV_Tmix_x052)RWKV_Tmix_x060)layer_idNln0T init_rwkvn_layergffffff?)gain皙?datatypebf16)-rr%__init__sizesrc_attn feed_forwardr norm1norm2norm3rDropoutdropoutnormalize_before concat_afterLinearconcat_linear1concat_linear2r-get!funasr.models.sense_voice.rwkv_v4r(r)!funasr.models.sense_voice.rwkv_v5r+!funasr.models.sense_voice.rwkv_v6r, self_attnargsr.n_embdprintinit constant_weight orthogonal_ receptancekeyvaluegatezeros_outputtotorchbfloat16) rr7r8r9 dropout_rater?r@r-rIkwargsr( RWKV_Tmixscalerrr r6;s\           zDecoderLayer.__init__c Cs(|jdkr|jdur||}|jdddkr|}|}||}|dur4|||j||d}nd}|||j||d}|dddddf}|jdddkr\|t j }|}| |}||| ||||}|}| |}||||}|durt j||gdd}||||fS) a"Compute decoded features. Args: tgt (torch.Tensor): Input tensor (#batch, maxlen_out, size). tgt_mask (torch.Tensor): Mask for input tensor (#batch, maxlen_out). memory (torch.Tensor): Encoded memory, float32 (#batch, maxlen_in, size). memory_mask (torch.Tensor): Encoded memory mask (#batch, maxlen_in). cache (List[torch.Tensor]): List of cached tensors. Each tensor shape should be (#batch, maxlen_out - 1, size). Returns: torch.Tensor: Output tensor(#batch, maxlen_out, size). torch.Tensor: Mask for output tensor (#batch, maxlen_out). torch.Tensor: Encoded memory (#batch, maxlen_in, size). torch.Tensor: Encoded memory mask (#batch, maxlen_in). rNr4r5)maskr0dim)r-r.rIrDrXr:r>rHrVrWfloat32r;r8r<r9cat) rtgttgt_maskmemory memory_maskcacheresidualr tgt_q_maskrrr rs,      zDecoderLayer.forwardr)r!r"r#__doc__r6rr$rrrr r%%sGr%cseZdZdZddddedfdedededed ed ed effd d Z de j de j de j de j de e j e j ff ddZ d#de j de j de j dee j de e j ee j ff ddZddZde j deed e j de e j eeffd!d"ZZS)$BaseTransformerDecoderaBase class of Transfomer decoder module. Args: vocab_size: output dim encoder_output_size: dimension of attention attention_heads: the number of heads of multi head attention linear_units: the number of units of position-wise feed forward num_blocks: the number of decoder blocks dropout_rate: dropout rate self_attention_dropout_rate: dropout rate for attention input_layer: input layer type use_output_layer: whether to use output layer pos_enc_class: PositionalEncoding or ScaledPositionalEncoding normalize_before: whether to use layer_norm before the first block concat_after: whether to concat attention layer's input and output if True, additional linear will be applied. i.e. x -> x + linear(concat(x, att(x))) if False, no additional linear will be applied. i.e. x -> x + att(x) r3embedT vocab_sizeencoder_output_sizerYpositional_dropout_rate input_layeruse_output_layerr?c st|} |dkrtjtj|| || ||_n*|dkr?tjtj|| tj| tj |tj || ||_nt d|||_ |j rQt| |_ |r\tj| ||_nd|_d|_dS)Nrllinearz'only 'embed' or 'linear' is supported: )rr6rWr Sequential EmbeddingrlrAr r=ReLU ValueErrorr? after_norm output_layerdecoders) rrmrnrYrorprq pos_enc_classr? attention_dimrrr r6s.       zBaseTransformerDecoder.__init__hs_padhlens ys_in_pad ys_in_lensreturnc C|}t|dddddf|j}t|d|jdd}||@}|}t||dddddddf|j} | jd|jdkr^|jd| jd} tjj | d| fdd} | |} | | ||| \} }}} |j rw|| } |jdur|| } |d} | | fS a@Forward decoder. Args: hs_pad: encoded memory, float32 (batch, maxlen_in, feat) hlens: (batch) ys_in_pad: input token ids, int64 (batch, maxlen_out) if input_layer == "embed" input tensor (batch, maxlen_out, #mels) in the other cases ys_in_lens: (batch) Returns: (tuple): tuple containing: x: decoded token score before softmax (batch, maxlen_out, token) if use_output_layer is True, olens: (batch, ) Nr^devicerr0)maxlenconstantFrrVrrr7 unsqueezeshaperWr functionalpadrlryr?rwrxsum rr|r}r~rrcrdmrerfpadlenrolensrrr r"$.     zBaseTransformerDecoder.forwardNrcrdrergc C||}|durdgt|j}g}t||jD]\}}||||d|d\}}}} ||q|jr?||dddf} n|dddf} |jdurVtj || dd} | |fSa5Forward one step. Args: tgt: input token ids, int64 (batch, maxlen_out) tgt_mask: input token mask, (batch, maxlen_out) dtype=torch.uint8 in PyTorch 1.2- dtype=torch.bool in PyTorch 1.2+ (include 1.2) memory: encoded memory, float32 (batch, maxlen_in, feat) cache: cached output list of (batch, max_time_out-1, size) Returns: y, cache: NN output value and cache per `self.decoders`. y.shape` is (batch, maxlen_out, token) Nrgr^r_ rllenryzipappendr?rwrxrW log_softmax rrcrdrergr new_cachecdecoderrfyrrr forward_one_step*   z'BaseTransformerDecoder.forward_one_stepcCsHtt||jdd}|j|d||d|d\}}|d|fS)zScore.rrr)rrrrrsqueeze)rysstaterys_masklogprrr scoreOs"zBaseTransformerDecoder.scorerstatesxscst|t|jddurd}n fddtD}t|d|jdd}|j||||d\}fddtD}||fS) aScore new token batch. Args: ys (torch.Tensor): torch.int64 prefix tokens (n_batch, ylen). states (List[Any]): Scorer states for prefix tokens. xs (torch.Tensor): The encoder feature that generates ys (n_batch, xlen, n_feat). Returns: tuple[torch.Tensor, List[Any]]: Tuple of batchfied scores for next token with shape of `(n_batch, n_vocab)` and next state list for ys. rNcs*g|]tfddtDqS)cg|]}|qSrr).0b)irrr nABaseTransformerDecoder.batch_score...)rWstackranger)n_batchr)rr rmsz6BaseTransformerDecoder.batch_score..r^rrcs$g|]fddtDqS)crrr)rr)rrrr rvrr)rr)n_layersr)rr rvs$)rryrrr7rrr)rrrr batch_staterr state_listr)rrrr batch_scoreUs   z"BaseTransformerDecoder.batch_scorer)r!r"r#rjr intrstrboolr6rWTensorrrrrrrrr$rrrr rksp ) 6 %rkdecoder_classesTransformerRWKVDecodercseZdZdddddddddeddf d ed ed ed ed ededededededededeffdd Zde j de j de j de j de e j e j ff ddZ d&d e j d!e j d"e j d#e e j de e j e e j ff d$d%ZZS)'rir3grlTFrmrnattention_heads linear_units num_blocksrYroself_attention_dropout_ratesrc_attention_dropout_raterprqr?r@c stj|||| | | d|di}t||t|fdd|_ddrEtdtj j |j dj dd d dSdS) N)rmrnrYrorprqrzr?rwkv_cfgc s(ttt|dS)N)rI)r%rr)lnumrIr{rr@rYrr?rrr s  z1TransformerRWKVDecoder.__init__..r/Trg-C6g-C6?)ar) rr6rDrcreaterryrKrrLuniform_rlrN)rrmrnrrrrYrorrrprqrzr?r@rZrrrr r6|s*  zTransformerRWKVDecoder.__init__r|r}r~rrc Crrrrrrr rrzTransformerRWKVDecoder.forwardNrcrdrergc Crrrrrrr rrz'TransformerRWKVDecoder.forward_one_stepr)r!r"r#r rrrrr6rWrrrrrr$rrrr rzs~    5 6))rjtypingrrrrrWr#funasr.models.transformer.attentionr,funasr.models.transformer.utils.dynamic_convr.funasr.models.transformer.utils.dynamic_conv2dr #funasr.models.transformer.embeddingr $funasr.models.transformer.layer_normr )funasr.models.transformer.utils.lightconvr +funasr.models.transformer.utils.lightconv2dr $funasr.models.transformer.utils.maskr*funasr.models.transformer.utils.nets_utilsr3funasr.models.transformer.positionwise_feed_forwardr&funasr.models.transformer.utils.repeatr2funasr.models.transformer.scorers.scorer_interfacer omegaconfrfunasr.registerrModuler%rkregisterrrrrr s8                    A