o i@spddlZddlmZmZmZddlmZddlmZddl m Z m Z Gdddej j ZGdd d ej j ZdS) N)DictOptionalTuple) LayerNorm) FeedForward)EncoderSelfAttentionDecoderSelfAttentionceZdZdZ   ddedededededed ed ed ed d ffdd Z ddejde ejd e eje ejffddZ Z S)RWKV)RWKV module. Args: size: Input/Output size. linear_size: Feed-forward hidden size. attention_size: SelfAttention hidden size. context_size: Context size for WKV computation. block_id: Block index. num_blocks: Number of blocks in the architecture. normalization_class: Normalization layer class. normalization_args: Normalization layer arguments. att_dropout_rate: Dropout rate for the attention module. ffn_dropout_rate: Dropout rate for the feed-forward module. size linear_sizeattention_size context_sizeblock_id num_blocksatt_dropout_rateffn_dropout_rate dropout_ratereturnNc htt||_t||_t|||||||_tjj | d|_ t ||||||_ tjj | d|_ dSzConstruct a RWKV object.)pN)super__init__rlayer_norm_attlayer_norm_ffnratttorchnnDropout dropout_attrffn dropout_ffn selfr rrrrrrrr __class__O/home/ubuntu/.local/lib/python3.10/site-packages/funasr/models/rwkv_bat/rwkv.pyr   z RWKV.__init__xstatecCT|j|||d\}}|||}|j|||d\}}|||}||fSa8Compute receptance weighted key value. Args: x: RWKV input sequences. (B, L, size) state: Decoder hidden states. [5 x (B, D_att/size, N)] Returns: x: RWKV output sequences. (B, L, size) x: Decoder hidden states. [5 x (B, D_att/size, N)] )r-rrr"r#rr$r&r,r-rr#r)r)r*forward9s z RWKV.forwardr r r N __name__ __module__ __qualname____doc__intfloatrrTensorrrr2 __classcell__r)r)r'r*r D   r cr )RWKVDecoderLayerr r r rrrrrrrrrNc rr)rrrrrrrrr r!r"rr#r$r%r'r)r*rar+zRWKVDecoderLayer.__init__r,r-cCr.r/r0r1r)r)r*r2{s zRWKVDecoderLayer.forwardr3r4r5r)r)r'r*r?Pr>r?)rtypingrrr$funasr.models.transformer.layer_normr(funasr.models.rwkv_bat.rwkv_feed_forwardr%funasr.models.rwkv_bat.rwkv_attentionrrr Moduler r?r)r)r)r*s  B