o ei}@sdZddlZddlmZddlZddlmZddlmZddl m Z ddl m Z dd l mZdd lmZmZmZmZmZmZmZd d lmZeeZdad dZGdddejjZ d+ddZ!d+ddZ"Gdddej#Z$Gdddej#Z%Gddde Z&eGdddeZ'eeddGd d!d!eZ(eed"dGd#d$d$eZ)eGd%d&d&e'Z*ed'dGd(d)d)e'e Z+gd*Z,dS),zPyTorch RWKV model.N) dataclass)nn)initialization)GenerationMixin)GradientCheckpointingLayer)PreTrainedModel) ModelOutputauto_docstringis_bitsandbytes_availableis_kernels_availableis_ninja_availableis_torch_cuda_availablelogging) RwkvConfigcCs,tstdddlm}|da|t_dS)NzFkernels is not installed, please install it with `pip install kernels`r) get_kernelzkernels-community/rwkv)r ImportErrorintegrations.hub_kernelsrrwkv_cuda_kernelmax_seq_length)context_lengthrrd/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/rwkv/modeling_rwkv.pyload_wkv_cuda_kernel-s   rc@s(eZdZedddZedddZdS) RwkvLinearAttentionNFc Cs|\}}} |tjkrtd|dtjd|| t| ddkr4td|d| dt| dd |j|_|jjd ksP|jjd ksP|jjd ksP|jjd krTtd t |  }|jt j krp| }| }| }| }| }| }t j|t jd } |s|dur|durt j|| d t j|jt jd}|dddddfd8<nt jdd|Ddd }|jt jkrtj} ntj} | ||||| |n|jt jkrtjntj} | ||||| |||||| |durddt j|d ddD}| |j|fS)NzCannot process a batch with z+ tokens at the same time, use a maximum of z with this model. rzThe product of batch size (z) and hidden size (z") needs to be a round multiple of .cudazUCalling the CUDA kernel for wkv attention requires all tensors to be on CUDA devices. memory_formatr)dtypedevicer 籡*GcSg|]}|dqSr#) unsqueeze.0srrr gz/RwkvLinearAttention.forward..)dimcSr%r&)squeezer(rrrr+tr,)sizerr ValueErrorminr! input_dtyper"typetorchexpfloat contiguousfloat16 empty_likecontiguous_formatzerosfloat32catbfloat16forward_with_state_bf16forward_with_state forward_bf16forwardsave_for_backwardchunkto) ctx time_decay time_firstkeyvaluestate return_state batch_sizeseq_len hidden_sizeoutput forward_funcrrrrB9sd         zRwkvLinearAttention.forwardc Cs|j}|j\}}}}}tj|tj|tjkrtjntjd} tj|tjd} tj|tjd} tj|tjd} |tjkr>|}|tjkrFt j nt j } | |||||| | | | | | || || || |ddfS)N)r r!r)r2 saved_tensorsr4r9r:r>r<r8r6r backward_bf16backwardr7rE)rFg_outputg_stater2rGrHrIrJrP g_time_decay g_time_firstg_keyg_value backward_funcrrrrTxs@ zRwkvLinearAttention.backwardNFN)__name__ __module__ __qualname__ staticmethodrBrTrrrrr8s  >rFcCs|\}}}t|}|dur=tj|dddftjd} tj|dddftjd} tj|dddftjdd} n|\} } } t| }t|D]p} |dd| f} |dd| f}t| | |}t| |}t| ||}|| ||}|| |}|||j |dd| f<t| || }t| ||}t| |}|| ||} || |} |} qL|s|dur| | | g}||fS)Nr)r!r$) r/r4 zeros_liker<r5ranger6maximumrEr!)rGrHrIrJrKrL_ seq_lengthrP num_state den_state max_state current_index current_key current_valuemax_for_outpute1e2 numerator denominator max_for_staterrrrwkv_linear_attention_cpus4 "       rscCs`tdd||||fD}|ddk}tdus|s|r&t||||||dSt||||||S)Ncss|] }|jjdkVqdS)rN)r"r3)r)trrr z(rwkv_linear_attention..rrKrL)anyr/rrsrapply)rGrHrIrJrKrLno_cuda one_tokenrrrrwkv_linear_attentions r|cs2eZdZd fdd Zd ddZd dd ZZS) RwkvSelfAttentionrcsDt||_tduotj|jk}tr0tr0|s0zt|jWnt y/t dYnw||_ |j }|jdur>|jn|}||_tt||_tt||_ttdd||_ttdd||_ttdd||_td|_tj||dd|_tj||dd|_tj||dd|_tj||dd|_dS)Nz9Could not load the custom CUDA kernel for RWKV attention.rrrrFbias)super__init__configrrrr rr Exceptionloggerinfolayer_idrOattention_hidden_sizer Parameterr4emptyrGrH time_mix_keytime_mix_valuetime_mix_receptance ZeroPad2d time_shiftLinearrIrJ receptancerP)selfrr kernel_loadedrOr __class__rrrs0   zRwkvSelfAttention.__init__NcCs|ddkr|dur|ddddd|jf}n||}|dur7|ddddd|jf|dddf<||j|d|j}||j|d|j}||j|d|j}||}||}t | |}|dur|dddf|ddddd|jf<||||fSNrrr) r/rrrrrrIrJr4sigmoidr)rhiddenrKshiftedrIrJrrrrextract_key_values (  ( z#RwkvSelfAttention.extract_key_valueFc sj||d\}}}}|durtfdd|ddDnd}tjj||||d\}}|durb|d|dddddjf<|d|dddddjf<|d|d ddddjf<|||fS) NrKc3s(|]}|ddddjfVqdSr]rr(rrrru s&z,RwkvSelfAttention.forward..r#rwrrr)rtupler|rGrHrrP) rrrK use_cacherrIrJ layer_staterwkvrrrrB s*    zRwkvSelfAttention.forwardrr]r\)r^r_r`rrrB __classcell__rrrrr}s r}cs(eZdZdfdd ZdddZZS) RwkvFeedForwardrcst||_||_|j}|jdur|jnd|j}td|_t t dd||_ t t dd||_ tj||dd|_tj||dd|_tj||dd|_dS)Nrr~rFr)rrrrrOintermediate_sizerrrrr4rrrrrIrrJ)rrrrOrrrrr s  zRwkvFeedForward.__init__NcCs |ddkr|dur|ddddd|jf}n||}|dur7|ddddd|jf|dddf<||j|d|j}||j|d|j}tt||}| |}t | |}|dur|dddf|ddddd|jf<|||fSr) r/rrrrr4squarerelurIrJrr)rrrKrrIrrJrrrrB1s ( ( zRwkvFeedForward.forwardrr]r^r_r`rrBrrrrrrsrcs&eZdZfddZdddZZS) RwkvBlockcsvt||_||_|dkrtj|j|jd|_tj|j|jd|_ tj|j|jd|_ t |||_ t |||_dS)Nr)eps)rrrrr LayerNormrOlayer_norm_epsilonpre_lnln1ln2r} attentionr feed_forward)rrrrrrrFs  zRwkvBlock.__init__NFcCs||jdkr ||}|j||||d\}}||}|j|||d\}}||}||f}|r8||f7}|S|d7}|S)Nr)rKrrr])rrrrrr)rrrKroutput_attentionsrroutputsrrrrBTs   zRwkvBlock.forward)NFFrrrrrrEs rc@sHeZdZUeed<dZdgZddgZdZdZ e de j fdd Zd S) RwkvPreTrainedModelrrrrGrHTmodulec st|tr|j}|jj}|jj|j||dd||}tjfddt D|j j |j j d}|ddddf}fddt D}tj||j j |j j d}tjddt D|jj |jj dd }t|j |t|jt|jtd |t|j t||t|jt||d t|jt|d |dSt|tr|j}|jj}|jjd||}tjfd dt D|j j |j j d}|ddddf}t|j t||t|jt||dSt|tjrI|jj}d} d} |jdurt|j|d |dkr't|d |d} |d |jjkr;|d|jjkr;d } | | 9} tj |j| d dSt|tj!rl|jj}dtt"|d |d} tj |j| d dSt|tj#rt$|jt|jdSdS)zInitialize the weights.rg?cg|]}|qSrrr)irOrrr+}z5RwkvPreTrainedModel._init_weights..r!r"Ncs,g|]}dd|dddqS)rgffffff?g?r)r)h)r ratio_0_to_1rrr+scSsg|] }|dddqS)rrrrrrrr+sg?g333333?crrrrrrrr+rr)gaing-C6?)% isinstancer}rrnum_hidden_layersrOrr4tensorrcrr!r"rGrHinitcopy_ ones_likemathlogpowrrrrrweightshaperzeros_sqrt vocab_size orthogonal_ Embeddingmaxrones_) rrrrratio_1_to_almost0 time_weight decay_speedzigzagrrscaler)rrOrr _init_weightsps~     $    $ z!RwkvPreTrainedModel._init_weightsN)r^r_r`r__annotations__base_model_prefix_no_split_modules_keep_in_fp32_modulessupports_gradient_checkpointing _is_statefulr4no_gradrModulerrrrrrgs rz+ Class for the RWKV model outputs. ) custom_introc@sneZdZUdZdZejdBed<dZe ejdBed<dZ e ejdfdBed<dZ e ejdfdBed<dS) RwkvOutputa  state (list of five `torch.FloatTensor` of shape `(batch_size, hidden_size, num_hidden_layers)`): The state of the model at the last time step. Can be used in a forward method with the next `input_ids` to avoid providing the old `input_ids`. Nlast_hidden_staterK. hidden_states attentions) r^r_r`__doc__rr4 FloatTensorrrKlistrrrrrrrrs rzK Base class for causal language model (or autoregressive) outputs. c@seZdZUdZdZejdBed<dZejdBed<dZ e ejdBed<dZ e ejdfdBed<dZ e ejdfdBed<dS) RwkvCausalLMOutputap loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). state (list of five `torch.FloatTensor` of shape `(batch_size, hidden_size, num_hidden_layers)`): The state of the model at the last time step. Can be used in a forward method with the next `input_ids` to avoid providing the old `input_ids`. NlosslogitsrK.rr)r^r_r`rrr4rrrrKrrrrrrrrrs  rcseZdZfddZddZddZe        ddejdBd ejdBd ej dBd e ej dBd e dBd e dBde dBde dBde e BfddZddZddZZS) RwkvModelcsdttjj|_tfddtj D|_ t j|_ d|_ d|_|dS)Ncsg|]}t|dqS)r)r)r)idxrrrr+sz&RwkvModel.__init__..F)rrrrrrO embeddings ModuleListrcrblocksrln_outlayers_are_rescaledgradient_checkpointing post_initrrrrrrs   zRwkvModel.__init__cC|jSr]rrrrrget_input_embeddingszRwkvModel.get_input_embeddingscC ||_dSr]rrnew_embeddingsrrrset_input_embeddings zRwkvModel.set_input_embeddingsN input_idsattention_mask inputs_embedsrKrroutput_hidden_states return_dictreturnc  s|dur|n|jj}|dur|n|jj}|dur|n|js!|jjnd}|dur)|n|jj}|dur6td|j|jkr@| |durLdurLt d|durXdurXt ddura| ||r|dur d|jj |jjffddtd D}|d d 8<|jr|jr|rtd d}} |rd nd} |rd nd} t|jD]4\} }|| |||d\} }}|jr|jjdkr| d|jjdkr| d} |r| | f} |r| |f} q|| } |r| | f} |stdd| || | fDSt| || | dS)a input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values.get_seq_length()` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only `input_ids` that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) state (tuple of five `torch.FloatTensor` of shape `(batch_size, hidden_size, num_hidden_layers)`, *optional*): If passed along, the model uses the previous state in all the blocks (which will give the output for the `input_ids` provided as if the model add `state_input_ids + input_ids` as context). use_cache (`bool`, *optional*): If set to `True`, the last state is returned and can be used to quickly generate the next logits. NFz<`attention_mask` was passed, but it is unused in this model.zDYou cannot specify both input_ids and inputs_embeds at the same timez5You have to specify either input_ids or inputs_embedsrcs0g|]}tj|dkrjntjjdqS)rr)r4r;r!r<r"rrrrrr+5s z%RwkvModel.forward..rgꌠ9Y>)FzZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...r)rKrrrr#css|] }|dur|VqdSr]r)r)xrrrru`rvz$RwkvModel.forward..)rrKrr)rrrtrainingruse_return_dictr warning_oncer_rescale_layersr0rr/rOrrcr enumerater rescale_everyrrr)rrrrrKrrrrkwargsrall_self_attentionsall_hidden_statesrblockrrrrrBsn               zRwkvModel.forwardcCsx|j|j kr dS|jjdkrtt|jD]\}}|jrA|jj j dt ||jj|j jj dt ||jjqt|jj j drl|jj j jdt ||jj|j jj jdt ||jjqt|jj j dr||jj |||j j|q|jj j dt ||jj|j jj dt ||jjqWdn1swY|j |_dS)Nrr#SCB quant_state)rr rrr4rrrrrPrmul_intrrJhasattrrdiv_ _bnb_4bit_dequantize_and_rescale)rblock_idrrrrris&   ""$ "zRwkvModel._rescale_layerscCsttstdddl}|j|jj|jj}|dt ||j j |j j |ddd|j}t|d|dS) z Perform the dequantization and rescaling of the weights of a given layer. After that operation the layer will be quantized again. z/Please install bitsandbytes to use this method.rNr#cpuF) requires_gradr)r r bitsandbytes functionaldequantize_4bitrdatarrrrrr Params4bitrEr"setattr)r target_layerrbnbdequant_weights quant_weightrrrrsz*RwkvModel._bnb_4bit_dequantize_and_rescale)NNNNNNNN)r^r_r`rrrr r4 LongTensorrrboolrrrBrrrrrrrrsD    jrz The RWKV Model transformer with a language modeling head on top (linear layer with weights tied to the input embeddings). cseZdZddiZfddZddZddZe dd ej d Bd ej d Bd ej d Bde ej d Bdej d Bde d Bde d Bde d Bde d Bde ejBdeeBfddZZS)RwkvForCausalLMz head.weightzrwkv.embeddings.weightcs8t|t||_tj|j|jdd|_| dS)NFr) rrrrrrrOrheadrrrrrrs   zRwkvForCausalLM.__init__cCrr]r,rrrrget_output_embeddingsrz%RwkvForCausalLM.get_output_embeddingscCrr]r-rrrrset_output_embeddingsrz%RwkvForCausalLM.set_output_embeddingsNrrrrrKlabelsrrrrlogits_to_keeprc  Ks| dur| n|jj} |j||||||| d} | d} t| tr%t| dn| }|| dd|ddf}d}|durI|jd|||jjd| }| s_|f| dd}|dur]|f|S|St ||| j | j | j dS)aJ input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values.get_seq_length()` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only `input_ids` that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) state (tuple of five `torch.FloatTensor` of shape `(batch_size, hidden_size, num_hidden_layers)`, *optional*): If passed along, the model uses the previous state in all the blocks (which will give the output for the `input_ids` provided as if the model add `state_input_ids + input_ids` as context). labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for language modeling. Note that the labels **are shifted** inside the model, i.e. you can set `labels = input_ids` Indices are selected in `[-100, 0, ..., config.vocab_size]` All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` use_cache (`bool`, *optional*): If set to `True`, the last state is returned and can be used to quickly generate the next logits. N)rrKrrrrr)rr0rr)rrrKrrr) rr rrrslicer, loss_functionrrrKrr)rrrrrKr0rrrrr1r rwkv_outputsr slice_indicesrrrPrrrrBs4& zRwkvForCausalLM.forward) NNNNNNNNNr)r^r_r`_tied_weights_keysrr.r/r r4r)rrr*rTensorrrrBrrrrrr+sN      r+)r+rrr\)-rr dataclassesrr4rrr generationrmodeling_layersrmodeling_utilsrutilsr r r r r rrconfiguration_rwkvr get_loggerr^rrrautogradFunctionrrsr|rr}rrrrrrr+__all__rrrrsP      $  j , F&"U / Y