o TiI@snddlZddlmZddlmZmZmZmZddlm Z Gdddej Z Gddde Z Gd d d e Z dS) N)AutoModelForCausalLMAutoModelForSeq2SeqLMis_torch_npu_availableis_torch_xpu_available)PreTrainedModelWrappercs(eZdZdZfddZddZZS) ValueHeadze The ValueHead class implements a head for GPT2 that returns a scalar for each output token. c stt|ds|dd}n|j}|rt|nt|_t|dr(|j }t|dr1|j }nt|drH|j rHt|drHt|j drH|j j }t |d|_t|_dS)Nsummary_dropout_probg? hidden_sizeword_embed_proj_dimis_encoder_decoderdecoderr)super__init__hasattrpopr nnDropoutIdentitydropoutr r r r LinearsummaryFlattenflatten)selfconfigkwargsr r  __class__R/home/ubuntu/.local/lib/python3.10/site-packages/trl/models/modeling_value_head.pyrs      zValueHead.__init__cCs8||}|j|jjjkr||jjj}||}|SN)rdtyperweightto)r hidden_statesoutputrrr forward2s  zValueHead.forward)__name__ __module__ __qualname____doc__rr' __classcell__rrrr rs rcsbeZdZdZeZdZfddZddZ    dd d Z d d Z d dZ ddZ ddZ ZS)!AutoModelForCausalLMWithValueHeadaW An autoregressive model with a value head in addition to the language model head. This class inherits from `~trl.PreTrainedModelWrapper` and wraps a `transformers.PreTrainedModel` class. The wrapper class supports classic functions such as `from_pretrained`, `push_to_hub` and `generate`. To call a method of the wrapped model, simply manipulate the `pretrained_model` attribute of this class. Class attributes: - **transformers_parent_class** (`transformers.PreTrainedModel`) -- The parent class of the wrapped model. This should be set to `transformers.AutoModelForCausalLM` for this class. - **supported_args** (`tuple`) -- A tuple of strings that are used to identify the arguments that are supported by the `ValueHead` class. Currently, the supported args are: - **summary_dropout_prob** (`float`, `optional`, defaults to `None`) -- The dropout probability for the `ValueHead` class. - **v_head_initializer_range** (`float`, `optional`, defaults to `0.2`) -- The initializer range for the `ValueHead` if a specific initialization strategy is selected. - **v_head_init_strategy** (`str`, `optional`, defaults to `None`) -- The initialization strategy for the `ValueHead`. Currently, the supported strategies are: - **`None`** -- Initializes the weights of the `ValueHead` with a random distribution. This is the default strategy. - **"normal"** -- Initializes the weights of the `ValueHead` with a normal distribution. r v_head_initializer_rangev_head_init_strategyc sNtj|fi|||\}}}t|jjfi||_|jdi|dS)a Initializes the model. Args: pretrained_model (`transformers.PreTrainedModel`): The model to wrap. It should be a causal language model such as GPT2. or any model mapped inside the `AutoModelForCausalLM` class. kwargs (`dict`, `optional`): Additional keyword arguments, that are passed to the `ValueHead` class. Nr)rr _split_kwargsrpretrained_modelrv_head _init_weightsrr2r v_head_kwargs_rrr r\s z*AutoModelForCausalLMWithValueHead.__init__cKZ|dd}|dd}|durdS|dkr+|jjjjjd|d|jjjjdSdS)av Initializes the weights of the value head. The default initialization strategy is random. Users can pass a different initialization strategy by passing the `v_head_init_strategy` argument when calling `.from_pretrained`. Supported strategies are: - `normal`: initializes the weights with a normal distribution. Args: **kwargs (`dict`, `optional`): Additional keyword arguments, that are passed to the `ValueHead` class. These arguments can contain the `v_head_init_strategy` argument as well as the `v_head_initializer_range` argument. r/皙?r0Nnormalmeanstdrr3rr#datanormal_biaszero_rrinitializer_range init_strategyrrr r4ls z/AutoModelForCausalLMWithValueHead._init_weightsNFc Ksd|d<||d<|jr|jjjdkr|d|jd||d|}|jd}|j}|j} |j|j j j jkr?| |j j j j}| | d} |jtjkrQ|}|rZ|| | |jfS|| | fS) aQ Applies a forward pass to the wrapped model and returns the logits of the value head. Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. past_key_values (`tuple(tuple(torch.FloatTensor))`, `optional`): Contains pre-computed hidden-states (key and values in the attention blocks) as computed by the model (see `past_key_values` input) to speed up sequential decoding. attention_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, `optional`): Mask to avoid performing attention on padding token indices. Mask values selected in ``[0, 1]``: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. return_past_key_values (bool): A flag indicating if the computed hidden-states should be returned. kwargs (`dict`, `optional`): Additional keyword arguments, that are passed to the wrapped model. Toutput_hidden_statespast_key_values PREFIX_TUNING) input_idsattention_maskNr) is_peft_modelr2active_peft_config peft_typerr%logitslossdevicer3rr#r$squeezer"torchfloat32floatrH rrJrHrKreturn_past_key_valuesrbase_model_outputlast_hidden_state lm_logitsrQvaluerrr r's*    z)AutoModelForCausalLMWithValueHead.forwardcO|jj|i|S)a/ A simple wrapper around the `generate` method of the wrapped model. Please refer to the [`generate`](https://huggingface.co/docs/transformers/internal/generation_utils) method of the wrapped model for more information about the supported arguments. Args: *args (`list`, *optional*): Positional arguments passed to the `generate` method of the wrapped model. **kwargs (`dict`, *optional*): Keyword arguments passed to the `generate` method of the wrapped model. r2generaterargsrrrr r_s z*AutoModelForCausalLMWithValueHead.generatecOT|js |jj|i|}ni}|jj|i|}|D] \}}||d|<q|Sz Returns the state dictionary of the model. We add the state dictionary of the value head to the state dictionary of the wrapped model by prepending the key with `v_head.`. v_head.rMr2 state_dictr3itemsrrarpretrained_model_state_dictv_head_state_dictkvrrr rfz,AutoModelForCausalLMWithValueHead.state_dictcO|j|j_|jj|i|Sr!r3r2 push_to_hubr`rrr rp z-AutoModelForCausalLMWithValueHead.push_to_hubcst|D]}d|vr||||dd<q|jj|dd~t|jdrzd|jj vs7d|jj vr;t dtt |jj d t t rctrUd ntr^d nd |j|_fd d}||d|_dSdS) We add the state dictionary of the value head to the state dictionary of the wrapped model by prepending the key with `v_head.`. This function removes the `v_head.` prefix from the keys of the value head state dictionary. rdFstrict hf_device_mapcpudiskdThe model is offloaded on CPU or disk - CPU & disk offloading is not supported for ValueHead models.rznpu:zxpu:zcuda:c:d}|D]}t|tjr||f7}q||f7}q|S)Nr isinstancerTTensorr$moduleinputoutputs new_outputr& first_devicerr set_device_hooks   zDAutoModelForCausalLMWithValueHead.post_init..set_device_hookTN)listkeysrreplacer3load_state_dictrr2rvvalues ValueErrorsetr|intrrr$register_forward_hookis_sequential_parallel)rrfrkrrrr post_inits0       z+AutoModelForCausalLMWithValueHead.post_initNNNF)r(r)r*r+rtransformers_parent_classsupported_argsrr4r'r_rfrprr,rrrr r->s  7r-csreZdZdZeZgdZdZfddZddZ dd Z d d Z d d Z ddZ    dddZddZZS)"AutoModelForSeq2SeqLMWithValueHeada A seq2seq model with a value head in addition to the language model head. This class inherits from `~trl.PreTrainedModelWrapper` and wraps a `transformers.PreTrainedModel` class. The wrapper class supports classic functions such as `from_pretrained` and `push_to_hub` and also provides some additional functionalities such as `generate`. Args: pretrained_model (`transformers.PreTrainedModel`): The model to wrap. It should be a causal language model such as GPT2. or any model mapped inside the `AutoModelForSeq2SeqLM` class. kwargs: Additional keyword arguments passed along to the `ValueHead` class. )lm_head embed_outoutput_projectionr.c sdtj|fi|||\}}}d|_|stdt|jjfi||_ |j di|dS)NTzOThe model does not have a language model head, please use a model that has one.r) rrr1r _has_lm_headrrr2rr3r4r5rrr r sz+AutoModelForSeq2SeqLMWithValueHead.__init__cs6|jD]\}tfdd|jDrdSqdS)Nc3|]}|vVqdSr!r.0 attributenamerr /zBAutoModelForSeq2SeqLMWithValueHead._has_lm_head..TF)r2 named_modulesanylm_head_namings)r_modulerrr r,s z/AutoModelForSeq2SeqLMWithValueHead._has_lm_headcst|D]}d|vr||||dd<q|jj|dd~t|jdrnd|jj vs7d|jj vr;t d|j D]\}t fd d |j DrV|jjnq@|j|_fd d }||d |_dSdS)rrrdrsFrtrvrwrxryc3rr!rrrrr rJrz?AutoModelForSeq2SeqLMWithValueHead.post_init..crz)a A hook that sets the device of the output of the model to the device of the first parameter of the model. Args: module (`nn.Module`): The module to which the hook is attached. input (`tuple`): The input to the module. outputs (`tuple`): The output of the module. rr{r~)lm_head_devicerr rQs    zEAutoModelForSeq2SeqLMWithValueHead.post_init..set_device_hookTN)rrrrr3rrr2rvrrrrrr#rRr$rr)rrfrkrrr)rrr r3s,    z,AutoModelForSeq2SeqLMWithValueHead.post_initcOrbrcrerhrrr rfirmz-AutoModelForSeq2SeqLMWithValueHead.state_dictcOrnr!ror`rrr rpyrqz.AutoModelForSeq2SeqLMWithValueHead.push_to_hubcKr8)z> We initialize the weights of the value head. r/r9r0Nr:r;r<r?rDrrr r4~s  z0AutoModelForSeq2SeqLMWithValueHead._init_weightsNFc Ks||d<|jr|jjjdkr|d|jd||dd|}|jd}|j}|j} || d} |j t j kr<| }|rE|| | |jfS|| | fS)NrHrIT)rJrKrGrLr)rMr2rNrOrdecoder_hidden_statesrPrQr3rSr"rTrUrVrHrWrrr r's&    z*AutoModelForSeq2SeqLMWithValueHead.forwardcOr])z: We call `generate` on the wrapped model. r^r`rrr r_sz+AutoModelForSeq2SeqLMWithValueHead.generater)r(r)r*r+rrrrrrrrfrpr4r'r_r,rrrr r s"  6 "r)rTtorch.nnr transformersrrrr modeling_baserModulerr-rrrrr s  (L