o
TieM @ s d dl Z d dlmZ d dlmZ d dlmZ d dlmZm Z m
Z
m Z m Z d dl
mZ d dlmZ d dlmZmZmZmZ dd lmZmZ eefZerbd d
lmZ d d l m Z d d l
m Z d d
l m Z eG dd dZ!de!iZ" d4dedede e
d de e# de$eef f
ddZ% d5dedede&de e# de$eee'e# f f
ddZ(d6d d Z)d7d d!Z*d7d"d#Z+d6d$d%Z,e &d8de d' d(d)d*e-fd+d,Z.d9d.d/Z/d0d1 Z0G d2d3 d3Z1dS ): N)contextmanager)deepcopy) dataclass)
TYPE_CHECKINGAnyLiteralOptionalUnion)version)
AddedToken
AutoTokenizerPreTrainedModelPreTrainedTokenizer )!AutoModelForCausalLMWithValueHead"AutoModelForSeq2SeqLMWithValueHead) Accelerator)DeepSpeedEngine)Module)DistributedDataParallelc @ sf e Zd ZU dZdZeed< dZeed< dZeed< e dd Z
e d d
Z e d d Z e d
d Z
dS )ChatMlSpecialTokensziDataclass for special tokens used in ChatML, including system, user, assistant, bos, eos, and pad tokens.z <|im_start|> bos_tokenz
<|im_end|> eos_token pad_tokenc C | j dS )Nsystemr self r D/home/ubuntu/.local/lib/python3.10/site-packages/trl/models/utils.pyr 1 zChatMlSpecialTokens.systemc C r )Nuserr r r r r r" 5 r! zChatMlSpecialTokens.userc C r )N assistantr r r r r r# 9 r! z ChatMlSpecialTokens.assistantc C s d| j d| j d| j dS )Nz {% for message in messages %}{{'z2' + message['role'] + '
' + message['content'] + 'z7' + '
'}}{% endfor %}{% if add_generation_prompt %}{{ 'z
' }}{% endif %})r r r# r r r r
chat_template= s z!ChatMlSpecialTokens.chat_templateN)__name__
__module__ __qualname____doc__r str__annotations__r r propertyr r" r# r$ r r r r r ) s
r chatmlmodel tokenizerformatresize_to_multiple_ofreturnc C s |j dur td|tvrtd| dt t| }|j|_|j|_|j|_|d|j|jgi |j |_ | jt |j
|durE|ndd t | dddur`|j | j
_ |j| j
_|j| j
_t | dddurw|j| j_|j| j_|j | j_ | |fS ) a
Setup chat format by adding special tokens to the tokenizer, setting the correct format, and extending the
embedding layer of the model based on the new special tokens.
We recommend using [`clone_chat_template`] instead of this function.
If the model already has a chat template, this will throw an error. If you want to overwrite it, please set
`tokenizer.chat_template` to `None`.
Args:
model (`~transformers.PreTrainedModel`): The model to be modified.
tokenizer (`~transformers.PreTrainedTokenizer`): The tokenizer to be modified.
format (`Optional[Literal["chatml"]]`): The format to be set. Defaults to "chatml".
resize_to_multiple_of (`int` or `None`): Number to resize the embedding layer to. Defaults to None.
Returns:
model (`~transformers.PreTrainedModel`):
The modified model.
tokenizer (`~transformers.PreTrainedTokenizer`):
The modified tokenizer.
NzcChat template is already added to the tokenizer. If you want to overwrite it, please set it to NonezFormat z" not available. Please use one of additional_special_tokensnew_num_tokenspad_to_multiple_ofconfiggeneration_config)r$
ValueErrorFORMAT_MAPPINGkeysr r r add_special_tokensresize_token_embeddingslenvocabgetattr pad_token_idr6 bos_token_id eos_token_idr7 )r- r. r/ r0 chat_formatr r r setup_chat_formatL s2
rD @ source_tokenizer_pathc s t |}| _ fdd|j D } | |j _ j| j _ j| j
_| j t j
|dur5|ndd d}| jt j
kretd| d} |}|d7 }|dkr]|| | jt j
ksCt j
| jkr|td t j
d
| j d d d |D } |}| |fS )
a
Clones a chat template from a source tokenizer to the target tokenizer and updates the model accordingly.
This function:
- Copies the chat template from a source tokenizer to the target tokenizer.
- Adds any new tokens from the source tokenizer to the target tokenizer.
- Sets and synchronizes the EOS token across the tokenizer and model.
- Resizes the model's token embeddings to match the new vocabulary size, optionally rounding it up to a multiple of
a specified value. In such cases, dummy tokens are added to the tokenizer to ensure the vocabulary size matches
the embedding dimensions.
Args:
model (`PreTrainedModel`):
Model to update.
tokenizer (`PreTrainedTokenizer`):
Tokenizer to update.
source_tokenizer_path (`str`):
Path or identifier of the pretrained tokenizer to clone from.
resize_to_multiple_of (`int` or `None`, *optional*, defaults to `64`):
The embedding layer will be resized to the new vocabulary size. If this is not `None`, it will round up the
new vocabulary size to the nearest multiple of this value.
Returns:
model (`PreTrainedModel`):
Updated model with resized token embeddings and EOS token configured.
tokenizer (`~transformers.PreTrainedTokenizer`):
Updated tokenizer with the chat template and special tokens applied.
added_tokens (`list[int]`):
List of tokens that were added to the tokenizer from the source tokenizer.
Example:
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
from trl import clone_chat_template
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.2-1B")
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.2-1B")
model, tokenizer, added_tokens = clone_chat_template(model, tokenizer, "Qwen/Qwen3-0.6B")
```
c s g | ]
}|j jvr|qS r )contentr> .0tokenr. r r
s z'clone_chat_template..Nr3 r z
r zAVocabulary size mismatch after resizing: tokenizer vocab size is z , but model embedding size is zB. This indicates an internal error in the token alignment process.c S s g | ]}|j qS r )rG rH r r r rL s )r from_pretrainedget_chat_templater$ added_tokens_decodervalues
add_tokensr rB r6 r7 r< r= r>
vocab_sizer append RuntimeErrorconvert_tokens_to_ids) r- r. rF r0 tokenizer_source added_tokensidx dummy_tokenis_addedr rK r clone_chat_template s<
/
r\ r c C s t | dsdS | jdurt | jdr| jj}n
| jdur | j}ntdt|jddD ]}|j q+|jD ]}| q6|j
D ]}| q@g |_g |_
dS )z:Removes the optimizer hooks from a DeepSpeed ZeRO-3 model. optimizerNparameter_offload8The model optimizer is None, which is not yet supported.Trecurse) hasattrr] r^ rU iter_paramsmoduleds_active_sub_modulesclear
forward_hooksremovebackward_hooks)r- optimizer_offloadparamhookr r r remove_hooks s
rm Fc C s t | j|d| S )Nr` ) itertoolschainnamed_parametersds_external_parameters)
sub_modulera r r r get_all_parameters s rs c C s dd t | |D S )Nc S s g | ]\}}|qS r r )rI _rk r r r rL s z iter_params..)rs )rd ra r r r rc s rc c C s ddl }t| ds dS | jdurt| jdr| jj}n
| jdur$| j}ntdt|jtdkr;||j dS |
|j dS )z7Adds the optimizer hooks from a DeepSpeed ZeRO-3 model.r Nr] r^ r_ z0.16.4) deepspeedrb r] r^ rU r
parse __version___register_deepspeed_modulerd _register_hooks_recursively)r- ru rj r r r add_hooks s
rz T)r r acceleratorr gather_deepspeed3_paramsc c s | | }|jjdurJ|jjjdkrJ|s | | V dS ddl}|j| t| | | V t | W d dS 1 sCw Y dS |V dS )a
Context manager to unwrap distributed or accelerated models for generation tasks.
Args:
model (`Union[DistributedDataParallel, DeepSpeedEngine]`):
Model to be unwrapped.
accelerator (`~accelerate.Accelerator`):
Accelerator instance managing the model.
gather_deepspeed3_params (`bool`, *optional*, defaults to `True`):
Whether to gather weights for DeepSpeed ZeRO Stage 3 models. If `False`, skips parameter gathering, which
can be more memory-efficient but may lead to slower generation times.
Yields:
Unwrapped model.
Example:
```python
with unwrap_model_for_generation(model, accelerator) as unwrapped_model:
generated_outputs = unwrapped_model.generate(input_ids)
```
N r )
unwrap_modelstatedeepspeed_plugin
zero_stageru zeroGatheredParameters
parametersrm rz )r- r{ r| unwrapped_modelru r r r unwrap_model_for_generation" s
"
r r c C s ddl }|jj}t|j}|d d }| durDt| jddr$t| jjnt| jdd}|durD|dkrD| || d| d | | d
|dkrNd|d d< |j
| |d ^} }| | S ) a Prepares the model for DeepSpeed inference or evaluation by initializing it with the appropriate configuration.
Adapted from accelerate:
https://github.com/huggingface/accelerate/blob/739b135f8367becb67ffaada12fe76e3aa60fefd/src/accelerate/accelerator.py#L1473
r Nzero_optimizationstage hidden_sizes hidden_sizer}
g?)z$zero_optimization.reduce_bucket_sizez4zero_optimization.stage3_param_persistence_thresholdz-zero_optimization.stage3_prefetch_bucket_size)r- r6 ) ru r r r deepspeed_configr? r6 maxr update
initializeeval)r- r{ ru r
config_kwargsr r rt r r r prepare_deepspeedL s*
r c
C s ddl m} ddlm} t| |sGt| |sG|jj| |jj}|jp&|j |j
|j |j |j
|j|j|j|j|j|j|jd }|| fi |} | | S )Nr )
FSDPModule)FullyShardedDataParallel) sharding_strategy cpu_offloadauto_wrap_policymixed_precisionsync_module_statesbackward_prefetchforward_prefetchuse_orig_params
param_init_fnignored_moduleslimit_all_gathers device_id)torch.distributed.fsdpr 2torch.distributed.fsdp.fully_sharded_data_parallelr
isinstancer fsdp_pluginset_auto_wrap_policyr reshard_after_forwardr r mixed_precision_policyr r r r r r r devicer )r- r{ r FSDPr kwargsr r r prepare_fsdpt s*
r c
@ sf e Zd ZdZdejdejdededef
ddZdejdejd d
fd d Z dejdejd d
fd
dZ
d
S )_ForwardRedirectionaI Implements the `forward-redirection`.
Taken from Pytorch-lightning:
https://github.com/Lightning-AI/pytorch-lightning/blob/02311d03fb982560246eead7c08104481fac9579/src/lightning/pytorch/strategies/strategy.py#L602
A method call to a wrapped module gets rerouted through the wrapper's `forward` method instead.
wrapper_moduleoriginal_modulemethodargsr c sL j dtdtdtf fdd }|_ |i |} |S )a Reroutes a method call through the `wrapper_module`'s `forward` method.
Args:
wrapper_module: The module that has `original_module` wrapped.
original_module: The module that was wrapped inside `wrapper_module`.
method_name: The name of the method that should be called on the `original_module` after inputs get
redirected through the `wrapper_module`'s `forward` method.
*args: The positional arguments to the method `method_name`. They will get passed to a patched
`forward` method instead.
**kwargs: The keyword arguments to the method `method_name`. They will get passed to a patched
`forward` method instead.
_args_kwargsr1 c s$ _ | i |} |S N)forwardon_after_inner_forward)r r outr original_forwardr r r r r wrapped_forward s z5_ForwardRedirection.__call__..wrapped_forward)r r on_after_outer_forward)r r r r r r r wrapper_outputr r r __call__ s "
z _ForwardRedirection.__call__r1 Nc C d S r r r r r r r r r z*_ForwardRedirection.on_after_inner_forwardc C r r r r r r r r r z*_ForwardRedirection.on_after_outer_forward) r% r& r' r( nnr callabler r r r r r r r r s
" r )r, N)rE )r- r r1 N)F)T)r- r r{ r )2rn
contextlibr copyr dataclassesr typingr r r r r torch.nnr packagingr
transformersr r r
r modeling_value_headr r SUPPORTED_ARCHITECTURES
accelerater deepspeed.runtime.enginer r torch.nn.parallel.distributedr r r9 inttuplerD r) listr\ rm rs rc rz boolr r r r r r r r sz
K
]
)(