o
̳i @ sv d dl mZmZmZmZmZ d dlmZmZ d dl m
Z
d dl m Z d dl
mZmZmZ g dZG dd dee Zd S )
)AnyListMappingOptionalTuple)MessagePromptTemplate)Llama2ChatTemplate) Transform)ModelTokenizerSentencePieceBaseTokenizer#tokenize_messages_no_special_tokens)
c @ s e Zd ZdZde fdedee dee fddZ e
dd Z e
d
d Z e
d d
Z
e
dd Z d&dededededee f
ddZdee defddZddd d ee d ed edeee ee f fd d!Z d'd"eeef d#edeeef fd$d%ZdS )(Llama2Tokenizera
Llama2's implementation of the SentencePiece tokenizer. Llama2Tokenizer does
not include any additional special tokens. The prompt template described in
https://llama.meta.com/docs/model-cards-and-prompt-formats/meta-llama-2/ describes
[INST][/INST] and <><> as special tokens but these are not registered
as unique ids and are tokenized as normal text. When using this tokenizer on the
pre-trained model for inference, the prompt template
:class:`~torchtune.models.llama2.Llama2ChatTemplate` is by default applied to your data
before tokenization to add the [INST] and <> tags for optimal performance.
For more details, see https://pytorch.org/torchtune/main/tutorials/chat.html#tokenizing-prompt-templates-special-tokens.
Args:
path (str): Path to pretrained SentencePiece tokenizer file.
max_seq_len (Optional[int]): A max sequence length to truncate tokens to.
Default: None
prompt_template (Optional[PromptTemplate]): template used to format the messages based on their role. This is used
to add structured text around the actual messages. The structured text is used in three scenarios:
- Task-specific templates to gear models for a particular task that it will expect after training
- Model-specific templates that are required whenever the model is prompted, such as the [INST]
tags in Llama2 and in Mistral
- Community standardized templates, such as :class:`~torchtune.data.ChatMLTemplate`
The extra text will still get tokenized as normal text, not as special tokens.
Default is :class:`~torchtune.models.llama2.Llama2ChatTemplate`.
Examples:
>>> tokenizer = Llama2Tokenizer("/path/to/spm_model")
>>> tokenized_text = tokenizer.encode("Hello world!", add_bos=True, add_eos=True)
>>> print(tokenized_text)
[1, 31587, 29644, 102, 2]
Npath max_seq_lenprompt_templatec C s, t || _d| j_| jg| _|| _|| _d S )Nr )r
_spm_modelpad_ideos_id stop_tokensr r )selfr r r r V/home/ubuntu/.local/lib/python3.10/site-packages/torchtune/models/llama2/_tokenizer.py__init__7 s
zLlama2Tokenizer.__init__c C | j jS N)r r r r r r r I zLlama2Tokenizer.eos_idc C r r )r bos_idr! r r r r# M r" zLlama2Tokenizer.bos_idc C r r )r r r! r r r r Q r" zLlama2Tokenizer.pad_idc C r r )r
vocab_sizer! r r r r$ U r" zLlama2Tokenizer.vocab_sizeTFtextadd_bosadd_eostrim_leading_whitespacereturnc C s | j j||||dS )N)r& r' r( )r encode)r r% r& r' r( r r r r* Y s zLlama2Tokenizer.encode token_idsc C s | j |S r )r decode)r r+ r r r r, g s zLlama2Tokenizer.decode)add_start_tokensadd_end_tokensmessagesr- r. c C s@ | j dur
| |n|}t| ||r| jnd|r | jdS ddS )a Tokenize a list of messages one at a time then concatenate them,
returning a list of tokens and a list of masks.
Note:
sentencepiece has problems where in general
encode(s1 + s2) != encode(s1) + encode(s2) due to whitespace handling.
We can get around this by prepending s2 with a known token and slicing the
beginning off the tokenized s2.
Example:
>>> tokenizer = Llama2Tokenizer(tokenizer_path, max_seq_len)
>>> messages = [
Message(role="system", content="system message\n", masked=True),
Message(role="user", content="user prompt\n", masked=True),
Message(role="assistant", content="assistant response\n"),
]
>>> # tokenize_messages encodes messages separately and concats
>>> tokenizer.tokenize_messages(messages)[0]
[1, 1788, 2643, 13, 1792, 9508, 13, 465, 22137, 2933, 2]
>>> # Same result as encoding the full string in one go
>>> tokenizer.encode(''.join([message.content for message in messages]))
[1, 1788, 2643, 13, 1792, 9508, 13, 465, 22137, 2933, 2]
Args:
messages (List[Message]): A list of messages, each containing role, content,
and masked attributes.
add_start_tokens (bool): Whether to add BOS token to the beginning of the first message.
Default True.
add_end_tokens (bool): Whether to add EOS token to the end of the last message. Default True.
Returns:
Tuple[List[int], List[bool]]: The tokenized messages
N) tokenizerr/ r# r )r r
r# r )r r/ r- r. templated_messagesr r r tokenize_messagesm s
-
z!Llama2Tokenizer.tokenize_messagessample inferencec C s2 | d}| j|| d\}}||d< ||d< |S )a
Apply ``tokenize_messages`` to the "messages" field in the sample.
Args:
sample (Mapping[str, Any]): A sample with a "messages" field containing
a List[Message] to tokenize
inference (bool): Whether the template is being used for inference or not.
Returns:
Mapping[str, Any]: The sample with added "tokens" and "mask" fields
and the "messages" field removed.
r/ )r. tokensmask)popr2 )r r3 r4 r/ r5 r6 r r r __call__ s
zLlama2Tokenizer.__call__)TTF)F)__name__
__module__ __qualname____doc__r strr intr r propertyr r# r r$ boolr r* r, r r r2 r r r8 r r r r r sr $
8
r N)typingr r r r r torchtune.datar r (torchtune.models.llama2._prompt_templater torchtune.modules.transformsr
'torchtune.modules.transforms.tokenizersr r r
WHITESPACE_CHARSr r r r r s