o ̳i~3@sddlZddlmZmZmZmZmZmZddlm Z m Z m Z ddl m Z ddlmZmZdZddd d d d d ddddddd ZdZddeeeeDZieeZGdddee ZdS)N)AnyDictListMappingOptionalTuple)MessagePromptTemplatetruncate) Transform)ModelTokenizerTikTokenBaseTokenizerzs(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\r\n\p{L}\p{N}]?\p{L}+|\p{N}{1,3}| ?[^\s\p{L}\p{N}]+[\r\n]*|\s*[\r\n]+|\s+(?!\S)|\s+iiiiiiiiii i ii ) <|begin_of_text|><|end_of_text|>z<|reserved_special_token_0|>z<|reserved_special_token_1|><|finetune_right_pad_id|> <|step_id|><|start_header_id|><|end_header_id|> <|eom_id|> <|eot_id|><|python_tag|> <|image|>z <|video|>cCs"i|] }dd|dd|qS)z<|reserved_special_token_z|>i ).0irrV/home/ubuntu/.local/lib/python3.10/site-packages/torchtune/models/llama3/_tokenizer.py &src @seZdZdZ   d2dedeeeefdeedeefddZ d d Z d ed efd dZ e d efddZ e d efddZ  d3d ededed eefddZ  d3deededed efddZded eefddZded eefd d!Zded eefd"d#Zddd$ded%ed&ed eefd'd(Zdd)d*eed&ed eeeeeffd+d,Z -d4d.eeefd/ed eeeffd0d1ZdS)5Llama3Tokenizera& tiktoken tokenizer configured with Llama3 Instruct's special tokens, as described in https://llama.meta.com/docs/model-cards-and-prompt-formats/meta-llama-3 Args: path (str): Path to pretrained tiktoken tokenizer file. special_tokens (Optional[Dict[str, int]]): mapping containing special text tokens and their registered token IDs. If left as None, this will be set to the canonical Llama3 special tokens. max_seq_len (Optional[int]): maximum sequence length for tokenizing a single list of messages, after which the input will be truncated. Default is None. prompt_template (Optional[PromptTemplate]): template used to format the messages based on their role. This is used to add structured text around the actual messages. The structured text is used in three scenarios: - Task-specific templates to gear models for a particular task that it will expect after training - Model-specific templates that are required whenever the model is prompted, such as the [INST] tags in Llama2 and in Mistral - Community standardized templates, such as :class:`~torchtune.data.ChatMLTemplate` The extra text will still get tokenized as normal text, not as special tokens. Default is None. Examples: >>> tokenizer = Llama3Tokenizer("/path/to/tt_model") >>> tokenized_text = tokenizer.encode("Hello world!", add_bos=True, add_eos=True) >>> print(tokenized_text) [1, 31587, 29644, 102, 2] Npathspecial_tokens max_seq_lenprompt_templatecCs|dur|nt|_||jd|_|jd|_|jd|_|jd|_|jd|_|jd|_|jd|_ |jd|_ |jd |_ |jd |_ |j|j |j g|_ t|d t|j|j|jd |_||_||_td |_td|_dS)Nrrrrrrrrrrllama3_tiktoken)r namepatternbos_ideos_idr!z <\|.*?\|>z/<\|start_header_id\|>.*?<\|end_header_id\|>\n\n)LLAMA3_SPECIAL_TOKENSr!_validate_special_tokensr'r(pad_idstep_idstart_header_id end_header_ideot_ideom_id python_tagimage_id stop_tokensr CL100K_PATTERNtt_modelr"r#recompile_special_token_regex_special_token_header_regex)selfr r!r"r#rrr__init__Ks8            zLlama3Tokenizer.__init__cCs&dD]}||jvrt|dqdS)zV Validate that required special tokens are passed into the tokenizer. )rrrrrrrz missing from special_tokensN)r! ValueError)r:tokenrrrr*~s  z(Llama3Tokenizer._validate_special_tokenstextreturncCs|jd|jd|S)z@ Remove special tokens from the decoded string. )r8subr9)r:r>rrr_remove_special_tokenssz&Llama3Tokenizer._remove_special_tokenscC|jjSN)r5base_vocab_sizer:rrrrEzLlama3Tokenizer.base_vocab_sizecCrCrD)r5 vocab_sizerFrrrrHrGzLlama3Tokenizer.vocab_sizeTadd_bosadd_eoscCs|jj|||dS)N)r>rIrJ)r5encode)r:r>rIrJrrrrKszLlama3Tokenizer.encode token_idstruncate_at_eosskip_special_tokenscCs"|jj||d}|r||S|S)a Decode a list of token ids into a string. Args: token_ids (List[int]): The list of token ids. truncate_at_eos (bool): Whether to truncate the string at the end of sequence token. Default is True. skip_special_tokens (bool): Whether to show or skip special tokens in the decoded string. Default is True. Returns: str: The decoded string. )rLrM)r5decoderB)r:rLrMrNdecoded_stringrrrrOs zLlama3Tokenizer.decodemessagecCs6|jg|j|jddd|jg|jddddS)zT Tokenize header start, message role, and header end as list of ids FrIrJz )r-rKrolestripr.r:rQrrr_tokenize_headersz Llama3Tokenizer._tokenize_headercCs|jr|jgS|jgS)z> Add eot or eom id at the end of the message. )eotr/r0rUrrr _tokenize_endszLlama3Tokenizer._tokenize_endcCszg}|jD],}|ddkr||j|dddd7}q|ddkr)||jg7}qtd|d|jr;|jg|}|S)z9 Tokenize message content as list of ids typer>contentFrRimagez"Unsupported message content type: )rZrKrTr2 RuntimeErroripythonr1)r:rQtokenized_bodyitemrrr_tokenize_bodys    zLlama3Tokenizer._tokenize_body)add_start_tokensadd_end_tokensrarbcCs>|r||ng}||}|r||ng}|||}|S)a Tokenize a message into a list of token ids. Args: message (Message): The message to tokenize. add_start_tokens (bool): Whether to prepend a tokenized header to the message. Default is True. add_end_tokens (bool): Whether to append eot or eom id at the end of the message. Default is True. Returns: List[int]: The list of token ids. )rVr`rX)r:rQrarbtokenized_headerr^ tokenized_endtokenized_messagerrrtokenize_messages   z Llama3Tokenizer.tokenize_messagerbmessagesc Cs|jdur ||n|}|jg}dg}t|}t|D]/\}}||dkr'|nd} |j|| d} || }||jgt| }|jrJt||jkrJnq|rX||jg}|dg}|jrrt||j|rd|jnd}t||j|rodnd}||fS)a Tokenize a list of messages into a list of token ids and masks. Args: messages (List[Message]): The list of messages to tokenize. add_end_tokens (bool): Whether to append end tokens ids (end-of-seq, end-of-turn, end-of-message) at the end of the last assistant message. This value should be set to False for generation. Default is True. Examples: >>> # Tokenize a list of messages with default settings >>> messages = [ ... Message(role="user", content="Hello world!", masked=True), ... Message(role="assistant", content="How are you?", masked=False), ... ] >>> tokenizer = Llama3Tokenizer("/path/to/tt_model") >>> tokenizer.tokenize_messages(messages) ([1, 31587, 29644, 102, 1, 31587, 29644, 102, 2], [True, True, True, True, True, False, False, False, True]) >>> # Tokenize a list of messages with add_end_tokens set to False >>> tokenizer.tokenize_messages(messages, add_end_tokens=False) ([1, 31587, 29644, 102, 1, 31587, 29644], [True, True, True, True, True, False, False]) Returns: Tuple[List[int], List[bool]]: The list of token ids and the list of masks. NTrg) r#r'len enumeraterfmaskedr"r(r ) r:rhrbtemplated_messagestokensmask num_messagesrrQadd_end_tokens_to_messagererrrtokenize_messagess6 !   z!Llama3Tokenizer.tokenize_messagesFsample inferencecCs2|d}|j|| d\}}||d<||d<|S)a Apply ``tokenize_messages`` to the "messages" field in the sample. Args: sample (Mapping[str, Any]): A sample with a "messages" field containing a List[Message] to tokenize inference (bool): Whether the template is being used for inference or not. Returns: Mapping[str, Any]: The sample with added "tokens" and "mask" fields and the "messages" field removed. rhrgrnro)poprr)r:rsrtrhrnrorrr__call__Ls zLlama3Tokenizer.__call__)NNN)TT)F)__name__ __module__ __qualname____doc__strrrintr r;r*rBpropertyrErHboolrrKrOrrVrXr`rfrrrrrrvrrrrr.s 3    !   E  r)r6typingrrrrrrtorchtune.datarr r torchtune.modules.transformsr 'torchtune.modules.transforms.tokenizersr r r4SPECIAL_TOKENSNUM_RESERVED_SPECIAL_TOKENSrangerjRESERVED_TOKENSr)rrrrrs4