o }oiH@sNddlmZmZddlmZddlmZddlm Z dgZ GdddeZdS))ListOptional) AutoTokenizer) TokenizerSpec)loggingrc @seZdZdZdddddddddgddddfdedeedeedeed eed eed eed eed eedeedeedeedeededeefddZ     dLdedeedeedeedeedeef ddZ e ddZ de de fddZe ddZd d!Zd"d#Zd$d%Zd&d'Zd(d)Zd*d+Zd,d-ZdMd.d/Ze d0d1Ze d2d3Ze d4d5Ze d6d7Ze d8d9Ze d:d;Ze dd?Ze d@dAZ e dBdCZ!e dDdEZ"dNdFedGefdHdIZ#dFefdJdKZ$dS)Orz Wrapper of HuggingFace AutoTokenizer https://huggingface.co/docs/transformers/main/en/model_doc/auto#transformers.AutoTokenizer. NTFpretrained_model_name vocab_file merges_file mask_token bos_token eos_token pad_token sep_token cls_token unk_tokenadditional_special_tokensuse_fasttrust_remote_codeinclude_special_tokens chat_templatec Cslz||||| | ||jsJdWn4tyGz||||| | ||js-JdWntyD}z td|d|d}~wwYnw||_t|j|_i}| dur[| |d<|durc||d<|durk||d<|durt||d<n|jjdur|jjr|jj|d<|dur||d <n|jjdur|jjr|jj|d <|dur||d <n|jj dur|jj r|jj |d <| dur| |d <n|jj dur|jj r|jj |d <| dur| |d <g}|||||| | fD]}|dur||j vr| |q| D]}|dur ||j vr | |qt|d kr t |d|||d|d|dk|_i|_dS)a. Args: pretrained_model_name: corresponds to HuggingFace-AutoTokenizer's 'pretrained_model_name_or_path' input argument. For more details please refer to the documentation of the `from_pretrained` method here: https://huggingface.co/docs/transformers/main/en/model_doc/auto#transformers.AutoTokenizer. The list of all supported models can be found here: https://huggingface.co/models vocab_file: path to file with vocabulary which consists of characters separated by newlines. mask_token: mask token bos_token: the beginning of sequence token eos_token: the end of sequence token. Usually equal to sep_token pad_token: token to use for padding sep_token: token used for separating sequences cls_token: class token. Usually equal to bos_token unk_token: token to use for unknown tokens additional_special_tokens: list of other tokens beside standard special tokens (bos, eos, pad, etc.). For example, sentinel tokens for T5 (, , etc.) use_fast: whether to use fast HuggingFace tokenizer include_special_tokens: when True, converting text to ids will include special tokens / prompt tokens (if any), yielding self.tokenizer(text).input_ids chat_template: The chat template string to format "messages" with against the underlying HF tokneizer with apply_chat_template function ztokenizer not initializedz4Unable to instantiate HuggingFace AUTOTOKENIZER for z . Exception: Nrr r rr r rrrzt will be added to the vocabulary. Please resize your model accordingly, see NLP_Tokenizers.ipynb for more details.zx yxy)_initialize_tokenizer tokenizer Exception ValueErrorrlenoriginal_vocab_sizerr r r get_vocabappendrwarningadd_special_tokenstext_to_tokensspace_sensitive_inv_vocab_dict)selfrrr r r r r rrrrrrrrespecial_tokens_dictnew_tokens_in_vocabtokenr*q/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/common/tokenizers/huggingface/auto_tokenizer.py__init__"s~)                zAutoTokenizer.__init__cCs|durtj|||d|_n|durtj||||d|_n tj|||||d|_|durDt|jdddur:td||j_d|j_dSdS)N)pretrained_model_name_or_pathrr)r-rrr)r-rr rrrzHYou are overwriting tokenizer's chat template, confirm this is intended.jinja) AUTOTOKENIZERfrom_pretrainedrgetattrrinforchat_template_format)r%rrr rrrr*r*r+rs4     z#AutoTokenizer._initialize_tokenizercCs t|jS)z Returns the size of the tokenizer's vocabulary. Returns: int: The number of tokens in the vocabulary. )rrr%r*r*r+ vocab_sizes zAutoTokenizer.vocab_sizer'returncCsL|j|}|dkrt|d|jjD] }t||t|j|dq|S)a Adds a dictionary of special tokens (eos, pad, cls...). If special tokens are NOT in the vocabulary, they are added to it (indexed starting from the last index of the current vocabulary). Args: special_tokens_dict: dict of string. Keys should be in the list of predefined special attributes: [``bos_token``, ``eos_token``, ``unk_token``, ``sep_token``, ``pad_token``, ``cls_token``, ``mask_token``, ``additional_special_tokens``]. Tokens are only added if they are not already in the vocabulary. Returns: Number of tokens added to the vocabulary. rz5 special tokens added, resize your model accordingly.N)rr!rr2SPECIAL_TOKENS_ATTRIBUTESsetattrr1)r%r'num_tokens_addedkr*r*r+r!s  z AutoTokenizer.add_special_tokenscsfddjDS)z Returns a list of the additional special tokens' IDs (excluding bos, eos, pad, unk). Returns: List[int]: List of token IDs for additional special tokens, such as sentinel tokens for T5. csg|]}|qSr*) token_to_id).0r)r4r*r+ z?AutoTokenizer.additional_special_tokens_ids..)rr4r*r4r+additional_special_tokens_idssz+AutoTokenizer.additional_special_tokens_idscC|j|}|S)z Converts text into a list of tokens. Args: text (str): Input text to be tokenized. Returns: List[str]: List of tokens. )rtokenize)r%texttokensr*r*r+r" zAutoTokenizer.text_to_tokenscCr@)z Converts a list of tokens back into text. Args: tokens (List[str]): List of tokens to be converted. Returns: str: The reconstructed text. )rconvert_tokens_to_string)r%rCrBr*r*r+tokens_to_textrDzAutoTokenizer.tokens_to_textcCs||gdS)z Converts a single token to its corresponding ID. Args: token (str): The token to convert. Returns: int: The ID corresponding to the token. r) tokens_to_ids)r%r)r*r*r+r;s zAutoTokenizer.token_to_idcCr@)z Converts a list of tokens to their corresponding IDs. Args: tokens (List[str]): List of tokens to convert. Returns: List[int]: List of token IDs. )rconvert_tokens_to_ids)r%rCidsr*r*r+rGrDzAutoTokenizer.tokens_to_idscCr@)z Converts a list of token IDs back to tokens. Args: ids (List[int]): List of token IDs to convert. Returns: List[str]: List of tokens. )rconvert_ids_to_tokens)r%rIrCr*r*r+ ids_to_tokens,rDzAutoTokenizer.ids_to_tokenscCs*|jr ||jS||}||}|S)a+ Converts text directly to token IDs. Args: text (str): Input text to be converted to IDs. Returns: List[int]: List of token IDs. If include_special_tokens is True, will include special tokens from the tokenizer's configuration. )rr input_idsr"rG)r%rBrCrIr*r*r+ text_to_ids9s    zAutoTokenizer.text_to_idscOs|jj|i|S)z*Appies chat template and tokenizes results)rapply_chat_template)r%argskwargsr*r*r+rNJsz!AutoTokenizer.apply_chat_templatecs4|}|rfdd|D}n|}|}|S)aF Converts token IDs back to text. Args: ids (List[int]): List of token IDs to convert to text. remove_special_tokens (bool): Whether to remove special tokens (like [PAD], [CLS], etc.) from the output text. Returns: str: The reconstructed text. csg|] }|jjvr|qSr*)rall_special_tokens)r<tr4r*r+r=\sz-AutoTokenizer.ids_to_text..)rKrF)r%rIremove_special_tokensrC tokens_cleanrBr*r4r+ ids_to_textNs  zAutoTokenizer.ids_to_textcs0dd|jjDfddttDS)z Returns the vocabulary as a list where the index corresponds to the token ID. Returns: List[str]: List of tokens in the vocabulary. cSi|]\}}||qSr*r*r<r:vr*r*r+ jr>z'AutoTokenizer.vocab..csg|]}|qSr*r*)r<iid2vocabr*r+r=ksz'AutoTokenizer.vocab..)rvocabitemsrangerr4r*r[r+r]bszAutoTokenizer.vocabcCs(|jikrdd|jjD|_|jS)z Returns the inverse vocabulary mapping (token to ID). Returns: Dict[str, int]: Dictionary mapping tokens to their IDs. cSrVr*r*rWr*r*r+rYvr>z+AutoTokenizer.inv_vocab..)r$rr]r^r4r*r*r+ inv_vocabms zAutoTokenizer.inv_vocabcC(t|ddur dS|t|dgdS)z Gets the ID of the padding token. Returns: int or None: The ID of the padding token if it exists, None otherwise. r Nrr1rGr4r*r*r+pad_idyzAutoTokenizer.pad_idcCra)z Gets the ID of the beginning-of-sequence token. Returns: int or None: The ID of the BOS token if it exists, None otherwise. r Nrrbr4r*r*r+bos_idrdzAutoTokenizer.bos_idcCra)z Gets the ID of the end-of-sequence token. Returns: int or None: The ID of the EOS token if it exists, None otherwise. r Nrrbr4r*r*r+eos_idrdzAutoTokenizer.eos_idcCs|t|dgdS)z Gets the ID of the end-of-document token (same as EOS token). Required for megatron-core compatibility. Returns: int: The ID of the EOD/EOS token. r r)rGr1r4r*r*r+eodszAutoTokenizer.eodcCra)z Gets the ID of the separator token. Returns: int or None: The ID of the separator token if it exists, None otherwise. rNrrbr4r*r*r+sep_idrdzAutoTokenizer.sep_idcCra)z Gets the ID of the classifier token. Returns: int or None: The ID of the classifier token if it exists, None otherwise. rNrrbr4r*r*r+cls_idrdzAutoTokenizer.cls_idcCra)z Gets the ID of the unknown token. Returns: int or None: The ID of the unknown token if it exists, None otherwise. rNrrbr4r*r*r+unk_idrdzAutoTokenizer.unk_idcCra)z Gets the ID of the mask token. Returns: int or None: The ID of the mask token if it exists, None otherwise. r Nrrbr4r*r*r+mask_idrdzAutoTokenizer.mask_idcCs t|jjS)z Returns the name of the underlying HuggingFace tokenizer class. Returns: str: Name of the tokenizer class. )typer__name__r4r*r*r+names zAutoTokenizer.namesave_directoryfilename_prefixcCs|jj||dS)KSaves tokenizer's vocabulary and other artifacts to the specified directory)rorp)rsave_vocabulary)r%rorpr*r*r+rrszAutoTokenizer.save_vocabularycCs |j|S)rq)rsave_pretrained)r%ror*r*r+rss zAutoTokenizer.save_pretrained)NNFFN)T)N)%rm __module__ __qualname____doc__strrrboolr,rpropertyr5dictintr!r?r"rFr;rGrKrMrNrUr]r`rcrerfrgrhrirjrkrnrrrsr*r*r*r+rs         &                      N) typingrr transformersrr/1nemo.collections.common.tokenizers.tokenizer_specr nemo.utilsr__all__r*r*r*r+s