o iZ@sxdZddlmZddlmZddlmZddlmZddl m Z e e Z d d d d Zd diZGdddeZdgZdS)zTokenization classes for Qwen2.)Optional) AddedToken)PreTrainedTokenizerFast)logging)Qwen2Tokenizerz vocab.jsonz merges.txtztokenizer.json) vocab_file merges_filetokenizer_filezqwen/qwen-tokenizerics`eZdZdZeZddgZeZ       d fdd Z dde d e e d e e fd d Z ZS)Qwen2TokenizerFastac Construct a "fast" Qwen2 tokenizer (backed by HuggingFace's *tokenizers* library). Based on byte-level Byte-Pair-Encoding. Same with GPT2Tokenizer, this tokenizer has been trained to treat spaces like parts of the tokens so a word will be encoded differently whether it is at the beginning of the sentence (without space) or not: ```python >>> from transformers import Qwen2TokenizerFast >>> tokenizer = Qwen2TokenizerFast.from_pretrained("Qwen/Qwen-tokenizer") >>> tokenizer("Hello world")["input_ids"] [9707, 1879] >>> tokenizer(" Hello world")["input_ids"] [21927, 1879] ``` This is expected. This tokenizer inherits from [`PreTrainedTokenizerFast`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: vocab_file (`str`, *optional*): Path to the vocabulary file. merges_file (`str`, *optional*): Path to the merges file. tokenizer_file (`str`, *optional*): Path to [tokenizers](https://github.com/huggingface/tokenizers) file (generally has a .json extension) that contains everything needed to load the tokenizer. unk_token (`str`, *optional*, defaults to `"<|endoftext|>"`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. Not applicable to this tokenizer. bos_token (`str`, *optional*): The beginning of sequence token. Not applicable for this tokenizer. eos_token (`str`, *optional*, defaults to `"<|endoftext|>"`): The end of sequence token. pad_token (`str`, *optional*, defaults to `"<|endoftext|>"`): The token used for padding, for example when batching sequences of different lengths. input_idsattention_maskN <|endoftext|>c st|trt|dddddn|}t|trt|dddddn|}t|tr.t|dddddn|}t|tr>t|dddddn|}tjd|||||||d|dS)NFT)lstriprstripspecial normalized)r r r unk_token bos_token eos_token pad_token) isinstancestrrsuper__init__) selfr r r rrrrkwargs __class__re/home/ubuntu/.local/lib/python3.10/site-packages/transformers/models/qwen2/tokenization_qwen2_fast.pyrSs6 zQwen2TokenizerFast.__init__save_directoryfilename_prefixreturncCs|jjj||d}t|S)N)name) _tokenizermodelsavetuple)rr"r#filesrrr!save_vocabularysz"Qwen2TokenizerFast.save_vocabulary)NNNrNrr)N)__name__ __module__ __qualname____doc__VOCAB_FILES_NAMESvocab_files_namesmodel_input_namesrslow_tokenizer_classrrrr)r+ __classcell__rrrr!r %s)(1r N)r/typingrtokenization_utilsrtokenization_utils_fastrutilsrtokenization_qwen2r get_loggerr,loggerr0MAX_MODEL_INPUT_SIZESr __all__rrrr!s       d