o i7@s`ddlmZddlmZddlmZddlmZee Z ddd d Z Gd d d eZ d gZ d S))Optional)PreTrainedTokenizerFast)logging)HerbertTokenizerz vocab.jsonz merges.txtztokenizer.json) vocab_file merges_filetokenizer_filec seZdZdZeZeZ        dfdd Z dd e e d e e e d e e fd dZ dd e e d e e e de d e e ffdd Zddede ed eefddZZS)HerbertTokenizerFastam Construct a "Fast" BPE tokenizer for HerBERT (backed by HuggingFace's *tokenizers* library). Peculiarities: - uses BERT's pre-tokenizer: BertPreTokenizer splits tokens on spaces, and also on punctuation. Each occurrence of a punctuation character will be treated separately. This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the methods. Users should refer to the superclass for more information regarding methods. Args: vocab_file (`str`): Path to the vocabulary file. merges_file (`str`): Path to the merges file. Nc s(tj||f||||||d| dS)N)r cls_token unk_token pad_token mask_token sep_token)super__init__) selfrr r rrrrrkwargs __class__i/home/ubuntu/.local/lib/python3.10/site-packages/transformers/models/herbert/tokenization_herbert_fast.pyr2s  zHerbertTokenizerFast.__init__ token_ids_0 token_ids_1returncCs8|jg}|jg}|dur|||S|||||S)a Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. An HerBERT, like BERT sequence has the following format: - single sequence: ` X ` - pair of sequences: ` A B ` Args: token_ids_0 (`List[int]`): List of IDs to which the special tokens will be added. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens. N) cls_token_id sep_token_id)rrrclsseprrr build_inputs_with_special_tokensJs  z5HerbertTokenizerFast.build_inputs_with_special_tokensFalready_has_special_tokenscsf|r tj||ddS|durdgdgt|dgSdgdgt|dgdgt|dgS)a Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. T)rrr&Nrr)rget_special_tokens_masklen)rrrr&rrrr'es.z,HerbertTokenizerFast.get_special_tokens_masksave_directoryfilename_prefixcCs|jjj||d}t|S)N)name) _tokenizermodelsavetuple)rr)r*filesrrrsave_vocabularysz$HerbertTokenizerFast.save_vocabulary)NNNr r rrr)N)NF)__name__ __module__ __qualname____doc__VOCAB_FILES_NAMESvocab_files_namesrslow_tokenizer_classrlistintrr%boolr'strr/r1 __classcell__rrrrr s@   (r N)typingrtokenization_utils_fastrutilsrtokenization_herbertr get_loggerr2loggerr6r __all__rrrrs       i