o ei@sdddlmZmZmZddlmZddlmZddlm Z e e Z ddiZ Gdd d eZd gZd S) ) Tokenizerdecoders normalizers)BPE)TokenizersBackend)loggingtokenizer_fileztokenizer.jsoncseZdZdZeZdZddgZeZ      dd e e e e fBdBd e e e BdBd e de de de de ffdd Zde fddZZS)GemmaTokenizeru Construct a fast Gemma tokenizer (backed by HuggingFace's tokenizers library). This tokenizer uses a BPE model with byte fallback, no prefix space, and a normalizer that replaces spaces with "▁". Args: tokenizer_file (`str`, optional): A tokenizers JSON file containing the serialization of a tokenizer. unk_token (`str`, optional, defaults to ""): The unknown token. bos_token (`str`, optional, defaults to ""): The beginning of sequence token. eos_token (`str`, optional, defaults to ""): The end of sequence token. pad_token (`str`, optional, defaults to ""): The padding token. mask_token (`str`, optional, defaults to ""): The mask token. add_bos_token (`bool`, optional, defaults to True): Whether or not to add a `bos_token` at the start of sequences. add_eos_token (`bool`, optional, defaults to False): Whether or not to add an `eos_token` at the end of sequences. vocab (`str` or `dict[str, int]`, optional): Custom vocabulary dict. If not provided, a minimal vocabulary is created using the special tokens. left input_idsattention_maskNvocabmerges unk_token bos_token eos_token pad_token mask_tokenc s|durt|dt|dt|dt|dt|di}||_|p g|_tt|j|jdt|ddd|_ttdd t t g|j_ t d d|j_ tjd |||||d |dS) NrrT)rrfuse_unkrdropout byte_fallbacku▁ )rrrrr)str_vocab_mergesrr _tokenizerrSequenceReplace ByteFallbackFusedecoderr normalizersuper__init__) selfrrrrrrrkwargs __class__r!j/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/gemma/tokenization_gemma.pyr-;s@    zGemmaTokenizer.__init__returncCsdS)Nrr!)r.r!r!r2_unk_idiszGemmaTokenizer._unk_id)NNrrrrr)__name__ __module__ __qualname____doc__VOCAB_FILES_NAMESvocab_files_names padding_sidemodel_input_namesrmodelr"dictintlistr-r4 __classcell__r!r!r0r2r s:.r N) tokenizersrrrtokenizers.modelsrtokenization_utils_tokenizersrutilsr get_loggerr5loggerr9r __all__r!r!r!r2s     T