o i@sdZddlZddlZddlZddlZddlmZddlmZm Z m Z ddl m Z ddl mZddlmZmZdd lmZdd lmZdd lmZdd lmZmZmZmZmZmZd dlm Z ddl!m"Z"ddl#m$Z$m%Z%m&Z&m'Z'm(Z(er|ddl)m*Z*ndZ*e+e,Z-ee.e/e e.e e.ffgdderdndffderdnderdndffdderdndffdderdndffdderdndffd derd!ndffd"derdndffd#d$erd%nderd&ndffd'd(derdndffd)er d*nddffd+d,d-erd.nder d/ndffd0d1er,d2ndffd3d4der:d5ndffd6d7d8derJdndffd9d:erVd;ndffddernd5ndffd?d@erzdAndffdBderdndffdCdDerdEnderdFndffdGdHerdnderdndffdIderdndffdJd@erdAndffdKderdndffdLderdndffdMdNerdOnderdPndffdQdRerdSndffdTder d!ndffdUderd!ndffdVder"dndffdWdXer.dYndffdZd[er:d\ndffd]erEd^nderLd_ndffd`daderZd5ndffdbdcddd@erjdAndffded:ervd;ndffdfdgerdhndffdierdjnderdkndffdlerdnderdndffdmerdnderdndffdnerdnderdndffdoerdnderdndffdpdqerdnderdndffdrdserdtndffdudverdwndffdxdyerdzndffd{d:er d;ndffd|der,dndffd}der8dndffd~derDdndffderOdnddffdder]d:nderdd;ndffdderpd5ndffdder|dndffderdnddffddderd;ndffdderdndffddderdndffderdnderdndffderdnderdndffderdnderdndffderdnderdndffderdnder dndffderdnderdndffdder(dndffdder4d5ndffdder@d5ndffdderLd5ndffdderXd5ndffdderdd5ndffderodnddffdd:er|d;ndffdd:erd;ndffdd:erd;ndffdderdndffddderd5ndffdd:erd;ndffddddddderdndffdderdndffdderd5ndffdderdndffddd@erdAndffdderdndffdderdndffdder&dndffdd:er2d;ndffdd:er>d;ndffddXerJdYndffderUdnder\dndffdderhdndffdersdnderzdndffdderdnderdndffdderd5ndffdderdndffdderdndffdderdndffdderdndffdderdndffdderdndffderdnderdndffderdnderdndffderdnderdndffdder$dndffdder0dndffdderdXe rCdYndffd?dXe rPdYndffd@dXe r]dYndffdAdBdCe rodDndffdEe r{dnde rdndffdFe rdGnde rdHndffdIe rdJnde rdKndffdLdMe rdNndffdOd@e rdAndffdPd@e rdAndffdQdRdSe rdTndffdUde rdndffdVe rdWnde r dXndffdYe rdWnde rdXndffdZe r*dnde r1dndffd[e r>d\nddffd]e rKdnde rRdndffd^de r_d5ndffd_e rld`nddffdadbe r}dcnddffdddedfe rdgndffdhde rdndffdid:e rd;ndffdje rdnde rdndffdke rdnde rdndffdle rdnde rdndffdmdndodpde rdndffdqe rdrnde rdsndffdte rdnde r!dndffdude r.dndffdvde r;dndffdwde rHdndffdxde rUdndffdydze rddnde roe sod5ndffd{d|d}d~dde rdndffdde rdndffde rdnde rdndffdde rdnddffde rdnde rdndffde rdnde rdndffde rdnde rdndffdde rdndffde rdnde rdndffde r$dnde r+dndffde r7dnde r>dndffde rJdnde rQdndffZ0e"e$e0Z1dde$2DZ3de.de e4edffddZ5       dde e.ej6e.fde e e.ej6e.fde7de e7de e8e.e.fde e e7e.fde e.de7de.de8e.effddZ9GdddZ:ddgZ;dS(zAuto Tokenizer class.N) OrderedDict)AnyOptionalUnion)is_mistral_common_available)PretrainedConfig)get_class_from_dynamic_moduleresolve_trust_remote_code)load_gguf_checkpoint)PreTrainedTokenizer)TOKENIZER_CONFIG_FILE) cached_fileextract_commit_hashis_g2p_en_availableis_sentencepiece_availableis_tokenizers_availablelogging)EncoderDecoderConfig)_LazyAutoMapping)CONFIG_MAPPING_NAMES AutoConfigconfig_class_to_model_typemodel_type_to_module_name!replace_list_option_in_docstrings)PreTrainedTokenizerFastaimv2 CLIPTokenizerCLIPTokenizerFastalbertAlbertTokenizerAlbertTokenizerFastalign BertTokenizerBertTokenizerFastarceeLlamaTokenizerLlamaTokenizerFastaria aya_visionCohereTokenizerFastbark)bart) BartTokenizerBartTokenizerFastbarthezBarthezTokenizerBarthezTokenizerFast)bartpho)BartphoTokenizerNbertzbert-generationBertGenerationTokenizer)z bert-japanese)BertJapaneseTokenizerN)bertweet)BertweetTokenizerNbig_birdBigBirdTokenizerBigBirdTokenizerFastbigbird_pegasusPegasusTokenizerPegasusTokenizerFast)biogpt)BioGptTokenizerNbitnetr) blenderbot)BlenderbotTokenizerBlenderbotTokenizerFast)zblenderbot-small)BlenderbotSmallTokenizerNblipzblip-2 GPT2TokenizerGPT2TokenizerFastbloomBloomTokenizerFastblt bridgetowerRobertaTokenizerRobertaTokenizerFastbros)byt5) ByT5TokenizerN camembertCamembertTokenizerCamembertTokenizerFast)canine)CanineTokenizerN chameleon chinese_clipclapclipclipseg)clvp) ClvpTokenizerN code_llamaCodeLlamaTokenizerCodeLlamaTokenizerFastcodegenCodeGenTokenizerCodeGenTokenizerFastcoherecohere2colpalicolqwen2Qwen2TokenizerQwen2TokenizerFastconvbertConvBertTokenizerConvBertTokenizerFastcpm CpmTokenizerCpmTokenizerFast)cpmant)CpmAntTokenizerNcsm)ctrl) CTRLTokenizerN)zdata2vec-audioWav2Vec2CTCTokenizerNz data2vec-textdbrxdebertaDebertaTokenizerDebertaTokenizerFastz deberta-v2DebertaV2TokenizerDebertaV2TokenizerFast deepseek_v2 deepseek_v3 deepseek_vldeepseek_vl_hybrid)dia) DiaTokenizerN diffllama distilbertDistilBertTokenizerDistilBertTokenizerFastdprDPRQuestionEncoderTokenizerDPRQuestionEncoderTokenizerFastelectraElectraTokenizerElectraTokenizerFastemu3ernieernie4_5 ernie4_5_moeernie_mErnieMTokenizer)esm) EsmTokenizerNexaone4falcon falcon_mambaGPTNeoXTokenizerFastfastspeech2_conformerFastSpeech2ConformerTokenizer)flaubert)FlaubertTokenizerN flex_olmofnet FNetTokenizerFNetTokenizerFast)fsmt) FSMTTokenizerNfunnelFunnelTokenizerFunnelTokenizerFastgemmaGemmaTokenizerGemmaTokenizerFastgemma2gemma3 gemma3_textgemma3n gemma3n_textgitglmglm4glm4_moeglm4v glm4v_moezgpt-sw3GPTSw3Tokenizergpt2 gpt_bigcodegpt_neogpt_neox)gpt_neox_japanese)GPTNeoXJapaneseTokenizerNgpt_ossgptj)zgptsan-japanese)GPTSanJapaneseTokenizerN)graniterIN) granitemoer)granitemoehybridr)granitemoesharedrzgrounding-dinogroupvitheliumherbertHerbertTokenizerHerbertTokenizerFast)hubertrwibertideficsidefics2idefics3 instructblipinstructblipvideointernvljambajanusjetmoe)jukebox)JukeboxTokenizerNzkosmos-2XLMRobertaTokenizerXLMRobertaTokenizerFastz kosmos-2.5layoutlmLayoutLMTokenizerLayoutLMTokenizerFast layoutlmv2LayoutLMv2TokenizerLayoutLMv2TokenizerFast layoutlmv3LayoutLMv3TokenizerLayoutLMv3TokenizerFast layoutxlmLayoutXLMTokenizerLayoutXLMTokenizerFastled LEDTokenizerLEDTokenizerFastliltllamallama4 llama4_textllava llava_nextllava_next_videollava_onevision longformerLongformerTokenizerLongformerTokenizerFastlongt5 T5TokenizerT5TokenizerFast)luke) LukeTokenizerNlxmertLxmertTokenizerLxmertTokenizerFastm2m_100M2M100Tokenizermambamamba2marianMarianTokenizermbartMBartTokenizerMBartTokenizerFastmbart50MBart50TokenizerMBart50TokenizerFastmegaz megatron-bert metaclip_2)zmgp-str)MgpstrTokenizerNminimax ministralMistralCommonTokenizermistralmistral3mixtralmllamamlukeMLukeTokenizerzmm-grounding-dino mobilebertMobileBertTokenizerMobileBertTokenizerFast modernbert moonshinemoshimpnetMPNetTokenizerMPNetTokenizerFastmptmramt5 MT5TokenizerMT5TokenizerFastmusicgenmusicgen_melodymvp MvpTokenizerMvpTokenizerFast)myt5) MyT5TokenizerNnemotronnezhanllb NllbTokenizerNllbTokenizerFastznllb-moe nystromformerolmoolmo2olmo3olmoez omdet-turbo oneformerz openai-gptOpenAIGPTTokenizerOpenAIGPTTokenizerFastoptowlv2owlvit paligemma)parakeet)ParakeetCTCTokenizerNpegasus pegasus_x) perceiver)PerceiverTokenizerN persimmonphiphi3phimoe)phobert)PhobertTokenizerN pix2structpixtralplbartPLBartTokenizer) prophetnet)ProphetNetTokenizerNqdqbertqwen2 qwen2_5_omni qwen2_5_vl qwen2_audio qwen2_moeqwen2_vlqwen3 qwen3_moe qwen3_nextqwen3_omni_moeqwen3_vl qwen3_vl_moe)rag) RagTokenizerNrealmRealmTokenizerRealmTokenizerFastrecurrent_gemmareformerReformerTokenizerReformerTokenizerFastrembertRemBertTokenizerRemBertTokenizerFast retribertRetriBertTokenizerRetriBertTokenizerFastrobertazroberta-prelayernorm)roc_bert)RoCBertTokenizerNroformerRoFormerTokenizerRoFormerTokenizerFastrwkv seamless_m4tSeamlessM4TTokenizerSeamlessM4TTokenizerFastseamless_m4t_v2 shieldgemma2siglipSiglipTokenizersiglip2smollm3speech_to_textSpeech2TextTokenizer)speech_to_text_2)Speech2Text2TokenizerNspeecht5SpeechT5Tokenizer)splinter)SplinterTokenizerSplinterTokenizerFast squeezebertSqueezeBertTokenizerSqueezeBertTokenizerFaststablelm starcoder2switch_transformerst5t5gemma)tapas)TapasTokenizerN)tapex)TapexTokenizerN)z transfo-xl)TransfoXLTokenizerNtvpudop UdopTokenizerUdopTokenizerFastumt5 video_llavaviltvipllava visual_bert)vits) VitsTokenizerNvoxtral)wav2vec2rw)z wav2vec2-bertrw)zwav2vec2-conformerrw)wav2vec2_phoneme)Wav2Vec2PhonemeCTCTokenizerNwhisperWhisperTokenizerWhisperTokenizerFastxclipxglm XGLMTokenizerXGLMTokenizerFast)xlm) XLMTokenizerNzxlm-prophetnetXLMProphetNetTokenizerz xlm-robertazxlm-roberta-xlxlnetXLNetTokenizerXLNetTokenizerFastxlstmxmodyosozambazamba2cCsi|]\}}||qSr).0kvrrg/home/ubuntu/veenaModal/venv/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py $sr class_namereturnc Cs|dkrtStD]6\}}||vr@t|}|dvr%|dkr%tdd}n td|d}zt||WSty?Yq wq tj D]}|D]}t|dd|krZ|SqJqFtd}t ||rkt||SdS) Nr)r rr r z.tokenization_mistral_common transformers.ztransformers.models__name__) rTOKENIZER_MAPPING_NAMESitemsr importlib import_modulegetattrAttributeErrorTOKENIZER_MAPPING_extra_contentvalueshasattr)r module_name tokenizersmodule tokenizer main_modulerrrtokenizer_class_from_name's0      rFpretrained_model_name_or_path cache_dirforce_downloadresume_downloadproxiestokenrevisionlocal_files_only subfolderc Ks| dd} | durtdt|durtd| }| d} t|t||||||||ddd| d} | dur Passing `token=True` is required when you want to use a private model. Returns: `dict`: The configuration of the tokenizer. Examples: ```python # Download configuration from huggingface.co and cache. tokenizer_config = get_tokenizer_config("google-bert/bert-base-uncased") # This model does not have a tokenizer config so the result will be an empty dict. tokenizer_config = get_tokenizer_config("FacebookAI/xlm-roberta-base") # Save a pretrained tokenizer locally and you can reload its config from transformers import AutoTokenizer tokenizer = AutoTokenizer.from_pretrained("google-bert/bert-base-cased") tokenizer.save_pretrained("tokenizer-test") tokenizer_config = get_tokenizer_config("tokenizer-test") ```use_auth_tokenNrThe `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.V`token` and `use_auth_token` are both specified. Please set only the argument `token`. _commit_hashF) rrrrrrrr _raise_exceptions_for_gated_repo%_raise_exceptions_for_missing_entries'_raise_exceptions_for_connection_errorsrz\Could not locate the tokenizer configuration file, will try to use the model config instead.zutf-8)encoding)popwarningswarn FutureWarning ValueErrorgetrr loggerinforopenjsonload)rrrrrrrrrkwargsr commit_hashresolved_config_filereaderresultrrrget_tokenizer_configEsF I    rc@s:eZdZdZddZeeeddZe d dd Z dS) AutoTokenizera This is a generic tokenizer class that will be instantiated as one of the tokenizer classes of the library when created with the [`AutoTokenizer.from_pretrained`] class method. This class cannot be instantiated directly using `__init__()` (throws an error). cCstd)Nz}AutoTokenizer is designed to be instantiated using the `AutoTokenizer.from_pretrained(pretrained_model_name_or_path)` method.)OSError)selfrrr__init__szAutoTokenizer.__init__cOsn|dd}|durtdt|ddurtd||d<|dd}d|d<|d d}|d d}|d d}|d } |durd} t|d} | durctd |ddddtDd| \} } |rw| durrt| } nt d| durt| } | durtd| d| j |g|Ri|St |fi|}d|vr|d|d<|d}d}d|vrt |dttfr|d}n|ddd}|durt |ts| rt|| fi|}t|ddd}tjd)i|}n tj |fd |i|}|j}t|drd|jvr|jd}|du}t|tvp-|duo-t|dup-t|ddu}|r[|r@|ddur@|d}n|d}d|vrQ|dd}nd}t|||||}|r|rt||fi|} |d d}| | j |g|Rd |i|S|durd} |r|ds|d}t|} | dur|}t|} | durtd|d!| j |g|Ri|St |trt|j t|j!urt d"|j!j"d#|j j"d$|j!}t#t|j$}|dur#tt|\}}|r|s|dur|j |g|Ri|S|dur|j |g|Ri|Std%td&|j"d'dd(dtDd)*a] Instantiate one of the tokenizer classes of the library from a pretrained model vocabulary. The tokenizer class to instantiate is selected based on the `model_type` property of the config object (either passed as an argument or loaded from `pretrained_model_name_or_path` if possible), or when it's missing, by falling back to using pattern matching on `pretrained_model_name_or_path`: List options Params: pretrained_model_name_or_path (`str` or `os.PathLike`): Can be either: - A string, the *model id* of a predefined tokenizer hosted inside a model repo on huggingface.co. - A path to a *directory* containing vocabulary files required by the tokenizer, for instance saved using the [`~PreTrainedTokenizer.save_pretrained`] method, e.g., `./my_model_directory/`. - A path or url to a single saved vocabulary file if and only if the tokenizer only requires a single vocabulary file (like Bert or XLNet), e.g.: `./my_model_directory/vocab.txt`. (Not applicable to all derived classes) inputs (additional positional arguments, *optional*): Will be passed along to the Tokenizer `__init__()` method. config ([`PretrainedConfig`], *optional*) The configuration object used to determine the tokenizer class to instantiate. cache_dir (`str` or `os.PathLike`, *optional*): Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used. force_download (`bool`, *optional*, defaults to `False`): Whether or not to force the (re-)download the model weights and configuration files and override the cached versions if they exist. resume_download: Deprecated and ignored. All downloads are now resumed by default when possible. Will be removed in v5 of Transformers. proxies (`dict[str, str]`, *optional*): A dictionary of proxy servers to use by protocol or endpoint, e.g., `{'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}`. The proxies are used on each request. revision (`str`, *optional*, defaults to `"main"`): The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so `revision` can be any identifier allowed by git. subfolder (`str`, *optional*): In case the relevant files are located inside a subfolder of the model repo on huggingface.co (e.g. for facebook/rag-token-base), specify it here. use_fast (`bool`, *optional*, defaults to `True`): Use a [fast Rust-based tokenizer](https://huggingface.co/docs/tokenizers/index) if it is supported for a given model. If a fast tokenizer is not available for a given model, a normal Python-based tokenizer is returned instead. tokenizer_type (`str`, *optional*): Tokenizer type to be loaded. trust_remote_code (`bool`, *optional*, defaults to `False`): Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to `True` for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine. kwargs (additional keyword arguments, *optional*): Will be passed to the Tokenizer `__init__()` method. Can be used to set special tokens like `bos_token`, `eos_token`, `unk_token`, `sep_token`, `pad_token`, `cls_token`, `mask_token`, `additional_special_tokens`. See parameters in the `__init__()` for more details. Examples: ```python >>> from transformers import AutoTokenizer >>> # Download vocabulary from huggingface.co and cache. >>> tokenizer = AutoTokenizer.from_pretrained("google-bert/bert-base-uncased") >>> # Download vocabulary from huggingface.co (user-uploaded) and cache. >>> tokenizer = AutoTokenizer.from_pretrained("dbmdz/bert-base-german-cased") >>> # If vocabulary files are in a directory (e.g. tokenizer was saved using *save_pretrained('./test/saved_model/')*) >>> # tokenizer = AutoTokenizer.from_pretrained("./test/bert_saved_model/") >>> # Download vocabulary from huggingface.co and define model-specific arguments >>> tokenizer = AutoTokenizer.from_pretrained("FacebookAI/roberta-base", add_prefix_space=True) ```rNrrrconfigT _from_autouse_fasttokenizer_typetrust_remote_code gguf_filezPassed `tokenizer_type` z3 does not exist. `tokenizer_type` should be one of z, css|]}|VqdSNrrcrrr +sz0AutoTokenizer.from_pretrained..rzt`use_fast` is set to `True` but the tokenizer class does not have a fast version. Falling back to the slow version.zTokenizer class z is not currently imported.rtokenizer_classauto_maprF)return_tensorsFastrrz-- code_revisionz- does not exist or is not currently imported.z The encoder model config class: z3 is different from the decoder model config class: z. It is not recommended to use the `AutoTokenizer.from_pretrained()` method in this case. Please use the encoder and decoder specific tokenizer classes.zzThis tokenizer cannot be instantiated. Please make sure you have `sentencepiece` installed in order to use this tokenizer.z!Unrecognized configuration class z8 to build an AutoTokenizer. Model type should be one of css|]}|jVqdSr)rrrrrrsr)%rrrrrrrjoinrrwarningfrom_pretrainedr isinstancetuplelistrrr r for_modelrrrtypersplitr r register_for_auto_classendswithrdecoderencoder __class__rr)clsrinputsrrrrrrrrtokenizer_class_tupletokenizer_class_nametokenizer_fast_class_nametokenizer_configconfig_tokenizer_classtokenizer_auto_map gguf_path config_dicthas_remote_codehas_local_code class_ref upstream_repo_tokenizer_class_candidate model_typetokenizer_class_pytokenizer_class_fastrrrrs M                             zAutoTokenizer.from_pretrainedNFcCs|dur |dur td|durt|trtd|dur&t|tr&td|durD|durDt|trD|j|krDtd|jd|d|tjvr[t|\}}|durU|}|dur[|}tj|||f|ddS) a Register a new tokenizer in this mapping. Args: config_class ([`PretrainedConfig`]): The configuration corresponding to the model to register. slow_tokenizer_class ([`PretrainedTokenizer`], *optional*): The slow tokenizer to register. fast_tokenizer_class ([`PretrainedTokenizerFast`], *optional*): The fast tokenizer to register. NzKYou need to pass either a `slow_tokenizer_class` or a `fast_tokenizer_classz:You passed a fast tokenizer in the `slow_tokenizer_class`.z:You passed a slow tokenizer in the `fast_tokenizer_class`.zThe fast tokenizer class you are passing has a `slow_tokenizer_class` attribute that is not consistent with the slow tokenizer class you passed (fast tokenizer has z and you passed z!. Fix one of those so they match!)exist_ok)r issubclassrr slow_tokenizer_classrrregister) config_classr"fast_tokenizer_classr  existing_slow existing_fastrrrr#s2   zAutoTokenizer.register)NNF) r __module__ __qualname____doc__r classmethodrrr staticmethodr#rrrrrs crr)NFNNNNFr)s>                 "()*+,-./01235;=CELSY[abcdefhnopqrstv} !#)*+,-./178:=>?@ACJQX_flmnopqrstuvwxyz{|}~    #$%&'()+29?@ABCEKLMNOPRY_`bhjqz  &'()*+-4;ABCDFIKNOPQRT[aiopqrsu}~   "()*+,.5<BCEHIJLSZ`bhijklnqrt{    !'()*+,.457>EKMT[b l&   o!