o ei@s:UdZddlZddlZddlZddlmZddlmZddlm Z ddl m Z ddl m Z mZdd lmZdd lmZdd lmZmZmZmZmZdd lmZd dlmZddlmZddlm Z m!Z!m"Z"m#Z#m$Z$ersddl%m&Z&ndZ&erddl'm(Z(ndZ(e)e*Z+iZ,e-e.e/efe0d<iZ1e-e.e/efe0d<ee.e.dBfgderdndfderdndfderdndfderdndfderdndfd erdndfd!erd"ndfd#erd$ndfd%d&erdndfd'erd(ndfd)d*d+er d,ndfd-erd.ndfd/d0er#d1ndfd2d3er/dndfd4er9d5ndfd6d7erEdndfd8d9erQd:ndfd;deridndfd?ersdndfd@dAerdBndfdCerd5ndfdDerdndfdEerdndfdFerdndfdGerdndfdHerdIndfdJdKdLdMdNerd5ndfdOerdPndfdQerdRndfdSdTerdndfdUerdVndfdWerdndfdXer d5ndfdYerdndfdZd[er!d\ndfd]er+d^ndfd_d`er7dndfdaerAd5ndfdberKdcndfdderUdendfdfdgeradhndfdierkdjndfdkerudjndfdlerdjndfdmerdjndfdnerdjndfdoerdjndfdperdndfdqerdrndfdserdrndfdterdrndfduerdrndfdverdrndfdwerdrndfdxerdrndfdyerdrndfdzerdrndfd{er d|ndfd}erd5ndfd~erd5ndfder)d5ndfder3d\ndfdder?d5ndfddddderQdndfder[dndfderedndfdddersdndfder}dndfderd5ndfderd5ndfderdndfderd5ndfderdndfderdndfderdndfderdndfderdndfderdndfderdndfderdndfderdndfder d"ndfderd"ndfderdndfdder)dndfder3dndfder=d\ndfderGd\ndfderQdndfder[dndfderedndfderodndfdder{dndfderdndfdde rdnerdrndfde rdnerdrndfde rdnerdrndfde rdnerdrndfderdndfderdndfderdndfderdndfderd\ndfdȑderdndfderdndfderdndfder#dndfdΑder/dndfder9dndfderCdndfderMdndfderWdndfderad\ndfderkd\ndfderud5ndfderd\ndfderdndfderdndfderdndfderd5ndfderdndfderdndfderdndfderd.ndfderd.ndfdderd5ndfdderdndfde rdnerdrndfder dndfdderdndfder!dndfder+dndfder5dndfder?dndfderIdndfderSdndfder]dndfdergdndfderqdndfder{dndfderdndfderdndfderdndfderdndfdderdndfderdjndfderĐdndfderАdndfderdndfdddderdndfd erd\ndfd erdndfd erdndfd erd ndfder)d ndfder4djndfder@dndfderLdndfderXdndfderddndfdderrdndfder}d\ndfderd5ndfderdndfderdndfderdjndfdd erdndfd!erdndfd"erΐd#ndfd$erdndfd%d&d'erdndfd(erdndfd)d*e rdner drndfd+e rdnerdrndfd,d-d.d/d0er2d1ndfd2er=dndfd3erId4ndfd5d6erWdndfd7erbdndfd8ernd9ndfd:eryd\ndfd;erdndfde 4DZ5d?d@Z6dAdBZ7dCe.dDe/edBfdEdFZ8  G    G HdVdIe.ej9e.BdJe.ej9e.BdBdKe:dLe-e.e.fdBdMe:e.BdBdNe.dBdOe:dPe.dDe-e.effdQdRZ;GdSdTdTZ<dUdTgZ=dS(WzAuto Tokenizer class.N) OrderedDict)Any)is_mistral_common_available)PreTrainedConfig)get_class_from_dynamic_moduleresolve_trust_remote_code)load_gguf_checkpoint)TOKENIZER_CONFIG_FILE)extract_commit_hashis_g2p_en_availableis_sentencepiece_availableis_tokenizers_availablelogging) cached_file)EncoderDecoderConfig)_LazyAutoMapping)CONFIG_MAPPING_NAMES AutoConfigconfig_class_to_model_typemodel_type_to_module_name!replace_list_option_in_docstrings)TokenizersBackend)SentencePieceBackendREGISTERED_TOKENIZER_CLASSESREGISTERED_FAST_ALIASESaimv2 CLIPTokenizeralbertAlbertTokenizeralign BertTokenizeraudioflamingo3Qwen2Tokenizer aya_visionCohereTokenizerbarkbartRobertaTokenizerbarthezBarthezTokenizer)bartphoBartphoTokenizerbertzbert-generationBertGenerationTokenizer)z bert-japaneseBertJapaneseTokenizer)bertweetBertweetTokenizerbig_birdBigBirdTokenizerbigbird_pegasusPegasusTokenizer)biogptBioGptTokenizer blenderbotBlenderbotTokenizer)zblenderbot-smallBlenderbotSmallTokenizerblipzblip-2 GPT2Tokenizer) bridgetowerr*bros)byt5 ByT5Tokenizer camembertCamembertTokenizer)canineCanineTokenizer chinese_clip)clapr*clipclipseg)clvp ClvpTokenizer code_llamaCodeLlamaTokenizercodegencoherecohere2colqwen2convbertcpm CpmTokenizer)cpmantCpmAntTokenizer)ctrl CTRLTokenizer)zdata2vec-audioWav2Vec2CTCTokenizer)z data2vec-textr*dbrxdebertaDebertaTokenizerz deberta-v2DebertaV2Tokenizer)dia DiaTokenizer distilbertdprDPRQuestionEncoderTokenizerelectraemu3ernie)esm EsmTokenizer falcon_mambaGPTNeoXTokenizerfastspeech2_conformerFastSpeech2ConformerTokenizer)flaubertFlaubertTokenizerflava flex_olmo florence2 BartTokenizerfnet FNetTokenizer)fsmt FSMTTokenizerfunnelFunnelTokenizergemmaGemmaTokenizergemma2gemma3 gemma3_textgemma3n gemma3n_textgitglmrglm4glm4_moe glm4_moe_liteglm4v glm4v_moe glm_imageglmasrgot_ocr2zgpt-sw3GPTSw3Tokenizergpt2 gpt_bigcodegpt_neogpt_neox)gpt_neox_japaneseGPTNeoXJapaneseTokenizergptj)graniter>) granitemoer>)granitemoehybridr>)granitemoesharedr>zgrounding-dinogroupvitherbertHerbertTokenizer)hubertrZ)ibertr*ideficsLlamaTokenizeridefics2 instructblipinstructblipvideointernvljais2zkosmos-2XLMRobertaTokenizerlasr_ctcParakeetTokenizer lasr_encoderlayoutlm layoutlmv2LayoutLMv2Tokenizer layoutlmv3LayoutLMv3Tokenizer layoutxlmLayoutXLMTokenizerled LEDTokenizer lighton_ocrQwen2TokenizerFastlilt longformerlongt5 T5Tokenizer)luke LukeTokenizerlxmertLxmertTokenizerm2m_100M2M100Tokenizermambamamba2marianMarianTokenizermarkuplmMarkupLMTokenizermbartMBartTokenizermbart50MBart50Tokenizer)megar*z megatron-bert metaclip_2)zmgp-strMgpstrTokenizer ministral3MistralCommonBackendmistralmistral3mixtralmlukeMLukeTokenizerzmm-grounding-dino mobilebertMobileBertTokenizermpnetMPNetTokenizermpt)mrar*mt5musicgenmusicgen_melodymvp MvpTokenizer)myt5 MyT5Tokenizernezhanllb NllbTokenizerznllb-moenougatNougatTokenizer nystromformerolmoolmo2olmo3olmoez omdet-turbo oneformerz openai-gptOpenAIGPTTokenizeroptovis2owlv2owlvitpegasus pegasus_x) perceiverPerceiverTokenizerphi)phobertPhobertTokenizer pix2structpixtralplbartPLBartTokenizer) prophetnetProphetNetTokenizerqdqbertqwen2 qwen2_5_omni qwen2_5_vl qwen2_audio qwen2_moeqwen2_vlqwen3qwen3_5Qwen3_5Tokenizer qwen3_5_moe qwen3_moe qwen3_nextqwen3_omni_moeqwen3_vl qwen3_vl_moe)rag RagTokenizerrealmrecurrent_gemmareformerReformerTokenizerrembertRemBertTokenizer retribert)robertar*)zroberta-prelayernormr*)roc_bertRoCBertTokenizerroformerRoFormerTokenizerrwkvsam3 sam3_video seamless_m4tSeamlessM4TTokenizerseamless_m4t_v2 shieldgemma2siglipSiglipTokenizersiglip2Siglip2Tokenizerspeech_to_textSpeech2TextTokenizerspeecht5SpeechT5Tokenizer)splinterSplinterTokenizer squeezebertstablelm starcoder2switch_transformerst5t5gemma)tapasTapasTokenizertrocrtvpudop UdopTokenizerumt5) unispeechrZ)z unispeech-satrZvilt visual_bert)vits VitsTokenizervoxtralvoxtral_realtime)wav2vec2rZ)z wav2vec2-bertrZ)zwav2vec2-conformerrZ)wav2vec2_phonemeWav2Vec2PhonemeCTCTokenizerwhisperWhisperTokenizerxclipxglm XGLMTokenizer)xlm XLMTokenizerz xlm-robertazxlm-roberta-xlxlnetXLNetTokenizerxlstmxmodyosocCsi|]\}}||qSrK).0kvrKrKh/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py PsrPcCs<t|ddd }t|WdS1swYdS)z*Loads a vocabulary file into a dictionary.rutf-8encodingN)openjsonload) vocab_filereaderrKrKrO load_vocabSs$rZcCsjg}t|ddd"}|D]}|}|r"|ds"|t|q Wd|S1s.wY|S)z Loads a merges file into a list.rQrRrS#N)rUstrip startswithappendtuplesplit) merges_filemergesrYlinerKrKrO load_mergesYs rd class_namereturnc Cs|dvrtS|tvrt|S|tvrt|S|dkrtStD]6\}}||krVt|}|dvr;|dkr;tdd}n td|d}zt||WSt yUYq wq t j D]}t|d d|krj|Sq\td}t ||rzt||SdS) N>BloomTokenizerBloomTokenizerFastr)rrr ministralrrr:rz.tokenization_mistral_common transformers.ztransformers.models__name__)rrrTOKENIZER_MAPPING_NAMESitemsr importlib import_modulegetattrAttributeErrorTOKENIZER_MAPPING_extra_contentvalueshasattr)re module_nametokenizer_classmodule tokenizer main_modulerKrKrOtokenizer_class_from_nameds:    r|Fpretrained_model_name_or_path cache_dirforce_downloadproxiestokenrevisionlocal_files_only subfolderc Ks|d} t|t|||||||ddd| d } | dur!tdiSt| | } t| dd } t| } Wdn1s Passing `token=True` is required when you want to use a private model. Returns: `dict`: The configuration of the tokenizer. Examples: ```python # Download configuration from huggingface.co and cache. tokenizer_config = get_tokenizer_config("google-bert/bert-base-uncased") # This model does not have a tokenizer config so the result will be an empty dict. tokenizer_config = get_tokenizer_config("FacebookAI/xlm-roberta-base") # Save a pretrained tokenizer locally and you can reload its config from transformers import AutoTokenizer tokenizer = AutoTokenizer.from_pretrained("google-bert/bert-base-cased") tokenizer.save_pretrained("tokenizer-test") tokenizer_config = get_tokenizer_config("tokenizer-test") ``` _commit_hashF) rrrrrrr _raise_exceptions_for_gated_repo%_raise_exceptions_for_missing_entries'_raise_exceptions_for_connection_errorsrNz\Could not locate the tokenizer configuration file, will try to use the model config instead.rRrS) getrr loggerinfor rUrVrW) r~rrrrrrrkwargs commit_hashresolved_config_filerYresultrKrKrOget_tokenizer_configs2 E   rc@sFeZdZdZddZeeedee BfddZ e d d d Z dS) AutoTokenizera This is a generic tokenizer class that will be instantiated as one of the tokenizer classes of the library when created with the [`AutoTokenizer.from_pretrained`] class method. This class cannot be instantiated directly using `__init__()` (throws an error). cCstd)Nz}AutoTokenizer is designed to be instantiated using the `AutoTokenizer.from_pretrained(pretrained_model_name_or_path)` method.)OSError)selfrKrKrO__init__szAutoTokenizer.__init__rfc Os|dd}d|d<|dd}|dd}|dd}|d}|dur^t|d} | durBtd |d d d d tDdt| } | durRtd| d| j|g|Ri|S|rzt||fi|} t| ddd} t j d)i| }n$|durz t j|fd|i|}Wnt yt j|fi|}Ynw|j } t|fi|}|dd}d}d|vrt|dttfr|d}n|ddd}|dur,|dur,| dur,| dkr,t| ddd|ddkr,tdurz tj|g|Ri|WSt y}ztd|WYd}~nd}~wwt|j|g|Ri|Sd|vr7|d|d<|r@|dd}|du}t|tvp^|duo^t|dup^t|ddu}|r|ddurn|d}n|d}d|vr|dd}nd}t|||||}|r|rt||fi|} |dd}| | j|g|Rd|i|S|dur|}t|} | dur|dst|d} | dur| jdkrt} | durt} | j|g|Ri|St|ddr|j}d|vr|dd}t|} | j|g|Ri|St|t r5t|j!t|j"ur2t#d |j"j$d!|j!j$d"|j"}t%t|jpBt|d#d}|duratt|t} | dura| j|g|Ri|S|dd}|dur|d$kr|d|vr||dd%}t|} | dur|dst|d} | dur| jdkrt} | durt} | j|g|Ri|Std&|j$d'd d(d tDd)*a Instantiate one of the tokenizer classes of the library from a pretrained model vocabulary. The tokenizer class to instantiate is selected based on the `model_type` property of the config object (either passed as an argument or loaded from `pretrained_model_name_or_path` if possible), or when it's missing, by falling back to using pattern matching on `pretrained_model_name_or_path`: List options Params: pretrained_model_name_or_path (`str` or `os.PathLike`): Can be either: - A string, the *model id* of a predefined tokenizer hosted inside a model repo on huggingface.co. - A path to a *directory* containing vocabulary files required by the tokenizer, for instance saved using the [`~PreTrainedTokenizer.save_pretrained`] method, e.g., `./my_model_directory/`. - A path or url to a single saved vocabulary file if and only if the tokenizer only requires a single vocabulary file (like Bert or XLNet), e.g.: `./my_model_directory/vocab.txt`. (Not applicable to all derived classes) inputs (additional positional arguments, *optional*): Will be passed along to the Tokenizer `__init__()` method. config ([`PreTrainedConfig`], *optional*) The configuration object used to determine the tokenizer class to instantiate. cache_dir (`str` or `os.PathLike`, *optional*): Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used. force_download (`bool`, *optional*, defaults to `False`): Whether or not to force the (re-)download the model weights and configuration files and override the cached versions if they exist. proxies (`dict[str, str]`, *optional*): A dictionary of proxy servers to use by protocol or endpoint, e.g., `{'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}`. The proxies are used on each request. revision (`str`, *optional*, defaults to `"main"`): The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so `revision` can be any identifier allowed by git. subfolder (`str`, *optional*): In case the relevant files are located inside a subfolder of the model repo on huggingface.co (e.g. for facebook/rag-token-base), specify it here. tokenizer_type (`str`, *optional*): Tokenizer type to be loaded. backend (`str`, *optional*, defaults to `"tokenizers"`): Backend to use for tokenization. Valid options are: - `"tokenizers"`: Use the HuggingFace tokenizers library backend (default) - `"sentencepiece"`: Use the SentencePiece backend trust_remote_code (`bool`, *optional*, defaults to `False`): Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to `True` for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine. kwargs (additional keyword arguments, *optional*): Will be passed to the Tokenizer `__init__()` method. Can be used to set special tokens like `bos_token`, `eos_token`, `unk_token`, `sep_token`, `pad_token`, `cls_token`, `mask_token`, `additional_special_tokens`. See parameters in the `__init__()` for more details. Examples: ```python >>> from transformers import AutoTokenizer >>> # Download vocabulary from huggingface.co and cache. >>> tokenizer = AutoTokenizer.from_pretrained("google-bert/bert-base-uncased") >>> # Download vocabulary from huggingface.co (user-uploaded) and cache. >>> tokenizer = AutoTokenizer.from_pretrained("dbmdz/bert-base-german-cased") >>> # If vocabulary files are in a directory (e.g. tokenizer was saved using *save_pretrained('./test/saved_model/')*) >>> # tokenizer = AutoTokenizer.from_pretrained("./test/bert_saved_model/") >>> # Download vocabulary from huggingface.co and define model-specific arguments >>> tokenizer = AutoTokenizer.from_pretrained("FacebookAI/roberta-base", add_prefix_space=True) >>> # Explicitly use the tokenizers backend >>> tokenizer = AutoTokenizer.from_pretrained("hf-internal-testing/llama-tokenizer", backend="tokenizers") >>> # Explicitly use the sentencepiece backend >>> tokenizer = AutoTokenizer.from_pretrained("hf-internal-testing/llama-tokenizer", backend="sentencepiece") ```configNT _from_autouse_fasttokenizer_typetrust_remote_code gguf_filezPassed `tokenizer_type` z3 does not exist. `tokenizer_type` should be one of z, css|]}|VqdSNrKrLcrKrKrO `sz0AutoTokenizer.from_pretrained..rkzTokenizer class z is not currently imported.F)return_tensorsrxauto_maprr}Fastz!Failed to use TokenizersBackend: rrrz-- code_revision PythonBackendPreTrainedTokenizerFastz The encoder model config class: z3 is different from the decoder model config class: z. It is not recommended to use the `AutoTokenizer.from_pretrained()` method in this case. Please use the encoder and decoder specific tokenizer classes. model_typerz!Unrecognized configuration class z8 to build an AutoTokenizer. Model type should be one of css|]}|jVqdSr)rlrrKrKrOrsrK)&poprrm ValueErrorjoinr|from_pretrainedrr r for_model Exceptionrrr isinstancer_listreplacerrdebugtypersr`rrregister_for_auto_classendswithrlrqrxrdecoderencoderwarning __class__r)clsr~inputsrr_rrrtokenizer_class_namerx gguf_path config_dictconfig_model_typetokenizer_configtokenizer_config_classtokenizer_auto_mapehas_remote_codehas_local_code class_ref upstream_repotokenizer_class_candidate_classrrKrKrOrs R                                      zAutoTokenizer.from_pretrainedNFcCs||dur|dur |}n |dur|}ntd|||fD] }|dur&|t|j<q|dur4|dur4|t|j<tj|||ddS)a Register a new tokenizer in this mapping. Args: config_class ([`PreTrainedConfig`]): The configuration corresponding to the model to register. tokenizer_class: The tokenizer class to register (V5 - preferred parameter). slow_tokenizer_class: (Deprecated) The slow tokenizer to register. fast_tokenizer_class: (Deprecated) The fast tokenizer to register. Nz$You need to pass a `tokenizer_class`)exist_ok)rrrlrrsregister) config_classrxslow_tokenizer_classfast_tokenizer_classr candidaterKrKrOrs  zAutoTokenizer.register)NNNF) rl __module__ __qualname____doc__r classmethodrrmrrr staticmethodrrKrKrKrOrsvrrs)NFNNNFr})>rrorVos collectionsrtypingrtransformers.utils.import_utilsrconfiguration_utilsrdynamic_module_utilsrrmodeling_gguf_pytorch_utilsr tokenization_utils_baser utilsr r r rr utils.hubrencoder_decoderr auto_factoryrconfiguration_autorrrrrtokenization_utils_tokenizersr tokenization_utils_sentencepiecer get_loggerrlrrdictstrr__annotations__rrmrsrnCONFIG_TO_TYPErZrdr|PathLikeboolrr__all__rKrKrKrOs2                  !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{}  !"#$%&'()*+,-./0123457<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqsy~       " .   `(