o -i@sUddlZddlZddlmZddlmZddlmZmZm Z ddl m Z ddl m Z ee BZeed<d ed efd d ZGd dde ZdS)N)Path) TypeAlias) AutoTokenizerPreTrainedTokenizerPreTrainedTokenizerFast))get_sentence_transformer_tokenizer_config) TokenizerLike HfTokenizer tokenizerreturncst}jjtttdr9t t tj Wdn1s4wYGfdddj }dj j |_ ||_ |S)z By default, transformers will recompute multiple tokenizer properties each time they are called, leading to a significant slowdown. This proxy caches these properties for faster access. vocab_sizeNcseZdZedeeffdd Zedeeffdd Zedeffdd Z de eefffdd Z deffd d Z fd d Z dS)z-get_cached_tokenizer..CachedTokenizerr cSNself)tokenizer_all_special_idsrO/home/ubuntu/veenaModal/venv/lib/python3.10/site-packages/vllm/tokenizers/hf.pyall_special_ids(z=get_cached_tokenizer..CachedTokenizer.all_special_idscrrrr)tokenizer_all_special_tokensrrall_special_tokens,rz@get_cached_tokenizer..CachedTokenizer.all_special_tokenscrrrr) max_token_idrrr0rz:get_cached_tokenizer..CachedTokenizer.max_token_idcrrrr)tokenizer_vocabrr get_vocab4z7get_cached_tokenizer..CachedTokenizer.get_vocabcrrrr) tokenizer_lenrr__len__7rz5get_cached_tokenizer..CachedTokenizer.__len__cs tffSr)get_cached_tokenizerr)r rr __reduce__:s z8get_cached_tokenizer..CachedTokenizer.__reduce__N)__name__ __module__ __qualname__propertylistintrstrrrdictrrr rrr rrrrrrCachedTokenizer'sr*Cached)copyrrrlenmaxvalueshasattr contextlibsuppressNotImplementedErrorr __class__r!)r cached_tokenizerr*rr)rrs     rc @sDeZdZedddddeeBdededBdedBdef d d ZdS) CachedHfTokenizerFN)trust_remote_coderevision download_dirpath_or_repo_idr7r8r9r c Osztj|g|R|||d|}Wn#ty5}z|s/dt|vs(dt|vr/d} t| ||d}~wwt||} t| trU| ddrUdd|j D} | | t |S) N)r7r8 cache_dirz,does not exist or is not currently imported.z*requires you to execute the tokenizer filezFailed to load the tokenizer. If the tokenizer is a custom tokenizer not yet available in the HuggingFace transformers library, consider setting `trust_remote_code=True` in LLM or using the `--trust-remote-code` flag in the CLI. do_lower_caseFcSsi|] \}}||qSr)lower).0kvrrr rsz5CachedHfTokenizer.from_pretrained..) rfrom_pretrained ValueErrorr' RuntimeErrorr isinstancer(getspecial_tokens_mapitemsadd_special_tokensr) clsr:r7r8r9argskwargsr eerr_msgencoder_configrGrrrrBDsB      z!CachedHfTokenizer.from_pretrained) r!r"r# classmethodr'rboolr rBrrrrr6Cs r6)r1r,pathlibrtypingr transformersrrrvllm.transformers_utils.configrprotocolr r __annotations__rr6rrrrs     2