o i@sUddlZddlZddlmZddlmZddlmZmZm Z ddl m Z ddl m Z ee BZeed<d ed efd d ZGd dde ZdS)N)Path) TypeAlias) AutoTokenizerPreTrainedTokenizerPreTrainedTokenizerFast))get_sentence_transformer_tokenizer_config) TokenizerLike HfTokenizer tokenizerreturncst}jjtttddDtdrBt t tj Wdn1s=wYGfdddj }dj j |_ ||_ |S)z By default, transformers will recompute multiple tokenizer properties each time they are called, leading to a significant slowdown. This proxy caches these properties for faster access. css|]}t|VqdSN)len).0tokrH/home/ubuntu/vllm_env/lib/python3.10/site-packages/vllm/tokenizers/hf.py sz'get_cached_tokenizer.. vocab_sizeNcseZdZedeeffdd Zedeeffdd Zedeffdd Z edeffdd Z de eefffd d Z deffd d Z fddZdS)z-get_cached_tokenizer..CachedTokenizerr cSr rself)tokenizer_all_special_idsrrall_special_ids*z=get_cached_tokenizer..CachedTokenizer.all_special_idscrr rr)tokenizer_all_special_tokensrrall_special_tokens.rz@get_cached_tokenizer..CachedTokenizer.all_special_tokenscrr rr) max_token_idrrr2rz:get_cached_tokenizer..CachedTokenizer.max_token_idcrr rr)max_chars_per_tokenrrr6rzAget_cached_tokenizer..CachedTokenizer.max_chars_per_tokencrr rr)tokenizer_vocabrr get_vocab:z7get_cached_tokenizer..CachedTokenizer.get_vocabcrr rr) tokenizer_lenrr__len__=r!z5get_cached_tokenizer..CachedTokenizer.__len__cs tffSr )get_cached_tokenizerr)r rr __reduce__@s z8get_cached_tokenizer..CachedTokenizer.__reduce__N)__name__ __module__ __qualname__propertylistintrstrrrrdictr r#r%rrrr rrr"rrrCachedTokenizer)sr/Cached)copyrrr rmaxvalueshasattr contextlibsuppressNotImplementedErrorr __class__r&)r cached_tokenizerr/rr.rr$s    "r$c @sDeZdZedddddeeBdededBdedBdef d d ZdS) CachedHfTokenizerFN)trust_remote_coderevision download_dirpath_or_repo_idr;r<r=r c Osztj|g|R|||d|}Wn#ty5}z|s/dt|vs(dt|vr/d} t| ||d}~wwt||} t| trU| ddrUdd|j D} | | t |S) N)r;r< cache_dirz,does not exist or is not currently imported.z*requires you to execute the tokenizer filezFailed to load the tokenizer. If the tokenizer is a custom tokenizer not yet available in the HuggingFace transformers library, consider setting `trust_remote_code=True` in LLM or using the `--trust-remote-code` flag in the CLI. do_lower_caseFcSsi|] \}}||qSr)lower)rkvrrr xsz5CachedHfTokenizer.from_pretrained..) rfrom_pretrained ValueErrorr, RuntimeErrorr isinstancer-getspecial_tokens_mapitemsadd_special_tokensr$) clsr>r;r<r=argskwargsr eerr_msgencoder_configrJrrrrEJsB      z!CachedHfTokenizer.from_pretrained) r&r'r( classmethodr,rboolr rErrrrr:Is r:)r5r1pathlibrtypingr transformersrrrvllm.transformers_utils.configrprotocolr r __annotations__r$r:rrrrs     8