o c۷i%@sddlZddlZddlZddlmZddlmZddlZddl m Z ddl m Z m Z ddlmZmZmZddlmZmZddlmZejd ed d ejd ed d e r[dd lmZdeeBdefddZddeeBdedefddZdeeBdedBfddZGdddeZ dS)N)cached_property)Path)TokenizerException)assert_sentencepiece_installedis_sentencepiece_installed)SpecialTokenPolicy TokenizerTokenizerVersion) ImageConfigMultiModalVersion)ModelSettingsBuilderoncez%.*`get_control_token` is deprecated.*)actioncategorymessagez#.*`_control_tokens` is deprecated.*)SentencePieceProcessorpathreturncsbttr tttj}ttjdgfdd|Ddg}o0tfdd|DS)z1Check if the given path is a SentencePiece model.cs$g|]}D] }d||qqS)z.model.).0vm) mm_versionsrd/home/ubuntu/vllm_env/lib/python3.10/site-packages/mistral_common/tokens/tokenizers/sentencepiece.py *s$z$is_sentencepiece..z.modelc3s|] }j|VqdSN)nameendswith)rsuffix)rrr ,sz#is_sentencepiece..) isinstancestrrlistr __members__r is_fileany)rinstruct_versionssuffixesr)rrris_sentencepiece#s  r)Ftokenizer_filenameraise_deprecatedcCspt|}|dd}|dkr|dd}|dkr(|r$td|dtdS|tjvr4td |t|S) z3Get the version of the tokenizer from the filename..modelrrz4Make sure to rename your tokenizer file to end with z.v1.v1!Unrecognized tokenizer filename: )r"splitrr r$)r*r+ _version_strrrrget_spm_version/s r3cCs^t|}|dd}|dksd|vrdSd|dd}|tjvr*td|t|jS)z1Get the image config from the tokenizer filename.r,r-r.rNr0)r"r1r r$rconfig)r*r2_mm_version_strrrrget_image_configDs  r6c seZdZdZd7deeBdedBddffdd Zedefdd Z edefd d Z ede dBfd d Z dede fddZdede fddZede fddZede fddZdeefddZede fddZede fddZde ejBeBdefddZedee fd d!Zedee fd"d#Zded$ed%edee fd&d'Zej fd(ee d)edefd*d+Z!d,e defd-d.Z"d(ee d)edefd/d0Z#d(ee defd1d2Z$ede fd3d4Z%ede fd5d6Z&Z'S)8SentencePieceTokenizerzC[SentencePiece](https://github.com/google/sentencepiece) tokenizer.N model_pathtokenizer_versionrcsttjj_tj|sJ|t t |t r|n| d_ j j ks0JfddtjD_|pDt|dd_t|_tdS)zInitialize the `SentencePieceTokenizer`. Args: model_path: The path to the `SentencePiece` model. tokenizer_version: The version of the tokenizer. If not provided, it will be inferred from the model path. ) model_filecsg|]}j|qSr_model id_to_piece)riselfrrrhsz3SentencePieceTokenizer.__init__..F)r+N)rlogging getLogger __class____name___loggerosrisfilerr!r"as_posixr< vocab_sizeget_piece_sizerangen_words_vocabr3_versionr _file_pathsuper__init__)r@r8r9rCr?rrQWs zSentencePieceTokenizer.__init__cC|jS)z The path to the tokenizer model.)rOr?rrr file_pathoz SentencePieceTokenizer.file_pathcCrS)zThe version of the tokenizer.)rNr?rrrversiontrUzSentencePieceTokenizer.versioncCs|jjr td|jdS)zOAlways returns None as SentencePiece does not support `model_settings_builder`.zCSentencePieceTokenizer does not support model settings for version N)rVsupports_model_settings ValueErrorr?rrrmodel_settings_builderysz-SentencePieceTokenizer.model_settings_builderscC |j|S)z+Get the special token for the given string.)r< piece_to_idr@rZrrrget_special_token z(SentencePieceTokenizer.get_special_tokencCstdt||S)NzC`get_control_token` is deprecated. Use `get_special_token` instead.)warningswarn FutureWarningr^r]rrrget_control_tokens  z(SentencePieceTokenizer.get_control_tokencC |jS)z!Vocabulary size of the tokenizer.)r<rIr?rrrrL zSentencePieceTokenizer.n_wordscCs t|jS)z.The number of special tokens of the tokenizer.)len special_idsr?rrrnum_special_tokensrez)SentencePieceTokenizer.num_special_tokenscCrS)z,Get all tokens in the vocabulary as strings.)rMr?rrrvocabszSentencePieceTokenizer.vocabcCrd)z#The beginning of sentence token id.)r<bos_idr?rrrrjrezSentencePieceTokenizer.bos_idcCrd)zThe end of sentence token id.)r<eos_idr?rrrrkrezSentencePieceTokenizer.eos_idtokencCsVt|ttjfr|jt|St|tr!|j|}|j|Stdt |j )z7Return `True` if the passed `token` is a special token.zExpected int or str, got ) r!intnpintegerr< IsControlr"r\ TypeErrortyperD)r@rl token_intrrr is_specials    z!SentencePieceTokenizer.is_specialcCstdt|jS)NzS`_control_tokens` is deprecated. Make use of `is_special` or `special_ids` instead.)r`rarbrgr?rrr_control_tokenssz&SentencePieceTokenizer._control_tokenscsfddtjDS)zIds of the special tokens.csh|] }j|r|qSr)r<rprtokr?rr sz5SentencePieceTokenizer.special_ids..)rKrLr?rr?rrgsz"SentencePieceTokenizer.special_idsboseoscCs@t|tsJ|j|}|r|jg|}|rg||j}|S)aEncode the given string into a list of token ids. Args: s: The string to encode. bos: Whether to add the beginning of sentence token. eos: Whether to add the end of sentence token. Returns: The list of token ids. )r!r"r<encoderjrk)r@rZryrztrrrr{s  zSentencePieceTokenizer.encodetokensspecial_token_policycCsJt|ttfstdt|d|tjtjfvr|||S|j |S)aDecode the given list of token ids into a string. Note: Using `special_token_policy=SpecialTokenPolicy.KEEP` will keep the special tokens and the normal tokens as SentencePiece pieces. Args: tokens: The list of token ids. special_token_policy: The policy to use for special tokens. Returns: The decoded string. z@Expected `special_token_policy` to be a SpecialTokenPolicy, got r,) r!r"rrXrrKEEPRAISE_decode_with_special_tokensr<decode)r@r}r~rrrrs  zSentencePieceTokenizer.decodetoken_idcCr[)z,Convert the given token id to a token piece.r;)r@rrrrr=r_z"SentencePieceTokenizer.id_to_piececsg}g}|D].}|r/|tjkrtd|r&|fdd|Dg}||q||q|rC|fdd|Dd|S)NzNDecoding `tokens` that contain special tokens with special_token_policy=RAISE.cg|]}|qSrr=rvr?rrrzFSentencePieceTokenizer._decode_with_special_tokens..crrrrvr?rrrrr)rtrrrXextendappendr=join)r@r}r~ text_list curr_tokensrwrr?rrs    z2SentencePieceTokenizer._decode_with_special_tokenscCs|j|tjdS)N)r~)rrr)r@r}rrr _to_stringsz!SentencePieceTokenizer._to_stringcCrd)zThe padding token id.)r<pad_idr?rrrrrezSentencePieceTokenizer.pad_idcCrd)zThe unknown token id.)r<unk_idr?rrrrrezSentencePieceTokenizer.unk_idr)(rD __module__ __qualname____doc__r"rr rQpropertyrTrVr rYrmr^rcrLrhr#rirrjrkrnroboolrtsetrurgr{rIGNORErr=rrrr __classcell__rrrRrr7TsD$  r7)F)!rArFr` functoolsrpathlibrnumpyrnmistral_common.exceptionsrmistral_common.importsrr%mistral_common.tokens.tokenizers.baserrr &mistral_common.tokens.tokenizers.imager r 7mistral_common.tokens.tokenizers.model_settings_builderr filterwarningsrb sentencepiecerr"rr)r3r6r7rrrrs6