o c۷iS @sNddlZddlZddlZddlZddlmZddlmZddlm Z ddl m Z ddl Z ddlZddlmZmZddlmZmZmZmZddlmZdd lmZejd ed d eeZd e e Bde!fddZ"Gddde Z#Gddde Z$Gddde Z%Gddde Z&GdddeZ' dde(e#de)dBde*e+e)ffddZ,dS) N)cached_property)groupby)Path) TypedDict) AudioConfigAudioSpectrogramConfig)SpecialTokenPolicy SpecialTokens TokenizerTokenizerVersion) ImageConfig)ModelSettingsBuilderoncez%.*`get_control_token` is deprecated.*)actioncategorymessagepathreturncCs.t|tr t|}|od|jvo|jdkS)z3Check if the given path is a tekken tokenizer file.tekken.json) isinstancestrris_filenamesuffix)rr]/home/ubuntu/vllm_env/lib/python3.10/site-packages/mistral_common/tokens/tokenizers/tekken.py is_tekken!s rc@s.eZdZUdZeed<eed<edBed<dS) TokenInfozToken information in the JSON file. Attributes: rank: The rank of the token. token_bytes: The token in bytes, base64 encoded. token_str: The token in string format. rank token_bytesN token_str)__name__ __module__ __qualname____doc__int__annotations__rrrrrr)s rc@s*eZdZUdZeed<eed<eed<dS)SpecialTokenInfozSpecial token information in the JSON file. Attributes: rank: The rank of the token. token_str: The token in string format. is_control: Whether the token is a control token. rr! is_controlN)r"r#r$r%r&r'rboolrrrrr(7s  r(c@s:eZdZUdZeed<eed<eed<eed<eed<dS) TekkenConfigaXTekken configuration in the JSON file. Attributes: pattern: The pattern of the tokenizer. num_vocab_tokens: The number of vocabulary tokens. default_vocab_size: The default vocabulary size. default_num_special_tokens: The default number of special tokens. version: The version of the tokenizer. patternnum_vocab_tokensdefault_vocab_sizedefault_num_special_tokensversionN)r"r#r$r%rr'r&rrrrr+Es   r+c@sVeZdZUdZeeed<eedBed<eed<e ed<e ed<e ed<e ed <dS) ModelDataa2The data of the tekken tokenizer model. Attributes: vocab: The vocabulary of the tokenizer. config: The configuration of the tokenizer. version: The version of the tokenizer. type: The type of the tokenizer. image: The image configuration of the tokenizer. vocabNspecial_tokensconfigr0typeimageaudio) r"r#r$r%listrr'r(r+r&rr rrrrrr1Ws   r1c@seZdZdZedejddedejddedejddedej ddedej dded ej dded ej dded ej dded ejdded ejddedejddedejddedejddedejddedejddedejddedejddedejddedejddedejddfZdZdddddddeedeededed ed!e d"ed#ee!BdBd$e"dBd%e#dBd&e$dBfd'd(Z%e&d)e!fd*d+Z'e&d)e$dBfd,d-Z(e)d.e*dd/ee!Bd)dfd0d1Z+e&d)e"dBfd2d3Z,e,j-d4e"d)dfd5d3Z,e&d)e#dBfd6d7Z.e.j-d4e#d)dfd8d7Z.e&d)efd9d:Z/e&d)efd;d<Z0e1d)e2efd=d>Z3e&d)e fd?d@Z4e1d)efdAdBZ5e1d)efdCdDZ6e1d)efdEdFZ7e1d)efdGdHZ8d)eefdIdJZ9dKedLe:dMe:d)eefdNdOZ;dPeedQedKed)efdWdXZ?dYee@jABeBd)e:fdZd[ZBdKed)efd\d]ZCe tekkenizerN)r_path image_config audio_configmodel_settings_builderr2r3r, vocab_sizenum_special_tokensr0rrOrPrQrRcs|js| durtd|d| |t||ks"J|t||f|_ttdd|D} t|| ks=Jd|t||ksEJfddtt||D} | rhtd| d d d | d d || }ttd d|Dt|kr|ksJ|J|||}td|d|dt||d_ tt|tj ksJ|j ft j ||j id_ |_| _| _|_dd|D_dd|D_fddt|D_tj_|durt|nd_| _dS)aInitialize the tekken tokenizer. Args: vocab: The vocabulary of the tokenizer. special_tokens: The special tokens of the tokenizer. pattern: The pattern of the tokenizer. vocab_size: The vocabulary size of the tokenizer. num_special_tokens: The number of special tokens of the tokenizer. version: The version of the tokenizer. name: The name of the tokenizer. image_config: The image configuration of the tokenizer. audio_config: The audio configuration of the tokenizer. model_settings_builder: The builder for model settings, or None if unsupported. N4model_settings_builder is not supported for version= but got model_settings_builder=cSg|]}|dqSr!r.0trrr z'Tekkenizer.__init__..zSpecial tokens must be unique: cs$g|]}t|jj|dddqS))idTr:)r(SPECIAL_TOKEN_TEMPLATEformatrZiselfrrr\szAdding special tokens rr!z, ..., cSrWrXrrYrrrr\r]zNon special vocabulary size is z with z special tokens.) max_vocab)rpat_strmergeable_ranksr3cSh|]}|dqSrrrYrrr r]z&Tekkenizer.__init__..cSsi|] }|d|dqS)r!rrrYrrr sz'Tekkenizer.__init__..csg|]}|qSr) id_to_piecerarcrrr\)supports_model_settings ValueErrorlen _vocab_sizesetrangeloggerinfo_reload_mergeable_ranks_tekken_token2id_nospecialvaluestiktokenEncoding_model_version _image_config _audio_config_all_special_tokens_special_token_ids_special_tokens_reverse_vocab_vocabrIGNORE_special_token_policyr _file_path_model_settings_builder)rdr2r3r,rSrTr0rrOrPrQrRnum_defined_special_tokensspecial_fillerinner_vocab_sizerrcr__init__sb  * zTekkenizer.__init__rcCs|jdur td|jS)zThe path to the tokenizer file.Nz)The tokenizer was not loaded from a file.)rrprcrrr file_paths zTekkenizer.file_pathcC|jS)zCThe model settings builder, or None if unsupported by this version.)rrcrrrrRz!Tekkenizer.model_settings_builderclsrc Cs(t|tr t|}|sJ|t|ddd }t|}Wdn1s(wY|dd}|tj vrIt d|d|d t tj |dusOJt|}|d d}|durp|tj krjt d |d t t j}nd d|D}||d <|d}r|tjkrt d|d|jdtd!i||d<n|d} rtd!i| |d<|d} r| d} td!i| } td!d| i| |d<|d} dur|jst d|d| | durt| } |} || d|| dd| dd| dd||jdd| d| d| |d S)"zLoad the tekken tokenizer from a file. Args: path: The path to the tokenizer file. Returns: The tekken tokenizer. rutf-8)encodingNr4r0zUnknown version: z in z+. Make sure to use a valid version string: r3zSpecial tokens not found in zL. Please update your tokenizer file and include all special tokens you need.cSsg|]}|qSrrrZtokenrrrr\sz(Tekkenizer.from_file.. multimodalz-The image config has to be called 'image' in z for tokenizers of version .r6r7audio_encoding_configencoding_configrRrUrVr2r,r.r/r) r2r3r,rSrTr0rrPrQrRrOr)rrrexistsopenjsonloadgetr __members__rpr8v7r9DEPRECATED_SPECIAL_TOKENSv11valuer poprrror model_validaterreplace)rrfuntyped _version_strr0special_tokens_dictsr3mmr6r7rrR model_datarrr from_filesx                zTekkenizer.from_filecCr)z)The image configuration of the tokenizer.)r~rcrrrr6DrzTekkenizer.imagercCtd)Nz!Can only set Image config at initrprdrrrrr6IcCr)zThe audio configuration of the tokenizer. Returns: The audio configuration object if it exists, otherwise None. )rrcrrrr7MszTekkenizer.audiocCr)Nz!Can only set Audio config at initrrrrrr7VrcCs t|jS)z.The number of special tokens of the tokenizer.)rqrrcrrrrTZ zTekkenizer.num_special_tokenscCr)z!Vocabulary size of the tokenizer.)rrrcrrrn_words_rzTekkenizer.n_wordscCsdd|jDS)zIds of the special tokens.cSrirjrrrrrrkgr]z)Tekkenizer.special_ids..rrcrrr special_idsdszTekkenizer.special_idscCr)zThe version of the tokenizer.)r}rcrrrr0irzTekkenizer.versioncC |dS)z#The beginning of sentence token id.zget_special_tokenrcrrrbos_idnrzTekkenizer.bos_idcCr)zThe end of sentence token id.zrrcrrreos_idsrzTekkenizer.eos_idcCr)zThe padding token id.zrrcrrrpad_idxrzTekkenizer.pad_idcCr)zThe unknown token id.zrrcrrrunk_id}rzTekkenizer.unk_idcCr)a6Get all tokens in the vocabulary as strings. Note: This will collapse all tokens for which we have a decoding error into the string. This is bad and results in things like len(set(vocab)) != len(vocab)). Returns: The vocabulary of the tokenizer. )rrcrrrr2szTekkenizer.vocabsboseoscsDj|}fdd|D}|rjg|}|r g|j}|S)aEncode a string into a list of token ids. Args: s: The string to encode. bos: Whether to add the beginning of sentence token. eos: Whether to add the end of sentence token. Returns: The list of token ids. csg|]}|jqSrrTrYrcrrr\rnz%Tekkenizer.encode..)r|encoderr)rdrrrtokensrrcrrs  zTekkenizer.encoderspecial_token_policycsg}t|fddD]>\}}|r9|tjkr tdt|d|tjkr2|fdd|Dq |tjkr8q q |j fdd|Dq |S) Ncs |jkSNr)r[rcrrs z(Tekkenizer._decode_all..z/Decoding `tokens` that contain special tokens (a) is not allowed. Either make sure `tokens` do not include any special tokens or, if you want to decode `tokens` that includes special tokens, change the tokenizer's special token policy to IGNORE or KEEP: ``` from mistral_common.tokens.tokenizers.mistral import MistralTokenizer from mistral_common.tokens.tokenizers.tekken import SpecialTokenPolicy tokenizer = MistralTokenizer.v3(is_tekken=True) tekken = tokenizer.instruct_tokenizer.tokenizer tekken.special_token_policy = SpecialTokenPolicy.IGNORE # or SpecialTokenPolicy.KEEP ```c3s|] }j|dVqdS)r!NrrYrcrr sz)Tekkenizer._decode_all..csg|]}|jqSrrrYrcrrr\rnz*Tekkenizer._decode_all..) rrRAISErpr8KEEPextendrappendr|decode)rdrrdecoded is_specialgrouprrcr _decode_alls   "zTekkenizer._decode_alltoken_idcCsd||jko dkSS)z$Check if a token id is a byte token.rrrdrrrris_byteszTekkenizer.is_bytecCs"||jvr |j|Std|)z$Get the token id of a special token.zUnknown control token )rrprdrrrrrs  zTekkenizer.get_special_tokenrcCsBt|ttjfr ||jvSt|tr||jvStdt|j )z7Return `True` if the passed `token` is a special token.zExpected int or str, got ) rr&npintegerrrr TypeErrorr5r")rdrrrrrs    zTekkenizer.is_specialcCstdt||S)NzC`get_control_token` is deprecated. Use `get_special_token` instead.)warningswarn FutureWarningrrrrrget_control_tokens  zTekkenizer.get_control_tokencCs6t|ttfstdt|dd|j||dS)zDecode a list of token ids into a string. Args: tokens: The list of token ids to decode. special_token_policy: The policy for handling special tokens. Returns: The decoded string. z>Expected `special_token_policy` to be SpecialTokenPolicy, got rrr)rrrrpr5joinr)rdrrrrrrs  zTekkenizer.decodecCs|j|tjdS)Nrrrr)rdrrrr _to_stringszTekkenizer._to_stringcCs|j|gtjdS)z0Convert a token id to its string representation.rrrrrrrmszTekkenizer.id_to_piececCsn||jkr.|tjkr|j|ddS|tjkr t|d|tjkr'dStd||j ||jS)aConvert a token id to its byte representation. Args: token_id: The token id to convert. special_token_policy: The policy for handling special tokens. Returns: The byte representation of the token. r!rz is a special tokenzUnknown special token policy ) rTrrrrrrprr|decode_single_token_bytes)rdrrrrrid_to_byte_pieces   zTekkenizer.id_to_byte_piece)Jr"r#r$r%r(r unkrr begin_instend_inst begin_tools end_toolsbegin_tool_resultsend_tool_results tool_callsimgpad img_breakimg_endprefixmiddler begin_system end_systembegin_tool_contentrr_r8rrr&r rr rr rpropertyrrR classmethodr5rr6setterr7rTrrrsrr0rrrrr2r*rrrrrrrrrrrrrmbytesrrrrrr9ks      Y Q r9r2rfcCs|dur)t||ksJt||ft||kr)|d|}tdt|di}t|D]4\}}|hdks=J|d|ksEJt|d}|dks]|t|gks]J||f|d||<q/t|t|ksnJt| tt t|ks~J|S)zAReload our tokenizer JSON file and convert it to Tiktoken format.Nz(Cutting non special vocabulary to first z tokens.>rr!r rr r) rqrurv enumeratekeysbase64 b64decoderrsryrt)r2rfranksrbxmergerrrrw s  " rwr)-rrloggingr functoolsr itertoolsrpathlibrtypingrnumpyrrz&mistral_common.tokens.tokenizers.audiorr%mistral_common.tokens.tokenizers.baserr r r &mistral_common.tokens.tokenizers.imager 7mistral_common.tokens.tokenizers.model_settings_builderr filterwarningsr getLoggerr"rurr*rrr(r+r1r9r8r&dictrrwrrrrsJ       %