o *i_Z @sBddlZddlZddlZddlZddlmZddlmZddlm Z ddl m Z ddl Z ddlZddlmZmZddlmZmZmZmZddlmZejd ed d eeZd ee Bd efddZ Gddde Z!Gddde Z"Gddde Z#Gddde Z$GdddeZ% dde&e!de'dBd e(e)e'ffddZ*dS)N)cached_property)groupby)Path) TypedDict) AudioConfigAudioSpectrogramConfig)SpecialTokenPolicy SpecialTokens TokenizerTokenizerVersion) ImageConfigoncez%.*`get_control_token` is deprecated.*)actioncategorymessagepathreturncCs.t|tr t|}|od|jvo|jdkS)z3Check if the given path is a tekken tokenizer file.tekken.json) isinstancestrris_filenamesuffix)rrd/home/ubuntu/veenaModal/venv/lib/python3.10/site-packages/mistral_common/tokens/tokenizers/tekken.py is_tekken s rc@s.eZdZUdZeed<eed<edBed<dS) TokenInfozToken information in the JSON file. Attributes: rank: The rank of the token. token_bytes: The token in bytes, base64 encoded. token_str: The token in string format. rank token_bytesN token_str)__name__ __module__ __qualname____doc__int__annotations__rrrrrr(s rc@s*eZdZUdZeed<eed<eed<dS)SpecialTokenInfozSpecial token information in the JSON file. Attributes: rank: The rank of the token. token_str: The token in string format. is_control: Whether the token is a control token. rr is_controlN)r!r"r#r$r%r&rboolrrrrr'6s  r'c@s:eZdZUdZeed<eed<eed<eed<eed<dS) TekkenConfigaXTekken configuration in the JSON file. Attributes: pattern: The pattern of the tokenizer. num_vocab_tokens: The number of vocabulary tokens. default_vocab_size: The default vocabulary size. default_num_special_tokens: The default number of special tokens. version: The version of the tokenizer. patternnum_vocab_tokensdefault_vocab_sizedefault_num_special_tokensversionN)r!r"r#r$rr&r%rrrrr*Ds   r*c@sVeZdZUdZeeed<eedBed<eed<e ed<e ed<e ed<e ed <dS) ModelDataa2The data of the tekken tokenizer model. Attributes: vocab: The vocabulary of the tokenizer. config: The configuration of the tokenizer. version: The version of the tokenizer. type: The type of the tokenizer. image: The image configuration of the tokenizer. vocabNspecial_tokensconfigr/typeimageaudio) r!r"r#r$listrr&r'r*r%rr rrrrrr0Vs   r0c@seZdZdZedejddedejddedejddedej ddedej dded ej dded ej dded ej dded ejdded ejddedejddedejddedejddedejddedejddedejddedejddedejddedejddedejddfZdZddddddeedeededed ed!e d"ed#ee!BdBd$e"dBd%e#dBfd&d'Z$e%d(e!fd)d*Z&e'd+e(dd,ee!Bd(dfd-d.Z)e%d(e"dBfd/d0Z*e*j+d1e"d(dfd2d0Z*e%d(e#dBfd3d4Z,e,j+d1e#d(dfd5d4Z,e%d(efd6d7Z-e%d(efd8d9Z.e/d(e0efd:d;Z1e%d(e fdd?Z4e4j+d@e3d(dfdAd?Z4e/d(efdBdCZ5e/d(efdDdEZ6e/d(efdFdGZ7e/d(efdHdIZ8d(eefdJdKZ9dLedMe:dNe:d(eefdOdPZ;dQeedRe3d(eefdSdTZdZee?j@BeBd(e:fd[d\ZAdLed(efd]d^ZBdidQeedRe3dBd(efd_d`ZCdQeed(efdadbZDdQeed(efdcddZEdUed(efdedfZFdidUedRe3dBd(eGfdgdhZHdS)j TekkenizerzTekken tokenizer. This tokenizer is based on the [tiktoken](https://github.com/openai/tiktoken) library. It fastens the tokenization for multiple languages. rTrr r( z tekkenizerN)r_path image_config audio_configr1r2r+ vocab_sizenum_special_tokensr/rrNrOrPcs|t||ksJ|t||f|_ttdd|D} t|| ks,Jd|t||ks4Jfddtt||D} | rWtd| ddd| d d|| }ttd d|Dt|krs|ksxJ|J|||} td | d |d t|| d_tt| tjksJ| jft j ||jid_ |_ | _ | _|_dd|D_dd|D_fddt|D_tj_|durt|_dSd_dS)aInitialize the tekken tokenizer. Args: vocab: The vocabulary of the tokenizer. special_tokens: The special tokens of the tokenizer. pattern: The pattern of the tokenizer. vocab_size: The vocabulary size of the tokenizer. num_special_tokens: The number of special tokens of the tokenizer. version: The version of the tokenizer. name: The name of the tokenizer. image_config: The image configuration of the tokenizer. cSg|]}|dqSr r.0trrr z'Tekkenizer.__init__..zSpecial tokens must be unique: cs$g|]}t|jj|dddqS))idTr9)r'SPECIAL_TOKEN_TEMPLATEformatrViselfrrrXszAdding special tokens rr z, ..., cSrSrTrrUrrrrXrYzNon special vocabulary size is z with z special tokens.) max_vocab)rpat_strmergeable_ranksr2cSh|]}|dqSrrrUrrr rYz&Tekkenizer.__init__..cSsi|] }|d|dqS)r rrrUrrr sz'Tekkenizer.__init__..csg|]}|qSr) id_to_piecer]r_rrrXN)len _vocab_sizesetrangeloggerinfo_reload_mergeable_ranks_tekken_token2id_nospecialvaluestiktokenEncoding_model_version _image_config _audio_config_all_special_tokens_special_token_ids_special_tokens_reverse_vocab_vocabrIGNORE_special_token_policyr _file_path)r`r1r2r+rQrRr/rrNrOrPnum_defined_special_tokensspecial_fillerinner_vocab_sizerr_r__init__sX  * zTekkenizer.__init__rcCs|jdur td|jS)zThe path to the tokenizer file.Nz)The tokenizer was not loaded from a file.)r ValueErrorr_rrr file_paths zTekkenizer.file_pathclsrc Cst|tr t|}|sJ|t|ddd }t|}Wdn1s(wY|dd}|tj vrIt d|d|d t tj |dusOJt|}|d d}|durq|td krkt d |d t t j }ndd|D}||d <|d}r|tdkrt d|d|jdtd i||d<n|d} rtd i| |d<|d} r| d} td i| } td d| i| |d<|} || d|| dd| dd| dd||jdd| d| d|d S)!zLoad the tekken tokenizer from a file. Args: path: The path to the tokenizer file. Returns: The tekken tokenizer. rutf-8)encodingNr3r/zUnknown version: z in z+. Make sure to use a valid version string: r2v7zSpecial tokens not found in zL. Please update your tokenizer file and include all special tokens you need.cSsg|]}|qSrrrVtokenrrrrXsz(Tekkenizer.from_file.. multimodalv11z-The image config has to be called 'image' in z for tokenizers of version .r5r6audio_encoding_configencoding_configr1r+r-r.r) r1r2r+rQrRr/rrOrPrNr)rrrexistsopenjsonloadgetr __members__rr7r8DEPRECATED_SPECIAL_TOKENSvaluer poprrrreplace) rrfuntyped _version_strr/special_tokens_dictsr2mmr5r6r model_datarrr from_filesb              zTekkenizer.from_filecC|jS)z)The image configuration of the tokenizer.)rxr_rrrr5+zTekkenizer.imagercCtd)Nz!Can only set Image config at initrr`rrrrr50cCr)zThe audio configuration of the tokenizer. Returns: The audio configuration object if it exists, otherwise None. )ryr_rrrr64szTekkenizer.audiocCr)Nz!Can only set Audio config at initrrrrrr6=rcCs t|jS)z.The number of special tokens of the tokenizer.)rkrzr_rrrrRA zTekkenizer.num_special_tokenscCr)z!Vocabulary size of the tokenizer.)rlr_rrrn_wordsFrzTekkenizer.n_wordscCsdd|jDS)zIds of the special tokens.cSrerfrrrrrrgNrYz)Tekkenizer.special_ids..rzr_rrr special_idsKszTekkenizer.special_idscCr)zThe version of the tokenizer.)rwr_rrrr/PrzTekkenizer.versioncCr)z'The policy for handling special tokens.)rr_rrrspecial_token_policyUrzTekkenizer.special_token_policypolicycCs4t|tstdt|dtdt||_dS)z+Set the policy for handling special tokens.z!Expected SpecialTokenPolicy, got rzThe attributed `special_token_policy` is deprecated and will be removed in 1.10.0. Please pass a special token policy explicitly to the relevant methods.N)rrrr4warningswarn FutureWarningr)r`rrrrrZs  cC |dS)z#The beginning of sentence token id.zget_special_tokenr_rrrbos_idjrzTekkenizer.bos_idcCr)zThe end of sentence token id.zrr_rrreos_idorzTekkenizer.eos_idcCr)zThe padding token id.zrr_rrrpad_idtrzTekkenizer.pad_idcCr)zThe unknown token id.zrr_rrrunk_idyrzTekkenizer.unk_idcCr)a6Get all tokens in the vocabulary as strings. Note: This will collapse all tokens for which we have a decoding error into the string. This is bad and results in things like len(set(vocab)) != len(vocab)). Returns: The vocabulary of the tokenizer. )r}r_rrrr1~szTekkenizer.vocabsboseoscsDj|}fdd|D}|rjg|}|r g|j}|S)aEncode a string into a list of token ids. Args: s: The string to encode. bos: Whether to add the beginning of sentence token. eos: Whether to add the end of sentence token. Returns: The list of token ids. csg|]}|jqSrrRrUr_rrrXrjz%Tekkenizer.encode..)rvencoderr)r`rrrtokensrr_rrs  zTekkenizer.encoderrcsg}t|fddD]>\}}|r9|tjkr tdt|d|tjkr2|fdd|Dq |tjkr8q q |j fdd|Dq |S) Ncs |jkSNr)rWr_rrs z(Tekkenizer._decode_all..z/Decoding `tokens` that contain special tokens (a) is not allowed. Either make sure `tokens` do not include any special tokens or, if you want to decode `tokens` that includes special tokens, change the tokenizer's special token policy to IGNORE or KEEP: ``` from mistral_common.tokens.tokenizers.mistral import MistralTokenizer from mistral_common.tokens.tokenizers.tekken import SpecialTokenPolicy tokenizer = MistralTokenizer.v3(is_tekken=True) tekken = tokenizer.instruct_tokenizer.tokenizer tekken.special_token_policy = SpecialTokenPolicy.IGNORE # or SpecialTokenPolicy.KEEP ```c3s|] }j|dVqdS)r NrrUr_rr sz)Tekkenizer._decode_all..csg|]}|jqSrrrUr_rrrXrjz*Tekkenizer._decode_all..) rrRAISErr7KEEPextendr~appendrvdecode)r`rrdecoded is_specialgrouprr_r _decode_alls   "zTekkenizer._decode_alltoken_idcCsd||jko dkSS)z$Check if a token id is a byte token.rrr`rrrris_byteszTekkenizer.is_bytecCs"||jvr |j|Std|)z$Get the token id of a special token.zUnknown control token )r|rr`rrrrrs  zTekkenizer.get_special_tokenrcCsBt|ttjfr ||jvSt|tr||jvStdt|j )z7Return `True` if the passed `token` is a special token.zExpected int or str, got ) rr%npintegerr{rr| TypeErrorr4r!)r`rrrrrs    zTekkenizer.is_specialcCtdt||S)NzC`get_control_token` is deprecated. Use `get_special_token` instead.)rrrrrrrrget_control_tokens  zTekkenizer.get_control_tokencCsb|durt|ttfstdt|d|dur'td|jdt|j}d |j ||dS)aDecode a list of token ids into a string. Args: tokens: The list of token ids to decode. special_token_policy: The policy for handling special tokens. Use the tokenizer's [attribute][mistral_common.tokens.tokenizers.tekken.Tekkenizer.special_token_policy] if `None`. Passing `None` is deprecated and will be changed to `SpecialTokenPolicy.IGNORE` in `mistral_common=1.10.0`. Returns: The decoded string. NzFExpected `special_token_policy` to be None or SpecialTokenPolicy, got r,Using the tokenizer's special token policy () is deprecated. It will be removed in 1.10.0. Please pass a special token policy explicitly. Future default will be SpecialTokenPolicy.IGNORE.rr) rrrrr4rrrrjoinr)r`rrrrrrs   zTekkenizer.decodecCr)z[DEPRECATED] Converts a list of token ids into a string, keeping special tokens. Use `decode` with `special_token_policy=SpecialTokenPolicy.KEEP` instead. This is a convenient method for debugging. z`to_string` is deprecated and will be removed in 1.10.0. Use `decode` with `special_token_policy=SpecialTokenPolicy.KEEP` instead.)rrr _to_stringr`rrrr to_strings  zTekkenizer.to_stringcCs|j|tjdS)NrrrrrrrrrszTekkenizer._to_stringcCs|j|gtjdS)z0Convert a token id to its string representation.rrrrrrri szTekkenizer.id_to_piececCs|durtd|jdt|j}||jkr@|tjkr&|j|ddS|tj kr2t |d|tj kr9dSt d||j ||jS) aConvert a token id to its byte representation. Args: token_id: The token id to convert. special_token_policy: The policy for handling special tokens. Use the tokenizer's [attribute][mistral_common.tokens.tokenizers.tekken.Tekkenizer.special_token_policy] if `None`. Passing `None` is deprecated and will be changed to `SpecialTokenPolicy.IGNORE` in `mistral_common=1.10.0`. Returns: The byte representation of the token. Nrrr rz is a special tokenzUnknown special token policy )rrrrrRrrrzrrrr~rvdecode_single_token_bytes)r`rrrrrid_to_byte_piece s      zTekkenizer.id_to_byte_piecer)Ir!r"r#r$r'r unkrr begin_instend_inst begin_tools end_toolsbegin_tool_resultsend_tool_results tool_callsimgpad img_breakimg_endprefixmiddler begin_system end_systembegin_tool_contentrr[r7rrr%r rr rrpropertyr classmethodr4rr5setterr6rRrrrmrr/rrrrrrr1r)rrrrrrrrrrrribytesrrrrrr8js     P G   r8r1rbcCs|dur)t||ksJt||ft||kr)|d|}tdt|di}t|D]4\}}|hdks=J|d|ksEJt|d}|dks]|t|gks]J||f|d||<q/t|t|ksnJt| tt t|ks~J|S)zAReload our tokenizer JSON file and convert it to Tiktoken format.Nz(Cutting non special vocabulary to first z tokens.>rr rrrr) rkrorp enumeratekeysbase64 b64decoderrmrsrn)r1rbranksr^xmergerrrrq3s  " rqr)+rrloggingr functoolsr itertoolsrpathlibrtypingrnumpyrrt&mistral_common.tokens.tokenizers.audiorr%mistral_common.tokens.tokenizers.baserr r r &mistral_common.tokens.tokenizers.imager filterwarningsr getLoggerr!rorr)rrr'r*r0r8r7r%dictrrqrrrrsH      N