o *i8@s|ddlmZmZddlmZddlmZddlmZm Z m Z ddl Z ddl mZmZddlmZddlmZdd lmZdd lmZdd lmZmZdd lmZdd lmZddlm Z ddl!m"Z"ddl#m$Z$Gddde%eZ&Gddde%eZ'Gddde%eZ(Gddde%eZ)GdddeZ*GdddeZ+e dedZ,e dedZ-e d e*dZ.Gd!d"d"e e,e-e.efZ/dS)#)ABCabstractmethod)Enum)Path)AnyGenericTypeVarN) ConfigDictField)Audio) MistralBase) FIMRequest)UserContentChunk)AssistantMessageType UserMessage)InstructRequest)Tool)TranscriptionRequest) AudioEncoder) ImageEncoderc@seZdZdZdZdZdS)UserMessagePositionzWhere to encode available toolsfirstlastN)__name__ __module__ __qualname____doc__rrrrb/home/ubuntu/veenaModal/venv/lib/python3.10/site-packages/mistral_common/tokens/tokenizers/base.pyrsrc@seZdZdZdZdZdZdZdZdZ dZ d Z d Z d Z d Zd ZdZdZdZdZdZdZdZdZdZdZdZdZdZdZdZdZdZ dS) SpecialTokensa(Enum of special tokens used in the tokenizer. Attributes: unk: The unknown token. bos: The beginning of string token. eos: The end of string token. begin_inst: The beginning of instruction token. end_inst: The end of instruction token. begin_tools: The beginning of tools token. end_tools: The end of tools token. begin_tool_results: The beginning of tool results token. end_tool_results: The end of tool results token. tool_calls: The tool calls token. img: The image token. pad: The pad token. img_break: The image break token. img_end: The image end token. prefix: The prefix token for FIM. middle: The middle token for FIM. suffix: The suffix token for FIM. begin_system: The beginning of system prompt token. end_system: The end of system prompt token. begin_tool_content: The beginning of tool content token. args: The args token. call_id: The call id token. audio: The audio token. begin_audio: The beginning of audio token. transcribe: The transcribe token. begin_think: The beginning of think token. end_think: The end of think token. Examples: >>> unk = SpecialTokens.unk zzzz[INST]z[/INST]z[AVAILABLE_TOOLS]z[/AVAILABLE_TOOLS]z[TOOL_RESULTS]z[/TOOL_RESULTS]z [TOOL_CALLS]z[IMG]zz [IMG_BREAK]z [IMG_END]z[PREFIX]z[MIDDLE]z[SUFFIX]z[SYSTEM_PROMPT]z[/SYSTEM_PROMPT]z[TOOL_CONTENT]z[ARGS]z [CALL_ID]z[AUDIO]z [BEGIN_AUDIO]z [TRANSCRIBE]z[THINK]z[/THINK]z[STREAMING_PAD]z[STREAMING_WORD]N)!rrrrunkboseos begin_instend_inst begin_tools end_toolsbegin_tool_resultsend_tool_results tool_callsimgpad img_breakimg_endprefixmiddlesuffix begin_system end_systembegin_tool_contentargscall_idaudio begin_audio transcribe begin_think end_think streaming_padstreaming_wordrrrrrs>#rcs:eZdZdZdZdZdZededeffdd Z Z S) SpecialTokenPolicyzWhat to do with special tokens when encoding/decoding. Attributes: IGNORE: Ignore special tokens. KEEP: Keep special tokens. RAISE: Raise an error if special tokens are found. ignorekeepraisevaluereturncs:|dkr tjSdkrtjSdkrtjSt|S)Nr)r=IGNOREKEEPRAISEsuper _missing_)clsrA __class__rrrIos   zSpecialTokenPolicy._missing_) rrrrrErFrG classmethodrrI __classcell__rrrKrr=bs r=c@seZdZdZedefddZdddefddZdddefd d Z dddefd d Z dddefd dZ dZ dZ dZdZdZdZdS)TokenizerVersionaEnum of tokenizer versions. Allow to distinguish between different versions of the tokenizer and maintain backward compatibility. Attributes: v1: The first version of the tokenizer. v2: The second version of the tokenizer that includes special control tokens [INST], [\INST]. v3: The third version of the tokenizer that includes improved function calling. v7: The seventh version of the tokenizer that includes improved system prompt and function calling. v11: The eleventh version of the tokenizer that includes improved function calling. v13: The thirteenth version of the tokenizer that includes no call id tokenization and better prompt caching. Examples: >>> version = TokenizerVersion.v1 rBcCst|jddS)NrC)intrAselfrrr _version_numszTokenizerVersion._version_numotherzstr | TokenizerVersioncCst|tr t|}|j|jkSN isinstancestrrOrSrRrTrrr__lt__s  zTokenizerVersion.__lt__cCs"t|trt|}|j|jkSdSrUrVrYrrr__le__  zTokenizerVersion.__le__cCs"t|trt|}|j|jkSdSrUrVrYrrr__gt__r\zTokenizerVersion.__gt__cCs"t|trt|}|j|jkSdSrUrVrYrrr__ge__r\zTokenizerVersion.__ge__v1v2v3v7v11v13N)rrrrpropertyrPrSboolrZr[r]r^r_r`rarbrcrdrrrrrO|srOc@szeZdZUdZeddZeeed<dZ e dBed<dZ eedBed<e edZ eejed <e edZeeed <dS) TokenizedaA tokenized [`InstructRequest`][mistral_common.tokens.instruct.request]. Attributes: tokens: The token ids. text: The text representation of the tokens. prefix_ids: The prefix ids for FIM. images: The loaded images associated with the tokens. Examples: >>> tokenized = Tokenized(tokens=[1, 2, 3], text="Hello world", prefix_ids=[1], images=[]) T)arbitrary_types_allowedtokensNtext prefix_ids)default_factoryimagesaudios)rrrrr model_configlistrP__annotations__rjrXrkr rmnpndarrayrnr rrrrrgs  rgc @seZdZeedefddZeedeefddZeedefddZ ede e fdd Z ed ede fd d Z eedefd dZeedefddZeedefddZeedefddZede dedede efddZed,de ededBde fddZede defdd Zed!eejBe Bdefd"d#Zeedefd$d%Zede ede fd&d'Zede ede fd(d)Zeedefd*d+ZdS)- TokenizerrBcCdS)z!Vocabulary size of the tokenizer.NrrQrrrn_wordszTokenizer.n_wordscCru)zIds of the special tokens.NrrQrrr special_idsrwzTokenizer.special_idscCru)z.The number of special tokens of the tokenizer.NrrQrrrnum_special_tokensrwzTokenizer.num_special_tokenscCru)z(All tokens in the vocabulary as strings.NrrQrrrvocabrwzTokenizer.vocabtoken_idcCru)z$Convert a token id to the token str.Nr)rRr{rrr id_to_piecerwzTokenizer.id_to_piececCru)z$id of the Beginning of String token.NrrQrrrbos_idrwzTokenizer.bos_idcCru)zid of the End of String token.NrrQrrreos_idrwzTokenizer.eos_idcCru)zid of the Pad token.NrrQrrrpad_idrwzTokenizer.pad_idcCru)zid of the Unk token.NrrQrrrunk_idrwzTokenizer.unk_idsr!r"cCru)z(Convert a string to a list of token ids.Nr)rRrr!r"rrrencoderwzTokenizer.encodeNrispecial_token_policycCru)aDecode the token ids to a string. Args: tokens: The token ids to decode. special_token_policy: The policy to use for special tokens. Passing `None` will default to `self._special_token_policy` for [Tekkenizer][mistral_common.tokens.tokenizers.tekken.Tekkenizer] and `SpecialTokenPolicy.IGNORE` for [SentencePieceTokenizer][mistral_common.tokens.tokenizers.sentencepiece.SentencePieceTokenizer]. Note that passing `None` will be deprecated and `special_token_policy` will default to `SpecialTokenPolicy.IGNORE` in `mistral_common=1.10.0`. Returns: The decoded string. NrrRrirrrrdecoderwzTokenizer.decodecCru)zGet the id of a control token.Nr)rRrrrrget_special_tokenrwzTokenizer.get_special_tokentokencCru)z2Check if token id or token str is a special token.Nr)rRrrrr is_specialrwzTokenizer.is_specialcCru)z!Get the version of the tokenizer.NrrQrrrversion rwzTokenizer.versioncCru)z[DEPRECATED] Converts a list of token ids into a string, keeping special tokens. Use `decode` with `special_token_policy=SpecialTokenPolicy.KEEP` instead. This is a convenient method for debugging. NrrRrirrr to_stringszTokenizer.to_stringcCdSrUrrrrr _to_stringzTokenizer._to_stringcCru)zThe file path of the tokenizer.NrrQrrr file_pathszTokenizer.file_pathrU) rrrrerrPrvsetrxryrprXrzr|r}r~rrrfrr=rrrrintegerrrOrrrrrrrrrrtsX " rtInstructRequestType)boundFIMRequestType TokenizedTypec@seZdZUdZeed<edBed<edBed<ede fddZ dededBdedBddfd d Z e d e defd d Ze d edefddZe d$deededBdefddZe d edefddZe  d%dedeedBdedededBdedeeeeejeeffddZ e  d%deee!BdededBdedeeeeejeeff d d!Z"e deedefd"d#Z#dS)&InstructTokenizerzBase class for instruct tokenizers. Attributes: tokenizer: The tokenizer to use. image_encoder: The image encoder to use if any. audio_encoder: The audio encoder to use if any. tokenizerN image_encoder audio_encoderrBcCs|jjS)zThe version of the tokenizer.)rrrQrrrr6szInstructTokenizer.versioncCru)zInitialize the instruct tokenizer. Args: tokenizer: The tokenizer to use. image_encoder: The image encoder to use if any. audio_encoder: The audio encoder to use if any. Nr)rRrrrrrr__init__;rwzInstructTokenizer.__init__requestcCru)zInstruct request to Tokenized object Args: request: The instruct request to encode. Returns: The tokenized instruct request. NrrRrrrrencode_instructFrwz!InstructTokenizer.encode_instructcCru)a Encodes an audio transcription request into a tokenized format. This method processes a transcription request containing audio data, encodes the user message, and returns the tokenized output. Args: request: The transcription request object containing the audio data to be encoded. Returns: Tokenized: The tokenized representation of the audio data, including processed audio and tokens Nrrrrrencode_transcriptionQsz&InstructTokenizer.encode_transcriptionrircCru)aConvert token ids to string Args: tokens: The token ids to decode. special_token_policy: The policy to use for special tokens. Passing `None` will default to `self._special_token_policy` for [Tekkenizer][mistral_common.tokens.tokenizers.tekken.Tekkenizer] and `SpecialTokenPolicy.IGNORE` for [SentencePieceTokenizer][mistral_common.tokens.tokenizers.sentencepiece.SentencePieceTokenizer]. Note that passing `None` will be deprecated and `special_token_policy` will default to `SpecialTokenPolicy.IGNORE` in `mistral_common=1.10.0`. Returns: The decoded string. NrrrrrrbrwzInstructTokenizer.decodecCru)zFIM request to Tokenized object Args: request: The FIM request to encode. Returns: The tokenized FIM request. Nrrrrr encode_fimsrwzInstructTokenizer.encode_fimFmessageavailable_toolsis_lastis_first system_promptforce_img_firstcCru)aEncode a user message. Args: message: The user message to encode. available_tools: The available tools. is_last: Whether the message is the last one. is_first: Whether the message is the first one. system_prompt: The system prompt. force_img_first: Whether to force the image to be first. Returns: The encoded tokens and images. Nr)rRrrrrrrrrrencode_user_message~sz%InstructTokenizer.encode_user_messagecontentcCru)aIEncode a user content. Args: content: The user content to encode. is_last: Whether the content is the last one. system_prompt: The system prompt. force_img_first: Whether to force the image to be first. Returns: The encoded tokens and images. Nr)rRrrrrrrrencode_user_contentsz%InstructTokenizer.encode_user_contentcCrrUrrrrrrrzInstructTokenizer._to_stringrU)NF)$rrrrrtrqrrrerOrrrrrrrrrprPr=rXrrrrrrftuplerrrsr rrrrrrrrr)st      "     r)0abcrrenumrpathlibrtypingrrrnumpyrrpydanticr r mistral_common.audior mistral_common.baser #mistral_common.protocol.fim.requestr &mistral_common.protocol.instruct.chunkr)mistral_common.protocol.instruct.messagesrr(mistral_common.protocol.instruct.requestr+mistral_common.protocol.instruct.tool_callsr-mistral_common.protocol.transcription.requestr&mistral_common.tokens.tokenizers.audior&mistral_common.tokens.tokenizers.imagerrXrrr=rOrgrtrrrrrrrrs4           C1 b