o c۷i8@sddlmZmZddlmZddlmZddlmZm Z m Z ddl Z ddl mZmZddlmZddlmZdd lmZdd lmZdd lmZmZdd lmZdd lmZddlm Z ddl!m"Z"ddl#m$Z$ddl%m&Z&ddl'm(Z(Gddde)eZ*Gddde)eZ+Gddde)eZ,Gddde)eZ-GdddeZ.GdddeZ/e ded Z0e d!ed Z1e d"e.d Z2Gd#d$d$e e0e1e2efZ3dS)%)ABCabstractmethod)Enum)Path)AnyGenericTypeVarN) ConfigDictField)Audio) MistralBase) FIMRequest)UserContentChunk)AssistantMessageType UserMessage)InstructRequest)Tool) SpeechRequest)TranscriptionRequest) AudioEncoder) ImageEncoder)ModelSettingsBuilderc@seZdZdZdZdZdS)UserMessagePositionzWhere to encode available toolsfirstlastN)__name__ __module__ __qualname____doc__rrrr[/home/ubuntu/vllm_env/lib/python3.10/site-packages/mistral_common/tokens/tokenizers/base.pyrsrc@seZdZdZdZdZdZdZdZdZ dZ d Z d Z d Z d Zd ZdZdZdZdZdZdZdZdZdZdZdZdZdZdZdZdZdZ dZ!d Z"d!Z#d"Z$d#S)$ SpecialTokensaEnum of special tokens used in the tokenizer. Attributes: unk: The unknown token. bos: The beginning of string token. eos: The end of string token. begin_inst: The beginning of instruction token. end_inst: The end of instruction token. begin_tools: The beginning of tools token. end_tools: The end of tools token. begin_tool_results: The beginning of tool results token. end_tool_results: The end of tool results token. tool_calls: The tool calls token. img: The image token. pad: The pad token. img_break: The image break token. img_end: The image end token. prefix: The prefix token for FIM. middle: The middle token for FIM. suffix: The suffix token for FIM. begin_system: The beginning of system prompt token. end_system: The end of system prompt token. begin_tool_content: The beginning of tool content token. args: The args token. call_id: The call id token. audio: The audio token. begin_audio: The beginning of audio token. transcribe: The transcribe token. begin_think: The beginning of think token. end_think: The end of think token. streaming_pad: The streaming pad token. streaming_word: The streaming word token. text_to_audio: The text to audio token. audio_to_text: The audio to text token. Examples: >>> unk = SpecialTokens.unk zzzz[INST]z[/INST]z[AVAILABLE_TOOLS]z[/AVAILABLE_TOOLS]z[TOOL_RESULTS]z[/TOOL_RESULTS]z [TOOL_CALLS]z[IMG]zz [IMG_BREAK]z [IMG_END]z[PREFIX]z[MIDDLE]z[SUFFIX]z[SYSTEM_PROMPT]z[/SYSTEM_PROMPT]z[TOOL_CONTENT]z[ARGS]z [CALL_ID]z[AUDIO]z [BEGIN_AUDIO]z [TRANSCRIBE]z[THINK]z[/THINK]z[STREAMING_PAD]z[STREAMING_WORD]z[NEXT_AUDIO_TEXT]z[REPEAT_AUDIO_TEXT]z[MODEL_SETTINGS]z[/MODEL_SETTINGS]N)%rrrrunkboseos begin_instend_inst begin_tools end_toolsbegin_tool_resultsend_tool_results tool_callsimgpad img_breakimg_endprefixmiddlesuffix begin_system end_systembegin_tool_contentargscall_idaudio begin_audio transcribe begin_think end_think streaming_padstreaming_word text_to_audio audio_to_textbegin_model_settingsend_model_settingsrrrr r!!sF'r!cs:eZdZdZdZdZdZededeffdd Z Z S) SpecialTokenPolicyzWhat to do with special tokens when encoding/decoding. Attributes: IGNORE: Ignore special tokens. KEEP: Keep special tokens. RAISE: Raise an error if special tokens are found. ignorekeepraisevaluereturncs:|dkr tjSdkrtjSdkrtjSt|S)Nr)rCIGNOREKEEPRAISEsuper _missing_)clsrG __class__rr rOys   zSpecialTokenPolicy._missing_) rrrrrKrLrM classmethodrrO __classcell__rrrQr rCls rCc@seZdZdZedefddZedefddZdddefd d Z dddefd d Z dddefd dZ dddefddZ dZ dZdZdZdZdZdZdS)TokenizerVersionaEnum of tokenizer versions. Allow to distinguish between different versions of the tokenizer and maintain backward compatibility. Attributes: v1: The first version of the tokenizer. v2: The second version of the tokenizer that includes special control tokens [INST], [\INST]. v3: The third version of the tokenizer that includes improved function calling. v7: The seventh version of the tokenizer that includes improved system prompt and function calling. v11: The eleventh version of the tokenizer that includes improved function calling. v13: The thirteenth version of the tokenizer that includes no call id tokenization and better prompt caching. Examples: >>> version = TokenizerVersion.v1 rHcCst|jddS)NrI)intrGselfrrr _version_numszTokenizerVersion._version_numcCs |tjkSN)rUv15rWrrr supports_model_settingss z(TokenizerVersion.supports_model_settingsotherzstr | TokenizerVersioncCst|tr t|}|j|jkSrZ isinstancestrrUrYrXr]rrr __lt__s  zTokenizerVersion.__lt__cCs"t|trt|}|j|jkSdSrZr^rarrr __le__  zTokenizerVersion.__le__cCs"t|trt|}|j|jkSdSrZr^rarrr __gt__rdzTokenizerVersion.__gt__cCs"t|trt|}|j|jkSdSrZr^rarrr __ge__rdzTokenizerVersion.__ge__v1v2v3v7v11v13r[N)rrrrpropertyrVrYboolr\rbrcrerfrgrhrirjrkrlr[rrrr rUs"rUc@szeZdZUdZeddZeeed<dZ e dBed<dZ eedBed<e edZ eejed <e edZeeed <dS) TokenizedaA tokenized [`InstructRequest`][mistral_common.protocol.instruct.request.InstructRequest]. Attributes: tokens: The token ids. text: The text representation of the tokens. prefix_ids: The prefix ids for FIM. images: The loaded images associated with the tokens. Examples: >>> tokenized = Tokenized(tokens=[1, 2, 3], text="Hello world", prefix_ids=[1], images=[]) T)arbitrary_types_allowedtokensNtext prefix_ids)default_factoryimagesaudios)rrrrr model_configlistrV__annotations__rrr`rsr runpndarrayrvr rrrr ros  roc @seZdZeedefddZeedeefddZeedefddZ eede dBfd d Z ede e fd d Zed ede fddZeedefddZeedefddZeedefddZeedefddZede dedede efddZeejfde edede fdd Zede defd!d"Zed#eejBe Bdefd$d%Zeedefd&d'Zede ede fd(d)Zeede fd*d+Z!dS), TokenizerrHcCdS)z!Vocabulary size of the tokenizer.NrrWrrr n_wordszTokenizer.n_wordscCr})zIds of the special tokens.NrrWrrr special_idsrzTokenizer.special_idscCr})z.The number of special tokens of the tokenizer.NrrWrrr num_special_tokensrzTokenizer.num_special_tokensNcCr})zCThe model settings builder, or None if unsupported by this version.NrrWrrr model_settings_builderrz Tokenizer.model_settings_buildercCr})z(All tokens in the vocabulary as strings.NrrWrrr vocabrzTokenizer.vocabtoken_idcCr})z$Convert a token id to the token str.Nr)rXrrrr id_to_piecerzTokenizer.id_to_piececCr})z$id of the Beginning of String token.NrrWrrr bos_idrzTokenizer.bos_idcCr})zid of the End of String token.NrrWrrr eos_idrzTokenizer.eos_idcCr})zid of the Pad token.NrrWrrr pad_idrzTokenizer.pad_idcCr})zid of the Unk token.NrrWrrr unk_idrzTokenizer.unk_idsr#r$cCr})z(Convert a string to a list of token ids.Nr)rXrr#r$rrr encoderzTokenizer.encoderqspecial_token_policycCr})zDecode the token ids to a string. Args: tokens: The token ids to decode. special_token_policy: The policy to use for special tokens. Returns: The decoded string. NrrXrqrrrr decoderzTokenizer.decodecCr})zGet the id of a control token.Nr)rXrrrr get_special_tokenrzTokenizer.get_special_tokentokencCr})z2Check if token id or token str is a special token.Nr)rXrrrr is_specialrzTokenizer.is_specialcCr})z!Get the version of the tokenizer.NrrWrrr versionrzTokenizer.versioncCdSrZrrXrqrrr _to_stringzTokenizer._to_stringcCr})zThe file path of the tokenizer.NrrWrrr file_path"szTokenizer.file_path)"rrrrmrrVr~setrrrrrxr`rrrrrrrnrrCrKrrrzintegerrrUrrrrrrrr r|sZ " r|InstructRequestType)boundFIMRequestType TokenizedTypec@seZdZUdZeed<edBed<edBed<ede fddZ dededBdedBddfd d Z e d e defd d Ze d edefddZe d edefddZe deededefddZe d edefddZe  d&dedeedBdedededBdedeeeeej ee!ffdd Z"e  d&d!eee#BdededBdedeeeeej ee!ff d"d#Z$e deedefd$d%Z%dS)'InstructTokenizerzBase class for instruct tokenizers. Attributes: tokenizer: The tokenizer to use. image_encoder: The image encoder to use if any. audio_encoder: The audio encoder to use if any. tokenizerN image_encoder audio_encoderrHcCs|jjS)zThe version of the tokenizer.)rrrWrrr r;szInstructTokenizer.versioncCr})zInitialize the instruct tokenizer. Args: tokenizer: The tokenizer to use. image_encoder: The image encoder to use if any. audio_encoder: The audio encoder to use if any. Nr)rXrrrrrr __init__@rzInstructTokenizer.__init__requestcCr})zInstruct request to Tokenized object Args: request: The instruct request to encode. Returns: The tokenized instruct request. NrrXrrrr encode_instructKrz!InstructTokenizer.encode_instructcCr})a Encodes an audio transcription request into a tokenized format. This method processes a transcription request containing audio data, encodes the user message, and returns the tokenized output. Args: request: The transcription request object containing the audio data to be encoded. Returns: Tokenized: The tokenized representation of the audio data, including processed audio and tokens Nrrrrr encode_transcriptionVrz&InstructTokenizer.encode_transcriptioncCr})aEncodes a speech synthesis request into a tokenized format. This method processes a speech request containing text input and optional reference audio or voice preset, and returns the tokenized output. Args: request: The speech request object containing the text and voice/audio data. Returns: Tokenized: The tokenized representation of the speech request. Nrrrrr encode_speech_requestfrz'InstructTokenizer.encode_speech_requestrqrcCr})zConvert token ids to string Args: tokens: The token ids to decode. special_token_policy: The policy to use for special tokens. Returns: The decoded string. Nrrrrr rtrzInstructTokenizer.decodecCr})zFIM request to Tokenized object Args: request: The FIM request to encode. Returns: The tokenized FIM request. Nrrrrr encode_fimrzInstructTokenizer.encode_fimFmessageavailable_toolsis_lastis_first system_promptforce_img_firstcCr})aEncode a user message. Args: message: The user message to encode. available_tools: The available tools. is_last: Whether the message is the last one. is_first: Whether the message is the first one. system_prompt: The system prompt. force_img_first: Whether to force the image to be first. Returns: The encoded tokens and images. Nr)rXrrrrrrrrr encode_user_messagesz%InstructTokenizer.encode_user_messagecontentcCr})aIEncode a user content. Args: content: The user content to encode. is_last: Whether the content is the last one. system_prompt: The system prompt. force_img_first: Whether to force the image to be first. Returns: The encoded tokens and images. Nr)rXrrrrrrr encode_user_contentsz%InstructTokenizer.encode_user_contentcCrrZrrrrr rrzInstructTokenizer._to_string)NF)&rrrrr|ryrrrmrUrrrrrrrrrrrxrVrCr`rrrrrrntuplerzr{r rrrrrrrr r.sx             r)4abcrrenumrpathlibrtypingrrrnumpyrzpydanticr r mistral_common.audior mistral_common.baser #mistral_common.protocol.fim.requestr &mistral_common.protocol.instruct.chunkr)mistral_common.protocol.instruct.messagesrr(mistral_common.protocol.instruct.requestr+mistral_common.protocol.instruct.tool_callsr&mistral_common.protocol.speech.requestr-mistral_common.protocol.transcription.requestr&mistral_common.tokens.tokenizers.audior&mistral_common.tokens.tokenizers.imager7mistral_common.tokens.tokenizers.model_settings_builderrr`rr!rCrUror|rrrrrrrr s8             K6 X