o c۷i @sddlZddlZddlmZddlmZmZmZmZddl Z ddl m Z ddl mZmZmZmZddlmZddlmZmZmZmZmZmZmZmZddlmZmZm Z m!Z!m"Z"m#Z#dd l$m%Z%m&Z&dd l'm(Z(m)Z)dd l*m+Z+dd l,m-Z-m.Z.dd l/m0Z0m1Z1ddl2m3Z3m4Z4m5Z5m6Z6m7Z7m8Z8m9Z9m:Z:m;Z;ddlm?Z?Gddde5ee4e3e9e fZ@Gddde@ee4e3e9e fZAGdddeAee4e3e9e fZBGdddeBee4e3e9e fZCGdddeCZDGdddeDZEGdddeEZFGdd d eFZGdS)!N)abstractmethod)AnyGenericSequenceoverload)Audio) InvalidAssistantMessageException InvalidMessageStructureExceptionInvalidRequestExceptionTokenizerException) FIMRequest) AudioChunk AudioURLChunk ContentChunk ImageChunk ImageURLChunk TextChunk ThinkChunkUserContentChunk)UATSAssistantMessageAssistantMessageType SystemMessage ToolMessage UserMessage)InstructRequest ModelSettings)ToolToolCall) SpeechRequest) StreamingModeTranscriptionRequest) AudioEncoderTranscriptionFormat) FIMRequestTypeInstructRequestTypeInstructTokenizerSpecialTokenPolicy SpecialTokens Tokenized TokenizedType TokenizerUserMessagePosition) ImageEncoder) Tekkenizerc steZdZdZ  d*dededBdedBffdd ZededBfd d Z de e fd d Z e d edee e ffddZededede e fddZedededede e fddZedede e fddZde e e dBde ede de ddf ddZede eddfd d!Zd eeefdefd"d#Ze j!fd$e e d%e de"fd&d'Z#d$e e de"fd(d)Z$Z%S)+InstructTokenizerBasezBase instruct tokenizer.N tokenizer image_encoder audio_encodercs&||_||_||_t|||dS)zInitialize the instruct tokenizer. Args: tokenizer: The tokenizer to use. image_encoder: The image encoder to use if any. audio_encoder: The audio encoder to use. N)r0r1r2super__init__selfr0r1r2 __class___/home/ubuntu/vllm_env/lib/python3.10/site-packages/mistral_common/tokens/tokenizers/instruct.pyr4;s zInstructTokenizerBase.__init__returncCs|jSN)r1r6r9r9r: mm_encoderMsz InstructTokenizerBase.mm_encodercCs |jjgS)zReturn the start tokens.)r0bos_idr=r9r9r:startTs zInstructTokenizerBase.startrequestcCsBd}d}tt|jD]\}}t|tr|dkr|}|}q ||fS)zFind the first and last user message in the request. Args: request: The request to search for user messages. Returns: The index of the first and last user message. )list enumeratemessages isinstancer)rA last_user_idxfirst_user_idximsgr9r9r:find_first_last_userXs  z*InstructTokenizerBase.find_first_last_usermessageis_before_last_user_messagecCtd)zEncode a tool message. Raises: NotImplementedError: The tool message is not implemented for the base tokenizer. zTool message not implementedNotImplementedErrorr6rLrMr9r9r:encode_tool_messagekz)InstructTokenizerBase.encode_tool_messagecontinue_messagecCrN)zEncode an assistant message. Raises: NotImplementedError: The assistant message is not implemented for the base tokenizer. z!Assistant message not implementedrOr6rLrMrTr9r9r:encode_assistant_messagets z.InstructTokenizerBase.encode_assistant_messagechunkcCrN)zEncode a think chunk. Raises: NotImplementedError: The think chunk is not implemented for the base tokenizer. zThink chunk not implementedrOr6rWr9r9r: encode_thinkrSz"InstructTokenizerBase.encode_think tokenizedrE max_tokenslast_user_message_indexcCdSr<r9)r6rZrEr[r\r9r9r:_truncate_for_max_tokenssz.InstructTokenizerBase._truncate_for_max_tokenscCr]r<r9clsrEr9r9r:validate_messagesz'InstructTokenizerBase.validate_messagesc Csg}g}d}g}||j||\}}t|jD]\}} |jr3|t|jdkr3t| ts3tdt| t rY|j | |j ||k||k|j d|j d\} } } || || nIt| trg|| ||k} n;t| tr|jow|t|jdk} |j| ||k| d} |t|jdkr| }nt| tr|| } n tdt| || q|jdur|||j|j||}|D] }|dur||qt||j|tjd|||d S) zEncode an instruct request. Args: request: The request to encode. Returns: The encoded tokens. Nz?Cannot continue final message if it is not an assistant messageT) system_promptforce_img_firstsettings)rTzUnknown message type special_token_policy)tokenstext prefix_idsimagesaudios)rarErKrDcontinue_final_messagelenrFrr rencode_user_messageavailable_toolsrdrfextendrrRrVrencode_system_messager typeappendtruncate_at_max_tokensr^r@r)decoder'KEEP)r6rArlrmrk tokens_listrHrGmsg_idxrJ new_tokens new_images new_audiosrTritokr9r9r:encode_instructsx            z%InstructTokenizerBase.encode_instructrirhcCs|jj||dS)zDecode tokens to a string. Args: tokens: The tokens to decode. special_token_policy: The policy to use for special tokens. Returns: The decoded string. rg)r0rw)r6rirhr9r9r:rws zInstructTokenizerBase.decodecCs |j|Sr<)r0 _to_string)r6rir9r9r:rs z InstructTokenizerBase._to_stringNN)&__name__ __module__ __qualname____doc__r+r-r"r4propertyr>rCintr@ staticmethodrtuplerKrrboolrRrrVrrYr^ classmethodrrarr)rr'IGNOREstrrwr __classcell__r9r9r7r:r/6sd      R r/c@sLeZdZdZddefdedeedBde de de dBd e d ed e ee ee jeeffd d Zded ee fddZ  d#de eeBde de dBd e d e ee ee jeeff ddZdede d ee fddZdede de d ee fddZded ee fddZded efddZded efdd Zde d efd!d"Z!dS)$InstructTokenizerV1zrInstruct tokenizer V1. This tokenizer has basic for messages. It does not support tools or image inputs. NFrLrqis_lastis_firstrdrerfr;c Cstt|jts Jd|jdusJdd}|r!|r!|d|j}n|j}d|d} |j| ddd \} } } | | | fS) arEncode a user message. Args: message: The message to encode. available_tools: Not used. is_last: Not used. is_first: Whether the message is the first one. system_prompt: The system prompt. force_img_first: Not used. Returns: The encoded tokens and empty list. "Message content must be normalizedNz(InstructTokenizerV1 cannot encode images z[INST] z [/INST]F)contentrrd)rFrrr1encode_user_content) r6rLrqrrrdrerfr message_txt curr_tokensimageaudior9r9r:rps  z'InstructTokenizerV1.encode_user_messagecCtd|jj)Nz,System message encoding not implemented for )rPr8rr6rLr9r9r:rs%z)InstructTokenizerV1.encode_system_messagercCs>t|tsJ|r|r|d|}|jj|ddd}|ggfS)a*Encode a user content. Args: content: The content to encode. is_last: Whether the message is the last one. system_prompt: The system prompt. force_img_first: Not used. Returns: The encoded tokens and empty list. rFboseos)rFrr0encode)r6rrrdrerir9r9r:r(s   z'InstructTokenizerV1.encode_user_contentrMcCrN)zEncode a tool message. Raises: TokenizerException: The tool message is not implemented for this version. &Tools not implemented for tokenizer V1r rQr9r9r:rRBz'InstructTokenizerV1.encode_tool_messagerTcCst|ts J||jdurt|jdkrtd|r"|jr"td|jr:t|jts/Jd|j j |jddd}n t|jd|j|jsQ|sQ| |j j |S) [Encode an assistant message. Args: message: The message to encode. is_before_last_user_message: Not used. continue_message: Whether to continue the message generation. Only use this if the assistant message is the last message. Returns: The encoded tokens. NrrU`continue_message` is only supported for assistant messages that have `prefix=False`.z4Message content must be a string for tokenizer < V13Frz // ) rFr tool_callsror prefixrrrr0rrueos_idr6rLrMrTrr9r9r:rVJs  z,InstructTokenizerV1.encode_assistant_messagerWcCrN)zEncode a think chunk. Raises: TokenizerException: The think chunk is not implemented for this version. z*Think not implemented for tokenizer < V13.rrXr9r9r:rYhrz InstructTokenizerV1.encode_thinkrAcCr)zEncode a FIM request. Raises: TokenizerException: The FIM request is not implemented for this version. zFIM not available for r r0versionr6rAr9r9r: encode_fimpszInstructTokenizerV1.encode_fimcCr)Nz Transcription not available for rrr9r9r:encode_transcriptionxrz(InstructTokenizerV1.encode_transcriptioncCstd|jjj)Nz+Speech request not available for tokenizer )r r0rvaluerr9r9r:encode_speech_request{sz)InstructTokenizerV1.encode_speech_requestNF)"rrrrrnonerrCrrrrrnpndarrayrrprrsrrrrRrrVrrYr r)rr!rrrr9r9r9r:rsd   $   rcseZdZdZejZ  d0dededBde dBffdd Z dde fd e d eedBd ed ed edBdede deeeeejeeffddZdedefddZdeeeBdefddZdedeeeffddZd ededeefddZde deeeffdd Z!d e"deefd!d"Z#d e"deefd#d$Z$de deefd%d&Z%d e"ded'edeefd(d)Z&d*edeefd+d,Z'd-e(de)fd.d/Z*Z+S)1InstructTokenizerV2z`Instruct tokenizer V2. This tokenizer adds supports to images, tools and FIM requests. Nr0r1r2cst||||jtjj|_|jtjj|_ |jtj j|_ |jtj j|_ |jtjj|_|jtjj|_|jtjj|_|jtjj|_|jtjj|_|jtjj|_dSInitialize the tokenizer. Args: tokenizer: The tokenizer to use. image_encoder: The image encoder to use. audio_encoder: The audio encoder to use. N)r3r4r0get_special_tokenr( begin_instr BEGIN_INSTend_instEND_INST begin_toolsBEGIN_AVAILABLE_TOOLS end_toolsEND_AVAILABLE_TOOLSbegin_tool_resultsBEGIN_TOOL_RESULTSend_tool_resultsEND_TOOL_RESULTSr TOOL_CALLSrBOSrPREFIXsuffixSUFFIXr5r7r9r:r4s zInstructTokenizerV2.__init__FrLrqrrrdrerfr;cCsd}||o |jtjkO}||o|jtjkO}g} |r=|r=dd|D} |jjtj| ddddd} | |j g| |j |rH| |j |d|j |j |||d\} } }g| |j}|jg}|| |}|| |fS)aEncode a user message. Args: message: The message to encode. available_tools: The list of available tools if any. is_last: Whether the message is the last one. is_first: Not used. system_prompt: The system prompt. force_img_first: Whether to force the image to be first. Returns: The encoded tokens and the list of images. FcSs g|] }|jdddiidqS)functionstrictT)exclude) model_dump).0toolr9r9r: s z;InstructTokenizerV2.encode_user_message.. ensure_asciir)rf)rrrdre)*_message_position_to_encode_tools_settingsr,firstlastr0rjsondumpsrrrr_encode_settingsrrrr)r6rLrqrrrdrerfdo_encode_tools_settingstools_settings_tokenstoolstools_json_tokensrirr prefix_tokens suffix_tokensrr9r9r:rps>     z'InstructTokenizerV2.encode_user_messagercCs&zt|WStjy|YSwr<)rloadsJSONDecodeErrorr6rr9r9r:_parse_json_contents  z'InstructTokenizerV2._parse_json_contentcCs(t|trddd|D}||S)Nrcs|]}|jVqdSr<rjrrWr9r9r: z:InstructTokenizerV2._parse_tool_content..)rFrCjoinrrr9r9r:_parse_tool_contents  z'InstructTokenizerV2._parse_tool_content tool_messagecCs|j||jdS)z8Bit of a hack due to the way tool results are tokenized.)namer)rrrr6rr9r9r:_prepare_tool_results z(InstructTokenizerV2._prepare_tool_resultrMcCsB|rgStj||gdd}|jg|jj|ddd|j}|S)aEncode a tool message. Args: message: The message to encode. is_before_last_user_message: Whether the message is before the last user message. If true, the message is not encoded. Returns: The encoded tokens. Frrrrrrr0rrr6rLrMtool_result_strrr9r9r:rRs z'InstructTokenizerV2.encode_tool_message tool_callcCs|jj||jjdS)z:Bit of a hack due to the way function calls are tokenized.r arguments)rrrr)r6rr9r9r:_prepare_function_calls z*InstructTokenizerV2._prepare_function_callcCsB|js Jd|t|jtsJd|jj|jddddS)Nz)Assistant message must have content. Got 3Message content must be a string for tokenizer < V7 Fr)rrFrr0rrstriprr9r9r:(_encode_normal_content_assistant_messagesz.idxr;csJt|tr dS|krdS|}|dusJt|8d|<dSr<)rFrro)r)r~r\rEto_dropr'r9r:dropLs   z:InstructTokenizerV7._truncate_for_max_tokens..droprrcz+Input couldn't fit in truncate_at_max_token)sumrrorFrr )r6r'rEr[r\r, current_idxr9r*r:r^>s   z,InstructTokenizerV7._truncate_for_max_tokensrLcCsF|jg}t|j}trt|dg}|||d7}||j|S)zEncode a system message. Args: message: The message to encode. Returns: The encoded tokens. rr)rrFrrrrrur )r6rLrirr9r9r:rsgs    z)InstructTokenizerV7.encode_system_messageFrrrdrec sz|dusJdt|trt|||St|dko#t|dttf}|r0|r0|d|dg}||\}}}|||fS)rN?in Tokenizer V7 we don't encode system prompts in user messagesrrcr)rFrr3rrorrr) r6rrrdrerrirlrr7r9r:rxs  z'InstructTokenizerV7.encode_user_contentrqrrfc s:|dusJdtj||||d||d\}} } || | fS)aEncode a user message. Args: message: The message to encode. available_tools: The list of available tools if any. is_last: Whether the message is the last one. is_first: Whether the message is the first one. system_prompt: Not used. force_img_first: Whether to force the image to be first. Returns: The encoded tokens and the list of images. Nr/)rrrdrerf)r3rp) r6rLrqrrrdrerfrirlrr7r9r:rps z'InstructTokenizerV7.encode_user_messagerAcCsd|jdus Jd|j|jjjtjkr||S|jjjtjkr'||Std|jjjd)a Encodes an audio transcription request into a tokenized format. This method processes a transcription request containing audio data, encodes the user message, and returns the tokenized output. Args: request: The transcription request object containing the audio data to be encoded. Returns: Tokenized: The tokenized representation of the audio data, including processed audio and tokens Nz6Audio encoder must be defined, got self.audio_encoder=zxTranscription format should be one of 'instruct', 'streaming', got self.audio_encoder.audio_config.transcription_format=r) r2r$transcription_formatr#INSTRUCT_encode_instruct_transcription STREAMING_encode_streaming_transcriptionr rr9r9r:rs  z(InstructTokenizerV7.encode_transcriptioncCs|jtjksJd|j|jdusJ|jjd|}|jtt |j dgdgdddt d\}}}g||}|j durTd|j }||jj|ddd 7}||jt||j||d S) Nz=Request must not be in streaming mode, got request.streaming=z! needs to have a TRANSCRIBE token) input_audio)rT)rqrrrdrfzlang:Frrirjrm) streamingr DISABLEDr#r8rr@rprr rrrlanguager0rrur)r)r6rArrirrlanguage_stringr9r9r:r2s&    z2InstructTokenizerV7._encode_instruct_transcriptionrtranscription_delay_mscCsj|jdus Jd|jt|trt|nt|}|j||}|jdur,|jgng}t|j |dS)NFAudio encoder must be defined to encode audio, got self.audio_encoder=rirm) r2rFrr from_base64 from_bytes encode_audiorr)ri)r6rr;_audio audio_encrmr9r9r: _encode_audios  z!InstructTokenizerV7._encode_audiocCs<|jtjkr||jj|j}||j}|j }nw|jtj kr|j dus'J|j |j\}}||g}t |jjdkrhtd|jdtt|jjtsPJt|jj}tt|j|jf|j|jg}n|jjrpJd|j dus}Jd|j ||j |j}ntd|jt||j|tjd|dS) Nrz%Passing audio with request.streaming=zf is deprecated. Make sure to not pass any audio to `TranscriptionRequest` when doing online streaming.z~For online streaming, no audio bytes should be passed in the first request. Audio buffering is taken care of directly by vLLM.r<z9Request must be in streaming mode, got request.streaming=rgr6) r7r OFFLINErCrdatatarget_streaming_delay_msr@rirmONLINEr2get_padding_audiorowarningswarn FutureWarningrFrrr>r concatenate audio_array sampling_rateformatencode_streaming_tokensrr)rwr'rx)r6rArZrirmleft_pad right_pad request_audior9r9r:r4sD      z3InstructTokenizerV7._encode_streaming_transcriptioncCs,||rtdd|DrtddSdS)zKValidates that system prompts and audio chunks are not used together in v7.cs|]}t|tVqdSr<)rFrrrLr9r9r:r1z8InstructTokenizerV7.validate_messages..z9System messages are not yet allowed when audio is presentN) _has_audioanyrr_r9r9r:ra-s z%InstructTokenizerV7.validate_messagescCstdd|DS)Ncss:|]}t|tot|jtotdd|jDVqdS)csrTr<)rFr rr9r9r:r9rVz;InstructTokenizerV7._has_audio...N)rFrrrCrXrUr9r9r:r6s  z1InstructTokenizerV7._has_audio..)rX)rEr9r9r:rW4szInstructTokenizerV7._has_audiorMcCsr|jdusJt|jtsJd|jj|jddd}|jj|jddd}|jg||j}g|||j}|S)aEncode a tool message. Note: Same as [V3][mistral_common.tokens.tokenizers.instruct.InstructTokenizerV3.encode_tool_message] but tools are not wrapped in a list and history is also tokenized Args: message: The message to encode. is_before_last_user_message: Not used. Returns: The encoded tokens. NrFr) r rFrrr0rrr"r)r6rLrMtool_call_id_tokensrirrr9r9r:rR=s$z'InstructTokenizerV7.encode_tool_messagerTcCs|js |js td||r|jrtdg}|jr7t|jtr'||}nt|jtr7|| |jd7}|jrA|| |7}|jsM|sM| |j j |S)rrrr)rrr rrrFrrrCrrrur0rrr9r9r:rV\s"      z,InstructTokenizerV7.encode_assistant_message ref_audiovoicecCs|dus|dusJd|d||jdusJd|jd}|dur4t|tr/t|nt|}|j||}t|j|j durI|j gdSgdS)a7Encode reference audio or voice preset into a Tokenized object. Args: ref_audio: Base64-encoded string or raw bytes of reference audio, or None. voice: Preset voice name, or None. Returns: Tokenized object with audio tokens and optional audio data. NzIEither ref_audio or voice must be defined to encode audio, got ref_audio=z and voice=r<r=) r2rFrrr>r?encode_audio_for_speech_requestr)rir)r6rZr[rArBr9r9r: _encode_audio_for_speech_request~s    z4InstructTokenizerV7._encode_audio_for_speech_requestcCs|jdus Jd|j|}t|d}||j|j}|j|j|j|j|j}| |jj ||j j |j ddd| |jj| |jj||_|S)axEncode a speech synthesis request into a tokenized sequence. Produces: [BOS] + audio_tokens + [TEXT_TO_AUDIO] + text_tokens + [AUDIO_TO_TEXT] + [BEGIN_AUDIO]. Args: request: The speech request containing input text and voice/audio data. Returns: Tokenized object with the full token sequence and optional audio data. Nr<)riFr)r2r@r)r]rZr[rirrrmrutext_to_audio_tokenr0rinputaudio_to_text_tokenbegin_audio_token)r6rA init_tokensrZtokenized_audiorir9r9r:rs  z)InstructTokenizerV7.encode_speech_requestrrr<)/rrrrr+r-r"r4rCrrr^rrsrrrrrrrrrrrrrpr!r)rr2bytesfloatrCr4rrrarrWrrRrVr]rrrr9r9r7r:rs  ) %  %  4 ""rc sVeZdZdZ  d dededBdedBddffdd Zd ede e fd d Z Z S) InstructTokenizerV11zInstruct tokenizer V11. The difference with V7 tokenizer is that it encodes tool calls differently: Tool call results are encoded as : - [begin tool call] call_name_tokens [call id] call_id_tokens [args] content tokens Nr0r1r2r;cs8t||||jtjj|_|jtjj|_ dSr<) r3r4r0rr(argsrARGSr CALL_IDr5r7r9r:r4szInstructTokenizerV11.__init__rLcCs|js Jd|g}|jD]B}||}g}d|vr+|jg|jj|dddd}||jg|jj|dddd||j|jjtj|dddddd7}q|S)NrrFrrrr) rrrir0rrrhrr)r6rLrrpreparedidsr9r9r:rs&  z= v13rcsrr<rrr9r9r:rrz;InstructTokenizerV13.encode_tool_message..Fr) r rrFrrr0rrr)r6rLrMrrirr9r9r:rRs  z(InstructTokenizerV13.encode_tool_messagerWcCsZ|jdus Jd|jdusJd|jj|jddd}|jg|}|jr+||j|S)zEncode a thinking chunk. Args: chunk: The thinking chunk to encode. Returns: The encoded tokens. Nz2think tokens are not available for this tokenizer.Fr)rprqr0rthinkingclosedru)r6rWri think_tokensr9r9r:rYs  z!InstructTokenizerV13.encode_thinkrEcCsdS)z9Allows system prompts and audio chunks to coexist in v13.Nr9r_r9r9r:ra-rbz&InstructTokenizerV13.validate_messagesr)rrrrr,rrr+r-r"r4rrCrrrrrRrrYrrrarr9r9r7r:rls& rlc seZdZdZ  ddededBdedBddffdd Zd eddfd d Z d ede e fd d Z de de e ffdd ZZS)InstructTokenizerV15zInstruct tokenizer V15. Extends V13 with model settings encoding. Inherits V13's tool call behavior (no call ID in tool calls/results, first-position tools, think support). Nr0r1r2r;cs:tj|||d|jtjj|_|jtjj|_ dS)Nr) r3r4r0rr(begin_model_settingsrBEGIN_MODEL_SETTINGSend_model_settingsEND_MODEL_SETTINGSr5r7r9r:r4:szInstructTokenizerV15.__init__rfcCs2|jj}|durtd|jjd||dS)zGValidate model settings against the tokenizer's model settings builder.Nz Tokenizer z" needs a `model_settings_builder`.)r0rr rvalidate_settings)r6rfrr9r9r:_validate_settingsDsz'InstructTokenizerV15._validate_settingscCsZ|||tkr gStj|jddddd}|jj|ddd}|jg||j }|S)z=Encode model settings as special-token-delimited JSON tokens.T) exclude_noneF)r sort_keysr) r{rrrrrr0rrwry)r6rfdumped_settingssetting_json_tokenssettings_tokensr9r9r:rKs  z%InstructTokenizerV15._encode_settingsrLcs4t|jtrtdd|jDrtdt|S)z6Encode a system message, rejecting ThinkChunk content.csrTr<)rFrrr9r9r:r_rVz=InstructTokenizerV15.encode_system_message..zsR   (  ,   D  h8&X