o *i1 @sddlZddlZddlmZddlmZmZmZmZddl Z ddl m Z ddl mZmZmZmZddlmZddlmZmZmZmZmZmZmZmZddlmZmZm Z m!Z!m"Z"m#Z#dd l$m%Z%dd l&m'Z'm(Z(dd l)m*Z*m+Z+dd l,m-Z-m.Z.dd l/m0Z0m1Z1m2Z2m3Z3m4Z4m5Z5m6Z6m7Z7m8Z8ddl9m:Z:ddl;mGddde>ee1e0e6e fZ?Gddde?ee1e0e6e fZ@Gddde@ZAGdddeAZBGdddeBZCdS)N)abstractmethod)AnyGenericSequenceoverload)Audio) InvalidAssistantMessageException InvalidMessageStructureExceptionInvalidRequestExceptionTokenizerException) FIMRequest) AudioChunk AudioURLChunk ContentChunk ImageChunk ImageURLChunk TextChunk ThinkChunkUserContentChunk)UATSAssistantMessageAssistantMessageType SystemMessage ToolMessage UserMessage)InstructRequest)ToolToolCall) StreamingModeTranscriptionRequest) AudioEncoderTranscriptionFormat) FIMRequestTypeInstructRequestTypeInstructTokenizerSpecialTokenPolicy SpecialTokens Tokenized TokenizedType TokenizerUserMessagePosition) ImageEncoder) Tekkenizerc steZdZdZ  d*dededBdedBffdd ZededBfd d Z de e fd d Z e d edee e ffddZededede e fddZedededede e fddZedede e fddZde e e dBde ede de ddf ddZede eddfd d!Zd eeefdefd"d#Zd+d$e e d%e dBde!fd&d'Z"d$e e de!fd(d)Z#Z$S),InstructTokenizerBasezBase instruct tokenizer.N tokenizer image_encoder audio_encodercs&||_||_||_t|||dS)zInitialize the instruct tokenizer. Args: tokenizer: The tokenizer to use. image_encoder: The image encoder to use if any. audio_encoder: The audio encoder to use. N)r.r/r0super__init__selfr.r/r0 __class__f/home/ubuntu/veenaModal/venv/lib/python3.10/site-packages/mistral_common/tokens/tokenizers/instruct.pyr2:s zInstructTokenizerBase.__init__returncCs|jSN)r/r4r7r7r8 mm_encoderLsz InstructTokenizerBase.mm_encodercCs |jjgS)zReturn the start tokens.)r.bos_idr;r7r7r8startSs zInstructTokenizerBase.startrequestcCsBd}d}tt|jD]\}}t|tr|dkr|}|}q ||fS)zFind the first and last user message in the request. Args: request: The request to search for user messages. Returns: The index of the first and last user message. )list enumeratemessages isinstancer)r? last_user_idxfirst_user_idximsgr7r7r8find_first_last_userWs  z*InstructTokenizerBase.find_first_last_usermessageis_before_last_user_messagecCtd)zEncode a tool message. Raises: NotImplementedError: The tool message is not implemented for the base tokenizer. zTool message not implementedNotImplementedErrorr4rJrKr7r7r8encode_tool_messagejz)InstructTokenizerBase.encode_tool_messagecontinue_messagecCrL)zEncode an assistant message. Raises: NotImplementedError: The assistant message is not implemented for the base tokenizer. z!Assistant message not implementedrMr4rJrKrRr7r7r8encode_assistant_messagess z.InstructTokenizerBase.encode_assistant_messagechunkcCrL)zEncode a think chunk. Raises: NotImplementedError: The think chunk is not implemented for the base tokenizer. zThink chunk not implementedrMr4rUr7r7r8 encode_think~rQz"InstructTokenizerBase.encode_think tokenizedrC max_tokenslast_user_message_indexcCdSr:r7)r4rXrCrYrZr7r7r8_truncate_for_max_tokenssz.InstructTokenizerBase._truncate_for_max_tokenscCr[r:r7clsrCr7r7r8validate_messagessz'InstructTokenizerBase.validate_messagesc Csg}g}d}g}||j||\}}t|jD]\}} |jr3|t|jdkr3t| ts3tdt| t rW|j | |j ||k||k|j dd\} } } | | | | nIt| tre|| ||k} n;t| tr|jou|t|jdk} |j| ||k| d} |t|jdkr| }nt| tr|| } n tdt| || q|jdur|||j|j||}|D] }|dur| |qt||j|tjd|||d S) zEncode an instruct request. Args: request: The request to encode. Returns: The encoded tokens. Nz?Cannot continue final message if it is not an assistant messageT) system_promptforce_img_first)rRzUnknown message type special_token_policy)tokenstext prefix_idsimagesaudios)r_rCrIrBcontinue_final_messagelenrDrr rencode_user_messageavailable_toolsraextendrrPrTrencode_system_messager typeappendtruncate_at_max_tokensr\r>r'decoder%KEEP)r4r?rhrirg tokens_listrFrEmsg_idxrH new_tokens new_images new_audiosrRretokr7r7r8encode_instructsv             z%InstructTokenizerBase.encode_instructrerdcCs|jj||dS)aDecode tokens to a string. Args: tokens: The tokens to decode. special_token_policy: The policy to use for special tokens. Passing `None` will default to `self._special_token_policy` for [Tekkenizer][mistral_common.tokens.tokenizers.tekken.Tekkenizer] and `SpecialTokenPolicy.IGNORE` for [SentencePieceTokenizer][mistral_common.tokens.tokenizers.sentencepiece.SentencePieceTokenizer]. Note that passing `None` will be deprecated and `special_token_policy` will default to `SpecialTokenPolicy.IGNORE` in `mistral_common=1.10.0`. Returns: The decoded string. rc)r.rs)r4rerdr7r7r8rsszInstructTokenizerBase.decodecCs |j|Sr:)r. _to_string)r4rer7r7r8r|s z InstructTokenizerBase._to_stringNNr:)%__name__ __module__ __qualname____doc__r)r+r r2propertyr<rAintr> staticmethodrtuplerIrrboolrPrrTrrWr\ classmethodrr_rr'r{r%strrsr| __classcell__r7r7r5r8r-5sd      Qr-c@s0eZdZdZ  d dedeedBdedededBd ed e ee ee j ee ffd d Zded ee fd dZ  d deeeBdededBd ed e ee ee j ee ff ddZdeded ee fddZdededed ee fddZded ee fddZded efddZded efddZdS)!InstructTokenizerV1zrInstruct tokenizer V1. This tokenizer has basic for messages. It does not support tools or image inputs. NFrJrmis_lastis_firstrarbr9c Cstt|jts Jd|jdusJdd}|r!|r!|d|j}n|j}d|d}|j|ddd \} } } | | | fS) arEncode a user message. Args: message: The message to encode. available_tools: Not used. is_last: Not used. is_first: Whether the message is the first one. system_prompt: The system prompt. force_img_first: Not used. Returns: The encoded tokens and empty list. "Message content must be normalizedNz(InstructTokenizerV1 cannot encode images z[INST] z [/INST]F)contentrra)rDrrr/encode_user_content) r4rJrmrrrarbr message_txt curr_tokensimageaudior7r7r8rls  z'InstructTokenizerV1.encode_user_messagecCtd|jj)Nz,System message encoding not implemented for )rNr6r~r4rJr7r7r8ro'z)InstructTokenizerV1.encode_system_messagercCs>t|tsJ|r|r|d|}|jj|ddd}|ggfS)a*Encode a user content. Args: content: The content to encode. is_last: Whether the message is the last one. system_prompt: The system prompt. force_img_first: Not used. Returns: The encoded tokens and empty list. rFboseos)rDrr.encode)r4rrrarbrer7r7r8r*s   z'InstructTokenizerV1.encode_user_contentrKcCrL)zEncode a tool message. Raises: TokenizerException: The tool message is not implemented for this version. &Tools not implemented for tokenizer V1r rOr7r7r8rPDz'InstructTokenizerV1.encode_tool_messagerRcCst|ts J||jdurt|jdkrtd|r"|jr"td|jr:t|jts/Jd|j j |jddd}n t|jd|j|jsQ|sQ| |j j |S) [Encode an assistant message. Args: message: The message to encode. is_before_last_user_message: Not used. continue_message: Whether to continue the message generation. Only use this if the assistant message is the last message. Returns: The encoded tokens. NrrU`continue_message` is only supported for assistant messages that have `prefix=False`.z4Message content must be a string for tokenizer < V13Frz // ) rDr tool_callsrkr prefixrrrr.rrqeos_idr4rJrKrRrr7r7r8rTLs  z,InstructTokenizerV1.encode_assistant_messagerUcCrL)zEncode a think chunk. Raises: TokenizerException: The think chunk is not implemented for this version. z*Think not implemented for tokenizer < V13.rrVr7r7r8rWjrz InstructTokenizerV1.encode_thinkr?cCr)zEncode a FIM request. Raises: TokenizerException: The FIM request is not implemented for this version. zFIM not available for r r.versionr4r?r7r7r8 encode_fimrszInstructTokenizerV1.encode_fimcCr)Nz Transcription not available for rrr7r7r8encode_transcriptionzrz(InstructTokenizerV1.encode_transcriptionNF)r~rrrrrArrrrrnpndarrayrrlrrorrrrPrrTrrWr r'rrrr7r7r7r8rs\   #   rcs~eZdZdZejZ  d-dededBde dBffdd Z  d.d e d e e dBd ed ed edBdedee ee eje effddZdedefddZdee eBdefddZdedeeeffddZd edede efddZdedeeeffddZd e de efd d!Z!d e de efd"d#Z"d e ded$ede efd%d&Z#d'ede efd(d)Z$d*e%de&fd+d,Z'Z(S)/InstructTokenizerV2z`Instruct tokenizer V2. This tokenizer adds supports to images, tools and FIM requests. Nr.r/r0cst||||jtjj|_|jtjj|_ |jtj j|_ |jtj j|_ |jtjj|_|jtjj|_|jtjj|_|jtjj|_|jtjj|_|jtjj|_dSInitialize the tokenizer. Args: tokenizer: The tokenizer to use. image_encoder: The image encoder to use. audio_encoder: The audio encoder to use. N)r1r2r.get_special_tokenr& begin_instvalue BEGIN_INSTend_instEND_INST begin_toolsBEGIN_AVAILABLE_TOOLS end_toolsEND_AVAILABLE_TOOLSbegin_tool_resultsBEGIN_TOOL_RESULTSend_tool_resultsEND_TOOL_RESULTSr TOOL_CALLSrBOSrPREFIXsuffixSUFFIXr3r5r7r8r2s zInstructTokenizerV2.__init__FrJrmrrrarbr9cCsd}||o |jtjkO}||o|jtjkO}g}|r:|r:dd|D} |jjtj| ddddd} |jg| |j }|j |j |||d\} } } g||j }|j g}|| |}|| | fS)aEncode a user message. Args: message: The message to encode. available_tools: The list of available tools if any. is_last: Whether the message is the last one. is_first: Not used. system_prompt: The system prompt. force_img_first: Whether to force the image to be first. Returns: The encoded tokens and the list of images. FcSsg|]}|qSr7) model_dump).0toolr7r7r8 sz;InstructTokenizerV2.encode_user_message.. ensure_asciir)rrrarb)&_user_message_position_to_encode_toolsr*firstlastr.rjsondumpsrrrrrr)r4rJrmrrrarbdo_encode_tools tools_tokenstoolstools_json_tokensrerr prefix_tokens suffix_tokensrr7r7r8rls.   z'InstructTokenizerV2.encode_user_messagercCs&zt|WStjy|YSwr:)rloadsJSONDecodeErrorr4rr7r7r8_parse_json_contents  z'InstructTokenizerV2._parse_json_contentcCs(t|trddd|D}||S)Nrcs|]}|jVqdSr:rfrrUr7r7r8 z:InstructTokenizerV2._parse_tool_content..)rDrAjoinrrr7r7r8_parse_tool_contents  z'InstructTokenizerV2._parse_tool_content tool_messagecCs|j||jdS)z8Bit of a hack due to the way tool results are tokenized.)namer)rrrr4rr7r7r8_prepare_tool_results z(InstructTokenizerV2._prepare_tool_resultrKcCsB|rgStj||gdd}|jg|jj|ddd|j}|S)aEncode a tool message. Args: message: The message to encode. is_before_last_user_message: Whether the message is before the last user message. If true, the message is not encoded. Returns: The encoded tokens. Frrrrrrr.rrr4rJrKtool_result_strrr7r7r8rPs z'InstructTokenizerV2.encode_tool_message tool_callcCs|jj||jjdS)z:Bit of a hack due to the way function calls are tokenized.r arguments)functionrrr)r4rr7r7r8_prepare_function_calls z*InstructTokenizerV2._prepare_function_callcCsB|js Jd|t|jtsJd|jj|jddddS)Nz)Assistant message must have content. Got 3Message content must be a string for tokenizer < V7 Fr)rrDrr.rrstriprr7r7r8(_encode_normal_content_assistant_messageszs zInstructTokenizerV2.encode_fimr}r))r~rrrr*rrr)r+r r2rrArrrrrrrrrlrrrrrdictrrPrrrrrrTrr r'rrr7r7r5r8r~s^  2  $rcseZdZdZ  d%dededBdedBffdd Zded e e e ffd d Z d e d e e e ffd dZde ded eefddZdededed eeffdd Zede eBeBd eeeddffddZedeeBd eeeejdffddZedeeBd eeede ffddZde e!Bd eeeejdBe dBffddZde"e!d eeeeejee ffddZ#  d&de ee$Bd ed!e dBd"ed eeeeejee ff fd#d$ Z%Z&S)'InstructTokenizerV3zxInstruct tokenizer V3. The only difference with V2 tokenizer is that it encodes the tool messages differently. Nr.r/r0cstj|||ddS)r)r/r0N)r1r2r3r5r7r8r2[s zInstructTokenizerV3.__init__rr9cCs6|jj||jjd}|jr|jdkr|j|d<|S)Nrnullid)rrrrr)r4r function_callr7r7r8rjs   z*InstructTokenizerV3._prepare_function_callrcCs&|jdus Jd||j|jdS)Nz7Tool message has to have the tool call id defined in v3)rcall_id) tool_call_idrrrr7r7r8rus z(InstructTokenizerV3._prepare_tool_resultrJrKcCs8tj||dd}|jg|jj|ddd|j}|S)aEncode a tool message. Note: Same as [V2][mistral_common.tokens.tokenizers.instruct.InstructTokenizerV2.encode_tool_message] but tools are not wrapped in a list and the history is also tokenized. Args: message: The message to encode. is_before_last_user_message: Whether the message is before the last user message. If true, the message is not encoded. Returns: The encoded tokens. Frrrrr7r7r8rP}sz'InstructTokenizerV3.encode_tool_messagerRcst|d|S)a Encode an assistant message. Note: Same as [V2][mistral_common.tokens.tokenizers.instruct.InstructTokenizerV2.encode_assistant_message] but always encode the tool history. continue_message: Whether to continue the message generation. Only use this if the assistant message is the last message. Args: message: The message to encode. is_before_last_user_message: Not used. Returns: The encoded tokens. F)r1rTrSr5r7r8rTsz,InstructTokenizerV3.encode_assistant_messagerUcCr[r:r7rVr7r7r8_encode_content_chunkz)InstructTokenizerV3._encode_content_chunkcCr[r:r7rVr7r7r8rrcCr[r:r7rVr7r7r8rrcCst|tr|jj|dddddfSt|tr#|jj|jdddddfSt|tr0||ddfSt|tt frL|j dus@Jd| |}|j |j dfSt|t tfrh|jdus\Jd||}|j d|jfStd|)NFrz+Make sure to define a image encoder at initz+Make sure to define a audio encoder at initzUnknown chunk type: )rDrr.rrrfrrWrrr/rerr rr0rr)r4rU img_encodingaudio_encodingr7r7r8rs     rc Cs^g}g}g}|D]!}||\}}}|||dur |||dur)||q|||fSr:)rrnrq) r4rrerhrrU chunk_tokens maybe_image maybe_audior7r7r8_encode_content_chunkss    z*InstructTokenizerV3._encode_content_chunksFrrarbcs*t|tr t|||Sg}g}g}t|dko!t|dttf}|r.|r.|d|dg}d} |D]]} d} | rM|rM|rMd} |d} ||jj| ddd7}t| t t frn| r`Jd t | d | | \} } }| |nt| ttfr| | \} }} | |n| | d} || q2|||fS) HEncode a user content. Args: content: The content to encode. is_last: Whether the message is the last one. system_prompt: The system prompt. force_img_first: Whether to force the image to be first. Returns: The encoded tokens and the images. rr`rTrFrrzEIt is not possible that `content` is non-empty when chunk is of type .)rDrr1rrkrrr.rr rrprrqrn)r4rrrarbrerhrhas_one_img_one_text_first first_chunkrU content_strr_ chunk_audio chunk_imager5r7r8rs6      z'InstructTokenizerV3.encode_user_contentr}r)'r~rrrr)r+r r2rrrrrrrrrArrPrrTrrrrrrrrrr rrrrrrrrr7r7r5r8rSs` *(&.  rcseZdZdZ  d2dededBdedBddffdd Zd eee dBd ee d e d e ddf d dZ de dee fddZ  d3deeeBdededBdedeee eejeeff fdd Z  d3dedeedBdedededBdedeee eejeefffdd ZdedefddZdedefd d!Zd4d"eeBd#edBdefd$d%Zdedefd&d'Z e!d ee"ddfd(d)Z#e$d ee"defd*d+Z%de&d,edee fd-d.Z'de d,ed/edee fd0d1Z(Z)S)5InstructTokenizerV7a%Instruct tokenizer V7. The difference with V3 tokenizer is that it encodes the system prompts differently: - in V7 the system prompts are treated as separate SystemMessages - they are no longer prepended to the last user message - they are printed between special tokens Nr.r/r0r9cszt||||jtjj|_|jtjj|_ |jtj j|_ d|_ |dur9|j js;|jtjj|_ dSdSdSr)r1r2r.rr& begin_systemr BEGIN_SYSTEM end_system END_SYSTEMbegin_tool_contentBEGIN_TOOL_CONTENT TRANSCRIBE audio_config is_streaming transcriber3r5r7r8r2szInstructTokenizerV7.__init__tokenized_messagesrCrYrZcstddD|dtddffdd }d}dkrb|tkrb|||d7}t|dtrX|tkrXt|tsX|||d7}|tkrXt|trCdkrb|tks%dkrjtd dS) Ncss |] }|durt|VqdSr:)rk)rtr7r7r8r7sz?InstructTokenizerV7._truncate_for_max_tokens..idxr9csJt|tr dS|krdS|}|dusJt|8d|<dSr:)rDrrk)rrzrZrCto_droprr7r8drop9s   z:InstructTokenizerV7._truncate_for_max_tokens..droprr`z+Input couldn't fit in truncate_at_max_token)sumrrkrDrr )r4rrCrYrZr  current_idxr7rr8r\+s   z,InstructTokenizerV7._truncate_for_max_tokensrJcCsF|jg}t|j}trt|dg}|||d7}||j|S)zEncode a system message. Args: message: The message to encode. Returns: The encoded tokens. rr)rrDrrrrrqr)r4rJrerr7r7r8roTs    z)InstructTokenizerV7.encode_system_messageFrrrarbc sz|dusJdt|trt|||St|dko#t|dttf}|r0|r0|d|dg}||\}}}|||fS)rN?in Tokenizer V7 we don't encode system prompts in user messagesrr`r)rDrr1rrkrrr) r4rrrarbr rerhrr5r7r8res  z'InstructTokenizerV7.encode_user_contentrmrc s8|dusJdtj||||d|d\}}} ||| fS)aEncode a user message. Args: message: The message to encode. available_tools: The list of available tools if any. is_last: Whether the message is the last one. is_first: Whether the message is the first one. system_prompt: Not used. force_img_first: Whether to force the image to be first. Returns: The encoded tokens and the list of images. Nr#)rrrarb)r1rl) r4rJrmrrrarbrerhrr5r7r8rls z'InstructTokenizerV7.encode_user_messager?cCsd|jdus Jd|j|jjjtjkr||S|jjjtjkr'||Std|jjjd)a Encodes an audio transcription request into a tokenized format. This method processes a transcription request containing audio data, encodes the user message, and returns the tokenized output. Args: request: The transcription request object containing the audio data to be encoded. Returns: Tokenized: The tokenized representation of the audio data, including processed audio and tokens Nz6Audio encoder must be defined, got self.audio_encoder=zxTranscription format should be one of 'instruct', 'streaming', got self.audio_encoder.audio_config.transcription_format=r ) r0rtranscription_formatr!INSTRUCT_encode_instruct_transcription STREAMING_encode_streaming_transcriptionr rr7r7r8rs  z(InstructTokenizerV7.encode_transcriptioncCs|jtjksJd|j|jdusJ|jjd|}|jtt |j dgdgdddd\}}}g||}|j durQd|j }||j j |ddd 7}||jt||j ||d S) Nz=Request must not be in streaming mode, got request.streaming=z! needs to have a TRANSCRIBE token) input_audio)rT)rmrrrazlang:Frrerfri) streamingrDISABLEDrr6r~r>rlrr rlanguager.rrqr'r|)r4r?rrer rlanguage_stringr7r7r8r&s$     z2InstructTokenizerV7._encode_instruct_transcriptionrtranscription_delay_mscCsX|jdus Jd|jt|trt|nt|}|j||}t|j|j gdS)NFAudio encoder must be defined to encode audio, got self.audio_encoder=)reri) r0rDrr from_base64 from_bytes encode_audior'rer)r4rr/_audio audio_encr7r7r8 _encode_audios  z!InstructTokenizerV7._encode_audiocCs<|jtjkr||jj|j}||j}|j }nw|jtj kr|j dus'J|j |j\}}||g}t |jjdkrhtd|jdtt|jjtsPJt|jj}tt|j|jf|j|jg}n|jjrpJd|j dus}Jd|j ||j |j}ntd|jt||j|tjd|dS) Nrz%Passing audio with request.streaming=zf is deprecated. Make sure to not pass any audio to `TranscriptionRequest` when doing online streaming.z~For online streaming, no audio bytes should be passed in the first request. Audio buffering is taken care of directly by vLLM.r0z9Request must be in streaming mode, got request.streaming=rcr*) r+rOFFLINEr6rdatatarget_streaming_delay_msr>reriONLINEr0get_padding_audiorkwarningswarn FutureWarningrDrrr1r concatenate audio_array sampling_rateformatencode_streaming_tokensrr'rsr%rt)r4r?rXrerileft_pad right_pad request_audior7r7r8r(sD      z3InstructTokenizerV7._encode_streaming_transcriptioncCs,||rtdd|DrtddSdS)Ncs|]}t|tVqdSr:)rDrrrJr7r7r8rz8InstructTokenizerV7.validate_messages..z9System messages are not yet allowed when audio is present) _has_audioanyrr]r7r7r8r_s z%InstructTokenizerV7.validate_messagescCstdd|DS)Ncss:|]}t|tot|jtotdd|jDVqdS)csrGr:)rDr rr7r7r8r!rIz;InstructTokenizerV7._has_audio...N)rDrrrArKrHr7r7r8rs  z1InstructTokenizerV7._has_audio..)rK)rCr7r7r8rJszInstructTokenizerV7._has_audiorKcCsr|jdusJt|jtsJd|jj|jddd}|jj|jddd}|jg||j}g|||j}|S)aEncode a tool message. Note: Same as [V3][mistral_common.tokens.tokenizers.instruct.InstructTokenizerV3.encode_tool_message] but tools are not wrapped in a list and history is also tokenized Args: message: The message to encode. is_before_last_user_message: Not used. Returns: The encoded tokens. NrFr) rrDrrr.rrrr)r4rJrKtool_call_id_tokensrerrr7r7r8rP%s$z'InstructTokenizerV7.encode_tool_messagerRcCs|js |js td||r|jrtdg}|jr7t|jtr'||}nt|jtr7|| |jd7}|jrA|| |7}|jsM|sM| |j j |S)rrrr)rrr rrrDrrrArrrqr.rrr7r7r8rTDs"      z,InstructTokenizerV7.encode_assistant_messager}rr:)*r~rrrr)r+r r2rArrr\rrorrrrrrrrrrrlrr'rr&bytesfloatr6r(rrr_rrJrrPrTrr7r7r5r8r s  ) % #  4rc sVeZdZdZ  d dededBdedBddffdd Zd ede e fd d Z Z S) InstructTokenizerV11zInstruct tokenizer V11. The difference with V7 tokenizer is that it encodes tool calls differently: Tool call results are encoded as : - [begin tool call] call_name_tokens [call id] call_id_tokens [args] content tokens Nr.r/r0r9cs8t||||jtjj|_|jtjj|_ dSr:) r1r2r.rr&argsrARGSrCALL_IDr3r5r7r8r2oszInstructTokenizerV11.__init__rJcCs|js Jd|g}|jD]B}||}g}d|vr+|jg|jj|dddd}||jg|jj|dddd||j|jjtj|dddddd7}q|S)NrrFrrrr) rrrRr.rrrQrr)r4rJrrpreparedidsr7r7r8rys&  z= v13rcsrr:rrr7r7r8rrz;InstructTokenizerV13.encode_tool_message..Fr) rrrDrrr.rrr)r4rJrKrrerr7r7r8rPs  z(InstructTokenizerV13.encode_tool_messagerUcCsZ|jdus Jd|jdusJd|jj|jddd}|jg|}|jr+||j|S)zEncode a thinking chunk. Args: chunk: The thinking chunk to encode. Returns: The encoded tokens. Nz2think tokens are not available for this tokenizer.Fr)rYrZr.rthinkingclosedrq)r4rUre think_tokensr7r7r8rWs  z!InstructTokenizerV13.encode_thinkr})r~rrrr*rrr)r+r r2rrArrrrrPrrWrr7r7r5r8rUs"rU)Drr<abcrtypingrrrrnumpyrmistral_common.audiormistral_common.exceptionsrr r r #mistral_common.protocol.fim.requestr &mistral_common.protocol.instruct.chunkr rrrrrrr)mistral_common.protocol.instruct.messagesrrrrrr(mistral_common.protocol.instruct.requestr+mistral_common.protocol.instruct.tool_callsrr-mistral_common.protocol.transcription.requestrr&mistral_common.tokens.tokenizers.audior r!%mistral_common.tokens.tokenizers.baser"r#r$r%r&r'r(r)r*&mistral_common.tokens.tokenizers.imager+'mistral_common.tokens.tokenizers.tekkenr,r-rrrrrOrUr7r7r7r8sL   ( ,   H  V8_&