o ۾i;@slddlmZddlZddlmZddlmZddlm Z ddl m Z ddl m Z e eZGdd d e ZdS) )SequenceN)PreTrainedTokenizerBase)ChatCompletionRequest) DeltaMessage) init_logger)ReasoningParsercs eZdZdZdeffdd ZdededeedBedBffd d Z d ed ed ede e de e de e de dBfddZ dedefddZdedefddZd ed ede fddZd eded ede fddZd ededed ede de f dd Zd edeedBe dBedBffd!d"ZZS)#GraniteReasoningParserz Reasoning parser for IBM Granite. IBM granite models currently use "Here is my thought process:" and "Here is my response:" to separate its thinking / response outputs. tokenizercstj|g|Ri|d|_d|_t|jd|jdtj|_ddg|_ddg|_ d |_ d |_ t d d |jD|_ dS) Nz&(?:Here's|Here is) my thought process:z(?:Here's|Here is) my response:z(.*?)z(.*)zHere's my thought process:zHere is my thought process:zHere's my response:zHere is my response::Herecss|]}t|VqdSN)len.0 think_startr[/home/ubuntu/.local/lib/python3.10/site-packages/vllm/reasoning/granite_reasoning_parser.py 3s z2GraniteReasoningParser.__init__..)super__init__think_start_exprresponse_start_exprrecompileDOTALLreasoning_regexvalid_think_startsvalid_response_startsseq_boundary_endseq_boundary_startmaxlongest_think_start)selfr argskwargs __class__rrrs zGraniteReasoningParser.__init__ model_outputrequestreturnNcCs8|j|}|s d|fS|d\}}|s|dfS||fS)aExtract the reasoning content & content sections, respectively. If the sequence doesn't match what we expect, i.e., the model generates something else, all content is considered non-reasoning content. Args: model_output (str): Output of the model to be parsed. request (ChatCompletionRequest): Request being processed. Returns: tuple[Optional[str], Optional[str]]: Tuple pair containing the reasoning content and non-reasoning content. Nr)rfindall)r"r'r(re_match reasoningresponse_contentrrrextract_reasoning7s  z(GraniteReasoningParser.extract_reasoning previous_text current_text delta_textprevious_token_idscurrent_token_idsdelta_token_idsc Csh||\}}} |s|||} n| s||||} n|dus!J|||| ||} | js2| js2dS| S)aExtract the reasoning content / content emitted by granite models; If the sequence doesn't match what we expect, i.e., the model generates something else, all content is considered non-reasoning content. NOTE: Granite models do not use a special token to start their reasoning and response sections; instead they have token sequences, e.g., Here is my thought process: Foo Here is my response: Bar This increases the complexity of correctly handling streams, since we need to watch for specific sequences and correctly parse them without dropping content that is potentially overlapping & spanning multiple delta messages. Args: previous_text (str): Previous text outside of this delta message. current_text (str): Previous text + delta text. delta_text (str): Text to consider and parse content from. previous_token_ids (Sequence[int]): Token IDs of previous_text. current_token_ids (Sequence[int]): Token IDs of current_text. delta_token_ids (Sequence[int]): Token IDs of delta_text. Returns: Union[DeltaMessage, None] DeltaMessage with either reasoning content or content, or None. N)_get_content_sections+_get_delta_message_with_no_reasoning_bounds*_get_delta_message_with_no_response_bounds#_get_delta_message_with_both_boundscontentr,) r"r/r0r1r2r3r4r, resp_seq_lenr9 delta_messagerrrextract_reasoning_streamingNs #   z2GraniteReasoningParser.extract_reasoning_streamingtextctfdd|jDS)zCheck if a text matches one of the possible start reasoning seqs. Args: text (str): Text to check for leading substr. Returns: bool: True if any of the possible reasoning start seqs match. c3|]}|VqdSr  startswithrr=rrrs  zDGraniteReasoningParser._is_reasoning_start_substr..)anyrr"r=rrBr_is_reasoning_start_substrs z1GraniteReasoningParser._is_reasoning_start_substrcr>)zCheck if a text matches one of the possible start response seqs. Args: text (str): Text to check for leading substr. Returns: bool: True if any of the possible response start seqs match. c3r?r r@rresponse_startrBrrr  zCGraniteReasoningParser._is_response_start_substr..)rCrrDrrBr_is_response_start_substrs z0GraniteReasoningParser._is_response_start_substrcCs\t|t|}||}||d|}|r |s td|dS|r(tdddStd|dS)a^Parse the delta message when the current text has not yet completed its start of reasoning sequence. Args: current_text (str): The full previous + delta text. delta_text (str): Text to consider and parse content from. Returns: DeltaMessage: Message containing the parsed content. Nr,r9)r rEr)r"r0r1prev_longest_length is_substr was_substrrrrr6s   zBGraniteReasoningParser._get_delta_message_with_no_reasoning_boundsr,c s0tfdd|jD}|dus|rtdddS|dt| }|||j}||j}|dkr<|||dnd}|dkrK||dnd} |dkrZ|||dnd} | rdtdddS|sx| rrt|d|ddSt|ddS| r||d|d|}t|ddSt||d|ddS)a5Parse the delta message when the current text has both reasoning content with no (response) content. NOTE that we may have overlapping tokens with the start of reasoning / start of response sequences on either side of the delta text. Args: current_text (str): The full previous + delta text. reasoning (str): reasoning content from current_text. delta_text (str): Text to consider and parse content from. Returns: DeltaMessage: Message containing the parsed content. c3s|]}|VqdSr )endswithrFr0rrrrHzTGraniteReasoningParser._get_delta_message_with_no_response_bounds..NrJrF)rCrrr rfindrrI) r"r0r,r1ends_with_start_response_seqr/prev_idx delta_idxprev_was_substrdelta_continues_substrdelta_new_substrrrOrr7sD        zAGraniteReasoningParser._get_delta_message_with_no_response_boundsr-response_seq_lenc Cs|t| d}t|t||}|dkrd}n$t||t|d} t|t|} | | } | dkr8d} || |}t||dS)aParse the delta message when the current text has both reasoning content and normal (response) content. Args: delta_text: Text to consider and parse content from. reasoning: reasoning content from current_text. response_content: response content from current_text. current_text: The full previous + delta text. response_seq_len: Len of the complete response sequence used. Returns: DeltaMessage: Message containing the parsed content. NrrJ)r r) r"r1r,r-r0rW delta_contentreasoning_end_idxdelta_reasoningstart_reasoning_idx delta_offset start_offsetrrrr8s z:GraniteReasoningParser._get_delta_message_with_both_boundsc sd}d}d}fddt|D}|D]]}|||}|dur8jD]}||ddkr6|d}|d}nq"q|spjD]2} |t|  dd| ddkro|t| } ||| } ||dd} | t| | fSq=q|r~|s~||dddfSdS) aParse the text to extract the reasoning content / content if we have them. Args: current_text (str): The full previous + delta text. Returns: tuple[Optional[str], Optional[int], Optional[str]]: Tuple of len 3 containing the reasoning content, the length of the response seq (if there is one) and the non-reasoning content. rNFcsg|] \}}|jkr|qSr)r)ridxcharr"rr Qs  z@GraniteReasoningParser._get_content_sections..rX)NNN) enumeraterrr ) r"r0current_chunk_startstart_reasoningparsed_contentdelimiter_idxscurrent_chunk_end current_chunkrrG end_reasoningr,r-rrarr5@s8    "  z,GraniteReasoningParser._get_content_sections)__name__ __module__ __qualname____doc__rrstrrtupler.rintrr<boolrErIr6r7r8r5 __classcell__rrr%rrsz  <  " N +r)collections.abcrregexr transformersr0vllm.entrypoints.openai.chat_completion.protocolr'vllm.entrypoints.openai.engine.protocolr vllm.loggerrvllm.reasoningrrlloggerrrrrrs