o پi=@sTddlZddlZddlmZmZmZmZmZddlm Z e e Z GdddZ dS)N)AnyDictListOptionalUnion)GenerateReqInputc@seZdZ  ddeeeeeeefdeededee deee f dd Z d d Z d eed eeededeefddZ d de d edeededeee f ddZde deededeee fddZ      d!d eeeeefd eeeeeeeefdeeedededee deee fddZdeee fdeededee fddZdedeedeee ffddZdS)"TokenizerManagerMultiItemMixinFNpromptslabel_token_ids apply_softmaxrequestreturncst|tst|tr"|rt|dtr"|jd|||d|dIdHSt|tr>|r0t|dtr>|jg|||d|dIdHStd)a Score probabilities of specified token IDs after each *full prompt*. This is a thin wrapper over `score_request` that treats `prompts` as already-composed inputs (i.e., no query/item concatenation needed). Args: prompts: A single prompt string, a list of prompt strings, or a list of pre-tokenized prompt token ID sequences. label_token_ids: Token IDs to compute probabilities for. apply_softmax: Whether to normalize probabilities using softmax. request: Optional FastAPI request object. Returns: List of score lists, one for each prompt, each in the order of label_token_ids. rF)queryitemsr r item_firstr Nz'Invalid prompts type for score_prompts.) isinstancestrlist score_request ValueError)selfr r r r ri/home/ubuntu/.local/lib/python3.10/site-packages/sglang/srt/managers/tokenizer_manager_multiitem_mixin.py score_prompts s4      z,TokenizerManagerMultiItemMixin.score_promptsc Cst|jdrD|jjdurF|jdurHz|jj|jjgdd|_WdStyC}ztd|jjd|d|_WYd}~dSd}~wwdSdSdS)zMInitialize multi-item delimiter text from token ID after tokenizer is loaded.multi_item_scoring_delimiterNF)skip_special_tokensz!Failed to decode delimiter token z: ) hasattr server_argsr tokenizerdecodemulti_item_delimiter_text Exceptionloggerwarning)rerrr%_initialize_multi_item_delimiter_text<s&   zDTokenizerManagerMultiItemMixin._initialize_multi_item_delimiter_textrrdelimiter_token_idcCs8|dd}|D] }||||q|||S)ap Build a single token sequence for multi-item scoring. Format: queryitem1item2item3 Args: query: Query token IDs items: List of item token ID sequences delimiter_token_id: Token ID to use as delimiter Returns: Combined token sequence N)appendextend)rrrr'combined_sequenceitemrrr _build_multi_item_token_sequenceNs    z?TokenizerManagerMultiItemMixin._build_multi_item_token_sequenceresultsc Cst|tr |dn|}|ddg}|s#td|ddddg}t|tr.t|nd} | d} t|| krRtd | d | d t|d |dddt|dkrZdnd} t| D]} | | } || }|||}||||}||q`|S) a Process results from multi-item scoring request. Extracts logprobs at delimiter positions from input_token_ids_logprobs. Args: results: Results from generate_request items: List of items being scored label_token_ids: Token IDs to extract scores for apply_softmax: Whether to apply softmax normalization batch_request: The original batch request containing input sequence Returns: List of score lists, one for each item r meta_infoinput_token_ids_logprobszAinput_token_ids_logprobs is empty for multi-item scoring request id zV. This indicates token_ids_logprobs were not computed properly for Mutil Item Scoring.z Expected z6 input_token_ids_logprobs for multi-item scoring with z items, but got z. Request ID: ) rrget RuntimeErrorlenrange_extract_logprobs_for_tokens_convert_logprobs_to_scoresr()rr-rr r batch_request single_resultinput_logprobsscores num_itemsexpected_logprobs_count start_idxitem_idx logprob_idxitem_logprobs_datalogprobs score_listrrr#_process_multi_item_scoring_resultshs>   zBTokenizerManagerMultiItemMixin._process_multi_item_scoring_resultsc Csvg}|D]4}|ddg}|rt|dkr$td|dddd||d|}||||}||q|S)a Process results from single-item scoring request. Single-item scoring results are stored in output_token_ids_logprobs. Args: results: Results from generate_request label_token_ids: Token IDs to extract scores for apply_softmax: Whether to apply softmax normalization Returns: List of score lists, one for each result r.output_token_ids_logprobsrz%output_logprobs is empty for request r0r1.)r3r5r4r7r8r() rr-r r r<resultoutput_logprobsrCrDrrr$_process_single_item_scoring_resultss zCTokenizerManagerMultiItemMixin._process_single_item_scoring_resultsrcs|dur td|jdur&|jj}|D]}||kr%td|d|dq|jjduo0|jdu} t|d| r8dndd d did } ttrt|tsXt|t r|rXt|dtrt|tr`|gn|} | r{|j} | | } | | | }|g| _ n[|rfd d | D}n fdd | D}|| _ nBtt rt|t r|rt|dt r| r|jj}| ||}|g| _ n|rŇfdd |D}n fdd |D}|| _ ntd|| |IdH}| r|||||| S||||S)a Score the probability of specified token IDs appearing after the given (query + item) pair. This method supports two scoring approaches: 1. Single-Item scoring (default): Process each query+item pair independently 2. Multi-Item scoring: When multi_item_scoring_delimiter is set, combine query and multiple items into a single sequence using delimiter for efficient processing. Note: item_first parameter is ignored in multi-item scoring mode since it uses a fixed format: queryitem1item2item3 Multi-item scoring works with both text and pre-tokenized inputs: - Text: queryitem1item2item3 - Tokens: queryitem1item2item3 Args: query: The query text or pre-tokenized query token IDs items: The item text(s) or pre-tokenized item token IDs label_token_ids: List of token IDs to compute probabilities for apply_softmax: Whether to normalize probabilities using softmax item_first: If True, prepend items to query. Ignored for multi-item scoring. request: Optional FastAPI request object Returns: List of lists containing probabilities for each item and each label token Nz label_token_ids must be providedz Token ID z# is out of vocabulary (vocab size: )TrFmax_new_tokens)token_ids_logprobreturn_logproblogprob_start_lenstreamsampling_paramscsg|]}|qSrr.0r+rrr z@TokenizerManagerMultiItemMixin.score_request..csg|]}|qSrrrSrUrrrVrWcsg|]}|qSrrrSrUrrrV3csg|]}|qSrrrSrUrrrV5rXz;Invalid combination of query/items types for score_request.)rr vocab_sizerrr!rrrrjointextr, input_idsgenerate_request __anext__rErJ)rrrr r rr rYtoken_iduse_multi_item_scoringr9 items_list delimitercombined_items single_promptr r'combined_input_idsinput_ids_listr-rrUrrs"          z,TokenizerManagerMultiItemMixin.score_requestrCcsLddl}fdd|D}|r|j||dd}|Sdd|D}|S)a\ Convert logprobs dictionary to ordered score list. Args: logprobs: Dictionary mapping token_id to logprob label_token_ids: Token IDs in desired order apply_softmax: Whether to apply softmax normalization Returns: List of scores in the same order as label_token_ids rNcsg|] }|tdqS)-inf)r3float)rTr_rCrrrV\szNTokenizerManagerMultiItemMixin._convert_logprobs_to_scores..)dimcSs&g|]}|tdkrt|ndqS)rgg)rhmathexp)rTxrrrrVds)torchsoftmaxtensortolist)rrCr r rnrDrrirr8Is z:TokenizerManagerMultiItemMixin._convert_logprobs_to_scores logprobs_datacCs,i}|r|D] \}}}||vr|||<q|S)a% Extract logprobs for specified token IDs from logprobs data. Args: logprobs_data: List of (logprob, token_id, text) tuples label_token_ids: Token IDs to extract logprobs for Returns: Dictionary mapping token_id to logprob r)rrrr rClogprobr__rrrr7js z;TokenizerManagerMultiItemMixin._extract_logprobs_for_tokens)FN)N)NNNFFN)__name__ __module__ __qualname__rrrintboolrrrhrr&r,rErJrrr8r7rrrrr s  1     ?  '   }  ! r)loggingrktypingrrrrrsglang.srt.managers.io_structr getLoggerrur#rrrrrs