o ]۷i+@sddlmZmZddlZddlmZmZmZmZm Z m Z m Z m Z ddl Z ddlmZddlmZmZmZddlmZmZddlmZGd d d ZGd d d ZdS) ) dataclassfieldN)AnyCallableDictHashableListOptionalTupleUnion)LMFormatEnforcerException)CharacterLevelParserForceStopParserCharacterLevelParserConfig)TokenizerPrefixTreeTokenizerPrefixTreeNode) TokenListc @sVeZdZdZdeeeeefde eegefde eeefdedef ddZ d S) TokenEnforcerTokenizerDatazTokenEnforcerTokenizerData contains all of the preprocessing for preparing the TokenEnforcer to work with a specific tokenizer. It does some calculations, so it is recommended to reuse it for multiple TokenEnforcersregular_tokensdecoder eos_token_id use_bitmask vocab_sizecsdfdd|D}||_t|j||_||_||_ddd|jjjD|_ |_ ||_ dS)aA Create the tokenizer data that the TokenEnforcer needs. This can be reused for multiple TokenEnforcers if they work with the same tokenizer. :param regular_tokens: A list of tuples (token_id, token_string, is_new_word_token) for all the regular (not special) tokens in the tokenizer vocabulary. Note that token_string is expected to include leading / trailing whitespaces if relevant. :param decoder: A function that decodes a list of token ids into a string. :param eos_token_id: The token id(s) of the end-of-string token(s). csg|] }|dkr|qS)r).0 token_tuplerrT/home/ubuntu/vllm_env/lib/python3.10/site-packages/lmformatenforcer/tokenenforcer.py sz7TokenEnforcerTokenizerData.__init__..css |] }t|dkr|VqdS)r N)len)r token_strrrr !sz6TokenEnforcerTokenizerData.__init__..N) rrtokenizer_treerrjoinrootchildrenkeystokenizer_alphabetrr)selfrrrrrfiltered_regular_tokensrrr__init__s   z#TokenEnforcerTokenizerData.__init__N) __name__ __module__ __qualname____doc__rr intstrboolrr r,rrrrr src @seZdZdZeGdddZdedefddZde e d e fd d Z d e d dfddZdedede deefddZd dde e fddZdS) TokenEnforcerzTokenEnforcer provides a token filtering mechanism, given a CharacterLevelParser and some information about the tokenizer. It is the main entry point for extending lm-format-enforcer to new inference libraries. See __init__() and get_allowed_tokens()c@sBeZdZUeed<eddZedBed<eedZ e e ed<dS)TokenEnforcer.OutputTensorStateparserN)defaultallowed_tokens)default_factorycurrent_word_tokens) r-r.r/r__annotations__rr8rlistr:rr1rrrrOutputTensorState)s r=tokenizer_datar6cCsXi|_||_|j|_|j|_|j|_|j|_i|_|j|_|j|_t |j d}||_ dS)z Create a new TokenEnforcer. :param tokenizer_data: Per tokenizer data that the token enforcer needs in order to operate. :param parser: A CharacterLevelParser that defines the allowed strings. )alphabetN) prefix_states root_parserr$rrrallowed_token_cacherrrr)config)r*r>r6rCrrrr,0s  zTokenEnforcer.__init__token_sequencereturncCst|}|dd}||jvr|j|jS||jvr/tj|jd}||j|<||||jS|j|}|||}||j|<||||jS)a Get a list of allowed tokens, given a list of tokens that were already generated. :param token_sequence: The tokens that were already generated, and the next token will be generated for. :return: A list of token ids that are allowed to be selected next. Nr6)tupler@r8r4r=rA_compute_allowed_tokens_apply_new_characters)r*rD sent_tupleprev_step_tuplestateprev_step_state new_staterrrget_allowed_tokensDs          z TokenEnforcer.get_allowed_tokens state_tokensrMr5cCsLz^t|j|j}|j}|dur||jvr|j||_WdS|j}||j|j j |||j rGt |j trA||j n||j |sMtd||_|dur\||j|<WdSWdStyftytjtjd|t|}td|dt|j|j|_t |j tr|j|j YdS|j|j YdSw)Nz+Parser reached state with no allowed tokens)levelz+Unknown LMFormatEnforcer Problem. Prefix: 'z' Terminating the parser. Please open an issue at https://github.com/noamgat/lm-format-enforcer/issues with the prefix and CharacterLevelParser parameters)rrrr6 cache_keyrBr8 shortcut_key_collect_allowed_tokensr$r&can_end isinstancerr<extendappend ValueErrorr Exceptionlogging basicConfigERRORr exception)r*rQrMr8rSrTprefixrrrrIbs:         z%TokenEnforcer._compute_allowed_tokens tree_noder8rTcCs||j|}|j}t||}t|trV|ddkrVt |dks)J|\}} } } |j j } t | j td| | } t | j | | }|| | |j|dg}|D]}||}|j|}||||dqXdS)Nr json_freetext")rXtokensget_allowed_charactersr'r(set intersectionrWrHr!r$json_freetext_tokensmin max_token_lenmaxlookup_allowed_tokensr8 add_characterrU)r*r6rar8rTallowed_charactersrelevant_characterscharacters_to_explore_cur_lenmin_lenmax_lencache min_remainingmax_allowed_len character next_parsernext_tree_noderrrrUs"      z%TokenEnforcer._collect_allowed_tokensc Cstj|jd}|d}||jjvr|g|_|jj|}n|j|g|_||j}||j}|t|d}|D]-}z |j ||_Wq9t yf} zt d|d| dt |_WYd} ~ q9d} ~ ww|S)NrGrFzReceived an invalid character 'z+', switching to ForceStopParser (Exception:))r4r=r6r$new_word_tokensr:tokens_to_strsrr!rnr[r\debugr) r*rMrDrO new_tokennew_characters prev_decoded new_decodedryerrrrJs$   z#TokenEnforcer._apply_new_charactersN)r-r.r/r0rr=rrr,rr1rrPr rIrr rrUrJrrrrr4&s(r4) dataclassesrrsystypingrrrrrr r r r\ exceptionsr characterlevelparserrrrtokenizerprefixtreerr tokenlistrrr4rrrrs(