o ۾i@sBdZddlmZmZddlZddlZddlZGdddejZdS)z This file helps integrate xgrammar in HF transformers package by extending transformers.LogitsProcessor, which is to be fed to `model.generate()`. )ListUnionNc@sJeZdZdZdeejeejffddZde j de j de j fdd Z d S) LogitsProcessora LogitsProcessor for processing logits in transformers' generate() method. Example usage ------------- .. code:: python model_name = "Qwen/Qwen2.5-0.5B-Instruct" tokenizer = AutoTokenizer.from_pretrained(model_name) config = AutoConfig.from_pretrained(model_name) # This can be larger than tokenizer.vocab_size due to paddings full_vocab_size = config.vocab_size tokenizer_info = xgr.TokenizerInfo.from_huggingface(tokenizer, vocab_size=full_vocab_size) grammar_compiler = xgr.GrammarCompiler(tokenizer_info) compiled_grammar = grammar_compiler.compile_builtin_json_grammar() xgr_logits_processor = xgr.contrib.hf.LogitsProcessor(compiled_grammar) model.generate(prompt, logits_processor=[xgr_logits_processor]) For an end-to-end example, see folder `examples/hf_transformers/`. Notes ----- - Note that this LogitsProcessor can only be used once. For each `generate()` call, instantiate a new one. - Note that this implementation may contain extra overhead. compiled_grammarcCsBg|_t|tr |n|g|_|jdjj|_d|_d|_d|_ dS)aInitialize the LogitsProcessor. Parameters ---------- compiled_grammar : xgr.CompiledGrammar | List[xgr.CompiledGrammar] One or more grammars compiled according to the given grammar and the model's tokenizer_info. rNF) matchers isinstancelistcompiled_grammarstokenizer_info vocab_sizefull_vocab_size token_bitmask prefilled batch_size)selfrrG/home/ubuntu/.local/lib/python3.10/site-packages/xgrammar/contrib/hf.py__init__+s zLogitsProcessor.__init__ input_idsscoresreturncsvtjdkr@|jd_tjdkrjnjj_tjjks*JdfddtjD_tjj_ |jdjkrYt dd|jddjd j s`d _ ntjD]}j| s~||d }j| |s~JqetjD]}j| sj|j |q|jj}|d kr|d }t|j |j|d kr||}|S)z Accept token sampled in the last iteration, fill in bitmask, and apply bitmask to logits. Returns: scores: Logits modified with bitmask. rz@The number of compiled grammars must be equal to the batch size.csg|] }tj|qSr)xgrGrammarMatcherr ).0irrr Nsz,LogitsProcessor.__call__..z;Expect input_ids.shape[0] to be LogitsProcessor.batch_size.zGot z for the former, and z for the latter.Tcudacpu)lenrshaperr rangerallocate_token_bitmaskr r RuntimeErrorr is_terminated accept_tokenfill_next_token_bitmaskdevicetypetoapply_token_bitmask_inplace)rrrr sampled_token device_typerrr__call__<sJ      zLogitsProcessor.__call__N) __name__ __module__ __qualname____doc__rrCompiledGrammarrrtorch LongTensor FloatTensorr/rrrrrs r) r3typingrrr5 transformersxgrammarrrrrrrs