o ۾ibA@sddlmZddlZddlZddlZddlZddlmZddl Z ddl Z ddl Z ddlmZddlmZddlmZddlmZddlmZddlmZmZerqddlZddlmZddlm Z ddl!Z"dd l#m$Z$dd l%m&Z&ned e'd Z"ed e'dZede'dZ ede'dZee(Z)dZ*dd-d.Z2d?d0d1Z3d@d4d5Z4dAd6d7Z5dBd:d;Z6dS)C) annotationsN) TYPE_CHECKING)LRUCache)Cache) init_logger) LazyLoader) GrammarOutputSchedulerOutput) TokenizerLike) InputBatchxgrxgrammaroc outlines_core file_utilsztransformers.file_utilsconvert_slow_tokenizerz#transformers.convert_slow_tokenizerscheduler_outputr grammar_outputr input_batchr logits torch.TensorreturnNonecCs^|j}i}d}|j}t|j}t|jD]\} } | |} |t|| d7}| |vr.| || <qg} tj |j d|j dfd|j d} d}|jD]3} t|| d}|| }durtt d|D]}||}|||| |<| |q`|d|7}qGt| j|jdd}t| |j dk}d}|stj| tjd dd }|j|jdd}tj|||d dS) a- Apply grammar bitmask to output logits of the model with xgrammar function. Args: scheduler_output (SchedulerOutput): The result of engine scheduling. input_batch (InputBatch): The input of model runner. logits (torch.Tensor): The output logits of model forward. r)shape fill_valuedtypeNT) non_blockingcpu)rdevice pin_memory)indices)grammar_bitmaskscheduled_spec_decode_tokenssetstructured_output_request_ids enumeratereq_idslengetnpfullrrrangeappendtorch from_numpytor!tensorint32r apply_token_bitmask_inplace)rrrrr$struct_out_req_batch_indicescumulative_offset spec_tokensstruct_out_req_ids batch_indexreq_id logit_index out_indicessorted_bitmaskcumulative_indexnum_spec_tokens logit_idxi bitmask_indexskip_out_indices index_tensorrrS/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/structured_output/utils.pyapply_grammar_bitmask,sJ      rGc@seZdZdZd ddZdS) OutlinesVocabularyzo Wrapper class for `outlines_core.Vocabulary`, which allows us to store a hash with the vocabulary vocabulary oc.VocabularyrrcCs2||_t|d}t|d}||_dS)Nutf-8)innerhashlibsha256__repr__encode hexdigestint_hash)selfrIhex_strhash_intrrrF__init__s  zOutlinesVocabulary.__init__N)rIrJrr)__name__ __module__ __qualname____doc__rXrrrrFrHzsrHstrcCsxtd}td}tjd}|r|S|rtj|ddStj|r0|dkr0tj|ddSt}tj|ddS)zFGet the context object that contains previously-computed return valuesOUTLINES_CACHE_DIRXDG_CACHE_HOME~z.cacheoutlines/)osgetenvpath expanduserjoinisdirtempfile gettempdir)outlines_cache_dirxdg_cache_homehome_dirtempdirrrrFget_outlines_cache_paths   rocCsft}tjr.tdt|ddd}tjd}| dd}||kr&| | d||St dd S) z3Get the Cache instance to be used for index cachingzEnabling outlines cache. This is an unbounded on-disk cache. It may consume a lot of disk space and should not be used with untrusted clients.noner)eviction_policy cull_limitr __version__N)maxsize) roenvsVLLM_V1_USE_OUTLINES_CACHEloggerwarningr importlibmetadataversionr+clearr&r) cache_dircacheoutlines_versioncached_versionrrrFget_outlines_caches    rz^<0x[0-9A-F]{2}>$u^.{0,6}�+.{0,6}$ tokenizerr eos_token_idrSdict[bytes, list[int]]c s*ddtDdfdd }i}g}D]v\}}|jvr&q||}|rt|ttfr8t|}nG|dkr@|dksId|vrzt |szt |r[tt |d d d g}n$fd d |D} d| vrut d|d|d|t| }n| d}||kr||g|q||q|S)zCreate a map from vocabulary tokens to lists of equivalent token ids. Returns: A Dict of token string -> equivalent token ids cSsi|]\}}||qSrr).0kvrrrF s z'_reduced_vocabulary..tokenr]rcs8|g}t|tur|tjs|dkrd|S|S)Nz<0x20> )convert_tokens_to_stringtyper] startswithrSPIECE_UNDERLINE)rstring)rrrFconvert_token_to_strings   z4_reduced_vocabulary..convert_token_to_stringu�rLcsg|]}|qSr)r+rc)unicode_to_bytesrrF sz'_reduced_vocabulary..NzCannot convert token `z` (z ) to bytes: rK)rr]rr])rbytes_to_unicodeitems get_vocaball_special_tokens isinstancebytes bytearrayre_replacement_seqmatchre_llama_byte_tokenrS RuntimeErrorrQ setdefaultr/) rrrrIempty_token_idsr token_idx token_str token_bytes byte_valsr)rrrF_reduced_vocabularysB        rrJc Cst|dr|jSz+t|dr|jdur|j}n tdt|dt||}tt||}||_|WSt yJ}z tdt|d|d}~ww)z2Get the `Vocabulary` object for a given tokenizer._outlines_vocabularyrNz?Error during structured outputs setup for outlines: Tokenizer (zi) has no `eos_token_id` property, but `eos_token_id` is required for structured outputs to work properly.z,Cannot get the vocabulary of the tokenizer (z0). The tokenizer should have a get_vocab method.) hasattrrr ValueErrorrrrHr VocabularyAttributeError)rr reduced_vocabrIerrrFget_outlines_vocabularys6 r grammar_strboolcCsL|rt|ts dS|dD]}tdd|}|sqd|vr#dSqdS)aV Check if grammar appears to use Lark syntax. Args: grammar_str: Input grammar string Returns: bool: True if grammar appears to be in Lark format, False otherwise Examples: >>> grammar_is_likely_lark("rule: 'abc'") True >>> grammar_is_likely_lark("rule ::= 'abc'") False F (#|//).*$z::=T)rr]splitresubstrip)rlinerrrFgrammar_is_likely_lark!src st|tstdt||stdt}t}g}d.ddd/d d}d0dd}fdd|dD}d}t|dD]F\}} | rK| drLq@d| vrz | dddd} | | |duri| }| dkrod}Wq@t y} z td|d| d} ~ wwq@|std| d|d} g} t|dD]\}} | sqz|d| vr| ds| r| | d d! | | dd\} }| d} ||d"| d#|t d$d%|}||||g} n5| dr | std&|d'| dd}||d(| d#|t d$d%|}|||| |Wqty;} z td)|d*t| | d} ~ ww| rL| | d d! | ||d+h}|rbtd,d- t|d |S)1a Convert a Lark grammar string to EBNF format. EBNF reference: https://github.com/ggerganov/llama.cpp/blob/master/grammars/README.md Lark grammar reference: https://lark-parser.readthedocs.io/en/latest/grammar.html Args: grammar_str: Input grammar in Lark format Returns: str: Converted grammar in EBNF format Examples: >>> print(convert_lark_to_ebnf("rule: 'hello'")) root ::= rule rule ::= "hello" zGrammar must be a string, got zGrammar string cannot be emptyrr]rcSstdd|S)z)Remove comments and whitespace from line.rr)rrr)rrrrF clean_line^sz(convert_lark_to_ebnf..clean_linetext rule_nameline_numrSrcSs<|dddks|dddkrtd|d|dS)z Validate quote matching in text.'r"zMismatched quotes in z on line N)countr)rrrrrrF check_quotesbs$z*convert_lark_to_ebnf..check_quotesset[str]cSs,tdd|}tdd|}ttd|S)z"Extract rule references from text.z"[^"]*"rz[+*?()|\[\]{}]rz\b[a-zA-Z_][a-zA-Z0-9_]*\b)rrr&findall)rrrrFextract_referencesgsz0convert_lark_to_ebnf..extract_referencescsg|]}|qSrr)rrrrrFrosz(convert_lark_to_ebnf..rNr|:r?startzInvalid rule format on line z". Expected 'rule_name: definition'zNo valid rules found in grammar root ::= z ::=  | zrule 'rz '([^']*)'z"\1"zAlternative '|' on line z$ without a preceding rule definitionzalternative for rule 'zError on line z: rootz"Referenced rules are not defined: z, )rr]rr])rr]rr]rrSrr)rr]rr)rr]rrrr&rr(radd IndexErrorr/rgrrupdatesorted)r defined_rulesreferenced_rules output_linesrrlines first_rulerrnamer current_rulecurrent_definition definitionalt_defundefined_rulesrrrFconvert_lark_to_ebnfAs           rchoice list[str]cs8d ddfdd|D}dd d d|D}|S) Nsr]rcSstdd|S)z+Escape special characters in a EBNF string.z(["\\])z\\\1)rr)rrrrFescape_ebnf_stringsz-choice_as_grammar..escape_ebnf_stringc3s|]}|VqdS)NrrrrrF sz$choice_as_grammar..rrcss|] }d|dVqdS)rNrrrrrFrs)rr]rr])rg)rescaped_choicesgrammarrrrFchoice_as_grammars r) rr rrrr rrrr)rr])rr rrSrr)rr rrJ)rr]rr)rr]rr])rrrr])7 __future__rrNimportlib.metadatarzrcritypingrnumpyr,regexrr0 cachetoolsr diskcacher vllm.envsrv vllm.loggerrvllm.utils.import_utilsrvllm.v1.core.sched.outputrr rr#transformers.convert_slow_tokenizerrtransformers.file_utilsrr r vllm.tokenizersr vllm.v1.worker.gpu_input_batchr globalsrYrxCACHErGrHrorcompilerrrrrrrrrrrFsT           N     F