o i<@s6ddlmZmZmZmZddlmZGdddZdS))ListOptionalSequenceAny)TokenIdc@sfeZdZUeed<eeed<eeed<eeed<de ddfdd Z d e de efd d Z e ZdS) TokenizerWrapper eos_token_id bos_token_idtokensspecial_token_ids gtokenizerreturnNcCsZ|j|_|j|_|j|_t|dg|_d|_||_d|_z|dWdSd|_YdS)Nr TstestF)r r r getattrr is_tokenizer_wrapper _gtokenizer_accepts_bytes)selfr rR/home/ubuntu/veenaModal/venv/lib/python3.10/site-packages/llguidance/_tokenizer.py__init__ s zTokenizerWrapper.__init__scCs(|jr ||d}|S||}|S)Nzutf-8)rrencode)rrrrrr_encode_strings  zTokenizerWrapper._encode_string)__name__ __module__ __qualname__r__annotations__rrbytesintrrstrrr__call__rrrrrs     rN)typingrrrr_utilrrrrrrs