o ÓÙ¾iäã@sNddlmZmZmZmZmZmZmZddlZ ddl m Z Gdd„deeƒZ dS)é)ÚDictÚHashableÚListÚProtocolÚSetÚTupleÚUnionN)ÚNDArrayc@s¢eZdZUeed<eed<eed<eeefed<eeed<deee efde e e j e e j ffdd „Zd e e j de efd d „Zd edefdd„ZdS)Ú TokenizerÚ eos_tokenÚ eos_token_idÚ pad_token_idÚ vocabularyÚspecial_tokensÚpromptÚreturncCódS)zHTranslate the input prompts into arrays of token ids and attention mask.N©)ÚselfrrrúM/home/ubuntu/.local/lib/python3.10/site-packages/outlines/models/tokenizer.pyÚencodeszTokenizer.encodeÚ token_idscCr)z?Translate an array of token ids to a string or list of strings.Nr)rrrrrÚdecodeszTokenizer.decodeÚtokencCr)uConvert a token to its equivalent string. This is for instance useful for BPE tokenizers where whitespaces are represented by the special characted `Ä `. This prevents matching a raw token that includes `Ä ` with a string. Nr)rrrrrÚconvert_token_to_stringsz!Tokenizer.convert_token_to_stringN)Ú__name__Ú __module__Ú __qualname__ÚstrÚ__annotations__Úintrrrrrr ÚnpÚint64rrrrrrrr s  ÿ þr ) ÚtypingrrrrrrrÚnumpyr!Ú numpy.typingr r rrrrÚs$