o 5ti@s,ddlmZddlmZGdddeZdS)) lru_cache) BaseTokenizerc@s.eZdZddZddZeddddZd S) TokenizerCharcCsdS)NcharselfrrW/home/ubuntu/.local/lib/python3.10/site-packages/sacrebleu/tokenizers/tokenizer_char.py signaturezTokenizerChar.signaturecCsdSNrrrrr __init__ r zTokenizerChar.__init__i)maxsizecCsddd|DS)zTokenizes all the characters in the input line. :param line: a segment to tokenize :return: the tokenized line  css|]}|VqdSr r).0rrrr sz)TokenizerChar.__call__..)join)r linerrr __call__ szTokenizerChar.__call__N)__name__ __module__ __qualname__r rrrrrrr rs rN) functoolsrtokenizer_baserrrrrr s