o 5ti@s4ddlmZddlZddlmZGdddeZdS)) lru_cacheN) BaseTokenizerc@s.eZdZddZddZeddddZd S) TokenizerRegexpcCsdS)NreselfrrU/home/ubuntu/.local/lib/python3.10/site-packages/sacrebleu/tokenizers/tokenizer_re.py signature szTokenizerRegexp.signaturecCs:tddftddftddftddfg|_dS)Nz([\{-\~\[-\` -\&\(-\+\:-\@\/])z \1 z([^0-9])([\.,])z\1 \2 z([\.,])([^0-9])z \1 \2z ([0-9])(-))rcompile_rerrrr __init__ s     zTokenizerRegexp.__init__i)maxsizecCs*|jD] \}}|||}qd|S)zCommon post-processing tokenizer for `13a` and `zh` tokenizers. :param line: a segment to tokenize :return: the tokenized line  )r subjoinsplit)r liner replrrr __call__szTokenizerRegexp.__call__N)__name__ __module__ __qualname__r rrrrrrr rs r) functoolsrrtokenizer_baserrrrrr s