o 5ti@s8ddlmZddlmZddlmZGdddeZdS)) lru_cache) BaseTokenizer)TokenizerRegexpc@s.eZdZddZddZeddddZd S) Tokenizer13acCsdS)N13aselfrrV/home/ubuntu/.local/lib/python3.10/site-packages/sacrebleu/tokenizers/tokenizer_13a.py signatureszTokenizer13a.signaturecCs t|_dS)N)r_post_tokenizerr rrr __init__ s zTokenizer13a.__init__i)maxsizecCsn|dd}|dd}|dd}d|vr.|dd}|d d}|d d }|d d }|d|dS)zTokenizes an input line using a relatively minimal tokenization that is however equivalent to mteval-v13a, used by WMT. :param line: a segment to tokenize :return: the tokenized line z z-   &z""z&z<)replacer )r linerrr __call__s      zTokenizer13a.__call__N)__name__ __module__ __qualname__r rrrrrrr rs rN) functoolsrtokenizer_baser tokenizer_rerrrrrr s