o wiM@s4ddlmZddlZddlmZGdddeZdS)) lru_cacheN) BaseTokenizerc@s<eZdZdZddZddZeddded efd d Zd S) TokenizerV14InternationalaETokenizes a string following the official BLEU implementation. See github.com/moses-smt/mosesdecoder/blob/master/scripts/generic/mteval-v14.pl#L954-L983 In our case, the input string is expected to be just one line. We just tokenize on punctuation and symbols, except when a punctuation is preceded and followed by a digit (e.g. a comma/dot as a thousand/decimal separator). We do not recover escaped forms of punctuations such as ' or > as these should never appear in MT system outputs (see issue #138) Note that a number (e.g., a year) followed by a dot at the end of sentence is NOT tokenized, i.e. the dot stays with the number because `s/(\p{P})(\P{N})/ $1 $2/g` does not match this case (unless we add a space after each sentence). However, this error is already in the original mteval-v14.pl and we want to be consistent with it. The error is not present in the non-international version, which uses `$norm_text = " $norm_text "`. :param line: the input string to tokenize. :return: The tokenized string. cCsdS)Nintlselfrr`/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/sacrebleu/tokenizers/tokenizer_intl.py signature sz#TokenizerV14International.signaturecCs.tddftddftddfg|_dS)Nz(\P{N})(\p{P})z\1 \2 z(\p{P})(\P{N})z \1 \2z(\p{S})z \1 )regexcompile_rerrrr __init__#s    z"TokenizerV14International.__init__i)maxsizelinereturncCs*|jD] \}}|||}qd|S)N )rsubjoinsplit)r rrreplrrr __call__-sz"TokenizerV14International.__call__N) __name__ __module__ __qualname____doc__r rrstrrrrrr rs  r) functoolsrr tokenizer_baserrrrrr s