o 5t¾iMã@s4ddlmZddlZddlmZGdd„deƒZdS)é)Ú lru_cacheNé)Ú BaseTokenizerc@s<eZdZdZdd„Zdd„Zeddded efd d „ƒZd S) ÚTokenizerV14InternationalaETokenizes a string following the official BLEU implementation. See github.com/moses-smt/mosesdecoder/blob/master/scripts/generic/mteval-v14.pl#L954-L983 In our case, the input string is expected to be just one line. We just tokenize on punctuation and symbols, except when a punctuation is preceded and followed by a digit (e.g. a comma/dot as a thousand/decimal separator). We do not recover escaped forms of punctuations such as ' or > as these should never appear in MT system outputs (see issue #138) Note that a number (e.g., a year) followed by a dot at the end of sentence is NOT tokenized, i.e. the dot stays with the number because `s/(\p{P})(\P{N})/ $1 $2/g` does not match this case (unless we add a space after each sentence). However, this error is already in the original mteval-v14.pl and we want to be consistent with it. The error is not present in the non-international version, which uses `$norm_text = " $norm_text "`. :param line: the input string to tokenize. :return: The tokenized string. cCsdS)NÚintl©©ÚselfrrúW/home/ubuntu/.local/lib/python3.10/site-packages/sacrebleu/tokenizers/tokenizer_intl.pyÚ signature sz#TokenizerV14International.signaturecCs.t d¡dft d¡dft d¡dfg|_dS)Nz(\P{N})(\p{P})z\1 \2 z(\p{P})(\P{N})z \1 \2z(\p{S})z \1 )ÚregexÚcompileÚ_rerrrr Ú__init__#s    úz"TokenizerV14International.__init__i)ÚmaxsizeÚlineÚreturncCs*|jD] \}}| ||¡}qd | ¡¡S)Nú )rÚsubÚjoinÚsplit)r rrÚreplrrr Ú__call__-sz"TokenizerV14International.__call__N) Ú__name__Ú __module__Ú __qualname__Ú__doc__r rrÚstrrrrrr rs  r)Ú functoolsrr Útokenizer_baserrrrrr Ús