o yiU3@sddlZddlmZddlmZmZddlZddlmZmZddl m Z ddl m Z m Z ddlmZdZd ZGd d d Z  ddeedeeededede ddedeeedefddZdS)N)partial)OptionalSequence)Tensortensor)Literal)_bleu_score_compute_bleu_score_update)_REGEX_AVAILABLEnone13azhintlchar))u㐀u䶵)u一u龥)u龦u龻)u豈u鶴)u侮u頻)u並u龎)u 0u⩭6)u⾀0u⾡d)u＀u￯)u⺀u⻿)u　u〿)u㇀u㇯)u⼀u⿟)u⿰u⿿)u㄀uㄯ)uㆠuㆿ)u︐u︟)u︰u﹏)u☀u⛿)u✀u➿)u㈀u㋿)u㌀u㏿c @seZdZdZeddfeddfeddfeddffZer:d d lZed dfed dfed dffZ ddddddZ d0de dde dd fddZ dedeefddZe d0dede dde deefddZededefdd Zed!ede fd"d#Zededefd$d%Zededefd&d'Zededefd(d)Zededefd*d+Zededefd,d-Zedede defd.d/Zd S)1_SacreBLEUTokenizerzTokenizer used for SacreBLEU calculation. Source: https://github.com/mjpost/sacrebleu/tree/master/sacrebleu/tokenizers z([\{-\~\[-\` -\&\(-\+\:-\@\/])z \1 z([^0-9])([\.,])z\1 \2 z([\.,])([^0-9])z \1 \2z ([0-9])(-)rNz(\P{N})(\p{P})z(\p{P})(\P{N})z(\p{S})_tokenize_base _tokenize_13a _tokenize_zh_tokenize_international_tokenize_charr Ftokenize lowercasereturncCst||j||_||_dSN)getattr _TOKENIZE_FN tokenize_fnr)selfrrr[/home/ubuntu/.local/lib/python3.10/site-packages/torchmetrics/functional/text/sacre_bleu.py__init__xs z_SacreBLEUTokenizer.__init__linecCs||}|||jSr)r_lowerrsplit)rr"tokenized_linerrr __call__|s z_SacreBLEUTokenizer.__call__cCs(t||j|}||}|||Sr)rrr#r$)clsr"rrrr%rrr rsz_SacreBLEUTokenizer.tokenizecC*|jD] \}}|||}qd|S)zCommon post-processing tokenizer for `13a` and `zh` tokenizers. Args: line: a segment to tokenize Return: the tokenized line  )_REGEXsubjoinr$r'r"_rereplrrr _tokenize_regexs z#_SacreBLEUTokenizer._tokenize_regexucharcCs.tD]\}}||kr|krdSqqdS)z Args: uchar: input char in unicode Return: whether the input char is a Chinese character. TF) _UCODE_RANGES)r1startendrrr _is_chinese_chars z$_SacreBLEUTokenizer._is_chinese_charcCs|S)zTokenizes an input line with the tokenizer. Args: line: a segment to tokenize Return: the tokenized line rr'r"rrr rs z"_SacreBLEUTokenizer._tokenize_basecCsf|dd}|dd}|dd}d|vr.|dd}|d d}|d d }|d d }||S)zTokenizes an input line using a relatively minimal tokenization that is however equivalent to mteval-v13a, used by WMT. Args: line: input sentence Return: tokenized sentence z z-  r)&z""z&z<)replacer0r6rrr rs       z!_SacreBLEUTokenizer._tokenize_13acCsL|}d}|D]}||r|d7}||7}|d7}q||7}q||S)ajThe tokenization of Chinese text in this script contains two steps: separate each Chinese characters (by utf-8 encoding); tokenize the Chinese part (following the `13a` i.e. mteval tokenizer). Author: Shujian Huang huangsj@nju.edu.cn Args: line: input sentence Return: tokenized sentence r7r))stripr5r0)r'r" line_in_charsrrrr rs    z _SacreBLEUTokenizer._tokenize_zhcCr()aTokenizes a string following the official BLEU implementation. See github.com/moses-smt/mosesdecoder/blob/master/scripts/generic/mteval-v14.pl#L954-L983 In our case, the input string is expected to be just one line. We just tokenize on punctuation and symbols, except when a punctuation is preceded and followed by a digit (e.g. a comma/dot as a thousand/decimal separator). We do not recover escaped forms of punctuations such as ' or > as these should never appear in MT system outputs (see issue #138) Note that a number (e.g., a year) followed by a dot at the end of sentence is NOT tokenized, i.e. the dot stays with the number because `s/(\p{P})(\P{N})/ $1 $2/g` does not match this case (unless we add a space after each sentence). However, this error is already in the original mteval-v14.pl and we want to be consistent with it. The error is not present in the non-international version, which uses `$norm_text = " $norm_text "`. Args: line: the input string to tokenize. Return: The tokenized string. r)) _INT_REGEXr+r,r$r-rrr rsz+_SacreBLEUTokenizer._tokenize_internationalcCsddd|DS)zTokenizes all the characters in the input line. Args: line: a segment to tokenize Return: the tokenized line r)css|]}|VqdSrr).0rrrr sz5_SacreBLEUTokenizer._tokenize_char..)r,r6rrr rs z"_SacreBLEUTokenizer._tokenize_charcCs|r|S|Sr)lower)r"rrrr r#sz_SacreBLEUTokenizer._lower)F)__name__ __module__ __qualname____doc__recompiler*r regexr@rrboolr!strrr& classmethodrr0 staticmethodr5rrrrrr#rrrr rPs`            rFr predstargetn_gramsmoothrrweightsrc Cs(|tvrtdtd|d|tjvr"tdttjt|t|kr8tdt|dt||dkrBtsBtd|d urXt||krXtd t|d||d urcd |g|}t |}t |}t d } t d } t tj ||d } t||||| | || \} } t| | |||||S)uCalculate `BLEU score`_ [1] of machine translated text with one or more references. This implementation follows the behaviour of SacreBLEU [2] implementation from https://github.com/mjpost/sacrebleu. Args: preds: An iterable of machine translated corpus target: An iterable of iterables of reference corpus n_gram: Gram value ranged from 1 to 4 smooth: Whether to apply smoothing – see [2] tokenize: Tokenization technique to be used. Supported tokenization: ['none', '13a', 'zh', 'intl', 'char'] lowercase: If ``True``, BLEU score over lowercased text is calculated. weights: Weights used for unigrams, bigrams, etc. to calculate BLEU score. If not provided, uniform weights are used. Return: Tensor with BLEU Score Raises: ValueError: If ``preds`` and ``target`` corpus have different lengths. ValueError: If a length of a list of weights is not ``None`` and not equal to ``n_gram``. Example: >>> from torchmetrics.functional import sacre_bleu_score >>> preds = ['the cat is on the mat'] >>> target = [['there is a cat on the mat', 'a cat is on the mat']] >>> sacre_bleu_score(preds, target) tensor(0.7598) References: [1] BLEU: a Method for Automatic Evaluation of Machine Translation by Papineni, Kishore, Salim Roukos, Todd Ward, and Wei-Jing Zhu `BLEU`_ [2] A Call for Clarity in Reporting BLEU Scores by Matt Post. [3] Automatic Evaluation of Machine Translation Quality Using Longest Common Subsequence and Skip-Bigram Statistics by Chin-Yew Lin and Franz Josef Och `Machine Translation Evolution`_ z*Argument `tokenize` expected to be one of z but got .z6Unsupported tokenizer selected. Please, choose one of zCorpus has different size z != rzv`'intl'` tokenization requires that `regex` is installed. Use `pip install regex` or `pip install torchmetrics[text]`.Nz5List of weights has different weights than `n_gram`: g?g)rr)AVAILABLE_TOKENIZERS ValueErrorrrkeyslistlenr ModuleNotFoundErrortorchzerosrrrr r) rPrQrRrSrrrT numerator denominator preds_len target_lenrrrr sacre_bleu_scores@0    rb)rOFr FN)rH functoolsrtypingrrr\rrtyping_extensionsr!torchmetrics.functional.text.bleurr torchmetrics.utilities.importsr rVr2rrLintrKfloatrbrrrr sD'   K