o .wiN@s ddlZddlZddlZddlmZddlmZddlmZm Z m Z ddl Z ddl m Z m Z ddlmZddlmZmZddlmZmZmZmZmZmZd Zed Zd Zejed Z d d dZ!GdddZ"     ddee#deee#de$de%dede%de ee&de fddZ'dS)N)Sequence)partial)AnyClassVarOptional)Tensortensor)Literal)_bleu_score_compute_bleu_score_update)_IPADIC_AVAILABLE_MECAB_AVAILABLE_MECAB_KO_AVAILABLE_MECAB_KO_DIC_AVAILABLE_REGEX_AVAILABLE_SENTENCEPIECE_AVAILABLE none13azhintlcharja-mecabko-mecab flores101 flores200))u㐀u䶵)u一u龥)u龦u龻)u豈u鶴)u侮u頻)u並u龎)u 0u⩭6)u⾀0u⾡d)u＀u￯)u⺀u⻿)u　u〿)u㇀u㇯)u⼀u⿟)u⿰u⿿)u㄀uㄯ)uㆠuㆿ)u︐u︟)u︰u﹏)u☀u⛿)u✀u➿)u㈀u㋿)u㌀u㏿ztorchmetrics-floreszRhttps://dl.fbaipublicfiles.com/fairseq/models/flores/sacrebleu_tokenizer_spm.modelz)https://tinyurl.com/flores200sacrebleuspmrrc @seZdZUdZeddfeddfeddfeddffZer;d d lZed dfed dfed dffZ dddddddddd Z e e e d<d d dZe e eeefe d<dGdededd fdd Zd!edeefd"d#Ze dGd$edd!edededeef d%d&Zed$edd!edefd'd(Zed)edefd*d+Zed$edd!edefd,d-Zed$edd!edefd.d/Zed$edd!edefd0d1Zed$edd!edefd2d3Z ed$edd!edefd4d5Z!ed$edd!edefd6d7Z"ed$edd!edefd8d9Z#ed$edd!ede$ddefd:d;Z%ed$edd!edefdd?Z'ed!ededefd@dAZ(ed$eddedd fdBdCZ)edDe$ddd fdEdFZ*d S)H_SacreBLEUTokenizerzTokenizer used for SacreBLEU calculation. Source: https://github.com/mjpost/sacrebleu/tree/master/sacrebleu/tokenizers z([\{-\~\[-\` -\&\(-\+\:-\@\/])z \1 z([^0-9])([\.,])z\1 \2 z([\.,])([^0-9])z \1 \2z ([0-9])(-)rNz(\P{N})(\p{P})z(\p{P})(\P{N})z(\p{S})_tokenize_base _tokenize_13a _tokenize_zh_tokenize_international_tokenize_char_tokenize_ja_mecab_tokenize_ko_mecab_tokenize_flores_101_tokenize_flores_200r _TOKENIZE_FNrsentencepiece_processorsFtokenize lowercasereturncCs&||t||j||_||_dSN)_check_tokenizers_validitygetattrr' tokenize_fnr*)selfr)r*r1d/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/torchmetrics/functional/text/sacre_bleu.py__init__s  z_SacreBLEUTokenizer.__init__linecCs||}|||jSr,)r/_lowerr*split)r0r4tokenized_liner1r1r2__call__s z_SacreBLEUTokenizer.__call__clscCs2||t||j|}||}|||Sr,)r-r.r'r5r6)r9r4r)r*r/r7r1r1r2r)s z_SacreBLEUTokenizer.tokenizecC*|jD] \}}|||}qd|S)zPost-processing tokenizer for `13a` and `zh` tokenizers. Args: line: a segment to tokenize Return: the tokenized line  )_REGEXsubjoinr6r9r4_rereplr1r1r2_tokenize_regexs z#_SacreBLEUTokenizer._tokenize_regexucharcstfddtDS)zCheck if character is chinese. Args: uchar: input char in unicode. Return: whether the input char is a Chinese character. c3s,|]\}}|ko|knVqdSr,r1).0startendrCr1r2 s*z7_SacreBLEUTokenizer._is_chinese_char..)any _UCODE_RANGESrGr1rGr2_is_chinese_chars z$_SacreBLEUTokenizer._is_chinese_charcCs|S)zTokenizes an input line with the tokenizer. Args: line: a segment to tokenize Return: the tokenized line r1r9r4r1r1r2rs z"_SacreBLEUTokenizer._tokenize_basecCsn|dd}|dd}|dd}d|vr.|dd}|d d}|d d }|d d }|d|dS)zTokenizes a line using a relatively minimal tokenization that is equivalent to mteval-v13a, used by WMT. Args: line: input sentence Return: tokenized sentence z z-  r;&z""z&z<)replacerBrLr1r1r2rs      z!_SacreBLEUTokenizer._tokenize_13acCsL|}d}|D]}||r|d7}||7}|d7}q||7}q||S)apTokenization of Chinese text. This is done in two steps: separate each Chinese characters (by utf-8 encoding) and afterwards tokenize the Chinese part (following the `13a` i.e. mteval tokenizer). Author: Shujian Huang huangsj@nju.edu.cn. Args: line: input sentence Return: tokenized sentence rMr;)striprKrB)r9r4 line_in_charsrr1r1r2r s    z _SacreBLEUTokenizer._tokenize_zhcCr:)aTokenizes a string following the official BLEU implementation. See github.com/moses-smt/mosesdecoder/blob/master/scripts/generic/mteval-v14.pl#L954-L983 In our case, the input string is expected to be just one line. We just tokenize on punctuation and symbols, except when a punctuation is preceded and followed by a digit (e.g. a comma/dot as a thousand/decimal separator). We do not recover escaped forms of punctuation such as ' or > as these should never appear in MT system outputs (see issue #138) Note that a number (e.g., a year) followed by a dot at the end of sentence is NOT tokenized, i.e. the dot stays with the number because `s/(\\p{P})(\\P{N})/ $1 $2/g` does not match this case (unless we add a space after each sentence). However, this error is already in the original mteval-v14.pl and we want to be consistent with it. The error is not present in the non-international version, which uses `$norm_text = " $norm_text "`. Args: line: the input string to tokenize. Return: The tokenized string. r;) _INT_REGEXr=r>r6r?r1r1r2r!sz+_SacreBLEUTokenizer._tokenize_internationalcCsddd|DS)zTokenizes all the characters in the input line. Args: line: a segment to tokenize Return: the tokenized line r;css|]}|VqdSr,r1)rDrr1r1r2rH4sz5_SacreBLEUTokenizer._tokenize_char..)r>rLr1r1r2r")s z"_SacreBLEUTokenizer._tokenize_charcCs6ddl}ddl}||jd}|}||S)zTokenizes a Japanese string line using MeCab morphological analyzer. Args: line: the input string to tokenize. Return: The tokenized string. rN -Owakati)ipadicMeCabTagger MECAB_ARGSrTparse)r9r4rXrYtaggerr1r1r2r#6  z&_SacreBLEUTokenizer._tokenize_ja_mecabcCs6ddl}ddl}||jd}|}||S)zTokenizes a Korean string line using MeCab-korean morphological analyzer. Args: line: the input string to tokenize. Return: The tokenized string. rNrW)mecab_ko mecab_ko_dicrZr[rTr\)r9r4r_r`r]r1r1r2r$Ir^z&_SacreBLEUTokenizer._tokenize_ko_mecabcCs|ddl}|j|dur3||j|<tjtt|dd}tj |s+| ||j| |d|j| |S)zTokenizes a string line using sentencepiece tokenizer. Args: line: the input string to tokenize. tokenize: Tokenization technique to be used. Return: The tokenized string. rN/r;) sentencepiecer(SentencePieceProcessorospathr>_FLORES_LOCAL_DIR_FLORES_MODELS_URLr6existsdownload_flores_fileLoadEncodeAsPieces)r9r4r)rc file_pathr1r1r2_tokenize_flores\s  z$_SacreBLEUTokenizer._tokenize_florescC ||dS)zTokenizes a string line using sentencepiece tokenizer according to `FLORES-101`_ dataset. Args: line: the input string to tokenize. Return: The tokenized string. rrnrLr1r1r2r%w z(_SacreBLEUTokenizer._tokenize_flores_101cCro)zTokenizes a string line using sentencepiece tokenizer according to `FLORES-200`_ dataset. Args: line: the input string to tokenize. Return: The tokenized string. rrprLr1r1r2r&rqz(_SacreBLEUTokenizer._tokenize_flores_200cCs|r|S|Sr,)lower)r4r*r1r1r2r5sz_SacreBLEUTokenizer._lowercCs||jvrtdt|j|dkrtstd|dkr'tr#ts'td|dkr3tr/t s3tdd|vr=t s?td d Sd S) z}Check if a supported tokenizer is chosen. Also check all dependencies of a given tokenizers are installed. z6Unsupported tokenizer selected. Please, choose one of rzv`'intl'` tokenization requires that `regex` is installed. Use `pip install regex` or `pip install torchmetrics[text]`.rz`'ja-mecab'` tokenization requires that `MeCab` and `ipadic` are installed. Use `pip install mecab-python3 ipadic` or `pip install torchmetrics[text]`.rz`'ko-mecab'` tokenization requires that `mecab_ko` and `mecab_ko_dic` are installed. Use `pip install mecab_ko mecab_ko_dic` or `pip install torchmetrics[text]`.floresz`'flores101' and 'flores200'` tokenizations require that `sentencepiece` is installed. Use `pip install sentencepiece` or `pip install torchmetrics[text]`.N) r' ValueErrorlistkeysrModuleNotFoundErrorr r rrr)r9r)r1r1r2r-s&   z._SacreBLEUTokenizer._check_tokenizers_validity model_namec Csddl}ddl}tjtddt|}tjt|dd}z@t |d0}|j |}| | Wdn1s>wYWdWdSWdWdS1sXwYWdS|jyu}z td|d |d}~ww) zGDownload necessary files for `flores` tokenization via `sentencepiece`.rNT)exist_okrarbwbzFailed to download z model.)sslurllib.requestremakedirsrgrhrfr>r6openrequesturlopenwritereadSSLErrorOSError)rxr{urllib model_urlrmout_file remote_fileer1r1r2rjsVz(_SacreBLEUTokenizer.download_flores_file)F)+__name__ __module__ __qualname____doc__recompiler<rregexrVr'rdict__annotations__r(strrr_TokenizersLiteralboolr3rr8 classmethodtyper)rB staticmethodrKrrr r!r"r#r$r rnr%r&r5r-rjr1r1r1r2rcs         "          !rFrpredstargetn_gramsmoothr)r*weightsr+c Cst|t|krtdt|dt||dur,t||kr,tdt|d||dur7d|g|}t|}t|}td} td} ttj||d} t||||| | || \} } t | | |||||S)a^ Calculate `BLEU score`_ [1] of machine translated text with one or more references. This implementation follows the behaviour of SacreBLEU [2] implementation from https://github.com/mjpost/sacrebleu. .. note:: In the original SacreBLEU, references are passed as a list of reference sets (grouped by reference index). In TorchMetrics, references are passed grouped per prediction (each prediction has its own list of references). For example:: # Predictions preds = ['The dog bit the man.', "It wasn't surprising.", 'The man had just bitten him.'] # Original SacreBLEU: refs = [ ['The dog bit the man.', 'It was not unexpected.', 'The man bit him first.'], # First set ['The dog had bit the man.', 'No one was surprised.', 'The man had bitten the dog.'], # Second set ] # TorchMetrics SacreBLEU: target = [ ['The dog bit the man.', 'The dog had bit the man.'], # References for first prediction ['It was not unexpected.', 'No one was surprised.'], # References for second prediction ['The man bit him first.', 'The man had bitten the dog.'], # References for third prediction ] Args: preds: An iterable of machine translated corpus target: An iterable of iterables of reference corpus n_gram: Gram value ranged from 1 to 4 smooth: Whether to apply smoothing - see [2] tokenize: Tokenization technique to be used. Choose between ``'none'``, ``'13a'``, ``'zh'``, ``'intl'``, ``'char'``, ``'ja-mecab'``, ``'ko-mecab'``, ``'flores101'`` and ``'flores200'``. lowercase: If ``True``, BLEU score over lowercased text is calculated. weights: Weights used for unigrams, bigrams, etc. to calculate BLEU score. If not provided, uniform weights are used. Return: Tensor with BLEU Score Raises: ValueError: If ``preds`` and ``target`` corpus have different lengths. ValueError: If a length of a list of weights is not ``None`` and not equal to ``n_gram``. Example: >>> from torchmetrics.functional.text import sacre_bleu_score >>> preds = ['the cat is on the mat'] >>> target = [['there is a cat on the mat', 'a cat is on the mat']] >>> sacre_bleu_score(preds, target) tensor(0.7598) References: [1] BLEU: a Method for Automatic Evaluation of Machine Translation by Papineni, Kishore, Salim Roukos, Todd Ward, and Wei-Jing Zhu `BLEU`_ [2] A Call for Clarity in Reporting BLEU Scores by Matt Post. [3] Automatic Evaluation of Machine Translation Quality Using Longest Common Subsequence and Skip-Bigram Statistics by Chin-Yew Lin and Franz Josef Och `Machine Translation Evolution`_ zCorpus has different size z != Nz5List of weights has different weights than `n_gram`: g?g)r)r*) lenrttorchzerosrrrr)r r ) rrrrr)r*r numerator denominator preds_len target_lenr/r1r1r2sacre_bleu_scores,G   r)rFrFN)(rertempfilecollections.abcr functoolsrtypingrrrrrrtyping_extensionsr !torchmetrics.functional.text.bleur r torchmetrics.utilities.importsr r rrrrAVAILABLE_TOKENIZERSrrJrfr> gettempdirrgrhrrintrfloatrr1r1r1r2sV'     m