o yiZ@sddlZddlmZddlmZmZmZmZmZm Z m Z ddl m Z m Z ddlmZmZmZmZdZdZdZGd d d Zd ed ed efddZdeedeed ee eeeffddZdeeefdeedeedededed efddZdeedededed eef dd Zdeedeed!ed"ed e eeeeff d#d$Zdeedeed e fd%d&Zdeedeeed e e e ffd'd(Z d)e d*e d e fd+d,Z! d>d-e eeefdee eeefd ed.e d/e d0eee d e e e eee ffd1d2Z"d.e d/e d e fd3d4Z# 5 5 6 5 5d?d-e eeefdee eeefd7ed8ed9ed:ed;ed e e e e ee fffd)z([{-~[-` -&(-+:-@/]) \1 )z's z 's )z's$z 's)z([^0-9])([\.,])\1 \2 )z([\.,])([^0-9])z \1 \2)z ([0-9])(-)r-resub)rrulespattern replacementrrrr!ws  z/_TercomTokenizer._normalize_general_and_westerncCstdd|}tdd|}tdd|}tdd|}tdd|}tdd|}td d|}t|jd|}t|jd|}|S) z?Split Chinese chars and Japanese kanji down to character level.z([\u4e00-\u9fff\u3400-\u4dbf])r,z([\u31c0-\u31ef\u2e80-\u2eff])z+([\u3300-\u33ff\uf900-\ufaff\ufe30-\ufe4f])z([\u3200-\u3f22])z<(^|^[\u3040-\u309f])([\u3040-\u309f]+)(?=$|^[\u3040-\u309f])r-z<(^|^[\u30a0-\u30ff])([\u30a0-\u30ff]+)(?=$|^[\u30a0-\u30ff])z<(^|^[\u31f0-\u31ff])([\u31f0-\u31ff]+)(?=$|^[\u31f0-\u31ff])r/r0_ASIAN_PUNCTUATION_FULL_WIDTH_PUNCTUATIONclsrrrrr"sz!_TercomTokenizer._normalize_asiancCstdd|S)z1Remove punctuation from an input sentence string.z[\.,\?:;!\"\(\)]rr.)rrrrr#sz_TercomTokenizer._remove_punctcCs$t|jd|}t|jd|}|S)z7Remove asian punctuation from an input sentence string.rr4r7rrrr$sz$_TercomTokenizer._remove_asian_punct)FFTF)__name__ __module__ __qualname____doc__r5r6boolrrstrr' staticmethodr! classmethodr"r#r$rrrrr9s:  rr tokenizerrcCs ||S)zGiven a sentence, apply tokenization. Args: sentence: The input sentence string. tokenizer: An instance of ``_TercomTokenizer`` handling a sentence tokenization. Return: The pre-processed output sentence string. )rstrip)rrArrr_preprocess_sentences rC pred_words target_wordsccstt|D]J}tt|D]A}t||tkrqtdtD]0}|||d|||dkr3n|||fVt|||k}t|||k}|sM|rOnqqqdS)aFind matching word sub-sequences in two lists of words. Ignores sub-sequences starting at the same position. Args: pred_words: A list of a tokenized hypothesis sentence. target_words: A list of a tokenized reference sentence. Return: Yields tuples of ``target_start, pred_start, length`` such that: ``target_words[target_start : target_start + length] == pred_words[pred_start : pred_start + length]`` pred_start: A list of hypothesis start indices. target_start: A list of reference start indices. length: A length of a word span to be considered. N)rangelenabs_MAX_SHIFT_DIST_MAX_SHIFT_SIZE)rDrE pred_start target_startlength_hyp_refrrr_find_shifted_pairss"  rQ alignments pred_errors target_errorsrLrMrNcCs`t||||dkrdSt||||dkrdS|||kr+||kr.dSdSdS)aJA helper function which returns ``True`` if any of corner cases has been met. Otherwise, ``False`` is returned. Args: alignments: A dictionary mapping aligned positions between a reference and a hypothesis. pred_errors: A list of error positions in a hypothesis. target_errors: A list of error positions in a reference. pred_start: A hypothesis start index. target_start: A reference start index. length: A length of a word span to be considered. Return: An indication whether any of conrner cases has been met. rTF)sum)rRrSrTrLrMrNrrr$_handle_corner_cases_during_shiftingsrVwordsstarttargetc Csdttdtdtdtdttf dd}dttdtdtdtdttf dd }dttdtdtdtdttf d d }||krD|||||S|||krQ|||||S|||||S) a<Perform a shift in ``words`` from ``start`` to ``target``. Args: words: A words to shift. start: An index where to start shifting from. length: A number of how many words to be considered. target: An index where to end shifting. Return: A list of shifted words. rWrXrYrNrcSs8|d|||||||||||dSNrrWrXrYrNrrr$_shift_word_before_previous_position#8z<_perform_shift.._shift_word_before_previous_positioncSs8|d|||||||||||dSrZrr[rrr#_shift_word_after_previous_position&r]z;_perform_shift.._shift_word_after_previous_positioncSsP|d|}||||||7}|||||7}||||d7}|SrZr)rWrXrYrN shifted_wordsrrr!_shift_word_within_shifted_string)s z9_perform_shift.._shift_word_within_shifted_string)rr>int)rWrXrNrYr\r^r`rrr_perform_shifts& && rbcached_edit_distancechecked_candidatescCs||\}}t|}t|\}}} d} t||D]b\} } } t|| || | | r'qd}td| D]E}| |dkr9d}n| ||vrH|| |d}nn*||krOq.|}t|| | |}|||d| | | |f}|d7}| rq|| krs|} q.|tkrznq| sd||fS| \}}}}}|||fS)aAttempt to shift words to match a hypothesis with a reference. It returns the lowest number of required edits between a hypothesis and a provided reference, a list of shifted words and number of checked candidates. Note that the filtering of possible shifts and shift selection are heavily based on somewhat arbitrary heuristics. The code here follows as closely as possible the logic in Tercom, not always justifying the particular design choices. (The paragraph copied from https://github.com/mjpost/sacrebleu/blob/master/sacrebleu/metrics/lib_ter.py) Args: pred_words: A list of tokenized hypothesis sentence. target_words: A list of lists of tokenized reference sentences. cached_edit_distance: A pre-computed edit distance between a hypothesis and a reference. checked_candidates: A number of checked hypothesis candidates to match a provided reference. Return: best_score: The best (lowest) number of required edits to match hypothesis and reference sentences. shifted_words: A list of shifted words in hypothesis sentences. checked_candidates: A number of checked hypothesis candidates to match a provided reference. NrrF)r rrQrVrGrb_MAX_SHIFT_CANDIDATES)rDrErcrd edit_distanceinverted_tracetracerRrTrSbestrLrMrNprev_idxoffsetidxr_ candidate best_score_rrr _shift_words7sJ       rqc Cszt|dkr tdSt|}d}d}|} t||||\}}}|tks'|dkr(n|d7}|}q||\}} ||} t| S)a=Compute translation edit rate between hypothesis and reference sentences. Args: pred_words: A list of a tokenized hypothesis sentence. target_words: A list of lists of tokenized reference sentences. Return: A number of required edits to match hypothesis and reference sentences. rTrF)rHr r rqrf) rDrErc num_shiftsrd input_wordsdeltanew_input_wordsrgrp total_editsrrr_translation_edit_rates$   rxcCsPtd}td}|D]}t||}|t|7}||kr|}q |t|}||fS)aCompute sentence TER statistics between hypothesis and provided references. Args: pred_words: A list of tokenized hypothesis sentence. target_words: A list of lists of tokenized reference sentences. Return: best_num_edits: The best (lowest) number of required edits to match hypothesis and reference sentences. avg_tgt_len: Average length of tokenized reference sentences. rrg7yQC)r rxrH)rDrE tgt_lengthsbest_num_edits tgt_words num_edits avg_tgt_lenrrr_compute_sentence_statisticss    r~r| tgt_lengthcCsD|dkr|dkr||}|S|dkr|dkrtd}|Std}|S)aPCompute TER score based on pre-computed a number of edits and an average reference length. Args: num_edits: A number of required edits to match hypothesis and reference sentences. tgt_length: An average length of reference sentences. Return: A corpus-level TER score or 1 if reference_length == 0. rg?rr)r )r|rscorerrr"_compute_ter_score_from_statisticss rpredstotal_num_editstotal_tgt_length sentence_terc st||\}}t||D]2\}}fdd|D}t|} t| |\} } || 7}|| 7}|dur>|t| | dq |||fS)aUpdate TER statistics. Args: preds: An iterable of hypothesis corpus. target: An iterable of iterables of reference corpus. tokenizer: total_num_edits: A total number of required edits to match hypothesis and reference sentences. total_tgt_length: A total average length of reference sentences. Return: total_num_edits: A total number of required edits to match hypothesis and reference sentences. total_tgt_length: A total average length of reference sentences. sentence_ter: (Optionally) A list of sentence-level TER. Raises: ValueError: If length of ``preds`` and ``target`` differs. csg|] }t|qSr)rCr&).0_tgtrArr sz_ter_update..Nr)rziprCr&r~appendr unsqueeze) rrYrArrrpredtgt tgt_words_ pred_words_r|rrrr _ter_updates rcCs t||S)aMCompute TER based on pre-computed a total number of edits and a total average reference length. Args: total_num_edits: A total number of required edits to match hypothesis and reference sentences. total_tgt_length: A total average length of reference sentences. Return: A corpus-level TER score. )r)rrrrr _ter_computes rFTrrrrreturn_sentence_level_scorec Cst|ts td|dt|tstd|dt|ts'td|dt|ts4td|dt||||}td}td} |rGgnd} t||||| | \}} } t|| } | r`| | fS| S)aCalculate Translation edit rate (`TER`_) of machine translated text with one or more references. This implementation follows the implmenetaions from https://github.com/mjpost/sacrebleu/blob/master/sacrebleu/metrics/ter.py. The `sacrebleu` implmenetation is a near-exact reimplementation of the Tercom algorithm, produces identical results on all "sane" outputs. Args: preds: An iterable of hypothesis corpus. target: An iterable of iterables of reference corpus. normalize: An indication whether a general tokenization to be applied. no_punctuation: An indication whteher a punctuation to be removed from the sentences. lowercase: An indication whether to enable case-insesitivity. asian_support: An indication whether asian characters to be processed. return_sentence_level_score: An indication whether a sentence-level TER to be returned. Return: A corpus-level translation edit rate (TER). (Optionally) A list of sentence-level translation_edit_rate (TER) if `return_sentence_level_score=True`. Example: >>> preds = ['the cat is on the mat'] >>> target = [['there is a cat on the mat', 'a cat is on the mat']] >>> translation_edit_rate(preds, target) tensor(0.1538) References: [1] A Study of Translation Edit Rate with Targeted Human Annotation by Mathew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla and John Makhoul `TER`_ zrCrarQr=rVrbrqrxr~rrrrrrrrs# $, &  &%! O*"  *