o yiD@sbddlZddlZddlmZddlmZmZmZmZm Z ddl m Z m Z m Z ddlmZddlmZdeded efd d Z  d2dedededededed efddZded efddZded efddZdee d e fddZd e eeefd!ee eeefd"e ed#ed$fd ee eeefee eeefffd%d&Z  d2ded'e eeefdedededed e fd(d)Z #   d3d e eeefd!ee eeefd"ed*dedededed+eee d ee fd,d-Z # .  d4d e eeefd!ee eeefd"ed*d/ededededed e e ee e fffd0d1ZdS)5N)inf)ListOptionalSequenceTupleUnion)Tensorstacktensor)Literal)_validate_inputs preds_word target_wordreturncCs t||kS)a,Distance measure used for substitutions/identity operation. Code adapted from https://github.com/rwth-i6/ExtendedEditDistance/blob/master/EED.py. Args: preds_word: hypothesis word string target_word: reference word string Return: 0 for match, 1 for no match )int)r rrT/home/ubuntu/.local/lib/python3.10/site-packages/torchmetrics/functional/text/eed.py_distance_between_wordsds r@333333?皙??hyprefalpharhodeletion insertionc shdgt|d}dgt|d}d|d<tgt|d}tdt|dD]o} tdt|dD]2} | dkr]t|| d||| dt|| d|| d|| ||| <q3|| d|| <q3|t|} || d7<|| ddkr||| fdd|D}|}tgt|d}q(|td d |D} td|d| tt|| S) a9Computes extended edit distance score for two lists of strings: hyp and ref. Code adapted from: https://github.com/rwth-i6/ExtendedEditDistance/blob/master/EED.py. Args: hyp: A hypothesis string ref: A reference string alpha: optimal jump penalty, penalty for jumps between characters rho: coverage cost, penalty for repetition of characters deletion: penalty for deletion of character insertion: penalty for insertion or substitution of character Return: Extended edit distance score as float rr csg|]}t|qSr)min.0xjumprr sz!_eed_function..css |] }|dkr |ndVqdS)rrNrr#rrr sz _eed_function..)lenrranger"rindexsumfloat) rrrrrrnumber_of_visitsrownext_rowwi min_indexcoveragerr&r _eed_functionrs,$   "r6sentencecCst|tstdt|d|}gd}|D] \}}|||}qgd}|D] \}}t|||}q*gd}|D] \}}|||}q.csg|] }fdd|DqS)crZrr)r$rr\rrr(r^z4_preprocess_sentences...r)r$ referencer\rrr(s)r rKrOrA)rSrTrUrr\r_preprocess_sentencessr` target_wordsc Cs4t}|D]}t||||||}||kr|}qt|S)aCompute scores for ExtendedEditDistance. Args: target_words: An iterable of reference words preds_word: A hypothesis word alpha: An optimal jump penalty, penalty for jumps between characters rho: coverage cost, penalty for repetition of characters deletion: penalty for deletion of character insertion: penalty for insertion or substitution of character Return: best_score: best (lowest) sentence-level score as a Tensor )rr6r ) r rarrrr best_scorer_scorerrr_compute_sentence_statisticssrd)rVrW sentence_eedc Cslt|||\}}|durg}dt|t|dfvr|St||D]\}} t|| ||||} || q!|S)aCompute scores for ExtendedEditDistance. Args: preds: An iterable of hypothesis corpus target: An iterable of iterables of reference corpus language: Language used in sentences. Only supports English (en) and Japanese (ja) for now. Defaults to en alpha: optimal jump penalty, penalty for jumps between characters rho: coverage cost, penalty for repetition of characters deletion: penalty for deletion of character insertion: penalty for insertion or substitution of character sentence_eed: list of sentence-level scores Return: individual sentence scores as a list of Tensors Nr)r`r*ziprdappend) rSrTrUrrrrre hypothesisrarcrrr _eed_update<s riFreturn_sentence_level_scorec Cs|tgd||||gD]\}} t| trt| tr%| dkr%td|dq t|||||||} t| } |r<| t| fS| S)uRComputes extended edit distance score (`ExtendedEditDistance`_) [1] for strings or list of strings. The metric utilises the Levenshtein distance and extends it by adding a jump operation. Args: preds: An iterable of hypothesis corpus. target: An iterable of iterables of reference corpus. language: Language used in sentences. Only supports English (en) and Japanese (ja) for now. Defaults to en return_sentence_level_score: An indication of whether sentence-level EED score is to be returned. alpha: optimal jump penalty, penalty for jumps between characters rho: coverage cost, penalty for repetition of characters deletion: penalty for deletion of character insertion: penalty for insertion or substitution of character Return: Extended edit distance score as a tensor Example: >>> from torchmetrics.functional import extended_edit_distance >>> preds = ["this is the prediction", "here is an other sample"] >>> target = ["this is the reference", "here is another one"] >>> extended_edit_distance(preds=preds, target=target) tensor(0.3078) References: [1] P. Stanchev, W. Wang, and H. Ney, “EED: Extended Edit Distance Measure for Machine Translation”, submitted to WMT 2019. `ExtendedEditDistance`_ )rrrrrz Parameter `z)` is expected to be a non-negative float.)rfr?r.rArirRr ) rSrTrUrjrrrr param_nameparamrPrQrrrextended_edit_distancees& rm)rrrr)rVrrrrN)rVFrrrr)rErMmathrtypingrrrrrtorchrr r typing_extensionsr #torchmetrics.functional.text.helperr r@rrr.r6rKrOrRr`rdriboolrmrrrrsX    ;,& ( "  ,