o .wi0e*@sddlmZddlmZddlmZddlmZmZm Z ddl Z ddl m Z m Z ddl mZe dZed Zd ed ed eeee feee feee feee feee feee fffd dZdeded eefddZded eefddZded eefddZdeeded eeeeedfe fffddZded ed ededed eeeeeedfe ffeeeeedfe ffeee feee fff ddZd eeeeedfe ffd!eeeeedfe ffd eee ffd"d#Zd$eee fd%eee fd eee ffd&d'Zd(eee fd)eee fd*eee fd+eee fd,eee fd-eee fd.e d/e d e fd0d1Z!d2eed3eeeeedfe ffd4eeeeedfe ffd5eee fd6eee fd ed ed.e d/e deded ee eee feee feee feee fffd7d8Z" dMd9e eeefd:e eeeeefd;eee fdeee fd?eee fd@eee fd ed ed.e d/e dededAeee d eeee feee feee feee feee feee feee ff dBdCZ#d;eee fdeee fd?eee fd@eee fd.e d/e d e fdDdEZ$ F G H I I IdNd9e eeefd:ee eeefd ed ed/e dededJed e e ee e fffdKdLZ%dS)O defaultdict)Sequence)chain)ListOptionalUnionN)Tensortensor)_validate_inputsgؗҜ?@[\]^_`{|}~ n_char_order n_word_orderreturncCs|ddt|D}ddt|D}ddt|D}ddt|D}ddt|D}ddt|D}||||||fS)ahPrepare dictionaries with default zero values for total ref, hypothesis and matching character and word n-grams. Args: n_char_order: A character n-gram order. n_word_order: A word n-gram order. Return: Dictionaries with default zero values for total reference, hypothesis and matching character and word n-grams. cSi|] }|dtdqSr .0nr^/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/torchmetrics/functional/text/chrf.py 6z*_prepare_n_grams_dicts..cSrrrrrrrr7rcSrrrrrrrr8rcSrrrrrrrr9rcSrrrrrrrr:rcSrrrrrrrr;r)range)r r total_preds_char_n_gramstotal_preds_word_n_gramstotal_target_char_n_gramstotal_target_word_n_gramstotal_matching_char_n_gramstotal_matching_word_n_gramsrrr_prepare_n_grams_dicts&sr"sentence whitespacecCs |rt|St|ddS)aSplit sentence into individual characters. Args: sentence: An input sentence to split. whitespace: An indication whether to keep whitespaces during character n-gram extraction. Return: A list of separated characters.  )liststripreplace)r#r$rrr_get_charactersGs r*wordcCsXt|dkr |gS|dtvr|dd|dgS|dtvr)|d|ddgS|gS)axSeparates out punctuation from beginning and end of words for chrF. Adapted from https://github.com/m-popovic/chrF and https://github.com/mjpost/sacrebleu/blob/master/sacrebleu/metrics/chrf.py. Args: word: An input word to be separated from a punctuation if present. Return: A list of a single word or a separated word and punctuation. rNr)len _PUNCTUATIONS)r+rrr_separate_word_and_punctuationWs   r/cCs ttdd|DS)zSeparates out punctuation from beginning and end of words for chrF for all words in the sentence. Args: sentence: An input sentence to split Return: An aggregated list of separated words and punctuation. css|]}t|VqdSN)r/)rr+rrr xsz-_get_words_and_punctuation..)r'r from_iterabler(split)r#rrr_get_words_and_punctuationns r4char_or_word_list n_gram_order.csftdd}td|dD]#fddttdDD]}||td7<q!q |S)zCalculate n-gram counts. Args: char_or_word_list: A list of characters of words n_gram_order: The largest number of n-gram. Return: A dictionary of dictionaries with a counts of given n-grams. cSs tddS)NcStdSNrrrrrrz1_ngram_counts....rrrrrr9s z_ngram_counts..rc3s$|] }t||VqdSr0)tuple)rir5rrrr1s"z _ngram_counts..)rrr-r )r5r6ngramsngramrr=r _ngram_counts{s (r@ lowercasec sdtdtdtdtdttttttdftfftttttdftffff fdd }d tttttdftffdtttffd d }|||||\}}||} ||} ||| | fS) aGet n-grams and total n-grams. Args: sentence: An input sentence n_char_order: A character n-gram order. n_word_order: A word n-gram order. lowercase: An indication whether to enable case-insensitivity. whitespace: An indication whether to keep whitespaces during character n-gram extraction. Return: char_n_grams_counts: A dictionary of dictionaries with sentence character n-grams. word_n_grams_counts: A dictionary of dictionaries with sentence word n-grams. total_char_n_grams: A dictionary containing a total number of sentence character n-grams. total_word_n_grams: A dictionary containing a total number of sentence word n-grams. r#r r rAr.cs2|r|}tt||}tt||}||fS)z@Get a dictionary of dictionaries with a counts of given n-grams.)lowerr@r*r4)r#r r rAchar_n_grams_countsword_n_grams_countsr$rr_char_and_word_ngrams_countss zJ_get_n_grams_counts_and_total_ngrams.._char_and_word_ngrams_countsn_grams_countscSs6tdd}|D]}t||||<q|S)z.Get total sum of n-grams over n-grams w.r.t n.cSr7r8rrrrrr9r:zQ_get_n_grams_counts_and_total_ngrams.._get_total_ngrams..)rsumvaluesdetachclone)rG total_n_gramsrrrr_get_total_ngramss z?_get_n_grams_counts_and_total_ngrams.._get_total_ngrams)strintboolr;dictr ) r#r r rAr$rFrMrCrDtotal_char_n_gramstotal_word_n_gramsrrEr$_get_n_grams_counts_and_total_ngramss$:2  rThyp_n_grams_countsref_n_grams_countscsHtdd}D]fddD}t||<q|S)zGet a number of n-gram matches between reference and hypothesis n-grams. Args: hyp_n_grams_counts: n-grams counts for hypothesis ref_n_grams_counts: n-grams counts for reference Return: matching_n_grams cSr7r8rrrrrr9r:z$_get_ngram_matches..cs(g|]}t||qSr)torchmin)rn_gramrUrrVrr sz&_get_ngram_matches..)rrHrJrK)rUrVmatching_n_grams min_n_gramsrrZr_get_ngram_matchess r^rLn_gramscCs"|D] }||||7<q|S)aBAggregate total n-grams to keep corpus-level statistics. Args: total_n_grams: A dictionary containing a total corpus-level number of n-grams. n_grams: A dictionary containing a sentence-level number of n-grams. Return: A dictionary containing a total corpus-level number of n-grams. r)rLr_rrrr_sum_over_dictss r`matching_char_n_gramsmatching_word_n_gramshyp_char_n_gramshyp_word_n_gramsref_char_n_gramsref_word_n_gramsn_orderbetac Cszdtttfdtttfdtttfdtdtttff dd}|||||} |||||} t| t| t|S)aCalculate sentence-level chrF/chrF++ score. For given hypothesis and reference statistics (either sentence-level or corpus-level) the chrF/chrF++ score is returned. Args: matching_char_n_grams: A total number of matching character n-grams between the best matching reference and hypothesis. matching_word_n_grams: A total number of matching word n-grams between the best matching reference and hypothesis. hyp_char_n_grams: A total number of hypothesis character n-grams. hyp_word_n_grams: A total number of hypothesis word n-grams. ref_char_n_grams: A total number of reference character n-grams. ref_word_n_grams: A total number of reference word n-grams. n_order: A sum of character and word n-gram order. beta: A parameter determining an importance of recall w.r.t. precision. If `beta=1`, their importance is equal. Return: A chrF/chrF++ score. This function is universal both for sentence-level and corpus-level calculation. r\ ref_n_grams hyp_n_gramsrhrcsZfddDfddDfddDfddD}|S)zGet n-gram level f-score.cs2i|]}||dkr||ntdqSrrrr)rjr\rrr &zA_calculate_fscore.._get_n_gram_fscore..cs2i|]}||dkr||ntdqSrkrr)r\rirrrrlcs.i|]}|td||tqS))rWmax_EPS_SMOOTHINGr)rh precisionrecallrrrs"cs2i|]}|dd|||qS)rrmrr)rh denominatorrprqrrrrlr)r\rirjrhf_scorer)rhrrrjr\rprqrir_get_n_gram_fscores  z-_calculate_fscore.._get_n_gram_fscore)rQrOr floatrHrIr ) rarbrcrdrerfrgrhrtchar_n_gram_f_scoreword_n_gram_f_scorerrr_calculate_fscores      rxtargetspred_char_n_grams_countspred_word_n_grams_countspred_char_n_gramspred_word_n_gramsc  Cstd} tdd} tdd} tdd}tdd}|D]1}t|||| | \}}}}t||}t||}t||||||||}|| krO|} |} |} |}|}q| | | ||fS)aSCalculate the best sentence-level chrF/chrF++ score. For a given pre-processed hypothesis, all references are evaluated and score and statistics for the best matching reference is returned. Args: targets: An iterable of references. pred_char_n_grams_counts: A dictionary of dictionaries with hypothesis character n-grams. pred_word_n_grams_counts: A dictionary of dictionaries with hypothesis word n-grams. pred_char_n_grams: A total number of hypothesis character n-grams. pred_word_n_grams: A total number of hypothesis word n-grams. n_char_order: A character n-gram order. n_word_order: A word n-gram order. n_order: A sum of character and word n-gram order. beta: A parameter determining an importance of recall w.r.t. precision. If `beta=1`, their importance is equal. lowercase: An indication whether to enable case-insensitivity. whitespace: An indication whether to keep whitespaces during character n-gram extraction. Return: Return chrF/chrF++ score and statistics for the best matching hypothesis and reference. f_score: A sentence-level chrF/chrF++ score. matching_char_n_grams: A total number of matching character n-grams between the best matching reference and hypothesis. matching_word_n_grams: A total number of matching word n-grams between the best matching reference and hypothesis. target_char_n_grams: A total number of reference character n-grams. target_word_n_grams: A total number of reference word n-grams. rcSr7r8rrrrrr9Lr:z6_calculate_sentence_level_chrf_score..cSr7r8rrrrrr9Mr:cSr7r8rrrrrr9Nr:cSr7r8rrrrrr9Or:)r rrTr^rx)ryrzr{r|r}r r rgrhrAr$ best_f_scorebest_matching_char_n_gramsbest_matching_word_n_gramsbest_target_char_n_gramsbest_target_word_n_gramstargettarget_char_n_grams_countstarget_word_n_grams_countstarget_char_n_gramstarget_word_n_gramsrarbrsrrr$_calculate_sentence_level_chrf_score sJ+       rpredsrrrrrr r!sentence_chrf_scorec Cst||\}}t||D]M\}}t||| | | \}}}}t||}t||}t||||||| | | | | \}}}}}|durE||dt||}t||}t||}t||}q |||||||fS)aUpdate function for chrf score. Args: preds: An iterable of hypothesis corpus. target: An iterable of iterables of reference corpus. total_preds_char_n_grams: A dictionary containing a total number of hypothesis character n-grams. total_preds_word_n_grams: A dictionary containing a total number of hypothesis word n-grams. total_target_char_n_grams: A dictionary containing a total number of reference character n-grams. total_target_word_n_grams: A dictionary containing a total number of reference word n-grams. total_matching_char_n_grams: A dictionary containing a total number of matching character n-grams between references and hypotheses. total_matching_word_n_grams: A dictionary containing a total number of total matching word n-grams between references and hypotheses. n_char_order: A character n-gram order. n_word_order: A word n-gram order. n_order: Sum of character and word n-gram order. beta: A parameter determining an importance of recall w.r.t. precision. If `beta=1`, their importance is equal. lowercase: An indication whether to enable case-insensitivity. whitespace: An indication whether to keep whitespaces during character n-gram extraction. sentence_chrf_score: A list of sentence-level chrF/chrF++ scores. Return: total_target_char_n_grams: number of reference character n-grams. total_target_word_n_grams: number of reference word n-grams. total_preds_char_n_grams: number of hypothesis character n-grams. total_preds_word_n_grams: number of hypothesis word n-grams. total_matching_char_n_grams: number of matching character n-grams between references and hypotheses. total_matching_word_n_grams: number of total matching word n-grams between references and hypotheses. sentence_chrf_score: A list of sentence-level chrF/chrF++ scores. Raises: ValueError: If length of ``preds`` and ``target`` differs. Nr)r ziprTr`rappend unsqueeze)rrrrrrr r!r r rgrhrAr$r target_corpuspredryrzr{r|r}sentence_level_f_scorerarbrrrrr_chrf_score_updatevsV<      rc Cst||||||||S)akCompute chrF/chrF++ score based on pre-computed target, prediction and matching character and word n-grams. Args: total_preds_char_n_grams: number of hypothesis character n-grams. total_preds_word_n_grams: number of hypothesis word n-grams. total_target_char_n_grams: number of reference character n-grams. total_target_word_n_grams: number of reference word n-grams. total_matching_char_n_grams: number of matching character n-grams between references and hypotheses. total_matching_word_n_grams: number of total matching word n-grams between references and hypotheses. n_order: A sum of character and word n-gram order. beta: A parameter determining an importance of recall w.r.t. precision. If `beta=1`, their importance is equal. Return: A corpus-level chrF/chrF++ score. )rx)rrrrr r!rgrhrrr_chrf_score_computesrrm@Freturn_sentence_level_scorecCst|tr |dkr tdt|tr|dkrtd|dkr"tdt||}t||\} } } } } }|r7gnd}t||| | | | | ||||||||\} } } } } }}t| | | | | |||}|rf|t|fS|S)uaCalculate `chrF score`_ of machine translated text with one or more references. This implementation supports both chrF score computation introduced in [1] and chrF++ score introduced in `chrF++ score`_. This implementation follows the implementations from https://github.com/m-popovic/chrF and https://github.com/mjpost/sacrebleu/blob/master/sacrebleu/metrics/chrf.py. Args: preds: An iterable of hypothesis corpus. target: An iterable of iterables of reference corpus. n_char_order: A character n-gram order. If `n_char_order=6`, the metrics refers to the official chrF/chrF++. n_word_order: A word n-gram order. If `n_word_order=2`, the metric refers to the official chrF++. If `n_word_order=0`, the metric is equivalent to the original chrF. beta: A parameter determining an importance of recall w.r.t. precision. If `beta=1`, their importance is equal. lowercase: An indication whether to enable case-insensitivity. whitespace: An indication whether to keep whitespaces during character n-gram extraction. return_sentence_level_score: An indication whether a sentence-level chrF/chrF++ score to be returned. Return: A corpus-level chrF/chrF++ score. (Optionally) A list of sentence-level chrF/chrF++ scores if `return_sentence_level_score=True`. Raises: ValueError: If ``n_char_order`` is not an integer greater than or equal to 1. ValueError: If ``n_word_order`` is not an integer greater than or equal to 0. ValueError: If ``beta`` is smaller than 0. Example: >>> from torchmetrics.functional.text import chrf_score >>> preds = ['the cat is on the mat'] >>> target = [['there is a cat on the mat', 'a cat is on the mat']] >>> chrf_score(preds, target) tensor(0.8640) References: [1] chrF: character n-gram F-score for automatic MT evaluation by Maja Popović `chrF score`_ [2] chrF++: words helping character n-grams by Maja Popović `chrF++ score`_ rzMExpected argument `n_char_order` to be an integer greater than or equal to 1.rzMExpected argument `n_word_order` to be an integer greater than or equal to 0.z.Expected argument `beta` to be greater than 0.N) isinstancerO ValueErrorrur"rrrWcat)rrr r rhrAr$rrgrrrrr r!r chrf_f_scorerrr chrf_score sl7     rr0)rrmrFFF)& collectionsrcollections.abcr itertoolsrtypingrrrrWr r #torchmetrics.functional.text.helperr rosetr.rOr;rQr"rNrPr'r*r/r4r@rTr^r`rurxrrrrrrrrsh    > !2    3  .       9     0 e                    o       *