o yipe*@sddlmZddlmZmZmZmZmZmZddl Z ddl m Z m Z ddl m Z e dZedZded ed eeee feee feee feee feee feee fffd d Zd eded eefddZded eefddZd ed eefddZdeeded eeeeedfe fffddZd eded ededed eeeeeedfe ffeeeeedfe ffeee feee fff ddZdeeeeedfe ffdeeeeedfe ffd eee ffd d!Zd"eee fd#eee fd eee ffd$d%Zd&eee fd'eee fd(eee fd)eee fd*eee fd+eee fd,ed-ed e fd.d/Zd0eed1eeeeedfe ffd2eeeeedfe ffd3eee fd4eee fded ed,ed-ededed ee eee feee feee feee fffd5d6Z dKd7eeeefd8eeeeeefd9eee fd:eee fd;eee fdeee fded ed,ed-ededed?eee d eeee feee feee feee feee feee feee ff d@dAZd9eee fd:eee fd;eee fdeee fd,ed-ed e fdBdCZ D E F G G GdLd7eeeefd8eeeeefded ed-edededHed ee ee e fffdIdJZ!dS)M defaultdict)DictListOptionalSequenceTupleUnionN)Tensortensor)_validate_inputsgؗҜ?@[\]^_`{|}~ n_char_order n_word_orderreturncCs|ddt|D}ddt|D}ddt|D}ddt|D}ddt|D}ddt|D}||||||fS)a~Prepare dictionaries dictionaries with default zero values for total reference, hypothesis and matching character and word n-grams. Args: n_char_order: A character n-gram order. n_word_order: A word n-gram order. Return: Dictionaries with default zero values for total reference, hypothesis and matching character and word n-grams. cSi|] }|dtdqSr .0nrU/home/ubuntu/.local/lib/python3.10/site-packages/torchmetrics/functional/text/chrf.py @z*_prepare_n_grams_dicts..cSrrrrrrrrArcSrrrrrrrrBrcSrrrrrrrrCrcSrrrrrrrrDrcSrrrrrrrrEr)range)r rtotal_preds_char_n_gramstotal_preds_word_n_gramstotal_target_char_n_gramstotal_target_word_n_gramstotal_matching_char_n_gramstotal_matching_word_n_gramsrrr_prepare_n_grams_dicts0sr#sentence whitespacecCs |rt|St|ddS)zSplit sentence into individual characters. Args: sentence: An input sentence to split. whitespace: An indication whether to keep whitespaces during character n-gram extraction. Return: A list of separated characters.  )liststripreplace)r$r%rrr_get_charactersQs r+wordcCsXt|dkr |gS|dtvr|dd|dgS|dtvr)|d|ddgS|gS)aw Separates out punctuations from beginning and end of words for chrF. Adapted from https://github.com/m-popovic/chrF and https://github.com/mjpost/sacrebleu/blob/master/sacrebleu/metrics/chrf.py. Args: word: An input word to be separated from a punctuation if present. Return: A list of a single word or a separated word and punctuation. rNr)len _PUNCTUATIONS)r,rrr_separate_word_and_punctiation`s   r0cCstdd|DgS)zSeparates out punctuations from beginning and end of words for chrF for all words in the sentence. Args: sentence: An input sentence to split Return: An aggregated list of separated words and punctuations. css|]}t|VqdSN)r0)rr,rrr }sz-_get_words_and_punctiation..)sumr)split)r$rrr_get_words_and_punctiationts r5char_or_word_list n_gram_order.csftdd}td|dD]#fddttdDD]}||td7<q!q |S)z Args: char_or_word_list: A list of characters of words n_gram_order: The largest number of n-gram. Return: A dictionary of dictionaries with a counts of given n-grams. cSs tddS)NcStdSNrrrrrrz1_ngram_counts....rrrrrr:s z_ngram_counts..rc3s$|] }t||VqdSr1)tuple)rir6rrrr2s"z _ngram_counts..)rrr.r )r6r7ngramsngramrr>r _ngram_countss (rA lowercasec sdtdtdtdtdttttttdftfftttttdftffff fdd }d tttttdftffdtttffd d }|||||\}}||} ||} ||| | fS) a Args: sentence: An input sentence n_char_order: A character n-gram order. n_word_order: A word n-gram order. lowercase: An indication whether to enable case-insensitivity. whitespace: An indication whether to keep whitespaces during character n-gram extraction. Return: char_n_grams_counts: A dictionary of dictionaries with sentence character n-grams. word_n_grams_counts: A dictionary of dictionaries with sentence word n-grams. total_char_n_grams: A dictionary containing a total number of sentence character n-grams. total_word_n_grams: A dictionary containing a total number of sentence word n-grams. r$r rrBr.cs2|r|}tt||}tt||}||fS)z@Get a dictionary of dictionaries with a counts of given n-grams.)lowerrAr+r5)r$r rrBchar_n_grams_countsword_n_grams_countsr%rr_char_and_word_ngrams_countss zJ_get_n_grams_counts_and_total_ngrams.._char_and_word_ngrams_countsn_grams_countscSs2tdd}|D]}tt||||<q|S)z.Get total sum of n-grams over n-grams w.r.t n.cSr8r9rrrrrr:r;zQ_get_n_grams_counts_and_total_ngrams.._get_total_ngrams..)rr r3values)rH total_n_gramsrrrr_get_total_ngramss z?_get_n_grams_counts_and_total_ngrams.._get_total_ngrams)strintboolrrr ) r$r rrBr%rGrKrDrEtotal_char_n_gramstotal_word_n_gramsrrFr$_get_n_grams_counts_and_total_ngramss$:2  rQhyp_n_grams_countsref_n_grams_countscs@tdd}D]ttfddD|<q|S)zGet a number of n-gram matches between reference and hypothesis n-grams. Args: hyp_n_grams_counts: ref_n_grams_counts: Return: matching_n_grams cSr8r9rrrrrr:r;z$_get_ngram_matches..c3s,|]}t||VqdSr1)torchmin)rn_gramrRrrSrrr2s  z%_get_ngram_matches..)rr r3)rRrSmatching_n_gramsrrWr_get_ngram_matchess  rYrJn_gramscCs"|D] }||||7<q|S)aAAggregate total n-grams to keep corpus-level statistics. Args: total_n_grams: A dictionary containing a total corpus-level number of n-grams. n_grams: A dictionary containing a sentence-level number of n-grams. Return: A dictionary containing a total corpus-level number of n-grams. r)rJrZrrrr_sum_over_dictss r[matching_char_n_gramsmatching_word_n_gramshyp_char_n_gramshyp_word_n_gramsref_char_n_gramsref_word_n_gramsn_orderbetac Cs~dtttfdtttfdtttfdtdtttff dd}|||||} |||||} t| t| t|} | S)aCalculate sentence-level chrF/chrF++ score. For given hypothesis and reference statistics (either sentence-level or corpus-level) the chrF/chrF++ score is returned. Args: matching_char_n_grams: A total number of matching character n-grams between the best matching reference and hypothesis. matching_word_n_grams: A total number of matching word n-grams between the best matching reference and hypothesis. hyp_char_n_grams: A total number of hypothesis character n-grams. hyp_word_n_grams: A total number of hypothesis word n-grams. ref_char_n_grams: A total number of reference character n-grams. ref_word_n_grams: A total number of reference word n-grams. n_order: A sum of character and word n-gram order. beta: A parameter determining an importance of recall w.r.t. precision. If `beta=1`, their importance is equal. Return: A chrF/chrF++ score. This function is universal both for sentence-level and corpus-level calucation. rX ref_n_grams hyp_n_gramsrcrcsZfddDfddDfddDfddD}|S)zGet n-gram level f-score.cs2i|]}||dkr||ntdqSrrrr)rerXrrr &zA_calculate_fscore.._get_n_gram_fscore..cs2i|]}||dkr||ntdqSrfrr)rXrdrrrrgcs.i|]}|td||tqS))rTmax_EPS_SMOOTHINGr)rc precisionrecallrrrs"cs2i|]}|dd|||qS)rrhrr)rc denominatorrkrlrrrrgr)rXrdrercf_scorer)rcrmrerXrkrlrdr_get_n_gram_fscores  z-_calculate_fscore.._get_n_gram_fscore)rrMr floatr3rIr ) r\r]r^r_r`rarbrcrochar_n_gram_f_scoreword_n_gram_f_scorernrrr_calculate_fscores      rstargetspred_char_n_grams_countspred_word_n_grams_countspreds_char_n_gramspreds_word_n_gramsc  Cstd} tdd} tdd} tdd}tdd}|D]1}t|||| | \}}}}t||}t||}t||||||||}|| krO|} |} |} |}|}q| | | ||fS)aTCalculate the best sentence-level chrF/chrF++ score. For a given pre-processed hypothesis, all references are evaluated and score and statistics for the best matching reference is returned. Args: targets: An iterable of references. preds_char_n_grams_counts: A dictionary of dictionaries with hypothesis character n-grams. preds_word_n_grams_counts: A dictionary of dictionaries with hypothesis word n-grams. pred_char_n_grams: A total number of hypothesis character n-grams. pred_word_n_grams: A total number of hypothesis word n-grams. n_char_order: A character n-gram order. n_word_order: A word n-gram order. n_order: A sum of character and word n-gram order. beta: A parameter determining an importance of recall w.r.t. precision. If `beta=1`, their importance is equal. lowercase: An indication whether to enable case-insensitivity. whitespace: An indication whether to keep whitespaces during character n-gram extraction. Return: Return chrF/chrF++ score and statistics for the best matching hypothesis and reference. f_score: A sentence-level chrF/chrF++ score. matching_char_n_grams: A total number of matching character n-grams between the best matching reference and hypothesis. matching_word_n_grams: A total number of matching word n-grams between the best matching reference and hypothesis. target_char_n_grams: A total number of reference character n-grams. target_word_n_grams: A total number of reference word n-grams. rcSr8r9rrrrrr:Mr;z6_calculate_sentence_level_chrf_score..cSr8r9rrrrrr:Nr;cSr8r9rrrrrr:Or;cSr8r9rrrrrr:Pr;)r rrQrYrs)rtrurvrwrxr rrbrcrBr% best_f_scorebest_matching_char_n_gramsbest_matching_word_n_gramsbest_target_char_n_gramsbest_target_word_n_gramstargettarget_char_n_grams_countstarget_word_n_grams_countstarget_char_n_gramstarget_word_n_gramsr\r]rnrrr$_calculate_sentence_level_chrf_score!sJ+       rpredsr~rrrr r!r"sentence_chrf_scorec Cst||\}}t||D]M\}}t||| | | \}}}}t||}t||}t||||||| | | | | \}}}}}|durE||dt||}t||}t||}t||}q |||||||fS)a Args: preds: An iterable of hypothesis corpus. target: An iterable of iterables of reference corpus. total_preds_char_n_grams: A dictionary containing a total number of hypothesis character n-grams. total_preds_word_n_grams: A dictionary containing a total number of hypothesis word n-grams. total_target_char_n_grams: A dictionary containing a total number of reference character n-grams. total_target_word_n_grams: A dictionary containing a total number of reference word n-grams. total_matching_char_n_grams: A dictionary containing a total number of matching character n-grams between references and hypotheses. total_matching_word_n_grams: A dictionary containing a total number of total matching word n-grams between references and hypotheses. n_char_order: A character n-gram order. n_word_order: A word n-gram order. n_order: Sum of character and word n-gram order. beta: A parameter determining an importance of recall w.r.t. precision. If `beta=1`, their importance is equal. lowercase: An indication whether to enable case-insensitivity. whitespace: An indication whether to keep whitespaces during character n-gram extraction. sentence_chrf_score: A list of sentence-level chrF/chrF++ scores. Return: total_target_char_n_grams: number of reference character n-grams. total_target_word_n_grams: number of reference word n-grams. total_preds_char_n_grams: number of hypothesis character n-grams. total_preds_word_n_grams: number of hypothesis word n-grams. total_matching_char_n_grams: number of matching character n-grams between references and hypotheses. total_matching_word_n_grams: number of total matching word n-grams between references and hypotheses. sentence_chrf_score: A list of sentence-level chrF/chrF++ scores. Raises: ValueError: If length of ``preds`` and ``target`` differs. Nr)r ziprQr[rappend unsqueeze)rr~rrrr r!r"r rrbrcrBr%r target_corpuspredrtrurvpred_char_n_gramspred_word_n_gramssentence_level_f_scorer\r]rrrrr_chrf_score_updatewsV:      rc Cst||||||||}|S)ajCompute chrF/chrF++ score based on pre-computed target, prediction and matching character and word n-grams. Args: total_preds_char_n_grams: number of hypothesis character n-grams. total_preds_word_n_grams: number of hypothesis word n-grams. total_target_char_n_grams: number of reference character n-grams. total_target_word_n_grams: number of reference word n-grams. total_matching_char_n_grams: number of matching character n-grams between references and hypotheses. total_matching_word_n_grams: number of total matching word n-grams between references and hypotheses. n_order: A sum of character and word n-gram order. beta: A parameter determining an importance of recall w.r.t. precision. If `beta=1`, their importance is equal. Return: A corpus-level chrF/chrF++ score. )rs) rrrr r!r"rbrc chrf_f_scorerrr_chrf_score_computes rrh@Freturn_sentence_level_scorecCst|tr |dkr tdt|tr|dkrtd|dkr"tdt||}t||\} } } } } }|r7gnd}t||| | | | | ||||||||\} } } } } }}t| | | | | |||}|rf|t|fS|S)uXCalculate `chrF score`_ of machine translated text with one or more references. This implementation supports both chrF score computation introduced in [1] and chrF++ score introduced in `chrF++ score`_. This implementation follows the implmenetaions from https://github.com/m-popovic/chrF and https://github.com/mjpost/sacrebleu/blob/master/sacrebleu/metrics/chrf.py. Args: preds: An iterable of hypothesis corpus. target: An iterable of iterables of reference corpus. n_char_order: A character n-gram order. If `n_char_order=6`, the metrics refers to the official chrF/chrF++. n_word_order: A word n-gram order. If `n_word_order=2`, the metric refers to the official chrF++. If `n_word_order=0`, the metric is equivalent to the original chrF. beta: A parameter determining an importance of recall w.r.t. precision. If `beta=1`, their importance is equal. lowercase: An indication whether to enable case-insesitivity. whitespace: An indication whether to keep whitespaces during character n-gram extraction. return_sentence_level_score: An indication whether a sentence-level chrF/chrF++ score to be returned. Return: A corpus-level chrF/chrF++ score. (Optionally) A list of sentence-level chrF/chrF++ scores if `return_sentence_level_score=True`. Raises: ValueError: If ``n_char_order`` is not an integer greater than or equal to 1. ValueError: If ``n_word_order`` is not an integer greater than or equal to 0. ValueError: If ``beta`` is smaller than 0. Example: >>> from torchmetrics.functional import chrf_score >>> preds = ['the cat is on the mat'] >>> target = [['there is a cat on the mat', 'a cat is on the mat']] >>> chrf_score(preds, target) tensor(0.8640) References: [1] chrF: character n-gram F-score for automatic MT evaluation by Maja Popović `chrF score`_ [2] chrF++: words helping character n-grams by Maja Popović `chrF++ score`_ rzMExpected argument `n_char_order` to be an integer greater than or equal to 1.rzMExpected argument `n_word_order` to be an integer greater than or equal to 0.z.Expected argument `beta` to be greater than 0.N) isinstancerM ValueErrorrpr#rrrTcat)rr~r rrcrBr%rrbrrrr r!r"rrrrr chrf_score sl5     rr1)rrhrFFF)" collectionsrtypingrrrrrr rTr r #torchmetrics.functional.text.helperr rjsetr/rMr#rLrNr+r0r5rArQrYr[rprsrrrrrrrrsd "  > !2    1  .       9     0 e                    m       *