o yiU@sxUddlZddlmZddlmZmZmZmZmZm Z m Z m Z ddl Z ddl m Z mZddlmZddlmZddgiZd d d d d ddddddd Zeee eeffed<dZdLddZdede efddZdedededeee ffd d!Z "dMd#e ed$e ed%ede ee e effd&d'Zd(e e ed#e ed$e ede efd)d*Zd+e e ed$e ede efd,d-Z   dNd.ed/eed0eegefd1eege efde ef d2d3Z d4e ed5e ed6edeee ffd7d8Z!d4e ed5e edeee ffd9d:Z"d4e e ed5e e edeee ffd;d<Z#   dNd=e ed5e e ed>ee eefd?ed/eed0eegefd1eege efdee eefeeee fffd@dAZ$dBeeee fdeee ffdCdDZ% E "   FdOd=e ee efd5e ee ee e efd?eddGed0eegefd1eege efdHe ee edIffdeee ffdJdKZ&dS)PN)Counter)AnyCallableDictListOptionalSequenceTupleUnion)Tensortensor)Literal)_NLTK_AVAILABLE) rouge_score_rouge_score_updatenltk LLsum) rouge1rouge2rouge3rouge4rouge5rouge6rouge7rouge8rouge9rougeL rougeLsumALLOWED_ROUGE_KEYS)avgbestreturnc Cs^ddl}z |jdWdSty.z|jddddddWYdSty-tdww) zxCheck whether `nltk` `punkt` is downloaded. If not, try to download if a machine is connected to the internet. rNztokenizers/punkt.zippunktTF)quietforce halt_on_errorraise_on_errorzz`nltk` resource `punkt` is not available on a disk and cannot be downloaded as a machine is not connected to the internet.)rdatafind LookupErrordownload ValueErrorOSError)rr7V/home/ubuntu/.local/lib/python3.10/site-packages/torchmetrics/functional/text/rouge.py _ensure_nltk_punkt_is_downloaded*s  r9xcCs2tstdddl}ttdd|||S)zdThe sentence is split to get rougeLsum scores matching published rougeL scores for BART and PEGASUS.zQROUGE-Lsum calculation requires that `nltk` is installed. Use `pip install nltk`.rNz)rModuleNotFoundErrorrr9resub sent_tokenize)r:rr7r7r8_split_sentence=s  r@ hits_or_lcspred_len target_lencCsp||}||}||krdkr!nn ttdtdtddSd||||}tt|t|t|dS)akThis computes precision, recall and F1 score based on hits/lcs, and the length of lists of tokenizer predicted and target sentences. Args: hits_or_lcs: A number of matches or a length of the longest common subsequence. pred_len: A length of a tokenized predicted sentence. target_len: A length of a tokenized target sentence.  precisionrecallfmeasurer)dictr )rArBrCrFrGrHr7r7r8_compute_metricsIs  rJF pred_tokens target_tokensreturn_full_tablecsfddtt|dD}tdt|dD]@}tdtdD]4}||d|dkrB||d|dd|||<q#t||d||||d|||<q#q|r]|S|ddS)zCommon DP algorithm to compute the length of the longest common subsequence. Args: pred_tokens: A tokenized predicted sentence. target_tokens: A tokenized target sentence. csg|] }dgtdqS)rr)len).0_rKr7r8 dsz_lcs..r)rangerNmax)rKrLrMlcsijr7rQr8_lcs[s ", rY lcs_tablecCst|}t|}g}|dkrP|dkrP||d||dkr/|d|d|d8}|d8}n|||d||d|krD|d8}n|d8}|dkrP|dks|S)zBacktrack LCS table. Args: lcs_table: A table containing information for the calculation of the longest common subsequence. pred_tokens: A tokenized predicted sentence. target_tokens: A tokenized target sentence. rr)rNinsert)rZrKrLrWrXbacktracked_lcsr7r7r8_backtracked_lcsps     r]pred_tokens_listcsndttdttdttfdddtttdttfdd}fd d |D}fd d ||D}|S) zFind union LCS between a target sentence and iterable of predicted tokens. Args: pred_tokens_list: A tokenized predicted sentence split by ' '. target_tokens: A tokenized single part of target sentence split by ' '. Return: rKrLr+cSst||dd}t|||}|S)zSReturns one of the longest of longest common subsequence via backtracked lcs table.T)rM)rYr])rKrLrZbacktracked_lcs_tabler7r7r8lcs_inds z_union_lcs..lcs_ind lcs_tablescSstttj|S)z#Find union LCS given a list of LCS.)sortedlistsetunion)rar7r7r8 find_unionsz_union_lcs..find_unioncsg|]}|qSr7r7)rOrKr`rLr7r8rRsz_union_lcs..csg|]}|qSr7r7rOrW)rLr7r8rR)rstrint)r^rLrfra union_lcsr7rgr8 _union_lcss " rmtextstemmer normalizer tokenizercsft|r||ntdd|}t|r||ntd|}r*fdd|D}dd|D}|S)aRouge score should be calculated only over lowercased words and digits. Optionally, Porter stemmer can be used to strip word suffixes to improve matching. The text normalization follows the implemantion from `Rouge score_Text Normalizition`_ Args: text: An input sentence. stemmer: Porter stemmer instance to strip word suffixes to improve matching. normalizer: A user's own normalizer function. If this is ``None``, replacing any non-alpha-numeric characters with spaces is default. This function must take a ``str`` and return a ``str``. tokenizer: A user's own tokenizer function. If this is ``None``, splitting by spaces is default This function must take a ``str`` and return ``Sequence[str]`` z [^a-z0-9]+ z\s+cs&g|]}t|dkr|n|qS)r)rNstemrOr:ror7r8rR&z0_normalize_and_tokenize_text..cSs&g|]}t|trt|dkr|qS)r) isinstancerjrNrtr7r7r8rRrv)callabler=r>lowersplit)rnrorprqtokensr7rur8_normalize_and_tokenize_texts "r|predtargetn_gramcsdttdtdtfdd}||||||tt}}d||fvr8ttdtdtddStfd d tD}t |t |d t |d S) zThis computes precision, recall and F1 score for the Rouge-N metric. Args: pred: A predicted sentence. target: A target sentence. n_gram: N-gram overlap. r{nr+csDt}fddttdDD] }||d7<q|S)Nc3s$|] }t||VqdSN)tuplerhrr{r7r8 s"z9_rouge_n_score.._create_ngrams..r)rrTrN)r{rngramsngramr7rr8_create_ngramss(z&_rouge_n_score.._create_ngramsrrDrEc3s"|] }t||VqdSr)min)rOw pred_ngrams target_ngramsr7r8rs z!_rouge_n_score..r) rrjrkrsumvaluesrIr rdrJrU)r}r~rrrBrChitsr7rr8_rouge_n_scores  rcCsNt|t|}}d||fvrttdtdtddSt||}t|||S)zThis computes precision, recall and F1 score for the Rouge-L metric. Args: pred: A predicted sentence. target: A target sentence. rrDrE)rNrIr rYrJ)r}r~rBrCrVr7r7r8_rouge_l_scores    rc Csttt|}ttt|}d||fvr!ttdtdtddSdtttdtfdd}||}||}d}|D],}t||} | D]"} || dkre|| dkre|d7}|| d8<|| d8<qCq:t |||S) a:This computes precision, recall and F1 score for the Rouge-LSum metric. More information can be found in Section 3.2 of the referenced paper [1]. This implementation follow the official implementation from: https://github.com/google-research/google-research/blob/master/rouge/rouge_scorer.py Args: pred: An iterable of predicted sentence split by ' '. target: An iterable target sentence split by ' '. References [1] ROUGE: A Package for Automatic Evaluation of Summaries by Chin-Yew Lin. https://aclanthology.org/W04-1013/ rrDrE sentencesr+cSst}|D]}||q|Sr)rupdate)rrsentencer7r7r8_get_token_countss z,_rouge_lsum_score.._get_token_countsr) rmaprNrIr rrjrrmrJ) r}r~rBrCrpred_tokens_counttarget_tokens_countrtgtrVtokenr7r7r8_rouge_lsum_scores$    rpredsrouge_keys_values accumulatecsdd|D}t||D]\}} dd|D} dd|D} g} t|} d|vr8fddt|D}| D]P}t|}d|vrTfddt|D}|D],}t|trdt| ||}n|d krnt| |}n |dkrwt||}|| |<| ||qV| | q:|d kr|d t fd d| D}tt | }|D] }||| ||qq |d krdd|D}| D].\}}i|D]}|D]\}}|vrg|<||qqчfddD||<q|D] }||||qq |S)a Update the rouge score with the current set of predicted and target sentences. Args: preds: An iterable of predicted sentences. target: An iterable of iterable of target sentences. rouge_keys_values: List of N-grams/'L'/'Lsum' arguments. accumulate: Useful incase of multi-reference rouge score. ``avg`` takes the avg of all references with respect to predictions ``best`` takes the best fmeasure score obtained between prediction and multiple corresponding references. Allowed values are ``avg`` and ``best``. stemmer: Porter stemmer instance to strip word suffixes to improve matching. normalizer: A user's own normalizer function. If this is ``None``, replacing any non-alpha-numeric characters with spaces is default. This function must take a `str` and return a `str`. tokenizer: A user's own tokenizer function. If this is ``None``, spliting by spaces is default This function must take a `str` and return `Sequence[str]` Example: >>> preds = "My name is John".split() >>> target = "Is your name John".split() >>> from pprint import pprint >>> score = _rouge_score_update(preds, target, rouge_keys_values=[1, 2, 3, 'L'], accumulate='best') >>> pprint(score) {1: [{'fmeasure': tensor(0.), 'precision': tensor(0.), 'recall': tensor(0.)}, {'fmeasure': tensor(0.), 'precision': tensor(0.), 'recall': tensor(0.)}, {'fmeasure': tensor(0.), 'precision': tensor(0.), 'recall': tensor(0.)}, {'fmeasure': tensor(0.), 'precision': tensor(0.), 'recall': tensor(0.)}], 2: [{'fmeasure': tensor(0.), 'precision': tensor(0.), 'recall': tensor(0.)}, {'fmeasure': tensor(0.), 'precision': tensor(0.), 'recall': tensor(0.)}, {'fmeasure': tensor(0.), 'precision': tensor(0.), 'recall': tensor(0.)}, {'fmeasure': tensor(0.), 'precision': tensor(0.), 'recall': tensor(0.)}], 3: [{'fmeasure': tensor(0.), 'precision': tensor(0.), 'recall': tensor(0.)}, {'fmeasure': tensor(0.), 'precision': tensor(0.), 'recall': tensor(0.)}, {'fmeasure': tensor(0.), 'precision': tensor(0.), 'recall': tensor(0.)}, {'fmeasure': tensor(0.), 'precision': tensor(0.), 'recall': tensor(0.)}], 'L': [{'fmeasure': tensor(0.), 'precision': tensor(0.), 'recall': tensor(0.)}, {'fmeasure': tensor(0.), 'precision': tensor(0.), 'recall': tensor(0.)}, {'fmeasure': tensor(0.), 'precision': tensor(0.), 'recall': tensor(0.)}, {'fmeasure': tensor(0.), 'precision': tensor(0.), 'recall': tensor(0.)}]} cSi|]}|gqSr7r7rO rouge_keyr7r7r8 Jz'_rouge_score_update..cSi|]}|iqSr7r7rr7r7r8rMrcSrr7r7rr7r7r8rNrrcg|] }t|qSr7r|)rO pred_sentencerprorqr7r8rRR z'_rouge_score_update..crr7r)rO tgt_sentencerr7r8rR[rrr*rcsg|]}|dqS)rHr7)rOv)key_currr7r8rRmsr)cSrr7r7rr7r7r8rtscs i|] }|t|qSr7)torchr mean)rO_type)_dict_metric_score_batchr7r8rs)zipr|r@rwrkrrrappendcopyrr argmaxitemitems)rr~rrrorprqresultspred_raw target_raw result_inner result_avg list_resultsr} pred_lsumtarget_raw_innerr target_lsumrscore all_fmeasure highest_idxnew_result_avgmetricsmetricrvaluer7)rrrprorqr8rsh3      rsentence_resultscCs8i}|ikr|S|D] \}}t|||<q |S)zCompute the combined ROUGE metric for all the input set of predicted and target sentences. Args: sentence_results: Rouge-N/Rouge-L/Rouge-LSum metrics calculated for single sentence. )rrr r)rrrscoresr7r7r8_rouge_score_computes rr*rrr&r' use_stemmer rouge_keys.c Csj|r tstdddl}|r|jjnd}t|ts|f}|D]} | t vr6t d| dt t q dd|D} t|t r[t dd |Dr[t|t rT|gnd d|D}t|t rc|g}t|t rl|gg}t||| ||||d } i} | D]} d D] }g| d | d|<qq{| D]\} }|D]}|D]\}}| d | d||qqqt| S)aw Calculate `Calculate Rouge Score`_ , used for automatic summarization. Args: preds: An iterable of predicted sentences or a single predicted sentence. target: An iterable of iterables of target sentences or an iterable of target sentences or a single target sentence. accumulate: Useful incase of multi-reference rouge score. - ``avg`` takes the avg of all references with respect to predictions - ``best`` takes the best fmeasure score obtained between prediction and multiple corresponding references. use_stemmer: Use Porter stemmer to strip word suffixes to improve matching. normalizer: A user's own normalizer function. If this is ``None``, replacing any non-alpha-numeric characters with spaces is default. This function must take a ``str`` and return a ``str``. tokenizer: A user's own tokenizer function. If this is ``None``, spliting by spaces is default This function must take a ``str`` and return ``Sequence[str]`` rouge_keys: A list of rouge types to calculate. Keys that are allowed are ``rougeL``, ``rougeLsum``, and ``rouge1`` through ``rouge9``. Return: Python dictionary of rouge scores for each input rouge key. Example: >>> from torchmetrics.functional.text.rouge import rouge_score >>> preds = "My name is John" >>> target = "Is your name John" >>> from pprint import pprint >>> pprint(rouge_score(preds, target)) {'rouge1_fmeasure': tensor(0.7500), 'rouge1_precision': tensor(0.7500), 'rouge1_recall': tensor(0.7500), 'rouge2_fmeasure': tensor(0.), 'rouge2_precision': tensor(0.), 'rouge2_recall': tensor(0.), 'rougeL_fmeasure': tensor(0.5000), 'rougeL_precision': tensor(0.5000), 'rougeL_recall': tensor(0.5000), 'rougeLsum_fmeasure': tensor(0.5000), 'rougeLsum_precision': tensor(0.5000), 'rougeLsum_recall': tensor(0.5000)} Raises: ModuleNotFoundError: If the python package ``nltk`` is not installed. ValueError: If any of the ``rouge_keys`` does not belong to the allowed set of keys. References: [1] ROUGE: A Package for Automatic Evaluation of Summaries by Chin-Yew Lin. https://aclanthology.org/W04-1013/ zBStemmer requires that `nltk` is installed. Use `pip install nltk`.rNzGot unknown rouge key z. Expected to be one of cSsg|]}t|qSr7)r()rOkeyr7r7r8rRrizrouge_score..css|]}t|tVqdSr)rwrjrOrr7r7r8rszrouge_score..cSsg|]}|gqSr7r7rr7r7r8rRr)rorprqr)rHrFrGrougerP)rr<rrsporter PorterStemmerrwrr(keysr5rcallrjrrrr)rr~rrrprqrrrorrroutputrtprrrr7r7r8rsN?     r)r+N)F)NNN)r*FNNr)'r= collectionsrtypingrrrrrrr r rr r typing_extensionsr torchmetrics.utilities.importsr__doctest_requires__r(rjrk__annotations__ALLOWED_ACCUMULATE_VALUESr9r@rJboolrYr]rmr|rrrrrrr7r7r7r8s (    "    &  *%&.-   &r