o yiu@sFddlmZddlmZmZmZmZmZddlZddlm Z m Z dee de defdd Z d e dee fd d Zd efdee deee de de de de de dee gee fdee e ffddZde de de de de deedede fddZ  ddee ee fdeee ee fde dedeeede f ddZdS))Counter)CallableOptionalSequenceTupleUnionN)Tensortensorngram_input_listn_gramreturncCs\t}td|dD]!}tt||dD]}t||||}||d7<qq |S)aCounting how many times each word appears in a given text with ngram. Args: ngram_input_list: A list of translated text or reference texts n_gram: gram value ranged 1 to 4 Return: ngram_counter: a collections.Counter object of ngram )rrangelentuple)r r ngram_counterij ngram_keyrU/home/ubuntu/.local/lib/python3.10/site-packages/torchmetrics/functional/text/bleu.py _count_ngrams rsentencecCs|S)zTokenizes sentence into list of words. Args: sentence: A sentence separated by white space. Return: List of words )split)rrrr _tokenize_fn/s rpredstarget numerator denominator preds_len target_len tokenizercsfdd|D}fdd|D} t| |D]c\} |t7}dd| D} fdd| D} || | t| 7}t|} t}| D] }|t||O}qF| |@}|D]}|t|d||7<qV| D]}|t|d| |7<qiq||fS)aBUpdates and returns variables required to compute the BLEU score. Args: preds: An iterable of machine translated corpus target: An iterable of iterables of reference corpus numerator: Numerator of precision score (true positives) denominator: Denominator of precision score (true positives + false positives) preds_len: count of words in a candidate prediction target: count of words in a reference translation n_gram: gram value ranged 1 to 4 tokenizer: A function that turns sentence into list of words csg|] }fdd|DqS)cg|] }|r |ngqSrr.0liner"rr Qz1_bleu_score_update...r)r%tr'rrr(Qsz&_bleu_score_update..cr#rrr$r'rrr(Rr)cSsg|]}t|qSr)rr%tgtrrrr(Vscsg|] }tt|qSr)absr)r%x)predrrr(Wr)r )ziprindexminrr)rrrrr r!r r"target_preds_targetstarget_len_listtarget_len_diff preds_countertarget_counterr,ngram_counter_clip counter_clipcounterr)r/r"r_bleu_score_update;s$  r=weightssmoothc Cs|j}t|dkrtd|dS|r4tt|tj||dt|tj||d}|d|d|d<n||}t||dt|} tt | } ||krUtd|dntd||} | | } | S)aComputes the BLEU score. Args: preds_len: count of words in a candidate translation target_len: count of words in a reference translation numerator: Numerator of precision score (true positives) denominator: Denominator of precision score (true positives + false positives) n_gram: gram value ranged 1 to 4 weights: Weights used for unigrams, bigrams, etc. to calculate BLEU score. smooth: Whether to apply smoothing )devicer?r ) rAr2r torchdivaddoneslogexpsum) r r!rrr r>r?rAprecision_scoreslog_precision_scoresgeometric_meanbrevity_penaltybleurrr_bleu_score_computejs  &rOFc Cst|tr|gn|}dd|D}t|t|kr'tdt|dt||dur=t||kr=tdt|d||durHd|g|}t|}t|}td} td} t||||| | |t\} } t | | |||||S) u/Calculate `BLEU score`_ of machine translated text with one or more references. Args: preds: An iterable of machine translated corpus target: An iterable of iterables of reference corpus n_gram: Gram value ranged from 1 to 4 smooth: Whether to apply smoothing – see [2] weights: Weights used for unigrams, bigrams, etc. to calculate BLEU score. If not provided, uniform weights are used. Return: Tensor with BLEU Score Raises: ValueError: If ``preds`` and ``target`` corpus have different lengths. ValueError: If a length of a list of weights is not ``None`` and not equal to ``n_gram``. Example: >>> from torchmetrics.functional import bleu_score >>> preds = ['the cat is on the mat'] >>> target = [['there is a cat on the mat', 'a cat is on the mat']] >>> bleu_score(preds, target) tensor(0.7598) References: [1] BLEU: a Method for Automatic Evaluation of Machine Translation by Papineni, Kishore, Salim Roukos, Todd Ward, and Wei-Jing Zhu `BLEU`_ [2] Automatic Evaluation of Machine Translation Quality Using Longest Common Subsequence and Skip-Bigram Statistics by Chin-Yew Lin and Franz Josef Och `Machine Translation Evolution`_ cSs g|] }t|tr |gn|qSr) isinstancestrr+rrrr(s zbleu_score..zCorpus has different size z != Nz5List of weights has different weights than `n_gram`: rBr@) rPrQr ValueErrorrCzerosr r=rrO) rrr r?r>r4r3rrr r!rrr bleu_scores '  rT)rFN) collectionsrtypingrrrrrrCrr rQintrrr=floatboolrOrTrrrrsx   / ,