o .wi@sJddlmZddlmZddlmZmZmZddlZddlm Z m Z dee de defd d Z d e dee fd d Zdefdee deee de de de de de dee gee fdee e ffddZde de de de de deedede fddZ   ddee ee fdeee ee fde dedeeede f ddZdS) )Counter)Sequence)CallableOptionalUnionN)Tensortensorngram_input_listn_gramreturncCs\t}td|dD]!}tt||dD]}t||||}||d7<qq |S)a Count how many times each word appears in a given text with ngram. Args: ngram_input_list: A list of translated text or reference texts n_gram: gram value ranged 1 to 4 Return: ngram_counter: a collections.Counter object of ngram )rrangelentuple)r r ngram_counterij ngram_keyr^/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/torchmetrics/functional/text/bleu.py _count_ngrams rsentencecCs|S)zTokenizes sentence into list of words. Args: sentence: A sentence separated by white space. Return: List of words )split)rrrr _tokenize_fn0s rpredstarget numerator denominator preds_len target_len tokenizercsfdd|D}fdd|D} t| |D]c\} |t7}dd| D} fdd| D} || | t| 7}t|} t}| D] }|t||O}qF| |@}|D]}|t|d||7<qV| D]}|t|d| |7<qiq||fS)aUpdate and returns variables required to compute the BLEU score. Args: preds: An iterable of machine translated corpus target: An iterable of iterables of reference corpus numerator: Numerator of precision score (true positives) denominator: Denominator of precision score (true positives + false positives) preds_len: count of words in a candidate prediction target_len: count of words in a reference translation target: count of words in a reference translation n_gram: gram value ranged 1 to 4 tokenizer: A function that turns sentence into list of words csg|] }fdd|DqS)cg|] }|r |ngqSrr.0liner!rr Uz1_bleu_score_update...r)r$tr&rrr'Usz&_bleu_score_update..cr"rrr#r&rrr'Vr(cSsg|]}t|qSr)rr$tgtrrrr'Zscsg|] }tt|qSr)absr)r$x)predrrr'[r(r )ziprindexminrr)rrrrrr r r!target_preds_targetstarget_len_listtarget_len_diff preds_countertarget_counterr+ngram_counter_clip counter_clipcounterr)r.r!r_bleu_score_update=s$  r<weightssmoothc Cs|j}t|dkrtd|dS|r4tt|tj||dt|tj||d}|d|d|d<n||}t||dt|} tt | } ||krUtd|dntd||} | | S)aCompute the BLEU score. Args: preds_len: count of words in a candidate translation target_len: count of words in a reference translation numerator: Numerator of precision score (true positives) denominator: Denominator of precision score (true positives + false positives) n_gram: gram value ranged 1 to 4 weights: Weights used for unigrams, bigrams, etc. to calculate BLEU score. smooth: Whether to apply smoothing )devicer?r ) r@r1rtorchdivaddoneslogexpsum) rr rrr r=r>r@precision_scoreslog_precision_scoresgeometric_meanbrevity_penaltyrrr_bleu_score_computens  &rMFc Cst|tr|gn|}dd|D}t|t|kr'tdt|dt||dur=t||kr=tdt|d||durHd|g|}t|}t|}td} td} t||||| | |t\} } t | | |||||S) a3Calculate `BLEU score`_ of machine translated text with one or more references. Args: preds: An iterable of machine translated corpus target: An iterable of iterables of reference corpus n_gram: Gram value ranged from 1 to 4 smooth: Whether to apply smoothing - see [2] weights: Weights used for unigrams, bigrams, etc. to calculate BLEU score. If not provided, uniform weights are used. Return: Tensor with BLEU Score Raises: ValueError: If ``preds`` and ``target`` corpus have different lengths. ValueError: If a length of a list of weights is not ``None`` and not equal to ``n_gram``. Example: >>> from torchmetrics.functional.text import bleu_score >>> preds = ['the cat is on the mat'] >>> target = [['there is a cat on the mat', 'a cat is on the mat']] >>> bleu_score(preds, target) tensor(0.7598) References: [1] BLEU: a Method for Automatic Evaluation of Machine Translation by Papineni, Kishore, Salim Roukos, Todd Ward, and Wei-Jing Zhu `BLEU`_ [2] Automatic Evaluation of Machine Translation Quality Using Longest Common Subsequence and Skip-Bigram Statistics by Chin-Yew Lin and Franz Josef Och `Machine Translation Evolution`_ cSs g|] }t|tr |gn|qSr) isinstancestrr*rrrr's zbleu_score..zCorpus has different size z != Nz5List of weights has different weights than `n_gram`: rAr?) rNrOr ValueErrorrBzerosrr<rrM) rrr r>r=r3r2rrrr rrr bleu_scores (  rR)rFN) collectionsrcollections.abcrtypingrrrrBrrrOintrrrr<floatboolrMrRrrrrsz    1 +