o .wi{m#@sddlZddlmZddlmZddlmZmZmZm Z ddl Z ddl m Z ddl m ZddlmZddlmZdd lmZmZmZmZdd lmZdd lmZer\er\dd lmZmZesad gZedZ eGdddeZ!GdddZ"de de de#de$de$def ddZ%ddde&e'e$ffddZ(de de$d e$d!e$de f d"d#Z)d$d%d&e&e'e fd'e*de#d(e&e'e$fde f d)d*Z+e ,d$d%d+ed'e*de#d(e&e'e$fd,e#de fd-d.Z-d/e e'ee'fd0e e'ee'fddd1e$de.e e e e ff d2d3Z/ 4dGd$d%d5ed6ed'e*de#d7e"d(e&e'e$fd,e#de fd8d9Z0 : ; < 4     = 4 >dHd/e e'ee'fd0e e'ee'fd?e e'ej1fd'e*d@e de#dAee*dBee*dCee e'e j2fd1ee$de$dDe$d,e#dEe#de e e.e e fffdFd Z3dS)IN)Sequence)unique) TYPE_CHECKINGListOptionalUnion)Tensor) functional) DataLoader)Literal)TokenizedDataset_get_progress_bar_input_data_collator_load_tokenizer_and_model)EnumStr)_TRANSFORMERS_GREATER_EQUAL_4_4)PreTrainedModelPreTrainedTokenizerBaseinfolm) kl_divergencealpha_divergencebeta_divergence ab_divergencerenyi_divergence l1_distance l2_distancel_infinity_distancefisher_rao_distancec@sFeZdZdZedefddZdZdZdZ dZ d Z d Z d Z d Zd ZdS)_IMEnumz8A helper Enum class for storing the information measure.returncCsdS)NzInformation measurer r r `/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/torchmetrics/functional/text/infolm.py_name:sz _IMEnum._namerrrrrrrrrN)__name__ __module__ __qualname____doc__ staticmethodstrr" KL_DIVERGENCEALPHA_DIVERGENCEBETA_DIVERGENCE AB_DIVERGENCERENYI_DIVERGENCE L1_DISTANCE L2_DISTANCEL_INFINITY_DISTANCEFISHER_RAO_DISTANCEr r r r!r6src @s(eZdZdZ  ddedeedeeddfddZd ed edefd d Z e d ed edefd dZ d ed edefddZ d ed edefddZ d ed edefddZd ed edefddZe d ed edefddZe d ed edefddZe d ed edefddZe d ed edefddZdS) _InformationMeasureuA wrapper class used for the calculation of different information measures. This metric can be used to measure the information between the discrete reference distributions of predicted and reference sentences. The class also handles input validation for `alpha` and `beta` parameters. Args: information_measure: A name of information measure to be used. Please use one of: ['kl_divergence', 'alpha_divergence', 'beta_divergence', 'ab_divergence', 'renyi_divergence', 'l1_distance', 'l2_distance', 'l_infinity_distance', 'fisher_rao_distance'] alpha: Alpha parameter of the divergence used for alpha, AB and Rényi divergence measures. beta: Beta parameter of the divergence used for beta and AB divergence measures. Raises: ValueError: If information measure is one from alpha, AB or Rényi divergence and parameter `alpha` is `None`. ValueError: If information measure is one from beta or divergence and parameter `beta` is `None`. ValueError: If information measure is alpha divergence and parameter `alpha` equals 0 or 1. ValueError: If information measure is beta divergence and parameter `beta` equals 0 or -1. ValueError: If information measure is AB divergence and parameter `alpha`, `beta` or `alpha + beta` equal 0. ValueError: If information measure is Rényi divergence and parameter `alpha` equals 1. Ninformation_measurealphabetarcCsbt||_tjtjtjf}|j|vr t|ts td|d|jtj tjfvr6t|ts6td|d|jtjkrMt|trE|dvrMtd|d|jtj krdt|tr\|dvrdtd|d|jtjkr|dus|dust dd ||fDsd ||||fvrtd |d|jtjkrt|tr|d krtd |d|pd |_ |pd |_ dS)Nz0Parameter `alpha` is expected to be defined for .z/Parameter `beta` is expected to be defined for )rzFParameter `alpha` is expected to be float differened from 0 and 1 for )rzFParameter `beta` is expected to be float differened from 0 and -1 for css|] }t|t VqdS)N) isinstancefloat).0pr r r! sz/_InformationMeasure.__init__..rzRParameters `alpha`, `beta` and their sum are expected to be differened from 0 for r7z@Parameter `alpha` is expected to be float differened from 1 for ) rfrom_strr3r*r,r-r9r: ValueErrorr+anyr4r5)selfr3r4r5 _bad_measuresr r r!__init__is:     z_InformationMeasure.__init__preds_distributiontarget_distributioncCs$t|d|jj}t|||S)N _calculate_)getattrr3valuetorch nan_to_num)rArDrEinformation_measure_functionr r r!__call__sz_InformationMeasure.__call__cCstj|t||ddS)aCalculate Kullback-Leibler divergence between discrete distributions of predicted and reference sentences. Args: preds_distribution: Discrete reference distribution of predicted sentences over the vocabulary. target_distribution: Discrete reference distribution of reference sentences over the vocabulary. Return: Kullback-Leibler divergence between discrete distributions of predicted and reference sentences. r8dim)rIsumlogrDrEr r r!_calculate_kl_divergencesz,_InformationMeasure._calculate_kl_divergencecCs:|j|jd}dtj||j|d|jdd|S)aCalculate alpha divergence between discrete distributions of predicted and reference sentences. Args: preds_distribution: Discrete reference distribution of predicted sentences over the vocabulary. target_distribution: Discrete reference distribution of reference sentences over the vocabulary. Return: Alpha divergence between discrete distributions of predicted and reference sentences. r7r8rM)r4rIrO)rArDrE _alpha_denomr r r!_calculate_alpha_divergences $z/_InformationMeasure._calculate_alpha_divergencecCsttj||j|jdd}||j|j|j}ttj||j|jdd}||j|j|j}ttj||j||jdd}||j|j}|||S)aCalculate AB divergence between discrete distributions of predicted and reference sentences. Args: preds_distribution: Discrete reference distribution of predicted sentences over the vocabulary. target_distribution: Discrete reference distribution of reference sentences over the vocabulary. Return: AB divergence between discrete distributions of predicted and reference sentences. r8rM)rIrPrOr5r4)rArDrEabcr r r!_calculate_ab_divergences  $ z,_InformationMeasure._calculate_ab_divergencecCsd|_|||S)aCalculate beta divergence between discrete distributions of predicted and reference sentences. Args: preds_distribution: Discrete reference distribution of predicted sentences over the vocabulary. target_distribution: Discrete reference distribution of reference sentences over the vocabulary. Return: Beta divergence between discrete distributions of predicted and reference sentences. g?)r4rXrArDrEr r r!_calculate_beta_divergences z._InformationMeasure._calculate_beta_divergencecCs2ttj||j|d|jdd|jdS)uCalculate Rényi divergence between discrete distributions of predicted and reference sentences. Args: preds_distribution: Discrete reference distribution of predicted sentences over the vocabulary. target_distribution: Discrete reference distribution of reference sentences over the vocabulary. Return: Rényi divergence between discrete distributions of predicted and reference sentences. r7r8rM)rIrPrOr4rYr r r!_calculate_renyi_divergences&z/_InformationMeasure._calculate_renyi_divergencecCtj||dddS)aCalculate L1 distance between discrete distributions of predicted and reference sentences. Args: preds_distribution: Discrete reference distribution of predicted sentences over the vocabulary. target_distribution: Discrete reference distribution of reference sentences over the vocabulary. Return: L1 distance between discrete distributions of predicted and reference sentences. r7r8r<rNrInormrQr r r!_calculate_l1_distancez*_InformationMeasure._calculate_l1_distancecCr\)aCalculate L2 distance between discrete distributions of predicted and reference sentences. Args: preds_distribution: Discrete reference distribution of predicted sentences over the vocabulary. target_distribution: Discrete reference distribution of reference sentences over the vocabulary. Return: L2 distance between discrete distributions of predicted and reference sentences. r8r]r^rQr r r!_calculate_l2_distanceraz*_InformationMeasure._calculate_l2_distancecCstj||tdddS)aCalculate L-infinity distance between discrete distributions of predicted and reference sentences. Args: preds_distribution: Discrete reference distribution of predicted sentences over the vocabulary. target_distribution: Discrete reference distribution of reference sentences over the vocabulary. Return: L-infinity distance between discrete distributions of predicted and reference sentences. infr8r])rIr_r:rQr r r!_calculate_l_infinity_distance sz2_InformationMeasure._calculate_l_infinity_distancec Cs(dttt||dddS)aCalculate Fisher-Rao distance between discrete distributions of predicted and reference sentences. Args: preds_distribution: Discrete reference distribution of predicted sentences over the vocabulary. target_distribution: Discrete reference distribution of reference sentences over the vocabulary. Return: Fisher-Rao distance between discrete distributions of predicted and reference sentences. rbr8rr7)rIacosclampsqrtrOrQr r r!_calculate_fisher_rao_distances(z2_InformationMeasure._calculate_fisher_rao_distance)NN)r#r$r%r&$_ALLOWED_INFORMATION_MEASURE_LITERALrr:rCrrLr'rRrTrXrZr[r`rcrerir r r r!r2Is8" $r2 input_idsattention_maskidf batch_size num_workersrcCst|||}t|||dS)aHPrepare dataloader. Args: input_ids: Indices of input sequence tokens in the vocabulary. attention_mask: Mask to avoid performing attention on padding token indices. idf: A bool indicating whether normalization using inverse document frequencies should be used. batch_size: A batch size used for model processing. num_workers: A number of workers to use for a dataloader. Return: An instance of ``torch.utils.data.DataLoader`` used for iterating over examples. )rnro)r r )rkrlrmrnrodatasetr r r!_get_dataloader+s rq tokenizerrcCs|j|j|j|jdS)aBuild a dictionary of model/tokenizer special tokens. Args: tokenizer: Initialized tokenizer from HuggingFace's `transformers package. Return: A dictionary containing: mask_token_id, pad_token_id, sep_token_id and cls_token_id.  mask_token_id pad_token_id sep_token_id cls_token_idrs)rrr r r!_get_special_tokens_mapDs  rxrurvrwcCs$||||B||B}|S)a(Generate a token mask for differentiating all special tokens in the input batch. There are 0s for special tokens and 1s otherwise. Args: input_ids: Indices of input sequence tokens in the vocabulary. pad_token_id: An id of ```` tokens that are used to make arrays of tokens the same size for batching purpose cls_token_id: An id of ```` token that represents the class of the input. (It might be ```` token for some models.) sep_token_id: An id of ```` token that separates two different sentences in the same input. (It might be ```` token for some models.) Return: Tensor mask of 0s and 1s that masks all special tokens in the ``input_ids`` tensor. )eq)rkrurvrw token_maskr r r!_get_token_maskWsr{modelrbatch temperaturespecial_tokens_mapcCsR|djd}g}t|d|d|d|d}t|D]P}|d} |d| dd|f<|| |dj} | dd|ddf} tj| |d d } |r_| |d dd|fd| j 9} | | d ~ ~ ~ qt j |dd } t d | |j |} |r||d |j } | jdd | jdd dS| jdd |jdd dS) a_Calculate a discrete probability distribution for a batch of examples. See `InfoLM`_ for details. Args: model: Initialized model from HuggingFace's `transformers package. batch: An input batch dictionary containing ``input_ids`` and ``attention_mask``. temperature: A temperature for calibrating language modelling. For more information, please reference `InfoLM`_ paper. max_length: A maximum length of input sequences. Sequences longer than `max_length` are to be trimmed. idf: An indication of whether normalization using inverse document frequencies should be used. special_tokens_map: A dictionary mapping tokenizer special tokens into the corresponding integer values. Return: A discrete probability distribution. rkr7rurvrwrtNrlr8rM input_ids_idfzbsv, bs -> bsv)shaper{rangeclonelogitsFsoftmax unsqueezetodeviceappendcpurIcateinsumrO)r|r}r~rmrseq_lenprob_distribution_batch_listrzmask_idxrklogits_distributionprob_distributionprob_distribution_batchmasked_input_ids_idfr r r!_get_batch_distributionps0  &r dataloaderverbosec CsH|j}g}t||D]}t||}|t|||||q tj|ddS)aCalculate a discrete probability distribution according to the methodology described in `InfoLM`_. Args: model: Initialized model from HuggingFace's `transformers package. dataloader: An instance of `torch.utils.data.DataLoader` used for iterating over examples. temperature: A temperature for calibrating language modelling. For more information, please reference `InfoLM`_ paper. max_length: A maximum length of input sequences. Sequences longer than `max_length` are to be trimmed. idf: An indication of whether normalization using inverse document frequencies should be used. special_tokens_map: A dictionary mapping tokenizer special tokens into the corresponding integer values. verbose: An indication of whether a progress bar to be displayed during the embeddings calculation. Return: A discrete probability distribution. rrM)rr rrrrIr) r|rr~rmrrrrr}r r r!_get_data_distributions  rpredstarget max_lengthcCsdt|ttfs t|}t|ttfst|}||d|ddd}||d|ddd}|j|j|j|jfS)a8Update the metric state by a tokenization of ``preds`` and ``target`` sentencens. Args: preds: An iterable of hypothesis corpus. target: An iterable of reference corpus. tokenizer: Initialized tokenizer from HuggingFace's `transformers package. max_length: A maximum length of input sequences. Sequences longer than `max_length` are to be trimmed. Return: Tokenizerd ``preds`` and ``target`` sentences represented with ``input_ids`` and ``attention_mask`` tensors. rTpt)paddingr truncationreturn_tensors)r9r(listrkrl)rrrrr preds_input target_inputr r r!_infolm_updatesrTpreds_dataloadertarget_dataloaderinformation_measure_clsc CsFt||||||}t||||||} ||jj}| |jj} ||| S)alCalculate selected information measure using the pre-trained language model. Args: model: Initialized model from HuggingFace's `transformers package. preds_dataloader: Loader iterating over tokenizer predicted sentences. target_dataloader: Loader iterating over tokenizer reference sentences. temperature: A temperature for calibrating language modelling. For more information, please reference `InfoLM`_ paper. idf: An indication of whether normalization using inverse document frequencies should be used. information_measure_cls: Information measure class containing all parameters necessary for calculating information measure values using ``preds_distribution`` and ``target_distribution``. special_tokens_map: A dictionary mapping tokenizer special tokens into the corresponding integer values. verbose: An indication of whether a progress bar to be displayed during the embeddings calculation. Return: A corpus-level InfoLM score. )rrpsorting_indices) r|rrr~rmrrrrDrEr r r!_infolm_computes#    rbert-base-uncased?r@Fmodel_name_or_pathr3r4r5r num_threadsreturn_sentence_level_scorec Cst||\}}t|||}| p|jj} t|}t|||| \}}}}t|||| | }t|||| | }t|||||||| }| rE||fS|S)uo Calculate `InfoLM`_ [1]. InfoML corresponds to distance/divergence between predicted and reference sentence discrete distribution using one of the following information measures: - `KL divergence`_ - `alpha divergence`_ - `beta divergence`_ - `AB divergence`_ - `Rényi divergence`_ - L1 distance - L2 distance - L-infinity distance - `Fisher-Rao distance`_ `InfoLM`_ is a family of untrained embedding-based metrics which addresses some famous flaws of standard string-based metrics thanks to the usage of pre-trained masked language models. This family of metrics is mainly designed for summarization and data-to-text tasks. If you want to use IDF scaling over the whole dataset, please use the class metric. The implementation of this metric is fully based HuggingFace `transformers`' package. Args: preds: An iterable of hypothesis corpus. target: An iterable of reference corpus. model_name_or_path: A name or a model path used to load `transformers` pretrained model. temperature: A temperature for calibrating language modelling. For more information, please reference `InfoLM`_ paper. information_measure: A name of information measure to be used. Please use one of: ['kl_divergence', 'alpha_divergence', 'beta_divergence', 'ab_divergence', 'renyi_divergence', 'l1_distance', 'l2_distance', 'l_infinity_distance', 'fisher_rao_distance'] idf: An indication of whether normalization using inverse document frequencies should be used. alpha: Alpha parameter of the divergence used for alpha, AB and Rényi divergence measures. beta: Beta parameter of the divergence used for beta and AB divergence measures. device: A device to be used for calculation. max_length: A maximum length of input sequences. Sequences longer than `max_length` are to be trimmed. batch_size: A batch size used for model processing. num_threads: A number of threads to use for a dataloader. verbose: An indication of whether a progress bar to be displayed during the embeddings calculation. return_sentence_level_score: An indication whether a sentence-level InfoLM score to be returned. Returns: A corpus-level InfoLM score. (Optionally) A list of sentence-level InfoLM scores if `return_sentence_level_score=True`. Example: >>> from torchmetrics.functional.text.infolm import infolm >>> preds = ['he read the book because he was interested in world history'] >>> target = ['he was interested in world history because he read the book'] >>> infolm(preds, target, model_name_or_path='google/bert_uncased_L-2_H-128_A-2', idf=False) tensor(-0.1784) References: [1] InfoLM: A New Metric to Evaluate Summarization & Data2Text Generation by Pierre Colombo, Chloé Clavel and Pablo Piantanida `InfoLM`_ ) rr2configrrxrrqrmean)rrrr~r3rmr4r5rrrnrrrrrr|rrpreds_input_idspreds_attention_masktarget_input_idstarget_attention_maskrr info_lm_scorer r r!r"s,W    )T) rrrTNNNNrrTF)4oscollections.abcrenumrtypingrrrrrIrtorch.nnr rtorch.utils.datar typing_extensionsr 4torchmetrics.functional.text.helper_embedding_metricr r rrtorchmetrics.utilities.enumsrtorchmetrics.utilities.importsr transformersrr__doctest_skip__rjrr2boolintrqdictr(rxr{r:rno_gradrtuplerrPathLikerrr r r r!s         c    9  ( *  1