o yin#@sddlZddlmZddlmZmZmZmZmZm Z ddl Z ddl m Z ddl m ZddlmZddlmZddlmZmZmZmZdd lmZdd lmZerYdd lmZmZndZZd gZed Z eGdddeZ!GdddZ"de de de#de$de$def ddZ%dedee&e$ffddZ'de de$de$de$de f d d!Z(d"ed#ee&e fd$e)de#d%ee&e$fde f d&d'Z*e +d"ed(ed$e)de#d%ee&e$fd)e#de fd*d+Z,d,e e&ee&fd-e e&ee&fded.e$dee e e e ff d/d0Z- 1dDd"ed2ed3ed$e)de#d4e"d%ee&e$fd)e#de fd5d6Z. 7 8 9 1     : 1 ;dEd,e e&ee&fd-e e&ee&fdee)d?ee)d@ee e&e j0fd.ee$de$dAe$d)e#dBe#de e ee e fffdCd Z1dS)FN)unique)DictListOptionalSequenceTupleUnion)Tensor) functional) DataLoader)Literal)TokenizedDataset_get_progress_bar_input_data_collator_load_tokenizer_and_model)EnumStr)_TRANSFORMERS_AVAILABLE)PreTrainedModelPreTrainedTokenizerBaseinfolm) kl_divergencealpha_divergencebeta_divergence ab_divergencerenyi_divergence l1_distance l2_distancel_infinity_distancefisher_rao_distancecsVeZdZdZdZdZdZdZdZdZ dZ d Z d Z e d ed ed ffdd ZZS)_IMEnumz8A helper Enum class for storing the information measure.rrrrrrrrrvaluereturnrcsFddtjD}t|}|dur||vr|Std|d|d)z Raises: ValueError: If required information measure is not among the supported options. cSsg|]}|qS)lower).0imr"r"W/home/ubuntu/.local/lib/python3.10/site-packages/torchmetrics/functional/text/infolm.py Jsz$_IMEnum.from_str..Nz-Invalid information measure. Expected one of z , but got .)r_member_names_superfrom_str ValueError)clsr _allowed_imenum_key __class__r"r&r+Cs  z_IMEnum.from_str)__name__ __module__ __qualname____doc__ KL_DIVERGENCEALPHA_DIVERGENCEBETA_DIVERGENCE AB_DIVERGENCERENYI_DIVERGENCE L1_DISTANCE L2_DISTANCEL_INFINITY_DISTANCEFISHER_RAO_DISTANCE classmethodstrrr+ __classcell__r"r"r0r&r5s$rc @s(eZdZdZ  d dedeedeeddfddZd ed edefd d Z e d ed edefddZ d ed edefddZ d ed edefddZ d ed edefddZd ed edefddZe d ed edefddZe d ed edefddZe d ed edefddZe d ed edefddZdS)!_InformationMeasureuA wrapper class used for the calculation the result of information measure between the discrete reference distributions of predicted and reference sentences. The class also handles input validation for `alpha` and `beta` parameters. Args: information_measure: A name of information measure to be used. Please use one of: ['kl_divergence', 'alpha_divergence', 'beta_divergence', 'ab_divergence', 'renyi_divergence', 'l1_distance', 'l2_distance', 'l_infinity_distance', 'fisher_rao_distance'] alpha: Alpha parameter of the divergence used for alpha, AB and Rényi divergence measures. beta: Beta parameter of the divergence used for beta and AB divergence measures. Raises: ValueError: If information measure is one from alpha, AB or Rényi divergence and parameter `alpha` is `None`. ValueError: If information measure is one from beta or divergence and parameter `beta` is `None`. ValueError: If information measure is alpha divergence and parameter `alpha` equals 0 or 1. ValueError: If information measure is beta divergence and parameter `beta` equals 0 or -1. ValueError: If information measure is AB divergence and parameter `alpha`, `beta` or `alpha + beta` equal 0. ValueError: If information measure is Rényi divergence and parameter `alpha` equals 1. Ninformation_measurealphabetar!cCsNt||_|jtjtjtjfvrt|tstd|d|jtj tjfvr4t|ts4td|d|jtjkrKt|trC|dvrKtd|d|jtj krbt|trZ|dvrbtd|d|jtjkrt dd ||fDs|d ||||fvrtd |d|jtjkrt|tr|d krtd |d|pd |_ |pd |_ dS)Nz0Parameter `alpha` is expected to be defined for r(z/Parameter `beta` is expected to be defined for )rzFParameter `alpha` is expected to be float differened from 0 and 1 for )rzFParameter `beta` is expected to be float differened from 0 and -1 for css|] }t|t VqdS)N) isinstancefloat)r$pr"r"r& sz/_InformationMeasure.__init__..rzRParameters `alpha`, `beta` and their sum are expected to be differened from 0 for rFz@Parameter `alpha` is expected to be float differened from 1 for ) rr+rCr7r9r:rHrIr,r8anyrDrE)selfrCrDrEr"r"r&__init__ps4      ( z_InformationMeasure.__init__preds_distributiontarget_distribtuioncCs"t|d|j}t|||S)N _calculate_)getattrrCtorch nan_to_num)rMrOrPinformation_measure_functionr"r"r&__call__sz_InformationMeasure.__call__target_distributioncCstj|t||ddS)aCalculate Kullback-Leibler divergence between discrete distributions of predicted and reference sentences. Args: preds_distribution: Discrete reference distribution of predicted sentences over the vocabulary. target_distribution: Discrete reference distribution of reference sentences over the vocabulary. Return: Kullback-Leibler divergence between discrete distributions of predicted and reference sentences. rGdim)rSsumlogrOrWr"r"r&_calculate_kl_divergencesz,_InformationMeasure._calculate_kl_divergencecCs>|j|jd}dtj||j|d|jdd|}|S)aCalculate alpha divergence between discrete distributions of predicted and reference sentences. Args: preds_distribution: Discrete reference distribution of predicted sentences over the vocabulary. target_distribution: Discrete reference distribution of reference sentences over the vocabulary. Return: Alpha divergence between discrete distributions of predicted and reference sentences. rFrGrX)rDrSrZ)rMrOrW _alpha_denomrr"r"r&_calculate_alpha_divergences  $z/_InformationMeasure._calculate_alpha_divergencecCsttj||j|jdd}||j|j|j}ttj||j|jdd}||j|j|j}ttj||j||jdd}||j|j}|||}|S)aCalculate AB divergence between discrete distributions of predicted and reference sentences. Args: preds_distribution: Discrete reference distribution of predicted sentences over the vocabulary. target_distribution: Discrete reference distribution of reference sentences over the vocabulary. Return: AB divergence between discrete distributions of predicted and reference sentences. rGrX)rSr[rZrErD)rMrOrWabcrr"r"r&_calculate_ab_divergences  $ z,_InformationMeasure._calculate_ab_divergencecCsd|_|||}|S)aCalculate beta divergence between discrete distributions of predicted and reference sentences. Args: preds_distribution: Discrete reference distribution of predicted sentences over the vocabulary. target_distribution: Discrete reference distribution of reference sentences over the vocabulary. Return: Beta divergence between discrete distributions of predicted and reference sentences. g?)rDrc)rMrOrWrr"r"r&_calculate_beta_divergences z._InformationMeasure._calculate_beta_divergencecCs6ttj||j|d|jdd|jd}|S)uCalculate Rényi divergence between discrete distributions of predicted and reference sentences. Args: preds_distribution: Discrete reference distribution of predicted sentences over the vocabulary. target_distribution: Discrete reference distribution of reference sentences over the vocabulary. Return: Rényi divergence between discrete distributions of predicted and reference sentences. rFrGrX)rSr[rZrD)rMrOrWrr"r"r&_calculate_renyi_divergences& z/_InformationMeasure._calculate_renyi_divergencecCtj||dddS)aCalculate L1 distance between discrete distributions of predicted and reference sentences. Args: preds_distribution: Discrete reference distribution of predicted sentences over the vocabulary. target_distribution: Discrete reference distribution of reference sentences over the vocabulary. Return: L1 distance between discrete distributions of predicted and reference sentences. rFrGrJrYrSnormr\r"r"r&_calculate_l1_distance z*_InformationMeasure._calculate_l1_distancecCrf)aCalculate L2 distance between discrete distributions of predicted and reference sentences. Args: preds_distribution: Discrete reference distribution of predicted sentences over the vocabulary. target_distribution: Discrete reference distribution of reference sentences over the vocabulary. Return: L2 distance between discrete distributions of predicted and reference sentences. rGrgrhr\r"r"r&_calculate_l2_distancerkz*_InformationMeasure._calculate_l2_distancecCstj||tdddS)aCalculate L-infinity distance between discrete distributions of predicted and reference sentences. Args: preds_distribution: Discrete reference distribution of predicted sentences over the vocabulary. target_distribution: Discrete reference distribution of reference sentences over the vocabulary. Return: L-infinity distance between discrete distributions of predicted and reference sentences. infrGrg)rSrirIr\r"r"r&_calculate_l_infinity_distance s z2_InformationMeasure._calculate_l_infinity_distancec Cs(dttt||dddS)aCalculate Fisher-Rao distance between discrete distributions of predicted and reference sentences. Args: preds_distribution: Discrete reference distribution of predicted sentences over the vocabulary. target_distribution: Discrete reference distribution of reference sentences over the vocabulary. Return: Fisher-Rao distance between discrete distributions of predicted and reference sentences. rlrGrrF)rSacosclampsqrtrZr\r"r"r&_calculate_fisher_rao_distances( z2_InformationMeasure._calculate_fisher_rao_distance)NN)r2r3r4r5$_ALLOWED_INFORMATION_MEASURE_LITERALrrIrNr rV staticmethodr]r_rcrdrerjrmrorsr"r"r"r&rBRs8  "rB input_idsattention_maskidf batch_size num_workersr!cCst|||}t|||d}|S)aPrepare dataloader. Args: input_ids: Indices of input sequence tokens in the vocabulary. attention_mask: Mask to avoid performing attention on padding token indices. idf: A batch size used for model processing. num_threads: A number of workers to use for a dataloader. Return: An instance of ``torch.utils.data.DataLoader`` used for iterating over examples. )ryrz)r r )rvrwrxryrzdataset dataloaderr"r"r&_get_dataloader,s r} tokenizercCs|j|j|j|jd}|S)aBuild a dictionary of model/tokenizer special tokens. Args: tokenizer: Initialized tokenizer from HuggingFace's `transformers package. Return: A dictionary containing: mask_token_id, pad_token_id, sep_token_id and cls_token_id.  mask_token_id pad_token_id sep_token_id cls_token_idr)r~special_tokens_mapsr"r"r&_get_special_tokens_mapCs  rrrrcCs$||||B||B}|S)a&Generate a token mask for differentiating all special tokens in the input batch. There are 0s for special tokens and 1s otherwise. Args: input_ids: Indices of input sequence tokens in the vocabulary. pad_token_id: An id of ```` tokens that are used to make arrays of tokens the same size for batching purpose cls_token_id: An id of ```` token that represents the class of the input. (It might be ```` token for some models.) sep_token_id: An id of ```` token that separates two different sentences in the same input. (It might be ```` token for some models.) Return: Tensor mask of 0s and 1s that masks all special tokens in the ``input_ids`` tensor. )eq)rvrrr token_maskr"r"r&_get_token_maskVsrmodelbatch temperaturespecial_tokens_mapcCsZ|djd}g}t|d|d|d|d}t|D]P}|d} |d| dd|f<|| |dj} | dd|ddf} tj| |d d } |r_| |d dd|fd| j 9} | | d ~ ~ ~ qt j |dd } t d | |j |} |r||d |j } | jdd | jdd d} | S| jdd |jdd d} | S) aCalculate a discrete probability distribution for a batch of examples according to the methodology described in `InfoLM`_. Args: model: Initialized model from HuggingFace's `transformers package. batch: An input batch dictionary containing ``input_ids`` and ``attention_mask``. temperature: A temperature for calibrating language modelling. For more information, please reference `InfoLM`_ paper. max_length: A maximum length of input sequences. Sequences longer than `max_length` are to be trimmed. idf: An indication of whether normalization using inverse document frequencies should be used. Return: A discrete probability distribution. rvrFrrrrNrwrGrX input_ids_idfzbsv, bs -> bsv)shaperrangeclonelogitsFsoftmax unsqueezetodeviceappendcpurScateinsumrZ)rrrrxrseq_lenprob_distribution_batch_listrmask_idxrvlogits_distributionprob_distributionprob_distribution_batchmasked_input_ids_idfr"r"r&_get_batch_distributionms4  &rr|verbosec CsH|j}g}t||D]}t||}|t|||||q tj|ddS)abCalculate a discrete probability distribution according to the methodology described in `InfoLM`_. Args: model: Initialized model from HuggingFace's `transformers package. dataloader: An instance of `torch.utils.data.DataLoader` used for iterating over examples. temperature: A temperature for calibrating language modelling. For more information, please reference `InfoLM`_ paper. max_length: A maximum length of input sequences. Sequences longer than `max_length` are to be trimmed. idf: An indication of whether normalization using inverse document frequencies should be used. verbose: An indication of whether a progress bar to be displayed during the embeddings calculation. Return: A discrete probability distribution. rrX)rrrrrrSr) rr|rrxrrrrrr"r"r&_get_data_distributions  rpredstarget max_lengthcCsdt|ttfs t|}t|ttfst|}||d|ddd}||d|ddd}|j|j|j|jfS)a7Update the metric state by a tokenization of ``preds`` and ``target`` sentencens. Args: preds: An iterable of hypothesis corpus. target: An iterable of reference corpus. tokenizer: Initialized tokenizer from HuggingFace's `transformers package. max_length: A maximum length of input sequences. Sequences longer than `max_length` are to be trimmed. Return: Tokenizerd ``preds`` and ``target`` sentences represented with ``input_ids`` and ``attention_mask`` tensors. rTpt)paddingr truncationreturn_tensors)rHr@listrvrw)rrr~r preds_input target_inputr"r"r&_infolm_updatesrTpreds_dataloadertarget_dataloaderinformation_measure_clsc CsJt||||||}t||||||} ||jj}| |jj} ||| } | S)akCalculate selected information measure using the pre-trained language model. Args: model: Initialized model from HuggingFace's `transformers package. preds_dataloader: Loader iterating over tokenizer predicted sentences. target_datalaoder: Loader iterating over tokenizer reference sentences. temperature: A temperature for calibrating language modelling. For more information, please reference `InfoLM`_ paper. idf: An indication of whether normalization using inverse document frequencies should be used. information_measure_cls: Information measure class containing all parameters necessary for calculating information measure values using ``preds_distribution`` and ``target_distribution``. special_tokens_map: A dictionary mapping tokenizer special tokens into the corresponding integer values. verbose: An indication of whether a progress bar to be displayed during the embeddings calculation. Return: A corpus-level InfoLM score. )rr{sorting_indices) rrrrrxrrrrOrW infolm_scorer"r"r&_infolm_computes"    rbert-base-uncased?r@Fmodel_name_or_pathrCrDrEr num_threadsreturn_sentence_level_scorec Cst||\}}t|||}| p|jj} t|}t|||| \}}}}t|||| | }t|||| | }t|||||||| }| rE||fS|S)uj Calculate `InfoLM`_ [1] - i.e. calculate a distance/divergence between predicted and reference sentence discrete distribution using one of the following information measures: - `KL divergence`_ - `alpha divergence`_ - `beta divergence`_ - `AB divergence`_ - `Rényi divergence`_ - L1 distance - L2 distance - L-infinity distance - `Fisher-Rao distance`_ `InfoLM`_ is a family of untrained embedding-based metrics which addresses some famous flaws of standard string-based metrics thanks to the usage of pre-trained masked language models. This family of metrics is mainly designed for summarization and data-to-text tasks. If you want to use IDF scaling over the whole dataset, please use the class metric. The implementation of this metric is fully based HuggingFace `transformers`' package. Args: preds: An iterable of hypothesis corpus. target: An iterable of reference corpus. model_name_or_path: A name or a model path used to load `transformers` pretrained model. temperature: A temperature for calibrating language modelling. For more information, please reference `InfoLM`_ paper. information_measure: A name of information measure to be used. Please use one of: ['kl_divergence', 'alpha_divergence', 'beta_divergence', 'ab_divergence', 'renyi_divergence', 'l1_distance', 'l2_distance', 'l_infinity_distance', 'fisher_rao_distance'] idf: An indication of whether normalization using inverse document frequencies should be used. alpha: Alpha parameter of the divergence used for alpha, AB and Rényi divergence measures. beta: Beta parameter of the divergence used for beta and AB divergence measures. device: A device to be used for calculation. max_length: A maximum length of input sequences. Sequences longer than `max_length` are to be trimmed. batch_size: A batch size used for model processing. num_threads: A number of threads to use for a dataloader. verbose: An indication of whether a progress bar to be displayed during the embeddings calculation. return_sentence_level_score: An indication whether a sentence-level InfoLM score to be returned. Returns: A corpus-level InfoLM score. (Optionally) A list of sentence-level InfoLM scores if `return_sentence_level_score=True`. Example: >>> from torchmetrics.functional.text.infolm import infolm >>> preds = ['he read the book because he was interested in world history'] >>> target = ['he was interested in world history because he read the book'] >>> infolm(preds, target, model_name_or_path='google/bert_uncased_L-2_H-128_A-2', idf=False) tensor(-0.1784) References: [1] InfoLM: A New Metric to Evaluate Summarization & Data2Text Generation by Pierre Colombo, Chloé Clavel and Pablo Piantanida `InfoLM`_ ) rrBconfigrrrr}rmean)rrrrrCrxrDrErrryrrrr~rrrpreds_input_idspreds_attention_masktarget_input_idstarget_attention_maskrr info_lm_scorer"r"r&rs,U    )T) rrrTNNNNrrTF)2osenumrtypingrrrrrrrSr torch.nnr rtorch.utils.datar typing_extensionsr 4torchmetrics.functional.text.helper_embedding_metricr rrrtorchmetrics.utilities.enumsrtorchmetrics.utilities.importsr transformersrr__doctest_skip__rtrrBboolintr}r@rrrIrno_gradrrrPathLikerrr"r"r"r&s         [    5  $ )  1