o .wi*@sddlZddlmZddlmZmZmZmZmZddl Z ddl m Z ddl m Z ddl mZmZmZmZmZmZddlmZddlmZdd lmZmZdd lmZmZesYd gZes_d d gZGd d d eZdS)N)Sequence)AnyClassVarListOptionalUnion)Tensor)_load_tokenizer_and_model)$_ALLOWED_INFORMATION_MEASURE_LITERAL_get_dataloader_get_special_tokens_map_infolm_compute_infolm_update_InformationMeasure)Metric) dim_zero_cat)_MATPLOTLIB_AVAILABLE_TRANSFORMERS_GREATER_EQUAL_4_4)_AX_TYPE_PLOT_OUT_TYPE InfoLM.plotInfoLMcseZdZUdZdZeeed<eeed<eeed<eeed<dddddddddd Ze ed <     d,de e e j fdedededeedeedee e ejfdeedededededee efdd ffdd Zedefd d!Zd"e e ee fd#e e ee fdd fd$d%Zde eeeefffd&d'Z d-d(ee eeefd)eedefd*d+Z Z!S).ru Calculate `InfoLM`_. InfoLM measures a distance/divergence between predicted and reference sentence discrete distribution using one of the following information measures: - `KL divergence`_ - `alpha divergence`_ - `beta divergence`_ - `AB divergence`_ - `Rényi divergence`_ - L1 distance - L2 distance - L-infinity distance - `Fisher-Rao distance`_ `InfoLM`_ is a family of untrained embedding-based metrics which addresses some famous flaws of standard string-based metrics thanks to the usage of pre-trained masked language models. This family of metrics is mainly designed for summarization and data-to-text tasks. The implementation of this metric is fully based HuggingFace ``transformers``' package. As input to ``forward`` and ``update`` the metric accepts the following input: - ``preds`` (:class:`~Sequence`): An iterable of hypothesis corpus - ``target`` (:class:`~Sequence`): An iterable of reference corpus As output of ``forward`` and ``compute`` the metric returns the following output: - ``infolm`` (:class:`~torch.Tensor`): If `return_sentence_level_score=True` return a tuple with a tensor with the corpus-level InfoLM score and a list of sentence-level InfoLM scores, else return a corpus-level InfoLM score Args: model_name_or_path: A name or a model path used to load ``transformers`` pretrained model. By default the `"bert-base-uncased"` model is used. temperature: A temperature for calibrating language modelling. For more information, please reference `InfoLM`_ paper. information_measure: A name of information measure to be used. Please use one of: ['kl_divergence', 'alpha_divergence', 'beta_divergence', 'ab_divergence', 'renyi_divergence', 'l1_distance', 'l2_distance', 'l_infinity_distance', 'fisher_rao_distance'] idf: An indication of whether normalization using inverse document frequencies should be used. alpha: Alpha parameter of the divergence used for alpha, AB and Rényi divergence measures. beta: Beta parameter of the divergence used for beta and AB divergence measures. device: A device to be used for calculation. max_length: A maximum length of input sequences. Sequences longer than ``max_length`` are to be trimmed. batch_size: A batch size used for model processing. num_threads: A number of threads to use for a dataloader. verbose: An indication of whether a progress bar to be displayed during the embeddings calculation. return_sentence_level_score: An indication whether a sentence-level InfoLM score to be returned. Example: >>> from torchmetrics.text.infolm import InfoLM >>> preds = ['he read the book because he was interested in world history'] >>> target = ['he was interested in world history because he read the book'] >>> infolm = InfoLM('google/bert_uncased_L-2_H-128_A-2', idf=False) >>> infolm(preds, target) tensor(-0.1784) Fpreds_input_idspreds_attention_masktarget_input_idstarget_attention_maskT) kl_divergencealpha_divergencebeta_divergence ab_divergencerenyi_divergence l1_distance l2_distancel_infinity_distancefisher_rao_distance%_information_measure_higher_is_betterbert-base-uncased?rN@rmodel_name_or_path temperatureinformation_measureidfalphabetadevice max_length batch_size num_threadsverbosereturn_sentence_level_scorekwargsreturnc  stjdi| ||_||_||_||_||_||_t |p d|_ | |_ | |_ | |_ | |_t||\|_|_t||||_|pE|jjj|_t|j|_|jdgdd|jdgdd|jdgdd|jdgdddS) Ncpurcat)dist_reduce_fxrrr)super__init__r)r*r+r,r-r.torchr/_devicer1r2r3r4r tokenizermodelrinformation_measure_clsconfigr0r special_tokens_map add_state)selfr)r*r+r,r-r.r/r0r1r2r3r4r5 __class__r:U/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/torchmetrics/text/infolm.pyr<s( zInfoLM.__init__cCs |j|jS)zReturns a bool indicating whether a higher value of the information measure is better. Done this way as depends on if the information measure is positive or negative. )r%r+)rEr:r:rHhigher_is_betters zInfoLM.higher_is_betterpredstargetcCsNt|||j|j\}}}}|j||j||j||j|dS)z*Update state with predictions and targets.N)rr?r0rappendrrr)rErJrKrrrrr:r:rHupdates    z InfoLM.updatec Cstt|jt|j|j|j|jd}tt|jt|j|j|j|jd}t |j |||j |j|j |j |j}|jr>||fS|S)zLCalculate selected information measure using the pre-trained language model.) input_idsattention_maskr,r1 num_workers)r rrrr,r1r2rrr r@r*rArCr3r4mean)rEpreds_dataloadertarget_dataloader info_lm_scorer:r:rHcomputes6 zInfoLM.computevalaxcCs |||S)a1Plot a single or multiple values from the metric. Args: val: Either a single result from calling `metric.forward` or `metric.compute` or a list of these results. If no value is provided, will automatically call `metric.compute` and plot that result. ax: An matplotlib axis object. If provided will add plot to that axis Returns: Figure and Axes object Raises: ModuleNotFoundError: If `matplotlib` is not installed .. plot:: :scale: 75 >>> # Example plotting a single value >>> from torchmetrics.text.infolm import InfoLM >>> metric = InfoLM('google/bert_uncased_L-2_H-128_A-2', idf=False) >>> preds = ['he read the book because he was interested in world history'] >>> target = ['he was interested in world history because he read the book'] >>> metric.update(preds, target) >>> fig_, ax_ = metric.plot() .. plot:: :scale: 75 >>> # Example plotting multiple values >>> from torchmetrics.text.infolm import InfoLM >>> metric = InfoLM('google/bert_uncased_L-2_H-128_A-2', idf=False) >>> preds = ["this is the prediction", "there is an other sample"] >>> target = ["this is the reference", "there is another one"] >>> values = [ ] >>> for _ in range(10): ... values.append(metric(preds, target)) >>> fig_, ax_ = metric.plot(values) )_plot)rErVrWr:r:rHplots *r) r&r'rTNNNNr(rTF)NN)"__name__ __module__ __qualname____doc__is_differentiablerr__annotations__r%rrstrosPathLikefloatr boolrr=r/intdictrr<propertyrIrrMtuplerUrrrY __classcell__r:r:rFrHr*s G           '. ") racollections.abcrtypingrrrrrr=r4torchmetrics.functional.text.helper_embedding_metricr #torchmetrics.functional.text.infolmr r r r rrtorchmetrics.metricrtorchmetrics.utilities.datartorchmetrics.utilities.importsrrtorchmetrics.utilities.plotrr__doctest_skip__rr:r:r:rHs