o .wi*V0@sddlZddlZddlZddlmZmZddlmZddlm Z m Z m Z m Z m Z ddlZddlmZddlmZddlmZddlmZmZmZmZmZmZmZdd lmZdd lmZm Z dd l!m"Z"m#Z#ed edfd dZ$dZ%e#rddl&m'Z'm(Z(dJddZ)ere e)sdgZ*ndgZ*      dKdede+dede e e,ej-fde e+de.de.de.de e ee/e,efgefd e0eeffddZ1d ed!e,d"ed efd#d$Z2d%ed&ed'ed(ed e0eeeff d)d*Z3dLd+e e,de e+de.d e,fd,d-Z4d.e,d efd/d0Z5d1e,d efd2d3Z6 4   dMd5e,d+e e,d.e e,d1e e,d e ef d6d7Z7  dNd8ed9ed:ed;ede e+de.d e0eeeffd ?  4    dOd@e e,ee,e/e,effdAe e,ee,e/e,effd+e e,de e+de.de edBe de e ee/e,efgefde.de.de e e,ej-fdCe+dDe+dEe+dFe.d5e,dGe.d.e e,d1e e,dHe.d e/e,e ee9e:e,fff*dIdZ;dS)PN)IteratorSequence)contextmanager)AnyCallableListOptionalUnion)Tensor)Module) DataLoader) TextDatasetTokenizedDataset_check_shape_of_model_output_get_progress_bar_input_data_collator_output_data_collator*_process_attention_mask_for_special_tokens)rank_zero_warn)_SKIP_SLOW_DOCTEST_try_proceed_with_timeout)_TQDM_AVAILABLE_TRANSFORMERS_GREATER_EQUAL_4_4returnc csDtd}|}z|tjdVW||dS||w)z]Ignore irrelevant fine-tuning warning from transformers when loading the model for BertScore.ztransformers.modeling_utilsN)logging getLoggergetEffectiveLevelsetLevelERROR)loggeroriginal_levelr!^/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/torchmetrics/functional/text/bert.py_ignore_log_warning(s  r#z roberta-large) AutoModel AutoTokenizercCs@tttttWddS1swYdS)zDownload intensive operations.N)r#r%from_pretrained_DEFAULT_MODELr$r!r!r!r"_download_model_for_bert_score:s  "r( bert_scoreF dataloader target_lenmodeldevice num_layers all_layersidfverboseuser_forward_fnc  Csg} g} t||D]} tXt| |} |sB|s0|| d| ddd} | j|dur,|nd} n ||| } t| | d| d} n|rHtd|| d| ddd} tjd d | jDdd } Wdn1sjwY| | j dd d} t | | d|\} } t | }t d | |} | | |r| d |n|| j}||jddd}| |q t| }t| }||fS)a<Calculate sentence embeddings and the inverse-document-frequency scaling factor. Args: dataloader: dataloader instance. target_len: A length of the longest sequence in the data. Used for padding the model output. model: BERT model. device: A device to be used for calculation. num_layers: The layer of representation to use. all_layers: An indication whether representation from all model layers should be used for BERTScore. idf: An Indication whether normalization using inverse document frequencies should be used. verbose: An indication of whether a progress bar to be displayed during the embeddings' calculation. user_forward_fn: A user's own forward function used in a combination with ``user_model``. This function must take ``user_model`` and a python dictionary of containing ``"input_ids"`` and ``"attention_mask"`` represented by :class:`~torch.Tensor` as an input and return the model's output represented by the single :class:`~torch.Tensor`. Return: A tuple of :class:`~torch.Tensor`s containing the model's embeddings and the normalized tokens IDF. When ``idf = False``, tokens IDF is not calculated, and a matrix of mean weights is returned instead. For a single sentence, ``mean_weight = 1/seq_len``, where ``seq_len`` is a sum over the corresponding ``attention_mask``. Raises: ValueError: If ``all_layers = True`` and a model, which is not from the ``transformers`` package, is used. input_idsattention_maskT)output_hidden_statesNzQThe option `all_layers=True` can be used only with default `transformers` models.cSsg|]}|dqS)r7) unsqueeze).0or!r!r" sz1_get_embeddings_and_idf_scale..dimzblsd, bs -> blsd input_ids_idf)keepdim)rtorchno_gradr hidden_statesrr8 ValueErrorcatnormrreinsumappendcputypedtypesum)r*r+r,r-r.r/r0r1r2embeddings_listidf_scale_listbatchoutr4processed_attention_maskr> embeddings idf_scaler!r!r"_get_embeddings_and_idf_scaleFs@'      rScos_simmetricrRcCsB|dkrdnd}|j|dj}td||d}|ddS) zOCalculate precision or recall, transpose it and scale it with idf_scale factor. precisionr<zbls, bs -> blsr6rr7)maxvaluesr@rFrK transposesqueeze)rTrUrRr=resr!r!r"_get_scaled_precision_or_recallsr^preds_embeddingstarget_embeddingspreds_idf_scaletarget_idf_scalecCsVtd||}t|d|}t|d|}d||||}|t|d}|||fS)aCalculate precision, recall and F1 score over candidate and reference sentences. Args: preds_embeddings: Embeddings of candidate sentences. target_embeddings: Embeddings of reference sentences. preds_idf_scale: An IDF scale factor for candidate sentences. target_idf_scale: An IDF scale factor for reference sentences. Return: Tensors containing precision, recall and F1 score, respectively. zblpd, blrd -> blprrVrecallrX)r@rFr^ masked_fillisnan)r_r`rarbrTrVrcf1_scorer!r!r"_get_precision_recall_f1s    rhmodel_name_or_pathcCs|d||r dSdS)z,Compute `BERT_score`_ (copied and adjusted)._L_idfz_no-idfr!)rir.r0r!r!r" _get_hashsrl baseline_pathcCs^t|}t|}ddt|D}Wdn1swYt|ddddfS)zqRead baseline from csv file from the local file. This method implemented to avoid `pandas` dependency. cSs&g|]\}}|dkrdd|DqS)rcSg|]}t|qSr!floatr9itemr!r!r"r;z8_read_csv_from_local_file...r!r9idxrowr!r!r"r;s&z-_read_csv_from_local_file..Nr7)opencsvreader enumerater@tensor)rmfnamecsv_file baseline_listr!r!r"_read_csv_from_local_files  r baseline_urlcCs\tj|}ddt|D}t|ddddfWdS1s'wYdS)ziRead baseline from csv file from URL. This method is implemented to avoid `pandas` dependency. cSs6g|]\}}|dkrdd|ddDqS)rcSrnr!rorqr!r!r"r;rsz1_read_csv_from_url...zutf-8,)stripdecodesplitrtr!r!r"r;s z&_read_csv_from_url..Nr7)urllibrequesturlopenrzr@r{)r http_requestr~r!r!r"_read_csv_from_urls $renlangcCsZ|rt|}|S|rt|}|S|r'|r'd}|d|d|d}t|}|StddS)ztdi|d| i}tdi|| |jd}ntdt|| | d}t|| | d}t||j || ||| || \}}t||j || ||| || \} }!| |j!j"} ||j!j"}|!|j!j"}!||j!j"}t#| ||!|\}"}#}$|durt$|"|#|$|||\}"}#}$|"|#|$d}|r|dt||| i|S)a`Bert_score Evaluating Text Generation`_ for text similirity matching. This metric leverages the pre-trained contextual embeddings from BERT and matches words in candidate and reference sentences by cosine similarity. It has been shown to correlate with human judgment on sentence-level and system-level evaluation. Moreover, BERTScore computes precision, recall, and F1 measure, which can be useful for evaluating different language generation tasks. This implementation follows the original implementation from `BERT_score`_. Args: preds: Either an iterable of predicted sentences or a ``Dict[input_ids, attention_mask]``. target: Either an iterable of target sentences or a ``Dict[input_ids, attention_mask]``. model_name_or_path: A name or a model path used to load ``transformers`` pretrained model. num_layers: A layer of representation to use. all_layers: An indication of whether the representation from all model's layers should be used. If ``all_layers = True``, the argument ``num_layers`` is ignored. model: A user's own model. user_tokenizer: A user's own tokenizer used with the own model. This must be an instance with the ``__call__`` method. This method must take an iterable of sentences (``List[str]``) and must return a python dictionary containing ``"input_ids"`` and ``"attention_mask"`` represented by :class:`~torch.Tensor`. It is up to the user's model of whether ``"input_ids"`` is a :class:`~torch.Tensor` of input ids or embedding vectors. his tokenizer must prepend an equivalent of ``[CLS]`` token and append an equivalent of ``[SEP]`` token as `transformers` tokenizer does. user_forward_fn: A user's own forward function used in a combination with ``user_model``. This function must take ``user_model`` and a python dictionary of containing ``"input_ids"`` and ``"attention_mask"`` represented by :class:`~torch.Tensor` as an input and return the model's output represented by the single :class:`~torch.Tensor`. verbose: An indication of whether a progress bar to be displayed during the embeddings' calculation. idf: An indication of whether normalization using inverse document frequencies should be used. device: A device to be used for calculation. max_length: A maximum length of input sequences. Sequences longer than ``max_length`` are to be trimmed. batch_size: A batch size used for model processing. num_threads: A number of threads to use for a dataloader. return_hash: An indication of whether the correspodning ``hash_code`` should be returned. lang: A language of input sentences. It is used when the scores are rescaled with a baseline. rescale_with_baseline: An indication of whether bertscore should be rescaled with a pre-computed baseline. When a pretrained model from ``transformers`` model is used, the corresponding baseline is downloaded from the original ``bert-score`` package from `BERT_score`_ if available. In other cases, please specify a path to the baseline csv/tsv file, which must follow the formatting of the files from `BERT_score`_ baseline_path: A path to the user's own local csv/tsv file with the baseline scale. baseline_url: A url path to the user's own csv/tsv file with the baseline scale. truncation: An indication of whether the input sequences should be truncated to the maximum length. Returns: Python dictionary containing the keys ``precision``, ``recall`` and ``f1`` with corresponding values. Raises: ValueError: If ``len(preds) != len(target)``. ModuleNotFoundError: If `tqdm` package is required and not installed. ModuleNotFoundError: If ``transformers`` package is required and not installed. ValueError: If ``num_layer`` is larger than the number of the model layers. ValueError: If invalid input is provided. Example: >>> from pprint import pprint >>> from torchmetrics.functional.text.bert import bert_score >>> preds = ["hello there", "general kenobi"] >>> target = ["hello there", "master kenobi"] >>> pprint(bert_score(preds, target)) {'f1': tensor([1.0000, 0.9961]), 'precision': tensor([1.0000, 0.9961]), 'recall': tensor([1.0000, 0.9961])} zLExpected number of predicted and reference sententes to be the same, but gotz and z1Expected argument `idf` to be a boolean, but got .zcAn argument `verbose = True` requires `tqdm` package be installed. Install with `pip install tqdm`.Nz`bert_score` metric with default models requires `transformers` package be installed. Either install with `pip install transformers>=4.4` or `pip install torchmetrics[text]`.zThe argument `model_name_or_path` was not specified while it is required when default `transformers` model are used.It is, therefore, used the default recommended model - num_hidden_layersz num_layers=z is forbidden for z. Please use num_layers <= zhModel config does not have `num_hidden_layers` as an integer attribute. Unable to validate `num_layers`.zXIt was not possible to retrieve the parameter `num_layers` from the model specification.css&|]}t|tot|dkVqdSrN) isinstancelistlenr9textr!r!r" s$zbert_score..css4|]}t|tot|dkot|dtVqdSr)rrrstrrr!r!r"rs$ css(|]}t|tot|dtVqdS)r3N)rdictr rr!r!r"rs z%Predictions and references are empty.rd)rVrcf1hash)r0r)r0 tokens_idfrr0)r0rzInvalid input provided.)r num_workersr!)%rrCrrrrboolrModuleNotFoundErrorrrr'r#r%r&r$evaltohasattrconfigrintAttributeErrorallupdaterlrr rrr rSrdatasetsorting_indicesrhr)%rrrir.r/r,rr2r1r0r-rrrrrrrmrr tokenizer_are_empty_lists_are_valid_lists_are_valid_tensors output_dictrtarget_dataset preds_dataset target_loader preds_loaderr`rbr_rarVrcrgr!r!r"r)s^           )rN)NNFFFN)NNF)rNNN)NF)NNFNNNFFNrrrFrFNNF)s>     $      Q   $