o yiAO-@sJddlZddlZddlmZmZmZmZmZmZm Z ddl m Z ddl Z ddl m Z ddlmZddlmZddlmZmZmZmZmZmZmZddlmZmZdd lmZmZd Z ersdd l!m"Z"m#Z#dFd dZ$erree$srdgZ%ndgZ%      dGdede&dedee e'e j(fdee&de)de)de)deeee'e fge fd ee e ffddZ*de de'de d e fdd Z+d!e d"e d#e d$e d ee e e ff d%d&Z,dHd'ee'dee&de)d e'fd(d)Z-d*e'd e fd+d,Z.d-e'd e fd.d/Z/ 0   dId1e'd'ee'd*ee'd-ee'd ee f d2d3Z0  dJd4e d5e d6e d7e dee&de)d ee e e ffd8d9Z1          : ; <  0   dKd=e ee'ee'e ffd>e ee'ee'e ffd'ee'dee&de)deed?edeeee'e fge fde)de)dee e'e j(fd@e&dAe&dBe&dCe)d1e'dDe)d*ee'd-ee'd ee'e ee2e'fff(dEdZ3dS)LN)AnyCallableDictListOptionalTupleUnion)warn)Tensor)Module) DataLoader) TextDatasetTokenizedDataset_check_shape_of_model_output_get_progress_bar_input_data_collator_output_data_collator*_process_attention_mask_for_special_tokens)_SKIP_SLOW_DOCTEST_try_proceed_with_timeout)_TQDM_AVAILABLE_TRANSFORMERS_AVAILABLEz roberta-large) AutoModel AutoTokenizerreturncCsttttdS)zDownload intensive operations.N)rfrom_pretrained_DEFAULT_MODELrrrU/home/ubuntu/.local/lib/python3.10/site-packages/torchmetrics/functional/text/bert.py_download_model*s r bert_scoreF dataloader target_lenmodeldevice num_layers all_layersidfverboseuser_forward_fnc  Csg} g} t||D]} tXt| |} |sB|s0|| d| ddd} | j|dur,|nd} n ||| } t| | d| d} n|rHtd|| d| ddd} tjd d | jDdd } Wdn1sjwY| | j dd d} t | | d|\} } t | }t d | |} | | |r| d |n|| j}||jddd}| |q t| }t| }||fS)a:Calculate sentence embeddings and the inverse-document-frequency scaling factor. Args: dataloader: dataloader instance. target_len: A length of the longest sequence in the data. Used for padding the model output. model: BERT model. device: A device to be used for calculation. num_layers: The layer of representation to use. all_layers: An indication whether representation from all model layers should be used for BERTScore. idf: An Indication whether normalization using inverse document frequencies should be used. verbose: An indication of whether a progress bar to be displayed during the embeddings' calculation. user_forward_fn: A user's own forward function used in a combination with ``user_model``. This function must take ``user_model`` and a python dictionary of containing ``"input_ids"`` and ``"attention_mask"`` represented by :class:`~torch.Tensor` as an input and return the model's output represented by the single :class:`~torch.Tensor`. Return: A tuple of :class:`~torch.Tensor`s containing the model's embeddings and the normalized tokens IDF. When ``idf = False``, tokens IDF is not calculated, and a matrix of mean weights is returned instead. For a single sentence, ``mean_weight = 1/seq_len``, where ``seq_len`` is a sum over the corresponding ``attention_mask``. Raises: ValueError: If ``all_layers = True`` and a model, which is not from the ``transformers`` package, is used. input_idsattention_maskT)output_hidden_statesNzQThe option `all_layers=True` can be used only with default `transformers` models.cSsg|]}|dqS)r.) unsqueeze).0orrr nsz1_get_embeddings_and_idf_scale..dimzblsd, bs -> blsd input_ids_idf)keepdim)rtorchno_gradr hidden_statesrr/ ValueErrorcatnormrreinsumappendcputypedtypesum)r!r"r#r$r%r&r'r(r)embeddings_listidf_scale_listbatchoutr+processed_attention_maskr5 embeddings idf_scalerrr_get_embeddings_and_idf_scale5s@%      rJcos_simmetricrIcCsF|dkrdnd}|j|dj}td||d}|dd}|S) zeHelper function that calculates precision or recall, transpose it and scale it with idf_scale factor. precisionr3zbls, bs -> blsr-rr.)maxvaluesr7r=rB transposesqueeze)rKrLrIr4resrrr_get_scaled_precision_or_recalls rUpreds_embeddingstarget_embeddingspreds_idf_scaletarget_idf_scalecCsVtd||}t|d|}t|d|}d||||}|t|d}|||fS)aCalculate precision, recall and F1 score over candidate and reference sentences. Args: preds_embeddings: Embeddings of candidate sentences. target_embeddings: Embeddings of reference sentences. preds_idf_scale: An IDF scale factor for candidate sentences. target_idf_scale: An IDF scale factor for reference sentences. Return: Tensors containing precision, recall and F1 score, respectively. zblpd, blrd -> blprrMrecallrO)r7r=rU masked_fillisnan)rVrWrXrYrKrMrZf1_scorerrr_get_precision_recall_f1s    r_model_name_or_pathcCs|d||r dnd}|S)z+Compute `BERT_score`_ (copied and adjusted)_L_idfz_no-idfr)r`r%r'msgrrr _get_hashsrd baseline_pathcCsbt|}t|}ddt|D}Wdn1swYt|ddddf}|S)zHelper function which reads baseline the csv file from the local file. This method implemented to avoid `pandas` dependency. cSs&g|]\}}|dkrdd|DqS)rcSg|]}t|qSrfloatr0itemrrrr2z8_read_csv_from_local_file...rr0idxrowrrrr2s&z-_read_csv_from_local_file..Nr.)opencsvreader enumerater7tensor)refnamecsv_file baseline_listbaselinerrr_read_csv_from_local_files  rx baseline_urlcCs^tj|}ddt|D}t|ddddf}Wd|S1s(wY|S)z~Helper function which reads the baseline csv file from URL. This method is implemented to avoid `pandas` dependency. cSs6g|]\}}|dkrdd|ddDqS)rcSrfrrgrirrrr2rkz1_read_csv_from_url...zutf-8,)stripdecodesplitrlrrrr2s z&_read_csv_from_url..Nr.)urllibrequesturlopenrrr7rs)ry http_requestrvrwrrr_read_csv_from_urls renlangcCs^|rt|}|S|rt|}|S|r'|r'd}|d|d|d}t|}|Sd}td|S)z>> from torchmetrics.functional.text.bert import bert_score >>> preds = ["hello there", "general kenobi"] >>> target = ["hello there", "master kenobi"] >>> score = bert_score(preds, target) >>> from pprint import pprint >>> rounded_score = {k: [round(v, 3) for v in vv] for k, vv in score.items()} >>> pprint(rounded_score) {'f1': [1.0, 0.996], 'precision': [1.0, 0.996], 'recall': [1.0, 0.996]} z=Number of predicted and reference sententes must be the same!zcAn argument `verbose = True` requires `tqdm` package be installed. Install with `pip install tqdm`.Nz`bert_score` metric with default models requires `transformers` package be installed. Either install with `pip install transformers>=4.0` or `pip install torchmetrics[text]`.zThe argument `model_name_or_path` was not specified while it is required when default `transformers` model are used.It is, therefore, used the default recommended model - .z num_layers=z is forbidden for z. Please use num_layers <= zXIt was not possible to retrieve the parameter `num_layers` from the model specification.css&|]}t|tot|dkVqdSrN) isinstancelistlenr0textrrr ss$zbert_score..css4|]}t|tot|dkot|dtVqdSr)rrrstrrrrrrts$ css(|]}t|tot|dtVqdS)r*N)rdictr rrrrrws z%Predictions and references are empty.r[)rMrZf1hash)r')r' tokens_idfr'zInvalid input provided.)r num_workersr)rr:rModuleNotFoundErrorrr rrrrevaltoconfignum_hidden_layersAttributeErrorallupdaterdrr rrr rJrr_rtolist)$rrr`r%r&r#rr)r(r'r$rrrrrrrery tokenizer_are_empty_lists_are_valid_lists_are_valid_tensors output_dictrwtarget_dataset preds_dataset target_loader preds_loaderrWrYrVrXrMrZr^rrrr s]      )rN)NNFFFN)NNF)rNNN)NF)NNFNNNFFNrrrFrFNN)4rpr~typingrrrrrrrwarningsr r7r torch.nnr torch.utils.datar 4torchmetrics.functional.text.helper_embedding_metricr rrrrrrtorchmetrics.utilities.checksrrtorchmetrics.utilities.importsrrr transformersrrr__doctest_skip__intrr$boolrJrUr_rdrxrrrrhr rrrrs. $    $     O   $