o
5ti^% @ s d dl mZmZmZmZmZmZmZmZ d dl m
Z
zdZ W n e y) dZ Y nw d dl
mZ ddlmZ ddlmZmZ dd lmZ d
d lmZmZmZ d
d lmZ erbd d
l m Z m Z m Z m Z ee!Z"dZ#G dd deZ$dS ) )
TYPE_CHECKINGAnyCallableDictListOptionalTupleUnion)DatasetTF)Literal )EvaluationModule)add_end_docstringsadd_start_docstrings)
get_logger )"EVALUATOR_COMPUTE_RETURN_DOCSTRING EVALUTOR_COMPUTE_START_DOCSTRING Evaluator)
DatasetColumn)PipelinePreTrainedModelPreTrainedTokenizerTFPreTrainedModela0
Examples:
```python
>>> from evaluate import evaluator
>>> from datasets import load_dataset
>>> task_evaluator = evaluator("question-answering")
>>> data = load_dataset("squad", split="validation[:2]")
>>> results = task_evaluator.compute(
>>> model_or_pipeline="sshleifer/tiny-distilbert-base-cased-distilled-squad",
>>> data=data,
>>> metric="squad",
>>> )
```
Datasets where the answer may be missing in the context are supported, for example SQuAD v2 dataset. In this case, it is safer to pass `squad_v2_format=True` to
the compute() call.
```python
>>> from evaluate import evaluator
>>> from datasets import load_dataset
>>> task_evaluator = evaluator("question-answering")
>>> data = load_dataset("squad_v2", split="validation[:2]")
>>> results = task_evaluator.compute(
>>> model_or_pipeline="mrm8488/bert-tiny-finetuned-squadv2",
>>> data=data,
>>> metric="squad_v2",
>>> squad_v2_format=True,
>>> )
```
c ( s> e Zd ZdZi Zd- fdd Zdededed ed
ef
d d Zd.ded
efddZ de
de de
fddZ e
eeee
d/deed ed d f deeef d ee d ee d!eeef d"eeed#f d$ed% d&ed'ed(ed)ee deded ed
edee d*eeeef ef f"d+d,Z Z S )0QuestionAnsweringEvaluatora
Question answering evaluator. This evaluator handles
[**extractive** question answering](https://huggingface.co/docs/transformers/task_summary#extractive-question-answering),
where the answer to the question is extracted from a context.
This question answering evaluator can currently be loaded from [`evaluator`] using the default task name
`question-answering`.
Methods in this class assume a data format compatible with the
[`~transformers.QuestionAnsweringPipeline`].
question-answeringNc s t j||d d S )N)default_metric_name)super__init__)selftaskr __class__ Y/home/ubuntu/.local/lib/python3.10/site-packages/evaluate/evaluator/question_answering.pyr ] s z#QuestionAnsweringEvaluator.__init__dataquestion_columncontext_column id_column label_columnc s^ |du rt d| ||| d t } fdd|D |d< |t||t||dfS )z
Prepare data.NzXPlease specify a valid `data` object - either a `str` with a name or a `Dataset` object.)r& r' r( r) c s g | ] }| | d qS ))idanswersr# ).0elementr( r) r# r$
s s z;QuestionAnsweringEvaluator.prepare_data..
references)questioncontext)
ValueErrorcheck_required_columnsdictr )r r% r& r' r( r)
metric_inputsr# r. r$ prepare_data` s(
z'QuestionAnsweringEvaluator.prepare_datar+ c s. |j }|j fddddj }||krdS dS )z
Check if the provided dataset follows the squad v2 data schema, namely possible samples where the answer is not in the context.
In this case, the answer text list should be `[]`.
c s t | d dkS )Ntextr )len)xr) r# r$ s z?QuestionAnsweringEvaluator.is_squad_v2_format..F)load_from_cache_fileT)num_rowsfilter)r r% r) original_num_rowsnonempty_num_rowsr# r; r$ is_squad_v2_format| s z-QuestionAnsweringEvaluator.is_squad_v2_format predictionssquad_v2_formatidsc C sR g }t t|D ] }|| d || d}|r || d |d< || qd|iS )Nanswer)prediction_textr* scoreno_answer_probabilityrC )ranger9 append)r rC rD rE resultipredr# r# r$ predictions_processor s z0QuestionAnsweringEvaluator.predictions_processorsimpleffffff?' r1 r2 r* model_or_pipeliner r r subsetsplitmetric tokenizerr strategy)rP bootstrapconfidence_level n_resamplesdevice random_statereturnc C s& i }| |
| | j|||d}| j|| |
||d\}}|du r0| j||d}td| d | j|||
d}| |}|rI|jdkrItd |sU|jd
krUtd |r]d | j d
< nd| j d
< | j
|fi |\}}| j |||| d}| | | j
||||| | d}| | | | |S )a
question_column (`str`, defaults to `"question"`):
The name of the column containing the question in the dataset specified by `data`.
context_column (`str`, defaults to `"context"`):
The name of the column containing the context in the dataset specified by `data`.
id_column (`str`, defaults to `"id"`):
The name of the column containing the identification field of the question and answer pair in the
dataset specified by `data`.
label_column (`str`, defaults to `"answers"`):
The name of the column containing the answers in the dataset specified by `data`.
squad_v2_format (`bool`, *optional*, defaults to `None`):
Whether the dataset follows the format of squad_v2 dataset. This is the case when the provided dataset
has questions where the answer is not in the context, more specifically when are answers as
`{"text": [], "answer_start": []}` in the answer column. If all questions have at least one answer, this parameter
should be set to `False`. If this parameter is not provided, the format will be automatically inferred.
)r% rT rU )r% r& r' r( r) N)r% r) z~`squad_v2_format` parameter not provided to QuestionAnsweringEvaluator.compute(). Automatically inferred `squad_v2_format` as .)rS rW r\ squadzkThe dataset has SQuAD v2 format but you are using the SQuAD metric. Consider passing the 'squad_v2' metric.squad_v2zkThe dataset has SQuAD v1 format but you are using the SQuAD v2 metric. Consider passing the 'squad' metric.Thandle_impossible_answerF)rD rE )rV r6 rX rZ r[ r] )"check_for_mismatch_in_device_setup load_datar7 rB loggerwarningprepare_pipelineprepare_metricnamePIPELINE_KWARGS
call_pipelinerO updatecompute_metric)r rS r% rT rU rV rW rX rZ r[ r\ r] r& r' r( r) rD rL r6 pipe_inputspiperC perf_resultsmetric_resultsr# r# r$ compute sT '
z"QuestionAnsweringEvaluator.compute)r N)r+ )NNNNNNrP rQ rR NNr1 r2 r* r+ N) __name__
__module__ __qualname____doc__rj r r
strr7 rB r boolrO r r r r TASK_DOCUMENTATIONr r r r
r floatintr r r rr
__classcell__r# r# r! r$ r N s
r N)%typingr r r r r r r r datasetsr
TRANSFORMERS_AVAILABLE ImportErrortyping_extensionsr moduler
utils.file_utilsr r
utils.loggingr baser r r utilsr transformersr r r r rs re ry r r# r# r# r$ s$ ( $