o 5tiaY@s(ddlmZmZddlmZddlmZmZmZm Z m Z m Z ddl m Z mZddlmZz ddlmZdZWn eyAdZYnwzdd lZdd lmZmZdZWn ey]dZYnwdd lmZdd lmZd dlmZd dlm Z d dl!m"Z"ddl#m$Z$e"e%Z&dZ'dZ(GdddeZ)d S))ABCabstractmethod)Number)AnyCallableDictListOptionalUnion)Dataset load_dataset) choose_split) bootstrapTFN)Pipelinepipeline) perf_counter)Literal)load)EvaluationModule) get_logger) DatasetColumna Compute the metric for a given pipeline and dataset combination. Args: model_or_pipeline (`str` or `Pipeline` or `Callable` or `PreTrainedModel` or `TFPreTrainedModel`, defaults to `None`): If the argument in not specified, we initialize the default pipeline for the task (in this case `text-classification` or its alias - `sentiment-analysis`). If the argument is of the type `str` or is a model instance, we use it to initialize a new `Pipeline` with the given model. Otherwise we assume the argument specifies a pre-initialized pipeline. data (`str` or `Dataset`, defaults to `None`): Specifies the dataset we will run evaluation on. If it is of type `str`, we treat it as the dataset name, and load it. Otherwise we assume it represents a pre-loaded dataset. subset (`str`, defaults to `None`): Defines which dataset subset to load. If `None` is passed the default subset is loaded. split (`str`, defaults to `None`): Defines which dataset split to load. If `None` is passed, infers based on the `choose_split` function. metric (`str` or `EvaluationModule`, defaults to `None`): Specifies the metric we use in evaluator. If it is of type `str`, we treat it as the metric name, and load it. Otherwise we assume it represents a pre-loaded metric. tokenizer (`str` or `PreTrainedTokenizer`, *optional*, defaults to `None`): Argument can be used to overwrite a default tokenizer if `model_or_pipeline` represents a model for which we build a pipeline. If `model_or_pipeline` is `None` or a pre-initialized pipeline, we ignore this argument. strategy (`Literal["simple", "bootstrap"]`, defaults to "simple"): specifies the evaluation strategy. Possible values are: - `"simple"` - we evaluate the metric and return the scores. - `"bootstrap"` - on top of computing the metric scores, we calculate the confidence interval for each of the returned metric keys, using `scipy`'s `bootstrap` method https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.bootstrap.html. confidence_level (`float`, defaults to `0.95`): The `confidence_level` value passed to `bootstrap` if `"bootstrap"` strategy is chosen. n_resamples (`int`, defaults to `9999`): The `n_resamples` value passed to `bootstrap` if `"bootstrap"` strategy is chosen. device (`int`, defaults to `None`): Device ordinal for CPU/GPU support of the pipeline. Setting this to -1 will leverage CPU, a positive integer will run the model on the associated CUDA device ID. If `None` is provided it will be inferred and CUDA:0 used if available, CPU otherwise. random_state (`int`, *optional*, defaults to `None`): The `random_state` value passed to `bootstrap` if `"bootstrap"` strategy is chosen. Useful for debugging. a` Return: A `Dict`. The keys represent metric keys calculated for the `metric` spefied in function arguments. For the `"simple"` strategy, the value is the metric score. For the `"bootstrap"` strategy, the value is a `Dict` containing the score, the confidence interval and the standard error calculated for each metric key. c#@sjeZdZdZiZiZdEdedefddZe   dFd e ed e d e d e e d e eeff ddZede de de d e eeffddZed e fddZeddZ               dGdeededdfd eeefd!e ed"e ed#eeefd$e eed%fd&e eed'fd(ed)d e d e d*e d e e d+ed,ed-e e eefd e ee ff d.d/Zed0d1Zd eeefd2e eeffd3d4ZedHd5d6ZdHd eeefd!ed"efd7d8Zd ed+ed,efd9d:Z   dIdeededdfd$ed;d&ed;d*e fdd?Z!d@dAZ"    dJd#edBe d(ed)d e d e d e e f dCdDZ#dS)K Evaluatorz The [`Evaluator`] class is the class from which all evaluators inherit. Refer to this class for methods shared across different evaluators. Base class implementing evaluator operations. Ntaskdefault_metric_namecCs(tstdts td||_||_dS)NzbIf you want to use the `Evaluator` you need `transformers`. Run `pip install evaluate[evaluator]`.zbIf you want to use the `Evaluator` you need `scipy>=1.7.1`. Run `pip install evaluate[evaluator]`.)TRANSFORMERS_AVAILABLE ImportErrorSCIPY_AVAILABLErr)selfrrr K/home/ubuntu/.local/lib/python3.10/site-packages/evaluate/evaluator/base.py__init__ps zEvaluator.__init__ffffff?' metric_keysconfidence_level n_resamples random_statereturnc Csbdd}i}|D]&}tt||||fi|dd|||d} | jj| jjf| jd||<q|S)z A utility function enabling the confidence interval calculation for metrics computed by the evaluator based on `scipy`'s `bootstrap` method. csfdd}|S)Ncs(jdiddt|DS)NcSsi|]\}}||qSr r ).0kvr r r! szjEvaluator._compute_confidence_interval..build_args_metric..args_metric..r )computezipkeys)argskeykwargsmetricr r! args_metrics(zVEvaluator._compute_confidence_interval..build_args_metric..args_metricr )r5r3r4r6r r2r!build_args_metricszAEvaluator._compute_confidence_interval..build_args_metricTF)data statisticpaired vectorizedr&r'r()confidence_intervalstandard_error)rlistvaluesr<lowhighr=) r5 metric_inputsr%r&r'r(r7bootstrap_dictr3bsr r r!_compute_confidence_interval|s    z&Evaluator._compute_confidence_interval start_timeend_time num_samplescCs$||}||}d|}|||dS)a} A utility function computing time performance metrics: - `total_time_in_seconds` - pipeline inference runtime for the evaluation data in seconds, - `samples_per_second` - pipeline throughput in the number of samples per second. - `latency_in_seconds` - pipeline inference runtime for the evaluation data in seconds per sample, g?)total_time_in_secondssamples_per_secondlatency_in_secondsr )rFrGrHlatency throughputlatency_sampler r r!_compute_time_perfs zEvaluator._compute_time_perfc Cszddl}|jr d}nd}Wn*ty:zddl}t|jddkr)d}nd}Wn ty7d}YnwYnw|dkrFt d|St d|S)zBHelper function to check if GPU or CPU is available for inference.rNGPUzFNo GPU found. The default device for pipeline inference is set to CPU.zLGPU found. The default device for pipeline inference is set to GPU (CUDA:0).) torchcuda is_availabler tensorflowlenconfiglist_physical_devicesloggerinfo)rRdevicetfr r r! _infer_devices.     zEvaluator._infer_devicecOst)z A core method of the `Evaluator` class, which processes the pipeline outputs for compatibility with the metric. )NotImplementedError)rr1r4r r r!predictions_processorszEvaluator.predictions_processorsimpletextlabelmodel_or_pipelinerPreTrainedModelTFPreTrainedModelr8subsetsplitr5 tokenizerPreTrainedTokenizerfeature_extractorFeatureExtractionMixinstrategy)r`rr[ input_column label_column label_mappingcCsi}|| ||j|||d}|j|| |d\}}|j|||| d}||}|||\}}|||}|||j|||| | | d}t |t urQ|j |i}|||||S)Nr8rfrg)r8rmrn)rcrhrjr[)r5rBrlr&r'r() "check_for_mismatch_in_device_setup load_data prepare_dataprepare_pipelineprepare_metric call_pipeliner_updatecompute_metrictypefloatname)rrcr8rfrgr5rhrjrlr&r'r[r(rmrnroresultrB pipe_inputspipe predictions perf_resultsmetric_resultsr r r!r.s6       zEvaluator.computecCsd|dur*|dkr,t|tr.|jjdkrtd||jjkr0td|jjd|ddSdSdSdS)NrPcpuaThe value of the `device` kwarg passed to `compute` suggests that this pipe should be run on an accelerator, but the pipe was instantiated on CPU. Pass `device` to the pipeline during initialization to use an accelerator, or pass `device=None` to `compute`. z)This pipeline was instantiated on device z but device=z was passed to `compute`.) isinstancerr[ry ValueErrorindex)r[rcr r r!rqs   z,Evaluator.check_for_mismatch_in_device_setup columns_namesc Cs>|D]\}}||jvrtd|d|d|jdqdS)a Ensure the columns required for the evaluation are present in the dataset. Args: data (`str` or [`Dataset`]): Specifies the dataset we will run evaluation on. columns_names (`List[str]`): List of column names to check in the dataset. The keys are the arguments to the [`evaluate.EvaluationModule.compute`] method, while the values are the column names to check. Example: ```py >>> from datasets import load_dataset >>> from evaluate import evaluator >>> data = load_dataset("rotten_tomatoes', split="train") >>> evaluator.check_required_columns(data, {"input_column": "text", "label_column": "label"}) ``` z Invalid `z` z8 specified. The dataset contains the following columns: .N)items column_namesr)rr8r input_name column_namer r r!check_required_columns's z Evaluator.check_required_columnscCs*|durt||}td||S)a Infers which split to use if `None` is given. Args: data (`str`): Name of dataset. subset (`str`): Name of config for datasets with multiple configurations (e.g. 'glue/cola'). split (`str`, defaults to `None`): Split to use. Returns: `split`: `str` containing which split to use Example: ```py >>> from evaluate import evaluator >>> evaluator("text-classification").get_dataset_split(data="rotten_tomatoes") WARNING:evaluate.evaluator.base:Dataset split not defined! Automatically evaluating with split: TEST 'test' ``` Nz@Dataset split not defined! Automatically evaluating with split: )r rYwarningupperrpr r r!get_dataset_splitAs zEvaluator.get_dataset_splitcCsZt|tr||||}t|||d}|St|tr)|dus"|dur'td|Std)a Load dataset with given subset and split. Args: data ([`Dataset`] or `str`, defaults to `None`): Specifies the dataset we will run evaluation on. If it is of type `str`, we treat it as the dataset name, and load it. Otherwise we assume it represents a pre-loaded dataset. subset (`str`, defaults to `None`): Specifies dataset subset to be passed to `name` in `load_dataset`. To be used with datasets with several configurations (e.g. glue/sst2). split (`str`, defaults to `None`): User-defined dataset split by name (e.g. train, validation, test). Supports slice-split (`test[:n]`). If not defined and data is a `str` type, will automatically select the best one via `choose_split()`. Returns: data ([`Dataset`]): Loaded dataset which will be used for evaluation. Example: ```py >>> from evaluate import evaluator >>> evaluator("text-classification").load_data(data="rotten_tomatoes", split="train") Dataset({ features: ['text', 'label'], num_rows: 8530 }) ``` )r{rgNz=`data` is a preloaded Dataset! Ignoring `subset` and `split`.zXPlease specify a valid `data` object - either a `str` with a name or a `Dataset` object.)rstrrr r rYrr)rr8rfrgr r r!rr^s   zEvaluator.load_datacOs(||||dd||it||fS)a Prepare data. Args: data ([`Dataset`]): Specifies the dataset we will run evaluation on. input_column (`str`, defaults to `"text"`): The name of the column containing the text feature in the dataset specified by `data`. second_input_column(`str`, *optional*): The name of the column containing the second text feature if there is one. Otherwise, set to `None`. label_column (`str`, defaults to `"label"`): The name of the column containing the labels in the dataset specified by `data`. Returns: `dict`: metric inputs. `list`: pipeline inputs. Example: ```py >>> from evaluate import evaluator >>> from datasets import load_dataset >>> ds = load_dataset("rotten_tomatoes", split="train") >>> evaluator("text-classification").prepare_data(ds, input_column="text", second_input_column=None, label_column="label") ``` )rmrn references)rr)rr8rmrnr1r4r r r!rsszEvaluator.prepare_data)PreTrainedTokenizerBaserkcCs|dur|}t|tst|tjst|tjr$t|j||||d}n|dur0t|j|d}n|}|dur?|dur?t d|j|jkrY|jdkrP|j dsYt d|jd|S)a Prepare pipeline. Args: model_or_pipeline (`str` or [`~transformers.Pipeline`] or `Callable` or [`~transformers.PreTrainedModel`] or [`~transformers.TFPreTrainedModel`], defaults to `None`): If the argument in not specified, we initialize the default pipeline for the task. If the argument is of the type `str` or is a model instance, we use it to initialize a new [`~transformers.Pipeline`] with the given model. Otherwise we assume the argument specifies a pre-initialized pipeline. preprocessor ([`~transformers.PreTrainedTokenizerBase`] or [`~transformers.FeatureExtractionMixin`], *optional*, defaults to `None`): Argument can be used to overwrite a default preprocessor if `model_or_pipeline` represents a model for which we build a pipeline. If `model_or_pipeline` is `None` or a pre-initialized pipeline, we ignore this argument. Returns: The initialized pipeline. Example: ```py >>> from evaluate import evaluator >>> evaluator("text-classification").prepare_pipeline(model_or_pipeline="distilbert-base-uncased") ``` N)modelrhrjr[)r[zUIgnoring the value of the preprocessor argument (`tokenizer` or `feature_extractor`). translationzZIncompatible `model_or_pipeline`. Please specify `model_or_pipeline` compatible with the `z` task.) r]rr transformersrdrerrrYr startswithr)rrcrhrjr[r~r r r!rts2   " zEvaluator.prepare_pipelinecCs>|dur|jdur tdt|j}|St|trt|}|S)a Prepare metric. Args: metric (`str` or [`EvaluationModule`], defaults to `None`): Specifies the metric we use in evaluator. If it is of type `str`, we treat it as the metric name, and load it. Otherwise we assume it represents a pre-loaded metric. Returns: The loaded metric. Example: ```py >>> from evaluate import evaluator >>> evaluator("text-classification").prepare_metric("accuracy") ``` NzW`Evaluator` doesn't specify a default metric. Please specify a valid `metric` argument.)rrrrr)rr5r r r!rus   zEvaluator.prepare_metriccOs6t}||i||j}t}||||t|fSN)rPIPELINE_KWARGSrOrV)rr~r1r4rF pipe_outputrGr r r!rvszEvaluator.call_pipelinerBc Cs\|jdi||j}|dkr,|}|||||||} |D] } || | | d<q| S|S)zCompute and return metrics.rscoreNr )r. METRIC_KWARGSr0rE) rr5rBrlr&r'r(r|r%rCr3r r r!rxs zEvaluator.compute_metricr)r#r$N)NNNNNNNr`r#r$NNrarbN)NN)NNN)r`r#r$N)$__name__ __module__ __qualname____doc__rrrr" staticmethodrrzintr rrrErOr]rr_r rr rrrr.rqrrrrrsrtrurvrxr r r r!rfs   %$            ? "   (# : r)*abcrrnumbersrtypingrrrrr r datasetsr r evaluate.evaluator.utilsr scipy.statsrrrrrrrtimertyping_extensionsrloadingrmoduler utils.loggingrutilsrrrY EVALUTOR_COMPUTE_START_DOCSTRING"EVALUATOR_COMPUTE_RETURN_DOCSTRINGrr r r r!s8            )