o 5ti @sZzddlmZddlmZddlmZdZWn ey!dZYnwddlm Z m Z ddl m Z dd l mZdd lmZdd lmZdd lmZdd lmZmZmZddlmZddlmZddlmZeddeddeddeddeddeddeddeddedde ddd Zde e fddZ!de de fddZd#de defd!d"Z"d S)$)SUPPORTED_TASKS) TASK_ALIASES) check_taskTF)DictList)AudioClassificationEvaluator)#AutomaticSpeechRecognitionEvaluator) Evaluator)ImageClassificationEvaluator)QuestionAnsweringEvaluator)SummarizationEvaluatorText2TextGenerationEvaluatorTranslationEvaluator)TextClassificationEvaluator)TextGenerationEvaluator)TokenClassificationEvaluatoraccuracy)implementationdefault_metric_namesquadseqeval word_countbleurougewer) ztext-classificationzimage-classificationzquestion-answeringztoken-classificationztext-generationztext2text-generation summarization translationzautomatic-speech-recognitionzaudio-classificationreturncCs ttS)z3 Returns a list of supported task strings. )listSUPPORTED_EVALUATOR_TASKSkeysr"r"O/home/ubuntu/.local/lib/python3.10/site-packages/evaluate/evaluator/__init__.pyget_supported_tasksRs r$taskcCsh|tvrt|}t|std|dtd|tvr(|tvr(t|Std|dtd)a Checks an incoming task string, to validate it's correct and returns the default Evaluator class and default metric name. It first performs a check to validata that the string is a valid `Pipeline` task, then it checks if it's a valid `Evaluator` task. `Evaluator` tasks are a substet of `Pipeline` tasks. Args: task (`str`): The task defining which evaluator will be returned. Currently accepted tasks are: - `"image-classification"` - `"question-answering"` - `"text-classification"` (alias `"sentiment-analysis"` available) - `"token-classification"` Returns: task_defaults: `dict`, contains the implementasion class of a give Evaluator and the default metric name. z Unknown task z, available tasks are: .)rcheck_pipeline_taskKeyErrorr$r r!SUPPORTED_PIPELINE_TASKS)r%r"r"r#rYsrNcCs0tstdt|}|d}|d}|||dS)a Utility factory method to build an [`Evaluator`]. Evaluators encapsulate a task and a default metric name. They leverage `pipeline` functionality from `transformers` to simplify the evaluation of multiple combinations of models, datasets and metrics for a given task. Args: task (`str`): The task defining which evaluator will be returned. Currently accepted tasks are: - `"image-classification"`: will return a [`ImageClassificationEvaluator`]. - `"question-answering"`: will return a [`QuestionAnsweringEvaluator`]. - `"text-classification"` (alias `"sentiment-analysis"` available): will return a [`TextClassificationEvaluator`]. - `"token-classification"`: will return a [`TokenClassificationEvaluator`]. Returns: [`Evaluator`]: An evaluator suitable for the task. Examples: ```python >>> from evaluate import evaluator >>> # Sentiment analysis evaluator >>> evaluator("sentiment-analysis") ```zeIf you want to use the `Evaluator` you need `transformers`. Run `pip install evaluate[transformers]`.rr)r%r)TRANSFORMERS_AVAILABLE ImportErrorr)r% targeted_taskevaluator_classrr"r"r# evaluatorqs r.)N)#transformers.pipelinesrr)rrr'r*r+typingrraudio_classificationrautomatic_speech_recognitionr baser image_classificationr question_answeringr text2text_generationr rrtext_classificationrtext_generationrtoken_classificationrr strr$r.r"r"r"r#sh            ,