o 7ti> @s^ddlZddlmZddlZddlmZddlmZGdddeZ de dee fd d Z dS) N)ListInstance)ConfigurableTaskcs|eZdZdZdZdZfddZddZdd Zd d Z d d Z ddZ ddZ dddZ ddZddZddZZS)SWDErzhazyresearch/based-swde-v2defaultc stjdd|jiiddS)Nmetadataversion)config)super__init__VERSION)selfkwargs __class__K/home/ubuntu/.local/lib/python3.10/site-packages/lm_eval/tasks/swde/task.pyr sz SWDE.__init__cCdSNFrrrrrhas_training_docszSWDE.has_training_docscCr)NTrrrrrhas_validation_docsrzSWDE.has_validation_docscCrrrrrrr has_test_docsrzSWDE.has_test_docscCs |jdS)N validation)datasetrrrrvalidation_docss zSWDE.validation_docscC|dS)Ntextrrdocrrr doc_to_textzSWDE.doc_to_textcCr)Nvaluerr rrr doc_to_target!r#zSWDE.doc_to_targetNFcKs&tdd||dgddfdd|gS) aUses RequestFactory to construct Requests and returns an iterable of Requests which will be sent to the LM. :param doc: The document as returned from training_docs, validation_docs, or test_docs. :param ctx: str The context string, generated by fewshot_context. This includes the natural language description, as well as the few shot examples, and the question part of the document for `doc`. generate_until 0)until max_gen_toksr) request_typer! argumentsidxNrr)rr!ctx chat_templateapply_chat_templaterrrrconstruct_requests$szSWDE.construct_requestscCs|}dt|d|dgiS)aTake a single document and the LM results and evaluates, returning a dict where keys are the names of submetrics and values are the values of the metric for that one document :param doc: The document as returned from training_docs, validation_docs, or test_docs. :param results: The results of the requests created in construct_requests. containsrr$)contains_score)rr!results continuationrrrprocess_results<s zSWDE.process_resultscCs dtjiS)z :returns: {str: [float] -> float} A dictionary where keys are the names of submetrics and values are functions that aggregate a list of metrics r2)npmeanrrrr aggregationKszSWDE.aggregationcCsddiS)z :returns: {str: bool} A dictionary where keys are the names of submetrics and values are whether a higher value of the submetric is better r2Trrrrrhigher_is_betterUszSWDE.higher_is_betterr)__name__ __module__ __qualname__r DATASET_PATH DATASET_NAMEr rrrrr"r%r1r6r9r: __classcell__rrrrr s    r predictionlabelscstfdd|DS)Nc 3s4|]}ttttt|tjVqdS)N)intboolresearchcompileescape IGNORECASE).0labelrArr as " z!contains_score..)max)rArBrrLrr3`s r3) rEtypingrnumpyr7lm_eval.api.instancerlm_eval.api.taskrrstrr3rrrrs   V