o 7ti @sjddlZddlmZddlmZddlZddlmZddl m Z Gddde Z de d ee fd d Z dS) N)deepcopy)List)Instance)ConfigurableTaskcs|eZdZdZdZdZfddZddZdd Zd d Z d d Z ddZ ddZ dddZ ddZddZddZZS)SQUADCompletionrzhazyresearch/based-squaddefaultc stjdd|jiiddS)Nmetadataversion)config)super__init__VERSION)selfkwargs __class__W/home/ubuntu/.local/lib/python3.10/site-packages/lm_eval/tasks/squad_completion/task.pyr szSQUADCompletion.__init__cCdSNFrrrrrhas_training_docsz!SQUADCompletion.has_training_docscCr)NTrrrrrhas_validation_docsrz#SQUADCompletion.has_validation_docscCrrrrrrr has_test_docsrzSQUADCompletion.has_test_docscCs |jdS)N validation)datasetrrrrvalidation_docss zSQUADCompletion.validation_docscC|dS)Ntextrrdocrrr doc_to_textzSQUADCompletion.doc_to_textcCr)Nvaluerr rrr doc_to_target"r#zSQUADCompletion.doc_to_targetNFcKsLt|jj}|ddg|d<|dd|d<td d|||fdd|gS) aUses RequestFactory to construct Requests and returns an iterable of Requests which will be sent to the LM. :param doc: The document as returned from training_docs, validation_docs, or test_docs. :param ctx: str The context string, generated by fewshot_context. This includes the natural language description, as well as the few shot examples, and the question part of the document for `doc`. until max_gen_toks0generate_untilr) request_typer! argumentsidxNr)rr generation_kwargsgetr)rr!ctx chat_templateapply_chat_templaterr,rrrconstruct_requests%s z"SQUADCompletion.construct_requestscCs|}dt|d|dgiS)aTake a single document and the LM results and evaluates, returning a dict where keys are the names of submetrics and values are the values of the metric for that one document :param doc: The document as returned from training_docs, validation_docs, or test_docs. :param results: The results of the requests created in construct_requests. containsrr$)contains_score)rr!results continuationrrrprocess_results?s zSQUADCompletion.process_resultscCs dtjiS)z :returns: {str: [float] -> float} A dictionary where keys are the names of submetrics and values are functions that aggregate a list of metrics r4)npmeanrrrr aggregationNszSQUADCompletion.aggregationcCsddiS)z :returns: {str: bool} A dictionary where keys are the names of submetrics and values are whether a higher value of the submetric is better r4Trrrrrhigher_is_betterXsz SQUADCompletion.higher_is_betterr)__name__ __module__ __qualname__r DATASET_PATH DATASET_NAMEr rrrrr"r%r3r8r;r< __classcell__rrrrr s    r predictionlabelscstfdd|DS)Nc 3s4|]}ttttt|tjVqdS)N)intboolresearchcompileescape IGNORECASE).0labelrCrr ds " z!contains_score..)max)rCrDrrNrr5cs r5)rGcopyrtypingrnumpyr9lm_eval.api.instancerlm_eval.api.taskrrstrr5rrrrs    X