o 7ti`!@spdZddlmZddlmZddlZddlmZddlm Z ddl m Z dZ d d Z d d ZGd dde ZdS)u Know What You Don’t Know: Unanswerable Questions for SQuAD https://arxiv.org/pdf/1806.03822.pdf Stanford Question Answering Dataset (SQuAD) is a reading comprehension dataset, consisting of questions posed by crowdworkers on a set of Wikipedia articles, where the answer to every question is a segment of text, or span, from the corresponding reading passage, or the question might be unanswerable. SQuAD2.0 combines the 100,000 questions in SQuAD1.1 with over 50,000 unanswerable questions written adversarially by crowdworkers to look similar to answerable ones. To do well on SQuAD2.0, systems must not only answer questions when possible, but also determine when no answer is supported by the paragraph and abstain from answering. Homepage: https://rajpurkar.github.io/SQuAD-explorer/ )partialexpN)versionInstance)ConfigurableTaskz @misc{rajpurkar2018know, title={Know What You Don't Know: Unanswerable Questions for SQuAD}, author={Pranav Rajpurkar and Robin Jia and Percy Liang}, year={2018}, eprint={1806.03822}, archivePrefix={arXiv}, primaryClass={cs.CL} } cCs ddl}|d}|j||dS)Nrsquad_v2 predictions references)evaluateloadcompute)r r r squad_metricrN/home/ubuntu/.local/lib/python3.10/site-packages/lm_eval/tasks/squadv2/task.py _squad_metric's rcCs t|\}}t||d|dS)Nr r)ziprget)keyitemsr r rrr _squad_agg.s rcseZdZdZdZdZd#fdd Zee j edks Jddd Z d d Z d d Z ddZddZddZddZddZddZ d$ddZddZdd Zd!d"ZZS)%SQuAD2zlighteval/squad_v2Ncstjdd|jiiddS)Nmetadatar)config)super__init__VERSION)selfr __class__rrr9szSQuAD2.__init__z1.11.0z,datasets v1.11.0 or later required for SQuADcCdSNTrr rrrhas_training_docsAzSQuAD2.has_training_docscCr#r$rr%rrrhas_validation_docsDr'zSQuAD2.has_validation_docscCr#NFrr%rrr has_test_docsGr'zSQuAD2.has_test_docscC |jdS)Ntraindatasetr%rrr training_docsJ zSQuAD2.training_docscCr+)N validationr-r%rrrvalidation_docsMr0zSQuAD2.validation_docscCs4d|ddd|ddd|dddS) NzTitle: titlez z Background: contextz Question: questionzAnswer:rr docrrr doc_to_textPs( zSQuAD2.doc_to_textcCr#r$rr%rrrshould_decontaminate^r'zSQuAD2.should_decontaminatecCs|dS)Nr4rr6rrrdoc_to_decontamination_queryasz#SQuAD2.doc_to_decontamination_querycCs4|dd}t|dkr|d}d|Sd}d|S)Nanswerstextr unanswerable )len)r r7 answer_listanswerrrr doc_to_targetds  zSQuAD2.doc_to_targetFcKs>td d||ddgifdd|td d||dfdd|gS) aUses RequestFactory to construct Requests and returns an iterable of Requests which will be sent to the LM. :param doc: The document as returned from training_docs, validation_docs, or test_docs. :param ctx: str The context string, generated by fewshot_context. This includes the natural language description, as well as the few shot examples, and the question part of the document for `doc`. generate_untiluntil r) request_typer7 argumentsidx loglikelihoodz unanswerableNrr)r r7ctx chat_templateapply_chat_templatekwargsrrrconstruct_requestsls" zSQuAD2.construct_requestsc Csl|\}\}}t|}|d||d}|d|dd}||f||f||f||f||f||f||f||fdS)aTake a single document and the LM results and evaluates, returning a dict where keys are the names of submetrics and values are the values of the metric for that one document :param doc: The document as returned from training_docs, validation_docs, or test_docs. :param results: The results of the requests created in construct_requests. id)rOprediction_textno_answer_probabilityr;)rOr;exactf1 HasAns_exact HasAns_f1 NoAns_exactNoAns_f1 best_exactbest_f1r) r r7results continuationlogprob_unanswerable_rQr r rrrprocess_resultss@ zSQuAD2.process_resultsc CsFttdttdttdttdttdttdttdttdd S) z :returns: {str: [float] -> float} A dictionary where keys are the names of submetrics and values are functions that aggregate a list of metrics rSrTrUrVrWrXrYrZrR)rrr%rrr aggregations2zSQuAD2.aggregationc CsdddddddddS)z :returns: {str: bool} A dictionary where keys are the names of submetrics and values are whether a higher value of the submetric is better TrRrr%rrrhigher_is_betterszSQuAD2.higher_is_better)Nr))__name__ __module__ __qualname__r DATASET_PATH DATASET_NAMErrparsedatasets __version__r&r(r*r/r2r8r9r:rBrNr_r`ra __classcell__rrr!rr4s, :!r)__doc__ functoolsrmathrrh packagingrlm_eval.api.instancerlm_eval.api.taskr _CITATIONrrrrrrrs