o yiy&@s\ddlZddlZddlmZddlmZmZmZmZm Z m Z ddl m Z m Z ddlmZeeefZe eeefZeee eeee eeeeffffZe eeefZe eeee eeffeeee eeeefffZdgdgdd d d d d ZdedefddZdedeefddZdedede fddZdedede fddZdeeege fdedeede fddZd ed!ede eeefeeeeeeeeeefffffd"d#Zd eeefd$eeeeeeeeeefffde e e e ffd%d&Zd'e d(e d)e deee ffd*d+Zd ed$edeee ffd,d-Z dS).N)Counter)AnyCallableDictListTupleUnion)Tensortensor)rank_zero_warnzThis is a test text) answer_starttextzThis is a test context.1zIs this a test?z train test)answerscontextidquestiontitlesreturncCs\dtdtfdd}dtdtfdd}dtdtfdd}dtdtfd d }|||||S) zALower text and remove punctuation, articles and extra whitespace.rrcSstdd|S)Nz\b(a|an|the)\b )resubrrV/home/ubuntu/.local/lib/python3.10/site-packages/torchmetrics/functional/text/squad.pyremove_articles,z(_normalize_text..remove_articlescSsd|S)Nr)joinsplitrrrrwhite_space_fix/rz(_normalize_text..white_space_fixcs"ttjdfdd|DS)Nc3s|] }|vr|VqdSNr).0chexcluderr 4sz7_normalize_text..remove_punc..)setstring punctuationrrrr&r remove_punc2s z$_normalize_text..remove_punccSs|Sr#)lowerrrrrr-6sz_normalize_text..lower)str)rrr!r,r-rrr_normalize_text)s r/cCs|sgSt|S)z&Split a sentence into separate tokens.)r/r )rrrr _get_tokens<sr0predicted_answer target_answerc Cst|}t|}t|t|@}tt|}t|dks$t|dkr,tt||kS|dkr4tdSd|tt|}d|tt|}d||||}|S)z#Compute F1 Score for two sentences.rg?)r0rr sumvalueslenint) r1r2 target_tokenspredicted_tokenscommonnum_same precisionrecallf1rrr_compute_f1_scoreAsr@ prediction ground_truthcCsttt|t|kS)z&Compute Exact Match for two sentences.)r r8r/)rArBrrr_compute_exact_match_scoreRsrC metric_fn ground_truthscstfdd|DS)zJCalculate maximum score for a predicted answer with all reference answers.c3s|]}|VqdSr#r)r$truthrDrArrr([sz1_metric_max_over_ground_truths..)max)rDrArErrGr_metric_max_over_ground_truthsWsrIpredstargetscst|tr|g}t|tr|g}|D]}|}d|vs d|vr$tdq|D]&}|}d|vs5d|vrsz&_squad_input_check..cSs"tdd|ddD|ddS)NcSsg|]}t|dqS)rdict)r$txtrrr sz8_squad_input_check....rrr)rrrN)tgtrrrs"z$_squad_input_check.. paragraphsqascsg|]}|qSrr)r$target _fn_answerrrrQsz&_squad_input_check..) isinstancerkeysKeyError SQuAD_FORMAT)rJrKpredrZrVr preds_dict targets_dictrrWr_squad_input_check^s>   r`rVc Cstd}td}td}|D]F}|dD]?}|dD]8}|d7}|d|vr1td|ddqttd d |d }||d} |tt| |7}|tt| |7}qqq|||fS) atCompute F1 Score and Exact Match for a collection of predictions and references. Args: preds: A dictionary mapping an `id` to the predicted `answer`. target: A list of dictionary mapping `paragraphs` to list of dictionary mapping `qas` to a list of dictionary containing `id` and list of all possible `answers`. Return: Tuple containing F1 score, Exact match score and total number of examples. Example: >>> from torchmetrics.functional.text.squad import _squad_update >>> preds = [{"prediction_text": "1976", "id": "56e10a3be3433e1400422b22"}] >>> target = [{"answers": {"answer_start": [97], "text": ["1976"]}, "id": "56e10a3be3433e1400422b22"}] >>> preds_dict = {pred["id"]: pred["prediction_text"] for pred in preds} >>> targets_dict = [ ... dict(paragraphs=[dict(qas=[dict(answers=[ ... {"text": txt} for txt in tgt["answers"]["text"]], id=tgt["id"]) for tgt in target ... ])]) ... ] >>> _squad_update(preds_dict, targets_dict) (tensor(1.), tensor(1.), tensor(1)) r3rrTrUr rzUnanswered question z will receive score 0.cSs|dS)Nrr)xrrrrSsz_squad_update..r)r r listmaprIrCr@) rJrVr? exact_matchtotalarticle paragraphqarEr]rrr _squad_updates"     rir?rdrecCs"d||}d||}||dS)zAggregate the F1 Score and Exact match for the batch. Return: Dictionary containing the F1 score, Exact match score for the batch. gY@)rdr?r)r?rdrerrr_squad_computes   rjcCs*t||\}}t||\}}}t|||S)aCalculate `SQuAD Metric`_ . Args: preds: A Dictionary or List of Dictionary-s that map `id` and `prediction_text` to the respective values. Example prediction: .. code-block:: python {"prediction_text": "TorchMetrics is awesome", "id": "123"} target: A Dictionary or List of Dictionary-s that contain the `answers` and `id` in the SQuAD Format. Example target: .. code-block:: python { 'answers': [{'answer_start': [1], 'text': ['This is a test answer']}], 'id': '1', } Reference SQuAD Format: .. code-block:: python { 'answers': {'answer_start': [1], 'text': ['This is a test text']}, 'context': 'This is a test context.', 'id': '1', 'question': 'Is this a test?', 'title': 'train test' } Return: Dictionary containing the F1 score, Exact match score for the batch. Example: >>> from torchmetrics.functional.text.squad import squad >>> preds = [{"prediction_text": "1976", "id": "56e10a3be3433e1400422b22"}] >>> target = [{"answers": {"answer_start": [97], "text": ["1976"]},"id": "56e10a3be3433e1400422b22"}] >>> squad(preds, target) {'exact_match': tensor(100.), 'f1': tensor(100.)} Raises: KeyError: If the required keys are missing in either predictions or targets. References: [1] SQuAD: 100,000+ Questions for Machine Comprehension of Text by Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, Percy Liang `SQuAD Metric`_ . )r`rirj)rJrVr^ target_dictr?rdrerrrsquads6 rl)!rr* collectionsrtypingrrrrrrtorchr r torchmetrics.utilitiesr r.SINGLE_PRED_TYPE PREDS_TYPEr8SINGLE_TARGET_TYPE TARGETS_TYPEUPDATE_METHOD_SINGLE_PRED_TYPEr\r/r0r@rCrIr`rirjrlrrrrs\    ,:   6 , &  "."