o .wi@sddlmZddlmZmZmZddlZddlmZddlm Z ddl m Z m Z m Z mZmZddlmZddlmZmZesAd gZGd d d e ZdS) )Sequence)AnyOptionalUnionN)Tensor)Metric) PREDS_TYPE TARGETS_TYPE_squad_compute_squad_input_check _squad_update)_MATPLOTLIB_AVAILABLE)_AX_TYPE_PLOT_OUT_TYPE SQuAD.plotcseZdZUdZdZeed<dZeed<dZeed<dZ e ed<d Z e ed <e ed <e ed <e ed <de ddffdd ZdededdfddZdeee ffddZ ddeee ee fdeedefddZZS)SQuADaCalculate `SQuAD Metric`_ which is a metric for evaluating question answering models. This metric corresponds to the scoring script for version 1 of the Stanford Question Answering Dataset (SQuAD). As input to ``forward`` and ``update`` the metric accepts the following input: - ``preds`` (:class:`~Dict`): A Dictionary or List of Dictionary-s that map ``id`` and ``prediction_text`` to the respective values Example ``prediction``: .. code-block:: python {"prediction_text": "TorchMetrics is awesome", "id": "123"} - ``target`` (:class:`~Dict`): A Dictionary or List of Dictionary-s that contain the ``answers`` and ``id`` in the SQuAD Format. Example ``target``: .. code-block:: python { 'answers': [{'answer_start': [1], 'text': ['This is a test answer']}], 'id': '1', } Reference SQuAD Format: .. code-block:: python { 'answers': {'answer_start': [1], 'text': ['This is a test text']}, 'context': 'This is a test context.', 'id': '1', 'question': 'Is this a test?', 'title': 'train test' } As output of ``forward`` and ``compute`` the metric returns the following output: - ``squad`` (:class:`~Dict`): A dictionary containing the F1 score (key: "f1"), and Exact match score (key: "exact_match") for the batch. Args: kwargs: Additional keyword arguments, see :ref:`Metric kwargs` for more info. Example: >>> from torchmetrics.text import SQuAD >>> preds = [{"prediction_text": "1976", "id": "56e10a3be3433e1400422b22"}] >>> target = [{"answers": {"answer_start": [97], "text": ["1976"]}, "id": "56e10a3be3433e1400422b22"}] >>> squad = SQuAD() >>> squad(preds, target) {'exact_match': tensor(100.), 'f1': tensor(100.)} Fis_differentiableThigher_is_betterfull_state_updategplot_lower_boundgY@plot_upper_boundf1_score exact_matchtotalkwargsreturnNc sjtjdi||jdtjdtjddd|jdtjdtjddd|jdtjdtjddddS) Nrr)dtypesum)namedefaultdist_reduce_fxrr)super__init__ add_statetorchtensorfloatint)selfr __class__r!T/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/torchmetrics/text/squad.pyr#hs zSQuAD.__init__predstargetcCsLt||\}}t||\}}}|j|7_|j|7_|j|7_dS)z*Update state with predictions and targets.N)r r rrr)r)r-r. preds_dict target_dictrrrr!r!r,updaters z SQuAD.updatecCst|j|j|jS)z5Aggregate the F1 Score and Exact match for the batch.)r rrr)r)r!r!r,computezsz SQuAD.computevalaxcCs |||S)aPlot a single or multiple values from the metric. Args: val: Either a single result from calling `metric.forward` or `metric.compute` or a list of these results. If no value is provided, will automatically call `metric.compute` and plot that result. ax: An matplotlib axis object. If provided will add plot to that axis Returns: Figure and Axes object Raises: ModuleNotFoundError: If `matplotlib` is not installed .. plot:: :scale: 75 >>> # Example plotting a single value >>> from torchmetrics.text import SQuAD >>> metric = SQuAD() >>> preds = [{"prediction_text": "1976", "id": "56e10a3be3433e1400422b22"}] >>> target = [{"answers": {"answer_start": [97], "text": ["1976"]}, "id": "56e10a3be3433e1400422b22"}] >>> metric.update(preds, target) >>> fig_, ax_ = metric.plot() .. plot:: :scale: 75 >>> # Example plotting multiple values >>> from torchmetrics.text import SQuAD >>> metric = SQuAD() >>> preds = [{"prediction_text": "1976", "id": "56e10a3be3433e1400422b22"}] >>> target = [{"answers": {"answer_start": [97], "text": ["1976"]}, "id": "56e10a3be3433e1400422b22"}] >>> values = [ ] >>> for _ in range(10): ... values.append(metric(preds, target)) >>> fig_, ax_ = metric.plot(values) )_plot)r)r3r4r!r!r,plot~s *r)NN)__name__ __module__ __qualname____doc__rbool__annotations__rrrr'rrrr#rr r1dictstrr2rrrrrr6 __classcell__r!r!r*r,r#s2  :     r)collections.abcrtypingrrrr%r torchmetricsr"torchmetrics.functional.text.squadrr r r r torchmetrics.utilities.importsr torchmetrics.utilities.plotrr__doctest_skip__rr!r!r!r,s