o .wiE @sddlmZddlmZmZmZmZmZddlZddlm Z ddl m Z ddl m Z ddlmZddlmZdd lmZdd lmZmZdd lmZdd lmZmZmZesZd dgZde de de dedee e ff ddZ Gddde Z!Gddde!Z"dS))Sequence)AnyCallableListOptionalUnionN)Tensor)Literal)Metric) retrieval_precision_recall_curve)_retrieval_aggregate)_check_retrieval_inputs)_flexible_bincount dim_zero_cat)_MATPLOTLIB_AVAILABLE)_AX_TYPE_PLOT_OUT_TYPE plot_curve"RetrievalPrecisionRecallCurve.plot$RetrievalRecallAtFixedPrecision.plot precisionrecalltop_k min_precisionreturncsztfddt|||D\}}Wnty-tjd|j|jd}tt|}Ynw|dkr>tjt||j|jd}||fS)aCompute maximum recall with condition that corresponding precision >= `min_precision`. Args: top_k: tensor with all possible k precision: tensor with all values precisions@k for k from top_k tensor recall: tensor with all values recall@k for k from top_k tensor min_precision: float value specifying minimum precision threshold. Returns: Maximum recall value, corresponding it best k c3s&|]\}}}|kr||fVqdS)N).0prkrrj/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/torchmetrics/retrieval/precision_recall_curve.py 4s$z7_retrieval_recall_at_fixed_precision..)devicedtype)maxzip ValueErrortorchtensorr$r%len)rrrr max_recallbest_krr r!$_retrieval_recall_at_fixed_precision!s& r.cseZdZUdZdZeed<dZeed<dZeed<e e ed<e e ed<e e ed <  dd e e dede de e deedefdedd ffdd Zde d e de dd fddZdee e e ffddZ d de ee e e fde edefddZZS)!RetrievalPrecisionRecallCurveaCompute precision-recall pairs for different k (from 1 to `max_k`). In a ranked retrieval context, appropriate sets of retrieved documents are naturally given by the top k retrieved documents. Recall is the fraction of relevant documents retrieved among all the relevant documents. Precision is the fraction of relevant documents among all the retrieved documents. For each such set, precision and recall values can be plotted to give a recall-precision curve. As input to ``forward`` and ``update`` the metric accepts the following input: - ``preds`` (:class:`~torch.Tensor`): A float tensor of shape ``(N, ...)`` - ``target`` (:class:`~torch.Tensor`): A long or bool tensor of shape ``(N, ...)`` - ``indexes`` (:class:`~torch.Tensor`): A long tensor of shape ``(N, ...)`` which indicate to which query a prediction belongs As output to ``forward`` and ``compute`` the metric returns the following output: - ``precisions`` (:class:`~torch.Tensor`): A tensor with the fraction of relevant documents among all the retrieved documents. - ``recalls`` (:class:`~torch.Tensor`): A tensor with the fraction of relevant documents retrieved among all the relevant documents - ``top_k`` (:class:`~torch.Tensor`): A tensor with k from 1 to `max_k` All ``indexes``, ``preds`` and ``target`` must have the same dimension and will be flatten at the beginning, so that for example, a tensor of shape ``(N, M)`` is treated as ``(N * M, )``. Predictions will be first grouped by ``indexes`` and then will be computed as the mean of the metric over each query. Args: max_k: Calculate recall and precision for all possible top k from 1 to max_k (default: `None`, which considers all possible top k) adaptive_k: adjust `k` to `min(k, number of documents)` for each query empty_target_action: Specify what to do with queries that do not have at least a positive ``target``. Choose from: - ``'neg'``: those queries count as ``0.0`` (default) - ``'pos'``: those queries count as ``1.0`` - ``'skip'``: skip those queries; if all queries are skipped, ``0.0`` is returned - ``'error'``: raise a ``ValueError`` ignore_index: Ignore predictions where the target is equal to this number. aggregation: Specify how to aggregate over indexes. Can either a custom callable function that takes in a single tensor and returns a scalar value or one of the following strings: - ``'mean'``: average value is returned - ``'median'``: median value is returned - ``'max'``: max value is returned - ``'min'``: min value is returned kwargs: Additional keyword arguments, see :ref:`Metric kwargs` for more info. Raises: ValueError: If ``empty_target_action`` is not one of ``error``, ``skip``, ``neg`` or ``pos``. ValueError: If ``ignore_index`` is not `None` or an integer. ValueError: If ``max_k`` parameter is not `None` or not an integer larger than 0. Example: >>> from torch import tensor >>> from torchmetrics.retrieval import RetrievalPrecisionRecallCurve >>> indexes = tensor([0, 0, 0, 0, 1, 1, 1]) >>> preds = tensor([0.4, 0.01, 0.5, 0.6, 0.2, 0.3, 0.5]) >>> target = tensor([True, False, False, True, True, False, True]) >>> r = RetrievalPrecisionRecallCurve(max_k=4) >>> precisions, recalls, top_k = r(preds, target, indexes=indexes) >>> precisions tensor([1.0000, 0.5000, 0.6667, 0.5000]) >>> recalls tensor([0.5000, 0.5000, 1.0000, 1.0000]) >>> top_k tensor([1, 2, 3, 4]) Fis_differentiableThigher_is_betterfull_state_updateindexespredstargetNnegmeanmax_k adaptive_kempty_target_action ignore_index aggregationr7medianminr&kwargsrc stjdi|d|_d}||vrtd|d||_|dur*t|ts*td||_|dur>t|tr:|dks>td||_t|t sJtd||_ |d vs]t |s]td |d ||_ |j d gdd |j dgdd |j dgdd dS)NF)errorskipr6posz7Argument `empty_target_action` received a wrong value `z`.z3Argument `ignore_index` must be an integer or None.rz,`max_k` has to be a positive integer or Nonez `adaptive_k` has to be a booleanr=zArgument `aggregation` must be one of `mean`, `median`, `min`, `max` or a custom callable functionwhich takes tensor of values, but got .r3)defaultdist_reduce_fxr4r5r)super__init__allow_non_binary_targetr(r: isinstanceintr;r8boolr9callabler< add_state)selfr8r9r:r;r<r@empty_target_action_options __class__rr!rHs2  z&RetrievalPrecisionRecallCurve.__init__cCsT|durtdt||||j|jd\}}}|j||j||j|dS)zGCheck shape, check and convert dtypes, flatten and add to accumulators.Nz!Argument `indexes` cannot be None)rIr;)r(r rIr;r3appendr4r5)rOr4r5r3rrr!updates  z$RetrievalPrecisionRecallCurve.updatecst|j}t|jt|j}t|\}}|||}t| }|j }|dur3t |}gg}}t tj |ddtj ||ddD]^\}} | s|jdkrZtd|jdkrv|tj|jd|tj|jdqI|jdkr|tj|jd|tj|jdqIt|| ||j\} } } || || qI|rttfd d |D|jdd nt|} |rttfd d |D|jdd nt|} tjd |d jd} | | | fS)Compute metric.Nr)dimrAzC`compute` method was provided with a query with no positive target.rC)r$r6cg|]}|qSrtorxr4rr! z9RetrievalPrecisionRecallCurve.compute..)r<rVcrWrrXrZr\rr!r]r^)rr3r4r5r)sortrdetachcputolistr8r&r'splitsumr:r(rSonesr$zerosr r9r stackr<rYarange)rOr3r5indices split_sizesr8 precisionsrecalls mini_preds mini_targetrr_rrr\r!computesJ          && z%RetrievalPrecisionRecallCurve.computecurveaxcCs |p|}t||d|jjdS)aPlot a single or multiple values from the metric. Args: curve: the output of either `metric.compute` or `metric.forward`. If no value is provided, will automatically call `metric.compute` and plot that result. ax: An matplotlib axis object. If provided will add plot to that axis Returns: Figure and Axes object Raises: ModuleNotFoundError: If `matplotlib` is not installed .. plot:: :scale: 75 >>> import torch >>> from torchmetrics.retrieval import RetrievalPrecisionRecallCurve >>> # Example plotting a single value >>> metric = RetrievalPrecisionRecallCurve() >>> metric.update(torch.rand(10,), torch.randint(2, (10,)), indexes=torch.randint(2,(10,))) >>> fig_, ax_ = metric.plot() )zFalse positive ratezTrue positive rate)rs label_namesname)rqrrR__name__)rOrrrsrrr!plots r)NFr6Nr7NN)rv __module__ __qualname____doc__r0rL__annotations__r1r2rrrrKstrrr rrrHrTtuplerqrrrw __classcell__rrrQr!r/@sN  M     * 7r/cseZdZdZdZ     ddedeed ed e d eed e d dffdd Z d e e e fffdd Z ddeee ee fdeed efddZZS)RetrievalRecallAtFixedPrecisiona! Compute `IR Recall at fixed Precision`_. As input to ``forward`` and ``update`` the metric accepts the following input: - ``preds`` (:class:`~torch.Tensor`): A float tensor of shape ``(N, ...)`` - ``target`` (:class:`~torch.Tensor`): A long or bool tensor of shape ``(N, ...)`` - ``indexes`` (:class:`~torch.Tensor`): A long tensor of shape ``(N, ...)`` which indicate to which query a prediction belongs .. important:: All ``indexes``, ``preds`` and ``target`` must have the same dimension. .. attention:: Predictions will be first grouped by ``indexes`` and then `RetrievalRecallAtFixedPrecision` will be computed as the mean of the `RetrievalRecallAtFixedPrecision` over each query. As output to ``forward`` and ``compute`` the metric returns the following output: - ``max_recall`` (:class:`~torch.Tensor`): A tensor with the maximum recall value retrieved documents. - ``best_k`` (:class:`~torch.Tensor`): A tensor with the best k corresponding to the maximum recall value Args: min_precision: float value specifying minimum precision threshold. max_k: Calculate recall and precision for all possible top k from 1 to max_k (default: `None`, which considers all possible top k) adaptive_k: adjust `k` to `min(k, number of documents)` for each query empty_target_action: Specify what to do with queries that do not have at least a positive ``target``. Choose from: - ``'neg'``: those queries count as ``0.0`` (default) - ``'pos'``: those queries count as ``1.0`` - ``'skip'``: skip those queries; if all queries are skipped, ``0.0`` is returned - ``'error'``: raise a ``ValueError`` ignore_index: Ignore predictions where the target is equal to this number. kwargs: Additional keyword arguments, see :ref:`Metric kwargs` for more info. Raises: ValueError: If ``empty_target_action`` is not one of ``error``, ``skip``, ``neg`` or ``pos``. ValueError: If ``ignore_index`` is not `None` or an integer. ValueError: If ``min_precision`` parameter is not float or between 0 and 1. ValueError: If ``max_k`` parameter is not `None` or an integer larger than 0. Example: >>> from torch import tensor >>> from torchmetrics.retrieval import RetrievalRecallAtFixedPrecision >>> indexes = tensor([0, 0, 0, 0, 1, 1, 1]) >>> preds = tensor([0.4, 0.01, 0.5, 0.6, 0.2, 0.3, 0.5]) >>> target = tensor([True, False, False, True, True, False, True]) >>> r = RetrievalRecallAtFixedPrecision(min_precision=0.8) >>> r(preds, target, indexes=indexes) (tensor(0.5000), tensor(1)) Tr#NFr6rr8r9r:r;r@rc sTtjd||||d|t|tr!d|kr dks%tdtd||_dS)N)r8r9r:r;r#g?z:`min_precision` has to be a positive float between 0 and 1r)rGrHrJfloatr(r)rOrr8r9r:r;r@rQrr!rHjs  z(RetrievalRecallAtFixedPrecision.__init__cs t\}}}t||||jS)rU)rGrqr.r)rOrlrmrrQrr!rqsz'RetrievalRecallAtFixedPrecision.computevalrscCs|p|d}|||S)aPlot a single or multiple values from the metric. Args: val: Either a single result from calling `metric.forward` or `metric.compute` or a list of these results. If no value is provided, will automatically call `metric.compute` and plot that result. ax: An matplotlib axis object. If provided will add plot to that axis Returns: Figure and Axes object Raises: ModuleNotFoundError: If `matplotlib` is not installed .. plot:: :scale: 75 >>> import torch >>> from torchmetrics.retrieval import RetrievalRecallAtFixedPrecision >>> # Example plotting a single value >>> metric = RetrievalRecallAtFixedPrecision(min_precision=0.5) >>> metric.update(torch.rand(10,), torch.randint(2, (10,)), indexes=torch.randint(2,(10,))) >>> fig_, ax_ = metric.plot() .. plot:: :scale: 75 >>> import torch >>> from torchmetrics.retrieval import RetrievalRecallAtFixedPrecision >>> # Example plotting multiple values >>> metric = RetrievalRecallAtFixedPrecision(min_precision=0.5) >>> values = [] >>> for _ in range(10): ... values.append(metric(torch.rand(10,), torch.randint(2, (10,)), indexes=torch.randint(2,(10,)))[0]) >>> fig, ax = metric.plot(values) r)rq_plot)rOrrsrrr!rws( r)r#NFr6Nrx)rvryrzr{r1rrrKrLr}rrHr~rrqrrrrrwrrrrQr!r)s@>r)#collections.abcrtypingrrrrrr)rtyping_extensionsr torchmetricsr 8torchmetrics.functional.retrieval.precision_recall_curver torchmetrics.retrieval.baser torchmetrics.utilities.checksr torchmetrics.utilities.datarrtorchmetrics.utilities.importsrtorchmetrics.utilities.plotrrr__doctest_skip__rr~r.r/rrrrr!s8          j