o .wiC@sddlmZddlmZmZmZmZddlmZddl m Z ddl m Z ddl mZmZmZmZmZmZmZmZmZddlmZddlmZdd lmZdd lmZdd lm Z m!Z!es`d d gZ"GdddeZ#GdddeZ$Gddde Z%dS))Sequence)AnyListOptionalUnion)Tensor)Literal)_ClassificationTaskWrapper) (_binary_calibration_error_arg_validation+_binary_calibration_error_tensor_validation _binary_calibration_error_update_binary_confusion_matrix_format _ce_compute,_multiclass_calibration_error_arg_validation/_multiclass_calibration_error_tensor_validation$_multiclass_calibration_error_update#_multiclass_confusion_matrix_format)Metric) dim_zero_cat)ClassificationTaskNoMultilabel)_MATPLOTLIB_AVAILABLE)_AX_TYPE_PLOT_OUT_TYPEBinaryCalibrationError.plotMulticlassCalibrationError.plotc seZdZUdZdZeed<dZeed<dZeed<dZ e ed<dZ e ed <e e ed <e e ed <  d#dededdeedededdf fdd Zde de ddfddZde fddZ d$deee ee fd eedefd!d"ZZS)%BinaryCalibrationErrora `Top-label Calibration Error`_ for binary tasks. The expected calibration error can be used to quantify how well a given model is calibrated e.g. how well the predicted output probabilities of the model matches the actual probabilities of the ground truth distribution. Three different norms are implemented, each corresponding to variations on the calibration error metric. .. math:: \text{ECE} = \sum_i^N b_i \|(p_i - c_i)\|, \text{L1 norm (Expected Calibration Error)} .. math:: \text{MCE} = \max_{i} (p_i - c_i), \text{Infinity norm (Maximum Calibration Error)} .. math:: \text{RMSCE} = \sqrt{\sum_i^N b_i(p_i - c_i)^2}, \text{L2 norm (Root Mean Square Calibration Error)} Where :math:`p_i` is the top-1 prediction accuracy in bin :math:`i`, :math:`c_i` is the average confidence of predictions in bin :math:`i`, and :math:`b_i` is the fraction of data points in bin :math:`i`. Bins are constructed in an uniform way in the [0,1] range. As input to ``forward`` and ``update`` the metric accepts the following input: - ``preds`` (:class:`~torch.Tensor`): A float tensor of shape ``(N, ...)`` containing probabilities or logits for each observation. If preds has values outside [0,1] range we consider the input to be logits and will auto apply sigmoid per element. - ``target`` (:class:`~torch.Tensor`): An int tensor of shape ``(N, ...)`` containing ground truth labels, and therefore only contain {0,1} values (except if `ignore_index` is specified). The value 1 always encodes the positive class. As output to ``forward`` and ``compute`` the metric returns the following output: - ``bce`` (:class:`~torch.Tensor`): A scalar tensor containing the calibration error Additional dimension ``...`` will be flattened into the batch dimension. Args: n_bins: Number of bins to use when computing the metric. norm: Norm used to compare empirical and expected probability bins. ignore_index: Specifies a target value that is ignored and does not contribute to the metric calculation validate_args: bool indicating if input arguments and tensors should be validated for correctness. Set to ``False`` for faster computations. kwargs: Additional keyword arguments, see :ref:`Metric kwargs` for more info. Example: >>> from torch import tensor >>> from torchmetrics.classification import BinaryCalibrationError >>> preds = tensor([0.25, 0.25, 0.55, 0.75, 0.75]) >>> target = tensor([0, 0, 1, 1, 1]) >>> metric = BinaryCalibrationError(n_bins=2, norm='l1') >>> metric(preds, target) tensor(0.2900) >>> bce = BinaryCalibrationError(n_bins=2, norm='l2') >>> bce(preds, target) tensor(0.2918) >>> bce = BinaryCalibrationError(n_bins=2, norm='max') >>> bce(preds, target) tensor(0.3167) Fis_differentiablehigher_is_betterfull_state_updateplot_lower_bound?plot_upper_bound confidences accuraciesl1NTn_binsnormr&l2max ignore_index validate_argskwargsreturnc s^tjdi||rt|||||_||_||_||_|jdgdd|jdgdddSNr#cat)dist_reduce_fxr$)super__init__r r-r'r(r, add_state)selfr'r(r,r-r. __class__r3j/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/torchmetrics/classification/calibration_error.pyr5ps zBinaryCalibrationError.__init__predstargetcCsV|jr t|||jt||d|jdd\}}t||\}}|j||j|dS)2Update metric states with predictions and targets.rF) thresholdr,convert_to_labelsN)r-r r,r r r#appendr$r7r;r<r#r$r3r3r:updates   zBinaryCalibrationError.updatecC(t|j}t|j}t|||j|jdSzCompute metric.)r(rr#r$rr'r(r7r#r$r3r3r:compute  zBinaryCalibrationError.computevalaxcC |||S)awPlot a single or multiple values from the metric. Args: val: Either a single result from calling `metric.forward` or `metric.compute` or a list of these results. If no value is provided, will automatically call `metric.compute` and plot that result. ax: An matplotlib axis object. If provided will add plot to that axis Returns: Figure object and Axes object Raises: ModuleNotFoundError: If `matplotlib` is not installed .. plot:: :scale: 75 >>> from torch import rand, randint >>> # Example plotting a single value >>> from torchmetrics.classification import BinaryCalibrationError >>> metric = BinaryCalibrationError(n_bins=2, norm='l1') >>> metric.update(rand(10), randint(2,(10,))) >>> fig_, ax_ = metric.plot() .. plot:: :scale: 75 >>> from torch import rand, randint >>> # Example plotting multiple values >>> from torchmetrics.classification import BinaryCalibrationError >>> metric = BinaryCalibrationError(n_bins=2, norm='l1') >>> values = [ ] >>> for _ in range(10): ... values.append(metric(rand(10), randint(2,(10,)))) >>> fig_, ax_ = metric.plot(values) _plotr7rIrJr3r3r:plot (rr%r&NTNN)__name__ __module__ __qualname____doc__rbool__annotations__rrr floatr"rrintrrrr5rBrGrrrrrO __classcell__r3r3r8r:r*sH  <       rcseZdZUdZdZeed<dZeed<dZeed<dZ e ed<dZ e ed <d Z e ed <eeed <eeed <    d&dedededdeedededdffdd ZdededdfddZdefd d!Z d'd"eeeeefd#eedefd$d%ZZS)(MulticlassCalibrationErrora `Top-label Calibration Error`_ for multiclass tasks. The expected calibration error can be used to quantify how well a given model is calibrated e.g. how well the predicted output probabilities of the model matches the actual probabilities of the ground truth distribution. Three different norms are implemented, each corresponding to variations on the calibration error metric. .. math:: \text{ECE} = \sum_i^N b_i \|(p_i - c_i)\|, \text{L1 norm (Expected Calibration Error)} .. math:: \text{MCE} = \max_{i} (p_i - c_i), \text{Infinity norm (Maximum Calibration Error)} .. math:: \text{RMSCE} = \sqrt{\sum_i^N b_i(p_i - c_i)^2}, \text{L2 norm (Root Mean Square Calibration Error)} Where :math:`p_i` is the top-1 prediction accuracy in bin :math:`i`, :math:`c_i` is the average confidence of predictions in bin :math:`i`, and :math:`b_i` is the fraction of data points in bin :math:`i`. Bins are constructed in an uniform way in the [0,1] range. As input to ``forward`` and ``update`` the metric accepts the following input: - ``preds`` (:class:`~torch.Tensor`): A float tensor of shape ``(N, C, ...)`` containing probabilities or logits for each observation. If preds has values outside [0,1] range we consider the input to be logits and will auto apply softmax per sample. - ``target`` (:class:`~torch.Tensor`): An int tensor of shape ``(N, ...)`` containing ground truth labels, and therefore only contain values in the [0, n_classes-1] range (except if `ignore_index` is specified). .. tip:: Additional dimension ``...`` will be flattened into the batch dimension. As output to ``forward`` and ``compute`` the metric returns the following output: - ``mcce`` (:class:`~torch.Tensor`): A scalar tensor containing the calibration error Args: num_classes: Integer specifying the number of classes n_bins: Number of bins to use when computing the metric. norm: Norm used to compare empirical and expected probability bins. ignore_index: Specifies a target value that is ignored and does not contribute to the metric calculation validate_args: bool indicating if input arguments and tensors should be validated for correctness. Set to ``False`` for faster computations. kwargs: Additional keyword arguments, see :ref:`Metric kwargs` for more info. Example: >>> from torch import tensor >>> from torchmetrics.classification import MulticlassCalibrationError >>> preds = tensor([[0.25, 0.20, 0.55], ... [0.55, 0.05, 0.40], ... [0.10, 0.30, 0.60], ... [0.90, 0.05, 0.05]]) >>> target = tensor([0, 1, 2, 0]) >>> metric = MulticlassCalibrationError(num_classes=3, n_bins=3, norm='l1') >>> metric(preds, target) tensor(0.2000) >>> mcce = MulticlassCalibrationError(num_classes=3, n_bins=3, norm='l2') >>> mcce(preds, target) tensor(0.2082) >>> mcce = MulticlassCalibrationError(num_classes=3, n_bins=3, norm='max') >>> mcce(preds, target) tensor(0.2333) Frrrrr r!r"Classplot_legend_namer#r$r%r&NT num_classesr'r(r)r,r-r.r/c sftjdi||rt||||||_||_||_||_||_|jdgdd|jdgdddSr0) r4r5rr-r_r'r(r,r6)r7r_r'r(r,r-r.r8r3r:r5 s z#MulticlassCalibrationError.__init__r;r<cCsX|jr t|||j|jt|||jdd\}}t||\}}|j||j|dS)r=F)r,r?N) r-rr_r,rrr#r@r$rAr3r3r:rBs   z!MulticlassCalibrationError.updatecCrCrDrErFr3r3r:rG(rHz"MulticlassCalibrationError.computerIrJcCrK)aPlot a single or multiple values from the metric. Args: val: Either a single result from calling `metric.forward` or `metric.compute` or a list of these results. If no value is provided, will automatically call `metric.compute` and plot that result. ax: An matplotlib axis object. If provided will add plot to that axis Returns: Figure object and Axes object Raises: ModuleNotFoundError: If `matplotlib` is not installed .. plot:: :scale: 75 >>> from torch import randn, randint >>> # Example plotting a single value >>> from torchmetrics.classification import MulticlassCalibrationError >>> metric = MulticlassCalibrationError(num_classes=3, n_bins=3, norm='l1') >>> metric.update(randn(20,3).softmax(dim=-1), randint(3, (20,))) >>> fig_, ax_ = metric.plot() .. plot:: :scale: 75 >>> from torch import randn, randint >>> # Example plotting a multiple values >>> from torchmetrics.classification import MulticlassCalibrationError >>> metric = MulticlassCalibrationError(num_classes=3, n_bins=3, norm='l1') >>> values = [] >>> for _ in range(20): ... values.append(metric(randn(20,3).softmax(dim=-1), randint(3, (20,)))) >>> fig_, ax_ = metric.plot(values) rLrNr3r3r:rO.rPrrQrR)rSrTrUrVrrWrXrrr rYr"r^strrrrZrrrr5rBrGrrrrrOr[r3r3r8r:r\sN  @        r\c@s^eZdZdZ     ddeddedd ed ed d eed eedede de fddZ dS)CalibrationErrora`Top-label Calibration Error`_. The expected calibration error can be used to quantify how well a given model is calibrated e.g. how well the predicted output probabilities of the model matches the actual probabilities of the ground truth distribution. Three different norms are implemented, each corresponding to variations on the calibration error metric. .. math:: \text{ECE} = \sum_i^N b_i \|(p_i - c_i)\|, \text{L1 norm (Expected Calibration Error)} .. math:: \text{MCE} = \max_{i} (p_i - c_i), \text{Infinity norm (Maximum Calibration Error)} .. math:: \text{RMSCE} = \sqrt{\sum_i^N b_i(p_i - c_i)^2}, \text{L2 norm (Root Mean Square Calibration Error)} Where :math:`p_i` is the top-1 prediction accuracy in bin :math:`i`, :math:`c_i` is the average confidence of predictions in bin :math:`i`, and :math:`b_i` is the fraction of data points in bin :math:`i`. Bins are constructed in an uniform way in the [0,1] range. This function is a simple wrapper to get the task specific versions of this metric, which is done by setting the ``task`` argument to either ``'binary'`` or ``'multiclass'``. See the documentation of :class:`~torchmetrics.classification.BinaryCalibrationError` and :class:`~torchmetrics.classification.MulticlassCalibrationError` for the specific details of each argument influence and examples. r%r&NTclstask)binary multiclassr'r(r)r_r,r-r.r/cKs|t|}|||||d|tjkrtdi|S|tjkr7t|ts/tdt |dt |fi|Std|)zInitialize task metric.)r'r(r,r-z+`num_classes` is expected to be `int` but `z was passed.`zNot handled value: Nr3) rfrom_strrBBINARYr MULTICLASS isinstancerZ ValueErrortyper\)rbrcr'r(r_r,r-r.r3r3r:__new__us    zCalibrationError.__new__)r%r&NNT) rSrTrUrVrkrrZrrWrrrlr3r3r3r:raYs4 raN)&collections.abcrtypingrrrrtorchrtyping_extensionsr torchmetrics.classification.baser 8torchmetrics.functional.classification.calibration_errorr r r r rrrrrtorchmetrics.metricrtorchmetrics.utilities.datartorchmetrics.utilities.enumsrtorchmetrics.utilities.importsrtorchmetrics.utilities.plotrr__doctest_skip__rr\rar3r3r3r:s$    ,