o yi@sddlZddlZddlZddlmZddlmZddlmZm Z m Z m Z m Z m Z mZmZddlmZddlZddlmZddlmZmZddlmZejd d Zeejd dZd ed edefddZd ed eddfddZd ed ede de ede e!ddf ddZ"d ed edeee!ffddZ#de!de eddfddZ$d ed ede!de ede!ddf ddZ%de!de ede!ddfd d!Z&d"e!d#e'de!de ed$eddf d%d&Z( dQd ed ede de e!de ed"e e!de e!defd'd(Z)d ed edeeeffd)d*Z* +    dRd ed ede d"e e!de e!de ede e!deeeeffd,d-Z+ + .dSde!d ed ede d/edeeeff d0d1Z, .dTd ed ed2edeeeffd3d4Z- . dUd5ed ed ed2ede e!deeeeff d6d7Z. .dTd ed ed2edeeeffd8d9Z/dVd;edd?Z0eiigd@dAfdBe e'efdCe e'efdDe e!dEe!def dFdGZ1dHe'dIe2dJe2defdKdLZ3efdMe dNe!defdOdPZ4dS)WN)partial) perf_counter)AnyCallableDictMappingOptionalSequenceTuple no_type_check)Mock)Tensor) select_topk to_onehot)DataTypeDOCTEST_DOWNLOAD_TIMEOUTxSKIP_SLOW_DOCTESTpredstargetreturncCs(||krdkrdSdSdS)NrTF)numelrrrQ/home/ubuntu/.local/lib/python3.10/site-packages/torchmetrics/utilities/checks.py_check_for_empty_tensors srcCs*|j|jkrtd|jd|jddS)zHCheck that predictions and target have the same shape, else raise error.zEPredictions and targets are expected to have the same shape, but got z and .N)shape RuntimeErrorrrrr_check_same_shape&s r threshold multiclass ignore_indexcCst||rdS|rtd|dur|dkrtd|dur/|dkr/|dkr/td|}|s?|dkr?td|jd|jdksMtd|dur[|dkr[td |durk|sm|dkrotd dSdSdS) zhPerform basic validation of inputs that does not require deducing any information of the type of inputs.Nz)The `target` has to be an integer tensor.rz-The `target` has to be a non-negative tensor.z6If `preds` are integers, they have to be non-negative.z>The `preds` and `target` should have the same first dimension.FzAIf you set `multiclass=False`, then `target` should not exceed 1.zYIf you set `multiclass=False` and `preds` are integers, then `preds` should not exceed 1.)ris_floating_point ValueErrorminrmax)rrr r!r" preds_floatrrr_basic_input_validation.s$  r)cCsT|}|j|jkrf|j|jkrtdd|jd|jd|r0|dkr0|dkr0td|jdkr;|r;tj}n|jdkrF|sFtj}n|jdkrQ|rQtj }ntj }|dkr`|dnd}||fS|j|jdkr|sttd|jd d |jdd krtd |dkr|jdnd}|jd krtj}||fStj }||fStd ) aThis checks that the shape and type of inputs are consistent with each other and fall into one of the allowed input types (see the documentation of docstring of ``_input_format_classification``). It does not check for consistency of number of classes, other functions take care of that. It returns the name of the case in which the inputs fall, and the implied number of classes (from the ``C`` dim for multi-class data, or extra dim(s) for multi-label data). z4The `preds` and `target` should have the same shape,z got `preds` with shape=z and `target` with shape=rrr#z`If `preds` and `target` are of shape (N, ...) and `preds` are floats, `target` should be binary.zSIf `preds` have one dimension more than `target`, `preds` should be a float tensor.NzIf `preds` have one dimension more than `target`, the shape of `preds` should be (N, C, ...), and the shape of `target` should be (N, ...).zEither `preds` and `target` both should have the (same) shape (N, ...), or `target` should be (N, ...) and `preds` should be (N, C, ...).) r$ndimrr%rr'rBINARY MULTICLASS MULTILABELMULTIDIM_MULTICLASS)rrr(caseimplied_classesrrr!_check_shape_and_type_consistencyLsH    r2 num_classescCs@|dkrtd|dkr|std|dkr|rtddSdS)zgThis checks that the consistency of `num_classes` with the data and `multiclass` param for binary data.r*z8Your data is binary, but `num_classes` is larger than 2.zYour data is binary and `num_classes=2`, but `multiclass` is not True. Set it to True if you want to transform binary data to multi-class format.r#zYou have binary data and have set `multiclass=True`, but `num_classes` is 1. Either set `multiclass=None`(default) or set `num_classes=2` to transform binary data to multi-class format.Nr%)r3r!rrr_check_num_classes_binarys  r5r1cCs|dkr |dur td|dkr:|dur||krtd|dkr,||kr,td|j|jkr<||kr>tddSdSdS) zThis checks that the consistency of `num_classes` with the data and `multiclass` param for (multi- dimensional) multi-class data.r#FzYou have set `num_classes=1`, but predictions are integers. If you want to convert (multi-dimensional) multi-class data with 2 classes to binary/multi-label, set `multiclass=False`.aUYou have set `multiclass=False`, but the implied number of classes (from shape of inputs) does not match `num_classes`. If you are trying to transform multi-dim multi-class data with 2 classes to multi-label, `num_classes` should be either None or the product of the size of extra dimensions (...). See Input Types in Metrics documentation.rzCThe highest label in `target` should be smaller than `num_classes`.z@The size of C dimension of `preds` does not match `num_classes`.N)r%rr'r)rrr3r!r1rrr_check_num_classes_mcs  r6cCs0|r |dkr td|s||krtddSdS)ztThis checks that the consistency of ``num_classes`` with the data and ``multiclass`` param for multi-label data.r*zYour have set `multiclass=True`, but `num_classes` is not equal to 2. If you are trying to transform multi-label data to 2 class multi-dimensional multi-class, you should set `num_classes` to either 2 or None.zPThe implied number of classes (from shape of inputs) does not match num_classes.Nr4)r3r!r1rrr_check_num_classes_mls  r7top_kr0r(cCsr|tjkr tdt|tr|dkrtd|std|dur$td|tjkr/|r/td||kr7tddS) Nz3You can not use `top_k` parameter with binary data.rz/The `top_k` has to be an integer larger than 0.zBYou have set `top_k`, but you do not have probability predictions.Fz7If you set `multiclass=False`, you can not set `top_k`.zIf you want to transform multi-label data to 2 class multi-dimensionalmulti-class data using `multiclass=True`, you can not use `top_k`.zIThe `top_k` has to be strictly smaller than the `C` dimension of `preds`.)rr,r% isinstanceintr.)r8r0r1r!r(rrr _check_top_ks r;c Cst|||||t||\}}|j|jkr+|dur!|dkr!td||kr+td|rR|tjkr8t||n|tjtj fvrIt |||||n |j rRt ||||dur`t ||||||S)a' Performs error checking on inputs for classification. This ensures that preds and target take one of the shape/type combinations that are specified in ``_input_format_classification`` docstring. It also checks the cases of over-rides with ``multiclass`` by checking (for multi-class and multi-dim multi-class cases) that there are only up to 2 distinct labels. In case where preds are floats (probabilities), it is checked whether they are in ``[0,1]`` interval. When ``num_classes`` is given, it is checked that it is consistent with input cases (binary, multi-label, ...), and that, if available, the implied number of classes in the ``C`` dimension is consistent with it (as well as that max label in target is smaller than it). When ``num_classes`` is not specified in these cases, consistency of the highest target value against ``C`` dimension is checked for (multi-dimensional) multi-class cases. If ``top_k`` is set (not None) for inputs that do not have probability predictions (and are not binary), an error is raised. Similarly, if ``top_k`` is set to a number that is higher than or equal to the ``C`` dimension of ``preds``, an error is raised. Preds and target tensors are expected to be squeezed already - all dimensions should be greater than 1, except perhaps the first one (``N``). Args: preds: Tensor with predictions (labels or probabilities) target: Tensor with ground truth labels, always integers (labels) threshold: Threshold value for transforming probability/logit predictions to binary (0,1) predictions, in the case of binary or multi-label inputs. num_classes: Number of classes. If not explicitly set, the number of classes will be inferred either from the shape of inputs, or the maximum label in the ``target`` and ``preds`` tensor, where applicable. top_k: Number of the highest probability entries for each sample to convert to 1s - relevant only for inputs with probability predictions. The default value (``None``) will be interpreted as 1 for these inputs. If this parameter is set for multi-label inputs, it will take precedence over threshold. Should be left unset (``None``) for inputs with label predictions. multiclass: Used only in certain special cases, where you want to treat inputs as a different type than what they appear to be. See the parameter's :ref:`documentation section ` for a more detailed explanation and examples. Return: case: The case the inputs fall in, one of 'binary', 'multi-class', 'multi-label' or 'multi-dim multi-class' Fr*zpYou have set `multiclass=False`, but have more than 2 classes in your data, based on the C dimension of `preds`.z^The highest label in `target` should be smaller than the size of the `C` dimension of `preds`.N)r)r2rr%r'rr,r5r-r/r6r.r7r;r$) rrr r3r!r8r"r0r1rrr_check_classification_inputss*>     r<cCsN|jddkr|d|d}}||fS||}}||fS)zRemove excess dimensions.rr#)rsqueeze unsqueezerrrr_input_squeeze5s r??c Cst||\}}|jtjkr|}t|||||||d}|tjtjfvr2|s2||k }|s0|nd}|tjkr>|r>t ||}|tj tj fvsH|r| rY|jd}t ||pVd}n|r]|n t||d}t|td|}t|td|}|dur|ddddf|ddddf}}t||s|tj tj fvr|dus|r||jd|jdd}||jd|jdd}n||jdd}||jdd}|jdkr|d|d}}| | |fS) aUConvert preds and target tensors into common format. Preds and targets are supposed to fall into one of these categories (and are validated to make sure this is the case): * Both preds and target are of shape ``(N,)``, and both are integers (multi-class) * Both preds and target are of shape ``(N,)``, and target is binary, while preds are a float (binary) * preds are of shape ``(N, C)`` and are floats, and target is of shape ``(N,)`` and is integer (multi-class) * preds and target are of shape ``(N, ...)``, target is binary and preds is a float (multi-label) * preds are of shape ``(N, C, ...)`` and are floats, target is of shape ``(N, ...)`` and is integer (multi-dimensional multi-class) * preds and target are of shape ``(N, ...)`` both are integers (multi-dimensional multi-class) To avoid ambiguities, all dimensions of size 1, except the first one, are squeezed out. The returned output tensors will be binary tensors of the same shape, either ``(N, C)`` of ``(N, C, X)``, the details for each case are described below. The function also returns a ``case`` string, which describes which of the above cases the inputs belonged to - regardless of whether this was "overridden" by other settings (like ``multiclass``). In binary case, targets are normally returned as ``(N,1)`` tensor, while preds are transformed into a binary tensor (elements become 1 if the probability is greater than or equal to ``threshold`` or 0 otherwise). If ``multiclass=True``, then both targets are preds become ``(N, 2)`` tensors by a one-hot transformation; with the thresholding being applied to preds first. In multi-class case, normally both preds and targets become ``(N, C)`` binary tensors; targets by a one-hot transformation and preds by selecting ``top_k`` largest entries (if their original shape was ``(N,C)``). However, if ``multiclass=False``, then targets and preds will be returned as ``(N,1)`` tensor. In multi-label case, normally targets and preds are returned as ``(N, C)`` binary tensors, with preds being binarized as in the binary case. Here the ``C`` dimension is obtained by flattening all dimensions after the first one. However, if ``multiclass=True``, then both are returned as ``(N, 2, C)``, by an equivalent transformation as in the binary case. In multi-dimensional multi-class case, normally both target and preds are returned as ``(N, C, X)`` tensors, with ``X`` resulting from flattening of all dimensions except ``N`` and ``C``. The transformations performed here are equivalent to the multi-class case. However, if ``multiclass=False`` (and there are up to two classes), then the data is returned as ``(N, X)`` binary tensors (multi-label). Note: Where a one-hot transformation needs to be performed and the number of classes is not implicitly given by a ``C`` dimension, the new ``C`` dimension will either be equal to ``num_classes``, if it is given, or the maximum label value in preds and target. Args: preds: Tensor with predictions (labels or probabilities) target: Tensor with ground truth labels, always integers (labels) threshold: Threshold value for transforming probability/logit predictions to binary (0 or 1) predictions, in the case of binary or multi-label inputs. num_classes: Number of classes. If not explicitly set, the number of classes will be inferred either from the shape of inputs, or the maximum label in the ``target`` and ``preds`` tensor, where applicable. top_k: Number of the highest probability entries for each sample to convert to 1s - relevant only for (multi-dimensional) multi-class inputs with probability predictions. The default value (``None``) will be interpreted as 1 for these inputs. Should be left unset (``None``) for all other types of inputs. multiclass: Used only in certain special cases, where you want to treat inputs as a different type than what they appear to be. See the parameter's :ref:`documentation section ` for a more detailed explanation and examples. Returns: preds: binary tensor of shape ``(N, C)`` or ``(N, C, X)`` target: binary tensor of shape ``(N, C)`` or ``(N, C, X)`` case: The case the inputs fall in, one of ``'binary'``, ``'multi-class'``, ``'multi-label'`` or ``'multi-dim multi-class'`` )r r3r!r8r"r*r#FN.r)r?dtypetorchfloat16floatr<rr,r.r:rr-r/r$rr'rrreshaper+r=)rrr r8r3r!r"r0rrr_input_format_classificationAsFZ     &  rGF multilabelcCs|j|j|jdfvrtd|j|jdkrtj|dd}|j|jkr@|jtjtjfvr@|dkr@|s@t||d}t||d}n|j|jkrP|rP||k}|jdkra| dd}| dd}| |d| |dfS)aConvert preds and target tensors into one hot spare label tensors. Args: num_classes: number of classes preds: either tensor with labels, tensor with probabilities/logits or multilabel tensor target: tensor with ground-true labels threshold: float used for thresholding multilabel input multilabel: boolean flag indicating if input is multilabel Raises: ValueError: If ``preds`` and ``target`` don't have the same number of dimensions or one additional dimension for ``preds``. Returns: preds: one hot tensor of shape [num_classes, -1] with predicted labels target: one hot tensors of shape [num_classes, -1] with true labels r#z[preds and target must have same number of dimensions, or one additional dimension for preds)dim)r3rrA) r+r%rCargmaxrBlongr:rr$ transposerF)r3rrr rHrrr$_input_format_classification_one_hots*     rMallow_non_binary_targetcCs:|j|jkr td|r|stdt|||dS)a8Check ``preds`` and ``target`` tensors are of the same shape and of the correct data type. Args: preds: either tensor with scores/logits target: tensor with ground true labels allow_non_binary_target: whether to allow target to contain non-binary values Raises: ValueError: If ``preds`` and ``target`` don't have the same shape, if they are empty or not of the correct ``dtypes``. Returns: preds: as torch.float32 target: as torch.long if not floating point else torch.float32 z.`preds` and `target` must be of the same shapez=`preds` and `target` must be non-empty and non-scalar tensorsrN)rr%rsize,_check_retrieval_target_and_prediction_typesrrrNrrr"_check_retrieval_functional_inputss rSindexescCs|j|jks |j|jkrtd|jtjurtd|dur0||k}||||||}}}|r8|sres1res2atolcs|ttr tj|dSttrkSttr'tddtDSttr:tfdd DSkS)z[Utility function for recursively asserting that two results are within a certain tolerance.)r\css|] \}}t||VqdSN_allclose_recursive).0r1r2rrr tsz&_allclose_recursive..c3s"|] }t||VqdSr]r^)r`krZr[rrrcvs ) r9r rCallclosestrr allziprkeys)rZr[r\rrerr_ls    r_) di init_args input_argsnum_update_to_comparerepsc Cs$Gddd|}Gddd|}|di|}|di|}d} t|dD]&} |di|} z |di|} Wn tyDd} Yn w| t| | @} q&|} z|}Wn tybd} Ynw| t| |@} | srtdd Std t||}t||gD]5\}}t|D],\}}t|D]#}t }t|D] } |di|} qt }||||||f<| qqqt |d }t |d }tt|D]3}td ||d |d|fd|d|fdtd||d |d|fdd|d|fdq|d|dk }td| dd S)aUtility function for checking if the new ``full_state_update`` property can safely be set to ``False`` which will for most metrics results in a speedup when using ``forward``. Args: metric_class: metric class object that should be checked init_args: dict containing arguments for initializing the metric class input_args: dict containing arguments to pass to ``forward`` num_update_to_compare: if we successfully detech that the flag is safe to set to ``False`` we will run some speedup test. This arg should be a list of integers for how many steps to compare over. reps: number of repetitions of speedup test Example (states in ``update`` are independent, save to set ``full_state_update=False``) >>> from torchmetrics.classification import MulticlassConfusionMatrix >>> check_forward_full_state_property( # doctest: +ELLIPSIS ... MulticlassConfusionMatrix, ... init_args = {'num_classes': 3}, ... input_args = {'preds': torch.randint(3, (100,)), 'target': torch.randint(3, (100,))}, ... ) Full state for 10 steps took: ... Partial state for 10 steps took: ... Full state for 100 steps took: ... Partial state for 100 steps took: ... Full state for 1000 steps took: ... Partial state for 1000 steps took: ... Recommended setting `full_state_update=False` Example (states in ``update`` are dependend meaning that ``full_state_update=True``): >>> from torchmetrics.classification import MulticlassConfusionMatrix >>> class MyMetric(MulticlassConfusionMatrix): ... def update(self, preds, target): ... super().update(preds, target) ... # by construction make future states dependent on prior states ... if self.confmat.sum() > 20: ... self.reset() >>> check_forward_full_state_property( ... MyMetric, ... init_args = {'num_classes': 3}, ... input_args = {'preds': torch.randint(3, (10,)), 'target': torch.randint(3, (10,))}, ... ) Recommended setting `full_state_update=True` c@eZdZdZdS)z4check_forward_full_state_property..FullStateTN__name__ __module__ __qualname__full_state_updaterrrr FullStaterxc@rr)z4check_forward_full_state_property..PartStateFNrsrrrr PartStateryrzTrFz,Recommended setting `full_state_update=True`Nr*rAzFull state for z steps took: z+-z0.3fzPartial state for r#)r#rA)rrAz'Recommended setting `full_state_update=`r)rangerr_computeprintrCzeroslen enumeraterresetmeanstditem) metric_classrnrorprqrxrz fullstate partstateequal_out1out2rZr[resimetricjtrstartendrrfasterrrr!check_forward_full_state_propertyzsV3       04r method_nameinstanceparentcCs~t||d}|dur dSt|dr|j}t|tr|j}nt|tr%|j}|dur+dSt||d}|dur9td|j |j kS)zRCheck if a method has been overridden by an instance compared to its parent class.NF __wrapped__z#The parent should define the method) getattrhasattrrr9r _mock_wrapsrfuncr%__code__)rrr instance_attr parent_attrrrr is_overriddens      rfntimeoutcCsRtj|d}||||sdStd|jd||dS)agFunction for checking if a certain function is taking too long to execute. Function will only be executed if running inside a doctest context. Currently does not support Windows. Args: fn: function to check timeout: timeout for function Returns: Bool indicating if the function finished within the specified timeout )rTz running `z`... let's kill it...F) multiprocessingProcessrjoinis_aliveloggingwarningrt terminate)rrprocrrr_try_proceed_with_timeouts  rr])r@NNNN)r@F)F)FN)rY)5rros functoolsrtimertypingrrrrrr r r unittest.mockr rCr torchmetrics.utilities.datarrtorchmetrics.utilities.enumsrenvironget_DOCTEST_DOWNLOAD_TIMEOUTrX_SKIP_SLOW_DOCTESTrrrEr:r)r2r5r6r7rgr;r<r?rGrMrSrWrQr_robjectrrrrrrsD   (    9 & _      4  "  2     g