o yiV@sddlmZddlmZddlmZmZmZmZm Z m Z m Z m Z m Z ddlZddlmZddlmZmZddlmZddlmZdd lmZmZGd d d eZdS) ) OrderedDict)deepcopy) AnyDictHashableIterableListOptionalSequenceTupleUnionN)Tensor)Module ModuleDict)Metric)rank_zero_warn) _flatten_dictallclosecseZdZUdZeeeefed<ddddde e e e eee ffde de ed e ed e e eeefd df fd d Zejjdeded eeeffddZdeded dfddZdFddZede de d e fddZdGde d dfddZd eeeffddZdFd d!ZdHde ed e ed dfd"d#ZdId$e d dfd%d&Zde e e e eee ffde d dfd'd(ZdFd)d*Zed eeeeffd+d,Z d-ed efd.d/Z!d e"fd0d1Z#dGd2e d e$e%fd3d4Z&dJd2e d5e d e$e'ee(ffd6d7Z)dId5e d e$e(fd8d9Z*dId:ed5e d e(fd;d<Z+ed=e ed>ed e efd?d@Z,d effdAdB Z-dCe eej.fd dfdDdEZ/Z0S)KMetricCollectionaMetricCollection class can be used to chain metrics that have the same call pattern into one single class. Args: metrics: One of the following * list or tuple (sequence): if metrics are passed in as a list or tuple, will use the metrics class name as key for output dict. Therefore, two metrics of the same class cannot be chained this way. * arguments: similar to passing in as a list, metrics passed in as arguments will use their metric class name as key for the output dict. * dict: if metrics are passed in as a dict, will use each key in the dict as key for output dict. Use this format if you want to chain together multiple of the same metric with different parameters. Note that the keys in the output dict will be sorted alphabetically. prefix: a string to append in front of the keys of the output dict postfix: a string to append after the keys of the output dict compute_groups: By default the MetricCollection will try to reduce the computations needed for the metrics in the collection by checking if they belong to the same **compute group**. All metrics in a compute group share the same metric state and are therefore only different in their compute step e.g. accuracy, precision and recall can all be computed from the true positives/negatives and false positives/negatives. By default, this argument is ``True`` which enables this feature. Set this argument to `False` for disabling this behaviour. Can also be set to a list of lists of metrics for setting the compute groups yourself. .. note:: The compute groups feature can significatly speedup the calculation of metrics under the right conditions. First, the feature is only available when calling the ``update`` method and not when calling ``forward`` method due to the internal logic of ``forward`` preventing this. Secondly, since we compute groups share metric states by reference, calling ``.items()``, ``.values()`` etc. on the metric collection will break this reference and a copy of states are instead returned in this case (reference will be reestablished on the next call to ``update``). .. note:: Metric collections can be nested at initilization (see last example) but the output of the collection will still be a single flatten dictionary combining the prefix and postfix arguments from the nested collection. Raises: ValueError: If one of the elements of ``metrics`` is not an instance of ``pl.metrics.Metric``. ValueError: If two elements in ``metrics`` have the same ``name``. ValueError: If ``metrics`` is not a ``list``, ``tuple`` or a ``dict``. ValueError: If ``metrics`` is ``dict`` and additional_metrics are passed in. ValueError: If ``prefix`` is set and it is not a string. ValueError: If ``postfix`` is set and it is not a string. Example (input as list): >>> import torch >>> from pprint import pprint >>> from torchmetrics import MetricCollection, MeanSquaredError >>> from torchmetrics.classification import MulticlassAccuracy, MulticlassPrecision, MulticlassRecall >>> target = torch.tensor([0, 2, 0, 2, 0, 1, 0, 2]) >>> preds = torch.tensor([2, 1, 2, 0, 1, 2, 2, 2]) >>> metrics = MetricCollection([MulticlassAccuracy(num_classes=3, average='micro'), ... MulticlassPrecision(num_classes=3, average='macro'), ... MulticlassRecall(num_classes=3, average='macro')]) >>> metrics(preds, target) # doctest: +NORMALIZE_WHITESPACE {'MulticlassAccuracy': tensor(0.1250), 'MulticlassPrecision': tensor(0.0667), 'MulticlassRecall': tensor(0.1111)} Example (input as arguments): >>> metrics = MetricCollection(MulticlassAccuracy(num_classes=3, average='micro'), ... MulticlassPrecision(num_classes=3, average='macro'), ... MulticlassRecall(num_classes=3, average='macro')) >>> metrics(preds, target) # doctest: +NORMALIZE_WHITESPACE {'MulticlassAccuracy': tensor(0.1250), 'MulticlassPrecision': tensor(0.0667), 'MulticlassRecall': tensor(0.1111)} Example (input as dict): >>> metrics = MetricCollection({'micro_recall': MulticlassRecall(num_classes=3, average='micro'), ... 'macro_recall': MulticlassRecall(num_classes=3, average='macro')}) >>> same_metric = metrics.clone() >>> pprint(metrics(preds, target)) {'macro_recall': tensor(0.1111), 'micro_recall': tensor(0.1250)} >>> pprint(same_metric(preds, target)) {'macro_recall': tensor(0.1111), 'micro_recall': tensor(0.1250)} Example (specification of compute groups): >>> metrics = MetricCollection( ... MulticlassRecall(num_classes=3, average='macro'), ... MulticlassPrecision(num_classes=3, average='macro'), ... MeanSquaredError(), ... compute_groups=[['MulticlassRecall', 'MulticlassPrecision'], ['MeanSquaredError']] ... ) >>> metrics.update(preds, target) >>> pprint(metrics.compute()) {'MeanSquaredError': tensor(2.3750), 'MulticlassPrecision': tensor(0.0667), 'MulticlassRecall': tensor(0.1111)} >>> pprint(metrics.compute_groups) {0: ['MulticlassRecall', 'MulticlassPrecision'], 1: ['MeanSquaredError']} Example (nested metric collections): >>> metrics = MetricCollection([ ... MetricCollection([ ... MulticlassAccuracy(num_classes=3, average='macro'), ... MulticlassPrecision(num_classes=3, average='macro') ... ], postfix='_macro'), ... MetricCollection([ ... MulticlassAccuracy(num_classes=3, average='micro'), ... MulticlassPrecision(num_classes=3, average='micro') ... ], postfix='_micro'), ... ], prefix='valmetrics/') >>> pprint(metrics(preds, target)) # doctest: +NORMALIZE_WHITESPACE {'valmetrics/MulticlassAccuracy_macro': tensor(0.1111), 'valmetrics/MulticlassAccuracy_micro': tensor(0.1250), 'valmetrics/MulticlassPrecision_macro': tensor(0.0667), 'valmetrics/MulticlassPrecision_micro': tensor(0.1250)} _groupsNT)prefixpostfixcompute_groupsmetricsadditional_metricsrrrreturncsNt||d|_||d|_||_d|_d|_|j|g|RdS)NrrF) super__init__ _check_argrr_enable_compute_groups_groups_checked_state_is_copy add_metrics)selfrrrrr __class__L/home/ubuntu/.local/lib/python3.10/site-packages/torchmetrics/collections.pyrs zMetricCollection.__init__argskwargscs<fddjdddD}t|}fdd|DS)zIteratively call forward for each metric. Positional arguments (args) will be passed to every metric in the collection, while keyword arguments (kwargs) will be filtered based on the signature of the individual metric. c s,i|]\}}||i|jdiqS)r&)_filter_kwargs.0km)r(r)r&r' s,z,MetricCollection.forward..TF keep_base copy_stateci|] \}}||qSr& _set_namer,r-vr#r&r'r/itemsr)r#r(r)resr&)r(r)r#r'forwardszMetricCollection.forwardcOs|jr0|jD]\}}t||d}|j|i|jdi|q|jr.|d|_dSdS|jdddD]\}}|jdi|}|j|i|q7|jr\| |d|_dSdS)zIteratively call update for each metric. Positional arguments (args) will be passed to every metric in the collection, while keyword arguments (kwargs) will be filtered based on the signature of the individual metric. rFTr0Nr&) r rr;getattrupdater*r! _compute_groups_create_state_refr_merge_compute_groups)r#r(r)_cgm0r.m_kwargsr&r&r'r?s   zMetricCollection.updatec Cst|j} t|jD]@\}}t|jD]+\}}||kr!qt||d}t||d}|||rC|j||j|nqt|j|krMnq t|j|krVnt|j}qt|j}i|_t| D] \} } | |j| <qjdS)aAIterates over the collection of metrics, checking if the state of each metric matches another. If so, their compute groups will be merged into one. The complexity of the method is approximately ``O(number_of_metrics_in_collection ** 2)``, as all metrics need to be compared to all other metrics. TrN) lenrrr;r>_equal_metric_statesextendpop enumeratevalues) r#n_groupscg_idx1 cg_members1cg_idx2 cg_members2metric1metric2tempidxrKr&r&r'rAs0     z&MetricCollection._merge_compute_groupsrQrRcCst|jdkst|jdkrdS|j|jkrdS|jD]F}t||}t||}t|t|kr8dSt|trOt|trO|j|jkoLt||St|t rgt|t rgt ddt ||DSq!dS)z6Check if the metric state of two metrics are the same.rFcss*|]\}}|j|jkot||VqdSN)shaper)r,s1s2r&r&r' s(z8MetricCollection._equal_metric_states..T) rF _defaultskeysr>type isinstancer rVrlistallzip)rQrRkeystate1state2r&r&r'rGs  z%MetricCollection._equal_metric_statesFcopyc Cs|jsJ|jD]A\}}t||d}tdt|D].}t|||}|jD]}t||}t|||r6t|n|q&t|d|rDt|j n|j qq||_dS)zCreate reference between metrics in the same compute group. Args: copy: If `True` the metric state will between members will be copied instead of just passed by reference r _update_countN) r!rr;r>rangerFrZsetattrrrf) r#rdrBrCrDimistatem0_stater&r&r'r@ s   z1MetricCollection._compute_groups_create_state_refcs6ddjdddD}t|}fdd|DS)z5Compute the result for each metric in the collection.cSsi|] \}}||qSr&)computer+r&r&r'r/sz,MetricCollection.compute..TFr0cr3r&r4r6r8r&r'r/ r9r:)r#r<r&r8r'rmszMetricCollection.computecCs@|jdddD]\}}|q|jr|jr|dSdSdS)z'Iteratively call reset for each metric.TFr0N)r;resetrr r@)r#rBr.r&r&r'rn"s    zMetricCollection.resetcCs0t|}|r ||d|_|r||d|_|S)zMake a copy of the metric collection Args: prefix: a string to append in front of the metric keys postfix: a string to append after the keys of the output dict rr)rrrr)r#rrmcr&r&r'clone*s zMetricCollection.clonemodecCs&|jdddD] \}}||qdS)zRMethod for post-init to change if metric states should be saved to its state_dict.TFr0N)r; persistent)r#rqrBr.r&r&r'rr8s zMetricCollection.persistentc Gst|tr|g}t|tr/t|}g}|D]}t|tr|n||q|r.td|dn |r.rtz_ in `compute_groups` argument does not match a metric in the collection. Please make sure that z matches TrscSsi|] \}}|t|gqSr&)strr~r&r&r'r/r9N) r]rr^rJrrKrvr[r )r#r7r}r&r&r'rz|s    z%MetricCollection._init_compute_groupscCs|jS)z@Return a dict with the current compute groups in the collection.)rr8r&r&r'rszMetricCollection.compute_groupsbasecCs8|jdur|n|j|}|jdur|}|S||j}|S)z3Adjust name of metric with both prefix and postfix.N)rr)r#rr|r&r&r'r5s  zMetricCollection._set_namecCs,t}|jD] \}}||||<q|SrU)r_modulesr;r5)r#odr-r7r&r&r'_to_renamed_ordered_dictsz)MetricCollection._to_renamed_ordered_dictr1cCs|r|jS|S)zReturn an iterable of the ModuleDict key. Args: keep_base: Whether to add prefix/postfix on the items collection. )rr[r)r#r1r&r&r'r[s  zMetricCollection.keysr2cCs$|||r |jS|S)a%Return an iterable of the ModuleDict key/value pairs. Args: keep_base: Whether to add prefix/postfix on the collection. copy_state: If metric states should be copied between metrics in the same compute group or just passed by reference )r@rr;r)r#r1r2r&r&r'r;s   zMetricCollection.itemscCs|||jS)zReturn an iterable of the ModuleDict values. Args: copy_state: If metric states should be copied between metrics in the same compute group or just passed by reference )r@rrK)r#r2r&r&r'rKs  zMetricCollection.valuesracCs|||j|S)aRetrieve a single metric from the collection. Args: key: name of metric to retrieve copy_state: If metric states should be copied between metrics in the same compute group or just passed by reference )r@r)r#rar2r&r&r' __getitem__s  zMetricCollection.__getitem__argr|cCs.|dus t|tr |Std|dt|)NzExpected input `z` to be a string, but got )r]rrvr\)rr|r&r&r'rszMetricCollection._check_argcsbtdd}|jr|d|j|jrdnd7}|jr-||js$dndd|j7}|dS)Nz , prefix=,z postfix=z ))r__repr__rr)r#repr_strr$r&r'rs zMetricCollection.__repr__dst_typecCs&|jdddD] \}}||q|S)zTransfer all metric state to specific dtype. Special version of standard `type` method. Arguments: dst_type (type or string): the desired type. TFr0)r; set_dtype)r#rrBr.r&r&r'rs zMetricCollection.set_dtype)rN)F)NN)T)FT)1ry __module__ __qualname____doc__rintrr__annotations__r rr r boolrtorchjitunusedrr=r?rA staticmethodrGr@rmrnrprrr"rzpropertyrr5rrrrr[r rr;rKrrrdtyper __classcell__r&r&r$r'rsd u  $   ?$    $r) collectionsrrdrtypingrrrrrr r r r rr torch.nnrrtorchmetrics.metricrtorchmetrics.utilitiesrtorchmetrics.utilities.datarrrr&r&r&r's  ,