o }o™i“)ã@sdddlmZmZmZmZddlZddlmZddlm Z ddgZ Gdd„deƒZ Gdd„de ƒZ dS) é)ÚAnyÚDictÚOptionalÚTupleN)ÚMetric)Ú METRIC_EPSÚClassificationReportÚMultiLabelClassificationReportc sŒeZdZdZdZ    ddedeeefded ed e e f ‡fd d „ Z d e j de j ddfdd„Zdee j e j e j e j ffdd„Z‡ZS)raã This metric computes the number of True Positive, False Negative, and False Positive examples per class. When doing distributed training/evaluation the result of res=ClassificationReport(predictions, labels) calls will be all-reduced between all workers using SUM operations. If used with PytorchLightning LightningModule, include TPs, FNs, and FPs inside validation_step results. Then aggregate them at the end of validation epoch to correctly compute validation precision, recall, f1 using get_precision_recall_f1(). Example: def validation_step(self, batch, batch_idx): ... tp, fn, fp, _ = self.classification_report(preds, labels) return {'val_loss': val_loss, 'tp': tp, 'fn': fn, 'fp': fp} def on_validation_epoch_end(self): ... # calculate metrics and classification report precision, recall, f1, report = self.classification_report.compute() logging.info(report) self.log('val_loss', avg_loss, prog_bar=True) self.log('precision', precision) self.log('f1', f1) self.log('recall', recall) Args: num_classes: number of classes in the dataset label_ids (optional): label name to label id mapping mode: how to compute the average dist_sync_on_step: sync across ddp process_group: which processes to sync across Return: aggregated precision, recall, f1, report TNÚmacroFÚ num_classesÚ label_idsÚmodeÚdist_sync_on_stepÚ process_groupcs tƒj||d||_|rdd„| ¡Dƒ|_nd|_||_|jdt |¡ddd|jdt |¡ddd|jd t |¡ddd|jd t |¡ddddS) N)rrcSsi|]\}}||“qS©r)Ú.0ÚkÚvrrúf/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/nlp/metrics/classification_report.pyÚ Lsz1ClassificationReport.__init__..ÚtpÚsumF)ÚdefaultÚdist_reduce_fxÚ persistentÚfnÚfpÚnum_examples_per_class) ÚsuperÚ__init__r ÚitemsÚ ids_to_labelsr Ú add_stateÚtorchÚzeros)Úselfr r r rr©Ú __class__rrrAs ÿzClassificationReport.__init__Ú predictionsÚlabelsÚreturnc Csèg}g}g}t|jƒD]+}||k}||k}| ||k| ¡¡| ||k| ¡¡| ||k| ¡¡q t |¡ |j¡} t |¡ |j¡} t |¡ |j¡} | | } |j| 7_|j | 7_ |j | 7_ |j | 7_ dS)á  Updates attributes needed for new classification report (true positive, false negative, false postive, examples per class) Args: predictions: predicted labels labels: actual labels Return: None N) Úranger Úappendrr#ÚtensorÚtoÚdevicerrrr) r%r(r)ÚTPÚFNÚFPÚlabel_idÚ current_labelÚlabel_predictedrrrrrrrÚupdateXs" zClassificationReport.updatec CsŽt |j¡}t |j¡ d¡}t |jd|j|jt¡}t |jd|j|j t¡}t d||||t¡}d  ddddd ¡}t t |jƒƒD].}d |›}|j re||j vre|j |›d |›d }|d   ||||||||j|¡7}qKt t |j¡dt |j|j¡t¡} t t |j¡dt |j|j ¡t¡} t d| | | | t¡} t |¡|} t |¡|} t |¡|}t ||j¡|}t ||j¡|}t ||j¡|}|d7}|d   d| | | |¡7}|d   d| | ||¡7}|d   d||||¡d7}||_|jdkr| | ||fS|jdkr'||||fS|jdkr3| | | |fS|jdkr?||||fSt|j›dƒ‚)zì Aggregates and then calculates logs classification report similar to sklearn.metrics.classification_report. Typically used during epoch_end. Return: aggregated precision, recall, f1, report rédéz+ {:50s} {:10s} {:10s} {:10s} {:10s}ÚlabelÚ precisionÚrecallÚf1Úsupportz label_id: z (ú)z/ {:50s} {:8.2f} {:8.2f} {:8.2f} {:8.0f}z -------------------z micro avgz macro avgz weighted avgÚ r ÚweightedÚmicroÚallzs mode is not supported. Choose "macro" to get aggregated numbers or "all" to get values for each class.)r#rrÚnonzeroÚsizeÚ true_dividerrrrÚformatr,Úlenr!Útotal_examplesr Ú ValueError)r%rIÚnum_non_empty_classesr;r<r=ÚreportÚir:Úmicro_precisionÚ micro_recallÚmicro_f1Úmacro_precisionÚ macro_recallÚmacro_f1Úweighted_precisionÚweighted_recallÚ weighted_f1rrrÚcomputexs`  ÿ** ÿ ÿ ÿýÿ         ÿzClassificationReport.compute)Nr FN)Ú__name__Ú __module__Ú __qualname__Ú__doc__Úfull_state_updateÚintrÚstrÚboolrrrr#ÚTensorr7rrWÚ __classcell__rrr&rrs(&úþ ýüûú* c@s*eZdZdZdejdejddfdd„ZdS)r a This metric computes the number of True Positive, False Negative, and False Positive examples per class for a multi-label dataset. When doing distributed training/evaluation the result of res=ClassificationReport(predictions, labels) calls will be all-reduced between all workers using SUM operations. If used with PytorchLightning LightningModule, include TPs, FNs, and FPs inside validation_step results. Then aggregate them at the end of validation epoch to correctly compute validation precision, recall, f1 using get_precision_recall_f1(). Example: def validation_step(self, batch, batch_idx): ... tp, fn, fp, _ = self.classification_report(preds, labels) return {'val_loss': val_loss, 'tp': tp, 'fn': fn, 'fp': fp} def on_validation_epoch_end(self): ... # calculate metrics and classification report precision, recall, f1, report = self.classification_report.compute() logging.info(report) self.log('val_loss', avg_loss, prog_bar=True) self.log('precision', precision) self.log('f1', f1) self.log('recall', recall) Args: num_classes: number of classes in the dataset label_ids (optional): label name to label id mapping mode: how to compute the average dist_sync_on_step: sync across ddp process_group: which processes to sync across Return: aggregated precision, recall, f1, report r(r)r*Nc Csü| ¡}g}g}g}t|jƒD]1}||}||}| ||k|dk ¡¡| ||k|dk ¡¡| ||k|dk ¡¡qt |¡ |j¡} t |¡ |j¡} t |¡ |j¡} | | } |j | 7_ |j | 7_ |j | 7_ |j | 7_ dS)r+éN) Útr,r r-rr#r.r/r0rrrr) r%r(r)r1r2r3r4r5Úlabels_predictedrrrrrrrr7æs$ z%MultiLabelClassificationReport.update)rXrYrZr[r#r`r7rrrrr ½s() Útypingrrrrr#Ú torchmetricsrÚtorchmetrics.utilities.datarÚ__all__rr rrrrÚs  &