o }oi%@srddlZddlZddlmZddlmZmZmZmZddl m Z ddl Z ddlmZmZmZmZmZmZmZdeeee jfdeeee jfdefdd Zdeeee jfdeeee jfdefd d Zdeeee jfdeeee jfdefd d Zdeeee jfdeeee jfdefddZ  d.deeee jfdeeee jfdededeeeeeeeeeffff ddZ   d/deeee jfdeeee jfdedededeeeeeeeeeffeeeefeeeeeeeeeffff ddZd0deeee jfdee efde fddZ! d1deeee jfdeeee jfdee efde fdd Z" !d2deeee jfdeeee jfdee efde fd"d#Z# $d3deeee jfdeeee jfdee efde fd%d&Z$ '  d4d(eeee jfd)eeee jfdee efde d*ee d+ee f d,d-Z%dS)5N)Path)ListOptionalTupleUnion)PrecisionRecallDisplayRocCurveDisplayaverage_precision_scorelog_lossprecision_recall_curve roc_auc_score roc_curvey_truey_scorereturncCsrt|}t|}t|t|ksJt|dkr"t|dks$Jt|dks2t|dkr4dSt||S)zCompute Area Under the Receiver Operating Characteristic Curve (ROC AUC) from prediction scores. Note: If only one class is present in y_true, 0.5 is returned. rg?)nparraylenallr rrrg/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/asr/parts/utils/confidence_metrics.pyauc_roc!s    rcCsdt|}t|}t|t|ksJt|dkr"t|dks$Jt|dkr-dSt||S)zCompute Area Under the Precision-Recall Curve (PR AUC) from prediction scores. Note: If only regatives are present in y_true, 0.0 is returned. rrrrrrr rrrrauc_pr/s    rcCslt|}t|}t|t|ksJt|dkr"t|dks$Jt|dkr-dStd|d|S)aCompute Area Under the Negative Predictive Value vs. True Negative Rate Curve (NT AUC) from prediction scores. This metric can be thought of as a PR AUC in which errors are treated as positives. Note: If only positives are present in y_true, 0.0 is returned. rrrrrrrrauc_nt=s   rcCst|}t|}t|t|ksJt|dkr"t|dks$Jt|dks2t|dkr6tj S|}d}t|||td||d| }|t|||S)aCompute Normalized Cross Entropy (NCE) from prediction scores. Also known as the Normalized Mutual Information. NCE measures how close the correct prediction scores are to one and the incorrect prediction scores are to zero. Negative NCE values indicate that the classifier performs worse than the setting all prediction scores as the proportion of correct predictions. Note: If only one class is present in y_true, 0.5 is returned. rrgV瞯<) rrrrmathinfmeanlogr )rrpepsHprrrnceMs  .r%dFn_bins return_curvec Cst|}t|}t|t|ksJt|dkr"t|dks$Jtd||gj}t|t|}}t|}g}g} t|D]s} | || d|} } | | d} | | |jd| kt }|t t||f}|| k|| k@ d}t||| <|| dkr||||k || || <|| || || <||| t || || qFt |t |}|r|| |ffS|S)aCompute Expected Calibration Error (ECE) from prediction scores. ECE measures how close the correct prediction scores are to one and the incorrect prediction scores are to zero. ECE ranges from zero to one with the best value zero (the lower the value, the better). rr)rrrrTzerosrangeappendastypeintarangenonzerosumabs)rrr'r(pyaccconfBm ece_curve thresholdsmab thresholdpy_indexpy_value bin_rangeecerrrrAbs2         & rAreturn_std_maximumcCst|}t|}tdd|d}t|t|ksJt|dkr+t|dks-Jt|dks;t|dkr_|rL|rLddd|tt|ffS|rPdS|r]d|tt|ffSdS|dk}tt|dd}tt||d}||} ||} g} |D]#} t| | kd|} t| | kd|}| t | |qt| } |r|r| | | || ffS|r| | | fS|r| || ffS| S)aYCompute Area Under the Youden's Curve (YC AUC) from prediction scores. YC AUC represents the rate of the effective threshold range. If return_std_maximum is set to True, std and maximum values of the Youden's Curve are returned with the AUC. Note: If only one class is present in y_true, zeroes are returned for every entity. rrr)rrr) rrlinspacerrr+maxr1r-r3r std)rrr'rBr(r9 mask_correct count_correctcount_incorrecty_score_correcty_score_incorrectycr=tnrfnrrrrauc_ycs>     rNhistplot_dirnamecCsltj|ddtjt|dddt|tdtdtj t |t |dd d t dS) NTexist_ok2)rr)r,zConfidence scoreCount.png,dpi) osmakedirspltrOrrtitlexlabelylabelsavefigrclf)rrPrQrrrsave_confidence_hists    rbroccCt|t|ks Jtj|ddtdt|dt|\}}}t||dt |tj t |t |dddt dS)NTrRr)fprtprrVrWrX) rrZr[r rrrplotr\r]r`rra)rrrPrQrerf_rrrsave_roc_curve$  riprcCst|t|ks Jtj|ddtt|t|\}}}t||dt |tj t |t |dddt dS)NTrR precisionrecallrVrWrX rrZr[r rrrrgr\r]r`rrarrrPrQrmrnrhrrr save_pr_curves  rqntcCrd)NTrRrrlrVrWrXrorprrr save_nt_curverjrsmy_awesome_curver9valuesr^r_cCst|t|ks Jtj|ddt||tddgtddgt||dur3t||durs  $2 222" *  .7