o 6ti@sfddlZddlmZmZmZejdkrejnejZddZ ddZ dd Z d d Z d d Z ddZdS)N)f1_scoreprecision_score recall_scorez2.0.0cCstdd|D}tdd|D}|dkr"tdd|Dd}n tdd|D|}|dkr=tdd|Dd}n td d|D|}d}||dkrZd ||||}|S) a_ Custom aggregation to compute corpus level metrics for the lexical substitution task predictions is a list of tuples (prec, has_answ, has_annotation) prec is the precision before dividing by |A| has_answ is 0 if the model did not produce any answer has_annotation is 0 if the gold answer is empty: no synonims from annotators cSg|]}|dqS).0prrU/home/ubuntu/.local/lib/python3.10/site-packages/lm_eval/tasks/evalita_llm/metrics.py z_aggreg_ls..cSr)rrrrr r r rcSrrrrrrr r r rcSrrrrrrr r r cSrrrrrrr r r cSrrrrrrr r r r)sum) predictionsATprecrecf1rrr _aggreg_ls s  rcCst|\}}t||dd}|S)a  This aggregation considers the sentiment analysis task as a multiple choice one with four classes the f1 score is computed as the average of the f1 scores for each class weighted by the number of samples See sklearn.metrics.f1_score for more details weightedaverage)zipr)r referencesrrrr _aggreg_sa_v2%s rcCst|\}} dd}||\}}||\}}t||dd}t||ddd}t|dkr2|d} nd} t||ddgdd\} } t||ddgdd\} } t||dd}t||ddd}t|dkrlt||ddd}nd}|| d}||d}||d}|S) a Custom aggregation function for the sentiment analysis task The original tasks compute the F1 score for each class and then average them Since the prompt cast the task to a multple choice one we need to aggregate the results in a different way cSsg}g}|D]?}|dkr|d|dq|dkr&|d|dq|dkr5|d|dq|dkrD|d|dq q||fS)zf Return two separate list of labels for opos and oneg x is a list of integers rrr)append)xoposonegirrr _map_to_original_labels?s"        z+_aggreg_sa.._map_to_original_labelsNrrr)labelsrr)rrlenrr)rrr$ pred_opos pred_onegref_oposref_onegopos_f1 opos_f1_c0 opos_f1_c1 oneg_prec_c0 oneg_prec_c1 oneg_rec_c0 oneg_rec_c1oneg_f1 oneg_f1_c0 oneg_f1_c1 f1_score_opos f1_score_onegf1_finalrrr _aggreg_sa1s2             r8c Cst|\}}g}|D]}||q g}|D]}||qt||dd}t|dkr;t|ddt|d}|S|d}|S)Nrrr)rextendrr&r) rpredrefall_predr all_refrrf1_sumrrr _aggreg_ner|s    rAcCsNt|\}}g}|D]}||q g}|D]}||qt||dd}|S)Nmacror)rr:r)rr;r<r=r r>r?rrrr _aggreg_rels   rCcCs.tt|}|d}|d}t||dd}|S)NrrrBr)listrr)items unzipped_listgoldspredsfscorerrr _aggreg_dds rJ)torchsklearn.metricsrrr __version__inference_modeno_gradinference_decoratorrrr8rArCrJrrrr s K