o }oid:@sdZddlZddlZddlmZedgdZdZdZ dZ d Z d Z d Z d Zd ZdZdZdZdZdZdZdZdZdZdZdZdZdZdZddZddZd d!Z d"d#Z!d$d%Z"d&d'Z#d(d)Z$d*d+Z%dS),aEvaluation metrics for Schema-guided dialogue. This library provides functions for calculating the evaluation metrics for a single dialogue. The following metrics are defined: (1) Active intent accuracy: The fraction of user turns for which the active intent has been correctly predicted. (2) Slot tagging F1: The macro-averaged F1 score for tagging slot values for non-categorical slots. This metric is optional to report in the final paper if participants decide not to use slot tagging. (3) Requested slots F1: The macro-averaged F1 score for requested slots over the turns. For a turn, if there are no requested slots in both the ground truth and the prediction, that turn is skipped. The reported number is the average F1 score for all un-skipped user turns. This metric is optional to report in the final paper. (4) Average goal accuracy: For each turn, participants must predict a single value for each slot present in the dialogue state. The slots which have a non-empty assignment in the ground truth dialogue state are only considered. This is the average accuracy of predicting the value of a slot correctly. A fuzzy matching based score is used for non-categorical slots. (5) Joint goal accuracy: This is the average accuracy of predicting all slot assignments for a turn correctly. A fuzzy matching based score is used for non-categorical slots. This is the primary evaluation metric used for ranking submissions. More details to follow with the evaluation script. This file contains code artifacts adapted from the original implementation: https://github.com/google-research/google-research/blob/master/schema_guided_dst/metrics.py N)fuzzF1Scoresf1 precisionrecallactive_intent_accuracyslot_tagging_f1slot_tagging_precisionslot_tagging_recallrequested_slots_f1requested_slots_precisionrequested_slots_recallaverage_goal_accuracyaverage_cat_accuracyaverage_noncat_accuracyjoint_goal_accuracyjoint_cat_accuracyjoint_noncat_accuracyaverage_cat_status_accuracyaverage_cat_value_accuracyaverage_noncat_status_accuracyaverage_noncat_value_accuracyjoint_cat_status_accuracyjoint_cat_value_accuracyjoint_noncat_status_accuracyjoint_noncat_value_accuracyNAc Cst|}t|}t|}t|}t||@}|r&t||nd}|r0t||nd}||dkrCd||||} nd} t| ||dS)aCompute F1 score from reference (grouth truth) list and hypothesis list. Args: list_ref: List of true elements. list_hyp: List of postive (retrieved) elements. Returns: A F1Scores object containing F1, precision, and recall scores. ?g@r) collectionsCountersumvaluesfloatr) list_reflist_hyprefhyptruepositive true_positiverrrr,\/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/nlp/metrics/sgd_metrics.py compute_f1Ws    r.cCst||dS)zReturns fuzzy string similarity score in range [0.0, 1.0]. Args: str_ref: reference string str_hyp: hypothesis string Returns: fuzzy string similarity gY@)rtoken_sort_ratio)str_refstr_hypr,r,r-fuzzy_string_matchos r2cCs8d}|D]}|rt||}nt||k}t||}q|S)a/Calculate non-categorical slots correctness. Args: str_ref_list: a list of reference strings. str_hyp: the hypothesis string. use_fuzzy_match: whether to use fuzzy string matching. Returns: score: The highest fuzzy string match score of the references and hypotheis. r)r2r$max) str_ref_listr1use_fuzzy_matchscorer0 match_scorer,r,r-noncat_slot_value_match|s    r8cCshg}g}g}g}g}|dD]z} | d} || d| |vre|d| |vrU|| } || d} | dr?t| d| k} nt| | |} || |d|| q|d|d|dq|d | |vry|d|dn |d|d|dqt|t|dksJt|t|dksJt|t|dksJ|||||fS) aQCompare and get correctness of goal state's slot_values. Args: slot_values_ref: goal state slot_values from reference (ground truth). slot_values_hyp: goal state slot_values from hypothesis (prediction). service: a service data structure in the schema. We use it to obtain the list of slots in the service and infer whether a slot is categorical. use_fuzzy_match: whether to use fuzzy string matching for non-categorical slot values Returns: list_cor: list of corectness scores, each corresponding to one slot in the service. The score is a float either 0.0 or 1.0 for categorical slot, and in range [0.0, 1.0] for non-categorical slot. slot_active: list indicating whether the element in list_cor corresponds to an active ground-truth slot. slot_cat: list indicating whether the element in list_cor corresponds to a categorical slot. list_cor_status: list of correct slot statuses list_cor_value: list of correctness score only for active slots. Monactive slots are assigned -1. slotsnameis_categoricalTrrrgF)appendr$r8len)slot_values_refslot_values_hypservicer5list_corlist_cor_statuslist_cor_value slot_activeslot_catslot slot_namevalue_ref_list value_hypcorr,r,r-compare_slot_valuessB                rKcCst|dd|ddkS)aGet active intent accuracy of a frame. Args: frame_ref: single semantic frame from reference (ground truth) file. frame_hyp: single semantic frame from hypothesis (prediction) file. Returns: 1.0 if the intent prediction is correct, otherwise 0.0. state active_intent)r$ frame_ref frame_hypr,r,r-get_active_intent_accuracys rQcsXdd|dDd|vrdSfdd|dD}fdd|dD}t||S)aGet slot tagging (non-categorical slots only) F1 scores of a frame. Args: frame_ref: single semantic frame from reference (ground truth) file. frame_hyp: single semantic frame from hypothesis (prediction) file. utt: user utterance. Slot tagging annotations are the character positions in the utterance. service: a service data structure in the schema. We use it to infer whether a slot is non-categorical. Returns: A F1Scores object containing F1, precision, and recall scores. cSsg|] }|ds|dqS)r;r:r,.0sr,r,r- sz'get_slot_tagging_f1..r9Nc4g|]}|dvr|d|d|dfqSrFstart exclusive_endr,rRlist_noncat_slotsuttr,r-rU  crVrWr,rRrZr,r-rUr]r.)rOrPr\r@r%r&r,rZr-get_slot_tagging_f1s   r_cCst|dd|ddS)a!Get requested slots F1 scores of a frame. Args: frame_ref: single semantic frame from reference (ground truth) file. frame_hyp: single semantic frame from hypothesis (prediction) file. Returns: A F1Scores object containing F1, precision, and recall scores. rLrequested_slotsr^rNr,r,r-get_requested_slots_f1s racCszi}t|dd|dd||\}}}}} ddt||D} | r't| nt|t<ddt|||D} | r=t| nt|t<ddt|||D} | rSt| nt|t<|r^t|nt|t <ddt||D} | rst| nt|t <ddt||D}|rt|nt|t <d dt|||D}|rt|nt|t <d dt||D}|rt|nt|t <d dt|||D}|rt|nt|t<d dt||D}|rt|nt|t<d dt| ||D}|rt|nt|t<ddt| |D}|r t|nt|t<ddt| ||D}|r!t|nt|t<ddt| |D}|r7t|nt|t<|S)aqGet average and joint goal accuracies of a frame. Args: frame_ref: single semantic frame from reference (ground truth) file. frame_hyp: single semantic frame from hypothesis (prediction) file. service: a service data structure in the schema. We use it to obtain the list of slots in the service and infer whether a slot is categorical. use_fuzzy_match: whether to use fuzzy string matching for comparing non-categorical slot values. Returns: goal_acc: a dict whose values are average / joint all-goal / categorical-goal / non-categorical-goal accuracies. rL slot_valuescSg|]\}}|r|qSr,r,)rSaccactiver,r,r-rUz7get_average_and_joint_goal_accuracy..cSsg|] \}}}|r|r|qSr,r,rSrdrecatr,r,r-rU!cSsg|] \}}}|r|s|qSr,r,rgr,r,r-rU$ricSrcr,r,rSrdrhr,r,r-rU*rfcSg|]\}}|s|qSr,r,rjr,r,r-rU-rfcSsg|] \}}}|r|r|qSr,r,rgr,r,r-rU2ricSrcr,r,rjr,r,r-rU5rfcSsg|] \}}}|s|r|qSr,r,rgr,r,r-rU9s cSrkr,r,rjr,r,r-rU@rfcSs&g|]\}}}|r|dkr|r|qSgr,rgr,r,r-rUD cSs g|] \}}|r|dkr|qSrlr,rjr,r,r-rUI cSs&g|]\}}}|s|dkr|r|qSrlr,rgr,r,r-rUMrmcSs g|] \}}|s|dkr|qSrlr,rjr,r,r-rURrn)rKzipnpmeanNAN_VALAVERAGE_GOAL_ACCURACYAVERAGE_CAT_ACCURACYAVERAGE_NONCAT_ACCURACYprodJOINT_GOAL_ACCURACYJOINT_CAT_ACCURACYJOINT_NONCAT_ACCURACYAVERAGE_CAT_STATUS_ACCURACYJOINT_CAT_STATUS_ACCURACYAVERAGE_NONCAT_STATUS_ACCURACYJOINT_NONCAT_STATUS_ACCURACYAVERAGE_CAT_VALUE_ACCURACYJOINT_CAT_VALUE_ACCURACYAVERAGE_NONCAT_VALUE_ACCURACYJOINT_NONCAT_VALUE_ACCURACY)rOrPr@r5goal_acclist_accrDrElist_status_acclist_value_acc active_accactive_cat_accactive_noncat_acccat_acc noncat_accactive_cat_status_acccat_status_accactive_noncat_status_accnoncat_status_accactive_cat_val_acc cat_val_accactive_noncat_val_accnoncat_val_accr,r,r-#get_average_and_joint_goal_accuracysN   r)&__doc__r numpyrp rapidfuzzr namedtuplerACTIVE_INTENT_ACCURACYSLOT_TAGGING_F1SLOT_TAGGING_PRECISIONSLOT_TAGGING_RECALLREQUESTED_SLOTS_F1REQUESTED_SLOTS_PRECISIONREQUESTED_SLOTS_RECALLrsrtrurwrxryrzr~r|rr{rr}rrrr.r2r8rKrQr_rarr,r,r,r-sF  @