o 6ti@spddlZddlmZmZddlZddZdedeeeeffddZdedeeeeffd d Z d d Z dS) N)ListTuplecKs.|d}|s tdtjd|dd}d|iS)z Load the graphwalks dataset with specific data file. Args: kwargs: Must contain 'data_file' key specifying which parquet file to load Returns: Dictionary with 'train' split containing the dataset data_filez-data_file must be specified in dataset_kwargszopenai/graphwalkstrain) data_filessplit)get ValueErrordatasets load_dataset)kwargsrdatasetrR/home/ubuntu/.local/lib/python3.10/site-packages/lm_eval/tasks/graphwalks/utils.pyr s r responsereturncCsr|ddd}d|vrgdfStd|}|r5|d}|s'gdfSdd |d D}|dfSgdfS) a Extract the answer list from a model response. Args: response: The model's generated response Returns: Tuple of (list of nodes, is_error) - list of nodes: extracted node IDs - is_error: True if parsing failed, False otherwise  z Final Answer:TFinal Answer:\s*\[(.*)\]FcS"g|] }|r|dqSz'"strip.0itemrrr 7  z'extract_answer_list..,)rstriprresearchgroupr)rlinematchbracket_content result_listrrrextract_answer_lists   r(cCst|dd}t|D])}td|}|r5|d}|s%gdfSdd|dD}|dfSq gdfS) a Extract the answer list from a model response (flexible version). Searches backwards through all lines to find "Final Answer:" pattern. More lenient than extract_answer_list which only checks the last line. Args: response: The model's generated response Returns: Tuple of (list of nodes, is_error) - list of nodes: extracted node IDs - is_error: True if parsing failed, False otherwise rrrFcSrrrrrrrrYrz0extract_answer_list_flexible..rT)r rreversedr!r"r#r)rlinesr$r%r&r'rrrextract_answer_list_flexibleAs     r+cCs|d}|d}t|\}}t|}t|}t||@}t|} t|} | dkr,|| nd} | dkr6|| nd} | | dkrHd| | | | nd} t|\}}t|}t||@}t|}| dkrf|| nd}|dkrp||nd}||dkrd||||nd}| |dS)aB Process results and compute set-based F1 scores. Returns both strict F1 (last line only) and flexible F1 (search all lines). Args: doc: Document containing ground truth answer_nodes results: List containing model generation Returns: Dictionary with f1 and flexible_f1 scores r answer_nodesg)f1 flexible_f1)r(setlenr+)docresultsr gold_nodespredicted_nodes_strict_sampled_set_strict truth_setn_overlap_strictn_sampled_strictn_golden recall_strictprecision_strict f1_strictpredicted_nodes_flexiblesampled_set_flexiblen_overlap_flexiblen_sampled_flexiblerecall_flexibleprecision_flexible f1_flexiblerrrprocess_resultsdsB        rF) r!typingrrr r strboolr(r+rFrrrrs& #