o 6ti@sddlZddlmZmZddlmZddlmZmZm Z ddl Z z ddl Z e Z Wn ey7edYnweGdddZdeee jffd d Zd edefd d Zde de edefddZdededefddZdededefddZdededefddZde ededeeeffddZde edeeeffddZde edeeeffdd Zde edeeeffd!d"ZdS)#N)Counter defaultdict) dataclass)CallableDictListzOCan not import pymorphy3. If you try to score libra, do `pip install pymorphy3`c@s*eZdZUeed<eeed<eed<dS)PredictionResult pred_answeranswerslengthN)__name__ __module__ __qualname__str__annotations__rrrM/home/ubuntu/.local/lib/python3.10/site-packages/lm_eval/tasks/libra/utils.pyrs   rreturncs|dg|dd}|dd}|dd|dd}ttj|||d d j}|vr;td d |rStj|||d d fd d}d |iStj|||d d }d |iS)zFilter dataset by page lengths for Libra task. in CLI metadata --metadata '{"valid_pages": ["8p", "32p"], "dataset_repo_name": "ai-forever/LIBRA"}' valid_pagesdataset_repo_namezai-forever/LIBRA dataset_nameNfilter_colnamer token)rtestzColumn z not found in dataset cs|vSN)get)docrrrr1sz0filter_dataset_by_page_lengths..)rlistdatasets load_datasetfeatureskeys ValueErrorfilter)argskwargsrrrdataset_columnsdataset_filteredrrrfilter_dataset_by_page_lengthss8     r*sentencecCsTt|}g}|D]}tdd|}t|dj}||q d |S)a@Normalize an input sentence by removing punctuation and converting words to their base (lemmatized) form. :param sentence: str Input sentence. :return: str A normalized sentence where: - All characters except letters, digits, and underscores are removed. - All words are converted to lowercase. - Words are lemmatized using `normalizer`. :raises ValueError: If `sentence` is not a string. :example: >>> normalize_answer("Hello, world! This is a test sentence.") 'hello world this is a test sentence' u[^a-zа-яй0-9_]+r ) rsplitresublower normalizerparse normal_formappendjoin)r+ new_sentencewordrrrrnormalize_answer9s   r9rresultscCs$|d}||d|dd}d|iS)aProcesses evaluation results by extracting prediction and relevant metadata. :param doc: A single instance from the evaluation dataset, containing reference answers and metadata. :param results: A list containing the predicted answer(s). The first element is used as the main prediction. :return: A dictionary where the key is the metric name ("libra_score") and the value is a dictionary with the predicted answer, reference answers, and context length. rpositive_outputsr )r r r libra_scorer)rr: prediction data_dictrrrprocess_resultsQs r?r= ground_truthcCsd}t|t|vr d}|S)N?r9)r=r@resultrrrexact_match_scoredsrEcCs`t|t|@}t|}|dkrdSd|t|}d|t|}d||||}|S)NrrB)rsumvalueslen)r=r@commonnum_same precisionrecallf1rrrf1_scoreks rOcCsVtd|}d}|D]}t|t|kr|d7}q t|dkr!dn|t|}t|S)Nz\d+rrA)r/findallrrIfloat)r=r@numbers right_numnumber final_scorerrr count_scorevs rWscoring_functionc Cstdd}|D]<}|d}t|d}tdd|dD}||dd7<|D]}|||d }|d krC||d |7<nq+qd d |DS) a9Aggregates score by 'length' by scoring_function. :param results: List of dictionaries containing 'pred_answer', 'answers', and 'length'. :return: Dictionary with 'length' as keys and average score as values. :example: >>> results = [ ... {"pred_answer": "1", "answers": ["1", "one"], "length": "8p"}, ... {"pred_answer": "0", "answers": ["zero", "none"], "length": "8p"}, ... {"pred_answer": "one", "answers": ["1", "one"], "length": "16p"} ... ] >>> aggregate_results(results=results) {'8p': 0.5, '16p': 1.0} cSsddgS)Nrrrrrrrsz#aggregate_results..r r cSsg|]}t|qSrrC).0textrrr sz%aggregate_results..r rP)r=r@rcSsi|] \}\}}|||qSrr)rYkeycorrecttotalrrr sz%aggregate_results..)rr9setitems) r:rXscoresrDr r r answermetricrrraggregate_resultss   recC t|tSr)rerEr:rrraggregate_results_em rhcCrfr)rerOrgrrraggregate_results_f1rirjcCrfr)rerWrgrrraggregate_results_count_scorerirk)r/ collectionsrr dataclassesrtypingrrrr pymorphy3 MorphAnalyzerr2 ImportErrorprintrrDatasetr*r9r?rRrErOrWrerhrjrkrrrrs>       !"