o 6tiM@sdZddlZddlmZddlmZmZmZz ddlm Z dZ Wne y/dZ e dYnwz dd l mZ dZWne yIdZe d Ynwd ed efd dZdeedeed eeeffddZdeedeed eeeffddZdeeefdeed eeeffddZded efddZdeeefd efddZdeeefd eefddZd d!Zd"ed efd#d$ZdS)%zD Utility functions for CNN/DailyMail summarization task evaluation. N) defaultdict)AnyDictList) rouge_scorerTFzIWarning: rouge_score not installed. Install with: pip install rouge-score)scorezGWarning: bert_score not installed. Install with: pip install bert-scoretextreturncCstdd|}|}|S)z Normalize text for evaluation by removing extra whitespace and lowercasing. Args: text: Input text string Returns: Normalized text string \s+ )resubstrip)rrU/home/ubuntu/.local/lib/python3.10/site-packages/lm_eval/tasks/cnn_dailymail/utils.pynormalize_texts r predictions referencesc CstsddddStjgddd}g}g}g}t||D]*\}}t|}t|}|||}||dj||dj||djq|rQt|t |nd|r\t|t |nd|rit|t |dSddS)z Calculate ROUGE scores for a list of predictions and references. Args: predictions: List of generated summaries references: List of reference summaries Returns: Dictionary with rouge1, rouge2, and rougeL scores rouge1rouge2rougeLT use_stemmerrrr) ROUGE_AVAILABLEr RougeScorerziprrappendfmeasuresumlen) rrscorer rouge1_scores rouge2_scores rougeL_scorespredrefscoresrrrcalculate_rouge_scores,s&  r)cCshtsddddSdd|D}dd|D}t||dddd \}}}|||dS) z Calculate BERTScore for a list of predictions and references. Args: predictions: List of generated summaries references: List of reference summaries Returns: Dictionary with precision, recall, and F1 scores rbertscore_precisionbertscore_recall bertscore_f1cSg|]}t|qSrr).0prrr gz'calculate_bertscore..cSr.rr/)r0rrrrr2hr3endistilbert-base-uncasedFlang model_typeverbose)BERTSCORE_AVAILABLE bert_scoremeanitem)rrPRF1rrrcalculate_bertscoreRs$   rBdocresultsc Cs|rt|dkr d}n t|tr|dn|}|dd}t|}t|}i}trFtjgddd}|||}|dj |dj |d j d}i}t rjt |g|gd d d d \}} } |d | d | d d}t |} i||d| iS)a] Process results for a single document. This function is called by lm-eval-harness for each document after generation. Args: doc: The document dictionary containing 'highlights' (reference summary) results: List containing the generated text(s) Returns: Dictionary with metric scores for this document r highlightsrTrrrrr5r6Fr7r*summary_length)r! isinstancelistgetrrrrrrr;r<r>calculate_summary_length) rCrDgenerated_summaryreference_summary rouge_resultsr"r(bertscore_resultsr?r@rArGrrrprocess_results|s>     rP generationcCs(|}tdd|}tdd|}|S)z Post-process the generated text to clean it up. Args: generation: Raw generated text Returns: Cleaned generated text z\n+r r )rr r )rQrrrpostprocess_generations rRcCs |dd}t|}|dkS)z Filter out articles that are too long. Args: doc: Document dictionary Returns: True if document should be kept, False if it should be filtered out articlerEi)rJr!split)rCrS word_countrrrfilter_long_articless rVcCsgS)z For multiple-choice format (not used in summarization, but kept for compatibility). Args: doc: Document dictionary Returns: List of choices (empty for generation tasks) r)rCrrr doc_to_choices rWcCsdd}||S)z Pre-process the entire dataset before evaluation. Args: dataset: HuggingFace dataset Returns: Processed dataset cSsN|dd}t|}|dd}t|}i|||t|t|dS)zProcess a single document.rSrErF)rSrFarticle_lengthrG)rJrr!rT)rCrSrFrrr _process_docs    z"process_docs.._process_doc)map)datasetrYrrr process_docss r\ generatedcCstt|S)z Calculate the length of generated summary in words. Args: generated: Generated summary text Returns: Number of words in summary )r!rrT)r]rrrrK s rK)__doc__r collectionsrtypingrrr rouge_scorerr ImportErrorprintr<rr;strrfloatr)rBrPrRboolrVrWr\intrKrrrrsN         &  **>