o 7ti(@s&ddlZddlZddlZddlZddlmZddlZzddlm Z Wn e y+Ynwe ddZ e ddZ dZd d Zd d Zd/ddZddZddZddZddZddZddZddZddZd d!Zd"d#Zd$d%Zd&ed'eed(ed)efd*d+Zd,ed'eed(ed)efd-d.Z dS)0N)ListOpenAIAPI_TYPEopenai MODEL_VERSIONz gpt-4.1-minia You are an impartial grader for multiple-choice questions. You are given: 1) the model's free-form output (student_answer), 2) the available options (each with a letter and text), 3) the correct answer (by letter and/or text). Your job: - Extract which single option the student intended (by letter if present, otherwise by best semantic match to the option text). - Compare that choice to the correct answer. - Output only a single character: 1 if correct, 0 if incorrect. No explanation. No extra characters. Just 1 or 0.cCs8tddD]}d|d}d}||vr|||}q|S)Nzz)rangereplace) input_stringi question_text query_textrL/home/ubuntu/.local/lib/python3.10/site-packages/lm_eval/tasks/pisa/utils.pyreplace_images_tokenss  rcCs4ddtt|D}dddt||D}|S)NcSsg|] }ttd|qS)A)chrord).0rrrr )sz!parse_options.. cSg|] \}}|d|qS)z. r)r option_letteroptionrrrr+s )r lenjoinzip)optionsoption_letters choices_strrrr parse_options(sr#cCs6|d}tt|d}d|d|d|}|S)Nquestionchoicesz>Given the provided image , answer following questions: rz )r#ast literal_eval)doc mc_promptr%parsed_optionsrrrconstruct_prompt3sr,cCs t|}|S)N)r,)r)r%rrrpisa_doc_to_text:sr-cCsd}||dur dS||gS)Nimager)r) image_keyrrrpisa_doc_to_visual?s  r0c Ksftt|d\}}t|d||}|d}||vr!||nd}|dur+||knd}dt|iS)z:Default evaluation of answers based on substring matching.r&ranswerNFacc)get_multi_choice_infor'r(parse_multi_choice_responseindexfloat) r)resultskwargs index2ans all_choices parsed_predgold_ipred_i is_correctrrrpisa_process_resultsFsr?c Kstddus Jdzddlm}Wn tytdwtt|d\}}|d}|||}t|dd d | Dt t d |d |d k}dt |iS)z.Evaluation of answers based on LLM as a judge.OPENAI_API_KEYNz/OPENAI_API_KEY environment variable is not set.rrz1Please install openai package to use LLM judging.r&r1cSr)) r)rkvrrrrgsz3pisa_process_results_llm_judged..rrArr2) osgetenvrr ImportErrorr3r'r( judge_mcqitemsrrr6) r)r7r8rr9r:r<correct_answerr>rrrpisa_process_results_llm_judgedUs.   rJcCs>d}t|tr|D] }||krd}|Sq |S||krd}|S)z$Evaluate a multiple choice instance.FT) isinstancelist)r<r=correctr1rrreval_multi_choicers rNcCsd}t|trg}|D] }|t|q nt|}|D]'}t|tr8|D]}t|tr6||vr6|s4d}nq%q||vrC|s@d}|Sq|S)z Evaluate an open question instance https://github.com/pisa-Benchmark/pisa/blob/51ce7f3e829c16bb44bc5445782686b4c3508794/eval/eval_utils.py#L191 FT)rKrLextend normalize_strstr)r<r=rM norm_answersr1prednorm_ansrrr eval_opens,  rUc CsdD]}||}qd|d}d}d}g}|D]}d|d|vr)||d}qt|dkrA|D]}|d|vr@||q2t|dkrX|D]}|d|vrW||qIt|dkr~t|d kr~|D]\}} | |vr}||d}qjt|dkrt|} | St|d krg} |r|r|D]} |d| d}| |qn)|D]} |d| d}| |qn|D]} ||| }| |q|t | } | S|d} | S) z Parse the prediction from the generated response. Return the predicted index e.g., A, B, C, D. https://github.com/pisa-Benchmark/pisa/blob/51ce7f3e829c16bb44bc5445782686b4c3508794/eval/eval_utils.py#L10 ),.!?;:' TF()rrWr) stripappendrsplitrHlowerrandomchoicerfindnpargmax) responser:r9char index_ansans_with_brack candidatesrfr5ans pred_index start_indexescanrrrr4sb              r4cCs@d}d}d}t||}t||}t||}|||}|S)z> Exact all forms of numbers from a string with regex. z-?\b\d{1,3}(?:,\d{3})+\bz-?\d+(?:\.\d+)?[eE][+-]?\d+z3-?(?:\d+\.\d+|\.\d+|\d+\b)(?![eE][+-]?\d+)(?![,\d]))refindall)stringpattern_commaspattern_scientificpattern_simplenumbers_with_commasnumbers_scientificnumbers_simple all_numbersrrrextract_numberss    r}cCs,z t|ddWdStyYdSw)NrVr$TF)r6r ValueError)rurrrcheck_is_numbers  rcCsb|}t|}|r|dd}t|}t|d}|gS|}t|dkr.d||dgS|gS)zHNormalize the str to lower case and make them float numbers if possible.rVr$rr])rarr r6roundrdr)ru is_numberrrrrPs   rPcCsPd}g}i}t|D]\}}||tt||<|tt||q ||fS)zi Given the list of options for multiple choice question Return the index2ans and all_choices r) enumeraterrrb)r start_chrr:r9rrrrrr3sr3student_answerr rMreturncCs*d|dtd|d|dS)z options: like ["A) red", "B) blue", "C) green", "D) yellow"] correct: either a letter like "B" or the full option text. Both are provided to help you. zStudent Answer: z Options: z' Correct Answer (letter and/or text): z Instructions: - If student gives multiple letters, pick the *final* one. - If no clear letter, pick the best-matching option by meaning. - Output only 1 or 0. )rarr)rr rMrrrbuild_user_prompt#s rrScCsdttdd}t|||}|jjjtdddtdd|dgd}|j dj j }|d kr0dSdS) Nr@)api_keyrrsystem)rolecontentuser)model temperature max_tokensmessages1) rrDrErchat completionscreater SYSTEM_PROMPTr&messagerra)rSr rMclient user_promptresprawrrrrG8s  rG)r$)!r'rDrerstypingrnumpyrhrrrFrErrrrr#r,r-r0r?rJrNrUr4r}rrPr3rQrintrGrrrrs<      = "