o 6ti @snddlZddlZddedefddZdeedeefd d Z dded edefd d Z deefddZ dS)NFqsheet no_contextcCst|d}d}|D]$}|d|dd7}|dD]}||dd|d7}|d7}qq |r>|d d |d }|S|d d |d d |d }|S) N questions prompt subpromptsquestionpart_n questionpreamblez z context)jsonloads)rr subquestionsall_subquestionssqsprrO/home/ubuntu/.local/lib/python3.10/site-packages/lm_eval/tasks/lingoly/utils.pyload_questionsheets&    rquestionpart_nsanswerscCs@i}i}t|D]\}}d||<||||<qt|}||fS)Nr) enumeraterdumps)rrformatted_outputformatted_answersiqnrrrformat_answerss rquestion_indexc Cst|d}||}d}g}g}|d|dd7}|dD]!}||dd|d7}||d||d |d7}qt||\} } t||} d | d |d | d } | | fS)Nrrrrrr r r answeraBelow is a problem sheet from a lingusitics exam. You will first see the entire sheet, then be asked to respond to specific questions from the sheet. Your answers to the questions should rely only on reasoning about the information provided in the sheet. zJ Now respond to the following questions: zg Format your response as a json file with the keys as provided below: z )rrappendrr) rr rrrrrrrrr question_bodyrrrr load_question+s*    r$question_sheetsc Csg}g}g}g}|D]:}ttt|dD],}t||dd\}}t||dd\} } || |||t|||dqq ||||d} tj | } | S)NrF)rToverall_question_n)r nc_promptrindex) rangelenrrr$r"strdatasetsDataset from_dict) r%prompts nc_promptsrindicesrrrr!r'_qsheetsdatasetrrrload_all_questionsMs(    r5)F) rr,dictboolrlistr+rintr$r5rrrrs "