o 6ti\@s`ddlZddlmZejGdddZejGdddZddZd d Zd d Zd dZ dS)N)instructions_registryc@s2eZdZUeed<eeed<eed<eed<dS) InputExamplekeyinstruction_id_listpromptkwargsN)__name__ __module__ __qualname__int__annotations__liststrrr[/home/ubuntu/.local/lib/python3.10/site-packages/lm_eval/tasks/ifeval/multilingual/utils.pyrs   rc@s>eZdZUeeed<eed<eed<eed<eeed<dS) OutputExamplerrresponsefollow_all_instructionsfollow_instruction_listN)rr r r rr boolrrrrrs rc Cs|j}g}t|D]F\}}tj|}||}dd|j|D}|jd i||} | r;d| vr;|j|jd| rJ| |rJ| dq | dq t |j|j|t ||dS) z3Tests response to see if instructions are followed.cSi|] \}}|r||qSrr.0kvrrr $z5test_instruction_following_strict..rrTFrrrrrNr)r enumeraterINSTRUCTION_DICTritemsbuild_descriptionget_instruction_argsrstripcheck_followingappendrall) inprinstruction_listis_following_listindexinstruction_idinstruction_cls instructionrargsrrr!test_instruction_following_stricts(    r0cCsT|d}d|dd}d|dd}d|dd}|dd}|dd}|dd}|dd} |||||||| g} |j} g} t| D]K\} }tj|}||}dd|j|  D}|j di|| }|rd |vr|j |j d d }| D]}|r| |rd }nq| |qQt|j|j |t| | d S)z=Tests response for an upper bound for following instructions. N*cSrrrrrrrrWrz4test_instruction_following_loose..rrFTrr)splitjoinr$replacerrrr rr!r"r#rr%r&rr')r(rrresponse_remove_firstresponse_remove_lastresponse_remove_bothrevised_responserevised_response_remove_firstrevised_response_remove_lastrevised_response_remove_both all_responsesr)r*r+r,r-r.rr/ is_followingrrr test_instruction_following_loose8sR         rCcCsRt|d|d|d|dd}|d}t||}t||}|j|j|j|jdS)Nrrrr)rrrrr)prompt_level_strict_accinst_level_strict_accprompt_level_loose_accinst_level_loose_acc)rr0rCrr)docresultsr(r out_strict out_looserrrprocess_resultsns  rLcCs"dd|D}t|t|}|S)NcSsg|] }|D]}|qqSrr)rsublistitemrrr rz&agg_inst_level_acc..)sumlen)r! flat_itemsinst_level_accrrragg_inst_level_accsrT) dataclasses!lm_eval.tasks.ifeval.multilingualr dataclassrrr0rCrLrTrrrrs !6