o 6ti@sdZddlZddlZddlZddlZddlZddlZddlmZm Z m Z m Z ddl Z ddl mZeeZe eee eee effZejZdZdZdZdZd Zd Zd ZdZd ZdZ dZ!d Z"dZ#dZ$dZ%dZ&dZ'dZ(GdddZ)Gddde)Z*Gddde)Z+Gddde)Z,Gddde)Z-Gddde)Z.Gdd d e)Z/Gd!d"d"e)Z0Gd#d$d$e)Z1Gd%d&d&e)Z2Gd'd(d(e)Z3Gd)d*d*e)Z4Gd+d,d,e)Z5Gd-d.d.e)Z6Gd/d0d0e)Z7Gd1d2d2e)Z8Gd3d4d4e)Z9Gd5d6d6e)Z:Gd7d8d8e)Z;Gd9d:d:e)Zd>e)Z>Gd?d@d@e)Z?GdAdBdBe)Z@GdCdDdDe)ZAGdEdFdFe)ZBGdGdHdHe)ZCGdIdJdJe)ZDGdKdLdLe)ZEGdMdNdNe)ZFdS)OzLibrary of instructions.N)DictOptionalSequenceUnion)instructions_util)z less thanzat least)zMy answer is yes.zMy answer is no.zMy answer is maybe.)z I would sayz My answer isz I believez In my opinionzI thinkzI reckonzI feelzFrom my perspectivez As I see itzAccording to mezAs far as I'm concernedzTo my understandingz In my viewzMy take on it iszAs per my perception)zAny other questions?z'Is there anything else I can help with?)SectionSECTION)P.S.P.P.S dic@s8eZdZdZddZddZddZdd Zd d Zd S) InstructionzAn instruction template.cCs ||_dSN)id)selfinstruction_idra/home/ubuntu/.local/lib/python3.10/site-packages/lm_eval/tasks/leaderboard/ifeval/instructions.py__init__q zInstruction.__init__cKtd)Nz$`build_description` not implemented.NotImplementedError)rkwargsrrrbuild_descriptiontzInstruction.build_descriptioncCr)Nz'`get_instruction_args` not implemented.rrrrrget_instruction_argswr z Instruction.get_instruction_argscCr)Nz,`get_instruction_args_keys` not implemented.rr!rrrget_instruction_args_keyszr z%Instruction.get_instruction_args_keyscCr)Nz"`check_following` not implemented.rrvaluerrrcheck_following}r zInstruction.check_followingN) __name__ __module__ __qualname____doc__rrr"r#r&rrrrrns rc@6eZdZdZddddZddZdd Zd d ZdS) ResponseLanguageCheckerz*Check the language of the entire response.NlanguagecCs>||_|jdurttt|_d|_|jjt|jdS)aBuild the instruction description. Args: language: A string representing the expected language of the response. The language has to comply to the 97 types defined in `langid.py` (https://pypi.org/project/langid/1.1.5/), which follows ISO 639-1 codes (https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes); for example, `en` for English, `zh` for Chinese, `fr` for French. Returns: A string representing the instruction description. NzTYour ENTIRE response should be in {language} language, no other language is allowed.r-) _languagerandomchoicelist _LANGUAGESkeys_description_patternformat)rr.rrrrs  z)ResponseLanguageChecker.build_descriptioncC d|jiS)0Returns the keyword args of `build_description`.r.)r/r!rrrr" z,ResponseLanguageChecker.get_instruction_argscCdgS)-Returns the args keys of `build_description`.r.rr!rrrr#z1ResponseLanguageChecker.get_instruction_args_keysc CsXt|tsJz t||jkWStjy+}ztd||WYd}~dSd}~ww)zCheck if the language of the entire response follows the instruction. Args: value: A string representing the response. Returns: True if the language of `value` follows instruction; otherwise False. /Unable to detect language for text %s due to %sNT) isinstancestr langdetectdetectr/LangDetectExceptionloggingerrorrr%errrr&s z'ResponseLanguageChecker.check_followingr'r(r)r*rr"r#r&rrrrr,  r,c@8eZdZdZdddddZddZdd Zd d ZdS) NumberOfSentenceszCheck the number of sentences.N num_sentencesrelationcCs|||_|jdus |jdkrtdt|_|durtt|_n|tvr.tdtd|d||_d|_|jj |j|jdS) a:Build the instruction description. Args: num_sentences: An integer specifying the number of sentences as a threshold. relation: A string in (`less than`, `at least`), defining the relational operator for comparison. Two relational comparisons are supported for now: if 'less than', the actual number of sentences < the threshold; if 'at least', the actual number of sentences >= the threshold. Returns: A string representing the instruction description. Nr1The supported relation for comparison must be in , but is given.zBYour response should contain {relation} {num_sentences} sentences.)rMrL) _num_sentences_thresholdr0randint_MAX_NUM_SENTENCESr1_COMPARISON_RELATION_comparison_relation ValueErrorr5r6)rrLrMrrrrs(z#NumberOfSentences.build_descriptioncC|j|jdS)r8rK)rRrVr!rrrr"z&NumberOfSentences.get_instruction_argscCddgS)r;rLrMrr!rrrr#z+NumberOfSentences.get_instruction_args_keyscC>t|}|jtdkr||jkS|jtdkr||jkSdS)aLCheck if the number of sentences follows the instruction. Args: value: A string representing the response. Returns: True if the response follows the instruction. Raise: ValueError if the string in `instruction_args` is not in [`less_than`, `at_least`]. rrNN)rcount_sentencesrVrUrR)rr%rLrrrr&s   z!NumberOfSentences.check_followingrGrrrrrJs & rJc@r+) PlaceholderCheckerz+Check the placeholders in template writing.Nnum_placeholderscC>||_|jdus |jdkrtdt|_d|_|jj|jdS)aBuild the instruction description. Args: num_placeholders: An integer denoting the minimum number of placeholders required in the response. Returns: A string representing the instruction description. NrrNzuThe response must contain at least {num_placeholders} placeholders represented by square brackets, such as [address].r_)_num_placeholdersr0rS_NUM_PLACEHOLDERSr5r6)rr`rrrrs  z$PlaceholderChecker.build_descriptioncCr7)r8r`)rbr!rrrr"r9z'PlaceholderChecker.get_instruction_argscCr:)r;r`rr!rrrr#r<z,PlaceholderChecker.get_instruction_args_keyscCstd|}t|}||jkS)a+Check if the number of placeholders follows the instruction. Args: value: A string representing the response. Returns: True if the actual number of placeholders in the response is greater than or equal to `num_placeholders`; otherwise, False. z\[.*?\])refindalllenrb)rr% placeholdersr`rrrr&s  z"PlaceholderChecker.check_followingrGrrrrr^  r^c@r+) BulletListCheckerz%Checks the bullet list in the prompt.N num_bulletscCra)aBuild the instruction description. Args: num_bullets: An integer specifying the exact number of bullet lists that is required to appear in the response. Returns: A string representing the instruction description. NrrNzYour answer must contain exactly {num_bullets} bullet points. Use the markdown bullet points such as: * This is point 1. * This is point 2rj) _num_bulletsr0rS _NUM_BULLETSr5r6)rrkrrrr0s  z#BulletListChecker.build_descriptioncCr7)r8rk)rlr!rrrr"Er9z&BulletListChecker.get_instruction_argscCr:)r;rkrr!rrrr#Ir<z+BulletListChecker.get_instruction_args_keyscCs>tjd|tjd}tjd|tjd}t|t|}||jkS)aVCheck if the number of bullet lists meets the requirement. Args: value: A string representing the response. The response is expected to contain some bullet lists that start with `\*`. Returns: True if the actual number of bullet lists in the response meets the requirement. z^\s*\*[^\*].*$flagsz^\s*-.*$)rdre MULTILINErfrl)rr% bullet_listsbullet_lists_2num_bullet_listsrrrr&Ms  z!BulletListChecker.check_followingrGrrrrri-  ric@0eZdZdZddZddZddZdd Zd S) ConstrainedResponseCheckerz Checks the constrained response.cCst|_d|_|jj|jdS)"Build the instruction description.zr?r_starterr0r1_STARTER_OPTIONSr5r6)rrrrrrs   z)ConstrainedStartChecker.build_descriptioncCr7)r8r)rr!rrrr"r9z,ConstrainedStartChecker.get_instruction_argscCr:)r;rrr!rrrr#r<z1ConstrainedStartChecker.get_instruction_args_keyscCs,d|jd}tj||tjd}|rdSdS)a0Checks if the response starts with the constrained keyword or phrase. Args: value: A string representing the response. Returns: True if the response starts with the given phrase or keyword that is contained in `instruction_args`; otherwise, False. z^\s*.*$rnTF)rrdsearchrp)rr%response_patternresponse_with_constrained_startrrrr&s   z'ConstrainedStartChecker.check_followingrGrrrrrrhrc@r+) HighlightSectionCheckerzChecks the highlighted section.Nnum_highlightscCra)zBuild the instruction description. Args: num_highlights: An integer specifying the minimum number of highlighted sections. Returns: A string representing the instruction description. NrrNzfHighlight at least {num_highlights} sections in your answer with markdown, i.e. *highlighted section*.r)_num_highlightsr0rS_NUM_HIGHLIGHTED_SECTIONSr5r6)rrrrrrs  z)HighlightSectionChecker.build_descriptioncCr7)r8r)rr!rrrr"r9z,HighlightSectionChecker.get_instruction_argscCr:)r;rrr!rrrr#r<z1HighlightSectionChecker.get_instruction_args_keyscCsld}td|}td|}|D] }|dr|d7}q|D]}|ddr0|d7}q ||jkS)aChecks if the number of highlighted sections meets the requirement. Args: value: a string representing the response. The response is expected to contain highlighted sections in the format of *highlighted*. Returns: True if the actual number of highlighted sections in the format of *highlighted sections* meets the minimum requirement; otherwise False. rz \*[^\n\*]*\*z\*\*[^\n\*]*\*\**rNz**)rdrer removeprefix removesuffixr)rr%r highlightsdouble_highlights highlightrrrr&s   z'HighlightSectionChecker.check_followingrGrrrrrrtrc@rI) SectionCheckerzChecks the sections.Nsection_spliter num_sectionscCspt|tr |n||_|jdurtt|_||_|jdus$|jdkr+tdt |_d|_ |j j |j|jdS)a^Build the instruction description. Args: section_spliter: A string represents the section spliter keyword that marks a new section, i.e., `Section` or `SECTION`. num_sections: An integer specifying the number of sections. Returns: A string representing the instruction description. NrrNzYour response must have {num_sections} sections. Mark the beginning of each section with {section_spliter} X, such as: {section_spliter} 1 [content of section 1] {section_spliter} 2 [content of section 2])rr) r>r?r_section_spliterr0r1_SECTION_SPLITER _num_sectionsrS _NUM_SECTIONSr5r6)rrrrrrrs    z SectionChecker.build_descriptioncCrX)r8r)rrr!rrrr"rYz#SectionChecker.get_instruction_argscCrZ)r;rrrr!rrrr#r[z(SectionChecker.get_instruction_args_keyscCs0d|jd}t||}t|d}||jkS)aChecks the response contains multiple sections. Args: value: A string representing the response. The response is expected to contain multiple sections (number of sections is greater than 1). A new section starts with `Section 1`, where the number denotes the section index. Returns: True if the number of sections in the response is greater than or equal to the minimum number of sections; otherwise, False. z\s?z \s?\d+\s?rN)rrdsplitrfr)rr%section_splitter_pattensectionsrrrrr&s   zSectionChecker.check_followingrGrrrrrs $ rc@r+) ParagraphCheckerzChecks the paragraphs.Nnum_paragraphscCra)zBuild the instruction description. Args: num_paragraphs: An integer specifying the number of paragraphs. Returns: A string representing the instruction description. NrrNzdThere should be {num_paragraphs} paragraphs. Paragraphs are separated with the markdown divider: ***r)_num_paragraphsr0rS_NUM_PARAGRAPHSr5r6)rrrrrr4s  z"ParagraphChecker.build_descriptioncCr7)r8r)rr!rrrr"Hr9z%ParagraphChecker.get_instruction_argscCr:)r;rrr!rrrr#Lr<z*ParagraphChecker.get_instruction_args_keyscCs`td|}t|}t|D]\}}|s*|dks"|t|dkr'|d8}qdSq||jkS)adChecks the response contains required number of paragraphs. Args: value: A string representing the response. The response may contain paragraphs that are separated by the markdown divider: `***`. Returns: True if the actual number of paragraphs is the same as required; otherwise, False. z \s?\*\*\*\s?rrNF)rdrrf enumeraterr)rr% paragraphsrindex paragraphrrrr&Ps   z ParagraphChecker.check_followingrGrrrrr1s  rc@r+) PostscriptCheckerzChecks the postscript.N)postscript_markercCr)aBuild the instruction description. Args: postscript_marker: A string containing the keyword that marks the start of the postscript section. Returns: A string representing the instruction description. NzZAt the end of your response, please explicitly add a postscript starting with {postscript}) postscript) r>r?r_postscript_markerr0r1_POSTSCRIPT_MARKERr5r6)rrrrrrks   z#PostscriptChecker.build_descriptioncCr7)r8r)rr!rrrr"r9z&PostscriptChecker.get_instruction_argscCr:)r;rrr!rrrr#r<z+PostscriptChecker.get_instruction_args_keyscCsX|}|jdkr d}n|jdkrd}n d|jd}tj||tjd}|r*dSd S) arChecks if the response follows the postscript format. Args: value: a string representing the response. The response is expected to contain a postscript section. Returns: True if the response contains a postscript section starting with the keyword containing in the `instruction_args`; otherwise False. r z\s*p\.\s?p\.\s?s.*$r z\s*p\.\s?s\..*$z\s*rrnTF)lowerrrdrerp)rr%postscript_patternrrrrr&s   z!PostscriptChecker.check_followingrGrrrrrhs  rc@s@eZdZdZddZddZddZdd Zd d Zd d Z dS)RephraseCheckerzChecks the rephrase.cCs,||s td|d||_d|_|jS)aBuild the instruction description. Args: original_message: A string representing the original message. The rephrased response should only change its words/sentences in between its two asterisks, for example, *change me*. Both original and rephrased messages should contain the changes in the form of *change me*. Returns: A string representing the instruction description. zMessage 5 does not contain changes in the form of *change me*.zvRephrasing: Your rephrased response should onlychange the words/sentences in between two asteriskssuch as *change me*.) is_changerW_reference_without_change _description)roriginal_messagerrrrs  z!RephraseChecker.build_descriptioncCr7)r8r)rr!rrrr"r9z$RephraseChecker.get_instruction_argscCr:)r;rrr!rrrr#r<z)RephraseChecker.get_instruction_args_keyscCs8||s td|d||}||j}||kS)ayChecks if the rephrasing follows the instruction. Args: value: A string representing the response, which is expected to rephras the string of `instruction_args`. Returns: True if `value` and `instruction_args` only differ by the words/sentences in between two asterisks such as *change me*; otherwise, False. zvalue r)rrW strip_changesr)rr%response_without_changesreference_without_changesrrrr&s    zRephraseChecker.check_followingcCs td|S)zDCheck if there is change in the response in the form of *change me*.\*.*\*rdrrresponserrrrs zRephraseChecker.is_changecCstdd|S)zStrips off the changes.r)rdsubrrrrrzRephraseChecker.strip_changesN) r'r(r)r*rr"r#r&rrrrrrrs rc@r+) KeywordCheckerz)Check the exisitence of certain keywords.NkeywordscCs<|s tjtd|_n||_t|j|_d|_|jj|jdS)zBuild the instruction description. Args: keywords: A sequence of strings representing the keywords that are expected in the response. Returns: A string representing the instruction description.  num_keywordsz,Include keywords {keywords} in the response.r)rgenerate_keywords _NUM_KEYWORDS _keywordssortedr5r6)rrrrrrs   z KeywordChecker.build_descriptioncCr7)r8r)rr!rrrr"r9z#KeywordChecker.get_instruction_argscCr:)r;rrr!rrrr#r<z(KeywordChecker.get_instruction_args_keyscCs(|jD]}tj||tjdsdSqdS)z4Check if the response contain the expected keywords.rnFT)rrdr IGNORECASE)rr%keywordrrrr&s zKeywordChecker.check_followingrGrrrrrrHrc@:eZdZdZddddddZddZdd Zd d ZdS) KeywordFrequencyCheckerzCheck the keyword frequency.Nr frequencyrMcCs|s tjddd|_n||_||_|jdus|jdkr%tdt|_|dur0tt |_ n|t vr?t dt d|d||_ d|_ |j j |j|j |jd S) aBuild the instruction description. Args: keyword: A string representing a keyword that is expected in the response. frequency: An integer specifying the number of times `keyword` is expected to appear in the response. relation: A string in (`less than`, `at least`), defining the relational operator for comparison. Two relational comparisons are supported for now: if 'less than', the actual number of occurrences < frequency; if 'at least', the actual number of occurrences >= frequency. Returns: A string representing the instruction description. rNrrNrOrPrQzPIn your response, the word {keyword} should appear {relation} {frequency} times.)rrMr)rr_keywordr _frequencyr0rS_KEYWORD_FREQUENCYr1rUrVrWr5r6)rrrrMrrrrs0 z)KeywordFrequencyChecker.build_descriptioncC|j|j|jdS)r8r)rrrVr!rrrr"Az,KeywordFrequencyChecker.get_instruction_argscCgdS)r;rrr!rrrr#Ir[z1KeywordFrequencyChecker.get_instruction_args_keyscCsLttj|j|tjd}|jtdkr||jkS|jtdkr$||jkSdS)zCChecks if the response contain the keyword with required frequency.rnrrNN)rfrdrerrrVrUr)rr%actual_occurrencesrrrr&Ms   z'KeywordFrequencyChecker.check_followingrGrrrrrs . rc@rI) NumberOfWordszChecks the number of words.N num_wordsrMcCs|||_|jdus |jdkrttt|_|durtt|_n|tvr.tdtd|d||_d|_ |j j |j|jdS)a-Build the instruction description. Args: num_words: An integer specifying the number of words contained in the response. relation: A string in (`less than`, `at least`), defining the relational operator for comparison. Two relational comparisons are supported for now: if 'less than', the actual number of words < num_words; if 'at least', the actual number of words >= num_words. Returns: A string representing the instruction description. NrrOrPrQz)Answer with {relation} {num_words} words.)rMr) _num_wordsr0rS_NUM_WORDS_LOWER_LIMIT_NUM_WORDS_UPPER_LIMITr1rUrVrWr5r6)rrrMrrrrZs(zNumberOfWords.build_descriptioncCrX)r8r)rrVr!rrrr"rz"NumberOfWords.get_instruction_argscCrZ)r;rrMrr!rrrr#r[z'NumberOfWords.get_instruction_args_keyscCr\)z=Checks if the response contains the expected number of words.rrNN)r count_wordsrVrUr)rr%rrrrr&s   zNumberOfWords.check_followingrGrrrrrWs & rc@ru) JsonFormatzCheck the Json format.cC d|_|jS)NzWEntire output should be wrapped in JSON format. You can use markdown ticks such as ```.r5r!rrrrszJsonFormat.build_descriptioncCr{r|rr!rrrr"r}zJsonFormat.get_instruction_argscCr~rrr!rrrr#r}z$JsonFormat.get_instruction_args_keyscCsP|ddddd}zt|WdSty'YdSw)Nz```jsonz```Jsonz```JSONz```FT)rrrjsonloadsrWr$rrrr&s  zJsonFormat.check_followingNrGrrrrr  rc@s4eZdZdZ d ddZddZddZd d ZdS) ParagraphFirstWordCheckz.!"',?rrr) rdrrfrrlstriprrr) rr%rrrr punctuationwordletterrrrr&s,    z'ParagraphFirstWordCheck.check_following)NNNrGrrrrrs . rc@2eZdZdZd ddZddZddZd d ZdS) KeySentenceCheckerz-Check the existence of certain key sentences.NcCsR|s tdg|_n||_|stdt|j|_n||_d|_|jj|j|jdS)avBuild the instruction description. Args: key_sentences: A sequences of strings representing the key sentences that are expected in the response. num_sentences: The number of key sentences that are expected to be seen in the response. Returns: A string representing the instruction description. zFor now, this is fine.rNzBInclude {num_sentences} of the following sentences {key_sentences}rL key_sentences)set_key_sentencesr0rSrf_num_sentencesr5r6)rrrLrrrr#s z$KeySentenceChecker.build_descriptioncCs|jt|jdS)r8r)rr2rr!rrrr"Csz'KeySentenceChecker.get_instruction_argscCrZ)r;rLrrr!rrrr#Jr[z,KeySentenceChecker.get_instruction_args_keyscCs4d}t|}|jD] }||vr|d7}q ||jkS)z;Checks if the response contains the expected key sentences.rrN)rsplit_into_sentencesrr)rr%count sentencessentencerrrr&Ns   z"KeySentenceChecker.check_followingNNrGrrrrr s    rc@r) ForbiddenWordsz5Checks that specified words are not used in response.NcCsD|s tjtd|_ntt||_t|j|_d|_|jj|jdS)aBuild the instruction description. Args: forbidden_words: A sequences of strings representing words that are not allowed in the response. Returns: A string representing the instruction description. rz:Do not include keywords {forbidden_words} in the response.)forbidden_words) rrr_forbidden_wordsr2rrr5r6)rrrrrr\s   z ForbiddenWords.build_descriptioncCr7)r8r)rr!rrrr"tr9z#ForbiddenWords.get_instruction_argscCr:)r;rrr!rrrr#xr<z(ForbiddenWords.get_instruction_args_keyscCs0|jD]}tjd|d|tjdrdSqdS)z=Check if the response does not contain the expected keywords.z\brnFT)rrdrr)rr%rrrrr&|s zForbiddenWords.check_followingrrGrrrrrYs   rc@ru) RephraseParagraphz'Checks that the paragraph is rephrased.cCs.||_||_||_d|_|jj||j|jdS)aBuilds the instruction description. Args: original_paragraph: A string presenting the original paragraph. The rephrases response should have betweeb low-high words in common. low: An integer presenting the lower bound of similar words. high: An integer representing the upper bound of similar words. Returns: A string representing the instruction description. aRephrase the following paragraph: {original_paragraph} Your response should have between {low} and {high} of the same words. Words are the same if and only if all of the letters, ignoring cases, are the same. For example, 'run' is the same as 'Run' but different to 'ran'.original_paragraphlowhigh)_original_paragraph_low_highrr6)rrrrrrrrs  z#RephraseParagraph.build_descriptioncCr)r8r)r r r r!rrrr"rz&RephraseParagraph.get_instruction_argscCr)r;rrr!rrrr#r[z+RephraseParagraph.get_instruction_args_keyscCsntd|}td|j}d}t|}t|}|D] }|t||||7}q||jko6||jkS)Nz\w+r) rdrerr  collectionsCounterminr r )rr% val_wordsoriginal_words similar_wordsdict_val dict_originalrrrrr&s  z!RephraseParagraph.check_followingNrGrrrrrs  rc@ru) TwoResponsesCheckerz$Check that two responses were given.cCr)rwzmGive two different responses. Responses and only responses should be separated by 6 asterisk symbols: ******.rr!rrrrz%TwoResponsesChecker.build_descriptioncCr{r|rr!rrrr"r}z(TwoResponsesChecker.get_instruction_argscCr~rrr!rrrr#r}z-TwoResponsesChecker.get_instruction_args_keyscCsxt}|d}t|D]\}}|s$|dkr#|t|dkr#dSq ||q t|dko;|d|dkS)zChecks if the response has two different answers. Args: value: A string representing the response. Returns: True if two responses are detected and false otherwise. z******rrNFr)r2rrrrfappend)rr%valid_responses responsesrrrrrr&s   z#TwoResponsesChecker.check_followingNrGrrrrrs  rc@r+) RepeatPromptThenAnswerz3Checks that Prompt is first repeated then answered.N)prompt_to_repeatcCs|std||_d|_|jS)zBuild the instruction description. Args: prompt_to_repeat: The prompt that is meant to be repeated. Returns: A string representing the instruction description. zprompt_to_repeat must be set.zFirst repeat the request word for word without change, then give your answer (1. do not say any words or characters before repeating the request; 2. the request you need to repeat does not include this sentence))rW_prompt_to_repeatr5)rrrrrrs  z(RepeatPromptThenAnswer.build_descriptioncCr7)Nr)rr!rrrr"rz+RepeatPromptThenAnswer.get_instruction_argscCr:)r;rrr!rrrr#r<z0RepeatPromptThenAnswer.get_instruction_args_keyscCs$||jrdSdS)NTF)rr startswithrr$rrrr& sz&RepeatPromptThenAnswer.check_followingrGrrrrrs  rc@r+) EndCheckerz0Checks that the prompt ends with a given phrase.N) end_phrasecCr)zBuild the instruction description. Args: end_phrase: A string representing the phrase the response should end with. Returns: A string representing the instruction description. Nz^Finish your response with this exact phrase {ender}. No other words should follow this phrase.)ender) r>r?r _end_phraser0r1_ENDING_OPTIONSr5r6)rrrrrrs   zEndChecker.build_descriptioncCr7)Nr)r r!rrrr"&rzEndChecker.get_instruction_argscCr:)r;rrr!rrrr#)r<z$EndChecker.get_instruction_args_keyscCs.|d}|j|_||jS)z5Checks if the response ends with the expected phrase.r)rrr endswithr$rrrr&-s zEndChecker.check_followingrGrrrrrs  rc@ru) TitleCheckerz Checks the response for a title.cCr)rwz^Your answer must contain a title, wrapped in double angular brackets, such as <>.rr!rrrr7rzTitleChecker.build_descriptioncCdSrrr!rrrr"?z!TitleChecker.get_instruction_argscCr~rrr!rrrr#Br}z&TitleChecker.get_instruction_args_keyscCsBd}t|}t||}|D]}|ddrdSqdS)z(Checks if the response contains a title.z <<[^\n]+>><>TF)rdcompilererrstripr)rr%pattern re_patterntitlestitlerrrr&Fs  zTitleChecker.check_followingNrGrrrrr#4  r#c@r) LetterFrequencyCheckerzChecks letter frequency.Nr let_frequency let_relationcCs|rt|dkst|dkst|dkr"tttj|_n| |_|j|_||_ |j dus:|j dkrAt dt |_ |durLtt |_n|t vr[tdt d|d||_d |_|jj|j|j |jd S) aBuild the instruction description. Args: letter: A string representing a letter that is expected in the response. let_frequency: An integer specifying the number of times `keyword` is expected to appear in the response. let_relation: A string in (`less than`, `at least`), defining the relational operator for comparison. Two relational comparisons are supported for now; if 'less than', the actual number of occurrences < frequency; if 'at least', the actual number of occurrences >= frequency. Returns: A string representing the instruction description. rNazNrrOrPrQzYIn your response, the letter {letter} should appear {let_relation} {let_frequency} times.r0)rfordrr0r1r2string ascii_letters_letterrrrS_LETTER_FREQUENCYrUrVrWr5r6)rrr1r2rrrrUs:   z(LetterFrequencyChecker.build_descriptioncCr).Returns the keyword args of build description.r0)r8rrVr!rrrr"rz+LetterFrequencyChecker.get_instruction_argscCr)r;r0rr!rrrr#r[z0LetterFrequencyChecker.get_instruction_args_keyscCs@|}t|}|jtdkr||j|jkS||j|jkS)zDChecks that the response contains the letter at the right frequency.r)rr r rVrUr8r)rr%lettersrrrr&s  z&LetterFrequencyChecker.check_followingrGrrrrr/Rs 4 r/c@ru) CapitalLettersEnglishCheckerzEChecks that the response is in english and is in all capital letters.cCr)rwzFYour entire response should be in English, and in all capital letters.rr!rrrrz.CapitalLettersEnglishChecker.build_descriptioncCr$rrr!rrrr"r%z1CapitalLettersEnglishChecker.get_instruction_argscCr~rrr!rrrr#r}z6CapitalLettersEnglishChecker.get_instruction_args_keysc C^t|tsJz |ot|dkWStjy.}ztd||WYd}~dSd}~ww)zBChecks that the response is in English and in all capital letters.enr=NT)r>r?isupperr@rArBrCrDrErrrr&z,CapitalLettersEnglishChecker.check_followingNrGrrrrr<  r<c@ru) LowercaseLettersEnglishCheckerzGChecks that the response is in english and is in all lowercase letters.cCr)rwzhYour entire response should be in English, and in all lowercase letters. No capital letters are allowed.rr!rrrrrz0LowercaseLettersEnglishChecker.build_descriptioncCr$rrr!rrrr"r%z3LowercaseLettersEnglishChecker.get_instruction_argscCr~rrr!rrrr#r}z8LowercaseLettersEnglishChecker.get_instruction_args_keysc Cr>)zDChecks that the response is in English and in all lowercase letters.r?r=NT)r>r?islowerr@rArBrCrDrErrrr&rAz.LowercaseLettersEnglishChecker.check_followingNrGrrrrrCr.rCc@ru) CommaCheckerz"Checks the response for no commas.cCr)rwz-sz?CapitalWordFrequencyChecker.check_following..r)rnltk word_tokenizerfrVrUr)rr%words capital_wordsrrrr&)s   z+CapitalWordFrequencyChecker.check_followingrrGrrrrrFs & rFc@ru) QuotationCheckerz7Checks response is wrapped with double quotation marks.cCr)rwz6Wrap your entire response with double quotation marks.rr!rrrr:r=z"QuotationChecker.build_descriptioncCr{)r:Nrr!rrrr"Ar}z%QuotationChecker.get_instruction_argscCr~rrr!rrrr#Er}z*QuotationChecker.get_instruction_args_keyscCs,|}t|dko|ddko|ddkS)z>Checks if the response is wrapped with double quotation marks.rNrr)rrfr$rrrr&Is$z QuotationChecker.check_followingNrGrrrrrP7rrP)Gr*r rrCr0rdr6typingrrrrr@lm_eval.tasks.ifevalr getLoggerr'loggerr?int_InstructionArgsDtypeLANGUAGE_CODESr3rUrTrcrmryrr!rrrrrrrr9rGrrrr,rJr^rirvrrrrrrrrrrrrrrrrrr#r/r<rCrErFrPrrrrsz  7H-1'/8E7:D*G;#k9+<*%%N B