o 7ti ,@sddlZddlmZiddddddddd d d d d d d ddddddddddddd dddddd d d dddZiddddddddd dd dd dd ddddd dd!dd"dd#dd$dd%dd&dd'd(d)d*d+dZidd,dd,dd,dd-d d,d d,d d,d d.dd/dd/dd/dd.dd0dd,dd/dd0d1d2d3d4d5d5d6Zgd7Zd8d9Zd:Ze d;krbeZ eZ e eZ eD]Zed<redd=neZed>vZeed?d@dAZeedBdCdDdEdFdGdHdIZdDeedDdJdKdLgZed<rdMndNgdOedPdQeedRdSeeeeedTdUidV Ze jdYieZee jedWdX ZeeWdn 1s\wYqdSdS)ZN) Environment narrativeqaqaspermultifieldqa_en@multifieldqa_zhhotpotqa 2wikimqamusiquedureader gov_reportiqmsum multi_newsvcsumtrectriviaqasamsumlsht passage_count)passage_retrieval_enpassage_retrieval_zhlcc repobench-pa}You are given a story, which can be either a novel or a movie script, and a question. Answer the question asconcisely as you can, using a single phrase if possible. Do not provide any explanation. Story: {context} Now, answer the question based on the story asconcisely as you can, using a single phrase if possible. Do not provide any explanation. Question: {question} Answer:aYou are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation. Article: {context} Answer the question based on the above article as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation. Question: {question} Answer:zRead the following text and answer briefly. {context} Now, answer the following question based on the above text, only give me the answer and do not output any other words. Question: {question} Answer:u阅读以下文字并用中文简短回答： {context} 现在请基于上面的文章回答下面的问题，只告诉我答案，不要输出任何其他字词。 问题：{question} 回答：a#Answer the question based on the given passages. Only give me the answer and do not output any other words. The following are given passages. {context} Answer the question based on the given passages. Only give me the answer and do not output any other words. Question: {question} Answer:u请基于给定的文章回答下述问题。 文章：{context} 请基于上述文章回答下面的问题。 问题：{question} 回答：zYou are given a report by a government agency. Write a one-page summary of the report. Report: {context} Now, write a one-page summary of the report. Summary:aYou are given a meeting transcript and a query containing a question or instruction. Answer the query in one or more sentences. Transcript: {context} Now, answer the query based on the above meeting transcript in one or more sentences. Query: {question} Answer:zYou are given several news passages. Write a one-page summary of all news. News: {context} Now, write a one-page summary of all the news. Summary:u下面有一段会议记录，请你阅读后，写一段总结，总结会议的内容。 会议记录： {context} 会议总结：zkPlease determine the type of the question below. Here are some examples of questions. {context} {question}zAnswer the question based on the given passage. Only give me the answer and do not output any other words. The following are some examples. {context} {question}ziSummarize the dialogue into a few short sentences. The following are some examples. {context} {question}uO请判断给定新闻的类别，下面是一些例子。 {context} {question}aThere are some paragraphs below sourced from Wikipedia. Some of them may be duplicates. Please carefully read these paragraphs and determine how many unique paragraphs there are after removing duplicates. In other words, how many non-repeating paragraphs are there in total? {context} Please enter the final count of unique paragraphs after removing duplicates. The output format should only contain the number, such as 1, 2, 3, and so on. The final answer is:aBHere are 30 paragraphs from Wikipedia, along with an abstract. Please determine which paragraph the abstract is from. {context} The following is an abstract. {question} Please enter the number of the paragraph that the abstract is from. The answer format must be like "Paragraph 1", "Paragraph 2", etc. The answer is:u以下是若干段落文字，以及其中一个段落的摘要。请确定给定的摘要出自哪一段。 {context} 下面是一个摘要 {question} 请输入摘要所属段落的编号。答案格式必须是"段落1"，"段落2"等格式 答案是：zBPlease complete the code given below. {context}Next line of code:zLPlease complete the code given below. {context}{question}Next line of code: qa_f1_scoreqa_f1_zh_scorerouge_zh_score rouge_scoreclassification_scorerretrieval_score count_scoreretrieval_zh_scorecode_sim_score)rrrr)"r 2wikimqa_er r gov_report_er hotpotqa_erlcc_err multi_news_ermultifieldqa_en_err rrpassage_count_erpassage_retrieval_en_errqasper_errz repobench-p_ersamsum_ertrec_er triviaqa_ercCst}|jddd|S)Nz--save_prefix_path)default)argparseArgumentParser add_argument parse_args)parserr7\/home/ubuntu/.local/lib/python3.10/site-packages/lm_eval/tasks/longbench/_generate_config.pyr5sr5a tag: - {{ tag[0] }} task: {{ task }} dataset_path: {{ dataset_path }} test_split: {{ test_split }} dataset_name: {{ dataset_name }} doc_to_text: "{{ doc_to_text }}" doc_to_target: '{{ doc_to_target }}' process_results: {{ process_results }} generation_kwargs: max_gen_toks: {{ generation_kwargs.max_gen_toks }} temperature: {{ generation_kwargs.temperature }} do_sample: {{ generation_kwargs.do_sample }} until: {% if has_newline %}["\n"]{% else %}[]{% endif %} metric_list: - metric: {{ metric_list[0].metric }} aggregation: {{ metric_list[0].aggregation }} higher_is_better: {{ metric_list[0].higher_is_better }} metadata: version: {{ metadata.version }} __main___e)rrrrF) max_gen_toks temperature do_sample z\n"z\"{z{{}z}}meanT)metric aggregationhigher_is_better longbench_e longbench longbench_zXnhyacinth/LongBenchtestz {{answers}}z!function metrics.get_versionz5.0) tagtask dataset_path test_split dataset_name doc_to_text doc_to_targetprocess_resultsgeneration_kwargs has_newline metric_listmetadataz.yamlwr7)r2jinja2rdataset2maxlendataset2promptdataset2metricDATASETSr5 template_str__name__argsenv from_stringtemplatedsendswithdfrVrUreplaceraw_doc_to_textrWdatarender rendered_yamlopensave_prefix_pathfwriter7r7r7r8sN                &