o
پi/ @ s d Z ddlZddlZddlZddlZddlmZmZmZm Z ddl
m Z ddl m
Z ddlmZmZmZmZmZmZ h dZdZd Zd
ed efd d
Zded e e fddZ G dd deZ dS )a
LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-Context Multitasks
Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng, Xiaozhi Wang, Xin Lv, Shulin Cao, Jiazheng Xu, Lei Hou, Yuxiao Dong, Jie Tang, Juanzi Li
https://arxiv.org/abs/2412.15204
N)AnyDictListOptional)
AutoTokenizer)simple_eval_common)ANSWER_PATTERN_MULTICHOICE
HTML_JINJAEval
EvalResult SamplerBaseSingleEvalResult> multi_document_qasingle_document_qalong_structured_datalong_dialogue_historycode_repo_understandinglong_in_context_learningzTHUDM/LongBench-v2trainrowreturnc
C s | dd}| dd}d| v rE| d }t|dkr |d nd}t|dkr*|d nd}t|dkr6|d nd}t|dkrB|d nd}n(| d | d
d}| d | d d}| d
| dd}| d| dd}d| d| d| d| d| d| d
}|S )z;Format a LongBench-v2 question using the official template.context questionchoicesr Achoice_ABchoice_BCchoice_CDchoice_DzF
Please read the following text and answer the question below.
z7
What is the correct answer to this question: z
Choices:
(A) z
(B) z
(C) z
(D) zP
Format your response as follows: "The correct answer is (insert answer here)".)getlenstrip) r r r r r r! r# r% prompt r* X/home/ubuntu/.local/lib/python3.10/site-packages/sglang/test/simple_eval_longbench_v2.py format_longbench_v2_question) s4
r, responsec C s | dd} td| tj}|r|d S td| tj}|r(|d S tt| }|r7|d S td| tj}|rH|d S dS )zFExtract answer from model response using official LongBench-v2 method.*r z!The correct answer is \(([A-D])\)r z The correct answer is ([A-D])zanswer\s+is\s*\(?([A-D])\)?N)replaceresearch
IGNORECASEgroupupperr )r- matchr* r* r+ extract_longbench_v2_answerN s r6 c @ s e Zd ZdZdeddddddfdededee deded eee d
ee d ee fd d
Z dedee
ee f fddZ dedee
ee f fddZ
dedee
ee f fddZde
ee f de
ee f fddZdeded ee d ee def
d d Zd edefd!d"ZdS )#LongBenchV2Evalz
Evaluation utility for LongBench-v2 dataset.
LongBench-v2 is designed to assess the ability of LLMs to handle long-context problems
requiring deep understanding and reasoning across real-world multitasks.
Nr model data_source num_examples num_threads n_repeats
categoriesmax_context_lengthmin_context_lengthc
s t j|dd| _|| _|| _| |} r fdd| D } |r3|dks(J d| dt|t| } | | } | s=td| | _ || _
|| _ t d t| j d
rZt d |s^|rkt d | d
| d dS dS )a<
Initialize LongBench-v2 evaluation.
Args:
data_source: HuggingFace dataset name, local file path (CSV/JSON)
num_examples: Number of examples to evaluate (None for all)
num_threads: Number of threads for parallel processing
n_repeats: Number of times to repeat evaluation for error bars
categories: List of task categories to include (None for all)
max_context_length: Maximum context length in characters
min_context_length: Minimum context length in characters
T)trust_remote_codec s g | ] }| d v r|qS )category)r& ).0exr= r* r+
z,LongBenchV2Eval.__init__..r z3n_repeats only supported when not sampling examplesNzANo examples available for LongBench-v2 evaluation after filteringzLoaded z examples from LongBench-v2zFiltered to categories: zContext length filter: -z characters)
r from_pretrained tokenizerr? r>
_load_datasetminr'
ValueErrorexamplesr< r; print)
selfr8 r9 r: r; r< r= r> r? rM r* rD r+ __init__q s2
zLongBenchV2Eval.__init__r c s< |st }tj|r |}n |} fdd|D S )z1Load dataset from HuggingFace hub or local files.c s g | ]} |qS r* )_normalize_example)rB examplerO r* r+ rE s z1LongBenchV2Eval._load_dataset..)DEFAULT_DATASETospathexists_load_local_file_load_hf_dataset)rO r9 raw_examplesr* rS r+ rJ s
z LongBenchV2Eval._load_datasetrV c C sv t j|d }|dv r7t|ddd}|dkr"dd |D }nt|}W d n1 s1w Y nn|d
kr\t|ddd}t|}t |}W d n1 sVw Y nIz t|ddd
}t|}W d n1 stw Y W n* tj
y t|ddd}t|}t |}W d n1 sw Y Y nw t |t r|
d g }t |t std |S )
z/Load examples from a local CSV/JSON/JSONL file.r > .json.jsonlrzutf-8)encodingr\ c S s g | ] }| rt|qS r* )r( jsonloads)rB liner* r* r+ rE rF z4LongBenchV2Eval._load_local_file..Nz.csvdataz)Expected list of examples from local file)rU rV splitextloweropenr_ loadcsv
DictReaderlistJSONDecodeError
isinstancedictr& rL )rO rV suffixfhrb readerr* r* r+ rX s@
z LongBenchV2Eval._load_local_file
identifierc
C s| |j ddd}|d }t|dkr|d nt}zddlm} W n ty0 } ztd|d}~ww |||d }d
d |D S ) z&Load the dataset from HuggingFace Hub.:r )maxsplitr r ) load_datasetzaPlease install the 'datasets' package to load LongBench-v2 from HuggingFace: pip install datasetsN)splitc S s g | ]}t |qS r* )rl )rB r r* r* r+ rE s z4LongBenchV2Eval._load_hf_dataset..)rt r' DEFAULT_DATASET_SPLITdatasetsrs ImportError)rO rp parts dataset_namert rs excdatasetr* r* r+ rY s z LongBenchV2Eval._load_hf_datasetrR c C s t |}dD ]}d| }||vr||v r|| ||< qd|vr*d|v r*|d |d< |d}t|tr>| |d< |S t|trXd| krMdk rXn |S g d| |d< |S )z.Ensure each example exposes the expected keys.r r r" r$ choice_rA domainanswerr )rl r& rk strr( r4 int)rO rR
normalizedletter
choice_keyr r* r* r+ rQ s
z"LongBenchV2Eval._normalize_exampleformatted_questionrI
min_length
max_lengthc C s> | |}t|}|dur||k rdS |dur ||kr dS dS )z9Filter examples by context length measured in characters.NFT)encoder' )rO r rI r r input_idscontext_lengthr* r* r+ _check_context_length s
z%LongBenchV2Eval._check_context_lengthsamplerc s0 dt f fdd }t|jj}t|S )zRun the evaluation.r c s\ t | }js
jr|jjjsd S j|ddg} |}|d u r)d}t|}| dd}t|t r?|
}nt|t rVd| krNdk rVn ng d| }||kr\dnd }t
jtj|t|d
d|||d }|t|d
dg }d t|i} | d
| dd}
|
tv r|| |
< | d} t| t r| r|| d| < t|||| dS )Nuser)contentroler r r r r| g ?g assistant)prompt_messages next_messagescorecorrect_answerextracted_answercharsrA r~ unknown
difficulty difficulty_)htmlr convometrics)r, r? r> r rI
_pack_messager6 r& rk r r( r4 r common jinja_env from_stringr renderrl r' TASK_CATEGORIESrd r
) r r r
response_textr r r r r r rA r r rO r* r+ fn sT
"
z$LongBenchV2Eval.__call__..fn)rl r map_with_progressrM r; aggregate_results)rO r r resultsr* r r+ __call__ s D
zLongBenchV2Eval.__call__)__name__
__module__ __qualname____doc__rT r r r r rP r r rJ rX rY rQ r boolr r r r r* r* r* r+ r7 i sV
:
"
r7 ) r rg r_ rU r0 typingr r r r transformersr sglang.testr r sglang.test.simple_eval_commonr r r
r r r
r rT ru rl r r, r6 r7 r* r* r* r+ s
%