o پis-@sdZddlZddlZddlZddlmZmZmZddlm Z m Z m Z deee effddZ deee effdd ZGd d d Zdd d ZdddZdddZdddZdddZdddZdddZedkroedSdS)z Validation script for LongBench-v2 implementation. This script validates our implementation against official LongBench-v2 format and benchmarks. N)AnyDictList)LongBenchV2Evalextract_longbench_v2_answerformat_longbench_v2_questionreturncCs`ddddddddd d d d d d dddddddddddddd dddddd d!d"d#d$d d%d gS)&zBCreate sample data in official LongBench-v2 format for validation.test_001sciencephysicshardmediumzMWhat is the fundamental force responsible for holding atomic nuclei together?zElectromagnetic forcezStrong nuclear forcezWeak nuclear forcezGravitational forceBzFNuclear physics studies the components and behavior of atomic nuclei. d) _iddomain sub_domain difficultylengthquestionchoice_Achoice_Bchoice_Cchoice_Danswercontexttest_002 literatureanalysislongz?What literary technique is primarily used in the given passage?Metaphor Alliteration SymbolismIronyCzWLiterary analysis involves examining various techniques authors use to convey meaning. test_003code algorithmseasyshortz-What is the time complexity of binary search?zO(n)zO(log n)uO(n²)zO(1)a Binary search is a fundamental algorithm in computer science. Binary search is a fundamental algorithm in computer science. Binary search is a fundamental algorithm in computer science. Binary search is a fundamental algorithm in computer science. Binary search is a fundamental algorithm in computer science. Binary search is a fundamental algorithm in computer science. Binary search is a fundamental algorithm in computer science. Binary search is a fundamental algorithm in computer science. Binary search is a fundamental algorithm in computer science. Binary search is a fundamental algorithm in computer science. Binary search is a fundamental algorithm in computer science. Binary search is a fundamental algorithm in computer science. Binary search is a fundamental algorithm in computer science. Binary search is a fundamental algorithm in computer science. Binary search is a fundamental algorithm in computer science. Binary search is a fundamental algorithm in computer science. Binary search is a fundamental algorithm in computer science. Binary search is a fundamental algorithm in computer science. Binary search is a fundamental algorithm in computer science. Binary search is a fundamental algorithm in computer science. Binary search is a fundamental algorithm in computer science. Binary search is a fundamental algorithm in computer science. Binary search is a fundamental algorithm in computer science. Binary search is a fundamental algorithm in computer science. Binary search is a fundamental algorithm in computer science. Binary search is a fundamental algorithm in computer science. Binary search is a fundamental algorithm in computer science. Binary search is a fundamental algorithm in computer science. Binary search is a fundamental algorithm in computer science. Binary search is a fundamental algorithm in computer science. Binary search is a fundamental algorithm in computer science. Binary search is a fundamental algorithm in computer science. Binary search is a fundamental algorithm in computer science. Binary search is a fundamental algorithm in computer science. Binary search is a fundamental algorithm in computer science. Binary search is a fundamental algorithm in computer science. Binary search is a fundamental algorithm in computer science. Binary search is a fundamental algorithm in computer science. Binary search is a fundamental algorithm in computer science. Binary search is a fundamental algorithm in computer science. Binary search is a fundamental algorithm in computer science. Binary search is a fundamental algorithm in computer science. Binary search is a fundamental algorithm in computer science. Binary search is a fundamental algorithm in computer science. Binary search is a fundamental algorithm in computer science. Binary search is a fundamental algorithm in computer science. Binary search is a fundamental algorithm in computer science. Binary search is a fundamental algorithm in computer science. Binary search is a fundamental algorithm in computer science. Binary search is a fundamental algorithm in computer science. r+r+r+b/home/ubuntu/.local/lib/python3.10/site-packages/sglang/test/longbench_v2/validate_longbench_v2.pycreate_sample_official_datasXr-cCs,ddgddddddd gd dd d dgS) zJCreate sample data in alternative format (choices as list) for validation.alt_001zWhat is 2 + 2?)3456rsingle_document_qaafBasic arithmetic operations. Basic arithmetic operations. Basic arithmetic operations. Basic arithmetic operations. Basic arithmetic operations. Basic arithmetic operations. Basic arithmetic operations. Basic arithmetic operations. Basic arithmetic operations. Basic arithmetic operations. Basic arithmetic operations. Basic arithmetic operations. Basic arithmetic operations. Basic arithmetic operations. Basic arithmetic operations. Basic arithmetic operations. Basic arithmetic operations. Basic arithmetic operations. Basic arithmetic operations. Basic arithmetic operations. Basic arithmetic operations. Basic arithmetic operations. Basic arithmetic operations. Basic arithmetic operations. Basic arithmetic operations. Basic arithmetic operations. Basic arithmetic operations. Basic arithmetic operations. Basic arithmetic operations. Basic arithmetic operations. )rrchoicesrcategoryralt_002zWhat color is the sky?)RedBlueGreenYellowmulti_document_qaaColor perception and atmospheric science. Color perception and atmospheric science. Color perception and atmospheric science. Color perception and atmospheric science. Color perception and atmospheric science. Color perception and atmospheric science. Color perception and atmospheric science. Color perception and atmospheric science. Color perception and atmospheric science. Color perception and atmospheric science. Color perception and atmospheric science. Color perception and atmospheric science. Color perception and atmospheric science. Color perception and atmospheric science. Color perception and atmospheric science. Color perception and atmospheric science. Color perception and atmospheric science. Color perception and atmospheric science. Color perception and atmospheric science. Color perception and atmospheric science. Color perception and atmospheric science. Color perception and atmospheric science. Color perception and atmospheric science. Color perception and atmospheric science. Color perception and atmospheric science. Color perception and atmospheric science. Color perception and atmospheric science. Color perception and atmospheric science. Color perception and atmospheric science. Color perception and atmospheric science. Color perception and atmospheric science. Color perception and atmospheric science. Color perception and atmospheric science. Color perception and atmospheric science. Color perception and atmospheric science. Color perception and atmospheric science. Color perception and atmospheric science. Color perception and atmospheric science. Color perception and atmospheric science. Color perception and atmospheric science. r+r+r+r+r,create_alternative_format_dataFs r<c@sbeZdZdZdeeeffddZdededeeeffdd Zd eeeefdefd d Z d S) MockSamplerz call_count)selfr>r+r+r,__init___s zMockSampler.__init__contentrolercCs ||dS)N)rBrCr+)r@rBrCr+r+r, _pack_messagecs zMockSampler._pack_messagemessagescCsf|dd}|jd7_d|vrdSd|vrdSd|vrdSd |vr%dSd |vr+dSd |vr1dSd S) z5Return a mock response based on the question content.rrBz atomic nucleiThe correct answer is (B)zliterary techniquezThe correct answer is (C)z binary searchz2 + 2zcolor is the skyComplex reasoning questionzThe correct answer is (A))r?)r@rEpromptr+r+r,__call__fs zMockSampler.__call__N) __name__ __module__ __qualname____doc__rstrrArDrrJr+r+r+r,r=\s "r=cCstddddddddd }t|}d|vsJd|vsJd |vs$Jd |vs*Jd |vs0Jtd ddgddd}t|}d|vsGJd |vsMJtddS)zLTest that our implementation handles official LongBench-v2 format correctly.z(Testing official format compatibility...z Test contextzTest question?Option AOption BOption COption DA)rrrrrrrz (A) Option Az (B) Option BzThe correct answer isu*✓ Official format compatibility verified)rPrQrRrS)rrr4ru-✓ Alternative format compatibility verifiedN)printr)official_sample formatted alt_sample formatted_altr+r+r,test_format_compatibilityzs2        rZcCsTtdgd}|D]\}}t|}||ks#Jd|d|d|q tddS)z5Test answer extraction with various response formats.zTesting answer extraction...))rGr)zThe correct answer is Cr$)z)After analysis, The correct answer is (D)D)z*The correct answer is (A)*rT)zI think the answer is Br)zNo clear answer hereNz Failed for 'z': got z , expected u✓ Answer extraction verifiedN)rUr) test_casesresponseexpectedresultr+r+r,test_answer_extractions   r`c Cstdt}tjdddd}t|||j}Wdn1s#wYz>t|ddd }ti}||}|j d ksAJd t |j dksLJd d |j vsUJdtd|j ddWt |dSt |w)z5Test the complete evaluation pipeline with mock data.zTesting evaluation pipeline...w.jsonFmodesuffixdeleteNrF) data_source num_examples num_threadsrzExpected positive scorez"Expected 3 evaluated conversationscharszExpected chars metricu)✓ Evaluation pipeline verified (score: .3f))rUr-tempfileNamedTemporaryFilejsondumpnamerr=scorelenconvosmetricsosunlink) official_dataf temp_fileeval_obj mock_samplerr_r+r+r,test_evaluation_pipelines r~c Cstdt}tjdddd}t|||j}Wdn1s#wYz*t|dgdd }t|j dkss&3  #    & /