o پi2J@sdZddlZddlZddlZddlZddlZddlm Z ddl m Z m Z e Z ddZddZd d Zd d Zd dZddZd,ddZddZddZddZd,ddZddZddZd d!Zd"d#Zd$d%Zd&d'Zd(d)Zd*d+Z dS)-z:This file contains the SGL programs used for unit testing.N)is_hip)download_and_cache_file read_jsonlcCstjdd}|jdd}d|dvs Jd|d|jdd idd idd igd d }dd|D}|gdksDJd|dS)NcSsd|d7}|d7}|d7}|d7}|d7}|d7}|d7}|d|d 7}|d tjd d d d 7}dS)Nz+The following are questions with answers. z"Q: What is the capital of France? z A: Paris z#Q: What is the capital of Germany? z A: Berlin z!Q: What is the capital of Italy? zA: Rome zQ:  zA:answerr)stop temperaturesglgensquestionrM/home/ubuntu/.local/lib/python3.10/site-packages/sglang/test/test_programs.py few_shot_qasz%test_few_shot_qa..few_shot_qa)What is the capital of the United States?)r washingtonrzanswer: rzWhat is the capital of Japan?z*What is the capital of the United Kingdom?z"What is the capital city of China?皙?rcSsg|] }|dqS)r)striplower).0xrrr (sz$test_few_shot_qa..)tokyolondonbeijingz answers: )r functionrunrr run_batch)rretretsanswersrrrtest_few_shot_qas  &r$cCs@tjdd}d}d}|j||ddd}t|dvsJdS) NcSs|td7}|t|7}|ttd7}| ||7}Wdn1s+wY||td7}WddS1sGwYdS)NYou are a helpful assistant.answer_1answer_2r systemuser assistantr )r question_1 question_2rrranswer_mt_bench-s   "z&test_mt_bench..answer_mt_benchCompose an engaging travel blog post about a recent trip to Hawaii, highlighting cultural experiences and must-see attractions.zGRewrite your previous response. Start every sentence with the letter A.gffffff?@)r,r-rmax_new_tokens))r rrlenmessages)r.r,r-r!rrr test_mt_bench,s  r6cCstjdd}|jdd}|r|ddksJ|n|ddvs$J|jdd}|r9|dd ks8J|n|ddvsAJ|jd d}|rW|dd ksUJ|dS|ddvs_JdS) NcSsD|d7}|d7}|d7}|d|d7}|dtdgd7}dS) NzBDetermine whether the statement below is True, False, or Unknown. z+Statement: The capital of France is Pairs. z Answer: True z Statement: rzAnswer:rTrueFalseUnknown)r select)r statementrrr true_or_falseBs z"test_select..true_or_falsez!The capital of Germany is Berlin.)r<rr8r7zThe capital of Canada is Tokyo.r9z$Purple is a better color than green.r:)r rrtext) check_answerr=r!rrr test_selectAs( r@cCsVtjdd}|jdd}t|ddksJ|t|ddks)J|dS) NcSs0|dtdd7}|dtdd7}dS)Nz The number of hours in a day is hoursrz The number of days in a year is days)r gen_intr rrr decode_intdsz#test_decode_int..decode_intrrrArBim)r rrintr>)rEr!rrrtest_decode_intcs    rHcCxtjdd}|jdd}z t|d}Wntjjy'td|dwt|dt s1Jt|dt s:JdS) NcSsddlm}m}m}|d7}|d7}|dJ|d7}|dtj|dd 7}|d tj|d d gd d 7}|dtj|d d gd d 7}|dtj|d d gd d 7}|d7}WddS1sdwYdS)Nr) REGEX_FLOAT REGEX_INT REGEX_STRHGenerate a JSON object to describe the basic city information of Paris. zHere are the JSON object: json_output{ "name": regex,  "population":  r)rRr "area": z "latitude": })sglang.lang.irrJrKrL var_scoper r )r rJrKrLrrr decode_jsonos  "z+test_decode_json_regex..decode_jsongrrNJSONDecodeErrorname population r rrjsonloadsdecoderr[print isinstancestrrGrZr!js_objrrrtest_decode_json_regexns  rgcCrI) NcSs|d7}|dG|d7}|dtd7}|dtd7}|dtjtdd7}|d td7}|d tjtdd 7}|d 7}WddS1sSwYdS) NrMrNrOrPrSrTrVdtypez "country": z "timezone": rrW)rYr gen_stringrCr rGrdrDrrrrZs  "z%test_decode_json..decode_jsonr0)r1rNr[r\r]r^rerrrtest_decode_jsons  rkTcCsJtjdd}|jddd}|r!d|vs#Jd|dSdS)NcSsX|d|d7}|dtjdddgdd7}|d|dd tjd ddd7}dS) N Question: rz(A good person to answer this question isexpert.r. z For example,z would answer that rr r rrr expert_answers& z)test_expert_answer..expert_answerzWhat is the capital of France?rrrpariszAnswer: )r rrr>r)r?rqr!rrrtest_expert_answers "rtcsLddtjfdd}d\}}|||dd}t|d||ks$JdS) NcSs t|S)N)eval) expressionrrr calculates z test_tool_use..calculatecs|d7}|d7}|dt|dt|d7}|dtjddd d 7}|d ||d7}WddS1s>wYdS) NzfPlease perform computations using a calculator. You can use calculate(expression) to get the results. z0For example, calculate(1+2)=3 calculate(3*4)=12 z!Question: What is the product of z and z? z Answer: The answer is calculate(rv)roz) = r)rdr r rY)r lhsrhsrwrrtool_uses   "ztest_tool_use..tool_use)iir)ryrzrr)r rrG)r|ryrzr!rr{r test_tool_uses  r}cCsDtjdd}|jddd}|d}d|vsd|vs JdSdS) NcSs|d7}|d|d7}tddD]^}|d|dtjddgd d 7}|d |d td |ddg7}|d |dkr`|dtjdd d7}|d|dtjddgd d 7}q|dtjddd d7}dSdS)Na Question: Which country does the founder of Microsoft live in? Thought 1: I need to search for the founder of Microsoft. Action 1: Search [Founder of Microsoft]. Observation 1: The founder of Microsoft is Bill Gates. Thought 2: I need to search for the country where Bill Gates lives in. Action 2: Search [Where does Bill Gates live]. Observation 2: Bill Gates lives in the United States. Thought 3: The answer is the United States. Action 3: Finish [United States]. rlrr3zThought :rnrorpzAction : action_SearchFinishz []z]. z Observation r)ranger r r;)r rirrrreacts $&&ztest_react..reactz/What country does the creator of Linux live in?rrrrfinlandstates)r rrr)rr!rrrr test_reacts  rcs@ddtjfdd}|jddd}t|dtsJdS) Nr0r3c s|d7}|d|d7}|dtd7}tddD]}||dtjddd gd d7}q|}tD]$}||d |dd |dd 7<||tjddgd7<q;||d7}tD]}|d|dd ||dd 7}ql|dtjddd7}dS)NzAct as a helpful assistant. zUSER: Give some tips for rpzASSISTANT: Okay. Here are z# concise tips, each under 8 words: r~rnr) max_tokensrzNow, I expand tip z into a detailed paragraph: Tip r detailed_tip roz/Here are these tips with detailed explanation: zTip z In summary,summaryir)rdrr r forkjoin)r topicrforks fork_sizerrrparallel_decodings0&    &z1test_parallel_decoding..parallel_decodingzwriting a good blog postg333333?)rrr)r rrrcrd)rr!rrrtest_parallel_decodings rcsJdtjfdd}|jdddddd }|d }|r!d |vs#JdSdS) Nr0cs~|d7}|d7}|d7}|||g|t}|fdd7}|jdd|d7}|d |d 7}|d tjd d 7}dS)Nz6USER: I will ask a question based on some statements. z)ASSISTANT: Sure. I will give the answer. z(USER: Please memorize these statements. csd|d|dS)Nz Statement rrr)rcontextsrr%szCtest_parallel_encoding..parallel_encoding..concate_and_append)modez?Now, please answer the following question. Do not list options.z Question: rz ASSISTANT:rr)rr4rr r )r r context_0 context_1 context_2rrrrparallel_encodings  z1test_parallel_encoding..parallel_encodingzWho is the father of Julian?zEthan is the father of Liam.zNoah is the father of Julian.zOliver is the father of Carlos.r)rrrrrrNoah)r rr)r?rr!rrrrtest_parallel_encodings rcCsdtjdd}|jdddd}d|dd vs.d |dd vs0J|dd dSdS) NcSs0|ttd|7}|ttd7}dS)Nzexample_image.pngr)r r*imager+r r rrrimage_qa:sztest_image_qa..image_qaz+Please describe this image in simple words.rr0)rrr1taxicontentcar)r rrr5)rstaterrr test_image_qa9s rcCs`tjdd}|ddd}d}|D]}||7}q|ddd}d}|dD]}||7}q'dS)NcSs4|td7}|t|7}|ttd7}dS)Nr%rr(r rrrqaLsztest_stream..qar/T)rstreamr)r r text_iter)rr!outchunkrrr test_streamKs"    rcs:dtjfdd}|}|d}t|sJdS)Nz@((25[0-5]|2[0-4]\d|[01]?\d\d?).){3}(25[0-5]|2[0-4]\d|[01]?\d\d?)cs$|d7}|dtjddd7}dS)Nz5Q: What is the IP address of the Google DNS servers? A: rr)rrRr rDrQrr regex_genfs ztest_regex..regex_genr)r rrrematch)rrrrrQr test_regexcs rcCsftjdd}|}zt|d|d<t|d|d<t|d|d<WdSty2t|w)NcSs|d7}|dtjdtddd7}|d7}|dtjdtd d7}|d 7}|dtjd td d7}|d 7}|dtjd td d7}dS)Nz!Q: What is the full name of DNS? zA: The full names is str_resr)rirz Q: Which year was DNS invented? rint_resrhzQ: What is the value of pi? float_reszQ: Is the sky blue? bool_res)r r rdrGfloatboolrDrrr dtype_genusz!test_dtype_gen..dtype_genrrr)r rrrGrr ValueErrorrb)rrrrrtest_dtype_gents   rcCsztjdddd}tjdd}tjjj}|||j}|||j}||ks;J|d|dS)Nr0num_api_spec_tokenscS^|d7}|d7}|d7}|dtjddddtjd dd7}|d tjd ddd7}dS Nz3Construct a character within the following format: ?Name: Steve Jobs. Birthday: February 24, 1955. Job: Apple CEO. z- Please generate new Name, Birthday and Job. Name:r\rro Birthday:birthday Job:jobr rDrrrgen_character_spec  z7test_completion_speculative..gen_character_speccSrrr rDrrrgen_character_no_specrz:test_completion_speculative..gen_character_no_specz vs )r r global_configdefault_backend token_usageresetsync prompt_tokens)rrrusage_with_specusage_with_no_specrrrtest_completion_speculatives        rcCs"tjdddd}|dS)NrcSs||td7}|td7}|td7}|td7}|tdtjdddd tjd ddd tjd dd7}dS) Nr%z2Construct a character within the following format:rz,Please generate new Name, Birthday and Job. rr\rrorrrrr(rDrrrrs&    z.gen_character_spec)r rr)rrrr test_chat_completion_speculatives  rcsddfdd}d}t|}tt|}d}d}|||g}g}g}tt|d|D]} ||| d ||| d ||| d q/d d t||D} ddljfdd} t } | j | dddd d} g}t | D]\} }|||  |dqyt | }tt|t|k}t } | j | ddddd} g}t | D]\} }|||  |dqt | }tt|t|k}td|d|t||dksJtst||dksJ ||fS||fS)z>Benchmark the accuracy of sgl.select on the HellaSwag dataset.cSsD||dd||dd}|r |||d||d7}|S)Nactivity_labelrctxrUendingslabelr)linesrinclude_answerr!rrrget_one_examples z.test_hellaswag_select..get_one_examplecs*d}t|D] }|||dd7}q|S)NrTr)r)rkr!r)rrrget_few_shot_exampless z4test_hellaswag_select..get_few_shot_exampleszRhttps://raw.githubusercontent.com/rowanz/hellaswag/master/data/hellaswag_val.jsonlNFrrcSsg|] \}}||dqS))rchoicesr)rqcrrrrsz)test_hellaswag_select..rcs"||7}|jd|d7}dS)Nr)r)r;)r rr)few_shot_examplesr rrfew_shot_hellaswags z1test_hellaswag_select..few_shot_hellaswagr0T)r num_threads progress_bargenerator_stylerz accuracy=z, accuracy_gen=rr~)rlistrrr4appendzipsglangrtime perf_counterr enumerateindexnpmeanarrayrbabs_is_hip)rurlfilenamer num_questions num_shots questionsrlabelsr argumentsrticr"predsr!latencyaccuracy preds_gen latency_gen accuracy_genr)rrr rtest_hellaswag_selectsb      rcsnddlddlm}jjj}d\jfdd}fdd}||}|}|||d dS) av Validate sgl.gen(min_tokens) functionality. The test asks a question where, without a min_tokens constraint, the generated answer is expected to be short. By enforcing the min_tokens parameter, we ensure the generated answer has at least the specified number of tokens. We verify that the number of tokens in the answer is >= the min_tokens threshold. rN) get_tokenizer)r0cs,|d7}|jdd7}dS)Nrr) min_tokensr)r*r+r rD MAX_TOKENS MIN_TOKENSr rrconvo_12s z(test_gen_min_new_tokens..convo_1cs8||}t|ksJdt|dd|dS)Nz Generated z tokens, min required: z. Text: )encoder4) tokenizerr> token_ids)r rrassert_min_tokens9s z2test_gen_min_new_tokens..assert_min_tokensr) r&sglang.srt.utils.hf_transformers_utilsrrrendpointget_model_namerr)r model_pathrrrrrr rtest_gen_min_new_tokens$s  r)T)!__doc__r_rrnumpyrrr sglang.srt.utilsr sglang.utilsrrrr$r6r@rHrgrkrtr}rrrrrrrrrrrrrrrs8 "  $ & * Y