o پiI@sdZddlZddlZddlZddlmZddlZddlZddlm Z ej GdddZ de fdd Z e d krMeZe ee eZe edSdS) z Run one test prompt. Usage: python3 -m sglang.test.send_one python3 -m sglang.test.send_one --profile --profile-steps 5 python3 -m sglang.test.send_one --profile --profile-by-stage N)Optional) run_profilec@s"eZdZUdZeed<dZeed<dZeed<dZ e ed<d Z e eed <d Z eed <d Zeed<d Zeed<d Zeed<dZe ed<dZe ed<dZeed<dZe ed<dZe ed<dZe ed<dZe ed<dZeed<dZe ed<d Ze eed<edejfddZe d ej!fd!d"Z"d S)# BenchArgs localhosthosti0uport batch_sizeFdifferent_promptsNseedg temperatureimax_new_tokensfrequency_penaltypresence_penaltyjsonreturn_logprobzSHuman: Give me a fully functional FastAPI server. Show the python code. Assistant:promptimage many_imagesstreamprofile profile_stepsprofile_by_stageprofile_prefixparsercCs>|jdttjd|jdttjd|jdttjd|jddtjd|jdttjd|jd t tj d|jd ttj d|jd t tj d|jd t tj d|jd dd|jddd|jdttjd|jddd|jddd|jddd|jddd|jdttjd|jddd|jdttjddS)Nz--host)typedefaultz--portz --batch-sizez--different-prompts store_true)actionrz--seedz --temperaturez--max-new-tokensz--frequency-penaltyz--presence-penaltyz--json)rz--return-logprobz--promptz--imagez --many-imagesz--streamz --profilez--profile-stepsz--profile-by-stagez--profile-prefix) add_argumentstrrrintrr r r floatr r rrrrr)rr$H/home/ubuntu/.local/lib/python3.10/site-packages/sglang/test/send_one.py add_cli_args-sB zBenchArgs.add_cli_argsargscs0ddt|D}|difdd|DS)NcSsg|]}|jqSr$)name.0attrr$r$r% Ssz+BenchArgs.from_cli_args..csi|]}|t|qSr$)getattrr)r'r$r% Tsz+BenchArgs.from_cli_args..r$) dataclassesfields)clsr'attrsr$r.r% from_cli_argsQszBenchArgs.from_cli_args)#__name__ __module__ __qualname__rr!__annotations__rr"r r boolr rr r#r rrrrrrrrrrrr staticmethodargparseArgumentParserr& classmethod Namespacer4r$r$r$r%rs2                #rr'c sd|jd|j}|jrd|_d}n |jrd|_gd}nd}|j|jr+dd}nd}|jd krH|js<g|jn fd d t|jD||j |j |j |j |j |gd d |j|jd}|jr{td|jdt||jddg|j|jdtj|d||jd}|jrd}|jddD]5}|d}|r|dr|dkrn"t|ddd}|d|d} t|d}t| dd d!qn|}|jd kr|d}|jd"krt|d#Sd$|d%vr|d%d$dkr|d%d&|d%d$} nd'} |d%d(} |d%d&| } |d%d&} |jst|dtgd)}| d*| | d*| d+gg}tj||d,d-}t|| | fS).Nzhttp://:z@Human: Describe this image in a very short sentence. Assistant:[https://raw.githubusercontent.com/sgl-project/sglang/main/examples/assets/example_image.pngzsHuman: I have one reference image and many images.Describe their relationship in a very short sentence. Assistant:)r@r@r@r@zHuman: What is the capital of France and how is that city like. Give me 3 trivial information about that city. Write in a format of json. Assistant:z$$ANY$$rcs g|] }d|ddqS)z Test case rz: r$)r*irr$r%r,~s z#send_one_prompt..)Questionz Assistant:z <|separator|>z<|eos|>) sampling_seedr r rr json_schemastop)text image_datasampling_paramsrrzRunning profiler with z steps...CPUGPU)url num_steps activitiesrrz /generate)rrrF)decode_unicodezutf-8zdata:z data: [DONE] rGT)endflush)rrspec_verify_ct meta_infocompletion_tokensg? e2e_latency)z Latency (s)Tokensz Acc LengthzSpeed (token/s)z.3fz.2fpretty)headerstablefmt) rrrrrrr r ranger r r rrrrrprintrrrrrequestspost iter_linesdecode startswithloadsstriplen status_codetabulate)r'base_urlrHrE json_dataresponselast_lenchunkret chunk_str acc_lengthlatencyspeedtokensr\rowsmsgr$rBr%send_one_promptWs           rw__main__)__doc__r;r0rtypingrr`risglang.profilerr dataclassrrwr5r<rr&r4 parse_argsr'r$r$r$r%s"   Ax