o پi5@sddlZddlZddlZddlZddlZddlZddlZddlZddlZddl m Z ddl m Z m Z mZmZddlZddlZddlmZddlmZddlmZmZddlmZmZmZddlmZdd lm Z dd l!m"Z"dd l#m$Z$m%Z%dd l&m'Z'm(Z(d Z)de*dee+fddZ,dee+dee+dee-fddZ.ej/GdddZ0GdddeZ1de de"fddZ2de de"fddZ3  ! dLde*d"e e e4d#e4d$e-d%e*d&ee d'e*d(ee*fd)d*Z5d+e0j6d+e0j7e0j8e0j9e0j:d+e0j;d!de0je0j?fde*d,e4d#e4d-e4d.e-d/e@d0e4d1e-d2e*d3e*d4eeBd5e@d6e4d7e@d8ee*d9ee*d%e*d:e*d;e@d$e-d'e*d(ee*de4d?e4f4d@dAZAdBdCZBdDdEZCdFe e1dGe0de"fdHdIZDe fde"dGe0de fdJdKZEdS)MN)SimpleNamespace)CallableListOptionalTuple) BaseModel)tabulate) AutoProcessorPreTrainedTokenizer) get_dataset get_processor get_tokenizer) run_profile) launch_server) ServerArgs) is_blackwellkill_process_tree)is_in_ciwrite_github_step_summaryiXurlreturnc CszLtj|ddd}|d}d}|jdD]/}|dr1td|}|r0|t| d7}q|d rGtd |}|rG|t| d7}q||fWSt yf}zt d |WYd }~d Sd }~ww) z Get cached_tokens_total and prompt_tokens_total from Prometheus /metrics endpoint. Returns (cached_tokens_total, prompt_tokens_total) or None if metrics are not available. z/metricstimeout zsglang:cached_tokens_total{z2sglang:cached_tokens_total\{[^}]*\}\s+([\d.eE+-]+)zsglang:prompt_tokens_total{z2sglang:prompt_tokens_total\{[^}]*\}\s+([\d.eE+-]+)z2Warning: Failed to get cache tokens from metrics: N) requestsgetraise_for_statustextsplit startswithresearchfloatgroup Exceptionprint)rresponsecached_tokens_totalprompt_tokens_totallinematcher/_/home/ubuntu/.local/lib/python3.10/site-packages/sglang/test/bench_one_batch_server_internal.pyget_cache_tokens_from_metrics!s4   r1beforeaftercCsH|dus|dur dS|d|d}|d|d}|dkr"||SdS)z Calculate cache hit rate from before/after metrics snapshots. Returns cached_tokens_delta / prompt_tokens_delta for the benchmark run. Nrrr/)r2r3 cached_delta prompt_deltar/r/r0calculate_cache_hit_rateDsr6c@seZdZUdZeed<dZeeed<dZ eeed<dZ eeed<d Z e ed <d Z eed <d Zeed<d Ze ed<dZeed<d Zeed<d Zeed<d Zeed<dZeed<d Zeed<dZeeed<dZeeed<dZeed<dZeed<d Zeed<dZeed <d!Zeed"<d#Zeed$<d Z eed%<d&Z!eed'<dZ"eeed(<d)Z#eed*<d+Z$eed,<d Z%e ed-<d.Z&eed/<dZ'ee(eed0<e)d1e*j+fd2d3Z,e-d4e*j.fd5d6Z/dS)7 BenchArgsdefaultrun_name)r batch_size) input_len) output_lenr temperatureFreturn_logprobrclient_stream_intervalinput_len_step_percentagebase_url skip_warmup show_reportprofiler profile_stepsprofile_by_stageNprofile_prefixprofile_output_dir dataset_pathrandom dataset_namegsp_num_groupsigsp_system_prompt_lengsp_question_lengsp_output_lenparallel_batchz result.jsonlresult_filenamepydantic_result_filenameTappend_to_github_summary*seedcache_hit_ratesglangbackendserver_args_for_metricsparsercCs0|jdttjd|jdtdtjd|jdtdtjd|jdtdtjd|jdttj d|jd d d |jd ttj d|jd ttj d|jdttj d|jdd d |jdd d |jdd d |jdttj d|jdd d |jdttjd|jdttjd|jdttjdd|jdttjgddd|jdttjdd|jdttjd d|jd!ttjd"d|jd#ttjd$d|jd%d d |jd&ttjd'd|jd(ttjd)d|jd*d+d,d-d.|jd/ttjd|jd0ttjd1d|jd2ttjd3d4gd5d|jd6td7dd8d9dS):Nz --run-name)typer8z --batch-size+)r`nargsr8z --input-lenz --output-lenz --temperaturez--return-logprob store_true)actionz--client-stream-intervalz--input-len-step-percentagez --base-urlz --skip-warmupz --show-reportz --profilez--profile-stepsz--profile-by-stagez--profile-prefixz--profile-output-dirz--dataset-pathzPath to the dataset.)r`r8helpz--dataset-name)mmmurMgenerated-shared-prefixz$Name of the dataset to benchmark on.)r`r8choicesrez--gsp-num-groupszRNumber of shared prefix groups. batch_size requests are distributed across groups.z--gsp-system-prompt-lenz7Length of the shared system prompt in tokens per group.z--gsp-question-lenz;Length of the unique question suffix in tokens per request.z--gsp-output-lenz=Output length in tokens for generated-shared-prefix requests.z--parallel-batchz--result-filenamezDStore the results line by line in the JSON Line format to this file.z--pydantic-result-filenamezEStore the results as pydantic models in the JSON format to this file.z--no-append-to-github-summary store_falserXz=Disable appending the output of this run to github ci summary)rddestrez--seedz--cache-hit-ratezzCache hit rate for benchmarking (0.0-1.0). 0.0 means no cache hits (flush all), 0.4 means 40%% of input tokens are cached.z --backendr\vllmz%Backend server type (sglang or vllm).z--server-args-for-metrics*zRServer launch arguments to record in metrics output (for tracking configurations).)r`rbr8re) add_argumentstrr7r9intr:r<r>r%r?rArBrDrHrJrKrLrNrOrPrRrTrVrWrZr[r])r_r/r/r0 add_cli_argsws    zBenchArgs.add_cli_argsargscs0ddt|D}|difdd|DS)NcSg|]}|jqSr/)name.0attrr/r/r0 z+BenchArgs.from_cli_args..csi|]}|t|qSr/)getattrrtrqr/r0 z+BenchArgs.from_cli_args..r/) dataclassesfields)clsrqattrsr/rzr0 from_cli_argsszBenchArgs.from_cli_args)0__name__ __module__ __qualname__r9rn__annotations__r:rror<r>r?r%r@boolrArBrDrErFrGrHrIrJrrKrLrNrOrPrRrTrUrVrWrXrZr[r]r^r staticmethodargparseArgumentParserrp classmethod Namespacerr/r/r/r0r7VsF                       xr7c@seZdZUeed<eed<eed<eed<eed<eed<eed<eed<eed <eed <eed <d Zeeed <d Z eeed<defddZ d S)BenchOneCaseResultr9r:r<r>latencyinput_throughputoutput_throughputoverall_throughput last_ttftlast_gen_throughput acc_lengthNr[ profile_linkrVcCst|dM}|j|j|j|jt|jdt|jdt|jdt|j dt|j dt|j dt|j d|j durrrrrrrrr[r)openr9r:r<r>roundrrrrrrrr[writejsondumps)selfrVfoutresr/r/r0 dump_to_jsonls$          "z BenchOneCaseResult.dump_to_jsonl) rrrrnrror%r[rrrr/r/r/r0rs rlaunch_server_func server_argsc CsTz z||Wnty}z|d}~wwWttdddSttddw)NF)include_parent)r'rosgetpid)rrr.r/r/r0launch_server_internals &rcCstjt||fd}|d|jd|j}t}t|tkrVzddi}tj |d|td}|j dkr=||fWSWn tj yHYnwt d t|tks#t d ) N)targetrqzhttp://:z Content-Typezapplication/json; charset=utf-8 /v1/models)headersr z1Server failed to start within the timeout period.)multiprocessingProcessrstarthostporttimeDEFAULT_TIMEOUTrr status_codeRequestExceptionsleep TimeoutError)rrprocrD start_timerr)r/r/r0launch_server_process%s2     rrMr\ input_idsr<r[rN image_datar] model_namec st||dkr dStd|ddddfdd |D}|d kr4||d d d d} |d} n|d d ddd d} |dkrJ|durJ|| d<|d} tj| | td} | tddS)aWarm up the cache by sending prefix tokens to populate the radix/prefix cache. Args: url: Server URL input_ids: List of input token id lists input_len: Length of input tokens cache_hit_rate: Fraction of input tokens to cache (0.0-1.0) dataset_name: Name of the dataset (used to determine if image data should be included) image_data: Optional image data for VLM models backend: Backend server type ("sglang" or "vllm") model_name: Model name (required for vllm backend) rNzWarming up cache with d.1fz % hit rate (z tokens per request)csg|]}|dqSNr/ruidscached_token_lenr/r0rw`r|z!_warmup_cache..rkrrF)modelprompt max_tokensr?stream/v1/completionsT)r?max_new_tokens ignore_eos)rsampling_paramsrrfr /generate)rrzCache warmup completed)ror(rpostrr) rrr<r[rNrr]rcache_warmup_input_idscache_warmup_payloadgen_urlwarmup_responser/rr0 _warmup_cacheAsD    rFr:r>r?r@stream_intervalrBr9rV tokenizerrGrHrIrJrKrLrUrOrPrRrTc=s|dkrtj|dtd}|n tj|dtd}|d}||vr/td|d|t||}t||||d||d v|tj|||d ||||d }t d |p\t d d}t ||d}|dkr|d|}fdd|D} t dd| Dt | }|}d}!n|dkrfdd|D} dd|D}!n dd|D} d}!|dkr|| ||ddd}"|rd |"d<|d}#n?d}$|$rg}%t |D] }&|%dddqd }'nd}'||d|'|d!|dd"|rd#|ini}"| |"d$<|!dur|!|"d%<|d&}#|d'krt|| ||||!||d(d}(| r t|| d)d*g|| |d+}(t|})t}*tj|#|"dtd,}|d'}+|dkrt},|jdd-D]W}-|-d.}-|-r|-d/r|-d0d}.|.d1krdn9t|.}/d2|/vrvtd3|/d4|/d5gD]}0|0d6}1|1|,vr|,|1t |,|krt|*}+q|qEn]|jdd-D]V}-|-d.}-|-r|-d/r|-d7krn@t|-d0dd8}/d2|/vrtd3|/d4|/d9d:dus|/d9d:d;d}6d>}7n.tj|d?td}||}8|8d@ig}9|9dAdBdpCd>}6|9dAdCdpNd>}7t|}:t|)|:};tdD|tdE|tdF|tdG|2dHdItdJ|3dHdK|d krtdL|4dHdKtdM|+dHdItdN|6dHdK|7dAkrtdO|7dHdP|;durtdQ|;dRt |||||2|3|4|5|+|6|7|;|(dS }<| r|.css|]}t|VqdSr)lenrr/r/r0 szrun_one_case..rfcrr/rr) tok_innerr/r0rwr|cSrrr/)rrr/r/r0rwrxcSrrr/)rrr/r/r0rwrxT)rrrr?rrlogprobsrFzHuman: What is the capital city of france? can you give as many trivial information as possible about that city? answer in json. 2z Assistant:z$$ANY$$)r?rr json_schemar)rr@rrUrrrr)rrr<r[rNrr]rCPUGPU)r num_steps activities output_dirrIrJ)rrr)decode_unicodezutf-8zdata:rz[DONE]errorzRequest has failed. .rhindexz data: [DONE]r meta_info finish_reasonr`lengthcompletion_tokens/get_server_infointernal_statesrravg_spec_accept_lengthz batch size: z input_len: z output_len: z latency: .2fz szinput throughput: z tok/szoutput throughput: z last_ttft: zlast generation throughput: z acc_length:  zcache hit rate: .4f) r9r:r<r>rrrrrrrr[r)"rrrr ValueErrorminrr7rZryr sumrrangeappendrrr1r perf_counterset iter_linesdecoder"striprloads RuntimeErrorraddr6r(rr)=rr:r<r>r?r@rrBr9rVrrGrHrIrJrKrNrLrUr[r]rrOrPrRrTr)supported_datasetsactual_gsp_groups dataset_argsdataset_model_idinput_requestsrrpayloadruse_structured_outputstexts_rrmetrics_beforeticrfirst_token_indiceschunkdata_strdatachoiceidxrrrrrr server_infointernal_state metrics_aftermetrics_cache_hit_rateresultr/)rrr0 run_one_cases|                                r c CsP||||kr&tdd|d|d|d|||d|d ddSd S) N========Skip benchmark batch_size=z * (input_len=z + output_len=z) = z! > skip_token_capacity_threshold=z due to kv cache limit.TFr()r:r<r>skip_token_capacity_thresholdr/r/r0!should_skip_due_to_token_capacitys*r%cCs.||krtdd|d|dddSdS)Nr!r"z' > skip_max_running_requests_threshold=z# due to max running requests limit.TFr#)r:#skip_max_running_requests_thresholdr/r/r0'should_skip_due_to_max_running_requestssr'results bench_argsc Csd|jd|jd}|jdkr|d|jddd7}|d 7}tr&d }nd }d }|jd ddg}gd}|jr@|d|D]s}||j} |jdkrS|jdnd} d|j |j } d|j |d| } d|j d| } |jdur{|jdnd}|j |j|j d|j d|j d| | d| d| d|g }|jr|j r|d|j dn|d||qB|t||dd7}|d 7}|S)Nz Input lens: z. Output lens: rrz Cache hit rate: rrz%.rrrgffffff?cSs|jSr)r<)xr/r/r0sz$get_report_summary..)key) z batch sizez input lenz latency (s)zinput throughput (tok/s)zoutput throughput (tok/s)z acc lengthzITL (ms)zinput cost ($/1M)zoutput cost ($/1M)zcache hit raterGrrzn/aig.Airz [Profile]()github)rtablefmt)r<r>r[rsortrGrtp_sizerr:rrrrr)r(r)rsummaryhourly_cost_per_gpu input_utilrowsrr hourly_cost accept_lengthitl_ms input_cost output_costr[rowr/r/r0get_report_summarysR      r<cCst|jtj|j|jrd|j}}nt||\}}|jdkrztd|dtj|dt d}| | dg}|sFt d|dd }td |td |d|j d kret|}nt|}td d|i} td} td} nhd}tj|dt d}| | } d| vr| d} n d| vr| ddd} |j d krt| }nt| }| dig} | ddidd} | ddd}| ddpd}|dksJd|||} t|j|j|j|jd}td|td|td| td | |jsJtt|j}td!td"||D]#}t|f|d#d$|j|j|j|jd%d%||j |j|j |j|d&|q"td'g}g}zt!"|j|j#|j$D]=\}}}t%|| smt&|||| roqZ|'t||||f|j|j|j|j|j(|j)||j |j|j |j*|j|d( |qZ|j+rLzt!"|j|j#|j$D]w\}}}t%|| st&|||| rq|j,pd%d)|d*|}|'t||||fid+|jd,|jd-|jd.|jd/|j(d0|j)d1|d2|j d3|jd4|j d5|j*d6|j+d7|j-d8|j.d9|d:|j/d;|jd||qWnt0y:}z td<|WYd}~nd}~wwt1||d=d>D] \}}|j2|_2qBW|rUt3|j4n |r_t3|j4wwtd?|j)|j5sp|| fSt6|||}t|t7r|j8rt9||| fS)@NrkzConnecting to vLLM server at z...rrrz-No models found on vLLM server via /v1/modelsridz Found model: zLoading tokenizer for rfzTokenizer loaded.rinfrtokenizer_pathprefillr memory_usagetoken_capacityiʚ;%effective_max_running_requests_per_dprdp_sizerzNeffective_max_running_requests_per_dp is not set, max_running_requests_per_dp=)rOrPrRrTzmax_running_requests_per_dp=zdp_size=z$skip_max_running_requests_threshold=zskip_token_capacity_threshold=z======== Warmup Begin ========zWarmup with batch_size=r;r=rC)r:r<r>r?r@rrBr9rVrrNrLrUr]rz======== Warmup End ======== ) r?r@rrBr9rVrrNrLrUr[r]rzbs-z-il-r?r@rrBr9rVrrNrLrUr[rGrHrIrJrKr]z8Error profiling, some profile traces may not be dumped: F)strictz Results are saved to ):rMrZnprDrr]r(rrrrrrrNr r r%dictrOrPrRrTrElistrr:r r?r@rArBrLrU itertoolsproductr<r>r'r%rr9rVr[rGrJrHrIrKr'ziprrpidrFr<rrXr)rr)rrrDr) model_listrrrr$r&r?rmax_running_requests_per_dprD gsp_kwargsbatch_size_uniquebsr(profile_resultsilolrJr.r profile_resr2r/r/r0run_benchmark_internals                   '    rV)rMNr\N)Frr}rIrrrrMr#rtypesrtypingrrrrnumpyrFrpydanticrr transformersr r sglang.bench_servingr r r sglang.profilerr"sglang.srt.entrypoints.http_serverrsglang.srt.server_argsrsglang.srt.utilsrrsglang.test.test_utilsrrrrntupler1r%r6 dataclassr7rrrrorrHrJrKrNrLr[rOrPrRrTrr r%r'r<rVr/r/r/r0s"      #  & !  N         H