o ۾i* @sddlZddlZddlmZddlmZddlmZddlm Z ddl m Z ddl m Z ddlmZd d lmZmZd d lmZzddlmZddlZddlZWneyhed d ZedZedZYnwdeee fde!efddZ"dddeee fde!ede#de$dBfddZ%deee fdedBde$dBfddZ&deee fdedBde$fddZ'deee fd ede$fd!d"Z(d#e!eee fdedBdedBde)e!eee fe*ffd$d%Z+d&d'd(d)d*ed+ed,e$dd)f d-d.Z,d/e d0e)e)eefd1fde fd2d3Z-d/e d4e)e)e)eefd1fe!eee ffd5e!ed6e#fd7d8Z.d9e dedBdedBd5e!ed6e#f d:d;Z/eGde0fd?d@Z1d>ej2fdAdBZ3e4dCkrpej5e0j6dDZ7e08e7e3e79dSdS)EN)ProcessPoolExecutor) dataclass)partial)Path)ClassVar) full_groupby)PlaceholderModule) DummyExecutor_json_load_bytes)sanitize_filename matplotlibpyplotpandasseabornrun_datakeyscCsF|D]}||dd|ddhD]}||vr||SqqdS)N_-)replace)rrkey candidaterU/home/ubuntu/.local/lib/python3.10/site-packages/vllm/benchmarks/sweep/plot_pareto.py_first_presentsrT allow_zerorreturnc Cspt||}|dur dSzt|}Wnttfy-}ztd|d|d||d}~ww|s6|dkr6dS|S)Nz"Expected numeric value for one of z , but found z in run_data=r)rfloat TypeError ValueError)rrrvaluenumericexcrrr _get_numeric#s(   r$user_count_varcCsD|r|gng}|dgt||dd}|dur|St|dgddS)N request_rateFrmax_concurrent_requests)extendr$)rr% candidates user_countrrr_infer_user_count;s  r+ gpu_count_varcCsz|r|gng}t||dd}|r|St|ddg}t|ddg}t|ddg}d }|r/||9}|r5||9}|r;||9}|S) NFrtensor_parallel_sizetppipeline_parallel_sizeppdata_parallel_sizedpg?)r$)rr,direct_candidatesdirect_gpu_counttp_sizepp_sizedp_size world_sizerrr_infer_gpu_countIsr9throughput_varcCs0t||g}|durtd|dt||S)NzCannot find throughput metric z in run data. Available keys: )r$r sorted)rr: throughputrrr_get_throughput`s r=all_datac Csvg}d}|D]0}t|d}t||}|dur|d7}qt||}||} ||} |i|| | ||dq||fS)Nroutput_throughputr )tokens_per_usertokens_per_gpuuser_count_estimate gpu_count)r=r+r9append) r>r%r,preparedskipped_missing_usersrecordr<r*rCr@rArrr_prepare_recordsns,    rHg& .>)epsilondfz pd.DataFramex_coly_colrIc Csf|j||gddgd}g}tj }|D]\}}||} | ||kr-||t|| }q|j|S)NF) ascending) sort_valuesmathinfiterrowsrDmaxloc) rJrKrLrI sorted_dffrontier_indicesbest_yidxrowy_valrrr_pareto_frontiers    rZfig_dir fig_group.cCs8dg}|r|dd|Dtd|d}||S)NPARETOcss"|] \}}|d|VqdS)=Nr).0kvrrr s z _get_fig_path..rz.png)r(r join)r[r\partsfilenamerrr _get_fig_paths rffig_group_datalabel_bydry_runc Cs|\}}t||}tdtdt|td||r%tddStj|}|jddgd}|jr@tdtddSt|dd}| d}t \} } t j |ddd d | d d t j|ddd | dd|r|D]-\} } g} |D]}|| vr| |d| |qv| r| j| d| dd| ddqn| d| d| jdddd d| | |t | tdt|dt|dtddS)Nz[BEGIN FIGURE]zGroup: z Output file: z [END FIGURE]r@rA)subsetz3No data points available after filtering; skipping.z0.5g333333?zAll runs)dataxycoloralphaaxlabelozPareto frontier)rkrlrmmarkerrprqr^ )fontsizez Tokens/s/userz Tokens/s/GPUTz--g?) linestyle linewidthrozPlotted z points; Pareto frontier size: .)rfprintdictpd DataFrame from_recordsdropnaemptyrZrNpltsubplotssns scatterplotlineplotrQrDtextrc set_xlabel set_ylabelgrid tight_layoutsavefigcloselen)r[rgrhrir\fig_datafig_pathrJfrontierfigrprrX label_partsrrrr _plot_figsv            r output_dirc Cs|d}dd|dD}|std||jdddt|||d\}}|r1td |d |s7td t|d d d} t| dkrHtnt} t | t t |||d| WddS1sewYdS)NparetocSsg|] }t|D]}|qqSr)r )r_pathrrrr szplot_pareto..z**/summary.jsonz/Did not find any parameter sweep results under T)parentsexist_ok)r%r,zSkipped zL runs without a user count (`max_concurrency` or `max_concurrent_requests`).zUNo data points with both throughput and user count available to plot Pareto frontier.cSstSN)tuple)itemrrr!szplot_pareto..)rr )rhri) rglobr mkdirrHrzrrr rallmaprr) rr%r,rhrir[raw_data prepared_datarF fig_groupsexecutorrrr plot_paretosJ  "rc@seZdZUeed<edBed<edBed<eeed<eed<dZe eed<d Z e eed <e d e j fd d Ze de jfddZdS)SweepPlotParetoArgsrNr%r,rhrir parser_namezYPlot Pareto frontier between tokens/s/user and tokens/s/GPU from parameter sweep results. parser_helpargscCsNt|j}|std||jsgn|jd}|||j|j||jdS)Nz!No parameter sweep results under ,rr%r,rhri) r OUTPUT_DIRexistsr rhsplitr%r,ri)clsrrrhrrr from_cli_args@s z!SweepPlotParetoArgs.from_cli_argsparsercCs\|jdtddd|jdtddd|jdtdd d|jd td d d|jd ddd|S)Nrresultsz3The directory containing the sweep results to plot.)typedefaulthelpz--user-count-varmax_concurrencyz_Result key that stores concurrent user count. Falls back to max_concurrent_requests if missing.z--gpu-count-varzResult key that stores GPU count. If not provided, falls back to num_gpus/gpu_count or tensor_parallel_size * pipeline_parallel_size.z --label-byzmax_concurrency,gpu_countzEComma-separated list of fields to annotate on Pareto frontier points.z --dry-run store_truez8If set, prints the figures to plot without drawing them.)actionr) add_argumentstr)rrrrr add_cli_argsPs<z SweepPlotParetoArgs.add_cli_args)__name__ __module__ __qualname__r__annotations__rlistboolrrr classmethodargparse NamespacerArgumentParserrrrrrr2s    rrcCst|j|j|j|j|jdS)Nr)rrr%r,rhrirrrrrun_mainwsrcCstt|dSr)rrrrrrrmainsr__main__) description):rrOconcurrent.futuresr dataclassesr functoolsrpathlibrtypingrvllm.utils.collection_utilsrvllm.utils.import_utilsrplotr r utilsr matplotlib.pyplotrrrr|rr ImportErrorplaceholder_attrr{robjectrrrrr$r+r9r=rintrHrZrfrrrrrrrrrrr parse_argsrrrrs                      &   & K 9D