o ۾i4@sddlZddlZddlZddlmZmZddlmZddlmZddl m Z m Z m Z ddl mZddlmZmZdd lmZmZmZdd lmZdd lmZmZdd lmZzddlZWn eyied ZYnwzddl m!Z!Wneyed"d"dZ!YnwdededefddZ#dedede$de%dBfddZ&dede%dBfdd Z'dede$fd!d"Z(ded#ed$ede$def d%d&Z)d'edBd(e*e$dededed)e%d*e+fd+d,Z,e d-Z-Gd.d/d/e.e%e/fZ0ded0e*e.e$e1ffd1d2Z2dd3d4d'edBd(e*e$dedededed)e%d*e+de-d5e%d6e%fd7d8Z3d'edBd(e*e$dededede-ded)e%d*e+fd9d:Z4d;e*e$d(e*e$ded?ed@ede-ded)e%d*e+fdAdBZ5eGdCdDdDeZ6dEe6fdFdGZ7dEej8fdHdIZ9e:dJkrej;e6je=e9e=?dSdS)LN)asdict dataclass)datetime)Path)ClassVarLiteralget_args)PlaceholderModule)ParameterSweepParameterSweepItem)SweepServeArgs run_benchmark run_server) ServerProcess)SLASweep SLASweepItem)sanitize_filenamepandas)PchipInterpolatorscipy interpolater output_dir serve_comb bench_combcCsPtt}|r|d|jddf|r|d|jddf|td|S)NzSERVE--sepzBENCH-)liststrextendas_textrjoin)rrrpartsr$S/home/ubuntu/.local/lib/python3.10/site-packages/vllm/benchmarks/sweep/serve_sla.py_get_sla_base_path"s r& base_pathsla_comb sla_variable sla_valuecCs6|dur|jdd}|d|dS||d|S)NrrzSLA--.json=)r!)r'r(r)r*prefixr$r$r%_get_sla_iter_path0s r. iter_path run_numbercCs |dur|dS|d|dS)N summary.jsonzrun=r+r$)r/r0r$r$r%_get_sla_run_path=sr2ccsL||dD]}t|j|d}|d}|r#||fVq dS)Nz=*r,r1)globintname removeprefixexists)r'r)r/r* summary_pathr$r$r%_iter_sla_val_pathsDs r9 bench_combs sla_combscCs@|D]}t|||}|D]}t|||ddsdSq qdS)Nr*TF)r&r.r7)rr:r;r)rrr'r(r$r$r%_sla_needs_serverLs  r=server bench_cmdnum_runsdry_runc Csttttf}t|D]}t|||||t|||d} | dur&|| q |r+dSt|ddd} t j || ddWd|S1sHwY|S)N)serve_overridesbench_overridesr0 output_pathrA)r0windent) rdictrobjectrangerr2appendopenjsondump) r>r?rrr/r@rA iter_datar0run_datafr$r$r%run_slaas,   rS) request_ratemax_concurrencycsdeZdZdededdffdd ZdeeeeeffddZdefd d Z defd d Z Z S) SLAHistory min_value max_valuereturnNcst||_||_dSN)super__init__rWrX)selfrWrX __class__r$r%r\s  zSLAHistory.__init__cCsFtt}tt}t|D]\}}||||q||fSrZ)rr4floatsorteditemsrL)r]xsysxyr$r$r%get_xys    zSLAHistory.get_xycCtdd|D|jdS)Ncss |] \}}|dkr|VqdSrNr$.0valmarginr$r$r% z-SLAHistory.get_max_passing..default)maxrbrWr]r$r$r%get_max_passingzSLAHistory.get_max_passingcCrh)Ncss |] \}}|dkr|VqdSrir$rjr$r$r%rnroz-SLAHistory.get_min_failing..rp)minrbrXrsr$r$r%get_min_failingruzSLAHistory.get_min_failing) __name__ __module__ __qualname__r4r\tuplerr`rgrtrw __classcell__r$r$r^r%rVs  rVrPcs<sJdfdd|Dfdd|D}t|S)NzSummary should not be emptycs,i|]tfddDtqS)c3s|] }t|VqdSrZ)r`)rkrQkr$r%rnsz-_compute_margin...)sumlen)rk)rPr}r% sz#_compute_margin..csg|] \}}||qSr$)print_and_compute_margin)rkr~ criterion)iter_data_meanr$r% s z#_compute_margin..)rbrr)r(rP sla_marginsr$)rPrr%_compute_margins   ri ) sla_min_value sla_max_valuerrc  Csttttf} t| | d} t||D]%\} }|d }t|}Wdn1s-wYt ||| | <q| d| krt | | d| krO| }n9t | | d| krZ| }n.t| ddi}|}t|dkrzt| | d}nt|d}|| vr|d7}t | t || }td |d |d t||||||iBt||||||d }|durdSt ||}|dkrtd |ddn td|dd| ||| |<| d| ksD| | fS)N)rWrXrbr rp extrapolateFrzTesting z: z req/s)rrr/r@rAzSLA criteria are met. (margin=z.2f)z"SLA criteria are not met. (margin=)rrIrrJrVr9rMrNloadrrtrwrrrvrrgsolverr4printrSr.r )r>r?rrr(r'r@rAr)rrsla_datahistorypast_sla_valuepathrRpast_iter_datarlspl spl_rootsrPrmr$r$r% solve_slasP          +rc Cstdtd|t|||||||||d } | dur,|s"JtdtddS| \} } | } td|d| dt|||dd d } tj| | d d Wdn1s]wYtd| S) Nz [SLA START]zSLA criteria: )rrr(r'r@rAr)zOmitting SLA search.z [SLA END]zMaximum z for SLA: z req/s.r<rErFrG)rr!rrtr.rMrNrO)r>r?rrr(r)r'r@rAresultr sla_historyr*rRr$r$r% search_slasD  r serve_cmdafter_bench_cmd show_stdout serve_params bench_params sla_paramscstfdd|Drtddttttf} |D]M} t| |||r1t|||| | dnt .} |D]#}|D]}t || |}t | || |||| | d }|durZ| |q.zYou should not override `zU` in `bench_params` in SLA mode, since it is supposed to be determined automatically.)rrBrA)rrr(r)r'r@rAz summary.csv)any ValueErrorrrIrrJr=r contextlib nullcontextr&rr pd DataFrame from_recordsto_csv)rr?rrrrrr)rr@rAall_datarr>rr(r' comb_data combined_dfr$rr%run_slas/sb    $ rcsteZdZUeed<eed<dZeeed<dZ eeed<e de j fdd Z e d e jd e jffd d ZZS)SweepServeSLAArgsrr) serve_sla parser_namez5Tune a variable to meet SLAs under multiple settings. parser_helpargscCsFt|}|jrt|j}ntg}|dit|||jdS)N)rr)r$)r from_cli_argsrr read_jsonrrr))clsr base_argsrr$r$r%rzs   zSweepServeSLAArgs.from_cli_argsparserrYcsDt|}|d}|jdtddd|jdtttddd |S) Nz sla optionsz --sla-paramsTaPath to JSON file containing a list of SLA constraints to satisfy. Each constraint is expressed in `{"": ""}` format, e.g.: `{"p99_e2el_ms": "<=500"}` means that the E2E latency should be less than 500ms 99%% of the time. Setting this option runs this script in SLA mode, which searches for the maximum `sla_variable` that satisfies the constraints for each combination of `serve_params`, `bench_params`, and `sla_params`.)typerequiredhelpz--sla-variablerTzSWhether to tune request rate or maximum concurrency to satisfy the SLA constraints.)rchoicesrqr)r[ add_cli_argsadd_argument_group add_argumentrr SLAVariable)rr sla_groupr^r$r%rs    zSweepServeSLAArgs.add_cli_args)rxryrzr__annotations__rrrrr classmethodargparse NamespacerArgumentParserrr|r$r$r^r%rrs $rrc Cs|jp td}|j|}|jr|std|dzt|j|j |j |j |j |j |j|j||j|jd WStyM}z td|d|d}~ww)Nz %Y%m%d_%H%M%Sz+Cannot resume from non-existent directory (r) rr?rrrrrr)rr@rAz/The script was terminated early. Use `--resume z2` to continue the script from its last checkpoint.)resumernowstrftimerr7rrrr?rrrrrr)r@rA BaseException RuntimeError)r timestamprexcr$r$r%run_mains4   rcCstt|dSrZ)rrr)rr$r$r%mainsr__main__) description)@rrrN dataclassesrrrpathlibrtypingrrrvllm.utils.import_utilsr param_sweepr r server rrr>r sla_sweeprrutilsrrr ImportErrorscipy.interpolaterplaceholder_attrr&rr4r.r2r9r=rboolrSrrIr`rVrJrrrrrrrrrxrrrr parse_argsr$r$r$r%s2              #     G  1    C4