o i-@sdZddlZddlZddlZddlZddlZddlZddlZddlZddl m Z ddl m Z ddl ZddlmZddlmZmZddlmZe dd Zd d Zd ejd eee fddfddZdejfddZd ejfddZdS)aBBenchmark the cold and warm startup time of vLLM models. This script measures total startup time (including model loading, compilation, and cache operations) for both cold and warm scenarios: - Cold startup: Fresh start with no caches (temporary cache directories) - Warm startup: Using cached compilation and model info N)contextmanager)Any)tqdm)#convert_to_pytorch_benchmark_format write_to_json) EngineArgsc csddlm}tjd}tjdd}z5|tjd<| dVWdn1s*wYWtj|dd|r@|tjd<dStj dddStj|dd|rX|tjd<wtj ddw) z Context manager to measure cold startup time: 1. Uses a temporary directory for vLLM cache to avoid any pollution between cold startup iterations. 2. Uses inductor's fresh_cache to clear torch.compile caches. r) fresh_cacheVLLM_CACHE_ROOTvllm_startup_bench_cold_)prefixNT) ignore_errors) torch._inductor.utilsrosenvirongettempfilemkdtempshutilrmtreepop)roriginal_cache_roottemp_cache_dirrM/home/ubuntu/vllm_env/lib/python3.10/site-packages/vllm/benchmarks/startup.py cold_startup s$     rc Csz?ddlm}t}|dit|}t|}d}t|jdr5|jj}t|dr5|j dur5|j j }| ||dWdSt y^}z| d| t |WYd}~dSd}~ww) z Run LLM startup in a subprocess and return timing metrics via a queue. This ensures complete isolation between iterations. r)LLMg vllm_configcompilation_configN)total_startup_timecompilation_timer)vllmrtime perf_counter dataclassesasdicthasattr llm_enginerrrput Exceptionstr) engine_args result_queuer start_timellmrrrerrrrun_startup_in_subprocess:s,      r/argsresultsreturncCstj|jd}t|d|dgi|d|ddd}|r&t|d|t|d|dgi|d |d d d}|rCt|d |t|d |d gi|d|ddd}|r`t|d|t|d|dgi|d|ddd}|rt|d|dSdS)Nravg_cold_startup_timecold_startup_timescold_startup_percentiles)r4r5)r0metrics extra_infoz.cold_startup.pytorch.jsonavg_cold_compilation_timecold_compilation_timescold_compilation_percentiles)r9r:z.cold_compilation.pytorch.jsonavg_warm_startup_timewarm_startup_timeswarm_startup_percentiles)r<r=z.warm_startup.pytorch.jsonavg_warm_compilation_timewarm_compilation_timeswarm_compilation_percentiles)r?r@z.warm_compilation.pytorch.json)rpathsplitext output_jsonrr)r0r1 base_namecold_startup_recordscold_compilation_recordswarm_startup_recordswarm_compilation_recordsrrr save_to_pytorch_benchmark_format`s\          rIparsercCsV|jdtddd|jdtddd|jdtdd d|jd tdd dt|}|S) Nz--num-iters-coldz"Number of cold startup iterations.)typedefaulthelpz--num-iters-warmupz>Number of warmup iterations before benchmarking warm startups.z--num-iters-warmz"Number of warm startup iterations.z --output-jsonz5Path to save the startup time results in JSON format.) add_argumentintr)r add_cli_args)rJrrrrRs4 rRcstjdddt|fdd}dtjd<tdtd g}g}tt|j d d D]&}t |}| |d | |d Wdn1sMwYq,tdtt|j dd D]}|q`tdg}g}tt|j dd D]}|}| |d | |d qwt|} t|} t|} t|} t| } t| }t| }t| }gd}t| |}t| |}t| |}t| |}tdtdtdtdtd| ddtd|ddtdt||D]\}}td|d|ddqtdt||D]\}}td|d|ddqtdtd|ddtd|ddtdt||D]\}}td|d|ddqEtdt||D]\}}td|d|ddq`td|jrt| t|||tt||tt||t|t|||tt||tt||d }t|jd!}tj||d"d#Wdn 1swYt||dSdS)$NspawnT)forcecstt}tjt|fd}|||s6|}|dur4|s0|}td|td|Std)z Create LLM instance in a subprocess and measure startup time. Returns timing metrics, using subprocess for complete isolation. )targetr0NzSubprocess failed: z$Subprocess failed with unknown errorz"Subprocess did not return a result) multiprocessingQueueProcessr/startjoinemptyr RuntimeError)r+processresult error_msgr*rrcreate_llm_and_measure_startups$z,main..create_llm_and_measure_startup0VLLM_ENABLE_V1_MULTIPROCESSINGzESetting VLLM_ENABLE_V1_MULTIPROCESSING=0 to collect startup metrics. zMeasuring cold startup time... zCold startup iterations)descrrz, Warming up for warm startup measurement... zWarmup iterationsz Measuring warm startup time... zWarm startup iterations) 2KZcz= ============================================================zSTARTUP TIME BENCHMARK RESULTSz<============================================================z COLD STARTUP:zAvg total startup time: z.2fz secondszAvg compilation time: zStartup time percentiles:z z%: zCompilation time percentiles:z WARM STARTUP:) r3r8r4r9r5r:r;r>r<r?r=r@w)indent)rVset_start_methodr from_cli_argsrrprintrrangenum_iters_coldrappendnum_iters_warmupnum_iters_warmnparraymean percentileziprCfloatdicttolistopenjsondumprI)r0rar4r9ir6_r<r?cold_startup_arraycold_compilation_arraywarm_startup_arraywarm_compilation_arrayavg_cold_startupavg_cold_compilationavg_warm_startupavg_warm_compilation percentagesr5r:r=r@ percentageryr1frr`rmains                   r)__doc__argparser#rrVrrrr! contextlibrtypingrnumpyrvrvllm.benchmarks.lib.utilsrrvllm.engine.arg_utilsrrr/ Namespacer|r)rIArgumentParserrRrrrrrs6     &  >