o iT @sUddlZddlZddlmZmZddlmZddlmZddl m Z m Z m Z ddl mZddlmZddlmZmZddlmZmZdd lmZdd lmZmZdd lmZdd lmZdd l m!Z!ddl"m#Z#m$Z$m%Z%m&Z&m'Z'ddl(m)Z)m*Z*ee+Z,eee-gdfZ.e/dZ0e0e.BZ1GdddeZ2de3e1fddZ4Gddde2Z5Gddde2Z6Gddde6e5Z7Gddde5Z8Gddde5Z9e e Be BZ:ee;d<d e:d!e3e-d"ed%e3e-d&e-de3e-fd'd(Z?d&e-de3e-fd)d*Z@Gd+d,d,ZAdS)-N)ABCabstractmethod)Callable) TypeAlias)CounterGauge Histogram)CUDAGraphLogging)SupportsMetricsInfo VllmConfig)KVConnectorLoggingKVConnectorPrometheus) init_logger)STAT_LOGGER_PLUGINS_GROUPload_plugins_by_group) FinishReason)PerfMetricsLogging)unregister_vllm_metrics)CachingMetricsIterationStatsMultiModalCacheStatsPromptTokenStatsSchedulerStats)SpecDecodingLoggingSpecDecodingPromStatLoggerBaseAggregateStatLoggerBasec @s~eZdZdZeddedefddZe  ddedBd e dBd e dBd efd d Z eddZ ddZ dedefddZdS)raInterface for logging metrics. API users may define custom loggers that implement this interface. However, note that the `SchedulerStats` and `IterationStats` classes are not considered stable interfaces and may change in future versions. r vllm_config engine_indexcCdSN)selfrrr!r!M/home/ubuntu/vllm_env/lib/python3.10/site-packages/vllm/v1/metrics/loggers.py__init__0zStatLoggerBase.__init__Nscheduler_statsiteration_statsmm_cache_stats engine_idxcCrr r!r"r&r'r(r)r!r!r#record3szStatLoggerBase.recordcCrr r!r"r!r!r#log_engine_initialized<r%z%StatLoggerBase.log_engine_initializedcCrr r!r,r!r!r#log?r%zStatLoggerBase.logis_awakelevelcCrr r!)r"r/r0r!r!r#record_sleep_stateBr%z!StatLoggerBase.record_sleep_staterNr)__name__ __module__ __qualname____doc__rr intr$rrrr+r-r.r1r!r!r!r#r(s(  returncCsRg}ttD]\}}t|trt|ts!td|d|d||q|S)NzStat logger plugin z+ must be a subclass of StatLoggerBase (got z).) rritems isinstancetype issubclassr TypeErrorappend) factoriesname plugin_classr!r!r#!load_stat_logger_plugin_factoriesFs  rCc@s*eZdZdZededeefddZdS)rzNAbstract base class for loggers that aggregate across multiple DP engines.rengine_indexescCrr r!r"rrDr!r!r#r$[r%z AggregateStatLoggerBase.__init__N) r4r5r6r7rr listr8r$r!r!r!r#rWsc @seZdZd"dedefddZddZdefd d Zd e fd d Z dede de fddZ e ddZ  d#dedBd e dBdedBdefddZddZddZddZd d!ZdS)$LoggingStatLoggerrrrcCs||_||_|tt|_t|_t|_ t|_ t |_ |jj }t||_d|_|jjjr>t|jjj|jjj|_d|_d|_d|_d|_|rUt||_dSdS)NF)rr_resettime monotonicrlast_scheduler_statsrprefix_caching_metrics connector_prefix_caching_metricsmm_caching_metricsrspec_decoding_loggingkv_transfer_configr kv_connector_loggingcudagraph_loggingobservability_configcudagraph_metricsr compilation_configcudagraph_modecudagraph_capture_sizeslast_prompt_throughputlast_generation_throughputengine_is_idle aggregated_enable_perf_statsrperf_metrics_logging)r"rrrQr!r!r#r$`s.  zLoggingStatLogger.__init__cCs"||_d|_d|_d|_d|_dSr3) last_log_timenum_prompt_tokensnum_generation_tokensnum_corrupted_reqsnum_preemptions)r"nowr!r!r#rI~s  zLoggingStatLogger._resetr9cCs |jjjSr )rrTenable_mfu_metricsr,r!r!r#r]s z$LoggingStatLogger._enable_perf_statsr'cCsF|j|jj7_|j|j7_|j|j7_|j|j7_dSr )r`prompt_token_statscomputedrarbrcnum_preempted_reqs)r"r'r!r!r#_track_iteration_statssz(LoggingStatLogger._track_iteration_stats tracked_statsrdcCs"||j}|dkr dSt||S)NrH)r_float)r"rjrd delta_timer!r!r#_get_throughputs  z!LoggingStatLogger._get_throughputcCs d|jS)NzEngine {:03d}: )formatrr,r!r!r# log_prefixs zLoggingStatLogger.log_prefixNr&r(r)cCs|r|||dur[|j|j|jdur|j|j|jdur*|j|j|j}r5|j ||j durF|j durF|j |j |j sL||_ |j}r[|r[|j||re|j|dSdS)zLog Stats to standard output.N)rirMobserveprefix_cache_statsconnector_prefix_cache_statsrNspec_decoding_statsrPkv_connector_statsrRrScudagraph_statsr\rL perf_statsr]r^rO)r"r&r'r(r)rtrvr!r!r#r+s,        zLoggingStatLogger.recordcCsVt}||j|}||j|}||t|||j|jf |_ ||_||_dSr ) rJrKrmr`rarIanyrYrZr[)r"rdprompt_throughputgeneration_throughputr!r!r# _update_statss  zLoggingStatLogger._update_statscCrr r!r,r!r!r#aggregate_scheduler_statsz+LoggingStatLogger.aggregate_scheduler_statscCsd|||jrtjntj}gd}|j|j|jj |jj g}|j dkr1| d| |j | ddg| |jjd|jjdgtjrU| d| |j|jjsg| d| |jjd|jjsy| d | |jjd||jd |g|R|jj|d |jj|d |jdur|jj|d |r|jj||jd dSdS) N)z$Avg prompt throughput: %.1f tokens/sz(Avg generation throughput: %.1f tokens/szRunning: %d reqszWaiting: %d reqsrzPreemptions: %dzGPU KV cache usage: %.1f%%zPrefix cache hit rate: %.1f%%dzCorrupted: %d reqsz&External prefix cache hit rate: %.1f%%zMM cache hit rate: %.1f%%z, )log_fn)r~ro)rzr{r[loggerdebuginforYrZrLnum_running_reqsnum_waiting_reqsrcr?extendkv_cache_usagerMhit_rateenvsVLLM_COMPUTE_NANS_IN_LOGITSrbrNemptyrOrojoinrPr.rRrSr]r^)r"r~ log_partslog_argsr!r!r#r.sV          zLoggingStatLogger.logcCs(|jjjrtd|j|jjjdSdS)NzSEngine %03d: vllm cache_config_info with initialization after num_gpu_blocks is: %d)r cache_confignum_gpu_blocksrrrr,r!r!r#r-s z(LoggingStatLogger.log_engine_initializedr2r3)r4r5r6r r8r$rIboolr]rrirkrmpropertyrorrr+rzr{r.r-r!r!r!r#rG_s.   # :rGc @seZdZdedeefddZeddZde fdd Z dd e d Bd e d Bde d BdefddZddZddZddZd S)AggregatedLoggingStatLoggerrrDcCs2||_dd|jD|_tj||ddd|_dS)NcSsi|]}|tqSr!)r.0idxr!r!r# sz8AggregatedLoggingStatLogger.__init__..)rT)rDlast_scheduler_stats_dictrGr$r\rEr!r!r#r$s  z$AggregatedLoggingStatLogger.__init__cCsdt|jS)Nz{} Engines Aggregated: )rnlenrDr,r!r!r#ro&sz&AggregatedLoggingStatLogger.log_prefixr9cCsdS)NFr!r,r!r!r#r]*r|z.AggregatedLoggingStatLogger._enable_perf_statsNrr&r'r(r)cCsH||jvr td|dStj|||||d|dur"||j|<dSdSNzUnexpected engine_idx: %dr(r))rDrwarningrGr+rr*r!r!r#r+.s  z"AggregatedLoggingStatLogger.recordcCsht|_|jD]}|jj|j7_|jj|j7_|jj|j7_q |jjt|j_dSr )rrLrvaluesrrrr)r"rLr!r!r#r{Bsz5AggregatedLoggingStatLogger.aggregate_scheduler_statscCst|dSr )rGr.r,r!r!r#r.PszAggregatedLoggingStatLogger.logcCs,|jjjrtdt|j|jjjdSdS)NzR%d Engines: vllm cache_config_info with initialization after num_gpu_blocks is: %d)rrrrrrrDr,r!r!r#r-Ss z2AggregatedLoggingStatLogger.log_engine_initializedr3)r4r5r6r rFr8r$rrorr]rrrr+r{r.r-r!r!r!r#rs.    rc @sfeZdZdedeededdfddZ  dd edBd e dBd e dBd efd dZ ddZ ddZ dS)PerEngineStatLoggerAdapterrrDper_engine_stat_logger_factoryr9NcCs*i|_||_|D] }||||j|<qdSr )per_engine_stat_loggersrD)r"rrDrrr!r!r#r$^s z#PerEngineStatLoggerAdapter.__init__rr&r'r(r)cCs6||jvr td|dS|j|j||||ddSr)rrrr+r*r!r!r#r+ks    z!PerEngineStatLoggerAdapter.recordcC|jD]}|qdSr )rrr.r"per_engine_stat_loggerr!r!r#r.| zPerEngineStatLoggerAdapter.logcCrr )rrr-rr!r!r#r-rz1PerEngineStatLoggerAdapter.log_engine_initializedr3)r4r5r6r rFr8PerEngineStatLoggerFactoryr$rrrr+r.r-r!r!r!r#r]s.   rc @seZdZeZeZeZe Z e Z dde deedBfddZdedefdd Z  dd edBd edBd edBdefddZddedefddZddZdS)PrometheusStatLoggerNrrDc*s0|durdg}||_t||_|jj|_|jj|_ddg}|jj|jj }fdd|D}| |j |||_ | ||||_|jddd|d }t|||_|jd d d|d }t|||_|jd d |dgddi|_gd}|D]fdd|D|j<qu||jddd|d } t| ||_tjr|jdd|d} t| ||_|jdd|d} t| ||_|jdd|d} t| ||_|jdd|d} t| ||_|jdd|d}t|||_|jdd |d}t|||_|jd!d"|d}t|||_|jd#d$|d}t|||_ |jd%d&|d}t|||_!|jd'd(|d)gdi|_"t#j$D]fd*d|D|j"<q5|jd+d,|d}t|||_%|jd-d.|d}t|||_&|jd/d0|d}t|||_'i|_(|jd1d2|d3gdt)D]fd4d|D|j(<q|j*d5d&t+||d6}t|||_,|j*d7d0t+||d6}t|||_-|j*d8d9gd:|d6}t|||_.|j*d;dgd?|d6}t|||_0|j*d@dAt+||d6}t|||_1|j*dBdCgdD|d6}t|||_2|j*dEdFgdG|d6}t|||_3|j*dHdIgdG|d6}t|||_4gdJ}|j*dKdL||d6} t| ||_5|j*dMdN||d6}!t|!||_6|j*dOdP||d6}"t|"||_7|j*dQdR||d6}#t|#||_8|j*dSdT||d6}$t|$||_9|j*dUdVt+||d6}%t|%||_:|jrgdW}&|j*dXdY|&|d6}'t|'||_;|j*dZd[|&|d6}(t|(||_<|j*d\d]|&|d6})t|)||_=n i|_;i|_|j?durt@|jd^krtABd_d`|_Cda|_Ddb|_E|j?jF|_G|jdcddde|jC|jD|jEgd |_>dSdS)fNr model_nameenginecsi|] }|t|gqSr!)strr)rr!r#rsz1PrometheusStatLogger.__init__..zvllm:num_requests_runningz.Number of requests in model execution batches. mostrecentrA documentationmultiprocess_mode labelnameszvllm:num_requests_waitingz+Number of requests waiting to be processed.zvllm:engine_sleep_statezEngine sleep state; awake = 0 means engine is sleeping; awake = 1 means engine is awake; weights_offloaded = 1 means sleep level 1; discard_all = 1 means sleep level 2. sleep_state)rArrr)awakeweights_offloaded discard_allcsi|] }|j|dqS))rrr)labelsr)gauge_engine_sleep_statersr!r#rs zvllm:kv_cache_usage_percz*KV-cache usage. 1 means 100 percent usage.zvllm:corrupted_requestszMCorrupted requests, in terms of total number of requests with NaNs in logits.)rArrzvllm:prefix_cache_queriesz;Prefix cache queries, in terms of number of queried tokens.zvllm:prefix_cache_hitsz7Prefix cache hits, in terms of number of cached tokens.z"vllm:external_prefix_cache_querieszsExternal prefix cache queries from KV connector cross-instance cache sharing, in terms of number of queried tokens.zvllm:external_prefix_cache_hitszoExternal prefix cache hits from KV connector cross-instance cache sharing, in terms of number of cached tokens.zvllm:mm_cache_queriesz?Multi-modal cache queries, in terms of number of queried items.zvllm:mm_cache_hitsz;Multi-modal cache hits, in terms of number of cached items.zvllm:num_preemptionsz0Cumulative number of preemption from the engine.zvllm:prompt_tokensz#Number of prefill tokens processed.zvllm:prompt_tokens_by_sourcez"Number of prompt tokens by source.sourcecs i|] }|t|qSr!rrr)counter_prompt_tokens_by_sourcerrr!r#r[s  zvllm:prompt_tokens_cachedz2Number of cached prompt tokens (local + external).zvllm:prompt_tokens_recomputedz4Number of cached tokens recomputed for forward pass.zvllm:generation_tokensz&Number of generation tokens processed.zvllm:request_successz)Count of successfully processed requests.finished_reasonc s$i|]}|t|tqSr!rr)counter_request_success_baserreasonr!r#rs zvllm:request_prompt_tokens)rArbucketsrzvllm:request_generation_tokenszvllm:iteration_tokens_totalz.Histogram of number of tokens per engine_step.)  @iiiii i@z&vllm:request_max_num_generation_tokensz;Histogram of maximum number of requested generation tokens.zvllm:request_params_nz%Histogram of the n request parameter.)r zvllm:request_params_max_tokensz.Histogram of the max_tokens request parameter.z vllm:time_to_first_token_secondsz,Histogram of time to first token in seconds.)MbP?{Gzt?{Gz?{Gz?g{Gz?gQ?g{Gz?皙?g????@@@$@4@D@T@gd@g@g@z vllm:inter_token_latency_secondsz,Histogram of inter-token latency in seconds.)rg?皙?g333333?rg333333?皙?333333?g?rrrrrrrrrrz*vllm:request_time_per_output_token_secondsz7Histogram of time_per_output_token_seconds per request.)rrg?rg?g@rrrg.@rg>@rgI@gN@g^@gn@g~@g@g@g@z vllm:e2e_request_latency_secondsz,Histogram of e2e request latency in seconds.zvllm:request_queue_time_secondsz5Histogram of time spent in WAITING phase for request.z#vllm:request_inference_time_secondsz5Histogram of time spent in RUNNING phase for request.z!vllm:request_prefill_time_secondsz5Histogram of time spent in PREFILL phase for request.z vllm:request_decode_time_secondsz4Histogram of time spent in DECODE phase for request.z'vllm:request_prefill_kv_computed_tokenszMHistogram of new KV tokens computed during prefill (excluding cached tokens).)rgMb`?rrrrrrrrrrrr<xi,iXiizvllm:kv_block_lifetime_secondsz|Histogram of KV cache block lifetime from allocation to eviction. Sampled metrics (controlled by --kv-cache-metrics-sample).z'vllm:kv_block_idle_before_evict_secondszqHistogram of idle time before KV cache block eviction. Sampled metrics (controlled by --kv-cache-metrics-sample).zvllm:kv_block_reuse_gap_secondszHistogram of time gaps between consecutive KV cache block accesses. Only the most recent accesses are recorded (ring buffer). Sampled metrics (controlled by --kv-cache-metrics-sample).rzfvllm:lora_requests_info prometheus metrics may be incorrect/misleading with data parallel deployments.max_lorawaiting_lora_adaptersrunning_lora_adapterszvllm:lora_requests_infozRunning stats on lora requests.sum)HrDrrrTshow_hidden_metricskv_cache_metricskv_cache_metrics_enabled model_configserved_model_name max_model_len_spec_decoding_clsspeculative_configspec_decoding_prom_kv_connector_clskv_connector_prom _gauge_clsmake_per_enginegauge_scheduler_runninggauge_scheduler_waitingrr1gauge_kv_cache_usagerr _counter_clscounter_corrupted_requestscounter_prefix_cache_queriescounter_prefix_cache_hits&counter_connector_prefix_cache_queries#counter_connector_prefix_cache_hitscounter_mm_cache_queriescounter_mm_cache_hitscounter_num_preempted_reqscounter_prompt_tokensrr ALL_SOURCEScounter_prompt_tokens_cached counter_prompt_tokens_recomputedcounter_generation_tokenscounter_request_successr_histogram_clsbuild_1_2_5_buckets#histogram_num_prompt_tokens_request'histogram_num_generation_tokens_requesthistogram_iteration_tokens+histogram_max_num_generation_tokens_requesthistogram_n_requesthistogram_max_tokens_requesthistogram_time_to_first_tokenhistogram_inter_token_latency'histogram_request_time_per_output_tokenhistogram_e2e_time_requesthistogram_queue_time_request histogram_inference_time_requesthistogram_prefill_time_requesthistogram_decode_time_request%histogram_prefill_kv_computed_requesthistogram_kv_block_lifetime$histogram_kv_block_idle_before_evicthistogram_kv_block_reuse_gapgauge_lora_info lora_configrrrlabelname_max_loralabelname_waiting_lora_adapterslabelname_running_lora_adapters max_lorasr)*r"rrDrrper_engine_labelvaluesrrrrrrrrrrrrrrrrrrrrrrrrr request_latency_bucketsr r r r rrkv_cache_residency_bucketsrrrr!)rrrrrrrr#r$s            zPrometheusStatLogger.__init__r< config_objcCs|}d|d<d\}}|dkrd}d}|dusJd||j||d|d }|jD]}|}t||d<|jd i|d q-dS) NrNNrzvllm:cache_config_infoz(Information of the LLMEngine CacheConfigzUnknown metrics info type rrrr!) metrics_inforkeysrDrrset)r"r<rrrAr info_gaugerr!r!r#log_metrics_infos$  z%PrometheusStatLogger.log_metrics_inforr&r'r(r)cCs|dur|j||j|j||j|j||j|j||j j |j ||j j |j durL|j||j j |j||j j |jdurY|j|j||jdurf|j|j||jr|jr|j|}|j|}|j|}|jD]}||j||j|jD]} || qq~|jdurd|j } d|j! } |j"| |j#| |j$|j%i} |jj&di| '|dur|j(||j |j)||j |durdSt*j+r|j,||j-|j.||j/|j0||j1|j2} t3j4D]}|j5||| 6|q|j7|| j8|j9|| j:|j;||j<|j=||j1|j<|j>D] }|j?||q?|j@D] }|jA||qN|jBD] }|jC||q]|jDD] }|jE||ql|jFD]s}|jG|jH||jI||jJ|jK||jL|jM||jN|jO||jP|jQ||jR|j1tS|jTd}|jU|||jV||j1|jW||j<|jX||jY|jZr|j[||jZq{dS)zLog to prometheus.N,rr!)\rr!rrrrrrincrqqueriesrhitsrrrrrsrrprtrrkv_cache_eviction_eventsrrrlifetime_seconds idle_secondsreuse_gaps_secondsrrrr rrrrrrset_to_current_timerrrrrrbrrhrr`rfrrr get_by_sourcer cached_tokensrrecomputed_tokensrrarmax_num_generation_tokens_iterr n_params_iterrtime_to_first_tokens_iterrinter_token_latencies_iterrfinished_requestsr finish_reasonr  e2e_latencyr  queued_timer  prefill_timer inference_timer decode_timemaxnum_cached_tokensrrrr mean_time_per_output_tokenmax_tokens_paramr)r"r&r'r(r) lifetime_hist idle_hist reuse_histeventgaprrlora_info_labelsptsrmax_gen_tokensn_paramttftitlfinished_requestprefill_kv_computedr!r!r#r+s                                         zPrometheusStatLogger.recordsleepr0cCs~d}d}d}|dkrd}|dkrd}n|dkrd}|jD] }|jd|||jd|||jd||qdS)Nrrrrrr)rDrr!)r"rLr0rrrr)r!r!r#r1s  z'PrometheusStatLogger.record_sleep_statecCs|d|jjdS)Nr)r#rrr,r!r!r#r-sz+PrometheusStatLogger.log_engine_initializedr r3rr)r4r5r6rrrrrrrrr rr rFr8r$rr r#rrrr+r1r-r!r!r!r#rs>  b  r PromMetricmetric engine_idxsrcsfdd|DS)Ncsi|] }|t|qSr!rrrOrr!r#rsz#make_per_engine..r!)rOrPrr!rQr#rsr mantissa_lst max_valuecCsDd}g} |D]}|d|}||kr||q|S|d7}q)z Builds a list of buckets with increasing powers of 10 multiplied by mantissa values until the value exceeds the specified maximum. rTrr)r?)rRrSexponentrmvaluer!r!r# build_bucketss  rWcCstgd|S)zR Example: >>> build_1_2_5_buckets(100) [1, 2, 5, 10, 20, 50, 100] )rrr)rW)rSr!r!r#rsrc @seZdZdZ     ddedeedBdeedBd ed ed ef d d Z  dde dBde dBde dBdedBfddZ ddedefddZddZddZdS) StatLoggerManagera StatLoggerManager: Logging happens at the level of the EngineCore (per scheduler). * DP: >1 EngineCore per AsyncLLM - loggers for each EngineCore. * With Local Logger, just make N copies for N EngineCores. * With Prometheus, we need a single logger with N "labels" This class abstracts away this implementation detail from the AsyncLLM, allowing the AsyncLLM to just call .record() and .log() to a simple interface. NTFrrrPcustom_stat_loggersenable_default_loggersaggregate_engine_logging client_countc Cs|r|ndg|_g|_g}|dur|||r3ttjr3|dkr(tdn |r,tnt }| |d} |D])} t | t rRt | trR| ||jd} t | trQd} nt||j| d} |j | q7| so|j t||jdSdS)NrrzfAsyncLLM created with api_server_count more than 1; disabling stats logging to avoid incomplete stats.F)rrDT)rrDr)rD stat_loggersrr isEnabledForloggingINFOrrrGr?r;r<r=rrr) r"rrPrYrZr[r\stat_logger_factoriesdefault_logger_factorycustom_prometheus_loggerstat_logger_factoryglobal_stat_loggerr!r!r#r$sL      zStatLoggerManager.__init__r&r'r(r)cCs.|durd}|jD] }|j||||dq dS)Nrr)r]r+)r"r&r'r(r)rr!r!r#r+s zStatLoggerManager.recordrrLr0cCs|jD]}|||qdSr )r]r1)r"rLr0rr!r!r#r1$s z$StatLoggerManager.record_sleep_statecC|jD]}|qdSr )r]r.)r"rr!r!r#r.(  zStatLoggerManager.logcCrfr )r]r-)r" agg_loggerr!r!r#r-,rgz(StatLoggerManager.log_engine_initialized)NNTFrrrM)r4r5r6r7r rFr8StatLoggerFactoryrr$rrrr+r1r.r-r!r!r!r#rXsD   7  rX)Br_rJabcrrcollections.abcrtypingrprometheus_clientrrr vllm.envsrvllm.compilation.cuda_graphr vllm.configr r 4vllm.distributed.kv_transfer.kv_connector.v1.metricsr r vllm.loggerr vllm.pluginsrrvllm.v1.enginervllm.v1.metrics.perfrvllm.v1.metrics.prometheusrvllm.v1.metrics.statsrrrrrvllm.v1.spec_decode.metricsrrr4rr8rr<AggregateStatLoggerFactoryrirrFrCrrGrrrrN__annotations__objectdictrrWrrXr!r!r!r#s`         ;D(0