o .i$ @s UddlZddlZddlmZmZddlmZddlmZddl m Z m Z m Z ddl mZddlmZddlmZmZddlmZmZdd lmZdd lmZmZdd lmZdd lmZdd l m!Z!ddl"m#Z#m$Z$m%Z%m&Z&ddl'm(Z(m)Z)ee*Z+eee,gdfZ-e.dZ/e/e-BZ0GdddeZ1de2e0fddZ3Gddde1Z4Gddde1Z5Gddde5e4Z6Gddde4Z7Gddde4Z8e e Be BZ9ee:d<d e9d!e2e,d"e;ded&e,de2e,fd)d*Z?Gd+d,d,Z@dS)-N)ABCabstractmethod)Callable) TypeAlias)CounterGauge Histogram)CUDAGraphLogging)SupportsMetricsInfo VllmConfig)KVConnectorLoggingKVConnectorPrometheus) init_logger)STAT_LOGGER_PLUGINS_GROUPload_plugins_by_group) FinishReason)PerfMetricsLogging)unregister_vllm_metrics)CachingMetricsIterationStatsMultiModalCacheStatsSchedulerStats)SpecDecodingLoggingSpecDecodingPromStatLoggerBaseAggregateStatLoggerBasec @s~eZdZdZeddedefddZe  ddedBd e dBd e dBd efd d Z eddZ ddZ dedefddZdS)raInterface for logging metrics. API users may define custom loggers that implement this interface. However, note that the `SchedulerStats` and `IterationStats` classes are not considered stable interfaces and may change in future versions. r vllm_config engine_indexcCdSN)selfrrr r T/home/ubuntu/veenaModal/venv/lib/python3.10/site-packages/vllm/v1/metrics/loggers.py__init__/zStatLoggerBase.__init__Nscheduler_statsiteration_statsmm_cache_stats engine_idxcCrrr r!r%r&r'r(r r r"record2szStatLoggerBase.recordcCrrr r!r r r"log_engine_initialized;r$z%StatLoggerBase.log_engine_initializedcCrrr r+r r r"log>r$zStatLoggerBase.logis_awakelevelcCrrr )r!r.r/r r r"record_sleep_stateAr$z!StatLoggerBase.record_sleep_staterNr)__name__ __module__ __qualname____doc__rr intr#rrrr*r,r-r0r r r r"r's(  returncCsRg}ttD]\}}t|trt|ts!td|d|d||q|S)NzStat logger plugin z+ must be a subclass of StatLoggerBase (got z).) rritems isinstancetype issubclassr TypeErrorappend) factoriesname plugin_classr r r"!load_stat_logger_plugin_factoriesEs  rBc@s*eZdZdZededeefddZdS)rzNAbstract base class for loggers that aggregate across multiple DP engines.rengine_indexescCrrr r!rrCr r r"r#Zr$z AggregateStatLoggerBase.__init__N) r3r4r5r6rr listr7r#r r r r"rVsc @seZdZd"dedefddZddZdefd d Zd e fd d Z dede de fddZ e ddZ  d#dedBd e dBdedBdefddZddZddZddZd d!ZdS)$LoggingStatLoggerrrrcCs||_||_|tt|_t|_t|_ t|_ t |_ |jj }t||_d|_|jjjr>t|jjj|jjj|_d|_d|_d|_d|_|rUt||_dSdS)NF)rr_resettime monotonicrlast_scheduler_statsrprefix_caching_metrics connector_prefix_caching_metricsmm_caching_metricsrspec_decoding_loggingkv_transfer_configr kv_connector_loggingcudagraph_loggingobservability_configcudagraph_metricsr compilation_configcudagraph_modecudagraph_capture_sizeslast_prompt_throughputlast_generation_throughputengine_is_idle aggregated_enable_perf_statsrperf_metrics_logging)r!rrrPr r r"r#_s.  zLoggingStatLogger.__init__cCs"||_d|_d|_d|_d|_dSr2) last_log_timenum_prompt_tokensnum_generation_tokensnum_corrupted_reqsnum_preemptions)r!nowr r r"rH}s  zLoggingStatLogger._resetr8cCs |jjjSr)rrSenable_mfu_metricsr+r r r"r\s z$LoggingStatLogger._enable_perf_statsr&cCsD|j|j7_|j|j7_|j|j7_|j|j7_dSr)r_r`rarbnum_preempted_reqs)r!r&r r r"_track_iteration_statssz(LoggingStatLogger._track_iteration_stats tracked_statsrccCs"||j}|dkr dSt||S)NrG)r^float)r!rgrc delta_timer r r"_get_throughputs  z!LoggingStatLogger._get_throughputcCs d|jS)NzEngine {:03d}: )formatrr+r r r" log_prefixs zLoggingStatLogger.log_prefixNr%r'r(cCs|r|||dur[|j|j|jdur|j|j|jdur*|j|j|j}r5|j ||j durF|j durF|j |j |j sL||_ |j}r[|r[|j||re|j|dSdS)zLog Stats to standard output.N)rfrLobserveprefix_cache_statsconnector_prefix_cache_statsrMspec_decoding_statsrOkv_connector_statsrQrRcudagraph_statsr[rK perf_statsr\r]rN)r!r%r&r'r(rqrsr r r"r*s,        zLoggingStatLogger.recordcCsVt}||j|}||j|}||t|||j|jf |_ ||_||_dSr) rIrJrjr_r`rHanyrXrYrZ)r!rcprompt_throughputgeneration_throughputr r r" _update_statss  zLoggingStatLogger._update_statscCrrr r+r r r"aggregate_scheduler_statsz+LoggingStatLogger.aggregate_scheduler_statscCsd|||jrtjntj}gd}|j|j|jj |jj g}|j dkr1| d| |j | ddg| |jjd|jjdgtjrU| d| |j|jjsg| d| |jjd|jjsy| d | |jjd||jd |g|R|jj|d |jj|d |jdur|jj|d |r|jj||jd dSdS) N)z$Avg prompt throughput: %.1f tokens/sz(Avg generation throughput: %.1f tokens/szRunning: %d reqszWaiting: %d reqsrzPreemptions: %dzGPU KV cache usage: %.1f%%zPrefix cache hit rate: %.1f%%dzCorrupted: %d reqsz&External prefix cache hit rate: %.1f%%zMM cache hit rate: %.1f%%z, )log_fn)r{rl)rwrxrZloggerdebuginforXrYrKnum_running_reqsnum_waiting_reqsrbr>extendkv_cache_usagerLhit_rateenvsVLLM_COMPUTE_NANS_IN_LOGITSrarMemptyrNrljoinrOr-rQrRr\r])r!r{ log_partslog_argsr r r"r-sV          zLoggingStatLogger.logcCs(|jjjrtd|j|jjjdSdS)NzSEngine %03d: vllm cache_config_info with initialization after num_gpu_blocks is: %d)r cache_confignum_gpu_blocksr|r}rr+r r r"r, s z(LoggingStatLogger.log_engine_initializedr1r2)r3r4r5r r7r#rHboolr\rrfrhrjpropertyrlrrr*rwrxr-r,r r r r"rF^s.   # :rFc @seZdZdedeefddZeddZde fdd Z dd e d Bd e d Bde d BdefddZddZddZddZd S)AggregatedLoggingStatLoggerrrCcCs2||_dd|jD|_tj||ddd|_dS)NcSsi|]}|tqSr )r.0idxr r r" sz8AggregatedLoggingStatLogger.__init__..)rT)rClast_scheduler_stats_dictrFr#r[rDr r r"r#s  z$AggregatedLoggingStatLogger.__init__cCsdt|jS)Nz{} Engines Aggregated: )rklenrCr+r r r"rl$sz&AggregatedLoggingStatLogger.log_prefixr8cCsdS)NFr r+r r r"r\(ryz.AggregatedLoggingStatLogger._enable_perf_statsNrr%r&r'r(cCsH||jvr td|dStj|||||d|dur"||j|<dSdSNzUnexpected engine_idx: %dr'r()rCr|warningrFr*rr)r r r"r*,s  z"AggregatedLoggingStatLogger.recordcCsht|_|jD]}|jj|j7_|jj|j7_|jj|j7_q |jjt|j_dSr)rrKrvaluesrrrr)r!rKr r r"rx@sz5AggregatedLoggingStatLogger.aggregate_scheduler_statscCst|dSr)rFr-r+r r r"r-NszAggregatedLoggingStatLogger.logcCs,|jjjrtdt|j|jjjdSdS)NzR%d Engines: vllm cache_config_info with initialization after num_gpu_blocks is: %d)rrrr|r~rrCr+r r r"r,Qs z2AggregatedLoggingStatLogger.log_engine_initializedr2)r3r4r5r rEr7r#rrlrr\rrrr*rxr-r,r r r r"rs.    rc @sfeZdZdedeededdfddZ  dd edBd e dBd e dBd efd dZ ddZ ddZ dS)PerEngineStatLoggerAdapterrrCper_engine_stat_logger_factoryr8NcCs*i|_||_|D] }||||j|<qdSr)per_engine_stat_loggersrC)r!rrCrrr r r"r#\s z#PerEngineStatLoggerAdapter.__init__rr%r&r'r(cCs6||jvr td|dS|j|j||||ddSr)rr|rr*r)r r r"r*is    z!PerEngineStatLoggerAdapter.recordcC|jD]}|qdSr)rrr-r!per_engine_stat_loggerr r r"r-z zPerEngineStatLoggerAdapter.logcCrr)rrr,rr r r"r,~rz1PerEngineStatLoggerAdapter.log_engine_initializedr2)r3r4r5r rEr7PerEngineStatLoggerFactoryr#rrrr*r-r,r r r r"r[s.   rc @seZdZeZeZeZe Z e Z dde deedBfddZdedefdd Z  dd edBd edBd edBdefddZddedefddZddZdS)PrometheusStatLoggerNrrCc(s|durdg}||_t||_|jj|_|jj|_ddg}|jj|jj }fdd|D}| |j |||_ | ||||_|jddd|d }t|||_|jd d d|d }t|||_|jd d |dgddi|_gd}|D]fdd|D|j<qu||jddd|d } t| ||_tjr|jdd|d} t| ||_|jdd|d} t| ||_|jdd|d} t| ||_|jdd|d} t| ||_|jdd|d}t|||_|jdd |d}t|||_|jd!d"|d}t|||_|jd#d$|d}t|||_ |jd%d&|d}t|||_!|jd'd(|d}t|||_"i|_#|jd)d*|d+gdt$D]fd,d|D|j#<qC|j%d-d&t&||d.}t|||_'|j%d/d(t&||d.}t|||_(|j%d0d1gd2|d.}t|||_)|j%d3d4t&||d.}t|||_*|j%d5d6gd7|d.}t|||_+|j%d8d9t&||d.}t|||_,|j%d:d;gd<|d.}t|||_-|j%d=d>gd?|d.}t|||_.|j%d@dAgd?|d.}t|||_/gdB}|j%dCdD||d.}t|||_0|j%dEdF||d.}t|||_1|j%dGdH||d.} t| ||_2|j%dIdJ||d.}!t|!||_3|j%dKdL||d.}"t|"||_4|j%dMdNt&||d.}#t|#||_5|jrgdO}$|j%dPdQ|$|d.}%t|%||_6|j%dRdS|$|d.}&t|&||_7|j%dTdU|$|d.}'t|'||_8n i|_6i|_7i|_8d|_9|j:durt;|jdVkrt<=dWdX|_>dY|_?dZ|_@|j:jA|_B|jd[d\d]|j>|j?|j@gd |_9dSdS)^Nr model_nameenginecsi|] }|t|gqSr )strr)rr r"rsz1PrometheusStatLogger.__init__..zvllm:num_requests_runningz.Number of requests in model execution batches. mostrecentr@ documentationmultiprocess_mode labelnameszvllm:num_requests_waitingz+Number of requests waiting to be processed.zvllm:engine_sleep_statezEngine sleep state; awake = 0 means engine is sleeping; awake = 1 means engine is awake; weights_offloaded = 1 means sleep level 1; discard_all = 1 means sleep level 2. sleep_state)r@rrr)awakeweights_offloaded discard_allcsi|] }|j|dqS))rrr)labelsr)gauge_engine_sleep_statersr r"rs zvllm:kv_cache_usage_percz*KV-cache usage. 1 means 100 percent usage.zvllm:corrupted_requestszMCorrupted requests, in terms of total number of requests with NaNs in logits.)r@rrzvllm:prefix_cache_queriesz;Prefix cache queries, in terms of number of queried tokens.zvllm:prefix_cache_hitsz7Prefix cache hits, in terms of number of cached tokens.z"vllm:external_prefix_cache_querieszsExternal prefix cache queries from KV connector cross-instance cache sharing, in terms of number of queried tokens.zvllm:external_prefix_cache_hitszoExternal prefix cache hits from KV connector cross-instance cache sharing, in terms of number of cached tokens.zvllm:mm_cache_queriesz?Multi-modal cache queries, in terms of number of queried items.zvllm:mm_cache_hitsz;Multi-modal cache hits, in terms of number of cached items.zvllm:num_preemptionsz0Cumulative number of preemption from the engine.zvllm:prompt_tokensz#Number of prefill tokens processed.zvllm:generation_tokensz&Number of generation tokens processed.zvllm:request_successz)Count of successfully processed requests.finished_reasonc s$i|]}|t|tqSr rrr)counter_request_success_baserreasonr r"ras zvllm:request_prompt_tokens)r@rbucketsrzvllm:request_generation_tokenszvllm:iteration_tokens_totalz.Histogram of number of tokens per engine_step.)  @iiiii i@z&vllm:request_max_num_generation_tokensz;Histogram of maximum number of requested generation tokens.zvllm:request_params_nz%Histogram of the n request parameter.)r zvllm:request_params_max_tokensz.Histogram of the max_tokens request parameter.z vllm:time_to_first_token_secondsz,Histogram of time to first token in seconds.)MbP?{Gzt?{Gz?{Gz?g{Gz?gQ?g{Gz?皙?g????@@@$@4@D@T@gd@g@g@z vllm:inter_token_latency_secondsz,Histogram of inter-token latency in seconds.)rg?皙?g333333?rg333333?皙?333333?g?rrrrrrrrrrz*vllm:request_time_per_output_token_secondsz7Histogram of time_per_output_token_seconds per request.)rrg?rg?g@rrrg.@rg>@rgI@gN@g^@gn@g~@g@g@g@z vllm:e2e_request_latency_secondsz,Histogram of e2e request latency in seconds.zvllm:request_queue_time_secondsz5Histogram of time spent in WAITING phase for request.z#vllm:request_inference_time_secondsz5Histogram of time spent in RUNNING phase for request.z!vllm:request_prefill_time_secondsz5Histogram of time spent in PREFILL phase for request.z vllm:request_decode_time_secondsz4Histogram of time spent in DECODE phase for request.z'vllm:request_prefill_kv_computed_tokenszMHistogram of new KV tokens computed during prefill (excluding cached tokens).)rgMb`?rrrrrrrrrrrr<xi,iXiizvllm:kv_block_lifetime_secondsz|Histogram of KV cache block lifetime from allocation to eviction. Sampled metrics (controlled by --kv-cache-metrics-sample).z'vllm:kv_block_idle_before_evict_secondszqHistogram of idle time before KV cache block eviction. Sampled metrics (controlled by --kv-cache-metrics-sample).zvllm:kv_block_reuse_gap_secondszHistogram of time gaps between consecutive KV cache block accesses. Only the most recent accesses are recorded (ring buffer). Sampled metrics (controlled by --kv-cache-metrics-sample).rzfvllm:lora_requests_info prometheus metrics may be incorrect/misleading with data parallel deployments.max_lorawaiting_lora_adaptersrunning_lora_adapterszvllm:lora_requests_infozRunning stats on lora requests.sum)CrCrrrSshow_hidden_metricskv_cache_metricskv_cache_metrics_enabled model_configserved_model_name max_model_len_spec_decoding_clsspeculative_configspec_decoding_prom_kv_connector_clskv_connector_prom _gauge_clsmake_per_enginegauge_scheduler_runninggauge_scheduler_waitingrr0gauge_kv_cache_usagerr _counter_clscounter_corrupted_requestscounter_prefix_cache_queriescounter_prefix_cache_hits&counter_connector_prefix_cache_queries#counter_connector_prefix_cache_hitscounter_mm_cache_queriescounter_mm_cache_hitscounter_num_preempted_reqscounter_prompt_tokenscounter_generation_tokenscounter_request_successr_histogram_clsbuild_1_2_5_buckets#histogram_num_prompt_tokens_request'histogram_num_generation_tokens_requesthistogram_iteration_tokens+histogram_max_num_generation_tokens_requesthistogram_n_requesthistogram_max_tokens_requesthistogram_time_to_first_tokenhistogram_inter_token_latency'histogram_request_time_per_output_tokenhistogram_e2e_time_requesthistogram_queue_time_request histogram_inference_time_requesthistogram_prefill_time_requesthistogram_decode_time_request%histogram_prefill_kv_computed_requesthistogram_kv_block_lifetime$histogram_kv_block_idle_before_evicthistogram_kv_block_reuse_gapgauge_lora_info lora_configrr|rlabelname_max_loralabelname_waiting_lora_adapterslabelname_running_lora_adapters max_lorasr)(r!rrCrrper_engine_labelvaluesrrrrrrrrrrrrrrrrrrrrrrrrequest_latency_bucketsrrrrrrkv_cache_residency_bucketsrr r r )rrrrrr"r#s           zPrometheusStatLogger.__init__r; config_objcCs|}d|d<d\}}|dkrd}d}|dusJd||j||d|d }|jD]}|}t||d<|jd i|d q-dS) NrNNrzvllm:cache_config_infoz(Information of the LLMEngine CacheConfigzUnknown metrics info type rrrr ) metrics_inforkeysrCrrset)r!r;rrr@r info_gaugerr r r"log_metrics_infos$  z%PrometheusStatLogger.log_metrics_inforr%r&r'r(cCs|dur|j||j|j||j|j||j|j||j j |j ||j j |j durL|j||j j |j||j j |jdurY|j|j||jdurf|j|j||jr|jr|j|}|j|}|j|}|jD]}||j||j|jD]} || qq~|jdurd|j } d|j! } |j"| |j#| |j$|j%i} |jj&di| '|dur|j(||j |j)||j |durdSt*j+r|j,||j-|j.||j/|j0||j1|j2||j3|j4||j1|j3|j5D] } |j6|| q|j7D] }|j8||q%|j9D] }|j:||q4|j;D] }|j<||qC|j=D]s}|j>|j?||j@||jA|jB||jC|jD||jE|jF||jG|jH||jI|j1tJ|jKd}|jL|||jM||j1|jN||j3|jO||jP|jQr|jR||jQqRdS)zLog to prometheus.N,rr )Srrrrrrrrincrnqueriesrhitsrorrrprrmrqrrkv_cache_eviction_eventsrr r lifetime_seconds idle_secondsreuse_gaps_secondsr rrrrrrr rrset_to_current_timerrrrrrarrerr_rr`rmax_num_generation_tokens_iterr n_params_iterrtime_to_first_tokens_iterrinter_token_latencies_iterrfinished_requestsr finish_reasonr e2e_latencyr queued_timer prefill_timerinference_timer decode_timemaxnum_cached_tokensrrrrmean_time_per_output_tokenmax_tokens_paramr)r!r%r&r'r( lifetime_hist idle_hist reuse_histeventgaprrlora_info_labelsmax_gen_tokensn_paramttftitlfinished_requestprefill_kv_computedr r r"r*s                                        zPrometheusStatLogger.recordsleepr/cCs~d}d}d}|dkrd}|dkrd}n|dkrd}|jD] }|jd|||jd|||jd||qdS)Nrrrrrr)rCrr)r!r@r/rrrr(r r r"r0ks  z'PrometheusStatLogger.record_sleep_statecCs|d|jjdS)Nr)rrrr+r r r"r,~sz+PrometheusStatLogger.log_engine_initializedrr2rr)r3r4r5rrrrrrrrr rr rEr7r#rr rrrrr*r0r,r r r r"rs>  ?  r PromMetricmetric engine_idxsrcsfdd|DS)Ncsi|] }|t|qSr rrrCrr r"rsz#make_per_engine..r )rCrDrr rEr"rsr mantissa_lst max_valuecCsDd}g} |D]}|d|}||kr||q|S|d7}q)z Builds a list of buckets with increasing powers of 10 multiplied by mantissa values until the value exceeds the specified maximum. rTrr)r>)rFrGexponentrmvaluer r r" build_bucketss  rKcCstgd|S)zR Example: >>> build_1_2_5_buckets(100) [1, 2, 5, 10, 20, 50, 100] )rrr)rK)rGr r r"rsrc @seZdZdZ     ddedeedBdeedBd ed ed ef d d Z  dde dBde dBde dBdedBfddZ ddedefddZddZddZdS) StatLoggerManagera StatLoggerManager: Logging happens at the level of the EngineCore (per scheduler). * DP: >1 EngineCore per AsyncLLM - loggers for each EngineCore. * With Local Logger, just make N copies for N EngineCores. * With Prometheus, we need a single logger with N "labels" This class abstracts away this implementation detail from the AsyncLLM, allowing the AsyncLLM to just call .record() and .log() to a simple interface. NTFrrrDcustom_stat_loggersenable_default_loggersaggregate_engine_logging client_countc Cs|r|ndg|_g|_g}|dur|||r3ttjr3|dkr(tdn |r,tnt }| |d} |D])} t | t rRt | trR| ||jd} t | trQd} nt||j| d} |j | q7| so|j t||jdSdS)NrrzfAsyncLLM created with api_server_count more than 1; disabling stats logging to avoid incomplete stats.F)rrCT)rrCr)rC stat_loggersrr| isEnabledForloggingINFOrrrFr>r:r;r<rrr) r!rrDrMrNrOrPstat_logger_factoriesdefault_logger_factorycustom_prometheus_loggerstat_logger_factoryglobal_stat_loggerr r r"r#sL      zStatLoggerManager.__init__r%r&r'r(cCs.|durd}|jD] }|j||||dq dS)Nrr)rQr*)r!r%r&r'r(r|r r r"r*s zStatLoggerManager.recordrr@r/cCs|jD]}|||qdSr)rQr0)r!r@r/r|r r r"r0s z$StatLoggerManager.record_sleep_statecC|jD]}|qdSr)rQr-)r!r|r r r"r-  zStatLoggerManager.logcCrZr)rQr,)r! agg_loggerr r r"r,r[z(StatLoggerManager.log_engine_initialized)NNTFrrrA)r3r4r5r6r rEr7StatLoggerFactoryrr#rrrr*r0r-r,r r r r"rLsD   7  rL)ArSrIabcrrcollections.abcrtypingrprometheus_clientrrr vllm.envsrvllm.compilation.cuda_graphr vllm.configr r 4vllm.distributed.kv_transfer.kv_connector.v1.metricsr r vllm.loggerr vllm.pluginsrrvllm.v1.enginervllm.v1.metrics.perfrvllm.v1.metrics.prometheusrvllm.v1.metrics.statsrrrrvllm.v1.spec_decode.metricsrrr3r|r7rr;AggregateStatLoggerFactoryr]rrErBrrFrrrrB__annotations__objectdictrrKrrLr r r r"s`         :D(