o ۷i<@sddlZddlmZddlZddlmZddlmZddl m Z ddl m Z m Z mZmZeGdddeZd efd d Zd efd d Zd efddZded efddZdeedee dededeedeeefdeeeeffddZdS)N) dataclass)PreTrainedTokenizerBase) SampleRequest)RequestFuncOutput)"MILLISECONDS_TO_SECONDS_CONVERSIONTERM_PLOTLIB_AVAILABLEBenchmarkMetricsTaskTypec@seZdZUdZeed<dZeed<dZeed<dZe e eefed<dZ eed<dZ e ed <dZeed <dZeed <dZeed <dZeed <dZe e eefed<dZeed<dZeed<dZeed<dZe e eefed<dS)MultiModalsBenchmarkMetricsmean_audio_ttfp_msmedian_audio_ttfp_msstd_audio_ttfp_msNpercentiles_audio_ttfp_mstotal_audio_duration_srtotal_audio_framesaudio_throughputmean_audio_rtfmedian_audio_rtf std_audio_rtfpercentiles_audio_rtfmean_audio_duration_smedian_audio_duration_sstd_audio_duration_spercentiles_audio_duration_s)__name__ __module__ __qualname__r float__annotations__r rrlisttuplerrintrrrrrrrrrr#r#Z/home/ubuntu/vllm_env/lib/python3.10/site-packages/vllm_omni/benchmarks/metrics/metrics.pyr s            r metricscCstdjddddtdd|jtdd|j|dur(tdd ||td kr6td d |td d |td d|j|rRtd d|jt|tr`td d|j |t j ksid|vrnt d|t ||||t j kr~t||tddS)N {s:{c}^{n}}z Serving Benchmark Result 2=snc {:<40} {:<10}zSuccessful requests:zFailed requests:zMaximum request concurrency:inf{:<40} {:<10.2f}zRequest rate configured (RPS):zBenchmark duration (s):zRequest throughput (req/s):zRequest goodput (req/s):Peak concurrent requests:e2elz2==================================================)printformat completedfailedrrequest_throughputrequest_goodput isinstancer max_concurrent_requestsr GENERATIONprocess_one_metricprint_text_metricsprint_audio_metrics) task_typeselected_percentile_metricsmax_concurrency request_ratebenchmark_durationgoodput_config_dictr%r#r#r$ print_metricss&        rDcCstdjddddtdd|jt|trr?r%metricr#r#r$r<<s"    r<cCsltdjddddtdd|jtdd |jtdd |j|D] }|d r3t||q'dS) Nr&z Audio Result r'r(r)r/z"Total audio duration generated(s):r-zTotal audio frames generated:z#Audio throughput(audio duration/s):rE)r2r3rrrrKr;)r?r%rLr#r#r$r=Ns  r=metric_attribute_namecCs^dddddddd}|||}td j|d d d |d k}|dk}d}d}|r-d}d}n|r3d}d}d||}t||d} td||ddd| dd||} t|| d} td||ddd| dd||} t|| g} | D]*\}}|rtt|nt|}d|d||d}t|dd|dqdS)NzTime to First Tokenz'Time per Output Token (excl. 1st token)zInter-token LatencyzEnd-to-end LatencyzTime to First PacketzReal Time FactorzAudio Duration)ttfttpotitlr1 audio_ttfp audio_rtfaudio_durationr&r'-r)rRrS_msz (ms)_sz (s)mean_r zMean :z<40 z<10.2fmedian_zMedian percentiles_P)getr2r3getattrupper is_integerstrr")rMr%metric_header_mapheader is_audio_rtfis_audio_durationsuffix unit_suffixmean_attr_name mean_valuemedian_attr_name median_valuepercentiles_attr_name percentiles percentilevaluep_strlabelr#r#r$r;XsB  & &  r;input_requestsoutputsdur_s tokenizerselected_percentilesrCreturnc .s"g} d} d} d}ggg}gggggg}tt|D]}||jr||j}|s:t|||jddj}| || ||j7} d}|dkra||j||j }||d}|||||j 7||j t ||ddt ||ddt ||dd|t ||d d||j | d7} q | dq |rg}g}d |vr|||d t d |vr|||d t d |vr||||d t d |vr|||d t t|D]}tddt||D}|r|d7}q| dkr1dPddt_tjdddd}d}dd|D}dd|D}|rtdd|D}tdd|D}tt||d}t|} t|}!t|D]g\}}"|"j|"j g}#|#d}$|"j D] }%|$|%7}$|#|$q|#D] }&t|&|}'d|'kr|krnq| |'d7<qt|"j|}(t|"j|"j |})t|(|)dD] }*|!|*d7<qːqqt| dkrtt| }tt|!}trddl}+|+},|,jt t| | dd|,jt t|!|!dd|,!nt"dt#dQid| d t|d!| d"t$| d#| |d$||d%t$| |d&| t$| |d't%pPdd(d)t&p[dd(d*t'pfdd(d+fd,d|Dd-t%p{dd(d.t&pdd(d/t'pdd(d0fd1d|Dd2t%pdd3t&pdd4t'pdd5fd6d|Dd7t$d8t$|d9t$|d:t%pdd;t&pddd|Dd?t%pdd(d@t&p dd(dAt'pdd(dBfdCd|DdDt%p,dd(dEt&p7dd(dFt'pBdd(dGfdHd|DdIt%pWdd(dJt&pbdd(dKt'pmdd(dLfdMd|DdN|dO|}-t(|||| | ||-|-| fS)RaCalculate the metrics for the benchmark. Args: input_requests: The input requests. outputs: The outputs of the requests. dur_s: The duration of the benchmark. tokenizer: The tokenizer to use. selected_percentiles: The percentiles to select. goodput_config_dict: The goodput configuration. Returns: A tuple of the benchmark metrics and the actual output lengths. rF)add_special_tokensrQr rRrS audio_framesrN audio_ttftrOr1cSsg|]\}}||kqSr#r#).0r*rr#r#r$ sz%calculate_metrics..NcSs|d|d|jd|dS)NrYz:  )r)msgcategoryfilenamelinenoliner#r#r$sz#calculate_metrics..zYAll requests failed. This is likely due to a misconfiguration on the benchmark arguments.) stacklevelcSsg|]}|jr|qSr#successr}outputr#r#r$rcSsg|]}|js|qSr#rrr#r#r$rrcss|]}|jVqdSN) start_timerr#r#r$ sz$calculate_metrics..css|] }|j|jVqdSr)rlatencyrr#r#r$rszOutput tokens per second)titlezConcurrent requests per secondz8tip: install termplotlib and gnuplot to plot the metricsr4r5rFrGr6r7rHrJ mean_ttft_ms std_ttft_msmedian_ttft_mspercentiles_ttft_msc$g|]}|tp d|dfqSrrnpror}p)ttftsr#r$r4$r rr rcrrrr) audio_ttfpsr#r$r8rrrrrc g|] }|tp d|fqSrrr)rSr#r$r< rrrrrrrcrrrr) audio_rtfsr#r$rCr mean_tpot_ms std_tpot_msmedian_tpot_mspercentiles_tpot_mscrrrr)tpotsr#r$rGr mean_itl_ms std_itl_ms median_itl_mspercentiles_itl_mscrrrr)itlsr#r$rKr mean_e2el_ms std_e2el_msmedian_e2el_mspercentiles_e2el_mscrrrr)e2elsr#r$rOrrIr9rr#))rangelenr output_tokensgenerated_text input_idsappend prompt_len text_latencyrNrPr_rrzipallwarnings formatwarningwarnminmaxr"rceilzeros enumeraterrr termplotlibfigureplotarangeshowr2r summeanstdmedianrD).rsrtrurvrwrCr>r?r@rArBactual_output_lensrFr4good_completed all_tpotsr{i output_lenrOlatency_minus_ttft valid_metrics slo_values req_metric is_good_reqrIr9successful_outputsfailed_outputsmin_start_time max_end_timeduration_secondstokens_per_secondconcurrent_requests_per_secondr token_times current_time itl_value token_time second_bucketrequest_start_secondrequest_end_secondsecondtplfigr%r#)rSrrrrrrr$calculate_metricss                              !"#$%&'()+ r)r dataclassesrnumpyr transformersrvllm.benchmarks.datasetsr)vllm.benchmarks.lib.endpoint_request_funcrvllm.benchmarks.serverrrr r rDr<r=rbr;r rdictr!r"rr#r#r#r$sD       -