o `۷i0@sddlZddlZddlZddlmZmZmZmZmZm Z ddl m Z ddl m Z mZmZddlmZmZddlmZeeZde eefded ed efd d Zded ededeeefd eeeeefff ddZdededed efddZde eefdedeeefd eeeeefffddZdeeefdeeefde eefded eeeeefff ddZdeeefdeeefd eeeffddZded eefdd Z d!eegee eefeeefffd eegeeeeeffffd"d#Z!d!eee efgeee e eeffeee efffd eee efgeee efee effffd$d%Z"ded eeeeefffd&d'Z#ed(d)ded ee eefeeefffd*d+Z$e$Z%dS),N)AnyCallableDictOptionalTupleUnion) DeploymentID)CONTROL_LOOP_INTERVAL_S'SERVE_AUTOSCALING_DECISION_COUNTERS_KEYSERVE_LOGGER_NAME)AutoscalingConfigAutoscalingContext) PublicAPIdesired_num_replicascurrent_num_replicasautoscaling_configreturncCsX||}|dkr |n|}t|||}tt||kr*||kr*|d8}|S)zApply scaling factors to the desired number of replicas. Returns the scaled number of replicas depending on the scaling factor. The computation uses the difference between desired and current to scale. r)get_upscaling_factorget_downscaling_factormathceilfloat)rrrreplicas_deltascaling_factorscaled_num_replicasrR/home/ubuntu/vllm_env/lib/python3.10/site-packages/ray/serve/autoscaling_policy.py_apply_scaling_factorss  rcurr_target_num_replicasconfig policy_statecCs|}|td}||kr$|dkrd}|d7}|t|jtkr#d}|}n9||kr[|dkr.d}|d8}|dk}|rE|jdurA|j}n |j}n|j}td|}|t|t krZd}|}nd}||t<||fS)z4Apply delay logic to the desired number of replicas.rrN)getr intupscale_delay_sr downscale_to_zero_delay_sdownscale_delay_smax)rrr r!decision_num_replicasdecision_counteris_scaling_to_zerodelay_srrr_apply_delay_logic.s8   r, num_replicascapacity_adjusted_min_replicascapacity_adjusted_max_replicascCst|t||S)zAClip replica count to be within capacity-adjusted min/max bounds.)r'min)r-r.r/rrr _apply_boundsisr1ctxcCs>t||j|j}t||j|j}t||j|j|\}}||fS)z?Apply the default parameters to the desired number of replicas.)rrr r1r.r/r,target_num_replicas)rr2r!bounded_num_replicasfinal_num_replicas updated_staterrr_apply_default_paramsus  r7user_policy_statecCs6t|tdi}t|||\}}|r||||fS)Nr)r r"r7update)r!r8rr2internal_policy_stater5r6rrr%_apply_default_params_and_merge_states  r;cCst|tdi}|||S)z~Merge user state with previous policy state, preserving internal keys. This mutates and returns `user_policy_state`. r)r r"r9)r!r8r:rrr%_merge_user_state_with_internal_states   r<cCs:|jdkr|jdkrttd|j|jS|jSdS)zm Returns the desired number of replicas if the cold start fast path applies, otherwise returns None. rrN)rtotal_num_requestsr'rrr rr3)r2rrr!_get_cold_start_scale_up_replicass  r> policy_funcc s4tdtdtttttffffdd }|S)z Wraps a custom policy function to automatically apply: - upscaling_factor / downscaling_factor - min_replicas / max_replicas bounds - upscale_delay_s / downscale_delay_s / downscale_to_zero_delay_s r2rcsJt|}|dur ||jfS|j}|\}}t||||\}}||fSN)r>r!copyr;)r2cold_start_replicasr!rupdated_custom_policy_stater5 final_stater?rrwrapped_policys   z1_apply_autoscaling_config..wrapped_policy) functoolswrapsr rr#rstrrr?rFrrEr_apply_autoscaling_configs (rKc sDtdtttfdttttftttffffdd }|S)z Wraps an application-level custom policy function to automatically apply per-deployment: - upscaling_factor / downscaling_factor - min_replicas / max_replicas bounds - upscale_delay_s / downscale_delay_s / downscale_to_zero_delay_s contextsrc si}|D] \}}|j||<q|\}}|pi}i}i}|D]@\}}||vr3||||<q$||i}t|} | durO| ||<t|||||<q$t||||||\} } | ||<| ||<q$||fSr@)itemsr!rAr"r>r<r;) rLstate_per_deploymentdep_idr2desired_num_replicas_dictrCfinal_decisionsrD"custom_policy_state_per_deploymentrBr5final_dep_staterErrrFs>   z;_apply_app_level_autoscaling_config..wrapped_policy)rGrHrrr rr#rJrrEr#_apply_app_level_autoscaling_configs ,rTcCsB|j}|j}|dkrtd||}|j|}||}|ifS)Nrz!Number of replicas cannot be zero)rr ValueErrorget_target_ongoing_requestsr=)r2num_running_replicasr target_num_requests error_ratiorrrr!_core_replica_queue_length_policy's  rZalpha) stabilitycCs"t|}|dur ||jfSt|S)aThe default autoscaling policy based on basic thresholds for scaling. There is a minimum threshold for the average queue length in the cluster to scale up and a maximum threshold to scale down. Each period, a 'scale up' or 'scale down' decision is made. This decision must be made for a specified number of periods in a row before the number of replicas is actually scaled. See config options for more details. Assumes `get_decision_num_replicas` is called once every CONTROL_LOOP_PERIOD_S seconds. N)r>r!rZ)r2rBrrr'replica_queue_length_autoscaling_policy4s r])&rGloggingrtypingrrrrrrray.serve._private.commonrray.serve._private.constantsr r r ray.serve.configr r ray.util.annotationsr getLoggerloggerr#rrrIr,r1r7r;r<r>rKrTrZr]default_autoscaling_policyrrrrs        ;                 C