o `۷i @sddlZddlZddlZddlZddlZddlmZddlmZm Z m Z m Z m Z m Z mZmZddlZddlmZddlmZddlmZddlmZddlmZmZdd lmZdd lmZm Z m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&m'Z'dd l(m)Z)dd l*m+Z+m,Z,m-Z-m.Z.m/Z/m0Z0m1Z1m2Z2m3Z3dd l4m5Z5ddl6m7Z7ddl8m9Z9ddl:m;Z;mZ>ddl?m@Z@ddlAmBZBddlCmDZDddlEmFZFmGZGmHZHmIZIddlJmKZKmLZLddlMmNZNddlOmPZPddlQmRZRddlSmTZTddlUmVZVmWZWmXZXmYZYddlZm[Z[m\Z\m]Z]m^Z^ddl_m`Z`maZambZbmcZcmdZemfZfddlgmhZhmiZimjZjmkZkmlZlmmZmmnZnmoZompZpmqZqmrZrmsZsmtZtmuZumvZvddlwmxZxeye2ZzdZ{d Z|d!Z}Gd"d#d#Z~d$e erd%erd&e ed'e e'fd(d)Zd*erd+erd'efd,d-Zd.e ed/e ed0e e'fd1d2ZdS)3N) defaultdict)AnyDictIterableListOptionalSetTupleUnion) build_address)run_background_task) GcsClient) ActorHandle)ApplicationStateManagerStatusOverview)AutoscalingStateManager) DeploymentIDDeploymentSnapshotHandleMetricReportNodeIdReplicaMetricReportRequestProtocolRequestRoutingInfoRunningReplicaInfoTargetCapacityDirection)DeploymentConfig) CONTROL_LOOP_INTERVAL_S)RAY_SERVE_CONTROLLER_CALLBACK_IMPORT_PATHRAY_SERVE_ENABLE_DIRECT_INGRESS*RAY_SERVE_RPC_LATENCY_WARNING_THRESHOLD_MS(RECOVERING_LONG_POLL_BROADCAST_TIMEOUT_SSERVE_CONTROLLER_NAMESERVE_DEFAULT_APP_NAMESERVE_LOGGER_NAMESERVE_NAMESPACE)ControllerHealthMetricsTracker)create_cluster_node_info_cache)DeploymentInfo)DeploymentReplicaDeploymentStateManager) EndpointState)ExternalScalerDisabledError)set_proxy_default_grpc_options)$configure_http_options_with_defaults)%configure_autoscaling_snapshot_loggerconfigure_component_logger#configure_component_memory_profilerget_component_logger_file_path) LongPollHostLongPollNamespace)NodePortManager)ProxyStateManager)RayInternalKVStore) ServeUsageTag)call_function_from_import_path"get_all_live_placement_group_namesget_head_node_idis_grpc_enabled)DeploymentMode HTTPOptions ProxyLocation gRPCOptions) ActorNameListApplicationArgsDeploymentArgsDeploymentRoute EndpointInfo EndpointSet)APITypeApplicationDetailsDeploymentDetailsHTTPOptionsSchema LoggingConfig ProxyDetailsReplicaDetails ReplicaRankServeActorDetailsServeApplicationSchemaServeDeploySchemaServeInstanceDetailsTarget TargetGroupgRPCOptionsSchema)metricszserve-app-config-checkpointzserve-logging-config-checkpointc @seZdZdZdddededeefddZdefd d Z dd d Z d e fddZ de fddZdefddZdefddZdefddZdefddZddZdeee ffd d!Zd"efd#d$Zd eeeeefffd%d&Zd efd'd(Zd eeeffd)d*Z d efd+d,Z!d-d.Z"dd/d0Z#dd1d2Z$dd3d4Z%d5e&d6e&d7e fd8d9Z'd:d;Z(dd?Z-d eee.e/ffd@dAZ0d e1fdBdCZ2d eeeffdDdEZ3dFed ee4fdGdHZ5dIed e&fdJdKZ6dIedLed e7fdMdNZ8d efdOdPZ9d efdQdRZ:dSdTZ;d eddZd[Z?d\d]Z@d^eee.efd_eeefd dfd`daZAdbedce.edded dfdedfZB gddhe+die&d dfdjdkZCddbedIed efdmdnZDd eee*eEefffdodpZFded eeGfdqdrZHd e.efdsdtZIdedue d dfdvdwZJddxeeKd efdydzZLd e.eMfd{d|ZN  }ddIeed~ed|_?t|_@d|_Ai|_BtCtDDd|_E|FtG|Hd|_Id|_J|KtL|_M|Ni|_Og|_P|Qg|_RdS)Nz$Controller must be on the head node.)addressz ray-serve- controller)component_name component_idz0Calling user-provided callback from import path .zODirect ingress is enabled in ServeController, enabling proxy on head node only.)rX head_node_idcluster_node_info_cachelogging_configrWT)all_namespacescSs g|] }|dtkr|dqS) namespacename)r$).0actorrgS/home/ubuntu/vllm_env/lib/python3.10/site-packages/ray/serve/_private/controller.py s  z,ServeController.__init__..)node_idnode_ipnode_instance_idactor_id actor_name worker_id log_file_pathF)controller_start_time)Srayget_runtime_context get_node_id_controller_node_idr:rcray_worker_namespacer gcs_address gcs_clientr6kv_storer2long_poll_hostasyncioEventdone_recovering_event_autoscaling_loggerrYgetLOGGING_CONFIG_CHECKPOINT_KEYpickleloads!reconfigure_global_logging_configr0strosgetpidrloggerinfor8r&r`updater_direct_ingress_enabledr<HeadOnlylocationr5r-r,proxy_state_managerr*endpoint_stateutillist_named_actorsrautoscaling_state_managerr)r9deployment_state_managerrapplication_state_managerrNget_node_ip_addressget_node_instance_id get_actor_idr! get_worker_idr1_actor_details_shutting_down_shutdown_event_shutdown_start_time_registered_cleanup_actorsr%time_health_metrics_tracker_create_control_loop_metricsr run_control_loop_target_capacity_target_capacity_direction_recover_state_from_checkpointset _proxy_nodes_update_proxy_nodes_last_autoscaling_snapshots&_autoscaling_enabled_deployments_cache&_refresh_autoscaling_deployments_cache_last_broadcasted_target_groups)selfrXrYrWkv_store_namespacelog_config_checkpointall_current_actorsall_serve_actor_namesrgrgrh__init__s                zServeController.__init__cCs|jr |j|kr dS|jtt|||_|jtj |it dt t |dtt t |d|_tjdtjdddidtd |jdS) Nr[)r\r]ra)r]razController starting (version='z'). log_to_stderrFextraz;Configure the serve controller logger with logging config: )rYryputrrdumpsrznotify_changedr3GLOBAL_LOGGING_CONFIGr/rrrr.r~rrrr __version__debug)rrYrgrgrhrs:     z1ServeController.reconfigure_global_logging_configreturncCsdS)z+No-op to check if this controller is alive.Nrgrrgrgrh check_alive=szServeController.check_alivecCstSN)rrrrgrgrhget_pidAszServeController.get_pidreplica_metric_reportc Cst|j}|d}|jj||jjj|jjj|jjdd|j ||t kr=t d|jd|jd|dt d |j|dS) N deployment applicationreplicatagsz*Received autoscaling metrics from replica  with timestamp which is Bms ago. This is greater than the warning threshold RPC latency of ms. This may indicate a performance issue with the controller try increasing the RAY_SERVE_RPC_LATENCY_WARNING_THRESHOLD_MS environment variable.)r timestampreplica_metrics_delay_gauger replica_id deployment_idrdapp_name unique_idrrecord_replica_metrics_delayrrwarningr"record_request_metrics_for_replica)rrlatency latency_msrgrgrh'record_autoscaling_metrics_from_replicaDs* z7ServeController.record_autoscaling_metrics_from_replicahandle_metric_reportc Cst|j}|d}|jj||jj|jj|jdd|j ||t kr>t d|jd|jd|jd|dt d |j |dS) Nrrrhandlerz)Received autoscaling metrics from handle z for deployment rrrr)rrhandle_metrics_delay_gaugerrrdr handle_idrrecord_handle_metrics_delayrrrr!record_request_metrics_for_handle)rrrrrgrgrh&record_autoscaling_metrics_from_handle_s* z6ServeController.record_autoscaling_metrics_from_handlercC |j|Sr)r%get_total_num_requests_for_deploymentrrrgrgrh2_get_total_num_requests_for_deployment_for_testingzszBServeController._get_total_num_requests_for_deployment_for_testingcCrr)rget_metrics_for_deploymentrrgrgrh'_get_metrics_for_deployment_for_testings z7ServeController._get_metrics_for_deployment_for_testingcCs|jj|jSr)r_deployment_states _replicasrrgrgrh _dump_replica_states_for_testingsz0ServeController._dump_replica_states_for_testingcCs|jj|dSr)rr%_stop_one_running_replica_for_testingrrgrgrhrs z5ServeController._stop_one_running_replica_for_testingkeys_to_snapshot_idsc.|js|jIdH|j|IdHS)aProxy long pull client's listen request. Args: keys_to_snapshot_ids (Dict[str, int]): Snapshot IDs are used to determine whether or not the host should immediately return the data or wait for the value to be changed. N)r}is_setwaitrzlisten_for_change)rrrgrgrhrs z!ServeController.listen_for_changekeys_to_snapshot_ids_bytescr)zProxy long pull client's listen request. Args: keys_to_snapshot_ids_bytes (Dict[str, int]): the protobuf bytes of keys_to_snapshot_ids (Dict[str, int]). N)r}rrrzlisten_for_change_java)rrrgrgrhrs   z&ServeController.listen_for_change_javacC |jS)2Returns a dictionary of deployment name to config.)r get_endpointsrrgrgrhget_all_endpointss z!ServeController.get_all_endpointscCs(|}dd|D}t|dS)rcSs"i|] \}}|jt|ddqS)route)r)rdEndpointInfoProto)re endpoint_tag endpoint_dictrgrgrh z:ServeController.get_all_endpoints_java..) endpoints)ritemsrESerializeToString)rrdatargrgrhget_all_endpoints_javas z&ServeController.get_all_endpoints_javacCs|jduriS|jS)z7Returns a dictionary of node ID to proxy actor handles.N)rget_proxy_handlesrrgrgrh get_proxiess  zServeController.get_proxiescCs*|jdurdSt|jd}|S)z9Returns the proxy actor name list serialized by protobuf.N)names)rr@get_proxy_namesvaluesr)ractor_name_listrgrgrhrs  zServeController.get_proxy_namescCs2|j}|t|j}||j||_dS)zUpdate the nodes set where proxy actors should run. Controller decides where proxy actors should run (head node and nodes with deployment replicas). N)rget_active_node_idsrr`get_draining_nodesaddrur)rnew_proxy_nodesrgrgrhrs   z#ServeController._update_proxy_nodescsg}t|jD])}|j|}|D]\}}t||d|jj}|r2| ||||fqq ||_ fdd|j D|_ dS)Nrdrcsi|] \}}|vr||qSrgrg)rekvactive_dep_idsrgrhrs zJServeController._refresh_autoscaling_deployments_cache..) rrlist_app_nameslist_deployment_detailsrrrdeployment_configautoscaling_configappendrr)rresultrdeployment_detailsdep_namedetailsrrgr rhrs"  z6ServeController._refresh_autoscaling_deployments_cachec Cs|jdurdSg}|jD]5\}}}}t||d}|j|}|dur#q |j|}|dur3||r3q ||j dd||j|<q |rN|j d|idSdS)zEEmit structured autoscaling snapshot logs in a single batch per loop.Nr T) exclude_none snapshots) r~rrrget_deployment_snapshotrris_scaling_equivalentrdictr) rsnapshots_to_logrrrrdep_iddeployment_snapshotlastrgrgrh&_emit_deployment_autoscaling_snapshotss,     z6ServeController._emit_deployment_autoscaling_snapshotsc st}d}t} t}z ||||IdHWn"ty<}ztd|tdIdHWYd}~nd}~wwt|}|dkrTtjd|ddd id |j ||j ||d7}|j |||j _t}ttIdHt|}|j |||j _q ) NrTz,There was an exception in the control loop:  z%The last control loop was slow (took z~s). This is likely caused by running a large number of replicas in a single Ray cluster. Consider using multiple Ray clusters.rFr)r rrun_control_loop_step Exceptionr exceptionr{sleeprcontrol_loop_duration_gauge_srrrecord_loop_durationnum_control_loops_gaugenum_control_loopsrsleep_duration_gauge_slast_sleep_duration_s) rrecovering_timeout num_loops start_timeloop_start_timee loop_durationsleep_start_timesleep_durationrgrgrhrs@       z ServeController.run_control_loopr/r-r.csz|jWntytdYnw|jr/z|Wnty.tdYnw|jsJt ||krJt d|d|j d}z?t }|j }t |}|j ||j||js|s|j |dkrtjdt |ddd d id Wntytd Ynwz%t }|j}|s|r|t |} |j | |j| Wntytd Ynwz|WntytdYnwt } |t | } |j | |j| |jr7|jr7z t } |jj|jdt | } |j | |j| Wnty6tdYnw|d urJ|j|j |j B|j!ra|j "}t#$|t#%|&dSdS)Nz+Exception updating cluster node info cache.zException during shutdown.z Replicas still recovering after z@s, setting done recovering event to broadcast long poll updates.rz&Finished recovering deployments after z.2fzs.rFrz$Exception updating deployment state.z%Exception updating application state.z4Exception emitting deployment autoscaling snapshots.) proxy_nodeszException updating proxy state.)'r`rr$rr%rshutdownr}rrrrrdsm_update_duration_gauge_srrecord_dsm_update_durationrrrasm_update_duration_gauge_srecord_asm_update_durationr rnode_update_duration_gauge_srecord_node_update_durationrrproxy_update_duration_gauge_srecord_proxy_update_durationrdrop_stale_handle_metricsget_alive_replica_actor_idsget_alive_proxy_actor_idsrget_ingress_replicas_infor4 update_portsprune!_get_node_id_to_alive_replica_ids)rr/r-r.any_recoveringdsm_update_start_time dsm_durationasm_update_start_timeany_target_state_changed asm_durationnode_update_start_timenode_update_durationproxy_update_start_timeproxy_update_durationingress_replicas_info_listrgrgrhr#.s                         z%ServeController.run_control_loop_stepcCstjddd|_tjddd|_tjddd|_tjdd d|_tjd d d|_tjd d d|_tjdddd|_|j dt itjdddd|_ tjdddd|_dS)N'serve_controller_node_update_duration_sz:The control loop time spent on collecting proxy node info.) description.serve_controller_proxy_state_update_duration_sz4The control loop time spent on updating proxy state.3serve_controller_deployment_state_update_duration_sz9The control loop time spent on updating deployment state.4serve_controller_application_state_update_duration_sz:The control loop time spent on updating application state.!serve_controller_sleep_duration_sz.The duration of the last control loop's sleep.(serve_controller_control_loop_duration_sz&The duration of the last control loop."serve_controller_num_control_loopszpThe number of control loops performed by the controller. Increases monotonically over the controller's lifetime.)rm)rRtag_keysrm*serve_autoscaling_replica_metrics_delay_mszpTime taken for the replica metrics to be reported to the controller. High values may indicate a busy controller.r)serve_autoscaling_handle_metrics_delay_mszoTime taken for the handle metrics to be reported to the controller. High values may indicate a busy controller.r)rUGauger;r=r7r9r+r'r)set_default_tagsrrrsrrrrrgrgrhrsT z,ServeController._create_control_loop_metricscCsD|\}}}||_|dur tjdddid|j||ddSdS)Nz!Recovered config from checkpoint.rFr)deployment_time)_read_config_checkpointrrr apply_config)rr^ serve_configtarget_capacity_directionrgrgrhrsz.ServeController._recover_state_from_checkpointcCsD|jt}|dur t|\}}}}|tt||d|fSdS)aReads the current Serve config checkpoint. The Serve config checkpoint stores active application configs and other metadata. Returns: If the GCS contains a checkpoint, tuple of: 1. A deployment timestamp. 2. A Serve config. This Serve config is reconstructed from the active application states. It may not exactly match the submitted config (e.g. the top-level http options may be different). 3. The target_capacity direction calculated after the Serve was submitted. If the GCS doesn't contain a checkpoint, returns (0, None, None). N) applicationstarget_capacity)NN)ryrCONFIG_CHECKPOINT_KEYrrrPlistr)r checkpointr^rdrbconfig_checkpoints_dictrgrgrhr_s   z'ServeController._read_config_checkpointcCr)z_Used for testing. Returned dictionary maps deployment names to replica infos. )rget_running_replica_infosrrgrgrh_all_running_replicass z%ServeController._all_running_replicascC|jS)z^Returns the actor details for this controller. Currently used for test only. )rrrgrgrhget_actor_detailssz!ServeController.get_actor_detailscCs,z|jWStytdw)a%Returns comprehensive health metrics for the controller. This method provides detailed performance metrics to help diagnose controller health issues, especially as cluster size increases. Returns: Dictionary containing health metrics including: - Control loop performance (iteration speed, durations) - Event loop health (task count, scheduling delay) - Component update latencies - Autoscaling metrics latency (handle/replica) - Memory usage z/Exception collecting controller health metrics.)rcollect_metricsrr$rr%rrgrgrhget_health_metricss   z"ServeController.get_health_metricsrjcCs|jdurdS|j|S)zReturns the proxy details for the proxy on the given node. Currently used for test only. Will return None if the proxy doesn't exist on the given node. N)rget_proxy_detailsr)rrjrgrgrhrp*s z!ServeController.get_proxy_detailsrcCs.|jD] \}}||kr|jSqdS)zcReturns the deployment timestamp for the given app. Currently used for test only. N)rlist_app_statusesrdeployment_timestamp)rr _app_nameapp_status_inforgrgrhget_deployment_timestamps5s  z)ServeController.get_deployment_timestampsdeployment_namecCs|j||S)zjReturns the deployment details for the app and deployment. Currently used for test only. )rr)rrrvrgrgrhget_deployment_detailsAs z&ServeController.get_deployment_detailscC|jdurtS|jS)z$Return the HTTP proxy configuration.N)rr= get_configrrgrgrhget_http_configL  zServeController.get_http_configcCrx)z$Return the gRPC proxy configuration.N)rr?get_grpc_configrrgrgrhr|Rr{zServeController.get_grpc_configcCsN|jdurdS|}|jdkr$|jdurdSdt|j|j|jS|jS)z+Return the root url for the serve instance.Nzhttp://)rrzroot_urlhostr port root_path)r http_configrgrgrh get_root_urlXs   zServeController.get_root_urlcCs|jtduS)zReturns whether the config checkpoint has been deleted. Get the config checkpoint from the kv store. If it is None, then it has been deleted. N)ryrrfrrgrgrhconfig_checkpoint_deletedgsz)ServeController.config_checkpoint_deleted actor_handlecCs|j}||j|<dS)aIRegister an actor to be killed on serve.shutdown(). This allows deployments to register auxiliary actors (like caches, coordinators, etc.) that should be cleaned up when Serve shuts down. The actors must use lifetime="detached" to survive replica restarts, but will be explicitly killed during serve.shutdown(). Note: Registered actors are NOT persisted across controller restarts. For full persistence, use controller-managed deployment-scoped actors (see https://github.com/ray-project/ray/issues/60359). If the same actor is registered multiple times (e.g., from multiple router instances sharing a tree actor via get_if_exists=True), it will only be stored once. Args: actor_handle: The actor handle to register for cleanup. N) _actor_idhexr)rrrmrgrgrh _register_shutdown_cleanup_actoros z0ServeController._register_shutdown_cleanup_actorc Cs8|jD]}z tj|ddWqtyYqwdS)z0Kill all actors registered for shutdown cleanup.T no_restartN)rrrrkillr$)rrfrgrgrh_kill_registered_cleanup_actorss z/ServeController._kill_registered_cleanup_actorscCs|jsdS|jdurt|_tjdddid|jt|jt|j |j |j |j r;|j |}|j }|j }|j }|j dupW|j }|r}|r}|r}|r}|r}|tjdddidtj}tj|dddSt|jd kr|stjtd ddid|stjd ddid|stjd ddid|stjd ddid|stjdddiddSdSdS)amShuts down the serve instance completely. This method will only be triggered when `self._shutting_down` is true. It deletes the kv store for config checkpoints, sets application state to deleting, delete all deployments, and shuts down all proxies. Once all these resources are released, it then kills the controller actor. NzController shutdown started.rFrz1All resources have shut down, controller exiting.Trr"z not yet deletedzapplication not yet shutdownzdeployment not yet shutdownzendpoint not yet shutdownzproxy_state not yet shutdown)rrrrrrydeleterfrrr6rrrris_ready_for_shutdownrrrrrs current_actorr)rrapplication_is_shutdowndeployment_is_shutdownendpoint_is_shutdownproxy_state_is_shutdown_controller_actorrgrgrhr6s              zServeController.shutdownname_to_deployment_args_listname_to_application_argsc Csi}|D]-\}}g}|D] }t|}||j|j|j|j|j| dr)|j nddq|||<qi} |D] \}} t | | |<q:|j || |j dS)a0 Takes in a list of dictionaries that contain deployment arguments. If same app name deployed, old application will be overwritten. Args: name: Application name. deployment_args_list: List of serialized deployment information, where each item in the list is bytes representing the serialized protobuf `DeploymentArgs` object. `DeploymentArgs` contains all the information for the single deployment. name_to_application_args: Dictionary mapping application names to serialized application arguments, where each item is bytes representing the serialized protobuf `ApplicationArgs` object. `ApplicationArgs` contains the information for the application. route_prefixN)rvdeployment_config_proto_bytesreplica_config_proto_bytesdeployer_job_idingressr)rrB FromStringrrvrreplica_configrrHasFieldrrAr deploy_appssave_checkpoint) rrrname_to_deployment_argsrddeployment_args_listdeployment_args_deserializeddeployment_args_bytesargs%name_to_application_args_deserializedapplication_args_bytesrgrgrhdeploy_applicationss0   z#ServeController.deploy_applicationsrdrapplication_argscCs|||i||idS)av Deploy a single application (as deploy_applications(), but it only takes a single name and deployment args). This primarily exists as a shim to avoid changing Java code in https://github.com/ray-project/ray/pull/49168, and could be removed if the Java code was refactored to use the new bulk deploy_applications API. N)r)rrdrrrgrgrhdeploy_applicationsz"ServeController.deploy_applicationreconfigr^c Cstjd|s t}i}|\}}}t|||jd|_t|j|j |j|j |_|j D]}|j dur<|j r<|j |_ |j dd}|||j <q.|jtt||j|j|f|jj|j ||j|jd|jdS)zApply the config described in `ServeDeploySchema`. This will upgrade the applications to the goal state specified in the config. If `deployment_time` is not provided, `time.time()` is used. v2) curr_config new_configcurr_target_capacity_directionNT exclude_unset)r^rdrb)r7 API_VERSIONrecordrr_#calculate_target_capacity_directionrlog_target_capacity_changerrdrcrarrdryrrfrrrapply_app_configsr)rrr^new_config_checkpoint_r app_configapp_config_dictrgrgrhr`sL    zServeController.apply_configr}cCsnt||d}|j|}|dur%|rd|dnd}td|d|d|j|}t||d }|S) aGet the current information about a deployment. Args: name: the name of the deployment. Returns: DeploymentRoute's protobuf serialized bytes Raises: KeyError: If the deployment doesn't exist. r Nz in application ''r}z Deployment 'z' does not existr^)deployment_infor) rrget_deploymentKeyErrorrget_endpoint_routerCto_protor)rrdridrapp_msgrdeployment_routergrgrhget_deployment_info\s  z#ServeController.get_deployment_infocsfddjDS)zGets the current information about all deployments. Returns: Dict(deployment_id, (DeploymentInfo, route)) cs"i|] \}}||j|fqSrg)rr)rerrrrgrhr}rz=ServeController.list_deployments_internal..)rget_deployment_infosrrrgrrhlist_deployments_internalus  z)ServeController.list_deployments_internalcCs|j|}|r |jSdS)aGet the deployment config for the given deployment id. Args: deployment_id: The deployment id to get the config for. Returns: A deployment config object if the deployment id exist, None otherwise. N)rrrr)rrrrgrgrhget_deployment_configs z%ServeController.get_deployment_configcCs |jjS)z6Gets the current list of all deployments' identifiers.)rrkeysrrgrgrhlist_deployment_idss z#ServeController.list_deployment_idstarget_num_replicascCsX|j}|j|std|d|j|s#td|jd|d|j||dS)aAUpdate the target number of replicas for a deployment. Args: deployment_id: The deployment to update. target_num_replicas: The new target number of replicas. Raises: ExternalScalerDisabledError: If external_scaler_enabled is set to False for the application. z Application 'z ' not foundz'Cannot update replicas for deployment 'z' in application 'a '. The external scaling API can only be used when 'external_scaler_enabled' is set to true in the application configuration. Current value: external_scaler_enabled=false. To use this API, redeploy your application with 'external_scaler_enabled: true' in the config.N) rrdoes_app_exist ValueErrorget_external_scaler_enabledr+rdrset_target_num_replicas)rrrrrgrgrhupdate_deployment_replicass    z*ServeController.update_deployment_replicassourcec Cs|}|}i}|jj|d}|r|ni}|D]2\}}t||j||||j |j |j | ||j ||j||j||j|d ||<qt|jdd} t|jdd} t|j|jt|j| | |jrv|jnd||djddS)aGets details on all applications on the cluster and system-level info. The information includes application and deployment statuses, config options, error messages, etc. Args: source: If provided, returns application statuses for applications matching this API type. Defaults to None, which means all applications are returned. Returns: Dict that follows the format of the schema ServeInstanceDetails. )r) rdr docs_pathstatusmessagelast_deployed_time_sdeployed_app_configr deploymentsexternal_scaler_enableddeployment_topologyTrN)rdcontroller_infoproxy_locationrXrWproxiesrc target_groups)rzr|rrqget_app_configsrrGget_route_prefix get_docs_pathrrrrrget_app_sourcerrget_deployment_topologyrI parse_objrrTrQrrr>_from_deployment_moderrrpget_target_groups'_get_user_facing_json_serializable_dict) rrr grpc_configrc app_statuses app_configsrrtrXrWrgrgrhget_serve_instance_detailssX      z*ServeController.get_serve_instance_detailscCs^g}|jr-|ttjd|jtjdt|r-|ttj d|jtj d|S)z*Get target groups for proxy-based routing./)protocolrtargets) rrprrSrHTTP get_targetsr;r|GRPC)rrrgrgrh_get_proxy_target_groupss(    z(ServeController._get_proxy_target_groupsFfrom_proxy_managercs}js |S|durddjD}n|g}fdd|D}|s*|Sg}|D]}j|}||}|rD||q.|||q.|S)aGet target groups for direct ingress deployments. This returns target groups that point directly to replica ports rather than proxy ports when direct ingress is enabled. Following situations are possible: 1. Direct ingress is not enabled. In this case, we just return the target groups from the proxy implementation. 2. Direct ingress is enabled and there are no applications. In this case, we return target groups for proxy. Serve controller is running but there are no applications to route traffic to. 3. Direct ingress is enabled and there are applications. All applications have atleast one running replica. In this case, we return target groups for all applications with targets pointing to the running replicas. 4. Direct ingress is enabled and there are applications. Some applications have no running replicas. In this case, for applications that have no running replicas, we return target groups for proxy and for applications that have running replicas, we return target groups for direct ingress. If there are multiple applications with no running replicas, we return one target group per application with unique route prefix. NcSsg|]\}}|qSrgrg)rersrrgrgrhri<sz5ServeController.get_target_groups..cs g|] }j|dur|qSr)rrreapprrgrhriEs ) rrrrqrr_get_target_groups_for_appextend3_get_target_groups_for_app_with_no_running_replicas)rrrproxy_target_groupsappsrrapp_target_groupsrgrrhrs2     z!ServeController.get_target_groupscs`|j|}t||d}|j|}|sgS|j}dd|j|gDfdd|DS)z7Get running replica details for a specific application.)rrdcSsh|]}|jjqSrg)rr)re replica_inforgrgrh jszVServeController._get_running_replica_details_for_ingress_deployment..csg|] }|jvr|qSrg)rrereplica_detailrunning_replica_idsrgrhrips  zWServeController._get_running_replica_details_for_ingress_deployment..)rget_ingress_deployment_namerrrwreplicasrjr)rringress_deployment_namerrreplica_detailsrgrrh3_get_running_replica_details_for_ingress_deployment^s     zCServeController._get_running_replica_details_for_ingress_deploymentrcCsz||}|s gSg}||tj}|r |ttj|||dt|r;||tj}|r;|ttj|||d|S)a Create HTTP and gRPC target groups for a specific application. This function can return empty list if there are no running replicas. Or replicas have not fully initialized yet, where their ports are not allocated yet. rrrr) r_get_targets_for_protocolrrrrSr;r|r)rrrrr http_targets grpc_targetsrgrgrhrvs@   z*ServeController._get_target_groups_for_appcCs\g}|jtj}|jtj}|r|ttj|||d|r,|ttj|||d|S)z For applications that have no running replicas, we return target groups for proxy. This will allow applications to be discoverable via the proxy in situations where their replicas have scaled down to 0. r)rrrrrrrS)rrrrrrrgrgrhrs,zCServeController._get_target_groups_for_app_with_no_running_replicasrrcsfdd|DS)z?Create targets for a specific protocol from a list of replicas.cs6g|]}|rt|j||j|jdqS))ipr instance_idrd)_is_port_allocatedrRrk _get_portrlrnrrrrgrhris  z=ServeController._get_targets_for_protocol..rg)rrrrgr rhrs z)ServeController._get_targets_for_protocolcCsXtt}|jjD]}|j}|D]}|j}|durq|jj }|| |qq |Sr) rrrrrrr actor_node_idrrr)rnode_id_to_alive_replica_idsdsrrrjreplica_unique_idrgrgrhrEs z1ServeController._get_node_id_to_alive_replica_idsrcCst|}|||S)z;Allocate an HTTP port for a replica in direct ingress mode.)r4get_node_manager allocate_port)rrjrr node_managerrgrgrhallocate_replica_ports  z%ServeController.allocate_replica_portr block_portcCst|}|||||dS)z:Release an HTTP port for a replica in direct ingress mode.N)r4r release_port)rrjrrrrrrgrgrhrelease_replica_ports z$ServeController.release_replica_portrcCt|j}||j|S)zGet the port for a replica.)r4rrjget_portrrrrrrgrgrhr  zServeController._get_portcCr)z-Check if the port for a replica is allocated.)r4rrjis_port_allocatedrrrgrgrhr rz"ServeController._is_port_allocatedcCs2|j|}|j|}t|||d}|S)zReturn application status Args: name: application name. If application name doesn't exist, app_status is NOT_STARTED. )rd app_statusdeployment_statuses)rget_app_status_infoget_deployments_statusesrrr)rrdrr status_inforgrgrhget_serve_status s  z ServeController.get_serve_statusrcCs"g}|D] }|||q|Sr)rr#)rrstatusesrdrgrgrhget_serve_statusessz"ServeController.get_serve_statusescCs(g}|jD] }|||q|Sr)rrqrr#)rr$rdrgrgrhlist_serve_statuses$sz#ServeController.list_serve_statusescCs<|jt}|dur iSt|\}}}}dd|DS)NcSsi|] \}}|t|qSrg)rOr)rerrrgrgrhr0s z3ServeController.get_app_configs..)ryrrfrrr)rrhrrirgrgrhr*s zServeController.get_app_configscCr)aGet the external_scaler_enabled flag value for an application. This is a helper method specifically for Java tests to verify the flag is correctly set, since Java cannot deserialize Python Pydantic objects. Args: app_name: Name of the application. Returns: True if external_scaler_enabled is set for the application, False otherwise. )rrrrrgrgrhr5s z+ServeController.get_external_scaler_enabledcCs|j}dd|DS)z6Gets deployment status bytes for all live deployments.cSsg|]}|qSrg)rr)rerrgrgrhriFsz?ServeController.get_all_deployment_statuses..)rget_deployment_statuses)rr$rgrgrhget_all_deployment_statusesCs z+ServeController.get_all_deployment_statusescCs2t||d}|j|g}|sdS|dS)zGet deployment status by deployment name. Args: name: Deployment name. app_name: Application name. Default is "" because 1.x deployments go through this API. r Nr)rrr(rr)rrdrrrrgrgrhget_deployment_statusHs z%ServeController.get_deployment_statuscCr)zqDocs path for application. Currently, this is the OpenAPI docs path for FastAPI-integrated applications.)rr)rrdrgrgrhrYs zServeController.get_docs_pathcCr)zName of the ingress deployment in an application. Returns: Ingress deployment name (str): if the application exists. None: if the application does not exist. )rrr'rgrgrhr_s z+ServeController.get_ingress_deployment_namecCs$|D]}|j|q|jdS)zhDelete applications based on names During deletion, the application status is DELETING N)r delete_appr)rrrdrgrgrh delete_appshszServeController.delete_appsrcCs|j|dS)zRecord replica routing information for a replica. Args: info: RequestRoutingInfo including deployment name, replica tag, multiplex model ids, and routing stats. N)rrecord_request_routing_info)rrrgrgrhr-rsz+ServeController.record_request_routing_infocCr)a Get the current rank mapping for all replicas in a deployment. Args: deployment_id: The deployment ID to get ranks for. Returns: Dictionary mapping replica_id to ReplicaRank object (with rank, node_rank, local_rank). )r_get_replica_ranks_mappingrrgrgrhr.{s z*ServeController._get_replica_ranks_mappingTrcs$d|_|sdS|jIdHdS)aSet the shutting down flag on controller to signal shutdown in run_control_loop(). This is used to signal to the controller that it should proceed with shutdown process, so it can shut down gracefully. It also waits until the shutdown event is triggered if wait is true. Raises: RayActorError: if wait is True, the caller waits until the controller is killed, which raises a RayActorError. TN)rrr)rrrgrgrhgraceful_shutdowns  z!ServeController.graceful_shutdowncCs0d}tjD] }t|tjjr|jj}q|j|fS)z5Get the logging configuration (for testing purposes).N)rhandlers isinstancelogging MemoryHandlertarget baseFilenamerY)rrphandlerrgrgrh_get_logging_configs   z#ServeController._get_logging_configcCrl)z=Gets the controller's scale direction (for testing purposes).)rrrgrgrh_get_target_capacity_directionsz.ServeController._get_target_capacity_direction)rN)re)r}r)NF)F)T)r__name__ __module__ __qualname____doc__r=rJrr?rrrintrrrrrrrrrrrrrbytesrrrrrrrrrrr rfloatr#rrr rPrr_rrrkrNrmrorKrprurHrwrzr|rboolrrrr6rrr`rr'rrrrrrFrrSrrrLrrrrrRrrrErrr r r"r#r%r&rOrrr)r r*rrrr,rr-rMr.r/r7r8rgrgrgrhrVvs  "       ) k7  *     H  3  A     J B  2                  rVrrrrcCsd}d}|dur>t||r>|j}|j}||kr|}|S|dur(|dur(tj}|S|dur0d}|S||kr9tj}|Stj}|S|jdurHtj}|Sd}|S)zCCompares two Serve configs to calculate the next scaling direction.N)applications_matchrdrDOWNUP)rrrcurr_target_capacitynext_target_capacity_directionnext_target_capacityrgrgrhrs0   rconfig1config2cCs(dd|jD}dd|jD}||kS)zzChecks whether the applications in config1 and config2 match. Two applications match if they have the same name. cSh|]}|jqSrgrdrrgrgrhrz%applications_match..cSrIrgrJrrgrgrhrrK)rc)rGrHconfig1_app_namesconfig2_app_namesrgrgrhrAsrArDrFrEc CsL||kr$t|trtd|jd|d|ddStddSdS)z$Logs changes in the target_capacity.zTarget capacity scaling z from z to r^z.Target capacity entering 100% at steady state.N)r1rrrvaluelower)rDrFrErgrgrhrs r)r{r2rrr collectionsrtypingrrrrrrr r rrray._common.network_utilsr ray._common.utilsr ray._rayletr ray.actorr$ray.serve._private.application_staterr$ray.serve._private.autoscaling_staterray.serve._private.commonrrrrrrrrrray.serve._private.configrray.serve._private.constantsrrrrr r!r"r#r$4ray.serve._private.controller_health_metrics_trackerr%ray.serve._private.default_implr&"ray.serve._private.deployment_infor'#ray.serve._private.deployment_stater(r)!ray.serve._private.endpoint_stater*ray.serve._private.exceptionsr+ray.serve._private.grpc_utilr,ray.serve._private.http_utilr- ray.serve._private.logging_utilsr.r/r0r1ray.serve._private.long_pollr2r3$ray.serve._private.node_port_managerr4ray.serve._private.proxy_stater5#ray.serve._private.storage.kv_storer6ray.serve._private.usager7ray.serve._private.utilsr8r9r:r;ray.serve.configr<r=r>r?ray.serve.generated.serve_pb2r@rArBrCrDrrEray.serve.schemarFrGrHrIrJrKrLrMrNrOrPrQrRrSrTray.utilrU getLoggerr#_CRASH_AFTER_CHECKPOINT_PROBABILITYrfrrVr?rr@rArrgrgrgrhs (     , ,           D  > #