o `۷i6O@sJddlZddlZddlmZmZmZmZddlmZm Z GdddZ dS)N)DictListOptionalSet)InstanceInstanceUpdateEventc @seZdZUdZdZeededfed<e d2de de de j d e d e f d d Z e d e fd dZe de j d efddZe de j d efddZe de j d efddZde j d efddZe d2de de j d e d efddZe de de j d e fddZe de ded efdd Ze d ededffd!d"Ze d3de d#edd ed$fd%d&Ze d3de d#edd ed$fd'd(Ze d3de d#edd eefd)d*Zede j d edfd+d,Ze de d-ed e fd.d/Z ed0d1Z!dS)4 InstanceUtilzt A helper class to group updates and operations on an Instance object defined in instance_manager.proto NzInstance.InstanceStatus_reachable_from instance_id instance_typestatusdetailsreturncCs0t}d|_||_||_||_t||||S)a Returns a new instance with the given status. Args: instance_id: The instance id. instance_type: The instance type. status: The status of the new instance. details: The details of the status transition. r)rversionr r r r_record_status_transition)r r r rinstancer_/home/ubuntu/vllm_env/lib/python3.10/site-packages/ray/autoscaler/v2/instance_manager/common.py new_instanceszInstanceUtil.new_instancecCs ttS)z/ Returns a random instance id. )struuiduuid4rrrrrandom_instance_id-s zInstanceUtil.random_instance_idinstance_statusc Cs>|tjksJ|tjtjtjtjtjtjtjtj tj tj h vS)z Returns True if the instance is in a status where there could exist a cloud instance allocated by the cloud provider. ) rUNKNOWN ALLOCATEDRAY_INSTALLING RAY_RUNNING RAY_STOPPINGRAY_STOP_REQUESTED RAY_STOPPED TERMINATINGRAY_INSTALL_FAILEDTERMINATION_FAILEDALLOCATION_TIMEOUTrrrris_cloud_instance_allocated4sz(InstanceUtil.is_cloud_instance_allocatedcCs:|tjksJ|ttjvrdS|ttjvrdSdS)z Returns True if the instance is in a status where the ray process is running on the cloud instance. i.e. RAY_RUNNING, RAY_STOP_REQUESTED, RAY_STOPPING FT)rrrget_reachable_statusesrrr&rrris_ray_runningHszInstanceUtil.is_ray_runningcCs:|tjksJtjt|vrdS|ttjvrdSdS)z Returns True if the instance is in a status where the ray process is pending to be started on the cloud instance. FT)rrrrr(r&rrris_ray_pending[szInstanceUtil.is_ray_pendingcCstjt|vS)zt Returns True if the instance is in a status where it may transition to RAY_RUNNING status. )rrrr(r&rrris_ray_running_reachableosz%InstanceUtil.is_ray_running_reachablernew_instance_statuscCs.|t|jvr dS||_t|||dS)aBTransitions the instance to the new state. Args: instance: The instance to update. new_instance_status: The new status to transition to. details: The details of the transition. Returns: True if the status transition is successful, False otherwise. FT)rget_valid_transitionsr r)rr,rrrr set_statusxs zInstanceUtil.set_statuscCs$t}|jtj|||ddS)zRecords the status transition. Args: instance: The instance to update. status: The new status to transition to. )r timestamp_nsrN)timetime_nsstatus_historyappendr StatusHistory)rr rnow_nsrrrrs z&InstanceUtil._record_status_transition timeout_sc Csr|j}tj||d}t|dks%Jd|jdt|dtj|dt|d}t ||dkr7d Sd S) al Returns True if the instance has been in the current status for more than the timeout_seconds. Args: instance: The instance to check. timeout_seconds: The timeout in seconds. Returns: True if the instance has been in the current status for more than the timeout_s seconds. )select_instance_statusz instance z has  z statusgeAFT) r rget_status_transition_times_nslenr rInstanceStatusNamesortedr0r1)rr6 cur_statusstatus_times_nsstatus_time_nsrrr has_timeouts  zInstanceUtil.has_timeoutcCstjtjtjhtjtjtjtjhtjtjtjtjtj tj tj tjhtjtjtj tj tj tjhtjtj tj tj tj tjhtj tj tj tjtjhtjtj htj tj tj tjhtj tj tjhtj tjtjhtjtj htjttjttj tj tjhtjtiSN)rQUEUED REQUESTED TERMINATEDrALLOCATION_FAILEDrrr%rr!r"r#r r$setrrrrrr-s     z"InstanceUtil.get_valid_transitionsr7zInstance.StatusHistorycCs.g}|jD]}|r|j|krq||q|S)aH Returns the status history of the instance. Args: instance: The instance. select_instance_status: The go-to status to search for, i.e. select only status history when the instance transitions into the status. If None, returns all status updates. )r2rr3)rr7history status_updaterrrget_status_transitionshs   z#InstanceUtil.get_status_transitionscCs,t||}|jddd|r|dSdS)z Returns the last status transition of the instance. Args: instance: The instance. instance_status: The status to search for. If None, returns the last status update. cSs|jSrDr/)xrrrsz9InstanceUtil.get_last_status_transition..)keyr:N)rrLsort)rr7rJrrrget_last_status_transitions z'InstanceUtil.get_last_status_transitioncCsddt||DS)aL Returns a list of timestamps of the instance status update. Args: instance: The instance. instance_status: The status to search for. If None, returns all status updates timestamps. Returns: The list of timestamps of the instance status updates. cSsg|]}|jqSrrM).0errr sz?InstanceUtil.get_status_transition_times_ns..)rrL)rr7rrrr;s z+InstanceUtil.get_status_transition_times_nscCs|jdur ||j|S)at Returns the set of instance status that is reachable from the given instance status following the status transitions. This method is memoized. Args: instance_status: The instance status to start from. Returns: The set of instance status that is reachable from the given instance status. N)r _compute_reachable)clsrrrrr(s  z#InstanceUtil.get_reachable_statusesupdatecCs|jr!dtj|jd|jd|jd|jd|jd|j Sdtj|j dtj|jd|jd|jd|jd|jd|j S) z3Returns a log string for the given instance update.z New instance z (id=z, type=z, cloud_instance_id=z , ray_id=z): zUpdate instance z->) upsertrr=r>r,r r cloud_instance_idnode_idrr )rrXrrrget_log_str_for_updates>   z#InstanceUtil.get_log_str_for_updatecsF|}fddi|_tjD]}t}||||j|<qdS)zo Computes and memorize the from status sets for each status machine with a DFS search. cs0||D]}||vr|||||q|S)z^ Regular DFS algorithm to find all reachable nodes from a given node. )add)graphstartvisited next_nodedfsrrrcs   z,InstanceUtil._compute_reachable..dfsN)r-r rr=valuesrI)rWvalid_transitionsr r`rrbrrVs  zInstanceUtil._compute_reachable)r rD)"__name__ __module__ __qualname____doc__r rrr__annotations__ staticmethodrrr=rrboolr'r)r*r+r.rintrCr-rrLrRr; classmethodr(rr\rVrrrrrs       '    r) r0rtypingrrrr'ray.core.generated.instance_manager_pb2rrrrrrrs