o $i@sddlZddlmZddlmZmZmZddlmZddl m Z ddl m Z m Z mZddlmZddlmZmZeeZGd d d ZdS) N) defaultdict)DictListOptional)EventLoggerAdapter)ResourceRequestUtil)ClusterResourceConstraintGangResourceRequestResourceRequest)LabelSelectorOperator) LaunchRequestTerminationRequestc@seZdZdZdefddZ     ddeeefde e e de e e d e e e d e e ed e e ed dfd dZdS)AutoscalerEventLoggerz Logs events related to the autoscaler. # TODO: - Add more logging for other events. - Rate limit the events if too spammy. loggercCs ||_dS)N)_logger)selfrr[/home/ubuntu/veenaModal/venv/lib/python3.10/site-packages/ray/autoscaler/v2/event_logger.py__init__s zAutoscalerEventLogger.__init__Ncluster_resourceslaunch_requeststerminate_requestsinfeasible_requestsinfeasible_gang_requests'infeasible_cluster_resource_constraintsreturnc Cs|r8tt}|D] }||j|j7<qt|D]\} \} } d| d| d} |j| t| q|rtt} |D]}| |j |jfd7<q@t j j dt j j dt j jdt j jdi}t| D]#\} \\}} } d | d | d ||d } |j| t| qg|s|r|d d}dt|d} d|vr| dt|dd7} d|vr| dt|dd7} |j| d|jdt|d|rKt|}d} t|D]`\} }t|j}| |d|j7} | t|dkr| d7} |jjr@g}|jjD]%}|jD]}t|j}d|j}||j d|d|dqq |r@| dd|d7} q| d7} |j!| |r|D]A}d j"|j#d!} t|j$}t|D]"\} }t|j}| |d|j7} | t|dkr| d7} qc| d7} |j!| qP|r|D]:}d"} t|j%D]#\}}t|j}| |d|j7} |t|j%dkr| d7} q| d7} |j!| qd#Sd#S)$az Log updates to the autoscaler scheduling state. Emits: - info logs for node launches and terminations (counts grouped by node type). - an info log summarizing the cluster size after a resize (CPUs/GPUs/TPUs). - warnings describing infeasible single resource requests, infeasible gang (placement group) requests, and infeasible cluster resource constraints. Args: cluster_resources: Mapping of resource name to total resources for the current cluster state. launch_requests: Node launch requests issued in this scheduling step. terminate_requests: Node termination requests issued in this scheduling step. infeasible_requests: Resource requests that could not be satisfied by any available node type. infeasible_gang_requests: Gang/placement group requests that could not be scheduled. infeasible_cluster_resource_constraints: Cluster-level resource constraints that could not be satisfied. Returns: None zAdding z node(s) of type .outdatedz"max number of worker nodes reachedz+max number of worker nodes per type reachedidlez Removing z nodes of type z (z).CPUrz Resized to z CPUsGPUz, z GPUsTPUz TPUszCurrent cluster resources: z6No available node types can fulfill resource requests *, z []z with label selectors: [z; z@. Add suitable node types to this cluster to resolve this issue.zQNo available node types can fulfill placement group requests (detail={details}): )detailsz8No available node types can fulfill cluster constraint: N)&rint instance_typecount enumerateitemsrinforcauser CauseOUTDATED MAX_NUM_NODESMAX_NUM_NODE_PER_TYPEIDLEgetdebugdictrgroup_by_countto_resource_maprequestlenlabel_selectorslabel_constraintsr Nameoperatorjoin label_valuesappend label_keywarningformatr'requestsresource_requests)rrrrrrrlaunch_type_countreqidxr)r*log_strtermination_by_causes_and_typecause_reason_mapr.num_cpusrequests_by_count req_count resource_map selector_strsselector constraintopvalues gang_requestinfeasible_constraintirrrlog_cluster_scheduling_updates&          z3AutoscalerEventLogger.log_cluster_scheduling_update)NNNNN)__name__ __module__ __qualname____doc__rrrstrfloatrrr r r r rrYrrrrrs2      r)logging collectionsrtypingrrrray._private.event.event_loggerrray.autoscaler.v2.utilsr!ray.core.generated.autoscaler_pb2rr r ray.core.generated.common_pb2r 'ray.core.generated.instance_manager_pb2r r getLoggerrZrrrrrrs