o biB @sddlZddlZddlZddlZddlmZddlmZmZm Z ddl Z ddl m Z m Z mZmZddlmZmZddlmZeeZdZdZd Zd Zd Zd Zd Zd ZdZ dZ!dZ"GdddZ#dee$efdee$effddZ%de$dee$effddZ&dee$effddZ'dee$efdee$effddZ(dee$efd e)dee$effd!d"Z*dee$efd e)dee$e+ffd#d$Z,d%ee$e$fd&ee$ee$e$ffd'e$de+fd(d)Z-d%ee$e$fd&ee$ee$e$ffde e+fd*d+Z.d%ee$e$fd&ee$ee$e$ffd'e$de e+fd,d-Z/d.ee$e+fd&ee$ee$e$ffde e+fd/d0Z0d1e$de+fd2d3Z1d%ee$efd'e$dee$e+ffd4d5Z2dS)6N)chain)AnyDictOptional)DISABLE_LAUNCH_CONFIG_CHECK_KEYDISABLE_NODE_UPDATERS_KEYFOREGROUND_NODE_LAUNCH_KEYWORKER_LIVENESS_CHECK_KEY) node_providerutils)validate_configautoscalerOptionsidleTimeoutSeconds upscalingMode AggressiveDefault Conservativez!cloud.google.com/gke-tpu-topologyz$cloud.google.com/gke-tpu-accelerator headgroupc@s6eZdZdZddZddZdeeeffddZ d S) AutoscalingConfigProduceraProduces an autoscaling config by reading data from the RayCluster CR. Used to fetch the autoscaling config at the beginning of each autoscaler iteration. In the context of Ray deployment on Kubernetes, the autoscaling config is an internal interface. The autoscaling config carries the strict subset of RayCluster CR data required by the autoscaler to make scaling decisions; in particular, the autoscaling config does not carry pod configuration data. This class is the only public object in this file. cCstj|d|_d||_dS)N) namespacez rayclusters/)r KubernetesHttpApiClientkubernetes_api_client _ray_cr_path)selfray_cluster_nameray_cluster_namespacerf/home/ubuntu/.local/lib/python3.10/site-packages/ray/autoscaler/_private/kuberay/autoscaling_config.py__init__6sz"AutoscalingConfigProducer.__init__cCs|}t|}|S)N)#_fetch_ray_cr_from_k8s_with_retries&_derive_autoscaling_config_from_ray_cr)rray_crautoscaling_configrrr__call__<sz"AutoscalingConfigProducer.__call__returnc CsvtdtdD]1}z |j|jWStjy8}z|tkr+tdt t n|dWYd}~qd}~wwt )zFetch the RayCluster CR by querying the K8s API server. Retry on HTTPError for robustness, in particular to protect autoscaler initialization. z1Failed to fetch RayCluster CR from K8s. Retrying.N) rangeMAX_RAYCLUSTER_FETCH_TRIESrgetrrequests HTTPErrorlogger exceptiontimesleepRAYCLUSTER_FETCH_RETRY_SAssertionError)rierrrr As   z=AutoscalingConfigProducer._fetch_ray_cr_from_k8s_with_retriesN) __name__ __module__ __qualname____doc__rr$rstrrr rrrrr's rr"r%c Cst|dd}t|d}tdd|D}t}|dti}t|vr/|td}nd}|tt kr;d}n|tt krEd }n |tt krOd }nd }||dd t ||||d |}t ||S) Nmetadatarspeccss|]}|dVqdS) max_workersNr).0 node_typerrr `s z9_derive_autoscaling_config_from_ray_cr..gN@g?r&iname)provider cluster_namehead_node_typeavailable_node_typesr;idle_timeout_minutesupscaling_speed)_generate_provider_config/_generate_available_node_types_from_ray_cr_specsumvalues*_generate_legacy_autoscaling_config_fieldsr)AUTOSCALER_OPTIONS_KEYIDLE_SECONDS_KEY UPSCALING_KEYUPSCALING_VALUE_CONSERVATIVEUPSCALING_VALUE_DEFAULTUPSCALING_VALUE_AGGRESSIVE_HEAD_GROUP_NAMEr ) r"provider_configrCglobal_max_workerslegacy_autoscaling_fieldsautoscaler_optionsrDrEr#rrrr!Ws>  r!rc Csddd|tdtdtdtdiS)zGenerates the `provider` field of the autoscaling config, which carries data required to instantiate the KubeRay node provider. typekuberayrTF)rrrr )rrrrrFsrFc Csigdggggggid S)zEGenerates legacy autoscaling config fields required for compatibiliy.F) file_mountscluster_synced_filesfile_mounts_sync_continuouslyinitialization_commandssetup_commandshead_setup_commandsworker_setup_commandshead_start_ray_commandsworker_start_ray_commandsauthrrrrrrJsrJ ray_cr_speccCs*|d}tt|ddidd|dDS)z[Formats autoscaler "available_node_types" field based on the Ray CR's group specs. headGroupSpecTis_headcSsi|] }|dt|ddqS) groupNameFrd)_node_type_from_group_spec)r<worker_group_specrrr s zC_generate_available_node_types_from_ray_cr_spec..workerGroupSpecs)rQrg)rbrcrrrrGs rG group_specrecCsp|rd}}n|d|dd}|d|dd}t||}||i|d}|t}|dur6t||d<|S) z/Converts CR group spec to autoscaler node type.r minReplicas numOfHostsr& maxReplicas) min_workersr; node_config resourcesNidle_timeout_s)r)"_get_ray_resources_from_group_specrLfloat)rkreror;rqr=rrrrrrgs    rgcCsj|di}|dddddi}|rtn|d}t|||}t|||}t||}t||}t||} i} t|ts@J|| d<|d urL|| d <|d urd |vrX|| d < d |ddvrt |ddd vrt |ddd vr|ddd t } |ddd t } t | | } | rd | d| d<n t dt dt d| d ur| | d<| || S)a' Infers Ray resources from rayStartCommands and K8s limits. The resources extracted are used in autoscaling calculations. TODO: Expose a better interface in the RayCluster CRD for Ray resource annotations. For now, we take the rayStartParams as the primary source of truth. rayStartParamstemplater: containersrrqrfCPUNGPUTPU nodeSelectorr&zTPU-z-headzPods using TPUs require both `z` and `z` node selectors. See https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/tpu.html#configuring-ray-pods-for-tpu-usage and https://cloud.google.com/kubernetes-engine/docs/how-to/tpus.memory)r)rQ _get_num_cpus _get_num_gpus_get_custom_resources _get_num_tpus _get_memory isinstanceintGKE_TPU_TOPOLOGY_LABELGKE_TPU_ACCELERATOR_LABELr tpu_node_selectors_to_typer,errorupdate)rkreray_start_params k8s_resources group_namenum_cpusnum_gpuscustom_resource_dictnum_tpusr|rqtopology acceleratoraccelerator_typerrrrssN         rsrrrcCsld|vr t|dSd|divr|dd}t|Sd|divr.|dd}t|Std|d)zgGet CPU annotation from ray_start_params or k8s_resources, with priority for ray_start_params. znum-cpuscpulimitsr*z6Autoscaler failed to detect `CPU` resources for group zY. Set the `--num-cpus` rayStartParam and/or the CPU resource limit for the Ray container.)rr)_round_up_k8s_quantity ValueError)rrr cpu_quantityrrrr},s    r}cCs`d|vr t|dSd|divr|dd}t|Sd|divr.|dd}t|SdS)sGet memory resource annotation from ray_start_params or k8s_resources, with priority for ray_start_params. r|rr*N)rr)r)rrmemory_quantityrrrrEs   rcCsfd|vr t|dSt|di|diD]\}}|dr0t|}|dkr0|SqdS)rznum-gpusrr*gpurN)rrr)itemsendswithr)rrrkeyresource_quantityrrrrr~Vs  r~rcCsPd|vr|dSdD]}||id}|dur%t|}|dkr%|Sq dS)zGet TPU custom resource annotation from custom_resource_dict in ray_start_params, or k8s_resources, with priority for custom_resource_dict. rz)rr*zgoogle.com/tpuNr)r)r)rrtyptpu_resource_quantityrrrrrssrquantitycCs t|}|jtjd}t|S)zRounds a Kubernetes resource quantity up to the nearest integer. Args: quantity: Resource quantity as a string in the canonical K8s form. Returns: The quantity, rounded up, as an integer. )rounding)r parse_quantityto_integral_valuedecimalROUND_UPr)rresource_decimalroundedrrrrs rc Csd|vriS|d}z0|dddd}t|}t|ts!J|D]\}}t|ts0Jt|ts7Jq%W|StyQ}z t d|d|d}~ww) a?Format custom resources based on the `resources` Ray start param. Currently, the value of the `resources` field must be formatted as follows: '"{"Custom1": 1, "Custom2": 5}"'. This method first converts the input to a correctly formatted json string and then loads that json string to a dict. rqr&\z1Error reading `resource` rayStartParam for group z. For the correct format, refer to example configuration at https://github.com/ray-project/ray/blob/master/python/ray/autoscaler/kuberay/ray-cluster.complete.yaml.N) replacejsonloadsrdictrr8r Exceptionr,r)rrresources_stringresources_jsonrqrvaluer3rrrrs&    r)3rrloggingr. itertoolsrtypingrrrr*!ray.autoscaler._private.constantsrrrr ray.autoscaler._private.kuberayr r ray.autoscaler._private.utilr getLoggerr4r,rKrLrMrPrOrNr(r0rrrQrr8r!rFrJrGboolrgrrsr}rr~rrrrrrrs   "07         O