o bi<@sddlZddlZddlZddlZddlZddlmZddlmZddl m Z ddl m Z ddl mZddlmZddlmZdd lmZdd lmZeeZd Zd efd dZdedefddZdddZdS)N) ray_constants)setup_component_logger)get_node_ip_address)try_to_create_directory) GcsClient)AutoscalingConfigProducer)Monitor)KubeRayConfigReader)is_autoscaler_v2returncCstjtjjtjjj dS)Nlogs) ospathjoinray_commonutilsget_ray_temp_dir_privaterSESSION_LATESTrrb/home/ubuntu/.local/lib/python3.10/site-packages/ray/autoscaler/_private/kuberay/run_autoscaler.py _get_log_dirs  r cluster_namecluster_namespacecCst}|d} ztddd|dgtdWntjy2tdtd t tYnwq t t ||}t |}t d|d r\d d lm}||jt|t|d dSt|j||dddS)zFWait until the Ray head container is ready. Then start the autoscaler.z:6379Trz health-checkz --addressz--skip-version-checkz/The Ray head is ready. Starting the autoscaler.z/The Ray head is not ready. Will check again in z seconds.)fetch_from_server gcs_clientr)AutoscalerMonitor)address config_readerlog_dir monitor_ipF)rautoscaling_configr"retry_on_failureN)r subprocess check_callloggerinfoCalledProcessErrorwarning BACKOFF_Stimesleep_setup_loggingrrr ray.autoscaler.v2.monitorrrr rrunr)rrhead_ip ray_addressautoscaling_config_producerr MonitorV2rrrrun_kuberay_autoscalersT       r5cCs|t}t|ttjtj|tjtjtjdt tj }t }| t tj||t j|t j|dS)zLog to autoscaler log file (typically, /tmp/ray/session_latest/logs/monitor.*) Also log to pod stdout (logs viewable with `kubectl logs -c autoscaler`). ) logging_levellogging_formatr!filename max_bytes backup_countN)rrrr LOGGER_LEVEL LOGGER_FORMATMONITOR_LOG_FILE_NAMELOGGING_ROTATE_BYTESLOGGING_ROTATE_BACKUP_COUNTlogging getLevelNameupper_StderrHandler setFormatter FormattersetLevelroot addHandler)r!levelstderr_handlerrrrr.Ys    r.)r N)r@rr%r,r ray._privaterray._private.ray_loggingrray._private.servicesrray._common.utilsr ray._rayletr2ray.autoscaler._private.kuberay.autoscaling_configrray.autoscaler._private.monitorr)ray.autoscaler.v2.instance_manager.configr ray.autoscaler.v2.utilsr getLogger__name__r'r+strrr5r.rrrrs&          ;