o bi8@s&dZddlZddlZddlZddlZddlZddlZddlZddlZddl m Z ddl Z ddl Z ddl Z ddlmZddlZddlmZddZdd Zd d Zd d ZddZddZddZddZddZddZ d N)Path)storage)RAYcCstjdd}|jdddd|jdtdd d |jd td d d |jdgdddd|jdtddd |jdtdd|}|jdkrN|jdkrNJd|j|j |j |j |j|jfS)z Check command line arguments and return the cluster configuration file path, the number of retries, the number of expected nodes, and the value of the --no-config-cache flag. zLaunch and verify a Ray cluster) description--no-config-cache store_truez3Pass the --no-config-cache flag to Ray CLI commands)actionhelpz --retriesz;Number of retries for verifying Ray is running (default: 3))typedefaultr z--num-expected-nodesz9Number of nodes for verifying Ray is running (default: 1)z--docker-override)disablelatestnightlycommitrzAOverride the docker image used for the head node and worker nodes)choicesr r z--wheel-overridez:Override the wheel used for the head node and worker nodescluster_configz&Path to the cluster configuration file)r r z%Cannot override both docker and wheel) argparseArgumentParser add_argumentintstr parse_argsdocker_overridewheel_overriderretriesno_config_cachenum_expected_nodes)parserargsr"\/home/ubuntu/.local/lib/python3.10/site-packages/ray/autoscaler/launch_and_verify_cluster.pycheck_arguments sX r$cCsj|dkrdS|dkr dS|dkr3tdtjr&dtjdtjd d d Std tjtd d S)a  Get the docker image to use for the head node and worker nodes. Args: docker_override: The value of the --docker-override flag. Returns: The docker image to use for the head node and worker nodes, or None if not applicable. rzrayproject/ray:latest-py39rzrayproject/ray:nightly-py39rz^[0-9]+.[0-9]+.[0-9]+$zrayproject/ray:.Nz-py39zGError: docker image is only available for release version, but we get: r )rematchray __version__ __commit__printsysexit)rr"r"r#get_docker_imageWs  r/cCs6|r t|tjstd|tddSdS)z Check if the provided file path is valid and readable. Args: file_path: The path of the file to check. Raises: SystemExit: If the file is not readable or does not exist. z/Error: Cannot read cluster configuration file: r N)is_fileosaccessR_OKr,r-r.) file_pathr"r"r# check_filers r5cCs*|dg}|d|d||d<dS)Nsetup_commandsz9pip3 uninstall -y ray && pip3 install -U "ray[default] @ ")getappend) config_yaml wheel_urlr6r"r"r#override_wheels_urls   r<cCsT|di}||d<d|d<|ddusJd|ddus$Jd||d<dS) Ndockerimage ray_containercontainer_name head_imagezCannot override head_image worker_imagezCannot override worker_image)r8)r: docker_image docker_configr"r"r#override_docker_images  rEcCs|tdtdtjddd}d}d}tjd|}tjtj|s/ttj|| |||t |d d S) z=Download the ssh key from the S3 bucket to the local machine.&======================================zDownloading ssh key...s3 us-west-2 region_namezaws-cluster-launcher-testzray-autoscaler_59_us-west-2.pem~/.ssh/N) r,boto3clientr1path expanduserexistsdirnamemakedirs download_filechmod) s3_client bucket_namekey_namelocal_key_pathr"r"r#download_ssh_key_awssrZcCstdtdt}d}d}||}||}tjd|}tjtj |s6t tj || |t |ddS)zGDownload the ssh key from the google cloud bucket to the local machine.rFzDownloading ssh key from GCP...z*gcp-cluster-launcher-release-test-ssh-keyszAray-autoscaler_gcp_us-west1_anyscale-bridge-cd812d38_ubuntu_0.pemrKrLN) r,rClient get_bucketget_blobr1rOrPrQrRrSdownload_to_filenamerU)rNrWrXbucketblobrYr"r"r#download_ssh_key_gcps   rac Cstdtdd}d}t|D]]}ztjddddt|gd d d t|WdStjym}z7td |d d |dt|jdt td|j dtd|j d|}WYd}~qd}~ww|)z Clean up the cluster using the given cluster configuration file. Args: cluster_config: The path of the cluster configuration file. rFzCleaning up cluster...Nr r)down-v-yTcheckcapture_outputzray down fails[r /z]: utf-8stdout: stderr: ) r,range subprocessrunrcleanup_security_groupsCalledProcessErroroutputdecode traceback print_excstdoutstderr)r:r last_error num_triesier"r"r#cleanup_clusters.  r{c Csd}|dkrYz |j|dWdStjjyR}z5|jdddkr:d|}td|d |t||d 7}ntd |WYd}~dSWYd}~nd}~ww|dksdSdS) Nr )GroupIdErrorCodeDependencyViolationzWaiting zGs for the instance to be terminated before deleting the security group r zError deleting security group: )delete_security_groupbotocore exceptions ClientErrorresponser,timesleep) ec2_clientidretryrz sleep_timer"r"r#cleanup_security_groups&    rc Cs|did}|dkrdSz*tjddd}|jdtgdd |d gdgd }|d D] }t||d q-WdStyS}ztd|WYd}~dSd}~ww)Nproviderr awsec2rHrIztag-key)NameValuesztag:ray-cluster-name cluster_name)FiltersSecurityGroupsr}z#Error cleaning up security groups: )r8rMrNdescribe_security_groupsrr Exceptionr,)config provider_typerrsecurity_grouprzr"r"r#ros* ror c Cstdtdgd}|r|d|t|td|z tj|dddWn*tjyU}zt|jtd|j d td |j d |d }~wwtdtd d }d}||krz!dddt|dd|dg}|r}|dtj|ddd}Wn"tjy|d7}td|d|dt dYnw||ksf|stdtd|dt ||tdtdtdtdtdt ||tdtdd S) ak Run the necessary Ray commands to start a cluster, verify Ray is running, and clean up the cluster. Args: cluster_config: The path of the cluster configuration file. retries: The number of retries for the verification step. no_config_cache: Whether to pass the --no-config-cache flag to the ray CLI commands. rFzStarting new cluster...)r)uprcrdr TrerjrirkNzVerifying Ray is running...Frr)execrcz2python -c 'import ray; ray.init("localhost:6379");z assert len(ray.nodes()) >= ')rfr z#Verification failed. Retry attempt z of z...<z!Error: Verification failed after z0 attempts. Cleaning up cluster before exiting...zExiting script.zRay verification successful.z'Finished executing script successfully.)r,r9rjoinrmrnrprqrurrrvrrr{r-r.) r:rrrrcmdrzsuccesscountr"r"r#run_ray_commandssn         r__main__z"Using cluster configuration file: z4Number of retries for 'verify ray is running' step: zUsing --no-config-cache flag: z6Number of expected nodes for 'verify ray is running': r-rFzOverriding ray wheel...: zOverriding docker image...: zUsing docker image: rr Fcache_stopped_nodesrgcp)gcloudr get-valueaccountT)rurfrizActive account email:)rrrprojectzInjecting GCP project 'z$' into cluster configuration file... project_idvspherezVSPHERE provider detected.z-Provider type not recognized. Exiting script.z.yaml)suffix)r )=__doc__rr1r'rmr-tempfilerrspathlibrrMryaml google.cloudrr)"ray.autoscaler._private.aws.configrr$r/r5r<rErZrar{rror__name__rrrrrrr, safe_load read_textr:rrdocker_override_imager8rrnPIPErurrstrip account_emailrr.NamedTemporaryFiletempwritedumpencodeflushnamer"r"r"r#s    7 * O            $