o $iu @s UddlmZddlmZddlZddlmZddlmZm Z m Z m Z m Z de fddZed Zd Zd Zd Zd ZdZeeeehZdZdZdZdZeed<dZdedZdefddZ dZ!dZ"dZ#dZ$dZ%dZ&dZ'd Z(d!Z)d"Z*d#Z+d$Z,d%Z-d&Z.d%Z/e!e"e$e&e'e(e)e*e+e,e.h Z0d'Z1d(Z2dS)))Path)AnyN)env_bool)0COPY_DIRECTORY_CHECKPOINTS_INSTEAD_OF_MOVING_ENVEVALUATION_DATASET_KEY MODEL_KEYPREPROCESSOR_KEYTRAIN_DATASET_KEYreturncCs(tsJdttjjjdS)Nz/Ray must be initialized to get the session dir. artifacts)rayis_initializedr_privateworker _global_nodeget_session_dir_pathas_posixrrP/home/ubuntu/veenaModal/venv/lib/python3.10/site-packages/ray/train/constants.py_get_ray_train_session_dirs rz ~/ray_resultscheckpoint_dir_name _time_total_s _hostname_node_ip_pidz results.json checkpoints_current_checkpoint_id DEPRECATED_DEPRECATED_VALUE&RAY_TRAIN_ENABLE_V2_MIGRATION_WARNINGSzSee this issue for more context and migration options: https://github.com/ray-project/ray/issues/49454. Disable these warnings by setting the environment variable: z=0cCs ttdS)NT)r$ENABLE_V2_MIGRATION_WARNINGS_ENV_VARrrrr_v2_migration_warnings_enabledDs r!/TRAIN_RESULT_ENABLE_DETAILED_AUTOFILLED_METRICS'TRAIN_ENABLE_SHARE_CUDA_VISIBLE_DEVICES&TRAIN_ENABLE_SHARE_HIP_VISIBLE_DEVICES+TRAIN_ENABLE_SHARE_NEURON_CORES_ACCELERATOR,TRAIN_ENABLE_SHARE_ASCEND_RT_VISIBLE_DEVICESTRAIN_PLACEMENT_GROUP_TIMEOUT_STRAIN_ENABLE_WORKER_SPREADRAY_CHDIR_TO_TRIAL_DIR%RAY_TRAIN_COUNT_PREEMPTION_AS_FAILURERAY_TRAIN_ENABLE_STATE_TRACKING*TUNE_ONLY_STORE_CHECKPOINT_SCORE_ATTRIBUTE&TORCH_PROCESS_GROUP_SHUTDOWN_TIMEOUT_S"JAX_DISTRIBUTED_SHUTDOWN_TIMEOUT_Scheckpoint_metadatacheckpoint_rank)3pathlibrtypingrr ray._private.ray_constantsrray.air.constantsrrrrr strr expanduserrDEFAULT_STORAGE_PATHCHECKPOINT_DIR_NAME TIME_TOTAL_SWORKER_HOSTNAMEWORKER_NODE_IP WORKER_PIDDETAILED_AUTOFILLED_KEYSRESULT_FILE_JSONTRAIN_CHECKPOINT_SUBDIRTUNE_CHECKPOINT_IDr__annotations__r V2_MIGRATION_GUIDE_MESSAGEboolr!&ENABLE_DETAILED_AUTOFILLED_METRICS_ENV%ENABLE_SHARE_CUDA_VISIBLE_DEVICES_ENV$ENABLE_SHARE_HIP_VISIBLE_DEVICES_ENV)ENABLE_SHARE_NEURON_CORES_ACCELERATOR_ENV'ENABLE_SHARE_NPU_RT_VISIBLE_DEVICES_ENV#TRAIN_PLACEMENT_GROUP_TIMEOUT_S_ENVTRAIN_ENABLE_WORKER_SPREAD_ENVr)r*r+r,r-.DEFAULT_TORCH_PROCESS_GROUP_SHUTDOWN_TIMEOUT_Sr/*DEFAULT_JAX_DISTRIBUTED_SHUTDOWN_TIMEOUT_STRAIN_ENV_VARSCHECKPOINT_METADATA_KEYCHECKPOINT_RANK_KEYrrrrsn