o `۷i@@sddlZddlZddlmZddlmZmZmZddlZddl m Z m Z er,ddl m Z e ddGdd d eZe d dGd d d Ze Gd ddeZe GdddeZdS)N)Enum) TYPE_CHECKINGOptionalTuple) DeveloperAPI PublicAPI)PathPartitionFilteralpha) stabilityc@seZdZdZdZ dZdS)CheckpointBackenda(Supported backends for storing and reading checkpoint files. Currently, only one type of backend is supported: * Batch-based backends: CLOUD_OBJECT_STORAGE and FILE_STORAGE. Their differences are as follows: 1. Writing checkpoints: Batch-based backends write a checkpoint file for each block. 2. Loading checkpoints and filtering input data: Batch-based backends load all checkpoint data into memory prior to dataset execution. The checkpoint data is then passed to each read task to perform filtering. CLOUD_OBJECT_STORAGE FILE_STORAGEN)__name__ __module__ __qualname____doc__r r rrT/home/ubuntu/vllm_env/lib/python3.10/site-packages/ray/data/checkpoint/interfaces.pyr s r betac@seZdZdZdZdZ  dddddddddeed eed ed ed d ee de de dedfddZ defddZ  dd ed ed d ee de e d ffddZdS)CheckpointConfigaConfiguration for checkpointing. Args: id_column: Name of the ID column in the input dataset. ID values must be unique across all rows in the dataset and must persist during all operators. checkpoint_path: Path to store the checkpoint data. It can be a path to a cloud object storage (e.g. `s3://bucket/path`) or a file system path. If the latter, the path must be a network-mounted file system (e.g. `/mnt/cluster_storage/`) that is accessible to the entire cluster. If not set, defaults to `RAY_DATA_CHECKPOINT_PATH_BUCKET/ray_data_checkpoint`. delete_checkpoint_on_success: If true, automatically delete checkpoint data when the dataset execution succeeds. Only supported for batch-based backend currently. override_filesystem: Override the :class:`pyarrow.fs.FileSystem` object used to read/write checkpoint data. Use this when you want to use custom credentials. override_backend: Override the :class:`CheckpointBackend` object used to access the checkpoint backend storage. filter_num_threads: Number of threads used to filter checkpointed rows. write_num_threads: Number of threads used to write checkpoint files for completed rows. checkpoint_path_partition_filter: Filter for checkpoint files to load during restoration when reading from `checkpoint_path`. RAY_DATA_CHECKPOINT_PATH_BUCKETray_data_checkpointNT)delete_checkpoint_on_successoverride_filesystemoverride_backendfilter_num_threadswrite_num_threads checkpoint_path_partition_filter id_columncheckpoint_pathrrzpyarrow.fs.FileSystemrrrrrc Cs||_t|jtrt|jdkrtd|j|dur$tjdtdd|p)||_ | |j ||\} } | |_ | |_ ||_ ||_||_||_dS)Nrz9Checkpoint ID column must be a non-empty string, but got zD`override_backend` is deprecated and will be removed in August 2025.) stacklevel)r isinstancestrlenInvalidCheckpointingConfigwarningswarn FutureWarning_get_default_checkpoint_pathr _infer_backend_and_fs filesystembackendrrrr) selfrr rrrrrrinferred_backend inferred_fsrrr__init__Ks4   zCheckpointConfig.__init__returncCs8tj|j}|durtd|jd|d|jS)N`zO` env var is not set, please explicitly set `CheckpointConfig.checkpoint_path`./)osenvironget&DEFAULT_CHECKPOINT_PATH_BUCKET_ENV_VARr&DEFAULT_CHECKPOINT_PATH_DIR)r.artifact_storagerrrr*us  z-CheckpointConfig._get_default_checkpoint_pathc CszH|durt|tjjsJdt||}n tjj|\}}|dur6t|ts3Jdt||}nt|tjjrAtj}ntj }||fWSt y]}z t d|d|d}~ww)NzLoverride_filesystem must be an instance of `pyarrow.fs.FileSystem`, but got zEoverride_backend must be an instance of `CheckpointBackend`, but got zInvalid checkpoint path: z. ) r#pyarrowfs FileSystemtypefrom_urir LocalFileSystemr r Exceptionr&)r.r rrr<_r-errrr+~s8   z&CheckpointConfig._infer_backend_and_fs)NN)rrrrr8r9rr$boolr intr1r*rr+rrrrr-sT  *  rc@eZdZdZdS)r&zNException which indicates that the checkpointing configuration is invalid.Nrrrrrrrrr&r&c@rF)InvalidCheckpointingOperatorszxException which indicates that the DAG is not eligible for checkpointing, due to one or more incompatible operators.NrGrrrrrIrHrI)r5r'enumrtypingrrrr;ray.util.annotationsrrray.data.datasourcerr rrAr&rIrrrrs   s