o ci@sddlZddlmZmZddlmZmZmZmZm Z Gddde Z Gddde Z Gdd d e Z Gd d d e ZGd d d e ZGddde ZGddde ZGdddeZGddde ZdS)N)ListOptional)$DEFAULT_WORKER_GROUP_START_TIMEOUT_S%DEFAULT_WORKER_HEALTH_CHECK_TIMEOUT_S REPORT_BARRIER_TIMEOUT_S_ENV_VAR$WORKER_GROUP_START_TIMEOUT_S_ENV_VAR%WORKER_HEALTH_CHECK_TIMEOUT_S_ENV_VARc@eZdZdZdS) RayTrainErrorz(Base class for all Ray Train exceptions.N__name__ __module__ __qualname____doc__rrU/home/ubuntu/.local/lib/python3.10/site-packages/ray/train/v2/_internal/exceptions.pyr r cs eZdZdZfddZZS)WorkerHealthCheckTimeoutErrorzBException raised when a worker health check hangs for long enough.cs2ttt}|dtd|d7}t|dS)Nz Set the z> environment variable to increase the timeout (current value: z seconds).)osgetenvrrsuper__init__)selfmessagetimeout __class__rrrsz&WorkerHealthCheckTimeoutError.__init__)r r rrr __classcell__rrrrrsrcs6eZdZdZdeffdd ZddZddZZS) WorkerHealthCheckFailedErrorz2Exception raised when a worker health check fails.failurecst|||_||_dSN)rr_messagehealth_check_failure)rrrrrrr$s  z%WorkerHealthCheckFailedError.__init__cC|j|j|jffSr )rr!r"rrrr __reduce__)z'WorkerHealthCheckFailedError.__reduce__cCs|jdt|jS)N )r!strr"r$rrr__str__,sz$WorkerHealthCheckFailedError.__str__) r r rr Exceptionrr%r)rrrrrr!s rcs.eZdZdZdeffdd ZddZZS)WorkerGroupStartupTimeoutErrora1Exception raised when the worker group startup times out. Example scenario: 4 GPUs are detected in the cluster, but when the worker are actually scheduled, one of the nodes goes down and only 3 GPUs are available. One of the worker tasks may be stuck pending, until a timeout is reached. num_workersc s<ttjtt}||_td|d|dtddS)Nz)The worker group startup timed out after z seconds waiting for a workers. Potential causes include: (1) temporary insufficient cluster resources while waiting for autoscaling (ignore this warning in this case), (2) infeasible resource request where the provided `ScalingConfig` cannot be satisfied), and (3) transient network issues. Set the z. environment variable to increase the timeout.) floatrenvirongetrrr,rr)rr,rrrrr8sz'WorkerGroupStartupTimeoutError.__init__cCs|j|jffSr )rr,r$rrrr%Msz)WorkerGroupStartupTimeoutError.__reduce__)r r rrintrr%rrrrrr+0sr+c@r )WorkerGroupStartupFailedErrorzException raised when the worker group fails to start. Example scenario: A worker is scheduled onto a node that dies while the worker actor is initializing. Nr rrrrr1Qrr1c@r )$CheckpointManagerInitializationErroraException raised when the checkpoint manager fails to initialize from a snapshot. Example scenarios: 1. The checkpoint manager snapshot version is old and incompatible with the current version of Ray Train. 2. The checkpoint manager snapshot JSON file is corrupted. 3. The checkpoint manager snapshot references checkpoints that cannot be found in the run storage path. Nr rrrrr2Yrr2c@r )CollectiveTimeoutErrorzhException raised when an internal Ray Train collective operation of the worker group times out. Nr rrrrr3err3cs>eZdZdZdeedeedeffdd ZddZ Z S) BroadcastCollectiveTimeoutErroraException raised when the broadcast operation times out. There are two main timeout examples: 1. If not all workers call `ray.train.report`, the entire worker group will hang until the timeout before raising. This prevents indefinite worker group hangs. 2. If a worker is slow in the training loop and fails to reach the broadcast time, the collective will time out. time_elapsed missing_ranks timeout_sc sD||_||_||_d|dd|dtd|dd }t|dS)Nz(The broadcast operation timed out after z.2fzn seconds. Please make sure all worker ranks call `ray.train.report`. The following ranks have not called it: z# You can set this timeout with the z& environment variable (current value: z s).) _time_elapsed_missing_ranks _timeout_srrr)rr5r6r7rrrrrvs z(BroadcastCollectiveTimeoutError.__init__cCs|j|j|j|jffSr )rr8r9r:r$rrrr%sz*BroadcastCollectiveTimeoutError.__reduce__) r r rrrr-rr0rr%rrrrrr4ks r4c@s2eZdZdZdedefddZddZdd Zd S) UserExceptionWithTracebacka"This class wraps a user code exception raised on the worker with its original traceback string, for logging and debugging purposes. This is needed because the original exception traceback is not serialized with the exception when it is *returned* back to the main process. exc traceback_strcCs||_||_dSr ) _base_exc_traceback_str)rr<r=rrrrs z#UserExceptionWithTraceback.__init__cCr#r )rr>r?r$rrrr%r&z%UserExceptionWithTraceback.__reduce__cCs|jSr )r?r$rrrr)sz"UserExceptionWithTraceback.__str__N) r r rr BaseExceptionr(rr%r)rrrrr;s  r;)rtypingrr ray.train.v2._internal.constantsrrrrrr*r rrr+r1r2r3r4r;rrrrs ! "