o `۷ir# @sddlZddlZddlZddlZddlZddlmZddlmZmZm Z m Z m Z m Z m Z mZmZmZddlZddlmZddlmZddlmZeeZedZded efd d Z d3d eegefee eefgeffdee eefde deed egeff ddZGddde eZ d4de!fddZ"ddZ#ddZ$deee efd eee effddZ%ej&d e e d e d!fd"d#Z'd$e(d efd%d&Z)d'ed efd(d)Z* d5d*e+d+e,d efd,d-Z-d e!fd.d/Z.d4d0e!d efd1d2Z/dS)6N)datetime) AnyCallableContextManagerDict GeneratorGenericListOptionalTypeVarUnion)count_required_parameters)UserExceptionWithTraceback) ObjectRefTbundlereturncCs<|}|dd|dd|ddd}|r||d<|S)zConvert a bundle of resources to Ray actor/task arguments. >>> bundle_to_remote_args({"GPU": 1, "memory": 1, "custom": 0.1}) {'num_cpus': 0, 'num_gpus': 1, 'memory': 1, 'resources': {'custom': 0.1}} CPUrGPUmemory)num_cpusnum_gpusr resources)copypop)rargsrQ/home/ubuntu/vllm_env/lib/python3.10/site-packages/ray/train/v2/_internal/util.pybundle_to_remote_argss   rtrain_loop_per_worker train_funcconfigtrain_func_context fn_arg_namecsrt}|dkr|d|d}t||dkr+pitfdd}|Stfdd}|S)aValidates and constructs the training function to execute. Args: train_func: The training function to execute. This can either take in no arguments or a ``config`` dict. config (Optional[Dict]): Configurations to pass into ``train_func``. If None then an empty Dict will be created. train_func_context: Context manager for user's `train_func`, which executes backend-specific logic before and after the training function. fn_arg_name (Optional[str]): The name of training function to use for error messages. Returns: A valid training function. Raises: ValueError: if the input ``train_func`` is invalid. z: should take in 0 or 1 required arguments, but it accepts z required arguments instead.cs2 WdS1swYdSNrrr!r r"rrtrain_fnQs$z&construct_train_func..train_fncs0 WdS1swYdSr%rr)r r"rrr'Xs$)r ValueError functoolswraps)r r!r"r#num_required_paramserr_msgr'rr&rconstruct_train_func0s r-c@s,eZdZdZdefddZdefddZdS) ObjectRefWrapperz>Thin wrapper around ray.put to manually control dereferencing.objcCst||_dSr%)rayput_ref)selfr/rrr__init__cszObjectRefWrapper.__init__rcCs t|jSr%)r0getr2)r3rrrr5fs zObjectRefWrapper.getN)__name__ __module__ __qualname____doc__rr4r5rrrrr.`sr.F include_mscCsd}|r|d7}t|S)Nz%Y-%m-%d_%H-%M-%Sz.%f)rtodaystrftime)r:patternrrrdate_strjsr>cCstSr%)time monotonicrrrrtime_monotonicqsrAcsfdd}|S)Ncs j|_|Sr%)r9)func copy_funcrrwrappedvsz_copy_doc..wrappedr)rDrErrCr _copy_docus rF object_refscst|t}|r |n|g}|}i|r0tj|dd\}}|r.t|t|D]\}}||<q%|stt|ks:Jfdd|D}|rG|S|dS)aThis is a safe version of `ray.get` that raises an exception immediately if an input task dies, while the others are still running. TODO(ml-team, core-team): This is NOT a long-term solution, and we should not maintain this function indefinitely. This is a mitigation for a Ray Core bug, and should be removed when that is fixed. See here: https://github.com/ray-project/ray/issues/47204 Args: object_refs: A single or list of object refs to wait on. Returns: task_outputs: The outputs of the tasks. Raises: `RayTaskError`/`RayActorError`: if any of the tasks encounter a runtime error or fail due to actor/task death (ex: node failure). r$) num_returnscsg|]}|qSrr).0tasktask_to_outputrr sz ray_get_safe..r) isinstancelistr0waitzipr5len)rGis_listunreadyreadyrJ task_outputordered_outputsrrKr ray_get_safe}s  rXcontext_managers)NNNccsLt}|D]}||qdVWddS1swYdS)z Utility to invoke a list of context managers and yield sequentially. Args: context_managers: List of context managers to invoke. N) contextlib ExitStack enter_context)rYstackcontext_managerrrrinvoke_context_managerss  "r_r/cCs|jd|jS)zReturns the full module name of the given object, including its qualified name. Args: obj: The object (class, function, etc.) whose module name is required. Returns: Full module and qualified name as a string. .)r7r8)r/rrrget_module_names rafncCs.t|tjr t|jSt|dr|jS|jjS)aReturns a readable name for any callable. Examples: >>> get_callable_name(lambda x: x) '' >>> def foo(a, b): pass >>> get_callable_name(foo) 'foo' >>> from functools import partial >>> bar = partial(partial(foo, a=1), b=2) >>> get_callable_name(bar) 'foo' >>> class Dummy: ... def __call__(self, a, b): pass >>> get_callable_name(Dummy()) 'Dummy' r6)rNr)partialget_callable_namerBhasattrr6 __class__rbrrrrds   rdeexclude_framescCs:tjtt|j| d}td|t||dS)a@Construct a UserExceptionWithTraceback from a base exception. Args: e: The base exception to construct a UserExceptionWithTraceback from. exclude_frames: The number of frames to exclude from the beginnning of the traceback. Returns: A UserExceptionWithTraceback object. )limitzError in training function: ) traceback_str) traceback format_excrR extract_tb __traceback__loggererrorr)rhriexc_traceback_strrrr'construct_user_exception_with_tracebacks  rscCs.ddlm}z|WdStyYdSw)z6Check if the current process is a Ray Train V2 worker.rget_train_fn_utilsTF)/ray.train.v2._internal.execution.train_fn_utilsru RuntimeErrorrtrrr_in_ray_train_workers  rxraise_in_tune_sessioncsdtdtffdd }|S)aCheck that the caller is a Ray Train worker spawned by Ray Train, with access to training function utilities. Args: raise_in_tune_session: Whether to raise a specific error message if the caller is in a Tune session. If True, will raise a DeprecationWarning. Returns: A decorator that performs this check, which raises an error if the caller is not a Ray Train worker. rbrcstfdd}|S)NcsNddlm}r|rtdjdts tdjd|i|S)Nr)_in_tune_sessionz `ray.train.z` is deprecated when running in a function passed to Ray Tune. Please use the equivalent `ray.tune` API instead. See this issue for more context: https://github.com/ray-project/ray/issues/49454`z` cannot be used outside of a Ray Train training function. You are calling this API from the driver or another non-training process. These utilities are only available within a function launched by `trainer.fit()`.)%ray.tune.trainable.trainable_fn_utilsrzDeprecationWarningr6rxrw)rkwargsrz)rbryrr _wrapped_fn s    z9requires_train_worker.._wrap.._wrapped_fn)r)r*)rbrryrgr_wrap sz$requires_train_worker.._wrap)r)ryrrrrrequires_train_workers r)r)F)r)0rZr)loggingr?rlrtypingrrrrrrr r r r r0ray.train._internal.utilsr !ray.train.v2._internal.exceptionsr ray.typesr getLoggerr6rprdictrstrr-r.boolr>rArFrXcontextmanagerr_objectrard BaseExceptionintrsrxrrrrrsj 0    $  0  &