o ci @sddlZddlZddlZddlmZddlmZmZmZmZm Z m Z m Z m Z m Z mZddlZddlmZddlmZe dZdedefd d Z d)d eegefeeeefgeffd e eeefdede edegeff ddZGddde eZd*defddZddZddZdeee efdeee effddZejde ede d fd!d"Z d#e!defd$d%Z"d&edefd'd(Z#dS)+N)datetime) AnyCallableContextManagerDict GeneratorGenericListOptionalTypeVarUnion)count_required_parameters) ObjectRefTbundlereturncCs<|}|dd|dd|ddd}|r||d<|S)zConvert a bundle of resources to Ray actor/task arguments. >>> bundle_to_remote_args({"GPU": 1, "memory": 1, "custom": 0.1}) {'num_cpus': 0, 'num_gpus': 1, 'memory': 1, 'resources': {'custom': 0.1}} CPUrGPUmemory)num_cpusnum_gpusr resources)copypop)rargsrO/home/ubuntu/.local/lib/python3.10/site-packages/ray/train/v2/_internal/util.pybundle_to_remote_argss   rtrain_loop_per_worker train_funcconfigtrain_func_context fn_arg_namecsrt}|dkr|d|d}t||dkr+pitfdd}|Stfdd}|S)aValidates and constructs the training function to execute. Args: train_func: The training function to execute. This can either take in no arguments or a ``config`` dict. config (Optional[Dict]): Configurations to pass into ``train_func``. If None then an empty Dict will be created. train_func_context: Context manager for user's `train_func`, which executes backend-specific logic before and after the training function. fn_arg_name (Optional[str]): The name of training function to use for error messages. Returns: A valid training function. Raises: ValueError: if the input ``train_func`` is invalid. z: should take in 0 or 1 required arguments, but it accepts z required arguments instead.cs2 WdS1swYdSNrrr rr!rrtrain_fnKs$z&construct_train_func..train_fncs0 WdS1swYdSr$rr)rr!rrr&Rs$)r ValueError functoolswraps)rr r!r"num_required_paramserr_msgr&rr%rconstruct_train_func*s r,c@s,eZdZdZdefddZdefddZdS) ObjectRefWrapperz>Thin wrapper around ray.put to manually control dereferencing.objcCst||_dSr$)rayput_ref)selfr.rrr__init__]szObjectRefWrapper.__init__rcCs t|jSr$)r/getr1)r2rrrr4`s zObjectRefWrapper.getN)__name__ __module__ __qualname____doc__rr3r4rrrrr-Zsr-F include_mscCsd}|r|d7}t|S)Nz%Y-%m-%d_%H-%M-%Sz.%f)rtodaystrftime)r9patternrrrdate_strdsr=cCstSr$)time monotonicrrrrtime_monotonicksr@csfdd}|S)Ncs j|_|Sr$)r8)func copy_funcrrwrappedpsz_copy_doc..wrappedr)rCrDrrBr _copy_docos rE object_refscst|t}|r |n|g}|}i|r0tj|dd\}}|r.t|t|D]\}}||<q%|stt|ks:Jfdd|D}|rG|S|dS)aThis is a safe version of `ray.get` that raises an exception immediately if an input task dies, while the others are still running. TODO(ml-team, core-team): This is NOT a long-term solution, and we should not maintain this function indefinitely. This is a mitigation for a Ray Core bug, and should be removed when that is fixed. See here: https://github.com/ray-project/ray/issues/47204 Args: object_refs: A single or list of object refs to wait on. Returns: task_outputs: The outputs of the tasks. Raises: `RayTaskError`/`RayActorError`: if any of the tasks encounter a runtime error or fail due to actor/task death (ex: node failure). r#) num_returnscsg|]}|qSrr).0tasktask_to_outputrr sz ray_get_safe..r) isinstancelistr/waitzipr4len)rFis_listunreadyreadyrI task_outputordered_outputsrrJr ray_get_safews  rWcontext_managers)NNNccsLt}|D]}||qdVWddS1swYdS)z Utility to invoke a list of context managers and yield sequentially. Args: context_managers: List of context managers to invoke. N) contextlib ExitStack enter_context)rXstackcontext_managerrrrinvoke_context_managerss  "r^r.cCs|jd|jS)zReturns the full module name of the given object, including its qualified name. Args: obj: The object (class, function, etc.) whose module name is required. Returns: Full module and qualified name as a string. .)r6r7)r.rrrget_module_names r`fncCs.t|tjr t|jSt|dr|jS|jjS)aReturns a readable name for any callable. Examples: >>> get_callable_name(lambda x: x) '' >>> def foo(a, b): pass >>> get_callable_name(foo) 'foo' >>> from functools import partial >>> bar = partial(partial(foo, a=1), b=2) >>> get_callable_name(bar) 'foo' >>> class Dummy: ... def __call__(self, a, b): pass >>> get_callable_name(Dummy()) 'Dummy' r5)rMr(partialget_callable_namerAhasattrr5 __class__)rarrrrcs   rc)r)F)$rYr(r>rtypingrrrrrrr r r r r/ray.train._internal.utilsr ray.typesrrdictrstrr,r-boolr=r@rErWcontextmanagerr^objectr`rcrrrrsN 0  $  0  &