o ci@sddlZddlZddlZddlZddlZddlmZddlmZm Z m Z m Z m Z m Z mZmZmZddlZddlmZddlmZmZmZddlmZddlmZedZeeZd e ed ee e e!ffd d Z"d ee#e$ffd dZ%deded efddZ&de e#effddZ'de d e$fddZ(  d%dee gefe e e#efgeffde e e#efde de e#de d e geff dd Z)Gd!d"d"ej*Z+Gd#d$d$Z,dS)&N)Path) AnyCallableContextManagerDictListOptionalTupleTypeVarUnion) ActorHandle)StartTracebackStartTracebackWithWorkerRankfind_free_port) RayActorError) ObjectRefT remote_valuesreturnc Cs|}t|dkr_t|\}}|D]E}zt|WqtyB}z||}td|dd|fWYd}~Sd}~wt yX}z ||}t |d|d}~wwt|dks dS)a$Check for actor failure when retrieving the remote values. Args: remote_values: List of object references from Ray actor methods. Returns: A tuple of (bool, Exception). The bool is True if evaluating all object references is successful, False otherwise. rzWorker z has failed.FN) worker_rank)TN) copylenraywaitgetrindexloggerinfo Exceptionr)r unfinishedfinished object_refexcfailed_actor_rankfailed_worker_rankr%M/home/ubuntu/.local/lib/python3.10/site-packages/ray/train/_internal/utils.pycheck_for_failure"s,    r'cCstj}t}||fS)z4Returns the IP address and a free port on this node.)rutilget_node_ip_addressr)addrportr%r%r&get_address_and_portHs r,path parent_pathcCs*|r |S||S)zConstructs a path relative to a parent. Args: path: A relative or absolute path. parent_path: A relative path or absolute path. Returns: An absolute path. ) expanduser is_absoluteresolvejoinpath)r-r.r%r%r&construct_pathPs r3env_varscCs"dd|D}tj|dS)zyUpdates the environment variables on this worker process. Args: env_vars: Environment variables to set. cSsi|] \}}|t|qSr%)str).0kvr%r%r& esz#update_env_vars..N)itemsosenvironupdate)r4 sanitizedr%r%r&update_env_vars_sr?fncs<t|j}tjjtjjtjjhtfdd|DS)aCounts the number of required parameters of a function. NOTE: *args counts as 1 required parameter. Examples -------- >>> def fn(a, b, /, c, *args, d=1, e=2, **kwargs): ... pass >>> count_required_parameters(fn) 4 >>> fn = lambda: 1 >>> count_required_parameters(fn) 0 >>> def fn(config, a, b=1, c=2): ... pass >>> from functools import partial >>> count_required_parameters(partial(fn, a=0)) 1 cs(g|]}|jtjjkr|jvr|qSr%)defaultinspect Parameteremptykind)r6ppositional_param_kindsr%r& s z-count_required_parameters..) rB signature parametersvaluesrCPOSITIONAL_ONLYPOSITIONAL_OR_KEYWORDVAR_POSITIONALr)r@paramsr%rGr&count_required_parametersis rQ train_funcFconfigtrain_func_context fn_arg_namediscard_returnsc st}|rtfdd}|n|dkr&|d|d}t||dkrAdur0intfdd}|Stfd d}|S) aValidates and constructs the training function to execute. Args: train_func: The training function to execute. This can either take in no arguments or a ``config`` dict. config (Optional[Dict]): Configurations to pass into ``train_func``. If None then an empty Dict will be created. train_func_context: Context manager for user's `train_func`, which executes backend-specific logic before and after the training function. fn_arg_name (Optional[str]): The name of training function to use for error messages. discard_returns: Whether to discard any returns from train_func or not. Returns: A valid training function. Raises: ValueError: if the input ``train_func`` is invalid. c s4z |i|WdSty}zt|d}~wwNrr )argskwargse)rRr%r&discard_return_wrappers z4construct_train_func..discard_return_wrapperz: should take in 0 or 1 required arguments, but it accepts z required arguments instead.Nc sVz WdWS1swYWdSty*}zt|d}~wwrWrXr[)rSrTwrapped_train_funcr%r&train_fns(z&construct_train_func..train_fnc sTz WdWS1swYWdSty)}zt|d}~wwrWrXr^)rTr_r%r&r`s()rQ functoolswraps ValueError) rRrSrTrUrVnum_required_paramsr\err_msgr`r%)rSrRrTr_r&construct_train_funcs(rfcs$eZdZdZiZfddZZS) SingletonzSingleton Abstract Base Class https://stackoverflow.com/questions/33364070/implementing -singleton-as-metaclass-but-for-abstract-classes cs0||jvrtt|j|i||j|<|j|SrW) _instancessuperrg__call__)clsrYrZ __class__r%r&rjs  zSingleton.__call__)__name__ __module__ __qualname____doc__rhrj __classcell__r%r%rlr&rgsrgc@s&eZdZdZdefddZddZdS) ActorWrapperzDWraps an actor to provide same API as using the base class directly.actorcCs ||_dSrW)rt)selfrtr%r%r&__init__s zActorWrapper.__init__cst|j|fddS)Ncstj|i|SrW)rrremote)rYrZ actor_methodr%r&sz*ActorWrapper.__getattr__..)getattrrt)ruitemr%rxr& __getattr__s  zActorWrapper.__getattr__N)rnrorprqr rvr}r%r%r%r&rss rs)rRF)-abcrarBloggingr;pathlibrtypingrrrrrrr r r r ray.actorr ray.air._internal.utilr rrray.exceptionsr ray.typesrr getLoggerrnrboolrr'r5intr,r3r?rQrfABCMetargrsr%r%r%r&sP ,     & +$  F