o `۷i@sddlZddlZddlZddlZddlZddlZddlmZmZm Z m Z m Z m Z m Z mZmZddlZddlmZmZddlmZddlmZmZddlmZddlmZedZeeZ d e ed e e!e e"ffd d Z#d e e$e%ffd dZ&de e$effddZ'ded e%fddZ(  d!deegefee e$efgeffde e e$efde de e$de!d egeff ddZ)Gdddej*Z+Gdd d Z,dS)"N) AnyCallableContextManagerDictListOptionalTupleTypeVarUnion)find_free_portis_ipv6) ActorHandle)StartTracebackStartTracebackWithWorkerRank) RayActorError) ObjectRefT remote_valuesreturnc Cs|}t|dkr_t|\}}|D]E}zt|WqtyB}z||}td|dd|fWYd}~Sd}~wt yX}z ||}t |d|d}~wwt|dks dS)a$Check for actor failure when retrieving the remote values. Args: remote_values: List of object references from Ray actor methods. Returns: A tuple of (bool, Exception). The bool is True if evaluating all object references is successful, False otherwise. rzWorker z has failed.FN) worker_rank)TN) copylenraywaitgetrindexloggerinfo Exceptionr)r unfinishedfinished object_refexcfailed_actor_rankfailed_worker_rankr%O/home/ubuntu/vllm_env/lib/python3.10/site-packages/ray/train/_internal/utils.pycheck_for_failure"s,    r'cCs*tj}tt|r tjntj}||fS)z4Returns the IP address and a free port on this node.)rutilget_node_ip_addressr r socketAF_INET6AF_INET)addrportr%r%r&get_address_and_portHs r/env_varscCs"dd|D}tj|dS)zyUpdates the environment variables on this worker process. Args: env_vars: Environment variables to set. cSsi|] \}}|t|qSr%)str).0kvr%r%r& Usz#update_env_vars..N)itemsosenvironupdate)r0 sanitizedr%r%r&update_env_varsOsr;fncs<t|j}tjjtjjtjjhtfdd|DS)aCounts the number of required parameters of a function. NOTE: *args counts as 1 required parameter. Examples -------- >>> def fn(a, b, /, c, *args, d=1, e=2, **kwargs): ... pass >>> count_required_parameters(fn) 4 >>> fn = lambda: 1 >>> count_required_parameters(fn) 0 >>> def fn(config, a, b=1, c=2): ... pass >>> from functools import partial >>> count_required_parameters(partial(fn, a=0)) 1 cs(g|]}|jtjjkr|jvr|qSr%)defaultinspect Parameteremptykind)r2ppositional_param_kindsr%r& xs z-count_required_parameters..) r> signature parametersvaluesr?POSITIONAL_ONLYPOSITIONAL_OR_KEYWORDVAR_POSITIONALr)r<paramsr%rCr&count_required_parametersYs rM train_funcFconfigtrain_func_context fn_arg_namediscard_returnsc st}|rtfdd}|n|dkr&|d|d}t||dkrAdur0intfdd}|Stfd d}|S) aValidates and constructs the training function to execute. Args: train_func: The training function to execute. This can either take in no arguments or a ``config`` dict. config (Optional[Dict]): Configurations to pass into ``train_func``. If None then an empty Dict will be created. train_func_context: Context manager for user's `train_func`, which executes backend-specific logic before and after the training function. fn_arg_name (Optional[str]): The name of training function to use for error messages. discard_returns: Whether to discard any returns from train_func or not. Returns: A valid training function. Raises: ValueError: if the input ``train_func`` is invalid. c s4z |i|WdSty}zt|d}~wwNrr)argskwargse)rNr%r&discard_return_wrappers z4construct_train_func..discard_return_wrapperz: should take in 0 or 1 required arguments, but it accepts z required arguments instead.Nc sVz WdWS1swYWdSty*}zt|d}~wwrSrTrW)rOrPwrapped_train_funcr%r&train_fns(z&construct_train_func..train_fnc sTz WdWS1swYWdSty)}zt|d}~wwrSrTrZ)rPr[r%r&r\s()rM functoolswraps ValueError) rNrOrPrQrRnum_required_paramsrXerr_msgr\r%)rOrNrPr[r&construct_train_funcs(rbcs$eZdZdZiZfddZZS) SingletonzSingleton Abstract Base Class https://stackoverflow.com/questions/33364070/implementing -singleton-as-metaclass-but-for-abstract-classes cs0||jvrtt|j|i||j|<|j|SrS) _instancessuperrc__call__)clsrUrV __class__r%r&rfs  zSingleton.__call__)__name__ __module__ __qualname____doc__rdrf __classcell__r%r%rhr&rcsrcc@s&eZdZdZdefddZddZdS) ActorWrapperzDWraps an actor to provide same API as using the base class directly.actorcCs ||_dSrS)rp)selfrpr%r%r&__init__s zActorWrapper.__init__cst|j|fddS)Ncstj|i|SrS)rrremote)rUrV actor_methodr%r&sz*ActorWrapper.__getattr__..)getattrrp)rqitemr%rtr& __getattr__s  zActorWrapper.__getattr__N)rjrkrlrmr rrryr%r%r%r&ros ro)rNF)-abcr]r>loggingr7r*typingrrrrrrrr r rray._common.network_utilsr r ray.actorr ray.air._internal.utilrrray.exceptionsr ray.typesrr getLoggerrjrboolrr'r1intr/r;rMrbABCMetarcror%r%r%r&sP,      & +$  F