o $i>@sdddlZddlZddlZddlZddlZddlmZddlm Z ddl m Z ddl m Z mZmZmZmZmZmZmZmZmZddlZddlmZddlmZddlmZdd lmZm Z dd l!m"Z"dd l#m$Z$m%Z%dd l&m'Z'm(Z(m)Z)m*Z*dd l+m,Z,m-Z-e rddl.Z/ddlm0Z0ddl1m2Z2e3e4Z5ddZ6e-GdddZ7dee7ee7effddZ8dS)N)partial)Path) PicklingError) TYPE_CHECKINGAnyCallableDictListMappingOptionalSequenceTypeUnion)RpcError)StorageContext)DEFAULT_STORAGE_PATH)CheckpointConfig SyncConfig) TuneError)is_function_trainableregister_trainable)CombinedStopperFunctionStopperStopperTimeoutStopper) Deprecated DeveloperAPI)PlacementGroupFactory)TrialcCs|s d}}||fSt|tr|rd}d}||fSt|tr&|}}||fSt|trBt|dkr:tdt||\}}||fStdt|)zsValidate ``tune.RunConfig``'s ``log_to_file`` parameter. Return validated relative stdout and stderr filenames.NstdoutstderrzIf you pass a Sequence to `log_to_file` it has to have a length of 2 (for stdout and stderr, respectively). The Sequence you passed has length {}.zqYou can pass a boolean, a string, or a Sequence of length 2 to `log_to_file`, but you passed something else ({}).) isinstanceboolstrr len ValueErrorformattype) log_to_file stdout_file stderr_filer,[/home/ubuntu/veenaModal/venv/lib/python3.10/site-packages/ray/tune/experiment/experiment.py_validate_log_to_file*s0   r.c&@s,eZdZdZhdZeZdddddddddddddddddded eee e fd e ee e e ee geffd e eeeejfd e eeefd ede eeeee ffdfdede ede dde eeefde eeefde e dgefde e dgefdede edede ede ef$ddZededefd d!Zed"eee e ffd#d$Zed"eee e ffd%d&Zed'd(Zed)e efd*d+Z ee!d,d-d.Z"ed)e efd/d0Z#ed)e efd1d2Z$ed3d4Z%ee!d,d5d6Z&ed7d8Z'ed)eeeffd9d:Z(dS); ExperimentaTracks experiment specifications. Implicitly registers the Trainable if needed. The args here take the same meaning as the arguments defined `tune.py:run`. .. code-block:: python experiment_spec = Experiment( "my_experiment_name", my_func, stop={"mean_accuracy": 100}, config={ "alpha": tune.grid_search([0.2, 0.4, 0.6]), "beta": tune.grid_search([1, 2]), }, resources_per_trial={ "cpu": 1, "gpu": 0 }, num_samples=10, local_dir="~/ray_results", checkpoint_freq=10, max_failures=2) >stop num_samples time_budget_sNFr)r0r2configresources_per_trialr1 storage_pathstorage_filesystem sync_configcheckpoint_configtrial_name_creatortrial_dirname_creatorr)export_formats max_failuresrestore local_dirnamerunr0r2r4r5rr1r6r7zpyarrow.fs.FileSystemr8r9r:rr;r)r<r=r>r?cCsDt| tr tdi| } n| pt} t|r$| jrtd| jr$tdzt||_ Wnt yJ}z|j t j jkrDtdt|d}~ww|sRt|}|pUt}|j|| | |d|_td|j|pli}t|ts|tdt|dd|_i}|snZt|trdd |D}|rd d |D}td |d t||_n7t|tr|}n/t|rt|rt||_n t|t r||_ntd t|d td|dt|d |r|jrt|jt!||_nt!||_t"|\}}|j |||||| | | ||f|p g||rt#|$%&nd|jd}||_'dS)Nz'checkpoint_at_end' cannot be used with a function trainable. You should include one last call to `ray.tune.report(metrics=..., checkpoint=...)` at the end of your training loop to get this behavior.z'checkpoint_frequency' cannot be set for a function trainable. You will need to report a checkpoint every `checkpoint_frequency` iterations within your training loop using `ray.tune.report(metrics=..., checkpoint=...)` to get this behavior.aThe Trainable/training function is too large for grpc resource limit. Check that its definition is not implicitly capturing a large array or other object in scope. Tip: use tune.with_parameters() to put large objects in the Ray object store. Original exception: )r6r7r8experiment_dir_namezStorageContext on the DRIVER: z*`Experiment(config)` must be a dict, got: zB. Please convert your search space to a dict before passing it in.cSsg|] }t|ts|qSr,)r"r.0sr,r,r- sz'Experiment.__init__..cSsg|]}t|qSr,)r(rCr,r,r-rFszIf you pass a list as the `stop` argument to `tune.RunConfig()`, each element must be an instance of `tune.stopper.Stopper`. Got .zaProvided stop object must be either a dict, a function, or a subclass of `ray.tune.Stopper`. Got zInvalid stop criteria: z". Must be a callable or dict. Got )rAr0r2r4r5r1r9r:r;r)r<r=r>storager,)(r"dictrrcheckpoint_at_endr&checkpoint_frequencyr/register_if_needed_run_identifierrrpc_coderay_raylet#GRPC_STATUS_CODE_RESOURCE_EXHAUSTEDr traceback format_excrget_experiment_dir_namer_storage_context_clsrHloggerdebugr(_stopperlistrcallableris_valid_functionrrr.r expanduserabsoluteas_posixspec)selfr@rAr0r2r4r5r1r6r7r8r9r:r;r)r<r=r>r?estopping_criteria bad_stoppers stopper_typesr*r+r_r,r,r-__init__fs               zExperiment.__init__r_c Csd|vrtdd|vr|di|d<|d|dd<|d=d|vr5t|dtr5td i|d|d<d|vrKt|dtrKtd i|d|d<t|}|d}z |||fi|}W|St yz}z tdt |d|d }~ww) zGenerates an Experiment object from JSON. Args: name: Name of Experiment. spec: JSON configuration of experiment. rAzNo trainable specified!envr4r8r9z>Failed to load the following Tune experiment specification: zd. Please check that the arguments are valid. Experiment creation failed with the following error: Nr,) rgetr"rIrrcopydeepcopypop TypeErrorpppformat)clsr@r_ run_valueexprar,r,r- from_jsons2   zExperiment.from_json run_objectcCsddlm}t|tst||r|St|tst|rdd}t|dr'|j}|St|drD|j}|dkr7d}|S| dr@d}|S|}|St|t rZt|d rZt|j drZ|j j}|St d ||Std ) aGet Trainable name. Args: run_object: Trainable to run. If string, assumes it is an ID and does not modify it. Otherwise, returns a string corresponding to the run_object name. Returns: A string representing the trainable identifier. Raises: TuneError: if ``run_object`` passed in is invalid. rDomainDEFAULT_name__name__zlambda<funcz(No name detected on trainable. Using {}.z*Improper 'run' - not string nor trainable.)ray.tune.search.samplertr"r$r(rZhasattrrvrw startswithrrzrVwarningr'r)rnrrrtr@fn_namer,r,r-get_trainable_names:       zExperiment.get_trainable_namec Csddlm}t|tr |St||rtd|S||}zt||W|Stt fyC}zd}t |t|d|dd}~ww)aRegisters Trainable or Function at runtime. Assumes already registered if run_object is a string. Also, does not inspect interface of given run_object. Args: run_object: Trainable to run. If string, assumes it is an ID and does not modify it. Otherwise, returns a string corresponding to the run_object name. Returns: A string representing the trainable identifier. rrsz0Not registering trainable. Resolving as variant.zOther options: -Try reproducing the issue by calling `pickle.dumps(trainable)`. -If the error is typing-related, try removing the type annotations and try again. N) r{rtr"r$rVr~rrrkrr()rnrrrtr@ra extra_msgr,r,r-rLGs       zExperiment.register_if_neededcC|jSN)rXr`r,r,r-stopperkszExperiment.stopperreturncC|jjSr)rHexperiment_driver_staging_pathrr,r,r- local_pathozExperiment.local_pathzReplaced by `local_path`cCtd)Nz(Use `local_path` instead of `local_dir`.DeprecationWarningrr,r,r-r?szExperiment.local_dircCrr)rHexperiment_fs_pathrr,r,r- remote_pathyrzExperiment.remote_pathcCs |jp|jSr)rrrr,r,r-path} zExperiment.pathcCs |jdS)Nr9)r_rgrr,r,r-r9rzExperiment.checkpoint_configcCr)Nz-Use `local_path` instead of `checkpoint_dir`.rrr,r,r-checkpoint_dirrzExperiment.checkpoint_dircCr)z7Returns a string representing the trainable identifier.)rMrr,r,r-run_identifierszExperiment.run_identifiercsfddjDS)zReturns the spec dict with only the public-facing keys. Intended to be used for passing information to callbacks, Searchers and Schedulers. cs i|] \}}|jvr||qSr,) PUBLIC_KEYS)rDkvrr,r- s z*Experiment.public_spec..)r_itemsrr,rr- public_specszExperiment.public_spec))rw __module__ __qualname____doc__rrrUr$rrr r r rr#intfloatdatetime timedeltarrrrIrr re classmethodrqrrLpropertyrrrr?rrr9rrrr,r,r,r-r/Fs     &*#     r/ experimentscCs|}|dur g}nt|tr|g}nt|tur!dd|D}t|tur=tdd|Dr=t|dkr;t d|St d |) a%Produces a list of Experiment objects. Converts input from dict, single experiment, or list of experiments to list of experiments. If input is None, will return an empty list. Arguments: experiments: Experiments to run. Returns: List of experiments. NcSsg|] \}}t||qSr,)r/rq)rDr@r_r,r,r-rFsz/_convert_to_experiment_list..css|]}t|tVqdSr)r"r/)rDrpr,r,r- sz._convert_to_experiment_list..r3zeRunning with multiple concurrent experiments. All experiments will be using the same SearchAlgorithm.zInvalid argument: {}) r"r/r(rIrrYallr%rVinforr')rexp_listr,r,r-_convert_to_experiment_lists     r)9rhrloggingpprintrlrR functoolsrpathlibrpicklertypingrrrrr r r r r rrOray.exceptionsrray.train._internal.storagerray.train.constantsrray.tunerrray.tune.errorrray.tune.registryrrray.tune.stopperrrrrray.util.annotationsrr pyarrow.fspyarrowrray.tune.experimentr getLoggerrwrVr.r/rr,r,r,r-s:   0        U