o ci(@sddlmZmZddlZddlZddlmZddlm Z ddl m Z ddl m Z ddlmZddlmZed d Gd d d ZdS) )OptionalUnionN)Result) RayTaskError)ExperimentAnalysis) TuneError)Trial) PublicAPIbeta) stabilityc @s"eZdZdZdefddZedefddZede j j fdd Z d)d e ede edededef ddZ d*de ede edejfddZdefddZdedefddZeddZeddZed d!Zed"ede eeeffd#d$Zd"edefd%d&Z defd'd(Z!d S)+ ResultGrida2A set of ``Result`` objects for interacting with Ray Tune results. You can use it to inspect the trials and obtain the best result. The constructor is a private API. This object can only be created as a result of ``Tuner.fit()``. Example: .. testcode:: import random from ray import tune def random_error_trainable(config): if random.random() < 0.5: return {"loss": 0.0} else: raise ValueError("This is an error") tuner = tune.Tuner( random_error_trainable, run_config=tune.RunConfig(name="example-experiment"), tune_config=tune.TuneConfig(num_samples=10), ) try: result_grid = tuner.fit() except ValueError: pass for i in range(len(result_grid)): result = result_grid[i] if not result.error: print(f"Trial finishes successfully with metrics" f"{result.metrics}.") else: print(f"Trial failed with error {result.error}.") .. testoutput:: :hide: ... You can also use ``result_grid`` for more advanced analysis. >>> # Get the best result based on a particular metric. >>> best_result = result_grid.get_best_result( # doctest: +SKIP ... metric="loss", mode="min") >>> # Get the best checkpoint corresponding to the best result. >>> best_checkpoint = best_result.checkpoint # doctest: +SKIP >>> # Get a dataframe for the last reported results of all of the trials >>> df = result_grid.get_dataframe() # doctest: +SKIP >>> # Get a dataframe for the minimum loss seen for each trial >>> df = result_grid.get_dataframe(metric="loss", mode="min") # doctest: +SKIP Note that trials of all statuses are included in the final result grid. If a trial is not in terminated state, its latest result and checkpoint as seen by Tune will be provided. See :doc:`/tune/examples/tune_analyze_results` for more usage examples. experiment_analysiscs"|_fddjjD_dS)Ncsg|]}|qS)_trial_to_result).0trialselfrH/home/ubuntu/.local/lib/python3.10/site-packages/ray/tune/result_grid.py Os z'ResultGrid.__init__..)_experiment_analysistrials_results)rr rrr__init__Js  zResultGrid.__init__returncC|jjS)zPath pointing to the experiment directory on persistent storage. This can point to a remote storage location (e.g. S3) or to a local location (path on the head node).)rexperiment_pathrrrrrSszResultGrid.experiment_pathcCr)zReturn the filesystem that can be used to access the experiment path. Returns: pyarrow.fs.FileSystem implementation. )r_fsrrrr filesystem[szResultGrid.filesystemNlastTmetricmodescopefilter_nan_and_infcCst|jjdkr||jjdS|s|jjstd|s%|jjs%td|jj||||d}|sGd|p7|jjd}||r@dnd 7}t|||S) aGet the best result from all the trials run. Args: metric: Key for trial info to order on. Defaults to the metric specified in your Tuner's ``TuneConfig``. mode: One of [min, max]. Defaults to the mode specified in your Tuner's ``TuneConfig``. scope: One of [all, last, avg, last-5-avg, last-10-avg]. If `scope=last`, only look at each trial's final step for `metric`, and compare across trials based on `mode=[min,max]`. If `scope=avg`, consider the simple average over all steps for `metric` and compare across trials based on `mode=[min,max]`. If `scope=last-5-avg` or `scope=last-10-avg`, consider the simple average over the last 5 or 10 steps for `metric` and compare across trials based on `mode=[min,max]`. If `scope=all`, find each trial's min/max score for `metric` based on `mode`, and compare trials based on `mode=[min,max]`. filter_nan_and_inf: If True (default), NaN or infinite values are disregarded and these trials are never selected as the best trial. rzNo metric is provided. Either pass in a `metric` arg to `get_best_result` or specify a metric in the `TuneConfig` of your `Tuner`.z|No mode is provided. Either pass in a `mode` arg to `get_best_result` or specify a mode in the `TuneConfig` of your `Tuner`.)r r!r"r#z*No best trial found for the given metric: z3. This means that no trial has reported this metricz~, or all values reported for this metric are NaN. To not ignore NaN values, you can set the `filter_nan_and_inf` arg to False..) lenrrrdefault_metric ValueError default_modeget_best_trial RuntimeError)rr r!r"r# best_trial error_msgrrrget_best_resultds8    zResultGrid.get_best_result filter_metric filter_modecCs|jj||dS)a4Return dataframe of all trials with their configs and reported results. Per default, this returns the last reported results for each trial. If ``filter_metric`` and ``filter_mode`` are set, the results from each trial are filtered for this metric and mode. For example, if ``filter_metric="some_metric"`` and ``filter_mode="max"``, for each trial, every received result is checked, and the one where ``some_metric`` is maximal is returned. Example: .. testcode:: import ray.tune def training_loop_per_worker(config): ray.tune.report({"accuracy": 0.8}) result_grid = ray.tune.Tuner( trainable=training_loop_per_worker, run_config=ray.tune.RunConfig(name="my_tune_run") ).fit() # Get last reported results per trial df = result_grid.get_dataframe() # Get best ever reported accuracy per trial df = result_grid.get_dataframe( filter_metric="accuracy", filter_mode="max" ) .. testoutput:: :hide: ... Args: filter_metric: Metric to filter best result for. filter_mode: If ``filter_metric`` is given, one of ``["min", "max"]`` to specify if we should find the minimum or maximum result. Returns: Pandas DataFrame with each trial as a row and their results as columns. )r r!)r dataframe)rr/r0rrr get_dataframes3zResultGrid.get_dataframecCs t|jSN)r&rrrrr__len__s zResultGrid.__len__icCs |j|S)z$Returns the i'th result in the grid.)r)rr5rrr __getitem__s zResultGrid.__getitem__cCsdd|DS)z)Returns the exceptions of errored trials.cSsg|]}|jr|jqSr)errorrresultrrrrz%ResultGrid.errors..rrrrrerrorsszResultGrid.errorscCtdd|jjDS)z%Returns the number of errored trials.cSg|] }|jtjkr|qSr)statusrERRORrtrrrrsz)ResultGrid.num_errors..r&rrrrrr num_errorsszResultGrid.num_errorscCr<)z:Returns the number of terminated (but not errored) trials.cSr=r)r>r TERMINATEDr@rrrrs  z-ResultGrid.num_terminated..rBrrrrnum_terminateds zResultGrid.num_terminatedrcCs |jtjkrdS|p|Sr3)r>rrDget_pickled_error get_error)rrrr_populate_exceptions zResultGrid._populate_exceptionc Csj|jj}d}|jr |jj}|j}dd|D}|jj|j}t ||j | ||j |jj||d}|S)NcSsg|]}|j|jfqSr) checkpointmetrics)rcheckpoint_resultrrrrs z/ResultGrid._trial_to_result..)rIrJr7path_storage_filesystemmetrics_dataframebest_checkpoints) run_metadatacheckpoint_managerlatest_checkpoint_resultrIbest_checkpoint_resultsrtrial_dataframesgettrial_idr last_resultcopyrHrLr)rrcpmrIrSrO metrics_dfr9rrrrs& zResultGrid._trial_to_resultcCs$dd|D}d|}d|dS)NcSsg|]}|jddqS))indent)_reprr8rrrrr:z'ResultGrid.__repr__..z, z ResultGrid<[ z ]>)join)rall_results_reprrrr__repr__s  zResultGrid.__repr__)NNrT)NN)"__name__ __module__ __qualname____doc__rrpropertystrrpyarrowfs FileSystemrrboolrr.pd DataFramer2intr4r6r;rCrE staticmethodrrrrrHrr`rrrrr sZ:    C 7    r )typingrrpandasrkrgray.air.resultrray.exceptionsrray.tune.analysisrray.tune.errorrray.tune.experimentrray.utilr r rrrrs