o $iol@s^ddlZddlZddlZddlZddlZddlmZddlmZddl m Z m Z m Z m Z mZmZddlZddlmZmZmZddlmZmZddlmZddlmZdd lmZdd lm Z dd l!m"Z"m#Z#dd l$m%Z%dd l&m'Z'ddl(m)Z)m*Z*m+Z+ddl,m-Z-z ddl.Z/ddl.m0Z0Wn e1ydZ/dZ0Ynwe2e3Z4e-ddGdddZ5dS)N)Number)Path)AnyDictListOptionalTupleUnion)EXPR_PROGRESS_FILEEXPR_RESULT_FILETRAINING_ITERATION)_exists_at_fs_pathget_fs_and_path) Checkpoint)"_find_newest_experiment_checkpoint)TuneController)Trial) CONFIG_PREFIXDEFAULT_METRIC) flatten_dict)TuneFunctionDecoder)is_nan is_nan_or_infunflattened_lookup) PublicAPI) DataFramebeta) stabilityc @seZdZdZddddddeeejfdee j j dee e deedeef d d Zd e e fd d Zde d efddZd eeeffddZdIded eeeffddZed efddZed e fddZed efddZed efddZed efdd Zed efd!d"Zd#d$Zed efd%d&Zed eeeffd'd(Z ed efd)d*Z!ed eeeffd+d,Z" dJd-eed.eed efd/d0Z# dKde d-eed e e$ee%ffd1d2Z&  dJde d-eed.eed eefd3d4Z'   5 6dLd-eed.eed7ed8ed ee f d9d:Z(   5dMd-eed.eed7ed eefd;d<Z) >dNd eefd?d@Z*d-ed efdAdBZ+d.ed efdCdDZ, dJd-eed.eed eee-ffdEdFZ.d eee-ffdGdHZ/dS)OExperimentAnalysisaAnalyze results from a Ray Train/Tune experiment. To use this class, the run must store the history of reported metrics in log files (e.g., `result.json` and `progress.csv`). This is the default behavior, unless default loggers are explicitly excluded with the `TUNE_DISABLE_AUTO_CALLBACK_LOGGERS=1` environment variable. Parameters: experiment_checkpoint_path: Path to an `experiment_state.json` file, or a directory that contains an `experiment_state.json` file. default_metric: Default metric for comparing results. Can be overwritten with the ``metric`` parameter in the respective functions. default_mode: Default mode for comparing results. Has to be one of [min, max]. Can be overwritten with the ``mode`` parameter in the respective functions. trials: List of trials that can be accessed via `analysis.trials`. N)storage_filesystemtrialsdefault_metric default_modeexperiment_checkpoint_pathrr r!r"cCs||_|r |dvr td||_|jdur|jdurt|_|r#||_nt|\|_}t|}|dr>tj ||_ ||_ n)||_ t |j |jd}|durdtjd}td|d|jjd|j d ||_ |pl||_||_||_dS) Nminmaxz2`default_mode` has to be None or one of [min, max]z.json)experiment_pathfs*z%No experiment snapshot file of form 'z' was found at: (z, z) Please check if you specified the correct experiment path, which should be a combination of the `storage_path` and `name` specified in your run.)r! ValueErrorr"r_fsrstrendswithospathdirname_experiment_fs_path_experiment_json_fs_pathrrCKPT_FILE_TMPLformat type_name _load_trialsr _fetch_trial_dataframes_trial_dataframesget_all_configs_configs)selfr#rr r!r"experiment_json_fs_pathpatternr>b/home/ubuntu/veenaModal/venv/lib/python3.10/site-packages/ray/tune/analysis/experiment_analysis.py__init__8s@     zExperimentAnalysis.__init__returnc Cs|j|j}tj|td}Wdn1swYt|j}g}|d}|D].\}}t j |dd}| |t |j } |j| _|j| _|j| _|| ||q-|S)N)cls trial_dataTstub)r+open_input_streamr2jsonloadsreadallrrr1rfrom_json_staterestore_run_metadatacopystorageparentas_posixstorage_fs_pathrnameexperiment_dir_name set_storageappend) r;fexperiment_stateexperiment_fs_pathr trial_statestrial_json_statetrial_runtime_metadatatrial new_storager>r>r?r6ls        zExperimentAnalysis._load_trialsr[c CsPdti}|jdur tSt|jjt}t|jjt}t |jj |re|jj |(}| dd}|sCtWdSdd|dD}Wdn1sWwYtj|dd}|St |jj |r|jj |}| d} Wdn1swYtjt| |d}|Std |d td td |jj) Ntrial_idzutf-8 cSsg|]}t|qSr>)rGrH).0rowr>r>r? z=ExperimentAnalysis._fetch_trial_dataframe../)sep)dtypezCould not fetch metrics for z: both z and z were not found at )r, last_resultrrrM trial_fs_pathr rOr r rrFrIdecoderstripsplitpdjson_normalizeread_csvioStringIOFileNotFoundError) r;r[ force_dtype json_fs_path csv_fs_pathrUcontent json_listdfcsv_strr>r>r?_fetch_trial_dataframes8  z)ExperimentAnalysis._fetch_trial_dataframec Csg}i}|jD]+}z ||||j<Wqty2}z|||ft||j<WYd}~qd}~ww|rLddd|D}tdt |d||S)zyFetches trial dataframes from files. Returns: A dictionary mapping trial_id -> pd.DataFrame Nr^cSs$g|]\}}d|dt|qS)z- z: )repr)r_r[errorr>r>r?ras$z>ExperimentAnalysis._fetch_trial_dataframes..zFailed to fetch metrics for z trial(s): ) r rxr] ExceptionrTrjoinloggerwarninglen)r;failures trial_dfsr[efail_strr>r>r?r7s&    z*ExperimentAnalysis._fetch_trial_dataframesFprefixcsfdd|jDS)zReturns all trial hyperparameter configurations. Args: prefix: If True, flattens the config dict and prepends `config/`. Returns: Dict[str, Dict]: Mapping trial_id -> config dict cs(i|]}|jrtt|jin|jqSr>)r]rrconfigr_r[rr>r? s z6ExperimentAnalysis.get_all_configs..r )r;rr>rr?r9s z"ExperimentAnalysis.get_all_configscC|jS)zPath pointing to the experiment directory on persistent storage. This can point to a remote storage location (e.g. S3) or to a local location (path on the head node).)r1r;r>r>r?r'sz"ExperimentAnalysis.experiment_pathcC$|jr|js td||j|jS)a)Get the best trial of the experiment The best trial is determined by comparing the last trial results using the `metric` and `mode` parameters passed to `tune.run()`. If you didn't pass these parameters, use `get_best_trial(metric, mode, scope)` instead. zTo fetch the `best_trial`, pass a `metric` and `mode` parameter to `tune.run()`. Alternatively, use the `get_best_trial(metric, mode)` method to set the metric and mode explicitly.)r!r"r*get_best_trialrr>r>r? best_trial zExperimentAnalysis.best_trialcCr)a8Get the config of the best trial of the experiment The best trial is determined by comparing the last trial results using the `metric` and `mode` parameters passed to `tune.run()`. If you didn't pass these parameters, use `get_best_config(metric, mode, scope)` instead. zTo fetch the `best_config`, pass a `metric` and `mode` parameter to `tune.run()`. Alternatively, use the `get_best_config(metric, mode)` method to set the metric and mode explicitly.)r!r"r*get_best_configrr>r>r? best_configrzExperimentAnalysis.best_configcCsJ|jr|js td|j}|std|jd|jd|||j|jS)aGet the checkpoint path of the best trial of the experiment The best trial is determined by comparing the last trial results using the `metric` and `mode` parameters passed to `tune.run()`. If you didn't pass these parameters, use `get_best_checkpoint(trial, metric, mode)` instead. Returns: :class:`Checkpoint ` object. zTo fetch the `best_checkpoint`, pass a `metric` and `mode` parameter to `tune.run()`. Alternatively, use the `get_best_checkpoint(trial, metric, mode)` method to set the metric and mode explicitly.zONo best trial found. Please check if you specified the correct default metric (z ) and mode (z).)r!r"r*rget_best_checkpoint)r;rr>r>r?best_checkpoints  z"ExperimentAnalysis.best_checkpointcCs"|jr|js td|j|jjS)aGet the full result dataframe of the best trial of the experiment The best trial is determined by comparing the last trial results using the `metric` and `mode` parameters passed to `tune.run()`. If you didn't pass these parameters, use `get_best_trial(metric, mode)` and use it to look for the dataframe in the `self.trial_dataframes` dict. zQTo fetch the `best_result`, pass a `metric` and `mode` parameter to `tune.run()`.)r!r"r*trial_dataframesrr]rr>r>r?best_dataframes z!ExperimentAnalysis.best_dataframecCs|jr|js td|jjS)aHGet the last result of the best trial of the experiment The best trial is determined by comparing the last trial results using the `metric` and `mode` parameters passed to `tune.run()`. If you didn't pass these parameters, use `get_best_trial(metric, mode, scope).last_result` instead. zTo fetch the `best_result`, pass a `metric` and `mode` parameter to `tune.run()`. Alternatively, use `get_best_trial(metric, mode).last_result` to set the metric and mode explicitly and fetch the last result.)r!r"r*rrfrr>r>r? best_result)s zExperimentAnalysis.best_resultcCstjddS)NTUNE_RESULT_DELIMrc)r.environgetrr>r>r? _delimiter<szExperimentAnalysis._delimitercCs0tstdt|j|d}tjj|gddS)aMGet the best result of the experiment as a pandas dataframe. The best trial is determined by comparing the last trial results using the `metric` and `mode` parameters passed to `tune.run()`. If you didn't pass these parameters, use `get_best_trial(metric, mode, scope).last_result` instead. zD`best_result_df` requires pandas. Install with `pip install pandas`. delimiterr]index)rkr*rrrr from_records)r;rr>r>r?best_result_df?s  z!ExperimentAnalysis.best_result_dfcCsdd|jDS)z7Get the last result of the all trials of the experimentcSsi|]}|j|jqSr>)r]rfrr>r>r?rUrbz.ExperimentAnalysis.results..rrr>r>r?resultsRszExperimentAnalysis.resultscs,tstdtjjfddjDddS)z/Get all the last results as a pandas dataframe.z@`results_df` requires pandas. Install with `pip install pandas`.csg|] }t|jdqS)r)rrfrrrr>r?ra_sz1ExperimentAnalysis.results_df..r]r)rkr*rrr rr>rr? results_dfWs zExperimentAnalysis.results_dfcCr)zList of all dataframes of the trials. Each dataframe is indexed by iterations and contains reported metrics. )r8rr>r>r?rfsz#ExperimentAnalysis.trial_dataframesmetricmodecCs|r |dvr td|r|std|j||d}|jdd}|D]\}}||vr:|||||j|dq#tt|S)aReturns a pandas.DataFrame object constructed from the trials. This function will look through all observed results of each trial and return the one corresponding to the passed ``metric`` and ``mode``: If ``mode=min``, it returns the result with the lowest *ever* observed ``metric`` for this trial (this is not necessarily the last)! For ``mode=max``, it's the highest, respectively. If ``metric=None`` or ``mode=None``, the last result will be returned. Args: metric: Key for trial info to order on. If None, uses last result. mode: One of [None, "min", "max"]. Returns: pd.DataFrame: Constructed from a result dict of each trial. r$*If set, `mode` has to be one of [min, max]z^If a `mode` is passed to `ExperimentAnalysis.dataframe(), you'll also have to pass a `metric`!)rrTr)logdir) r*_retrieve_rowsr9itemsupdaterkrlistvalues)r;rrrows all_configsr/rr>r>r? dataframeos  zExperimentAnalysis.dataframecs8p|jpt|jjj}dd|D}fdd|DS)aGet all checkpoints and a specified metric of a trial. Args: trial: The log directory of a trial, or a trial instance. metric: key for trial info to return, e.g. "mean_accuracy". "training_iteration" is used by default if no value was passed to ``self.default_metric``. Returns: List of [Checkpoint, metric] for all checkpoints of the trial. cSsg|]}|j|jfqSr>) checkpointmetrics)r_checkpoint_resultr>r>r?ras zIExperimentAnalysis._get_trial_checkpoints_with_metric..csg|] \}}|t|fqSr>)r)r_rrrr>r?ras )r!r run_metadatacheckpoint_managerbest_checkpoint_results)r;r[rrbest_checkpointsr>rr?"_get_trial_checkpoints_with_metrics z5ExperimentAnalysis._get_trial_checkpoints_with_metriccs||p|jpt}||}|||}ttdd|}|s(td|ddS|dkr.dndt|fd dd \}}|S) aJGets best persistent checkpoint path of provided trial. Any checkpoints with an associated metric value of ``nan`` will be filtered out. Args: trial: The log directory of a trial, or a trial instance. metric: key of trial info to return, e.g. "mean_accuracy". "training_iteration" is used by default if no value was passed to ``self.default_metric``. mode: One of [min, max]. Defaults to ``self.default_mode``. Returns: A :class:`Checkpoint ` object cSst|d SN)rxr>r>r?sz8ExperimentAnalysis.get_best_checkpoint..z)No checkpoints have been found for trial .Nr%rcs |dSrr>rscore_order_factorr>r?rs )key) r!r _validate_moderrfilterr}rzr&)r;r[rrcheckpoints_and_metricsr_r>rr?rs     z&ExperimentAnalysis.get_best_checkpointlastTscopefilter_nan_and_infc Cst|jdkr |jdS||}||}|dvr"td||d}d}|jD]D}||jvr1q)|dvr=|j||}n|j||}|rKt|rKq)|durT|}|}q)|dkra||kra|}|}q)|dkrm||krm|}|}q)|sut d |S) aRetrieve the best trial object. Compares all trials' scores on ``metric``. If ``metric`` is not specified, ``self.default_metric`` will be used. If `mode` is not specified, ``self.default_mode`` will be used. These values are usually initialized by passing the ``metric`` and ``mode`` parameters to ``tune.run()``. Args: metric: Key for trial info to order on. Defaults to ``self.default_metric``. mode: One of [min, max]. Defaults to ``self.default_mode``. scope: One of [all, last, avg, last-5-avg, last-10-avg]. If `scope=last`, only look at each trial's final step for `metric`, and compare across trials based on `mode=[min,max]`. If `scope=avg`, consider the simple average over all steps for `metric` and compare across trials based on `mode=[min,max]`. If `scope=last-5-avg` or `scope=last-10-avg`, consider the simple average over the last 5 or 10 steps for `metric` and compare across trials based on `mode=[min,max]`. If `scope=all`, find each trial's min/max score for `metric` based on `mode`, and compare trials based on `mode=[min,max]`. filter_nan_and_inf: If True (default), NaN or infinite values are disregarded and these trials are never selected as the best trial. Returns: The best trial for the provided metric. If no trials contain the provided metric, or if the value for the metric is NaN for all trials, then returns None. rr)allravg last-5-avg last-10-avgzExperimentAnalysis: attempting to get best trial for metric {} for scope {} not in ["all", "last", "avg", "last-5-avg", "last-10-avg"]. If you didn't pass a `metric` parameter to `tune.run()`, you have to pass one when fetching the best trial.N)rrrrr&r%zGCould not find best trial. Did you pass the correct `metric` parameter?) rr _validate_metricrr*r4metric_analysisrr}r~) r;rrrrrbest_metric_scorer[ metric_scorer>r>r?rsJ&       z!ExperimentAnalysis.get_best_trialcCs||||}|r |jSdS)a.Retrieve the best config corresponding to the trial. Compares all trials' scores on `metric`. If ``metric`` is not specified, ``self.default_metric`` will be used. If `mode` is not specified, ``self.default_mode`` will be used. These values are usually initialized by passing the ``metric`` and ``mode`` parameters to ``tune.run()``. Args: metric: Key for trial info to order on. Defaults to ``self.default_metric``. mode: One of [min, max]. Defaults to ``self.default_mode``. scope: One of [all, last, avg, last-5-avg, last-10-avg]. If `scope=last`, only look at each trial's final step for `metric`, and compare across trials based on `mode=[min,max]`. If `scope=avg`, consider the simple average over all steps for `metric` and compare across trials based on `mode=[min,max]`. If `scope=last-5-avg` or `scope=last-10-avg`, consider the simple average over the last 5 or 10 steps for `metric` and compare across trials based on `mode=[min,max]`. If `scope=all`, find each trial's min/max score for `metric` based on `mode`, and compare trials based on `mode=[min,max]`. N)rr)r;rrrrr>r>r?r4sz"ExperimentAnalysis.get_best_configtraining_iterationr&cCs|p|||}||tdS)aGets the last checkpoint of the provided trial, i.e., with the highest "training_iteration". If no trial is specified, it loads the best trial according to the provided metric and mode (defaults to max. training iteration). Args: trial: If None, load the best trial automatically. metric: If no trial is specified, use this metric to identify the best trial and load the last checkpoint from this trial. mode: If no trial is specified, use the metric and this mode to identify the best trial and load the last checkpoint from it. Returns: Path for last checkpoint of trial r&)rrr )r;r[rrr>r>r?get_last_checkpointTsz&ExperimentAnalysis.get_last_checkpointcCs|s |js td|p |jS)NzjNo `metric` has been passed and `default_metric` has not been set. Please specify the `metric` parameter.)r!r*)r;rr>r>r?rjs  z#ExperimentAnalysis._validate_metriccCs0|s |js td|r|dvrtd|p|jS)NzdNo `mode` has been passed and `default_mode` has not been set. Please specify the `mode` parameter.r$r)r"r*)r;rr>r>r?rrs   z!ExperimentAnalysis._validate_modec Cs|dus |dvs J|r|sJi}|jD]C\}}|jrq||vr&d}n|dkr1||}n |dkr<||}nd}z |j|||<WqtyZt d |Yqw|S)N)r&r%rr&r%z2Warning: Non-numerical value(s) encountered for {}) rremptyidxmaxidxminilocto_dict TypeErrorr}r~r4)r;rrrr/rvidxr>r>r?r|s*  z!ExperimentAnalysis._retrieve_rowscs:|j}dtdtfddfdd|dD|d<|S)zEnsure that trials are marked as stubs when pickling, so that they can be loaded later without the trainable being registered. r[rAcSs*|jr|St|jdd}|||S)NTrD)rErtrainable_name __setstate__ __getstate__)r[ trial_copyr>r>r?make_stub_if_neededs z.make_stub_if_neededcsg|]}|qSr>r>)r_trr>r?rasz3ExperimentAnalysis.__getstate__..r )__dict__rLr)r;stater>rr?rs zExperimentAnalysis.__getstate__)F)NN)N)NNrT)NNr)Nrr&)0__name__ __module__ __qualname____doc__r r,r.PathLikerpyarrowr( FileSystemrrr@r6rrxrr7boolr9propertyr'rrrrrrrrrrrrrrrrrrrrrrrrr>r>r>r?r#s    4  ( ! * Z !    r)6rLrnrGloggingr.numbersrpathlibrtypingrrrrrr pyarrow.fsrray.air.constantsr r r ray.train._internal.storager rray.tuner#ray.tune.execution.experiment_stater"ray.tune.execution.tune_controllerrray.tune.experimentrray.tune.resultrrray.tune.utilsrray.tune.utils.serializationrray.tune.utils.utilrrrray.util.annotationsrpandasrkr ImportError getLoggerrr}rr>r>r>r?s<