o ci@sddlZddlZddlZddlZddlZddlZddlZddlZddlm Z m Z m Z m Z m Z mZmZmZddlZddlmZmZmZmZddlm ZddlmZddlZddlZddlZddlmZddlm Z m!Z!ddl"m#Z#m$Z$dd l%m&Z&m'Z'dd l(m)Z)dd l*m+Z+m,Z,m-Z-dd l.m/Z/m0Z0m1Z1m2Z2m3Z3m4Z4dd l5m6Z6ddl7m8Z8ddl9m:Z:ddl;mZ>m?Z?ddl@mAZAe+\ZBZCe,\ZDZEZFe-\ZGZCeHeIZJ    dede ejKdeLdeMdeMdejKf ddZNdfdd ZO dgd!d"ZPdhd$eMfd%d&ZQde0d'fd(d)d*eLd+e eRd,eSfd-d.ZT /did0e6d1eLd2eLde eLfd3d4ZUd5e6ddfd6d7ZVe)d5e6fd8d9ZW djd:e eSe fd0e6d;eRdeRfdd?d@dAe ejYd:e e dBe e dCe edDe e dEeRd;eRdee6ejZj[ffdFdGZ\dldHdIZ]dJdKdLedMdNd@dOe eSe fdPeMddf dQdRZ^dmdSeMddTfdUdVZ_GdWdXdXZ`dYdMdZejadejafd[d\Zb ]   dndYeSd^d@d_eRd`eRdae eeSdbeRf dcddZcdS)oN) TYPE_CHECKINGAnyDictListOptionalTupleTypeUnion)BoxDiscrete MultiDiscrete MultiBinary)r)r)tune)WandbLoggerCallback WANDB_ENV_VAR)DEFAULT_MODULE_IDColumns)is_atari wrap_deepmind) OldAPIStack)try_import_jax try_import_tftry_import_torch)'DIFF_NUM_GRAD_UPDATES_VS_SAMPLER_POLICYENV_RUNNER_RESULTSEPISODE_RETURN_MEANEVALUATION_RESULTSNUM_ENV_STEPS_TRAINEDNUM_ENV_STEPS_SAMPLED_LIFETIME) ResultDict)UnsupportedSpaceException) CLIReporter)TRAINING_ITERATION) AlgorithmAlgorithmConfig) DatasetReaderY@順parserdefault_reward default_itersdefault_timestepsreturncCs|durt}|jdtddd|jdddd |jd gd d d d|jdtddd|jdtddd|jdtddd|jdtddd|jdtddd|jdtddd|jddddd d|jd!td"d"d#gd$d%|jd&dd'd |jd(tdd)d|jd*tdgd+d,d%|jd-dd.d |jd/td0d1d|jd2tdd3d|jd4td5d6d|jd7tdd8d|jd9dd:d |jd;tddd|jd?tdd@d|jdAt|dBd|jdCt|dDd|jdEt|dFd|jdGddHd |jdIddJd |jdKtddLd|jdMtddNd|jdOtddPd|jdQtddRd|jdStddT|jdUddVd |jdWtddXd|S)YacAdds RLlib-typical (and common) examples scripts command line args to a parser. TODO (sven): This function should be used by most of our examples scripts, which already mostly have this logic in them (but written out). Args: parser: The parser to add the arguments to. If None, create a new one. default_reward: The default value for the --stop-reward option. default_iters: The default value for the --stop-iters option. default_timesteps: The default value for the --stop-timesteps option. Returns: The altered (or newly created) parser object. Nz--algoPPOz&The RLlib-registered algorithm to use.)typedefaulthelpz--enable-new-api-stack store_truezAWhether to use the `enable_rl_module_and_learner` config setting.)actionr1z --framework)tftf2torchr6zThe DL framework specifier.)choicesr0r1z--envz2The gym.Env identifier to run the experiment with.z--num-env-runnersz 0, will run as multi-agent with the environment simply cloned n times and each agent acting independently at every single timestep. The overall reward for this experiment is then the sum over all individual agents' rewards.z--evaluation-num-env-runnerszGThe number of evaluation (remote) EnvRunners to use for the experiment.z--evaluation-intervalz`Every how many iterations to run one round of evaluation. Use 0 (default) to disable evaluation.z--evaluation-durationcSs|dkr|St|S)Nauto)int)vr;N/home/ubuntu/.local/lib/python3.10/site-packages/ray/rllib/utils/test_utils.pyz/add_rllib_example_script_args.. zThe number of evaluation units to run each evaluation round. Use `--evaluation-duration-unit` to count either in 'episodes' or 'timesteps'. If 'auto', will run as many as possible during train pass (`--evaluation-parallel-to-training` must be set then).z--evaluation-duration-unitepisodes timestepszThe evaluation duration unit to count by. One of 'episodes' or 'timesteps'. This unit will be run `--evaluation-duration` times in each evaluation round. If `--evaluation-duration=auto`, this setting does not matter.)r/r0r7r1z!--evaluation-parallel-to-trainingzWhether to run evaluation parallel to training. This might help speed up your overall iteration time. Be aware that when using this option, your reported evaluation results are referring to one iteration before the current one.z--outputz\The output directory to write trajectories to, which are collected by the algo's EnvRunners.z --log-level)INFODEBUGWARNERRORz-The log-level to be used by the RLlib logger.z --no-tunezWhether to NOT use tune.Tuner(), but rather a simple for-loop calling `algo.train()` repeatedly until one of the stop criteria is met.z --num-sampleszMHow many (tune.Tuner.fit()) experiments to execute - if possible in parallel.z--max-concurrent-trialsz1How many (tune.Tuner) trials to run concurrently.z --verbosezBThe verbosity level for the `tune.Tuner()` running the experiment.z--checkpoint-freqzThe frequency (in training iterations) with which to create checkpoints. Note that if --wandb-key is provided, all checkpoints will automatically be uploaded to WandB.z--checkpoint-at-endzWhether to create a checkpoint at the very end of the experiment. Note that if --wandb-key is provided, all checkpoints will automatically be uploaded to WandB.z --wandb-keyz/The WandB API key to use for uploading results.z--wandb-projectzThe WandB project name to use.z--wandb-run-namezThe WandB run name to use.z --stop-rewardz0Reward at which the script should stop training.z --stop-itersz"The number of iterations to train.z--stop-timestepsz8The number of (environment sampling) timesteps to train.z --as-testzWhether this script should be run as a test. If set, --stop-reward must be achieved within --stop-timesteps AND --stop-iters, otherwise this script will throw an exception at the end.z--as-release-testzWhether this script should be run as a release test. If set, all that applies to the --as-test option is true, plus, a short JSON summary will be written into a results file whose location is given by the ENV variable `TEST_OUTPUT_JSON`.z--num-learnerszLThe number of Learners to use. If `None`, use the algorithm's default value.z--num-cpus-per-learnerzTThe number of CPUs per Learner to use. If `None`, use the algorithm's default value.z--num-gpus-per-learnerzThe number of GPUs per Learner to use. If `None` and there are enough GPUs for all required Learners (--num-learners), use a value of 1, otherwise 0.z#--num-aggregator-actors-per-learnerzaThe number of Aggregator actors to use per Learner. If `None`, use the algorithm's default value.z --num-cpus)r/r0z --local-modez,Init Ray in local mode for easier debugging.z --num-gpusz6The number of GPUs to use (only on the old API stack).)argparseArgumentParser add_argumentstrr9float)r)r*r+r,r;r;r<add_rllib_example_script_args:s         rMFc Cst|trJt|tsJdt|}|D]#\}}||vs*Jd|d|t|||||||d||q|rHJdt||d St|t tfrt|t tfs\Jdt |t |kspJdt |t |t |D]\} }t||| ||||dqtd St|t j tfr|durt|t|usJd |d |d d St|t|usJd |d |d d S|d us|d ust|tst|trt|tr|dur||ksJd |d|d d S||ksJd |d|d d St|dr|jtkst|jdst|trKzt j|||dur-Jd |d|d Wd StyJ} z|dur?| WYd } ~ d Sd } ~ wwtd urt|tjtjfrhtrd|}ntdt|tjtjfrtr||}n$t } | !|}t||||||dWd S1swYt"d urt|t"jr|#$}t|t"jr|#$}ddl%m&} t|| r|'}t|| r|'}|d ur |d ur z t jj(|||dWnty } z|dur| WYd } ~ d Sd } ~ ww|durJd |d|d d S|d ur'd}|d ur.d}z t jj)||||dWntyU} z|durJ| WYd } ~ d Sd } ~ ww|durfJd |d|d d S)ao Checks two structures (dict, tuple, list, np.array, float, int, etc..) for (almost) numeric identity. All numbers in the two structures have to match up to `decimal` digits after the floating point. Uses assertions. Args: x: The value to be compared (to the expectation: `y`). This may be a Tensor. y: The expected value to be compared to `x`. This must not be a tf-Tensor, but may be a tf/torch-Tensor. decimals: The number of digits after the floating point up to which all numeric values have to match. atol: Absolute tolerance of the difference between x and y (overrides `decimals` if given). rtol: Relative tolerance of the difference between x and y (overrides `decimals` if given). false: Whether to check that x and y are NOT the same. z2ERROR: If x is dict, y needs to be a dict as well!z ERROR: y does not have x's key='z'! y=)decimalsatolrtolfalsez3ERROR: y contains keys ({}) that are not in x! y={}z>ERROR: If x is tuple/list, y needs to be a tuple/list as well!z7ERROR: y does not have the same length as x ({} vs {})!Tz ERROR: x (z) is y ()!z ) is not y (Nz) is the same as y (z) is not the same as y (dtypez)rPrQ)* isinstancedictsetkeysitemscheckremoveformatlisttuplelen enumeratenpbool_boolrKr9hasattrrTobject startswithbytestestingassert_array_equalAssertionErrortf1TensorVariabler4executing_eagerlynumpy ValueErrorSessionrunr6detachcpuray.rllib.utils.metrics.statsrUpeekassert_almost_equalassert_allclose) xyrOrPrQrRy_keyskeyvalueiesessrUr;r;r<r\Ns      **""          "          r\csddlm}m|zttjWn ty&j Ynwj j fdd}fD]}|urit dd}|sNJ|j sXj}n |j jddd }t |d |}nj}d g|urud gngD]e}d D]`} |urddgndgD]R} tdd} dD]G} | rdgngdD];} tdtd|td|td| td| td| td| ||||| | | | | qqqq|qxq>dS)aTests different combinations of args for algorithm.compute_single_action. Args: algorithm: The Algorithm object to test. include_state: Whether to include the initial state of the Policy's Model in the `compute_single_action` call. include_prev_action_reward: Whether to include the prev-action and -reward in the `compute_single_action` call. Raises: ValueError: If anything unexpected happens. r)DEFAULT_POLICY_ID SampleBatchc sTi}|ur||d<|d<|} t|trt| dd} d} rN} | sNg} d} d| jvrN| jd| j| d7} d| jvs4rTnd} rZdnd} |dkr|usfJj | i}rw| |j <| |j <| r|j d d r| |d <nt| D] \} }||d| <qtd d |}jd|||d|}t|dtrt|d|d|df}tdd |}zjd|||d|}|sj dst||WntyYnw|j| | f| | ||||d|}d}| s |s |ur|\}}}|r;tt| t|D]\}}t|r.|j}nt|j}t||jq|durE|j d}|durO|j d}|dkr|kr|sx|sk|skttsxtd|d|dttr|s|j drtt |dkrtd|d|ddSdSdSdSdSdS)N full_fetch policy_id?r state_in_rF input_dictenable_rl_module_and_learnerFstate_incSs t|dSNr)rc expand_dimssr;r;r<r=%s z._test..)rexploretimesteprGcSs|dSrr;rr;r;r<r=2snoisy) prev_action prev_rewardrrunsquash_action clip_actionnormalize_actions clip_actionssinglezReturned action (z) of algorithm/policy z not in Env's action_space g.@zC should be in normalized space, but seems too large/small for that!r;)!samplerWr rcclipget_initial_stateview_requirementsappendspaceOBS PREV_ACTIONS PREV_REWARDSconfiggetrbtree map_structurecompute_actions_from_input_dictr_arraycompute_single_actionr\ TypeErrorzipflattenr4 is_tensorshapeas_listcontainsrranyabs)whatmethod_to_test obs_spacerrrunsquashr call_kwargsobsrr action_in reward_inrrinput_dict_batchedr3action2 state_out_sisosi_shaper action_space algorithminclude_prev_action_reward include_statemodelpidpolr;r<_tests                       z*check_compute_single_action.._testenv_runner_groupNcSs|jSN)observation_space)pr;r;r<r=sz-check_compute_single_action..)roriginal_spacerr)TFFTr()TFNzP--------------------------------------------------------------------------------zwhat=zmethod_to_test=zexplore=z full_fetch=z unsquash=zclip=)ray.rllib.policy.sample_batchrrnextiter env_runnerget_policies_to_train get_policyAttributeErrorpolicyrrgetattrlocal_env_runnerr for_policyrandomrandintprint)rrrrrr worker_setrrrrrrrr;rr<check_compute_single_actionsf   ~  rd max_stepsc Csddlm}t|}t|r!t||jdd|jddd}|\}}d\}}} d} |s^| sb| |krd||dd|||| |d} | | dd\}}}} }| d 7} |s`| sf| |ks6d Sd Sd Sd Sd Sd S) a[Checks whether the given policy can infer actions from an env with connectors. Args: policy: The policy to check. env_name: Name of the environment to check max_steps: The maximum number of steps to run the environment for. Raises: ValueError: If the policy cannot infer actions from the environment. r)local_policy_inferencerdim framestack)rr)FF)env_idagent_idrreward terminated truncatedinforFN) ray.rllib.utils.policyrgymmakerrrrresetstep) renv_namerrenvrrrrrts action_outr;r;r<check_inference_w_connectorss2     (rz/episode_return_mean tune_resultsztune.ResultGrid min_value evaluationmetriccCsg}|D]'\}}|s|dur(td||vr(||td|q|||qt|}||krCtd|d|dtd|d|ddS)akThrows an error if `min_reward` is not reached within tune_results. Checks the last iteration found in tune_results for its "episode_return_mean" value and compares it to `min_reward`. Args: tune_results: The tune.Tuner().fit() returned results object. min_reward: The min reward that must be reached. evaluation: If True, use `evaluation/env_runners/[metric]`, if False, use `env_runners/[metric]`, if None, use evaluation sampler results if available otherwise, use train sampler results. Raises: ValueError: If `min_reward` not reached. N/`` of not reached! reached! ok) get_dataframeiterrowsrrmaxrrr)rrrrrecorded_valuesrrow best_valuer;r;r<check_learning_achievedsrrresults upper_limit lower_limitcCsrddlm}ddlm}|d|}||vrdS||t}||kr(|ks7ntd|d|d|d |S) ayVerifies that the off-policy'ness of some update is within some range. Off-policy'ness is defined as the average (across n workers) diff between the number of gradient updates performed on the policy used for sampling vs the number of gradient updates that have been performed on the trained policy (usually the one on the local worker). Uses the published DIFF_NUM_GRAD_UPDATES_VS_SAMPLER_POLICY metric inside a training results dict and compares to the given bounds. Note: Only works with single-agent results thus far. Args: results: The training results dict. upper_limit: The upper limit to for the off_policy_ness value. lower_limit: The lower limit to for the off_policy_ness value. Returns: The off-policy'ness value (described above). Raises: AssertionError: If the value is out of bounds. rr LEARNER_INFOrNz`off_policy_ness` (z) is outside the given bounds (z - rS)rr$ray.rllib.utils.metrics.learner_infor rrl)rrr rr  learner_infooff_policy_nessr;r;r<check_off_policynesss"   r train_resultsc Csddlm}m}m}m}||||tdfD]}||vs%Jd|d|dqt|dts1Jdddlm }| d |dd ij }||}|s_t |dks_t |vs_Jdt d |d |D])\} } | d krlqc| D]\}} |d s~|drt| sJd| dqpqc|S)Checks proper structure of a Algorithm.train() returned dict. Args: train_results: The train results dict to check. Raises: AssertionError: If `train_results` doesn't have the proper structure or data in it. r)rFAULT_TOLERANCE_STATSLEARNER_RESULTSTIMERSr' ' not found in `train_results` (rS&`config` in results not a python dict!r$policiesz' not found in train_results['z']!__all_modules___min_max'key' value not a scalar ()ray.rllib.utils.metricsrrrrr"rWrX%ray.rllib.algorithms.algorithm_configr$update_from_dictis_multi_agentrarr[endswithrcisscalar) rrrrrr~r$r"learner_results module_idmodule_metricsrr;r;r<!check_train_results_new_api_stack(sJ    r(c Csddlm}ddlm}m}ddtdddd d d d d tf D]}||vs-Jd|d|dqdD]}||tvsEJd|d|tdq0t|dtsQJdddl m }| d|ddij }|d}||vstJd|dd|vst |vsJd|d||}|st|dks||vsJd|d|d|D]9\} } | dkrq| dkrq|| vr| |} n| } | D]\}} |ds|drt| sJd| dqq|S)rrr )r LEARNER_STATS_KEYrcustom_metricsriterations_since_restorenum_healthy_workersperftime_since_restoretime_this_iter_stimers time_total_srrrS) episode_len_meanepisode_reward_maxepisode_reward_meanepisode_reward_min hist_statspolicy_reward_maxpolicy_reward_meanpolicy_reward_min sampler_perfz4' not found in `train_results[ENV_RUNNER_RESULTS]` (rrrz)'learner' not in train_results['infos'] (num_steps_trainedz:'num_(env_)?steps_trained' not in train_results['infos'] (z2' not found in train_results['infos']['learner'] ( batch_count__all__min_max_r)rrr r r)rr"rWrXr r$r!r"rrar[rhrcr$) rrr r)r~r$r"rrr policy_stats learner_statsrr;r;r<check_train_resultsnsx      rBstop keep_ray_upc Cs|D]P\}}|}|dD]}|}t|tr#||vr#||}qd}|dur+qz t|sJ||krJtd|d|d|sFt WdSWqt yTYqwdS)aChecks stopping criteria on `ResultDict` Args: stop: Dictionary of stopping criteria. Each criterium is a mapping of a metric in the `ResultDict` of the algorithm to a certain criterium. results: An RLlib `ResultDict` containing all results from a training step. keep_ray_up: Optionally shutting down the runnin Ray instance. Returns: True, if any stopping criterium is fulfilled. Otherwise, False. rNzStop criterion (= ) fulfilled!TF) r[splitstriprWrXrcisnanrrayshutdownr)rCrrDr~ thresholdvalkr;r;r< should_stops*   rO)rCsuccess_metric trainabletune_callbacks keep_configrD schedulerprogress_reporter base_configr$argsrPrQrRrSc& s`|dur t} | }|jrd|_tj|jpd|jdd|dur6tdt |j tdt |j t |ji}|} |sG| |j|jdurQ| jdurQ| |j|js[| jddd|jdurg| j|jd|jdurs| j|jd| jr |jdur|jd krtd td d } |jdur|jn| jpd } |jpd | }|jdur|jnd | }| jd d |jdur| j|jd|j dur| j|j d|jdur| |kr| jd dn"| jd dn| |krtd|jd|jd| d| j|jd|j!dur | j|j!dn |jdur| j|jd |j"d kr-| j#|j$|j"|j%|j&|j'd|j(dur:| j)|j(d|j*durG| j+|j*d|j,r|jsS|jrUJ| -}t.|t |jD]}|/}t|vr|tt t0j1}t2d|d|ddt3|vrt|t3vr|t3tt }t2d|ddt2|4D]K\}}|}|5dD]}z||}Wqt6yd}Ynw|durt07|s||krt2d|d|d |st8|Sqqb|st8|S|pg}t9|d!rK|j:dust;tp3|j?@d"tABd#d"tC| j@}|DtEdE||dd$|jFrFd%|jFini| durr|jGd krrtHit d&d'd(t d)tdt d*id+d,| jIDd-} d.td?d@5}#ztef|"|#Wn$tgyd dAlhmi}$fdBd,|$jjD}%|%|"dC<tef|"|#YnwWdn 1swY|std9d:|dD|S)Fav Given an algorithm config and some command line args, runs an experiment. There are some constraints on what properties must be defined in `args`. It should ideally be generated via calling `args = add_rllib_example_script_args()`, which can be found in this very module here. The function sets up an Algorithm object from the given config (altered by the contents of `args`), then runs the Algorithm via Tune (or manually, if `args.no_tune` is set to True) using the stopping criteria in `stop`. At the end of the experiment, if `args.as_test` is True, checks, whether the Algorithm reached the `success_metric` (if None, use `env_runners/ episode_return_mean` with a minimum value of `args.stop_reward`). See https://github.com/ray-project/ray/tree/master/rllib/examples for an overview of all supported command line options. Args: base_config: The AlgorithmConfig object to use for this experiment. This base config will be automatically "extended" based on some of the provided `args`. For example, `args.num_env_runners` is used to set `config.num_env_runners`, etc.. args: A argparse.Namespace object, ideally returned by calling `args = add_rllib_example_script_args()`. It must have the following properties defined: `stop_iters`, `stop_reward`, `stop_timesteps`, `no_tune`, `verbose`, `checkpoint_freq`, `as_test`. Optionally, for WandB logging: `wandb_key`, `wandb_project`, `wandb_run_name`. stop: An optional dict mapping ResultDict key strings (using "/" in case of nesting, e.g. "env_runners/episode_return_mean" for referring to `result_dict['env_runners']['episode_return_mean']` to minimum values, reaching of which will stop the experiment). Default is: { "env_runners/episode_return_mean": args.stop_reward, "training_iteration": args.stop_iters, "num_env_steps_sampled_lifetime": args.stop_timesteps, } success_metric: Only relevant if `args.as_test` is True. A dict mapping a single(!) ResultDict key string (using "/" in case of nesting, e.g. "env_runners/episode_return_mean" for referring to `result_dict['env_runners']['episode_return_mean']` to a single(!) minimum value to be reached in order for the experiment to count as successful. If `args.as_test` is True AND this `success_metric` is not reached with the bounds defined by `stop`, will raise an Exception. trainable: The Trainable sub-class to run in the tune.Tuner. If None (default), use the registered RLlib Algorithm class specified by args.algo. tune_callbacks: A list of Tune callbacks to configure with the tune.Tuner. In case `args.wandb_key` is provided, appends a WandB logger to this list. keep_config: Set this to True, if you don't want this utility to change the given `base_config` in any way and leave it as-is. This is helpful for those example scripts which demonstrate how to set config settings that are otherwise taken care of automatically in this function (e.g. `num_env_runners`). Returns: The last ResultDict from a --no-tune run OR the tune.Tuner.fit() results. NT)num_cpus local_modeignore_reinit_errorrF)r"enable_env_runner_and_connector_v2)num_env_runners)num_envs_per_env_runnerrz--num-gpus is not supported on the new API stack! To train on GPUs, use the command line options `--num-gpus-per-learner=1` and `--num-learners=[your number of available GPUs]`, instead.GPUrFnum_gpus) num_learners)!num_aggregator_actors_per_learnernum_gpus_per_learnerz0You are running your script with --num-learners=z and --num-gpus-per-learner=z, but your cluster only has z GPUs!)num_cpus_per_learner)evaluation_num_env_runnersevaluation_intervalevaluation_durationevaluation_duration_unitevaluation_parallel_to_training) log_level)outputziter=z R=)endz R(eval)=zStop criterium (rErF wandb_key-z\W+)api_keyprojectupload_checkpointsnamerr1ztotal time (s)rzcombined returncSs"i|] }td|d|qS)z/module_episode_returns_mean/zreturn )r).0rr;r;r< s  z7run_rllib_example_script_experiment..)metric_columns0RAY_AIR_NEW_OUTPUT)checkpoint_frequencycheckpoint_at_end)rCverbose callbackscheckpoint_configrU) num_samplesmax_concurrent_trialsrT) param_space run_config tune_configz;Running the example script resulted in one or more errors! cSsg|] }|jdjdqSrrG)rW)rurr;r;r< Jsz7run_rllib_example_script_experiment..c3s|] \}}|VqdSrr;)rurr)success_metric_keyr;r< as  z6run_rllib_example_script_experiment..rrrr) time_taken trial_states last_updatestatspassed not_passedfailuresTEST_OUTPUT_JSONz/tmp/learning_test.jsonwt)r#csi|] }|vr||qSr;r;)rurN)rr;r<rvsrrr;)krM parse_argsas_release_testas_testrJinitrXrYrr stop_rewardrstop_timestepsr" stop_iters frameworkr environmentenable_new_api_stack api_stackr\ env_runnersr]rr`rrcluster_resourcesrrard resourceslearnersrbrergrrfrhrirjrk debuggingrl offline_datano_tunebuildrangetrainrcnanrrr[rGKeyErrorrIrKrfrorosenviron wandb_projectalgolowerresubrKrrwandb_run_name num_agentsr!rtimerTuner algo_class RunConfigr|CheckpointConfigcheckpoint_freqr{ TuneConfigrrfiterrors RuntimeErrorrrrrr_experiment_analysistrials last_resultpoprLstatusopenjsondump Exceptionray.rllib.algorithms.algorithmr#_progress_metrics)&rVrWrCrPrQrRrSrDrTrUr)rnum_gpus_availablenum_actual_learnersnum_gpus_requestednum_gpus_needed_if_availablerrr mean_returnRevalr~rLrMrNrorr start_timer test_passedtry_itsuccess_metric_valuertrial json_summaryfr#simplified_statsr;)rrr<#run_rllib_example_script_experiment sH                       $                   rc Csddlm}m}t|t|us Jdtt|tt|d dd}t||ur2|||dSt||urs|j|jks@Jt}|j D]\}}|||j||| |qHt|j }| |} | rqJd| ddSt d tt|) aCheck if both batches are (almost) identical. For MultiAgentBatches, the step count and individual policy's SampleBatches are checked for identity. For SampleBatches, identity is checked as the almost numerical key-value-pair identity between batches with ray.rllib.utils.test_utils.check(). unroll_id is compared only if both batches have an unroll_id. Args: batch1: Batch to compare against batch2 batch2: Batch to compare against batch1 r)MultiAgentBatchrz.Input batches are of different types {} and {}Nc Ss|dd}|dd}|dur|dur||ksJt}|D]\}}|dkr*q!t|||||q!t|}|d||} |rU| rSJd|| dS| r^Jd| dS)N unroll_idz]SampleBatches for policy with ID {} don't share information on the following information: {}zGSampleBatches don't share information on the following information: {}) rrYr[r\addrZdiscardsymmetric_differencer^) _batch1_batch2 _policy_id unroll_id_1 unroll_id_2 batch1_keysrNr: batch2_keys _differencer;r;r<check_sample_batchess,       z.check_same_batch..check_sample_batchesz:MultiAgentBatches don't share the following information: .zUnsupported batch type r)rrrr/r^rKcountrYpolicy_batchesr[rrZrrr) batch1batch2rrr batch1_idsr policy_batch batch2_ids differencer;r;r<check_same_batchs4  #     rrF)training_iterationrr# algo_config fw_kwargsrc Csdddlm}ddlm}t|i}dD]}|jddj|ddjtt j d d d j tt j d d d }t d |jd|dt dt|tj||tj|ddd}|j}tj||tj|ddd} | j} t|td| td|jrt|d||| d||qt|d||d| d||dqdS)aCheck if the algorithm is reproducible across different testing conditions: frameworks: all input frameworks num_gpus: int(os.environ.get("RLLIB_NUM_GPUS", "0")) num_workers: 0 (only local workers) or 4 ((1) local workers + (4) remote workers) num_envs_per_env_runner: 2 Args: algo_class: Algorithm class to test. algo_config: Base config to use for the algorithm. fw_kwargs: Framework iterator keyword arguments. training_iteration: Number of training iterations to run. Returns: None Raises: It raises an AssertionError if the algorithm is not reproducible. rr r r*)seedrG)r\r]RLLIB_NUM_GPUSrxrcr_zTesting reproducibility of z with z workersz /// configrF)rCr|)rrr6rrAN)rrr r r"rrrr9rrrrr__name__pprintto_dictrrrrget_best_resultmetricsr\rr) rrrrrr  stop_dict num_workersresults1results2r;r;r<check_reproducibiltysf          r batch_sizer%c Cszddlm}ddlm}ddlm}m}d}d|d}||jd|d \}}||j|d jd d dd } ||| } | S)zReturns a DatasetReader for the cartpole dataset. Args: batch_size: The batch size to use for the reader. Returns: A rllib DatasetReader for the cartpole dataset. rr) IOContext)r%get_dataset_and_shardsztests/data/cartpole/large.jsonr)r^pathsdataset)input_ input_config)train_batch_sizeT)actions_in_input_normalized)r worker_index) ray.rllib.algorithmsr$ray.rllib.offliner ray.rllib.offline.dataset_readerr%rrtraining) rr$rr%rpathrrrioctxreaderr;r;r<get_cartpole_dataset_reader7s$    rc@s4eZdZdZddZddedefd d Zd d Zd S) ModelCheckeraHelper class to compare architecturally identical Models across frameworks. Holds a ModelConfig, such that individual models can be added simply via their framework string (by building them with config.build(framework=...). A call to `check()` forces all added models to be compared in terms of their number of trainable and non-trainable parameters, as well as, their computation results given a common weights structure and values and identical inputs to the models. cCs,||_i|_i|_tjdd|_i|_dS)Ng{Gzg{Gz?)r param_counts output_valuesrcruniformrandom_fill_input_valuemodels)selfrr;r;r<__init__as  zModelChecker.__init__r6TFrr-c Cs|jj|d}|j|<tdg|rdgngt|jj|j}|r(tj |i}|r5t dd||tj <|dkrCddl m}||}||}|j|jfd||}||j|<|dkrkt d d||j|<|St d d||j|<|S) z+Builds a new Model for the given framework.)rrFcSstjdgt|dS)NrF)r)rczerosr_rr;r;r<r=~sz"ModelChecker.add..r6r)convert_to_torch_tensor)value_sequencecSs|dur |SdSr)rurqrr;r;r<r=scSs|dur|SdSr)rqrr;r;r<r=r>)rrrrcfullr_ input_dimsrrrrrSTATE_INray.rllib.utils.torch_utilsr_set_to_dummy_weightsget_num_parametersrr) rrrstaterinputsroutputscomparable_outputsr;r;r<rqs8     zModelChecker.addcCsZtt|j}|jD] }t||j|q|jD] }t||j|ddqdS)zECompares all added Models with each other and possibly raises errors.gMb@?)rPN)rrrrZrvaluesr\r)rmain_keycr:r;r;r<r\s zModelChecker.checkN)r6TF) r __module__ __qualname____doc__rrKrrr\r;r;r;r<rVs   .ralgrcCs0g}tdD] }|t||qt|S)aQReturns the mean action computed by the given algorithm. Note: This makes calls to `Algorithm.compute_single_action` Args: alg: The constructed algorithm to run inference on. obs: The observation to compute the action for. Returns: The mean action computed by the algorithm over 5000 samples. i)rrrLrrcmean)r*routrr;r;r<_get_mean_action_from_algorithms  r-Trr check_bounds frameworksuse_gpuc sddlm}ddlmddlmddlmtdt ddd t j d t dd d t j d t gd ttdtd t ddd t j d gttd t dddt j d tdttdtd giddtgdtdt ddd t j d t dddt j d t dddt j d ttdt ddd t j d gttdt ddd t j d ddgdddgd}}d|d<||d<f dd } |sd!}t| } | j|rd"ndd#} D]} |} t| ||| | qوD]} |} t| ||| | qd$S)%aDChecks whether the given algorithm supports different action and obs spaces. Performs the checks by constructing an rllib algorithm from the config and checking to see that the model inside the policy is the correct one given the action and obs spaces. For example if the action space is discrete and the obs space is an image, then the model should be a vision network with a categorical action distribution. Args: alg: The name of the algorithm to test. config: The config to use for the algorithm. train: Whether to train the algorithm for a few iterations. check_bounds: Whether to check the bounds of the action space. frameworks: The frameworks to test the algorithm with. use_gpu: Whether to check support for training on a gpu. r) RandomEnv)ComplexInputNetwork)FullyConnectedNetwork) VisionNetworkrNrr)rN)rT)rGr5)rFrGr5rGrFa) action_choice parametersyet_another_nested_dict)discrete continuous int_actions multidiscreter`rX)r5r?r?)rNrN)Tr@rFr?)taskposition) multi_binaryr<r=vector2dimager`rX)rCr<r=rEr`rXr<r=rErkrc s|}||jr'|vrtd|dS|vr'td|dS|d}|}|}td||||t}|t t ||t dddt j dddd d } z| } Wn?tjjy} z't| jd kr{t| jd tr{d } nt| jd jd trd } nWYd} ~ nKd} ~ wtyd } Yn=w|dvr|dvrt| jsJn|dkrt| jsJn|dkrt| jfsJr| | td| t|dS)Nz1Skipping PPO test with RLModules for obs space {}z4Skipping PPO test with RLModules for action space {}rz7=== Testing {} (fw={}) action_space={} obs_space={} ===rr;)rrT)rr reward_space p_terminatedcheck_action_bounds) env_configokrG unsupportedr)SACr.)atarirEr=rDzTest: {}, ran in {}s)copyvalidaterloggerwarningr^rrr!rXr rcfloat32rrJ exceptions RayActorErrorrarWrWr rrrrC) r*ra_nameo_name config_copyfwrrt0statrr TorchComplexNet TorchFCNetTorchVisionNetaction_spaces_to_testr.observation_spaces_to_test rlmodule_supported_action_spaces%rlmodule_supported_observation_spacesrr;r< _do_checksz   z)check_supported_spaces.._do_check)r5r4r6rFr_N)*ray.rllib.examples.envs.classes.random_envr1(ray.rllib.models.torch.complex_input_netr2ray.rllib.models.torch.fcnetr3 ray.rllib.models.torch.visionnetr4r r rcrRint32r GymTupleGymDictr rJremoteoptionsrZr) r*rrr.r/r0r1default_observation_spacedefault_action_spacerc_do_check_remoterUrVr;r[r<check_supported_spacessb       I   rp)Nr&r'r()rNNNF)FF)r)r)Fr)r-Nr7)TFNF)drHrloggingrrrrrtypingrrrrrrrr gymnasiumrgymnasium.spacesr r r r rjrirqrcrrJrray.air.integrations.wandbrrray.rllib.corerr%ray.rllib.env.wrappers.atari_wrappersrrray.rllib.utils.annotationsrray.rllib.utils.frameworkrrrrrrrrrrray.rllib.utils.typingrray.rllib.utils.errorr ray.tuner!ray.tune.resultr"rr#r$r r%jaxrrmr4tfvr6 getLoggerrrPrIrLr9rMr\rrrerKrrr(rBrO Namespace result_grid ResultGridrrrrrndarrayr-rpr;r;r;r<s8(                  H. ( 3Fj  3   S  \X