o ci'@sddlZddlZddlmZmZmZmZmZddlZ ddl Z ddl Z ddl m Z ddlmZddlmZddlmZddlmZmZddlmZdd lmZdd lmZmZdd lmZm Z erfdd l!m"Z"e#d Z$e\Z%Z&Z'dZ(dZ)eddGdddeej*dZ+dS)N)AnyDictOptionalTuple TYPE_CHECKING)COMPONENT_RL_MODULE)FaultAwareApply)update_global_seed_if_necessary) try_import_tf)ENV_RESET_TIMERENV_STEP_TIMER) MetricsLogger)convert_to_torch_tensor) StateDict TensorType) PublicAPI DeveloperAPI)AlgorithmConfigz ray.rllibenv_reset_failureenv_step_failurealpha) stabilityc seZdZdZd#fdd ZejddZdd Zd d Z ejd e fd dZ d e fddZ e d eejeeffddZejd eeeejejfffddZd$ddZd$ddZddddeedeed ee e ffddZdd Zd efd!d"Z Z!S)% EnvRunneraBase class for distributed RL-style data collection from an environment. The EnvRunner API's core functionalities can be summarized as: - Gets configured via passing a AlgorithmConfig object to the constructor. Normally, subclasses of EnvRunner then construct their own environment (possibly vectorized) copies and RLModules/Policies and use the latter to step through the environment in order to collect training data. - Clients of EnvRunner can use the `sample()` method to collect data for training from the environment(s). - EnvRunner offers parallelism via creating n remote Ray Actors based on this class. Use `ray.remote([resources])(EnvRunner)` method to create the corresponding Ray remote class. Then instantiate n Actors using the Ray `[ctor].remote(...)` syntax. - EnvRunner clients can get information about the server/node on which the individual Actors are running. configrc s|jdd|_|d|_|d|jj|_d|_t|_t t r5|jj dks-|j r5t s5t d|_|jjdurPt|jj|jpFdd|jj|_t|jj |jd dS) zInitializes an EnvRunner instance. Args: config: The AlgorithmConfig to use to setup this EnvRunner. **kwargs: Forward compatibility kwargs. F) copy_frozen worker_index num_workersNtf2rg.A) frameworkseed)copyrgetrnum_env_runnersrenvr metricssuper__init__tf1 framework_strenable_tf1_exec_eagerlyexecuting_eagerlyenable_eager_execution_seedrint in_evaluationr )selfrkwargs __class__L/home/ubuntu/.local/lib/python3.10/site-packages/ray/rllib/env/env_runner.pyr&2s2     zEnvRunner.__init__cCdS)aHChecks that self.__init__() has been completed properly. Useful in case an `EnvRunner` is run as @ray.remote (Actor) and the owner would like to make sure the Ray Actor has been properly initialized. Raises: AssertionError: If the EnvRunner Actor has NOT been properly initialized. Nr3r/r3r3r4assert_healthy`zEnvRunner.assert_healthycCr5)aCreates the RL environment for this EnvRunner and assigns it to `self.env`. Note that users should be able to change the EnvRunner's config (e.g. change `self.config.env_config`) and then call this method to create new environments with the updated configuration. It should also be called after a failure of an earlier env in order to clean up the existing env (for example `close()` it), re-create a new one, and then continue sampling with that new env. Nr3r6r3r3r4make_envls zEnvRunner.make_envcCr5)a(Creates the RLModule for this EnvRunner and assigns it to `self.module`. Note that users should be able to change the EnvRunner's config (e.g. change `self.config.rl_module_spec`) and then call this method to create a new RLModule with the updated configuration. Nr3r6r3r3r4 make_moduleyszEnvRunner.make_modulereturncKr5)a`Returns experiences (of any form) sampled from this EnvRunner. The exact nature and size of collected data are defined via the EnvRunner's config and may be overridden by the given arguments. Args: **kwargs: Forward compatibility kwargs. Returns: The collected experience in any form. Nr3)r/r0r3r3r4sampler8zEnvRunner.samplecCr5)zReturns metrics (in any form) of the thus far collected, completed episodes. Returns: Metrics of any form. Nr3r6r3r3r4 get_metricszEnvRunner.get_metricscCs,|}|jtd}|}t|||fS)a Convenience method for fast, async algorithms. Use this in Algorithms that need to sample Episode lists as ray.ObjectRef, but also require (in the same remote call) the metrics and the EnvRunner states, except for the module weights. )not_components)r< get_staterr=rayput)r/ _episodes_connector_states_metricsr3r3r4sample_get_state_and_metricss z&EnvRunner.sample_get_state_and_metricscCr5)zFReturns a dict mapping ModuleIDs to 2-tuples of obs- and action space.Nr3r6r3r3r4 get_spacesr8zEnvRunner.get_spacesNcCr5)zReleases all resources used by this EnvRunner. For example, when using a gym.Env in this EnvRunner, you should make sure that its `close()` method is called. Nr3r6r3r3r4stopr>zEnvRunner.stopcCr5)z:If this Actor is deleted, clears all resources used by it.Nr3r6r3r3r4__del__szEnvRunner.__del__roptionsrrKc Csz%|jt|jj||d\}}Wdn1swY||fWStyS}z"|jjrMt d|j d| |j ||dWYd}~S|d}~ww)aTries resetting the env and - if an error occurs - handles it gracefully. Args: seed: An optional seed (int) to be passed to the Env.reset() call. options: An optional options-dict to be passed to the Env.reset() call. Returns: The results of calling `Env.reset()`, which is a tuple of observations and info dicts. Raises: Exception: In case `config.restart_failed_sub_environments` is False and `Env.reset()` resulted in an error. rJNz?Resetting the env resulted in an error! The original error is: r) r$log_timer r#reset Exceptionrrestart_failed_sub_environmentslogger exceptionargsr9_try_env_reset)r/rrKobsinfoser3r3r4rSs" zEnvRunner._try_env_resetc Csz!|jt|j|}Wd|WS1swY|WStyJ}z|jjrDt d|j d| t WYd}~S|d}~ww)zHTries stepping the env and - if an error orrurs - handles it gracefully.Nz>Stepping the env resulted in an error! The original error is: r) r$rLr r#steprNrrOrPrQrRr9ENV_STEP_FAILURE)r/actionsresultsrVr3r3r4 _try_env_steps& zEnvRunner._try_env_stepcCs"|jjdkr t|Sttj|S)z0Converts structs to a framework-specific tensor.torch)rr(rtree map_structuretfconvert_to_tensor)r/structr3r3r4_convert_to_tensors zEnvRunner._convert_to_tensor)rr)r;N)"__name__ __module__ __qualname____doc__r&abcabstractmethodr7r9r:rr<r=rrrA ObjectRefrrFrstrgymSpacerGrHrIrr-dictrSr[rrb __classcell__r3r3r1r4r s:.     $    )r) metaclass),rgloggingtypingrrrrr gymnasiumrkr]rAray.rllib.corerray.rllib.utils.actor_managerrray.rllib.utils.debugr ray.rllib.utils.frameworkr ray.rllib.utils.metricsr r &ray.rllib.utils.metrics.metrics_loggerr ray.rllib.utils.torch_utilsrray.rllib.utils.typingrrray.util.annotationsrr%ray.rllib.algorithms.algorithm_configr getLoggerrPr'r__ENV_RESET_FAILURErXABCMetarr3r3r3r4s.