o cil@s ddlZddlmZmZmZmZmZmZddlm Z m Z ddl m Z ddl mZddlmZddlmZddlmZdd lmZdd lmZdd lmZdd lmZdd lmZmZddl m!Z!m"Z"ddl#m$Z$ddl%m&Z&m'Z'e!\Z(Z)Z*e"Z+e,e-Z.Gddde Z/Gddde Z0dS)N)AnyDictOptionalTupleTypeUnion)AlgorithmConfig NotProvided)DQN) SACTFPolicy)"AddObservationsFromEpisodesToBatch)+AddNextObservationsFromEpisodesToTrainBatch)Learner) RLModuleSpec)Policy) deep_update)override)DEPRECATED_VALUEdeprecation_warning) try_import_tftry_import_tfp)EpisodeReplayBuffer)LearningRateOrScheduleRLModuleSpecTypec,seZdZdZd,fdd Zeeeeeeeeeeeeeeeeeeeeeeddee dee e e fdee e e fd ee d ee d eee e fd eeeeeeffd ee dee e e fdee dee dee dee e e fdeedeedeedeedee dee deeddf*fddZeed-fdd Zeed.d edefd!d"Zeedefd#d$Zeedeed%e ffd&d'Zee d,fd(d) Zefd*d+ZZS)/ SACConfiga Defines a configuration class from which an SAC Algorithm can be built. .. testcode:: config = ( SACConfig() .environment("Pendulum-v1") .env_runners(num_env_runners=1) .training( gamma=0.9, actor_lr=0.001, critic_lr=0.002, train_batch_size_per_learner=32, ) ) # Build the SAC algo object from the config and run 1 training iteration. algo = config.build() algo.train() Ncsddi|_tj|p tdd|_ddgdgddid|_ddgdgddid|_d|_d |_d |_ d |_ d |_ d t dddd|_ d|_d|_dddd|_d|_d|_d|_d|_d|_d|_d |_d|_d|_d|_d |_d|_d|_d|_t|_ t|_!dS)NtypeStochasticSampling) algo_classTrelu) fcnet_hiddensfcnet_activationpost_fcnet_hiddenspost_fcnet_activation custom_modelcustom_model_configFg{Gzt?g?autoPrioritizedEpisodeReplayBufferg.Ag333333?g?)rcapacityalphabetaga2U0*3?)actor_learning_ratecritic_learning_rateentropy_learning_rategiUMu>rid)"exploration_configsuper__init__SACtwin_qq_model_configpolicy_model_config clip_actionstau initial_alphatarget_entropyn_stepintreplay_buffer_configstore_buffer_in_checkpointstraining_intensity optimizationactor_lr critic_lralpha_lrlr grad_cliptarget_network_update_freqrollout_fragment_lengthtrain_batch_size_per_learnertrain_batch_size(num_steps_sampled_before_learning_startsmin_time_s_per_iteration"min_sample_timesteps_per_iteration_deterministic_loss_use_beta_distributionruse_state_preprocessorworker_side_prioritization)selfr __class__P/home/ubuntu/.local/lib/python3.10/site-packages/ray/rllib/algorithms/sac/sac.pyr22sd    zSACConfig.__init__)r4r5r6r8r9r:r;r>r=r?r7rEoptimization_configrArBrCrFrMrNrJr4r5r6r8r9r:r;r>r=r?r7rErVrArBrCrFrMrNrJreturnc s\tjdi||tur||_|tur|j||tur$|j||tur+||_|tur2||_|tur9||_ |tur@||_ |turG||_ | tur_t d|j id| iddgdg}|d|_ | turf| |_| turm| |_| turt| |_| tur{| |_|tur||_|tur||_|tur||_|tur||_|tur||_|tur||_|tur||_|S)u(Sets the training related configuration. Args: twin_q: Use two Q-networks (instead of one) for action-value estimation. Note: Each Q-network will have its own target network. q_model_config: Model configs for the Q network(s). These will override MODEL_DEFAULTS. This is treated just as the top-level `model` dict in setting up the Q-network(s) (2 if twin_q=True). That means, you can do for different observation spaces: `obs=Box(1D)` -> `Tuple(Box(1D) + Action)` -> `concat` -> `post_fcnet` obs=Box(3D) -> Tuple(Box(3D) + Action) -> vision-net -> concat w/ action -> post_fcnet obs=Tuple(Box(1D), Box(3D)) -> Tuple(Box(1D), Box(3D), Action) -> vision-net -> concat w/ Box(1D) and action -> post_fcnet You can also have SAC use your custom_model as Q-model(s), by simply specifying the `custom_model` sub-key in below dict (just like you would do in the top-level `model` dict. policy_model_config: Model options for the policy function (see `q_model_config` above for details). The difference to `q_model_config` above is that no action concat'ing is performed before the post_fcnet stack. tau: Update the target by au * policy + (1- au) * target_policy. initial_alpha: Initial value to use for the entropy weight alpha. target_entropy: Target entropy lower bound. If "auto", will be set to `-|A|` (e.g. -2.0 for Discrete(2), -3.0 for Box(shape=(3,))). This is the inverse of reward scale, and will be optimized automatically. n_step: N-step target updates. If >1, sars' tuples in trajectories will be postprocessed to become sa[discounted sum of R][s t+n] tuples. An integer will be interpreted as a fixed n-step value. If a tuple of 2 ints is provided here, the n-step value will be drawn for each sample(!) in the train batch from a uniform distribution over the closed interval defined by `[n_step[0], n_step[1]]`. store_buffer_in_checkpoints: Set this to True, if you want the contents of your buffer(s) to be stored in any saved checkpoints as well. Warnings will be created if: - This is True AND restoring from a checkpoint that contains no buffer data. - This is False AND restoring from a checkpoint that does contain buffer data. replay_buffer_config: Replay buffer config. Examples: { "_enable_replay_buffer_api": True, "type": "MultiAgentReplayBuffer", "capacity": 50000, "replay_batch_size": 32, "replay_sequence_length": 1, } - OR - { "_enable_replay_buffer_api": True, "type": "MultiAgentPrioritizedReplayBuffer", "capacity": 50000, "prioritized_replay_alpha": 0.6, "prioritized_replay_beta": 0.4, "prioritized_replay_eps": 1e-6, "replay_sequence_length": 1, } - Where - prioritized_replay_alpha: Alpha parameter controls the degree of prioritization in the buffer. In other words, when a buffer sample has a higher temporal-difference error, with how much more probability should it drawn to use to update the parametrized Q-network. 0.0 corresponds to uniform probability. Setting much above 1.0 may quickly result as the sampling distribution could become heavily “pointy” with low entropy. prioritized_replay_beta: Beta parameter controls the degree of importance sampling which suppresses the influence of gradient updates from samples that have higher probability of being sampled via alpha parameter and the temporal-difference error. prioritized_replay_eps: Epsilon parameter sets the baseline probability for sampling so that when the temporal-difference error of a sample is zero, there is still a chance of drawing the sample. training_intensity: The intensity with which to update the model (vs collecting samples from the env). If None, uses "natural" values of: `train_batch_size` / (`rollout_fragment_length` x `num_env_runners` x `num_envs_per_env_runner`). If not None, will make sure that the ratio between timesteps inserted into and sampled from th buffer matches the given values. Example: training_intensity=1000.0 train_batch_size=250 rollout_fragment_length=1 num_env_runners=1 (or 0) num_envs_per_env_runner=1 -> natural value = 250 / 1 = 250.0 -> will make sure that replay+train op will be executed 4x asoften as rollout+insert op (4 * 250 = 1000). See: rllib/algorithms/dqn/dqn.py::calculate_rr_weights for further details. clip_actions: Whether to clip actions. If actions are already normalized, this should be set to False. grad_clip: If not None, clip gradients during optimization at this value. optimization_config: Config dict for optimization. Set the supported keys `actor_learning_rate`, `critic_learning_rate`, and `entropy_learning_rate` in here. actor_lr: The learning rate (float) or learning rate schedule for the policy in the format of [[timestep, lr-value], [timestep, lr-value], ...] In case of a schedule, intermediary timesteps will be assigned to linearly interpolated learning rate values. A schedule config's first entry must start with timestep 0, i.e.: [[0, initial_value], [...]]. Note: It is common practice (two-timescale approach) to use a smaller learning rate for the policy than for the critic to ensure that the critic gives adequate values for improving the policy. Note: If you require a) more than one optimizer (per RLModule), b) optimizer types that are not Adam, c) a learning rate schedule that is not a linearly interpolated, piecewise schedule as described above, or d) specifying c'tor arguments of the optimizer that are not the learning rate (e.g. Adam's epsilon), then you must override your Learner's `configure_optimizer_for_module()` method and handle lr-scheduling yourself. The default value is 3e-5, one decimal less than the respective learning rate of the critic (see `critic_lr`). critic_lr: The learning rate (float) or learning rate schedule for the critic in the format of [[timestep, lr-value], [timestep, lr-value], ...] In case of a schedule, intermediary timesteps will be assigned to linearly interpolated learning rate values. A schedule config's first entry must start with timestep 0, i.e.: [[0, initial_value], [...]]. Note: It is common practice (two-timescale approach) to use a smaller learning rate for the policy than for the critic to ensure that the critic gives adequate values for improving the policy. Note: If you require a) more than one optimizer (per RLModule), b) optimizer types that are not Adam, c) a learning rate schedule that is not a linearly interpolated, piecewise schedule as described above, or d) specifying c'tor arguments of the optimizer that are not the learning rate (e.g. Adam's epsilon), then you must override your Learner's `configure_optimizer_for_module()` method and handle lr-scheduling yourself. The default value is 3e-4, one decimal higher than the respective learning rate of the actor (policy) (see `actor_lr`). alpha_lr: The learning rate (float) or learning rate schedule for the hyperparameter alpha in the format of [[timestep, lr-value], [timestep, lr-value], ...] In case of a schedule, intermediary timesteps will be assigned to linearly interpolated learning rate values. A schedule config's first entry must start with timestep 0, i.e.: [[0, initial_value], [...]]. Note: If you require a) more than one optimizer (per RLModule), b) optimizer types that are not Adam, c) a learning rate schedule that is not a linearly interpolated, piecewise schedule as described above, or d) specifying c'tor arguments of the optimizer that are not the learning rate (e.g. Adam's epsilon), then you must override your Learner's `configure_optimizer_for_module()` method and handle lr-scheduling yourself. The default value is 3e-4, identical to the critic learning rate (`lr`). target_network_update_freq: Update the target network every `target_network_update_freq` steps. _deterministic_loss: Whether the loss should be calculated deterministically (w/o the stochastic action sampling step). True only useful for continuous actions and for debugging. _use_beta_distribution: Use a Beta-distribution instead of a `SquashedGaussian` for bounded, continuous action spaces (not recommended; for debugging only). Returns: This updated AlgorithmConfig object. r=FNrT)r1trainingr r4r5updater6r8r9r:r;r>rr=r?r7rEr@rArBrCrFrMrNrJ)rQr4r5r6r8r9r:r;r>r=r?r7rErVrArBrCrFrMrNrJkwargsnew_replay_buffer_configrRrTrUrXsf<   zSACConfig.trainingc stt|jtr|jd}n|j}|js8|jdkr8|j|kr8td|jd|jd|jd|jdd |jt krFt dd d t |_|j durT|j d krTtd |j d vrpt durptdtretjnddtdd|jr|jddvr|jrt|jtst|jtrt|jdtr|jdkr|jstd|jst|jdtrd|jdvst|jdtrt|jdtrtd|jr|jdurtdtddSdS)Nr'r&z Your `rollout_fragment_length` (z') is smaller than needed for `n_step` (zB)! If `n_step` is an integer try setting `rollout_fragment_length=z@`. If `n_step` is a tuple, try setting `rollout_fragment_length=z`.z config['use_state_preprocessor']F)olderrorgz `grad_clip` value must be > 0.0!)tftf2zYou need `tensorflow_probability` in order to run SAC! Install it via `pip install tensorflow_probability`. Your tf.__version__=z5.Trying to import tfp results in the following error:T)r]r)rr(MultiAgentEpisodeReplayBuffer(MultiAgentPrioritizedEpisodeReplayBufferrsamplerz[When using the new `EnvRunner API` the replay buffer must be of type `EpisodeReplayBuffer`.EpisodeadWhen using the old API stack the replay buffer must not be of type `EpisodeReplayBuffer`! We suggest you use the following config to run SAC on the old API stack: `config.training(replay_buffer_config={'type': 'MultiAgentPrioritizedReplayBuffer', 'prioritized_replay_alpha': [alpha], 'prioritized_replay_beta': [beta], 'prioritized_replay_eps': [eps], })`.zBasic learning rate parameter `lr` is not `None`. For SAC use the specific learning rate parameters `actor_lr`, `critic_lr` and `alpha_lr`, for the actor, critic, and the hyperparameter `alpha`, respectively and set `config.lr` to None.aaYou are running SAC on the new API stack! This is the new default behavior for this algorithm. If you don't want to use the new API stack, set `config.api_stack(enable_rl_module_and_learner=False, enable_env_runner_and_connector_v2=False)`. For a detailed migration guide, see here: https://docs.ray.io/en/master/rllib/new-api-stack-migration-guide.html)r1validate isinstancer;tuple in_evaluationrG ValueErrorrOrrrE frameworktfploggerwarningr^ __version__r"enable_env_runner_and_connector_v2r=input_strlistenable_rl_module_and_learnerr issubclassrrD)rQmin_rollout_fragment_lengthrRrTrUrds             zSACConfig.validater worker_indexcCs0|jdkrt|jttfr|jdS|jS|jS)Nr&r')rGrer;rfrq)rQrurTrTrUget_rollout_fragment_lengths  z%SACConfig.get_rollout_fragment_lengthcCs2|jdkrddlm}t|dStd|jd)Ntorchr)DefaultSACTorchRLModule) module_classThe framework  is not supported. Use `torch`.) framework_str:ray.rllib.algorithms.sac.torch.default_sac_torch_rl_modulerxrrh)rQrxrTrTrUget_default_rl_module_specs    z$SACConfig.get_default_rl_module_specrcCs,|jdkr ddlm}|Std|jd)Nrwr)SACTorchLearnerrzr{)r|0ray.rllib.algorithms.sac.torch.sac_torch_learnerrrh)rQrrTrTrUget_default_learner_classs   z#SACConfig.get_default_learner_classcs$tj|||d}|tt|S)N)input_observation_spaceinput_action_spacedevice)r1build_learner_connector insert_afterr r )rQrrrpipelinerRrTrUrsz!SACConfig.build_learner_connectorcstjd|jiBS)Nr4)r1_model_config_auto_includesr4)rQrRrTrUr(sz%SACConfig._model_config_auto_includesN)rWN)r)__name__ __module__ __qualname____doc__r2rrr rboolrrprfloatrr<rrrXrdrvrr~rrrpropertyr __classcell__rTrTrRrUrsZ     so   rcs`eZdZdZfddZeeedefddZ eeedede e e fdd Z ZS) r3a0Soft Actor Critic (SAC) Algorithm class. This file defines the distributed Algorithm class for the soft actor critic algorithm. See `sac_[tf|torch]_policy.py` for the definition of the policy loss. Detailed documentation: https://docs.ray.io/en/master/rllib-algorithms.html#sac cs(|jddg7_tj|i|dS)Nr6r5)_allow_unknown_subkeysr1r2)rQargsrZrRrTrUr28sz SAC.__init__rWcCstSr)r)clsrTrTrUget_default_config<szSAC.get_default_configconfigcCs |ddkrddlm}|StS)Nrirwr)SACTorchPolicy))ray.rllib.algorithms.sac.sac_torch_policyrr )rrrrTrTrUget_default_policy_classAs  zSAC.get_default_policy_class)rrrrr2 classmethodrr rrrrrrrrTrTrRrUr3-s  r3)1loggingtypingrrrrrr%ray.rllib.algorithms.algorithm_configrr ray.rllib.algorithms.dqn.dqnr &ray.rllib.algorithms.sac.sac_tf_policyr Cray.rllib.connectors.common.add_observations_from_episodes_to_batchr Oray.rllib.connectors.learner.add_next_observations_from_episodes_to_train_batchr ray.rllib.core.learnerr"ray.rllib.core.rl_module.rl_modulerray.rllib.policy.policyrray.rllib.utilsrray.rllib.utils.annotationsrray.rllib.utils.deprecationrrray.rllib.utils.frameworkrr4ray.rllib.utils.replay_buffers.episode_replay_bufferrray.rllib.utils.typingrrtf1r^tfvrj getLoggerrrkrr3rTrTrTrUs2