o `۷i@sdZddlZddlmZddlmZmZmZmZm Z m Z m Z m Z ddl ZddlmZddlmZddlmZddlmZmZdd lmZdd lmZdd lmZdd lmZdd l m!Z!ddl"m#Z#m$Z$ddl%m&Z&ddl'm(Z(ddl)m*Z*ddl+m,Z,ddl-m.Z.m/Z/m0Z0m1Z1m2Z2m3Z3m4Z4m5Z5m6Z6m7Z7m8Z8m9Z9m:Z:m;Z;mZ>m?Z?m@Z@ddlAmBZBddlCmDZDmEZEmFZFmGZGddlHmIZImJZJmKZKmLZLeMeNZOGdddeZPdedeeQfddZRGdddeZSdS)ap Deep Q-Networks (DQN, Rainbow, Parametric DQN) ============================================== This file defines the distributed Algorithm class for the Deep Q-Networks algorithm. See `dqn_[tf|torch]_policy.py` for the definition of the policies. Detailed documentation: https://docs.ray.io/en/master/rllib-algorithms.html#deep-q-networks-dqn-rainbow-parametric-dqn N) defaultdict)AnyCallableDictListOptionalTupleTypeUnion)Self)DEPRECATED_VALUE) Algorithm)AlgorithmConfig NotProvided) DQNTFPolicy)DQNTorchPolicy)Learner) RLModuleSpec)synchronous_parallel_sample)multi_gpu_train_one_steptrain_one_step)Policy)MultiAgentBatch) deep_update)override) ALL_MODULESENV_RUNNER_RESULTSENV_RUNNER_SAMPLING_TIMERLAST_TARGET_UPDATE_TSLEARNER_RESULTSLEARNER_UPDATE_TIMERNUM_AGENT_STEPS_SAMPLED NUM_AGENT_STEPS_SAMPLED_LIFETIMENUM_ENV_STEPS_SAMPLEDNUM_ENV_STEPS_SAMPLED_LIFETIMENUM_TARGET_UPDATESREPLAY_BUFFER_ADD_DATA_TIMERREPLAY_BUFFER_RESULTSREPLAY_BUFFER_SAMPLE_TIMER REPLAY_BUFFER_UPDATE_PRIOS_TIMER SAMPLE_TIMERSYNCH_WORKER_WEIGHTS_TIMER TD_ERROR_KEYTIMERS)convert_to_numpy)sample_min_n_steps_from_buffer*update_priorities_in_episode_replay_buffer"update_priorities_in_replay_buffervalidate_buffer_config)LearningRateOrSchedule ResultDictRLModuleSpecTypeSampleBatchTypec2seZdZdZd-fdd Zeeeeeeeeeeeeeeeeeeeeeeeeeddee dee dee d ee e e e efd eed eed ee d ee deedee deedeedee deedee dee dee dee e ee e ffdeeee eeee geefdeedeedeedee def0fddZeed.fd d! Zeed/d#e de fd$d%Zeedefd&d'Zeeedeeefffd(d) Zeede ed*effd+d,Z Z!S)0 DQNConfiga/Defines a configuration class from which a DQN Algorithm can be built. .. testcode:: from ray.rllib.algorithms.dqn.dqn import DQNConfig config = ( DQNConfig() .environment("CartPole-v1") .training(replay_buffer_config={ "type": "PrioritizedEpisodeReplayBuffer", "capacity": 60000, "alpha": 0.5, "beta": 0.5, }) .env_runners(num_env_runners=1) ) algo = config.build() algo.train() algo.stop() .. testcode:: from ray.rllib.algorithms.dqn.dqn import DQNConfig from ray import tune config = ( DQNConfig() .environment("CartPole-v1") .training( num_atoms=tune.grid_search([1,]) ) ) tune.Tuner( "DQN", run_config=tune.RunConfig(stop={"training_iteration":1}), param_space=config, ).fit() .. testoutput:: :hide: ... Ncs(ddddd|_tj|ptdd|_dd g|_d |_d |_d |_d |_ |j t j dddd|_ d|_d|_d|_d|_d|_d|_d|_d|_d|_d|_d|_d|_dg|_d|_d|_d|_d|_d|_d|_ d|_!dddd d!|_"d|_#t$|_%t$|_&t$|_'t$|_(d|_)t$|_*t$|_+t$|_,dS)"z!Initializes a DQNConfig instance. EpsilonGreedy?g{Gz?')typeinitial_epsilon final_epsilonepsilon_timesteps) algo_classauto)rr9)r:g?gD@ global_normgMb@? F)explore)evaluation_configNiig:0yE>g$g$@g?ThuberrPrioritizedEpisodeReplayBufferiPg333333?g?)r;capacityalphabeta)-exploration_configsuper__init__DQNrollout_fragment_lengthepsilon grad_clip grad_clip_bylrtrain_batch_size evaluationr overridesmin_time_s_per_iteration"min_sample_timesteps_per_iterationtarget_network_update_freq(num_steps_sampled_before_learning_startsstore_buffer_in_checkpoints adam_epsilontau num_atomsv_minv_maxnoisysigma0duelinghiddensdouble_qn_stepbefore_learn_on_batchtraining_intensitytd_error_loss_fn$categorical_distribution_temperature burn_in_lenreplay_buffer_config lr_scheduler buffer_sizeprioritized_replaylearning_startsreplay_batch_sizereplay_sequence_lengthprioritized_replay_alphaprioritized_replay_betaprioritized_replay_eps)selfr? __class__R/home/ubuntu/vllm_env/lib/python3.10/site-packages/ray/rllib/algorithms/dqn/dqn.pyrN{s`   zDQNConfig.__init__)rZrmr\rnrQr]rRr[r^r_r`rarbrcrdrerfrgrhrirjrkrlrZrmr\rnrQr]rRr[r^r_r`rarbrcrdrerfrgrhrirjrkrlreturnc sztjdi||tur||_|tur(td|jid|iddgdg}|d|_|tur/||_|tur6||_|tur=||_|turD||_ |turK||_ |turR||_ | turY| |_ | tur`| |_ | turg| |_| turn| |_| turu| |_|tur|||_|tur||_|tur||_|tur||_|tur||_|tur||_|tur||_|tur||_|tur||_|tur||_|S)uWSets the training related configuration. Args: target_network_update_freq: Update the target network every `target_network_update_freq` sample steps. replay_buffer_config: Replay buffer config. Examples: { "_enable_replay_buffer_api": True, "type": "MultiAgentReplayBuffer", "capacity": 50000, "replay_sequence_length": 1, } - OR - { "_enable_replay_buffer_api": True, "type": "MultiAgentPrioritizedReplayBuffer", "capacity": 50000, "prioritized_replay_alpha": 0.6, "prioritized_replay_beta": 0.4, "prioritized_replay_eps": 1e-6, "replay_sequence_length": 1, } - Where - prioritized_replay_alpha: Alpha parameter controls the degree of prioritization in the buffer. In other words, when a buffer sample has a higher temporal-difference error, with how much more probability should it drawn to use to update the parametrized Q-network. 0.0 corresponds to uniform probability. Setting much above 1.0 may quickly result as the sampling distribution could become heavily “pointy” with low entropy. prioritized_replay_beta: Beta parameter controls the degree of importance sampling which suppresses the influence of gradient updates from samples that have higher probability of being sampled via alpha parameter and the temporal-difference error. prioritized_replay_eps: Epsilon parameter sets the baseline probability for sampling so that when the temporal-difference error of a sample is zero, there is still a chance of drawing the sample. store_buffer_in_checkpoints: Set this to True, if you want the contents of your buffer(s) to be stored in any saved checkpoints as well. Warnings will be created if: - This is True AND restoring from a checkpoint that contains no buffer data. - This is False AND restoring from a checkpoint that does contain buffer data. epsilon: Epsilon exploration schedule. In the format of [[timestep, value], [timestep, value], ...]. A schedule must start from timestep 0. adam_epsilon: Adam optimizer's epsilon hyper parameter. grad_clip: If not None, clip gradients during optimization at this value. num_steps_sampled_before_learning_starts: Number of timesteps to collect from rollout workers before we start sampling from replay buffers for learning. Whether we count this in agent steps or environment steps depends on config.multi_agent(count_steps_by=..). tau: Update the target by au * policy + (1- au) * target_policy. num_atoms: Number of atoms for representing the distribution of return. When this is greater than 1, distributional Q-learning is used. v_min: Minimum value estimation v_max: Maximum value estimation noisy: Whether to use noisy network to aid exploration. This adds parametric noise to the model weights. sigma0: Control the initial parameter noise for noisy nets. dueling: Whether to use dueling DQN. hiddens: Dense-layer setup for each the advantage branch and the value branch double_q: Whether to use double DQN. n_step: N-step target updates. If >1, sars' tuples in trajectories will be postprocessed to become sa[discounted sum of R][s t+n] tuples. An integer will be interpreted as a fixed n-step value. If a tuple of 2 ints is provided here, the n-step value will be drawn for each sample(!) in the train batch from a uniform distribution over the closed interval defined by `[n_step[0], n_step[1]]`. before_learn_on_batch: Callback to run before learning on a multi-agent batch of experiences. training_intensity: The intensity with which to update the model (vs collecting samples from the env). If None, uses "natural" values of: `train_batch_size` / (`rollout_fragment_length` x `num_env_runners` x `num_envs_per_env_runner`). If not None, will make sure that the ratio between timesteps inserted into and sampled from the buffer matches the given values. Example: training_intensity=1000.0 train_batch_size=250 rollout_fragment_length=1 num_env_runners=1 (or 0) num_envs_per_env_runner=1 -> natural value = 250 / 1 = 250.0 -> will make sure that replay+train op will be executed 4x asoften as rollout+insert op (4 * 250 = 1000). See: rllib/algorithms/dqn/dqn.py::calculate_rr_weights for further details. td_error_loss_fn: "huber" or "mse". loss function for calculating TD error when num_atoms is 1. Note that if num_atoms is > 1, this parameter is simply ignored, and softmax cross entropy loss will be used. categorical_distribution_temperature: Set the temperature parameter used by Categorical action distribution. A valid temperature is in the range of [0, 1]. Note that this mostly affects evaluation since TD error uses argmax for return calculation. burn_in_len: The burn-in period for a stateful RLModule. It allows the Learner to utilize the initial `burn_in_len` steps in a replay sequence solely for unrolling the network and establishing a typical starting state. The network is then updated on the remaining steps of the sequence. This process helps mitigate issues stemming from a poor initial state - zero or an outdated recorded state. Consider setting this parameter to a positive integer if your stateful RLModule faces convergence challenges or exhibits signs of catastrophic forgetting. Returns: This updated AlgorithmConfig object. rmFNrz)rMtrainingrrZrrmr\rnrQr]rRr[r^r_r`rarbrcrdrerfrgrhrirjrkrl)rwrZrmr\rnrQr]rRr[r^r_r`rarbrcrdrerfrgrhrirjrkrlkwargsnew_replay_buffer_configrxrzr{r}st zDQNConfig.trainingc st|jr|jdur|dn |jst||jddkr3|jdkr+|d|j r3|d|j dvr=|d|js]|j d kr]|j |j kr]|d |j d |j d |j d d|j vrd|j dkrp|jkrnntd|jd|j ddddlm}|jrt|jdtst|jd|s|ddS|jst|jdtrd|jdvst|jd|r|ddSdSdS)NzW`lr_schedule` is deprecated and must be None! Use the `lr` setting to setup a schedule.r;ParameterNoisecomplete_episodeszParameterNoise Exploration requires `batch_mode` to be 'complete_episodes'. Try setting `config.env_runners(batch_mode='complete_episodes')`.zOParameterNoise Exploration and `noisy` network cannot be used at the same time!)rGmsez,`td_error_loss_fn` must be 'huber' or 'mse'!r@z Your `rollout_fragment_length` (z) is smaller than `n_step` (z:)! Try setting config.env_runners(rollout_fragment_length=z). max_seq_lenrzYour defined `burn_in_len`=z" is larger or equal `max_seq_len`=zC! Either decrease the `burn_in_len` or increase your `max_seq_len`.)EpisodeReplayBufferz[When using the new `EnvRunner API` the replay buffer must be of type `EpisodeReplayBuffer`.EpisodeadWhen using the old API stack the replay buffer must not be of type `EpisodeReplayBuffer`! We suggest you use the following config to run DQN on the old API stack: `config.training(replay_buffer_config={'type': 'MultiAgentPrioritizedReplayBuffer', 'prioritized_replay_alpha': [alpha], 'prioritized_replay_beta': [beta], 'prioritized_replay_eps': [eps], })`.)rMvalidateenable_rl_module_and_learnerrn _value_error in_evaluationr2rL batch_moderbrjrPrg model_configrl ValueError4ray.rllib.utils.replay_buffers.episode_replay_bufferr"enable_env_runner_and_connector_v2 isinstancermstr issubclass)rwrrxrzr{rsv           zDQNConfig.validater worker_indexcCs0|jdkrt|jttfr|jdS|jS|jS)Nr@rE)rPrrgtuplelist)rwrrzrzr{get_rollout_fragment_lengths  z%DQNConfig.get_rollout_fragment_lengthcCs6|jdkrddlm}t||jdStd|jd)Ntorchr)DefaultDQNTorchRLModule) module_classrThe framework ; is not supported! Use `config.framework('torch')` instead.) framework_str:ray.rllib.algorithms.dqn.torch.default_dqn_torch_rl_modulerrrr)rwrrzrzr{get_default_rl_module_spec s   z$DQNConfig.get_default_rl_module_specc s*tj|j|j|j|j|j|j|jdBS)N)rfrdrQr_std_initrar`) rM_model_config_auto_includesrfrdrQr_rcrar`rwrxrzr{rsz%DQNConfig._model_config_auto_includesrcCs,|jdkr ddlm}|Std|jd)Nrr)DQNTorchLearnerrr)r0ray.rllib.algorithms.dqn.torch.dqn_torch_learnerrr)rwrrzrzr{get_default_learner_class)s   z#DQNConfig.get_default_learner_classNr|N)r)"__name__ __module__ __qualname____doc__rNrrrrintdictboolrr floatr3rrr rrrr r}rrr5rpropertyrrrr __classcell__rzrzrxr{r7Ks/Y     P[  $r7configr|cCsl|jsddgS|j||jt|jdd}|j|}|dkr-ttd|dgSdtt|gS)zACalculate the round robin weights for the rollout and train stepsrE) ritotal_train_batch_sizernum_envs_per_env_runnermaxnum_env_runnersrnpround)r native_ratiosample_and_train_weightrzrzr{calculate_rr_weights8s rcseZdZeeedefddZeeedede e e fddZ eededdffdd Z eedd d Zd d ZdefddZZS)rOr|cCstSr)r7)clsrzrzr{get_default_configTszDQN.get_default_configrcCs|ddkrtStS)N frameworkr)rr)rrrzrzr{get_default_policy_classYs zDQN.get_default_policy_classNcs`t||jjr,|jr.|jdur#|jjdddgddd|_dS|jj |_dSdSdS)NcSs |jSr)module is_stateful)errzrzr{js zDQN.setup..rEF)remote_worker_idslocal_env_runnerr) rMsetuprrenv_runner_group env_runnerforeach_env_runner_module_is_statefulrr)rwrrxrzr{rcs   z DQN.setupcCs|jjs|S|S)a-DQN training iteration function. Each training iteration, we: - Sample (MultiAgentBatch) from workers. - Store new samples in replay buffer. - Sample training batch (MultiAgentBatch) from replay buffer. - Learn on training batch. - Update remote workers' new policy weights. - Update target network every `target_network_update_freq` sample steps. - Return all collected metrics for the iteration. Returns: The results dict from executing the training iteration. )rr_training_step_old_api_stack_training_step_new_api_stackrrzrzr{ training_stepqszDQN.training_stepc Cs2t|j\}}t|D]N}|jttft|jd|jj ddd\}}Wdn1s.wY|jj |t d|jtt f|j |Wdn1sTwYq |jjdkrot|jjt tfid}n |jjt tfdd}||jjkrt|D]}|jttfE|j j|jj|jj|j|jjddt|jt|jdr|jjnd|jj |jj!d dd }|j "}|jj |gt#dWdn1swY|jtt$fX|j%j&|t|jt tft|jt tfid }t't(} |D] } | )D]\} } t*| vr| | +t,| -t*qqd d | )D} |jj |t.dWdn 1s?wY|jtt/ft0|j | dWdn 1s_wYq|jtt1ft2|d3t4h} |jj5|j%| dddWddS1swYdSdS)NT) worker_setconcatsample_timeout_s_uses_new_env_runners_return_metrics)key agent_steps)defaultrrrlrK) num_itemsrgbatch_length_Tlookbackmin_batch_length_TgammarKsample_episodes)episodes timestepscSs$i|]\}}|ttj|ddiqS)r)axis)r,r concatenate).0 module_idsrzrzr{ sz4DQN._training_step_new_api_stack..) replay_buffer td_errors)from_worker_or_learner_grouppolicies global_varsinference_only)6rrrangemetricslog_timer-rrrr aggregaterr&local_replay_bufferaddcount_steps_bysumpeekr"valuesr$r[r(samplerrgrrgetrhasattrrlrrm get_metricsr'r learner_groupupdaterritemsr,extendr.poprr)r0r+setkeysr sync_weights)rw store_weightr_renv_runner_results current_tsreplay_buffer_resultslearner_resultsrresrmodule_resultsmodules_to_updaterzrzr{rs           %$z DQN._training_step_new_api_stackc Csi}t|j\}}t|D]C}|jtt|jd|jjd}Wdn1s)wY|s4iS|jt | 7<|jt | 7<|j |q d|jt i}|j|jjdkrbt nt }||jjkrt|D]}t|j |jj|jjdkd}|jdpdd } | ||j|j}|jd durt||}nt||}t|j |j|||jt} || |jjkr|j} |j| fd d |jtd 7<||jt<|jt|jj|d Wdn1swYqo|S)zvTraining step for the old API stack. More specifically this training step relies on `RolloutWorker`. T)rrrNtimestepr)count_by_agent_stepsrhcWs|Srrz)barzrzr{r7sz2DQN._training_step_old_api_stack..simple_optimizercSs||vo|Sr) update_target)ppid to_updaterzrzr{rNsrE)r)rrr_timersr*rrr _countersr!rr# env_stepsrrrr[r/rrrrr1rrZrget_policies_to_trainforeach_policy_to_trainr%r+r) rw train_resultsrrrnew_sample_batchrcur_ts train_batchpost_fn last_updaterrzrzr{rsj              z DQN._training_step_old_api_stackr)rrr classmethodrr r7rrrr rrrrrr4rrrzrzrxr{rOSs"  zrO)Trlogging collectionsrtypingrrrrrrr r numpyrtyping_extensionsr ray._common.deprecationr ray.rllib.algorithms.algorithmr %ray.rllib.algorithms.algorithm_configrr&ray.rllib.algorithms.dqn.dqn_tf_policyr)ray.rllib.algorithms.dqn.dqn_torch_policyrray.rllib.core.learnerr"ray.rllib.core.rl_module.rl_modulerray.rllib.execution.rollout_opsrray.rllib.execution.train_opsrrray.rllib.policy.policyrray.rllib.policy.sample_batchrray.rllib.utilsrray.rllib.utils.annotationsrray.rllib.utils.metricsrrrrrr r!r"r#r$r%r&r'r(r)r*r+r,r-ray.rllib.utils.numpyr.$ray.rllib.utils.replay_buffers.utilsr/r0r1r2ray.rllib.utils.typingr3r4r5r6 getLoggerrloggerr7rrrOrzrzrzr{s< (            T  p