o $i]@sfdZddlZddlmZmZmZmZmZmZm Z ddl m Z ddl m Z ddlmZddlmZmZddlmZdd lmZmZdd lmZmZdd lmZdd lmZmZdd l m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+m,Z,ddl-m.Z.ddl/m0Z0ddl1m2Z2ddl3m4Z4erddl5m6Z6e7e8Z9dZ:dZ;dZGdddeZ?GdddeZ@dS)a3 Proximal Policy Optimization (PPO) ================================== This file defines the distributed Algorithm class for proximal policy optimization. See `ppo_[tf|torch]_policy.py` for the definition of the policy loss. Detailed documentation: https://docs.ray.io/en/master/rllib-algorithms.html#ppo N) TYPE_CHECKINGAnyDictListOptionalTypeUnion)Self)DEPRECATED_VALUE) Algorithm)AlgorithmConfig NotProvided) RLModuleSpec)standardize_fieldssynchronous_parallel_sample)multi_gpu_train_one_steptrain_one_step)Policy) OldAPIStackoverride) ALL_MODULESENV_RUNNER_RESULTSENV_RUNNER_SAMPLING_TIMERLEARNER_RESULTSLEARNER_UPDATE_TIMERNUM_AGENT_STEPS_SAMPLEDNUM_ENV_STEPS_SAMPLEDNUM_ENV_STEPS_SAMPLED_LIFETIME!NUM_MODULE_STEPS_TRAINED_LIFETIME SAMPLE_TIMERSYNCH_WORKER_WEIGHTS_TIMERTIMERS)LEARNER_STATS_KEY) Scheduler) ResultDict)log_once)Learnervf_loss_unclippedvf_explained_var mean_kl_loss curr_kl_coeffcurr_entropy_coeffc!sXeZdZdZdfdd ZeedefddZeede e de ffd d Z eee e e e e e e e e e e e e ed d eed eedeedeedeedeedeedeedeeee eefdeedeedeedeeee eefdeffddZeed fdd Zeeedee efffdd ZZS)! PPOConfigaTDefines a configuration class from which a PPO Algorithm can be built. .. testcode:: from ray.rllib.algorithms.ppo import PPOConfig config = PPOConfig() config.environment("CartPole-v1") config.env_runners(num_env_runners=1) config.training( gamma=0.9, lr=0.01, kl_coeff=0.3, train_batch_size_per_learner=256 ) # Build a Algorithm object from the config and run 1 training iteration. algo = config.build() algo.train() .. testcode:: from ray.rllib.algorithms.ppo import PPOConfig from ray import tune config = ( PPOConfig() # Set the config object's env. .environment(env="CartPole-v1") # Update the config object's training parameters. .training( lr=0.001, clip_param=0.2 ) ) tune.Tuner( "PPO", run_config=tune.RunConfig(stop={"training_iteration": 1}), param_space=config, ).fit() .. testoutput:: :hide: ... Ncsddi|_tj|p tdd|_d|_d|_d|_d|_d|_ d |_ d|_ d |_ d|_ d |_d |_d |_d |_d|_d|_d|_d|_d|jd<d|_d|_t|_t|_dS)z!Initializes a PPOConfig instance.typeStochasticSampling) algo_classg-C6 ?autoiTg?g?g{Gz?g333333?g$@NFvf_share_layers)exploration_configsuper__init__PPOlrrollout_fragment_lengthtrain_batch_size use_criticuse_gae num_epochsminibatch_sizeshuffle_batch_per_epochlambda_ use_kl_losskl_coeff kl_target vf_loss_coeff entropy_coeff clip_param vf_clip_param grad_clipnum_env_runnersmodelentropy_coeff_schedule lr_scheduler sgd_minibatch_sizer5)selfr/ __class__Y/home/ubuntu/veenaModal/venv/lib/python3.10/site-packages/ray/rllib/algorithms/ppo/ppo.pyr8ls4   zPPOConfig.__init__returncCs2|jdkrddlm}t|dStd|jd)Ntorchr)DefaultPPOTorchRLModule) module_classThe framework z/ is not supported. Use either 'torch' or 'tf2'.) framework_str:ray.rllib.algorithms.ppo.torch.default_ppo_torch_rl_modulerWr ValueError)rPrWrSrSrTget_default_rl_module_specs    z$PPOConfig.get_default_rl_module_specr&cCs>|jdkr ddlm}|S|jdvrtdtd|jd)NrVr)PPOTorchLearner)tf2tfzPTensorFlow is no longer supported on the new API stack! Use `framework='torch'`.rYz+ is not supported. Use `framework='torch'`.)rZ0ray.rllib.algorithms.ppo.torch.ppo_torch_learnerr^r\)rPr^rSrSrTget_default_learner_classs    z#PPOConfig.get_default_learner_class)r=r>rBrCrDrErFrGrMrHrIrJrNr5r=r>rBrCrDrErFrGrMrHrIrJrNc stjdi||tur||_|tur||_|tur||_|tur%||_|tur,||_|tur3||_|tur:||_ |turA||_ | turH| |_ | turO| |_ | turV| |_ | tur]| |_| turd| |_|S)a Sets the training related configuration. Args: use_critic: Should use a critic as a baseline (otherwise don't use value baseline; required for using GAE). use_gae: If true, use the Generalized Advantage Estimator (GAE) with a value function, see https://arxiv.org/pdf/1506.02438.pdf. lambda_: The lambda parameter for General Advantage Estimation (GAE). Defines the exponential weight used between actually measured rewards vs value function estimates over multiple time steps. Specifically, `lambda_` balances short-term, low-variance estimates against long-term, high-variance returns. A `lambda_` of 0.0 makes the GAE rely only on immediate rewards (and vf predictions from there on, reducing variance, but increasing bias), while a `lambda_` of 1.0 only incorporates vf predictions at the truncation points of the given episodes or episode chunks (reducing bias but increasing variance). use_kl_loss: Whether to use the KL-term in the loss function. kl_coeff: Initial coefficient for KL divergence. kl_target: Target value for KL divergence. vf_loss_coeff: Coefficient of the value function loss. IMPORTANT: you must tune this if you set vf_share_layers=True inside your model's config. entropy_coeff: The entropy coefficient (float) or entropy coefficient schedule in the format of [[timestep, coeff-value], [timestep, coeff-value], ...] In case of a schedule, intermediary timesteps will be assigned to linearly interpolated coefficient values. A schedule config's first entry must start with timestep 0, i.e.: [[0, initial_value], [...]]. clip_param: The PPO clip parameter. vf_clip_param: Clip param for the value function. Note that this is sensitive to the scale of the rewards. If your expected V is large, increase this. grad_clip: If specified, clip the global norm of gradients by this amount. Returns: This updated AlgorithmConfig object. NrS)r7trainingr r=r>rBrCrDrErFrGrHrIrJrNrM)rPr=r>rBrCrDrErFrGrMrHrIrJrNr5kwargsrQrSrTrcs8;zPPOConfig.trainingc s8t||js)|j|jkr)|d|jd|jd|jd|jd n-|jrV|j}|jp4|j}t |t rVt |t rV||krV|d|d|d|d|jd |j sf|j dkrf|j sf|d|jr|jdurs|d |jdur}|d tj|jd d d t |jtr|jdkr|ddSdSdS)Nz`minibatch_size` (z!) must be <= `train_batch_size` (z.). In PPO, the train batch will be split into zG chunks, each of which is iterated over (used for updating the policy) z times.z-) must be <= `train_batch_size_per_learner` (truncate_episodeszEpisode truncation is not supported without a value function (to estimate the return at the end of the truncated trajectory). Consider setting batch_mode=complete_episodes.zW`lr_schedule` is deprecated and must be None! Use the `lr` setting to setup a schedule.zm`entropy_coeff_schedule` is deprecated and must be None! Use the `entropy_coeff` setting to setup a schedule.rGzentropy coefficient)fixed_value_or_schedule setting_name descriptionr3z`entropy_coeff` must be >= 0.0)r7validate4validate_train_batch_size_vs_rollout_fragment_lengthenable_rl_module_and_learnerr@r< _value_errorr?train_batch_size_per_learner isinstanceint in_evaluation batch_moder>rNrMr#rGfloat)rPmbstbsrQrSrTrish        zPPOConfig.validatecstjddiBS)Nr5F)r7_model_config_auto_includesrPrQrSrTrugsz%PPOConfig._model_config_auto_includesNrUN)__name__ __module__ __qualname____doc__r8rr rr]rrstrrbr r rboolrrrror rcripropertyrrru __classcell__rSrSrQrTr,?sp,1      ]J&r,c@speZdZeeedefddZeeedede e e fddZ eed dd Z edefd d ZdS) r9rUcCstSrw)r,)clsrSrSrTget_default_confignszPPO.get_default_configconfigcCsH|ddkrddlm}|S|ddkrddlm}|Sddlm}|S)N frameworkrVr)PPOTorchPolicyr`) PPOTF1Policy) PPOTF2Policy))ray.rllib.algorithms.ppo.ppo_torch_policyr&ray.rllib.algorithms.ppo.ppo_tf_policyrr)rrrrrrSrSrTget_default_policy_classss     zPPO.get_default_policy_classNc Cs|jjs|S|jttfF|jjdkr*t|j |jj |jj |jjdd\}}nt|j |jj |jj |jjdd\}}|sG WddS|jj |t dWdn1sYwY|jttf4|jj|t|jt tft|jjtttfddi|jj|jj|jjd}|jj |tdWdn1swY|jttft|dth}|j j|j|dd WddS1swYdS) N agent_stepsT) worker_setmax_agent_stepssample_timeout_s_uses_new_env_runners_return_metrics)r max_env_stepsrrr)keyr)default)episodes timestepsr?r@rA)from_worker_or_learner_grouppoliciesinference_only)r"enable_env_runner_and_connector_v2_training_step_old_api_stackmetricslog_timer!rcount_steps_byrenv_runner_grouptotal_train_batch_sizer aggregaterr learner_groupupdaterpeekrrrr?r@rAr setkeys sync_weights)rPrenv_runner_resultslearner_resultsmodules_to_updaterSrSrT training_stepsj      "zPPO.training_stepc sPjtRjjdkrtjjjjjd}n tjjjjjd}|s0iWdS|}j t | 7<j t | 7<t|dg}Wdn1sZwYjjrit|}nt|}t|}j t fdd|Dd}jtjdkrd}jj|||d Wdn1swY|D]q\}}|td }||jj|td } |td } td rjdidr| dkrtd || | |j!|"d|j!|d#} tdr| jj$krd_%td| djdd|dqj&'||S)Nr)rrr)rrr advantagescsi|] }|jj|jqSrS) env_runner policy_mapnum_grad_updates).0pidrvrSrT sz4PPO._training_step_old_api_stack..)timestepnum_grad_updates_per_policyr)rr global_varsklvf_loss policy_lossppo_warned_lr_ratiorLr5dzThe magnitude of your value function loss for policy: {} is extremely large ({}) compared to the policy loss ({}). This can prevent the policy from learning. Consider scaling down the VF loss by reducing vf_loss_coeff, or disabling vf_share_layers.rewardsppo_warned_vf_clipTz1The mean reward returned from the environment is z! but the vf_clip_param is set to rIz%. Consider increasing it for policy: z' to improve value function convergence.)(_timersrrrrrrras_multi_agent _countersrrr env_stepsrsimple_optimizerrrlistrr num_remote_workersritemsr"get get_policy update_klrFr%loggerwarningformatpolicy_batchesset_get_interceptormeanrIwarned_vf_cliprset_global_vars) rP train_batch train_resultspolicies_to_updaterr policy_id policy_info kl_divergencescaled_vf_lossr mean_rewardrSrvrTrs           z PPO._training_step_old_api_stackrx)ryrzr{ classmethodrr r,rr rrrrrrr$rrSrSrSrTr9ms  Qr9)Ar|loggingtypingrrrrrrrtyping_extensionsr ray._common.deprecationr ray.rllib.algorithms.algorithmr %ray.rllib.algorithms.algorithm_configr r "ray.rllib.core.rl_module.rl_modulerray.rllib.execution.rollout_opsrrray.rllib.execution.train_opsrrray.rllib.policy.policyrray.rllib.utils.annotationsrrray.rllib.utils.metricsrrrrrrrrrrr r!$ray.rllib.utils.metrics.learner_infor"#ray.rllib.utils.schedules.schedulerr#ray.rllib.utils.typingr$ray.util.debugr%ray.rllib.core.learner.learnerr& getLoggerryr%LEARNER_RESULTS_VF_LOSS_UNCLIPPED_KEY$LEARNER_RESULTS_VF_EXPLAINED_VAR_KEYLEARNER_RESULTS_KL_KEY!LEARNER_RESULTS_CURR_KL_COEFF_KEY&LEARNER_RESULTS_CURR_ENTROPY_COEFF_KEYr,r9rSrSrSrTs: $     8      0