o `۷iJ@sdZddlZddlmZmZddlmZddlmZm Z ddl m Z m Z ddl mZmZddlmZdd lmZdd lmZdd lmZmZmZmZmZeeZd Zd Z dZ!GdddeZ"GdddeZ#dS)aOAsynchronous Proximal Policy Optimization (APPO) The algorithm is described in [1] (under the name of "IMPACT"): Detailed documentation: https://docs.ray.io/en/master/rllib-algorithms.html#appo [1] IMPACT: Importance Weighted Asynchronous Architectures with Clipped Target Networks. Luo et al. 2020 https://arxiv.org/pdf/1912.00167 N)OptionalType)Self)DEPRECATED_VALUEdeprecation_warning)AlgorithmConfig NotProvided)IMPALA IMPALAConfig) RLModuleSpec)Policy)override)LAST_TARGET_UPDATE_TSLEARNER_STATS_KEYNUM_AGENT_STEPS_SAMPLEDNUM_ENV_STEPS_SAMPLEDNUM_TARGET_UPDATES mean_kl_loss curr_kl_coeffold_action_distc s$eZdZdZdfdd Zeeeeeeeeeeeeeeeeeedde e de e de e d e e d e e d e e d e e d e e de e de e de e de e de e de e de ffddZeed fdd ZeeddZeedefddZeeefddZZS)! APPOConfiga Defines a configuration class from which an APPO Algorithm can be built. .. testcode:: from ray.rllib.algorithms.appo import APPOConfig config = ( APPOConfig() .training(lr=0.01, grad_clip=30.0, train_batch_size_per_learner=50) ) config = config.learners(num_learners=1) config = config.env_runners(num_env_runners=1) config = config.environment("CartPole-v1") # Build an Algorithm object from the config and run 1 training iteration. algo = config.build() algo.train() del algo .. testcode:: from ray.rllib.algorithms.appo import APPOConfig from ray import tune config = APPOConfig() # Update the config object. config = config.training(lr=tune.grid_search([0.001,])) # Set the config object's env. config = config.environment(env="CartPole-v1") # Use to_dict() to get the old-style python config dict when running with tune. tune.Tuner( "APPO", run_config=tune.RunConfig( stop={"training_iteration": 1}, verbose=0, ), param_space=config.to_dict(), ).fit() .. testoutput:: :hide: ... Ncsddi|_tj|p tdd|_d|_d|_d|_d|_d|_ d|_ d |_ d|_ d |_ d |_d |_d |_d |_d |_d|_d|_d|_d|_d|_d|_d|_d|_d|_d|_d|_d|_d|_d |_ d |_!d|_"d|_#d|_$d|_%t&|_'t&|_(dS)z"Initializes a APPOConfig instance.typeStochasticSampling) algo_classTg?g?Fg{Gz?g@ gD@ global_normadamgMb@?gGz?g?g?Nrdi,))exploration_configsuper__init__APPOvtraceuse_gaelambda_ clip_param use_kl_losskl_coeff kl_targettarget_worker_clippinguse_circular_buffercircular_buffer_num_batches$circular_buffer_iterations_per_batchsimple_queue_sizenum_env_runnerstarget_network_update_freqbroadcast_interval grad_clip grad_clip_byopt_typelrdecaymomentumepsilon vf_loss_coeff entropy_coefftau lr_scheduleentropy_coeff_schedulenum_gpusnum_multi_gpu_tower_stacksminibatch_buffer_sizereplay_proportionreplay_buffer_num_slotslearner_queue_sizelearner_queue_timeoutrtarget_update_frequency use_critic)selfr __class__T/home/ubuntu/vllm_env/lib/python3.10/site-packages/ray/rllib/algorithms/appo/appo.pyr%VsN  zAPPOConfig.__init__)r'r(r)r*r+r,r-r4r?r.r/r0r1r2rIrJr'r(r)r*r+r,r-r4r?r.r/r0r1r2returnc s|tkr tdddd|tkrtddddtjd i||tur&||_|tur-||_|tur4||_|tur;||_|turB||_ |turI||_ |turP||_ |turW||_ | tur^| |_ | ture| |_| turl| |_| turs| |_| turz| |_|tur||_|S) udSets the training related configuration. Args: vtrace: Whether to use V-trace weighted advantages. If false, PPO GAE advantages will be used instead. use_gae: If true, use the Generalized Advantage Estimator (GAE) with a value function, see https://arxiv.org/pdf/1506.02438.pdf. Only applies if vtrace=False. lambda_: GAE (lambda) parameter. clip_param: PPO surrogate slipping parameter. use_kl_loss: Whether to use the KL-term in the loss function. kl_coeff: Coefficient for weighting the KL-loss term. kl_target: Target term for the KL-term to reach (via adjusting the `kl_coeff` automatically). target_network_update_freq: NOTE: This parameter is only applicable on the new API stack. The frequency with which to update the target policy network from the main trained policy network. The metric used is `NUM_ENV_STEPS_TRAINED_LIFETIME` and the unit is `n` (see [1] 4.1.1), where: `n = [circular_buffer_num_batches (N)] * [circular_buffer_iterations_per_batch (K)] * [train batch size]` For example, if you set `target_network_update_freq=2`, and N=4, K=2, and `train_batch_size_per_learner=500`, then the target net is updated every 2*4*2*500=8000 trained env steps (every 16 batch updates on each learner). The authors in [1] suggests that this setting is robust to a range of choices (try values between 0.125 and 4). target_network_update_freq: The frequency to update the target policy and tune the kl loss coefficients that are used during training. After setting this parameter, the algorithm waits for at least `target_network_update_freq` number of environment samples to be trained on before updating the target networks and tune the kl loss coefficients. NOTE: This parameter is only applicable when using the Learner API (enable_rl_module_and_learner=True). tau: The factor by which to update the target policy network towards the current policy network. Can range between 0 and 1. e.g. updated_param = tau * current_param + (1 - tau) * target_param target_worker_clipping: The maximum value for the target-worker-clipping used for computing the IS ratio, described in [1] IS = min(π(i) / π(target), ρ) * (π / π(i)) use_circular_buffer: Whether to use a circular buffer for storing training batches. If false, a simple Queue will be used. Defaults to True. circular_buffer_num_batches: The number of train batches that fit into the circular buffer. Each such train batch can be sampled for training max. `circular_buffer_iterations_per_batch` times. circular_buffer_iterations_per_batch: The number of times any train batch in the circular buffer can be sampled for training. A batch gets evicted from the buffer either if it's the oldest batch in the buffer and a new batch is added OR if the batch reaches this max. number of being sampled. simple_queue_size: The size of the simple queue (if `use_circular_buffer` is False) for storing training batches. Returns: This updated AlgorithmConfig object. rIr4T)oldnewerrorrJzM`use_critic` no longer supported! APPO always uses a value function (critic).)rQhelprSNrN)rrr$trainingrr'r(r)r*r+r,r-r4r?r.r/r0r1r2)rKr'r(r)r*r+r,r-r4r?r.r/r0r1r2rIrJkwargsrLrNrOrUsVOzAPPOConfig.trainingcsbt|jr-|jdks|jdkr|d|jdkr!|d|jdkr/|ddSdSdS)Nrr a`minibatch_buffer_size/replay_proportion` not valid on new API stack with APPO! Use `circular_buffer_num_batches` for the number of train batches in the circular buffer. To change the maximum number of times any batch may be sampled, set `circular_buffer_iterations_per_batch`.aO`num_multi_gpu_tower_stacks` not supported on new API stack with APPO! In order to train on multi-GPU, use `config.learners(num_learners=[number of GPUs], num_gpus_per_learner=1)`. To scale the throughput of batch-to-GPU-pre-loading on each of your `Learners`, set `num_gpu_loader_threads` to a higher number (recommended values: 1-8).r"aE`learner_queue_size` not supported on new API stack with APPO! In order set the size of the circular buffer (which acts as a 'learner queue'), use `config.training(circular_buffer_num_batches=..)`. To change the maximum number of times any batch may be sampled, set `config.training(circular_buffer_iterations_per_batch=..)`.)r$validateenable_rl_module_and_learnerrDrE _value_errorrCrGrKrLrNrOrW"s   zAPPOConfig.validatecCs>|jdkr ddlm}|S|jdvrtdtd|jd)Ntorchr)APPOTorchLearner)tf2tfzPTensorFlow is no longer supported on the new API stack! Use `framework='torch'`.The framework z+ is not supported. Use `framework='torch'`.) framework_str2ray.rllib.algorithms.appo.torch.appo_torch_learnerr\ ValueError)rKr\rNrNrOget_default_learner_classEs    z$APPOConfig.get_default_learner_classcCs4|jdkr ddlm}n td|jdt|dS)Nr[r)APPOTorchRLModuler_z/ is not supported. Use either 'torch' or 'tf2'.) module_class)r`4ray.rllib.algorithms.appo.torch.appo_torch_rl_modulerdrbr )rKRLModulerNrNrOget_default_rl_module_specXs   z%APPOConfig.get_default_rl_module_speccstjddiBS)Nvf_share_layersF)r$_model_config_auto_includesrZrLrNrOrjfsz&APPOConfig._model_config_auto_includesNrPN)__name__ __module__ __qualname____doc__r%r r rrrboolfloatintrrUrWrcr rhpropertyrrj __classcell__rNrNrLrOr(sz-K     "  rcsreZdZfddZeed fdd ZeeedefddZ eeed e de e e fd d ZZS) r&cs:tj|g|Ri||jjs|jdddSdS)zInitializes an APPO instance.cS|Srk update_targetp_rNrNrOvzAPPO.__init__..N)r$r%configrX env_runnerforeach_policy_to_train)rKr~argsrVrLrNrOr%msz APPO.__init__rPNcs|jjr tSt|jt}|j|jjdkrtnt}|jj |jj }|||krT|jt d7<||jt<|j dd|jjrTfdd}|j |S)N agent_stepsrcSrvrkrwryrNrNrOr|r}z$APPO.training_step..csjtvs Jdtf|vr+|td}|dus$J|f||dStd|dS)Nz'{} should be nested under policy id keyklzNo data for {}, not updating kl)rformatget update_klloggerwarning)pipi_idr train_resultsrNrOupdates z"APPO.training_step..update)r~rXr$ training_step _countersrcount_steps_byrr num_epochsrDrrrr+)rK last_updatecur_tstarget_update_freqrrLrrOrxs$        zAPPO.training_stepcCstSrk)r)clsrNrNrOget_default_configszAPPO.get_default_configr~cCsV|ddkrddlm}|S|ddkr#|jrtdddlm}|Sddlm}|S) N frameworkr[r)APPOTorchPolicyr^zWRLlib's RLModule and Learner API is not supported for tf1. Use framework='tf2' instead.) APPOTF1Policy) APPOTF2Policy)+ray.rllib.algorithms.appo.appo_torch_policyrrXrb(ray.rllib.algorithms.appo.appo_tf_policyrr)rr~rrrrNrNrOget_default_policy_classs     zAPPO.get_default_policy_classrl)rmrnror%r r r classmethodrrrrrr rrurNrNrLrOr&ls  0 r&)$rploggingtypingrrtyping_extensionsrray._common.deprecationrr%ray.rllib.algorithms.algorithm_configrr"ray.rllib.algorithms.impala.impalar r "ray.rllib.core.rl_module.rl_moduler ray.rllib.policy.policyr ray.rllib.utils.annotationsr ray.rllib.utils.metricsrrrrr getLoggerrmrLEARNER_RESULTS_KL_KEY!LEARNER_RESULTS_CURR_KL_COEFF_KEYOLD_ACTION_DIST_KEYrr&rNrNrNrOs&      F