o `۷i !@sddlmZmZmZddlmZmZddlmZm Z ddl m Z ddl m Z ddlmZddlmZddlmZdd lmZmZGd d d e ZGd d d eZdS))OptionalTypeUnion)AlgorithmConfig NotProvided)MARWIL MARWILConfig)"AddObservationsFromEpisodesToBatch)+AddNextObservationsFromEpisodesToTrainBatch)Learner) RLModuleSpec)override)LearningRateOrScheduleRLModuleSpecTypecseZdZdZdfdd Zeeeeeeeeeddee dee dee d ee d ee d ee d ee d dffddZ eed eedeffddZeed efddZee dfdd Zeedfdd ZefddZZS) IQLConfigaDefines a configuration class from which a new IQL Algorithm can be built .. testcode:: :skipif: True from ray.rllib.algorithms.iql import IQLConfig # Run this from the ray directory root. config = IQLConfig().training(actor_lr=0.00001, gamma=0.99) config = config.offline_data( input_="./rllib/offline/tests/data/pendulum/pendulum-v1_enormous") # Build an Algorithm object from the config and run 1 training iteration. algo = config.build() algo.train() .. testcode:: :skipif: True from ray.rllib.algorithms.iql import IQLConfig from ray import tune config = IQLConfig() # Print out some default values. print(config.beta) # Update the config object. config.training( lr=tune.grid_search([0.001, 0.0001]), beta=0.75 ) # Set the config object's data path. # Run this from the ray directory root. config.offline_data( input_="./rllib/offline/tests/data/pendulum/pendulum-v1_enormous" ) # Set the config object's env, used for evaluation. config.environment(env="Pendulum-v1") # Use to_dict() to get the old-style python config dict # when running with tune. tune.Tuner( "IQL", param_space=config.to_dict(), ).fit() NcsLtj|ptdd|_d|_d|_d|_d|_d|_d|_ d|_ d|_ dS)N) algo_classg?g?ga2U0*3?Tr?) super__init__IQLbeta expectileactor_lr critic_lrvalue_lrlrtwin_qtarget_network_update_freqtau)selfr __class__R/home/ubuntu/vllm_env/lib/python3.10/site-packages/ray/rllib/algorithms/iql/iql.pyr<s zIQLConfig.__init__)rrrrrrrrrrrrrrreturnc  sxtjdi||tur||_|tur||_|tur||_|tur%||_|tur,||_|tur3||_|tur:||_ |S)aSets the training related configuration. Args: beta: The temperature to scaling advantages in exponential terms. Must be >> 0.0. The higher this parameter the less greedy (exploitative) the policy becomes. It also means that the policy is fitting less to the best actions in the dataset. twin_q: If a twin-Q architecture should be used (advisable). expectile: The expectile to use in expectile regression for the value function. For high expectiles the value function tries to match the upper tail of the Q-value distribution. actor_lr: The learning rate for the actor network. Actor learning rates greater than critic learning rates work well in experiments. critic_lr: The learning rate for the Q-network. Critic learning rates greater than value function learning rates work well in experiments. value_lr: The learning rate for the value function network. target_network_update_freq: The number of timesteps in between the target Q-network is fixed. Note, too high values here could harm convergence. The target network is updated via Polyak-averaging. tau: The update parameter for Polyak-averaging of the target Q-network. The higher this value the faster the weights move towards the actual Q-network. Return: This updated `AlgorithmConfig` object. Nr") rtrainingrrrrrrrr) rrrrrrrrkwargsr r"r#r%Ys 'zIQLConfig.trainingr cCs,|jdkr ddlm}|Std|jd)Ntorchr)IQLTorchLearnerThe framework z) is not supported. Use `'torch'` instead.) framework_str0ray.rllib.algorithms.iql.torch.iql_torch_learnerr( ValueError)rr(r"r"r#get_default_learner_classs   z#IQLConfig.get_default_learner_classcCs2|jdkrddlm}t|dStd|jd)Nr'r)DefaultIQLTorchRLModule) module_classr)z' is not supported. Use `torch` instead.)r*:ray.rllib.algorithms.iql.torch.default_iql_torch_rl_moduler.r r,)rr.r"r"r#get_default_rl_module_specs    z$IQLConfig.get_default_rl_module_speccs8tj|||d}|d|d|tt|S)N)input_observation_spaceinput_action_spacedeviceAddOneTsToEpisodesAndTruncateGeneralAdvantageEstimation)rbuild_learner_connectorremove insert_afterr r )rr2r3r4pipeliner r"r#r7s  z!IQLConfig.build_learner_connectorcsHt|jdkr|dd|jkrdks"n|ddSdS)NgzFFor meaningful results, `beta` (temperature) parameter must be >> 0.0!rz@For meaningful results, `expectile` parameter must be in (0, 1).)rvalidater _value_errorrrr r"r#r;s  zIQLConfig.validatecstjd|jiBS)Nr)r_model_config_auto_includesrr=r r"r#r>sz%IQLConfig._model_config_auto_includesN)r$N)__name__ __module__ __qualname____doc__rr rrrboolfloatrintr%rrstrr-rr1r7r;propertyr> __classcell__r"r"r r#rsN*  9  rc@s*eZdZdZeeedefddZdS)rzOImplicit Q-learning (derived from MARWIL). Uses MARWIL training step. r$cCstSr?)r)clsr"r"r#get_default_configszIQL.get_default_configN) r@rArBrC classmethodr rrrKr"r"r"r#rs rN)typingrrr%ray.rllib.algorithms.algorithm_configrr"ray.rllib.algorithms.marwil.marwilrrCray.rllib.connectors.common.add_observations_from_episodes_to_batchr Oray.rllib.connectors.learner.add_next_observations_from_episodes_to_train_batchr ray.rllib.core.learner.learnerr "ray.rllib.core.rl_module.rl_moduler ray.rllib.utils.annotationsr ray.rllib.utils.typingrrrrr"r"r"r#s     K