o `۷i\@sdZddlZddlmZmZmZddlmZddlm Z m Z ddl m Z m Z ddlmZddlmZdd lmZdd lmZeeZGd d d e ZGd dde ZdS)a4 TQC (Truncated Quantile Critics) Algorithm. Paper: https://arxiv.org/abs/2005.04269 "Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Critics" TQC extends SAC by using distributional RL with quantile regression to control overestimation bias in the Q-function. N)OptionalTypeUnion) Algorithm)AlgorithmConfig NotProvided)SAC SACConfig)Learner) RLModuleSpec)override)RLModuleSpecTypec seZdZdZdfdd Zeeeeeddee dee dee ffd d Z ee dfd d Z ee d e fddZee d eedeffddZeee fddZZS) TQCConfigaConfiguration for the TQC algorithm. TQC extends SAC with distributional critics using quantile regression. Example: >>> from ray.rllib.algorithms.tqc import TQCConfig >>> config = ( ... TQCConfig() ... .environment("Pendulum-v1") ... .training( ... n_quantiles=25, ... n_critics=2, ... top_quantiles_to_drop_per_net=2, ... ) ... ) >>> algo = config.build() Ncs(tj|ptdd|_d|_d|_dS)z!Initializes a TQCConfig instance.) algo_classN)super__init__TQC n_quantiles n_criticstop_quantiles_to_drop_per_net)selfr __class__R/home/ubuntu/vllm_env/lib/python3.10/site-packages/ray/rllib/algorithms/tqc/tqc.pyr-s zTQCConfig.__init__rrrrrrc s@tjdi||tur||_|tur||_|tur||_|S)a8Sets the training-related configuration. Args: n_quantiles: Number of quantiles for each critic network. Default is 25. n_critics: Number of critic networks. Default is 2. top_quantiles_to_drop_per_net: Number of quantiles to drop per network when computing the target Q-value. This controls the overestimation bias. Default is 2. **kwargs: Additional arguments passed to SACConfig.training(). Returns: This updated TQCConfig object. Nr)rtrainingrrrr)rrrrkwargsrrrr6szTQCConfig.trainingreturncst|jdkrtd|j|jdkrtd|j|jdkr,td|j|j|j}|j|j}||krGtd|d|dd S) z Validates the TQC configuration.z `n_quantiles` must be >= 1, got z`n_critics` must be >= 1, got rz2`top_quantiles_to_drop_per_net` must be >= 0, got z Cannot drop z quantiles when only zp total quantiles are available. Reduce `top_quantiles_to_drop_per_net` or increase `n_quantiles` or `n_critics`.N)rvalidater ValueErrorrr)rtotal_quantilesquantiles_to_droprrrr"Xs(      zTQCConfig.validatecCs2|jdkrddlm}t|dStd|jd)Ntorchr)DefaultTQCTorchRLModule) module_classThe framework  is not supported. Use `torch`.) framework_str:ray.rllib.algorithms.tqc.torch.default_tqc_torch_rl_moduler'r r#)rr'rrrget_default_rl_module_specus    z$TQCConfig.get_default_rl_module_specr cCs,|jdkr ddlm}|Std|jd)Nr&r)TQCTorchLearnerr)r*)r+0ray.rllib.algorithms.tqc.torch.tqc_torch_learnerr.r#)rr.rrrget_default_learner_classs   z#TQCConfig.get_default_learner_classcstj|j|j|jdBS)Nr)r_model_config_auto_includesrrr)rrrrr1s z%TQCConfig._model_config_auto_includesN)r N)__name__ __module__ __qualname____doc__rr r rrintrrr"r r-rrstrr0propertyr1 __classcell__rrrrrs. !  rc@s*eZdZdZeeedefddZdS)raTQC (Truncated Quantile Critics) Algorithm. TQC extends SAC by using distributional critics with quantile regression and truncating the top quantiles to control overestimation bias. Key differences from SAC: - Uses multiple critic networks, each outputting multiple quantiles - Computes target Q-values by sorting and truncating top quantiles - Uses quantile Huber loss for critic training See the paper for more details: https://arxiv.org/abs/2005.04269 r cCstSr2)r)clsrrrget_default_configszTQC.get_default_configN) r3r4r5r6 classmethodr rrr<rrrrrs r)r6loggingtypingrrrray.rllib.algorithms.algorithmr%ray.rllib.algorithms.algorithm_configrrray.rllib.algorithms.sac.sacrr ray.rllib.core.learnerr "ray.rllib.core.rl_module.rl_moduler ray.rllib.utils.annotationsr ray.rllib.utils.typingr getLoggerr3loggerrrrrrrs