o ci!@sbdZddlmZddlZddlZddlmZddlm Z m Z ddlm Z eGdddej Z dS) z;This is the next version of action distribution base class.)TupleN)ExperimentalAPI) TensorTypeUnion)overridec @s eZdZdZejddddeedfdede e ee e fffd d Z ejddddeedfdede e ee e fffd d Z ejd e de fddZ ejddde fddZejde fddZeejdejdefddZede ddfddZe   d ddZd!ddZdS)" DistributionaEThe base class for distribution over a random variable. Examples: .. testcode:: import torch from ray.rllib.core.models.configs import MLPHeadConfig from ray.rllib.models.torch.torch_distributions import TorchCategorical model = MLPHeadConfig(input_dims=[1]).build(framework="torch") # Create an action distribution from model logits action_logits = model(torch.Tensor([[1]])) action_dist = TorchCategorical.from_logits(action_logits) action = action_dist.sample() # Create another distribution from a dummy Tensor action_dist2 = TorchCategorical.from_logits(torch.Tensor([0])) # Compute some common metrics logp = action_dist.logp(action) kl = action_dist.kl(action_dist2) entropy = action_dist.entropy() NF) sample_shape return_logpr.r returncKdS)aDraw a sample from the distribution. Args: sample_shape: The shape of the sample to draw. return_logp: Whether to return the logp of the sampled values. **kwargs: Forward compatibility placeholder. Returns: The sampled values. If return_logp is True, returns a tuple of the sampled values and its logp. Nselfrr kwargsr r R/home/ubuntu/.local/lib/python3.10/site-packages/ray/rllib/models/distributions.pysample'zDistribution.samplecKr )aDraw a re-parameterized sample from the action distribution. If this method is implemented, we can take gradients of samples w.r.t. the distribution parameters. Args: sample_shape: The shape of the sample to draw. return_logp: Whether to return the logp of the sampled values. **kwargs: Forward compatibility placeholder. Returns: The sampled values. If return_logp is True, returns a tuple of the sampled values and its logp. Nr r r r rrsample;rzDistribution.rsamplevaluecKr )a The log-likelihood of the distribution computed at `value` Args: value: The value to compute the log-likelihood at. **kwargs: Forward compatibility placeholder. Returns: The log-likelihood of the value. Nr )rrrr r rlogpRrzDistribution.logpothercKr )zThe KL-divergence between two distributions. Args: other: The other distribution. **kwargs: Forward compatibility placeholder. Returns: The KL-divergence between the two distributions. Nr )rrrr r rkl^rzDistribution.klcKr )zThe entropy of the distribution. Args: **kwargs: Forward compatibility placeholder. Returns: The entropy of the distribution. Nr )rrr r rentropyjrzDistribution.entropyspacecKr )aReturns the required length of an input parameter tensor. Args: space: The space this distribution will be used for, whose shape attributes will be used to determine the required shape of the input parameter tensor. **kwargs: Forward compatibility placeholder. Returns: size of the required input vector (minus leading batch dimension). Nr )rrr r rrequired_input_dimurzDistribution.required_input_dimlogitscKst)aCreates a Distribution from logits. The caller does not need to have knowledge of the distribution class in order to create it and sample from it. The passed batched logits vectors might be split up and are passed to the distribution class' constructor as kwargs. Args: logits: The logits to create the distribution from. **kwargs: Forward compatibility placeholder. Returns: The created distribution. .. testcode:: import numpy as np from ray.rllib.models.distributions import Distribution class Uniform(Distribution): def __init__(self, lower, upper): self.lower = lower self.upper = upper def sample(self): return self.lower + (self.upper - self.lower) * np.random.rand() def logp(self, x): ... def kl(self, other): ... def entropy(self): ... @staticmethod def required_input_dim(space): ... def rsample(self): ... @classmethod def from_logits(cls, logits, **kwargs): return Uniform(logits[:, 0], logits[:, 1]) logits = np.array([[0.0, 1.0], [2.0, 3.0]]) my_dist = Uniform.from_logits(logits) sample = my_dist.sample() )NotImplementedError)clsrrr r r from_logitss4zDistribution.from_logits parent_clsc s&Gfddd}d|_|S)zReturns a partial child of TorchMultiActionDistribution. This is useful if inputs needed to instantiate the Distribution from logits are available, but the logits are not. csreZdZfddZefddZeedej de ffdd Z eed e dd ffd d Z ZS) z>Distribution.get_partial_dist_cls..DistributionPartialcstj|i|dS)N)super__init__)rargsr __class__r rr!szGDistribution.get_partial_dist_cls..DistributionPartial.__init__cs4t|t@}|rtd|di|}|S)z9Checks if keys in kwargs don't clash with partial_kwargs.z&Cannot override the following kwargs: zS. This is because they were already set at the time this partial class was defined.)set ValueError)roverlap merged_kwargs)partial_kwargsr r _merge_kwargss  zLDistribution.get_partial_dist_cls..DistributionPartial._merge_kwargsrr cs0|jdi|}||dksJjdi|S)Nrr )r*r)rrrr(rr rrszQDistribution.get_partial_dist_cls..DistributionPartial.required_input_dimrDistributionPartialcs,|jdi|}j|fi|}||_|S)Nr )r*rr$)rrrr( distributionr+r rrszJDistribution.get_partial_dist_cls..DistributionPartial.from_logits)__name__ __module__ __qualname__r! staticmethodr* classmethodrgymSpaceintrrr __classcell__r rr)r#rr,s  r,Partial)r.)rr)r,r r7rget_partial_dist_clss (z!Distribution.get_partial_dist_clscCs|S)a^Returns a deterministic equivalent for this distribution. Specifically, the deterministic equivalent for a Categorical distribution is a Deterministic distribution that selects the action with maximum logit value. Generally, the choice of the deterministic replacement is informed by established conventions. r )rr r rto_deterministicszDistribution.to_deterministic)rrr r)r r)r.r/r0__doc__abcabstractmethodrr5boolrrrrrrrr1r3r4rr2rr9r:r r r rr sP        5 5r)r;typingr gymnasiumr3r<ray.rllib.utils.annotationsrray.rllib.utils.typingrrrABCrr r r rs