o $ix @sddlZddlZddlZddlmZmZmZmZmZddl Z ddl Z ddl Z ddlmZmZddlmZddlmZddlmZmZmZddlmZddlmZdd lmZmZm Z m!Z!m"Z"ertdd l#m$Z$m%Z%dd l&m'Z'dd l(m)Z)e*e+Z,e\Z-Z.d Z/dZ0e-re1dZ2ne3dZ2edddede"dee4e"ffddZ5eddddddee6de4de"fddZ7ed d!de"fd"d#Z8eded$dee4e"ffd%d&Z9e  ' ' dmd(ee4d)e:d*e:d+eed,fd-d.Z;ednd/e!d(ee4fd0d1Zed;ee"de"fdKs z%apply_grad_clipping..params grad_gnorm)confignpinf param_groupslistfilternnutilsclip_grad_norm_ isinstancetorchTensorcpunumpyminlen) rrrr& clip_valuenum_none_grads param_groupr$ global_normr!r!r"apply_grad_clipping-s      r;value)r grad_clip_bygradients_dictrrr=c Cs&|durdS|dvrtd|d|dkr1|D]\}}|dur$dnt|| |||<qdS|dkrY|D]\}}|durV|djdd d }||krV|||q9dSt|}t |}t |d krk|S|tj |d |d }tj |dd} |D]} | dur| | | jq|S)a?Performs gradient clipping on a grad-dict based on a clip value and clip mode. Changes the provided gradient dict in place. Args: gradients_dict: The gradients dict, mapping str to gradient tensors. grad_clip: The value to clip with. The way gradients are clipped is defined by the `grad_clip_by` arg (see below). grad_clip_by: One of 'value', 'norm', or 'global_norm'. Returns: If `grad_clip_by`="global_norm" and `grad_clip` is not None, returns the global norm of all tensors, otherwise returns None. N)r<normr:z`grad_clip_by` (z*) must be one of [value|norm|global_norm]!r<r?eeAneginfposinfrgư>)r5?)max) ValueErroritemsr1clipr? nan_to_nummul_r+valuescompute_global_normr6clampdetachtodevice) r>rr=kvr?gradients_list total_norm clip_coeffclip_coeff_clampedgr!r!r"clip_gradients`s<   rZrUrcsHdt|dkr tdSttfdd|Djddd}|S) zComputes the global norm for a gradients dict. Args: gradients_list: The gradients list containing parameters. Returns: Returns the global norm of all tensors in `gradients_list`. @rcs.g|]}|durt|jdddqS)NrArBrC)r1r?rPrK).0rY norm_typer!r" sz'compute_global_norm..rArBrC)r6r1tensorr?stackrK)rUrVr!r^r"rNs     rN)rrcs4tjfddjDdd}|_|t|dS)azConcatenates multi-GPU (per-tower) TD error tensors given TorchPolicy. TD-errors are extracted from the TorchPolicy via its tower_stats property. Args: policy: The TorchPolicy to extract the TD-error values from. Returns: A dict mapping strings "td_error" and "mean_td_error" to the corresponding concatenated and mean-reduced values. cs*g|]}|jdtdgjqS)td_errorr\) tower_statsgetr1rarQrRr]trr!r"r`sz.concat_multi_gpu_td_errors..rdim)rc mean_td_error)r1catmodel_gpu_towersrcmean)rrcr!rhr"concat_multi_gpu_td_errorss roFrR pin_memory use_streamstream)ztorch.cuda.Streamztorch.cuda.classes.Streamcsdur tntdjdkotjrE|r=dur7ttjjtjjjfs6Jdtdntjn tjjdndfddt |S) aj Converts any (possibly nested) structure to torch.Tensors. Args: x: The input structure whose leaves will be converted. device: The device to create the tensor on (e.g. "cuda:0" or "cpu"). pin_memory: If True, calls `pin_memory()` on the created tensors. use_stream: If True, uses a separate CUDA stream for `Tensor.to()`. stream: An optional CUDA stream for the host-to-device copy in `Tensor.to()`. Returns: A new structure with the same layout as `x` but with all leaves converted to torch.Tensors. Leaves that are None are left unchanged. Nr3cudaz-`stream` must be a torch.cuda.Stream but got .rRcsb|dur|St|trtt|j|j|jSt|r |}nEt|t j r]|j t ks2|j j t jur4|S|jjsWttdt|}Wdn1sQwYnt|}ntt |}|rs|j tjkrs|}r{r{|}rdurtj|jdd}Wd|S1swY|S|jdd}|S|}|S)NignoreT) non_blocking)r0r tree map_structurerMlengthsmax_lenr1 is_tensorr(ndarraydtypeobjecttypestr_flags writeablewarningscatch_warnings simplefilter from_numpyasarrayis_floating_pointfloat16floatrprsrrrQ)itemrarRis_cudamappingrprrr!r"rsJ          z(convert_to_torch_tensor..mapping) r1rRrrs is_availabler0Streamclassesdefault_streamrxry)xrRrprqrrr!rr"convert_to_torch_tensors  4rrcsfdd}t||S)aCreates a copy of `x` and makes deep copies torch.Tensors in x. Also moves the copied tensors to the specified device (if not None). Note if an object in x is not a torch.Tensor, it will be shallow-copied. Args: x : Any (possibly nested) struct possibly containing torch.Tensors. device : The device to move the tensors to. Returns: Any: A new struct with the same structure as `x`, but with all torch.Tensors deep-copied and moved to the specified device. cs4t|tjrdurt|S|S|Sr)r0r1r2clonerPrQ)rrur!r"rXs   z#copy_torch_tensors..mapping)rxry)rrRrr!rur"copy_torch_tensorsFs  rypredcCs\|}tj|dd}tj||dd}tdg|j}t|d||tdS)a,Computes the explained variance for a pair of labels and predictions. The formula used is: max(-1.0, 1.0 - (std(y - pred)^2 / std(y)^2)) Args: y: The labels. pred: The predictions. Returns: The explained variance given a pair of labels and predictions. rrigr%)squeezer1varrarQrRrGr)rr squeezed_yy_vardiff_varmin_r!r!r"explained_variancees rinputs spaces_struct time_axisc Cs8t|}|durt|ndgt|}d}d}g}t||D]g\}} |dur4|jd}|r4|jd}t| trO|rDt|||g}| t ||  q t| t rk|r`t|||dg}| t ||  q |rxt|||dg}nt||dg}| | q tj |dd} |rt| ||dg} | S)ao Flattens arbitrary input structs according to the given spaces struct. Returns a single 1D tensor resulting from the different input components' values. Thereby: - Boxes (any shape) get flattened to (B, [T]?, -1). Note that image boxes are not treated differently from other types of Boxes and get flattened as well. - Discrete (int) values are one-hot'd, e.g. a batch of [1, 0, 3] (B=3 with Discrete(4) space) results in [[0, 1, 0, 0], [1, 0, 0, 0], [0, 0, 0, 1]]. - MultiDiscrete values are multi-one-hot'd, e.g. a batch of [[0, 2], [1, 4]] (B=2 with MultiDiscrete([2, 5]) space) results in [[1, 0, 0, 0, 1, 0, 0], [0, 1, 0, 0, 0, 0, 1]]. Args: inputs: The inputs to be flattened. spaces_struct: The structure of the spaces that behind the input time_axis: Whether all inputs have a time-axis (after the batch axis). If True, will keep not only the batch axis (0th), but the time axis (1st) as-is and flatten everything from the 2nd axis up. Returns: A single 1D tensor resulting from concatenating all flattened/one-hot'd input components. Depending on the time_axis flag, the shape is (B, n) or (B, T, n). .. testcode:: from gymnasium.spaces import Discrete, Box from ray.rllib.utils.torch_utils import flatten_inputs_to_1d_tensor import torch struct = { "a": np.array([1, 3]), "b": ( np.array([[1.0, 2.0], [4.0, 5.0]]), np.array( [[[8.0], [7.0]], [[5.0], [4.0]]] ), ), "c": { "cb": np.array([1.0, 2.0]), }, } struct_torch = tree.map_structure(lambda s: torch.from_numpy(s), struct) spaces = dict( { "a": gym.spaces.Discrete(4), "b": (gym.spaces.Box(-1.0, 10.0, (2,)), gym.spaces.Box(-1.0, 1.0, (2, 1))), "c": dict( { "cb": gym.spaces.Box(-1.0, 1.0, ()), } ), } ) print(flatten_inputs_to_1d_tensor(struct_torch, spaces_struct=spaces)) .. testoutput:: tensor([[0., 1., 0., 0., 1., 2., 8., 7., 1.], [0., 0., 0., 1., 4., 5., 5., 4., 2.]]) Nrr%ri)rxflattenr6zipshaper0rr1reshapeappendone_hotrrrl) rrr flat_inputs flat_spacesBToutinput_spacemergedr!r!r"flatten_inputs_to_1d_tensorzs: H     rtensorscCs(dd|D}ttdd|DdS)aNReturns the global L2 norm over a list of tensors. output = sqrt(SUM(t ** 2 for t in tensors)), where SUM reduces over all tensors and over all elements in tensors. Args: tensors: The list of tensors to calculate the global norm over. Returns: The global L2 norm over the given tensor list. c Ss&g|]}ttt|ddqS)r[?r1powsumrfr!r!r"r`s&zglobal_norm..css|] }t|dVqdS)r[N)r1r)r]l2r!r!r" szglobal_norm..rr)r single_l2sr!r!r"r:sr:rFdeltacCs6tt||kt|dd|t|d|S)aComputes the huber loss for a given term and delta parameter. Reference: https://en.wikipedia.org/wiki/Huber_loss Note that the factor of 0.5 is implicitly included in the calculation. Formula: L = 0.5 * x^2 for small abs x (delta threshold) L = delta * (abs(x) - 0.5*delta) for larger abs x (delta threshold) Args: x: The input term, e.g. a TD error. delta: The delta parmameter in the above formula. Returns: The Huber loss resulting from `x` and `delta`. r[r)r1whereabsr)rrr!r!r" huber_losss  rcCsdtt|dS)zComputes half the L2 norm over a tensor's values without the sqrt. output = 0.5 * sum(x ** 2) Args: x: The input tensor. Returns: 0.5 times the L2 norm over the given tensor's values (w/o sqrt). rr[)r1rrrr!r!r"l2_losss rrcst|trtj|jSt|tr@t|jdt j r-t |j} j ddn|j}tjfddt|DddStd|)aReturns a one-hot tensor, given and int tensor and a space. Handles the MultiDiscrete case as well. Args: x: The input tensor. space: The space to use for generating the one-hot tensor. Returns: The resulting one-hot tensor. Raises: ValueError: If the given space is not a discrete one. .. testcode:: import torch import gymnasium as gym from ray.rllib.utils.torch_utils import one_hot x = torch.IntTensor([0, 3]) # batch-dim=2 # Discrete space with 4 (one-hot) slots per batch item. s = gym.spaces.Discrete(4) print(one_hot(x, s)) x = torch.IntTensor([[0, 1, 2, 3]]) # batch-dim=1 # MultiDiscrete space with 5 + 4 + 4 + 7 = 20 (one-hot) slots # per batch item. s = gym.spaces.MultiDiscrete([5, 4, 4, 7]) print(one_hot(x, s)) .. testoutput:: tensor([[1, 0, 0, 0], [0, 0, 0, 1]]) tensor([[1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0]]) rrcs.g|]\}}tjdd|f|qSr)r- functionalrlong)r]inrr!r"r`Vs.zone_hot..riz#Unsupported space for `one_hot`: {})r0rr-rrrrrnvecr(r}ravelrrr1rl enumeraterHformat)rrrr!rr"r(s %  raxiscCs@t|td}t||t|}t||t||S)zSame as torch.mean() but ignores -inf values. Args: x: The input tensor to reduce mean over. axis: The axis over which to reduce. None for all axes. Returns: The mean reduced inputs, ignoring inf values. z-inf)r1nerr zeros_liker)rrmaskx_zeroedr!r!r"reduce_mean_ignore_inf]s rrzmaxlen time_majorcCsh|dur|}tt|jt|f}||jjdd |k}|s*| }| |p0tj |S)alOffers same behavior as tf.sequence_mask for torch. Thanks to Dimitris Papatheodorou (https://discuss.pytorch.org/t/pytorch-equivalent-for-tf-sequence-mask/ 39036). Args: lengths: The tensor of individual lengths to mask by. maxlen: The maximum length to use for the time axis. If None, use the max of `lengths`. dtype: The torch dtype to use for the resulting mask. time_major: Whether to return the mask as [B, T] (False; default) or as [T, B] (True). Returns: The sequence mask resulting from the given input and parameters. Nr%ri) rGr1onestuplerintrQrRcumsumrgrbool)rzrr~rrr!r!r" sequence_maskmsrmain_net target_nettaucs2|fdd|D}||dS)aUpdates a torch.nn.Module target network using Polyak averaging. .. code-block:: text new_target_net_weight = ( tau * main_net_weight + (1.0 - tau) * current_target_net_weight ) Args: main_net: The nn.Module to update from. target_net: The target network to update. tau: The tau value to use in the Polyak averaging formula. cs*i|]\}}||d|qS)r%r!)r]rSrT state_dictrr!r" sz)update_target_network..N)rrIload_state_dict)rrrnew_state_dictr!rr"update_target_networks   r kl_divergencecCs&|r|rtddSdSdS)Na}KL divergence is non-finite, this will likely destabilize your model and the training process. Action(s) in a specific state have near-zero probability. This can happen naturally in deterministic environments where the optimal policy has zero mass for a specific action. To fix this issue, consider setting the coefficient for the KL loss term to zero or increasing policy entropy.)loss_initializedisinfloggerwarning)rrr!r!r"warn_if_infinite_kl_divergences rseedcCs|durOtrQt|tjj}|dur-ttjjdkr-dtjd<tj|tj|nttj tdkr>t dnt ddtj j _dtj j _dSdSdS)ztSets the torch random seed to the given value. Args: seed: The seed to use or None for no seeding. Ngffffff$@z:4096:8CUBLAS_WORKSPACE_CONFIGz1.8.0TF)r1 manual_seedr rsrosenvironmanual_seed_allVersion __version__use_deterministic_algorithmsset_deterministicbackendscudnn deterministic benchmark)r cuda_versionr!r!r"set_torch_seeds       rlogitslabelscCst| tj|ddS)zSame behavior as tf.nn.softmax_cross_entropy_with_logits. Args: x: The input predictions. labels: The labels corresponding to `x`. Returns: The resulting softmax cross-entropy given predictions and labels. r)r1rr-r log_softmax)rrr!r!r"!softmax_cross_entropy_with_logitssr torch.TensorcCst|tt|dS)zThe symlog function as described in [1]: [1] Mastering Diverse Domains through World Models - 2023 D. Hafner, J. Pasukonis, J. Ba, T. Lillicrap https://arxiv.org/pdf/2301.04104v1.pdf r%)r1signlogrrr!r!r"symlogsrcCst|tt|dS)zInverse of the `symlog` function as desribed in [1]: [1] Mastering Diverse Domains through World Models - 2023 D. Hafner, J. Pasukonis, J. Ba, T. Lillicrap https://arxiv.org/pdf/2301.04104v1.pdf r%)r1rexpr)rr!r!r"inverse_symlogsr44@ num_buckets lower_bound upper_boundcCsDt|||}tjd|jd|d}|||d}| ||}t|}t|} t|| | d| } t| || d| } |||} || |} || | | } d| } tj ||gdd}tj || gdd}tj ||gdd }tj | | gdd}tj |jd||d}|||dddf|dddff<|S) a+Returns a two-hot vector of dim=num_buckets with two entries that are non-zero. See [1] for more details: [1] Mastering Diverse Domains through World Models - 2023 D. Hafner, J. Pasukonis, J. Ba, T. Lillicrap https://arxiv.org/pdf/2301.04104v1.pdf Entries in the vector represent equally sized buckets within some fixed range (`lower_bound` to `upper_bound`). Those entries not 0.0 at positions k and k+1 encode the actual `value` and sum up to 1.0. They are the weights multiplied by the buckets values at k and k+1 for retrieving `value`. Example: num_buckets=11 lower_bound=-5 upper_bound=5 value=2.5 -> [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.5, 0.5, 0.0, 0.0] -> [-5 -4 -3 -2 -1 0 1 2 3 4 5] (0.5*2 + 0.5*3=2.5) Example: num_buckets=5 lower_bound=-1 upper_bound=1 value=0.1 -> [0.0, 0.0, 0.8, 0.2, 0.0] -> [-1 -0.5 0 0.5 1] (0.2*0.5 + 0.8*0=0.1) Args: value: The input tensor of shape (B,) to be two-hot encoded. num_buckets: The number of buckets to two-hot encode into. lower_bound: The lower bound value used for the encoding. If input values are lower than this boundary, they will be encoded as `lower_bound`. upper_bound: The upper bound value used for the encoding. If input values are higher than this boundary, they will be encoded as `upper_bound`. Returns: The two-hot encoded tensor of shape (B, num_buckets). rrur%rFr[rriN) r1rOarangerrfloorceilreqrbrlrzeros)r<rrrrR batch_indices bucket_deltaidxrSkp1values_k values_kp1 weights_k weights_kp1 indices_k indices_kp1indicesupdatesoutputr!r!r"two_hots&1    $rcCs(z ddlm}WdStyYdSw)NrTF) torch._dynamo_dynamo ImportError)dynamor!r!r"_dynamo_is_availablens   r)NFFNr)NF)rF)NNF)rrrr)rrrr)rrrN)Ologgingrrtypingrrrrr gymnasiumgymr4r(rxgymnasium.spacesrr packagingr ray.rllib.models.repeated_valuesr ray.rllib.utils.annotationsr r r ray.rllib.utils.frameworkrray.rllib.utils.numpyrray.rllib.utils.typingrrrrrray.rllib.core.learner.learnerrrray.rllib.policy.torch_policyr ray.rllib.policy.torch_policy_v2r getLogger__name__rr1r- FLOAT_MIN FLOAT_MAXparseTORCH_COMPILE_REQUIRED_VERSIONrHstrr;rrZrNrorrrrrr:rrSpacerrrrrrrrrrrrr!r!r!r"s>           2 C#    ` r4 (      X