o $i8N@sddlmZddlmZddlmZmZddlZddl Z ddl m Z m Z ddl mZddlmZddlmZmZdd lmZmZmZmZe\ZZZe\ZZd Zd Zd Z d Z!eeddddKde"de"dej#fddZ$eeddd dLdeej#dee%dej#fddZ&edMdede%defddZ'e  dNdej#d ej#d!eej#d"ee(dej#f d#d$Z)e   dOd%ed&eed'e%d(e%def d)d*Z*ed+d,Z+edPdej#d.e,dej#fd/d0Z-edej#dej#fd1d2Z.e    -dQd ej#d!eej#d3eej#de%d4e,f d5d6Z/edd-d7ej0fdeee"fd8e"d9e,d:e,d;e1dej#f dd?Z3edRdej#d@e,dej#fdAdBZ4edSdej#dCe%dej#fdDdEZ5e dTdeej#e6fdGe"dHee,dej#fdIdJZ7dS)U) OrderedDict)MappingProxyType)ListOptionalN)Discrete MultiDiscrete) Deprecated) PublicAPI) try_import_tftry_import_torch) SpaceStructTensorStructType TensorTypeUniongư>iz)RLlib itself has no use for this anymore.F)helperror@sizealignreturncCs||j}tj||dtjd}|jj|}|dkrdn||}|dkr5|||ddd|}n |||||}t||ksLJt||jj|dksZJ|jj|S)aReturns an array of a given size that is 64-byte aligned. The returned array can be efficiently copied into GPU memory by TensorFlow. Args: size: The size (total number of items) of the array. For example, array([[0.0, 1.0], [2.0, 3.0]]) would have size=4. dtype: The numpy dtype of the array. align: The alignment to use. Returns: A np.ndarray with the given specifications. dtyper)itemsizenpemptyuint8ctypesdataviewlen)rrrnr data_alignoffsetoutputr'R/home/ubuntu/veenaModal/venv/lib/python3.10/site-packages/ray/rllib/utils/numpy.py aligned_arrays   r)items time_majorcCsvt|dkrgSt|dkr|dSt|dtjr|djtjtjtjfvr|dj}tt dd|D|}|durx|dur\t dd|D}|dj d|f|dj dd}n1t d d|D}||dj df|dj dd}nt d d|D}|f|dj dd}| |}|j j d dksJ|j j tj|||rdndd |Stj||rdd Sdd S)a Concatenate arrays, ensuring the output is 64-byte aligned. We only align float arrays; other arrays are concatenated as normal. This should be used instead of np.concatenate() to improve performance when the output array is likely to be fed into TensorFlow. Args: items: The list of items to concatenate and align. time_major: Whether the data in items is time-major, in which case, we will concatenate along axis=1. Returns: The concat'd and aligned array. rrcss|]}|jVqdSN)r.0sr'r'r( asz!concat_aligned..NTcs|]}|jdVqdS)rNshaper-r'r'r(r0drcsr1rNr2r-r'r'r(r0ir4csr1r5r2r-r'r'r(r0nr4r)outaxisr7)r" isinstancerndarrayrfloat32float64rr)sumr3reshaperr concatenate)r*r+rflat batch_dim new_shaper&r'r'r(concat_aligned=s@    rCTx reduce_typecsfdd}t||S)aConverts values in `stats` to non-Tensor numpy or python types. Args: x: Any (possibly nested) struct, the values in which will be converted and returned as a new struct with all torch/tf tensors being converted to numpy types. reduce_type: Whether to automatically reduce all float64 and int64 data into float32 and int32 data, respectively. Returns: A new struct with the same structure as `x`, but with all values converted to numpy arrays (on CPU). cstrt|tjrt|dkr|n|}nt r:t|t jt j fr:t |dr:t s5J|}n|}rat|t jrat |jt jrT|t j}|St |jtra|t j}|S)Nrnumpy)torchr9Tensorr"rcpuitemdetachrFtfVariablehasattrexecuting_eagerlyrr: issubdtyperfloatingastyper;intint32)rJretrEr'r(mappings*     z!convert_to_numpy..mapping)tree map_structure)rDrErWr'rVr(convert_to_numpyxs  rZweightsbiases frameworkcCstd dd}||}|dko |jd|jdko |jd|jdk}|||d}||}t|||dur7d S|S) aCalculates FC (dense) layer outputs given weights/biases and input. Args: x: The input to the dense layer. weights: The weights matrix. biases: The biases vector. All 0s if None. framework: An optional framework hint (to figure out, e.g. whether to transpose torch weight matrices). Returns: The dense layer's output. FcSsRtrt|tjr|}tr tr t|tjr |}|r't |}|Sr,) rGr9rHrIrKrFrLrOrMr transpose)r r^r'r'r(map_s    zfc..map_rGrr)r^NF)r3rmatmul)rDr[r\r]r_r^r'r'r(fcs  & rcinputs spaces_struct time_axis batch_axisc s|r|sJt|}|durt|ndgt|}d}d}g}t||D]\} |dur<|r<jd}|r<jd}t| tr[|rLt||g| t | j d tj q&t| tr|rlt||dg|r| tjfddt| jDddq&| tjfd dt| jDddq&ttrtg|rt||dgn|rt|dgntdg| tj q&tj|dd} |rt| ||dg} | S) a Flattens arbitrary input structs according to the given spaces struct. Returns a single 1D tensor resulting from the different input components' values. Thereby: - Boxes (any shape) get flattened to (B, [T]?, -1). Note that image boxes are not treated differently from other types of Boxes and get flattened as well. - Discrete (int) values are one-hot'd, e.g. a batch of [1, 0, 3] (B=3 with Discrete(4) space) results in [[0, 1, 0, 0], [1, 0, 0, 0], [0, 0, 0, 1]]. - MultiDiscrete values are multi-one-hot'd, e.g. a batch of [[0, 2], [1, 4]] (B=2 with MultiDiscrete([2, 5]) space) results in [[1, 0, 0, 0, 1, 0, 0], [0, 1, 0, 0, 0, 0, 1]]. Args: inputs: The inputs to be flattened. spaces_struct: The (possibly nested) structure of the spaces that `inputs` belongs to. time_axis: Whether all inputs have a time-axis (after the batch axis). If True, will keep not only the batch axis (0th), but the time axis (1st) as-is and flatten everything from the 2nd axis up. batch_axis: Whether all inputs have a batch axis. If True, will keep that batch axis as-is and flatten everything from the other dims up. Returns: A single 1D tensor resulting from concatenating all flattened/one-hot'd input components. Depending on the time_axis flag, the shape is (B, n) or (B, T, n). .. testcode:: :skipif: True # B=2 from ray.rllib.utils.tf_utils import flatten_inputs_to_1d_tensor from gymnasium.spaces import Discrete, Box out = flatten_inputs_to_1d_tensor( {"a": [1, 0], "b": [[[0.0], [0.1]], [1.0], [1.1]]}, spaces_struct=dict(a=Discrete(2), b=Box(shape=(2, 1))) ) print(out) # B=2; T=2 out = flatten_inputs_to_1d_tensor( ([[1, 0], [0, 1]], [[[0.0, 0.1], [1.0, 1.1]], [[2.0, 2.1], [3.0, 3.1]]]), spaces_struct=tuple([Discrete(2), Box(shape=(2, ))]), time_axis=True ) print(out) .. testoutput:: [[0.0, 1.0, 0.0, 0.1], [1.0, 0.0, 1.0, 1.1]] # B=2 n=4 [[[0.0, 1.0, 0.0, 0.1], [1.0, 0.0, 1.0, 1.1]], [[1.0, 0.0, 2.0, 2.1], [0.0, 1.0, 3.0, 3.1]]] # B=2 T=2 n=4 Nrrdepthcs0g|]\}}tdd|f|dtjqS)Nrhone_hotrRrr;r.ir#input_r'r( -sz/flatten_inputs_to_1d_tensor..r8cs(g|]\}}t||dtjqS)rhrkrmror'r(rq7s)rXflattenr"zipr3r9rrr>appendrlr#rRr;rr? enumeratenvecfloatarray) rdrerfrg flat_inputs flat_spacesBTr6spacemergedr'ror(flatten_inputs_to_1d_tensorsd B           rcCsHt|tjr|jdd|St|trtt|St|tr"t|S|S)a{Flags actions immutable to notify users when trying to change them. Can also be used with any tree-like structure containing either dictionaries, numpy arrays or already immutable objects per se. Note, however that `tree.map_structure()` will in general not include the shallow object containing all others and therefore immutability will hold only for all objects contained in it. Use `tree.traverse(fun, action, top_down=False)` to include also the containing object. Args: obj: The object to be made immutable. Returns: The immutable object. .. testcode:: :skipif: True import tree import numpy as np from ray.rllib.utils.numpy import make_action_immutable arr = np.arange(1,10) d = dict(a = 1, b = (arr, arr)) tree.traverse(make_action_immutable, d, top_down=False) F)write)r9rr:setflagsrrdict)objr'r'r(make_action_immutableSs     r?deltacCs6tt||kt|dd|t|d|S)z4Reference: https://en.wikipedia.org/wiki/Huber_loss.@g?)rwhereabspower)rDrr'r'r( huber_losszs.rcCstt|dS)zComputes half the L2 norm of a tensor (w/o the sqrt): sum(x**2) / 2. Args: x: The input tensor. Returns: The l2-loss output according to the above formula given `x`. r)rr=square)rDr'r'r(l2_losss rinitial_internal_states forget_biascCs|j|rdnd}|j|rdnd}|jdd}|dur.tj||fd} tj||fd} n|d} |d} |rBtj|||fd} n tj|||fd} t|D]} |r^|| ddddfn |dd| ddf} tj| | fdd} t| ||}t|dd|d|df|}t| |} t|ddd|f}t|dd||df}t | t||} t|dd|d|df}t|t| } |r| | | ddddf<qO| | dd| ddf<qO| | | ffS) aCalculates LSTM layer output given weights/biases, states, and input. Args: x: The inputs to the LSTM layer including time-rank (0th if time-major, else 1st) and the batch-rank (1st if time-major, else 0th). weights: The weights matrix. biases: The biases vector. All 0s if None. initial_internal_states: The initial internal states to pass into the layer. All 0s if None. time_major: Whether to use time-major or not. Default: False. forget_bias: Gets added to first sigmoid (forget gate) output. Default: 1.0. Returns: Tuple consisting of 1) The LSTM layer's output and 2) Tuple: Last (c-state, h-state). rrNr2r8r) r3rzerosranger?rbsigmoidmultiplytanhadd)rDr[r\rr+rsequence_length batch_sizeunitsc_statesh_statesunrolled_outputst input_matrixinput_matmul_matrix sigmoid_1 sigmoid_2tanh_3 sigmoid_4r'r'r(lstms4 0$   rr`rion_value off_valuerc CsDt|trtj|tjd}n trt|tjr|}|jtj kr(| tj }d}|dkr3t |d}t ||ksEJd t |||j}tjg||Rd|}g}t|jD]5}dg|j} dg|j} d| |<t||| } |dkr||d| |d<t| | } || q\||||t|<| |S)aOne-hot utility function for numpy. Thanks to qianyizhang: https://gist.github.com/qianyizhang/07ee1c15cad08afb03f5de69349efc30. Args: x: The input to be one-hot encoded. depth: The max. number to be one-hot encoded (size of last rank). on_value: The value to use for on. Default: 1.0. off_value: The value to use for off. Default: 0.0. Returns: The one-hot encoded equivalent of the input array. rrrrztilerttuple) rDrirrrr3r6indicesrntilesr/rr'r'r(rls:           rlcs@tr ttjr jtjfddt|DddS)NcsDg|]\}}ttdkr|ndd|f|dtjqS)rNrh)rlr"rRrr;rmr3rDr'r(rqs2z)one_hot_multidiscrete..rjr8)rGr9rHrFr3rr?ru)rDdepthsr'rr(one_hot_multidiscretes ralphacCst||||S)zImplementation of the leaky ReLU function. y = x * alpha if x < 0 else x Args: x: The input values. alpha: A scaling ("leak") factor to use for negative x. Returns: The leaky ReLU output for x. )rmaximum)rDrr'r'r(relu"s r derivativecCs$|r|d|Sddt| S)aY Returns the sigmoid function applied to x. Alternatively, can return the derivative or the sigmoid function. Args: x: The input to the sigmoid function. derivative: Whether to return the derivative or not. Default: False. Returns: The sigmoid function (or its derivative) applied to x. r)rexp)rDrr'r'r(r2s rrjr7epsiloncCs.|pt}t|}t|tj||dd|S)a{Returns the softmax values for x. The exact formula used is: S(xi) = e^xi / SUMj(e^xj), where j goes over all elements in x. Args: x: The input to the softmax function. axis: The axis along which to softmax. epsilon: Optional epsilon as a minimum value. If None, use `SMALL_NUMBER`. Returns: The softmax over x. T)keepdims) SMALL_NUMBERrrrr=)rDr7rx_expr'r'r(softmaxFs r)rr,)T)NN)NFT)r)NNFr)r`ra)rjN)8 collectionsrtypesrtypingrrrFrrXgymnasium.spacesrrray._common.deprecationrray.rllib.utils.annotationsr ray.rllib.utils.frameworkr r ray.rllib.utils.typingr r rrtf1rLtfvrG_r LARGE_INTEGERMIN_LOG_NN_OUTPUTMAX_LOG_NN_OUTPUTrSr:r)boolrCrZstrrcrrrwrrrr;typerlrrrlistrr'r'r'r(s     6) )  &  E  <