o cihL@sddlZddlmZmZddlmZmZddlZ ddl m Z ddl Z ddl mZmZmZmZe Gddde jZe dejd ejfd d Ze dejd efd d Ze deeefdejd eeeeffddZe deeefdejd eeeeffddZe dejd eejfddZe ddZe d5ddddddejdedeeeefdeededed e jfd d!Ze d"d#Z e dd$d%eed&eeeffd'd(Z!e d)d*Z"e d+d,Z#e d-d.Z$e d/d0Z%e d1ed2ed efd3d4Z&dS)6N)TupleDict)ActTypeObsType) DeveloperAPI)AnyListOptionalUnionc@seZdZdZddZdS)BatchedNdArrayaA ndarray-wrapper the usage of which indicates that there a batch dim exists. This is such that our `batch()` utility can distinguish between having to stack n individual batch items (each one w/o any batch dim) vs having to concatenate n already batched items (each one possibly with a different batch dim, but definitely with some batch dim). TODO (sven): Maybe replace this by a list-override instead. cCst||}|SN)npasarrayview)cls input_arrayobjrV/home/ubuntu/.local/lib/python3.10/site-packages/ray/rllib/utils/spaces/space_utils.py__new__szBatchedNdArray.__new__N)__name__ __module__ __qualname____doc__rrrrrr s r spacereturncCst|dr t|jS|S)aRReturns the original space of a space, if any. This function recursively traverses the given space and returns the original space at the very end of the chain. Args: space: The space to get the original space for. Returns: The original space or the given space itself if no original space is found. original_space)hasattrget_original_spacerrrrrrs rcCs,t|tjjtjjtjjtjjfvrdSdS)aaReturns true, if the space is composite. Note, we follow here the glossary of `gymnasium` by which any spoace that holds other spaces is defined as being 'composite'. Args: space: The space to be checked for being composed of other spaces. Returns: True, if the space is composed of other spaces, otherwise False. TF)typegymspacesrGraphSequencerrrrris_composite_space/s r%samplecCst|r ||gS|S)aReturns a jsonabled space sample, if the space is composite. Checks, if the space is composite and converts the sample to a jsonable struct in this case. Otherwise return the sample as is. Args: sample: Any action or observation type possible in `gymnasium`. space: Any space defined in `gymnasium.spaces`. Returns: The `sample` as-is, if the `space` is composite, otherwise converts the composite sample to a JSONable data type. )r% to_jsonabler&rrrrto_jsonable_if_neededGs r)cCst|r ||dS|S)aReturns a jsonabled space sample, if the space is composite. Checks, if the space is composite and converts the sample to a JSONable struct in this case. Otherwise return the sample as is. Args: sample: Any action or observation type possible in `gymnasium`, or a JSONable data type. space: Any space defined in `gymnasium.spaces`. Returns: The `sample` as-is, if the `space` is not composite, otherwise converts the composite sample jsonable to an actual `space` sample.. r)r% from_jsonabler(rrrfrom_jsonable_if_needed_sr+csfddg}|||S)aFlattens a gym.Space into its primitive components. Primitive components are any non Tuple/Dict spaces. Args: space: The gym.Space to flatten. This may be any supported type (including nested Tuples and Dicts). Returns: List[gym.Space]: The flattened list of primitive Spaces. This list does not contain Tuples or Dicts anymore. cslddlm}t|tr|D]}||q dSt|t|fr/t|jD] }|||q#dS||dS)Nr)FlexDict)ray.rllib.utils.spaces.flexdictr, isinstancerrsortedr"append)space_ return_listr,sk_helper_flattenrrr6s   z&flatten_space.._helper_flattenr)rretrr5r flatten_spacexs  r8csfdd|S)aaReturns a Tuple/Dict Space as native (equally structured) py tuple/dict. Args: space: The Space to get the python struct for. Returns: Union[dict,tuple,gym.Space]: The struct equivalent to the given Space. Note that the returned struct still contains all original "primitive" Spaces (e.g. Box, Discrete). .. testcode:: :skipif: True get_base_struct_from_space(Dict({ "a": Box(), "b": Tuple([Discrete(2), Discrete(3)]) })) .. testoutput:: dict(a=Box(), b=tuple(Discrete(2), Discrete(3))) csDttrtfddDSttr fddjDSS)Nc3s|]}|VqdSr r).0r3_helper_structrr szEget_base_struct_from_space.._helper_struct..csi|] }||qSrr)r9r4)r;r1rr szFget_base_struct_from_space.._helper_struct..)r.rtuplerr"r1r:r?rr;s  z2get_base_struct_from_space.._helper_structrrrr:rget_base_struct_from_spaces r@ F) fill_value time_size time_majorone_hot_discrete batch_sizerCrDrErFcsttjjtjjttfr,}ttjjtjjfrt}t fdd|SrXttjj rBtj ddj ft jnttjjrXtj ddt jft jdkrdurdkrhdksjJr~t jfdd tDjd St jfd d tDjd St jdkrfd d tDnjd SdurÈdkrdksJrg}ng}n dkrʈgng}t j|tjjd S)aReturns batched dummy data (using `batch_size`) for the given `space`. Note: The returned batch will not pass a `space.contains(batch)` test as an additional batch dimension has to be added at axis 0, unless `batch_size` is set to 0. Args: space: The space to get a dummy batch for. batch_size: The required batch size (B). Note that this can also be 0 (only if `time_size` is None!), which will result in a non-batched sample for the given space (no batch dim). fill_value: The value to fill the batch with or "random" for random values. time_size: If not None, add an optional time axis of `time_size` size to the returned batch. This time axis might either be inserted at axis=1 (default) or axis=0, if `time_major` is True. time_major: If True AND `time_size` is not None, return batch as shape [T x B x ...], otherwise as [B x T x ...]. If `time_size` if None, ignore this setting and return [B x ...]. one_hot_discrete: If True, will return one-hot vectors (instead of int-values) for those sub-components of a (possibly complex) `space` that are Discrete or MultiDiscrete. Note that in case `fill_value` is 0.0, this will result in zero-hot vectors (where all slots have a value of 0.0). Returns: The dummy batch of size `bqtch_size` matching the given space. cst|dS)N)rrGrCrDrErF)get_dummy_batch_for_spacer3)rGrCrFrErDrrsz+get_dummy_batch_for_space..rB?randomNrcs"g|] }fddtDqS)cg|]}qSrr&r9_rrr 8get_dummy_batch_for_space...ranger9t)rGrrrrQz-get_dummy_batch_for_space..dtypecs"g|] }fddtDqS)crMrrNrVrrrrQrRrSrTrO)rrDrrrQrXcrMrrNrOrrrrQrR)rCrZ)r.r!r"rrdictr>r@tree map_structureDiscreteBoxnr float32 MultiDiscretesumnvecarrayrUrZr&fulllistshape)rrGrCrDrErF base_structrhr)rGrCrFrrErDrrHsX&     rHcCsNt|tttfr%g}t|D] }|t|dgqtj |dd}|S)a:Returns a single np.ndarray given a list/tuple of np.ndarrays. Args: input_ (Union[List[np.ndarray], np.ndarray]): The list of ndarrays or a single ndarray. Returns: np.ndarray: The result after concatenating all single arrays in input_. .. testcode:: :skipif: True flatten_to_single_ndarray([ np.array([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]]), np.array([7, 8, 9]), ]) .. testoutput:: np.array([ 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0 ]) raxis) r.rgr>r[r\flattenr0r reshape concatenate)input_expandedin_rrrflatten_to_single_ndarray#s rs)'individual_items_already_have_batch_dimlist_of_structsrtcs^|std|dkrt|d}t|dt}|rtjntjtjfddg|R}|S)aConverts input from a list of (nested) structs to a (nested) struct of batches. Input: List of structs (each of these structs representing a single batch item). [ {"a": 1, "b": (4, 7.0)}, <- batch item 1 {"a": 2, "b": (5, 8.0)}, <- batch item 2 {"a": 3, "b": (6, 9.0)}, <- batch item 3 ] Output: Struct of different batches (each batch has size=3 b/c there were 3 items in the original list): { "a": np.array([1, 2, 3]), "b": (np.array([4, 5, 6]), np.array([7.0, 8.0, 9.0])) } Args: list_of_structs: The list of (possibly nested) structs. Each item in this list represents a single batch item. individual_items_already_have_batch_dim: True, if the individual items in `list_of_structs` already have a batch dim. In this case, we will concatenate (instead of stack) at the end. In the example above, this would look like this: Input: [{"a": [1], "b": ([4], [7.0])}, ...] -> Output: same as in above example. If the special value "auto" is used, Returns: The struct of component batches. Each leaf item in this struct represents the batch for a single component (in case struct is tuple/dict). If the input is a simple list of primitive items, e.g. a list of floats, a np.array of floats will be returned. z3Input `list_of_structs` does not contain any items.autorcst|ddS)Nrrk)r ascontiguousarrayrInp_funcrrrJvszbatch..) ValueErrorr\rmr.r r rostackr])rurtflatr7rrxrbatchEs& r}c sRt|g}ttdD]|t|fddttDq|S)aConverts input from (nested) struct of batches to batch of structs. Input: Struct of different batches (each batch has size=3): { "a": np.array([1, 2, 3]), "b": (np.array([4, 5, 6]), np.array([7.0, 8.0, 9.0])) } Output: Batch (list) of structs (each of these structs representing a single action): [ {"a": 1, "b": (4, 7.0)}, <- action 1 {"a": 2, "b": (5, 8.0)}, <- action 2 {"a": 3, "b": (6, 9.0)}, <- action 3 ] Args: batches_struct: The struct of component batches. Each leaf item in this struct represents the batch for a single component (in case struct is tuple/dict). Alternatively, `batches_struct` may also simply be a batch of primitives (non tuple/dict). Returns: The list of individual structs. Each item in the returned list represents a single (maybe complex) batch item. rcsg|]}|qSrr)r9i batch_pos flat_batchesrrrQszunbatch..)r\rmrUlenr0 unflatten_as)batches_structoutrrrunbatch{s rcCdd}t|||S)aClips all components in `action` according to the given Space. Only applies to Box components within the action space. Args: action: The action to be clipped. This could be any complex action, e.g. a dict or tuple. action_space: The action space struct, e.g. `{"a": Distrete(2)}` for a space: Dict({"a": Discrete(2)}). Returns: Any: The input action, but clipped by value according to the space's bounds. cSs$t|tjjrt||j|j}|Sr )r.r!r"r_r cliplowhighar3rrrmap_szclip_action..map_r\r])action action_spacerrrr clip_actionsrcCr)aUnsquashes all components in `action` according to the given Space. Inverse of `normalize_action()`. Useful for mapping policy action outputs (normalized between -1.0 and 1.0) to an env's action space. Unsquashing results in cont. action component values between the given Space's bounds (`low` and `high`). This only applies to Box components within the action space, whose dtype is float32 or float64. Args: action: The action to be unsquashed. This could be any complex action, e.g. a dict or tuple. action_space_struct: The action space struct, e.g. `{"a": Box()}` for a space: Dict({"a": Box()}). Returns: Any: The input action, but unsquashed, according to the space's bounds. An unsquashed action is ready to be sent to the environment (`BaseEnv.send_actions([unsquashed actions])`). cSst|tjjrFt|jrFt|jrF|jtj ks|jtj kr9|j |d|j |j d}t ||j |j }|St|jtjrF|j |}|S)NrK@)r.r!r"r_r all bounded_below bounded_aboverZrafloat64rrr issubdtypeintegerrrrrrs    zunsquash_action..map_rraction_space_structrrrrunsquash_actionsrcCr)aNormalizes all (Box) components in `action` to be in [-1.0, 1.0]. Inverse of `unsquash_action()`. Useful for mapping an env's action (arbitrary bounded values) to a [-1.0, 1.0] interval. This only applies to Box components within the action space, whose dtype is float32 or float64. Args: action: The action to be normalized. This could be any complex action, e.g. a dict or tuple. action_space_struct: The action space struct, e.g. `{"a": Box()}` for a space: Dict({"a": Box()}). Returns: Any: The input action, but normalized, according to the space's bounds. cSsHt|tjjr"|jtjks|jtjkr"||jd|j |jd}|S)NrrK) r.r!r"r_rZr rarrrrrrrrsznormalize_action..map_rrrrrnormalize_actionsrelementsampled_elementcCsdd}tj|||ddS)aKConvert all the components of the element to match the space dtypes. Args: element: The element to be converted. sampled_element: An element sampled from a space to be matched to. Returns: The input element, but with all its components converted to match the space dtypes. cSst|tjrAt|tjs3t|ttfsJd|d|jdkr*tj||jd}|Std t ||j|jkr?| |j}|St|tsLt|tj rdt|trY| rYt|}t|tjrdt|}|S)NzERROR: `elem` (z!) must be np.array, float or int!rrYzZElement should be of type np.ndarray but is instead of type {})r.r ndarrayfloatintrhrerZrzformatr astypeint_ is_integerfloat_int64)elemr3rrrrs2         z+convert_element_to_space_type..map_F) check_typesr)rrrrrrconvert_element_to_space_typesr)rA)' gymnasiumr!gymnasium.spacesrrgymnasium.corerrnumpyr ray.rllib.utils.annotationsrr\typingrrr r rr Spacerboolr%r)r+r8r@rrstrrHrsr}rrrrrrrrrs        #  f !  5 (  +