o `۷i?F@s|ddlmZmZmZmZddlZddlZddl Z ddl m Z e Gdddej Z e dejdejfdd Ze dejdefd d Ze dejdeejfd d Ze ddZe d.ddddddejdedeeeefdeedededej fddZe ddZe dddeedeeeffd d!Ze d"d#Ze d$d%Ze d&d'Ze d(d)Ze d*ed+edefd,d-ZdS)/)AnyListOptionalUnionN) DeveloperAPIc@seZdZdZddZdS)BatchedNdArrayaA ndarray-wrapper the usage of which indicates that there a batch dim exists. This is such that our `batch()` utility can distinguish between having to stack n individual batch items (each one w/o any batch dim) vs having to concatenate n already batched items (each one possibly with a different batch dim, but definitely with some batch dim). TODO (sven): Maybe replace this by a list-override instead. cCst||}|SN)npasarrayview)cls input_arrayobjrX/home/ubuntu/vllm_env/lib/python3.10/site-packages/ray/rllib/utils/spaces/space_utils.py__new__szBatchedNdArray.__new__N)__name__ __module__ __qualname____doc__rrrrrr s rspacereturncCst|dr t|jS|S)aRReturns the original space of a space, if any. This function recursively traverses the given space and returns the original space at the very end of the chain. Args: space: The space to get the original space for. Returns: The original space or the given space itself if no original space is found. original_space)hasattrget_original_spacerrrrrrs rcCs,t|tjjtjjtjjtjjfvrdSdS)aaReturns true, if the space is composite. Note, we follow here the glossary of `gymnasium` by which any spoace that holds other spaces is defined as being 'composite'. Args: space: The space to be checked for being composed of other spaces. Returns: True, if the space is composed of other spaces, otherwise False. TF)typegymspacesDictGraphSequenceTuplerrrris_composite_space/s r#csfddg}|||S)aFlattens a gym.Space into its primitive components. Primitive components are any non Tuple/Dict spaces. Args: space: The gym.Space to flatten. This may be any supported type (including nested Tuples and Dicts). Returns: List[gym.Space]: The flattened list of primitive Spaces. This list does not contain Tuples or Dicts anymore. cstddlm}t|tjjr|D]}||qdSt|tjj|fr3t|jD] }|||q'dS||dS)Nr)FlexDict) ray.rllib.utils.spaces.flexdictr$ isinstancerrr"rsortedappend)space_ return_listr$sk_helper_flattenrrr.Vs  z&flatten_space.._helper_flattenr)rretrr-r flatten_spaceGs  r0csfdd|S)aaReturns a Tuple/Dict Space as native (equally structured) py tuple/dict. Args: space: The Space to get the python struct for. Returns: Union[dict,tuple,gym.Space]: The struct equivalent to the given Space. Note that the returned struct still contains all original "primitive" Spaces (e.g. Box, Discrete). .. testcode:: :skipif: True get_base_struct_from_space(Dict({ "a": Box(), "b": Tuple([Discrete(2), Discrete(3)]) })) .. testoutput:: dict(a=Box(), b=tuple(Discrete(2), Discrete(3))) csLttjjrtfddDSttjjr$fddjDSS)Nc3s|]}|VqdSrr).0r+_helper_structrr szEget_base_struct_from_space.._helper_struct..csi|] }||qSrr)r1r,)r3r)rr szFget_base_struct_from_space.._helper_struct..)r&rrr"tuplerr)r2r7rr3s z2get_base_struct_from_space.._helper_structrrrr2rget_base_struct_from_spacegs r8 F) fill_value time_size time_majorone_hot_discrete batch_sizer;r<r=r>csttjjtjjttfr,}ttjjtjjfrt}t fdd|SrXttjj rBtj ddj ft jnttjjrXtj ddt jft jdkrdurdkrhdksjJr~t jfdd tDjd St jfd d tDjd St jdkrfd d tDnjd SdurÈdkrdksJrg}ng}n dkrʈgng}t j|tjjd S)aReturns batched dummy data (using `batch_size`) for the given `space`. Note: The returned batch will not pass a `space.contains(batch)` test as an additional batch dimension has to be added at axis 0, unless `batch_size` is set to 0. Args: space: The space to get a dummy batch for. batch_size: The required batch size (B). Note that this can also be 0 (only if `time_size` is None!), which will result in a non-batched sample for the given space (no batch dim). fill_value: The value to fill the batch with or "random" for random values. time_size: If not None, add an optional time axis of `time_size` size to the returned batch. This time axis might either be inserted at axis=1 (default) or axis=0, if `time_major` is True. time_major: If True AND `time_size` is not None, return batch as shape [T x B x ...], otherwise as [B x T x ...]. If `time_size` if None, ignore this setting and return [B x ...]. one_hot_discrete: If True, will return one-hot vectors (instead of int-values) for those sub-components of a (possibly complex) `space` that are Discrete or MultiDiscrete. Note that in case `fill_value` is 0.0, this will result in zero-hot vectors (where all slots have a value of 0.0). Returns: The dummy batch of size `bqtch_size` matching the given space. cst|dS)N)rr?r;r<r=r>)get_dummy_batch_for_spacer+)r?r;r>r=r<rrsz+get_dummy_batch_for_space..r:?randomNrcs"g|] }fddtDqS)cg|]}qSrsampler1_rrr 8get_dummy_batch_for_space...ranger1t)r?rrrrJz-get_dummy_batch_for_space..dtypecs"g|] }fddtDqS)crErrFrOrrrrJrKrLrMrH)rr<rrrJrQcrErrFrHrrrrJrK)r;rS)r&rrrr"dictr6r8tree map_structureDiscreteBoxnr float32 MultiDiscretesumnvecarrayrNrSrGfulllistshape)rr?r;r<r=r> base_structrar)r?r;r>rr=r<rr@sX&     r@cCsNt|tttfr%g}t|D] }|t|dgqtj |dd}|S)a:Returns a single np.ndarray given a list/tuple of np.ndarrays. Args: input_ (Union[List[np.ndarray], np.ndarray]): The list of ndarrays or a single ndarray. Returns: np.ndarray: The result after concatenating all single arrays in input_. .. testcode:: :skipif: True flatten_to_single_ndarray([ np.array([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]]), np.array([7, 8, 9]), ]) .. testoutput:: np.array([ 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0 ]) raxis) r&r`r6rTrUflattenr(r reshape concatenate)input_expandedin_rrrflatten_to_single_ndarrays rl)'individual_items_already_have_batch_dimlist_of_structsrmcs^|std|dkrt|d}t|dt}|rtjntjtjfddg|R}|S)aConverts input from a list of (nested) structs to a (nested) struct of batches. Input: List of structs (each of these structs representing a single batch item). [ {"a": 1, "b": (4, 7.0)}, <- batch item 1 {"a": 2, "b": (5, 8.0)}, <- batch item 2 {"a": 3, "b": (6, 9.0)}, <- batch item 3 ] Output: Struct of different batches (each batch has size=3 b/c there were 3 items in the original list): { "a": np.array([1, 2, 3]), "b": (np.array([4, 5, 6]), np.array([7.0, 8.0, 9.0])) } Args: list_of_structs: The list of (possibly nested) structs. Each item in this list represents a single batch item. individual_items_already_have_batch_dim: True, if the individual items in `list_of_structs` already have a batch dim. In this case, we will concatenate (instead of stack) at the end. In the example above, this would look like this: Input: [{"a": [1], "b": ([4], [7.0])}, ...] -> Output: same as in above example. If the special value "auto" is used, Returns: The struct of component batches. Each leaf item in this struct represents the batch for a single component (in case struct is tuple/dict). If the input is a simple list of primitive items, e.g. a list of floats, a np.array of floats will be returned. z3Input `list_of_structs` does not contain any items.autorcst|ddS)Nrrd)r ascontiguousarrayrAnp_funcrrrBEszbatch..) ValueErrorrUrfr&rr rhstackrV)rnrmflatr/rrqrbatchs& rvc sRt|g}ttdD]|t|fddttDq|S)aConverts input from (nested) struct of batches to batch of structs. Input: Struct of different batches (each batch has size=3): { "a": np.array([1, 2, 3]), "b": (np.array([4, 5, 6]), np.array([7.0, 8.0, 9.0])) } Output: Batch (list) of structs (each of these structs representing a single action): [ {"a": 1, "b": (4, 7.0)}, <- action 1 {"a": 2, "b": (5, 8.0)}, <- action 2 {"a": 3, "b": (6, 9.0)}, <- action 3 ] Args: batches_struct: The struct of component batches. Each leaf item in this struct represents the batch for a single component (in case struct is tuple/dict). Alternatively, `batches_struct` may also simply be a batch of primitives (non tuple/dict). Returns: The list of individual structs. Each item in the returned list represents a single (maybe complex) batch item. rcsg|]}|qSrr)r1i batch_pos flat_batchesrrrJmszunbatch..)rUrfrNlenr( unflatten_as)batches_structoutrrxrunbatchJs rcCdd}t|||S)aClips all components in `action` according to the given Space. Only applies to Box components within the action space. Args: action: The action to be clipped. This could be any complex action, e.g. a dict or tuple. action_space: The action space struct, e.g. `{"a": Distrete(2)}` for a space: Dict({"a": Discrete(2)}). Returns: Any: The input action, but clipped by value according to the space's bounds. cSs$t|tjjrt||j|j}|Sr)r&rrrXr cliplowhighar+rrrmap_szclip_action..map_rUrV)action action_spacerrrr clip_actionssrcCr)aUnsquashes all components in `action` according to the given Space. Inverse of `normalize_action()`. Useful for mapping policy action outputs (normalized between -1.0 and 1.0) to an env's action space. Unsquashing results in cont. action component values between the given Space's bounds (`low` and `high`). This only applies to Box components within the action space, whose dtype is float32 or float64. Args: action: The action to be unsquashed. This could be any complex action, e.g. a dict or tuple. action_space_struct: The action space struct, e.g. `{"a": Box()}` for a space: Dict({"a": Box()}). Returns: Any: The input action, but unsquashed, according to the space's bounds. An unsquashed action is ready to be sent to the environment (`BaseEnv.send_actions([unsquashed actions])`). cSst|tjjrFt|jrFt|jrF|jtj ks|jtj kr9|j |d|j |j d}t ||j |j }|St|jtjrF|j |}|S)NrC@)r&rrrXr all bounded_below bounded_aboverSrZfloat64rrr issubdtypeintegerrrrrrs    zunsquash_action..map_rraction_space_structrrrrunsquash_actionsrcCr)aNormalizes all (Box) components in `action` to be in [-1.0, 1.0]. Inverse of `unsquash_action()`. Useful for mapping an env's action (arbitrary bounded values) to a [-1.0, 1.0] interval. This only applies to Box components within the action space, whose dtype is float32 or float64. Args: action: The action to be normalized. This could be any complex action, e.g. a dict or tuple. action_space_struct: The action space struct, e.g. `{"a": Box()}` for a space: Dict({"a": Box()}). Returns: Any: The input action, but normalized, according to the space's bounds. cSsHt|tjjr"|jtjks|jtjkr"||jd|j |jd}|S)NrrC) r&rrrXrSr rZrrrrrrrrsznormalize_action..map_rrrrrnormalize_actionsrelementsampled_elementcCsdd}tj|||ddS)aKConvert all the components of the element to match the space dtypes. Args: element: The element to be converted. sampled_element: An element sampled from a space to be matched to. Returns: The input element, but with all its components converted to match the space dtypes. cSst|tjrAt|tjs3t|ttfsJd|d|jdkr*tj||jd}|Std t ||j|jkr?| |j}|St|tsLt|tj rdt|trY| rYt|}t|tjrdt|}|S)NzERROR: `elem` (z!) must be np.array, float or int!rrRzZElement should be of type np.ndarray but is instead of type {})r&r ndarrayfloatintrar^rSrsformatrastypeint_ is_integerfloat_int64)elemr+rrrrs2         z+convert_element_to_space_type..map_F) check_typesr)rrrrrrconvert_element_to_space_typesr)r9)typingrrrr gymnasiumrnumpyr rUray.rllib.utils.annotationsrrrSpacerboolr#r0r8rrstrr@rlrvrrrrrrrrrsp  #  f !  5 (  +