o `۷ic% @sddlmZddlmZmZmZddlZddlZddl m Z m Z ddl m Z e dededeeeffd d Ze d eee ejjefd edeejjfd dZe dejjdeed edeejjfddZe deedejjdejjfddZdS))deque)ListTupleUnionN)BatchedNdArraybatch) DeveloperAPI episode_lenTreturncCsg}g}t||}||tdg|dg||tj}||||}|dkrTt||}||tdg|dg||tj}||||8}|dks,||fS)a7Creates loss mask and a seq_lens array, given an episode length and T. Args: episode_lens: A list of episode lengths to infer the loss mask and seq_lens array from. T: The maximum number of timesteps in each "row", also known as the maximum sequence length (max_seq_len). Episodes are split into chunks that are at most `T` long and remaining timesteps will be zero-padded (and masked out). Returns: Tuple consisting of a) list of the loss masks to use (masking out areas that are past the end of an episode (or rollout), but had to be zero-added due to the added extra time rank (of length T) and b) the list of sequence lengths resulting from splitting the given episodes into chunks of at most `T` timesteps. r)minappendnparraybool_)r r maskseq_lenslen_rowoverflow extra_rowra/home/ubuntu/vllm_env/lib/python3.10/site-packages/ray/rllib/utils/postprocessing/zero_padding.pycreate_mask_and_seq_lens s  "   " r item_list max_seq_lenc CsTtdd|d}g}g}d}t|}t|dkr|}||}t|}g} g} d} |D].} t| trV| | d|t| |krJt| } q/|} | | |dq/| | d} q/|| 7}|t || | rv| t || ||kr|t |ddg}d}t|dks|dkr||kr| |g|||t |dd|S)aSplits the contents of `item_list` into a new list of ndarrays and returns it. In the returned list, each item is one ndarray of len (axis=0) `max_seq_len`. The last item in the returned list may be (right) zero-padded, if necessary, to reach `max_seq_len`. If `item_list` contains one or more `BatchedNdArray` (instead of individual items), these will be split accordingly along their axis=0 to yield the returned structure described above. .. testcode:: from ray.rllib.utils.postprocessing.zero_padding import ( BatchedNdArray, split_and_zero_pad, ) from ray.rllib.utils.test_utils import check # Simple case: `item_list` contains individual floats. check( split_and_zero_pad([0, 1, 2, 3, 4, 5, 6, 7], 5), [[0, 1, 2, 3, 4], [5, 6, 7, 0, 0]], ) # `item_list` contains BatchedNdArray (ndarrays that explicitly declare they # have a batch axis=0). check( split_and_zero_pad([ BatchedNdArray([0, 1]), BatchedNdArray([2, 3, 4, 5]), BatchedNdArray([6, 7, 8]), ], 5), [[0, 1, 2, 3, 4], [5, 6, 7, 8, 0]], ) Args: item_list: A list of individual items or BatchedNdArrays to be split into `max_seq_len` long pieces (the last of which may be zero-padded). max_seq_len: The maximum length of each item in the returned list. Returns: A list of np.ndarrays (all of length `max_seq_len`), which contains the same data as `item_list`, but split into sub-chunks of size `max_seq_len`. The last item in the returned list may be zero-padded, if necessary. cSs tt|tr |dgS|S)Nr)r zeros_like isinstancer)srrrds z$split_and_zero_pad..rNr auto)'individual_items_already_have_batch_dim) tree map_structurerlenpopleftflattenrrr unflatten_as appendleftrextend) rr zero_elementretcurrent_time_row current_titemt item_flatitem_list_appendcurrent_time_row_flat_itemsadd_to_current_titmrrrsplit_and_zero_pad1sX2        + r6nd_array episode_lenscCsDg}d}|D]}t||||}|t|g|||7}q|S)aSplits and zero-pads a single np.ndarray based on episode lens and a maxlen. Args: nd_array: The single np.ndarray to be split into n chunks, based on the given `episode_lens` and the `max_seq_len` argument. For example, if `nd_array` has a batch dimension (axis 0) of 21, `episode_lens` is [15, 3, 3], and `max_seq_len` is 6, then the returned list would have np.ndarrays in it of batch dimensions (axis 0): [6, 6, 6 (zero-padded), 6 (zero-padded), 6 (zero-padded)]. Note that this function doesn't work on nested data, such as dicts of ndarrays. episode_lens: A list of episode lengths along which to split and zero-pad the given `nd_array`. max_seq_len: The maximum sequence length to split at (and zero-pad). Returns: A list of n np.ndarrays, resulting from splitting and zero-padding the given `nd_array`. r)rr*r6)r7r8rr,cursorr itemsrrrsplit_and_zero_pad_n_episodess r;datac Cst|jdkr |St|jdksJg}d}|jd}|D].}t||\}}t|D] }||||d7}q*|dkrK|||d|f|d7}qt|S)aRemoves right-side zero-padding from data based on `episode_lens`. ..testcode:: from ray.rllib.utils.postprocessing.zero_padding import unpad_data_if_necessary import numpy as np unpadded = unpad_data_if_necessary( episode_lens=[4, 2], data=np.array([ [2, 4, 5, 3, 0, 0, 0, 0], [-1, 3, 0, 0, 0, 0, 0, 0], ]), ) assert (unpadded == [2, 4, 5, 3, -1, 3]).all() unpadded = unpad_data_if_necessary( episode_lens=[1, 5], data=np.array([ [2, 0, 0, 0, 0], [-1, -2, -3, -4, -5], ]), ) assert (unpadded == [2, -1, -2, -3, -4, -5]).all() Args: episode_lens: A list of actual episode lengths. data: A 2D np.ndarray with right-side zero-padded rows. Returns: A 1D np.ndarray resulting from concatenation of the un-padded input data along the 0-axis. r rN)r%shapedivmodrangerr concatenate) r8r<new_datarow_idxr rnum_rowscol_idxirrrunpad_data_if_necessarys '    rG) collectionsrtypingrrrnumpyrr#"ray.rllib.utils.spaces.space_utilsrrray.util.annotationsrintr_typingNDArrayfloatr6r;rGrrrrsD   %  u  "