o $i2@sddlZddlmZmZddlmZmZmZmZm Z m Z m Z ddl Z ddlZddlZddlmZmZddlmZddlmZmZddlmZddlmZdd lmZdd lm Z m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&erdd l'm(Z(dd l)m*Z*dd l+m,Z,ddl-m.Z.ddl/m0Z0e1e2Z3e\Z4Z5Z6eddddde e7de8de e7fddZ9ede&de&de&fddZ:e  dode%de e$de;de&fd d!Zd'e ed(e e8de;d)e;dd*f d+d,Z?ed-e d.d/e d0e#fde e d1fd2d3Z@edpd5e&d6e7de&fd7d8ZAed5e&de&fd9d:ZBe dqd;e d<d=e;defd>d?ZCe @drdAe dBe&dCedDdEe7de!f dFdGZDed5e&d&e j>de&fdHdIZEedsd5e&dJe eFde&fdKdLZGe dqdMe e8dNfdOe;dedDfdPdQZHedtdSdTZIedudUdVZJe W X Y dvd'dRdZeFd[e7d\e7fd]d^ZKed_e"d`e"dae7ddfdbdcZLedde%de&fdedfZMedge d.dhe&ddfdidjZNdkdlZOeGdmdndnZPdS)wN) OrderedDictdeque) TYPE_CHECKINGAnyCallableListOptionalTypeUnion)Discrete MultiDiscrete) force_list) DeveloperAPI PublicAPI) try_import_tf) SMALL_NUMBER)get_base_struct_from_space)LocalOptimizerModelGradients NetworkTypePartialAlgorithmConfigDict SpaceStructTensorStructType TensorType)AlgorithmConfig) ParamDict EagerTFPolicyEagerTFPolicyV2TFPolicy) grad_clipgradients_dictrr" grad_clip_byreturncCs|durdS|dkr!|D]\}}t|| |||<qdS|dkr:|D] \}}t||||<q+dS|dks@Jtt||\}}t| |D]\}}|||<qU|S)a?Performs gradient clipping on a grad-dict based on a clip value and clip mode. Changes the provided gradient dict in place. Args: gradients_dict: The gradients dict, mapping str to gradient tensors. grad_clip: The value to clip with. The way gradients are clipped is defined by the `grad_clip_by` arg (see below). grad_clip_by: One of 'value', 'norm', or 'global_norm'. Returns: If `grad_clip_by`="global_norm" and `grad_clip` is not None, returns the global norm of all tensors, otherwise returns None. Nvaluenorm global_norm) copyitemstf clip_by_value clip_by_normclip_by_global_normlistvalueszipkeys)r#r"r$kv clipped_gradsr(r6U/home/ubuntu/veenaModal/venv/lib/python3.10/site-packages/ray/rllib/utils/tf_utils.pyclip_gradients$s"   r8ypredcCsHtjj|dgd\}}tjj||dgd\}}tdd||tS)a,Computes the explained variance for a pair of labels and predictions. The formula used is: max(-1.0, 1.0 - (std(y - pred)^2 / std(y)^2)) Args: y: The labels. pred: The predictions. Returns: The explained variance given a pair of labels and predictions. r)axesg)r+nnmomentsmaximumr)r9r:_y_vardiff_varr6r6r7explained_varianceUsrCFinputs spaces_struct time_axisc CsZt|}|durt|ndgt|}d}d}g}t||D]x\}} t|}t|} |dur<| d}|r<| d}t| trZ|rLt |||g}| t t || tj q t| try|rkt |||dg}| t t || tj q |rt |||dg}nt ||dg}| t |tj q tj|dd} |rt | ||dg} | S)aKFlattens arbitrary input structs according to the given spaces struct. Returns a single 1D tensor resulting from the different input components' values. Thereby: - Boxes (any shape) get flattened to (B, [T]?, -1). Note that image boxes are not treated differently from other types of Boxes and get flattened as well. - Discrete (int) values are one-hot'd, e.g. a batch of [1, 0, 3] (B=3 with Discrete(4) space) results in [[0, 1, 0, 0], [1, 0, 0, 0], [0, 0, 0, 1]]. - MultiDiscrete values are multi-one-hot'd, e.g. a batch of [[0, 2], [1, 4]] (B=2 with MultiDiscrete([2, 5]) space) results in [[1, 0, 0, 0, 1, 0, 0], [0, 1, 0, 0, 0, 0, 1]]. Args: inputs: The inputs to be flattened. spaces_struct: The structure of the spaces that behind the input time_axis: Whether all inputs have a time-axis (after the batch axis). If True, will keep not only the batch axis (0th), but the time axis (1st) as-is and flatten everything from the 2nd axis up. Returns: A single 1D tensor resulting from concatenating all flattened/one-hot'd input components. Depending on the time_axis flag, the shape is (B, n) or (B, T, n). .. testcode:: :skipif: True # B=2 from ray.rllib.utils.tf_utils import flatten_inputs_to_1d_tensor from gymnasium.spaces import Discrete, Box out = flatten_inputs_to_1d_tensor( {"a": [1, 0], "b": [[[0.0], [0.1]], [1.0], [1.1]]}, spaces_struct=dict(a=Discrete(2), b=Box(shape=(2, 1))) ) print(out) # B=2; T=2 out = flatten_inputs_to_1d_tensor( ([[1, 0], [0, 1]], [[[0.0, 0.1], [1.0, 1.1]], [[2.0, 2.1], [3.0, 3.1]]]), spaces_struct=tuple([Discrete(2), Box(shape=(2, ))]), time_axis=True ) print(out) .. testoutput:: [[0.0, 1.0, 0.0, 0.1], [1.0, 0.0, 1.0, 1.1]] # B=2 n=4 [[[0.0, 1.0, 0.0, 0.1], [1.0, 0.0, 1.0, 1.1]], [[1.0, 0.0, 2.0, 2.1], [0.0, 1.0, 3.0, 3.1]]] # B=2 T=2 n=4 Nrr<axis)treeflattenlenr1r+convert_to_tensorshape isinstancer reshapeappendcastone_hotfloat32r concat) rDrErF flat_inputs flat_spacesBToutinput_spacerNmergedr6r6r7flatten_inputs_to_1d_tensorhs> =     r^cCsZtdkrddlm}|}nztj}Wnty%tjj}Ynwdd|DS)zReturns a list of GPU device names, e.g. ["/gpu:0", "/gpu:1"]. Supports both tf1.x and tf2.x. Returns: List of GPU device names (str). r<r) device_libcSsg|] }d|jvr|jqS)GPU) device_typename).0dr6r6r7 sz#get_gpu_devices..) tfvtensorflow.python.clientr_list_local_devicesr+configlist_physical_devices Exception experimental)r_devicesr6r6r7get_gpu_devicess   rnT)r\r&rbrFrKr\r&rbrKztf1.placeholdercsddlm}|durGt|tjjtjjfr)|r||dSt fddt |St j d|r0dnd|j |jtjkrAtjdS|jdS|dusMJ|j d d}t j d|r[dndt|trd|nt||jtjkrvtjdS|jdS) aReturns a tf1.placeholder object given optional hints, such as a space. Note that the returned placeholder will always have a leading batch dimension (None). Args: space: An optional gym.Space to hint the shape and dtype of the placeholder. value: An optional value to hint the shape and dtype of the placeholder. name: An optional name for the placeholder. time_axis: Whether the placeholder should also receive a time dimension (None). flatten: Whether to flatten the given space into a plain Box space and then create the placeholder from the resulting space. Returns: The tf1 placeholder. r) ModelCatalogNcs$t|dddd|DdS)N.cSg|]}t|qSr6strrcpr6r6r7rez5get_placeholder....)r\rb)get_placeholderjoin)path componentrbr6r7 sz!get_placeholder..Nr6)rNdtyperbr<)ray.rllib.models.catalogrorOgymspacesDictTupleget_action_placeholderrJmap_structure_with_pathrtf1 placeholderrNr~npfloat64r+rTtupleas_list)r\r&rbrFrKrorNr6r{r7rws>     rworig_clsr!rir)r!rrcCs|}|dd}|dvrtstd|dkrbtstts$Jddlm}ddlm}dd l m }t |d rEt ||sE| }n t ||sKntd ||d rbt |||frb|}|S) a[Returns the corresponding tf-eager class for a given TFPolicy class. Args: orig_cls: The original TFPolicy class to get the corresponding tf-eager class for. config: The Algorithm config dict or AlgorithmConfig object. Returns: The tf eager policy class corresponding to the given TFPolicy class. frameworkr+)tf2r+zCould not import tensorflow!rrrrr as_eagerz0This policy does not support eager execution: {} eager_tracing)getr ImportErrorexecuting_eagerlyenable_eager_execution ray.rllib.policy.eager_tf_policyr#ray.rllib.policy.eager_tf_policy_v2rray.rllib.policy.tf_policyr!hasattr issubclassr ValueErrorformat with_tracing)rriclsrrrr!r6r6r7get_tf_eager_cls_if_necessary#s.         r?xdeltacCs6tt||ktj|d|t|d|S)aComputes the huber loss for a given term and delta parameter. Reference: https://en.wikipedia.org/wiki/Huber_loss Note that the factor of 0.5 is implicitly included in the calculation. Formula: L = 0.5 * x^2 for small abs x (delta threshold) L = delta * (abs(x) - 0.5*delta) for larger abs x (delta threshold) Args: x: The input term, e.g. a TD error. delta: The delta parmameter in the above formula. Returns: The Huber loss resulting from `x` and `delta`. ?)r+whereabsmathsquare)rrr6r6r7 huber_lossUs  rcCsdtt|dS)zComputes half the L2 norm over a tensor's values without the sqrt. output = 0.5 * sum(x ** 2) Args: x: The input tensor. Returns: 0.5 times the L2 norm over the given tensor's values (w/o sqrt). r@)r+ reduce_sumpowrr6r6r7l2_lossns rsession_or_nonez tf1.Session dynamic_shapecs4tr dus JndusJfdd}|S)aLReturns a function that can be executed in either graph or eager mode. The function must take only positional args. If eager is enabled, this will act as just a function. Otherwise, it will build a function that executes a session run with placeholders internally. Args: session_or_none: tf.Session if in graph mode, else None. dynamic_shape: True if the placeholders should have a dynamic batch dimension. Otherwise they will be fixed shape. Returns: A function that can be called in either eager or static-graph mode. Ncs4durgidgfdd}|SS)Nc sg}|D]}t|tur||q||q|}ddurjj=fdd}t||}t|D]}|q7t||}| D]\}}||<qIid<Wdn1sewYt t t|t fdd| d}|S)NrcsTrt|jdkrd|jdd}nd}n|j}tj|j|ddd|DdS) Nrr}r<r6rpcSrqr6rrrtr6r6r7rervzfmake_tf_callable..make_wrapper..call.._create_placeholders..)r~rNrb)rLrNrrr~rx)ryr&rN)rr6r7_create_placeholdersszRmake_tf_callable..make_wrapper..call.._create_placeholderscs ||Sr}) __setitem__)phr4 feed_dictr6r7r|s zFmake_tf_callable..make_wrapper..call..)typer/extendrQgraph as_defaultrJrrKr*dictr1 map_structurerun) argskwargs args_flatar placeholdersrr3ret)args_placeholdersrfnkwargs_placeholdersr symbolic_outrr7calls:         z4make_tf_callable..make_wrapper..callr6)rrrr)rrrrr7 make_wrappers3z&make_tf_callable..make_wrapper)r+r)rrrr6rr7make_tf_callable}s  Br$@ optimizer objectivevar_listz tf.Variableclip_valcsbdus dks Jtr!|j}ttt||||}n|j||d}fdd|DS)aComputes, then clips gradients using objective, optimizer and var list. Ensures the norm of the gradients for each variable is clipped to `clip_val`. Args: optimizer: Either a shim optimizer (tf eager) containing a tf.GradientTape under `self.tape` or a tf1 local optimizer object. objective: The loss tensor to calculate gradients on. var_list: The list of tf.Variables to compute gradients over. clip_val: The global norm clip value. Will clip around -clip_val and +clip_val. Returns: The resulting model gradients (list or tuples of grads + vars) corresponding to the input `var_list`. N)rcs4g|]\}}|durdurt|n||fqSr})r+r-)rcgr4rr6r7res z%minimize_and_clip..)r+rtaper/r1gradientcompute_gradients)rrrrrgrads_and_varsr6rr7minimize_and_clips rcst|trtj|jtjdSt|trBt|jdtj r/t |j}t j ddfn|j}tj fddt|DddStd|)ajReturns a one-hot tensor, given and int tensor and a space. Handles the MultiDiscrete case as well. Args: x: The input tensor. space: The space to use for generating the one-hot tensor. Returns: The resulting one-hot tensor. Raises: ValueError: If the given space is not a discrete one. .. testcode:: :skipif: True import gymnasium as gym import tensorflow as tf from ray.rllib.utils.tf_utils import one_hot x = tf.Variable([0, 3], dtype=tf.int32) # batch-dim=2 # Discrete space with 4 (one-hot) slots per batch item. s = gym.spaces.Discrete(4) one_hot(x, s) .. testoutput:: .. testcode:: :skipif: True x = tf.Variable([[0, 1, 2, 3]], dtype=tf.int32) # batch-dim=1 # MultiDiscrete space with 5 + 4 + 4 + 7 = 20 (one-hot) slots # per batch item. s = gym.spaces.MultiDiscrete([5, 4, 4, 7]) one_hot(x, s) .. testoutput:: r~rrGcs.g|]\}}tjdd|f|tjdqS)Nr)r+rSrT)rcinrr6r7re<s.zone_hot..rHz#Unsupported space for `one_hot`: {})rOr r+rSrrTr nvecrndarrayravelrPrNrU enumeraterr)rr\rr6rr7rSs ,  rSrIcCsJt|tjj}t||t|}tj||tjt|tj|S)zSame as tf.reduce_mean() but ignores -inf values. Args: x: The input tensor to reduce mean over. axis: The axis over which to reduce. None for all axes. Returns: The mean reduced inputs, ignoring inf values. ) r+ not_equalrTminr zeros_likerrrR)rrImaskx_zeroedr6r6r7reduce_mean_ignore_infCs  rscopeztf1.VariableScopetrainable_onlycCs2tj|rtjjntjjt|tr|dS|jdS)aGet variables inside a given scope. Args: scope: Scope in which the variables reside. trainable_only: Whether or not to return only the variables that were marked as trainable. Returns: The list of variables in the given `scope`. )r)rget_collection GraphKeysTRAINABLE_VARIABLES VARIABLESrOrsrb)rrr6r6r7 scope_varsUs  r tf.TensorcCs$tj|tjtj|dS)zThe symlog function as described in [1]: [1] Mastering Diverse Domains through World Models - 2023 D. Hafner, J. Pasukonis, J. Ba, T. Lillicrap https://arxiv.org/pdf/2301.04104v1.pdf r<)r+rsignlogrrr6r6r7symlogks$rcCs$tj|tjtj|dS)zInverse of the `symlog` function as desribed in [1]: [1] Mastering Diverse Domains through World Models - 2023 D. Hafner, J. Pasukonis, J. Ba, T. Lillicrap https://arxiv.org/pdf/2301.04104v1.pdf r<)r+rrexpr)r9r6r6r7inverse_symlogvs$r44@ num_buckets lower_bound upper_boundcCs6t|||}tjtdt|d|ptjd}|||d}| ||}tj|}tj|} t t || | d| } t t | || d| } |||} || |} || | | } d| } t ||gd}t || gd}t ||gd}t | | gd}tj t|tj|t|d|fdS)a+Returns a two-hot vector of dim=num_buckets with two entries that are non-zero. See [1] for more details: [1] Mastering Diverse Domains through World Models - 2023 D. Hafner, J. Pasukonis, J. Ba, T. Lillicrap https://arxiv.org/pdf/2301.04104v1.pdf Entries in the vector represent equally sized buckets within some fixed range (`lower_bound` to `upper_bound`). Those entries not 0.0 at positions k and k+1 encode the actual `value` and sum up to 1.0. They are the weights multiplied by the buckets values at k and k+1 for retrieving `value`. Example: num_buckets=11 lower_bound=-5 upper_bound=5 value=2.5 -> [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.5, 0.5, 0.0, 0.0] -> [-5 -4 -3 -2 -1 0 1 2 3 4 5] (0.5*2 + 0.5*3=2.5) Example: num_buckets=5 lower_bound=-1 upper_bound=1 value=0.1 -> [0.0, 0.0, 0.8, 0.2, 0.0] -> [-1 -0.5 0 0.5 1] (0.2*0.5 + 0.8*0=0.1) Args: value: The input tensor of shape (B,) to be two-hot encoded. num_buckets: The number of buckets to two-hot encode into. lower_bound: The lower bound value used for the encoding. If input values are lower than this boundary, they will be encoded as `lower_bound`. upper_bound: The upper bound value used for the encoding. If input values are higher than this boundary, they will be encoded as `upper_bound`. Returns: The two-hot encoded tensor of shape (B, num_buckets). rrr<rrrG)rN)r+r,rRrangerNrTrfloorceilrequalstackrU scatter_ndint32)r&rrrr~ batch_indices bucket_deltaidxr3kp1values_k values_kp1 weights_k weights_kp1 indices_k indices_kp1indicesupdatesr6r6r7two_hots01     r main_net target_nettaucCs:t|j|jD]\}}||d||}||qdS)a`Updates a keras.Model target network using Polyak averaging. new_target_net_weight = ( tau * main_net_weight + (1.0 - tau) * current_target_net_weight ) Args: main_net: The keras.Model to update from. target_net: The target network to update. tau: The tau value to use in the Polyak averaging formula. rN)r1 variablesassign)r r r old_var current_var updated_varr6r6r7update_target_networks ractionscCsNt|d}tj|tjd}t|jdkr%|dddf}t|jdks|S)aHelper function useful for returning dummy logp's (0) for some actions. Args: actions: The input actions. This can be any struct of complex action components or a simple tensor of different dimensions, e.g. [B], [B, 2], or {"a": [B, 4, 5], "b": [B]}. Returns: A 1D tensor of 0.0 (dummy logp's) matching the batch dim of `actions` (shape=[B]). rrr<N)rJrKr+rrTrLrN)raction_componentlogp_r6r6r7zero_logps_from_actionss rpolicymean_klcs<dd|rtjtj|ddfddddSdS)NcSstdtdS)Na}KL divergence is non-finite, this will likely destabilize your model and the training process. Action(s) in a specific state have near-zero probability. This can happen naturally in deterministic environments where the optimal policy has zero mass for a specific action. To fix this issue, consider setting the coefficient for the KL loss term to zero or increasing policy entropy.r)loggerwarningr+constantr6r6r6r7 print_warnings z5warn_if_infinite_kl_divergence..print_warningcSs tdS)Nr)r+rr6r6r6r7r|, z0warn_if_infinite_kl_divergence..csSr}r6r6rr6r7r|-s)false_fntrue_fn)loss_initializedr+condris_inf)rrr6rr7warn_if_infinite_kl_divergences    r$cCsbd}g}|D]}tj|tjd}|||||}||||7}qt||ks/Jd|S)Nrrz.Passed weight does not have the correct shape.)rprodint_rPrQrL)vectorshapesrarraysrNsizearrayr6r6r7 _unflatten1s  r,c@sPeZdZdZdddZddZddZd d Zd d Zd e fddZ ddZ dS)TensorFlowVariablesaA class used to set and get weights for Tensorflow networks. Attributes: sess (tf.Session): The tensorflow session used to run assignment. variables (Dict[str, tf.Variable]): Extracted variables from the loss or additional variables that are passed in. placeholders (Dict[str, tf.placeholders]): Placeholders for weights. assignment_nodes (Dict[str, tf.Tensor]): Nodes that assign weights. Nc s||_t|}t|}gt|}t|dkri|}|dur qt|dr(|j}|jD]}||vr;| || |q+|j D]}||vrO| || |q?d|j jvs\d|j jvrc |j j t|dkst|_fddtD} |dur| |7} ts| D] } | |j| jj j <qi|_i|_|jD]#\} } tj| j| d| d |j| <| |j| |j| <qdS| D]} | |j| j <qdS) aUCreates TensorFlowVariables containing extracted variables. The variables are extracted by performing a BFS search on the dependency graph with loss as the root node. After the tree is traversed and those variables are collected, we append input_variables to the collected variables. For each variable in the list, the variable has a placeholder and assignment operation created for it. Args: output (tf.Operation, List[tf.Operation]): The tensorflow operation to extract all variables from. sess (Optional[tf.Session]): Optional tf.Session used for running the get and set methods in tf graph mode. Use None for tf eager. input_variables (List[tf.Variables]): Variables to include in the list. rNopVariable VarHandlecsg|] }|jjjvr|qSr6)r.node_defrbrcr4variable_namesr6r7reysz0TensorFlowVariables.__init__.. Placeholder_r{)sessr rsetrLpopleftrr.rDrQaddcontrol_inputsr1rbrr rglobal_variablesrrassignment_nodesr*rr&r~ get_shaperr) selfoutputr6input_variablesqueueexplored_inputstf_objinput_opcontrol variable_listr4r3varr6r3r7__init__Is\            zTensorFlowVariables.__init__cCstdd|jDS)zReturns the total length of all of the flattened variables. Returns: The length of all flattened variables concatenated. css"|] }t|VqdSr})rr%r=rr2r6r6r7 s z4TensorFlowVariables.get_flat_size..)sumr r0r>r6r6r7 get_flat_sizesz!TensorFlowVariables.get_flat_sizecs>jstddjDStfddjDS)zGets the weights and returns them as a flat array. Returns: 1D Array containing the flattened weights. cSg|]}|qSr6)numpyrKr2r6r6r7rez0TensorFlowVariables.get_flat..csg|] }|jjdqS))session)evalr6rKr2rKr6r7res)r6r concatenater r0rKr6rKr7get_flatszTensorFlowVariables.get_flatcsddjD}t||}js&tj|D] \}}||qdSfddjD}jjtj t t||ddS)a;Sets the weights to new_weights, converting from a flat array. Note: You can only set all weights in the network using this function, i.e., the length of the array must match get_flat_size. Args: new_weights (np.ndarray): Flat array containing weights. cSrMr6)r=rr2r6r6r7rerOz0TensorFlowVariables.set_flat..csg|] \}}j|qSr6)r)rcr3r4rKr6r7resrN) r r0r,r6r1rr*rr/r<r)r> new_weightsr(r)r4rrr6rKr7set_flats     zTensorFlowVariables.set_flatcCs|js|jS|j|jS)zReturns a dictionary containing the weights of the network. Returns: Dictionary mapping variable names to their weights. )r6r rrKr6r6r7 get_weightsszTensorFlowVariables.get_weightsrTcCsR|jdur|jD] \}}|||q dS||\}}|jj||ddS)aSets the weights to new_weights. Note: Can set subsets of variables as well, by only passing in the variables you want to be set. Args: new_weights: Dictionary mapping variable names to their weights. Nr)r6r r*r_assign_weightsr)r>rTrbrG assign_listrr6r6r7 set_weightss zTensorFlowVariables.set_weightscsgitjddfdd}|D]3\vr*|qfddD}fddt|d d d D}|rO||d qsVJd tt|ksbJdfddDfS)a'Sets weigths using exact or closest assignable variable name Args: weights: Dictionary mapping variable names to their weights. Returns: Tuple[List, Dict]: assigned variables list, dict of placeholders and weights cstfdd|DS)Ncsg|]}|vr|qSr6r6)rcel2r6r7rerOzOTensorFlowVariables._assign_weights..nb_common_elem..)rL)l1r\r6r[r7nb_common_elemsz;TensorFlowVariables._assign_weights..nb_common_elemcs&|j|<||dSr})rrQremove)rbr&) assignableassignedrr>r6r7rs z3TensorFlowVariables._assign_weights..assigncs$i|]}|d|dqS)/)split)rcrG)rbr^r6r7 sz7TensorFlowVariables._assign_weights..cs.g|]\}}|dkrjj|jkr|qS)r)rNr<)rc close_varcn)r>r&r6r7res z7TensorFlowVariables._assign_weights..cSs |d S)Nr<r6)rr6r6r7r|rz5TensorFlowVariables._assign_weights..)keyrzNo variables in the input matched those in the network. Possible cause: Two networks were defined in the same TensorFlow graph. To fix this, place each network definition in its own tf.Graph.z^All weights couldn't be assigned because no variable had an exact/close name or had same shapecsg|]}j|qSr6)r<r2rKr6r7res)r7r<r2r*sortedrL)r>weightsrcommonselectr6)r`rarrbr^r>r&r7rWs0    z#TensorFlowVariables._assign_weights)NN) __name__ __module__ __qualname____doc__rHrLrSrUrVrrYrWr6r6r6r7r-=s I r-)NF)r)F)rr})rrr%r)r9rr%r)rrrN)Qlogging collectionsrrtypingrrrrrr r gymnasiumrrNrrJgymnasium.spacesr r ray.rllib.utilsr ray.rllib.utils.annotationsrrray.rllib.utils.frameworkrray.rllib.utils.numpyr"ray.rllib.utils.spaces.space_utilsrray.rllib.utils.typingrrrrrrr%ray.rllib.algorithms.algorithm_configrray.rllib.core.learner.learnerrrrrrrr! getLoggerrlrrr+rffloatrsr8rCboolr^rnSpacerwrrrrrrSintrrrrr rrr$r,r-r6r6r6r7s6$    $        0 h :   1 ` (;     \