o `۷i'B+@sddlmZmZmZmZmZmZmZddlZ ddl m Z m Z ddl mZddlmZddlmZddlmZddlmZdd lmZdd lmZdd lmZmZdd lmZm Z dd l!m"Z"ddl#m$Z$ddl%m&Z&m'Z'm(Z(e"\Z)Z*Z+edddddddddddddddddde dddde,deeeeeegee(ee(ffdeedge&fdeeeegee,e(ffdeeee&gdfdeeede(ge'fdeeede'gdfdeeeee'gee,e(ffdeeegee,e(ffdeeegee,e(ffdeeee j-e j-e&gdfdeeee j-e j-e&gdfdeeee j.j-e j.j-e&gdfd eeee j-e j-e&gdfd!eeee j.j-e j.j-e&gefd"eee(ee(gee(e(ffd#eeeee(e(e(gee(e/ee(ffd$eee/d%eeege0fd&eef(d'd(Z1dS)))CallableDictListOptionalTupleTypeUnionN)DEPRECATED_VALUEdeprecation_warning)ModelV2)TFActionDistribution)eager_tf_policy)DynamicTFPolicy)Policy) SampleBatch)TFPolicy) add_mixins force_list) OldAPIStackoverride) try_import_tf)LEARNER_STATS_KEY)AlgorithmConfigDictModelGradients TensorType)get_default_configpostprocess_fnstats_fn optimizer_fncompute_gradients_fnapply_gradients_fn grad_stats_fnextra_action_out_fnextra_learn_fetches_fnvalidate_spaces before_initbefore_loss_init after_init make_modelaction_sampler_fnaction_distribution_fnmixinsget_batch_divisibility_reqobs_include_prev_action_rewardextra_action_fetches_fn gradients_fnnameloss_fnrrrztf.keras.optimizers.Optimizerrr z tf.Operationr!r"r#r$r%r&r'r(r)r*r+r,returncsttt||tkrtddd|durtdddd|dur*tdd ddG fd d d }fd d }fdd}t||_t||_||_ ||_ |S)aSHelper function for creating a dynamic tf policy at runtime. Functions will be run in this order to initialize the policy: 1. Placeholder setup: postprocess_fn 2. Loss init: loss_fn, stats_fn 3. Optimizer init: optimizer_fn, gradients_fn, apply_gradients_fn, grad_stats_fn This means that you can e.g., depend on any policy attributes created in the running of `loss_fn` in later functions such as `stats_fn`. In eager mode, the following functions will be run repeatedly on each eager execution: loss_fn, stats_fn, gradients_fn, apply_gradients_fn, and grad_stats_fn. This means that these functions should not define any variables internally, otherwise they will fail in eager mode execution. Variable should only be created in make_model (if defined). Args: name: Name of the policy (e.g., "PPOTFPolicy"). loss_fn (Callable[[ Policy, ModelV2, Type[TFActionDistribution], SampleBatch], Union[TensorType, List[TensorType]]]): Callable for calculating a loss tensor. get_default_config (Optional[Callable[[None], AlgorithmConfigDict]]): Optional callable that returns the default config to merge with any overrides. If None, uses only(!) the user-provided PartialAlgorithmConfigDict as dict for this Policy. postprocess_fn (Optional[Callable[[Policy, SampleBatch, Optional[Dict[AgentID, SampleBatch]], Episode], None]]): Optional callable for post-processing experience batches (called after the parent class' `postprocess_trajectory` method). stats_fn (Optional[Callable[[Policy, SampleBatch], Dict[str, TensorType]]]): Optional callable that returns a dict of TF tensors to fetch given the policy and batch input tensors. If None, will not compute any stats. optimizer_fn (Optional[Callable[[Policy, AlgorithmConfigDict], "tf.keras.optimizers.Optimizer"]]): Optional callable that returns a tf.Optimizer given the policy and config. If None, will call the base class' `optimizer()` method instead (which returns a tf1.train.AdamOptimizer). compute_gradients_fn (Optional[Callable[[Policy, "tf.keras.optimizers.Optimizer", TensorType], ModelGradients]]): Optional callable that returns a list of gradients. If None, this defaults to optimizer.compute_gradients([loss]). apply_gradients_fn (Optional[Callable[[Policy, "tf.keras.optimizers.Optimizer", ModelGradients], "tf.Operation"]]): Optional callable that returns an apply gradients op given policy, tf-optimizer, and grads_and_vars. If None, will call the base class' `build_apply_op()` method instead. grad_stats_fn (Optional[Callable[[Policy, SampleBatch, ModelGradients], Dict[str, TensorType]]]): Optional callable that returns a dict of TF fetches given the policy, batch input, and gradient tensors. If None, will not collect any gradient stats. extra_action_out_fn (Optional[Callable[[Policy], Dict[str, TensorType]]]): Optional callable that returns a dict of TF fetches given the policy object. If None, will not perform any extra fetches. extra_learn_fetches_fn (Optional[Callable[[Policy], Dict[str, TensorType]]]): Optional callable that returns a dict of extra values to fetch and return when learning on a batch. If None, will call the base class' `extra_compute_grad_fetches()` method instead. validate_spaces (Optional[Callable[[Policy, gym.Space, gym.Space, AlgorithmConfigDict], None]]): Optional callable that takes the Policy, observation_space, action_space, and config to check the spaces for correctness. If None, no spaces checking will be done. before_init (Optional[Callable[[Policy, gym.Space, gym.Space, AlgorithmConfigDict], None]]): Optional callable to run at the beginning of policy init that takes the same arguments as the policy constructor. If None, this step will be skipped. before_loss_init (Optional[Callable[[Policy, gym.spaces.Space, gym.spaces.Space, AlgorithmConfigDict], None]]): Optional callable to run prior to loss init. If None, this step will be skipped. after_init (Optional[Callable[[Policy, gym.Space, gym.Space, AlgorithmConfigDict], None]]): Optional callable to run at the end of policy init. If None, this step will be skipped. make_model (Optional[Callable[[Policy, gym.spaces.Space, gym.spaces.Space, AlgorithmConfigDict], ModelV2]]): Optional callable that returns a ModelV2 object. All policy variables should be created in this function. If None, a default ModelV2 object will be created. action_sampler_fn (Optional[Callable[[TensorType, List[TensorType]], Tuple[TensorType, TensorType]]]): A callable returning a sampled action and its log-likelihood given observation and state inputs. If None, will either use `action_distribution_fn` or compute actions by calling self.model, then sampling from the so parameterized action distribution. action_distribution_fn (Optional[Callable[[Policy, ModelV2, TensorType, TensorType, TensorType], Tuple[TensorType, type, List[TensorType]]]]): Optional callable returning distribution inputs (parameters), a dist-class to generate an action distribution object from, and internal-state outputs (or an empty list if not applicable). If None, will either use `action_sampler_fn` or compute actions by calling self.model, then sampling from the so parameterized action distribution. mixins (Optional[List[type]]): Optional list of any class mixins for the returned policy class. These mixins will be applied in order and will have higher precedence than the DynamicTFPolicy class. get_batch_divisibility_req (Optional[Callable[[Policy], int]]): Optional callable that returns the divisibility requirement for sample batches. If None, will assume a value of 1. Returns: Type[DynamicTFPolicy]: A child class of DynamicTFPolicy based on the specified args. r-T)olderrorNr.r")r3newr4r/rc seZdZ  d f dd Zee dfdd ZeefddZeefdd Z eefd d Z eefd d Z ee fddZ dS)z#build_tf_policy..policy_clsNcst r ||||r||||fdd}tj|||| | ||dr5||||d|_dS)NcsRr ||||dus|jri}n|}t|dr$|j|dS||_dS)N_extra_action_fetches) _is_towerhasattrr6update)policy obs_space action_spaceconfigextra_action_fetches)r&r"Y/home/ubuntu/vllm_env/lib/python3.10/site-packages/ray/rllib/policy/tf_policy_template.pybefore_loss_init_wrappers  zNbuild_tf_policy..policy_cls.__init__..before_loss_init_wrapper) r;r<r=r1rr!r&r(r)r*existing_inputsexisting_modelr,r)r__init__global_timestep)selfr;r<r=rCrBrA) r*r)r'r%r&r"r,r!r1r(rr$r?r@rDs0 z,build_tf_policy..policy_cls.__init__cs"t||}r||||S|SN)rpostprocess_trajectory)rF sample_batchother_agent_batchesepisode)rr?r@rH s z:build_tf_policy..policy_cls.postprocess_trajectorycsTr ||j}n|}t|}|jr|j|}|sdS|jdr&|S|dSN%_tf_policy_handles_more_than_one_lossr)r= optimizerr explorationget_exploration_optimizer)rF optimizers)baserr?r@rNs   z-build_tf_policy..policy_cls.optimizercsLt|}t|}r|jdr|||S||d|dS|||SrL)rr= gradients)rFrNlossrQlosses)rRrr?r@rS)s  z-build_tf_policy..policy_cls.gradientscsr|||S|||SrG)build_apply_op)rFrNgrads_and_vars)r rRr?r@rV9s z2build_tf_policy..policy_cls.build_apply_opcst|fi|jSrG)dictextra_compute_action_fetchesr6rF)rRr?r@rY@s z@build_tf_policy..policy_cls.extra_compute_action_fetchescs&rttiifi|S|SrG)rXrextra_compute_grad_fetchesrZ)rRr#r?r@r[Fs z>build_tf_policy..policy_cls.extra_compute_grad_fetches)NN) __name__ __module__ __qualname__rDrrrHrrNrSrVrYr[r?)r*r)r'r rRr%r&rr"r#r,r!r1r(rrrr$r?r@ policy_clss"$3 r_cstditfi|S)aAllows creating a TFPolicy cls based on settings of another one. Keyword Args: **overrides: The settings (passed into `build_tf_policy`) that should be different from the class that this method is called on. Returns: type: A new TFPolicy sub-class. Examples: >> MySpecialDQNPolicyClass = DQNTFPolicy.with_updates( .. name="MySpecialDQNPolicyClass", .. loss_function=[some_new_loss_function], .. ) Nr?)build_tf_policyrX) overridesoriginal_kwargsr?r@ with_updatesTsz%build_tf_policy..with_updatescstjdiS)Nr?)r _build_eager_tf_policyr?rbr?r@as_eagergsz!build_tf_policy..as_eager) localscopyrrr r staticmethodrdrfr\r^)r0r1rrrrrr r!r"r#r$r%r&r'r(r)r*r+r,r-r.r/r_rdrfr?)r*r)r'r rRr%r&rr"r#r,r!r1r(rrcrrr$r@r`s& ,  6 ~   r`)2typingrrrrrrr gymnasiumgymray._common.deprecationr r ray.rllib.models.modelv2r "ray.rllib.models.tf.tf_action_distr ray.rllib.policyr "ray.rllib.policy.dynamic_tf_policyrray.rllib.policy.policyrray.rllib.policy.sample_batchrray.rllib.policy.tf_policyrray.rllib.utilsrrray.rllib.utils.annotationsrrray.rllib.utils.frameworkr$ray.rllib.utils.metrics.learner_inforray.rllib.utils.typingrrrtf1tftfvstrSpacespacestypeintr`r?r?r?r@s$               %(-0  67<