o `۷iN3@sddlmZmZmZmZmZmZmZmZddl Z ddl m Z ddl mZddlmZddlmZddlmZddlmZdd lmZdd lmZmZdd lmZmZdd lm Z m!Z!dd l"m#Z#ddl$m%Z%ddl&m'Z'm(Z(m)Z)e \Z*Z+e!\Z,Z+edddddddddddddddddddddde-de-deeeeeeegee)ee)ffdeege'fdeeeegee-e)ffdeeeeeeeefeegefdeeeee-e)fee)eegee-e)ffdeeede)gee-e)ffdeeegee-e)ffdeeee'gdfdeeee j.e j.e'gdfdeeee j.e j.e'gdfdeeee j/j.e j/j.e'gdfdeeee j.e j.e'gdfd eeee j/j.e j/j.e'gdfd!eee)ee)gee)e)ffd"eeeee)e)e)gee)e0ee)ffd#eeee j/j.e j/j.e'gefd$eeee j/j.e j/j.e'geeeeffd%eeeegee(e1ffd&eeedgdfd'eee0d(eeege2fd)eef0d*d+Z3dS),)AnyCallableDictListOptionalTupleTypeUnionN) ModelCatalog)ModelV2)TorchDistributionWrapper) TorchModelV2)Policy) SampleBatch) TorchPolicy)NullContextManager add_mixins) OldAPIStackoverride)try_import_jaxtry_import_torch)LEARNER_STATS_KEY)convert_to_numpy)AlgorithmConfigDictModelGradients TensorType)get_default_configstats_fnpostprocess_fnextra_action_out_fnextra_grad_process_fnextra_learn_fetches_fn optimizer_fnvalidate_spaces before_initbefore_loss_init after_init_after_loss_initaction_sampler_fnaction_distribution_fn make_modelmake_model_and_action_distcompute_gradients_fnapply_gradients_fnmixinsget_batch_divisibility_reqname frameworkloss_fnrrrrr ztorch.optim.Optimizerr!r"r#r$r%r&r'r(r)r*r+r,r-r.r/returncszttt|}G fddd|}fdd}t||_||_||_|S)aHelper function for creating a new Policy class at runtime. Supports frameworks JAX and PyTorch. Args: name: name of the policy (e.g., "PPOTorchPolicy") framework: Either "jax" or "torch". loss_fn (Optional[Callable[[Policy, ModelV2, Type[TorchDistributionWrapper], SampleBatch], Union[TensorType, List[TensorType]]]]): Callable that returns a loss tensor. get_default_config (Optional[Callable[[None], AlgorithmConfigDict]]): Optional callable that returns the default config to merge with any overrides. If None, uses only(!) the user-provided PartialAlgorithmConfigDict as dict for this Policy. postprocess_fn (Optional[Callable[[Policy, SampleBatch, Optional[Dict[Any, SampleBatch]], Optional[Any]], SampleBatch]]): Optional callable for post-processing experience batches (called after the super's `postprocess_trajectory` method). stats_fn (Optional[Callable[[Policy, SampleBatch], Dict[str, TensorType]]]): Optional callable that returns a dict of values given the policy and training batch. If None, will use `TorchPolicy.extra_grad_info()` instead. The stats dict is used for logging (e.g. in TensorBoard). extra_action_out_fn (Optional[Callable[[Policy, Dict[str, TensorType], List[TensorType], ModelV2, TorchDistributionWrapper]], Dict[str, TensorType]]]): Optional callable that returns a dict of extra values to include in experiences. If None, no extra computations will be performed. extra_grad_process_fn (Optional[Callable[[Policy, "torch.optim.Optimizer", TensorType], Dict[str, TensorType]]]): Optional callable that is called after gradients are computed and returns a processing info dict. If None, will call the `TorchPolicy.extra_grad_process()` method instead. # TODO: (sven) dissolve naming mismatch between "learn" and "compute.." extra_learn_fetches_fn (Optional[Callable[[Policy], Dict[str, TensorType]]]): Optional callable that returns a dict of extra tensors from the policy after loss evaluation. If None, will call the `TorchPolicy.extra_compute_grad_fetches()` method instead. optimizer_fn (Optional[Callable[[Policy, AlgorithmConfigDict], "torch.optim.Optimizer"]]): Optional callable that returns a torch optimizer given the policy and config. If None, will call the `TorchPolicy.optimizer()` method instead (which returns a torch Adam optimizer). validate_spaces (Optional[Callable[[Policy, gym.Space, gym.Space, AlgorithmConfigDict], None]]): Optional callable that takes the Policy, observation_space, action_space, and config to check for correctness. If None, no spaces checking will be done. before_init (Optional[Callable[[Policy, gym.Space, gym.Space, AlgorithmConfigDict], None]]): Optional callable to run at the beginning of `Policy.__init__` that takes the same arguments as the Policy constructor. If None, this step will be skipped. before_loss_init (Optional[Callable[[Policy, gym.spaces.Space, gym.spaces.Space, AlgorithmConfigDict], None]]): Optional callable to run prior to loss init. If None, this step will be skipped. after_init (Optional[Callable[[Policy, gym.Space, gym.Space, AlgorithmConfigDict], None]]): DEPRECATED: Use `before_loss_init` instead. _after_loss_init (Optional[Callable[[Policy, gym.spaces.Space, gym.spaces.Space, AlgorithmConfigDict], None]]): Optional callable to run after the loss init. If None, this step will be skipped. This will be deprecated at some point and renamed into `after_init` to match `build_tf_policy()` behavior. action_sampler_fn (Optional[Callable[[TensorType, List[TensorType]], Tuple[TensorType, TensorType]]]): Optional callable returning a sampled action and its log-likelihood given some (obs and state) inputs. If None, will either use `action_distribution_fn` or compute actions by calling self.model, then sampling from the so parameterized action distribution. action_distribution_fn (Optional[Callable[[Policy, ModelV2, TensorType, TensorType, TensorType], Tuple[TensorType, Type[TorchDistributionWrapper], List[TensorType]]]]): A callable that takes the Policy, Model, the observation batch, an explore-flag, a timestep, and an is_training flag and returns a tuple of a) distribution inputs (parameters), b) a dist-class to generate an action distribution object from, and c) internal-state outputs (empty list if not applicable). If None, will either use `action_sampler_fn` or compute actions by calling self.model, then sampling from the parameterized action distribution. make_model (Optional[Callable[[Policy, gym.spaces.Space, gym.spaces.Space, AlgorithmConfigDict], ModelV2]]): Optional callable that takes the same arguments as Policy.__init__ and returns a model instance. The distribution class will be determined automatically. Note: Only one of `make_model` or `make_model_and_action_dist` should be provided. If both are None, a default Model will be created. make_model_and_action_dist (Optional[Callable[[Policy, gym.spaces.Space, gym.spaces.Space, AlgorithmConfigDict], Tuple[ModelV2, Type[TorchDistributionWrapper]]]]): Optional callable that takes the same arguments as Policy.__init__ and returns a tuple of model instance and torch action distribution class. Note: Only one of `make_model` or `make_model_and_action_dist` should be provided. If both are None, a default Model will be created. compute_gradients_fn (Optional[Callable[ [Policy, SampleBatch], Tuple[ModelGradients, dict]]]): Optional callable that the sampled batch an computes the gradients w.r. to the loss function. If None, will call the `TorchPolicy.compute_gradients()` method instead. apply_gradients_fn (Optional[Callable[[Policy, "torch.optim.Optimizer"], None]]): Optional callable that takes a grads list and applies these to the Model's parameters. If None, will call the `TorchPolicy.apply_gradients()` method instead. mixins (Optional[List[type]]): Optional list of any class mixins for the returned policy class. These mixins will be applied in order and will have higher precedence than the TorchPolicy class. get_batch_divisibility_req (Optional[Callable[[Policy], int]]): Optional callable that returns the divisibility requirement for sample batches. If None, will assume a value of 1. Returns: Type[TorchPolicy]: TorchPolicy child class constructed from the specified args. cseZdZ fddZee dfdd Ze fddZe fdd Zefd d Z efd d Z e fddZ efddZ efddZ ddZddZZS)z&build_policy_class..policy_clsc s||_|_|jd< r ||||jr||||j r? dus)Jd |||||_tj||jdd\}}n( rL ||||\|_}ntj||jdd\}}tj||||jdd|_t}t|j|ssJd |_|jj |||||j|jdrdn||ddd |j |jj p}|r|||j |j ||jd |jdrdn d r||||d |_dS) Nr1zAEither `make_model` or `make_model_and_action_dist` must be None!model)r1) obs_space action_space num_outputs model_configr1z5ERROR: Generated Model must be a TorchModelV2 object! in_evaluation max_seq_len) observation_spacer6configr4lossaction_distribution_classr(r)r:r/T)auto_remove_unneeded_view_reqsrr)r<r1r4r get_action_dist get_model_v2r isinstance parent_cls__init__view_requirementsupdater;r6!_initialize_loss_from_dummy_batchglobal_timestep) selfr5r6r< dist_class_ logit_dim model_cls_before_loss_init)r'r)r(r&r$r%r1r/r2r*r+rCrr#V/home/ubuntu/vllm_env/lib/python3.10/site-packages/ray/rllib/policy/policy_template.pyrDsz          z/build_policy_class..policy_cls.__init__Ncs`|"t|||}r||||WdS|WdS1s)wYdSN)_no_grad_contextsuperpostprocess_trajectory)rI sample_batchother_agent_batchesepisode) __class__rrOrPrTLs  $z=build_policy_class..policy_cls.postprocess_trajectorycsr|||S|||S)zCalled after optimizer.zero_grad() and loss.backward() calls. Allows for gradient processing before optimizer.step() is called. E.g. for gradient clipping. )extra_grad_process)rI optimizerr=)r rCrOrPrY_s z9build_policy_class..policy_cls.extra_grad_processcs.rt|}ttiifi|S|SrQ)rdictrextra_compute_grad_fetches)rIfetches)r!rCrOrPr\ks  zAbuild_policy_class..policy_cls.extra_compute_grad_fetchescsr||S||SrQ)compute_gradients)rIbatch)r,rCrOrPr^ts  z8build_policy_class..policy_cls.compute_gradientscs"r ||dS||dSrQ)apply_gradients)rI gradients)r-rCrOrPr`{sz6build_policy_class..policy_cls.apply_gradientscs^|!r|||||}n |||||}||WdS1s(wYdSrQ)rRextra_action_out_convert_to_numpy)rI input_dict state_batchesr4 action_dist stats_dict)rrCrOrPrbs   $z7build_policy_class..policy_cls.extra_action_outcs"r ||j}|S|}|SrQ)r<rZ)rI optimizers)r"rCrOrPrZs   z0build_policy_class..policy_cls.optimizercsT|r ||}n|j||}||WdS1s#wYdSrQ)rRrCextra_grad_inforc)rI train_batchrg)rrOrPris  $z6build_policy_class..policy_cls.extra_grad_infocSs|jdkr tStSNtorch)r1rlno_gradr)rIrOrOrPrRs z7build_policy_class..policy_cls._no_grad_contextcSs|jdkr t|S|Srk)r1r)rIdatarOrOrPrcs z8build_policy_class..policy_cls._convert_to_numpy)NN)__name__ __module__ __qualname__rDrrrTrYr\r^r`rbrZrirRrc __classcell__rO)r'r)r(r&r-r$r%r,rr r!r1r/r2r*r+r"rCrrr#)rXrP policy_clss*&U  rscstditfi|S)aCreates a Torch|JAXPolicy cls based on settings of another one. Keyword Args: **overrides: The settings (passed into `build_torch_policy`) that should be different from the class that this method is called on. Returns: type: A new Torch|JAXPolicy sub-class. Examples: >> MySpecialDQNPolicyClass = DQNTorchPolicy.with_updates( .. name="MySpecialDQNPolicyClass", .. loss_function=[some_new_loss_function], .. ) NrO)build_policy_classr[) overrides)original_kwargsrOrP with_updatessz(build_policy_class..with_updates)localscopyrr staticmethodrwrorq)r0r1r2rrrrr r!r"r#r$r%r&r'r(r)r*r+r,r-r.r/basersrwrO)r'r)r(r&r-r$r%r,rr r!r1r/r2r*r+r"rvrCrrr#rPrt s S < 5 rt)4typingrrrrrrrr gymnasiumgymray.rllib.models.catalogr ray.rllib.models.modelv2r (ray.rllib.models.torch.torch_action_distr $ray.rllib.models.torch.torch_modelv2r ray.rllib.policy.policyrray.rllib.policy.sample_batchrray.rllib.policy.torch_policyrray.rllib.utilsrrray.rllib.utils.annotationsrrray.rllib.utils.frameworkrr$ray.rllib.utils.metrics.learner_inforray.rllib.utils.numpyrray.rllib.utils.typingrrrjaxrKrlstrSpacespacestyper[intrtrOrOrOrPs4(                   #'(+.169>A GLRU XYZ