o `Û·iH.ã @sHddlmZmZmZmZmZmZmZmZm Z m Z m Z m Z ddl ZddlmZer~ddlmZddlZddlZddlZddlmZddlmZddlmZddlmZddl m!Z!dd l"m#Z#dd l$m%Z%dd l&m'Z'dd l(m)Z)dd l*m+Z+m,Z,ddl-m.Z.e dZ/ e e/e0e1fZ2 e e e3dfee3fZ4e dZ5 e e6ddfZ7 e dZ8 ee6efZ9 e0Z: e0Z; e0Z< ee e3e e3e e3e3ffe e3e e3e3fffZ= e ee6e ee>e6ffe>e6fZ? e0Z@ e e3e6fZA e eejBfZC edgeeCfZD eZE e6ZF e6ZG eeFdfZH e dZI eeFedgeJfZK eeEeIgeGfZL e eeGeeGedgeJffZM ee6e2fZN e e dZO e e3e6fZP e3ZQ eeEefZR eeAeRfZS eZT eZU e0ZV eZW ee6dfZX eZY e dZZ eZZ[ e dZ\ eZ] ee]e\fZ^ ee\Z_ ee6e\fZ` e eaeee e3eafee e3e e3eafffZb e0Zc e0Zd e ee e/e/fee/fZe e0Zf ee6e/fZg e ddee6effZh e ejijjee6ejijjfe ejijjdffZk eeeZl e e2elefZm eGdd„dƒƒZneGd d!„d!ƒƒZoeGd"d#„d#ƒƒZpe d$ƒZqdS)%é) Ú TYPE_CHECKINGÚAnyÚCallableÚDictÚHashableÚListÚOptionalÚSequenceÚTupleÚTypeÚTypeVarÚUnionN)Ú OldAPIStack)ÚNDArray)ÚMultiRLModuleSpec)Ú RLModuleSpec)Ú EnvContext)ÚMultiAgentEpisode)ÚSingleAgentEpisode)ÚDynamicTFPolicyV2)ÚEagerTFPolicyV2)Ú PolicySpec)ÚMultiAgentBatchÚ SampleBatch)ÚViewRequirement)z NDArray[Any]z jnp.ndarrayz tf.Tensorú torch.Tensor.)ztorch.nn.Modulez keras.Modelz torch.deviceÚint)rrrr)rrr)rrr)ztorch.optim.Optimizerzkeras.optimizers.Optimizer)rz tf.Variablerc@s&eZdZdZdededefdd„ZdS)ÚAgentConnectorDataTypeaƒData type that is fed into and yielded from agent connectors. Args: env_id: ID of the environment. agent_id: ID to help identify the agent from which the data is received. data: A payload (``data``). With RLlib's default sampler, the payload is a dictionary of arbitrary data columns (obs, rewards, terminateds, truncateds, etc). Úenv_idÚagent_idÚdatacCs||_||_||_dS©N)rrr )Úselfrrr ©r#úL/home/ubuntu/vllm_env/lib/python3.10/site-packages/ray/rllib/utils/typing.pyÚ__init__5s zAgentConnectorDataType.__init__N)Ú__name__Ú __module__Ú __qualname__Ú__doc__Ústrrr%r#r#r#r$r)s rc@s*eZdZdZdedededefdd„ZdS) ÚActionConnectorDataTypea?Data type that is fed into and yielded from agent connectors. Args: env_id: ID of the environment. agent_id: ID to help identify the agent from which the data is received. input_dict: Input data that was passed into the policy. Sometimes output must be adapted based on the input, for example action masking. So the entire input data structure is provided here. output: An object of PolicyOutputType. It is is composed of the action output, the internal state output, and additional data fetches. rrÚ input_dictÚoutputcCs||_||_||_||_dSr!)rrr,r-)r"rrr,r-r#r#r$r%Ns z ActionConnectorDataType.__init__N)r&r'r(r)r*ÚTensorStructTypeÚPolicyOutputTyper%r#r#r#r$r+?s þýüûr+c@s*eZdZdZdeeefddfdd„ZdS)ÚAgentConnectorsOutputa=Final output data type of agent connectors. Args are populated depending on the AgentConnector settings. The branching happens in ViewRequirementAgentConnector. Args: raw_dict: The raw input dictionary that sampler can use to build episodes and training batches. This raw dict also gets passed into ActionConnectors in case it contains data useful for action adaptation (e.g. action masks). sample_batch: The SampleBatch that can be immediately used for querying the policy for next action. Úraw_dictÚ sample_batchrcCs||_||_dSr!)r1r2)r"r1r2r#r#r$r%os zAgentConnectorsOutput.__init__N)r&r'r(r)rr*r.r%r#r#r#r$r0_s ÿÿr0ÚT)rÚtypingrrrrrrrr r r r r Ú gymnasiumÚgymÚray.rllib.utils.annotationsrÚ jax.numpyÚnumpyÚjnpÚkerasÚ tensorflowÚtfÚtorchÚ numpy.typingrÚ(ray.rllib.core.rl_module.multi_rl_modulerÚ"ray.rllib.core.rl_module.rl_modulerÚray.rllib.env.env_contextrÚ!ray.rllib.env.multi_agent_episoderÚ"ray.rllib.env.single_agent_episoderÚ%ray.rllib.policy.dynamic_tf_policy_v2rÚ#ray.rllib.policy.eager_tf_policy_v2rÚray.rllib.policy.policyrÚray.rllib.policy.sample_batchrrÚ!ray.rllib.policy.view_requirementrÚ TensorTypeÚdictÚtupler.rÚ TensorShapeÚ NetworkTyper*Ú DeviceTypeÚRLModuleSpecTypeÚ StateDictÚAlgorithmConfigDictÚPartialAlgorithmConfigDictÚModelConfigDictÚConvFilterSpecÚtypeÚFromConfigSpecÚ EnvConfigDictÚEnvIDÚEnvÚEnvTypeÚ EnvCreatorÚAgentIDÚPolicyIDÚModuleIDÚMultiAgentPolicyConfigDictÚ EpisodeTypeÚboolÚIsPolicyToTrainÚAgentToModuleMappingFnÚShouldModuleBeUpdatedFnÚ PolicyStateÚTFPolicyV2TypeÚ EpisodeIDÚUnrollIDÚMultiAgentDictÚ MultiEnvDictÚ EnvObsTypeÚ EnvActionTypeÚ EnvInfoDictÚFileTypeÚViewRequirementsDictÚ ResultDictÚLocalOptimizerÚ OptimizerÚParamÚParamRefÚ ParamDictÚ ParamListÚNamedParamDictÚfloatÚLearningRateOrScheduleÚ GradInfoDictÚLearnerStatsDictÚModelGradientsÚ ModelWeightsÚModelInputDictÚSampleBatchTypeÚspacesÚSpaceÚ SpaceStructÚ StateBatchesr/rr+r0r3r#r#r#r$Ús 8              ,ÿ   ÿÿ        þÿ $ÿ