o ciZ@sddlZddlmZmZmZmZddlZddlZddl m Z ddl m Z ddl mZddlmZddlmZddlmZdd lmZdd lmZdd lmZmZdd lmZdd lmZeZ eGddde Z!dS)N)DictAnyOptionalList)Dataset)OffPolicyEstimator)compute_q_and_v_values)OfflineEvaluator) FQETorchModel)Policy) convert_ma_batch_to_sample_batch) SampleBatch) DeveloperAPIoverride)SampleBatchType)convert_to_numpyc seZdZdZee  ddedededee ffdd Z eed e d e e e ffd d Zeede d e e eeffddZddZeeded e e e ffddZeedddeded e e e ffddZZS) DirectMethodaThe Direct Method estimator. Let s_t, a_t, and r_t be the state, action, and reward at timestep t. This method trains a Q-model for the evaluation policy \pi_e on behavior data generated by \pi_b. Currently, RLlib implements this using Fitted-Q Evaluation (FQE). You can also implement your own model and pass it in as `q_model_config = {"type": your_model_class, **your_kwargs}`. This estimator computes the expected return for \pi_e for an episode as: V^{\pi_e}(s_0) = \sum_{a \in A} \pi_e(a | s_0) Q(s_0, a) and returns the mean and standard deviation over episodes. For more information refer to https://arxiv.org/pdf/1911.06854.pdfNpolicygammaepsilon_greedyq_model_configcsvt|||t|dr|jdddksJd|pi}|dt}|d ||d||_t|jds9Jdd S) a Initializes a Direct Method OPE Estimator. Args: policy: Policy to evaluate. gamma: Discount factor of the environment. epsilon_greedy: The probability by which we act acording to a fully random policy during deployment. With 1-epsilon_greedy we act according the target policy. q_model_config: Arguments to specify the Q-model. Must specify a `type` key pointing to the Q-model class. This Q-model is trained in the train() method and is used to compute the state-value estimates for the DirectMethod estimator. It must implement `train` and `estimate_v`. TODO (Rohan138): Unify this with RLModule API. config frameworktorchz,Framework must be torch to use DirectMethod.type)rr estimate_vz'self.model must implement `estimate_v`!N)super__init__hasattrrgetpopr model)selfrrrr model_cls __class__r^/home/ubuntu/.local/lib/python3.10/site-packages/ray/rllib/offline/estimators/direct_method.pyr's$  zDirectMethod.__init__episodereturncCs\i}|d}d}t|jD] }||||j|7}q ||dd}||d<||d<|S)Nrewardsr v_behaviorv_target)rangecountr_compute_v_target)r$r)estimates_per_epsioder+r-tr.rrr(estimate_on_single_episodeSsz'DirectMethod.estimate_on_single_episodebatchcCs.i}|d}|}||}||d<||d<|S)Nr+r-r.)r1)r$r5r2r+r-r.rrr(estimate_on_single_step_samplescs z,DirectMethod.estimate_on_single_step_samplescCs|j|}t|}|S)N)r#rr)r$ init_stepr.rrr(r1rs zDirectMethod._compute_v_targetcCs"t|}|j|}dt|iS)zTrains self.model on the given batch. Args: batch: A SampleBatchType to train on Returns: A dict with key "loss" and value as the mean training loss. loss)r r#trainnpmean)r$r5lossesrrr(r9ws zDirectMethod.train.) n_parallelismdatasetr=c Cs|t||d}|jt|d|jj|jddd}|d}|d}||}|d|t |}||||dS) aoCalculates the Direct Method estimate on the given dataset. Note: This estimate works for only discrete action spaces for now. Args: dataset: Dataset to compute the estimate on. Each record in dataset should include the following columns: `obs`, `actions`, `action_prob` and `rewards`. The `obs` on each row shoud be a vector of D dimensions. n_parallelism: The number of parallel workers to use. Returns: Dictionary with the following keys: v_target: The estimated value of the target policy. v_behavior: The estimated value of the behavior policy. v_gain: The estimated gain of the target policy over the behavior policy. v_std: The standard deviation of the estimated value of the target. r,pandasF) model_class model_statecompute_q_values) batch_size batch_format fn_kwargsr+v_values)r-r. v_gain_mean v_gain_ste) maxr0 map_batchesrr#r' get_stater;stdmathsqrt) r$r>r=rC updated_dsr-r.rGrHrrr(estimate_on_datasets( z DirectMethod.estimate_on_dataset)rN)__name__ __module__ __qualname____doc__rrr floatrrrr strrr4rr6r1rr9r rintrP __classcell__rrr&r(rsD+   r)"loggingtypingrrrrrMnumpyr:ray.datar1ray.rllib.offline.estimators.off_policy_estimatorr*ray.rllib.offline.offline_evaluation_utilsr#ray.rllib.offline.offline_evaluatorr ,ray.rllib.offline.estimators.fqe_torch_modelr ray.rllib.policyr ray.rllib.policy.sample_batchr r ray.rllib.utils.annotationsrrray.rllib.utils.typingrray.rllib.utils.numpyr getLoggerloggerrrrrr(s$