o ci @sddlZddlZddlmZmZmZmZddl m Z ddl m Z ddl mZddlmZer:ddlmZddlmZe dd ejd ed d eeefdedejf ddZed ejdeeefdeddejfddZed ejdejfddZdS)N)AnyDictType TYPE_CHECKING) SampleBatch)Policy)convert_to_numpy) DeveloperAPI) FQETorchModel)OffPolicyEstimatorTbatch model_classr model_statecompute_q_valuesreturncCsx||}ttjt|tjtjt|tjdi}||}t|}||d<|r:| |}t|}||d<|S)aComputes the Q and V values for the given batch of samples. This function is to be used with map_batches() to perform a batch prediction on a dataset of records with `obs` and `actions` columns. Args: batch: A sub-batch from the dataset. model_class: The model class to use for the prediction. This class should be a sub-class of FQEModel that implements the estimate_q() and estimate_v() methods. model_state: The state of the model to use for the prediction. compute_q_values: Whether to compute the Q values or not. If False, only the V is computed and returned. Returns: The modified batch with the Q and V values added as columns. v_valuesq_values) from_staterOBSnpvstackACTIONSsqueeze estimate_vr estimate_q)r r rrmodel sample_batchrrr^/home/ubuntu/.local/lib/python3.10/site-packages/ray/rllib/offline/offline_evaluation_utils.pycompute_q_and_v_valuess   r policy_stateestimator_classr c Cst|}||ddd}ttjt|djtjt|djdtj t|djdtj t|djdi}| |}|tj }|tj }||} | |} | |d<| |d <||d <||d <|S) a%Computes the importance sampling weights for the given batch of samples. For a lot of off-policy estimators, the importance sampling weights are computed as the propensity score ratio between the new and old policies (i.e. new_pi(act|obs) / old_pi(act|obs)). This function is to be used with map_batches() to perform a batch prediction on a dataset of records with `obs`, `actions`, `action_prob` and `rewards` columns. Args: batch: A sub-batch from the dataset. policy_state: The state of the policy to use for the prediction. estimator_class: The estimator class to use for the prediction. This class Returns: The modified batch with the importance sampling weights, weighted rewards, new and old propensities added as columns. r)policygammaepsilon_greedyobsactionsr action_probrewardsweightsweighted_rewardsnew_probold_prob) rrrrrrvaluesrr ACTION_PROBREWARDScompute_action_probs) r r!r"r# estimatorrr,r-r)r*r+rrrcompute_is_weights<s&    r3cCsJtjtjtjtjtjtjh}|jD]}||vr"||dd||<q|S)aRemoves the time dimension from the given sub-batch of the dataset. If each row in a dataset has a time dimension ([T, D]), and T=1, this function will remove the T dimension to convert each row to of shape [D]. If T > 1, the row is left unchanged. This function is to be used with map_batches(). Args: batch: The batch to remove the time dimension from. Returns: The modified batch with the time dimension removed (when applicable) cSst|dkr |dS|S)Nr)len)xrrrsz!remove_time_dim..) rrrr/r0NEXT_OBSDONEScolumnsapply)r BATCHED_KEYSkrrrremove_time_dimks r>)T)numpyrpandaspdtypingrrrrray.rllib.policy.sample_batchrray.rllib.policyrray.rllib.utils.numpyrray.rllib.utils.annotationsr ,ray.rllib.offline.estimators.fqe_torch_modelr 1ray.rllib.offline.estimators.off_policy_estimatorr DataFramestrboolr r3r>rrrrsF        ,  .