o ci9@sdZddlZddlmZmZddlmZddlm Z ddl m Z ddl m Z mZddlmZe \ZZ d"d d Zd dd d dddZd d ddedefddZd d d ddededdfddZddZddZddZd d!ZdS)#a [1] Mastering Diverse Domains through World Models - 2023 D. Hafner, J. Pasukonis, J. Ba, T. Lillicrap https://arxiv.org/pdf/2301.04104v1.pdf [2] Mastering Atari with Discrete World Models - 2021 D. Hafner, T. Lillicrap, M. Norouzi, J. Ba https://arxiv.org/pdf/2010.02193.pdf N)create_cartpole_dream_imagecreate_frozenlake_dream_image)DEFAULT_MODULE_ID)Columns)try_import_torch)LEARNER_RESULTSREPLAY_BUFFER_RESULTS)inverse_symlogtorchc Cs|j}|d}|d}|dkrItt|jjj}|jjt| ||df |t| ||f|jdd |d } n|jj| ||df| ||f|jddd} t | ||f|} | S)Returnsrr N)hz)shapenextiter world_modeldecoder parametersdevicer from_numpyreshapetodetachcpunumpynp) h_t0_to_H z_t0_to_H dreamer_modelobs_dims_shape frameworkrTBr!reconstructed_obs_distr_means_TxBreconstructed_obs_T_Br(b/home/ubuntu/.local/lib/python3.10/site-packages/ray/rllib/algorithms/dreamerv3/utils/summaries.pyreconstruct_obs_from_h_and_zs.  r*)rT) batch_indicesdescinclude_imagesr#cCs(|sdS|d}t|d|d|||d} |drtnt} |D]q} g} tt| dD]L} | | | | | |d| | |d| | |d | d| d |vrY|d | | nd|d | d| |d | | |d| | d d q,|d|rd|ndd| t j | ddiq dS)N dream_datah_states_t0_to_H_BxTz_states_prior_t0_to_H_BxTrr r!r"r#CartPoler values_dreamed_t0_to_H_BxT actions_ints_dreamed_t0_to_H_BxTrewards_dreamed_t0_to_H_BxT DISAGREE_intrinsic_rewards_H_BxTcontinues_dreamed_t0_to_H_BxTVALUE_TARGETS_H_BxTT) dreamed_obs dreamed_V dreamed_a dreamed_r_tp1dreamed_ri_tp1 dreamed_c_tp1 value_target initial_h as_tensordreamed_trajectories__Baxis) r* startswithrrrangelenappendrupdater concatenate)resultsenvr!r"r+r,r-r#r.dreamed_obs_H_Bfuncbimagestr(r(r)report_dreamed_trajectoryHsN    rU) symlog_obs do_reportrVrWc Csttdf}|j|dgdd}|j|ddd|d}|s'|j|dddSt|t|d |f|tjj d d|tjd d ||d dS) aSummarizes sampled data (from the replay buffer) vs world-model predictions. World model predictions are based on the posterior states (z computed from actual observation encoder input + the current h-states). Observations: Computes MSE (sampled vs predicted/recreated) over all features. For image observations, also creates direct image comparisons (sampled images vs predicted (posterior) ones). Rewards: Compute MSE (sampled vs predicted). Continues: Compute MSE (sampled vs predicted). Args: metrics: The MetricsLogger object of the DreamerV3 algo. sample: The sampled data (dict) from the replay buffer. Already tf-tensor converted. batch_size_B: The batch size (B). This is the number of trajectories sampled from the buffer. batch_length_T: The batch length (T). This is the length of an individual trajectory sampled from the buffer. do_report: Whether to actually log the report (default). If this is set to False, this function serves as a clean-up on the given metrics, making sure they do NOT contain anymore any (spacious) data relevant for producing the report/videos. /WORLD_MODEL_fwd_out_obs_distribution_means_b0xTNdefaultr F key_error.WORLD_MODEL_sampled_vs_predicted_posterior_b0x_videosr rrmetricscomputed_float_obs_B_T_dimssampled_obs_B_T_dims metrics_keyrV) rrpeekdelete _report_obsrrrOBSr) r`sample batch_size_Bbatch_length_TrVrWfwd_output_key*predicted_observation_means_single_examplefinal_result_keyr(r(r)report_predicted_vs_sampled_obss2"  rn)rVrWr#returncCsV|jttdfid}|jttdddd|d} d|d} d|d} |s>|j| dd|j| dd|j| ddd St|d d |d d ||tjjd d |d} |} | |}t|t | d dd d|tjd d| |f| |dt ||dd |tj d d | |f| dt ||dd d|dd d | |f| dd S)aLogs dreamed observations, rewards, continues and compares them vs sampled data. For obs, we'll try to create videos (side-by-side comparison) of the dreamed, recreated-from-prior obs vs the sampled ones (over dreamed_T timesteps). Args: metrics: The MetricsLogger object of the DreamerV3 algo. sample: The sampled data (dict) from the replay buffer. Already tf-tensor converted. burn_in_T: The number of burn-in timesteps (these will be skipped over in the reported video comparisons and MSEs). dreamed_T: The number of timesteps to produce dreamed data for. dreamer_model: The DreamerModel to use to create observation vectors/images from dreamed h- and (prior) z-states. symlog_obs: Whether to inverse-symlog the computed observations or not. Set this to True for environments, in which we should symlog the observations. do_report: Whether to actually log the report (default). If this is set to False, this function serves as a clean-up on the given metrics, making sure they do NOT contain anymore any (spacious) data relevant for producing the report/videos. r.rYFr[%EVALUATION_sampled_vs_dreamed_prior_H_obs _rewards_MSE_continues_MSENh_states_t0_to_H_Bx1rz_states_prior_t0_to_H_Bx1rr1r r_rewards_dreamed_t0_to_H_Bx1)r`computed_rewardssampled_rewardsrccontinues_dreamed_t0_to_H_Bx1? is_terminated)r`computed_continuessampled_continuesrc)rdrrrer*rrgrrfrswapaxes_report_rewardsREWARDS_report_continues)r`rh burn_in_T dreamed_Tr!rVrWr#r.final_result_key_obsfinal_result_key_rewfinal_result_key_contrPt0tHr(r(r))report_dreamed_eval_trajectory_vs_samplessZ           rcCsB|}|}|}|}|j|j||||dtdddS)N)capacitysize_num_episodessize_timestepsreplayed_steps added_stepsr )keywindow)get_num_episodesget_num_timestepsget_sampled_timestepsget_added_timestepslog_dictrr)r` replay_bufferepisodes_in_buffer ts_in_bufferrrr(r(r)!report_sampling_and_replay_buffer!s rcCst|jdvrit|jdkrdnd}|rt|}|s7|dd}|dd}t|ddtj}t||}t|ddtj}t||}tj||gd d }t|jd kr^t |d }|j ||d d dd Sd S)aSummarizes computed- vs sampled observations: MSE and (if applicable) images. Args: metrics: The MetricsLogger object of the DreamerV3 algo. computed_float_obs_B_T_dims: Computed float observations (not clipped, not cast'd). Shape=(B, T, [dims ...]). sampled_obs_B_T_dims: Sampled observations (as-is from the environment, meaning this could be uint8, 0-255 clipped images). Shape=(B, T, [dims ...]). metrics_key: The metrics key (or key sequence) under which to log ths resulting video sequence. symlog_obs: Whether to inverse-symlog the computed observations or not. Set this to True for environments, in which we should symlog the observations. )r)rr rr)rrr rrzggo@r rFrNr )reducer) rJrr rclipastypeuint8 transposerM expand_dims log_value)r`rarbrcrVtranspose_axescomputed_imagessampled_vs_computed_imagesr(r(r)rf5s:      rfcCs2tt||}t|}|j||dddSNr )r)rmeansquarer)r`rwrxrcmse_sampled_vs_computed_rewardsr(r(r)rus   rcCs0tt|||j}|j||dddSr)rrrrdtyper)r`r|r}rc!mse_sampled_vs_computed_continuesr(r(r)rs r)r )__doc__rr.ray.rllib.algorithms.dreamerv3.utils.debuggingrrray.rllib.corerray.rllib.core.columnsrray.rllib.utils.frameworkrray.rllib.utils.metricsrrray.rllib.utils.tf_utilsr r rCr*rUboolrnrrrfrrr(r(r(r)sJ       2 E G Z@