o `۷i7@sddlmZddlmZmZddlZddlZddlm Z ddl m Z ddl m Z mZe deejefdefd d Ze d d Ze Gd ddejZe GdddejZe GdddejZe GdddejZe GdddejZe GdddejZe GdddejZe GdddejZe GdddejZe Gdd d ejZ e Z!e ! "  #d.dejd$e"d%e"d&ee"d'edejf d(d)Z#e d/d,d-Z$dS)0)deque)OptionalUnionN)spaces) PublicAPI)resizergb2grayenvreturncCsVt|ts!t|jdr|jjdurt|jjdkrdSdt|vS|dp*|dS)aReturns, whether a given env object or env descriptor (str) is an Atari env. Args: env: The gym.Env object or a string descriptor of the env (for example, "ale_py:ALE/Pong-v5"). Returns: Whether `env` is an Atari environment. shapeNFz AtariEnvtj|||jddksJt|jdksJdS)zLTake action on reset. For environments that are fixed until firing.FIREN)rrrr.get_action_meaningsrr rrrreszFireResetEnv.__init__cKsz|jjdi||jd\}}}}}|s|r!|jjdi||jd\}}}}}|s0|r9|jjdi|||fS)Nr:r r)r r5r-)r!r6r1r7r2r3r4rrrr5mszFireResetEnv.resetcC |j|Srr r-r!acrrrr-w zFireResetEnv.stepNr&r'r(rr5r-rrrrr9cs r9c@s4eZdZddZdddddZddZd d ZdS) FrameStackcCs|tj||||_tg|d|_|jj}tj t j |jj |ddt j |jj |dd|d|d|d|f|jjd|_dS) zStack k last frames.)maxlen)repeatsaxisrr:r lowhighr dtypeN)rrrkrframesrr rBoxr#repeatrJrKrL)r!r rMshprrrr}s zFrameStack.__init__NseedoptionscCs<|jj||d\}}t|jD]}|j|q||fS)NrR)r r5rangerMrNappend_get_ob)r!rSrTobinfosr7rrrr5s zFrameStack.resetcCs4|j|\}}}}}|j||||||fSr)r r-rNrVrW)r!r0rXr%r2r3r4rrrr-s zFrameStack.stepcCs$t|j|jks Jtj|jddS)Nr rH)rrNrMr# concatenater!rrrrWszFrameStack._get_ob)r&r'r(rr5r-rWrrrrrD{s   rDc@r)FrameStackTrajectoryViewcCsNtj|||jj}|ddksJtjdd|d|df|jjd|_dS)z4No stacking. Trajectory View API takes care of this.r r:rrIN)rrrrr rrOrL)r!r rQrrrrs  z!FrameStackTrajectoryView.__init__cCstj|ddS)NrFrZ)r#squeezer! observationrrrrasz$FrameStackTrajectoryView.observationN)r&r'r(rrarrrrr]s r]c@&eZdZd ddZddZddZdS) MaxAndSkipEnvcCs4tj||tjd|jj|jjd|_||_ dS)z!Return only every `skip`-th frame)r )rLN) rrrr#zerosrr rL _obs_buffer_skip)r!r skiprrrrs  zMaxAndSkipEnv.__init__c Csd}d}}}t|jD]/}|j|\}}}}}||jdkr&||jd<||jdkr2||jd<||7}|s:|r<nq |jjdd} | ||||fS)z:Repeat action, sum reward, and max over last observations.gNr rr:rZ)rUrgr r-rfmax) r!r0 total_rewardr2r3r4ir1r% max_framerrrr-s   zMaxAndSkipEnv.stepcKs|jjdi|S)Nr)r r5)r!r6rrrr5r"zMaxAndSkipEnv.resetN)rdr8rrrrrcs  rcc@sFeZdZdddZddZddZdd Zd d Zd d ZddZ dS) MonitorEnvNcCs<tj||d|_d|_d|_g|_g|_d|_d|_ dS)z4Record episodes stats prior to EpisodicLifeEnv, etc.Nr) rrr_current_reward _num_steps _total_steps_episode_rewards_episode_lengths _num_episodes _num_returnedr rrrrs zMonitorEnv.__init__cKst|jjdi|\}}|jdurt|j|_|jdur0|j|j|j|j|j d7_ d|_d|_||fS)Nr:rr) r r5rpsumrrrnrqrVrors)r!r6r1r4rrrr5s   zMonitorEnv.resetcCsN|j|\}}}}}|j|7_|jd7_|jd7_|||||fS)Nr:)r r-rnrorp)r!r0r1rewr2r3r4rrrr-s zMonitorEnv.stepcC|jSr)rqr\rrrget_episode_rewardszMonitorEnv.get_episode_rewardscCrwr)rrr\rrrget_episode_lengthsryzMonitorEnv.get_episode_lengthscCrwr)rpr\rrrget_total_stepsryzMonitorEnv.get_total_stepsccs@t|jt|jD] }|j||j|fVq t|j|_dSr)rUrtrrqrr)r!rkrrrnext_episode_resultsszMonitorEnv.next_episode_resultsr) r&r'r(rr5r-rxrzr{r|rrrrrms   rmc@rb) NoopResetEnvcCs:tj||||_d|_d|_|jddksJdS)zsSample initial states by taking random number of no-ops on reset. No-op is assumed to be action 0. NrNOOP)rrrnoop_maxoverride_num_noops noop_actionr.r=)r!r rrrrrs zNoopResetEnv.__init__cKs|jjdi||jdur|j}n"z |jjd|jd}Wnty3|jjd|jd}Ynw|dks:Jd}t |D]}|j |j \}}}}}|sR|r]|jjdi|\}}q@||fS)z7Do no-op action for a number of steps in [1, noop_max].Nr:rr) r r5rr. np_randomintegersrAttributeErrorrandintrUr-r)r!r6noopsr1r7r2r3r4rrrr5 s     zNoopResetEnv.resetcCr>rr?r@rrrr-rBzNoopResetEnv.stepN)r~rCrrrrr}s  r}cs$eZdZfddZddZZS)NormalizedImageEnvcs2tj|i|tjjdd|jjtjd|_dS)Ng?)r rL) superrrrrOrr r#float32)r!argsr6 __class__rrr%s zNormalizedImageEnv.__init__cCs|tjddS)Ng`@r)astyper#rr`rrrra0szNormalizedImageEnv.observation)r&r'r(rra __classcell__rrrrr#s  rc@s$eZdZddefddZddZdS) GrayScaleAndResizeT grayscalecCsLtj||||_||_||_tjdd|j|j|rdndftj d|_ dS)z.Warp frames to the specified size (dim x dim).rr^r:r<rIN) rObservationWrapperrwidthheightrrrOr#uint8r)r!r dimrrrrr6s zGrayScaleAndResize.__init__cCsH|jrt|}t||j|jd}|dddddfSt||j|jdS)N)rr)rrrrr)r!framerrrraCs zGrayScaleAndResize.observationN)T)r&r'r(boolrrarrrrr4s r@rdTr frameskip framestackrcCstjj|dd}t|||d}t|}|dkr$|jdusJt||d}t|dd}t|}d |j vr9t |}|rAt ||d }|S) aeWraps `env` for new-API-stack-friendly RLlib Atari experiments. Note that we assume reward clipping is done outside the wrapper. Args: env: The env object to wrap. dim: Dimension to resize observations to (dim x dim). frameskip: Whether to skip n frames and max over them (keep brightest pixels). framestack: Whether to stack the last n (grayscaled) frames. Note that this step happens after(!) a possible frameskip step, meaning that if frameskip=4 and framestack=2, we would perform the following over this trajectory: actual env timesteps: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 -> ... frameskip: ( max ) ( max ) ( max ) ( max ) framestack: ( stack ) (stack ) Returns: The wrapped gym.Env. i)max_episode_steps)rrr:Nrhr~rr;)rM) rwrappers TimeLimit WarpFramerspecrcr}r*r.r=r9rD)r rrrrrrrwrap_atari_for_new_api_stackOs   rTFcCspt|}t|dd}|jdur|durt|dd}t|}d|jvr(t|}t||}|dur6t |d}|S)a Configure environment for DeepMind-style Atari. Note that we assume reward clipping is done outside the wrapper. Args: env: The env object to wrap. dim: Dimension to resize observations to (dim x dim). framestack: Whether to framestack observations. r~rNTrdrr;) rmr}rrcr*r.r=r9rrD)r rr noframeskiprrr wrap_deepminds    r)rrdNT)rTF)% collectionsrtypingrr gymnasiumrnumpyr#rray.rllib.utils.annotationsrray.rllib.utils.imagesrrEnvrrrrrrrr*r9rDrr]rcrmr}rrrintrrrrrrsh      %!2$ 5