o ci@sddlZddlZddlZddlmZddlmZddlmZddl m Z m Z ddl m Z ddlmZddlmZdd lmZmZe Gd d d ZdS) N)Optional)_TimerSimpleReplayBuffer)DEFAULT_POLICY_IDconcat_samples) OldAPIStack) ReplayMode) _ALL_POLICIES)PolicyIDSampleBatchTypec@sfeZdZdZejfdededefddZde dd fd d Z e fd e de e fd dZdefddZd S)MixInMultiAgentReplayBufferaThis buffer adds replayed samples to a stream of new experiences. - Any newly added batch (`add()`) is immediately returned upon the next `replay` call (close to on-policy) as well as being moved into the buffer. - Additionally, a certain number of old samples is mixed into the returned sample according to a given "replay ratio". - If >1 calls to `add()` are made without any `replay()` calls in between, all newly added batches are returned (plus some older samples according to the "replay ratio"). .. testcode:: from ray.rllib.execution.buffers.mixin_replay_buffer import ( MixInMultiAgentReplayBuffer) from ray.rllib.policy.sample_batch import SampleBatch # replay ratio 0.66 (2/3 replayed, 1/3 new samples): buffer = MixInMultiAgentReplayBuffer(capacity=100, replay_ratio=0.66) A, B, C = (SampleBatch({"obs": [1]}), SampleBatch({"obs": [2]}), SampleBatch({"obs": [3]})) buffer.add(A) buffer.add(B) buffer.add(B) print(buffer.replay()["obs"]) .. testoutput:: :hide: ... capacity replay_ratio replay_modecs|_||_d|_|jdkr|jd|j|_|dtjfvr#tj|_n|dtjfvr/tj|_ntd|fdd}t ||_ t |_ t |_t |_d|_t t|_dS) aInitializes MixInReplay instance. Args: capacity: Number of batches to store in total. replay_ratio: Ratio of replayed samples in the returned batches. E.g. a ratio of 0.0 means only return new samples (no replay), a ratio of 0.5 means always return newest sample plus one old one (1:1), a ratio of 0.66 means always return the newest sample plus 2 old (replayed) ones (1:2), etc... N?lockstep independentzUnsupported replay mode: {}cs tdS)N) num_slotsrrrc/home/ubuntu/.local/lib/python3.10/site-packages/ray/rllib/execution/buffers/mixin_replay_buffer.py new_bufferNs z8MixInMultiAgentReplayBuffer.__init__..new_bufferr)rrreplay_proportionr LOCKSTEPr INDEPENDENT ValueErrorformat collections defaultdictreplay_buffersradd_batch_timer replay_timerupdate_priorities_timer num_addedlistlast_added_batches)selfrrrrrrr__init__1s"     z$MixInMultiAgentReplayBuffer.__init__batchreturnNcCs|}|}|j9|jtjkr#|jt||j t |n|j D]\}}|j|||j | |q(Wdn1sGwY|j |j7_ dS)a2Adds a batch to the appropriate policy's replay buffer. Turns the batch into a MultiAgentBatch of the DEFAULT_POLICY_ID if it is not a MultiAgentBatch. Subsequently adds the individual policy batches to the storage. Args: batch: The batch to be added. N)copyas_multi_agentr!rr rr r add_batchr&appendpolicy_batchesitemsr$count)r'r) policy_id sample_batchrrradd^s   zMixInMultiAgentReplayBuffer.addr2cCs"|jtjkr|tkrtdt|j|}t|dks*t|j|dkr,|j dkr,dS|j X|j|}g|j|<|j dkrJt |WdS|j dkrZ| WdSt|}|j }t||kr||d8}|| t||ksit |WdS1swYdS)NzTrying to sample from single policy's buffer in lockstep mode. In lockstep mode, all policies' experiences are sampled from a single replay buffer which is accessed with the policy id `{}`rrg)rr rr rrr lenr&rr"rreplayrrandomr.)r'r2bufferoutput_batchesnum_newrrrrr7{s4      $z"MixInMultiAgentReplayBuffer.replaycCstS)zReturns the computer's network name. Returns: The computer's networks name or an empty string, if the network name could not be determined. )platformnode)r'rrrget_hostsz$MixInMultiAgentReplayBuffer.get_host)__name__ __module__ __qualname____doc__r rintfloatr(r r4rr rr7strr>rrrrr s$$ - +r )rr<r8typingrray.util.timerrray.rllib.execution.replay_opsrray.rllib.policy.sample_batchrrray.rllib.utils.annotationsr8ray.rllib.utils.replay_buffers.multi_agent_replay_bufferr ,ray.rllib.utils.replay_buffers.replay_bufferr ray.rllib.utils.typingr r r rrrrs