o ̳iD@sFddlmZmZddlZddlmZddlmZGdddejZ dS))OptionalTupleN)rlhfcseZdZdZ   ddededeffdd Z dd ejd ejd ejd ejdejdejdeejdeejde ejejejejejffddZ Z S)PPOLossa Proximal Policy Optimization (PPO) Loss module. This implementation uses the following references: https://arxiv.org/abs/1707.06347 eqn. 7 https://github.com/vwxyzjn/lm-human-preference-details/blob/ccc19538e817e98a60d3253242ac15e2a562cb49/lm_human_preference_details/train_policy_accelerate.py#L719 https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/ppo2/model.py#L68-L75 Args: epsilon (float): clipping range for PPO update. value_clip_range (float): clipping range for value function update. value_coeff (float): coefficient for the value function loss contribution. 皙?皙?epsilonvalue_clip_range value_coeffcs t||_||_||_dS)N)super__init__rr r )selfrr r  __class__K/home/ubuntu/.local/lib/python3.10/site-packages/torchtune/rlhf/loss/ppo.pyr s  zPPOLoss.__init__Npi_old_logprobs pi_logprobs advantagesphi_old_values phi_valuesreturns padding_masksvalue_padding_masksreturnc Cst||} t| d|jd|j} | | } | | } | | k|j} |dur.| nt| |} t | | }|durB|nt||}t|||j ||j }t ||d||d}|durmd|ndt||}|||j }|| | |  | fS)a Forward pass of the PPO loss module. Args: pi_old_logprobs (torch.Tensor): Log probabilities of the old policy. pi_logprobs (torch.Tensor): Log probabilities of the current policy. advantages (torch.Tensor): Advantage values. phi_old_values (torch.Tensor): Value predictions of the old value function. phi_values (torch.Tensor): Value predictions of the current value function. returns (torch.Tensor): Return values. padding_masks (Optional[torch.Tensor]): Padding token masks of the same shape as ``pi_logprobs``, where True indicates the corresponding loss values should participage in policy loss calculation. value_padding_masks (Optional[torch.Tensor]): Padding token masks of the same shape as ``pi_logprobs``, where True indicates the corresponding loss values should participage in value loss calculation. Returns: Tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor]: A tuple of five tensors: - loss: The total PPO loss. - policy_loss: The policy function loss. - value_loss: The value function loss. - ratios: The ratio between the current and old policy probabilities. - clipfrac: The fraction of ratios that were clipped. g?Ng?) torchexpclamprtodtypemeanr masked_meanmaximumr r detach)r rrrrrrrrratiosclipped_ratiospolicy_losses_clippedpolicy_losses_unclippedclipfrac policy_lossvalues_clipped value_losslossrrrforward+sF$       zPPOLoss.forward)rrr)NN) __name__ __module__ __qualname____doc__floatr rTensorrrr. __classcell__rrrrrsB  r) typingrrrtorch.nnnn torchtunerModulerrrrrs