o پij@s8dZddlZddlmZddlmZGdddeZdS)a  RMSProp modified to behave like Tensorflow impl Originally cut & paste from PyTorch RMSProp https://github.com/pytorch/pytorch/blob/063946d2b3f3f1e953a2a3b54e0b34f1393de295/torch/optim/rmsprop.py Licensed under BSD-Clause 3 (ish), https://github.com/pytorch/pytorch/blob/master/LICENSE References for added functionality: Cautious Optimizers: https://arxiv.org/abs/2411.16085 Why Gradients Rapidly Increase Near the End of Training: https://arxiv.org/abs/2506.02285 Modifications Copyright 2021 Ross Wightman N) Optimizer)ParamsTcseZdZdZ          dd ed ed ed ed ededededededeffdd ZfddZe dddZ Z S) RMSpropTFaImplements RMSprop algorithm (TensorFlow style epsilon) NOTE: This is a direct cut-and-paste of PyTorch RMSprop with eps applied before sqrt and a few other modifications to closer match Tensorflow for matching hyper-params. Noteworthy changes include: 1. Epsilon applied inside square-root 2. square_avg initialized to ones 3. LR scaling of update accumulated in momentum buffer Proposed by G. Hinton in his `course `_. The centered version first appears in `Generating Sequences With Recurrent Neural Networks `_. Args: params: iterable of parameters to optimize or dicts defining parameter groups lr: learning rate momentum: momentum factor alpha: smoothing (decay) constant eps: term added to the denominator to improve numerical stability centered: if ``True``, compute the centered RMSProp, the gradient is normalized by an estimation of its variance weight_decay: weight decay (L2 penalty) (default: 0) decoupled_decay: decoupled weight decay as per https://arxiv.org/abs/1711.05101 corrected_weight_decay: apply corrected weight decay (lr**2 / max_lr) when decoupled_decay is True lr_in_momentum: learning rate scaling is included in the momentum buffer update as per defaults in Tensorflow caution: apply caution {Gz??绽|=rFTparamslralphaeps weight_decaymomentumcentereddecoupled_decaycorrected_weight_decaylr_in_momentumcautionc sd|ks td|d|kstd|d|ks!td|d|ks,td|d|ks7td|t|||||||| | | d } tt||| dS)Nr zInvalid learning rate: {}zInvalid epsilon value: {}zInvalid momentum value: {}zInvalid weight_decay value: {}zInvalid alpha value: {}) r rr r rrrrrr) ValueErrorformatdictsuperr__init__) selfr r r r rrrrrrrdefaults __class__I/home/ubuntu/.local/lib/python3.10/site-packages/timm/optim/rmsprop_tf.pyr3s. zRMSpropTF.__init__csPtt|||jD]}|dd|dd|dd|ddq dS)NrrrFrr)rr __setstate__ param_groups setdefault)rstategrouprrrr Zs    zRMSpropTF.__setstate__NcCsd}|durt |}Wdn1swY|jD]}|dD]}|jdur0q'|j}|jr:td|j|}t|dkrhd|d<t||d<|ddkr]t ||d<|d rht ||d <|d}d |d }|dd 7<|ddkr|dr|dr|dd|j d} n|d} | d | |dn |j ||dd}|j |d||d|d r|d } | j || |d|j| | dd |d} n | |d} |ddkr4|d} | |ddd} |dr| j|| |dd|dr| | |} | | q'| || |dr)| | |} |j | |d dq'|j|| |d dq'q |S)zPerforms a single optimization step. Arguments: closure (callable, optional): A closure that reevaluates the model and returns the loss. Nr z)RMSprop does not support sparse gradientsrstep square_avgrmomentum_bufferrgrad_avgg?r rrrrr )r )valuer cSs2||dk|j}||jdd||S)NrgMbP?)min)todtypediv_meanclamp_)_m_gmaskrrr_apply_cautionsz&RMSpropTF.step.._apply_cautionrr)torch enable_gradr!grad is_sparse RuntimeErrorr#len ones_like zeros_likermul_addadd_powaddcmulsqrt_addcdiv_)rclosurelossr$pr8r#r&one_minus_alphawd_scaler(avgbufr5rrrr%bsh               EzRMSpropTF.step) rrrrr FFFTF)N) __name__ __module__ __qualname____doc__rfloatboolrr r6no_gradr% __classcell__rrrrrsL!     'r)rOr6 torch.optimr_typesrrrrrrs