o پiY@s4dZddlZddlmZddlZGdddeZdS)a6 Nvidia NovoGrad Optimizer. Original impl by Nvidia from Jasper example: - https://github.com/NVIDIA/DeepLearningExamples/blob/master/PyTorch/SpeechRecognition/Jasper Paper: `Stochastic Gradient Methods with Layer-wise Adaptive Moments for Training of Deep Networks` - https://arxiv.org/abs/1905.11286 N) OptimizercsLeZdZdZ      dfdd Zfd d Zedd d ZZ S) NvNovoGrada( Implements Novograd algorithm. Args: params (iterable): iterable of parameters to optimize or dicts defining parameter groups lr (float, optional): learning rate (default: 1e-3) betas (Tuple[float, float], optional): coefficients used for computing running averages of gradient and its square (default: (0.95, 0.98)) eps (float, optional): term added to the denominator to improve numerical stability (default: 1e-8) weight_decay (float, optional): weight decay (L2 penalty) (default: 0) grad_averaging: gradient averaging amsgrad (boolean, optional): whether to use the AMSGrad variant of this algorithm from the paper `On the Convergence of Adam and Beyond`_ (default: False) MbP?gffffff?g\(\?:0yE>rFc sd|ks td|d|kstd|d|dkr"dks,ntd|dd|dkr8dksBntd|dt||||||d }tt|||dS) NgzInvalid learning rate: {}zInvalid epsilon value: {}rg?z%Invalid beta parameter at index 0: {}z%Invalid beta parameter at index 1: {})lrbetaseps weight_decaygrad_averagingamsgrad) ValueErrorformatdictsuperr__init__) selfparamsrr r r r r defaults __class__I/home/ubuntu/.local/lib/python3.10/site-packages/timm/optim/nvnovograd.pyr s"  zNvNovoGrad.__init__cs,tt|||jD]}|ddq dS)Nr F)rr __setstate__ param_groups setdefault)rstategrouprrrr=s zNvNovoGrad.__setstate__NcCsd}|durt |}Wdn1swY|jD]}|dD]}|jdur.q&|j}|jr8td|d}|j|}t|dkrnd|d<t||d<t g |dj |d<|rnt g |dj |d <|d|d}} |r}|d } |d \} } |dd 7<t t |d } | dkr| | n | | j| d | d |rtj| | | d| |d}n | |d}|||ddkr|j||dd |dr|d | || ||j||d d q&q |S)zPerforms a single optimization step. Arguments: closure (callable, optional): A closure that reevaluates the model and returns the loss. Nrz#Sparse gradients are not supported.r rstepexp_avg exp_avg_sqmax_exp_avg_sqr r)alpha)outr r r r)torch enable_gradrgrad is_sparse RuntimeErrorrlen zeros_likezerostodevicesumpowcopy_mul_add_maxsqrtdiv_)rclosurelossrpr(r rr r!r"beta1beta2normdenomrrrrBsT          4zNvNovoGrad.step)rrrrFF)N) __name__ __module__ __qualname____doc__rrr&no_gradr __classcell__rrrrr s r)rBr&torch.optim.optimizerrmathrrrrrs