o پi<@s$dZddlZGdddejjZdS)z AdaHessian Optimizer Lifted from https://github.com/davda54/ada-hessian/blob/master/ada_hessian.py Originally licensed MIT, Copyright 2020, David Samuel NcspeZdZdZ        dfd d Zed d Zd dZddZe ddZ e dddZ Z S) Adahessiana Implements the AdaHessian algorithm from "ADAHESSIAN: An Adaptive Second OrderOptimizer for Machine Learning" Arguments: params (iterable): iterable of parameters to optimize or dicts defining parameter groups lr (float, optional): learning rate (default: 0.1) betas ((float, float), optional): coefficients used for computing running averages of gradient and the squared hessian trace (default: (0.9, 0.999)) eps (float, optional): term added to the denominator to improve numerical stability (default: 1e-8) weight_decay (float, optional): weight decay (L2 penalty) (default: 0.0) hessian_power (float, optional): exponent of the hessian trace (default: 1.0) update_each (int, optional): compute the hessian trace approximation only after *this* number of steps (to save time) (default: 1) n_samples (int, optional): how many times to sample `z` for the approximation of the hessian trace (default: 1) 皙?g?g+?:0yE>?Fc sd|ks td|d|kstd|d|dkr"dks,ntd|dd|dkr8dksBntd|dd|krLdksTntd |||_||_| |_d |_t|j|_t |||||d } t t | || | D] } d| _d|j| d <qdS) NrzInvalid learning rate: zInvalid epsilon value: rrz#Invalid beta parameter at index 0: rz#Invalid beta parameter at index 1: zInvalid Hessian power value: i)lrbetaseps weight_decay hessian_power hessian step) ValueError n_samples update_eachavg_conv_kernelseedtorch Generator manual_seed generatordictsuperr__init__ get_paramshessstate) selfparamsr r r r r rrrdefaultsp __class__I/home/ubuntu/.local/lib/python3.10/site-packages/timm/optim/adahessian.pyrs6  zAdahessian.__init__cCsdS)NTr$rr$r$r%is_second_orderFszAdahessian.is_second_ordercCsdd|jDS)zH Gets all parameters in all param_groups with gradients css(|]}|dD]}|jr|VqqdS)rN) requires_grad).0groupr!r$r$r% Os&z(Adahessian.get_params..) param_groupsr&r$r$r%rJszAdahessian.get_paramscCs@|D]}t|jts|j|d|jdkr|jqdS)z; Zeros out the accumulated hessian traces. rrN)r isinstancerfloatrrzero_)rr!r$r$r% zero_hessianQs $ zAdahessian.zero_hessianc sg}tddD]}j|djdkr||j|dd7<q t|dkr2dSjj|djkrHt |dj j _dd|D}t j D]3}fd d|D}tjj|||d |j dkd }t|||D]\}}}|j||j 7_quqTdS) z} Computes the Hutchinson approximation of the hessian trace and accumulates it for each trainable parameter. cSs |jduSNgrad)r!r$r$r%as z(Adahessian.set_hessian..rrrNcSsg|]}|jqSr$r2r)r!r$r$r% lsz*Adahessian.set_hessian..c s0g|]}tjdd|j|jdddqS)r)rdeviceg@r)rrandintsizerr8r5r&r$r%r6ps0T) grad_outputs only_inputs retain_graph)filterrrrappendlenrr8rrrrrangerautogradr3zipr) rrr!gradsizsh_zsh_zzr$r&r% set_hessianZs&  zAdahessian.set_hessianNcCsd}|dur |}|||jD]}|dD]}|jdus&|jdur'q|jrD|dkrDt|jj ddgdd |j |_| d|d |d |j |}t|dkrnd |d <t||d <t||d<|d |d}}|d\}} |d d7<| |j|jd|d| | j|j|jd| dd||d } d| |d } |d} || | d|d} |d | }|j|| | dqq|S)z Performs a single optimization step. Arguments: closure (callable, optional) -- a closure that reevaluates the model and returns the loss (default: None) Nrr7T)dimkeepdimrr r rstepexp_avgexp_hessian_diag_sqr )alpha)valuer r )r0rJr,r3rrrMrabsmean expand_asclonemul_rr@ zeros_likeadd_addcmul_pow_addcdiv_)rclosurelossr*r!rrPrQbeta1beta2bias_correction1bias_correction2kdenom step_sizer$r$r%rOvs<  (    &zAdahessian.step)rrrrrrrFr1)__name__ __module__ __qualname____doc__rpropertyr'rr0rno_gradrJrO __classcell__r$r$r"r%r s&,  r)rjroptim Optimizerrr$r$r$r%s