o ziN@sHddlZddlmZddlmZmZmZmZmZdZ GdddeZ dS)N) Optimizer)Betas2Nus2OptFloatOptLossClosureParams)QHAdamcs`eZdZdZ      dded ed ed ed ed edeffdd Z dde de fddZ Z S)r aImplements the QHAdam optimization algorithm. It has been proposed in `Adaptive methods for Nonconvex Optimization`__. Arguments: params: iterable of parameters to optimize or dicts defining parameter groups lr: learning rate (default: 1e-3) betas: coefficients used for computing running averages of gradient and its square (default: (0.9, 0.999)) nus: immediate discount factors used to estimate the gradient and its square (default: (1.0, 1.0)) eps: term added to the denominator to improve numerical stability (default: 1e-8) weight_decay: weight decay (L2 penalty) (default: 0) decouple_weight_decay: whether to decouple the weight decay from the gradient-based optimization step (default: False) Example: >>> import torch_optimizer as optim >>> optimizer = optim.QHAdam(model.parameters(), lr=0.1) >>> optimizer.zero_grad() >>> loss_fn(model(input), target).backward() >>> optimizer.step() __ https://arxiv.org/abs/1810.06801 Note: Reference code: https://github.com/facebookresearch/qhoptim MbP?g?g+??r F:0yE>paramslrbetasnus weight_decaydecouple_weight_decayepsc s|dkr td||dkrtd|d|dkr"dks,ntd|dd|dkr8dksBntd|d|dkrMtd |||||||d }tt|||dS) NrzInvalid learning rate: {}zInvalid epsilon value: {}rr z%Invalid beta parameter at index 0: {}rz%Invalid beta parameter at index 1: {}zInvalid weight_decay value: {})rrrrrr) ValueErrorformatsuperr __init__) selfrrrrrrrdefaults __class__J/home/ubuntu/.local/lib/python3.10/site-packages/torch_optimizer/qhadam.pyr)s0   zQHAdam.__init__NclosurereturncCsd}|dur |}|jD]}|d}|d\}}|d\}}|d} |d} |d} |dD]} | jdur6q.| jj} | jrAtd |j| }| d kr_| rW| jd || n| j| j| d | | }t |d krd |d<d |d<t | j|d<t | j|d<d||d|d<d||d|d<|d}|d}|d}|d}dd|}dd|}||j| d|d ||j|d|d ||}|dkr|j| d|d ||}|dkr|j|d|d | | d kr|| | jj ||| dq.q |S)zPerforms a single optimization step. Arguments: closure: A closure that reevaluates the model and returns the loss. NrrrrrrrzLQHAdam does not support sparse gradients, please consider SparseAdam insteadrr)alphar beta1_weight beta2_weightexp_avg exp_avg_sqr )value) param_groupsgraddata is_sparse RuntimeErrorstatemul_add_mullentorch zeros_likesqrt_addcdiv_)rr!lossgrouprbeta1beta2nu1nu2rrrpd_pr.d_p_sqr$r%r&r' beta1_adj beta2_adjavg_grad avg_grad_rmsrrr stepNsf             5z QHAdam.step)r r r rFr)N)__name__ __module__ __qualname____doc__rfloatrrboolrrrrD __classcell__rrrr r s0"%r ) r3torch.optim.optimizerrtypesrrrrr__all__r rrrr s