o zi&@s|ddlZddlZddlmZddlZddlmZddlmZm Z m Z dZ GdddeZ Gd d d eZ Gd d d eZdS) N)Optional) Optimizer)OptFloatOptLossClosureParams) A2GradUni A2GradInc A2GradExpc ReZdZdZ   ddedeededeffdd Zdd ed e fd d Z Z S)raImplements A2GradUni Optimizer Algorithm. It has been proposed in `Optimal Adaptive and Accelerated Stochastic Gradient Descent`__. Arguments: params: iterable of parameters to optimize or dicts defining parameter groups lr: not used for this optimizer (default: None) beta: (default: 10) lips: Lipschitz constant (default: 10) Example: >>> import torch_optimizer as optim >>> optimizer = optim.A2GradUni(model.parameters(), lips=10) >>> optimizer.zero_grad() >>> loss_fn(model(input), target).backward() >>> optimizer.step() __ https://arxiv.org/abs/1810.00553 Note: Reference code: https://github.com/severilov/A2Grad_optimizer N paramslrbetalipscsLt|||d}|dkrtd||dkrtd|t||dS)NrrrInvalid beta value: {}Invalid lips value: {})dict ValueErrorformatsuper__init__selfr rrrdefaults __class__J/home/ubuntu/.local/lib/python3.10/site-packages/torch_optimizer/a2grad.pyr(s zA2GradUni.__init__closurereturnc Csd}|dur |}|jD]}|dD]}|jdurq|jj}|j|}t|dkrDd|d<d|d<d|d<t||d<t|j|d <d |d |dd}|d}||d||| |ddt j ||d d } |dt | |  7<t|d} d |dd} d||d| } |d } | j|| d |jd| |jj| | d |jj|d|  |d| d | |d<|dd7<qq |SPerforms a single optimization step. Arguments: closure: A closure that reevaluates the model and returns the loss. Nr rstepralpha_kv_kavg_gradx_kralphar param_groupsgraddatastatelencopydeepcopymul_add_div_torchaddsumitemmathsqrtrr!lossgrouppr1r3gamma_kr(delta_kh_k alpha_k_1coefx_k_1rrr r%:sH      'zA2GradUni.stepNr r N __name__ __module__ __qualname____doc__rrfloatrrrr% __classcell__rrrr r src r )r aImplements A2GradInc Optimizer Algorithm. It has been proposed in `Optimal Adaptive and Accelerated Stochastic Gradient Descent`__. Arguments: params: iterable of parameters to optimize or dicts defining parameter groups lr: not used for this optimizer (default: None) beta: (default: 10) lips: Lipschitz constant (default: 10) Example: >>> import torch_optimizer as optim >>> optimizer = optim.A2GradInc(model.parameters(), lips=10) >>> optimizer.zero_grad() >>> loss_fn(model(input), target).backward() >>> optimizer.step() __ https://arxiv.org/abs/1810.00553 Note: Reference code: https://github.com/severilov/A2Grad_optimizer Nr r rrrcsP|dkr td||dkrtd|t|||d}tt|||dS)NrrzInvalid weight_decay value: {}r)rrrrr rrrrr rs zA2GradInc.__init__r!r"c Csd}|dur |}|jD]}|dD]}|jdurq|jj}|j|}t|dkrDd|d<d|d<d|d<t||d<t|j|d <d |d |dd}|d}||d||| |ddt j ||d d } |d|d|ddd 9<|dt | |  7<t|d} d |dd} d||d| } |d } | j|| d |jd| |jj| | d |jj|d|  |d| d | |d<|dd7<qq |Sr#r/r@rrr r%sJ      $(zA2GradInc.steprJrKrLrrrr r osr c sXeZdZdZ    ddedeededed ef fd d Zdd ed e fddZ Z S)r auImplements A2GradExp Optimizer Algorithm. It has been proposed in `Optimal Adaptive and Accelerated Stochastic Gradient Descent`__. Arguments: params: iterable of parameters to optimize or dicts defining parameter groups lr: not used for this optimizer (default: None) beta: (default: 10) lips: Lipschitz constant (default: 10) rho: represents the degree of weighting decrease, a constant smoothing factor between 0 and 1 (default: 0.5) Example: >>> import torch_optimizer as optim >>> optimizer = optim.A2GradExp(model.parameters(), lips=10) >>> optimizer.zero_grad() >>> loss_fn(model(input), target).backward() >>> optimizer.step() __ https://arxiv.org/abs/1810.00553 Note: Reference code: https://github.com/severilov/A2Grad_optimizer Nr ?r rrrrhocspt||||d}tt||||dkrtd||dkr'td||dks/|dkr6td|dS)N)rrrTrrrrg?zInvalid rho value: {})rrr rrr)rr rrrrTrrrr rs zA2GradExp.__init__r!r"c Csd}|dur |}|jD]}|dD]}|jdurq|jj}|j|}t|dkrDd|d<d|d<d|d<t||d<t|j|d <d |d |dd}|d}||d||| |ddt j ||d d } |ddkrt | |  |d<n|d|d9<|dd|dt | |  7<t|d|dg|d<t|dd|d} d |dd} d ||d| } |d } | j|| d |jd| |jj| | d |jj|d| |d| d | |d<|dd7<qq |S)r$Nr rr%rr&r'r(r)r*rr+r,v_kkrTr.r)r0r1r2r3r4r5r6r7r8r9r:r;r<r=maxr>r?r@rrr r%sX       0zA2GradExp.step)Nr r rSrKrLrrrr r s$r )r5r>typingrr:torch.optim.optimizerrtypesrrr__all__rr r rrrr s  b_