o oi@sHddlZddlZddlmZddlmZmZmZdZGdddeZ dS)N) Optimizer)OptFloatOptLossClosureParams)SGDPcseZdZdZ        ddeded ed ed ed ed edededdffdd ZeddZ eddZ eddZ ddZ d de defddZZS)!raImplements SGDP algorithm. It has been proposed in `Slowing Down the Weight Norm Increase in Momentum-based Optimizers`__ Arguments: params: iterable of parameters to optimize or dicts defining parameter groups lr: learning rate (default: 1e-3) momentum: momentum factor (default: 0) dampening: dampening for momentum (default: 0) eps: term added to the denominator to improve numerical stability (default: 1e-8) weight_decay: weight decay (L2 penalty) (default: 0) delta: threhold that determines whether a set of parameters is scale invariant or not (default: 0.1) wd_ratio: relative weight decay applied on scale-invariant parameters compared to that applied on scale-variant parameters (default: 0.1) nesterov: enables Nesterov momentum (default: False) Example: >>> import torch_optimizer as optim >>> optimizer = optim.SGDP(model.parameters(), lr=0.1) >>> optimizer.zero_grad() >>> loss_fn(model(input), target).backward() >>> optimizer.step() __ https://arxiv.org/abs/2006.08217 Note: Reference code: https://github.com/clovaai/AdamP MbP?r:0yE>皙?Fparamslrmomentum dampeningeps weight_decaydeltawd_rationesterovreturnNc s|dkr td||dkrtd||dkr!td||dkr,td||dkr7td||dkrBtd||dkrMtd |t|||||||| d } tt||| dS) NgzInvalid learning rate: {}zInvalid epsilon value: {}zInvalid momentum value: {}zInvalid dampening value: {}rzInvalid weight_decay value: {}zInvalid delta value: {}zInvalid wd_ratio value: {})r r rrrrrr) ValueErrorformatdictsuperr__init__) selfr r r rrrrrrdefaults __class__R/home/ubuntu/SoloSpeech/.venv/lib/python3.10/site-packages/torch_optimizer/sgdp.pyr.s6  z SGDP.__init__cCs||ddS)Nr)viewsizexrrr _channel_viewWszSGDP._channel_viewcCs |ddS)Nrr )r!r#rrr _layer_view[s zSGDP._layer_viewcCsT||}||}|jdd|}|jdd|}||jdd}|||S)Nrdim)normadd_sumabs)r$yr view_funcx_normy_normdotrrr_cosine_similarity_s zSGDP._cosine_similarityc Csd}dgdgt|jd}|j|jfD]G} |||j|| } | |t| |j dkr\|j| |jj dd | |} || | | |j dd |8}|}||fSq||fS)Nrr r')lenshaper%r&r2datamaxmathsqrtr"r)r!r*r+) rpgradperturbrrrwd expand_sizer. cosine_simp_nrrr _projectionjs""  zSGDP._projectionclosurec Cs<d}|dur |}|jD]}|d}|d}|d}|d}|dD]x}|jdur*q"|jj} |j|} t| dkrAt|j| d<| d} | |j| d|d |rZ| || } n| } d} t|j dkrw| || | |d |d |d \} } |dkr|jd|d |d| d||jj| |d  d q"q |S)zPerforms a single optimization step. Arguments: closure: A closure that reevaluates the model and returns the loss. Nrr rrr rr)alpharrrr ) param_groupsr:r5stater3torch zeros_likemul_r*r4r@)rrAlossgrouprr rrr9r:rDbufd_prrrrstep~sZ       ,z SGDP.step)rrrr rr r F)N)__name__ __module__ __qualname____doc__rfloatboolr staticmethodr%r&r2r@rrrL __classcell__rrrrr sN%   )    r) r7rEtorch.optim.optimizerrtypesrrr__all__rrrrrs