o oi'@sddlmZmZddlZddlmZddlmZmZm Z m Z dZ ee e e ge fZde de d e d e fd d Zde de d e d e fd dZde de d e d e fddZGdddeZdS))CallableDictN) Optimizer)OptFloatOptLossClosureParamsState) SGDLRDecaylrtalphareturncCs |||SNr r r rrX/home/ubuntu/SoloSpeech/.venv/lib/python3.10/site-packages/torch_optimizer/sgdlrdecay.py_exps rcCs|d||S)N?rrrrr_1tsrcCs|d||dS)Nrg?rrrrr_sqrtsrcseZdZdZ       ddeded ed ed ed ed ededdffdd Zde ddffdd Z dde de fddZ ZS)r a/Implements stochastic gradient descent with several step size decay schemes. It has been proposed in `Exponential Step Sizes for Non-Convex Optimization`__. Arguments: params: iterable of parameters to optimize or dicts defining parameter groups lr: initial learning rate (default: 1e-3) scheme: the decay scheme, currently only supports 'exp', 't', 'sqrt' alpha: decay factor. momentum: momentum factor (default: 0). weight_decay: weight decay (L2 penalty) (default: 0). dampening: dampening for momentum (default: 0). nesterov: enables Nesterov momentum (default: False). __ https://arxiv.org/abs/2002.05273 Note: Reference code: https://github.com/zhenxun-zhuang/SGD-Exponential-Stepsize # noqa MbP?exp+?Fparamsr schemer momentum dampening weight_decaynesterovrNc s|dkr td||dkrtd||dkr!td||dkr,td||dvr7td|t|||||||d} |rP|d ksL|d krPtd tt||| tttd|_dS) NrzInvalid learning rate: zInvalid alpha value: zInvalid momentum value: zInvalid weight_decay value: )r1tsqrtzInvalid scheme value: )r r rrrrr rz8Nesterov momentum requires a momentum and zero dampening) ValueErrordictsuperr __init__rrr_schemes) selfrr rr rrrr defaults __class__rrr&2s8   zSGDLRDecay.__init__statecs,tt|||jD]}|ddq dS)Nr F)r%r __setstate__ param_groups setdefault)r(r,groupr*rrr-]s zSGDLRDecay.__setstate__closurecCsBd}|dur |}|jD]}|d}|d}|d}|d}|d}|d} |j|d} |d D]l} | jdur9q1|j| } t| d krHd | d <| d d 7<| jj} |d kr_| || j|d krd | vrst|  }| d <n| d }| |d || |r| ||} n|} | | | d |}| j| | q1q |S)zPerforms a single optimization step. Arguments: closure: A closure that reevaluates the model and returns the loss. Nrrrr r r rrrsteprmomentum_buffer) r.r'gradr,lendataadd_torchclonedetachmul_add)r(r1lossr0rrrr r r decay_funcp param_stated_pbuf setep_sizerrrr2bsJ      zSGDLRDecay.step)rrrrrrFr)__name__ __module__ __qualname____doc__rfloatstrboolr&r r-rrr2 __classcell__rrr*rr s<  +r )typingrrr8torch.optim.optimizerrtypesrrrr __all__rHint DecayFuncrrrr rrrrs