o }oi$ @svddlmZddlmZddlmZmZmZmZddl m Z m Z gdZ eGdddZ eGdd d e ZeGd d d e ZeGd d d e ZeGddde ZeGddde ZeGddde ZeGddde ZeGddde ZeGddde ZeGddde Zdede fddZded eeeefd!e fd"d#Ze eeeeeeeeeed$ Zd%S)&) dataclass)partial)AnyDictOptionalTuple)MISSING OmegaConf) OptimizerParams AdamParamsNovogradParams SGDParamsAdadeltaParams AdamaxParams AdagradParams AdamWParams RMSpropParams RpropParamsc@s"eZdZUdZeZeeed<dS)r zw Base Optimizer params with no values. User can chose it to explicitly override via command line arguments lrN) __name__ __module__ __qualname____doc__rrrfloat__annotations__rrO/home/ubuntu/.local/lib/python3.10/site-packages/nemo/core/config/optimizers.pyr #s r c@sBeZdZUdZdZeed<dZeed<dZeed<dZ e ed<dS) r aN Default configuration for Adam optimizer. It is not derived from Config as it is not a NeMo object (and in particular it doesn't need a name). ..note: For the details on the function/meanings of the arguments, please refer to: https://pytorch.org/docs/stable/optim.html?highlight=sgd#torch.optim.SGD rmomentum dampening weight_decayFnesterovN) rrrrrrrrrr boolrrrrr -   r c@s6eZdZUdZdZeed<dZeed<dZe ed<dS) r aP Default configuration for Adam optimizer. It is not derived from Config as it is not a NeMo object (and in particular it doesn't need a name). ..note: For the details on the function/meanings of the arguments, please refer to: https://pytorch.org/docs/stable/optim.html?highlight=adam#torch.optim.Adam :0yE>epsrrFamsgradN) rrrrr$rrrr%r!rrrrr >s  r c@sJeZdZUdZdZeeefed<dZeed<dZ eed<dZ e ed <d S) raC Default configuration for AdamW optimizer. It is not derived from Config as it is not a NeMo object (and in particular it doesn't need a name). ..note: For the details on the function/meanings of the arguments, please refer to: https://pytorch.org/docs/stable/optim.html#torch.optim.AdamW ?g+?betasr#r$rrFr%N) rrrrr(rrrr$rr%r!rrrrrOs   rc@s6eZdZUdZdZeed<dZeed<dZeed<dS) raI Default configuration for Adadelta optimizer. It is not derived from Config as it is not a NeMo object (and in particular it doesn't need a name). ..note: For the details on the function/meanings of the arguments, please refer to: https://pytorch.org/docs/stable/optim.html#torch.optim.Adadelta r'rhoư>r$rrN) rrrrr)rrr$rrrrrr`s  rc@s>eZdZUdZdZeeefed<dZeed<dZ eed<dS) raE Default configuration for Adamax optimizer. It is not derived from Config as it is not a NeMo object (and in particular it doesn't need a name). ..note: For the details on the function/meanings of the arguments, please refer to: https://pytorch.org/docs/stable/optim.html#torch.optim.Adamax r&r(r#r$rrN) rrrrr(rrrr$rrrrrrps  rc@sBeZdZUdZdZeed<dZeed<dZeed<dZ eed<dS) raG Default configuration for Adagrad optimizer. It is not derived from Config as it is not a NeMo object (and in particular it doesn't need a name). ..note: For the details on the function/meanings of the arguments, please refer to: https://pytorch.org/docs/stable/optim.html#torch.optim.Adagrad rlr_decayrinitial_accumulator_valueg|=r$N) rrrrr+rrrr,r$rrrrrr"rc@sNeZdZUdZdZeed<dZeed<dZeed<dZ eed<d Z e ed <d S) raG Default configuration for RMSprop optimizer. It is not derived from Config as it is not a NeMo object (and in particular it doesn't need a name). ..note: For the details on the function/meanings of the arguments, please refer to: https://pytorch.org/docs/stable/optim.html#torch.optim.RMSprop gGz?alphar#r$rrrFcenteredN) rrrrr-rrr$rrr.r!rrrrrs    rc@s:eZdZUdZdZeeefed<dZeeefed<dS)raI Default configuration for RpropParams optimizer. It is not derived from Config as it is not a NeMo object (and in particular it doesn't need a name). ..note: For the details on the function/meanings of the arguments, please refer to: https://pytorch.org/docs/stable/optim.html#torch.optim.Rprop )g?g333333?etas)r*2 step_sizesN) rrrrr/rrrr1rrrrrs  rc@szeZdZUdZdZeeefed<dZeed<dZ eed<dZ e ed <dZ e ed <dZ e ed <d Zeed <dZeed<dS)r a" Configuration of the Novograd optimizer. It has been proposed in "Stochastic Gradient Methods with Layer-wise Adaptive Moments for Training of Deep Networks" (https://arxiv.org/abs/1905.11286) Args: lr (float, optional): learning rate (default: 1e-3) betas (Tuple[float, float], optional): coefficients used for computing running averages of gradient and its square (default: (0.9, 0.999)) eps (float, optional): term added to the denominator to improve numerical stability (default: 1e-8) weight_decay (float, optional): weight decay (L2 penalty) (default: 0) amsgrad (boolean, optional): whether to use the AMSGrad variant of this algorithm from the paper "On the Convergence of Adam and Beyond" )gffffff?g\(\?r(r#r$rrFgrad_averagingr%lucMbP? luc_trustluc_epsN)rrrrr(rrrr$rr2r!r%r3r5r6rrrrr s       r c@szeZdZUdZdZeed<dZeeefed<dZ eed<dZ eed <d Z eed <d Z e ed <dZe ed<dZe ed<dS)AdafactorParamsa Configuration of the Adafactor optimizer. It has been proposed in "Adafactor: Adaptive Learning Rates with Sublinear Memory Cost" (https://arxiv.org/abs/1804.04235) Args: lr (float, optional): learning rate (default: 1e-3) beta1 (float, optional): coefficients used for computing running averages of gradient and its square (default: None) eps (Tuple [float, float] optional) weight_decay (float, optional): weight decay (L2 penalty) (default: 0) scale_parameter (float, optional): scale parameter (default: False) relative_step (bool, optional): whether to use relative step sizes (default: False) warmup_init (bool, optional): whether to warmup the learning rate linearly (default: False) Nbeta1)gKH9r4r$g?clip_thresholdg? decay_raterrTscale_parameterF relative_step warmup_init)rrrrr8rrr$rr9r:rr;r!r<r=rrrrr7s       r7nameoptimizer_paramscCs"|tvr td||t|<dS)aF Checks if the optimizer param name exists in the registry, and if it doesnt, adds it. This allows custom optimizer params to be added and called by name during instantiation. Args: name: Name of the optimizer. Will be used as key to retrieve the optimizer. optimizer_params: Optimizer class zFCannot override pre-existing optimizers. Conflicting optimizer name = N)AVAILABLE_OPTIMIZER_PARAMS ValueError)r>r?rrrregister_optimizer_paramss  rBkwargsreturncKst|dur|S|tvrtd|dtt|}|dur0t|dkr0t|}t||t|fi|}|S)aC Convenience method to obtain a OptimizerParams class and partially instantiate it with optimizer kwargs. Args: name: Name of the OptimizerParams in the registry. kwargs: Optional kwargs of the optimizer used during instantiation. Returns: a partially instantiated OptimizerParams Nz%Cannot resolve optimizer parameters 'z('. Available optimizer parameters are : r)r@rAkeyslenr createmerger)r>rCscheduler_paramsrrrget_optimizer_configs  rJ) optim_params adam_paramsnovograd_params sgd_paramsadadelta_params adamax_paramsadagrad_params adamw_paramsrmsprop_params rprop_paramsadafactor_paramsN) dataclassesr functoolsrtypingrrrr omegaconfrr __all__r r r rrrrrrr r7strrBrJr@rrrrsR   "