o Ti@s8dZddlZddlZddlmZGdddejjZdS)zt Copyright NVIDIA/apex This file is adapted from NVIDIA/apex/optimizer/fused_adam and implements the LAMB optimizer N)FusedLambBuildercsHeZdZdZ         dfd d ZdddZddZZS) FusedLambaImplements the LAMB algorithm. Currently GPU-only. LAMB was proposed in `Large Batch Optimization for Deep Learning: Training BERT in 76 minutes. https://arxiv.org/abs/1904.00962 Arguments: params (iterable): iterable of parameters to optimize or dicts defining parameter groups. lr (float, optional): learning rate. (default: 1e-3) bias_correction (bool, optional): bias correction (default: True) betas (Tuple[float, float], optional): coefficients used for computing running averages of gradient and its square. (default: (0.9, 0.999)) eps (float, optional): term added to the denominator to improve numerical stability. (default: 1e-8) eps_inside_sqrt (boolean, optional): in the 'update parameters' step, adds eps to the bias-corrected second moment estimate before evaluating square root instead of adding it to the square root of second moment estimate as in the original paper. (default: False) weight_decay (float, optional): weight decay (L2 penalty) (default: 0) max_grad_norm (float, optional): value used to clip global grad norm (default: 0.0) max_coeff(float, optional): maximum value of the lamb coefficient (default: 10.0) min_coeff(float, optional): minimum value of the lamb coefficient (default: 0.01) amsgrad (boolean, optional): NOT SUPPORTED in FusedLamb! MbP?Tg?g+?:0yE>F$@{Gz?c sZt|_| r tdt||||||| | d} tt||| |r%dnd|_g|_ dS)Nz/FusedLamb does not support the AMSGrad variant.)lrbias_correctionbetaseps weight_decay max_grad_norm max_coeff min_coeffr) rloadfused_lamb_cuda RuntimeErrordictsuperr__init__eps_mode lamb_coeffs) selfparamsr r r r eps_inside_sqrtrrrramsgraddefaults __class__Q/home/ubuntu/.local/lib/python3.10/site-packages/deepspeed/ops/lamb/fused_lamb.pyr)s  zFusedLamb.__init__N?cCsd}|dur |}|durdgt|j}nt|tjr |g}nt|dtkr,|g}n|}|dur;dgt|j}nt|tjrE|g}nt|dtkrQ|g}n|}|dur_dgt|j}|jdd=t|j|||D]\} } } } | durdgt| d} | durdgt| d} | durdgt| d} nt| ts| g} | drdnd} t| d| | | D]\}}}}|}| ddkr||d| d}|dkr||}|j dur|durq|dur|j j }|j rt d|j |}t|dkr d|d <t|j |d <t|j |d <|d |d }}| d \}}| d }| d}|d d7<|dur9tjgtjdn|}|j|j ||||| d||||| d||d |j| | d}|j|qqm|S)aPerforms a single optimization step. Arguments: closure (callable, optional): A closure that reevaluates the model and returns the loss. grads (list of tensors, optional): weight gradient to use for the optimizer update. If gradients have type torch.half, parameters are expected to be in type torch.float. (default: None) output params (list of tensors, optional): A reduced precision copy of the updated weights written out in addition to the regular updated weights. Have to be of same type as gradients. (default: None) scale (float, optional): factor to divide gradient tensor values by before applying to weights. (default: 1) Nrrr rrgư>z+FusedLamb does not support sparse gradientsstepexp_avg exp_avg_sqr rr)dtyper r r)len param_groups isinstancetypes GeneratorTypetypelistrzipgraddata is_sparserstatetorch zeros_liketensorfloatrlambrappend)rclosuregrads output_paramsscale grad_normsloss grads_groupoutput_params_groupgroupgrads_this_groupoutput_params_this_groupgrad_norm_groupr pr1 output_param grad_normcombined_scaleclipr4r&r'beta1beta2rrout_p lamb_coeffr"r"r#r%Es          *zFusedLamb.stepcCsdd|jD}|S)NcSsg|]}|qSr")item).0rOr"r"r# sz-FusedLamb.get_lamb_coeffs..)r)rrr"r"r#get_lamb_coeffsszFusedLamb.get_lamb_coeffs) rTrrFrrrr F)NNNr$N)__name__ __module__ __qualname____doc__rr%rS __classcell__r"r"r r#rs gr)rWr,r5deepspeed.ops.op_builderroptim Optimizerrr"r"r"r#s