o Ti?"@sPdZddlZddlmZedZddlmZddlmZGdd d ej j Z dS) z[ Copyright NVIDIA/apex This file is adapted from fused adam in NVIDIA/apex, commit 6bd01c4 N)MultiTensorApplyi)get_accelerator)FusedAdamBuildercsHeZdZdZ        dfdd Zfd d Zdd dZZS) FusedAdama@ Implements Adam algorithm. Currently GPU-only. Requires Apex to be installed via ``pip install -v --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" ./``. This version of fused Adam implements 2 fusions. * Fusion of the Adam update's elementwise operations * A multi-tensor apply launch that batches the elementwise updates applied to all the model's parameters into one or a few kernel launches. :class:`apex.optimizers.FusedAdam` may be used as a drop-in replacement for ``torch.optim.AdamW``, or ``torch.optim.Adam`` with ``adam_w_mode=False``:: opt = apex.optimizers.FusedAdam(model.parameters(), lr = ....) ... opt.step() :class:`apex.optimizers.FusedAdam` may be used with or without Amp. If you wish to use :class:`FusedAdam` with Amp, you may choose any ``opt_level``:: opt = apex.optimizers.FusedAdam(model.parameters(), lr = ....) model, opt = amp.initialize(model, opt, opt_level="O0" or "O1 or "O2") ... opt.step() In general, ``opt_level="O1"`` is recommended. .. warning:: A previous version of :class:`FusedAdam` allowed a number of additional arguments to ``step``. These additional arguments are now deprecated and unnecessary. Adam was been proposed in `Adam: A Method for Stochastic Optimization`_. Arguments: params (iterable): iterable of parameters to optimize or dicts defining parameter groups. lr (float, optional): learning rate. (default: 1e-3) betas (Tuple[float, float], optional): coefficients used for computing running averages of gradient and its square. (default: (0.9, 0.999)) eps (float, optional): term added to the denominator to improve numerical stability. (default: 1e-8) weight_decay (float, optional): weight decay (L2 penalty) (default: 0) amsgrad (boolean, optional): whether to use the AMSGrad variant of this algorithm from the paper `On the Convergence of Adam and Beyond`_ (default: False) NOT SUPPORTED in FusedAdam! adam_w_mode (boolean, optional): Apply L2 regularization or weight decay True for decoupled weight decay(also known as AdamW) (default: True) set_grad_none (bool, optional): whether set grad to None when zero_grad() method is called. (default: True) .. _Adam - A Method for Stochastic Optimization: https://arxiv.org/abs/1412.6980 .. _On the Convergence of Adam and Beyond: https://openreview.net/forum?id=ryQu7f-RZ MbP?Tg?g+?:0yE>Fc sj|rtdt|||||d} tt||| |rdnd|_| |_t} t dg|_ | j |_ dS)Nz/FusedAdam does not support the AMSGrad variant.)lrbias_correctionbetaseps weight_decayrr) RuntimeErrordictsuperr__init__ adam_w_mode set_grad_nonerloadr IntTensor_dummy_overflow_bufmulti_tensor_adam) selfparamsr r r rrramsgradrdefaultsfused_adam_cuda __class__Q/home/ubuntu/.local/lib/python3.10/site-packages/deepspeed/ops/adam/fused_adam.pyrLs   zFusedAdam.__init__cs<|jr|jD] }|dD]}d|_q qdStt|dS)Nr)r param_groupsgradrr zero_grad)rgroupprr!r"r%cs  zFusedAdam.zero_gradNcCstdd||||fDrtdd}|dur|}|jD]c}t|ddkr)q|dr/dnd} |d \} } d |vr?d|d <ggggf\} } }}ggggf\}}}}ggggf\}}}}|dD]}|jduriqa|jjjrrtd |j|}t|dkr|d d|d <t |j|d <t |j|d <|j t j kr| |jj| |j| |d | |d qa|j t jkr| |j| || |d | |d qa|j t jkr| |jj| |j| |d | |d qatdt| dkr)|d d7<t|j|j| | ||g|d| | |d|d |j| |d t|dkrU|d d7<t|j|j||||g|d| | |d|d |j| |d t|dkr|d d7<t|j|j||||g|d| | |d|d |j| |d q|S)a+Performs a single optimization step. Arguments: closure (callable, optional): A closure that reevaluates the model and returns the loss. The remaining arguments are deprecated, and are only retained (for the moment) for error-checking purposes. css|]}|duVqdS)Nr!).0r'r!r!r" tsz!FusedAdam.step..zuFusedAdam has been updated. Simply initialize it identically to torch.optim.Adam, and call step() with no arguments.Nrrr rr stepzOFusedAdam does not support sparse gradients, please consider SparseAdam insteadexp_avg exp_avg_sqz+FusedAdam only support fp16, bf16 and fp32.r rr)anyrr#lenr$data is_sparsestategettorch zeros_likedtypefloat16appendbfloat16float32multi_tensor_applierrrr)rclosuregrads output_paramsscale grad_norms grad_scalerlossr&r beta1beta2g_16p_16m_16v_16g_bfp_bfm_bfv_bfg_32p_32m_32v_32r'r1r!r!r"r*ks               zFusedAdam.step)rTrr Tr FT)NNNNNN)__name__ __module__ __qualname____doc__rr%r* __classcell__r!r!rr"rs; r) rSr3multi_tensor_applyrr:deepspeed.acceleratorrdeepspeed.ops.op_builderroptim Optimizerrr!r!r!r"s