o Ti`@sPddlZddlmZddlmZddlmZddlmZGdddej j Z dS)N) get_cpu_info)logger) should_log_le)CPUAdamBuildercsXeZdZdZ        dfdd Zd d Zfd d ZedddZ Z S)DeepSpeedCPUAdamrMbP?Tg?g+?:0yE>Fc  st||||||d} tt||| t} d| vr | dnd|_d|jvrKt|jD]\} } t| dD]\}}|j t j krHt dnq7q-tj|_tjdt_||_| |_t|_|j|j||d|d|||td d S) af Fast vectorized implementation of two variations of Adam optimizer on CPU: * Adam: A Method for Stochastic Optimization: (https://arxiv.org/abs/1412.6980); * AdamW: Fixing Weight Decay Regularization in Adam (https://arxiv.org/abs/1711.05101) DeepSpeed CPU Adam(W) provides between 5x to 7x speedup over torch.optim.adam(W). In order to apply this optimizer, the model requires to have its master parameter (in FP32) reside on the CPU memory. To train on a heterogeneous system, such as coordinating CPU and GPU, DeepSpeed offers the ZeRO-Offload technology which efficiently offloads the optimizer states into CPU memory, with minimal impact on training throughput. DeepSpeedCPUAdam plays an important role to minimize the overhead of the optimizer's latency on CPU. Please refer to ZeRO-Offload tutorial (https://www.deepspeed.ai/tutorials/zero-offload/) for more information on how to enable this technology. For calling step function, there are two options available: (1) update optimizer's states and (2) update optimizer's states and copy the parameters back to GPU at the same time. We have seen that the second option can bring 30% higher throughput than the doing the copy separately using option one. .. note:: We recommend using our `config `_ to allow :meth:`deepspeed.initialize` to build this optimizer for you. Arguments: model_params (iterable): iterable of parameters to optimize or dicts defining parameter groups. lr (float, optional): learning rate. (default: 1e-3) betas (Tuple[float, float], optional): coefficients used for computing running averages of gradient and its square. (default: (0.9, 0.999)) eps (float, optional): term added to the denominator to improve numerical stability. (default: 1e-8) weight_decay (float, optional): weight decay (L2 penalty) (default: 0) amsgrad (boolean, optional): whether to use the AMSGrad variant of this algorithm from the paper `On the Convergence of Adam and Beyond`_ (default: False) NOT SUPPORTED in DeepSpeed CPUAdam! adamw_mode: select between Adam and AdamW implementations (default: AdamW) fp32_optimizer_states: creates momentum and variance in full precision regardless of the precision of the parameters (default: True) )lrbetaseps weight_decaybias_correctionamsgrad vendor_id_rawunknownamdparamsz0FP16 params for CPUAdam may not work on AMD CPUsrinfoN)dictsuperr__init__rlower cpu_vendor enumerate param_groupsdtypetorchhalfrwarning optimizer_idopt_id adam_w_modefp32_optimizer_statesrload ds_opt_adam create_adamr)self model_paramsr rr r r r adamw_moder$ default_argscpu_infogroup_idgroupparam_idp __class__O/home/ubuntu/.local/lib/python3.10/site-packages/deepspeed/ops/adam/cpu_adam.pyrs66     zDeepSpeedCPUAdam.__init__cCs|j|jdSN)r& destroy_adamr")r(r3r3r4__del__cszDeepSpeedCPUAdam.__del__cs,tt|||jD]}|ddq dS)NrF)rr __setstate__r setdefault)r(stater.r1r3r4r8hs zDeepSpeedCPUAdam.__setstate__Nc CsXd}|durt |}Wdn1swYtd}t|jD]\}}t|dD]w\}}|jdur;q1|j|ksIJd|jd|j|}t|dkrwd|d<|jr^tj n|j } tj |j | |d|d <tj |j | |d|d <|dd 7<|d \} } |j |j|d|d | | |d|d|d|j |jj |d |d  q1q'|S)aUpdate the model parameters. .. note:: This method will be called internally by ZeRO-Offload. DeepSpeed users should still use ``engine.step()`` as shown in the `Getting Started `_ guide. Args: closure (callable, optional): closure to compute the loss. Defaults to ``None``. Returns: loss: if ``closure`` is provided. Otherwise ``None``. NcpurzCPUAdam param is on zY and must be 'cpu', make sure you enabled 'offload_optimizer': 'cpu' in your ZeRO config.rstep)rdeviceexp_avg exp_avg_sqrr r r r r)r enable_gradr=rrgradr:lenr$floatr zeros_likedatar& adam_updater") r(closurelossr=r-r.r/r0r: state_dtypebeta1beta2r3r3r4r<ms2        zDeepSpeedCPUAdam.step)rTrr rFTTr5) __name__ __module__ __qualname__r!rr7r8rno_gradr< __classcell__r3r3r1r4r sS r) rcpuinfordeepspeed.utilsrdeepspeed.utils.loggingrdeepspeed.ops.op_builderroptim Optimizerrr3r3r3r4s