o Ti@sPdZddlZddlmZedZddlmZddlmZGdd d ej j Z dS) z* This file is modified from fused_adam.py N)MultiTensorApplyi)get_accelerator)FusedLionBuildercs8eZdZdZd fdd Zfdd Zdd d ZZS) FusedLionaImplements Lion algorithm. Currently GPU-only. Arguments: params (iterable): iterable of parameters to optimize or dicts defining parameter groups. lr (float, optional): learning rate. (default: 1e-3) betas (Tuple[float, float], optional): coefficients used for computing running averages of gradient and its square. (default: (0.9, 0.999)) weight_decay (float, optional): weight decay (L2 penalty) (default: 0) set_grad_none (bool, optional): whether set grad to None when zero_grad() method is called. (default: True) .. _Symbolic Discovery of Optimization Algorithms: https://doi.org/10.48550/arXiv.2302.06675 MbP?g?g+?TcsLt|||d}tt|||||_t}tdg|_ |j |_ dS)N)lrbetas weight_decayr) dictsuperr__init__ set_grad_nonerloadr IntTensor_dummy_overflow_bufmulti_tensor_lion)selfparamsr r r rdefaultsfused_lion_cuda __class__Q/home/ubuntu/.local/lib/python3.10/site-packages/deepspeed/ops/lion/fused_lion.pyr$s   zFusedLion.__init__cs<|jr|jD] }|dD]}d|_q qdStt|dS)Nr)r param_groupsgradrr zero_grad)rgroupprrrr/s  zFusedLion.zero_gradNc Cstdd||||fDrtdd}|dur|}|jD]"}t|ddkr)q|d\} } d|vr7d|d<ggg} } } ggg}}}ggg}}}|dD]~}|jdur[qS|jjjrdtd |j|}t|dkr| dd|d<t |j|d <|j t j kr| |jj| |j| |d qS|j t jkr||j||||d qS|j t jkr||jj||j||d qStd t| dkr|dd 7<t|j|j| | | g|d | | |d|dt|dkr|dd 7<t|j|j|||g|d | | |d|dt|dkr@|dd 7<t|j|j|||g|d | | |d|dq|S)a+Performs a single optimization step. Arguments: closure (callable, optional): A closure that reevaluates the model and returns the loss. The remaining arguments are deprecated, and are only retained (for the moment) for error-checking purposes. css|]}|duVqdS)Nr).0r!rrr @sz!FusedLion.step..zFusedLion has been updated.Nrrr stepz+FusedLion does not support sparse gradientsexp_avgz+FusedLion only support fp16, bf16 and fp32.rr r )any RuntimeErrorrlenrdata is_sparseNotImplementedErrorstategettorch zeros_likedtypefloat16appendbfloat16float32multi_tensor_applierrr)rclosuregrads output_paramsscale grad_norms grad_scalerlossr beta1beta2g_16p_16m_16g_bfp_bfm_bfg_32p_32m_32r!r,rrrr$7sj                zFusedLion.step)rrr T)NNNNNN)__name__ __module__ __qualname____doc__rrr$ __classcell__rrrrrs  r) rKr.multi_tensor_applyrr5deepspeed.acceleratorrdeepspeed.ops.op_builderroptim Optimizerrrrrrs