o TibL@sTddlZddlZddlZddlmZddlmZddlm Z Gdddej j Z dS)N)get_accelerator)required_torch_version)commcsVeZdZdZ           dfd d ZdddZfddZZS) ZeroOneAdama Implements the 0/1 Adam algorithm. Currently GPU-only. For usage example please see https://www.deepspeed.ai/tutorials/zero-one-adam/ For technical details please read https://arxiv.org/abs/2202.06009 Arguments: params (iterable): iterable of parameters to optimize or dicts defining parameter groups. lr (float, optional): learning rate. (default: 1e-3) betas (Tuple[float, float], optional): coefficients used for computing running averages of gradient and its square. (default: (0.9, 0.999)) eps (float, optional): term added to the denominator to improve numerical stability. (default: 1e-8) weight_decay (float, optional): weight decay (L2 penalty) (default: 0) var_freeze_step (int, optional): The latest step to update the variance, using the notation from https://arxiv.org/abs/2202.06009, it denotes the max{i|i in T_v}. Note that this is different from the freeze step from the 1-bit Adam. The var_freeze_step is usually the end of the learning rate warmup and thus does not require tuning. (default: 100000) var_update_scaler (int, optional): The interval to update the variance. Note that the update policy for variance follows an exponential rule, where var_update_scaler denotes the kappa in the 0/1 Adam paper. (default: 16) local_step_scaler (int, optional): The interval to scale the local steps interval according to the learning rate policy. (default: 32678) local_step_clipper (int, optional): The largest interval for local steps with learning rate policy. This corresponds to the variable H in the 0/1 Adam paper. (default: 16) amsgrad (boolean, optional): whether to use the AMSGrad variant of this algorithm from the paper `On the Convergence of Adam and Beyond`_ (default: False) NOT SUPPORTED in 0/1 Adam! eps_inside_sqrt (boolean, optional): in the 'update parameters' step, adds eps to the bias-corrected second moment estimate before evaluating square root instead of adding it to the square root of second moment estimate as in the original paper. (default: False) cuda_aware (boolean, required): Set True if the underlying MPI implementation supports CUDA-Aware communication. (default: False) comm_backend_name (string, optional): Set to 'mpi' if needed. (default: 'nccl') .. _Adam\: A Method for Stochastic Optimization: https://arxiv.org/abs/1412.6980 .. _On the Convergence of Adam and Beyond: https://openreview.net/forum?id=ryQu7f-RZ NMbP?Tg?g+?:0yE>F順ncclcs|rtdt|||||| d}tt||||rdnd|_||_d|_||_d|_ | |_ | |_ | |_ | |_ d|_d|_||_tsIJdd|_|jdkrotdd sZJd dd lm}t|jd |_ ||jj|_nD|jd krddlm}|||_n3|jdkrddlm}t|jd |_ ||jj|_n|jdkrddlm}t|jd |_ ||jj|_|jj|_t |jdt!"|jd|_#dS)Nz.0/1 Adam does not support the AMSGrad variant.)lrbias_correctionbetaseps weight_decay max_grad_normrFz0Please initialize the torch distributed backend.r g?) min_versionzPlease use torch 1.8 or greater to enable NCCL backend in 0/1 Adam. Alternatively, please specify 'mpi' as the 'comm_backend_name' in config file to proceed with the MPI backend) NcclBackend"pipeline_enable_backward_allreducempi) MpiBackendhccl) HcclBackend compressed)CompressedBackend)$ RuntimeErrordictsuperr__init__eps_mode deepspeed initialize cuda_awareusing_pipelinevar_freeze_stepvar_update_scalerlocal_step_scalerlocal_step_clipper freeze_keyreinitial_error_buffercomm_backend_namedistis_initializedcomm_backend_handlerdeepspeed.runtime.comm.ncclrhasattrmpudeepspeed.runtime.comm.mpirdeepspeed.runtime.comm.hcclr!deepspeed.runtime.comm.compressedrsizeintnpgcddivider)selfparamsr$rrrreps_inside_sqrtrrr(r)r*r+amsgradr&r.defaultsrrrr __class__X/home/ubuntu/.local/lib/python3.10/site-packages/deepspeed/runtime/fp16/onebit/zoadam.pyr":sZ          "zZeroOneAdam.__init__c Csd}|dur |}|durdgt|j}nt|tjr |g}nt|dtkr,|g}n|}t|j|D]N\}}|durFdgt|d}|drLdnd}t|d|D]\}} |jdurd| durdqU| durl|jj } | j rst d|j |} t| dkrd| d<t |j | d<t |j | d <|jrd | vr"d| d <d| d <d| d <d| d<d| d<t |j | d<| d| d<| d|j|jdkr| d|j|j| d|j|j7<| d|j| d<tt j| d|jd| d <t j| d|jd| d<t |j | d<t|js"tdkr"td| d| d } } | d} |d\}}| dd7<|jr|jdur| d| d dkrk| |d|| | | |j| d|dnl|jdkrt B|j !| | d | d|j"j#}d|vr|j|djkr|dj$|jd|d<||d| |d||Wdn 1swYn| |j| d|d| d|d7<d} |js|jdkr| %|j !| | d | d|j"j#d|vr| j|djkr|dj$| jd|d<| |d|jrk| | &|d}|ddkr3||d|j 7}t #|j |d ||jdurS| |d |Wdn 1s^wY| d| d dkr|jrt v|j d | | | &|d|jdkr| '|j !| | d | d|j"j#d|vr| j|djkr|dj$| jd|d<| |d| (j| | dd d|j | | &|d| (d| d<Wdn 1swY|jdurK| d| d dkr"| d d7<| d |j)kr"d| d <| d d!9<| dd| d dkr=|j*r8d|j"_+qUd|j"_,qU|j*rFd|j"_+qUd|j"_,qU| dd7<| d|j-krkd| d<t.|j/| d d!| d <qU|jstd"dd#d|_| 0d | 0dq4|jsd|_td$t|S|j |jdddd|j1krd|_|j*rd|j"_+nd|j"_,|jdur|j2dur|jD]}|dD]}|j |d (|j |d(q͐qd|_2|S)%aPerforms a single optimization step. Arguments: closure (callable, optional): A closure that reevaluates the model and returns the loss. grads (list of tensors, optional): weight gradient to use for the optimizer update. If gradients have type torch.half, parameters are expected to be in type torch.float. (default: None) output params (list of tensors, optional): A reduced precision copy of the updated weights written out in addition to the regular updated weights. Have to be of same type as gradients. (default: None) scale (float, optional): factor to divide gradient tensor values by before applying to weights. (default: 1) Nrr>rrz*0/1 Adam does not support sparse gradientsstepexp_avg exp_avg_sq worker_error var_interval var_counterlocal_step_intervallocal_step_counterlrs tensor_sizecorrected_tensor_sizeserver_chunk_size)device server_errormomentum_accumulatorz&Cupy Buffers Initialized Successfully.rF)alpha exp_avg_maskrrrr TzPop out errors)flushz)Finished the initialization step at rank )3len param_groups isinstancetypes GeneratorTypetypelistzipgraddata is_sparserstatetorch zeros_liker%keysnumelr8r<r empty_cachezerosrRr/get_rankprintr,mul_addcmul_add_no_gradr1compressed_allreducer$ local_ranktoset_sqrtcopy_zero_r)r'renable_backward_allreducer*minr+popr(r-)r=closuregradsloss grads_groupgroupgrads_this_grouprprbrerGrH comm_bufferbeta1beta2 grad_onebitupdaterDrDrErFs6                               $   zZeroOneAdam.stepcst|jD](\}}d|vr|d|d|d<qd|vr-d|d|vr-|d|dqt||j|jdddd|jkrd|_|j|jddddd|j|jdddddkru|jrpd |j _ n!d |j _ n|jr}d|j _ nd|j _ nd |_|jrd|j _ nd|j _ d|_ |jD]6}|dD]/}d |j|vr|j|d d |j|vr|j|d d |j|vr|j|d qqd S)z^ Overrides load_state_dict() to add special handling when loading checkpoints rVr[rr>rFFrrJTrIrSrTN) enumerater[r{r!load_state_dictrer(var_freeze_keyr'r$rryr-)r= state_dictirrrBrDrErCsL  "       zZeroOneAdam.load_state_dict)NrTrrFr r r r r r FFr )NN)__name__ __module__ __qualname____doc__r"rFr __classcell__rDrDrBrErs*- EEr)r]rfnumpyr:deepspeed.acceleratorrdeepspeed.utils.torchrr$rr/optim OptimizerrrDrDrDrEs