o TiN=@sTddlZddlZddlZddlmZddlmZddlm Z Gdddej j Z dS)N)get_accelerator)required_torch_version)commcsPeZdZdZ          dfd d Zdd dZfddZZS) OnebitAdamaQImplements the 1-bit Adam algorithm. Currently GPU-only. For usage example please see https://www.deepspeed.ai/tutorials/onebit-adam/ For technical details please read https://arxiv.org/abs/2102.02888 Arguments: params (iterable): iterable of parameters to optimize or dicts defining parameter groups. lr (float, optional): learning rate. (default: 1e-3) freeze_step (int, optional): Number of steps for warmup (uncompressed) stage before we start using compressed communication. (default 100000) betas (Tuple[float, float], optional): coefficients used for computing running averages of gradient and its square. (default: (0.9, 0.999)) eps (float, optional): term added to the denominator to improve numerical stability. (default: 1e-8) weight_decay (float, optional): weight decay (L2 penalty) (default: 0) amsgrad (boolean, optional): whether to use the AMSGrad variant of this algorithm from the paper `On the Convergence of Adam and Beyond`_ (default: False) NOT SUPPORTED in 1-bit Adam! eps_inside_sqrt (boolean, optional): in the 'update parameters' step, adds eps to the bias-corrected second moment estimate before evaluating square root instead of adding it to the square root of second moment estimate as in the original paper. (default: False) cuda_aware (boolean, required): Set True if the underlying MPI implementation supports CUDA-Aware communication. (default: False) comm_backend_name (string, optional): Set to 'mpi' if needed. (default: 'nccl') .. _Adam\: A Method for Stochastic Optimization: https://arxiv.org/abs/1412.6980 .. _On the Convergence of Adam and Beyond: https://openreview.net/forum?id=ryQu7f-RZ NMbP?順Tg?g+?:0yE>Fncclcs| rtdt||||| | d}tt||||rdnd|_d|_d|_d|_d|_ ||_ d|_ d|_ ||_ | |_d|_| |_tsIJdd|_|jdkrotd d sZJd dd lm}t|j d |_||j j|_nD|jdkrddlm}|| |_n3|jdkrddlm}t|j d |_||j j|_n|jdkrddlm}t|j d |_||j j|_|jj|_t |jdt!"|jd|_#dS)Nz01-bit Adam does not support the AMSGrad variant.)lrbias_correctionbetaseps weight_decay max_grad_normrr Fz0Please initialize the torch distributed backend.r g?) min_versionzPlease use torch 1.8 or greater to enable NCCL backend in 1-bit Adam. Alternatively, please specify 'mpi' as the 'comm_backend_name' in config file to proceed with the MPI backend) NcclBackend"pipeline_enable_backward_allreducempi) MpiBackendhccl) HcclBackend compressed)CompressedBackend)$ RuntimeErrordictsuperr__init__eps_mode comm_time step_timeave_stepbk_time deepspeedadam_freeze_key initialize freeze_step cuda_awareusing_pipelinecomm_backend_namedistis_initializedcomm_backend_handlerdeepspeed.runtime.comm.ncclrhasattrmpudeepspeed.runtime.comm.mpirdeepspeed.runtime.comm.hcclr!deepspeed.runtime.comm.compressedrsizeintnpgcddivider)selfparamsr&r r)r rreps_inside_sqrtrramsgradr*r,defaultsrrrr __class__V/home/ubuntu/.local/lib/python3.10/site-packages/deepspeed/runtime/fp16/onebit/adam.pyr .sZ          "zOnebitAdam.__init__c Csd}|dur |}d}d}d}|jdurd}|dur#dgt|j}nt|tjr-|g}nt|dtkr9|g}n|}t|j|D]\} } | durSdgt| d} | drYdnd} t| d| D]\} } | j durq| durqqb| dury| j j } | j rt d|j | }t|dkrd|d <t| j |d <t| j |d <|jr|jrd |vrt| j |d <|d |d<|d |j|jdkr|d|j|j|d |j|j7<|d|j|d<ttj|d| jd|d <tj|d| jd|d<td|_|jstdkrtd|d |d }}| d\}}|d d7<|jdur_||d|| ||d|| | d} |jr^||| d}nd| vr| ddurt | | dt!||d|| ||d|| | d} nK|jdur||d|| d} |jdkr|"|j#$||d |d|j%j&d| vr|j| djkr| dj'|jd| d<|| d|jr||| d}|jr!| ddkr|| d| j 7}t(| | d |Wdn 1swYqb|js6tddd|)d |)dqA|jsLd|_d|_tdt|S|jduro|d |j*krotdd|_|j+rkd|j%_,|Sd|j%_-|S)aPerforms a single optimization step. Arguments: closure (callable, optional): A closure that reevaluates the model and returns the loss. grads (list of tensors, optional): weight gradient to use for the optimizer update. If gradients have type torch.half, parameters are expected to be in type torch.float. (default: None) output params (list of tensors, optional): A reduced precision copy of the updated weights written out in addition to the regular updated weights. Have to be of same type as gradients. (default: None) scale (float, optional): factor to divide gradient tensor values by before applying to weights. (default: 1) NrFr r<r rz,1-bit Adam does not support sparse gradientsstepexp_avg exp_avg_sq worker_error tensor_sizecorrected_tensor_sizeserver_chunk_size)device server_errorTz&Cupy Buffers Initialized Successfully.rr non_freeze exp_avg_maskrr zPop out errors)flushz)Finished the initialization step at rank z.OnebitAdam - starting compressed communication).r'len param_groups isinstancetypes GeneratorTypetypelistzipgraddata is_sparserstatetorch zeros_liker(keysnumelr6r:r empty_cachezerosrKr-get_rankprintmul_add_addcmul_sqrt all_reduceget_world_sizeset_r/compressed_allreducer& local_ranktono_gradpopr)r+renable_backward_allreduce)r;closuregradsloss gather_timeallgather_timeall_time v_diff_buffer grads_groupgroupgrads_this_groupr prXr[rErFbeta1beta2updaterBrBrCrDps                    zOnebitAdam.stepcs^t|jD](\}}d|vr|d|d|d<qd|vr-d|d|vr-|d|dqt||j|jdddd|jkrdtdkrOt d|j durcd|_ |j r_d|j _ n#d|j _ntdkrnt d |j durd|_ |j r~d|j _ nd|j _|jD]'}|dD] }d |j|vr|j|d d |j|vr|j|d qqd S) z^ Overrides load_state_dict() to add special handling when loading checkpoints rNrQrr<rDz?Checkpoint loaded and OnebitAdam warmup stage starts/continues.TFzDCheckpoint loaded and OnebitAdam compression stage starts/continues.rGrLN) enumeraterQrorload_state_dictr[r)r-rbrcr'r+r&rrp)r; state_dictiryr{r@rBrCrs@  "       zOnebitAdam.load_state_dict) NrrTrr Fr r FFr )NN)__name__ __module__ __qualname____doc__r rDr __classcell__rBrBr@rCrs$! Br)rSr\numpyr8deepspeed.acceleratorrdeepspeed.utils.torchrr&rr-optim OptimizerrrBrBrBrCs