o Ti,@szdZddlZddlmZddlmZdZdZdZ dZ d Z d d Z Gd d d Z Gddde ZGddde ZddZdS)a Copyright (c) 2019, NVIDIA CORPORATION. All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Taken and modified for DeepSpeed from: https://github.com/NVIDIA/Megatron-LM/blob/master/fp16/loss_scaler.py Commit: 93ab4bea59dc5cbf97c079d313741866af4deac9 N)comm)logger init_scale scale_window delayed_shiftconsecutive_hysteresis min_scalecCst|dr |S|dS)Nitemr)hasattrr )tr V/home/ubuntu/.local/lib/python3.10/site-packages/deepspeed/runtime/fp16/loss_scaler.pyto_python_float$s rc@s>eZdZdZddZeddZddZdd Zdd d Z d S)LossScalerBasez4LossScalarBase Base class for a loss scaler cCs||_d|_dSNF) cur_scaledynamic)selfrr r r __init__/s zLossScalerBase.__init__cCs|jSN)rrr r r loss_scale3szLossScalerBase.loss_scalecstfdd|DS)Nc3s|]}j|VqdSr)r).0grr r 8sz0LossScalerBase.scale_gradient..)tuple)rmodulegrad_ingrad_outr rr scale_gradient7szLossScalerBase.scale_gradientcCsdSrr )roverflowr r r update_scale:zLossScalerBase.update_scaleFcCs||j}|j|ddS)N) retain_graph)rbackward)rlossr# scaled_lossr r r r$=s zLossScalerBase.backwardN)F) __name__ __module__ __qualname____doc__rpropertyrrr!r$r r r r r*s rcs2eZdZdZd fdd ZddZddZZS) LossScalerau Class that manages a static loss scale. This class is intended to interact with :class:`FP16_Optimizer`, and should not be directly manipulated by the user. Use of :class:`LossScaler` is enabled via the ``static_loss_scale`` argument to :class:`FP16_Optimizer`'s constructor. Args: scale (float, optional, default=1.0): The loss scale. cstt||dSr)superr,r)rscale __class__r r rOszLossScaler.__init__cCdSrr )rparamsr r r has_overflowSr"zLossScaler.has_overflowcCr2rr )xr r r _has_inf_or_nanWr"zLossScaler._has_inf_or_nan)r-)r'r(r)r*rr4r6 __classcell__r r r0r r,Cs  r,csLeZdZdZdddddddejffdd Zd d Zd d ZddZ Z S)DynamicLossScalera Class that manages dynamic loss scaling. It is recommended to use :class:`DynamicLossScaler` indirectly, by supplying ``dynamic_loss_scale=True`` to the constructor of :class:`FP16_Optimizer`. However, it's important to understand how :class:`DynamicLossScaler` operates, because the default options can be changed using the the ``dynamic_loss_args`` argument to :class:`FP16_Optimizer`'s constructor. Loss scaling is designed to combat the problem of underflowing gradients encountered at long times when training fp16 networks. Dynamic loss scaling begins by attempting a very high loss scale. Ironically, this may result in OVERflowing gradients. If overflowing gradients are encountered, :class:`DynamicLossScaler` informs :class:`FP16_Optimizer` that an overflow has occurred. :class:`FP16_Optimizer` then skips the update step for this particular iteration/minibatch, and :class:`DynamicLossScaler` adjusts the loss scale to a lower value. If a certain number of iterations occur without overflowing gradients detected, :class:`DynamicLossScaler` increases the loss scale once more. In this way :class:`DynamicLossScaler` attempts to "ride the edge" of always using the highest loss scale possible without incurring overflow. Args: init_scale (float, optional, default=2**32): Initial loss scale attempted by :class:`DynamicLossScaler.` scale_factor (float, optional, default=2.0): Factor used when adjusting the loss scale. If an overflow is encountered, the loss scale is readjusted to loss scale/``scale_factor``. If ``scale_window`` consecutive iterations take place without an overflow, the loss scale is readjusted to loss_scale*``scale_factor``. scale_window (int, optional, default=1000): Number of consecutive iterations without an overflow to wait before increasing the loss scale. consecutive_hysteresis (bool, optional, default=False): Whether to refill hysteresis if we reach an iteration that doesn't overflow lg@ir-FTc sVtt||d|_d|_||_||_||_||_||_ ||_ ||_ d|_ ||_ dS)NrT)r.r8rcur_iterlast_overflow_iter scale_factorrrrcur_hysteresisrraise_error_at_min_scalerdtype) rrr<rrrrr>r?r0r r rvs  zDynamicLossScaler.__init__cCs,|D]}|jdur||jjrdSqdS)NTF)gradr6data)rr3pr r r has_overflow_serials z%DynamicLossScaler.has_overflow_serialc Csrz t|}Wnty%}zd|jdvrWYd}~dSd}~ww|tdtd fvs5||kr7dSdS)Nzvalue cannot be convertedrTinfF)floatsum RuntimeErrorargs)r5cpu_suminstancer r r r6sz!DynamicLossScaler._has_inf_or_nancCs|r|jdks |jdkrO|j|jkr|jrtdt|j|j|j}t dkrKdt d}|j t j krF|dt |jdt |7}t|||_n4t dkr|dt d}|j t j krw|dt |jd|jd |jd7}t||jd8_|j|_n2|jrt dkrd |j}t||j|_|j|j|jdkr|js|j|_|j|j9_|jd7_dS) Nr-zSCurrent loss scale already at minimum - cannot decrease scale anymore. Exiting run.rzeepspeed] OVERFLOW! Rank z Skipping step.z Attempted loss scale: z, reducing to z, but hysteresis is z. Reducing hysteresis to z;Consecutive hysteresis is enabled. Restoring hysteresis to )rr=rrr> Exceptionmaxr<distget_rankr?torchhalfintrinfor:r;rr)rr next_scale overflow_msghysteresis_msgr r r r!s<     (     zDynamicLossScaler.update_scale) r'r(r)r*rOrPrrCr6r!r7r r r0r r8[sr8cCsN|tjkr|r|durt|dStdd|i|S|tjkr |nd}t|dS)N)r?r?g?)r/r )rOrPr8r,)r?static_loss_scaledynamic_scalingdynamic_loss_argsloss_scale_valuer r r CreateLossScalers   rZ)r*rO deepspeedrrMdeepspeed.utilsrINITIAL_LOSS_SCALE SCALE_WINDOW DELAYED_SHIFTCONSECUTIVE_HYSTERESISMIN_LOSS_SCALErrr,r8rZr r r r s  u