o i@sJddlmZmZmZmZmZddlmZmZddl m Z GdddZ dS))DictIterableListUnioncast) has_torch_amptorch)is_torch_arrayc @seZdZdZ     d$deded ed ed ef d d ZddZ d%de de dfde de dffddZ ddde ddefddZddZeddZdd Zd!d"Zd#S)&PyTorchGradScalera Gradient scaler for the PyTorch shim. Gradients with small magnitudes are not representable in half-precision and will underflow to zero. A gradient scaler counters this issue by scaling up the loss before backpropagation, increasing the gradients by the same magnitude. A large enough scale will avoid that the gradients underflow. The gradients are unscaled in single precision after backpropagation, to provide the unscaled gradients to the optimizer. F@?@enabled init_scalebackoff_factor growth_factorgrowth_intervalcCsD||_||_||_||_tjddtjd|_td||_d|_ dS)a Construct a gradient scaler for the PyTorch shim. enabled (bool): Sets whether the gradient scalar is enabled. If it is disabled, the methods of the grad scaler are no-ops. init_scale (float): The initial scale used to increase the gradient magnitude. backoff_factor (float): The scale will be multiplied by this factor if any of the gradients overflows. growth_factor (float): The scale will be multiplied by this factor when none of the gradients overflowed for "growth_interval" steps. growth_interval (int): When no overflows were found for this number of steps, the scale will be multiplied by "growth_factor". r)dtypeFN) _enabled_growth_factor_backoff_factor_growth_intervalr fullint_growth_tracker_scale _found_inf)selfrrrrrr"S/home/ubuntu/.local/lib/python3.10/site-packages/thinc/shims/pytorch_grad_scaler.py__init__s zPyTorchGradScaler.__init__cCs |j||_|j||_dSN)rtor)r!devicer"r"r#to_:szPyTorchGradScaler.to_tensors torch.TensorreturncCs~|jstd|Std}t}t|rtd|}||||St|tr=g}|D]}t|s0||||||q(|S|)z)Scale up the values in the given tensors.r*z>Input to gradient scaling must be a Tensor or Iterable[Tensor]) rr ValueErrordictr _scale_tensor isinstancerappend)r!r)inplaceincorrect_typescale_per_devicetensorscaled_tensorsr"r"r#scale>s&    zPyTorchGradScaler.scaler4r3)z torch.devicer*r1cCs\tstd|jsd}t||j}||vr|jj|d||<||}|r*||S||S)NzHGradient scaling is not supported, requires capable GPU and torch>=1.9.0zGradient scaling is only supported for CUDA tensors. If you are using PyTorch models, you can avoid this error by disabling mixed-precision support.r')rr,is_cudar'rr&mul_)r!r4r3r1msgr'r6r"r"r#r.^s zPyTorchGradScaler._scale_tensorcCs,t}|D]}||jg}||q|Sr%)r- setdefaultr'r0)r!r)tensors_per_devicer4device_tensorsr"r"r#_tensors_per_device|s  z%PyTorchGradScaler._tensors_per_devicecCs|jSr%)r )r!r"r"r# found_infszPyTorchGradScaler.found_infcCsz|jsdS|j}||}|D]"\}}tjdd|d}|j |d}t |||t |dkr9d|_ q|j S)zNUnscale the given tensors. Returns True if any of the gradients were infinite.Frr7rT) rrdouble reciprocalfloatr>itemsr rr&*_amp_foreach_non_finite_check_and_unscale_boolr )r!r) inv_scaler<r'r=found_inf_deviceinv_scale_devicer"r"r#unscales   zPyTorchGradScaler.unscalecCsP|jsdStjd|jr dnd|jjd}t|j|j||j|j |j d|_dS)z Update the scale factor and clear information about infinities. This method should be called after each optimization step. Nrg?r@r7F) rr rr rr'_amp_update_scale_rrrr)r!rHr"r"r#updates zPyTorchGradScaler.updateN)Fr r rr)F)__name__ __module__ __qualname____doc__rFrCrr$r(rrrr6rr.r>propertyr?rJrLr"r"r"r#r sJ  '     r N) typingrrrrrcompatrr utilr r r"r"r"r#s