o oi@sddlmZddlmZmZmZmZmZmZddl m Z ddl m Z ddl mZddlmZmZedZdd d d d Zed Zed ZeeeefZGdddZdS) nullcontext)AnyContextManagerDictLiteralOptionalUnion)Tensor)Module) Optimizer) _PARAMETERS Optimizable)@ 64-true32-true16-mixed bf16-mixed)643216bf16)ztransformer-engineztransformer-engine-float16z16-truerz bf16-truerrrc @sHeZdZUdZdZeed<dedefddZde fdd Z de fd d Z de fd d Z de de fddZde de fddZdedeede fddZdedeede de ddf ddZdedeede fddZdede de fdd Zdedefd!d"Zdeddfd#d$Zdeee ffd%d&Zd'eee fddfd(d)Zd,d*d+ZdS)- PrecisionzBase class for all plugins handling the precision-specific parts of the training. The class attribute precision must be overwritten in child classes. The default value reflects fp32 training. r precisionmodulereturncC|S)zConvert the module parameters to the precision type this plugin handles. This is optional and depends on the precision limitations during optimization. )selfrrr`/home/ubuntu/.local/lib/python3.10/site-packages/lightning/fabric/plugins/precision/precision.pyconvert_module0szPrecision.convert_modulecCtS)z1Controls how tensors get created (device, dtype).rr rrr!tensor_init_context8zPrecision.tensor_init_contextcCr#)zInstantiate module parameters or tensors in the precision type this plugin handles. This is optional and depends on the precision limitations during optimization. rr$rrr!module_init_context<szPrecision.module_init_contextcCr#)zWA contextmanager for managing model forward/training_step/evaluation_step/predict_step.rr$rrr!forward_contextDr&zPrecision.forward_contextdatacCr)zConvert model inputs (forward) to the floating point precision type of this plugin. This is a no-op in the base precision plugin, since we assume the data already has the desired type (default is torch.float32). rr r)rrr! convert_inputHzPrecision.convert_inputcCr)zConvert outputs to the floating point precision type expected after model's forward. This is a no-op in the base precision plugin, since we assume the data already has the desired type (default is torch.float32). rr*rrr!convert_outputQr,zPrecision.convert_outputtensorcCdS)zRuns before precision plugin executes backward. Args: tensor: The tensor that will be used for backpropagation module: The module that was involved in producing the tensor and whose parameters need the gradients Nrr r.rrrr! pre_backwardZzPrecision.pre_backwardmodelargskwargsNcOs|j|i|dS)zPerforms the actual backpropagation. Args: tensor: The tensor that will be used for backpropagation model: The module that was involved in producing the tensor and whose parameters need the gradients N)backward)r r.r3r4r5rrr!r6cszPrecision.backwardcCr/)zRuns after precision plugin executes backward. Args: tensor: The tensor that will be used for backpropagation module: The module that was involved in producing the tensor and whose parameters need the gradients Nrr0rrr! post_backwardmr2zPrecision.post_backward optimizercKs|jdi|S)zHook to run the optimizer step.Nr)step)r r8r5rrr!optimizer_stepvszPrecision.optimizer_stepccs |jD] }|dEdHqdS)zThe main params of the model. Returns the plain model params here. Maybe different in other precision plugins. paramsN) param_groups)r r8grouprrr! main_params~s zPrecision.main_paramscCsdS)Nr)r r8rrr!unscale_gradientsszPrecision.unscale_gradientscCsiS)zCalled when saving a checkpoint, implement to generate precision plugin state_dict. Returns: A dictionary containing precision plugin state. rr$rrr! state_dictr,zPrecision.state_dictr@cCr/)zCalled when loading a checkpoint, implement to reload precision plugin state given precision plugin state_dict. Args: state_dict: the precision plugin state returned by ``state_dict``. Nr)r r@rrr!load_state_dictszPrecision.load_state_dictcCr/)zThis method is called to teardown the training process. It is the right place to release memory and free other resources. Nrr$rrr!teardownr2zPrecision.teardown)rN) __name__ __module__ __qualname____doc__r_PRECISION_INPUT_STR__annotations__r r"rr%r'r(rr+r-r rr1r6r7rr:r r r>r?rstrr@rArBrrrr!r's0    "       rN) contextlibrtypingrrrrrr torchr torch.nnr torch.optimr lightning.fabric.utilities.typesr r_PRECISION_INPUT_INT%_PRECISION_INPUT_STR_ALIAS_CONVERSION_PRECISION_INPUT_STR_ALIASrG_PRECISION_INPUTrrrrr!s