o 8wi@sddlmZmZddlmZmZmZmZddlm Z ddl m Z ddl m Z ddlmZmZedZdd d d d Zed Zed ZeeeefZGdddZdS))AbstractContextManager nullcontext)AnyLiteralOptionalUnion)Tensor)Module) Optimizer) _PARAMETERS Optimizable)@ 64-true32-true16-mixed bf16-mixed)643216bf16)ztransformer-engineztransformer-engine-float16z16-truerz bf16-truerrrc @sHeZdZUdZdZeed<dedefddZde fdd Z de fd d Z de fd d Z de de fddZde de fddZdedeede fddZdedeede de ddf ddZdedeede fddZdede de fdd Zdedefd!d"Zdeddfd#d$Zdeee ffd%d&Zd'eee fddfd(d)Zd,d*d+ZdS)- PrecisionzBase class for all plugins handling the precision-specific parts of the training. The class attribute precision must be overwritten in child classes. The default value reflects fp32 training. r precisionmodulereturncC|S)zConvert the module parameters to the precision type this plugin handles. This is optional and depends on the precision limitations during optimization. )selfrrri/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/lightning_fabric/plugins/precision/precision.pyconvert_module0szPrecision.convert_modulecCtS)z1Controls how tensors get created (device, dtype).rrrrrtensor_init_context8zPrecision.tensor_init_contextcCr!)zInstantiate module parameters or tensors in the precision type this plugin handles. This is optional and depends on the precision limitations during optimization. r"r#rrrmodule_init_context<szPrecision.module_init_contextcCr!)zWA contextmanager for managing model forward/training_step/evaluation_step/predict_step.r"r#rrrforward_contextDr%zPrecision.forward_contextdatacCr)zConvert model inputs (forward) to the floating point precision type of this plugin. This is a no-op in the base precision plugin, since we assume the data already has the desired type (default is torch.float32). rrr(rrr convert_inputHzPrecision.convert_inputcCr)zConvert outputs to the floating point precision type expected after model's forward. This is a no-op in the base precision plugin, since we assume the data already has the desired type (default is torch.float32). rr)rrrconvert_outputQr+zPrecision.convert_outputtensorcCdS)zRuns before precision plugin executes backward. Args: tensor: The tensor that will be used for backpropagation module: The module that was involved in producing the tensor and whose parameters need the gradients Nrrr-rrrr pre_backwardZzPrecision.pre_backwardmodelargskwargsNcOs|j|i|dS)zPerforms the actual backpropagation. Args: tensor: The tensor that will be used for backpropagation model: The module that was involved in producing the tensor and whose parameters need the gradients N)backward)rr-r2r3r4rrrr5cszPrecision.backwardcCr.)zRuns after precision plugin executes backward. Args: tensor: The tensor that will be used for backpropagation module: The module that was involved in producing the tensor and whose parameters need the gradients Nrr/rrr post_backwardmr1zPrecision.post_backward optimizercKs|jdi|S)zHook to run the optimizer step.Nr)step)rr7r4rrroptimizer_stepvszPrecision.optimizer_stepccs |jD] }|dEdHqdS)zThe main params of the model. Returns the plain model params here. Maybe different in other precision plugins. paramsN) param_groups)rr7grouprrr main_params~s zPrecision.main_paramscCsdS)Nr)rr7rrrunscale_gradientsszPrecision.unscale_gradientscCsiS)zCalled when saving a checkpoint, implement to generate precision plugin state_dict. Returns: A dictionary containing precision plugin state. rr#rrr state_dictr+zPrecision.state_dictr?cCr.)zCalled when loading a checkpoint, implement to reload precision plugin state given precision plugin state_dict. Args: state_dict: the precision plugin state returned by ``state_dict``. Nr)rr?rrrload_state_dictszPrecision.load_state_dictcCr.)zThis method is called to teardown the training process. It is the right place to release memory and free other resources. Nrr#rrrteardownr1zPrecision.teardown)rN) __name__ __module__ __qualname____doc__r_PRECISION_INPUT_STR__annotations__r r rr$r&r'rr*r,rrr0r5r6r r9r r r=r>dictstrr?r@rArrrrr's0    "       rN) contextlibrrtypingrrrrtorchrtorch.nnr torch.optimr lightning_fabric.utilities.typesr r _PRECISION_INPUT_INT%_PRECISION_INPUT_STR_ALIAS_CONVERSION_PRECISION_INPUT_STR_ALIASrF_PRECISION_INPUTrrrrrs