o oiG@sddlmZddlmZmZmZmZmZmZddl Z ddl m Z ddl m Z ddl mZddlmZmZddlmZmZddlmZdd lmZdd lmZmZdd lmZdd lm Z dd l!m"Z"ddl#m$Z$ddl%m&Z&ddl'm(Z(erzddl)Z)e(Z*Gddde Z+dS)) nullcontext) TYPE_CHECKINGAnyCallableContextManagerOptionalUnionN)apply_to_collection)Tensor)Module)LBFGS Optimizer)get_argsoverride)_PRECISION_INPUT)_convert_fp_tensor_DtypeContextManager) Steppable) Precision)GradClipAlgorithmType)MisconfigurationException) is_overridden) WarningCachec @seZdZdZdeddfddZededefdd Zed e de fd d Z ede fd dZ ede fddZ ededddeede de ddf ddZededddege fde de f ddZedejfdedeeefdeddfdd ZdS)!DeepSpeedPrecisionatPrecision plugin for DeepSpeed integration. .. warning:: This is an :ref:`experimental ` feature. Args: precision: Full precision (32-true), half precision (16-true, bf16-true) or mixed precision (16-mixed, bf16-mixed). Raises: ValueError: If unsupported ``precision`` is provided. precisionreturnNcCsVtt}||vrtd|d|d||_tjtjtjtjtjd}||j|_dS)Nz)`Trainer(strategy='deepspeed', precision=z1)` is not supported. `precision` must be one of: .)z bf16-mixedz16-mixedz bf16-truez16-truez32-true) rr ValueErrorrtorchbfloat16float16float32_desired_dtype)selfrsupported_precisionprecision_to_typer&a/home/ubuntu/.local/lib/python3.10/site-packages/lightning/pytorch/plugins/precision/deepspeed.py__init__7szDeepSpeedPrecision.__init__modulecCsd|jvr |j|jdS|S)Ntrue)dtype)rtor")r#r)r&r&r'convert_moduleHs z!DeepSpeedPrecision.convert_moduledatacCst|tt|jdS)N)functionr+dst_type)r rr r")r#r.r&r&r' convert_inputNsz DeepSpeedPrecision.convert_inputcCsd|jvrtSt|jS)Nr*)rrrr"r#r&r&r'tensor_init_contextRs  z&DeepSpeedPrecision.tensor_init_contextcCs|S)N)r3r2r&r&r'module_init_contextXsz&DeepSpeedPrecision.module_init_contexttensormodelzpl.LightningModule optimizerargskwargscOs8td|r td|jj}|j|g|Ri|dS)aPerforms back-propagation using DeepSpeed's engine. Args: tensor: the loss tensor model: the model to be optimized optimizer: ignored for DeepSpeed \*args: additional positional arguments for the :meth:`deepspeed.DeepSpeedEngine.backward` call \**kwargs: additional keyword arguments for the :meth:`deepspeed.DeepSpeedEngine.backward` call backwardzYou have overridden the `LightningModule.backward` hook but it will be ignored since DeepSpeed handles the backward logic internally.N)r warning_cachewarntrainerr6r:)r#r5r6r7r8r9deepspeed_enginer&r&r'r:\s zDeepSpeedPrecision.backwardclosurecKsVt|tr td|}||||du}|jr|rtd|jj}|jdi|S)Nz5DeepSpeed and the LBFGS optimizer are not compatible.z_Skipping backward by returning `None` from your `training_step` is not supported by `DeepSpeed`r&) isinstancer r_after_closureautomatic_optimizationr=r6step)r#r7r6r?r9closure_resultskipped_backwardr>r&r&r'optimizer_stepws   z!DeepSpeedPrecision.optimizer_stepgclip_valgradient_clip_algorithmcCsdS)z/DeepSpeed handles gradient clipping internally.Nr&)r#r7rGrHr&r&r'clip_gradientssz!DeepSpeedPrecision.clip_gradients)__name__ __module__ __qualname____doc__rr(rr r-rr1rr3r4r rrr:rrFrNORMr rintfloatrIr&r&r&r'r(sb    r), contextlibrtypingrrrrrrrlightning_utilitiesr r torch.nnr torch.optimr r typing_extensionsrrlightning.pytorchpytorchpl,lightning.fabric.plugins.precision.deepspeedr(lightning.fabric.plugins.precision.utilsrr lightning.fabric.utilities.typesr-lightning.pytorch.plugins.precision.precisionrlightning.pytorch.utilitiesr&lightning.pytorch.utilities.exceptionsr)lightning.pytorch.utilities.model_helpersr%lightning.pytorch.utilities.rank_zeror deepspeedr;rr&r&r&r's*