o oiD @s*UddlZddlmZddlmZmZddlmZddlm Z m Z m Z m Z m Z mZmZmZmZmZmZmZddlZddlmZddlmZddlmZdd lmZdd lmZdd lmZdd l m!Z!dd l"m#Z#ddl$m%Z%ddl&m'Z'ddl(m)Z)ddl*m+Z+ddl,m-Z-ddl.m/Z/ddl0m1Z1ede e2e fdZ3dZ4dZ5e6e7d<GdddZ8Gddde/Z9GdddZ:d e d!e fd"d#Z;d$ee efd!eee ejd+e6d,e d!dfd-d.Z?d$e@d!e6fd/d0ZAd1e d!e fd2d3ZBeBejCe_CdS)4N)deepcopy)partialwraps) MethodType) AnyCallableDict GeneratorIteratorListMappingOptionalTupleTypeVarUnionoverload) is_overridden)apply_to_collection)Tensor)nn)OptimizedModule)_IncompatibleKeys) Optimizer) DataLoader)override) Precision)Strategy)move_data_to_device)_set_sampler_epoch)_DeviceDtypeModuleMixin) Optimizable T_destination)bound) training_stepvalidation_step test_step predict_stepF_in_fabric_backwardc @seZdZddededeeeddfddZe defdd Z de e e ffd d Zd e e e fddfd dZddeedefddZdedefddZdS)_FabricOptimizerN optimizerstrategy callbacksreturncCs8||_||_|p g|_td|jj|j|jfi|_dS)aFabricOptimizer is a thin wrapper around the :class:`~torch.optim.Optimizer` that delegates the optimizer step calls to the strategy. The underlying wrapped optimizer object can be accessed via the property :attr:`optimizer`. Args: optimizer: The optimizer to wrap strategy: Reference to the strategy for handling the optimizer step FabricN) _optimizer _strategy _callbackstype __class____name__)selfr)r*r+r5M/home/ubuntu/.local/lib/python3.10/site-packages/lightning/fabric/wrappers.py__init__:s  "z_FabricOptimizer.__init__cC|jSN)r.r4r5r5r6r)Kz_FabricOptimizer.optimizercCs|j|jSr9)r/get_optimizer_stater)r:r5r5r6 state_dictOsz_FabricOptimizer.state_dictr=cCs|j|dSr9)r)load_state_dict)r4r=r5r5r6r>Rsz _FabricOptimizer.load_state_dictclosurecCs|durd|ini}t|jdrt|jjtr|jj}n|j}|jj|fi|}|jD]}t|dd}t |r?||j|dq,|S)Nr?modelon_after_optimizer_step)r*r)) hasattrr/ isinstancer@r r)optimizer_stepr0getattrcallable)r4r?kwargsr)outputcallbackhookr5r5r6stepUs   z_FabricOptimizer.stepitemcCs t|j|Sr9)rEr.r4rLr5r5r6 __getattr__fs z_FabricOptimizer.__getattr__r9)r3 __module__ __qualname__rrr r rr7propertyr)rstrrr=r>rrKrNr5r5r5r6r(9s$r(c seZdZ d0dejdedeejddffdd Zedejfdd Z e d e d e de fd d Z e ddddedededefddZe ddddededeee ffddZe d1deedededeeee ffddZe d2deee fded e defddZdeeefddfdd Zd!edefd"d#Zded$edefd%d&Zd'edefd(d)Ze d*e de ffd+d, Ze d$ed-e ddffd.d/ ZZ S)3 _FabricModuleNforward_moduler*original_moduler,cs4t||_|p ||_||_tt|_d|_dS)aThe FabricModule is a thin wrapper around the :class:`torch.nn.Module` and handles precision / autocast automatically for the forward pass. The underlying wrapped module can be accessed via the property :attr:`module`. Args: forward_module: The module to wrap the ``forward`` method on. strategy: Reference to the strategy for handling precision etc. original_module: The original, unmodified module as passed into the :meth:`lightning.fabric.fabric.Fabric.setup` method. This is needed when attribute lookup on this wrapper should pass through to the original module. TN) superr7_forward_module_original_moduler/set_LIGHTNING_MODULE_STEP_METHODS_forward_methods_fabric_module_initialized)r4rTr*rUr2r5r6r7ks    z_FabricModule.__init__cCs |jp|jSr9)rXrWr:r5r5r6modules z_FabricModule.moduleargsrGcOsp|jj}|||f\}}||j|i|}Wdn1s$wY||}t|t|jd|S)ziCasts all inputs to the right precision and handles autocast for operations in the module forward method.Ndtypefunction) r/ precision convert_inputforward_contextrWconvert_outputrr_register_backward_hook)r4r_rGrcrHr5r5r6forwards  z_FabricModule.forward.)prefix keep_vars destinationrirjcCdSr9r5r4rkrirjr5r5r6r=z_FabricModule.state_dictcCrlr9r5)r4rirjr5r5r6r=rnFcCs|jj|||dS)N)rkrirj)rXr=rmr5r5r6r=s Tr=strictcKs|jjd||d|S)N)r=rpr5)rXr>)r4r=rprGr5r5r6r>sz_FabricModule.load_state_dictmethodcCst|ttfstdt|jt|tr|n|j}|dkr#tdtt|j|dts?t d|dt|jjd|d|j |dS) z]Mark a method as a 'forward' method to prevent it bypassing the strategy wrapper (e.g., DDP).z(Expected a method or a string, but got: rhz>You cannot mark the forward method itself as a forward method.Nz You marked 'z' as a forward method, but `.z$` does not exist or is not a method.) rCrrR TypeErrorr1r3 ValueErrorrErXAttributeErrorr[add)r4rqnamer5r5r6mark_forward_methodsz!_FabricModule.mark_forward_method method_namecsRdksJjjdtdtdtffdd dtdtdtffdd }|S) Nrhr_rGr,cs"j_tj}||i|Sr9)rXrhrE)r_rGrq)ryoriginal_forwardr4r5r6wrapped_forwards zC_FabricModule._redirection_through_forward..wrapped_forwardcsj_j|i|Sr9)rXrh)r_rG)r4r{r5r6call_forward_moduleszG_FabricModule._redirection_through_forward..call_forward_module)rXrhr)r4ryr|r5)ryrzr4r{r6_redirection_through_forwards  z*_FabricModule._redirection_through_forwardrwcsLddtdtddffdd tdtd tdtffd d }|S) zTracks whether any submodule in ``self._original_module`` was called during the execution of ``method`` by registering forward hooks on all submodules.F___r,NcsddS)NTr5)r~r) module_calledr5r6rJszA_FabricModule._wrap_method_with_module_call_tracker..hookr_rGcspg}jD] }||q|i|}r-tdtjjddd|D]}|q/|S)NzYou are calling the method `rrz()` from outside the model. To avoid issues with the currently selected strategy, explicitly mark it as a forward method with `fabric_model.mark_forward_method(z)` after `fabric.setup()`.)rXmodulesappendregister_forward_hook RuntimeErrorr1r3remove)r_rGhandlesr^rHhandlerJrqrrwr4r5r6_wrapped_methods zL_FabricModule._wrap_method_with_module_call_tracker.._wrapped_method)rr)r4rqrwrr5rr6%_wrap_method_with_module_call_trackers $z3_FabricModule._wrap_method_with_module_call_trackertensorcsL|js|Stdjtd}tfdddD}tt|p|}|||S)Nbackwardparentc3s"|] }t|jjtdVqdS)rN)rr/rcr).0rqr:r5r6 s  z8_FabricModule._register_backward_hook..) pre_backwardr post_backward) requires_gradrr/ranyr_backward_hook register_hook)r4rstrategy_requiresprecision_requiresrJr5r:r6rgs  z%_FabricModule._register_backward_hookrLcs|dkr||jvr|j|jkr||Szt|WStyBtd}t||}t |r>|j|jkr>| ||}|YSw)Nr[rX) r[rWrXr}rVrNrurEinspectismethodr)r4rLrUattrr]r5r6rNs       z_FabricModule.__getattr__valuecs|t|ddst||dS|j}t||}|t|v}|s)|s)t|||dS|r1t||||rrrrxrr}rrrgrNr __classcell__r5r5r]r6rSjsb $(     $rSc@steZdZd dedeejddfddZedeejfddZde fd d Z de e e ee ddfffd d ZdS)_FabricDataLoaderN dataloaderdevicer,cCs$|j|j||_||_d|_dS)aThe FabricDataLoader is a wrapper for the :class:`~torch.utils.data.DataLoader`. It moves the data to the device automatically if the device is specified. Args: dataloader: The dataloader to wrap device: The device to which the data should be moved. By default the device is `None` and no data transfers will be made (identical behavior as :class:`~torch.utils.data.DataLoader`). rN)__dict__update _dataloader_device_num_iter_calls)r4rrr5r5r6r7+s  z_FabricDataLoader.__init__cCr8r9)rr:r5r5r6r:r;z_FabricDataLoader.devicecCs t|jSr9)lenrr:r5r5r6__len__>s z_FabricDataLoader.__len__ccsZt|j|j|jd7_|jdurt|jEdHdS|jD] }t||jVq!dS)N)rrrriterrrMr5r5r6__iter__As  z_FabricDataLoader.__iter__r9)r3rOrPrr torchrr7rQintrrr rr rr5r5r5r6r*s (r collectionr,cCsNdttttfdttjttffdd}tttg}|t t |t ||dS)Nobjr,cSsHtt|d}trt|jdSt|tr|jSt|tr"|jS|SNr)rC_unwrap_compiledrSrWr(r)rr)r unwrappedr5r5r6_unwrapPs  z _unwrap_objects.._unwrapr`) rrSr(rrrrrrrrtuple)rrtypesr5r5r6_unwrap_objectsOs  rrcCs8t|trt|dd}durtd|j|fS|dfS)zRemoves the :class:`torch._dynamo.OptimizedModule` around the object if it is wrapped. Use this function before instance checks against e.g. :class:`_FabricModule`. _compile_kwargsNzFailed to determine the arguments that were used to compile the module. Make sure to import lightning before `torch.compile` is used.)rCrrEr _orig_mod)rcompile_kwargsr5r5r6ras  rr^rcCstj|fi|Sr9)rcompile)r^rr5r5r6 _to_compiledqsrrequires_backwardr~cGs|rts tddSdS)NzwThe current strategy and precision selection requires you to call `fabric.backward(loss)` instead of `loss.backward()`.)r'r)rr~r5r5r6rus rcCst|\}}t|tttfS)aZChecks if an object was set up by Fabric. A :class:`~torch.nn.Module` may be wrapped by a :class:`_FabricModule`, a :class:`~torch.optim.Optimizer` may be wrapped by a :class:`_FabricOptimizer`, or a :class:`~torch.utils.data.DataLoader` may be wrapped by :class:`_FabricDataLoader`. Args: obj: The object to test. )rrCrSr(r)rr~r5r5r6 is_wrapped}s r compile_fncs&tdtdtdtffdd }|S)aZWraps the ``torch.compile`` function and captures the compile arguments. We extract the compile arguments so that we can reapply ``torch.compile`` in ``Fabric.setup()`` with the same arguments as the user passed to the original call. The arguments get stored in a dictionary ``_compile_kwargs`` on the returned compiled module. r_rGr,csH|r t|dtjs|i|S|d}|fi|}t||_|Sr)rCrrrr)r_rGr@compiled_modelrr5r6_captures  z)_capture_compile_kwargs.._capture)rr)rrr5rr6_capture_compile_kwargss  r)Drcopyr functoolsrrrrtypingrrrr r r r r rrrrrlightning_utilitiesr#lightning_utilities.core.apply_funcrrr torch._dynamortorch.nn.modules.moduler torch.optimrtorch.utils.datartyping_extensionsrlightning.fabric.pluginsrlightning.fabric.strategiesrlightning.fabric.utilitiesrlightning.fabric.utilities.datar-lightning.fabric.utilities.device_dtype_mixinr lightning.fabric.utilities.typesr rRr!rZr'r__annotations__r(rSrrrrrrobjectrrrr5r5r5r6sF  8                1A%8