o zic@sxddlZddlmZmZddlmZddlmZmZm Z m Z m Z m Z m Z mZmZmZddlZddlmZddlmZddlmZddlZddlmZdd lmZdd lmZdd lm Z dd l!m"Z"dd l#m$Z$m%Z%ddl&m'Z'ddl(m)Z)m*Z*ddl+m,Z,ddl-m.Z.ddl/m0Z0ddl1m2Z2ddl3m4Z4ddl5m6Z6m7Z7edZ8edZ9e:e;ZdS)N)ABCabstractmethod)contextmanager) AnyCallableDict GeneratorListMappingOptionalTupleTypeVarUnion)Tensor)Module) Optimizer) CheckpointIO)_StrategyRegistry)move_data_to_device)ReduceOp) _EmptyInit)_optimizer_to_device_optimizers_to_device)_PATH)LightningOptimizer"_init_optimizers_and_lr_schedulers)TorchCheckpointIO)_WrappingCheckpointIO) Precision) _Launcher) TrainerFn) STEP_OUTPUTLRSchedulerConfig TBroadcastTReducec @seZdZdZ   ddeddeedeeddfdd Zedee fd d Z ededfd d Z e j ddd Z edefddZ e j deddfddZ edefddZej deeddfddZedeefddZej deeddfddZdddZdddZdd d!Zdd$d%Zdd&d'Zdd(d)Zd*edeeeffd+d,Zd-ed*eed.ed/edef d0d1Z dd*ed2egefdeede fd/edef d3d4Z!de deede"e eeffd5d6Z#de de fd7d8Z$d*edefd9d:Z%dde(defd?d@Z)ee*de&j'fdAdBZ+e*ddCdDZ,ee*de-fdEdFZ.e*  GddHeeefdIeedJeee/efdeeeffdKdLZ0e*ddMeeddfdNdOZ1e*ddPe2dQe(de2fdRdSZ3e*ddHedIeedUe-defdVdWZ4ddYe-dZe-de-fd[d\Z5d-eddfd]d^Z6d-eddfd_d`Z7edee fdadbZ8e8j dcee ddfdddbZ8ededfdedfZ9dge:deeeffdhdiZ;ddjed.ed/ede?fdpdqZ@ddrdsZAd.ed/ede?fdtduZBd.ed/ede?fdvdwZCd.ed/edefdxdyZDdzeEdeEfd{d|ZFede-fd}d~ZGede-fddZHede-fddZIdeeeffddZJ ddjeeefde:deeddfddZKde:ddfddZLeMddee-deNdfddZOeMdeNdfddZPdddZQeRdeSddfddZTdddZUdddZVdddZWdddZXdddZYdddZZdddZ[dddZ\dzStrategy.acceleratorcC ||_dSr+r?)r7r&r8r8r9r&L cCs4|jdur t|_|jSt|jtrt|j_|jSr+)r-r isinstancerr(r<r8r8r9r(Ps  zStrategy.checkpoint_ioiocCr@r+)r-)r7rCr8r8r9r(YrAcCs|jdur|jStSr+)r.rr<r8r8r9r)]szStrategy.precision_plugincCr@r+)r.)r7r)r8r8r9r)arAcCr;r+)r4r<r8r8r9 optimizerser>zStrategy.optimizersrDcs|_fdd|D_dS)Ncsg|]}t|qSr8)r_to_lightning_optimizer).0optr<r8r9 lsz'Strategy.optimizers..)r4r5)r7rDr8r<r9rDismodelpl.LightningModulecCs||_||_dS)z=Called by the Trainer to connect the strategy with the model.N)r/rIr7rIr8r8r9connectns zStrategy.connectcCdS)z&Attach the launcher based on Strategy.Nr8r<r8r8r9_configure_launcheruzStrategy._configure_launchercCs |jdusJ|j|jdS)zSetup any processes or distributed connections. This is called before the LightningModule/DataModule setup hook which allows the user to access the accelerator environment before setup is complete. N)r& setup_device root_devicer<r8r8r9setup_environmentxszStrategy.setup_environmenttrainer pl.TrainercCs$|jdusJt|j\|_|_dS)zCreates optimizers and schedulers. Args: trainer: the Trainer, these optimizers should be connected to N)lightning_modulerrDr6r7rSr8r8r9setup_optimizersszStrategy.setup_optimizerscCs|jdusJ|j||jdusJ|j|j|_|||j|_|jjt j kr3| || |jjt j krGt |j|jdSdS)zSets up the accelerator, plugins and initializes the optimizers (if needed). Args: trainer: the trainer instance N)r&setuprIr)convert_modulemodel_to_device _setup_modelstatefnr FITTINGrWsetup_precision_pluginrrDrQrVr8r8r9rXs  zStrategy.setupcCs@|jdusJ|j|j|j|j\}}}||_||_||_dS)z.Attaches the precision plugin to the strategy.N)rIr)rLrDr6)r7rIrDr6r8r8r9r_s   zStrategy.setup_precision_plugin optimizercCs<t|tr|j}t|dr||jr|SiS|S)zReturns state of an optimizer. Allows for syncing/collating optimizer state from processes in custom strategies. consolidate_state_dict)rBr _optimizerhasattrrais_global_zero state_dictr7r`r8r8r9optimizer_states  zStrategy.optimizer_state closure_lossargskwargscOsf|||jdus J|j||j}|jj||j|g|Ri||j||j}|||S)a'Forwards backward-calls to the precision plugin. Args: closure_loss: a tensor holding the loss value to backpropagate optimizer: An optional optimizer that gets passed down to the precision plugin's backward \*args: Positional arguments that get passed down to the precision plugin's backward, intended as arguments for the actual function that performs the backward, like :meth:`~torch.Tensor.backward`. \**kwargs: Keyword arguments for the same purpose as ``*args``. N) pre_backwardrUr)backward post_backward)r7rhr`rirjr8r8r9rls   zStrategy.backwardclosurecKs4|p|j}t|tjs J|jj|f||d|S)aHPerforms the actual optimizer step. Args: optimizer: the optimizer performing the step closure: closure calculating the loss value model: reference to the model, optionally defining optimizer step related hooks \**kwargs: Keyword arguments to ``optimizer.step`` )rIrn)rUrBplLightningModuler)optimizer_step)r7r`rnrIrjr8r8r9rqs zStrategy.optimizer_stepcs$|}fdd|D}||fS)zSetup a model and multiple optimizers together. The returned objects are expected to be in the same order they were passed in. The default implementation will call :meth:`_setup_model` and :meth:`_setup_optimizer` on the inputs. csg|]}|qSr8)_setup_optimizer)rFr`r<r8r9rHsz8Strategy._setup_model_and_optimizers..)r[)r7rIrDr8r<r9_setup_model_and_optimizerss z$Strategy._setup_model_and_optimizerscC|S)zDPerforms setup for the model, e.g., by wrapping it by another class.r8rKr8r8r9r[zStrategy._setup_modelcCrt)zHPerforms setup for the optimizer, e.g., by wrapping it by another class.r8rfr8r8r9rrruzStrategy._setup_optimizerrbatchdevicedataloader_idxcCs2|j}|p|j}|dur|j|||dSt||S)azMoves the batch to the correct device. The returned batch is of the same type as the input batch, just having all tensors on the correct device. Args: batch: The batch of samples to move to the correct device device: The target device dataloader_idx: The index of the dataloader to which the batch belongs. N)rwrx)rUrQ_apply_batch_transfer_handlerr)r7rvrwrxrIr8r8r9batch_to_devices   zStrategy.batch_to_devicecCrM)zReturns the root device.Nr8r<r8r8r9rQrOzStrategy.root_devicecCrM)z&Moves the model to the correct device.Nr8r<r8r8r9rZrOzStrategy.model_to_devicecCrM)zcWhether the current process is the rank zero process not only on the local node, but for all nodes.Nr8r<r8r8r9rd!rOzStrategy.is_global_zeromeantensorgroup reduce_opcCrM)a,Reduces the given tensor (e.g. across GPUs/processes). Args: tensor: the tensor to sync and reduce group: the process group to reduce reduce_op: the reduction operation. Defaults to 'mean'. Can also be a string 'sum' or ReduceOp. Nr8)r7r|r}r~r8r8r9reduce&rOzStrategy.reducenamecCrM)zSynchronizes all processes which blocks processes until the whole group enters this function. Args: name: an optional name to pass into barrier. Nr8)r7rr8r8r9barrier7rOzStrategy.barrierobjsrccCrM)zBroadcasts an object to all processes. Args: obj: the object to broadcast src: source rank Nr8)r7rrr8r8r9 broadcast@rOzStrategy.broadcastF sync_gradscCrM)aPerform an all_gather on all processes. Args: tensor: the tensor to all_gather group: the process group to gather results from sync_grads: flag that allows users to synchronize gradients for all_gather op Nr8)r7r|r}rr8r8r9 all_gatherJrOzStrategy.all_gatherTdecisionallcCrt)z/Reduce a boolean decision across all processes.r8)r7rrr8r8r9reduce_boolean_decisionUz Strategy.reduce_boolean_decisioncCrM)z.Run before precision plugin executes backward.Nr8r7rhr8r8r9rkYrOzStrategy.pre_backwardcCrM)z-Run after precision plugin executes backward.Nr8rr8r8r9rm\rOzStrategy.post_backwardcCs|jdur|jS|jS)z0Returns the potentially wrapped LightningModule.N)r0r/r<r8r8r9rI_szStrategy.model new_modelcCr@r+)r0)r7rr8r8r9rIdrAcCr;)zWhether the strategy handles gradient accumulation internally.Fr8r<r8r8r9handles_gradient_accumulationruz&Strategy.handles_gradient_accumulationcCs|jdusJ|jS)zReturns model state.N)rUrer<r8r8r9lightning_module_state_dicts z$Strategy.lightning_module_state_dictfilepathstorage_optionscCs |jr|jj|||ddSdS)a?Save model/training states as a checkpoint file through state-dump and file-write. Args: checkpoint: dict containing model and trainer state filepath: write-target file's path storage_options: parameter for how to save to storage, passed to ``CheckpointIO`` plugin )rN)rdr(save_checkpoint)r7rrrr8r8r9rs zStrategy.save_checkpointcCs|jr |j|dSdS)zqRemove checkpoint filepath from the filesystem. Args: filepath: Path to checkpoint N)rdr(remove_checkpoint)r7rr8r8r9rszStrategy.remove_checkpoint empty_init)NNNc cstt|d}|<|j |j dVWdn1s"wYWdn1s1wYWddSWddS1sIwYdS)aControls how tensors get created (device, dtype). Args: empty_init: Whether to initialize the model with empty weights (uninitialized memory). If ``None``, the strategy will decide. Some strategies may not support all options. )enabledN)rboolrQr)tensor_init_context)r7rempty_init_contextr8r8r9rs PzStrategy.tensor_init_contextccs dVdS)aProvide hook to create modules in a distributed aware context. This is useful for when we'd like to shard the model instantly, which is useful for extremely large models which can save memory and initialization time. Returns: Model parallel context. Nr8r<r8r8r9model_sharded_contexts zStrategy.model_sharded_contextcCsjt|jtd|jdurt|jjd|j |j |j dus)J|j |j dS)zThis method is called to teardown the training process. It is the right place to release memory and free other resources. cpuNz: moving model to CPU)rrDrrwrUlogdebug __class____name__rr)teardownr&r(r<r8r8r9r s    zStrategy.teardownstrategy_registrycCdSr+r8)clsrr8r8r9register_strategiesrzStrategy.register_strategiescCrM)zCalled when train begins.Nr8r<r8r8r9on_train_start!rzStrategy.on_train_startcCrM)zCalled when validation begins.Nr8r<r8r8r9on_validation_start%rzStrategy.on_validation_startcCrM)zCalled when test begins.Nr8r<r8r8r9 on_test_start)rzStrategy.on_test_startcCrM)zCalled when predict begins.Nr8r<r8r8r9on_predict_start-rzStrategy.on_predict_startcCrM)zCalled when train ends.Nr8r<r8r8r9 on_train_end1rzStrategy.on_train_endcCrM)zCalled when validation ends.Nr8r<r8r8r9on_validation_end5rzStrategy.on_validation_endcCrM)zCalled when test end.Nr8r<r8r8r9 on_test_end9rzStrategy.on_test_endcCrM)zCalled when predict ends.Nr8r<r8r8r9on_predict_end=rzStrategy.on_predict_end batch_idxcCrM)zCCalled in the training loop before anything happens for that batch.Nr8)r7rvrr8r8r9on_train_batch_startArzStrategy.on_train_batch_start exceptioncCrM)zACalled when the trainer execution is interrupted by an exception.Nr8)r7rr8r8r9 on_exceptionErzStrategy.on_exceptioncCsg|_g|_g|_dSr+)r4r5r6r<r8r8r9 _reset_optimizers_and_schedulersIs z)Strategy._reset_optimizers_and_schedulerscCstt|}g|d<|S)Nr5)dictvarsr7r\r8r8r9 __getstate__Ns zStrategy.__getstate__r\cCs||_|j|_dSr+)__dict__rDrr8r8r9 __setstate__Ts zStrategy.__setstate__)r&r'r*N)rIrJr*N)r*N)rSrTr*Nr+)Nr)Nr{)r)NF)T)cr __module__ __qualname____doc__r rrr:propertyrr=r&setterr(r)r rrDrLrNrRrWrXr_rstrrrgrrlrrrrqr rsr[rrrrwintrzrrQrZrrdrrrr#rrrrkrmrIrUrrr rrr!rrrrrobjectrrrrrrrrrrrr classmethodrrrrrrrrrrr BaseExceptionrrrrr8r8r8r9r%.s(           & "     "                      r%c @s^eZdZdZdedddedededef d d Zdedddd fd d Z dedddd fddZ d S)r2zImplements the `forward-redirection`. A method call to a wrapped module gets rerouted through the wrapper's `forward` method instead. wrapper_moduleoriginal_modulerJ method_namerirjr*csXdksJjdtdtdtffdd }|_|i|}|S)aReroutes a method call through the `wrapper_module`'s `forward` method. Args: wrapper_module: The module that has `original_module` wrapped. original_module: The module that was wrapped inside `wrapper_module`. method_name: The name of the method that should be called on the `original_module` after inputs get redirected through the `wrapper_module`'s `forward` method. *args: The positional arguments to the method `method_name`. They will get passed to a patched `forward` method instead. **kwargs: The keyword arguments to the method `method_name`. They will get passed to a patched `forward` method instead. forward_args_kwargsr*cs._t}||i|}|Sr+)rgetattron_after_inner_forward)rrmethodoutroriginal_forwardrr7rr8r9wrapped_forwardss   z5_ForwardRedirection.__call__..wrapped_forward)rron_after_outer_forward)r7rrrrirjrwrapper_outputr8rr9__call__`s "  z_ForwardRedirection.__call__NcCrr+r8r7rrr8r8r9rz*_ForwardRedirection.on_after_inner_forwardcCrr+r8rr8r8r9rrz*_ForwardRedirection.on_after_outer_forward) rrrrrrrr!rrrr8r8r8r9r2Ys" $r2)?loggingabcrr contextlibrtypingrrrrr r r r r rrrtorch.nnr torch.optimrpytorch_lightningrolightning_fabric.pluginsrlightning_fabric.strategiesrlightning_fabric.utilitiesr&lightning_fabric.utilities.distributedrlightning_fabric.utilities.initr$lightning_fabric.utilities.optimizerrr lightning_fabric.utilities.typesr pytorch_lightning.core.optimizerrrpytorch_lightning.pluginsr$pytorch_lightning.plugins.io.wrapperr#pytorch_lightning.plugins.precisionr/pytorch_lightning.strategies.launchers.launcherr pytorch_lightning.trainer.statesr !pytorch_lightning.utilities.typesr!r"r#r$ getLoggerrrr%r2r8r8r8r9s@  0               /