o 8wic@stddlZddlmZmZddlmZmZddlmZddl m Z m Z m Z m Z mZddlZddlmZddlmZddlmZddlZdd lmZdd lmZdd lmZdd lmZdd lm Z ddl!m"Z"m#Z#ddl$m%Z%ddl&m'Z'm(Z(ddl)m*Z*ddl+m,Z,ddl-m.Z.ddl/m0Z0ddl1m2Z2ddl3m4Z4m5Z5e dZ6e dZ7e8e9Z:GdddeZ;GdddZe(defd?d@Z)ee*de&j'fdAdBZ+e*ddCdDZ,ee*de-fdEdFZ.e*  GddHeeefdIeedJeee/efdeeeffdKdLZ0e*ddMeeddfdNdOZ1e*ddPe2dQe(de2fdRdSZ3e*ddHedIeedUe-defdVdWZ4ddYe-dZe-de-fd[d\Z5d-eddfd]d^Z6d-eddfd_d`Z7edee fdadbZ8e8j dcee ddfdddbZ8ededfdedfZ9dge:deeeffdhdiZ;ddjed.ed/ede?fdpdqZ@ddrdsZAd.ed/ede?fdtduZBd.ed/ede?fdvdwZCd.ed/edefdxdyZDdzeEdeEfd{d|ZFede-fd}d~ZGede-fddZHede-fddZIdeeeffddZJ ddjeeefde:deeddfddZKde:ddfddZLeMddee-deNdfddZOeMdeNdfddZPdddZQeRdeSddfddZTdddZUdddZVdddZWdddZXdddZYdddZZdddZ[dddZ\dcCs|jdur|jStSr()r+rr9r5r5r6r&^szStrategy.precision_plugincCr=r()r+)r4r&r5r5r6r&br>cCr8r()r1r9r5r5r6 optimizersfr;zStrategy.optimizersrAcs|_fdd|D_dS)Ncsg|]}t|qSr5)r_to_lightning_optimizer).0optr9r5r6 msz'Strategy.optimizers..)r1r2)r4rAr5r9r6rAjsmodelpl.LightningModulecCs||_||_dS)z=Called by the Trainer to connect the strategy with the model.N)r,rFr4rFr5r5r6connectos zStrategy.connectcCdS)z&Attach the launcher based on Strategy.Nr5r9r5r5r6_configure_launchervzStrategy._configure_launchercCs |jdusJ|j|jdS)zSetup any processes or distributed connections. This is called before the LightningModule/DataModule setup hook which allows the user to access the accelerator environment before setup is complete. N)r# setup_device root_devicer9r5r5r6setup_environmentyszStrategy.setup_environmenttrainer pl.TrainercCs$|jdusJt|j\|_|_dS)zCreates optimizers and schedulers. Args: trainer: the Trainer, these optimizers should be connected to N)lightning_modulerrAr3r4rPr5r5r6setup_optimizersszStrategy.setup_optimizerscCs|jdusJ|j||jdusJ|j|j|_|||j|_|jjt j kr3| || |jjt j krGt |j|jdSdS)zSets up the accelerator, plugins and initializes the optimizers (if needed). Args: trainer: the trainer instance N)r#setuprFr&convert_modulemodel_to_device _setup_modelstatefnrFITTINGrTsetup_precision_pluginrrArNrSr5r5r6rUs  zStrategy.setupcCs@|jdusJ|j|j|j|j\}}}||_||_||_dS)z.Attaches the precision plugin to the strategy.N)rFr&rIrAr3)r4rFrAr3r5r5r6r\s   zStrategy.setup_precision_plugin optimizercCs<t|tr|j}t|dr||jr|SiS|S)zReturns state of an optimizer. Allows for syncing/collating optimizer state from processes in custom strategies. consolidate_state_dict)r?r _optimizerhasattrr^is_global_zero state_dictr4r]r5r5r6optimizer_states  zStrategy.optimizer_state closure_lossargskwargscOsf|||jdus J|j||j}|jj||j|g|Ri||j||j}|||S)a'Forwards backward-calls to the precision plugin. Args: closure_loss: a tensor holding the loss value to backpropagate optimizer: An optional optimizer that gets passed down to the precision plugin's backward \*args: Positional arguments that get passed down to the precision plugin's backward, intended as arguments for the actual function that performs the backward, like :meth:`~torch.Tensor.backward`. \**kwargs: Keyword arguments for the same purpose as ``*args``. N) pre_backwardrRr&backward post_backward)r4rer]rfrgr5r5r6ris   zStrategy.backwardclosurecKs4|p|j}t|tjs J|jj|f||d|S)aHPerforms the actual optimizer step. Args: optimizer: the optimizer performing the step closure: closure calculating the loss value model: reference to the model, optionally defining optimizer step related hooks \**kwargs: Keyword arguments to ``optimizer.step`` )rFrk)rRr?plLightningModuler&optimizer_step)r4r]rkrFrgr5r5r6rns zStrategy.optimizer_stepcs$|}fdd|D}||fS)zSetup a model and multiple optimizers together. The returned objects are expected to be in the same order they were passed in. The default implementation will call :meth:`_setup_model` and :meth:`_setup_optimizer` on the inputs. csg|]}|qSr5)_setup_optimizer)rCr]r9r5r6rEsz8Strategy._setup_model_and_optimizers..)rX)r4rFrAr5r9r6_setup_model_and_optimizerss z$Strategy._setup_model_and_optimizerscC|S)zDPerforms setup for the model, e.g., by wrapping it by another class.r5rHr5r5r6rXzStrategy._setup_modelcCrq)zHPerforms setup for the optimizer, e.g., by wrapping it by another class.r5rcr5r5r6rorrzStrategy._setup_optimizerrbatchdevicedataloader_idxcCs2|j}|p|j}|dur|j|||dSt||S)azMoves the batch to the correct device. The returned batch is of the same type as the input batch, just having all tensors on the correct device. Args: batch: The batch of samples to move to the correct device device: The target device dataloader_idx: The index of the dataloader to which the batch belongs. N)rtru)rRrN_apply_batch_transfer_handlerr)r4rsrtrurFr5r5r6batch_to_devices   zStrategy.batch_to_devicecCrJ)zReturns the root device.Nr5r9r5r5r6rNrLzStrategy.root_devicecCrJ)z&Moves the model to the correct device.Nr5r9r5r5r6rWrLzStrategy.model_to_devicecCrJ)zcWhether the current process is the rank zero process not only on the local node, but for all nodes.Nr5r9r5r5r6ra"rLzStrategy.is_global_zeromeantensorgroup reduce_opcCrJ)a,Reduces the given tensor (e.g. across GPUs/processes). Args: tensor: the tensor to sync and reduce group: the process group to reduce reduce_op: the reduction operation. Defaults to 'mean'. Can also be a string 'sum' or ReduceOp. Nr5)r4ryrzr{r5r5r6reduce'rLzStrategy.reducenamecCrJ)zSynchronizes all processes which blocks processes until the whole group enters this function. Args: name: an optional name to pass into barrier. Nr5)r4r}r5r5r6barrier8rLzStrategy.barrierobjsrccCrJ)zBroadcasts an object to all processes. Args: obj: the object to broadcast src: source rank Nr5)r4rrr5r5r6 broadcastArLzStrategy.broadcastF sync_gradscCrJ)aPerform an all_gather on all processes. Args: tensor: the tensor to all_gather group: the process group to gather results from sync_grads: flag that allows users to synchronize gradients for all_gather op Nr5)r4ryrzrr5r5r6 all_gatherKrLzStrategy.all_gatherTdecisionallcCrq)z/Reduce a boolean decision across all processes.r5)r4rrr5r5r6reduce_boolean_decisionVz Strategy.reduce_boolean_decisioncCrJ)z.Run before precision plugin executes backward.Nr5r4rer5r5r6rhZrLzStrategy.pre_backwardcCrJ)z-Run after precision plugin executes backward.Nr5rr5r5r6rj]rLzStrategy.post_backwardcCs|jdur|jS|jS)z0Returns the potentially wrapped LightningModule.N)r-r,r9r5r5r6rF`szStrategy.model new_modelcCr=r()r-)r4rr5r5r6rFer>cCr8)zWhether the strategy handles gradient accumulation internally.Fr5r9r5r5r6handles_gradient_accumulationrrz&Strategy.handles_gradient_accumulationcCs|jdusJ|jS)zReturns model state.N)rRrbr9r5r5r6lightning_module_state_dicts z$Strategy.lightning_module_state_dictfilepathstorage_optionscCs |jr|jj|||ddSdS)a?Save model/training states as a checkpoint file through state-dump and file-write. Args: checkpoint: dict containing model and trainer state filepath: write-target file's path storage_options: parameter for how to save to storage, passed to ``CheckpointIO`` plugin )rN)rar%save_checkpoint)r4rrrr5r5r6rs zStrategy.save_checkpointcCs|jr |j|dSdS)zqRemove checkpoint filepath from the filesystem. Args: filepath: Path to checkpoint N)rar%remove_checkpoint)r4rr5r5r6rszStrategy.remove_checkpoint empty_init)NNNc cstt|d}|<|j |j dVWdn1s"wYWdn1s1wYWddSWddS1sIwYdS)aControls how tensors get created (device, dtype). Args: empty_init: Whether to initialize the model with empty weights (uninitialized memory). If ``None``, the strategy will decide. Some strategies may not support all options. )enabledN)rboolrNr&tensor_init_context)r4rempty_init_contextr5r5r6rs PzStrategy.tensor_init_contextccs dVdS)aProvide hook to create modules in a distributed aware context. This is useful for when we'd like to shard the model instantly, which is useful for extremely large models which can save memory and initialization time. Returns: Model parallel context. Nr5r9r5r5r6model_sharded_contexts zStrategy.model_sharded_contextcCsjt|jtd|jdurt|jjd|j |j |j dus)J|j |j dS)zThis method is called to teardown the training process. It is the right place to release memory and free other resources. cpuNz: moving model to CPU)rrArrtrRlogdebug __class____name__rr&teardownr#r%r9r5r5r6rs    zStrategy.teardownstrategy_registrycCdSr(r5)clsrr5r5r6register_strategiesrzStrategy.register_strategiescCrJ)zCalled when train begins.Nr5r9r5r5r6on_train_start"rzStrategy.on_train_startcCrJ)zCalled when validation begins.Nr5r9r5r5r6on_validation_start&rzStrategy.on_validation_startcCrJ)zCalled when test begins.Nr5r9r5r5r6 on_test_start*rzStrategy.on_test_startcCrJ)zCalled when predict begins.Nr5r9r5r5r6on_predict_start.rzStrategy.on_predict_startcCrJ)zCalled when train ends.Nr5r9r5r5r6 on_train_end2rzStrategy.on_train_endcCrJ)zCalled when validation ends.Nr5r9r5r5r6on_validation_end6rzStrategy.on_validation_endcCrJ)zCalled when test end.Nr5r9r5r5r6 on_test_end:rzStrategy.on_test_endcCrJ)zCalled when predict ends.Nr5r9r5r5r6on_predict_end>rzStrategy.on_predict_end batch_idxcCrJ)zCCalled in the training loop before anything happens for that batch.Nr5)r4rsrr5r5r6on_train_batch_startBrzStrategy.on_train_batch_start exceptioncCrJ)zACalled when the trainer execution is interrupted by an exception.Nr5)r4rr5r5r6 on_exceptionFrzStrategy.on_exceptioncCsg|_g|_g|_dSr()r1r2r3r9r5r5r6 _reset_optimizers_and_schedulersJs z)Strategy._reset_optimizers_and_schedulerscCstt|}g|d<|S)Nr2)dictvarsr4rYr5r5r6 __getstate__Os zStrategy.__getstate__rYcCs||_|j|_dSr()__dict__rArr5r5r6 __setstate__Us zStrategy.__setstate__)r#r$r'N)rFrGr'N)r'N)rPrQr'Nr()Nr)Nrx)r)NF)T)cr __module__ __qualname____doc__r rrr7propertyrr:r#setterr%r&listrrArIrKrOrTrUr\rstrr rdrrirr r rntuplerprXrorrtintrwrrNrWrrarr|r~r rrrrhrjrFrRrrrrrrrrrrrobjectrrrrrrrrrrrr classmethodrrrrrrrrrrr BaseExceptionrrrrr5r5r5r6r"/s(           & "     "                      r"c @s^eZdZdZdedddedededef d d Zdedddd fd d Z dedddd fddZ d S)r/zImplements the `forward-redirection`. A method call to a wrapped module gets rerouted through the wrapper's `forward` method instead. wrapper_moduleoriginal_modulerG method_namerfrgr'csXdksJjdtdtdtffdd }|_|i|}|S)aReroutes a method call through the `wrapper_module`'s `forward` method. Args: wrapper_module: The module that has `original_module` wrapped. original_module: The module that was wrapped inside `wrapper_module`. method_name: The name of the method that should be called on the `original_module` after inputs get redirected through the `wrapper_module`'s `forward` method. *args: The positional arguments to the method `method_name`. They will get passed to a patched `forward` method instead. **kwargs: The keyword arguments to the method `method_name`. They will get passed to a patched `forward` method instead. forward_args_kwargsr'cs._t}||i|}|Sr()rgetattron_after_inner_forward)rrmethodoutroriginal_forwardrr4rr5r6wrapped_forwardts   z5_ForwardRedirection.__call__..wrapped_forward)rron_after_outer_forward)r4rrrrfrgrwrapper_outputr5rr6__call__as "  z_ForwardRedirection.__call__NcCrr(r5r4rrr5r5r6rz*_ForwardRedirection.on_after_inner_forwardcCrr(r5rr5r5r6rrz*_ForwardRedirection.on_after_outer_forward) rrrrr rrrrrrr5r5r5r6r/Zs" $r/)=loggingabcrrcollections.abcrr contextlibrtypingrrr r r rr torch.nnr torch.optimrpytorch_lightningrllightning_fabric.pluginsrlightning_fabric.strategiesrlightning_fabric.utilitiesr&lightning_fabric.utilities.distributedrlightning_fabric.utilities.initr$lightning_fabric.utilities.optimizerrr lightning_fabric.utilities.typesr pytorch_lightning.core.optimizerrrpytorch_lightning.pluginsr$pytorch_lightning.plugins.io.wrapperr#pytorch_lightning.plugins.precisionr/pytorch_lightning.strategies.launchers.launcherr pytorch_lightning.trainer.statesr!pytorch_lightning.utilities.typesrrr r! getLoggerrrr"r/r5r5r5r6sB                 /