o zil@sxddlmZmZddlZddlmZddlmZddlmZGdddeZ Gdd d e Z Gd d d ej j j jZdS) )ABCabstractmethodN)Tensor)Module)overridec@<eZdZdZededefddZededefddZdS) LayerSynczxAbstract base class for creating plugins that wrap layers of a model with synchronization logic for multiprocessing.modelreturncCdS)zJOverride this method to apply synchronization to the layers of this model.Nselfr r r X/home/ubuntu/.local/lib/python3.10/site-packages/pytorch_lightning/plugins/layer_sync.pyapplyzLayerSync.applycCr )zEOverride this method to undo all modifications made in :meth:`apply`.Nr r r r rrevertrzLayerSync.revertN)__name__ __module__ __qualname____doc__rrrrr r r rrs rc@r) TorchSyncBatchNormzA plugin that wraps all batch normalization layers of a model with synchronization logic for multiprocessing. This plugin has no effect in single-device operation. r r cCstjj|S)aAdd global batchnorm for a model spread across multiple GPUs and nodes. Override this method to synchronize batchnorm layers between specific process groups instead of the whole world. Args: model: Reference to the current LightningModule Return: LightningModule with batchnorm layers synchronized within the process groups. )torchnn SyncBatchNormconvert_sync_batchnormr r r rr+szTorchSyncBatchNorm.applycCs|}t|tjjjjrLt|j|j|j |j |j }|j r7t |j |_ |j|_Wdn1s2wY|j|_|j|_|j|_t|drL|j|_|D] \}}||||qP~|S)aConvert the wrapped batchnorm layers back to regular batchnorm layers. Args: model: Reference to the current LightningModule Return: LightningModule with regular batchnorm layers that will no longer sync across processes. Nqconfig) isinstancerrmodules batchnormr _BatchNormXd num_featuresepsmomentumaffinetrack_running_statsno_gradweightbias running_mean running_varnum_batches_trackedhasattrrnamed_children add_moduler)rr converted_modulenamechildr r rr;s&    zTorchSyncBatchNorm.revertN)rrrrrrrrr r r rr$s rc@s"eZdZededdfddZdS)r inputr NcCsdS)Nr )rr2r r r_check_input_dim_sz_BatchNormXd._check_input_dim)rrrrrr3r r r rr ^sr )abcrrrrtorch.nnrtyping_extensionsrrrrrr _BatchNormr r r r rs    :