o þϯioã@srddlZddlmZddlmZddlmZddlmZGdd„dej ƒZ Gdd„dej ƒZ Gd d „d ej ƒZdS) éN)Údifferentiable_all_reduce)Úget_world_size)Únncó eZdZdZ‡fdd„Z‡ZS)ÚNaiveSyncBatchNorm1dzp An implementation of 1D naive sync batch normalization. See details in NaiveSyncBatchNorm2d below. c s8tƒdks|jstƒ |¡S|jd|jd}}tj|ddgd}tj||ddgd}|dks5Jdƒ‚tj||gdd}t|ƒdt  ¡}t  ||¡\}}|||}t  ||j ¡}|j |} |j|| } |  ddd¡} |  ddd¡} |j|j| ¡|j7_|j|j| ¡|j7_|| | S)Néré©Údimú/SyncBatchNorm does not support zero batch size.çð?éÿÿÿÿ©rÚtrainingÚsuperÚforwardÚshapeÚtorchÚmeanÚcatrÚdistÚsplitÚrsqrtÚepsÚweightÚbiasÚreshapeÚ running_meanÚmomentumÚdetachÚ running_var© ÚselfÚinputÚBÚCrÚmeansqrÚvecÚvarÚinvstdÚscaler©Ú __class__©úR/home/ubuntu/.local/lib/python3.10/site-packages/pytorchvideo/layers/batch_norm.pyrs$    zNaiveSyncBatchNorm1d.forward©Ú__name__Ú __module__Ú __qualname__Ú__doc__rÚ __classcell__r-r-r+r.r srcr)ÚNaiveSyncBatchNorm2daŽ An implementation of 2D naive sync batch normalization. In PyTorch<=1.5, ``nn.SyncBatchNorm`` has incorrect gradient when the batch size on each worker is different. (e.g., when scale augmentation is used, or when it is applied to mask head). This is a slower but correct alternative to `nn.SyncBatchNorm`. Note: This module computes overall statistics by using statistics of each worker with equal weight. The result is true statistics of all samples (as if they are all on one worker) only when all workers have the same (N, H, W). This mode does not support inputs with zero batch size. c s<tƒdks|jstƒ |¡S|jd|jd}}tj|gd¢d}tj||gd¢d}|dks5Jdƒ‚tj||gdd}t|ƒdt  ¡}t  ||¡\}}|||}t  ||j ¡}|j |} |j|| } |  dddd¡} |  dddd¡} |j|j| ¡|j7_|j|j| ¡|j7_|| | S)Nrr)rrér r r r rr!r+r-r.r<s$    zNaiveSyncBatchNorm2d.forwardr/r-r-r+r.r5,sr5cr)ÚNaiveSyncBatchNorm3dzp An implementation of 3D naive sync batch normalization. See details in NaiveSyncBatchNorm2d above. c s@tƒdks|jstƒ |¡S|jd|jd}}tj|gd¢d}tj||gd¢d}|dks5Jdƒ‚tj||gdd}t|ƒdt  ¡}t  ||¡\}}|||}t  ||j ¡}|j |} |j|| } |  ddddd¡} |  ddddd¡} |j|j| ¡|j7_|j|j| ¡|j7_|| | S)Nrr)rrr6ér r r r rr!r+r-r.r_s$    zNaiveSyncBatchNorm3d.forwardr/r-r-r+r.r7Xsr7)rÚtorch.distributedÚ distributedrÚfvcore.nn.distributedrÚpytorchvideo.layers.distributedrrÚ BatchNorm1drÚ BatchNorm2dr5Ú BatchNorm3dr7r-r-r-r.Ús    ",