o پi@sdZddlZddlmZddlmmZddlmZm Z m Z Gdddej Z Gdddej Z Gd d d ej ZGd d d ej ZdS) a Convolution with Weight Standardization (StdConv and ScaledStdConv) StdConv: @article{weightstandardization, author = {Siyuan Qiao and Huiyu Wang and Chenxi Liu and Wei Shen and Alan Yuille}, title = {Weight Standardization}, journal = {arXiv preprint arXiv:1903.10520}, year = {2019}, } Code: https://github.com/joe-siyuan-qiao/WeightStandardization ScaledStdConv: Paper: `Characterizing signal propagation to close the performance gap in unnormalized ResNets` - https://arxiv.org/abs/2101.08692 Official Deepmind JAX code: https://github.com/deepmind/deepmind-research/tree/master/nfnets Hacked together by / copyright Ross Wightman, 2021. N) get_paddingget_padding_valuepad_samec.eZdZdZ  d fdd Zdd ZZS) StdConv2dzConv2d with Weight Standardization. Used for BiT ResNet-V2 models. Paper: `Micro-Batch Training with Batch-Channel Normalization and Weight Standardization` - https://arxiv.org/abs/1903.10520v2 rNFư>c s:|dur t|||}tj||||||||d| |_dS)Nstridepaddingdilationgroupsbias)rsuper__init__eps) self in_channel out_channels kernel_sizer r r r rr __class__H/home/ubuntu/.local/lib/python3.10/site-packages/timm/layers/std_conv.pyr s  zStdConv2d.__init__c CsRtj|jd|jddddd|jd|j}t|||j|j |j |j |j }|SNrT)trainingmomentumr) F batch_normweightreshaperr reshape_asconv2drr r r r rxr!rrrforward*s zStdConv2d.forward)rNrrFr__name__ __module__ __qualname____doc__rr' __classcell__rrrrr  rcr) StdConv2dSamezConv2d with Weight Standardization. TF compatible SAME padding. Used for ViT Hybrid model. Paper: `Micro-Batch Training with Batch-Channel Normalization and Weight Standardization` - https://arxiv.org/abs/1903.10520v2 rSAMEFrc s@t||||d\}} tj||||||||d| |_| |_dS)Nr r r )rrrsame_padr) rrrrr r r r rr is_dynamicrrrr8s  zStdConv2dSame.__init__c Csl|jr t||j|j|j}tj|jd|j ddddd|j d |j}t |||j |j|j|j|j}|Sr)r2rrr r rr r!r"rrr#r$rr r r%rrrr'Bs zStdConv2dSame.forward)rr0rrFrr(rrrrr/2r.r/c.eZdZdZ  d fdd Zd d ZZS) ScaledStdConv2da4Conv2d layer with Scaled Weight Standardization. Paper: `Characterizing signal propagation to close the performance gap in unnormalized ResNets` - https://arxiv.org/abs/2101.08692 NOTE: the operations used in this impl differ slightly from the DeepMind Haiku impl. The impact is minor. rNT?rc sp|dur t|||}tj||||||||dtt|jdddf| |_| |j d d|_ | |_ dS)Nr rr) rrrnn Parametertorchfullrgainr!numelscaler) r in_channelsrrr r r r rgammar gain_initrrrrUs   zScaledStdConv2d.__init__c Cs^tj|jd|jddd|j|jddd|jd |j}t |||j |j |j |j|jSNrrTr)r!rrr)rr r!r"rr<r>viewrr#r$rr r r r r%rrrr'as zScaledStdConv2d.forward)rNrrTr6rr6r(rrrrr5L   r5cr4) ScaledStdConv2dSamea\Conv2d layer with Scaled Weight Standardization and Tensorflow-like SAME padding support Paper: `Characterizing signal propagation to close the performance gap in unnormalized ResNets` - https://arxiv.org/abs/2101.08692 NOTE: the operations used in this impl differ slightly from the DeepMind Haiku impl. The impact is minor. rr0Tr6rc svt||||d\}} tj||||||||dtt|jdddf| |_| |j d d|_ | |_ | |_ dS)Nr1r rrr7)rrrr8r9r:r;rr<r!r=r>r2r) rr?rrr r r r rr@rrAr3rrrrrs  zScaledStdConv2dSame.__init__c Csx|jr t||j|j|j}tj|jd|j ddd|j |j  ddd|j d|j}t|||j|j|j|j|jSrB)r2rrr r rr r!r"rr<r>rCrr#r$rr r r%rrrr'~s zScaledStdConv2dSame.forward)rr0rrTr6rr6r(rrrrrEirDrE)r,r:torch.nnr8torch.nn.functional functionalrr rrrConv2drr/r5rErrrrs