o پi-@sdZddlZddlmZddlZddlmZddlmmZ ddl m Z m Z m Z mZmZmZmZmZzddlmZWneyJddl mZYnwGdddejZGd d d ejZGd d d ejZGd ddejZdejdefddZdejdejdejdefddZdejdejdejdefddZGdddejZGdddej Z!Gdd d ej Z"Gd!d"d"ej Z#Gd#d$d$ej Z$dS)%z Normalization layers and wrappers Norm layer definitions that support fast norm and consistent channel arg order (always first arg). Hacked together by / Copyright 2022 Ross Wightman N)Tuple) is_fast_normfast_group_normfast_layer_norm fast_rms_norm rms_norm2dfast_rms_norm2dfast_simple_norm simple_norm)rms_normcs8eZdZUejjeed<d fdd ZddZ Z S) GroupNorm _fast_norm h㈵>Tcs tj||||dt|_dS)N)epsaffinesuper__init__rr)self num_channels num_groupsrr __class__D/home/ubuntu/.local/lib/python3.10/site-packages/timm/layers/norm.pyrs zGroupNorm.__init__cC8|jrt||j|j|j|jSt||j|j|j|jSNrrrweightbiasrF group_normrxrrrforward!zGroupNorm.forward)rrT) __name__ __module__ __qualname__torchjitFinalbool__annotations__rr& __classcell__rrrrr s r csHeZdZUdZejjeed<fddZ dej dej fddZ Z S) GroupNorm1zL Group Normalization with 1 group. Input: tensor in shape [B, C, *] rc s"tjd|fi|t|_dS)Nrr)rrkwargsrrrr.s zGroupNorm1.__init__r%returncCrrrr$rrrr&2r'zGroupNorm1.forward r(r)r*__doc__r+r,r-r.r/rTensorr&r0rrrrr1(s  r1cJeZdZUdZejjeed<d fdd Z dej dej fd d Z Z S) LayerNormz# LayerNorm w/ fast norm option rư>Tctj|||dt|_dSN)relementwise_affinerrrrrrrrr> zLayerNorm.__init__r%r3cCs@|jrt||j|j|j|j}|St||j|j|j|j}|Sr)rrnormalized_shaper r!rr" layer_normr$rrrr&Bs zLayerNorm.forwardr9Tr4rrrrr89s r8cr7) LayerNorm2dz5 LayerNorm for channels of '2D' spatial NCHW tensors rr9Tcr:r;rr=rrrrNr>zLayerNorm2d.__init__r%r3cCs^|dddd}|jrt||j|j|j|j}n t||j|j|j|j}|dddd}|SNrr) permuterrr?r r!rr"r@r$rrrr&Rs zLayerNorm2d.forwardrAr4rrrrrBJs rBtensorr3cCs tjr |S|jtjdS)N) memory_format)r+r, is_scripting is_contiguouscontiguous_format)rGrrr_is_contiguous\s rLr%r r!rcCsXtj|dddd\}}||t||}||ddddf|ddddf}|S)NrFT)dimunbiasedkeepdim)r+var_meanrsqrt)r%r r!rsurrr_layer_norm_cfds(rTcCsl|jddd}||jddd||d}||t||}||dddd|dddd}|S)NrT)rMrOr)meanclampr+rQview)r%r r!rrSrRrrr_layer_norm_cf_sqmks  $rYcs2eZdZdZdfdd ZdejfddZZS) LayerNormExp2da_ LayerNorm for channels_first tensors with 2d spatial dimensions (ie N, C, H, W). Experimental implementation w/ manual norm for tensors non-contiguous tensors. This improves throughput in some scenarios (tested on Ampere GPU), esp w/ channels_last layout. However, benefits are not always clear and can perform worse on other GPUs. r9cstj||ddS)N)r)rr)rrrrrrr|szLayerNormExp2d.__init__r3cCsVt|rt|dddd|j|j|j|jdddd}|St||j|j|j}|SrC) rLr"r@rFr?r r!rrTr$rrrr&s zLayerNormExp2d.forward)r9) r(r)r*r5rr+r6r&r0rrrrrZssrZcveZdZUdZgdZeedfed<eed<e ed<e ed<ddfd d Z dddZ de j d e j fddZZS)RmsNormz. RmsNorm w/ fast (apex) norm if available r?rr<r.r?rr<rr9TNr3c||d}t|}t|tjr|f}t||_||_||_t |_ |jr6t t j|jfi||_n|dd|dSN)devicedtyper rr isinstancenumbersIntegraltupler?rr<rrnn Parameterr+emptyr register_parameterreset_parametersrchannelsrrr`rafactory_kwargsr?rrrr      zRmsNorm.__init__cC|jr tj|jdSdSrr<rginitones_r rrrrrkzRmsNorm.reset_parametersr%cC6|jrt||j|j|j}|St||j|j|j}|Sr)rrr?r rr r$rrrr& zRmsNorm.forwardr9TNNr3Nr(r)r*r5 __constants__rintr/floatr.rrkr+r6r&r0rrrrr\  r\cr[) RmsNorm2daB RmsNorm2D for NCHW tensors, w/ fast apex or cast norm if available NOTE: It's currently (2025-05-10) faster to use an eager 2d kernel that does reduction on dim=1 than to permute and use internal PyTorch F.rms_norm, this may change if something like https://github.com/pytorch/pytorch/pull/150576 lands. r].r?rr<rr9TNr3cr^r_rbrlrrrrrozRmsNorm2d.__init__cCrprrqrtrrrrkruzRmsNorm2d.reset_parametersr%cCrvr)rr r?r rrr$rrrr&rwzRmsNorm2d.forwardrxryrzrrrrrs  rcr[) SimpleNormz SimpleNorm (x / std(x)) r].r?rr<rr9TNr3cr^r_rbrlrrrrrozSimpleNorm.__init__cCrprrqrtrrrrkruzSimpleNorm.reset_parametersr%cCrvr)rr r?r rr r$rrrr&s zSimpleNorm.forwardrxryrzrrrrrr~rcr[) SimpleNorm2dz! SimpleNorm for NCHW tensors r].r?rr<rr9TNr3cr^r_rbrlrrrrrozSimpleNorm2d.__init__cCrprrqrtrrrrk$ruzSimpleNorm2d.reset_parametersr%cCsT|dddd}|jrt||j|j|j}n t||j|j|j}|dddd}|SrC)rFrr r?r rr r$rrrr&(s zSimpleNorm2d.forwardrxryrzrrrrrr~r)%r5rdtypingrr+torch.nnrgtorch.nn.functional functionalr" fast_normrrrrrr r r r ImportErrorr r1r8rBr6r.rLr}rTrYrZModuler\rrrrrrrs0  (   *.(