o پi@s dZddlmZddlmZmZddlmZddlm Z m Z m Z m Z m Z ddlZddlmZddlmZmZddlmZmZmZmZmZmZmZmZmZd d lmZd d l m!Z!d d l"m#Z#d d l$m%Z%m&Z&ddgZ'eGdddZ(Gdddej)Z*dde+de,de fddZ-Gdddej)Z.e!Gdddej)Z/    dde0d e0d!e+d"e e d#e e d$e1de ej2e0e e+e fffd%d&Z3e4dd'd(d)d*d+d,d-d.d/d0d1d2d3d4d5Z5Gd6ddej)Z6 7  8  dd9e e0d:fd;e e0d:fde e e+e fde(fd?d@Z7dd9e e0d:fd;e e0d:fde(fdBdCZ8 D E F G H I dd9e e0d:fd;e e0d:fde e e+e fde(fdLdMZ9 D H dd9e e0d:fd;e e0d:fd#e+dNe1de(f dOdPZ:e4didQe:dRdSdTe:dUdSdVe:dWdSdXe:dYdSdZe:d[dSd\e:d]dSd^e:d_dSd`e9dRdSdae9dUdSdbe9dWdSdce9dYdSdde9d[dSdee9d]dSdfe9d_dSdge9dhdSdie9dRdjdkdle4dldmdndodpdqe9dRdjdkdldre4dodsdte9dUdudkdldre4dodsdve9dWdudkdldre4dodsdwe9dYdudkdldre4dodsdxe8dydSdze8d{dSd|e8d}d~dde8dddde8dddde8dddde7ddSde7ddSde7ddSde7ddIe4dddde7ddIe4dddde7ddIe4dddde7ddre4dde7ddre4dde7ddre4dde9dddjdmdle4dldmdndodZ;dde+de1de de6fddZ<dde+de de e+e ffddZ=e%ide=ddddddddde=ddddddddde=ddddddddde=ddddddddde=ddddddddde=ddddddddde=ddddddddd`e=dddddύdae=dddddύdbe=dddddύdce=dddddύdde=dddddύdee=dddddύdfe=dddddύdge=dddddύde=ddddddd֍de=ddddddd֍ide=ddddddd֍de=ddddddd֍dwe=ddddddލdxe=ddddddde=dddddddd|e=ddddddde=ddddddde=ddddddde=ddddddde=dddde=ddddddddde=dddde=dddde=dddde=dddde=dddde=ddde=ddde=ddddddddZ>e&dde1de de6fddQZ?e&dde1de de6fddTZ@e&dde1de de6fddVZAe&dde1de de6fddXZBe&dde1de de6fddZZCe&dde1de de6fdd\ZDe&dde1de de6fdd^ZEe&dde1de de6fdd`ZFe&dde1de de6fddaZGe&dde1de de6fddbZHe&dde1de de6fddcZIe&dde1de de6fdddZJe&dde1de de6fddeZKe&dde1de de6fddfZLe&dde1de de6fddgZMe&dde1de de6fddiZNe&dde1de de6fddqZOe&dde1de de6fddtZPe&dde1de de6fddvZQe&dde1de de6fddwZRe&dde1de de6fddxZSe&dde1de de6fddzZTe&dde1de de6fdd|ZUe&dde1de de6fd dZVe&dde1de de6fd dZWe&dde1de de6fd dZXe&dde1de de6fd dZYe&dde1de de6fd dZZe&dde1de de6fddZ[e&dde1de de6fddZ\e&dde1de de6fddZ]e&dde1de de6fddZ^e&dde1de de6fddZ_e&dde1de de6fddZ`e&dde1de de6fddZae&dde1de de6fddZbdS(a Normalization Free Nets. NFNet, NF-RegNet, NF-ResNet (pre-activation) Models Paper: `Characterizing signal propagation to close the performance gap in unnormalized ResNets` - https://arxiv.org/abs/2101.08692 Paper: `High-Performance Large-Scale Image Recognition Without Normalization` - https://arxiv.org/abs/2102.06171 Official Deepmind JAX code: https://github.com/deepmind/deepmind-research/tree/master/nfnets Status: * These models are a work in progress, experiments ongoing. * Pretrained weights for two models so far, more to come. * Model details updated to closer match official JAX code now that it's released * NF-ResNet, NF-RegNet-B, and NFNet-F models supported Hacked together by / copyright Ross Wightman, 2021. ) OrderedDict) dataclassreplace)partial)AnyCallableDictOptionalTupleNIMAGENET_DEFAULT_MEANIMAGENET_DEFAULT_STD) ClassifierHeadDropPath AvgPool2dSameScaledStdConv2dScaledStdConv2dSame get_act_layer get_act_fnget_attnmake_divisible)build_model_with_cfg)register_notrace_module)checkpoint_seq)generate_default_cfgsregister_model NormFreeNetNfCfgc@s6eZdZUdZeeeeefed<eeeeefed<dZeed<dZ e ed<dZ e eed <dZ e eed <dZe e ed <dZe ee efed <d Zeed<dZeed<dZeed<dZeed<dZeed<dZeed<dZeed<dZeed<dZeed<dZeed<dZeed<dZeed<d Ze ed!<dS)"rz.Configuration for Normalization-Free Networks.depthschannelsg?alpha3x3 stem_typeNstem_chs group_size attn_layer attn_kwargs@ attn_gain? width_factor? bottle_ratior num_featuresch_divFreg extra_conv gamma_in_act same_paddinggh㈵> std_conv_epsskipinit zero_init_fcsilu act_layer) __name__ __module__ __qualname____doc__r int__annotations__r!floatr#strr$r r%r&r'rrr)r+r-r.r0r1boolr2r3r4r5r6r7r9rCrCE/home/ubuntu/.local/lib/python3.10/site-packages/timm/models/nfnet.pyr&s.               csFeZdZdZddededeffdd Zd ej d ej fd d Z Z S)GammaActz.Activation function with gamma scaling factor.relur*Fact_typegammainplacecs$tt||_||_||_dS)zInitialize GammaAct. Args: act_type: Type of activation function. gamma: Scaling factor for activation output. inplace: Whether to perform activation in-place. N)super__init__ract_fnrHrI)selfrGrHrI __class__rCrDrKCs   zGammaAct.__init__xreturncCs|j||jd|jS)zzForward pass. Args: x: Input tensor. Returns: Scaled activation output. rI)rLrImul_rHrMrPrCrCrDforwardPs zGammaAct.forward)rFr*F) r:r;r<r=rAr@rBrKtorchTensorrU __classcell__rCrCrNrDrE@s rEr*rGrHrQcsddtdtffdd }|S)zCreate activation function factory with gamma scaling. Args: act_type: Type of activation function. gamma: Scaling factor for activation output. Returns: Activation function factory. FrIrQcst|dS)N)rHrI)rErRrGrHrCrD_createfszact_with_gamma.._createNF)rBrE)rGrHrZrCrYrDact_with_gamma\s r\c s^eZdZdZdddefdededededeed ef fd d Zd e j d e j fddZ Z S) DownsampleAvgzEAvgPool downsampling as in 'D' ResNet variants with dilation support.rNin_chsout_chsstridedilationfirst_dilation conv_layerc s|tt||dkr |nd}|dks|dkr.|dkr!|dkr!tntj}|d|ddd|_nt|_|||ddd|_dS)a7Initialize DownsampleAvg. Args: in_chs: Input channels. out_chs: Output channels. stride: Stride for downsampling. dilation: Dilation rate. first_dilation: First dilation rate (unused). conv_layer: Convolution layer type. rTF) ceil_modecount_include_pad)r`N) rJr]rKrnn AvgPool2dpoolIdentityconv) rMr^r_r`rarbrc avg_stride avg_pool_fnrNrCrDrKns zDownsampleAvg.__init__rPrQcCs|||S)ztForward pass. Args: x: Input tensor. Returns: Downsampled tensor. )rkrirTrCrCrDrUs zDownsampleAvg.forward) r:r;r<r=rr>r rrKrVrWrUrXrCrCrNrDr]ks(r]c%seZdZdZddddddddddddddded fd ed eed ed edeededededeedededededee dedee de def$fdd Z de j de j fd d!Z ZS)" NormFreeBlockz-Normalization-Free pre-activation block. Nrr*?TFr(r^r_r`rarbr!betar-r%r0r1r2r6r&r)r9rcdrop_path_ratecst|p|}|p |}t| r||n||| }| sdn|| }| r.| | dkr.| |}||_||_||_||ksC|dksC||krOt||||||d|_nd|_||_|||d|_ |dd|_ |||d|||d|_ | r|dd|_ |||dd||d|_ nd|_ d|_ | r|dur|||_nd|_||_|||d| rd nd d |_| s|dur|||_nd|_|dkrt|nt|_| rttd |_dSd|_dS) aInitialize NormFreeBlock. Args: in_chs: Input channels. out_chs: Output channels. stride: Stride for convolution. dilation: Dilation rate. first_dilation: First dilation rate. alpha: Alpha scaling factor for residual. beta: Beta scaling factor for pre-activation. bottle_ratio: Bottleneck ratio. group_size: Group convolution size. ch_div: Channel divisor for rounding. reg: Use RegNet-style configuration. extra_conv: Add extra 3x3 convolution. skipinit: Use skipinit initialization. attn_layer: Attention layer type. attn_gain: Attention gain factor. act_layer: Activation layer type. conv_layer: Convolution layer type. drop_path_rate: Stochastic depth drop rate. rr)r`rarbrcNTrR)r`ragroupsr*rp) gain_init)rJrKrr!rqr)r] downsampleact1conv1act2conv2act2bconv2battnact3conv3 attn_lastrrgrj drop_path ParameterrVtensor skipinit_gain)rMr^r_r`rarbr!rqr-r%r0r1r2r6r&r)r9rcrrmid_chsrtrNrCrDrKsN +        $zNormFreeBlock.__init__rPrQcCs|||j}|}|jdur||}||}|||}|jdur.|||}|jdur;|j ||}| | |}|j durP|j | |}| |}|jdur`||j||j|}|S)zoForward pass. Args: x: Input tensor. Returns: Output tensor. N)rwrqrvrxrzryr|r{r}r)rr~rrrrSr!)rMrPoutshortcutrCrCrDrUs$          zNormFreeBlock.forward)r:r;r<r=rr>r r@rBrrKrVrWrUrXrCrCrNrDrnsr     ZrnTr^r_r#rcr9preact_featurecCsd}t|ddd}t}|dvsJd|vrd|vr:d|vs J|d|d |d|f} d } d }t|ddd d}n$d |vrJd |d|d|f} n |d|d|f} d} t|dddd}t| d} tt| | D]'\} \} }||| d |d|d| d<| | kr|dd|d| d<| }qknd|vr|||d dd|d<n |||ddd|d<d|vrtjd ddd|d<d }t|||fS)aCreate stem module for NFNet models. Args: in_chs: Input channels. out_chs: Output channels. stem_type: Type of stem ('', 'deep', 'deep_tiered', 'deep_quad', '3x3', '7x7', etc.). conv_layer: Convolution layer type. act_layer: Activation layer type. preact_feature: Use pre-activation feature. Returns: Tuple of (stem_module, stem_stride, stem_feature_info). rd stem.convnum_chs reductionmodule) rdeep deep_tiered deep_quadr"7x7 deep_pool3x3_pool7x7_poolrquadrir/)rdrrrdz stem.conv3tieredrs)rdrrz stem.conv2r) kernel_sizer`rkTrRactr")r`padding)dictrlen enumerateziprg MaxPool2d Sequential)r^r_r#rcr9r stem_stride stem_featurestemr$strideslast_idxicsrCrCrD create_stems<   rg`U?gyX?g\9?g`aK?g?g?g `l?g`i?g|?g 7@g-?g@g`?g?)identityceluelugelu leaky_relu log_sigmoid log_softmaxrFrelu6selusigmoidr8softsignsoftplustanhcs eZdZdZ      d'deded ed ed ed ed edeffdd Z e j j d(de deeeffddZe j j d)de ddfddZe j j dejfddZd*ded eeddfddZde jde jfd d!Zd(de jd"e de jfd#d$Zde jde jfd%d&ZZS)+ra* Normalization-Free Network As described in : `Characterizing signal propagation to close the performance gap in unnormalized ResNets` - https://arxiv.org/abs/2101.08692 and `High-Performance Large-Scale Image Recognition Without Normalization` - https://arxiv.org/abs/2102.06171 This model aims to cover both the NFRegNet-Bx models as detailed in the paper's code snippets and the (preact) ResNet models described earlier in the paper. There are a few differences: * channels are rounded to be divisible by 8 by default (keep tensor core kernels happy), this changes channel dim and param counts slightly from the paper models * activation correcting gamma constants are moved into the ScaledStdConv as it has less performance impact in PyTorch when done with the weight scaling there. This likely wasn't a concern in the JAX impl. * a config option `gamma_in_act` can be enabled to not apply gamma in StdConv as described above, but apply it in each activation. This is slightly slower, numerically different, but matches official impl. * skipinit is disabled by default, it seems to have a rather drastic impact on GPU memory use and throughput for what it is/does. Approx 8-10% throughput loss. rsavg rpcfg num_classesin_chans global_pool output_stride drop_raterrkwargsc s4t||_||_d|_t|fi|}|jtvs$Jd|jd|jr)t nt } |j r@t |jt|jd} t | |jd} nt|j} t | t|j|jd} |jr_t t|jfi|jnd} t|jpi|jd|j|j} t|| |j| | d \|_} }|g|_d d td|t|j |jD}| }| }d }d }g}t!|jD]\}}|dkr| dkrd nd}||kr|d kr||9}d }||9}|dvrd nd}g}t"|j|D]}|dko|dk}t|j||j|j}|t#d/id|d|d|j$dd |dd|dkr |nd d|d|d|j%d|j&r |r d n|j'd|jd|j&d|j(d|j)d| d|j*d | d!| d"|||g7}|dkrRd }||j$d7}|}|}q|jt+||d#|d$g7_|t,j-|g7}qt,j-||_.|j/rt|j|j/|j|_/| ||j/d |_0t+|j/|d%d$|jd&<n||_/t,1|_0| |j/dkd'|_2|j/|_3t4|j/|||jd(|_5|6D]S\}}d)|vrt7|t,j8r|j9rt,j:;|j<n t,j:=|jdurt,j:;|j>qt7|t,j?rt,j:j@|jdurt,j:;|j>qdS)0a Args: cfg: Model architecture configuration. num_classes: Number of classifier classes. in_chans: Number of input channels. global_pool: Global pooling type. output_stride: Output stride of network, one of (8, 16, 32). drop_rate: Dropout rate. drop_path_rate: Stochastic depth drop-path rate. **kwargs: Extra kwargs overlayed onto cfg. Fz3Please add non-linearity constants for activation (z).)rH)eps)rHrNr)rcr9cSsg|]}|qSrC)tolist).0rPrCrCrD sz(NormFreeNet.__init__..rr*rd)rrdr^r_r!rqr,r`rarbr%r-r0r1r2r6r&r)r9rcrrzstages.r final_convrR) pool_typerfcrpg{Gz?fan_inlinear)mode nonlinearityrC)ArJrKrrgrad_checkpointingrr9 _nonlin_gammar4rrr3r\rr5rr&rr'rr$r r+r0rr#r feature_inforVlinspacesumrsplitrrangernr!r%r1r-r2r6r)rrgrstagesr.rrj final_acthead_hidden_sizerhead named_modules isinstanceLinearr7initzeros_weightnormal_biasConv2dkaiming_normal_)rMrrrrrrrrrrcr9r&r$r stem_featdrop_path_ratesprev_chs net_stridera expected_varr stage_idx stage_depthr`rbblocks block_idx first_blockr_nmrNrCrDrK}s  "&              zNormFreeNet.__init__FcoarserQcCs td|rdnddfdgd}|S)z"Group parameters for optimization.z^stemz^stages\.(\d+)z^stages\.(\d+)\.(\d+)N)z ^final_conv)i)rr)r)rMrmatcherrCrCrD group_matcherszNormFreeNet.group_matcherTenableNcCs ||_dS)z)Enable or disable gradient checkpointing.N)r)rMrrCrCrDset_grad_checkpointings z"NormFreeNet.set_grad_checkpointingcCs|jjS)zGet the classifier head.)rr)rMrCrCrDget_classifier szNormFreeNet.get_classifiercCs||_|j||dS)zReset the classifier head. Args: num_classes: Number of classes for new classifier. global_pool: Global pooling type. N)rrreset)rMrrrCrCrDreset_classifierszNormFreeNet.reset_classifierrPcCsJ||}|jrtjst|j|}n||}||}||}|S)zForward pass through feature extraction layers. Args: x: Input tensor. Returns: Feature tensor. ) rrrVjit is_scriptingrrrrrTrCrCrDforward_featuress    zNormFreeNet.forward_features pre_logitscCs|r |j||dS||S)zForward pass through classifier head. Args: x: Input features. pre_logits: Return features before final linear layer. Returns: Classification logits or features. )r)r)rMrPrrCrCrD forward_head,s zNormFreeNet.forward_headcCs||}||}|S)zoForward pass. Args: x: Input tensor. Returns: Output logits. )rrrTrCrCrDrU8s zNormFreeNet.forward)rrsrrrprpr[)T)N)r:r;r<r=rr>rAr@rrKrVrignorerBrrrrgModulerr rrWrrrUrXrCrCrNrDrgsF }   iirFr.r r%r&r'c Cs&|pi}t||ddd||||d }|S)arCreate NFNet ResNet configuration. Args: depths: Number of blocks in each stage. channels: Channel dimensions for each stage. group_size: Group convolution size. act_layer: Activation layer type. attn_layer: Attention layer type. attn_kwargs: Attention layer arguments. Returns: NFNet configuration. r@ro) rr r#r$r-r%r9r&r')r)rr r%r9r&r'rrCrCrD _nfres_cfgFs r0hc Cs:d|dd}tdd}t||dddd |d d |d }|S) zCreate NFNet RegNet configuration. Args: depths: Number of blocks in each stage. channels: Channel dimensions for each stage. Returns: NFNet configuration. irrr,rd_ratior"r/g?g@Tse) rr r#r%r+r-r.r1r&r')rr)rr r.r'rrCrCrD _nfreg_cfgjs  r rrrr,r(rr r- feat_multc CsHt|d|}|dur|ntdd}t||dd||d||||d } | S) aCreate NFNet configuration. Args: depths: Number of blocks in each stage. channels: Channel dimensions for each stage. group_size: Group convolution size. bottle_ratio: Bottleneck ratio. feat_mult: Feature multiplier for final layer. act_layer: Activation layer type. attn_layer: Attention layer type. attn_kwargs: Attention layer arguments. Returns: NFNet configuration. rNr,r rrT) rr r#r$r%r-r2r.r9r&r')r>rr) rr r%r-rr9r&r'r.rrCrCrD _nfnet_cfgs  rr6cCs:t||ddddddd|t|dd|dtddd }|S) aCreate DeepMind NFNet configuration. Args: depths: Number of blocks in each stage. channels: Channel dimensions for each stage. act_layer: Activation layer type. skipinit: Use skipinit initialization. Returns: NFNet configuration. rrr,Trr(r r )rr r#r$r%r-r2r3r4r6r.r9r&r')rr>r)rr r9r6rrCrCrD _dm_nfnet_cfgs"r dm_nfnet_f0)rrdrs)r dm_nfnet_f1)rdr r dm_nfnet_f2)rsr dm_nfnet_f3)rr/r dm_nfnet_f4)  dm_nfnet_f5)rr$r dm_nfnet_f6)r*nfnet_f0nfnet_f1nfnet_f2nfnet_f3nfnet_f4nfnet_f5nfnet_f6nfnet_f7)r/rrnfnet_l0g?rror/)r  rd_divisorr8)rrr%r-r'r9 eca_nfnet_l0eca)rrr%r-r&r'r9 eca_nfnet_l1rd eca_nfnet_l2 eca_nfnet_l3 nf_regnet_b0)rrsrr nf_regnet_b1)rdrrr nf_regnet_b2)rdrr/r/)8pi)rr nf_regnet_b3)rdrrr)r:ri nf_regnet_b4)rdr r@)r ih nf_regnet_b5)rsrr$r$)PiPi nf_resnet26)rdrdrdrd nf_resnet50)rsrrrs nf_resnet101)rsrrs nf_seresnet26g?r )rr&r' nf_seresnet50nf_seresnet101nf_ecaresnet26nf_ecaresnet50nf_ecaresnet101 test_nfnet)rrrr)rr`r)rr rr%r-r'r9Fvariant pretrainedrcKs,t|}tdd}tt||f||d|S)zCreate a NormFreeNet model. Args: variant: Model variant name. pretrained: Load pretrained weights. **kwargs: Additional model arguments. Returns: NormFreeNet model instance. T)flatten_sequential) model_cfg feature_cfg) model_cfgsrrr)rRrSrrUrVrCrCrD_create_normfreenets rXurlc Ks|dddddttddd |S) zCreate default configuration dictionary. Args: url: Model weight URL. **kwargs: Additional configuration options. Returns: Configuration dictionary. rrsr[rr?bicubicz stem.conv1zhead.fc) rYr input_size pool_sizecrop_pct interpolationmeanstd first_conv classifierr )rYrrCrCrD_dcfg,s rgzdm_nfnet_f0.dm_in1kztimm/zmhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-dnf-weights/dm_nfnet_f0-604f9c3a.pth)rr)rsrh)rsrrr]squash) hf_hub_idrYr`r_test_input_sizera crop_modezdm_nfnet_f1.dm_in1kzmhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-dnf-weights/dm_nfnet_f1-fc540f82.pthr\rZ)rs@rmgQ?zdm_nfnet_f2.dm_in1kzmhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-dnf-weights/dm_nfnet_f2-89875923.pth)r/r/)rs`rngq= ףp?zdm_nfnet_f3.dm_in1kzmhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-dnf-weights/dm_nfnet_f3-d74ab3aa.pth)rr)rsrogGz?zdm_nfnet_f4.dm_in1kzmhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-dnf-weights/dm_nfnet_f4-0ac5b10b.pth)rr)rsrp)rsrrg;On?zdm_nfnet_f5.dm_in1kzmhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-dnf-weights/dm_nfnet_f5-ecb20ab1.pth) rq)rs rrgI +?zdm_nfnet_f6.dm_in1kzmhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-dnf-weights/dm_nfnet_f6-e0f12116.pth)r$r$)rsrs)rs@rtgd;O?)rYr`r_rk)r r )rsru)rs`rvznfnet_l0.ra2_in1kzjhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights/nfnet_l0_ra2-45c6688d.pth)rsrBrB)rjrYr`r_rk test_crop_pctzeca_nfnet_l0.ra2_in1kzmhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights/ecanfnet_l0_ra2-e3e9ac50.pthzeca_nfnet_l1.ra2_in1kzmhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights/ecanfnet_l1_ra2-7dce93cd.pthzeca_nfnet_l2.ra3_in1kzmhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights/ecanfnet_l2_ra3-da781a61.pth)r@r@)rYr`r_rkrwr)rYr`r_rkreznf_regnet_b1.ra2_in1kzrhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights/nf_regnet_b1_256_ra2-ad85cfef.pth)rjrYr`r_rkre)rsrx)rsry)rr)rsrz)rYreznf_resnet50.ra2_in1kzmhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights/nf_resnet50_ra2-9f236009.pth)rjrYr`r_rkrare)r,r,r,gffffff?)rsr{)rr)rjrcrdrar_r`)rOztest_nfnet.r160_in1kcKtdd|i|S)z&NFNet-F0 (DeepMind weight compatible).rrSN)rrXrSrrCrCrDrcKr|)z&NFNet-F1 (DeepMind weight compatible).rrSN)rr}r~rCrCrDrrcKr|)z&NFNet-F2 (DeepMind weight compatible).rrSN)rr}r~rCrCrDrrcKr|)z&NFNet-F3 (DeepMind weight compatible).rrSN)rr}r~rCrCrDrrcKr|)z&NFNet-F4 (DeepMind weight compatible).rrSN)rr}r~rCrCrDrrcKr|)z&NFNet-F5 (DeepMind weight compatible).r!rSN)r!r}r~rCrCrDr!rcKr|)z&NFNet-F6 (DeepMind weight compatible).r#rSN)r#r}r~rCrCrDr#rcKr|)z NFNet-F0.r'rSN)r'r}r~rCrCrDr'rcKr|)z NFNet-F1.r(rSN)r(r}r~rCrCrDr(rcKr|)z NFNet-F2.r)rSN)r)r}r~rCrCrDr)rcKr|)z NFNet-F3.r*rSN)r*r}r~rCrCrDr*rcKr|)z NFNet-F4.r+rSN)r+r}r~rCrCrDr+rcKr|)z NFNet-F5.r,rSN)r,r}r~rCrCrDr,rcKr|)z NFNet-F6.r-rSN)r-r}r~rCrCrDr-rcKr|)z NFNet-F7.r.rSN)r.r}r~rCrCrDr.rcKr|)zNFNet-L0b w/ SiLU. My experimental 'light' model w/ F0 repeats, 1.5x final_conv mult, 64 group_size, .25 bottleneck & SE ratio r0rSN)r0r}r~rCrCrDr0cKr|)zECA-NFNet-L0 w/ SiLU. My experimental 'light' model w/ F0 repeats, 1.5x final_conv mult, 64 group_size, .25 bottleneck & ECA attn r2rSN)r2r}r~rCrCrDr2 rcKr|)zECA-NFNet-L1 w/ SiLU. My experimental 'light' model w/ F1 repeats, 2.0x final_conv mult, 64 group_size, .25 bottleneck & ECA attn r4rSN)r4r}r~rCrCrDr4rcKr|)zECA-NFNet-L2 w/ SiLU. My experimental 'light' model w/ F2 repeats, 2.0x final_conv mult, 64 group_size, .25 bottleneck & ECA attn r5rSN)r5r}r~rCrCrDr5rcKr|)zECA-NFNet-L3 w/ SiLU. My experimental 'light' model w/ F3 repeats, 2.0x final_conv mult, 64 group_size, .25 bottleneck & ECA attn r6rSN)r6r}r~rCrCrDr6&rcKr|)z"Normalization-Free RegNet-B0. r7rSN)r7r}r~rCrCrDr7/cKr|)z"Normalization-Free RegNet-B1. r8rSN)r8r}r~rCrCrDr86rcKr|)z"Normalization-Free RegNet-B2. r9rSN)r9r}r~rCrCrDr9=rcKr|)z"Normalization-Free RegNet-B3. r=rSN)r=r}r~rCrCrDr=DrcKr|)z"Normalization-Free RegNet-B4. r?rSN)r?r}r~rCrCrDr?KrcKr|)z"Normalization-Free RegNet-B5. rCrSN)rCr}r~rCrCrDrCRrcKr|)z"Normalization-Free ResNet-26. rFrSN)rFr}r~rCrCrDrFYrcKr|)z"Normalization-Free ResNet-50. rGrSN)rGr}r~rCrCrDrG`rcKr|)z#Normalization-Free ResNet-101. rHrSN)rHr}r~rCrCrDrHgrcKr|)zNormalization-Free SE-ResNet26.rJrSN)rJr}r~rCrCrDrJnrcKr|)zNormalization-Free SE-ResNet50.rKrSN)rKr}r~rCrCrDrKtrcKr|)z Normalization-Free SE-ResNet101.rLrSN)rLr}r~rCrCrDrLzrcKr|)z Normalization-Free ECA-ResNet26.rMrSN)rMr}r~rCrCrDrMrcKr|)z Normalization-Free ECA-ResNet50.rNrSN)rNr}r~rCrCrDrNrcKr|)z!Normalization-Free ECA-ResNet101.rOrSN)rOr}r~rCrCrDrOrcKr|)z%Test NFNet model for experimentation.rPrSN)rPr}r~rCrCrDrPr)r*)rNNT)rNrFNN)r)r rr,r(rr N)r rTrCr[)r)cr= collectionsr dataclassesrr functoolsrtypingrrrr r rVtorch.nnrg timm.datar r timm.layersrrrrrrrrr_builderr _features_fxr _manipulater _registryrr__all__rrrErAr@r\r]rnr>rBrrrrrrr rrrWrXrg default_cfgsrrrrrr!r#r'r(r)r*r+r,r-r.r0r2r4r5r6r7r8r9r=r?rCrFrGrHrJrKrLrMrNrOrPrCrCrCrDsv   ,   + =b   *$   -   $                ! ' ( ) * + , / 0 1345789;  A"          "$&(*,/37;?C EI K M O  RS  W Y Z [ ] ^ _ h"