o پi @sdZddlZddlmZddlmZmZmZddlm Z ddl m Z m Z m Z mZmZmZmZmZddlZddlmZddlmZdd lmZmZdd lmZmZmZmZmZm Z dd lm!Z!m"Z"m#Z#m$Z$m%Z%m&Z&dd lm'Z'm(Z(m)Z)m*Z*m+Z+dd lm,Z,m-Z-m.Z.m/Z/m0Z0ddl1m2Z2ddl3m4Z4ddl5m6Z6ddl7m8Z8m9Z9ddl:m;Z;meGdddZ?eGdddZ@GdddejAZBGdddejAZCGdd d ejAZDGd!d"d"ejAZEGd#d$d$ejAZFdd&ejAd'eGd(eGd)dfd*d+ZHGd,d-d-ejAZIdd&ejAd'eGd(eGd)dfd.d/ZJd0eeKd1eKd)eKfd2d3ZLGd4d5d5ejAZMGd6d7d7ejAZNd8ejOd9eeKd)ejOfd:d;ZPe6dd?ZQd8ejOd@eeKd)ejOfdAdBZRe6dd9eeKeKfd)ee fdFdGZTGdHdIdIejAZUGdJdKdKejAZVd8ejOd9eeKd)ejOfdLdMZWe6dd=eeKeKfd)e>fd^d_Z`dEe@d`e d)e@fdadbZaGdcddddejAZb e f g g h % i j k  l mddneGdoeGdpecdqecdreGdseGdtecdueGdveGdweeddxeGdyeKd)e eGe ffdzd{Ze | f g } % j k  ~  l mddneGdoeGdpecdeddseGdueGdveGd9eeeKeKfdeKdweeddxeGdyeKd)e eGe ffddZf | f j k j k  g mddneGdoeGdseGdeGdueGdveGd9eeeKeKfdecdweedeededffdxeGdyeKd)e eGe ffddZgd)e eGe ffddZheidide@dddddefdiddde@dddddefdediddde@dddddeedidgdde@dddddeed|didgdde@dddddeed|ddde@dddddeed|dddde@dddddeed|didgddde@dddddefdiddddde@dddddeed|ddde@dddddeeddidgdddde@dddddeed|ddmdde@dddddeed|ddddde@dddddeed|ddddde@ddddddeede@ddddddegdlddde@dddddde@dddddde@dddddde@dddddde@dddddde@dddddde@ddddddҜefde@ddddddҜefde@ddddddҜefde@ddddddҜefde@ddddddҜefddٍde@ddddddҜefddٍde@ddddddҜefddٍde@ddddddҜefdddde@dddddddޜefddٍde@dddddddegde@ddddddҜegde@ddddddҜegde@dddddddegdidldde@ddddddҜegdidde@dddddddޜegdidde@ddddddidmdehde@ddddddiddehde@ddddddiddehde@ddddddiddehde@ddddddiddehZjde eGejOfdejAd)e eGejOffddZkddeGdeeGdecd`e d)ebf ddZlddeGd`e d)e eGe ffddZme;idemd%ddemdddddemddd d emdd d d emdd demdd demdd demdddddddemddeedddemdddddemd%ddemddd demdd d d!emd%dd"emd%dd#emdd$ddd%emdd&d'id(emdd&d'd)emdd&d'd*emdd&d'd+emd%dd,emd%dd-emd%dd.emd%dd/emd%dd0emd%dd1emd%d2d3d4d5emdd6d2d3d7d8emdd9d d:emd%d2d3d4d;emd%d2d3d4demdd?d2d3d7d@emddAd2d3d7idBemddCdddDemd%d2d3d4dEemdd dFemdddddddGemdd&d'dHemddId2d3d7dJemd%d2d3d4dKemddLd2d3d7dMemdd2d3dNdOemdd dPemdddddddQemd%ddRemdd&d'dSemdeedTdUemdddddddVemddWdXddddYemdeedTidZemddddddd[emddWdXdddd\emdeedTd]emddddddd^emddWdXdddd_emdeedTd`emdddddddaemddWdXddddbemddcd'ddemdddddddeemddWdXddddfemddcd'dgemdddddddhemddWdddidjemddcd'dkemdddddddlemddWdXdddZne<ddecd`e d)ebfdmdnZoe<ddecd`e d)ebfdodpZpe<ddecd`e d)ebfdqdrZqe<ddecd`e d)ebfdsdtZre<ddecd`e d)ebfdudvZse<ddecd`e d)ebfdwdxZte<ddecd`e d)ebfdydzZue<ddecd`e d)ebfd{d|Zve<ddecd`e d)ebfd}d~Zwe<ddecd`e d)ebfddZxe<ddecd`e d)ebfddZye<ddecd`e d)ebfddZze<ddecd`e d)ebfddZ{e<ddecd`e d)ebfddZ|e<ddecd`e d)ebfddZ}e<ddecd`e d)ebfddZ~e<ddecd`e d)ebfddZe<ddecd`e d)ebfddZe<ddecd`e d)ebfddZe<ddecd`e d)ebfddZe<ddecd`e d)ebfddZe<ddecd`e d)ebfddZe<ddecd`e d)ebfddZe<ddecd`e d)ebfddZe<ddecd`e d)ebfddZe<ddecd`e d)ebfddZe<ddecd`e d)ebfddZe<ddecd`e d)ebfddZe<ddecd`e d)ebfddZe<ddecd`e d)ebfddZe<ddecd`e d)ebfddZe<ddecd`e d)ebfddZe<ddecd`e d)ebfddZe<ddecd`e d)ebfddZe<ddecd`e d)ebfddZe<ddecd`e d)ebfddZe<ddecd`e d)ebfddZe<ddecd`e d)ebfddZe<ddecd`e d)ebfddZe<ddecd`e d)ebfddZe<ddecd`e d)ebfddZe<ddecd`e d)ebfddZe<ddecd`e d)ebfdd„Ze<ddecd`e d)ebfdÐdĄZe<ddecd`e d)ebfdŐdƄZe<ddecd`e d)ebfdǐdȄZe<ddecd`e d)ebfdɐdʄZe<ddecd`e d)ebfdːd̄Ze<ddecd`e d)ebfd͐d΄Ze<ddecd`e d)ebfdϐdЄZe<ddecd`e d)ebfdѐd҄Ze<ddecd`e d)ebfdӐdԄZe<ddecd`e d)ebfdՐdքZe<ddecd`e d)ebfdאd؄Ze<ddecd`e d)ebfdِdڄZe<ddecd`e d)ebfdېd܄ZdS(a MaxVit and CoAtNet Vision Transformer - CNN Hybrids in PyTorch This is a from-scratch implementation of both CoAtNet and MaxVit in PyTorch. 99% of the implementation was done from papers, however last minute some adjustments were made based on the (as yet unfinished?) public code release https://github.com/google-research/maxvit There are multiple sets of models defined for both architectures. Typically, names with a `_rw` suffix are my own original configs prior to referencing https://github.com/google-research/maxvit. These configs work well and appear to be a bit faster / lower resource than the paper. The models without extra prefix / suffix' (coatnet_0_224, maxvit_tiny_224, etc), are intended to match paper, BUT, without any official pretrained weights it's difficult to confirm a 100% match. Papers: MaxViT: Multi-Axis Vision Transformer - https://arxiv.org/abs/2204.01697 @article{tu2022maxvit, title={MaxViT: Multi-Axis Vision Transformer}, author={Tu, Zhengzhong and Talebi, Hossein and Zhang, Han and Yang, Feng and Milanfar, Peyman and Bovik, Alan and Li, Yinxiao}, journal={ECCV}, year={2022}, } CoAtNet: Marrying Convolution and Attention for All Data Sizes - https://arxiv.org/abs/2106.04803 @article{DBLP:journals/corr/abs-2106-04803, author = {Zihang Dai and Hanxiao Liu and Quoc V. Le and Mingxing Tan}, title = {CoAtNet: Marrying Convolution and Attention for All Data Sizes}, journal = {CoRR}, volume = {abs/2106.04803}, year = {2021} } Hacked together by / Copyright 2022, Ross Wightman N) OrderedDict) dataclassreplacefield)partial)AnyCallableDictListOptionalSetTupleUnion)nn)Final)IMAGENET_DEFAULT_MEANIMAGENET_DEFAULT_STD)MlpConvMlpDropPath LayerNormClassifierHeadNormMlpClassifierHead) create_attn get_act_layerget_norm_layerget_norm_act_layer create_conv2d create_pool2d)trunc_normal_tf_ to_2tuple extend_tuplemake_divisible_assert) RelPosMlp RelPosBias RelPosBiasTfuse_fused_attnresize_rel_pos_bias_table)build_model_with_cfg)feature_take_indices)register_notrace_function) named_applycheckpoint_seq)generate_default_cfgsregister_model) MaxxVitCfgMaxxVitConvCfgMaxxVitTransformerCfgMaxxVitc@s2eZdZUdZdZeed<dZeed<dZ e ed<dZ eed<dZ eed <dZ eed <d Ze ed <d Ze ed <dZeed<dZeed<dZeed<dZeed<dZeeeefed<dZeeeefed<dZeed<dZeed<dZee ed<dZeed<dZeed<d Zeed!<d"Ze ed#<d$d%Z dS)&r3z-Configuration for MaxxVit transformer blocks. dim_headT head_first@ expand_ratio expand_first shortcut_bias attn_bias attn_drop proj_dropavg2 pool_typebias rel_pos_type rel_pos_dimpartition_ratioN window_size grid_sizeF no_block_attn use_nchw_attn init_valuesgelu act_layer layernorm2d norm_layer layernorm norm_layer_clư>norm_epscCsJ|jdur t|j|_|jdur!t|j|_|jdur#|j|_dSdSdSN)rHr rGselfrWG/home/ubuntu/.local/lib/python3.10/site-packages/timm/models/maxxvit.py __post_init__Vs      z#MaxxVitTransformerCfg.__post_init__)!__name__ __module__ __qualname____doc__r6int__annotations__r7boolr9floatr:r;r<r>r?rAstrrCrErFrGr r rHrIrJrKrMrOrQrSrYrWrWrWrXr3=s0                    r3c@seZdZUdZdZeed<dZeed<dZ e ed<dZ e ed <d Z e ed <d Ze ed <dZe ed<dZeed<dZeed<dZeed<dZeed<d Ze ed<dZeed<dZeed<dZeed<dZeeed<dZeed <dZeed!<dZeed"<d#Zeeed$<d%d&Zd#S)'r2z-Configuration for MaxxVit convolution blocks.mbconv block_typer8r9T expand_output kernel_sizer) group_sizeF pre_norm_act output_biasdw stride_moder@rAdownsample_pool_typepadding attn_earlyse attn_layersiluattn_act_layer? attn_ratiorRrKrLrMrOrQNrScCsf|jdvsJ|jdk}|js|rdnd|_|js|sd|_|jdur*|r'dnd|_|jp/|j|_dS)N)rcconvnextrc batchnorm2drNrPh㈵>rR)rdrOrQrSrmrA)rV use_mbconvrWrWrXrYws   zMaxxVitConvCfg.__post_init__)rZr[r\r]rdrbr_r9rarer`rgr^rhrirjrlrArmrorprrrtrvrKr rMrOrQrSrYrWrWrWrXr2_s.                    r2c@seZdZUdZdZeedfed<dZeedfed<dZ ee e ee dffdfed<d Z e eeeeffed <d Z eed <eed Zeed<eed Zeed<dZeeed<dZe ed<dS)r1z!Configuration for MaxxVit models.`. embed_dimrfrdepths)CrTrrd@ stem_widthF stem_bias)default_factoryconv_cfgtransformer_cfgNhead_hidden_sizevit_eff weight_init)rZr[r\r]rr r^r_rrdrrbrrr`rr2rr3rrr rrWrWrWrXr1s $ r1ceZdZUdZeeed<        ddedeed ed ed ed ed ee de de ffdd Z dde j dee j de j fddZZS) Attention2dz)Multi-head attention for 2D NCHW tensors. fused_attnNr5Tr=dimdim_outr6rBr:r7 rel_pos_clsr>r?c st|p|}|r |n|} | ||_||_||_|d|_t|_tj || dd|d|_ |r7||jdnd|_ t ||_ tj | |d|d|_t | |_dS) Args: dim: Input dimension. dim_out: Output dimension (defaults to input dimension). dim_head: Dimension per attention head. bias: Whether to use bias in qkv and projection. expand_first: Whether to expand channels before or after qkv. head_first: Whether heads are first in tensor layout. rel_pos_cls: Relative position class to use. attn_drop: Attention dropout rate. proj_drop: Projection dropout rate. rfr)rB num_headsN)super__init__rr6r7scaler'rrConv2dqkvrel_posDropoutr>projr? rVrrr6rBr:r7rr>r?dim_attn __class__rWrXrs     zAttention2d.__init__xshared_rel_posreturnc Cs|j\}}}}|jr"||||j|jddjddd\}}} n|||d|j|jdd\}}} |j r|d} |j durF|j } n|durL|} t j jj|dd|dd| dd| |jrm|jjndddd|d||}n9||j}|dd|} |j dur| | } n|dur| |} | jdd} || } | | dd|d||}||}||}|S) Nrfrrr)r= attn_mask dropout_p)shaper7rviewrr6chunkreshapeunbindrrget_biastorchr functionalscaled_dot_product_attention transpose contiguoustrainingr>prsoftmaxrr?) rVrrBrHWqkvr<attnrWrWrXforwards>0(         zAttention2d.forwardNr5TTTNr=r=rTrZr[r\r]rr`r_r^r rrarrTensorr __classcell__rWrWrrXrs>    *'rcr) AttentionClz/Channels-last multi-head attention (B, ..., C).rNr5Tr=rrr6rBr:r7rr>r?c st|p|}|r||kr|n|} | |dksJd| ||_||_||_|d|_t|_tj || d|d|_ |rD||jdnd|_ t ||_ tj | ||d|_t | |_dS)rrz(attn dim should be divisible by head_dimrrfrrN)rrrr6r7rr'rrLinearrrrr>rr?rrrWrXrs    zAttentionCl.__init__rrrc Csj|jd}|jdd}|jr+|||d|j|jdddjddd\}}}n|||dd|j|jdd d\}}}|j rmd}|j durS|j }n|durY|}t jjj|||||jrh|jjndd}n1||j}||d d} |j dur|j | |d } n|dur| |} | jdd} || } | |}|dd|d }||}||}|S) Nrrrfr)rrr=rrr)r)rr7rrrr6rrrrrrrrrrrrr>rrrrr?) rVrrr restore_shaperrrr<rrWrWrXrs: 80        zAttentionCl.forwardrrTrrWrWrrXrs>    *(rcFeZdZdZd dededeffdd Zd ej d ej fd d Z Z S) LayerScalezPer-channel scaling layer.ryFrrKinplacec*t||_t|t||_dSz Args: dim: Number of channels. init_values: Initial scaling value. inplace: Whether to perform inplace operations. Nrrrr ParameterronesgammarVrrKrrrWrXr8 zLayerScale.__init__rrcCs|j}|jr ||S||SrT)rrmul_rVrrrWrWrXrCszLayerScale.forwardryF rZr[r\r]r^rar`rrrrrrWrWrrXr5 rcr) LayerScale2dz)Per-channel scaling layer for 2D tensors.ryFrrKrcrrrrrrWrXrKrzLayerScale2d.__init__rrcCs*|jdddd}|jr||S||S)Nr)r)rrrrrrWrWrXrVszLayerScale2d.forwardrrrWrWrrXrHrrc sTeZdZdZ   ddedededed ef fd d Zd ej d ej fddZ Z S) Downsample2da5A downsample pooling module supporting several maxpool and avgpool modes. * 'max' - MaxPool2d w/ kernel_size 3, stride 2, padding 1 * 'max2' - MaxPool2d w/ kernel_size = stride = 2 * 'avg' - AvgPool2d w/ kernel_size 3, stride 2, padding 1 * 'avg2' - AvgPool2d w/ kernel_size = stride = 2 r@rnTrrrArorBcst|dvs J|dkrtddd|pdd|_n*|dkr*tdd|p%dd |_n|d kr;td ddd |p6dd |_n td d|pAdd |_||krUtj||d|d |_dSt|_dS)z Args: dim: Input dimension. dim_out: Output dimension. pool_type: Type of pooling operation. padding: Padding mode. bias: Whether to use bias in expansion conv. )maxmax2avgr@rrfrr))rgstriderorr)rorF)rgrcount_include_padrorN)rrrpoolrrexpandIdentity)rVrrrArorBrrWrXrds   zDownsample2d.__init__rrcC||}||}|SrT)rrrVrrWrWrXr  zDownsample2d.forward)r@rnT) rZr[r\r]r^rbr`rrrrrrWrWrrXr[s" !rrnmodulenameschemercCst|tjtjfr|dkr&tjj|jdd|jdur$tj|jdSdS|dkrAt |jdd|jdur?tj|jdSdS|dkr\tj |j|jdurZtj|jdSdStj |j|jdurd|vrwtjj|jdddStj|jdSdSdS) z&Initialize transformer module weights.normal{Gz?stdN trunc_normal xavier_normalmlprR) isinstancerrrinitnormal_weightrBzeros_rxavier_normal_xavier_uniform_)rrrrWrWrX_init_transformers.    rc seZdZdZddedfdedededeed ed ef fd d Z dde ddfddZ dde j dee j de j fddZZS)TransformerBlock2daYTransformer block with 2D downsampling. '2D' NCHW tensor layout Some gains can be seen on GPU using a 1D / CL block, BUT w/ the need to switch back/forth to NCHW for spatial pooling, the benefit is minimal so ended up using just this variant for CoAt configs. This impl was faster on TPU w/ PT XLA than the 1D experiment. r)Nr=rrrrcfg drop_pathc sZttt|j|jd}t|j}|dkr:t|||j |j d|_ t td||fdt|||j dfg|_n||ks@Jt |_ |||_t|||j|j|j||j|jd|_|jrft||jdnt |_|d krst|nt |_|||_t|t||j ||jd |_!|jrt||jdnt |_"|d krt||_#d St |_#d S) a Args: dim: Input dimension. dim_out: Output dimension. stride: Stride for downsampling. rel_pos_cls: Relative position class. cfg: Transformer block configuration. drop_path: Drop path rate. epsr)rArBnormdownrA)r6r:rBrr>r?rKr= in_featureshidden_featuresrMdropN)$rrrrrOrSrrMrrAr;shortcutr Sequentialrnorm1rrr6r:r<r>r?rrKrls1r drop_path1norm2rr^r9rls2 drop_path2) rVrrrrrrrOrMrrWrXrsB         $zTransformerBlock2d.__init__rnrrcCttt|d|dSNr)r-rrrVrrWrWrX init_weightszTransformerBlock2d.init_weightsrrc CsN|||||j|||d}||||||}|S)Nr) r rr rr rrrr)rVrrrWrWrXrs* zTransformerBlock2d.forwardrnrT)rZr[r\r]r3r^r rrarrbrrrrrrWrWrrXrs*7*rcCst|tjr|dkr#tjj|jdd|jdur!tj|jdSdS|dkr>t|jdd|jdurpermuterr7rWrWrXrs         zConvNeXtBlock.forward)rZr[r\r]r2r^r r r`rarrrrrrWrWrrXr8vs6  Ar8rrGcCs|j\}}}}t||ddkd|d|ddt||ddkd|d|dd||||d|d||d|d|}|ddddd d d |d|d|}|S) z'Partition into non-overlapping windows.rheight () must be divisible by window ()r)width (rfrrrrr#rrBr)rrGrrrrwindowsrWrWrXwindow_partition ((,,rJrIimg_sizecCsf|\}}|jd}|d||d||d|d|d|}|ddddddd|||}|S)zReverse window partition.rrr)rfrrGrrrrBrrIrGrLrrrrrWrWrXwindow_reverse  ,$rOrHc Cs|j\}}}}t||ddkd|d|dt||ddkd|d|d|||d||d|d||d|}|dddddd d |d|d|}|S) z6Partition into overlapping windows with grid striding.rheight  must be divisible by grid r)width rrGrfrrrH)rrHrrrrrIrWrWrXgrid_partition &&,,rTcCsf|\}}|jd}|d||d||d|d|d|}|ddddddd|||}|S)zReverse grid partition.rrr)rfrGrrrMrIrHrLrrrrrWrWrX grid_reverserPrWrcCsVd}|jdkrtt||jd}|S|jdkrtt|d}|S|jdkr)tt|d}|S)z,Get relative position class based on config.Nr)rG hidden_dimrB)rGbias_tf)rCrr$rEr%r&)rrGrrWrWrXget_rel_pos_clss   rZc sLeZdZdZdedfdedededeffdd Zd d Z d d Z Z S)PartitionAttentionClzRGrid or Block partition + Attn + FFN. NxC 'channels last' tensor layout. blockr=rpartition_typerrc s&ttt|j|jd}t|j}|dk|_t |jr |j n|j |_ t ||j }|||_t|||j|j|j||j|jd|_|jrLt||jdnt|_|dkrYt|nt|_|||_t|t||j ||jd|_!|jr|t||jdnt|_"|dkrt||_#dSt|_#dS)Nrr\r6rBr7rr>r?rr=r)$rrrrrQrSrrMpartition_blockr rGrHpartition_sizerZr rr6r<r7r>r?rrKrrrr rrrrr^r9rrrrVrr]rrrOrMrrrWrXrs8        $zPartitionAttentionCl.__init__cCsb|jdd}|jrt||j}nt||j}||}|jr(t||j|}|St||j|}|S)Nr)rf)rr_rJr`rTrrOrWrVrrL partitionedrWrWrX_partition_attn-  z$PartitionAttentionCl._partition_attnc CD||||||}||||||}|SrTrr rdr rrrrrrWrWrXr<  zPartitionAttentionCl.forward) rZr[r\r]r3r^rbrarrdrrrWrWrrXr[s &r[csbeZdZdZedfdededeffdd Zdej d ej fd d Z dej d ej fd d Z Z S)ParallelPartitionAttentionzQExperimental. Grid and Block partition + single FFN. NxC tensor layout. r=rrrc s^t|ddks Jtt|j|jd}t|j}|j|j ks$Jt |j|_ t ||j }|||_ t||d|j|j|j||j|jd|_t||d|j|j|j||j|jd|_|jrgt||jdnt|_|dkrtt|nt|_|||_t|t||j |||jd|_!|jrt||jdnt|_"|dkrt||_#dSt|_#dS) z Args: dim: Input dimension. cfg: Transformer block configuration. drop_path: Drop path rate. rrrr^rr=)rr out_featuresrMr N)$rrrrrQrSrrMrGrHr r`rZr rr6r<r7r>r? attn_block attn_gridrKrrrr rrrrr^r9rrr)rVrrrrOrMrrrWrXrHsP         $z#ParallelPartitionAttention.__init__rrcCsh|jdd}t||j}||}t||j|}t||j}||}t||j|}tj ||gddS)Nr)rfrr) rrJr`rkrOrTrlrWrcat)rVrrLpartitioned_blockx_windowpartitioned_gridx_gridrWrWrXrds    z*ParallelPartitionAttention._partition_attnc CrfrTrgrrWrWrXrrhz"ParallelPartitionAttention.forward) rZr[r\r]r3r^rarrrrdrrrWrWrrXriBs7 ric Cs|j\}}}}t||ddkd|d|ddt||ddkd|d|dd|||||d|d||d|d}|ddddd d d ||d|d}|S) z#Partition windows for NCHW tensors.rrCrDrEr)rFrrGrfrrrH)rrGrrrrrIrWrWrXwindow_partition_nchwrKrrc Csf|\}}|jd}|d||d||d||d|d}|ddddddd|||}|S)z*Reverse window partition for NCHW tensors.r)rrrfrGrrrMrNrWrWrXwindow_reverse_nchwrPrsc Cs|j\}}}}t||ddkd|d|dt||ddkd|d|d||||d||d|d||d}|dddddd d ||d|d}|S) z Grid partition for NCHW tensors.rrQrRr)rSrfrrrGrrH)rrHrrrrrIrWrWrXgrid_partition_nchwrUrtc Csf|\}}|jd}|d||d||d||d|d}|ddddddd|||}|S)z(Reverse grid partition for NCHW tensors.r)rrrfrGrrrMrVrWrWrXgrid_reverse_nchwrPruc sheZdZdZdedfdedededeffdd Zd e j d e j fd d Z d e j d e j fddZ Z S)PartitionAttention2dzHGrid or Block partition + Attn + FFN. '2D' NCHW tensor layout. r\r=rr]rrc s&ttt|j|jd}t|j}|dk|_t |jr |j n|j |_ t ||j }|||_t|||j|j|j||j|jd|_|jrLt||jdnt|_|dkrYt|nt|_|||_t|t||j ||jd|_!|jr|t||jdnt|_"|dkrt||_#dSt|_#dS)z Args: dim: Input dimension. partition_type: Partition type ('block' or 'grid'). cfg: Transformer block configuration. drop_path: Drop path rate. rr\r^rr=rN)$rrrrrOrSrrMr_r rGrHr`rZr rr6r<r7r>r?rrKrrrr rrrrr^r9rrrrarrWrXrs8        $zPartitionAttention2d.__init__rrcCsb|jdd}|jrt||j}nt||j}||}|jr(t||j|}|St||j|}|S)Nr)rr_rrr`rtrrsrurbrWrWrXrdrez$PartitionAttention2d._partition_attnc CrfrTrgrrWrWrXrrhzPartitionAttention2d.forward)rZr[r\r]r3r^rbrarrrrdrrrWrWrrXrvs -rvc sZeZdZdZdeedfdededededed ef fd d Zdd dZ ddZ Z S) MaxxVitBlockz;MaxVit conv, window partition + FFN , grid partition + FFN.r)r=rrrrrrc st|j|_|jdkrtnt}||||||d|_t|||d}|jr(t nt } |j r/dn| di||_ | dddi||_ dS)a^Initialize MaxxVitBlock. Args: dim: Input channel dimension. dim_out: Output channel dimension. stride: Stride for downsampling. conv_cfg: Configuration for convolutional blocks. transformer_cfg: Configuration for transformer blocks. drop_path: Drop path rate. rwrrrrrrNr]gridrW)rrrJ nchw_attnrdr8r!convdictrvr[rIrkrl) rVrrrrrrconv_clsr5partition_layerrrWrXrs zMaxxVitBlock.__init__rncCsJ|jdurttt|d|jttt|d|jttt|d|jdSr)rkr-rrrlrr|rrWrWrXr#s zMaxxVitBlock.init_weightscCsX||}|js|dddd}|jdur||}||}|js*|dddd}|SrA)r|r{rBrkrlrrWrWrXr)s    zMaxxVitBlock.forwardr) rZr[r\r]r2r3r^rarrrrrWrWrrXrws* rwcsxeZdZdZddeedfdedededed ed ed effd d Zdde ddfddZ de j de j fddZ ZS)ParallelMaxxVitBlockzYMaxVit block with parallel cat(window + grid), one FF. Experimental timm block. r)rr=rrrnum_convrrrc st|jdkr tnt}|dkr2||||||dg} | |||||dg|d7} tj| |_n ||||||d|_t|||d|_ dS)aa Args: dim: Input dimension. dim_out: Output dimension. stride: Stride for first conv block. num_conv: Number of convolution blocks. conv_cfg: Convolution block configuration. transformer_cfg: Transformer block configuration. drop_path: Drop path rate. rwr)rx)rrryN) rrrdr8r!rr r|rir) rVrrrrrrrr~convsrrWrXr=s zParallelMaxxVitBlock.__init__rnrrNcCs,ttt|d|jttt|d|jdSr)r-rrrrr|rrWrWrXr\sz!ParallelMaxxVitBlock.init_weightsrcCs8||}|dddd}||}|dddd}|SrA)r|rBrrrWrWrXr`s  zParallelMaxxVitBlock.forwardr)rZr[r\r]r2r3r^rarrbrrrrrrWrWrrXr7s0 rcseZdZdZddddeedfdeded ed ed eeefd ee ee fd ededee e e fffdd Z de jde jfddZZS) MaxxVitStagezEMaxxVit stage consisting of mixed convolution and transformer blocks.rrG)rrr=r#r$rdepth feat_size block_typesrrrc  std|_t||}g} t|D]p\} } | dkr|nd} | dvs%J| dkrA|jdkr0tnt}| |||| || | dg7} n@| dkrZt||}| t ||| ||| | d g7} n'| d krn| t ||| ||| | d g7} n| d kr| t ||| ||| | d g7} |}qt j | |_d S)a Args: in_chs: Input channels. out_chs: Output channels. stride: Stride for first block. depth: Number of blocks in stage. feat_size: Feature map size. block_types: Block types ('C' for conv, 'T' for transformer, etc). transformer_cfg: Transformer block configuration. conv_cfg: Convolution block configuration. drop_path: Drop path rate(s). Frr))rrMPMrrwrxr)rrrrr)rrrrrN)rrgrad_checkpointingr! enumeraterdr8r!rZrrwrrr blocks)rVr#r$rrrrrrrrit block_strider~rrrWrXrks\        zMaxxVitStage.__init__rrcCs.|jrtjst|j|}|S||}|SrT)rrjit is_scriptingr.rrrWrWrXrs   zMaxxVitStage.forward)rZr[r\r]r3r2r^r rrbrar rrrrrrWrWrrXrhs:   HrcszeZdZdZ      dded ed ed ed ed ededeffdd ZddeddfddZ de j de j fddZ Z S)Stemz"Stem layer for feature extraction.rfrnFrLrxryr#r$rgrorBrMrOrSc stt|ttfst|}tt|||d} |d|_d|_ t ||d|d||d|_ | |d|_ t |d|d|d||d|_ dS)ae Args: in_chs: Input channels. out_chs: Output channels. kernel_size: Kernel size for convolutions. padding: Padding mode. bias: Whether to use bias. act_layer: Activation layer. norm_layer: Normalization layer. norm_eps: Normalization epsilon. rrrr)rrorBr)N)rrrlisttupler rrr$rrconv1r conv2) rVr#r$rgrorBrMrOrSr/rrWrXrs  "z Stem.__init__rrNcCrrr6rrWrWrXrrzStem.init_weightsrcC"||}||}||}|SrT)rr rrrWrWrXr   z Stem.forward)rfrnFrLrxryr)rZr[r\r]r^rbr`rarrrrrrrWrWrrXrs6 "rcCsF|jdur |js J|S|d|j|d|jf}t|||d}|S)z>Configure window size based on image size and partition ratio.Nrr))rGrH)rGrHrFr)rrLr`rWrWrXcfg_window_sizes  rkwargscKsi}i}i}|D]$\}}|dr|||dd<q |dr*|||dd<q |||<q t|ft|jfi|t|jfi|d|}|S)z-Overlay keyword arguments onto configuration. transformer_rnconv_)rr)items startswithrrr)rrtransformer_kwargs conv_kwargs base_kwargsrrrWrWrX_overlay_kwargss$   rcseZdZdZ      d=dedeeeeeffd ed ed ed e d e de ffdd Z d>de j dededdfddZejjdeefddZejjd?dedeee ffddZejjd@d eddfd!d"Zejjde j fd#d$ZdAd ed eeddfd%d&Z    ' dBd(ejd)eeeeefd*ed+ed,ed-edeeejeejeejfffd.d/Z 0  dCd)eeeefd1ed2edeed3ffd4d5Zd(ejdejfd6d7Zd?d(ejd8edejfd9d:Z d(ejdejfd;d<Z!Z"S)Dr4z{CoaTNet + MaxVit base model. Highly configurable for different block compositions, tensor layouts, pooling types. rfrr=rrLin_chans num_classes global_pool drop_ratedrop_path_raterc sdtt|}|rt|fi|}t|j|} ||_||_|jd|_ |_||_ d|_ g|_ t ||j|jj|j|jj|jj|jjd|_|jj} |j t|jjdddg7_ tddt|t| D} t|j} t|j| ksvJd dtd |t|j |jD} |jj}g}t!| D]B}d|j|}tfd d| D} |t"|||j||j#||j| | | |d g7}| 9} |}|j t|| d |dg7_ qt$j%||_&t't(|jj|jjd}|j)rt$*|_+|j)|_)t,|j ||j)|||d|_-n|j |_)||j |_+t.|j |||d|_-|j/dvsJ|j/r0t0t'|j1|j/d|dSdS)a Args: cfg: Model configuration. img_size: Input image size. in_chans: Number of input channels. num_classes: Number of classification classes. global_pool: Global pooling type. drop_rate: Dropout rate. drop_path_rate: Drop path rate. **kwargs: Additional keyword arguments to overlay on config. rF)r#r$rorBrMrOrSrstem)num_chs reductionrcSsg|]\}}||qSrWrW).0rsrWrWrX <sz$MaxxVit.__init__..cSsg|]}|qSrW)tolist)rrrWrWrXr@srcsg|] }|ddqS)r)rW)rr stage_striderWrXrFs)rrrrrrzstages.r) hidden_sizerArrO)rAr)rnrrrrrN)2rrr rrrrrr num_featuresrr feature_inforrrrorrMrOrSrrr}r$rziplenrrlinspacesumsplitrangerrdrr stagesrrrrrrheadrrr- _init_weights)rVrrLrrrrrrrrr num_stagesdprr#rrr$final_norm_layerrrrXrs    &    "    zMaxxVit.__init__rnrrrrNcCs>t|drz |j|dWdSty|YdSwdS)Nrr)hasattrr TypeError)rVrrrrWrWrXrms  zMaxxVit._init_weightscCsdd|DS)Ncs*h|]\}tfdddDrqS)c3s|]}|vVqdSrTrW)rnrrWrX xsz4MaxxVit.no_weight_decay...)relative_position_bias_tablez rel_pos.mlp)any)r_rWrrX vs z*MaxxVit.no_weight_decay..)named_parametersrUrWrWrXno_weight_decaytszMaxxVit.no_weight_decayFcoarsecCstdddgd}|S)Nz^stem)z^stages\.(\d+)N)z^norm)i)rr)r})rVrmatcherrWrWrX group_matcherzs zMaxxVit.group_matcherTenablecCs|jD]}||_qdSrT)rr)rVrrrWrWrXset_grad_checkpointings zMaxxVit.set_grad_checkpointingcCs|jjSrT)rfcrUrWrWrXget_classifierszMaxxVit.get_classifiercCs||_|j||dSrT)rrreset)rVrrrWrWrXreset_classifierszMaxxVit.reset_classifierNCHWrindicesr stop_early output_fmtintermediates_onlycCs|dvsJdg}tt|jd|\}} d} ||}| |vr&||t|j} tjs2|s6|j} n|jd| } | D]!} | d7} | |}| |vr`|rY| | krY||}n|}||q?|re|S| | krn||}||fS)a Forward features that returns intermediates. Args: x: Input image tensor indices: Take last n blocks if int, all if None, select matching indices if sequence norm: Apply norm layer to compatible intermediates stop_early: Stop iterating over blocks when last desired intermediate hit output_fmt: Shape of intermediate feature outputs intermediates_only: Only return intermediate features Returns: )rzOutput shape must be NCHW.r)rN) r+rrrappendrrrr)rVrrrrrr intermediates take_indices max_indexfeat_idxlast_idxrstagex_interrWrWrXforward_intermediatess2       zMaxxVit.forward_intermediatesr) prune_norm prune_head.cCsLtt|jd|\}}|jd||_|rt|_|r$|dd|_|S)z6Prune layers not required for specified intermediates.r)Nrrn)r+rrrrrrr)rVrrrrrrWrWrXprune_intermediate_layerss z!MaxxVit.prune_intermediate_layerscCrrT)rrrrrWrWrXforward_featuresrzMaxxVit.forward_features pre_logitscCs|r |j||dS||S)N)r)r)rVrrrWrWrX forward_headszMaxxVit.forward_headcCrrT)rrrrWrWrXrrzMaxxVit.forward)rrfrrr=r=rF)TrT)NFFrF)r)FT)#rZr[r\r]r1rr^r rbrarrrModulerrrignorer rr`r rrrr rrr rrrrrrrWrWrrXr4 s ^  8  r4rr@FreluTrNrPrBrDrlrAconv_output_biasconv_attn_earlyconv_attn_act_layerconv_norm_layertransformer_shortcut_biastransformer_norm_layertransformer_norm_layer_clrKrCrEc Cs6tt||dd|||d|d td||| ||| | ddS)asRW variant configuration for CoAtNet models. These models were created and trained before seeing https://github.com/google-research/maxvit Common differences for initial timm models: - pre-norm layer in MZBConv included an activation after norm - mbconv expansion calculated from input instead of output chs - mbconv shortcut and final 1x1 conv did not have a bias - SE act layer was relu, not silu - mbconv uses silu in timm, not gelu - expansion in attention block done via output proj, not input proj Variable differences (evolved over training initial models): - avg pool with kernel_size=2 favoured downsampling (instead of maxpool for coat) - SE attention was between conv2 and norm/act - default to avg pool for mbconv downsample instead of 1x1 or dw conv - transformer block shortcut has no bias TFrs) rlrArirerjrprtrMrO)r:r;rArKrOrQrCrErrr}r2r3) rlrArrrrrrrrKrCrErWrWrX _rw_coat_cfgs.  rrk?r5conv_attn_ratior6c Cs4tt||d||d|dtd|||| ||| | d dS)aRW variant configuration for MaxViT models. These models were created and trained before seeing https://github.com/google-research/maxvit Differences of initial timm models: - mbconv expansion calculated from input instead of output chs - mbconv shortcut and final 1x1 conv did not have a bias - mbconv uses silu in timm, not gelu - expansion in attention block done via output proj, not input proj Frs)rlrArerjrvrMrO) r:rAr6rGrKrOrQrCrErr) rlrArrrrrrGr6rKrCrErWrWrX _rw_max_cfgs, rrRrconv_norm_layer_clrIc CsDt|}ttd||d|d||dtd||||d||| | d dS)z=Configuration for experimental ConvNeXt-based MaxxViT models.rwFr)rdrlrArerKrOrQr)) r:rArGrIrKrOrQrCrEr)r r}r2r3) rlrArrrrrGrIrKrCrErWrWrX _next_cfgLs. rcCs"ttddddtdddddd S) z0Configuration matching TensorFlow MaxViT models.gMbP? gelu_tanhsame)rSrMroryFrY)rSrMr7rCrrrWrWrWrX_tf_cfgssrcoatnet_pico_rw)rrDr)r5r)rrrru)rrcoatnet_nano_rw)rfrGrf)rlrr coatnet_0_rwr{)rrfr9r)rr coatnet_1_rw)rrrr)rlrr coatnet_2_rw)rrrD)rrrs)rlr coatnet_3_rw)r}r~r)r|r})rlrrKcoatnet_bn_0_rwrx)rlrrrcoatnet_rmlp_nano_rwr~)rrrCrEcoatnet_rmlp_0_rw)rlrCcoatnet_rmlp_1_rwr)rArrrCrEcoatnet_rmlp_1_rw2)rlrCrEcoatnet_rmlp_2_rw)rlrrKrCcoatnet_rmlp_3_rwcoatnet_nano_cc)rrrrr)rrrrdcoatnext_nano_rwr)rrrr)ryN)rCrK coatnet_0rr)rrrr coatnet_1 coatnet_2rr  coatnet_3r}r  coatnet_4)r r coatnet_5)rrDrmaxvit_pico_rw)r5rrr)rrrr)rrrr)r5)rrrdrmaxvit_nano_rw)r)rrfr)maxvit_tiny_rwmaxvit_tiny_pm)rrrrmaxvit_rmlp_pico_rw)rCmaxvit_rmlp_nano_rwmaxvit_rmlp_tiny_rwmaxvit_rmlp_small_rwmaxvit_rmlp_base_rw)rrrdrrmaxxvit_rmlp_nano_rw)rrrdrrmaxxvit_rmlp_tiny_rwmaxxvit_rmlp_small_rw)0r|maxxvitv2_nano_rw)rIrCmaxxvitv2_rmlp_base_rw)rrrr)rImaxxvitv2_rmlp_large_rw)i@ir)rrr)Pr1rmaxvit_tiny_tf)rrrdrrrmaxvit_small_tfmaxvit_base_tfmaxvit_large_tfmaxvit_xlarge_tf state_dictmodelcCs|}i}|D]V\}}|dr7||dd}|j|jjks-|jd|jdkr7t||j|jjd}||vr\|j||jkr\| || kr\|jdvsTJ| ||j}|||<q |S)z/Filter checkpoint state dict for compatibility.rNirr))new_window_sizenew_bias_shape)rrG) r9rendswith get_submodulerrrGr(ndimnumelr)r9r:model_state_dictout_dictrrmrWrWrXcheckpoint_filter_fns  ", rDvariant cfg_variant pretrainedcKsT|dur|tvr |}n d|ddd}tt||ft|tddtd|S)zCreate a MaxxVit model variant.NrrT)flatten_sequential) model_cfg feature_cfgpretrained_filter_fn) model_cfgsjoinrr*r4r}rD)rErFrGrrWrWrX_create_maxxvit srNurlc Ks |dddddddddd d |S) z$Create a default configuration dict.r)rfrr)r9r9ffffff?bicubic)?rRrRz stem.conv1zhead.fcT) rOr input_size pool_sizecrop_pct interpolationmeanr first_conv classifierfixed_input_sizerW)rOrrWrWrX_cfgsr[zcoatnet_pico_rw_224.untrained)rOzcoatnet_nano_rw_224.sw_in1kztimm/zyhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights-maxx/coatnet_nano_rw_224_sw-f53093b4.pthg?) hf_hub_idrOrUzcoatnet_0_rw_224.sw_in1kzvhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights-maxx/coatnet_0_rw_224_sw-a6439706.pth)r\rOzcoatnet_1_rw_224.sw_in1kzvhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights-maxx/coatnet_1_rw_224_sw-5cae1ea8.pthz!coatnet_2_rw_224.sw_in12k_ft_in1k)r\z'coatnet_rmlp_1_rw2_224.sw_in12k_ft_in1kz&coatnet_rmlp_2_rw_224.sw_in12k_ft_in1kz&coatnet_rmlp_2_rw_384.sw_in12k_ft_in1k)rfr~r~)rrg?squash)r\rSrTrU crop_modezcoatnet_bn_0_rw_224.sw_in1kzyhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights-maxx/coatnet_bn_0_rw_224_sw-c228e218.pthrP)r\rOrWrrUz coatnet_rmlp_nano_rw_224.sw_in1kz~https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights-maxx/coatnet_rmlp_nano_rw_224_sw-bd1d51b3.pthzcoatnet_rmlp_0_rw_224.untrainedzcoatnet_rmlp_1_rw_224.sw_in1kz{https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights-maxx/coatnet_rmlp_1_rw_224_sw-9051e6c3.pthzcoatnet_rmlp_2_rw_224.sw_in1kz{https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights-maxx/coatnet_rmlp_2_rw_224_sw-5ccfac55.pthzcoatnet_rmlp_3_rw_224.untrainedzcoatnet_nano_cc_224.untrainedzcoatnext_nano_rw_224.sw_in1kzzhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights-maxx/coatnext_nano_rw_224_ad-22cb71c2.pthzcoatnet_2_rw_224.sw_in12ki-.)r\rzcoatnet_3_rw_224.sw_in12kzcoatnet_rmlp_1_rw2_224.sw_in12kzcoatnet_rmlp_2_rw_224.sw_in12kzcoatnet_0_224.untrainedzcoatnet_1_224.untrainedzcoatnet_2_224.untrainedzcoatnet_3_224.untrainedzcoatnet_4_224.untrainedzcoatnet_5_224.untrainedzmaxvit_pico_rw_256.untrained)rfrr)r_)rOrSrTzmaxvit_nano_rw_256.sw_in1kzxhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights-maxx/maxvit_nano_rw_256_sw-fb127241.pth)r\rOrSrTzmaxvit_tiny_rw_224.sw_in1kzxhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights-maxx/maxvit_tiny_rw_224_sw-7d0dffeb.pthzmaxvit_tiny_rw_256.untrainedzmaxvit_tiny_pm_256.untrainedzmaxvit_rmlp_pico_rw_256.sw_in1kz}https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights-maxx/maxvit_rmlp_pico_rw_256_sw-8d82f2c6.pthzmaxvit_rmlp_nano_rw_256.sw_in1kz}https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights-maxx/maxvit_rmlp_nano_rw_256_sw-c17bb0d6.pthzmaxvit_rmlp_tiny_rw_256.sw_in1kz}https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights-maxx/maxvit_rmlp_tiny_rw_256_sw-bbef0ff5.pthz maxvit_rmlp_small_rw_224.sw_in1kz~https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights-maxx/maxvit_rmlp_small_rw_224_sw-6ef0ae4f.pthz"maxvit_rmlp_small_rw_256.untrainedz(maxvit_rmlp_base_rw_224.sw_in12k_ft_in1kz(maxvit_rmlp_base_rw_384.sw_in12k_ft_in1kz maxvit_rmlp_base_rw_224.sw_in12kz maxxvit_rmlp_nano_rw_256.sw_in1kz~https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights-maxx/maxxvit_rmlp_nano_rw_256_sw-0325d459.pthz"maxxvit_rmlp_tiny_rw_256.untrainedz!maxxvit_rmlp_small_rw_256.sw_in1kzhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights-maxx/maxxvit_rmlp_small_rw_256_sw-37e217ff.pthzmaxxvitv2_nano_rw_256.sw_in1k)r\rSrTz+maxxvitv2_rmlp_base_rw_224.sw_in12k_ft_in1kz+maxxvitv2_rmlp_base_rw_384.sw_in12k_ft_in1kz%maxxvitv2_rmlp_large_rw_224.untrainedz#maxxvitv2_rmlp_base_rw_224.sw_in12kzmaxvit_tiny_tf_224.in1k)r\rWrzmaxvit_tiny_tf_384.in1kzmaxvit_tiny_tf_512.in1k)rfrDrD)r2r2zmaxvit_small_tf_224.in1kzmaxvit_small_tf_384.in1kzmaxvit_small_tf_512.in1kzmaxvit_base_tf_224.in1kzmaxvit_base_tf_384.in1kzmaxvit_base_tf_512.in1kzmaxvit_large_tf_224.in1kzmaxvit_large_tf_384.in1kzmaxvit_large_tf_512.in1kzmaxvit_base_tf_224.in21kiSUz maxvit_base_tf_384.in21k_ft_in1kz maxvit_base_tf_512.in21k_ft_in1kzmaxvit_large_tf_224.in21kz!maxvit_large_tf_384.in21k_ft_in1kz!maxvit_large_tf_512.in21k_ft_in1k)r\rSrUr^zmaxvit_xlarge_tf_224.in21kz"maxvit_xlarge_tf_384.in21k_ft_in1kz"maxvit_xlarge_tf_512.in21k_ft_in1kcKtdd|i|S)z)CoatNet Pico model with RW configuration.coatnet_pico_rw_224rGN)rarNrGrrWrWrXra racKr`)z)CoatNet Nano model with RW configuration.coatnet_nano_rw_224rGN)rerbrcrWrWrXre rdrecKr`)z&CoatNet-0 model with RW configuration.coatnet_0_rw_224rGN)rfrbrcrWrWrXrf rdrfcKr`)z&CoatNet-1 model with RW configuration.coatnet_1_rw_224rGN)rgrbrcrWrWrXrg rdrgcKr`)z&CoatNet-2 model with RW configuration.coatnet_2_rw_224rGN)rhrbrcrWrWrXrh rdrhcKr`)z&CoatNet-3 model with RW configuration.coatnet_3_rw_224rGN)rirbrcrWrWrXri rdricKr`)z4CoatNet-0 model with BatchNorm and RW configuration.coatnet_bn_0_rw_224rGN)rjrbrcrWrWrXrj% rdrjcKr`)z.CoatNet Nano model with Relative Position MLP.coatnet_rmlp_nano_rw_224rGN)rkrbrcrWrWrXrk+ rdrkcKr`)z+CoatNet-0 model with Relative Position MLP.coatnet_rmlp_0_rw_224rGN)rlrbrcrWrWrXrl1 rdrlcKr`)z+CoatNet-1 model with Relative Position MLP.coatnet_rmlp_1_rw_224rGN)rmrbrcrWrWrXrm7 rdrmcKr`)z.CoatNet-1 model with Relative Position MLP v2.coatnet_rmlp_1_rw2_224rGN)rnrbrcrWrWrXrn= rdrncKr`)z+CoatNet-2 model with Relative Position MLP.coatnet_rmlp_2_rw_224rGN)rorbrcrWrWrXroC rdrocKr`)z6CoatNet-2 model with Relative Position MLP at 384x384.coatnet_rmlp_2_rw_384rGN)rprbrcrWrWrXrpI rdrpcKr`)z+CoatNet-3 model with Relative Position MLP.coatnet_rmlp_3_rw_224rGN)rqrbrcrWrWrXrqO rdrqcKr`)z(CoatNet Nano model with ConvNeXt blocks.coatnet_nano_cc_224rGN)rrrbrcrWrWrXrrU rdrrcKr`)z*CoAtNeXt Nano model with RW configuration.coatnext_nano_rw_224rGN)rsrbrcrWrWrXrs[ rdrscKr`)zCoatNet-0 model. coatnet_0_224rGN)rtrbrcrWrWrXrta rdrtcKr`)zCoatNet-1 model. coatnet_1_224rGN)rurbrcrWrWrXrug rdrucKr`)zCoatNet-2 model. coatnet_2_224rGN)rvrbrcrWrWrXrvm rdrvcKr`)zCoatNet-3 model. coatnet_3_224rGN)rwrbrcrWrWrXrws rdrwcKr`)zCoatNet-4 model. coatnet_4_224rGN)rxrbrcrWrWrXrxy rdrxcKr`)zCoatNet-5 model. coatnet_5_224rGN)ryrbrcrWrWrXry rdrycKr`)z(MaxViT Pico model with RW configuration.maxvit_pico_rw_256rGN)rzrbrcrWrWrXrz rdrzcKr`)z(MaxViT Nano model with RW configuration.maxvit_nano_rw_256rGN)r{rbrcrWrWrXr{ rdr{cKr`)z(MaxViT Tiny model with RW configuration.maxvit_tiny_rw_224rGN)r|rbrcrWrWrXr| rdr|cKr`)z3MaxViT Tiny model with RW configuration at 256x256.maxvit_tiny_rw_256rGN)r}rbrcrWrWrXr} rdr}cKr`)z3MaxViT Relative Position MLP Pico RW 256x256 model.maxvit_rmlp_pico_rw_256rGN)r~rbrcrWrWrXr~ rdr~cKr`)z3MaxViT Relative Position MLP Nano RW 256x256 model.maxvit_rmlp_nano_rw_256rGN)rrbrcrWrWrXr rdrcKr`)z3MaxViT Relative Position MLP Tiny RW 256x256 model.maxvit_rmlp_tiny_rw_256rGN)rrbrcrWrWrXr rdrcKr`)z4MaxViT Relative Position MLP Small RW 224x224 model.maxvit_rmlp_small_rw_224rGN)rrbrcrWrWrXr rdrcKr`)z9MaxViT Small model with Relative Position MLP at 256x256.maxvit_rmlp_small_rw_256rGN)rrbrcrWrWrXr rdrcKr`)z-MaxViT Base model with Relative Position MLP.maxvit_rmlp_base_rw_224rGN)rrbrcrWrWrXr rdrcKr`)z8MaxViT Base model with Relative Position MLP at 384x384.maxvit_rmlp_base_rw_384rGN)rrbrcrWrWrXr rdrcKr`)z'MaxViT Tiny model with parallel blocks.maxvit_tiny_pm_256rGN)rrbrcrWrWrXr rdrcKr`)z4MaxxViT Relative Position MLP Nano RW 256x256 model.maxxvit_rmlp_nano_rw_256rGN)rrbrcrWrWrXr rdrcKr`)z.MaxxViT Tiny model with Relative Position MLP.maxxvit_rmlp_tiny_rw_256rGN)rrbrcrWrWrXr rdrcKr`)z/MaxxViT Small model with Relative Position MLP.maxxvit_rmlp_small_rw_256rGN)rrbrcrWrWrXr rdrcKr`)zMaxxViT-V2 Nano model.maxxvitv2_nano_rw_256rGN)rrbrcrWrWrXr rdrcKr`)z1MaxxViT-V2 Base model with Relative Position MLP.maxxvitv2_rmlp_base_rw_224rGN)rrbrcrWrWrXr rdrcKr`)zs $  (       !$ QR 0 K eW (  ("  AP (  ( H51S/ \     :     1    '  (2>IT]is~ $*06<DKRYahov #.7CLU^g,s&"  #'(+./069<?DEFGHILMQTWZ^bfkpsx~  "%(+.147:>ADGJM PSV\                                                       $