o پi @sbddlZddlmZddlmZmZmZmZmZddl Z ddl m Z ddl m Z mZddlmZmZmZmZmZmZmZddlmZddlmZdd lmZdd lmZmZd gZ d d Z!Gddde j"Z#Gddde j"Z$e j%dfde&de&dee j"de'de j(f ddZ)Gddde j"Z*Gddde j"Z+Gddde j"Z,Gd d!d!e j"Z-Gd"d#d#e j"Z.Gd$d%d%e j"Z/Gd&d'd'e j"Z0Gd(d)d)e j"Z1Gd*d+d+e j"Z2Gd,d d e j"Z3dgd.d/Z4eid0e4d1d2d3e4d1d2d4e4d1d2d5e4d1d2d6e4d1d2d7e4d1d2d8e4d1d9d:d;e4d1d2de4d1d2d?e4d1d2d@e4d1d2dAe4d1d9d:dBe4dCdDd9dEdFdGdHdIe4dJdKd9dEdFdGdHdLe4dMdNd9dEdFdGdHZ5dOdPZ6dhdQdRZ7edhdSdTZ8edhdUdVZ9edhdWdXZ:edhdYdZZ;edhd[d\ZedhdadbZ?edhdcddZ@edhdedfZAdS)iN)partial)ListOptionalTupleTypeUnionIMAGENET_DEFAULT_MEANIMAGENET_DEFAULT_STD)DropPath trunc_normal_ create_conv2d ConvNormAct SqueezeExciteuse_fused_attnClassifierHead)build_model_with_cfg)feature_take_indices)checkpoint_seq)register_modelgenerate_default_cfgsFastVitcCs |sdS||dks J||S)Nrr) group_sizechannelsrrG/home/ubuntu/.local/lib/python3.10/site-packages/timm/models/fastvit.py num_groupssrcseZdZdZddddddddejf dededed ed ed ed ed ededededeej ddffdd Z de j de j fddZ ddZdee j e j ffddZdeejejfdee j e j ffddZZS) MobileOneBlocka#MobileOne building block. This block has a multi-branched architecture at train-time and plain-CNN style architecture at inference time For more details, please refer to our paper: `An Improved One millisecond Mobile Backbone` - https://arxiv.org/pdf/2206.04040.pdf rrFTin_chsout_chs kernel_sizestridedilationrinference_modeuse_seuse_actuse_scale_branchnum_conv_branches act_layerreturnNc s&tt|_t||_|_|__|_ |_ | _ |r*t |ddnt _|r?t||||jdd_nEd_||krP|dkrPt j|dnd_| dkrit fdd tj D_nd_d_dkr| rtj j djjd d _| r| _dSt _dS) aConstruct a MobileOneBlock module. Args: in_chs: Number of channels in the input. out_chs: Number of channels produced by the block. kernel_size: Size of the convolution kernel. stride: Stride size. dilation: Kernel dilation factor. group_size: Convolution group size. inference_mode: If True, instantiates model in inference mode. use_se: Whether to use SE-ReLU activations. use_act: Whether to use activation. Default: ``True`` use_scale_branch: Whether to use scale branch. Default: ``True`` num_conv_branches: Number of linear conv branches. r) rd_divisorTr!r"r#groupsbiasN) num_featuresrc s(g|]}tjjjjddqS)Fr!r"r- apply_act)rrr r"r-).0_r!selfrr msz+MobileOneBlock.__init__..Fr0)superr__init__r$rr-r"r#r!rr r(rnnIdentityser reparam_conv BatchNorm2didentity ModuleListrangeconv_kxk conv_scaleract) r5rr r!r"r#rr$r%r&r'r(r) __class__r4rr8,sR        zMobileOneBlock.__init__xcCs|jdur||||Sd}|jdur||}d}|jdur(||}||}|jdur=|jD]}|||7}q4|||S)zApply forward pass.Nr)r<rCr;r>rBrA)r5rF identity_out scale_outoutrcrrrforwards       zMobileOneBlock.forwardc Cs|jdurdS|\}}t|j|j|j|j|j|jdd|_||jj _ ||jj _ | D] \}}d|vr6q-| q-|d|dt|drO|dd|_dS)aFollowing works like `RepVGG: Making VGG-style ConvNets Great Again` - https://arxiv.org/pdf/2101.03697.pdf. We re-parameterize multi-branched architecture used at training time to obtain a plain CNN-like structure for inference. NT) in_channels out_channelsr!r"r#r-r.r<rArBr>)r<_get_kernel_biasr rr r!r"r#r-weightdatar.named_parametersdetach_ __delattr__hasattrr$)r5kernelr.namepararrrreparameterizes.         zMobileOneBlock.reparameterizec Csd}d}|jdur"||j\}}|jd}tjj|||||g}d}d}|jdur3||j\}}d}d}|jdurVt |j D]}||j|\} } || 7}|| 7}qA|||} |||} | | fS)zMethod to obtain re-parameterized kernel and bias. Reference: https://github.com/DingXiaoH/RepVGG/blob/main/repvgg.py#L83 Returns: Tuple of (kernel, bias) after fusing branches. rN) rB_fuse_bn_tensorr!torchr9 functionalpadr>rAr@r() r5 kernel_scale bias_scaler]kernel_identity bias_identity kernel_conv bias_convix_kernel_bias kernel_final bias_finalrrrrNs(       zMobileOneBlock._get_kernel_biasbranchc Cst|tr|jj}|jj}|jj}|jj}|jj}|jj}nSt|t j s&Jt |ds_|j |j }tj|j ||j|jf|jj|jjd} t|j D]} d| | | ||jd|jdf<qI| |_|j}|j}|j}|j}|j}|j}||} || dddd} || |||| fS)aMethod to fuse batchnorm layer with preceding conv layer. Reference: https://github.com/DingXiaoH/RepVGG/blob/main/repvgg.py#L95 Args: branch: Sequence of ops to be fused. Returns: Tuple of (kernel, bias) after fusing batchnorm. id_tensordtypedevicerrY) isinstancerconvrObn running_mean running_varr.epsr9r=rTrr-r[zerosr!rlrmr@rjsqrtreshape) r5rirUrrrsgammabetart input_dim kernel_valueistdtrrrrZs<     zMobileOneBlock._fuse_bn_tensor)__name__ __module__ __qualname____doc__r9GELUintboolrModuler8r[TensorrKrXrrNr Sequentialr=rZ __classcell__rrrDrr"s\     \##rcseZdZdZ    ddededededed eed ed eejd ed dffdd Z de j d e j fddZ d e e j e j ffddZdddZedejdejd e e j e j ffddZZS)ReparamLargeKernelConvzBuilding Block of RepLKNet This class defines overparameterized large kernel conv block introduced in `RepLKNet `_ Reference: https://github.com/DingXiaoH/RepLKNet-pytorch NFrr r!r"r small_kernelr%r)r$r*c stt|||_t|||_||_||_||_||_ | r,t ||||d|jdd|_ n)d|_ t ||||j|jdd|_ |durU||ksHJdt ||||j|jdd|_|r]t|dd nt|_|durl||_dSt|_dS) a!Construct a ReparamLargeKernelConv module. Args: in_chs: Number of input channels. out_chs: Number of output channels. kernel_size: Kernel size of the large kernel conv branch. stride: Stride size. Default: 1 group_size: Group size. Default: 1 small_kernel: Kernel size of small kernel conv branch. act_layer: Activation module. Default: ``nn.GELU`` inference_mode: If True, instantiates model in inference mode. Default: ``False`` rTr,NFr0zDThe kernel size for re-param cannot be larger than the large kernel!g?)rd_ratio)r7rr8r"rr-rr r!rr r<r large_conv small_convrr9r:r;rC) r5rr r!r"rrr%r)r$rDrrr8sP    "zReparamLargeKernelConv.__init__rFcCsP|jdur ||}n||}|jdur|||}||}||}|SN)r<rrr;rC)r5rFrIrrrrK`s      zReparamLargeKernelConv.forwardcCsn||jj|jj\}}t|dr3||jj|jj\}}||7}|tj||j |j dgd7}||fS)zMethod to obtain re-parameterized kernel and bias. Reference: https://github.com/DingXiaoH/RepLKNet-pytorch Returns: Tuple of (kernel, bias) after fusing branches. rrY) _fuse_bnrrprqrTrr9r\r]r!r)r5eq_keq_bsmall_ksmall_brrrget_kernel_biasks z&ReparamLargeKernelConv.get_kernel_biascCsf|\}}t|j|j|j|j|jdd|_||jj_ ||jj _ | dt |dr1| ddSdS)a Following works like `RepVGG: Making VGG-style ConvNets Great Again` - https://arxiv.org/pdf/2101.03697.pdf. We re-parameterize multi-branched architecture used at training time to obtain a plain CNN-like structure for inference. Tr!r"r-r.rrN) rr rr r!r"r-r<rOrPr.rSrT)r5rrrrrrX{s    z%ReparamLargeKernelConv.reparameterizerprqc Cs\|j}|j}|j}|j}|j}|j}||}||dddd} || ||||fS)zMethod to fuse batchnorm layer with conv layer. Args: conv: Convolutional kernel weights. bn: Batchnorm 2d layer. Returns: Tuple of (kernel, bias) after fusing batchnorm. rnr)rOrrrsr.rtrvrw) rprqrUrrrsrxryrtr}r~rrrrs  zReparamLargeKernelConv._fuse_bn)NFNFr*N)rrrrrrrr9rr8r[rrKrrrX staticmethodConv2dr=rrrrrDrrsL   D rFrr r)r$r*c Cs@tt||dd||dt||ddd||dt||dd||dS)a,Build convolutional stem with MobileOne blocks. Args: in_chs: Number of input channels. out_chs: Number of output channels. inference_mode: Flag to instantiate model in inference mode. Default: ``False`` Returns: nn.Sequential object with stem elements. rY)rr r!r"r)r$r)rr r!r"rr)r$)r9rr)rr r)r$rrrconvolutional_stems6 rc sleZdZUdZejjeed<    dde de ded e d e d d f fd d Z dej d ej fddZ ZS) AttentionzMulti-headed Self Attention module. Source modified from: https://github.com/rwightman/pytorch-image-models/blob/master/timm/models/vision_transformer.py fused_attn Fdimhead_dimqkv_bias attn_drop proj_dropr*Ncst||dksJd||_|||_|d|_t|_tj||d|d|_ t ||_ t|||_ t ||_ dS)a}Build MHSA module that can handle 3D or 4D input tensors. Args: dim: Number of embedding dimensions. head_dim: Number of hidden dimensions per head. Default: ``32`` qkv_bias: Use bias or not. Default: ``False`` attn_drop: Dropout rate for attention tensor. proj_drop: Dropout rate for projection tensor. rz#dim should be divisible by head_dimgr)r.N)r7r8r num_headsscalerrr9LinearqkvDropoutrprojr)r5rrrrrrDrrr8s    zAttention.__init__rFc Cs |j\}}}}||}|ddd}||||d|j|jddddd}|d\}} } |j rGt j j j || | |jrB|jjndd }n||j}|| dd} | jdd } || } | | }|dd|||}||}||}|dd||||}|S) NrYrnrrrrr) dropout_pr)shapeflatten transposerrwrrpermuteunbindrr[r9r\scaled_dot_product_attentiontrainingrprsoftmaxrr) r5rFBCHWNrqkvattnrrrrKs.     zAttention.forward)rFrr)rrrrr[jitFinalr__annotations__rfloatr8rrKrrrrDrrs* rcsneZdZdZejdddfdededededeejde d e d e d d ffd d Z de j d e j fddZ ZS) PatchEmbedz$Convolutional patch embedding layer.F patch_sizer"r embed_dimr) lkc_use_actr%r$r*Nc sLttt||||dd||r|nd|d t||ddd||d|_dS)a{Build patch embedding layer. Args: patch_size: Patch size for embedding computation. stride: Stride for convolutional embedding layer. in_chs: Number of channels of input tensor. embed_dim: Number of embedding dimensions. inference_mode: Flag to instantiate model in inference mode. Default: ``False`` rrN) rr r!r"rrr%r)r$F)rr r!r"r%r)r$)r7r8r9rrrr) r5rr"rrr)rr%r$rDrrr8s.    zPatchEmbed.__init__rFcCs||}|Sr)rr5rFrrrrKJs zPatchEmbed.forward)rrrrr9rrrrrr8r[rrKrrrrDrrs4  ,rcs&eZdZdfdd ZddZZS) LayerScale2dh㈵>Fcs.t||_t|t|dd|_dS)Nr)r7r8inplacer9 Parameterr[onesrx)r5r init_valuesrrDrrr8Ps zLayerScale2d.__init__cCs|jr ||jS||jSr)rmul_rxrrrrrKUzLayerScale2d.forward)rF)rrrr8rKrrrrDrrOsrcsNeZdZdZ   ddeffdd Zdejd ejfd d Zdd dZ Z S)RepMixerzReparameterizable token mixer. For more details, please refer to our paper: `FastViT: A Fast Hybrid Vision Transformer using Structural Reparameterization `_ rrFr$c st||_||_||_|r&tj|j|j|jd|jd|jdd|_dSd|_t|||ddddd|_ t|||ddd |_ |durKt |||_ dSt |_ dS) aBuild RepMixer Module. Args: dim: Input feature map dimension. :math:`C_{in}` from an expected input of size :math:`(B, C_{in}, H, W)`. kernel_size: Kernel size for spatial mixing. Default: 3 layer_scale_init_value: Initial value for layer scale. Default: 1e-5 inference_mode: If True, instantiates model in inference mode. Default: ``False`` rrYTr!r"paddingr-r.NFr)rr&r'r()rr&)r7r8rr!r$r9rr<rnormmixerr layer_scaler:)r5rr!layer_scale_init_valuer$rDrrr8`sD    zRepMixer.__init__rFr*cCs:|jdur ||}|S||||||}|Sr)r<rrrrrrrrKs  zRepMixer.forwardNcCs |jrdS|j|jt|jtr<|jj|jj d|jj j |jj j }t |jj|jj j|jj j}n|jj|jj j |jj j }|jj j|jj j}t|j|j|jd|jdd|_ ||j j _||j j_|D] \}}d|vrzqq|qq|d|d|d dS) ziReparameterize mixer and norm into a single convolutional layer for efficient inference. NrnrTrr<rrr)r$rrXrrorrrjrx unsqueezer<rOr[squeezer.r rr!rPrQrRrS)r5wbrVrWrrrrXsF         zRepMixer.reparameterize)rrFr) rrrrrr8r[rrKrXrrrrDrrYs 5rc s~eZdZdZddejdfdedeedeedeej de d df fd d Z d ej d dfd dZ de jd e jfddZZS)ConvMlpzConvolutional FFN Module.Nrrhidden_channelsr r)dropr*csvt|p|}|p |}t||d|dd|_tj||dd|_||_tj||dd|_t ||_ | |j dS)a_Build convolutional FFN module. Args: in_chs: Number of input channels. hidden_channels: Number of channels after expansion. Default: None out_chs: Number of output channels. Default: None act_layer: Activation layer. Default: ``GELU`` drop: Dropout rate. Default: ``0.0``. F)r!r-r1r)r!N) r7r8rrpr9rfc1rCfc2rrapply _init_weights)r5rrr r)rrDrrr8s  zConvMlp.__init__mcCs@t|tjrt|jdd|jdurtj|jddSdSdS)N{Gz?r}r)ror9rr rOr.init constant_r5rrrrrs  zConvMlp._init_weightsrFcCs@||}||}||}||}||}||}|Sr)rprrCrrrrrrrKs      zConvMlp.forward)rrrrr9rrrrrrr8rr[rrKrrrrDrrs*!rc sneZdZdZ   ddedeedeeeeeffddffd d Zd e j de j fd d Z dddZ Z S)RepConditionalPosEnca"Implementation of conditional positional encoding. For more details refer to paper: `Conditional Positional Encodings for Vision Transformers `_ In our implementation, we can reparameterize this module to eliminate a skip connection. NrrFrdim_out spatial_shaper*c stt|t|trt|gd}t|ts"Jdt|dt|dks2Jdt|d||_ ||_ |p;||_ ||_ |rYt j|j |j |j d|dd|j dd|_d Sd |_t j|j |j |dt|dd|j dd |_d S) atBuild reparameterizable conditional positional encoding Args: dim: Number of input channels. dim_out: Number of embedding dimensions. Default: 768 spatial_shape: Spatial shape of kernel for positional encoding. Default: (7, 7) inference_mode: Flag to instantiate block in inference mode. Default: ``False`` rYz/"spatial_shape" must by a sequence or int, get z instead.z+Length of "spatial_shape" should be 2, got rrTrN)r-r.)r7rr8rortuplertypelenrrrr-r9rr<pos_enc)r5rrrr$rDrrr8sJ       zRepConditionalPosEnc.__init__rFcCs*|jdur ||}|S|||}|Sr)r<rrrrrrK<s  zRepConditionalPosEnc.forwardc Cs|j|j}tj|j||jd|jdf|jjj|jjjd}t |jD]}d|||||jdd|jddf<q$|}||jj}|jj }t j |j|j |jdt|jdd|jdd|_||jj_||jj _|D] \}}d|vrvqm|qm|ddS) NrrrkrYTrr<r)rr-r[rurrrOrlrmr@r.r9rrrr<rPrQrRrS) r5rzr{r|rjw_finalb_finalrVrWrrrrXCsL       z#RepConditionalPosEnc.reparameterize)NrFr)rrrrrrrrr8r[rrKrXrrrrDrrs  6rcsbeZdZdZddejddddfdeded ed eej d ed ed ede ffdd Z ddZ Z S) RepMixerBlockzImplementation of Metaformer block with RepMixer as token mixer. For more details on Metaformer structure, please refer to: `MetaFormer Is Actually What You Need for Vision `_ r@rrFrr! mlp_ratior)r drop_pathrr$c sztt||||d|_t|t||||d|_|dur&t|||_nt |_|dkr6t ||_ dSt |_ dS)a,Build RepMixer Block. Args: dim: Number of embedding dimensions. kernel_size: Kernel size for repmixer. Default: 3 mlp_ratio: MLP expansion ratio. Default: 4.0 act_layer: Activation layer. Default: ``nn.GELU`` proj_drop: Dropout rate. Default: 0.0 drop_path: Drop path rate. Default: 0.0 layer_scale_init_value: Layer scale value at initialization. Default: 1e-5 inference_mode: Flag to instantiate block in inference mode. Default: ``False`` )r!rr$rrr)rNr) r7r8r token_mixerrrmlprrr9r:r r) r5rr!rr)rrrr$rDrrr8xs"   $zRepMixerBlock.__init__cCs(||}|||||}|Sr)rrrrrrrrrKs zRepMixerBlock.forward)rrrrr9rrrrrrr8rKrrrrDrrqs6  -rcsdeZdZdZdejejdddfdedede ej de ej d ed ed effd d Z ddZ Z S)AttentionBlockzImplementation of metaformer block with MHSA as token mixer. For more details on Metaformer structure, please refer to: `MetaFormer Is Actually What You Need for Vision `_ rrrrrr) norm_layerrrrcst|||_t|d|_|durt|||_nt|_|dkr(t |nt|_ t |t ||||d|_ |durEt|||_nt|_|dkrUt ||_dSt|_dS)aBuild Attention Block. Args: dim: Number of embedding dimensions. mlp_ratio: MLP expansion ratio. Default: 4.0 act_layer: Activation layer. Default: ``nn.GELU`` norm_layer: Normalization layer. Default: ``nn.BatchNorm2d`` proj_drop: Dropout rate. Default: 0.0 drop_path: Drop path rate. Default: 0.0 layer_scale_init_value: Layer scale value at initialization. Default: 1e-5 rNrr)r7r8rrrr layer_scale_1r9r:r drop_path1rrr layer_scale_2 drop_path2)r5rrr)rrrrrDrrr8s"      $zAttentionBlock.__init__c Cs>||||||}|||||}|Sr)rrrrrrrrrrrrKs zAttentionBlock.forward)rrrrr9rr=rrrrr8rKrrrrDrrs0 ,rc"seZdZdddddddejejddd ddfd ed ed ed ededededede ej dede de ej de ej de de de e f fdd Z ddZZS) FastVitStageTFrrYNrrrrrrdepthtoken_mixer_type downsample se_downsampledown_patch_size down_stride pos_emb_layerr!rr)rproj_drop_ratedrop_path_raterc std|_|rt|||||| ||d|_n ||ksJt|_| dur/| ||d|_nt|_g}t|D]3}|dkrR| t || | | |||||dq:|dkrg| t || | | ||||dq:t d |tj||_dS) aQFastViT stage. Args: dim: Number of embedding dimensions. depth: Number of blocks in stage token_mixer_type: Token mixer type. kernel_size: Kernel size for repmixer. mlp_ratio: MLP expansion ratio. act_layer: Activation layer. norm_layer: Normalization layer. proj_drop_rate: Dropout rate. drop_path_rate: Drop path rate. layer_scale_init_value: Layer scale value at initialization. inference_mode: Flag to instantiate block in inference mode. F)rr"rrr%r)rr$N)r$repmixer)r!rr)rrrr$ attention)rr)rrrrz"Token mixer type: {} not supported)r7r8grad_checkpointingrrr9r:pos_embr@appendrr ValueErrorformatrblocks)r5rrrrrrrrr r!rr)rr r rrr$r block_idxrDrrr8sZ $       zFastVitStage.__init__cCsB||}||}|jrtjst|j|}|S||}|Sr)rrrr[r is_scriptingrrrrrrrK=s    zFastVitStage.forward)rrrr9rr=rstrrrrrrr8rKrrrrDrrsb     Xrc1sVeZdZUejjeed< ddddddddd d d d d d d dddddej ej dfde de e dfde e dfde e dfde edfde edfde edfde de de eejdfde de ded ed!ed"ed#eded$ed%e d&eejd'eejd(ed)d*f0fd+d, Zd-ejd)d*fd.d/Zejjd0d1ZejjdPd2d3ZejjdQd5d6Zejjd)ejfd7d8ZdRde d%ee fd9d:Z *   ; dSded?ed@e dAed)eeeje ejeejfffdBdCZ D  4dTd=ee ee fdEedFefdGdHZdtj||_t |j|_||_|_|jrgd|_t|jD]&\}} |dkrtjddr t}!n|||}!d| }"||"|!qn't|d||_|_}#t|d|#ddd|d |dd! |_ t!|#| || d"|_"|#|j$dS)$NrrcSsg|]}|qSr)tolist)r2rFrrrr6ysz$FastVit.__init__..rrrrrrrr rr!rr)rr r rrr$rYrstages.)num_chs reductionmodulerrrYr FORK_LAST3rrnrT) rr r!r"rr$r%r)r() pool_typer-r)%r7r8r+rr/ feature_inforstemr[linspacesumsplitr@rrrdictr9rstages num_stagesr/head_hidden_size out_indices enumerateosenvirongetr: add_modulerr final_convrheadrr)$r5r#r$r%r&r'r(r)r*r+r,rrr-r r rrrr.r/rr)r$prev_dimrdprr>r|rstagei_embi_layerlayer layer_namefinal_featuresrDrrr8Ns "      &         zFastVit.__init__rcCsPt|tjr"t|jddt|tjr$|jdur&tj|jddSdSdSdS)zInit. for classificationrrNr)ror9rr rOr.rrrrrrrs zFastVit._init_weightscCstSr)setr5rrrno_weight_decayszFastVit.no_weight_decaycCstd|rddSgddS)Nz^stemz^stages\.(\d+)))z^stages\.(\d+).downsampler)z^stages\.(\d+).pos_embrT)z^stages\.(\d+)\.\w+\.(\d+)N)r9r)r=)r5coarserrr group_matchers zFastVit.group_matcherTcCs|jD]}||_qdSr)r>r)r5enablesrrrset_grad_checkpointings zFastVit.set_grad_checkpointingcCs|jjSr)rHfcrRrrrget_classifierszFastVit.get_classifiercCs||_|j||dSr)r+rHreset)r5r+r/rrrreset_classifierszFastVit.reset_classifierNCHWrFindicesr stop_early output_fmtintermediates_onlycCs|dvsJdg}tt|j|\}} ||}|jd} tjs%|s)|j} n |jd| d} d} t| D]\} } | |}| |vrI| |q8|rN|S| | krW| |}||fS)a Forward features that returns intermediates. Args: x: Input image tensor indices: Take last n blocks if int, all if None, select matching indices if sequence norm: Apply norm layer to compatible intermediates stop_early: Stop iterating over blocks when last desired intermediate hit output_fmt: Shape of intermediate feature outputs intermediates_only: Only return intermediate features Returns: )r^zOutput shape must be NCHW.rNr) rrr>r9r?r[rrrBrrG)r5rFr_rr`rarb intermediates take_indices max_indexlast_idxr>feat_idxrKrrrforward_intermediatess&    zFastVit.forward_intermediatesr prune_norm prune_headcCs<tt|j|\}}|jd|d|_|r|dd|S)z@ Prune layers not required for specified intermediates. Nrr)rrr>r])r5r_rirjrdrerrrprune_intermediate_layerss  z!FastVit.prune_intermediate_layerscCst||}g}t|jD]!\}}||}|jr-||jvr-t|d|}||}||q |jr3|S||}|S)Nr)r9rBr>rrAgetattrrrG)r5rFoutsidxblockrx_outrrrforward_features s    zFastVit.forward_features pre_logitscCs|r |j|ddS||S)NT)rs)rH)r5rFrsrrr forward_head1rzFastVit.forward_headcCs"||}|jr |S||}|Sr)rrrrtrrrrrK4s  zFastVit.forwardF)Tr)NFFr^F)rFT)#rrrr[rrrrr9r=rrrrrrrrr8rignorerSrVrYr[r]rrrrhrlrrrtrKrrrrDrrGs            s     0 rkc Ks|dddddttddd |S) Nr!)rrr)rwg?bicubic)zstem.0.conv_kxk.0.convzstem.0.conv_scale.convzhead.fc) urlr+ input_size pool_sizecrop_pct interpolationmeanr} first_conv classifierr)rykwargsrrr_cfg<s rzfastvit_t8.apple_in1kztimm/) hf_hub_idzfastvit_t12.apple_in1kzfastvit_s12.apple_in1kzfastvit_sa12.apple_in1kzfastvit_sa24.apple_in1kzfastvit_sa36.apple_in1kzfastvit_ma36.apple_in1kgffffff?)rr|zfastvit_t8.apple_dist_in1kzfastvit_t12.apple_dist_in1kzfastvit_s12.apple_dist_in1kzfastvit_sa12.apple_dist_in1kzfastvit_sa24.apple_dist_in1kzfastvit_sa36.apple_dist_in1kzfastvit_ma36.apple_dist_in1kzfastvit_mci0.apple_mclipzapple/mobileclip_s0_timmzXhttps://docs-assets.developer.apple.com/ml-research/datasets/mobileclip/mobileclip_s0.ptr)rrr)?rr)rryr|r+r~r}zfastvit_mci1.apple_mclipzapple/mobileclip_s1_timmzXhttps://docs-assets.developer.apple.com/ml-research/datasets/mobileclip/mobileclip_s1.ptzfastvit_mci2.apple_mclipzapple/mobileclip_s2_timmzXhttps://docs-assets.developer.apple.com/ml-research/datasets/mobileclip/mobileclip_s2.ptcCsfd|vr|S|d|}d|vrd}nd}ddl}ddl}g}|D]\}}|d|}|r9|t|d q#tt t |}i} |D]\}}|rY||vrSqH| |d}| d d }| d d }| dd}| dd}| dd}| dd}| dd}| dd}| dd}| dd|}| dr| dd}| d d!}|d"r|d#krt|jd$rt|jjtjr| d#d%}|j}t|jd| d&<n| d"d'}|d(|}d)\} } |rt|d*} ||| } | dur,d+| } d,| } | d-|vr| | d-| d.}n| d/|vr$| | d/| d0}n| | | d1}|| |<qH| S)2z$ Remap original checkpoints -> timm zstem.0.conv_kxk.0.conv.weight state_dictz8image_encoder.model.patch_embed.0.rbr_conv.0.conv.weightzimage_encoder.model.rkrNz^(.*?)network\.(\d+)\.proj.*rY patch_embedr9rbr_convrA rbr_scalerBrbr_skipr>conv_exprG lkb_originrconvffnrz se.reducezse.fc1z se.expandzse.fc2zlayer_scale_([0-9])zlayer_scale_\1.gammarzlayer_scale.gamma dist_head head_distzhead.z head.projrZzhead.fc.weightz head.fc.biaszhead.fc.z^network\.(\d+))NNrznetwork.r1z.projz.downsample.projz.pez.pos_emb.pos_encz.blocks)rErebisectitemsmatchrrgrouplistsortedrQreplacesubendswith startswithrTrHrorZr9rTr[rur bisect_right)rmodelprefixrr stage_endsrrrout_dict stage_idxnet_idx net_prefix stage_prefixrrrcheckpoint_filter_fnsn                $        rcKs2|dd}tt||fttd|dd|}|S)NrAr5T)flatten_sequentialrA)pretrained_filter_fn feature_cfg)poprrrr=)variant pretrainedrrArrrr_create_fastvits  rcK.tddddd}td d|it|fi|S) z%Instantiate FastViT-T8 model variant.)rYrYrrY)0`irrrrrr$r&r'r% fastvit_t8rN)rr=rrr model_argsrrrrrcKr) z&Instantiate FastViT-T12 model variant.rrrrr fastvit_t12rN)rrrrrrrrrcKr) z&Instantiate FastViT-S12 model variant.rrr rr fastvit_s12rN)rrrrrrrrrc K@tddddddttddfdd}td d |it|fi|S) z'Instantiate FastViT-SA12 model variant.rrr Nrrr r r r r$r&r'r,r% fastvit_sa12r)rr=rrrrrrrrrc Kr) z'Instantiate FastViT-SA24 model variant.)rr rrr Nrrrr fastvit_sa24r)rrrrrrrrrc Kr) z'Instantiate FastViT-SA36 model variant.rrrrr Nrrrr fastvit_sa36r)rrrrrrrrrc Kr) z'Instantiate FastViT-MA36 model variant.r)Li0i`r Nrrrr fastvit_ma36r)rrrrrrr+rrc KDtdddddddttddfdd d }td d |it|fi|S)zInstantiate MCi0 model variant.)rYr rYrrFFTTNrrrTr$r&r'r)r,r%r fastvit_mci0r)rrrrrrr8 rc Kr)zInstantiate MCi1 model variant.)rrrrrrNrrrTr fastvit_mci1r)rrrrrrrGrrc Kr)zInstantiate MCi2 model variant.)rrr)Pi@irrNrrrTr fastvit_mci2r)rrrrrrrVrr)rkru)BrC functoolsrtypingrrrrrr[torch.nnr9 timm.datar r timm.layersr r r rrrr_builderr _featuresr _manipulater _registryrr__all__rrrrrrrrrrrrrrrrrrrr default_cfgsrrrrrrrrrrrrrrrrs&  $    r -D4 p4t:9c v  !&-4> I