o oiS@spddlmZddlZddlmZmZmZddlZddlm m Z ddlm Z ddl m Z ddlmZmZmZmZddlmZmZddlmZd3d dZGddde jZGddde jZGdddeZGdddeZGdddeZGddde jZGdddeZ GdddeZ!Gdd d eZ"Gd!d"d"eZ#d4d&d'Z$d5d6d-d.Z%d5d6d/d0Z&d5d6d1d2Z'dS)7) annotationsN)AnyOptionalSequence)nn) checkpoint)DropPath LayerNorm2dwindow_partitionwindow_unpartition)ModuleTensor) KORNIA_CHECKxint | tuple[int, int]returntuple[int, int]cCst|tr ||fS|SN) isinstanceint)rrR/home/ubuntu/.local/lib/python3.10/site-packages/kornia/contrib/models/tiny_vit.py _make_pair$srcs*eZdZdddejfdfdd ZZS)ConvBNr in_channelsr out_channels kernel_sizestridepaddinggroups activation type[Module]rNonec s<ttj||||||dd|_t||_||_dS)NF)r bias)super__init__rConv2dc BatchNorm2dbnact)selfrrrrrr r! __class__rrr&)s   zConvBN.__init__)rrrrrrrrrrr rr!r"rr#)__name__ __module__ __qualname__rIdentityr& __classcell__rrr-rr(s rcs$eZdZejfd fdd ZZS) PatchEmbedrr embed_dimr!r"rr#c s@ttt||dddd|t|d|ddd|_dS)Nr)r%r&r Sequentialrseq)r,rr5r!r-rrr&:s ( zPatchEmbed.__init__)rrr5rr!r"rr#r/r0r1rGELUr&r3rrr-rr49sr4cs0eZdZejdfdfd d ZdddZZS)MBConvrrrexpansion_ratiofloatr!r" drop_pathrr#csbtt||}t||d|d|_t||ddd|||_t||d|_t||_||_ dS)Nrr!r7) r%r&rrconv1conv2conv3rr@r+)r,rrr>r!r@hidden_channelsr-rrr&Bs    zMBConv.__init__rr c Cs&|||||||Sr)r+r@rDrCrBr,rrrrforwardRs&zMBConv.forward) rrrrr>r?r!r"r@r?rr#rr rr r/r0r1rr;r&rGr3rrr-rr<As r<cs.eZdZejfdfd d ZdddZZS) PatchMerginginput_resolutionrdimrout_dimrr!r"rr#c s^t|dvdtt||_t||d|d|_t||d|d||d|_t||d|_dS)N)rr6zstride must be either 1 or 2rrAr7)r r!) rr%r&rrKrrBrCrD)r,rKrLrMrr!r-rrr&Ws   zPatchMerging.__init__rr cCsL|jdkr|ddd|j}||||}|ddd}|S)Nr7rr6)ndim transpose unflattenrKrDrCrBflattenrFrrrrGfs zPatchMerging.forward) rKrrLrrMrrrr!r"rr#rHrIrrr-rrJVsrJcs6eZdZejddddfdfdd ZdddZZS) ConvLayerr=NF@rLrdepthr!r"r@float | list[float] downsampleOptional[Module]use_checkpointboolconv_expand_ratior?rr#csRt||_ttsg|tfddt|D|_||_ dS)Nc s g|] }t|qSr)r<.0ir!rZrLr@rr s z&ConvLayer.__init__..) r%r&rXrlistr ModuleListrangeblocksrV)r,rLrTr!r@rVrXrZr-r^rr&os    zConvLayer.__init__rr cC>|jD]}|jrt||n||}q|jdur||}|SrrcrXrrVr,rblkrrrrG   zConvLayer.forward)rLrrTrr!r"r@rUrVrWrXrYrZr?rr#rHrIrrr-rrRnsrRcs&eZdZejdfdfd d ZZS)MLPr= in_featuresrhidden_features out_featuresr!r"dropr?rr#csVtt||_t|||_||_t||_ t|||_ t||_ dSr) r%r&r LayerNormnormLinearfc1act1Dropoutdrop1fc2drop2)r,rjrkrlr!rmr-rrr&s   z MLP.__init__) rjrrkrrlrr!r"rmr?rr#r:rrr-rrisricsVeZdZ   ddfdd ZedddZedd fdd Zd!ddZ Z S)" AttentionrSrzrLrkey_dim num_heads attn_ratior? resolutionrrr#c st||_|d|_||_|||_t|||_t||||_||_ |j|jd}t ||_ t |||_t |j||_||\}}t t|||_|jd|dd|d|_dS)Ngr6attention_bias_idxsF) persistent)r%r&r|scaler{nh_kdrddhr}rrnrorpqkvprojbuild_attention_bias Parametertorchzerosattention_biasesregister_bufferab) r,rLr{r|r}r~hindicesattn_offset_sizer-rrr&s"     zAttention.__init__tuple[Tensor, int]cCs|\}}t|}t|}||}||}|dddf|dddf}|dddf|dddf}|||} tj| dd\} } | ||||} | } | | fS)NT)return_inverse)rarangerepeat_interleaverepeatabsuniqueviewnumel)r~HWrowscolsrrccdrdckeys unique_keysinverserrrrrrs    $$ zAttention.build_attention_biasTmoderYcs>t||r|jdurd|_|S|jdd|jf|_|Sr)r%trainrrr)r,rr-rrrs zAttention.trainrr c Cs|j\}}}||}||}||||jddddd}|j|j|j|jgdd\}}}|j r<|j dd|j fn|j } || dd|j| } | jdd} | | dd|||j}||}|S)Nrr6rr7)rL)shaperorrr|permutesplitr{rtrainingrrrrOrsoftmaxreshaperr) r,rBN_rqkvr$attnrrrrGs       zAttention.forward)rxrSry) rLrr{rr|rr}r?r~rrr#)r~rrr)T)rrYrrwrH) r/r0r1r& staticmethodrrno_gradrrGr3rrr-rrws rwcs8eZdZdddddejfdfdd ZdddZZS) TinyViTBlockrSr=r7rLrrKrr| window_size mlp_ratior?rmr@local_conv_sizer!r"rr#c st||dkdtt||_||_||} t|| |d||f|_t||_ t |||d|d||_ t |t |||| ||_t||_dS)Nrz!dim must be divislbe by num_headsg?rr6)rr%r&rrKrrwrr drop_path1r local_convrirmlp drop_path2) r,rLrKr|rrrmr@rr!head_dimr-rrr&s   zTinyViTBlock.__init__rr c Cs|j\}}|j\}}}|}|||||}t||j\}}||dd}t||j|||f}||||}|||}| dd ||||}| |}|||| dd}|| | |}|S)Nrr6)rKrrr rrrQr rrOrrrr) r,rrrrLCres_xpad_hwrrrrGs   zTinyViTBlock.forward)rLrrKrr|rrrrr?rmr?r@r?rrr!r"rr#rHrIrrr-rrsrcs:eZdZddddddejfd!fdd Zd"dd ZZS)# BasicLayerrSr=NFr7rLrrKrrTr|rrr?rmr@rUrVrWrXrYrr!r"rr#c sHt| |_tf ddt|D|_| |_dS)Nc s6g|]}tttr|n qSr)rrr`r[ r!rLrmr@rKrrr|rrrr_(s z'BasicLayer.__init__..)r%r&rXrrarbrcrV) r,rLrKrTr|rrrmr@rVrXrr!r-rrr&s   zBasicLayer.__init__rr cCrdrrerfrrrrG;rhzBasicLayer.forward)rLrrKrrTrr|rrrrr?rmr?r@rUrVrWrXrYrrr!r"rr#rHrIrrr-rrs&rcs^eZdZdZdddddddd d d d d dejd fd0fd"d# Zd1d&d'Zed2d3d.d/Z Z S)4TinyViTaTinyViT model, as described in https://arxiv.org/abs/2207.10666. Args: img_size: Size of input image. in_chans: Number of input image's channels. num_classes: Number of output classes. embed_dims: List of embedding dimensions. depths: List of block count for each downsampling stage num_heads: List of attention heads used in self-attention for each downsampling stage. window_sizes: List of self-attention's window size for each downsampling stage. mlp_ratio: Ratio of MLP dimension to embedding dimension in self-attention. drop_rate: Dropout rate. drop_path_rate: Stochastic depth rate. use_checkpoint: Whether to use activation checkpointing to trade compute for memory. mbconv_expand_ratio: Expansion ratio used in MBConv block. local_conv_size: Kernel size of convolution used in TinyViTBlock activation: activation function. mobile_same: Whether to use modifications for MobileSAM. r7i)`ir6r6r6)r7r rrrzrrSr=Fimg_sizerin_chans num_classes embed_dims Sequence[int]depthsr| window_sizesrr? drop_ratedrop_path_raterXrYmbconv_expand_ratiorr!r" mobile_samrr#c st||_||_||r3gd}ttj|dddddtdtjdddddddtd|_ngd}d|_t ||d ||_ |d }d d t d | t |D}t|}g}tt|||||D]f\}\}}}}}|t|dt|d}||dkrt|||||nd}|||t |d|t |d|d|| |d }|d krtdd| i|}ntd||||| | d|}||||}qetj||_||_t|d|_t|d||_dS)N)r6r6rrrrF)r$r7)r6r6r6rrcSsg|]}|qSr)item)r\rrrrr_sz$TinyViT.__init__..)rLrTr@rVrXr!rZ)rKr|rrrmrr)r%r&rrrr8r'r neckr4 patch_embedrlinspacesumlen enumeratezipminrJrRrappendlayers feat_sizern norm_headrphead)r,rrrrrr|rrrrrXrrr!rstridesrKdprn_layersri_layerr5rT num_heads_irrrMrVkwargslayerr-rrr&Ysh  &   zTinyViT.__init__rr cCsf||}||}|jr$|d|j|jfdddd}||}|S|d}|| |}|S)zUClassify images if ``mobile_sam=False``, produce feature maps if ``mobile_sam=True``.rrr7r6) rrrrPrrrmeanrrrFrrrrGs     zTinyViT.forwardvariantstr pretrained bool | strrrcKs*t|dvdtttd||fi|S)a1Create a TinyViT model from pre-defined variants. Args: variant: TinyViT variant. Possible values: ``'5m'``, ``'11m'``, ``'21m'``. pretrained: whether to use pre-trained weights. Possible values: ``False``, ``True``, ``'in22k'``, ``'in1k'``. For TinyViT-21M (``variant='21m'``), ``'in1k_384'``, ``'in1k_512'`` are also available. **kwargs: other keyword arguments that will be passed to :class:`TinyViT`. .. note:: When ``img_size`` is different from the pre-trained size, bicubic interpolation will be performed on attention biases. When using ``pretrained=True``, ImageNet-1k checkpoint (``'in1k'``) is used. For feature extraction or fine-tuning, ImageNet-22k checkpoint (``'in22k'``) is preferred. )5m11m21mz+Only variant 5m, 11m, and 21m are supported)r _tiny_vit_5m _tiny_vit_11m _tiny_vit_21m)rrrrrr from_configszTinyViT.from_config) rrrrrrrrrrr|rrrrr?rr?rr?rXrYrr?rrr!r"rrYrr#rHF)rrrrrrrr) r/r0r1__doc__rr;r&rGrrr3rrr-rrCs* YrmodelurlrcCs(|}tj|}d|vr|d}dd|D}|D]I}||j\}}||j\}} t||kd|d||| krft|d} t| d} ||d|| | } t j | | | fdd } | || ||<q|d jd |j j krd } t j| dd t|j j|d <t|j j|d<|||S)NrcSsg|]}d|vr|qS)rr)r\rrrrr_sz$_load_pretrained..z Fail to load z/. Pre-trained checkpoint should have num_heads=g?rbicubic)sizerz head.weightrzaNumber of classes does not match pre-trained checkpoint's. Resetting classification head to zeros) stacklevelz head.bias) state_dictrhubload_state_dict_from_urlrrrrrF interpolaterrlwarningswarn zeros_likeweightr$load_state_dict)rrmodel_state_dictrab_keysrn_heads1L1n_heads2L2S1S2rmsgrrr_load_pretraineds.    rFrrrrcKVtd gdgdgdgddd|}|r)|durd}d d d |}t||}|S) N)@i@r)r6r rr=rrr|rrTin1kzchttps://github.com/wkcn/TinyViT-model-zoo/releases/download/checkpoints/tiny_vit_5m_22k_distill.pthzghttps://github.com/wkcn/TinyViT-model-zoo/releases/download/checkpoints/tiny_vit_5m_22kto1k_distill.pthin22kr"rrrrrrrrrrr&  rcKr) N)rrrir)r6rrxrzrg?r!Tr"zdhttps://github.com/wkcn/TinyViT-model-zoo/releases/download/checkpoints/tiny_vit_11m_22k_distill.pthzhhttps://github.com/wkcn/TinyViT-model-zoo/releases/download/checkpoints/tiny_vit_11m_22kto1k_distill.pthr#rr%r&rrrrr'rcKs~tdgdgdgdgddd|}|r=|dur/d}|d d }|d kr)d }|d kr/d}ddddd|}t||}|S)N)rrri@r)r7rrrg?r!Tr"rrrin1k_384iin1k_512zdhttps://github.com/wkcn/TinyViT-model-zoo/releases/download/checkpoints/tiny_vit_21m_22k_distill.pthzhhttps://github.com/wkcn/TinyViT-model-zoo/releases/download/checkpoints/tiny_vit_21m_22kto1k_distill.pthzlhttps://github.com/wkcn/TinyViT-model-zoo/releases/download/checkpoints/tiny_vit_21m_22kto1k_384_distill.pthzlhttps://github.com/wkcn/TinyViT-model-zoo/releases/download/checkpoints/tiny_vit_21m_22kto1k_512_distill.pth)r$r"r)r*r)rgetr)rrrrrrrrr(s4   r)rrrr)rrrrrrr)rrrrrr)( __future__rr typingrrrrtorch.nn.functionalr functionalr  torch.utilsrkornia.contrib.models.commonrr r r kornia.corer r kornia.core.checkrrr8rr4r<rJrRrirwrrrrrrrrrrrs4      C./  !