o oi-@sdZddlmZddlmZmZddlZddlmZddlm Z m Z m Z ddl m Z dgZGd d d e ZGd d d ejZGd dde ZGdddejZGddde ZGddde ZGddde ZgdZdddZdS)aeModule that implement Vision Transformer (ViT). Paper: https://paperswithcode.com/paper/an-image-is-worth-16x16-words-transformers-1 Based on: `https://towardsdatascience.com/implementing-visualttransformer-in-pytorch-184f9f16f632` Added some tricks from: `https://github.com/rwightman/pytorch-image-models/blob/master/timm/models/vision_transformer.py` ) annotations)AnyCallableN)nn)ModuleTensor concatenate) KORNIA_CHECKVisionTransformercs(eZdZd fdd Zdd d ZZS) ResidualAddfnCallable[..., Tensor]returnNonecst||_dSN)super__init__r )selfr  __class__F/home/ubuntu/.local/lib/python3.10/site-packages/kornia/contrib/vit.pyr)s  zResidualAdd.__init__xrkwargsrcKs"|}|j|fi|}||7}|Sr)r )rrrresrrrforward-szResidualAdd.forward)r r rr)rrrrrr__name__ __module__ __qualname__rr __classcell__rrrrr (sr cs eZdZd d fd d ZZS) FeedForward in_featuresinthidden_features out_features dropout_ratefloatrrc s8tt||tt|t||t|dSr)rrrLinearGELUDropout)rr#r%r&r'rrrr5s  zFeedForward.__init__)r") r#r$r%r$r&r$r'r(rrrrrrr rrrrr!4sr!cs(eZdZdfd d Zdd dZZS)MultiHeadAttentionemb_sizer$ num_headsatt_dropr( proj_droprrcst||_||_||}|d|_|j|jr'td|jd|jdt||d|_t ||_ t|||_ t ||_ dS)NgzySize of embedding inside the transformer decoder must be visible by number of headsfor correct multi-head attention Got: z embedding size and z numbers of heads) rrr.r/scale ValueErrorrr)qkvr+r0 projectionprojection_drop)rr.r/r0r1 head_sizerrrr@s"    zMultiHeadAttention.__init__rrc Cs|j\}}}||||d|j||jddddd}|d|d|d}}}td|||j} | jdd} | | } td | |} | dddd ||d} | | } | | } | S) Nr2rzbhqd, bhkd -> bhqkdimzbhal, bhlv -> bhav )shaper5reshaper/permutetorcheinsumr3softmaxr0 contiguousviewr6r7) rrBNCr5qkvattoutrrrrTs .    zMultiHeadAttention.forward) r.r$r/r$r0r(r1r(rrrrrrrrrrrr-?sr-cseZdZd fd d ZZS) TransformerEncoderBlock embed_dimr$r/r'r( dropout_attnrrc sdtttt|dt||||t|ttt|dt||d||dt|dS)Nư>r;)r') rrr r Sequential LayerNormr-r+r!)rrQr/r'rRrrrrjs    z TransformerEncoderBlock.__init__) rQr$r/r$r'r(rRr(rrr,rrrrrPisrPcs4eZdZ     ddfd d ZdddZZS)TransformerEncoder r"rQr$depthr/r'r(rRrrcs8ttjfddt|D|_g|_dS)Nc3s|] }tVqdSr)rP).0_rRr'rQr/rr sz.TransformerEncoder.__init__..)rrrrTrangeblocksresults)rrQrYr/r'rRrr\rr~s  zTransformerEncoder.__init__rrcCs2g|_|}|jD] }||}|j|q |Sr)r`r_childrenappend)rrrNmrrrrs zTransformerEncoder.forward)rWrXrXr"r") rQr$rYr$r/r$r'r(rRr(rrrOrrrrrrV}srVcsBeZdZdZ     ddfdd ZdddZdddZZS)PatchEmbeddingzJCompute the 2d image patch embedding ready to pass to transformer encoder.r2rWN in_channelsr$ out_channels patch_size image_sizebackbone Module | Nonerrcst||_||_||_|ptj||||d|_|dur,||||f\}}||_n||d}t t dd||_ t t |d||_ dS)N) kernel_sizestrider9r:)rrrgrhrirConv2drk_compute_feats_dims ParameterrBrandn cls_token positions)rrgrhrirjrk feat_sizerrrrs  zPatchEmbedding.__init__tuple[int, int, int]tuple[int, int]cCs:|tjdg|R}|jd|jd|jdfS)Nr:r<)rkrBzerosdetachr?)rrjrNrrrrpsz"PatchEmbedding._compute_feats_dimsrrcCs^||}|j\}}}}|||dddd}|j|dd}t||gdd}||j7}|S)Nr<rr9r:r=)rkr?rFrArsrepeatrrt)rrrGrHr[ cls_tokensrrrrs  zPatchEmbedding.forward)r2rWrerfN) rgr$rhr$rir$rjr$rkrlrr)rjrvrrwrO)rrr__doc__rrprr rrrrrds rdcs^eZdZdZ         d)d*fdd Zed+ddZd,ddZed-d.d'd(Z Z S)/r aVision transformer (ViT) module. The module is expected to be used as operator for different vision tasks. The method is inspired from existing implementations of the paper :cite:`dosovitskiy2020vit`. .. warning:: This is an experimental API subject to changes in favor of flexibility. Args: image_size: the size of the input image. patch_size: the size of the patch to compute the embedding. in_channels: the number of channels for the input. embed_dim: the embedding dimension inside the transformer encoder. depth: the depth of the transformer. num_heads: the number of attention heads. dropout_rate: dropout rate. dropout_attn: attention dropout rate. backbone: an nn.Module to compute the image patches embeddings. Example: >>> img = torch.rand(1, 3, 224, 224) >>> vit = VisionTransformer(image_size=224, patch_size=16) >>> vit(img).shape torch.Size([1, 197, 768]) rfrer2rWrXr"Nrjr$rirgrQrYr/r'r(rRrkrlrrc s`t||_||_||_||_t||||| |_|jj} t | |||||_ t | d|_ dS)NrS)rrrjrirg embed_sizerdpatch_embeddingrhrVencoderrrUnorm) rrjrirgrQrYr/r'rRrk hidden_dimrrrrs zVisionTransformer.__init__ list[Tensor]cCs|jjSr)rr`)rrrrencoder_resultssz!VisionTransformer.encoder_resultsrrc Cst|tstdt||jg|jddRvr7|jd|jkr7td|jd|jd|jd|j||}| |}| |}|S)Nz#Input x type is not a Tensor. Got: ryrxzInput image shape must be Bxrz. Got: ) isinstancer TypeErrortyperjr?rgr4rrr)rrrNrrrrs *"   zVisionTransformer.forwardFvariantstr pretrainedboolrrc Ks|d\}}t|}ddddddddddddd d d dd d d dd|}|j||dtdi|}|rHt|}tj|} || |S)aBuild ViT model based on the given config string. The format is ``vit_{size}/{patch_size}``. E.g. ``vit_b/16`` means ViT-Base, patch size 16x16. If ``pretrained=True``, AugReg weights are loaded. The weights are hosted on HuggingFace's model hub: https://huggingface.co/kornia. .. note:: The available weights are: ``vit_l/16``, ``vit_b/16``, ``vit_s/16``, ``vit_ti/16``, ``vit_b/32``, ``vit_s/32``. Args: variant: ViT model variant e.g. ``vit_b/16``. pretrained: whether to load pre-trained AugReg weights. kwargs: other keyword arguments that will be passed to :func:`kornia.contrib.vit.VisionTransformer`. Returns: The respective ViT model Example: >>> from kornia.contrib import VisionTransformer >>> vit_model = VisionTransformer.from_config("vit_b/16", pretrained=True) /rXr2)rQrYr/irWirei )vit_tivit_svit_bvit_lvit_h)riNr) splitr$updater _get_weight_urlrBhubload_state_dict_from_urlload_state_dict) rrr model_typepatch_size_strri model_configmodelurl state_dictrrr from_configs"       zVisionTransformer.from_config) rfrer2rWrXrXr"r"N)rjr$rir$rgr$rQr$rYr$r/r$r'r(rRr(rkrlrr)rrrO)F)rrrrrrrr ) rrrr~rpropertyrr staticmethodrr rrrrr s"  )zvit_l/16zvit_b/16zvit_s/16z vit_ti/16zvit_b/32zvit_s/32rrrcCs@t|tvd|d|d\}}d||d|d|dS)z$Return the URL of the model weights.zVariant z% does not have pre-trained checkpointrzhttps://huggingface.co/kornia/z_augreg_i21k_r224/resolve/main/-z.pth)r _AVAILABLE_WEIGHTSr)rrrirrrr8sr)rrrr)r~ __future__rtypingrrrBr kornia.corerrrkornia.core.checkr __all__r rTr!r-rPrVrdr rrrrrrs"     *+u