o €o™i>)ã @sâddlmZmZmZddlZddlmZddlmZmZde de defdd „Z dde de d e d e def dd„Z Gdd„deƒZ Gdd„deƒZ Gdd„deƒZGdd„deƒZGdd„deƒZGdd„deƒZGdd„deƒZdS)é)ÚAnyÚDictÚTupleN)Únn)ÚModuleÚTensorÚinpÚoupÚreturnc Cs*t tj||dddddt |¡t ¡¡S)z&Apply 1x1 Convolution with Batch Norm.érF©Úbias©rÚ SequentialÚConv2dÚ BatchNorm2dÚSiLU)rr ©rúM/home/ubuntu/.local/lib/python3.10/site-packages/kornia/contrib/vit_mobile.pyÚ conv_1x1_bnó*rér Ú kernal_sizeÚstridec Cs*t tj||||dddt |¡t ¡¡S)z&Apply NxN Convolution with Batch Norm.r Fr r)rr rrrrrÚ conv_nxn_bnrrcsHeZdZdededdf‡fdd„ Zdedeee fdefd d „Z ‡Z S) ÚPreNormÚdimÚfnr Ncs tƒ ¡t |¡|_||_dS©N)ÚsuperÚ__init__rÚ LayerNormÚnormr)Úselfrr©Ú __class__rrr %s   zPreNorm.__init__ÚxÚkwargscKs|j| |¡fi|¤ŽSr)rr")r#r&r'rrrÚforward*szPreNorm.forward) Ú__name__Ú __module__Ú __qualname__Úintrr rrÚstrrr(Ú __classcell__rrr$rr$s&rc sBeZdZd dedededdf‡fdd„ Zd edefd d „Z‡ZS) Ú FeedForwardçrÚ hidden_dimÚdropoutr Nc sBtƒ ¡t t ||¡t ¡t |¡t ||¡t |¡¡|_dSr)rr rrÚLinearrÚDropoutÚnet)r#rr1r2r$rrr /s * ÿzFeedForward.__init__r&cCs | |¡Sr)r5©r#r&rrrr(5s zFeedForward.forward©r0© r)r*r+r,Úfloatr rr(r.rrr$rr/.s r/c sFeZdZddededededd f ‡fd d „ Zd edefd d„Z‡ZS)Ú Attentionéé@r0rÚheadsÚdim_headr2r NcsŠtƒ ¡||}|dko||k }||_|d|_tjdd|_tj||ddd|_|r>t  t ||¡t  |¡¡|_ dSt  ¡|_ dS)Nr gà¿éÿÿÿÿ©rrFr ) rr r=ÚscalerÚSoftmaxÚattendr3Úto_qkvrr4ÚIdentityÚto_out)r#rr=r>r2Ú inner_dimÚ project_outr$rrr :s  2zAttention.__init__r&c s˜ˆ |¡jddd}|dj\‰‰‰‰‡‡‡‡‡fdd„|Dƒ\}}}t || dd¡¡ˆj}ˆ |¡}t ||¡}| dd¡ ˆˆˆˆ¡}ˆ  |¡S) Nrr?r@rc 3s2|]}| ˆˆˆˆjˆˆj¡ dd¡VqdS)érN)Úreshaper=Ú transpose)Ú.0Út©ÚbÚhdÚnÚpr#rrÚ Ks€0z$Attention.forward..éþÿÿÿrI) rDÚchunkÚshapeÚtorchÚmatmulrKrArCrJrF) r#r&ÚqkvÚqÚkÚvÚdotsÚattnÚoutrrNrr(Gs    zAttention.forward)r;r<r0r8rrr$rr:9s$ r:csReZdZdZddedededededed d f‡fd d „ Zd ed efdd„Z‡Z S)Ú Transformera¢Transformer block described in ViT. Paper: https://arxiv.org/abs/2010.11929 Based on: https://github.com/lucidrains/vit-pytorch Args: dim: input dimension. depth: depth for transformer block. heads: number of heads in multi-head attention layer. dim_head: head size. mlp_dim: dimension of the FeedForward layer. dropout: dropout ratio, defaults to 0. r0rÚdepthr=r>Úmlp_dimr2r Nc s\tƒ ¡t g¡|_t|ƒD]}|j t t|t||||ƒƒt|t |||ƒƒg¡¡qdSr) rr rÚ ModuleListÚlayersÚrangeÚappendrr:r/)r#rrar=r>rbr2Ú_r$rrr ds   þÿÿÿzTransformer.__init__r&cCs,|jD]\}}||ƒ|}||ƒ|}q|Sr)rd)r#r&r^Úffrrrr(qs zTransformer.forwardr7) r)r*r+Ú__doc__r,r9r rr(r.rrr$rr`Ts, r`c sJeZdZdZddededededd f ‡fd d „ Zd edefd d„Z‡ZS)ÚMV2BlockaqMV2 block described in MobileNetV2. Paper: https://arxiv.org/pdf/1801.04381 Based on: https://github.com/tonylins/pytorch-mobilenet-v2 Args: inp: input channel. oup: output channel. stride: stride for convolution, defaults to 1, set to 2 if down-sample. expansion: expansion ratio for hidden dimension, defaults to 4. r érr rÚ expansionr Ncsötƒ ¡||_t||ƒ}|jdko||k|_|dkrCt tj||d|d|ddt |¡t  ¡tj||dddddt |¡¡|_ dSt tj||dddddt |¡t  ¡tj||d|d|ddt |¡t  ¡tj||dddddt |¡¡|_ dS)Nr rF)Úgroupsr rr ) rr rr,Úuse_res_connectrrrrrÚconv)r#rr rrlr1r$rrr †s,   ù  õzMV2Block.__init__r&cCs|jr || |¡S| |¡Sr)rnror6rrrr(¦s zMV2Block.forward)r rk) r)r*r+rir,r rr(r.rrr$rrjxs$  rjcs`eZdZdZ ddededededeeefded ed d f‡fd d „ Zded efdd„Z ‡Z S)ÚMobileViTBlockayMobileViT block mentioned in MobileViT. Args: dim: input dimension of Transformer. depth: depth of Transformer. channel: input channel. kernel_size: kernel size. patch_size: patch size for folding and unfloding. mlp_dim: dimension of the FeedForward layer in Transformer. dropout: dropout ratio, defaults to 0. r0rraÚchannelÚ kernel_sizeÚ patch_sizerbr2r Ncsftƒ ¡|\|_|_t|||ƒ|_t||ƒ|_t||dd||ƒ|_ t||ƒ|_ td|||ƒ|_ dS)Nrkr;rI) rr ÚphÚpwrÚconv1rÚconv2r`Ú transformerÚconv3Úconv4)r#rrarqrrrsrbr2r$rrr »s   zMobileViTBlock.__init__r&c Csö| ¡}| |¡}| |¡}|j\}}}}||j||j}}| ||||j||j¡ dd¡}| |||||j|j¡ dd¡}| |¡}| dd¡ |||||j|j¡}| dd¡ ||||¡}|  |¡}t   ||fd¡}|  |¡}|S)Nr rIr) ÚclonervrwrVrtrurJrKrxryrWÚcatrz) r#r&ÚyrOÚdÚhÚwÚnhÚnwrrrr(Ðs  $$ $  zMobileViTBlock.forwardr7) r)r*r+rir,rr9r rr(r.rrr$rrp­s*øþýüû úùø ÷rpc sTeZdZdZ ddededeeefd ed d f ‡fd d „ Zde d e fdd„Z ‡Z S)Ú MobileViTa>Module MobileViT. Default arguments is for MobileViT XXS. Paper: https://arxiv.org/abs/2110.02178 Based on: https://github.com/chinhsuanwu/mobilevit-pytorch Args: mode: 'xxs', 'xs' or 's', defaults to 'xxs'. in_channels: the number of channels for the input image. patch_size: image_size must be divisible by patch_size. dropout: dropout ratio in Transformer. Example: >>> img = torch.rand(1, 3, 256, 256) >>> mvit = MobileViT(mode='xxs') >>> mvit(img).shape torch.Size([1, 320, 8, 8]) Úxxsr©rIrIr0ÚmodeÚ in_channelsrsr2r Nc s&tƒ ¡|dkrd}gd¢}gd¢}n|dkr#d}gd¢}gd¢}n|d kr1d}gd ¢}gd ¢}d }gd ¢} t||ddd|_t g¡|_|j t|d|dd|ƒ¡|j t|d|dd|ƒ¡|j t|d|d d|ƒ¡|j t|d|d d|ƒ¡|j t|d |dd|ƒ¡|j t|d|dd|ƒ¡|j t|d|dd|ƒ¡t g¡|_ |j  t |d| d|d||t |ddƒ|d¡|j  t |d| d|d||t |ddƒ|d¡|j  t |d| d|d||t |ddƒ|d¡t |d|dƒ|_ dS)Nr„rI)r<éPé`) érŠér‹é0rŒr<r<rˆrˆi@Úxsrk)r‰éxé) rŠé rŒrŒr<r<rˆrˆr‰r‰i€Ús)réÀéð) rŠrr<r<r‰r‰é€r”é r•i€r)rIrkrr)rr ééér;)r2é rTr?)rr rrvrrcÚmv2rfrjÚmvitrpr,rrw) r#r†r‡rsr2rlÚdimsÚchannelsrrrar$rrr sF     ,ÿ,ÿ,ÿzMobileViT.__init__r&cCs¤| |¡}|jd|ƒ}|jd|ƒ}|jd|ƒ}|jd|ƒ}|jd|ƒ}|jd|ƒ}|jd|ƒ}|jd|ƒ}|jd|ƒ}|jd|ƒ}| |¡}|S)Nrr rIrrkr–r—)rvršr›rwr6rrrr(.s  zMobileViT.forward)r„rr…r0) r)r*r+rir-r,rr9r rr(r.rrr$rrƒîsÿÿÿ ÿÿþ,rƒ)rr )ÚtypingrrrrWrÚ kornia.corerrr,rrrr/r:r`rjrprƒrrrrÚs    $5A