o پiP@sXdZddlZddlmZmZmZmZmZmZm Z ddl Z ddl m Z ddl m mZddlmZmZddlmZmZmZmZmZmZddlmZddlmZdd lmZdd l m!Z!m"Z"d gZ#Gd d d e j$Z%Gddde j$Z&Gddde j$Z'Gddde j$Z(Gddde j$Z)Gddde j$Z*de+dede j$fddZ,dqdee-dfde.d e-dee-e-e-e-ffd!d"Z/Gd#d$d$e j$Z0Gd%d&d&e j$Z1  '  ( ) *  +drd,ed-e-d.e-d/ee-d0e-d1e-d2e-d3e-d4e.d5e2d6e.d7e.d8ede j3fd9d:Z4 ) *  +dsd,ed-e-d.e-d/ee-d0e-d4e.d5e2d6e.d7e.d8ede j3fd;d<Z5Gd=d d e j$Z6dtd>e+d?e2d8ede6fd@dAZ7dudCe+d8edee+effdDdEZ8e"e8dFdGdHdIe8dFdJdKdLdMe8dFdNdHdIe8dFdOdKdLdMe8dFdPdHdIe8dFdQdKdRdMe8dFdSdHdIe8dFdTdUdRdMe8dFdVdHdIe8dFdWdUdRdMe8dFdXdUdYdMdZ Z9e!dtd?e2d8ede6fd[d\Z:e!dtd?e2d8ede6fd]d^Z;e!dtd?e2d8ede6fd_d`Ze!dtd?e2d8ede6fdedfZ?e!dtd?e2d8ede6fdgdhZ@e!dtd?e2d8ede6fdidjZAe!dtd?e2d8ede6fdkdlZBe!dtd?e2d8ede6fdmdnZCe!dtd?e2d8ede6fdodpZDdS)va5 Vision OutLOoker (VOLO) implementation Paper: `VOLO: Vision Outlooker for Visual Recognition` - https://arxiv.org/abs/2106.13112 Code adapted from official impl at https://github.com/sail-sg/volo, original copyright in comment below Modifications and additions for timm by / Copyright 2022, Ross Wightman N)AnyCallableDictListOptionalTupleUnionIMAGENET_DEFAULT_MEANIMAGENET_DEFAULT_STD)DropPathMlp to_2tuple to_ntuple trunc_normal_use_fused_attn)build_model_with_cfg)feature_take_indices) checkpoint)register_modelgenerate_default_cfgsVOLOcsfeZdZdZ      ddededed ed ed ed ed effdd Zdej dej fddZ Z S)OutlookAttentionz,Outlook attention mechanism for VOLO models.rFdim num_heads kernel_sizepaddingstrideqkv_bias attn_drop proj_dropc st||} ||_||_||_||_| d|_tj|||d|_ t||d||_ t ||_ t|||_ t ||_tj|||d|_tj||dd|_dS)aInitialize OutlookAttention. Args: dim: Input feature dimension. num_heads: Number of attention heads. kernel_size: Kernel size for attention computation. padding: Padding for attention computation. stride: Stride for attention computation. qkv_bias: Whether to use bias in linear layers. attn_drop: Attention dropout rate. proj_drop: Projection dropout rate. ࿩bias)rrr T)rr ceil_modeN)super__init__rrrr scalennLinearvattnDropoutr"projr#Unfoldunfold AvgPool2dpool) selfrrrrr r!r"r#head_dim __class__D/home/ubuntu/.local/lib/python3.10/site-packages/timm/models/volo.pyr**s    zOutlookAttention.__init__xreturnc Csp|j\}}}}||dddd}t||jt||j}}||||j||j|j |j ||ddddd}| |dddddddd} | | ||||j|j |j |j |j ddddd} | |j } | j dd} || } | |ddddd|||j |j ||}tj|||f|j |j|jd}||dddd}||}|S) Forward pass. Args: x: Input tensor of shape (B, H, W, C). Returns: Output tensor of shape (B, H, W, C). rrrr'r) output_sizerrr )shaper.permutemathceilr r3reshaperrr5r/r+softmaxr"Ffoldrr1r#) r6r<BHWCr.hwr/r:r:r;forwardSs0 " "     0 zOutlookAttention.forward)rrrFrr) __name__ __module__ __qualname____doc__intboolfloatr*torchTensorrQ __classcell__r:r:r8r;r's4 )rcszeZdZdZdddddejejdfdededed ed ed ed ed ede de de ffdd Z de j de j fddZZS) Outlookerz9Outlooker block that combines outlook attention with MLP.r@rFrrrr r mlp_ratior" drop_path act_layer norm_layerr!c st| ||_t|||||| |d|_|dkrt|nt|_| ||_ t |t ||| d|_ |dkr?t||_ dSt|_ dS)afInitialize Outlooker block. Args: dim: Input feature dimension. kernel_size: Kernel size for outlook attention. padding: Padding for outlook attention. stride: Stride for outlook attention. num_heads: Number of attention heads. mlp_ratio: Ratio for MLP hidden dimension. attn_drop: Attention dropout rate. drop_path: Stochastic depth drop rate. act_layer: Activation layer type. norm_layer: Normalization layer type. qkv_bias: Whether to use bias in linear layers. )rrr r!r"r in_featureshidden_featuresr`N)r)r*norm1rr/r r,Identity drop_path1norm2r rVmlp drop_path2) r6rrrr rr^r"r_r`rar!r8r:r;r*ys&    $zOutlooker.__init__r<r=cC8|||||}|||||}|SzoForward pass. Args: x: Input tensor. Returns: Output tensor. rgr/rerjrirhr6r<r:r:r;rQ zOutlooker.forward)rRrSrTrUr,GELU LayerNormrVrXrrWr*rYrZrQr[r:r:r8r;r\vsD    2r\c sheZdZUdZejjeed<    dde de ded e d e f fd d Z d ej dej fddZ ZS) Attentionz!Multi-head self-attention module. fused_attnFrrrr!r"r#csjt||_||}|d|_t|_tj||d|d|_t ||_ t|||_ t ||_ dS)a,Initialize Attention module. Args: dim: Input feature dimension. num_heads: Number of attention heads. qkv_bias: Whether to use bias in QKV projection. attn_drop: Attention dropout rate. proj_drop: Projection dropout rate. r$rr%N) r)r*rr+rrsr,r-qkvr0r"r1r#)r6rrr!r"r#r7r8r:r;r*s   zAttention.__init__r<r=c Cs|j\}}}}|||||d|j||jddddd}|d\}}} |jrrr?rrr'r) dropout_pr@rA)rCrurGrrDunbindrsrIscaled_dot_product_attentiontrainingr"pr+ transposerHr1r#) r6r<rKrLrMrNruqkr.r/r:r:r;rQs" 2     zAttention.forward)rtFrr)rRrSrTrUrYjitFinalrW__annotations__rVrXr*rZrQr[r:r:r8r;rrs& rrcsjeZdZdZddddejejfdededede d ed ed e d e ffd d Z de j de j fddZZS) Transformerz9Transformer block with multi-head self-attention and MLP.@Frrrr^r!r"r_r`rac st|||_t||||d|_|dkrt|nt|_|||_ t |t |||d|_ |dkr