o پi@sdZddlmZddlZddlmZddlmmZddl m Z m Z ddl m Z ddlmZdeefd d ZGd d d ejZGd ddejZdS)aP Bottleneck Self Attention (Bottleneck Transformers) Paper: `Bottleneck Transformers for Visual Recognition` - https://arxiv.org/abs/2101.11605 @misc{2101.11605, Author = {Aravind Srinivas and Tsung-Yi Lin and Niki Parmar and Jonathon Shlens and Pieter Abbeel and Ashish Vaswani}, Title = {Bottleneck Transformers for Visual Recognition}, Year = {2021}, } Based on ref gist at: https://gist.github.com/aravindsrinivas/56359b79f0ce4449bcb04ab4b56a57a2 This impl is a WIP but given that it is based on the ref gist likely not too far off. Hacked together by / Copyright 2021 Ross Wightman )ListN) to_2tuplemake_divisible) trunc_normal_)_assert permute_maskc Cs|j\}}}}||dd}|d|d|d}t|ddgd}t|d|dg}|d|dd|d}|ddd||ddf}|||d||dd|dd}||S)a Compute relative logits along one dimension As per: https://gist.github.com/aravindsrinivas/56359b79f0ce4449bcb04ab4b56a57a2 Originally from: `Attention Augmented Convolutional Networks` - https://arxiv.org/abs/1904.09925 Args: q: (batch, heads, height, width, dim) rel_k: (2 * width - 1, dim) permute_mask: permute output dim according to this rrN)shape transposereshapeFpadflattenexpandpermute) qrel_krBHWdimxx_padrO/home/ubuntu/.local/lib/python3.10/site-packages/timm/layers/bottleneck_attn.py rel_logits_1ds   rcs(eZdZdZfddZddZZS) PosEmbedRelz Relative Position Embedding As per: https://gist.github.com/aravindsrinivas/56359b79f0ce4449bcb04ab4b56a57a2 Originally from: `Attention Augmented Convolutional Networks` - https://arxiv.org/abs/1904.09925 cshtt|\|_|_||_tt |jdd|||_ tt |jdd|||_ dS)Nr r) super__init__rheightwidthdim_headnn Parametertorchrandn height_rel width_rel)self feat_sizer$scale __class__rrr!=s "&zPosEmbedRel.__init__cCsf|j\}}}|||j|jd}t||jdd}|dd}t||jdd}||}||||}|S)Nr )rrr )rrr )rr0rr1r )r rr"r#rr*r r))r+rrHW_ rel_logits_w rel_logits_h rel_logitsrrrforwardDs  zPosEmbedRel.forward)__name__ __module__ __qualname____doc__r!r7 __classcell__rrr.rr8s rcs6eZdZdZ  d fdd Zd d Zd d ZZS)BottleneckAttna Bottleneck Attention Paper: `Bottleneck Transformers for Visual Recognition` - https://arxiv.org/abs/2101.11605 The internal dimensions of the attention module are controlled by the interaction of several arguments. * the output dimension of the module is specified by dim_out, which falls back to input dim if not set * the value (v) dimension is set to dim_out // num_heads, the v projection determines the output dim * the query and key (qk) dimensions are determined by * num_heads * dim_head if dim_head is not None * num_heads * (dim_out * attn_ratio // num_heads) if dim_head is None * as seen above, attn_ratio determines the ratio of q and k relative to the output if dim_head not used Args: dim (int): input dimension to the module dim_out (int): output dimension of the module, same as dim if not set stride (int): output stride of the module, avg pool used if stride == 2 (default: 1). num_heads (int): parallel attention heads (default: 4) dim_head (int): dimension of query and key heads, calculated from dim_out * attn_ratio // num_heads if not set qk_ratio (float): ratio of q and k dimensions to output dimension when dim_head not set. (default: 1.0) qkv_bias (bool): add bias to q, k, and v projections scale_pos_embed (bool): scale the position embedding as well as Q @ K Nrr1?Fc st|dus Jd|p|}||dksJ||_|p't||dd||_||j|_||j|_||j|_|jd|_| |_ t j ||jd|jd|d|_ t ||j|jd |_|dkrht ddnt |_|dS) NzBA concrete feature size matching expected input (H, W) is requiredr)divisorr r)bias)r$r-)r r! num_headsr dim_head_qk dim_head_v dim_out_qk dim_out_vr-scale_pos_embedr%Conv2dqkvr pos_embed AvgPool2dIdentitypoolreset_parameters) r+rdim_outr,striderCr$qk_ratioqkv_biasrHr.rrr!js       zBottleneckAttn.__init__cCsFt|jj|jjjdddt|jj|jdt|jj|jddS)NrrA)std)rrJweightr rKr)r-r*)r+rrrrOszBottleneckAttn.reset_parametersc Cs |j\}}}}t||jjkdt||jjkd||}tj||j|j|j gdd\}}}| ||j |j d dd}| ||j |j d}| ||j |jd dd}|jrh|||||j} n |||j||} | jdd} | | dd ||j ||} || } | S)Nr)rr r )r rrKr"r#rJr'splitrFrGrrCrDr rErHr-softmaxrN) r+rrCrrrkvattnoutrrrr7s "  zBottleneckAttn.forward)NNrr1Nr>FF)r8r9r:r;r!rOr7r<rrr.rr=Tsr=)r;typingrr'torch.nnr%torch.nn.functional functionalrhelpersrr weight_initr trace_utilsrintrModulerr=rrrrs