o پi)@sdZddlmZddlZddlmZddlmmZddlm Z ddl m Z ddl m Z d eefd d ZGd d d ejZGdddejZdS)a] Halo Self Attention Paper: `Scaling Local Self-Attention for Parameter Efficient Visual Backbones` - https://arxiv.org/abs/2103.12731 @misc{2103.12731, Author = {Ashish Vaswani and Prajit Ramachandran and Aravind Srinivas and Niki Parmar and Blake Hechtman and Jonathon Shlens}, Title = {Scaling Local Self-Attention for Parameter Efficient Visual Backbones}, Year = {2021}, } Status: This impl is a WIP, there is no official ref impl and some details in paper weren't clear to me. The attention mechanism works but it's slow as implemented. Hacked together by / Copyright 2021 Ross Wightman )ListN)nn)make_divisible) trunc_normal_)_assert permute_maskc Cs|j\}}}}|jd}|dd}||dd} | d||} t| ddgd} t| d||g} | d|d|} | ddd||ddf} | ||d||dd|dd} | |S)a~ Compute relative logits along one dimension As per: https://gist.github.com/aravindsrinivas/56359b79f0ce4449bcb04ab4b56a57a2 Originally from: `Attention Augmented Convolutional Networks` - https://arxiv.org/abs/1904.09925 Args: q: (batch, height, width, dim) rel_k: (2 * window - 1, dim) permute_mask: permute output dim according to this rrN)shape transposereshapeFpadflattenexpandpermute) qrel_krBHWdimrel_sizewin_sizexx_padrI/home/ubuntu/.local/lib/python3.10/site-packages/timm/layers/halo_attn.py rel_logits_1ds    r cs(eZdZdZfddZddZZS) PosEmbedRelz Relative Position Embedding As per: https://gist.github.com/aravindsrinivas/56359b79f0ce4449bcb04ab4b56a57a2 Originally from: `Attention Augmented Convolutional Networks` - https://arxiv.org/abs/1904.09925 csZt||_||_tt|dd|||_tt|dd|||_ dS)z Args: block_size (int): block size win_size (int): neighbourhood window size dim_head (int): attention head dim scale (float): scale factor (for init) r rN) super__init__ block_sizedim_headr Parametertorchrandn height_rel width_rel)selfr$rr%scale __class__rrr#Cs  $zPosEmbedRel.__init__c Csl|j\}}}}|d|j|j|j}t||jdd}|dd}t||jdd}||}||||d}|S)Nr )rrr )rrr )rr/rr0r )r rr$r%r r*r r)) r+rrBBHW_ rel_logits_w rel_logits_h rel_logitsrrrforwardQs zPosEmbedRel.forward)__name__ __module__ __qualname____doc__r#r7 __classcell__rrr-rr!=s r!cs6eZdZdZ  dfdd Zd d Zd d ZZS)HaloAttna Halo Attention Paper: `Scaling Local Self-Attention for Parameter Efficient Visual Backbones` - https://arxiv.org/abs/2103.12731 The internal dimensions of the attention module are controlled by the interaction of several arguments. * the output dimension of the module is specified by dim_out, which falls back to input dim if not set * the value (v) dimension is set to dim_out // num_heads, the v projection determines the output dim * the query and key (qk) dimensions are determined by * num_heads * dim_head if dim_head is not None * num_heads * (dim_out * attn_ratio // num_heads) if dim_head is None * as seen above, attn_ratio determines the ratio of q and k relative to the output if dim_head not used Args: dim (int): input dimension to the module dim_out (int): output dimension of the module, same as dim if not set feat_size (Tuple[int, int]): size of input feature_map (not used, for arg compat with bottle/lambda) stride: output stride of the module, query downscaled if > 1 (default: 1). num_heads: parallel attention heads (default: 8). dim_head: dimension of query and key heads, calculated from dim_out * attn_ratio // num_heads if not set block_size (int): size of blocks. (default: 8) halo_size (int): size of halo overlap. (default: 3) qk_ratio (float): ratio of q and k dimensions to output dimension when dim_head not set. (default: 1.0) qkv_bias (bool) : add bias to q, k, and v projections avg_down (bool): use average pool downsample instead of strided query blocks scale_pos_embed (bool): scale the position embedding as well as Q @ K Nrr/?Fc sXt|p|}||dksJ|dvsJ||_|p%t|| dd||_||j|_||j|_||j|_|jd|_| |_ ||_ |_ ||_ ||d|_ d|_d} |dkrq| pb||dk} | rgdn||_|j |j|_ tj||jd|j| d |_tj||j|jd| d |_t|j |j |j|jd |_| rtddnt|_|dS) Nr)rr r>)divisorr rF)stridebias)rC)r$rr%r,)r"r# num_headsr dim_head_qk dim_head_v dim_out_qk dim_out_vr,scale_pos_embedr$ block_size_ds halo_sizer block_striderConv2drkvr! pos_embed AvgPool2dIdentitypoolreset_parameters)r+rdim_out feat_sizerBrDr%r$rKqk_ratioqkv_biasavg_downrI use_avg_poolr-rrr#}s6        zHaloAttn.__init__cCsZ|jjjdd}t|jj|dt|jj|dt|jj|jdt|jj|jddS)NrrA)std) rweightr rrNrOr)r,r*)r+rZrrrrSs zHaloAttn.reset_parameterscCs|j\}}}}t||jdkdt||jdkd||j}||j}||}||} | d|j||j||jdddddd} | ||j|jd| dd} | |} t | |j |j |j |j g} | d|j|jd|j|j||j|j|j|ddddd} tj| |j|jgdd \} } |jr| | dd || |j} n| | dd |j|| } | jdd } | |  dd}|d|j|j||}|ddddd||j||j||j}||}|S) Nrr rr/r r0)rr )r rr$rrrErJrrDr rNrrrKunfoldrrFr'splitrIrOr,softmax contiguousviewrHrLrR)r+rrCrr num_h_blocks num_w_blocks num_blocksrrNkvattnoutrrrr7sB        "   zHaloAttn.forward) NNrr>Nr>r/r?FFF)r8r9r:r;r#rSr7r<rrr-rr=as%r=)r;typingrr'rtorch.nn.functional functionalrhelpersr weight_initr trace_utilsrintr Moduler!r=rrrrs     $t