o پi~!@sddlmZmZmZddlZddlmZddlmZddl m Z ddl m Z ddej d eej fd d ZGd d d ejZGdddejZdS))FinalOptionalTypeN)nn) functional)use_fused_attn)apply_rot_embed_catscores attn_maskcCs|dur|S||SN)r r r r I/home/ubuntu/.local/lib/python3.10/site-packages/timm/layers/attention.pymaybe_add_mask srcseZdZUdZeeed<        dded ed ed ed ed ededede e e j ddffdd Z ddejde ejdejfddZZS) AttentionaStandard Multi-head Self Attention module with QKV projection. This module implements the standard multi-head attention mechanism used in transformers. It supports both the fused attention implementation (scaled_dot_product_attention) for efficiency when available, and a manual implementation otherwise. The module includes options for QK normalization, attention dropout, and projection dropout. fused_attnFTNdim num_headsqkv_biasqk_norm scale_norm proj_bias attn_drop proj_drop norm_layerreturnc st||dksJd|s|r| dusJd||_|||_|jd|_t|_tj||d|d|_ |r?| |jnt |_ |rK| |jnt |_ t ||_|r\| |nt |_tj|||d|_t ||_dS)agInitialize the Attention module. Args: dim: Input dimension of the token embeddings num_heads: Number of attention heads qkv_bias: Whether to use bias in the query, key, value projections qk_norm: Whether to apply normalization to query and key vectors proj_bias: Whether to use bias in the output projection attn_drop: Dropout rate applied to the attention weights proj_drop: Dropout rate applied after the output projection norm_layer: Normalization layer constructor for QK normalization if enabled rz$dim should be divisible by num_headsN|j jndd}n!||j}||dd } t| |} | jd d } | | } | | }|dd|||}||}||}||}|S) Nr rrrr dropout_pr)shaper(reshaperr%permuteunbindr*r+rFscaled_dot_product_attentiontrainingrpr& transposersoftmaxr-r.r) r/r2r BNCr(qkvattnr r rforwardBs* *       zAttention.forward)rFFFTrrNr )__name__ __module__ __qualname____doc__rbool__annotations__intfloatrrrModuler$torchTensorrK __classcell__r r r0rrsP     ,rcseZdZUdZejjeed<          dd e d e d ed ed e de de de e de e jdedeffdd Z  dde ejde ejfddZZS) AttentionRopez A Self Attention module with ROPE support. Includes options for: * QK normalization option * Attention output (scale) normalization * Fused or unfused QKV projection support rrTrrNFrrr qkv_fusednum_prefix_tokensrr attn_head_dimrrrc s4t| s | r| dusJd||_||} |dur|} | |j} | d|_||_t|_|rFtj|| d|d|_ d|_ |_ |_ nd|_ tj|| |d|_ tj|| |d|_ tj|| |d|_ | rj| | nt |_| ru| | nt |_t||_| r| | nt |_t| ||_t||_dS)aInitialize the Attention module. Args: dim: Input dimension of the token embeddings num_heads: Number of attention heads qkv_bias: Whether to add a bias term to the query, key, and value projections num_prefix_tokens: Number of reg/cls tokens at the beginning of the sequence that should not have position embeddings applied attn_drop: Dropout rate for attention weights proj_drop: Dropout rate for the output projection attn_head_dim: Dimension of each attention head (if None, computed as dim // num_heads) norm_layer: Normalization layer constructor to use for QK and scale normalization qk_norm: Enable normalization of query (Q) and key (K) vectors with norm_layer scale_norm: Enable normalization (scaling) of attention output with norm_layer Nrrr r!)r#r$rr&rZrrrr'r(q_projk_projv_projr)r*r+r,rr-r.r)r/rrrrYrZrrr[rrrr%attn_dimr0r rr$ks0    zAttentionRope.__init__roper c Cs(|j\}}}|jdur*||}|||d|jdddddd}|d\}} } n0|||||jddd}|||||jddd} | ||||jddd} | || | }} |dur|j } t j|ddddd| ddft|dddd| dddf|gdd| }t j| ddddd| ddft| dddd| dddf|gdd| } |jrtj|| | ||jr|jjnd d }n!||j}|| d d} t| |} | jdd} || } | | }|dd|||}||}||}||}|S) aForward pass for the attention module. Args: x: Input tensor of shape (batch_size, sequence_length, embedding_dim) rope: Rotary position embeddings tensor for position-aware attention attn_mask: Optional attention mask to apply during attention computation Returns: Tensor of shape (batch_size, sequence_length, embedding_dim) Nr r8r3rrr4r9rr5r7)r:r(r;rr<r=r\rBr]r^r*r+rZrUcatr type_asrr>r?r@rrAr&rrCr-r.r) r/r2r`r rDrErFr(rGrHrInptrJr r rrKs<   "   VV       zAttentionRope.forward) rTTrrrNNFF)NN)rLrMrNrOrUjitrrPrQrRrSrrrrTr$rVrKrWr r r0rrXasV     <rXr )typingrrrrUrtorch.nnrr>configrpos_embed_sincosr rVrrTrrXr r r rs    R