o پiu  @sdZddlZddlmZddlmZmZmZmZm Z m Z m Z m Z ddl Z ddlmZddlmmZddlmZmZmZmZddlmZmZmZmZmZmZmZm Z m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*ddl+m,Z,dd l-m.Z.dd l/m0Z0dd l1m2Z2m3Z3d gZ4Gd ddej5Z6Gdddej5Z7Gdddej5Z8Gdd d ej5Z9 ddee:e j;fdej5de:dee:e j;ffddZ<  ddee:e j;fdej5de:de=dee:e j;ff dd Z>dd"e:d#e=de9fd$d%Z?dd'e:dee:effd(d)Z@dd'e:dee:effd*d+ZAe2id,e@d-d.d/e@d-d0d1d2d3d4e@d-eed0d1d2d5d6e@d-eed7d1d2d5d8e@d-d9d1d2d3d:e@d-d9d1d2d3d;e@d-d9d1d2d3de@d-d0d1d=d?e@d-d9d1d=d@e@d-d9d1d=dAe@d-d9d1d=dBe@d-d9d1d2dCdDdEe@d-d9d1d2dCdDdFe@d-d9d1d2dCdDdGe@d-ddHdIe@d-ddHidJe@d-ddHdKe@d-ddHdLe@d-ddHdMe@d-dNdHdOe@d-dNdHdPe@d-dQdHdRe@d-dSdHdTe@d-d0d1dSdUdVe@d-dNdHdWe@d-dNdHdXe@ddYdZe@d-d[d\d]d]d^d_e@d-d[d\d]d]d^d`e@d-d[d\d=dae@d-d[d\d]d]d^dbeAd-dcdNdddeeAd-d0dNddeAd-d9dfddeAd-d9dddeAd-d9dddeAd-d9ddddgZBe3dd#e=de9fdhdiZCe3dd#e=de9fdjdkZDe3dd#e=de9fdldmZEe3dd#e=de9fdndoZFe3dd#e=de9fdpdqZGe3dd#e=de9fdrdsZHe3dd#e=de9fdtduZIe3dd#e=de9fdvdwZJe3dd#e=de9fdxdyZKe3dd#e=de9fdzd{ZLe3dd#e=de9fd|d}ZMe3dd#e=de9fd~dZNe3dd#e=de9fddZOe3dd#e=de9fddZPe3dd#e=de9fddZQe3dd#e=de9fddZRe3dd#e=de9fddZSe3dd#e=de9fddZTe3dd#e=de9fddZUe3dd#e=de9fddZVe3dd#e=de9fddZWe3dd#e=de9fddZXe3dd#e=de9fddZYe3dd#e=de9fddZZe3dd#e=de9fddZ[e3dd#e=de9fddZ\dS)a EVA EVA from https://github.com/baaivision/EVA , paper: https://arxiv.org/abs/2211.07636 @article{EVA, title={EVA: Exploring the Limits of Masked Visual Representation Learning at Scale}, author={Fang, Yuxin and Wang, Wen and Xie, Binhui and Sun, Quan and Wu, Ledell and Wang, Xinggang and Huang, Tiejun and Wang, Xinlong and Cao, Yue}, journal={arXiv preprint arXiv:2211.07636}, year={2022} } EVA-02: A Visual Representation for Neon Genesis - https://arxiv.org/abs/2303.11331 @article{EVA02, title={EVA-02: A Visual Representation for Neon Genesis}, author={Fang, Yuxin and Sun, Quan and Wang, Xinggang and Huang, Tiejun and Wang, Xinlong and Cao, Yue}, journal={arXiv preprint arXiv:2303.11331}, year={2023} } This file contains EVA & EVA02 model implementations evolved from BEiT, additional models in vision_transformer.py. Modifications by / Copyright 2023 Ross Wightman, original copyrights below N)partial)AnyCallableDictListOptionalSetTupleUnion)IMAGENET_DEFAULT_MEANIMAGENET_DEFAULT_STDOPENAI_CLIP_MEANOPENAI_CLIP_STD) PatchEmbedMlpGluMlpSwiGLU LayerNormDropPath PatchDropoutRotaryEmbeddingCatapply_rot_embed_catapply_keep_indices_nlc trunc_normal_resample_patch_embedresample_abs_pos_embedglobal_pool_nlc to_2tupleuse_fused_attn AttentionRopeAttentionPoolLatent)build_model_with_cfg)feature_take_indices) checkpoint)generate_default_cfgsregister_modelEvacseZdZUdZejjeed<           dd e d e d ed ed ede de de de e de e dedeffdd Z  dde ejde ejfddZZS) EvaAttentionzG EVA Attention with ROPE, no k-bias, and fused/unfused qkv options fused_attnTFr!Ndim num_headsqkv_bias qkv_fusedqkv_bias_separatenum_prefix_tokens attn_drop proj_drop attn_head_dim norm_layerqk_norm scale_normc st| s | r| dusJd||_||} | dur| } | |j}| d|_||_t|_||_|rrtj ||ddd|_ d|_ |_ |_ |rhtt||_|jdt|ddtt||_n1d|_|_|_n'tj |||d|_ tj ||dd|_ tj |||d|_ d|_ d|_|_|_| r| |jnt|_| r| |jnt|_t||_| r| |nt|_t |||_t||_dS) a, Args: dim: Input dimension of the token embeddings num_heads: Number of attention heads qkv_bias: Whether to add a bias term to the query, key, and value projections qkv_fused: Whether qkv projections are fused into one projection or separate qkv_bias_separate: Whether to apply bias to qkv as a separate addition or part of F.linear() call num_prefix_tokens: Number of reg/cls tokens at the beginning of the sequence that should not have position embeddings applied attn_drop: Dropout rate for attention weights proj_drop: Dropout rate for the output projection attn_head_dim: Dimension of each attention head (if None, computed as dim // num_heads) norm_layer: Normalization layer constructor to use for QK and scale normalization qk_norm: Enable normalization of query (Q) and key (K) vectors with norm_layer scale_norm: Enable normalization (scaling) of attention output with norm_layer Nz    zEvaAttention.__init__rope attn_maskc Cs|j\}}}|jdurR|jdur||}n"t|j|j|jf}|jr.||}||7}n tj ||jj |d}| ||d|j d ddddd}|d\} } } n0|| |||j ddd} || |||j ddd} || |||j ddd} || || } } |dur|j} tj| ddddd| ddft| dddd| dddf|gdd | } tj| ddddd| ddft| dddd| dddf|gdd | } |jrtj| | | ||jr|jjnd d }n:| |j} | | d d} |dur*|tj}| |ddddddft!d } | j"dd } || } | | }|dd |||}|#|}|$|}|%|}|S)aForward pass for the attention module. Args: x: Input tensor of shape (batch_size, sequence_length, embedding_dim) rope: Rotary position embeddings tensor for position-aware attention attn_mask: Optional attention mask to apply during attention computation Returns: Tensor of shape (batch_size, sequence_length, embedding_dim) N)weightr9r8rr!r,r+)rY dropout_pz-inf)&shaperArHrFcatr:rJr0FlinearrZreshaper-permuteunbindrB transposerCrDrMrNr1rtype_asr)scaled_dot_product_attentiontrainingr2pr>tobool masked_fillfloatsoftmaxrPrQr3)rRxrXrYBNCrAr.qkvnptattnrVrVrWforwardwsL      "   VV   &     zEvaAttention.forward) r*TTFr!r+r+NNFTNN)__name__ __module__ __qualname____doc__rFjitFinalrn__annotations__intrprrr=Tensorr{ __classcell__rVrVrTrWr(0s\      Er(c$seZdZddddddddddddejedfded ed ed ed ed edededede dededede ede de de ef"fdd Z dde e jde e jfddZZS) EvaBlockT@Fr!evar+Nr,r-r.r/ mlp_ratio swiglu_mlp scale_mlpscale_attn_innerr1 attn_typer3r2 drop_path init_values act_layerr5r4c s@t|||_| dkrtnt}|||||| | | |||d |_|dur/t|t |nd|_ | dkr:t | nt |_ |||_t||}|ro|r\t|||rU|nd| d|_n!t||d|re|ndtjd| d|_nt||||rw|nd| d |_|durt|t |nd|_| dkrt | |_dSt |_dS) a Initialize the EVA transformer block. Args: dim: Input dimension of the token embeddings num_heads: Number of attention heads qkv_bias: Whether to use bias terms in query, key, value projections qkv_fused: Whether to use a single projection for query, key, value mlp_ratio: Ratio of MLP hidden dimension to input dimension swiglu_mlp: Whether to use SwiGLU activation in the MLP scale_mlp: Whether to use normalization in the MLP scale_attn_inner: Whether to use normalization within the attention mechanism num_prefix_tokens: Number of tokens at the beginning of the sequence (class tokens, etc.) attn_type: Type of attention module to use ('eva' or 'rope') proj_drop: Dropout rate for projection layers attn_drop: Dropout rate for attention matrix drop_path: Stochastic depth rate init_values: Initial value for LayerScale, None = no LayerScale act_layer: Activation layer constructor norm_layer: Normalization layer constructor attn_head_dim: Dimension of each attention head (if None, computed as dim // num_heads) rX r-r.r/r1r2r3r4r5r7Nr+ in_featureshidden_featuresr5dropr\Frrr5r gate_lastrrrrr5r)r<r=norm1rr(rzr?rErFonesgamma_1rrL drop_path1norm2rrmlprSiLUrgamma_2 drop_path2)rRr,r-r.r/rrrrr1rr3r2rrrr5r4attn_clsrrTrVrWr=sX ) "         "$zEvaBlock.__init__rXrYc Cs|jdur$|||j||||d}|||||}|S|||j|j||||d}|||j|||}|SN)rXrY)rrrzrrrrrrRrrrXrYrVrVrWr{s  & zEvaBlock.forwardr|)r}r~rr?GELUrrrnrpstrrrr=rFrr{rrVrVrTrWrsh     ([rc$seZdZdZddddddddddddejejdfd ed ed ed ed e de dedededede de de de e de de de ef"fdd Z d de ejde ejfddZZS)!EvaBlockPostNormzF EVA block w/ post-norm and support for swiglu, MLP norm scale, ROPE. TrrFr!r+Nr,r-r.r/rrrrrr1r3r2rrrr5r4c st|dkr tnt}|||||| | | ||| d |_|||_| dkr)t| nt|_ t ||}|rY|rFt |||r?|nd| d|_ n!t ||d|rO|ndtjd| d|_ nt||||ra|nd| d |_ |||_| dkrwt| |_dSt|_dS) a Initialize the post-norm EVA transformer block. Args: dim: Input dimension of the token embeddings num_heads: Number of attention heads qkv_bias: Whether to use bias terms in query, key, value projections qkv_fused: Whether to use a single projection for query, key, value mlp_ratio: Ratio of MLP hidden dimension to input dimension swiglu_mlp: Whether to use SwiGLU activation in the MLP scale_mlp: Whether to use normalization in the MLP scale_attn_inner: Whether to use normalization within the attention mechanism num_prefix_tokens: Number of tokens at the beginning of the sequence (class tokens, etc.) attn_type: Type of attention module to use ('eva' or 'rope') proj_drop: Dropout rate for projection layers attn_drop: Dropout rate for attention matrix drop_path: Stochastic depth rate init_values: Initial value for LayerScale, None = no LayerScale (NOTE: ignored for post-norm block) act_layer: Activation layer constructor norm_layer: Normalization layer constructor attn_head_dim: Dimension of each attention head (if None, computed as dim // num_heads) rXrr+Nrr\Frr)r<r=rr(rzrrr?rLrrrrrrrrr)rRr,r-r.r/rrrrrr1r3r2rrrr5r4rrrTrVrWr=#sT )         $zEvaBlockPostNorm.__init__rXrYc Cs>||||j|||d}|||||}|Sr)rrrzrrrrrVrVrWr{|s zEvaBlockPostNorm.forwardr|)r}r~rrr?rrrrnrprrrr=rFrr{rrVrVrTrWr!sj     (YrcQseZdZdZddddddddd d d d d d d d d d d d d edd dd d d dd d ddddd d ddf'deeeeeffdeeeeeffdedededededede de de de de de d ed!e d"e d#e d$e d%e d&e d'e d(e e d)e d*ed+e d,e d-e d.ed/e d0e d1e e d2e e d3e ed4e e d5e d6e d7e eeeefefd8e fNfd9d: Z dhdejd;dfd?d@Zejjd;eefdAdBZejjdidCe d;dfdDdEZejjdjdFe d;eeeffdGdHZejjd;ejfdIdJZdkdede ed;dfdKdLZd;eeje ejffdMdNZ  O dldPejdQe eeeefdRe dSe dTe dUedVe d;eeejeejeejfffdWdXZ Y dmdQeeeefdZe d[e fd\d]Z!dkdPejd^e ed;ejfd_d`Z"dPejd;ejfdadbZ#djdPejdce d;ejfdddeZ$dPejd;ejfdfdgZ%Z&S)nr'a! Eva Vision Transformer w/ Abs & Rotary Pos Embed This class implements the EVA and EVA02 models that were based on the BEiT ViT variant * EVA - abs pos embed, global avg pool * EVA02 - abs + rope pos embed, global avg pool, SwiGLU, scale Norm in MLP (ala normformer) r8avg TrFrr+NrijgMbP?img_size patch_sizein_chans num_classes global_pool embed_dimdepthr-r.r/rrrrr drop_rate pos_drop_ratepatch_drop_rateproj_drop_rateattn_drop_ratedrop_path_rater5r class_tokennum_reg_tokensuse_abs_pos_embuse_rot_pos_embrope_grid_offsetrope_grid_indexing use_post_normuse_pre_transformer_normuse_post_transformer_norm use_fc_normattn_pool_num_headsattn_pool_mlp_ratiodynamic_img_sizedynamic_img_padref_feat_shapehead_init_scalec(-st|dvs J|_|____|rdnd|_|$_d_ |}(|!dur2|!})n|dk})| dur=| }*n|) }*i}+|$rM|+ t dddt d||||%| d |+_ j j},tj d rlj n| |rzttddnd_|rttd|nd_|ojdu_|rttd|,jnd_tj|d _|dkrt|jd d _nd_|r|&durt|&nd}&td|$rdnj j|&||d_ nd_ |(rnt!_"ddt#d||D|rt$nt%t& fddt'|D_( fddt'|D_)|*r0nt!_*|dkrQ|"p>}"|#pC}#t+j|"|#tj,d_-nd_-|)r[nt!_.t|_/|dkrqt0|nt!_12j3jdurt4jddjdurt4jddjdurt4jdd5t6j1tj0rt4j1j7ddj1j7j89|'j1j:j89|'dSdS)a Initialize the EVA Vision Transformer model. Args: img_size: Input image size (single int for square, or tuple for rectangular) patch_size: Patch size to divide image into tokens (single int for square, or tuple) in_chans: Number of input image channels num_classes: Number of classes (output dim) for classification head (final projection), 0 for pass-through global_pool: Type of global pooling for final sequence ('avg', 'token', 'map', etc.) embed_dim: Embedding dimension for tokens depth: Number of transformer blocks num_heads: Number of attention heads qkv_bias: Enable bias for query, key, value projections qkv_fused: Use a single projection for query, key, value mlp_ratio: Ratio of mlp hidden dim to embedding dim swiglu_mlp: Use SwiGLU activation in MLP scale_mlp: Apply scaling normalization in MLP (normformer style) scale_attn_inner: Apply scaling normalization inside attention attn_type: Type of attention module to use drop_rate: Dropout rate after final projection and pooling pos_drop_rate: Dropout rate for positional embeddings patch_drop_rate: Rate of dropping patches during training proj_drop_rate: Dropout rate for projections attn_drop_rate: Dropout rate for attention drop_path_rate: Stochastic depth rate norm_layer: Normalization layer constructor init_values: Initial layer-scale values class_token: Use class token num_reg_tokens: Number of additional learnable 'register' tokens to add to the sequence use_abs_pos_emb: Use absolute (learned) positional embeddings use_rot_pos_emb: Use rotary position embeddings rope_grid_offset: Offset for rotary position embedding grid rope_grid_indexing: Indexing mode for rotary position embeddings ('ij' or 'xy') use_post_norm: Use post-norm transformer block type use_pre_transformer_norm: Use normalization layer before transformer blocks use_post_transformer_norm: Use normalization layer after transformer blocks use_fc_norm: Use normalization layer after pooling, before final classifier attn_pool_num_heads: Number of heads in attention pooling attn_pool_mlp_ratio: MLP ratio in attention pooling dynamic_img_size: Support dynamic image sizes in forward pass dynamic_img_pad: Apply dynamic padding for irregular image sizes ref_feat_shape: Reference feature shape for rotary position embedding scale head_init_scale: Initialization scale for classification head weights )ravgmaxmaxtokenmapr!rFNrNHWC)strict_img_size output_fmt)rrrrrr9 feat_ratio)rlT)r1return_indices) in_pixels feat_shaper grid_offset grid_indexingcSsg|]}|qSrV)item).0rrrVrVrW "sz Eva.__init__..cs8g|]}  j |dqS))r,r-r.r/rrrrrr1r3r2rr5rr1ri)rrblock_fndprrrrr5r-rr.r/rrrRrrVrWr$s(cs g|] }td|dqS)blocks.)modulenum_chs reductiondictr)rrrVrWr7sr)r-rr5r{Gz?stdrV);r<r=rr num_featureshead_hidden_sizerr1rgrad_checkpointingupdaterr patch_embed num_patcheshasattrrr?rErFrG cls_token reg_token cls_embed pos_embedrOpos_dropr patch_droprr grid_sizerXrLnorm_prelinspacerr ModuleListrangeblocks feature_inforPr r attn_poolfc_norm head_dropr@headapply _init_weightsrfix_init_weight isinstancerZdatamul_r9)-rRrrrrrrrr-r.r/rrrrrrrrrrrr5rrrrrrrrrrrrrrrrractivate_pre_normactivate_fc_normactivate_post_norm embed_argsrrT)rrrrrrrr5r-rr.r/rrrrRrrWr=s U      ,           z Eva.__init__returncCsLdd}t|jD]\}}||jjjj|d||jjjj|dq dS)z=Fix initialization weights by rescaling based on layer depth.cSs|td|dS)Ng@)div_mathsqrt)paramlayer_idrVrVrWrescale\sz$Eva.fix_init_weight..rescaler!N) enumeraterrzrQrZr rfc2)rRrrlayerrVrVrWrZs zEva.fix_init_weightmcCs>t|tjrt|jdd|jdurtj|jdSdSdS)zbInitialize weights for Linear layers. Args: m: Module to initialize. rrN)rr?r@rrZr9initzeros_)rRrrVrVrWrcs  zEva._init_weightscCs ddh}|S)z(Parameters to exclude from weight decay.rrrV)rRnwdrVrVrWno_weight_decaynszEva.no_weight_decayenablecCs ||_dS)z)Enable or disable gradient checkpointing.N)r)rRrrVrVrWset_grad_checkpointingts zEva.set_grad_checkpointingcoarsecCstdddgd}|S)z(Create layer groupings for optimization.z ^cls_token|pos_embed|patch_embed)z^blocks\.(\d+)N)z^norm)i)stemrr)rRr matcherrVrVrW group_matcherys zEva.group_matchercCs|jSN)rrRrVrVrWget_classifierszEva.get_classifiercCs>||_|dur ||_|dkrt|j||_dSt|_dS)zReset the classifier head. Args: num_classes: Number of output classes. global_pool: Global pooling type. Nr)rrr?r@rrLr)rRrrrVrVrWreset_classifiers*zEva.reset_classifierc Csz|jr:|j\}}}}|jdur |jj}t|j||f||jd}nd}||d|}|jdur7|jj ||fdnd}n|j}|jdurG|j nd}|j dur`t j |j |jddd|fdd}|durh||}|jdurg} |j dur| |j |jddd| |j |jdddt j | |gdd}||}|jdur||\}} |dur| durt||| }||fS)N)new_sizeold_sizer1r[)rarr!r^)rrarrrrr1viewrX get_embedrrFrbexpandrappendrrr) rRrrrsHWruprev_grid_sizer rot_pos_embedto_cat keep_indicesrVrVrW _pos_embeds> " $     zEva._pos_embedNCHWrrindicesreturn_prefix_tokensrP stop_earlyrintermediates_onlycst|dvsJd|dk}g} ttj|\} } |j\} } }|}|\}}|}tj s7|s;j}n jd| d}t |D]*\}}j r\tj s\t |||d}n|||d}|| vrr| |ro|n|qHjrfdd| D}fd d| D} |rj| |f\fd d| D} tj s|rtt| |} |r| S|}|| fS) a) Forward features that returns intermediates. Args: x: Input image tensor indices: Take last n blocks if an int, if is a sequence, select by matching indices return_prefix_tokens: Return both prefix and spatial intermediate tokens norm: Apply norm layer to all intermediates stop_early: Stop iterating over blocks when last desired intermediate hit output_fmt: Shape of intermediate feature outputs intermediates_only: Only return intermediate features )r5NLCz>Output format for EVA-ViT features must be one of NCHW or NLC.r5Nr!rXcs"g|] }|dddjfqS)Nrrryr%rVrWr"z-Eva.forward_intermediates..cs"g|] }|ddjdfqSr$rr<r%rVrWrr>cs,g|]}|dddddqS)r[rr8r!r\)rerf contiguousr<)rsr.r/rVrWrs,)r#lenrrarr4rrFr is_scriptingrrr$r-rPr1dynamic_feat_sizelistzip)rRrrr6r7rPr8rr9re intermediates take_indices max_index_heightwidthr1rrblk prefix_tokensrV)rsr.r/rRrWforward_intermediatess<    zEva.forward_intermediatesr! prune_norm prune_headcCsZtt|j|\}}|jd|d|_|rt|_|r+d|_t|_|dd|S)z@ Prune layers not required for specified intermediates. Nr!rr) r#r@rr?rLrPrrr')rRr6rNrOrFrGrVrVrWprune_intermediate_layerss   zEva.prune_intermediate_layers pool_typecCs>|jdur ||}|S|dur|jn|}t|||jd}|S)N)rQr1)rrrr1)rRrrrQrVrVrWpool s  zEva.poolcCsh||}||\}}||}|jD]}|jr&tjs&t|||d}q|||d}q| |}|S)zForward pass through feature extraction layers. Args: x: Input tensor. Returns: Feature tensor. r;) rr4rrrrFrrAr$rP)rRrrr1rKrVrVrWforward_featuress    zEva.forward_features pre_logitscCs0||}||}||}|r|S||S)zForward pass through classifier head. Args: x: Feature tensor. pre_logits: Return pre-logits if True. Returns: Output tensor. )rRrrr)rRrrrTrVrVrW forward_head's  zEva.forward_headcCs||}||}|S)zoForward pass. Args: x: Input tensor. Returns: Output tensor. )rSrU)rRrrrVrVrWr{6s z Eva.forward)rN)TFr$)NFFFr5F)r!FT)'r}r~rrrr rr rrnrprrr=rr?ModulerrFrignorerrrrrr#r&r'rr4rrMrPrRrSrUr{rrVrVrTrWr'sT       !"#$%&'( Q   , @ visual. state_dictmodelprefixrc Cs|d|}dd|D}i}gd}t|}|D]\}}|r/||s)q||d}|D] }||d|d}q1|dr|d d}|d d }|d d }|dr|jdd} |drw|d| |d<|| d|d<n|dr|d| |d<|| d|d<qn/|dkrd}|dd}t |jd|d<n|dkrd}| d d}n |dkr| d}|||<q|S)zConvert Perception Encoder weights. Args: state_dict: State dictionary to convert. model: Target model instance. prefix: Prefix to strip from keys. Returns: Converted state dictionary. r[cSsi|] \}}|dd|qS)zmodule.r)replace)rrwrxrVrVrW Tsz_convert_pe..) )conv1patch_embed.proj)positional_embeddingr)ztransformer.resblocks.r)ln_prer)ln_postrP)ln_rP)z ls_1.gammar)z ls_2.gammar)in_proj_zqkv.)out_projrQ)zmlp.c_fcmlp.fc1)z mlp.c_projmlp.fc2Nrr!rzattn_pool.attnzattn_pool.layernormzattn_pool.normzattn_pool.probezattn_pool.latentz attn_pool.qkvr8rZzattn_pool.q.weightzattn_pool.kv.weightr9zattn_pool.q.biaszattn_pool.kv.biasrQz head.weightz head.biasclass_embeddingrr) getitemsr@ startswithr]raendswithrhrFrG unsqueeze) rZr[r\out_dictswaps len_prefixrwrxspr,rVrVrW _convert_peDsJ              rsbicubicT interpolation antialiasc Csi}|d|}|d|}|d|}|d|}d|vr#t||Sd|vr.t||ddSd |vr5d }n d |vrtddddd|ddd}td d |it|fi|}|S) zB EVA-g CLIP model (only difference from non-CLIP is the pooling) rrrrrrr)rrrr-rreva_giant_patch14_clip_224rN)rrrrrrVrVrWrs   rcKsLtdddddddddddd|d d d }tdd |it|fi|}|S)zU A EVA-CLIP specific variant that adds additional attn scale layernorm to eva02_base rrrrFrTrrr) rrrrr-r/rrrrrrreva02_base_patch16_clip_224rN)rrrrVrVrWr" rcKLtdddddddddddd |d d d }tdd|it|fi|}|S)V A EVA-CLIP specific variant that adds additional attn scale layernorm to eva02_large rrrrrrFTrrr rrrrr-rr/rrrrrreva02_large_patch14_clip_224rN)rrrrVrVrWrrrcKr)rrrrrrrFTrrrreva02_large_patch14_clip_336rN)rrrrVrVrWrrrc KsBtddddddd|dd d }tdd |it|fi|}|S)zD A EVA-CLIP specific variant that uses residual post-norm in blocks rri@rgI$I$!@Trr)rrrrr-rrreva02_enormous_patch14_clip_224rN)rrrrVrVrWrs  rcKDtdddddddddd ddd d }tdd |it|fi|}|S)Nrrrrr*Th㈵>Fr!r rrrrr-r/r.rrrrrr$vit_medium_patch16_rope_reg1_gap_256r)rrrrVrVrWr"rcKsDtdddddddddd ddd d }tdd |it|fi|}|S)Nrrrr*TFrr!rr%vit_mediumd_patch16_rope_reg1_gap_256r)rrrrVrVrWr*rrcKr)Nrrir TrFr]rr%vit_betwixt_patch16_rope_reg4_gap_256r)rrrrVrVrWr?rrcKsDtddddddddddddd d }td d |it|fi|}|S)NrrrrTrFr!rr"vit_base_patch16_rope_reg1_gap_256r)rrrrVrVrWrTrrcKsLtddddddddddd d d dttd d d}tdd|it|fi|S)NrrrrrrXT)rrrxyr*repsrrrr-rrrrrrrrrrr5vit_pe_core_base_patch16_224r)rrrrrrrrrVrVrWri$ rcKsLtdddddddddd d d d dttd dd}tdd|it|fi|S)NrrrrrrrXT)rrrrr*rrrvit_pe_core_large_patch14_336r)rrrrVrVrWrrrcKsLtddddddddd d d d d d ttddd}tdd|it|fi|S)Nr2rUUUUUU@rrXFT rrr*rrr)rrrr-rrrrrrrrrrr5 vit_pe_core_gigantic_patch14_448r)rrrrVrVrWrrrcKstd!iddddddddd d d d d ddddddddddddddddddttdd}td"d |it|fi|S)#Nrrrrrr-rrrrrXrTrrrrrrrrrFrr皙?r5rrvit_pe_lang_large_patch14_448rrV)rrrrVrVrWrsD      rcKLtddddddddd d dddd ttd d d}tdd|it|fi|S)Nrr/rrrXFTrrrrrrrrr-rrrrrrrrrrr5 vit_pe_lang_gigantic_patch14_448r)rrrrVrVrWrrrcKr)NrrrrrrXFTrrrrrr#vit_pe_spatial_gigantic_patch14_448r)rrrrVrVrWrrr)rY)rtTrV)r)]rr functoolsrtypingrrrrrrr r rFtorch.nnr?torch.nn.functional functionalrc timm.datar r r r timm.layersrrrrrrrrrrrrrrrrrr _builderr" _featuresr# _manipulater$ _registryr%r&__all__rWr(rrr'rrrsrnrrrr default_cfgsrrrrrrrrrrrrrrrrrrrrrrrrrrrVrVrVrWs ( P    gaH   I   _ #*/49>EJOV[`ejqx$).37>EL k