o پi̿@sdZddlZddlmZmZmZmZmZmZm Z m Z m Z ddl Z ddl mZddlmmZddlmZmZddlmZmZmZmZmZmZmZmZmZmZddl m!Z!ddl"m#Z#dd l$m%Z%dd l&m'Z'dd l(m)Z)m*Z*m+Z+d gZ,e e-e e-e-ffZ.d e j/de e-e-fde j/fddZ0e%de j/de e-e-fde e-e-fde j/fddZ1Gdddej2Z3Gdddej2Z4Gdddej2Z5Gdddej2Z6Gdd d ej2Z7dee8e j/fd ej2dee8e j/ffd!d"Z9dad$e8d%e:de7fd&d'Z;dbd)d*Zd?ed?dA Z=e*dad%e:de7fdBdCZ>e*dad%e:de7fdDdEZ?e*dad%e:de7fdFdGZ@e*dad%e:de7fdHdIZAe*dad%e:de7fdJdKZBe*dad%e:de7fdLdMZCe*dad%e:de7fdNdOZDe*dad%e:de7fdPdQZEe*dad%e:de7fdRdSZFe*dad%e:de7fdTdUZGe*dad%e:de7fdVdWZHe*dad%e:de7fdXdYZIe+eJdZd[d\d]d^d_d`dS)caK Swin Transformer V2 A PyTorch impl of : `Swin Transformer V2: Scaling Up Capacity and Resolution` - https://arxiv.org/abs/2111.09883 Code/weights from https://github.com/microsoft/Swin-Transformer, original copyright/license info below Modifications and additions for timm hacked together by / Copyright 2022, Ross Wightman N) AnyCallableDictListOptionalSetTupleTypeUnionIMAGENET_DEFAULT_MEANIMAGENET_DEFAULT_STD) PatchEmbedMlpDropPath to_2tuple trunc_normal_ClassifierHeadresample_patch_embedndgrid get_act_layer LayerType)build_model_with_cfg)feature_take_indices)register_notrace_function) checkpoint)generate_default_cfgsregister_modelregister_model_deprecationsSwinTransformerV2x window_sizereturncCsj|j\}}}}||||d|d||d|d|}|ddddddd|d|d|}|S)zPartition into non-overlapping windows. Args: x: Input tensor of shape (B, H, W, C). window_size: Window size (height, width). Returns: Windows tensor of shape (num_windows*B, window_size[0], window_size[1], C). rrshapeviewpermute contiguous)r!r"BHWCwindowsr3S/home/ubuntu/.local/lib/python3.10/site-packages/timm/models/swin_transformer_v2.pywindow_partition$s ,,r5r2img_sizecCsf|\}}|jd}|d||d||d|d|d|}|ddddddd|||}|S)a1Merge windows back to feature map. Args: windows: Windows tensor of shape (num_windows * B, window_size[0], window_size[1], C). window_size: Window size (height, width). img_size: Image size (height, width). Returns: Feature map tensor of shape (B, H, W, C). r(rrr$r%r&r'r))r2r"r6r/r0r1r!r3r3r4window_reverse4s  ,$r7cseZdZdZ     ddedeeefded ed ed ed ed eeefddffdd ZdddZ deeefddfddZ dde j de e j de j fddZZS)WindowAttentionzWindow based multi-head self attention (W-MSA) module with relative position bias. Supports both shifted and non-shifted window attention with continuous relative position bias and cosine attention. TFrrdimr" num_headsqkv_biasqkv_bias_separate attn_drop proj_droppretrained_window_sizer#Nc s&t||_||_t||_||_||_t t dt |ddf|_ ttjddddtjddtjd|dd|_tj||d dd|_|rjt t ||_|jd t |dd t t ||_n d |_d |_d |_t||_t|||_t||_tjd d|_|d S)a4Initialize window attention module. Args: dim: Number of input channels. window_size: The height and width of the window. num_heads: Number of attention heads. qkv_bias: If True, add a learnable bias to query, key, value. qkv_bias_separate: If True, use separate bias for q, k, v projections. attn_drop: Dropout ratio of attention weight. proj_drop: Dropout ratio of output. pretrained_window_size: The height and width of the window in pre-training. rr%iTbias)inplaceFr$k_bias persistentNr(r;)super__init__r;r"rrAr<r>nn Parametertorchlogones logit_scale SequentialLinearReLUcpb_mlpqkvzerosq_biasregister_bufferv_biasrFDropoutr?projr@Softmaxsoftmax"_make_pair_wise_relative_positions) selfr;r"r<r=r>r?r@rA __class__r3r4rKNs2  "    zWindowAttention.__init__c Cst|jdd |jdtj}t|jdd |jdtj}tt||}|ddd d}|j ddkrp|dddddddf|j dd<|dddddddf|j dd<n.|dddddddf|jdd<|dddddddf|jdd<|d9}t |t t |dt d}|jd|dd t|jd}t|jd}tt||}t|d}|dddddf|dddddf}|ddd}|dddddf|jdd7<|dddddf|jdd7<|dddddfd|jdd9<|d } |jd | dd dS) z?Create pair-wise relative position index and coordinates table.rrr%N?relative_coords_tableFrGr(relative_position_index)rNaranger"tofloat32stackrr,r- unsqueezerAsignlog2absmathrYflattensum) r`relative_coords_hrelative_coords_wrecoords_hcoords_wcoordscoords_flattenrelative_coordsrfr3r3r4r_s6&&.0..   ,((, z2WindowAttention._make_pair_wise_relative_positionscCs(t|}||jkr||_|dSdS)zUpdate window size and regenerate relative position tables. Args: window_size: New window size (height, width). N)rr"r_)r`r"r3r3r4set_window_sizes   zWindowAttention.set_window_sizer!maskcCs|j\}}}|jdur||}n"t|j|j|jf}|jr)||}||7}n tj ||jj |d}| ||d|j d ddddd}|d\}} } tj|dd tj| dd d d} tj|jtd d } | | } ||jd|j } | |jd|jd|jd|jd|jdd}| ddd}d t|}| |d} |dur|jd}| d||j |||dd} | d|j ||} || } n|| } || } | | dd |||}| |}|!|}|S)a#Forward pass of window attention. Args: x: Input features with shape of (num_windows*B, N, C). mask: Attention mask with shape of (num_windows, Wh*Ww, Wh*Ww) or None. Returns: Output features with shape of (num_windows*B, N, C). N)weightrDr$r(r%rrr&rIgY@)max)"r*rXrVrNcatrFrZr>Flinearr{reshaper<r,unbind normalize transposeclamprQrorOexprUrer+rfr"r-sigmoidrkr^r?r\r@)r`r!rzB_Nr1rVr=qkvattnrQrelative_position_bias_tablerelative_position_biasnum_winr3r3r4forwards>    "$& $     zWindowAttention.forward)TFr9r9r:r#NN)__name__ __module__ __qualname____doc__intrboolfloatrKr_ryrNTensorrr __classcell__r3r3rar4r8Gs:     7* r8cs4eZdZdZddddddddddejdf d ed ed ed ed ededede dede de de de de ej deffdd Z d*deejdeejfddZ d*ded eedeeeefeeefffd!d"Z d*d#eeefd eeefdeeddfd$d%Zdejdejfd&d'Zdejdejfd(d)ZZS)+SwinTransformerV2BlockzSwin Transformer V2 Block. A standard transformer block with window attention and shifted window attention for modeling long-range dependencies efficiently. rF@Tr9gelur;input_resolutionr<r" shift_sizealways_partition dynamic_mask mlp_ratior=r@r? drop_path act_layer norm_layerrAc st||_t||_||_t||_||_||_| ||\|_ |_ |j d|j d|_ ||_ t| } t|t|j || | | t|d|_|||_| dkrUt| nt|_t|t||| | d|_|||_| dkrtt| nt|_|jd|jrdn|dd dS) a Args: dim: Number of input channels. input_resolution: Input resolution. num_heads: Number of attention heads. window_size: Window size. shift_size: Shift size for SW-MSA. always_partition: Always partition into full windows and shift mlp_ratio: Ratio of mlp hidden dim to embedding dim. qkv_bias: If True, add a learnable bias to query, key, value. proj_drop: Dropout rate. attn_drop: Attention dropout rate. drop_path: Stochastic depth rate. act_layer: Activation layer. norm_layer: Normalization layer. pretrained_window_size: Window size in pretraining. rr)r"r<r=r?r@rAr9) in_featureshidden_featuresrdrop attn_maskNFrG)rJrKr;rrr<target_shift_sizerr_calc_window_shiftr"r window_arearrr8rnorm1rrLIdentity drop_path1rrmlpnorm2 drop_path2rY get_attn_mask)r`r;rr<r"rrrrr=r@r?rrrrArar3r4rKsF #      zSwinTransformerV2Block.__init__Nr!r#c Csft|jr|durtdg|jdR}ntjd|jd|jddf|j|jd}d}d|jd f|jd |jd f|jd dffD]<}d|jd f|jd |jd f|jd dffD]}||dd|d|d|d|dddf<|d7}qdqGt ||j}| d|j }| d| d}| |dktd |dktd}|Sd}|S) zGenerate attention mask for shifted window attention. Args: x: Input tensor for dynamic shape calculation. Returns: Attention mask or None if no shift. Nrr%)dtypedevicerr(gYr9)anyrrNrWrr*rrr"r5r+rrk masked_fillr)r`r!img_maskcnthw mask_windowsrr3r3r4r4s. (0  $z$SwinTransformerV2Block.get_attn_masktarget_window_sizercCst|}|dur|j}t|r|dd|ddf}nt|}|jr'||fSt|}t|}ddt|j|D}ddt|j||D}t|t|fS)aCalculate window size and shift size based on input resolution. Args: target_window_size: Target window size. target_shift_size: Target shift size. Returns: Tuple of (adjusted_window_size, adjusted_shift_size). Nrr%rcSs g|] \}}||kr |n|qSr3r3).0rrr3r3r4 us z=SwinTransformerV2Block._calc_window_shift..cSs"g|] \}}}||kr dn|qSrr3)rrrsr3r3r4rvs")rrrrziprtuple)r`rrr"rr3r3r4rXsz)SwinTransformerV2Block._calc_window_shift feat_sizecCsp||_|dur ||_|t|\|_|_|jd|jd|_|j|j|j d|j r/dn| dddS)zSet input size and update window configuration. Args: feat_size: New feature map size. window_size: New window size. always_partition: Override always_partition setting. NrrrFrG) rrrrr"rrrryrYrr)r`rr"rr3r3r4set_input_sizeys z%SwinTransformerV2Block.set_input_sizec Csp|j\}}}}t|j}|r!tj||jd |jd fdd}n|}|jd||jd|jd}|jd||jd|jd} tjj|ddd| d|f}|j\} } } } t ||j} | d|j |} t |ddrt| |}n|j}|j| |d}| d|jd|jd|}t||j| | f}|d d d |d |d d f}|rtj||jdd}|S|}|S) zApply windowed attention with optional shift. Args: x: Input tensor of shape (B, H, W, C). Returns: Output tensor of shape (B, H, W, C). rr)rr%)shiftsdimsr(rF)rzN)r*rrrNrollr"rL functionalpadr5r+rgetattrrrrr7r-)r`r!r.r/r0r1 has_shift shifted_xpad_hpad_w_HpWp x_windowsr attn_windowsr3r3r4_attns. &""   $zSwinTransformerV2Block._attncCsd|j\}}}}|||||}||d|}|||||}|||||}|S)Nr()r*rrrrrrr)r`r!r.r/r0r1r3r3r4rs zSwinTransformerV2Block.forwardr)rrrrrL LayerNormr_int_or_tuple_2_trrrr ModulerKrrNrrrrrrrrr3r3rar4rs       J' %   .rcsVeZdZdZdejfdedeedeej ffdd Z de j d e j fd d Z ZS) PatchMergingzPatch Merging Layer. Merges 2x2 neighboring patches and projects to higher dimension, effectively downsampling the feature maps. Nr;out_dimrcsFt||_|p d||_tjd||jdd|_||j|_dS)z Args: dim (int): Number of input channels. out_dim (int): Number of output channels (or 2 * dim if None) norm_layer (nn.Module, optional): Normalization layer. Default: nn.LayerNorm r%r&FrCN)rJrKr;rrLrS reductionnorm)r`r;rrrar3r4rKs zPatchMerging.__init__r!r#cCs|j\}}}}ddd|dd|df}tj||}|j\}}}}|||dd|dd|ddddddd}||}||}|S)Nrr%rr$r&r') r*rLrrrr,rprr)r`r!r.r/r0r1 pad_valuesrr3r3r4rs2  zPatchMerging.forward)rrrrrLrrrr rrKrNrrrr3r3rar4rs rc'seZdZdZdddddddddejddf ded ed ed ed ed edededede dede de de de e e fde ejdededdf&fdd Z d&deeefd edeeddfdd Zd!ejdejfd"d#Zd'd$d%ZZS)(SwinTransformerV2StagezA Swin Transformer V2 Stage. A single stage consisting of multiple Swin Transformer blocks with optional downsampling at the beginning. FrTr9rrr;rrdepthr<r"rr downsamplerr=r@r?rrrrA output_nchwr#Ncst| _| _| rtdd|Dn| _| _| _d _t tddD | r:t |d _ n |ks@Jt _ t  fddt|D _dS) a  Args: dim: Number of input channels. out_dim: Number of output channels. input_resolution: Input resolution. depth: Number of blocks. num_heads: Number of attention heads. window_size: Local window size. always_partition: Always partition into full windows and shift dynamic_mask: Create attention mask in forward based on current input size downsample: Use downsample layer at start of the block. mlp_ratio: Ratio of mlp hidden dim to embedding dim. qkv_bias: If True, add a learnable bias to query, key, value. proj_drop: Projection dropout rate attn_drop: Attention dropout rate. drop_path: Stochastic depth rate. act_layer: Activation layer type. norm_layer: Normalization layer. pretrained_window_size: Local window size in pretraining. output_nchw: Output tensors on NCHW format instead of NHWC. cs|]}|dVqdSr%Nr3rir3r3r4 $z2SwinTransformerV2Stage.__init__..FcSsg|]}|dqSr%r3)rrr3r3r4r)z3SwinTransformerV2Stage.__init__..)r;rrcsVg|]'}t j|ddkrdn  ttr"|n dqS)r%r)r;rr<r"rrrrr=r@r?rrrrA)routput_resolution isinstancelistrrrr?rrrrr<rrAr@r=r`rr"r3r4r3s(N)rJrKr;rrrrrgrad_checkpointingrrrrLr ModuleListrangeblocks)r`r;rrrr<r"rrrrr=r@r?rrrrArrarr4rKs *  *zSwinTransformerV2Stage.__init__rcCsb||_t|jtjr||_nt|jtsJtdd|D|_|jD] }|j |j||dq#dS)zUpdate resolution, window size and relative positions. Args: feat_size: New input (feature) resolution. window_size: New window size. always_partition: Always partition / shift the window. csrrr3rr3r3r4rYrz8SwinTransformerV2Stage.set_input_size..rr"rN) rrrrLrrrrrr)r`rr"rblockr3r3r4rGs  z%SwinTransformerV2Stage.set_input_sizer!cCs>||}|jD]}|jrtjst||}q||}q|S)zForward pass through the stage. Args: x: Input tensor of shape (B, H, W, C). Returns: Output tensor of shape (B, H', W', C'). )rrrrNjit is_scriptingr)r`r!blkr3r3r4ras   zSwinTransformerV2Stage.forwardcCsX|jD]&}tj|jjdtj|jjdtj|jjdtj|jjdqdS)z/Initialize residual post-normalization weights.rN)rrLinit constant_rrDr{r)r`rr3r3r4_init_respostnormss z(SwinTransformerV2Stage._init_respostnormrr)rrrrrLrrrrrr strrr rrKrrrrNrrrrr3r3rar4rs      T  rc+seZdZdZdddddddd d d d d d dddddejdfdedededededede edfde edfdede de de de d e d!e d"e d#e d$e ee fd%e d&e edff(fd'd( Zd)ejd*d+fd,d-Z + + + . +dSdee eefdee eefdee eefd/eedee f d0d1Zejjd*eefd2d3ZejjdTd4e d*eeeffd5d6ZejjdUd7e d*d+fd8d9Zejjd*ejfd:d;ZdVdedeed*d+fd dWd?ejd@ee eeefdAe dBe dCedDe d*e eeje ejeejfffdEdFZ G dXd@e eeefdHe dIe fdJdKZ!d?ejd*ejfdLdMZ"dTd?ejdNe d*ejfdOdPZ#d?ejd*ejfdQdRZ$Z%S)Yr a Swin Transformer V2. A hierarchical vision transformer using shifted windows for efficient self-attention computation with continuous position bias. A PyTorch impl of : `Swin Transformer V2: Scaling Up Capacity and Resolution` - https://arxiv.org/abs/2111.09883 r&r$avg`r%r%r%r$r rFTrr9g?r)rrrrr6 patch_sizein_chans num_classes global_pool embed_dimdepths.r<r"rstrict_img_sizerr= drop_rateproj_drop_rateattn_drop_ratedrop_path_raterrpretrained_window_sizesc st||_|dvsJ||_d|_t||_|_td|jd|_ |_ g|_ t t tfsAfddt|jDt|||d|| dd|_|jj}d dtd|t||D}g}d}d}t|jD]r}|}|td id |d |d |d||d|fd ||d|dkd||d| d| d| d| d| d|d|d||d|d|d||g7}|}|dkr|d9}|j t|d|d|dg7_ qqtj||_||j |_t|j ||||jd|_||j |jD]}|!qdS)!a] Args: img_size: Input image size. patch_size: Patch size. in_chans: Number of input image channels. num_classes: Number of classes for classification head. embed_dim: Patch embedding dimension. depths: Depth of each Swin Transformer stage (layer). num_heads: Number of attention heads in different layers. window_size: Window size. mlp_ratio: Ratio of mlp hidden dim to embedding dim. qkv_bias: If True, add a learnable bias to query, key, value. drop_rate: Head dropout rate. proj_drop_rate: Projection dropout rate. attn_drop_rate: Attention dropout rate. drop_path_rate: Stochastic depth rate. norm_layer: Normalization layer. act_layer: Activation layer type. patch_norm: If True, add normalization after patch embedding. pretrained_window_sizes: Pretrained window sizes of each layer. output_fmt: Output tensor format if not None, otherwise output 'NHWC' by default. )rNHWCr%rcsg|] }td|qSr)rrr r3r4rsz.SwinTransformerV2.__init__..r)r6r r r rr output_fmtcSsg|]}|qSr3)tolist)rr!r3r3r4rrr;rrrrr<r"rrrr=r@r?rrrrAr&layers.)num_chsrmodule) pool_typer input_fmtNr3)"rJrKr r rlen num_layersr r num_featureshead_hidden_size feature_inforrrrr patch_embed grid_sizerNlinspacerqsplitrdictrLrRlayersrrheadapply _init_weightsr)r`r6r r r r r rr<r"rrrr=rrrrrrrkwargsr%dprr)in_dimscalerrblyrarr4rKs .   "      &     zSwinTransformerV2.__init__mr#NcCsPt|tjr"t|jddt|tjr$|jdur&tj|jddSdSdSdS)zVInitialize model weights. Args: m: Module to initialize. g{Gz?)stdNr)rrLrSrr{rDrr)r`r2r3r3r4r,s zSwinTransformerV2._init_weightsrc window_ratioc s|dus|dur|jj||d|jj}|dur'dur'tfdd|D}t|jD]\}}dt|dd} |j|d| |d| f||dq,dS) aTUpdates the image resolution, window size, and so the pair-wise relative positions. Args: img_size (Optional[Tuple[int, int]]): New input resolution, if None current resolution is used patch_size (Optional[Tuple[int, int]): New patch size, if None use current patch size window_size (Optional[int]): New window size, if None based on new_img_size // window_div window_ratio (int): divisor for calculating window size from patch grid size always_partition: always partition / shift windows even if feat size is < window N)r6r csg|]}|qSr3r3)rrr4r3r4rrz4SwinTransformerV2.set_input_size..r%rrr)r$rr%r enumerater)r}) r`r6r r"r4rr%indexstage stage_scaler3r5r4rsz SwinTransformerV2.set_input_sizecs<t}|D]\}tfdddDr|q|S)zGet parameter names that should not use weight decay. Returns: Set of parameter names to exclude from weight decay. cg|]}|vqSr3r3)rkwnr3r4r.rz5SwinTransformerV2.no_weight_decay..)rUrQ)set named_modulesradd)r`nodr2r3r<r4no_weight_decay%s  z!SwinTransformerV2.no_weight_decaycoarsecCstd|rddSgddS)zCreate parameter group matcher for optimizer parameter groups. Args: coarse: If True, use coarse grouping. Returns: Dictionary mapping group names to regex patterns. z^absolute_pos_embed|patch_embedz^layers\.(\d+)))z^layers\.(\d+).downsampler)z^layers\.(\d+)\.\w+\.(\d+)N)z^norm)i)stemr)r()r`rCr3r3r4 group_matcher2s  zSwinTransformerV2.group_matcherenablecCs|jD]}||_qdS)z}Enable or disable gradient checkpointing. Args: enable: If True, enable gradient checkpointing. N)r)r)r`rFlr3r3r4set_grad_checkpointingEs z(SwinTransformerV2.set_grad_checkpointingcCs|jjS)z_Get the classifier head. Returns: The classification head module. )r*fc)r`r3r3r4get_classifierOsz SwinTransformerV2.get_classifiercCs||_|j||dS)zReset the classification head. Args: num_classes: Number of classes for new head. global_pool: Global pooling type. N)r r*reset)r`r r r3r3r4reset_classifierXsz"SwinTransformerV2.reset_classifierNCHWr!indicesr stop_earlyrintermediates_onlycCs|dvsJdg}tt|j|\}} ||}t|j} tjs%|s)|j} n |jd| d} t| D]+\} } | |}| |vra|rP| | dkrP||}n|}| dddd }| |q6|rf|S||}||fS)a Forward features that returns intermediates. Args: x: Input image tensor indices: Take last n blocks if int, all if None, select matching indices if sequence norm: Apply norm layer to compatible intermediates stop_early: Stop iterating over blocks when last desired intermediate hit output_fmt: Shape of intermediate feature outputs intermediates_only: Only return intermediate features Returns: )rMzOutput shape must be NCHW.Nrrr$r%) rrr)r$rNrrr6rr,r-append)r`r!rNrrOrrP intermediates take_indices max_index num_stagesstagesrr8x_interr3r3r4forward_intermediatesbs*     z'SwinTransformerV2.forward_intermediatesr prune_norm prune_headcCsJtt|j|\}}|jd|d|_|rt|_|r#|dd|S)z@ Prune layers not required for specified intermediates. Nrrr)rrr)rLrrrL)r`rNrYrZrSrTr3r3r4prune_intermediate_layerss  z+SwinTransformerV2.prune_intermediate_layerscCs"||}||}||}|S)zForward pass through feature extraction layers. Args: x: Input tensor of shape (B, C, H, W). Returns: Feature tensor of shape (B, H', W', C). )r$r)rr`r!r3r3r4forward_featuress  z"SwinTransformerV2.forward_features pre_logitscCs|r |j|ddS||S)aForward pass through classification head. Args: x: Feature tensor of shape (B, H, W, C). pre_logits: If True, return features before final linear layer. Returns: Logits tensor of shape (B, num_classes) or pre-logits. T)r^)r*)r`r!r^r3r3r4 forward_heads zSwinTransformerV2.forward_headcCs||}||}|S)zForward pass through the model. Args: x: Input tensor of shape (B, C, H, W). Returns: Logits tensor of shape (B, num_classes). )r]r_r\r3r3r4rs zSwinTransformerV2.forward)NNNrcNF)Tr)NFFrMF)rFT)&rrrrrLrrrrrrrr rrKrr,rrrNrignorerrBrrrErHrJrLrrrXr[r]r_rrr3r3rar4r |s         t        4  state_dictmodelc s|d|}|d|}d|v}i}ddl}|D]L\}tfdddDr*qd vrQ|jjjj\}}}}|jd |ksF|jd |krQt|||fd d d d}|sb| ddd dd||<q|S)aMFilter and process checkpoint state dict for loading. Handles resizing of patch embeddings and relative position tables when model size differs from checkpoint. Args: state_dict: Checkpoint state dictionary. model: Target model to load weights into. Returns: Filtered state dictionary. rcrbzhead.fc.weightrNcr:r3r3)rr=rr3r4rrz(checkpoint_filter_fn..)rfrerzpatch_embed.proj.weightr|r(bicubicT) interpolation antialiasverbosezlayers.(\d+).downsamplecSsdt|dddS)Nrrz .downsample)rgroup)r!r3r3r4sz&checkpoint_filter_fn..zhead.zhead.fc.) getreitemsrr$r\r{r*rsubreplace) rbrcnative_checkpointout_dictrlrrr/r0r3rdr4checkpoint_filter_fns.   rrFvariant pretrainedcKsPtddt|ddD}|d|}tt||fttd|dd|}|S) zCreate a Swin Transformer V2 model. Args: variant: Model variant name. pretrained: If True, load pretrained weights. **kwargs: Additional model arguments. Returns: SwinTransformerV2 model instance. css|]\}}|VqdSrr3)rrrr3r3r4rrz._create_swin_transformer_v2..r)rrrr out_indicesT)flatten_sequentialru)pretrained_filter_fn feature_cfg)rr6rkpoprr rrr()rsrtr-default_out_indicesrurcr3r3r4_create_swin_transformer_v2s  r{rc Ks"|ddddddttddd d |S) Nr)r$r|)rcrcg?reTzpatch_embed.projzhead.fcmit) urlr  input_size pool_sizecrop_pctrffixed_input_sizemeanr3 first_conv classifierlicenser )r~r-r3r3r4_cfg srztimm/z{https://github.com/SwinTransformer/storage/releases/download/v2.0.0/swinv2_base_patch4_window12to16_192to256_22kto1k_ft.pth) hf_hub_idr~z{https://github.com/SwinTransformer/storage/releases/download/v2.0.0/swinv2_base_patch4_window12to24_192to384_22kto1k_ft.pth)r$r)rrrd)rr~rrrz|https://github.com/SwinTransformer/storage/releases/download/v2.0.0/swinv2_large_patch4_window12to16_192to256_22kto1k_ft.pthz|https://github.com/SwinTransformer/storage/releases/download/v2.0.0/swinv2_large_patch4_window12to24_192to384_22kto1k_ft.pthzfhttps://github.com/SwinTransformer/storage/releases/download/v2.0.0/swinv2_tiny_patch4_window8_256.pthzghttps://github.com/SwinTransformer/storage/releases/download/v2.0.0/swinv2_tiny_patch4_window16_256.pthzghttps://github.com/SwinTransformer/storage/releases/download/v2.0.0/swinv2_small_patch4_window8_256.pthzhhttps://github.com/SwinTransformer/storage/releases/download/v2.0.0/swinv2_small_patch4_window16_256.pthzfhttps://github.com/SwinTransformer/storage/releases/download/v2.0.0/swinv2_base_patch4_window8_256.pthzghttps://github.com/SwinTransformer/storage/releases/download/v2.0.0/swinv2_base_patch4_window16_256.pthzkhttps://github.com/SwinTransformer/storage/releases/download/v2.0.0/swinv2_base_patch4_window12_192_22k.pthiQU)r$r)rr)rr~r rrzlhttps://github.com/SwinTransformer/storage/releases/download/v2.0.0/swinv2_large_patch4_window12_192_22k.pth) 2swinv2_base_window12to16_192to256.ms_in22k_ft_in1k2swinv2_base_window12to24_192to384.ms_in22k_ft_in1k3swinv2_large_window12to16_192to256.ms_in22k_ft_in1k3swinv2_large_window12to24_192to384.ms_in22k_ft_in1kzswinv2_tiny_window8_256.ms_in1kz swinv2_tiny_window16_256.ms_in1kz swinv2_small_window8_256.ms_in1kz!swinv2_small_window16_256.ms_in1kzswinv2_base_window8_256.ms_in1kz swinv2_base_window16_256.ms_in1k!swinv2_base_window12_192.ms_in22k"swinv2_large_window12_192.ms_in22kcK0tddddd}t d d|it|fi|S) z"Swin-T V2 @ 256x256, window 16x16.r~rrrr"r rr<swinv2_tiny_window16_256rtN)rr(r{rtr- model_argsr3r3r4rQrcKr) z Swin-T V2 @ 256x256, window 8x8.rcrrrrswinv2_tiny_window8_256rtN)rrrr3r3r4rYrrcKr) z"Swin-S V2 @ 256x256, window 16x16.r~rr%r%r%rrswinv2_small_window16_256rtN)rrrr3r3r4rarrcKr) z Swin-S V2 @ 256x256, window 8x8.rcrrrrswinv2_small_window8_256rtN)rrrr3r3r4rirrcKr) z"Swin-B V2 @ 256x256, window 16x16.r~rr&rcr~ rswinv2_base_window16_256rtN)rrrr3r3r4rqrrcKr) z Swin-B V2 @ 256x256, window 8x8.rcrrrrswinv2_base_window8_256rtN)rrrr3r3r4ryrrcKr) z"Swin-B V2 @ 192x192, window 12x12.rrrrrswinv2_base_window12_192rtN)rrrr3r3r4rrrcK2tdddddd}t d d|it|fi|S) zQSwin-B V2 @ 192x192, trained at window 12x12, fine-tuned to 256x256 window 16x16.r~rrrrrrrr"r rr<r!swinv2_base_window12to16_192to256rtN)rrrr3r3r4rrcKr) zQSwin-B V2 @ 192x192, trained at window 12x12, fine-tuned to 384x384 window 24x24.rrrrrr!swinv2_base_window12to24_192to384rtN)rrrr3r3r4rrrcKr) z"Swin-L V2 @ 192x192, window 12x12.rrrrrr0rswinv2_large_window12_192rtN)rrrr3r3r4rrrcKr) zQSwin-L V2 @ 192x192, trained at window 12x12, fine-tuned to 256x256 window 16x16.r~rrrrr"swinv2_large_window12to16_192to256rtN)rrrr3r3r4rrrcKr) zQSwin-L V2 @ 192x192, trained at window 12x12, fine-tuned to 384x384 window 24x24.rrrrrr"swinv2_large_window12to24_192to384rtN)rrrr3r3r4rrrrrrrrr)swinv2_base_window12_192_22k)swinv2_base_window12to16_192to256_22kft1k)swinv2_base_window12to24_192to384_22kft1kswinv2_large_window12_192_22k*swinv2_large_window12to16_192to256_22kft1k*swinv2_large_window12to24_192to384_22kft1kr`)r)Krrotypingrrrrrrrr r rNtorch.nnrLtorch.nn.functionalrr timm.datar r timm.layersrrrrrrrrrr_builderr _featuresr _features_fxr _manipulater _registryrrr__all__rrrr5r7rr8rrrr rrrrr{r default_cfgsrrrrrrrrrrrrrr3r3r3r4s, 0    "0h& ,R*  :