o پi+h@sdZddlmZddlmZmZmZmZmZddl Z ddl m Z ddl m Z mZddlmZmZmZmZmZmZddlmZdd lmZdd lmZmZdd lmZmZd gZ Gd dde j!Z"Gddde j!Z#Gddde j!Z$Gddde j!Z%Gddde j!Z&Gdd d e j!Z'dEddZ(dFddZ)ee)dde)dde)dde)dde)dde)dde)dd d!dd"d#e)dd d!dd"d#e)dd d!dd"d#e)dd d!dd"d#e)dd"d$e)ddd$d% Z*d&e'fd'd(Z+dGd*d+Z,edGd,e'fd-d.Z-edGd,e'fd/d0Z.edGd,e'fd1d2Z/edGd,e'fd3d4Z0edGd,e'fd5d6Z1edGd,e'fd7d8Z2edGd,e'fd9d:Z3edGd,e'fd;d<Z4edGd,e'fd=d>Z5edGd,e'fd?d@Z6edGd,e'fdAdBZ7edGd,e'fdCdDZ8dS)Ha[ FocalNet As described in `Focal Modulation Networks` - https://arxiv.org/abs/2203.11926 Significant modifications and refactoring from the original impl at https://github.com/microsoft/FocalNet This impl is/has: * fully convolutional, NCHW tensor layout throughout, seemed to have minimal performance impact but more flexible * re-ordered downsample / layer so that striding always at beginning of layer (stage) * no input size constraints or input resolution/H/W tracking through the model * torchscript fixed and a number of quirks cleaned up * feature extraction support via `features_only=True` )partial)CallableListOptionalTupleUnionNIMAGENET_DEFAULT_MEANIMAGENET_DEFAULT_STD)MlpDropPath LayerNorm2d trunc_normal_ClassifierHeadNormMlpClassifierHead)build_model_with_cfg)feature_take_indices) named_apply checkpoint)generate_default_cfgsregister_modelFocalNetcsTeZdZdddddefdedededed ed ed ed effd d ZddZ Z S)FocalModulationTFdim focal_level focal_factorbias use_post_normnormalize_modulator proj_drop norm_layerc s*t||_||_||_||_||_||_|||jdg|_t j |d||jdd|d|_ t j ||d|d|_ t |_t j ||dd|_t ||_t |_g|_t|jD]'} |j| |j} |jt t j ||| || dddt |j| q\|jr| ||_dSt |_dS)Nrr) kernel_sizer)r$F)r$groupspaddingr)super__init__r focal_windowrrr r! input_splitnnConv2dfhGELUactprojDropoutr" ModuleList focal_layers kernel_sizesrangeappend SequentialIdentitynorm) selfrr)rrrr r!r"r#kr$ __class__H/home/ubuntu/.local/lib/python3.10/site-packages/timm/models/focalnet.pyr(%s. "    "zFocalModulation.__init__c Cs||}t||jd\}}}d}t|jD]\}}||}|||dd||df}q||jddd}|||dd|jdf}|j rS||jd}|| |} | | } | | } | | } | S)Nrr)rT)keepdim)r-torchsplitr* enumerater4r0meanrr!r.r:r1r") r;xqctxgatesctx_alll focal_layer ctx_globalx_outr?r?r@forwardMs "   zFocalModulation.forward) __name__ __module__ __qualname__r intboolfloatrr(rP __classcell__r?r?r=r@r$s2  (rcs&eZdZdfdd ZddZZS) LayerScale2dh㈵>Fcs*t||_t|t||_dSN)r'r(inplacer+ ParameterrConesgamma)r;r init_valuesr[r=r?r@r(is zLayerScale2d.__init__cCs*|jdddd}|jr||S||S)Nr)r^viewr[mul_)r;rGr^r?r?r@rPnszLayerScale2d.forward)rYF)rQrRrSr(rPrWr?r?r=r@rXhsrXcsteZdZdZdddddddddejef ded ed ed ed e d e de dededede de ffdd Z ddZ Z S) FocalNetBlockz% Focal Modulation Network Block. @rrAF-C6?rr mlp_ratiorr)r use_post_norm_in_modulationr!layerscale_valuer" drop_path act_layerr#c s,t||_||_||_||_||_|s| |nt|_ t |||j||| | d|_ |r2| |nt|_ |dur@t ||nt|_| dkrMt| nt|_|sX| |nt|_t|t||| | dd|_|rq| |nt|_|durt ||nt|_| dkrt| |_dSt|_dS)ap Args: dim: Number of input channels. mlp_ratio: Ratio of mlp hidden dim to embedding dim. focal_level: Number of focal levels. focal_window: Focal window size at first focal level. use_post_norm: Whether to use layer norm after modulation. use_post_norm_in_modulation: Whether to use layer norm in modulation. layerscale_value: Initial layerscale value. proj_drop: Dropout rate. drop_path: Stochastic depth rate. act_layer: Activation layer. norm_layer: Normalization layer. )r)rr r!r"r#NrT) in_featureshidden_featuresrjdropuse_conv)r'r(rrfr)rr r+r9norm1r modulation norm1_postrXls1r drop_path1norm2r rTmlp norm2_postls2 drop_path2) r;rrfrr)r rgr!rhr"rirjr#r=r?r@r(ws<   $zFocalNetBlock.__init__c Cs`|}||}||}||}||||}|||||| |}|SrZ) rorprqrsrrrxrwrvrurt)r;rGshortcutr?r?r@rPs   &zFocalNetBlock.forward)rQrRrS__doc__r+r/r rTrVrUrr(rPrWr?r?r=r@rcssN     ?rccseZdZdZdddddddddddef ded ed ed ed ed edededededededededeffdd Z e j j dddZ ddZZS) FocalNetStagez4 A basic Focal Transformer layer for one stage. rdTrFrerrout_dimdepthrf downsamplerr)use_overlap_downr rgr!rhr"rir#c svt||_||_d|_|rt|d|d|_nt|_t  f ddt |D|_ dS)a8 Args: dim: Number of input channels. out_dim: Number of output channels. depth: Number of blocks. mlp_ratio: Ratio of mlp hidden dim to embedding dim. downsample: Downsample layer at start of the layer. focal_level: Number of focal levels focal_window: Focal window size at first focal level use_overlap_down: User overlapped convolution in downsample layer. use_post_norm: Whether to use layer norm after modulation. use_post_norm_in_modulation: Whether to use layer norm in modulation. layerscale_value: Initial layerscale value proj_drop: Dropout rate for projections. drop_path: Stochastic depth rate. norm_layer: Normalization layer. Fr)in_chsout_chsstrideoverlapr#cs<g|]}t ttr|nd qS)) rrfrr)r rgr!rhr"rir#)rc isinstancelist.0i rirr)rhrfr#r!r|r"r rgr?r@ s z*FocalNetStage.__init__..N) r'r(rr}grad_checkpointing Downsampler~r+r9r3r6blocks)r;rr|r}rfr~rr)rr rgr!rhr"rir#r=rr@r(s # "zFocalNetStage.__init__cCs ||_dSrZ)r)r;enabler?r?r@set_grad_checkpointing s z$FocalNetStage.set_grad_checkpointingcCs>||}|jD]}|jrtjst||}q||}q|SrZ)r~rrrCjit is_scriptingr)r;rGblkr?r?r@rPs    zFocalNetStage.forwardT)rQrRrSrzr rTrVrUrr(rCrignorerrPrWr?r?r=r@r{s`     D r{c sFeZdZ   d dededededeef fd d Zd d ZZ S)rFNrrrrr#cst||_d}|}|r%|dvsJ|dkrd\}}n|dkr%d\}}tj|||||d|_|dur;|||_dSt|_dS) a Args: in_chs: Number of input image channels. out_chs: Number of linear projection output channels. stride: Downsample stride. overlap: Use overlapping convolutions if True. norm_layer: Normalization layer. r)rrr)rr)rAr)r$rr&N)r'r(rr+r,r1r9r:)r;rrrrr#r&r$r=r?r@r(s   $zDownsample.__init__cC||}||}|SrZ)r1r:r;rGr?r?r@rP;  zDownsample.forward)rFN) rQrRrSrTrUrrr(rPrWr?r?r=r@rs rc'seZdZdZdddddddd d d d d d d d d d deeddfdededededeedfde deedfdeedfde de de de de ede d e e d!e d"e d#e d$e f&fd%d& Z ejjd'd(ZejjdHd)d*ZejjdId,d-Zejjd.ejfd/d0ZdJdede efd1d2Z 3 dKd4ejd5e eeeefd6e d7e d8ed9e d.eeejeejeejfffd:d;Z < +dLd5eeeefd=e d>e fd?d@ZdAdBZdHdCe fdDdEZdFdGZZ S)Mrz," Focal Modulation Networks (FocalNets) rAavg`rrrrd)rrrrrArArArAFN?rg?rY)epsin_chans num_classes global_pool embed_dimdepths.rf focal_levels focal_windowsrr rgr!head_hidden_sizehead_init_scalerh drop_rateproj_drop_ratedrop_path_rater#cstt||_fddt|jD||_|_d|_|_g|_ t |d| |d|_ d}ddt d|t|D}g}t|jD]N}|}t||||||dk||||| | | | |||t|d|t|d|d|d }|}||g7}|j t|d d |d |d g7_ qKtj||_| rt|_| |_t|j|| |||d|_n||j|_t|j|||d|_ttt|d|dS)aa Args: in_chans: Number of input image channels. num_classes: Number of classes for classification head. embed_dim: Patch embedding dimension. depths: Depth of each Focal Transformer layer. mlp_ratio: Ratio of mlp hidden dim to embedding dim. focal_levels: How many focal levels at all stages. Note that this excludes the finest-grain level. focal_windows: The focal window size at all stages. use_overlap_down: Whether to use convolutional embedding. use_post_norm: Whether to use layernorm after modulation (it helps stabilize training of large models) layerscale_value: Value for layer scale. drop_rate: Dropout rate. drop_path_rate: Stochastic depth rate. norm_layer: Normalization layer. csg|]}d|qS)rr?rrr?r@rnsz%FocalNet.__init__..r`r)rrrr#cSsg|]}|qSr?)item)rrGr?r?r@r}sNr)rr|r}rfr~rr)rr rgr!rhr"rir#rrlayers.)num_chs reductionmodule) hidden_size pool_typerr#)rr)r)r'r(len num_layersr6rr num_featuresr feature_inforstemrClinspacesumr{dictr+r8layersr9r:rheadrrr _init_weights)r;rrrrrrfrrrr rgr!rrrhrrrr#in_dimdprri_layerr|layerr=rr@r(Est & & *   zFocalNet.__init__cCsdhS)Nr?r;r?r?r@no_weight_decayszFocalNet.no_weight_decaycCs td|r ddgdSgddS)Nz^stem)z^layers\.(\d+)Nz^norm)i))z^layers\.(\d+).downsample)r)z^layers\.(\d+)\.\w+\.(\d+)Nr)rr)r)r;coarser?r?r@ group_matcherszFocalNet.group_matcherTcCs"||_|jD]}|j|dqdS)N)r)rrr)r;rrLr?r?r@rs zFocalNet.set_grad_checkpointingreturncCs|jjSrZ)rfcrr?r?r@get_classifierszFocalNet.get_classifiercCs||_|jj||ddS)N)r)rrreset)r;rrr?r?r@reset_classifierszFocalNet.reset_classifierNCHWrGindicesr: stop_early output_fmtintermediates_onlycCs|dvsJdg}tt|j|\}} ||}tjs |s$|j} n |jd| d} t|jd} t| D]\} } | |}| |vrW|rP| | krP||}n|}| |q8|r\|S| | kre||}||fS)a Forward features that returns intermediates. Args: x: Input image tensor indices: Take last n blocks if int, all if None, select matching indices if sequence norm: Apply norm layer to compatible intermediates stop_early: Stop iterating over blocks when last desired intermediate hit output_fmt: Shape of intermediate feature outputs intermediates_only: Only return intermediate features Returns: )rzOutput shape must be NCHW.Nr) rrrrrCrrrEr:r7)r;rGrr:rrr intermediates take_indices max_indexstageslast_idxfeat_idxstagex_interr?r?r@forward_intermediatess*     zFocalNet.forward_intermediatesr prune_norm prune_headcCsJtt|j|\}}|jd|d|_|rt|_|r#|dd|S)z@ Prune layers not required for specified intermediates. Nrrr)rrrr+r9r:r)r;rrrrrr?r?r@prune_intermediate_layerss  z"FocalNet.prune_intermediate_layerscCs"||}||}||}|SrZ)rrr:rr?r?r@forward_featuress   zFocalNet.forward_features pre_logitscCs|r |j||dS||S)N)r)r)r;rGrr?r?r@ forward_headszFocalNet.forward_headcCrrZ)rrrr?r?r@rPrzFocalNet.forwardFrrZ)NFFrF)rFT)!rQrRrSrzrr rTstrrrVrUrrr(rCrrrrrr+ModulerrTensorrrrrrrrPrWr?r?r=r@rAs        i     4 rcCst|tjrt|jdd|jdurtj|jdSdSt|tjrLt|jdd|jdur6tj|j|rNd|vrP|jj ||jj |dSdSdSdS)Ng{Gz?)stdhead.fc) rr+r,rweightrinitzeros_Lineardatarb)rnamerr?r?r@rs     rrc Ks |dddddttdddd |S) Nr)rAr)rrg?bicubicz stem.projrmit) urlr input_size pool_sizecrop_pct interpolationrFr first_conv classifierlicenser)rkwargsr?r?r@_cfg-srztimm/) hf_hub_id)rAr) riRU)rrrrr)rr) zfocalnet_tiny_srf.ms_in1kzfocalnet_small_srf.ms_in1kzfocalnet_base_srf.ms_in1kzfocalnet_tiny_lrf.ms_in1kzfocalnet_small_lrf.ms_in1kzfocalnet_base_lrf.ms_in1kzfocalnet_large_fl3.ms_in22kzfocalnet_large_fl4.ms_in22kzfocalnet_xlarge_fl3.ms_in22kzfocalnet_xlarge_fl4.ms_in22kzfocalnet_huge_fl3.ms_in22kzfocalnet_huge_fl4.ms_in22kmodelcCs|d|}d|vr |Sddl}i}|}|D]W\}}|dd|}|dd}|dd d |}d |vrC||vrC|d d |}|dd}|dd}||vrm|||krm||j|jkrm|||j}|||<q|S)Nrzstem.proj.weightrz gamma_([0-9])z ls\1.gamma patch_embedrzlayers.(\d+).downsamplecSsdt|dddS)Nrrz .downsample)rTgroup)rGr?r?r@esz&checkpoint_filter_fn..r:z norm([0-9])z norm\1_postzln.znorm.rr) getre state_dictitemssubreplacenumelshapereshape)rrrout_dict dest_dictr<vr?r?r@checkpoint_filter_fn[s$    , rFcKsPtddt|ddD}|d|}tt||fttd|dd|}|S) Ncss|]\}}|VqdSrZr?)rr_r?r?r@ qsz#_create_focalnet..r)rrrAr out_indicesT)flatten_sequentialr)pretrained_filter_fn feature_cfg)tuplerErpoprrrr)variant pretrainedrdefault_out_indicesrrr?r?r@_create_focalnetps  rrcK*tdgddd|}tdd|i|S)Nrrrrfocalnet_tiny_srfrr?)rrrrr model_kwargsr?r?r@r|rcKr)Nrrrrrfocalnet_small_srfrr?)r&r r!r?r?r@r&r#r&cKr)Nr$rfocalnet_base_srfrr?)r(r r!r?r?r@r(r#r(cK0tdgddgdd|}tdd|i|S) Nrrrrrrfocalnet_tiny_lrfrr?)r+r r!r?r?r@r+r+cKr)) Nr$rrr*focalnet_small_lrfrr?)r-r r!r?r?r@r-r,r-cKr)) Nr$r'rr*focalnet_base_lrfrr?)r.r r!r?r?r@r.r,r.c K>td gddgddgddddd|}td d |i|S) Nr$rrTrerrrrr rrhfocalnet_large_fl3rr?)r3r r!r?r?r@r3r3c K6td gddgddddd|}td d|i|S) Nr$r0rrrrTrerrrr rrhfocalnet_large_fl4rr?)r8r r!r?r?r@r8r8c Kr/) Nr$rr1rTrer2focalnet_xlarge_fl3rr?)r;r r!r?r?r@r;r4r;c Kr5) Nr$r:r6Trer7focalnet_xlarge_fl4rr?)r<r r!r?r?r@r<r9r<c Ks@td gddgddgdddddd|}td d |i|S) Nr$`rrArTre)rrrrr rgrrhfocalnet_huge_fl3rr?)r>r r!r?r?r@r>sr>c Ks8td gddgdddddd|}td d|i|S) Nr$r=r6Tre)rrrr rgrrhfocalnet_huge_fl4rr?)r?r r!r?r?r@r?sr?)Nr)rr)9rz functoolsrtypingrrrrrrCtorch.nnr+ timm.datar r timm.layersr r r rrr_builderr _featuresr _manipulaterr _registryrr__all__rrrXrcr{rrrr default_cfgsrrrr&r(r+r-r.r3r8r;r<r>r?r?r?r?r@s     D RV& _  #