o پi]@sdZddlZddlZddlmZmZmZmZmZm Z ddl Z ddl m Z ddl m m ZddlmZmZddlmZddlmZeeZGd d d e jZGd d d eZ  d)deedededefddZe j Z!e!fdeeefdeeefdedede j"de j#de j$fddZ%e!fde j$de j$deeefde j#d e j#de j$f d!d"Z&  d)de j$deedededef d#d$Z'Gd%d&d&e jZ(Gd'd(d(e jZ)dS)*a? Image to Patch Embedding using Conv2d A convolution based approach to patchifying a 2D image w/ embedding projection. Based on code in: * https://github.com/google-research/vision_transformer * https://github.com/google-research/big_vision/tree/main/big_vision Hacked together by / Copyright 2020 Ross Wightman N)CallableDictListOptionalTupleUnion)nn)Formatnchw_to) to_2tuple)_assertcs:eZdZUdZeed<ejje ed<       d d e e e e e ffd e d e de de ede de ede de de ffdd Zd e e e e e fffddZ  d!d e e e e e e ffd e e e e e e fffddZd"de e e e fe ffddZd e e e fde e e ffddZddZZS)# PatchEmbed! 2D Image to Patch Embedding output_fmtdynamic_img_padNTFimg_size patch_sizein_chans embed_dim norm_layerflattenbiasstrict_img_sizec stt||_||\|_|_|_|dur"d|_t ||_ n||_t j |_ | |_ | |_ tj|||||d|_|rC|||_dSt|_dS)NF kernel_sizestrider)super__init__r r_init_img_sizer grid_size num_patchesrr rNCHWrrrConv2dprojIdentitynorm) selfrrrrrrrrrr __class__K/home/ubuntu/.local/lib/python3.10/site-packages/timm/layers/patch_embed.pyr" s   zPatchEmbed.__init__cCsR|jsJ|dur dSt|}tddt||jD}|d|d}|||fS)N)NNNcSsg|]\}}||qSr.r.).0spr.r.r/ Csz-PatchEmbed._init_img_size..rr )rr tuplezip)r+rr$r%r.r.r/r#>s  zPatchEmbed._init_img_sizecCsd}|dur t|}|durZ||jkrZt8tj|jj|jj|||jj dud}|j t |jj |dd|jj durE|j |jj ||_Wdn1sRwY||_|p^|j }||j ksh|duru||\|_ |_|_dSdS)NrT)verbose)r rtorchno_gradrr'r( in_channels out_channelsrweightcopy_resample_patch_embedrr#r$r%)r+rrnew_patch_sizenew_projr.r.r/set_input_sizeGs,    zPatchEmbed.set_input_sizereturncCs|rt|jS|jSN)maxr)r+ as_scalarr.r.r/ feat_ratioas zPatchEmbed.feat_ratiocCsZ|jrt|d|jdt|d|jdfS|d|jd|d|jdfS)z Get grid (feature) size for given image size taking account of dynamic padding. NOTE: must be torchscript compatible so using fixed tuple indexing rr )rmathceilr)r+rr.r.r/dynamic_feat_sizegs0$zPatchEmbed.dynamic_feat_sizecCsv|j\}}}}|jdurg|jr8t||jdkd|d|jddt||jdkd|d|jddn/|jsgt||jddkd|d|jddt||jddkd|d|jdd|jr|jd||jd|jd}|jd||jd|jd}t|d|d|f}||}|j r| d dd}n |j t j krt||j }||}|S) NrzInput height (z) doesn't match model ().r z Input width (z%) should be divisible by patch size ()shaperrr rrFpadr(r transposerr r&r r*)r+xBCHWpad_hpad_wr.r.r/forwardps2 (*""    zPatchEmbed.forward) rrrrNTNTTF)NN)T)__name__ __module__ __qualname____doc__r __annotations__r7jitFinalboolrintrrrstrr"r#r@rErHrV __classcell__r.r.r,r/rs\      " rcseZdZUdZeed<        dd eed ed ed ed eede dee de ffdd Z de e jeeffddZZS)PatchEmbedWithSizerrrrrrNTrrrrrrrc s tj||||||||ddS)N)rrrrrrrr)r!r") r+rrrrrrrrr,r.r/r"s  zPatchEmbedWithSize.__init__rAcCs|j\}}}}|jdur8t||jddkd|d|jddt||jddkd|d|jdd||}|jdd}|jrQ|ddd}n |jtj kr]t ||j}| |}||fS) NrzInput image height (z#) must be divisible by patch size (rIr zInput image width (rJ) rKrr rr(rrNrr r&r r*)r+rOrPrQrRrS feat_sizer.r.r/rVs ,,    zPatchEmbedWithSize.forward)rrrrNTNT)rWrXrYrZr r[rr_rr^r`r"rr7TensorrrVrar.r.r,r/rbs:  $rbbicubicTFnew_size interpolation antialiasr6c s.ddlzddlm}Wntyddlm}Ynwt|jdks'Jdtdks1Jd|jdd}t|tkrB|S|rTt d |jd d d fd dfdd}||}tj j |j |jdfdd} ||| dddd} |j} |}| |}|| }|S)a/Resample the weights of the patch embedding kernel to target resolution. We resample the patch embedding kernel by approximately inverting the effect of patch resizing. Code based on: https://github.com/google-research/big_vision/blob/b00544b81f8694488d5f36295aeb7972f3755ffe/big_vision/models/proj/flexi/vit.py With this resizing, we can for example load a B/8 filter into a B/16 model and, on 2x larger input image, the result will match. Args: patch_embed: original parameter to be resized. new_size (tuple(int, int): target shape (height, width)-only. interpolation (str): interpolation for resize antialias (bool): use anti-aliasing filter in resize verbose (bool): log operation Returns: Resized patch embedding kernel. rN)vmapzFour dimensions expectedrJzNew shape should only be hwrczResize patch embedding z to z, w/ z interpolation.cs,t|d}tj||dd}|S)N)NN.)sizemoderi)rr.)r7rerL interpolatenumpy)x_np _new_sizex_tf x_upsampled)rirhr.r/resizesz(resample_patch_embed_old..resizecsTg}t|D]}|}d|||<|||dq |jS)Ng?)rangeprodzeros unravel_indexappendreshapestackT) _old_sizerqmati basis_vec)nprtr.r/get_resize_mats   z0resample_patch_embed_old..get_resize_mat)devicecs|d}|S)Nru)r{)kernelresampled_kernel)rgresize_mat_pinvr.r/resample_kernels z1resample_patch_embed_old..resample_kernelr )ror7rj ImportError functorchlenrKr4_loggerinfotensorlinalgpinvr}rdtypefloatto) patch_embedrgrhrir6rjold_sizer resize_matrv_resample_kernel orig_dtyper.)rirhrgrrtrr/resample_patch_embed_olds0    rrrrrAcCst|\}}|\}} ||} || } tj| ||d} | | d||} tj| |||dd}|dddd| | }|S)zKComputes the resize matrix basis vectors and interpolates them to new_size.)rrr F)rlrmri align_cornersrJr)r7eyer{rLrnsqueezepermute)rrgrhrirrold_hold_wnew_hnew_w old_total new_total eye_matrixbasis_vectors_batchresized_basis_vectors_batch resize_matrixr.r.r/_compute_resize_matrixs rr pinv_matrixnew_size_tuplerintermediate_dtypec CsV|j^}}}|||dj|d}|j|d}||}|j||g|Rj|d}|S)zW Simplified resampling w/o vmap use. As proposed by https://github.com/stas-sl ru)r)rKr{r) rrrrrc_outc_in_resampled_patch_embedr.r.r/_apply_resamplings  rc Cst|jdks Jdt|dksJdt|jdd}t|}||kr(|S|j}|j}t|||||t} tj | } t || ||t} | S)z5 Standalone function (computes matrix on each call). rkz/Input tensor should be 4D (out_ch, in_ch, h, w)rJz+New shape should only be hw (height, width)rcN) rrKr4rrrDTYPE_INTERMEDIATEr7rrr) rrgrhrir6old_size_tuplerrrrrrr.r.r/r=.s    r=c seZdZdZ  ddeeefdedeffdd Ze fd eeefd e j d e j d e j fd dZde j d eed e j fddZZS) PatchEmbedResamplerFixedOrigSizez Resample patch embedding weights from a fixed original size, caching the pseudoinverse matrix based on the target size. rfT orig_sizerhricsDtt|trt|dksJd||_||_||_i|_dS)z Args: orig_size (Tuple[int, int]): The expected original (height, width) of input patch_embed tensors. interpolation (str): Interpolation mode. antialias (bool): Use anti-aliasing filter in resize. rJz.`orig_size` must be a tuple of (height, width)N) r!r" isinstancer4rrrhri_pinv_cache_map)r+rrhrir,r.r/r"Qs  z)PatchEmbedResamplerFixedOrigSize.__init__rgrrrAcCs|}|j|}|r t||r t||}|j|kr |j|kr |St|j||j|j ||}t j |}d|dd|d}t||rHt |||||||j|<|S)zRRetrieves the cached pinv matrix or computes and caches it for the given new_size.pinv_rrOr )rgethasattrgetattrrrrrrhrir7rrdelattrregister_buffer)r+rgrr cache_key buffer_namerrr.r.r/_get_or_create_pinv_matrixfs        z;PatchEmbedResamplerFixedOrigSize._get_or_create_pinv_matrixrc Cst|jdks Jt|dksJt|jdd}||jks*Jd|d|jt|}|j|kr5|S|j}|j}|||}t||||}|S)a Resamples the patch embedding weights to new_size. Args: patch_embed (torch.Tensor): Original weights (out_ch, in_ch, H_orig, W_orig). new_size (List[int]): Target [height, width]. Returns: torch.Tensor: Resampled weights. rkrJrcNzInput patch_embed spatial size z0 does not match module's expected original size )rrKr4rrrrr) r+rrg input_sizerrrrrr.r.r/rVs     z(PatchEmbedResamplerFixedOrigSize.forward)rfT)rWrXrYrZrr_r`r^r"rr7rrrerrrVrar.r.r,r/rLs,   &rc seZdZdZ    ddeeefdeded ed ef fd d Zd e j deeefde j fddZ d e j deeefde j fddZ   dde j de j de e j de eeefdede j f ddZZS)PatchEmbedInterpolatora8Dynamically interpolates patch embedding weights for variable patch sizes. This module wraps patch embedding weight resampling functionality to support on-the-fly patch size variation during training. It handles both Conv2d and Linear patch embeddings. Args: base_patch_size: The original patch size the model was initialized with in_chans: Number of input channels embed_dim: Embedding dimension interpolation: Interpolation mode for resampling antialias: Whether to use antialiasing during interpolation rrrfTbase_patch_sizerrrhrics,t||_||_||_||_||_dSrB)r!r"rrrrhri)r+rrrrhrir,r.r/r"s  zPatchEmbedInterpolator.__init__r;target_patch_sizerAc Cs||jkr|S|jd}|j\}}|\}}|||||j}|dddd}t|||g|j|jdd} | dddd} | |d} | S)a/Resample linear patch embedding weights for a new patch size. Args: weight: Linear weight tensor of shape [embed_dim, patch_h * patch_w * in_chans] target_patch_size: Target (patch_h, patch_w) to resample to Returns: Resampled weight tensor rrr rJFrgrhrir6ru)rrKr{rrr=rhri) r+r;rrbase_phbase_pw target_ph target_pw weight_convweight_conv_resampledweight_resampledr.r.r/resample_linear_weights"    z-PatchEmbedInterpolator.resample_linear_weightcCs,||jkr|St|t||j|jdd}|S)a-Resample conv2d patch embedding weights for a new patch size. Args: weight: Conv2d weight tensor of shape [embed_dim, in_chans, patch_h, patch_w] target_patch_size: Target (patch_h, patch_w) to resample to Returns: Resampled weight tensor Fr)rr=listrhri)r+r;rrr.r.r/resample_conv_weights z+PatchEmbedInterpolator.resample_conv_weightNpatches proj_weight proj_biasr is_linearcCs|dur|j}|rV||jkr7|jdksJd|j\}}}} } |||} |||d} tjj| | |} | S|jdkrK|j\}}}} } |||d}tjj|||} | S||jkro| ||} tjjj || ||dd} | Stjjj ||||dd} | S)aWApply patch embedding with dynamic weight resampling. Args: patches: Input patches - For linear mode with resampling: [B, N, Ph, Pw, C] - For linear mode without resampling: [B, N, Ph*Pw*C] - For conv mode: [B, C, H, W] proj_weight: Original projection weight proj_bias: Optional projection bias patch_size: Current patch size (if None, uses base_patch_size) is_linear: Whether using linear (True) or conv2d (False) projection Returns: Embedded patches Nz0Patches must be [B, N, Ph, Pw, C] for resamplingrur)r padding) rndimrKrr{r7r functionallinearrconv2d)r+rrrrrrPNPhPwrQr patches_flatoutputr.r.r/rV s6     zPatchEmbedInterpolator.forward)rrrfT)NNT)rWrXrYrZrr_r`r^r"r7rerrrrVrar.r.r,r/rs^   *   r)rfTF)*rZloggingrFtypingrrrrrrr7rtorch.nn.functionalrrLformatr r helpersr trace_utilsr getLoggerrWrModulerrbr_r`r^rfloat32rrrrerrr=rrr.r.r.r/s      r/ F        ]