o ۷im~ @sdZddlZddlZddlmZmZddlZddlmZddl m Z ddl m Z ddl mZmZdd lmZdd lmZmZdd lmZd d lmZeeZGdddejZejjddZ ddZ!GdddejZ"d:dej#de$de%dej#fddZ&GdddejZ'Gd d!d!ejZ(Gd"d#d#ejZ)Gd$d%d%ejZ*d&d'Z+d(d)Z,Gd*d+d+e Z-Gd,d-d-ejZ.d.ejddfd/d0Z/eGd1d2d2eZ0eGd3d4d4e0Z1ed5d6Gd7d8d8e0eZ2gd9Z3dS);zPyTorch ViTDet backbone.N)OptionalUnion)nn)ACT2FN)GradientCheckpointingLayer)BackboneOutputBaseModelOutput)PreTrainedModel)auto_docstringlogging) BackboneMixin) VitDetConfigcs>eZdZdZfddZddZdejdejfdd ZZ S) VitDetEmbeddingsz This class turns `pixel_values` of shape `(batch_size, num_channels, height, width)` into the initial `hidden_states` (patch embeddings) to be consumed by a Transformer. cst|j|j}}|j|j}}t|tjj r|n||f}t|tjj r)|n||f}|d|d|d|d}||_ ||_||_||_ |j r]|d}t td||j|_nd|_t j||||d|_dS)Nrr) kernel_sizestride)super__init__pretrain_image_size patch_size num_channels hidden_size isinstance collectionsabcIterable image_size num_patches use_absolute_position_embeddingsr Parametertorchzerosposition_embeddingsConv2d projection)selfconfigrrrrr num_positions __class__`/home/ubuntu/vllm_env/lib/python3.10/site-packages/transformers/models/vitdet/modeling_vitdet.pyr*s  zVitDetEmbeddings.__init__cCs|r |ddddf}|jd}tt|}|||kr"tdtjs/||ks/||krOtj j | d||d dddd||fdd d }| ddddS| d||dS) a Calculate absolute positional embeddings. If needed, resize embeddings and remove cls_token dimension for the original embeddings. Args: abs_pos_embeddings (`torch.Tensor`): Absolute positional embeddings with (1, num_position, num_channels). has_cls_token (`bool`): If true, has 1 embedding in abs_pos_embeddings for cls token. height (`int`): Height of input image tokens. width (`int`): Width of input image tokens. Returns: Absolute positional embeddings after processing with shape (1, height, width, num_channels) Nrz5Absolute position embeddings must be a square number.rrbicubicF)sizemode align_corners) shapeintmathsqrt ValueErrorr!jit is_tracingr functional interpolatereshapepermute)r&abs_pos_embeddings has_cls_tokenheightwidth num_positionr0new_abs_pos_embeddingsr+r+r,get_absolute_positions@s  z'VitDetEmbeddings.get_absolute_positions pixel_valuesreturncCs|jd}||jkrtd|jd|d||}|jdurA|dddd}|||jd|jd|jd}|dddd}|S) NrzoMake sure that the channel dimension of the pixel values match with the one set in the configuration. Expected z but got .rr.rT)r3rr7r%r#r=rD)r&rEr embeddingsr+r+r,forwardfs"    zVitDetEmbeddings.forward) __name__ __module__ __qualname____doc__rrDr!TensorrI __classcell__r+r+r)r,r$s  &rcCstdt||d}|jd|kr3tjj|d|jddddd|dd}|d|dd}n|}t |dddft||d}t |dddft||d}|||dt||d}|| S) a Get relative positional embeddings according to the relative positions of query and key sizes. Args: q_size (`int`): Size of query q. k_size (`int`): Size of key k. rel_pos (`torch.Tensor`): Relative position embeddings (num_embeddings, num_channels). Returns: Extracted positional embeddings according to relative positions. r.rrr-linear)r0r1N?) r4maxr3rr:r;r<r=r!arangelong)q_sizek_sizerel_pos max_rel_distrel_pos_resizedq_coordsk_coordsrelative_coordsr+r+r, get_rel_pos|s$$ r]cCs|\}}|\}} t|||} t|| |} |j\} } }|| |||}td|| } td|| }|| |||| | dddddddddf|dddddddddf| |||| }|S)a Calculate decomposed Relative Positional Embeddings as introduced in [MViT2](https://github.com/facebookresearch/mvit/blob/19786631e330df9f3622e5402b4a419a263a2c80/mvit/models/attention.py). Args: attn (`torch.Tensor`): Attention map. queries (`torch.Tensor`): Query q in the attention layer with shape (batch_size, queries_height * queries_width, num_channels). rel_pos_h (`torch.Tensor`): Relative position embeddings (Lh, num_channels) for height axis. rel_pos_w (`torch.Tensor`): Relative position embeddings (Lw, num_channels) for width axis. q_size (`tuple[int]`): Spatial sequence size of query q with (queries_height, queries_width). k_size (`tuple[int]`): Spatial sequence size of key k with (keys_height, keys_width). Returns: attn (Tensor): attention map with added relative positional embeddings. zbhwc,hkc->bhwkzbhwc,wkc->bhwkN)r]r3r<r!einsumview)attnqueries rel_pos_h rel_pos_wrUrVqueries_height queries_width keys_height keys_widthrelative_heightrelative_width batch_size_dimr_qrelative_weightr+r+r,!add_decomposed_relative_positionss      rocs,eZdZdZdfdd Zd ddZZS) VitDetAttentionz=Multi-head Attention block with relative position embeddings.Ncst|j}|j}||_||}|d|_tj||d|jd|_ t|||_ |j |_ |j rSt t d|dd||_t t d|dd||_dSdS)z Args: config (`VitDetConfig`): Model configuration. input_size (`tuple[int]`, *optional*): Input resolution, only required in case relative position embeddings are added. grbiasr.rrN)rrrnum_attention_heads num_headsscalerLinearqkv_biasqkvproj use_relative_position_embeddingsr r!r"rbrc)r&r' input_sizerlrthead_dimr)r+r,rs   $zVitDetAttention.__init__FcCs&|j\}}}}|||||d|jdddddd}|d||j||dd\}} } ||j| dd} |jrMt | ||j |j ||f||f} | j dd} | | }| ||j||d}|ddddd}||||d}||}|r| ||j| jd| jd} || f} | S|f} | S) Nrr-r.rr)rl)r3rxr<rtr=unbindru transposerzrorbrcsoftmaxr_ry)r& hidden_stateoutput_attentionsrjr@rArkrxrakeysvaluesattention_scoresattention_probsoutputsr+r+r,rIs,,&  zVitDetAttention.forwardN)FrJrKrLrMrrIrOr+r+r)r,rpsrpFinput drop_probtrainingrFcCsd|dks|s|Sd|}|jdfd|jd}|tj||j|jd}||||}|S)aF Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks). Comment by Ross Wightman: This is the same as the DropConnect impl I created for EfficientNet, etc networks, however, the original name is misleading as 'Drop Connect' is a different form of dropout in a separate paper... See discussion: https://github.com/tensorflow/tpu/issues/494#issuecomment-532968956 ... I've opted for changing the layer and argument names to 'drop path' rather than mix DropConnect as a layer name and use 'survival rate' as the argument. rrr)r)dtypedevice)r3ndimr!randrrfloor_div)rrr keep_probr3 random_tensoroutputr+r+r, drop_path s rcsTeZdZdZd deeddffdd Zdejdejfdd Z de fd d Z Z S) VitDetDropPathzXDrop paths (Stochastic Depth) per sample (when applied in main path of residual blocks).NrrFcst||_dSr)rrr)r&rr)r+r,r!s  zVitDetDropPath.__init__ hidden_statescCst||j|jSr)rrr)r&rr+r+r,rI%szVitDetDropPath.forwardcCs d|jS)Nzp=)rr&r+r+r, extra_repr(s zVitDetDropPath.extra_reprr) rJrKrLrMrfloatrr!rNrIstrrrOr+r+r)r,rs rcs*eZdZdZdfdd ZddZZS)VitDetLayerNormaL A LayerNorm variant, popularized by Transformers, that performs point-wise mean and variance normalization over the channel dimension for inputs that have shape (batch_size, channels, height, width). https://github.com/facebookresearch/ConvNeXt/blob/d1fa8f6fef0a165b27399986cc2bdacc92777e40/models/convnext.py#L119 ư>cs@ttt||_tt||_||_ |f|_ dSr) rrrr r!onesweightr"rrepsnormalized_shape)r&rrr)r+r,r3s  zVitDetLayerNorm.__init__cCsn|jddd}||djddd}||t||j}|jddddf||jddddf}|S)NrT)keepdimr.)meanpowr!r6rrrr)r&xusr+r+r,rI:s ,zVitDetLayerNorm.forward)rrr+r+r)r,r,srcs(eZdZdZfddZddZZS)VitDetResBottleneckBlockz The standard bottleneck residual block without the last activation layer. It contains 3 conv layers with kernels 1x1, 3x3, 1x1. csttj||ddd|_t||_t|j|_ tj||dddd|_ t||_ t|j|_ tj||ddd|_ t||_dS)ar Args: config (`VitDetConfig`): Model configuration. in_channels (`int`): Number of input channels. out_channels (`int`): Number of output channels. bottleneck_channels (`int`): Number of output channels for the 3x3 "bottleneck" conv layers. rFrqr)paddingrrN)rrrr$conv1rnorm1r hidden_actact1conv2norm2act2conv3norm3)r&r' in_channels out_channelsbottleneck_channelsr)r+r,rHs     z!VitDetResBottleneckBlock.__init__cCs&|}|D]}||}q||}|Sr)children)r&routlayerr+r+r,rI`s   z VitDetResBottleneckBlock.forwardrr+r+r)r,rBs rcs@eZdZdededdffdd Zdejdejfdd ZZS) VitDetMlp in_featureshidden_featuresrFNcsDtt|||_t|j|_t|||_t |j |_ dSr) rrrrvfc1rractfc2Dropout dropout_probdrop)r&r'rrr)r+r,rjs  zVitDetMlp.__init__rcCs6||}||}||}||}||}|Sr)rrrr)r&rr+r+r,rIqs     zVitDetMlp.forward) rJrKrLr4rr!rNrIrOr+r+r)r,risrc Cs|j\}}}}||||}||||}tj|ddd|d|f}||||}} |||||| |||}|ddddddd|||} | || ffS)a  Partition into non-overlapping windows with padding if needed. Args: hidden_state (`torch.Tensor`): Input tokens with [batch_size, height, width, num_channels]. window_size (`int`): Window size. Returns: `tuple(torch.FloatTensor)` comprising various elements: - windows: windows after partition with [batch_size * num_windows, window_size, window_size, num_channels]. - (padded_height, padded_width): padded height and width before partition rrrr.r}r-)r3rr:padr_r= contiguous) r window_sizerjr@rAr pad_height pad_width padded_height padded_widthwindowsr+r+r,window_partition{s$ rc Cs|\}}|\}}|jd||||}||||||||d} | dddddd} | |||d} | ddd|d|ddf} | S) aB Window unpartition into original sequences and removing padding. Args: windows (`torch.Tensor`): Input tokens with [batch_size * num_windows, window_size, window_size, num_channels]. window_size (`int`): Window size. pad_height_width (`tuple[int]`): Padded height and width (padded_height, padded_width). height_width (`tuple[int]`): Original height and width before padding. Returns: hidden_state: unpartitioned sequences with [batch_size, height, width, num_channels]. rr-rrr.r}rN)r3r_r=r) rrpad_height_width height_widthrrr@rArjrr+r+r,window_unpartitions$rc s|eZdZdZ ddededededd f fd d Z dd e j d e e j dede e e j e j fe e j ffddZZS) VitDetLayerzCThis corresponds to the Block class in the original implementation.rFr'drop_path_rateruse_residual_blockrFNc st|j}|j}t|ttfr|n||f}|j}t|ttfr$|n||f}|d|d|d|df}tj ||j d|_ t ||dkrI|n||fd|_ |dkrXt|nt|_tj ||j d|_t||t||jd|_||_||_|jrt||||dd|_dSdS) Nrr)r)r{r)r'rrr.)r'rrr)rrrrrlisttuplerr LayerNormlayer_norm_epsrrp attentionrIdentityrrrr4 mlp_ratiomlprrrresidual) r&r'rrrrlrrr{r)r+r,rs0   zVitDetLayer.__init__r head_maskrc Cs|dddd}|}||}|jdkr'|jd|jd}}t||j\}}|j||d}|d}|dd} |jdkrGt||j|||f}|||}|||| |}|dddd}|j rk| |}|f| } | S)Nrr.rr)r) r=rrr3rrrrrrrr) r&rrrshortcutr@rArself_attention_outputsrr+r+r,rIs*      zVitDetLayer.forward)rrF)NF)rJrKrLrMrrr4boolrr!rNrrrrIrOr+r+r)r,rs2&rcsbeZdZdeddffdd Z    ddejd eejd ed ed ede e e ff d dZ Z S) VitDetEncoderr'rFNc st||_|j}ddtjd|j|ddD}g}t|D]}|t |||||j vr1|j nd||j vdq t ||_d|_dS)NcSsg|]}|qSr+)item).0rr+r+r, sz*VitDetEncoder.__init__..rcpu)r)rrrF)rrr'num_hidden_layersr!linspacerrangeappendrwindow_block_indicesrresidual_block_indicesr ModuleListrgradient_checkpointing)r&r'depthrlayersir)r+r,r s   zVitDetEncoder.__init__FTrrroutput_hidden_states return_dictc Cs|rdnd}|r dnd}t|jD](\}} |r||f}|dur$||nd} | || |} | d}|r9|| df}q|rA||f}|sOtdd|||fDSt|||dS)Nr+rrcss|] }|dur|VqdSrr+)rvr+r+r, >sz(VitDetEncoder.forward..last_hidden_stater attentions) enumeraterrr ) r&rrrrrall_hidden_statesall_self_attentionsr layer_modulelayer_head_mask layer_outputsr+r+r,rI"s(     zVitDetEncoder.forward)NFFT)rJrKrLrrr!rNrrrrr rIrOr+r+r)r,r s& rmodulecCs6tjj|jddd|jdurtj|jddSdS)a Initialize `module.weight` using the "MSRAFill" implemented in Caffe2. Also initializes `module.bias` to 0. Source: https://detectron2.readthedocs.io/en/latest/_modules/fvcore/nn/weight_init.html. Args: module (torch.nn.Module): module to initialize. fan_outrelu)r1 nonlinearityNr)rinitkaiming_normal_rrr constant_)rr+r+r,caffe2_msra_fillFs rc@sHeZdZUeed<dZdZdZgZde e j e j e j fddfdd ZdS) VitDetPreTrainedModelr'vitdetrETrrFNcCst|tjtjfr0tjj|jjt j d|j j d|jj |j_|jdur.|jjdSdSt|tjrE|jj|jjddSt|trdtjj|jjt j d|j j d|jj |j_dSt|tr|j jrtjj|jjt j d|j j d|j_tjj|jjt j d|j j d|j_dSt|tr|j|j|jfD]}t|q|j|jfD]}|jjd|jjq|jjj|jjjdSdS)zInitialize the weightsr)rstdNrQ) rrrvr$r trunc_normal_rdatator!float32r'initializer_rangerrrzero_rfill_rr#rprzrbrcrrrrrrrr)r&rrr+r+r, _init_weights\sP          z#VitDetPreTrainedModel._init_weights)rJrKrLr__annotations__base_model_prefixmain_input_namesupports_gradient_checkpointing_no_split_modulesrrrvr$rrr+r+r+r,rTs &rcseZdZdeffdd ZdefddZdeee efddfd d Z e     dd e e jd e e jd e ede ede edeeeff ddZZS) VitDetModelr'cs2t|||_t||_t||_|dSr)rrr'rrHrencoder post_initr&r'r)r+r,rs    zVitDetModel.__init__rFcC|jjSrrHr%rr+r+r,get_input_embeddingsz VitDetModel.get_input_embeddingsheads_to_pruneNcCs*|D]\}}|jj|j|qdS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrr prune_heads)r&rrheadsr+r+r, _prune_headsszVitDetModel._prune_headsrErrrrc Cs|dur|n|jj}|dur|n|jj}|dur|n|jj}|dur&td|||jj}||}|j|||||d}|d}|sL|f|ddSt ||j |j dS)a Examples: ```python >>> from transformers import VitDetConfig, VitDetModel >>> import torch >>> config = VitDetConfig() >>> model = VitDetModel(config) >>> pixel_values = torch.randn(1, 3, 224, 224) >>> with torch.no_grad(): ... outputs = model(pixel_values) >>> last_hidden_states = outputs.last_hidden_state >>> list(last_hidden_states.shape) [1, 768, 14, 14] ```Nz You have to specify pixel_values)rrrrrrr) r'rruse_return_dictr7 get_head_maskrrHrr rr) r&rErrrrembedding_outputencoder_outputssequence_outputr+r+r,rIs. zVitDetModel.forward)NNNNN)rJrKrLrrrrdictr4rr#r rr!rNrrrr rIrOr+r+r)r,rs.  rzF ViTDet backbone, to be used with frameworks like Mask R-CNN. ) custom_introc sbeZdZfddZdefddZe   d dejde e d e e d e e de f d d Z Z S)VitDetBackbonecsVttt|_t|_fddtjdD|_ | dS)Ncsg|]}jqSr+)r)rrkr'r+r,rsz+VitDetBackbone.__init__..r) rr_init_backbonerrHrrrr num_featuresrrr)r,r,rs     zVitDetBackbone.__init__rFcCrrrrr+r+r,rrz#VitDetBackbone.get_input_embeddingsNrErrrc Cs|dur|n|jj}|dur|n|jj}|dur|n|jj}||}|j|d||d}|r1|jn|d}d}t|j|D]\} } | |j vrK|| f7}q=|sf|r[|f|dd} | S|f|dd} | St ||rm|jnd|j dS)a Examples: ```python >>> from transformers import VitDetConfig, VitDetBackbone >>> import torch >>> config = VitDetConfig() >>> model = VitDetBackbone(config) >>> pixel_values = torch.randn(1, 3, 224, 224) >>> with torch.no_grad(): ... outputs = model(pixel_values) >>> feature_maps = outputs.feature_maps >>> list(feature_maps[-1].shape) [1, 768, 14, 14] ```NT)rrrrr+r.) feature_mapsrr) r'r$rrrHrrzip stage_names out_featuresrr) r&rErrrr&rrr/stagerrr+r+r,rIs8    zVitDetBackbone.forward)NNN)rJrKrLrrrr r!rNrrrrIrOr+r+r)r,r+s$  r+)rrr+)rF)4rMcollections.abcrr5typingrrr!r activationsrmodeling_layersrmodeling_outputsrr modeling_utilsr utilsr r utils.backbone_utilsr configuration_vitdetr get_loggerrJloggerModulerr8script_if_tracingr]rorprNrrrrrrrrrrrrrrr+__all__r+r+r+r,sL       X $) ?' Q;3W N