o eidt @sdZddlZddlZddlZddlmZddlmZddl m Z ddl m Z ddl mZdd lmZmZdd lmZdd lmZmZd d lmZeeZGdddejZejjddZ ddZ!GdddejZ"d7dej#de$de%dej#fddZ&GdddejZ'Gd d!d!ejZ(Gd"d#d#ejZ)Gd$d%d%ejZ*d&d'Z+d(d)Z,Gd*d+d+eZ-Gd,d-d-ejZ.eGd.d/d/eZ/eGd0d1d1e/Z0ed2d3Gd4d5d5e e/Z1gd6Z2dS)8zPyTorch ViTDet backbone.N)nn)initialization)ACT2FN) BackboneMixin)GradientCheckpointingLayer)BackboneOutputBaseModelOutput)PreTrainedModel)auto_docstringlogging) VitDetConfigcs>eZdZdZfddZddZdejdejfdd ZZ S) VitDetEmbeddingsz This class turns `pixel_values` of shape `(batch_size, num_channels, height, width)` into the initial `hidden_states` (patch embeddings) to be consumed by a Transformer. cst|j|j}}|j|j}}t|tjj r|n||f}t|tjj r)|n||f}|d|d|d|d}||_ ||_||_||_ |j r]|d}t td||j|_nd|_t j||||d|_dS)Nr r) kernel_sizestride)super__init__pretrain_image_size patch_size num_channels hidden_size isinstance collectionsabcIterable image_size num_patches use_absolute_position_embeddingsr Parametertorchzerosposition_embeddingsConv2d projection)selfconfigrrrrr num_positions __class__h/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/vitdet/modeling_vitdet.pyr)s  zVitDetEmbeddings.__init__cCs|r |ddddf}|jd}tt|}|||kr"tdtjs/||ks/||krOtj j | d||d dddd||fdd d }| ddddS| d||dS) a Calculate absolute positional embeddings. If needed, resize embeddings and remove cls_token dimension for the original embeddings. Args: abs_pos_embeddings (`torch.Tensor`): Absolute positional embeddings with (1, num_position, num_channels). has_cls_token (`bool`): If true, has 1 embedding in abs_pos_embeddings for cls token. height (`int`): Height of input image tokens. width (`int`): Width of input image tokens. Returns: Absolute positional embeddings after processing with shape (1, height, width, num_channels) Nr z5Absolute position embeddings must be a square number.rrbicubicF)sizemode align_corners) shapeintmathsqrt ValueErrorr jit is_tracingr functional interpolatereshapepermute)r%abs_pos_embeddings has_cls_tokenheightwidth num_positionr/new_abs_pos_embeddingsr*r*r+get_absolute_positions?s  z'VitDetEmbeddings.get_absolute_positions pixel_valuesreturncCs|jd}||jkrtd|jd|d||}|jdurA|dddd}|||jd|jd|jd}|dddd}|S) Nr zoMake sure that the channel dimension of the pixel values match with the one set in the configuration. Expected z but got .rr-rT)r2rr6r$r"r<rC)r%rDr embeddingsr*r*r+forwardes"    zVitDetEmbeddings.forward) __name__ __module__ __qualname____doc__rrCr TensorrH __classcell__r*r*r(r+r#s  &rcCstdt||d}|jd|kr3tjj|d|jddddd|dd}|d|dd}n|}t |dddft||d}t |dddft||d}|||dt||d}|| S) a Get relative positional embeddings according to the relative positions of query and key sizes. Args: q_size (`int`): Size of query q. k_size (`int`): Size of key k. rel_pos (`torch.Tensor`): Relative position embeddings (num_embeddings, num_channels). Returns: Extracted positional embeddings according to relative positions. r-r rr,linear)r/r0Ng?) r3maxr2rr9r:r;r<r arangelong)q_sizek_sizerel_pos max_rel_distrel_pos_resizedq_coordsk_coordsrelative_coordsr*r*r+ get_rel_pos{s$$ r[cCs|\}}|\}} t|||} t|| |} |j\} } }|| |||}td|| } td|| }|| |||| | dddddddddf|dddddddddf| |||| }|S)a Calculate decomposed Relative Positional Embeddings as introduced in [MViT2](https://github.com/facebookresearch/mvit/blob/19786631e330df9f3622e5402b4a419a263a2c80/mvit/models/attention.py). Args: attn (`torch.Tensor`): Attention map. queries (`torch.Tensor`): Query q in the attention layer with shape (batch_size, queries_height * queries_width, num_channels). rel_pos_h (`torch.Tensor`): Relative position embeddings (Lh, num_channels) for height axis. rel_pos_w (`torch.Tensor`): Relative position embeddings (Lw, num_channels) for width axis. q_size (`tuple[int]`): Spatial sequence size of query q with (queries_height, queries_width). k_size (`tuple[int]`): Spatial sequence size of key k with (keys_height, keys_width). Returns: attn (Tensor): attention map with added relative positional embeddings. zbhwc,hkc->bhwkzbhwc,wkc->bhwkN)r[r2r;r einsumview)attnqueries rel_pos_h rel_pos_wrSrTqueries_height queries_width keys_height keys_widthrelative_heightrelative_width batch_size_dimr_qrelative_weightr*r*r+!add_decomposed_relative_positionss      rmcs,eZdZdZdfdd Zd ddZZS) VitDetAttentionz=Multi-head Attention block with relative position embeddings.Ncst|j}|j}||_||}|d|_tj||d|jd|_ t|||_ |j |_ |j rSt t d|dd||_t t d|dd||_dSdS)z Args: config (`VitDetConfig`): Model configuration. input_size (`tuple[int]`, *optional*): Input resolution, only required in case relative position embeddings are added. grbiasr-rr N)rrrnum_attention_heads num_headsscalerLinearqkv_biasqkvproj use_relative_position_embeddingsrr r!r`ra)r%r& input_sizerjrrhead_dimr(r*r+rs   $zVitDetAttention.__init__FcCs&|j\}}}}|||||d|jdddddd}|d||j||dd\}} } ||j| dd} |jrMt | ||j |j ||f||f} | j dd} | | }| ||j||d}|ddddd}||||d}||}|r| ||j| jd| jd} || f} | S|f} | S) Nrr,r-rr )rj)r2rvr;rrr<unbindrs transposerxrmr`rasoftmaxr]rw)r% hidden_stateoutput_attentionsrhr?r@rirvr_keysvaluesattention_scoresattention_probsoutputsr*r*r+rHs,,&  zVitDetAttention.forwardNFrIrJrKrLrrHrNr*r*r(r+rnsrnFinput drop_probtrainingrEcCsd|dks|s|Sd|}|jdfd|jd}|tj||j|jd}||||}|S)zc Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks). rr r)r )dtypedevice)r2ndimr randrrfloor_div)rrr keep_probr2 random_tensoroutputr*r*r+ drop_paths rcsTeZdZdZd dedBddffdd Zdejdejfdd Zde fd d Z Z S) VitDetDropPathzXDrop paths (Stochastic Depth) per sample (when applied in main path of residual blocks).NrrEcst||_dSr)rrr)r%rr(r*r+rs  zVitDetDropPath.__init__ hidden_statescCst||j|jSr)rrr)r%rr*r*r+rHszVitDetDropPath.forwardcCs d|jS)Nzp=)rr%r*r*r+ extra_repr"s zVitDetDropPath.extra_reprr) rIrJrKrLfloatrr rMrHstrrrNr*r*r(r+rs rcs*eZdZdZdfdd ZddZZS)VitDetLayerNormaL A LayerNorm variant, popularized by Transformers, that performs point-wise mean and variance normalization over the channel dimension for inputs that have shape (batch_size, channels, height, width). https://github.com/facebookresearch/ConvNeXt/blob/d1fa8f6fef0a165b27399986cc2bdacc92777e40/models/convnext.py#L119 ư>cs@ttt||_tt||_||_ |f|_ dSr) rrrrr onesweightr!rpepsnormalized_shape)r%rrr(r*r+r-s  zVitDetLayerNorm.__init__cCsn|jddd}||djddd}||t||j}|jddddf||jddddf}|S)Nr T)keepdimr-)meanpowr r5rrrp)r%xusr*r*r+rH4s ,zVitDetLayerNorm.forward)rrr*r*r(r+r&srcs(eZdZdZfddZddZZS)VitDetResBottleneckBlockz The standard bottleneck residual block without the last activation layer. It contains 3 conv layers with kernels 1x1, 3x3, 1x1. csttj||ddd|_t||_t|j|_ tj||dddd|_ t||_ t|j|_ tj||ddd|_ t||_dS)ar Args: config (`VitDetConfig`): Model configuration. in_channels (`int`): Number of input channels. out_channels (`int`): Number of output channels. bottleneck_channels (`int`): Number of output channels for the 3x3 "bottleneck" conv layers. r Fror)paddingrpN)rrrr#conv1rnorm1r hidden_actact1conv2norm2act2conv3norm3)r%r& in_channels out_channelsbottleneck_channelsr(r*r+rBs     z!VitDetResBottleneckBlock.__init__cCs&|}|D]}||}q||}|Sr)children)r%routlayerr*r*r+rHZs   z VitDetResBottleneckBlock.forwardrr*r*r(r+r<s rcs@eZdZdededdffdd Zdejdejfdd ZZS) VitDetMlp in_featureshidden_featuresrENcsDtt|||_t|j|_t|||_t |j |_ dSr) rrrrtfc1rractfc2Dropout dropout_probdrop)r%r&rrr(r*r+rds  zVitDetMlp.__init__rcCs6||}||}||}||}||}|Sr)rrrr)r%rr*r*r+rHks     zVitDetMlp.forward) rIrJrKr3rr rMrHrNr*r*r(r+rcsrc Cs|j\}}}}||||}||||}tj|ddd|d|f}||||}} |||||| |||}|ddddddd|||} | || ffS)a  Partition into non-overlapping windows with padding if needed. Args: hidden_state (`torch.Tensor`): Input tokens with [batch_size, height, width, num_channels]. window_size (`int`): Window size. Returns: `tuple(torch.FloatTensor)` comprising various elements: - windows: windows after partition with [batch_size * num_windows, window_size, window_size, num_channels]. - (padded_height, padded_width): padded height and width before partition rr rr-r{r,)r2rr9padr]r< contiguous) r window_sizerhr?r@r pad_height pad_width padded_height padded_widthwindowsr*r*r+window_partitionus$ rc Cs|\}}|\}}|jd||||}||||||||d} | dddddd} | |||d} | ddd|d|ddf} | S) aB Window unpartition into original sequences and removing padding. Args: windows (`torch.Tensor`): Input tokens with [batch_size * num_windows, window_size, window_size, num_channels]. window_size (`int`): Window size. pad_height_width (`tuple[int]`): Padded height and width (padded_height, padded_width). height_width (`tuple[int]`): Original height and width before padding. Returns: hidden_state: unpartitioned sequences with [batch_size, height, width, num_channels]. rr,r rr-r{rN)r2r]r<r) rrpad_height_width height_widthrrr?r@rhrr*r*r+window_unpartitions$rc sleZdZdZ ddededededd f fd d Z dd e j d ede e j e j fe e j BfddZ Z S) VitDetLayerzCThis corresponds to the Block class in the original implementation.rFr&drop_path_rateruse_residual_blockrENc st|j}|j}t|ttfr|n||f}|j}t|ttfr$|n||f}|d|d|d|df}tj ||j d|_ t ||dkrI|n||fd|_ |dkrXt|nt|_tj ||j d|_t||t||jd|_||_||_|jrt||||dd|_dSdS) Nrr )r)ryr)r&rrr-)r&rrr)rrrrrlisttuplerr LayerNormlayer_norm_epsrrn attentionrIdentityrrrr3 mlp_ratiomlprrrresidual) r%r&rrrrjrrryr(r*r+rs0   zVitDetLayer.__init__rrc Cs|dddd}|}||}|jdkr'|jd|jd}}t||j\}}|j||d}|d}|dd}|jdkrGt||j|||f}|||}|||| |}|dddd}|j rk| |}|f|}|S)Nrr-rr )r) r<rrr2rrrrrrrr) r%rrshortcutr?r@rself_attention_outputsrr*r*r+rHs*      zVitDetLayer.forward)rrFr)rIrJrKrLrrr3boolrr rMrrHrNr*r*r(r+rs,&rc sReZdZdeddffdd Z   ddejd ed ed edee Bf d d Z Z S) VitDetEncoderr&rENc st||_|j}ddtjd|j|ddD}g}t|D]}|t |||||j vr1|j nd||j vdq t ||_d|_dS)NcSsg|]}|qSr*)item).0rr*r*r+ sz*VitDetEncoder.__init__..rcpu)r)rrrF)rrr&num_hidden_layersr linspacerrangeappendrwindow_block_indicesrresidual_block_indicesr ModuleListrgradient_checkpointing)r%r&depthrlayersir(r*r+rs   zVitDetEncoder.__init__FTrroutput_hidden_states return_dictc Cs|rdnd}|r dnd}t|jD]\}}|r||f}|||} | d}|r.|| df}q|r6||f}|sDtdd|||fDSt|||dS)Nr*rr css|] }|dur|VqdSrr*)rvr*r*r+ 4sz(VitDetEncoder.forward..last_hidden_stater attentions) enumeraterrr ) r%rrrrall_hidden_statesall_self_attentionsr layer_module layer_outputsr*r*r+rHs&     zVitDetEncoder.forward)FFT) rIrJrKrrr rMrrr rHrNr*r*r(r+rs rc@sReZdZUeed<dZdZdZdZgZ e de j e jBe jBddfd d ZdS) VitDetPreTrainedModelr&vitdetrD)imageTmodulerENcCslt|tjtjfr#tj|jd|jjd|j dur!t |j dSdSt|tj r7t |j t |jdSt|t rItj|jd|jjddSt|trj|jjrjtj|jd|jjdtj|jd|jjddSt|tr|j|j|jfD]}tj|jddd|j durt|j dqw|j|jfD]}t |jt |j qt |jjt |jj dSdS)zInitialize the weightsr)rstdNfan_outrelu)r0 nonlinearityr)rrrtr#init trunc_normal_rr&initializer_rangerpzeros_rones_rr"rnrxr`rarrrrkaiming_normal_ constant_rrr)r%rrr*r*r+ _init_weightsEs2       z#VitDetPreTrainedModel._init_weights)rIrJrKr__annotations__base_model_prefixmain_input_nameinput_modalitiessupports_gradient_checkpointing_no_split_modulesr no_gradrrtr#rr r*r*r*r+r<s &rc sreZdZdeffdd ZdefddZe    ddej dBd e dBd e dBd e dBde e Bf d d Z ZS) VitDetModelr&cs2t|||_t||_t||_|dSr)rrr&rrGrencoder post_initr%r&r(r*r+rcs    zVitDetModel.__init__rEcC|jjSrrGr$rr*r*r+get_input_embeddingsmz VitDetModel.get_input_embeddingsNrDrrrc Ks|dur|n|jj}|dur|n|jj}|dur|n|jj}|dur&td||}|j||||d}|d}|sC|f|ddSt||j|j dS)a Examples: ```python >>> from transformers import VitDetConfig, VitDetModel >>> import torch >>> config = VitDetConfig() >>> model = VitDetModel(config) >>> pixel_values = torch.randn(1, 3, 224, 224) >>> with torch.no_grad(): ... outputs = model(pixel_values) >>> last_hidden_states = outputs.last_hidden_state >>> list(last_hidden_states.shape) [1, 768, 14, 14] ```Nz You have to specify pixel_values)rrrrr r) r&rruse_return_dictr6rGrr rr) r%rDrrrkwargsembedding_outputencoder_outputssequence_outputr*r*r+rHps* zVitDetModel.forward)NNNN)rIrJrKrrrrr r rMrrr rHrNr*r*r(r+ras& rzF ViTDet backbone, to be used with frameworks like Mask R-CNN. ) custom_introc sbeZdZfddZdefddZe   d dejde dBd e dBd e dBde f d d Z Z S)VitDetBackbonecsJtt|_t|_fddtjdD|_| dS)Ncsg|]}jqSr*)r)rrir&r*r+rsz+VitDetBackbone.__init__..r ) rrrrGrrrr num_featuresrrr(r"r+rs    zVitDetBackbone.__init__rEcCrrrrr*r*r+rrz#VitDetBackbone.get_input_embeddingsNrDrrrc Ks|dur|n|jj}|dur|n|jj}|dur|n|jj}||}|j|d||d}|r1|jn|d}d} t|j|D]\} } | |j vrK| | f7} q=|sf|r[| f|dd} | S| f|dd} | St | |rm|jnd|j dS)a Examples: ```python >>> from transformers import VitDetConfig, VitDetBackbone >>> import torch >>> config = VitDetConfig() >>> model = VitDetBackbone(config) >>> pixel_values = torch.randn(1, 3, 224, 224) >>> with torch.no_grad(): ... outputs = model(pixel_values) >>> feature_maps = outputs.feature_maps >>> list(feature_maps[-1].shape) [1, 768, 14, 14] ```NT)rrrr r*r-) feature_mapsrr) r&rrrrGrrzip stage_names out_featuresrr) r%rDrrrrrrrr$stagerrr*r*r+rHs8    zVitDetBackbone.forward)NNN) rIrJrKrrrr r rMrrrHrNr*r*r(r+r!s$  r!)rrr!)rF)3rLcollections.abcrr4r rrr activationsrbackbone_utilsrmodeling_layersrmodeling_outputsrr modeling_utilsr utilsr r configuration_vitdetr get_loggerrIloggerModulerr7script_if_tracingr[rmrnrMrrrrrrrrrrrrrr!__all__r*r*r*r+sJ         X $) ?' P8$G N