o i @s dZddlZddlZddlZddlmZddlmZm Z ddl Z ddl m Z m Z ddl mZddlmZdd lmZdd lmZmZmZmZmZmZdd lmZdd lmZmZmZdd l m!Z!m"Z"m#Z#ddl$m%Z%ddl&m'Z'e"(e)Z*ee!ddGdddeZ+dRde j de,de-de j fddZ.Gddde j/Z0Gdd d e j/Z1Gd!d"d"e j/Z2Gd#d$d$e j/Z3Gd%d&d&e3Z4Gd'd(d(e j/Z5e3e4d)Z6Gd*d+d+e j/Z7Gd,d-d-e j/Z8Gd.d/d/e j/Z9Gd0d1d1eZ:Gd2d3d3e j/Z;Gd4d5d5e j/ZGd:d;d;e j/Z?e!dd>e=Z@e!d?dGd@dAdAe=ZAGdBdCdCe j/ZBGdDdEdEe j/ZCGdFdGdGe j/ZDGdHdIdIe j/ZEGdJdKdKe j/ZFe!GdLdMdMe=ZGe!dNdGdOdPdPe=e%ZHgdQZIdS)SzPyTorch BEiT model.N) dataclass)OptionalUnion)Tensornn)CrossEntropyLoss)ACT2FN)GradientCheckpointingLayer)BackboneOutputBaseModelOutputBaseModelOutputWithPoolingImageClassifierOutputMaskedLMOutputSemanticSegmenterOutput)PreTrainedModel)#compile_compatible_method_lru_cache find_pruneable_heads_and_indicesprune_linear_layer)auto_docstringlogging torch_int) BackboneMixin) BeitConfigz- Class for outputs of [`BeitModel`]. ) custom_introc@seZdZdZdS)BeitModelOutputWithPoolingaF pooler_output (`torch.FloatTensor` of shape `(batch_size, hidden_size)`): Average of the last layer hidden states of the patch tokens (excluding the *[CLS]* token) if *config.use_mean_pooling* is set to True. If set to False, then the final hidden state of the *[CLS]* token will be returned. N)__name__ __module__ __qualname____doc__r!r!Z/home/ubuntu/.local/lib/python3.10/site-packages/transformers/models/beit/modeling_beit.pyr/srFinput drop_probtrainingreturncCsd|dks|s|Sd|}|jdfd|jd}|tj||j|jd}||||}|S)aF Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks). Comment by Ross Wightman: This is the same as the DropConnect impl I created for EfficientNet, etc networks, however, the original name is misleading as 'Drop Connect' is a different form of dropout in a separate paper... See discussion: https://github.com/tensorflow/tpu/issues/494#issuecomment-532968956 ... I've opted for changing the layer and argument names to 'drop path' rather than mix DropConnect as a layer name and use 'survival rate' as the argument. r#rrr)dtypedevice)shapendimtorchrandr)r*floor_div)r$r%r& keep_probr+ random_tensoroutputr!r!r" drop_path>s r4csTeZdZdZd deeddffdd Zdejdejfdd Z de fd d Z Z S) BeitDropPathzXDrop paths (Stochastic Depth) per sample (when applied in main path of residual blocks).Nr%r'cst||_dSN)super__init__r%)selfr% __class__r!r"r8Us  zBeitDropPath.__init__ hidden_statescCst||j|jSr6)r4r%r&r9r<r!r!r"forwardYszBeitDropPath.forwardcCs d|jS)Nzp=)r%r9r!r!r" extra_repr\s zBeitDropPath.extra_reprr6) rrrr rfloatr8r-rr>strr@ __classcell__r!r!r:r"r5Rs r5c sveZdZdZdeddffdd Zdejded edejfd d Z  dd ejd e ej de e dejfddZ ZS)BeitEmbeddingszc Construct the CLS token, position and patch embeddings. Optionally, also the mask token. configr'Ncstttdd|j|_|jr!ttdd|j|_ nd|_ t ||_ |j |_ t |jtjjr8|jn|j|jf|_|j j}|jrUttd|d|j|_nd|_t|j|_dS)Nr)r7r8r Parameterr-zeros hidden_size cls_tokenuse_mask_token mask_tokenBeitPatchEmbeddingspatch_embeddings patch_size isinstance image_size collectionsabcIterable num_patches use_absolute_position_embeddingsposition_embeddingsDropouthidden_dropout_probdropout)r9rErTr:r!r"r8hs   zBeitEmbeddings.__init__ embeddingsheightwidthc Cs|jdd}|jjdd}tjs||kr||kr|jS|jddddf}|jddddf}|jd}||j} ||j} t|d} |d| | |}|dddd}t j j || | fdd d }|dddd dd|}tj ||fdd S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rNg?rrbicubicFsizemode align_cornersdim)r+rVr-jit is_tracingrNrreshapepermuter functional interpolateviewcat) r9rZr[r\rT num_positionsclass_pos_embedpatch_pos_embedre new_height new_widthsqrt_num_positionsr!r!r"interpolate_pos_encodings(     z'BeitEmbeddings.interpolate_pos_encoding pixel_valuesbool_masked_posrtcCs|jdur|durtd|j\}}}}||\}\}} |\} } }|durC|j| | d} |d | } |d| | | }|j | dd}t j ||fdd}|jdurb|| |||}||}||| ffS)Nz`interpolate_pos_encoding` argument has no effect for BEiTEmbeddings, embeddings are always interpolated to the input image size. The argument will be removed in transformers v4.51.0.r]rrd)rVwarningswarnr+rMrarKexpand unsqueezetype_asrIr-rmrtrY)r9rurvrt_r[r\rZ patch_height patch_width batch_sizeseq_len mask_tokensw cls_tokensr!r!r"r>s"   zBeitEmbeddings.forwardNN)rrrr rr8r-rintrtr BoolTensorboolr>rCr!r!r:r"rDbs+rDcs6eZdZdZfddZdejdejfddZZS)rLz This class turns `pixel_values` of shape `(batch_size, num_channels, height, width)` into the initial `hidden_states` (patch embeddings) of shape `(batch_size, seq_length, hidden_size)` to be consumed by a Transformer. cst|j|j}}|j|j}}t|tjj r|n||f}t|tjj r)|n||f}|d|d|d|d}|d|d|d|df}||_||_||_||_ ||_ t j ||||d|_dS)Nrr kernel_sizestride)r7r8rPrN num_channelsrHrOrQrRrSrT patch_shaperConv2d projection)r9rErPrNrrHrTrr:r!r"r8s   zBeitPatchEmbeddings.__init__rur'c Cs^|j\}}}}||jkrtd||}|jd|jd}}|ddd}|||ffS)NzeMake sure that the channel dimension of the pixel values match with the one set in the configuration.r^rr)r+r ValueErrorrflatten transpose) r9rurrr[r\rZr}r~r!r!r"r>s   zBeitPatchEmbeddings.forward) rrrr r8r-rr>rCr!r!r:r"rLs rLcseZdZddedeeddffdd Z     ddejd eejd e d eejd e d eee de eejeejejfffddZ Z S)BeitSelfAttentionNrE window_sizer'cst||_|j|jdkr"t|ds"td|jd|jd|j|_t|j|j|_|j|j|_ t |j|j |_ t j |j|j dd|_ t |j|j |_t |j|_t||_|jrkt||d|_dSdS) Nrembedding_sizezThe hidden size z4 is not a multiple of the number of attention heads .F)biasr)r7r8rErHnum_attention_headshasattrrrattention_head_size all_head_sizerLinearquerykeyvaluerWattention_probs_dropout_probrYrhas_relative_position_biasBeitRelativePositionBiasrelative_position_biasr9rErr:r!r"r8s&   zBeitSelfAttention.__init__Fr< head_maskoutput_attentionsrrt resolutioncCsl|j\}}} |||d|j|jdd} |||d|j|jdd} |||d|j|jdd} t | | dd} | t |j} |j rk|\}}||j j||j jf}| |j|||jdd} |durs| |} tjj| dd}||}|dur||}t || }|dddd}|dd|jf}|j|}|r||f}|S|f}|S) Nr]rr^dim_sizerdrr)r+rrlrrrrrr-matmulmathsqrtrrErNrrrjsoftmaxrYri contiguousrar)r9r<rrrrtrr seq_lengthr| query_layer key_layer value_layerattention_scoresr[r\rattention_probs context_layernew_context_layer_shapeoutputsr!r!r"r>sF      zBeitSelfAttention.forwardr6NFNFN)rrrrrtupler8r-rrrrr>rCr!r!r:r"rs,  rcsveZdZ     d dejdeejdedeejdedeeed e eejeejejffffd d Z Z S) BeitSdpaSelfAttentionNFr<rrrrtrr'c sv|s|durtdtj||||||dS|j\}}} |||d|j|j dd} | ||d|j|j dd} | ||d|j|j dd} d} |j rp|\}}||j j||j jf}|j|||jdd} |dur| dur{|} n| |7} dt|j}tjjj| | | | |jr|j jndd|d }|d ddd }|dd |jf}|j|}|dfS) Na`BeitSdpaSelfAttention` is used but `torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True` or `head_mask`. Falling back to the manual attention implementation, but specifying the manual implementation will be required from Transformers version v5.0.0 onwards. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.)r<rrrrtrr]rr^rr#F) attn_mask dropout_p is_causalscalerrr)logger warning_oncer7r>r+rrlrrrrrrrErNrrrr-rrjscaled_dot_product_attentionr&rrirrar)r9r<rrrrtrrrr|rrr attn_biasr[r\rscalingrrr:r!r"r>Hsd     zBeitSdpaSelfAttention.forwardr) rrrr-rrrrrrr>rCr!r!r:r"rGs* rcsHeZdZdZdeddffdd Zd dejdejdejfd d ZZ S) BeitSelfOutputz The residual connection is defined in BeitLayer instead of here (as is the case with other models), due to the layernorm applied before each block. rEr'Ncs.tt|j|j|_t|j|_dSr6) r7r8rrrHdenserWrXrYr9rEr:r!r"r8 zBeitSelfOutput.__init__r< input_tensorcC||}||}|Sr6rrY)r9r<rgammar!r!r"r>  zBeitSelfOutput.forwardr6) rrrr rr8r-rr>rCr!r!r:r"rs&r)eagersdpacseZdZddedeeddffdd ZddZ   dd ej d eej d e d eej de deee de eej eej ej fffddZ ZS) BeitAttentionNrErr'cs4tt|j||d|_t||_t|_dS)Nr) r7r8BEIT_SELF_ATTENTION_CLASSES_attn_implementation attentionrr3set pruned_headsrr:r!r"r8s   zBeitAttention.__init__cCst|dkrdSt||jj|jj|j\}}t|jj||j_t|jj||j_t|jj ||j_ t|j j |dd|j _ |jjt||j_|jj|jj|j_ |j ||_dS)Nrrrd)lenrrrrrrrrrr3rrunion)r9headsindexr!r!r" prune_headss zBeitAttention.prune_headsFr<rrrrtrc Cs:|||||||}||d|}|f|dd} | S)Nrr)rr3) r9r<rrrrtr self_outputsattention_outputrr!r!r"r>s  zBeitAttention.forwardr6r)rrrrrrr8rr-rrrrr>rCr!r!r:r"rs.  rc<eZdZdeddffdd ZdejdejfddZZS) BeitIntermediaterEr'NcsDtt|j|j|_t|jt rt |j|_ dS|j|_ dSr6) r7r8rrrHintermediate_sizerrO hidden_actrBr intermediate_act_fnrr:r!r"r8s   zBeitIntermediate.__init__r<cCrr6)rrr=r!r!r"r>rzBeitIntermediate.forward rrrrr8r-rr>rCr!r!r:r"rsrcr) BeitOutputrEr'Ncs.tt|j|j|_t|j|_ dSr6) r7r8rrrrHrrWrXrYrr:r!r"r8rzBeitOutput.__init__r<cCrr6rr=r!r!r"r>rzBeitOutput.forwardrr!r!r:r"rsrcseZdZdZddedeededdffdd Z   dd e j d ee j d e dee j de deee e fde ee j ee j e j fffddZZS) BeitLayerz?This corresponds to the Block class in the timm implementation.Nr#rErdrop_path_rater'cst|j|_d|_t||d|_t||_t||_ t j |j |j d|_|dkr/t|nt |_t j |j |j d|_|j}|dkrct j|t|j dd|_t j|t|j dd|_dSd\|_|_dS) Nrrepsr#rT) requires_gradr)r7r8chunk_size_feed_forward seq_len_dimrrr intermediaterr3r LayerNormrHlayer_norm_epslayernorm_beforer5Identityr4layernorm_afterlayer_scale_init_valuerFr-oneslambda_1lambda_2)r9rErr init_valuesr:r!r"r8s    zBeitLayer.__init__Fr<rrrrtrc Cs|j|||||||d}|d}|dd} |jdur"|j|}|||}||} || } || } |jdurB|j| } || |} | f| } | S)N)rrrtrrr)rrrr4rrr3r) r9r<rrrrtrself_attention_outputsrr layer_outputr!r!r"r>s*          zBeitLayer.forward)Nr#r)rrrr rrrrAr8r-rrrrr>rCr!r!r:r"rs.$rcsfeZdZdededdffdd Zedddeeefdej fd d Z dd e dej fd dZ Z S)rrErr'NcsRt||_d|ddd|ddd|_tt|j|j|_ dS)Nr^rrr) r7r8rnum_relative_distancerrFr-rGrrelative_position_bias_tablerr:r!r"r84s & z!BeitRelativePositionBias.__init__ )maxsizec Csd|ddd|ddd}|d|d}tjt|dt|ddd}t|}t|d}|dddddf|dddddf}|ddd}|dddddf|dd7<|dddddf|dd7<|dddddfd|dd9<tj|dfd|jd}| d |ddddf<|d|dddf<|d|dddf<|d|d <|S) z This method creates the relative position index, modified to support arbitrary window sizes, as introduced in [MiDaS v3.1](https://huggingface.co/papers/2307.14460). r^rrrij)indexingN)rar)r])rr) r-meshgridarangestackrrirrGr)sum) r9rr window_areagridcoordscoords_flattenrelative_coordsrelative_position_indexr!r!r" generate_relative_position_index=s $$  ,&&* z9BeitRelativePositionBias.generate_relative_position_indexFrtcCsfd|jdd}d|jdd}d|dd}d|dd}|j}|j} ||d} |d| d} | d||ddddd} tjj| t|t|fdd} | dddd| dd} t | || ddg} | |}| | d}| |d|dd|d|ddd}|ddd }|rtjj|d||fdd d d}|dS) zu Modification of timm.models.beit.py: Attention._get_rel_pos_bias to support arbitrary window sizes. r^rrrNr]bilinear)rarbFr`)rrrrhrirrjrkrr-rmrrlrrzsqueeze)r9rrtr old_height old_widthrqrr old_relative_position_bias_tableold_num_relative_distancenew_num_relative_distance old_sub_table new_sub_table new_relative_position_bias_tablerrr!r!r"r>Vs@  & z BeitRelativePositionBias.forward)FN)rrrrrr8rrr-rrrr>rCr!r!r:r"r3s  rcseZdZddedeeddffdd Z      dd ejd eejd e d e d e deee e fde de ee ffddZ ZS) BeitEncoderNrErr'cszt|_j|_|jrtd|_ddtjdj j ddDt fddt j D|_d|_dS) NrcSsg|]}|qSr!)item.0xr!r!r" sz(BeitEncoder.__init__..rcpu)r*cs(g|]}tjr nd|dqS)N)rr)ruse_relative_position_biasrirEdprrr!r"rs F)r7r8rE!use_shared_relative_position_biasrrrr-linspacernum_hidden_layersr ModuleListrangelayergradient_checkpointingrr:rr"r8s   zBeitEncoder.__init__FTr<rroutput_hidden_statesrtr return_dictc Cs|rdnd}|r dnd} t|jD]M\} } |r||f}|jr;|\} } | |jj| |jjf}|j|||jdd}nd}|durE|| nd}| ||||||d}|d}|r^| |df} q|rf||f}|sttdd||| fDSt||| dS) Nr!r)rtr)rrrrtrrcss|] }|dur|VqdSr6r!)rvr!r!r" sz&BeitEncoder.forward..)last_hidden_stater< attentions) enumerater$rrErNrr+rr )r9r<rrr&rtrr'all_hidden_statesall_self_attentionsr layer_moduler[r\rrlayer_head_mask layer_outputsr!r!r"r>sD     zBeitEncoder.forwardr6)NFFFNT)rrrrrrr8r-rrrrr r>rCr!r!r:r"rs2  rc@s:eZdZUeed<dZdZdZdgZdgZ dZ ddZ d S) BeitPreTrainedModelrEbeitruTrz.*relative_position_index.*cCsbt|tjtjtjfr%|jjjd|jj d|j dur#|j j dSdSt|tj rH|jjjd|jj d|j durF|jj|j  dSdSt|tjr]|j j |jjddSt|tr|jj |jdurs|jj |jdur|jj dSdSt|tr|jj dSt|tr|jdur|jj|jj|jj|jjdSdSdS)zInitialize the weightsr#)meanstdNg?)rOrrrConvTranspose2dweightdatanormal_rEinitializer_rangerzero_ Embedding padding_idxrfill_rDrIrKrVrrrrrr)r9moduler!r!r" _init_weightss8             z!BeitPreTrainedModel._init_weightsN) rrrr__annotations__base_model_prefixmain_input_namesupports_gradient_checkpointing_no_split_modules"_keys_to_ignore_on_load_unexpected_supports_sdpar@r!r!r!r"r2s  r2cseZdZddededdffdd Zdd Zd d Ze     dd e j de e j de e j de ede edede ede eeffddZZS) BeitModelTrEadd_pooling_layerr'Ncspt|||_t||_t||jjjd|_|j rt nt j |j |jd|_|r/t|nd|_|dS)zv add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer rrN)r7r8rErDrZrrMrencoderuse_mean_poolingrrrrHr layernorm BeitPoolerpooler post_init)r9rErIr:r!r"r8s   zBeitModel.__init__cC|jjSr6rZrMr?r!r!r"get_input_embeddingszBeitModel.get_input_embeddingscCs*|D]\}}|jj|j|qdS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrJr$rr)r9heads_to_pruner$rr!r!r" _prune_headsszBeitModel._prune_headsFrurvrrr&rtr'c Cs|dur|n|jj}|dur|n|jj}|dur|n|jj}|||jj}|j||d\}} |jdd} |j||||| ||d} | d} | | } |j durU| | nd} |sl| dura| | fn| f}|| ddSt | | | j | j dS)z bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, num_patches)`, *optional*): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). N)rvr^)rrr&rr'rtrr)r* pooler_outputr<r+)rErr&use_return_dict get_head_maskr!rZr+rJrLrNrr<r+)r9rurvrrr&rtr'embedding_outputr|rencoder_outputssequence_output pooled_output head_outputsr!r!r"r>s8 zBeitModel.forward)T)NNNNFN)rrrrrr8rRrVrr-rrrrrrr>rCr!r!r:r"rHs8 rHcr) rMrEr'Ncs2t|jrtj|j|jd|_dSd|_dS)Nr)r7r8rKrrrHrrLrr:r!r"r8Rs zBeitPooler.__init__r<cCsL|jdur|ddddddf}||d}|S|dddf}|S)Nrr)rLr4)r9r< patch_tokensr]r!r!r"r>Xs zBeitPooler.forwardrr!r!r:r"rMQsrMa Beit Model transformer with a 'language' modeling head on top. BEiT does masked image modeling by predicting visual tokens of a Vector-Quantize Variational Autoencoder (VQ-VAE), whereas other vision models like ViT and DeiT predict RGB pixel values. As a result, this class is incompatible with [`AutoModelForMaskedImageModeling`], so you will need to use [`BeitForMaskedImageModeling`] directly if you wish to do masked image modeling with BEiT. cseZdZdeddffdd ZddZe        dd eej d eej d eej d eej d ee dee de dee de e effddZZS)BeitForMaskedImageModelingrEr'NcsTt||j|_t|dd|_tj|j|jd|_ t |j|j |_ | dS)NFrIr)r7r8 num_labelsrHr3rrrHrrLr vocab_sizelm_headrOrr:r!r"r8ms  z#BeitForMaskedImageModeling.__init__cCsdSr6r!r?r!r!r"get_output_embeddingszsz0BeitForMaskedImageModeling.get_output_embeddingsFrurvrlabelsrr&rtr'c  Cs|dur|n|jj}|j|||||||d} | d} || } || ddddf} d} |dur 1` a classification loss is computed (Cross-Entropy). Examples: ```python >>> from transformers import AutoImageProcessor, BeitForMaskedImageModeling >>> import torch >>> from PIL import Image >>> import requests >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> image_processor = AutoImageProcessor.from_pretrained("microsoft/beit-base-patch16-224-pt22k") >>> model = BeitForMaskedImageModeling.from_pretrained("microsoft/beit-base-patch16-224-pt22k") >>> num_patches = (model.config.image_size // model.config.patch_size) ** 2 >>> pixel_values = image_processor(images=image, return_tensors="pt").pixel_values >>> # create random boolean mask of shape (batch_size, num_patches) >>> bool_masked_pos = torch.randint(low=0, high=2, size=(1, num_patches)).bool() >>> outputs = model(pixel_values, bool_masked_pos=bool_masked_pos) >>> loss, logits = outputs.loss, outputs.logits >>> list(logits.shape) [1, 196, 8192] ```N)rvrrr&rtr'rrlosslogitsr<r+) rErXr3rLrdrrr<r+)r9rurvrrfrr&rtr'rr\prediction_scoresmasked_lm_lossloss_fctr3r!r!r"r>}s4, z"BeitForMaskedImageModeling.forward)NNNNNNFN)rrrrr8rerrr-rrrrrrr>rCr!r!r:r"r`ds>    r`z Beit Model transformer with an image classification head on top (a linear layer on top of the average of the final hidden states of the patch tokens) e.g. for ImageNet. cseZdZdeddffdd Ze       ddeejdeejd eejd ee d ee d e d ee de e e ffddZ ZS)BeitForImageClassificationrEr'NcsRt||j|_t|dd|_|jdkrt|j|jnt|_ | dS)NTrar) r7r8rbrHr3rrrHr classifierrOrr:r!r"r8s $ z#BeitForImageClassification.__init__Frurrfrr&rtr'c Cs|dur|n|jj}|j||||||d}|r|jn|d} || } d} |dur1||| |j} |sG| f|dd} | durE| f| S| St| | |j|jdS)a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the image classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Nrrr&rtr'rr^rg) rErXr3rWrn loss_functionrr<r+) r9rurrfrr&rtr'rr]rirhr3r!r!r"r>s. z"BeitForImageClassification.forwardNNNNNFN)rrrrr8rrr-rrrrrr>rCr!r!r:r"rms6  rmcseZdZdZ   ddededeeeeeffdeeeeefefd ed eeeeeffd d ffd d Z de j d e j fddZ Z S)BeitConvModuleaD A convolutional block that bundles conv/norm/activation layers. This block simplifies the usage of convolution layers, which are commonly used with a norm layer (e.g., BatchNorm) and activation layer (e.g., ReLU). Based on OpenMMLab's implementation, found in https://github.com/open-mmlab/mmsegmentation. rFr in_channels out_channelsrpaddingrdilationr'Ncs<ttj||||||d|_t||_t|_dS)N)rsrtrrurrv) r7r8rrconv BatchNorm2dbnReLU activation)r9rsrtrrurrvr:r!r"r8s  zBeitConvModule.__init__r$cCs"||}||}||}|Sr6)rwryr{)r9r$r3r!r!r"r>)s   zBeitConvModule.forward)rFr)rrrr rrrrBrr8r-rr>rCr!r!r:r"rr s* rrcsDeZdZdedededdffdd Zdejdejfd d ZZS) BeitPyramidPoolingBlock pool_scalerschannelsr'NcsLtt|t||ddg|_t|jD] \}}|t||qdS)Nrr) r7r8rAdaptiveAvgPool2drrlayersr, add_modulerB)r9r}rsr~rr$r:r!r"r82s  z BeitPyramidPoolingBlock.__init__r$cCs|}|jD]}||}q|Sr6)r)r9r$ hidden_stater$r!r!r"r>;s  zBeitPyramidPoolingBlock.forward) rrrrr8r-rr>rCr!r!r:r"r|1s r|c sXeZdZdZdeedfdedededdf fd d Zd ej de ej fd d Z Z S)BeitPyramidPoolingModulea Pyramid Pooling Module (PPM) used in PSPNet. Args: pool_scales (tuple[int]): Pooling scales used in Pooling Pyramid Module. in_channels (int): Input channels. channels (int): Channels after modules, before conv_seg. align_corners (bool): align_corners argument of F.interpolate. Based on OpenMMLab's implementation, found in https://github.com/open-mmlab/mmsegmentation. pool_scales.rsr~rcr'Ncsht||_||_||_||_g|_t|D]\}}t|||d}|j || t ||qdS)N)r}rsr~) r7r8rrcrsr~blocksr,r|appendrrB)r9rrsr~rcrr}blockr:r!r"r8Ps  z!BeitPyramidPoolingModule.__init__rcCsHg}|jD]}||}tjj||ddd|jd}||q|S)Nr^r r`)rrrjrkrarcr)r9rppm_outsppmppm_outupsampled_ppm_outr!r!r"r>\s  z BeitPyramidPoolingModule.forward) rrrr rrrr8r-rlistr>rCr!r!r:r"rBs* " rcsHeZdZdZdeddffdd ZddZd ejdejfd d Z Z S) BeitUperHeadz Unified Perceptual Parsing for Scene Understanding. This head is the implementation of [UPerNet](https://huggingface.co/papers/1807.10221). Based on OpenMMLab's implementation, found in https://github.com/open-mmlab/mmsegmentation. rEr'Ncst|j|_|jgd|_|j|_d|_tj|j|j dd|_ t |j|jd|j|jd|_ t |jdt|j|j|jddd|_t|_t|_|jddD] }t ||jdd}t |j|jddd}|j||j|qWt t|j|j|jddd|_dS) NFrrr])rcrrru)r7r8rrHrsr~rcrrrbrnr psp_modulesrrr bottleneckr" lateral_convs fpn_convsrfpn_bottleneck)r9rErsl_convfpn_convr:r!r"r8os>     zBeitUperHead.__init__cCs:|d}|g}|||tj|dd}||}|S)Nr]rrd)extendrr-rmr)r9inputsrpsp_outsr3r!r!r" psp_forwards  zBeitUperHead.psp_forwardencoder_hidden_statescsfddtjDt}t|dddD]$}|djdd}|dtjj ||dj d|d<q fd dt|dD}|dt|dddD]}tjj |||djdddj d||<qbt j |dd } |}|}|S) Ncsg|] \}}||qSr!r!)rr lateral_conv)rr!r"rsz(BeitUperHead.forward..rrr]r^r r`csg|] }j||qSr!)rr)lateralsr9r!r"rsrd)r,rrrrr#r+rrjrkrcr-rmrrn)r9rused_backbone_levelsr prev_shapefpn_outsr3r!)rrr9r"r>s$   zBeitUperHead.forward) rrrr rr8rr-rr>rCr!r!r:r"rgs & rc s`eZdZdZ ddedededeeeeeffd d f fd d Zd e j d e j fddZ Z S) BeitFCNHeada Fully Convolution Networks for Semantic Segmentation. This head is implemented of [FCNNet](https://huggingface.co/papers/1411.4038>). Args: config (BeitConfig): Configuration. in_channels kernel_size (int): The kernel size for convs in the head. Default: 3. dilation (int): The dilation rate for convs in the head. Default: 1. Based on OpenMMLab's implementation, found in https://github.com/open-mmlab/mmsegmentation. r^rrrEin_indexrrvr'Nc st|j|_|j|_|j|_|j|_ ||_ |d|}g}| t |j|j|||dt |jdD]}| t |j|j|||dq5|jdkrQt|_ntj||_|j rjt |j|j|j||dd|_tj|j|jdd|_dS)Nr^)rrurvrrrr)r7r8rHrsauxiliary_channelsr~auxiliary_num_convs num_convsauxiliary_concat_input concat_inputrrrrr#rrconvs Sequentialconv_catrrbrn)r9rErrrv conv_paddingrrr:r!r"r8s6     zBeitFCNHead.__init__rcCs@||j}||}|jr|tj||gdd}||}|S)Nrrd)rrrrr-rmrn)r9rr<r3r!r!r"r>s   zBeitFCNHead.forward)r^rr) rrrr rrrrr8r-rr>rCr!r!r:r"rs"rcseZdZdeddffdd ZddZe       dd eej d eej d eej d ee d ee de dee de e e ffddZZS)BeitForSemanticSegmentationrEr'Nc st||j|_t|dd|_t|jjdkrtdt t j |j |j dddt |j t t j |j |j ddd|_t t j |j |j ddd|_t |_t jddd|_t||_|jrft|nd|_|dS)NFrarzBeitForSemanticSegmentation requires config.out_indices to be a list of 4 integers, specifying which features to use from the backbone. One can use [3, 5, 7, 11] in case of a base-sized architecture.r^r)r7r8rbrHr3rrE out_indicesrrrr6rHrxGELUfpn1fpn2rfpn3 MaxPool2dfpn4r decode_headuse_auxiliary_headrauxiliary_headrOrr:r!r"r8s*     z$BeitForSemanticSegmentation.__init__c Cstjj||jddddd}|dur"tjj||jddddd}t|jjd}|||}|}|durA|||} ||jj| 7}|S)Nrr Fr`) ignore_index)rrjrkr+rrEsemantic_loss_ignore_indexauxiliary_loss_weight) r9riauxiliary_logitsrfupsampled_logitsupsampled_auxiliary_logitsrl main_lossrhauxiliary_lossr!r!r" compute_losss  z(BeitForSemanticSegmentation.compute_lossFrurrfrr&rtr'cs|dur|njj}|dur|njj}|dur"jjdkr"tdj|||d||d}|r2|jn|d} fddt| D} |jdjj jj fd d| D} j j j jg} tt| D] } | | | | | | <qh| } d}jdur| }d}|dur| ||}|s|r| f|dd}n | f|d d}|dur|f|S|St|| |r|jnd|jd S) aD labels (`torch.LongTensor` of shape `(batch_size, height, width)`, *optional*): Ground truth semantic segmentation maps for computing the loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels > 1`, a classification loss is computed (Cross-Entropy). Examples: ```python >>> from transformers import AutoImageProcessor, BeitForSemanticSegmentation >>> from PIL import Image >>> import requests >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> image_processor = AutoImageProcessor.from_pretrained("microsoft/beit-base-finetuned-ade-640-640") >>> model = BeitForSemanticSegmentation.from_pretrained("microsoft/beit-base-finetuned-ade-640-640") >>> inputs = image_processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> # logits are of shape (batch_size, num_labels, height, width) >>> logits = outputs.logits ```Nrz/The number of labels should be greater than oneTrocs$g|]\}}|djjvr|qSr()rEr)ridxfeaturer?r!r"rcs$z7BeitForSemanticSegmentation.forward..rcs<g|]}|ddddddfddddqS)Nrrr^r])rirhr)rpatch_resolutionr!r"rfs0r^rg)rErXr&rbrr3r<r,r+rPrNrrrrr#rrrrrr+)r9rurrfrr&rtr'rrfeaturesopsrrirrhr3r!)rrr9r"r>,sR"       z#BeitForSemanticSegmentation.forwardrq)rrrrr8rrrr-rrrrrr>rCr!r!r:r"rs8  rzM BEiT backbone, to be used with frameworks like DETR and MaskFormer. c sZeZdZfddZddZe   d dedeedeed eed e f d d Z Z S) BeitBackbonec sttfddtjdD|_t|_t|jj j d|_ j rut |jjdkr9tdj}ttj||dddtj|jd ttj||ddd|_ttj||ddd|_t|_tjddd|_|dS) Ncsg|]}jqSr!)rH)rr|rEr!r"rsz)BeitBackbone.__init__..rrrzBeitBackbone requires config.out_indices to be a list of 4 integers, specifying which features to use from the backbone. One can use [3, 5, 7, 11] in case of a base-sized architecture.r^rr)r7r8_init_backboner#r! num_featuresrDrZrrMrrJadd_fpnrrErrrHrrr6rxbatch_norm_epsrrrrrrrrO)r9rErHr:rr"r8s*     zBeitBackbone.__init__cCrPr6rQr?r!r!r"rRrSz!BeitBackbone.get_input_embeddingsNrur&rr'r'cCs|dur|n|jj}|dur|n|jj}|dur|n|jj}|jd}||\}\}}|jdd} |j|d|| |d} |rB| jn| d} d} t|j | D].\} }| |j vr||jj rw|ddddddf}| ddd}| |d||}| |f7} qN|jjr|| d|| d|| d|| d g} t| } |s|r| f| dd}|S| f| dd}|St| |r| jnd| jd S) a: Examples: ```python >>> from transformers import AutoImageProcessor, AutoBackbone >>> import torch >>> from PIL import Image >>> import requests >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> processor = AutoImageProcessor.from_pretrained("microsoft/beit-base-patch16-224") >>> model = AutoBackbone.from_pretrained( ... "microsoft/beit-base-patch16-224", out_features=["stage1", "stage2", "stage3", "stage4"] ... ) >>> inputs = processor(image, return_tensors="pt") >>> outputs = model(**inputs) >>> feature_maps = outputs.feature_maps >>> list(feature_maps[-1].shape) [1, 768, 14, 14] ```Nrr^T)r&rrr'rr!r]r) feature_mapsr<r+)rErXr&rr+rZrJr<zip stage_names out_featuresreshape_hidden_statesrirhrrrrrrr r+)r9rur&rr'rrZr}r~rrr<rstagerr3r!r!r"r>sT         zBeitBackbone.forward)NNN) rrrr8rRrrrrr r>rCr!r!r:r"rs$ r)rmr`rrHr2r)r#F)Jr collections.abcrQrrw dataclassesrtypingrrr-rrtorch.nnr activationsr modeling_layersr modeling_outputsr r r rrrmodeling_utilsr pytorch_utilsrrrutilsrrrutils.backbone_utilsrconfiguration_beitr get_loggerrrrrArr4Moduler5rDrLrrrrrrrrrrr2rHrMr`rmrrr|rrrrr__all__r!r!r!r"s          f&YJ, ASL&W^<%%U; w