o wi @sdZddlZddlZddlZddlmZddlmZm Z ddl Z ddl Z ddl m Z m Z ddlmZmZmZddlmZdd lmZdd lmZmZmZmZmZmZdd lmZdd lm Z m!Z!m"Z"dd l#m$Z$m%Z%m&Z&ddl'm(Z(ddl)m*Z*e%+e,Z-ee$ddGdddeZ.dRde j de/de0de j fddZ1Gddde j2Z3Gdd d e j2Z4Gd!d"d"e j2Z5Gd#d$d$e j2Z6Gd%d&d&e6Z7Gd'd(d(e j2Z8e6e7d)Z9Gd*d+d+e j2Z:Gd,d-d-e j2Z;Gd.d/d/e j2ZGd4d5d5e j2Z?e$Gd6d7d7eZ@e$Gd8d9d9e@ZAGd:d;d;e j2ZBe$dd>e@ZCe$d?dGd@dAdAe@ZDGdBdCdCe j2ZEGdDdEdEe j2ZFGdFdGdGe j2ZGGdHdIdIe j2ZHGdJdKdKe j2ZIe$GdLdMdMe@ZJe$dNdGdOdPdPe@e(ZKgdQZLdS)SzPyTorch BEiT model.N) dataclass)OptionalUnion)Tensornn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)GradientCheckpointingLayer)BackboneOutputBaseModelOutputBaseModelOutputWithPoolingImageClassifierOutputMaskedLMOutputSemanticSegmenterOutput)PreTrainedModel)#compile_compatible_method_lru_cache find_pruneable_heads_and_indicesprune_linear_layer)auto_docstringlogging torch_int) BackboneMixin) BeitConfigz- Class for outputs of [`BeitModel`]. ) custom_introc@seZdZdZdS)BeitModelOutputWithPoolingaF pooler_output (`torch.FloatTensor` of shape `(batch_size, hidden_size)`): Average of the last layer hidden states of the patch tokens (excluding the *[CLS]* token) if *config.use_mean_pooling* is set to True. If set to False, then the final hidden state of the *[CLS]* token will be returned. N)__name__ __module__ __qualname____doc__r#r#c/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/transformers/models/beit/modeling_beit.pyr0srFinput drop_probtrainingreturncCsd|dks|s|Sd|}|jdfd|jd}|tj||j|jd}||||}|S)aF Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks). Comment by Ross Wightman: This is the same as the DropConnect impl I created for EfficientNet, etc networks, however, the original name is misleading as 'Drop Connect' is a different form of dropout in a separate paper... See discussion: https://github.com/tensorflow/tpu/issues/494#issuecomment-532968956 ... I've opted for changing the layer and argument names to 'drop path' rather than mix DropConnect as a layer name and use 'survival rate' as the argument. r%rrr)dtypedevice)shapendimtorchrandr+r,floor_div)r&r'r( keep_probr- random_tensoroutputr#r#r$ drop_path?s r6csTeZdZdZd deeddffdd Zdejdejfdd Z de fd d Z Z S) BeitDropPathzXDrop paths (Stochastic Depth) per sample (when applied in main path of residual blocks).Nr'r)cst||_dSN)super__init__r')selfr' __class__r#r$r:Vs  zBeitDropPath.__init__ hidden_statescCst||j|jSr8)r6r'r(r;r>r#r#r$forwardZszBeitDropPath.forwardcCs d|jS)Nzp=)r'r;r#r#r$ extra_repr]s zBeitDropPath.extra_reprr8) rr r!r"rfloatr:r/rr@strrB __classcell__r#r#r<r$r7Ss r7c sveZdZdZdeddffdd Zdejded edejfd d Z  dd ejd e ej de e dejfddZ ZS)BeitEmbeddingszc Construct the CLS token, position and patch embeddings. Optionally, also the mask token. configr)Ncstttdd|j|_|jr!ttdd|j|_ nd|_ t ||_ |j |_ t |jtjjr8|jn|j|jf|_|j j}|jrUttd|d|j|_nd|_t|j|_dS)Nr)r9r:r Parameterr/zeros hidden_size cls_tokenuse_mask_token mask_tokenBeitPatchEmbeddingspatch_embeddings patch_size isinstance image_size collectionsabcIterable num_patches use_absolute_position_embeddingsposition_embeddingsDropouthidden_dropout_probdropout)r;rGrVr<r#r$r:is   zBeitEmbeddings.__init__ embeddingsheightwidthc Cs|jdd}|jjdd}tjs||kr||kr|jS|jddddf}|jddddf}|jd}||j} ||j} t|d} |d| | |}|dddd}t j j || | fdd d }|dddd dd|}tj ||fdd S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rNg?rr bicubicFsizemode align_cornersdim)r-rXr/jit is_tracingrPrreshapepermuter functional interpolateviewcat) r;r\r]r^rV num_positionsclass_pos_embedpatch_pos_embedrg new_height new_widthsqrt_num_positionsr#r#r$interpolate_pos_encodings(     z'BeitEmbeddings.interpolate_pos_encoding pixel_valuesbool_masked_posrvcCs|jdur|durtd|j\}}}}||\}\}} |\} } }|durC|j| | d} |d | } |d| | | }|j | dd}t j ||fdd}|jdurb|| |||}||}||| ffS)Nz`interpolate_pos_encoding` argument has no effect for BEiTEmbeddings, embeddings are always interpolated to the input image size. The argument will be removed in transformers v4.51.0.r_rrf)rXwarningswarnr-rOrcrMexpand unsqueezetype_asrKr/rorvr[)r;rwrxrv_r]r^r\ patch_height patch_width batch_sizeseq_len mask_tokensw cls_tokensr#r#r$r@s"   zBeitEmbeddings.forwardNN)rr r!r"rr:r/rintrvr BoolTensorboolr@rEr#r#r<r$rFcs+rFcs6eZdZdZfddZdejdejfddZZS)rNz This class turns `pixel_values` of shape `(batch_size, num_channels, height, width)` into the initial `hidden_states` (patch embeddings) of shape `(batch_size, seq_length, hidden_size)` to be consumed by a Transformer. cst|j|j}}|j|j}}t|tjj r|n||f}t|tjj r)|n||f}|d|d|d|d}|d|d|d|df}||_||_||_||_ ||_ t j ||||d|_dS)Nrr kernel_sizestride)r9r:rRrP num_channelsrJrQrSrTrUrV patch_shaperConv2d projection)r;rGrRrPrrJrVrr<r#r$r:s   zBeitPatchEmbeddings.__init__rwr)c Cs^|j\}}}}||jkrtd||}|jd|jd}}|ddd}|||ffS)NzeMake sure that the channel dimension of the pixel values match with the one set in the configuration.r`r r)r-r ValueErrorrflatten transpose) r;rwrrr]r^r\rrr#r#r$r@s   zBeitPatchEmbeddings.forward) rr r!r"r:r/rr@rEr#r#r<r$rNs rNceZdZddedeeddffdd ZddZ   dd ej d eej d e d eej de deee de eej eej ej fffddZ ZS)BeitSelfAttentionNrG window_sizer)cst||_|j|jdkr"t|ds"td|jd|jd|j|_t|j|j|_|j|j|_ t |j|j |_ t j |j|j dd|_ t |j|j |_t |j|_t||_|jrkt||d|_dSdS) Nrembedding_sizezThe hidden size z4 is not a multiple of the number of attention heads .F)biasr)r9r:rGrJnum_attention_headshasattrrrattention_head_size all_head_sizerLinearquerykeyvaluerYattention_probs_dropout_probr[rhas_relative_position_biasBeitRelativePositionBiasrelative_position_biasr;rGrr<r#r$r:s&   zBeitSelfAttention.__init__cCs6|dd|j|jf}|j|}|ddddS)Nr_rr`rr )rcrrrnrk)r;x new_x_shaper#r#r$transpose_for_scoress z&BeitSelfAttention.transpose_for_scoresFr> head_maskoutput_attentionsrrv resolutioncCs.||}|||}|||} ||} t| |dd} | t|j } |j rL|\} } | |j j | |j j f}| |j |||jdd} |durT| |} tjj| dd}||}|duri||}t|| }|dddd}|dd|jf}|j|}|r||f}|S|f}|S) Nr_rdim_sizerfrr`r )rrrrr/matmulrmathsqrtrrrGrPrr-rrlsoftmaxr[rk contiguousrcrrn)r;r>rrrrvrmixed_query_layer key_layer value_layer query_layerattention_scoresr]r^rattention_probs context_layernew_context_layer_shapeoutputsr#r#r$r@ s4       zBeitSelfAttention.forwardr8NFNFN)rr r!rrtupler:rr/rrrrr@rEr#r#r<r$rs.  rcsveZdZ     d dejdeejdedeejdedeeed e eejeejejffffd d Z Z S) BeitSdpaSelfAttentionNFr>rrrrvrr)c s8|s|durtdtj||||||dS||}|||}|||} ||} d} |jrQ|\} } | |j j | |j j f}|j |||j dd} |dur`| dur\|} n| |7} dt |j}tjjj| || | |jrw|j jndd|d}|dd dd }|dd |jf}|j|}|dfS) Na`BeitSdpaSelfAttention` is used but `torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True` or `head_mask`. Falling back to the manual attention implementation, but specifying the manual implementation will be required from Transformers version v5.0.0 onwards. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.)r>rrrrvrrrr%F) attn_mask dropout_p is_causalscalerr`r r)logger warning_oncer9r@rrrrrrGrPrr-rrrr/rrlscaled_dot_product_attentionr(rrkrrcrrn)r;r>rrrrvrrrrr attn_biasr]r^rscalingrrr<r#r$r@CsR      zBeitSdpaSelfAttention.forwardr) rr r!r/rrrrrrr@rEr#r#r<r$rBs* rcsHeZdZdZdeddffdd Zd dejdejdejfd d ZZ S) BeitSelfOutputz The residual connection is defined in BeitLayer instead of here (as is the case with other models), due to the layernorm applied before each block. rGr)Ncs.tt|j|j|_t|j|_dSr8) r9r:rrrJdenserYrZr[r;rGr<r#r$r: zBeitSelfOutput.__init__r> input_tensorcC||}||}|Sr8rr[)r;r>rgammar#r#r$r@  zBeitSelfOutput.forwardr8) rr r!r"rr:r/rr@rEr#r#r<r$rs&r)eagersdpacr) BeitAttentionNrGrr)cs4tt|j||d|_t||_t|_dS)Nr) r9r:BEIT_SELF_ATTENTION_CLASSES_attn_implementation attentionrr5set pruned_headsrr<r#r$r:s   zBeitAttention.__init__cCst|dkrdSt||jj|jj|j\}}t|jj||j_t|jj||j_t|jj ||j_ t|j j |dd|j _ |jjt||j_|jj|jj|j_ |j ||_dS)Nrrrf)lenrrrrrrrrrr5rrunion)r;headsindexr#r#r$ prune_headss zBeitAttention.prune_headsFr>rrrrvrc Cs:|||||||}||d|}|f|dd} | S)Nrr)rr5) r;r>rrrrvr self_outputsattention_outputrr#r#r$r@s  zBeitAttention.forwardr8r)rr r!rrrr:rr/rrrrr@rEr#r#r<r$rs.  rc<eZdZdeddffdd ZdejdejfddZZS) BeitIntermediaterGr)NcsDtt|j|j|_t|jt rt |j|_ dS|j|_ dSr8) r9r:rrrJintermediate_sizerrQ hidden_actrDr intermediate_act_fnrr<r#r$r:s   zBeitIntermediate.__init__r>cCrr8)rrr?r#r#r$r@rzBeitIntermediate.forward rr r!rr:r/rr@rEr#r#r<r$rsrcr) BeitOutputrGr)Ncs.tt|j|j|_t|j|_ dSr8) r9r:rrrrJrrYrZr[rr<r#r$r:rzBeitOutput.__init__r>cCrr8rr?r#r#r$r@rzBeitOutput.forwardrr#r#r<r$rsrcseZdZdZddedeededdffdd Z   dd e j d ee j d e dee j de deee e fde ee j ee j e j fffddZZS) BeitLayerz?This corresponds to the Block class in the timm implementation.Nr%rGrdrop_path_rater)cst|j|_d|_t||d|_t||_t||_ t j |j |j d|_|dkr/t|nt |_t j |j |j d|_|j}|dkrct j|t|j dd|_t j|t|j dd|_dSd\|_|_dS) Nrrepsr%rT) requires_gradr)r9r:chunk_size_feed_forward seq_len_dimrrr intermediaterr5r LayerNormrJlayer_norm_epslayernorm_beforer7Identityr6layernorm_afterlayer_scale_init_valuerHr/oneslambda_1lambda_2)r;rGrr init_valuesr<r#r$r:s    zBeitLayer.__init__Fr>rrrrvrc Cs|j|||||||d}|d}|dd} |jdur"|j|}|||}||} || } || } |jdurB|j| } || |} | f| } | S)N)rrrvrrr)rrrr6rrr5r) r;r>rrrrvrself_attention_outputsrr layer_outputr#r#r$r@s*          zBeitLayer.forward)Nr%r)rr r!r"rrrrCr:r/rrrrr@rEr#r#r<r$rs.$rcsfeZdZdededdffdd Zedddeeefdej fd d Z dd e dej fd dZ Z S)rrGrr)NcsRt||_d|ddd|ddd|_tt|j|j|_ dS)Nr`rrr ) r9r:rnum_relative_distancerrHr/rIrrelative_position_bias_tablerr<r#r$r:#s & z!BeitRelativePositionBias.__init__ )maxsizec Csd|ddd|ddd}|d|d}tjt|dt|ddd}t|}t|d}|dddddf|dddddf}|ddd}|dddddf|dd7<|dddddf|dd7<|dddddfd|dd9<tj|dfd|jd}| d |ddddf<|d|dddf<|d|dddf<|d|d <|S) z This method creates the relative position index, modified to support arbitrary window sizes, as introduced in [MiDaS v3.1](https://huggingface.co/papers/2307.14460). r`rrr ij)indexingN)rcr+r_)rr) r/meshgridarangestackrrkrrIr+sum) r;rr window_areagridcoordscoords_flattenrelative_coordsrelative_position_indexr#r#r$ generate_relative_position_index,s $$  ,&&* z9BeitRelativePositionBias.generate_relative_position_indexFrvcCsfd|jdd}d|jdd}d|dd}d|dd}|j}|j} ||d} |d| d} | d||ddddd} tjj| t|t|fdd} | dddd| dd} t | || ddg} | |}| | d}| |d|dd|d|ddd}|ddd }|rtjj|d||fdd d d}|dS) zu Modification of timm.models.beit.py: Attention._get_rel_pos_bias to support arbitrary window sizes. r`rrr Nr_bilinear)rcrdFrb)rrrrjrkrrlrmrr/rorrnrr|squeeze)r;rrvr old_height old_widthrsrt old_relative_position_bias_tableold_num_relative_distancenew_num_relative_distance old_sub_table new_sub_table new_relative_position_bias_tabler rr#r#r$r@Es@  & z BeitRelativePositionBias.forward)FN)rr r!rrr:rrr/rrrr@rEr#r#r<r$r"s  rcseZdZddedeeddffdd Z      dd ejd eejd e d e d e deee e fde de ee ffddZ ZS) BeitEncoderNrGrr)cszt|_j|_|jrtd|_ddtjdj j ddDt fddt j D|_d|_dS) NrcSsg|]}|qSr#)item.0rr#r#r$ ~sz(BeitEncoder.__init__..rcpu)r,cs(g|]}tjr nd|dqS)N)rr)ruse_relative_position_biasrirGdprrr#r$rs F)r9r:rG!use_shared_relative_position_biasrrrr/linspacernum_hidden_layersr ModuleListrangelayergradient_checkpointingrr<r"r$r:vs   zBeitEncoder.__init__FTr>rroutput_hidden_statesrvr return_dictc Cs|rdnd}|r dnd} t|jD]M\} } |r||f}|jr;|\} } | |jj| |jjf}|j|||jdd}nd}|durE|| nd}| ||||||d}|d}|r^| |df} q|rf||f}|sttdd||| fDSt||| dS) Nr#r)rvr)rrrrvrrcss|] }|dur|VqdSr8r#)rvr#r#r$ sz&BeitEncoder.forward..)last_hidden_stater> attentions) enumerater)rrGrPrr-rr)r;r>rrr+rvrr,all_hidden_statesall_self_attentionsr! layer_moduler]r^rrlayer_head_mask layer_outputsr#r#r$r@sD     zBeitEncoder.forwardr8)NFFFNT)rr r!rrrr:r/rrrrrr@rEr#r#r<r$rus2  rc@s4eZdZeZdZdZdZdgZdgZ dZ ddZ dS) BeitPreTrainedModelbeitrwTrz.*relative_position_index.*cCsbt|tjtjtjfr%|jjjd|jj d|j dur#|j j dSdSt|tj rH|jjjd|jj d|j durF|jj|j  dSdSt|tjr]|j j |jjddSt|tr|jj |jdurs|jj |jdur|jj dSdSt|tr|jj dSt|tr|jdur|jj|jj|jj|jjdSdSdS)zInitialize the weightsr%)meanstdNg?)rQrrrConvTranspose2dweightdatanormal_rGinitializer_rangerzero_ Embedding padding_idxrfill_rFrKrMrXrrrrrr)r;moduler#r#r$ _init_weightss8             z!BeitPreTrainedModel._init_weightsN) rr r!r config_classbase_model_prefixmain_input_namesupports_gradient_checkpointing_no_split_modules"_keys_to_ignore_on_load_unexpected_supports_sdparEr#r#r#r$r7s r7cseZdZddededdffdd Zdd Zd d Ze     dd e j de e j de e j de ede edede ede eeffddZZS) BeitModelTrGadd_pooling_layerr)Ncspt|||_t||_t||jjjd|_|j rt nt j |j |jd|_|r/t|nd|_|dS)zv add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer rrN)r9r:rGrFr\rrOrencoderuse_mean_poolingrrrrJr layernorm BeitPoolerpooler post_init)r;rGrNr<r#r$r:s   zBeitModel.__init__cC|jjSr8r\rOrAr#r#r$get_input_embeddingszBeitModel.get_input_embeddingscCs*|D]\}}|jj|j|qdS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrOr)rr)r;heads_to_pruner)rr#r#r$ _prune_headsszBeitModel._prune_headsFrwrxrrr+rvr,c Cs|dur|n|jj}|dur|n|jj}|dur|n|jj}|||jj}|j||d\}} |jdd} |j||||| ||d} | d} | | } |j durU| | nd} |sl| dura| | fn| f}|| ddSt | | | j | j dS)z bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, num_patches)`, *optional*): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). N)rxr`)rrr+rr,rvrr)r/ pooler_outputr>r0)rGrr+use_return_dict get_head_maskr&r\r-rOrQrSrr>r0)r;rwrxrrr+rvr,embedding_outputr~rencoder_outputssequence_output pooled_output head_outputsr#r#r$r@s8 zBeitModel.forward)T)NNNNFN)rr r!rrr:rWr[rr/rrrrrrr@rEr#r#r<r$rMs8 rMcr) rRrGr)Ncs2t|jrtj|j|jd|_dSd|_dS)Nr)r9r:rPrrrJrrQrr<r#r$r:As zBeitPooler.__init__r>cCsL|jdur|ddddddf}||d}|S|dddf}|S)Nrr)rQr9)r;r> patch_tokensrbr#r#r$r@Gs zBeitPooler.forwardrr#r#r<r$rR@srRa Beit Model transformer with a 'language' modeling head on top. BEiT does masked image modeling by predicting visual tokens of a Vector-Quantize Variational Autoencoder (VQ-VAE), whereas other vision models like ViT and DeiT predict RGB pixel values. As a result, this class is incompatible with [`AutoModelForMaskedImageModeling`], so you will need to use [`BeitForMaskedImageModeling`] directly if you wish to do masked image modeling with BEiT. cseZdZdeddffdd Ze        ddeejdeej d eejd eejd ee d ee d e dee de e e ffddZZS)BeitForMaskedImageModelingrGr)NcsTt||j|_t|dd|_tj|j|jd|_ t |j|j |_ | dS)NFrNr)r9r: num_labelsrMr8rrrJrrQr vocab_sizelm_headrTrr<r#r$r:\s  z#BeitForMaskedImageModeling.__init__Frwrxrlabelsrr+rvr,c  Cs|dur|n|jj}|j|||||||d} | d} || } || ddddf} d} |dur 1` a classification loss is computed (Cross-Entropy). Examples: ```python >>> from transformers import AutoImageProcessor, BeitForMaskedImageModeling >>> import torch >>> from PIL import Image >>> import requests >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> image_processor = AutoImageProcessor.from_pretrained("microsoft/beit-base-patch16-224-pt22k") >>> model = BeitForMaskedImageModeling.from_pretrained("microsoft/beit-base-patch16-224-pt22k") >>> num_patches = (model.config.image_size // model.config.patch_size) ** 2 >>> pixel_values = image_processor(images=image, return_tensors="pt").pixel_values >>> # create random boolean mask of shape (batch_size, num_patches) >>> bool_masked_pos = torch.randint(low=0, high=2, size=(1, num_patches)).bool() >>> outputs = model(pixel_values, bool_masked_pos=bool_masked_pos) >>> loss, logits = outputs.loss, outputs.logits >>> list(logits.shape) [1, 196, 8192] ```N)rxrrr+rvr,rrlosslogitsr>r0) rGr]r8rQrirrr>r0)r;rwrxrrjrr+rvr,rraprediction_scoresmasked_lm_lossloss_fctr5r#r#r$r@is4, z"BeitForMaskedImageModeling.forward)NNNNNNFN)rr r!rr:rrr/rrrrrrr@rEr#r#r<r$reSs<    rez Beit Model transformer with an image classification head on top (a linear layer on top of the average of the final hidden states of the patch tokens) e.g. for ImageNet. cseZdZdeddffdd Ze       ddeejdeejd eejd ee d ee d e d ee de e e ffddZ ZS)BeitForImageClassificationrGr)NcsRt||j|_t|dd|_|jdkrt|j|jnt|_ | dS)NTrfr) r9r:rgrMr8rrrJr classifierrTrr<r#r$r:s $ z#BeitForImageClassification.__init__Frwrrjrr+rvr,cCsl|dur|n|jj}|j||||||d}|r|jn|d} || } d} |dur|jjdurS|jdkr9d|j_n|jdkrO|jtj ksJ|jtj krOd|j_nd|j_|jjdkrqt } |jdkrk| | | } n+| | |} n%|jjdkrt } | | d|j|d} n|jjdkrt} | | |} |s| f|dd} | dur| f| S| St| | |j|jd S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the image classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Nrrr+rvr,r regressionsingle_label_classificationmulti_label_classificationr_r`rk)rGr]r8r\rr problem_typergr+r/longrr rrrnrrr>r0)r;rwrrjrr+rvr,rrbrmrlrpr5r#r#r$r@sN    "       z"BeitForImageClassification.forwardNNNNNFN)rr r!rr:rrr/rrrrrr@rEr#r#r<r$rqs6  rqcseZdZdZ   ddededeeeeeffdeeeeefefd ed eeeeeffd d ffd d Z de j d e j fddZ Z S)BeitConvModuleaD A convolutional block that bundles conv/norm/activation layers. This block simplifies the usage of convolution layers, which are commonly used with a norm layer (e.g., BatchNorm) and activation layer (e.g., ReLU). Based on OpenMMLab's implementation, found in https://github.com/open-mmlab/mmsegmentation. rFr in_channels out_channelsrpaddingrdilationr)Ncs<ttj||||||d|_t||_t|_dS)N)r{r|rr}rr~) r9r:rrconv BatchNorm2dbnReLU activation)r;r{r|rr}rr~r<r#r$r:s  zBeitConvModule.__init__r&cCs"||}||}||}|Sr8)rrr)r;r&r5r#r#r$r@'s   zBeitConvModule.forward)rFr)rr r!r"rrrrDrr:r/rr@rEr#r#r<r$rz s* rzcsDeZdZdedededdffdd Zdejdejfd d ZZS) BeitPyramidPoolingBlock pool_scaler{channelsr)NcsLtt|t||ddg|_t|jD] \}}|t||qdS)Nrr) r9r:rAdaptiveAvgPool2drzlayersr1 add_modulerD)r;rr{rr!r)r<r#r$r:0s  z BeitPyramidPoolingBlock.__init__r&cCs|}|jD]}||}q|Sr8)r)r;r& hidden_stater)r#r#r$r@9s  zBeitPyramidPoolingBlock.forward) rr r!rr:r/rr@rEr#r#r<r$r/s rc sXeZdZdZdeedfdedededdf fd d Zd ej de ej fd d Z Z S)BeitPyramidPoolingModulea Pyramid Pooling Module (PPM) used in PSPNet. Args: pool_scales (tuple[int]): Pooling scales used in Pooling Pyramid Module. in_channels (int): Input channels. channels (int): Channels after modules, before conv_seg. align_corners (bool): align_corners argument of F.interpolate. Based on OpenMMLab's implementation, found in https://github.com/open-mmlab/mmsegmentation. pool_scales.r{rrer)Ncsht||_||_||_||_g|_t|D]\}}t|||d}|j || t ||qdS)N)rr{r) r9r:rrer{rblocksr1rappendrrD)r;rr{rrer!rblockr<r#r$r:Ns  z!BeitPyramidPoolingModule.__init__rcCsHg}|jD]}||}tjj||ddd|jd}||q|S)Nr`rrb)rrrlrmrcrer)r;rppm_outsppmppm_outupsampled_ppm_outr#r#r$r@Zs  z BeitPyramidPoolingModule.forward) rr r!r"rrrr:r/rlistr@rEr#r#r<r$r@s* " rcsHeZdZdZdeddffdd ZddZd ejdejfd d Z Z S) BeitUperHeadz Unified Perceptual Parsing for Scene Understanding. This head is the implementation of [UPerNet](https://huggingface.co/papers/1807.10221). Based on OpenMMLab's implementation, found in https://github.com/open-mmlab/mmsegmentation. rGr)Ncst|j|_|jgd|_|j|_d|_tj|j|j dd|_ t |j|jd|j|jd|_ t |jdt|j|j|jddd|_t|_t|_|jddD] }t ||jdd}t |j|jddd}|j||j|qWt t|j|j|jddd|_dS) NFrrr_)rer rr})r9r:rrJr{rrerrrgrrr psp_modulesrzr bottleneckr' lateral_convs fpn_convsrfpn_bottleneck)r;rGr{l_convfpn_convr<r#r$r:ms>     zBeitUperHead.__init__cCs:|d}|g}|||tj|dd}||}|S)Nr_rrf)extendrr/ror)r;inputsrpsp_outsr5r#r#r$ psp_forwards  zBeitUperHead.psp_forwardencoder_hidden_statescsfddtjDt}t|dddD]$}|djdd}|dtjj ||dj d|d<q fd dt|dD}|dt|dddD]}tjj |||djdddj d||<qbt j |dd } |}|}|S) Ncsg|] \}}||qSr#r#)rr! lateral_conv)rr#r$rsz(BeitUperHead.forward..rrr_r`rrbcsg|] }j||qSr#)rr )lateralsr;r#r$rsrf)r1rrrrr(r-rrlrmrer/rorrr)r;rused_backbone_levelsr! prev_shapefpn_outsr5r#)rrr;r$r@s$   zBeitUperHead.forward) rr r!r"rr:rr/rr@rEr#r#r<r$res & rc s`eZdZdZ ddedededeeeeeffd d f fd d Zd e j d e j fddZ Z S) BeitFCNHeada Fully Convolution Networks for Semantic Segmentation. This head is implemented of [FCNNet](https://huggingface.co/papers/1411.4038>). Args: config (BeitConfig): Configuration. in_channels kernel_size (int): The kernel size for convs in the head. Default: 3. dilation (int): The dilation rate for convs in the head. Default: 1. Based on OpenMMLab's implementation, found in https://github.com/open-mmlab/mmsegmentation. r`r rrGin_indexrr~r)Nc st|j|_|j|_|j|_|j|_ ||_ |d|}g}| t |j|j|||dt |jdD]}| t |j|j|||dq5|jdkrQt|_ntj||_|j rjt |j|j|j||dd|_tj|j|jdd|_dS)Nr`)rr}r~rrrr)r9r:rJr{auxiliary_channelsrauxiliary_num_convs num_convsauxiliary_concat_input concat_inputrrrzr(rrconvs Sequentialconv_catrrgrr)r;rGrrr~ conv_paddingrr!r<r#r$r:s6     zBeitFCNHead.__init__rcCs@||j}||}|jr|tj||gdd}||}|S)Nrrf)rrrrr/rorr)r;rr>r5r#r#r$r@s   zBeitFCNHead.forward)r`r r) rr r!r"rrrrr:r/rr@rEr#r#r<r$rs"rcseZdZdeddffdd ZddZe       dd eej d eej d eej d ee d ee de dee de e e ffddZZS)BeitForSemanticSegmentationrGr)Nc st||j|_t|dd|_t|jjdkrtdt t j |j |j dddt |j t t j |j |j ddd|_t t j |j |j ddd|_t |_t jddd|_t||_|jrft|nd|_|dS)NFrfrzBeitForSemanticSegmentation requires config.out_indices to be a list of 4 integers, specifying which features to use from the backbone. One can use [3, 5, 7, 11] in case of a base-sized architecture.r`r)r9r:rgrMr8rrG out_indicesrrrr;rJrGELUfpn1fpn2rfpn3 MaxPool2dfpn4r decode_headuse_auxiliary_headrauxiliary_headrTrr<r#r$r:s*     z$BeitForSemanticSegmentation.__init__c Cstjj||jddddd}|dur"tjj||jddddd}t|jjd}|||}|}|durA|||} ||jj| 7}|S)NrrFrb) ignore_index)rrlrmr-rrGsemantic_loss_ignore_indexauxiliary_loss_weight) r;rmauxiliary_logitsrjupsampled_logitsupsampled_auxiliary_logitsrp main_lossrlauxiliary_lossr#r#r$ compute_losss  z(BeitForSemanticSegmentation.compute_lossFrwrrjrr+rvr,cs|dur|njj}|dur|njj}|dur"jjdkr"tdj|||d||d}|r2|jn|d} fddt| D} |jdjj jj fd d| D} j j j jg} tt| D] } | | | | | | <qh| } d}jdur| }d}|dur| ||}|s|r| f|dd}n | f|d d}|dur|f|S|St|| |r|jnd|jd S) aD labels (`torch.LongTensor` of shape `(batch_size, height, width)`, *optional*): Ground truth semantic segmentation maps for computing the loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels > 1`, a classification loss is computed (Cross-Entropy). Examples: ```python >>> from transformers import AutoImageProcessor, BeitForSemanticSegmentation >>> from PIL import Image >>> import requests >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> image_processor = AutoImageProcessor.from_pretrained("microsoft/beit-base-finetuned-ade-640-640") >>> model = BeitForSemanticSegmentation.from_pretrained("microsoft/beit-base-finetuned-ade-640-640") >>> inputs = image_processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> # logits are of shape (batch_size, num_labels, height, width) >>> logits = outputs.logits ```Nrz/The number of labels should be greater than oneTrscs$g|]\}}|djjvr|qSr*)rGr)ridxfeaturerAr#r$ras$z7BeitForSemanticSegmentation.forward..rcs<g|]}|ddddddfddddqS)Nrrr`r_)rkrjr)rpatch_resolutionr#r$rds0r`rk)rGr]r+rgrr8r>r1r-rRrPrrrrr(rrrrrr0)r;rwrrjrr+rvr,rrfeaturesopsr!rmrrlr5r#)rrr;r$r@*sR"       z#BeitForSemanticSegmentation.forwardry)rr r!rr:rrrr/rrrrrr@rEr#r#r<r$rs8  rzM BEiT backbone, to be used with frameworks like DETR and MaskFormer. c sZeZdZfddZddZe   d dedeedeed eed e f d d Z Z S) BeitBackbonec sttfddtjdD|_t|_t|jj j d|_ j rut |jjdkr9tdj}ttj||dddtj|jd ttj||ddd|_ttj||ddd|_t|_tjddd|_|dS) Ncsg|]}jqSr#)rJ)rr~rGr#r$rsz)BeitBackbone.__init__..rrrzBeitBackbone requires config.out_indices to be a list of 4 integers, specifying which features to use from the backbone. One can use [3, 5, 7, 11] in case of a base-sized architecture.r`rr)r9r:_init_backboner(r& num_featuresrFr\rrOrrOadd_fpnrrGrrrJrrr;rbatch_norm_epsrrrrrrrrT)r;rGrJr<rr$r:s*     zBeitBackbone.__init__cCrUr8rVrAr#r#r$rWrXz!BeitBackbone.get_input_embeddingsNrwr+rr,r)cCs|dur|n|jj}|dur|n|jj}|dur|n|jj}|jd}||\}\}}|jdd} |j|d|| |d} |rB| jn| d} d} t|j | D].\} }| |j vr||jj rw|ddddddf}| ddd}| |d||}| |f7} qN|jjr|| d|| d|| d|| d g} t| } |s|r| f| dd}|S| f| dd}|St| |r| jnd| jd S) a: Examples: ```python >>> from transformers import AutoImageProcessor, AutoBackbone >>> import torch >>> from PIL import Image >>> import requests >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> processor = AutoImageProcessor.from_pretrained("microsoft/beit-base-patch16-224") >>> model = AutoBackbone.from_pretrained( ... "microsoft/beit-base-patch16-224", out_features=["stage1", "stage2", "stage3", "stage4"] ... ) >>> inputs = processor(image, return_tensors="pt") >>> outputs = model(**inputs) >>> feature_maps = outputs.feature_maps >>> list(feature_maps[-1].shape) [1, 768, 14, 14] ```Nrr`T)r+rrr,rr#r_r ) feature_mapsr>r0)rGr]r+rr-r\rOr>zip stage_names out_featuresreshape_hidden_statesrkrjrrrrrrr r0)r;rwr+rr,rr_rrrrr>rstagerr5r#r#r$r@sT         zBeitBackbone.forward)NNN) rr r!r:rWrrrrr r@rEr#r#r<r$rs$ r)rqrerrMr7r)r%F)Mr"collections.abcrSrry dataclassesrtypingrrr/torch.utils.checkpointrrtorch.nnrrr activationsr modeling_layersr modeling_outputsr rrrrrmodeling_utilsr pytorch_utilsrrrutilsrrrutils.backbone_utilsrconfiguration_beitr get_loggerrrrrCrr6Moduler7rFrNrrrrrrrrrrr7rMrRrerqrzrrrrrr__all__r#r#r#r$s         f&S>, ASL&W[N%%U; w