o eij @sdZddlZddlZddlmZddlZddlmZmZddl m Z ddl m Z ddlmZdd lmZdd lmZdd lmZmZmZmZmZmZdd lmZdd lmZddlm Z m!Z!m"Z"ddl#m$Z$e!%e&Z'ee ddGdddeZ(dRdejde)de*dejfddZ+Gdddej,Z-Gdd d ej,Z.Gd!d"d"ej,Z/Gd#d$d$ej,Z0Gd%d&d&e0Z1Gd'd(d(ej,Z2e0e1d)Z3Gd*d+d+ej,Z4Gd,d-d-ej,Z5Gd.d/d/ej,Z6Gd0d1d1eZ7Gd2d3d3ej,Z8Gd4d5d5ej,Z9e Gd6d7d7eZ:e Gd8d9d9e:Z;Gd:d;d;ej,Zd>e:Z=e d?dGd@dAdAe:Z>GdBdCdCej,Z?GdDdEdEej,Z@GdFdGdGej,ZAGdHdIdIej,ZBGdJdKdKej,ZCe GdLdMdMe:ZDe dNdGdOdPdPee:ZEgdQZFdS)SzPyTorch BEiT model.N) dataclass)Tensornn)CrossEntropyLoss)initialization)ACT2FN) BackboneMixin)GradientCheckpointingLayer)BackboneOutputBaseModelOutputBaseModelOutputWithPoolingImageClassifierOutputMaskedLMOutputSemanticSegmenterOutput)PreTrainedModel)#compile_compatible_method_lru_cache)auto_docstringlogging torch_int) BeitConfigz- Class for outputs of [`BeitModel`]. ) custom_introc@seZdZdZdS)BeitModelOutputWithPoolingaF pooler_output (`torch.FloatTensor` of shape `(batch_size, hidden_size)`): Average of the last layer hidden states of the patch tokens (excluding the *[CLS]* token) if *config.use_mean_pooling* is set to True. If set to False, then the final hidden state of the *[CLS]* token will be returned. N)__name__ __module__ __qualname____doc__rrd/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/beit/modeling_beit.pyr-srFinput drop_probtrainingreturncCsd|dks|s|Sd|}|jdfd|jd}|tj||j|jd}||||}|S)zc Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks). r rrr)dtypedevice)shapendimtorchrandr&r'floor_div)r!r"r# keep_probr( random_tensoroutputrrr drop_path<s r1csTeZdZdZd dedBddffdd Zdejdejfdd Zde fd d Z Z S) BeitDropPathzXDrop paths (Stochastic Depth) per sample (when applied in main path of residual blocks).Nr"r$cst||_dSN)super__init__r")selfr" __class__rrr5Ns  zBeitDropPath.__init__ hidden_statescCst||j|jSr3)r1r"r#r6r9rrrforwardRszBeitDropPath.forwardcCs d|jS)Nzp=)r"r6rrr extra_reprUs zBeitDropPath.extra_reprr3) rrrrfloatr5r*rr;strr= __classcell__rrr7rr2Ks r2csleZdZdZdeddffdd Zdejded edejfd d Z dd ejd ej dBdejfddZ Z S)BeitEmbeddingszc Construct the CLS token, position and patch embeddings. Optionally, also the mask token. configr$Ncstttdd|j|_|jr!ttdd|j|_ nd|_ t ||_ |j |_ t |jtjjr8|jn|j|jf|_|j j}|jrUttd|d|j|_nd|_t|j|_dS)Nr)r4r5r Parameterr*zeros hidden_size cls_tokenuse_mask_token mask_tokenBeitPatchEmbeddingspatch_embeddings patch_size isinstance image_size collectionsabcIterable num_patches use_absolute_position_embeddingsposition_embeddingsDropouthidden_dropout_probdropout)r6rBrQr7rrr5as   zBeitEmbeddings.__init__ embeddingsheightwidthc Cs|jdd}|jjdd}tjs||kr||kr|jS|jddddf}|jddddf}|jd}||j} ||j} t|d} |d| | |}|dddd}t j j || | fdd d }|dddd dd|}tj ||fdd S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rNg?rrbicubicFsizemode align_cornersdim)r(rSr*jit is_tracingrKrreshapepermuter functional interpolateviewcat) r6rWrXrYrQ num_positionsclass_pos_embedpatch_pos_embedrb new_height new_widthsqrt_num_positionsrrrinterpolate_pos_encodingxs(     z'BeitEmbeddings.interpolate_pos_encoding pixel_valuesbool_masked_poscCs|j\}}}}||\}\}}|\} } }|dur5|j| | d} |d| } |d| | | }|j| dd} tj | |fdd}|j durT|| |||}| |}|||ffSNrZrra) r(rJr^rHexpand unsqueezetype_asrFr*rjrSrqrV)r6rrrs_rXrYrW patch_height patch_width batch_sizeseq_len mask_tokensw cls_tokensrrrr;s   zBeitEmbeddings.forwardr3) rrrrrr5r*rintrq BoolTensorr;r@rrr7rrA[s+rAcs6eZdZdZfddZdejdejfddZZS)rIz This class turns `pixel_values` of shape `(batch_size, num_channels, height, width)` into the initial `hidden_states` (patch embeddings) of shape `(batch_size, seq_length, hidden_size)` to be consumed by a Transformer. cst|j|j}}|j|j}}t|tjj r|n||f}t|tjj r)|n||f}|d|d|d|d}|d|d|d|df}||_||_||_||_ ||_ t j ||||d|_dS)Nrr kernel_sizestride)r4r5rMrK num_channelsrErLrNrOrPrQ patch_shaperConv2d projection)r6rBrMrKrrErQrr7rrr5s   zBeitPatchEmbeddings.__init__rrr$c Csj|j\}}}}||jkrtd|||jjj}|jd|jd}}|ddd}|||ffS)NzeMake sure that the channel dimension of the pixel values match with the one set in the configuration.r[rr) r(r ValueErrorrtoweightr&flatten transpose) r6rrr{rrXrYrWryrzrrrr;s  zBeitPatchEmbeddings.forward) rrrrr5r*rr;r@rrr7rrIs rIceZdZddededBddffdd Z    ddejd ed ejdBd ed ee dBdeejeejejfBf d dZ Z S)BeitSelfAttentionNrB window_sizer$cst||_|j|jdkr"t|ds"td|jd|jd|j|_t|j|j|_|j|j|_ t |j|j |_ t j |j|j dd|_ t |j|j |_t |j|_t||_|jrkt||d|_dSdS) Nrembedding_sizezThe hidden size z4 is not a multiple of the number of attention heads .F)biasr)r4r5rBrEnum_attention_headshasattrrrattention_head_size all_head_sizerLinearquerykeyvaluerTattention_probs_dropout_probrVboolhas_relative_position_biasBeitRelativePositionBiasrelative_position_biasr6rBrr7rrr5s&   zBeitSelfAttention.__init__Fr9output_attentionsrrq resolutioncCs\|j\}}}|||d|j|jdd} |||d|j|jdd} |||d|j|jdd} t | | dd} | t |j} |j rk|\} }| |j j||j jf}| |j|||jdd} |durs| |} tjj| dd}||}t || }|dddd}|dd|jf}|j|}|r||f}|S|f}|S) NrZrr[dim_sizerarr)r(rrirrrrrr*matmulmathsqrtrrBrKrrrgsoftmaxrVrf contiguousr^r)r6r9rrrqrr{ seq_lengthrx query_layer key_layer value_layerattention_scoresrXrYrattention_probs context_layernew_context_layer_shapeoutputsrrrr;sB      zBeitSelfAttention.forwardr3FNFN rrrrtupler5r*rrrr;r@rrr7rrs&  rc@s^eZdZ    d dejdedejdBdedeedBdeejeejejfBf d d ZdS) BeitSdpaSelfAttentionFNr9rrrqrr$c Cs`|r t|jjd|j\}}}|||d|j|j dd} | ||d|j|j dd} | ||d|j|j dd} d} |j re|\} }| |j j||j jf}|j|||jdd} |durt| durp|} n| |7} dt|j}tjjj| | | | |jr|j jndd|d}|d ddd }|dd |jf}|j|}|dfS) Nz does not support `output_attentions=True`. The returned attention weights will be `None`. If you want to get attention weights, please set `attn_implementation='eager'` when loading the model.rZrr[rr F) attn_mask dropout_p is_causalscalerrr)logger warning_oncer8rr(rrirrrrrrrBrKrrrr*rrgscaled_dot_product_attentionr#rrfrr^r)r6r9rrrqrr{rrxrrr attn_biasrXrYrscalingrrrrrr;5sT     zBeitSdpaSelfAttention.forwardr) rrrr*rrrrr;rrrrr4s$ rcsHeZdZdZdeddffdd Zd dejdejdejfd d ZZ S) BeitSelfOutputz The residual connection is defined in BeitLayer instead of here (as is the case with other models), due to the layernorm applied before each block. rBr$Ncs.tt|j|j|_t|j|_dSr3) r4r5rrrEdenserTrUrVr6rBr7rrr5x zBeitSelfOutput.__init__r9 input_tensorcC||}||}|Sr3rrV)r6r9rgammarrrr;}  zBeitSelfOutput.forwardr3) rrrrrr5r*rr;r@rrr7rrrs&r)eagersdpacr) BeitAttentionNrBrr$cs,tt|j||d|_t||_dS)Nr)r4r5BEIT_SELF_ATTENTION_CLASSES_attn_implementation attentionrr0rr7rrr5s zBeitAttention.__init__Fr9rrrqrc Cs8||||||}||d|}|f|dd}|S)Nrr)rr0) r6r9rrrqr self_outputsattention_outputrrrrr;s  zBeitAttention.forwardr3rrrrr7rrs&  rc<eZdZdeddffdd ZdejdejfddZZS) BeitIntermediaterBr$NcsDtt|j|j|_t|jt rt |j|_ dS|j|_ dSr3) r4r5rrrEintermediate_sizerrL hidden_actr?rintermediate_act_fnrr7rrr5s   zBeitIntermediate.__init__r9cCrr3)rrr:rrrr;rzBeitIntermediate.forward rrrrr5r*rr;r@rrr7rrsrcr) BeitOutputrBr$Ncs.tt|j|j|_t|j|_ dSr3) r4r5rrrrErrTrUrVrr7rrr5rzBeitOutput.__init__r9cCrr3rr:rrrr;rzBeitOutput.forwardrrrr7rrsrcseZdZdZddededBdeddffdd Z  dd ej d e d ej dBde dee e fdBdeej eej ej fBf ddZ Z S) BeitLayerz?This corresponds to the Block class in the timm implementation.Nr rBrdrop_path_rater$cst|j|_d|_t||d|_t||_t||_ t j |j |j d|_|dkr/t|nt |_t j |j |j d|_|j}|dkrct j|t|j dd|_t j|t|j dd|_dSd\|_|_dS) Nrrepsr rT) requires_grad)NN)r4r5chunk_size_feed_forward seq_len_dimrrr intermediaterr0r LayerNormrElayer_norm_epslayernorm_beforer2Identityr1layernorm_afterlayer_scale_init_valuerCr*oneslambda_1lambda_2)r6rBrr init_valuesr7rrr5s    zBeitLayer.__init__Fr9rrrqrc Cs|j||||||d}|d}|dd}|jdur!|j|}|||}||} || } || } |jdurA|j| } || |} | f|}|S)Nrrrqrrr)rrrr1rrr0r) r6r9rrrqrself_attention_outputsrr layer_outputrrrr;s(         zBeitLayer.forward)Nr r)rrrrrrr>r5r*rrrr;r@rrr7rrs($rcsfeZdZdededdffdd Zedddeeefdej fd d Z dd e dej fd dZ Z S)rrBrr$NcsRt||_d|ddd|ddd|_tt|j|j|_ dS)Nr[rrr) r4r5rnum_relative_distancerrCr*rDrrelative_position_bias_tablerr7rrr5s & z!BeitRelativePositionBias.__init__ )maxsizec Csd|ddd|ddd}|d|d}tjt|dt|ddd}t|}t|d}|dddddf|dddddf}|ddd}|dddddf|dd7<|dddddf|dd7<|dddddfd|dd9<tj|dfd|jd}| d |ddddf<|d|dddf<|d|dddf<|d|d <|S) z This method creates the relative position index, modified to support arbitrary window sizes, as introduced in [MiDaS v3.1](https://huggingface.co/papers/2307.14460). r[rrrij)indexingN)r^r&rZ)rr) r*meshgridarangestackrrfrrDr&sum) r6rr window_areagridcoordscoords_flattenrelative_coordsrelative_position_indexrrr generate_relative_position_indexs $$  ,&&* z9BeitRelativePositionBias.generate_relative_position_indexFrqcCsfd|jdd}d|jdd}d|dd}d|dd}|j}|j} ||d} |d| d} | d||ddddd} tjj| t|t|fdd} | dddd| dd} t | || ddg} | |}| | d}| |d|dd|d|ddd}|ddd }|rtjj|d||fdd d d}|dS) zu Modification of timm.models.beit.py: Attention._get_rel_pos_bias to support arbitrary window sizes. r[rrrNrZbilinear)r^r_Fr])rrrrerfrrgrhrr*rjrrirrvsqueeze)r6rrqr old_height old_widthrnro old_relative_position_bias_tableold_num_relative_distancenew_num_relative_distance old_sub_table new_sub_table new_relative_position_bias_tablerrrrrr;!s@  & z BeitRelativePositionBias.forward)FN)rrrrrr5rrr*rrrr;r@rrr7rrs  rcsteZdZddededBddffdd Z     dd ejd ed ed ed ee e fdBdedee BfddZ Z S) BeitEncoderNrBrr$cszt|_j|_|jrtd|_ddtjdj j ddDt fddt j D|_d|_dS) NrcSsg|]}|qSr)item.0xrrr Zsz(BeitEncoder.__init__..rcpu)r'cs(g|]}tjr nd|dqS)N)rr)ruse_relative_position_biasrirBdprrrrr\s F)r4r5rB!use_shared_relative_position_biasrrrr*linspacernum_hidden_layersr ModuleListrangelayergradient_checkpointingrr7rrr5Rs   zBeitEncoder.__init__FTr9routput_hidden_statesrqr return_dictcCs|rdnd}|r dnd}t|jD]B\} } |r||f}|jr;|\} } | |jj| |jjf} |j| ||jdd}nd}| |||||d}|d}|rS||df}q|r[||f}|sitdd|||fDSt|||dS) Nrr)rqrrrcss|] }|dur|VqdSr3r)rvrrr sz&BeitEncoder.forward..)last_hidden_stater9 attentions) enumeraterrrBrKrr(rr )r6r9rrrqrrall_hidden_statesall_self_attentionsr layer_modulerXrYrr layer_outputsrrrr;gs@    zBeitEncoder.forwardr3)FFFNT) rrrrrr5r*rrrr r;r@rrr7rr Qs, r csNeZdZUeed<dZdZdZdZdgZ dgZ dZ e fdd ZZS) BeitPreTrainedModelrBbeit)imagerrTrz.*relative_position_index.*cst|t|tr+t|j|jdurt|j|jdur)t|jdSdSt|t r8t|j dSt|t rV|j durXt |j |jjt |j|jjdSdSdS)zInitialize the weightsN)r4 _init_weightsrLrAinitzeros_rFrHrSrrrr constant_rBrr)r6moduler7rrr,s         z!BeitPreTrainedModel._init_weights)rrrr__annotations__base_model_prefixinput_modalitiesmain_input_namesupports_gradient_checkpointing_no_split_modules"_keys_to_ignore_on_load_unexpected_supports_sdpar*no_gradr,r@rrr7rr)s r)cseZdZddededdffdd Zdd Ze    dd ej d ej dBd edBdedBdededBde e BfddZ ZS) BeitModelTrBadd_pooling_layerr$Ncspt|||_t||_t||jjjd|_|j rt nt j |j |jd|_|r/t|nd|_|dS)zv add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer rrN)r4r5rBrArWr rJrencoderuse_mean_poolingrrrrEr layernorm BeitPoolerpooler post_init)r6rBr;r7rrr5s   zBeitModel.__init__cC|jjSr3rWrJr<rrrget_input_embeddingszBeitModel.get_input_embeddingsFrrrsrrrqrcKs|dur|n|jj}|dur|n|jj}|dur|n|jj}|j||d\}} |jdd} |j|||| ||d} | d} || } |jdurL|| nd} |sc| durX| | fn| f}|| ddSt | | | j | j dS)z bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, num_patches)`, *optional*): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). N)rsr[)rrrrrqrr)r" pooler_outputr9r#) rBrruse_return_dictrWr(r<r>r@rr9r#)r6rrrsrrrqrkwargsembedding_outputrxrencoder_outputssequence_output pooled_output head_outputsrrrr;s4 zBeitModel.forward)T)NNNFN)rrrrrr5rDrr*rrrrr;r@rrr7rr:s0 r:cr) r?rBr$Ncs2t|jrtj|j|jd|_dSd|_dS)Nr)r4r5r=rrrErr>rr7rrr5s zBeitPooler.__init__r9cCsL|jdur|ddddddf}||d}|S|dddf}|S)Nrr)r>mean)r6r9 patch_tokensrLrrrr;s zBeitPooler.forwardrrrr7rr?sr?a Beit Model transformer with a 'language' modeling head on top. BEiT does masked image modeling by predicting visual tokens of a Vector-Quantize Variational Autoencoder (VQ-VAE), whereas other vision models like ViT and DeiT predict RGB pixel values. As a result, this class is incompatible with [`AutoModelForMaskedImageModeling`], so you will need to use [`BeitForMaskedImageModeling`] directly if you wish to do masked image modeling with BEiT. cseZdZdeddffdd ZddZe       dd ejdBd ej dBd ejdBd e dBd e dBde de dBde e BfddZ ZS)BeitForMaskedImageModelingrBr$NcsTt||j|_t|dd|_tj|j|jd|_ t |j|j |_ | dS)NFr;r)r4r5 num_labelsr:r*rrrErr>r vocab_sizelm_headrArr7rrr5s  z#BeitForMaskedImageModeling.__init__cCsdSr3rr<rrrget_output_embeddings'sz0BeitForMaskedImageModeling.get_output_embeddingsFrrrslabelsrrrqrcKs|dur|n|jj}|j||||||d} | d} || } || ddddf} d} |dur;t} | | ||} |sQ| f| dd}| durO| f|S|St| | | j| jdS)a bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, num_patches)`): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the image classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Examples: ```python >>> from transformers import AutoImageProcessor, BeitForMaskedImageModeling >>> import torch >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> image_processor = AutoImageProcessor.from_pretrained("microsoft/beit-base-patch16-224-pt22k") >>> model = BeitForMaskedImageModeling.from_pretrained("microsoft/beit-base-patch16-224-pt22k") >>> num_patches = (model.config.image_size // model.config.patch_size) ** 2 >>> pixel_values = image_processor(images=image, return_tensors="pt").pixel_values >>> # create random boolean mask of shape (batch_size, num_patches) >>> bool_masked_pos = torch.randint(low=0, high=2, size=(1, num_patches)).bool() >>> outputs = model(pixel_values, bool_masked_pos=bool_masked_pos) >>> loss, logits = outputs.loss, outputs.logits >>> list(logits.shape) [1, 196, 8192] ```N)rsrrrqrrrlosslogitsr9r#) rBrGr*r>rTrrr9r#)r6rrrsrVrrrqrrHrrKprediction_scoresmasked_lm_lossloss_fctr0rrrr;*s2. z"BeitForMaskedImageModeling.forward)NNNNNFN)rrrrr5rUrr*rrrrrr;r@rrr7rrPs8   rPz Beit Model transformer with an image classification head on top (a linear layer on top of the average of the final hidden states of the patch tokens) e.g. for ImageNet. cszeZdZdeddffdd Ze      ddejdBdejdBd edBd edBd ed edBde e Bfd dZ Z S)BeitForImageClassificationrBr$NcsRt||j|_t|dd|_|jdkrt|j|jnt|_ | dS)NTrQr) r4r5rRr:r*rrrEr classifierrArr7rrr5s $ z#BeitForImageClassification.__init__FrrrVrrrqrc Ks|dur|n|jj}|j|||||d}|r|jn|d} || } d} |dur0||| |j} |sF| f|dd} | durD| f| S| St| | |j|jdS)a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the image classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Nrrrqrrr[rW) rBrGr*rFr^ loss_functionrr9r#) r6rrrVrrrqrrHrrLrYrXr0rrrr;s, z"BeitForImageClassification.forwardNNNNFN) rrrrr5rr*rrrrr;r@rrr7rr]xs0  r]cseZdZdZ   ddededeeeefBdeeeefBeBd ed eeeefBd d ffd d Zde j d e j fddZ Z S)BeitConvModuleaD A convolutional block that bundles conv/norm/activation layers. This block simplifies the usage of convolution layers, which are commonly used with a norm layer (e.g., BatchNorm) and activation layer (e.g., ReLU). Based on OpenMMLab's implementation, found in https://github.com/open-mmlab/mmsegmentation. rFr in_channels out_channelsrpaddingrdilationr$Ncs<ttj||||||d|_t||_t|_dS)N)rcrdrrerrf) r4r5rrconv BatchNorm2dbnReLU activation)r6rcrdrrerrfr7rrr5s  zBeitConvModule.__init__r!cCs"||}||}||}|Sr3)rgrirk)r6r!r0rrrr;s   zBeitConvModule.forward)rFr) rrrrrrr?rr5r*rr;r@rrr7rrbs* rbcsDeZdZdedededdffdd Zdejdejfd d ZZS) BeitPyramidPoolingBlock pool_scalercchannelsr$NcsLtt|t||ddg|_t|jD] \}}|t||qdS)Nrr) r4r5rAdaptiveAvgPool2drblayersr$ add_moduler?)r6rmrcrnrrr7rrr5s  z BeitPyramidPoolingBlock.__init__r!cCs|}|jD]}||}q|Sr3)rq)r6r! hidden_staterrrrr;s  zBeitPyramidPoolingBlock.forward) rrrrr5r*rr;r@rrr7rrls rlc sXeZdZdZdeedfdedededdf fd d Zd ej de ej fd d Z Z S)BeitPyramidPoolingModulea Pyramid Pooling Module (PPM) used in PSPNet. Args: pool_scales (tuple[int]): Pooling scales used in Pooling Pyramid Module. in_channels (int): Input channels. channels (int): Channels after modules, before conv_seg. align_corners (bool): align_corners argument of F.interpolate. Based on OpenMMLab's implementation, found in https://github.com/open-mmlab/mmsegmentation. pool_scales.rcrnr`r$Ncsht||_||_||_||_g|_t|D]\}}t|||d}|j || t ||qdS)N)rmrcrn) r4r5rur`rcrnblocksr$rlappendrrr?)r6rurcrnr`rrmblockr7rrr5s  z!BeitPyramidPoolingModule.__init__rcCsHg}|jD]}||}tjj||ddd|jd}||q|S)Nr[rr])rvrrgrhr^r`rw)r6rppm_outsppmppm_outupsampled_ppm_outrrrr; s  z BeitPyramidPoolingModule.forward) rrrrrrrr5r*rlistr;r@rrr7rrts* " rtcsHeZdZdZdeddffdd ZddZd ejdejfd d Z Z S) BeitUperHeadz Unified Perceptual Parsing for Scene Understanding. This head is the implementation of [UPerNet](https://huggingface.co/papers/1807.10221). Based on OpenMMLab's implementation, found in https://github.com/open-mmlab/mmsegmentation. rBr$Ncst|j|_|jgd|_|j|_d|_tj|j|j dd|_ t |j|jd|j|jd|_ t |jdt|j|j|jddd|_t|_t|_|jddD] }t ||jdd}t |j|jddd}|j||j|qWt t|j|j|jddd|_dS) NFrrorZ)r`rrre)r4r5rurErcrnr`rrrRr^rt psp_modulesrblen bottleneckr lateral_convs fpn_convsrwfpn_bottleneck)r6rBrcl_convfpn_convr7rrr5s>     zBeitUperHead.__init__cCs:|d}|g}|||tj|dd}||}|Srt)extendrr*rjr)r6inputsrpsp_outsr0rrr psp_forwardBs  zBeitUperHead.psp_forwardencoder_hidden_statescsfddtjDt}t|dddD]$}|djdd}|dtjj ||dj d|d<q fd dt|dD}|dt|dddD]}tjj |||djdddj d||<qbt j |dd } |}|}|S) Ncsg|] \}}||qSrr)rr lateral_conv)rrrrMsz(BeitUperHead.forward..rrrZr[rr]csg|] }j||qSr)rr)lateralsr6rrrZsra)r$rrwrrrr(rrgrhr`r*rjrr^)r6rused_backbone_levelsr prev_shapefpn_outsr0r)rrr6rr;Ks$   zBeitUperHead.forward) rrrrrr5rr*rr;r@rrr7rr~s & r~c s\eZdZdZ ddedededeeeefBd d f fd d Zd ej d ej fddZ Z S) BeitFCNHeada Fully Convolution Networks for Semantic Segmentation. This head is implemented of [FCNNet](https://huggingface.co/papers/1411.4038>). Args: config (BeitConfig): Configuration. in_channels kernel_size (int): The kernel size for convs in the head. Default: 3. dilation (int): The dilation rate for convs in the head. Default: 1. Based on OpenMMLab's implementation, found in https://github.com/open-mmlab/mmsegmentation. r[rrrBin_indexrrfr$Nc st|j|_|j|_|j|_|j|_ ||_ |d|}g}| t |j|j|||dt |jdD]}| t |j|j|||dq5|jdkrQt|_ntj||_|j rjt |j|j|j||dd|_tj|j|jdd|_dS)Nr[)rrerfrrrro)r4r5rErcauxiliary_channelsrnauxiliary_num_convs num_convsauxiliary_concat_input concat_inputrrwrbrrrconvs Sequentialconv_catrrRr^)r6rBrrrf conv_paddingrrr7rrr5xs6     zBeitFCNHead.__init__rcCs@||j}||}|jr|tj||gdd}||}|S)Nrra)rrrrr*rjr^)r6rr9r0rrrr;s   zBeitFCNHead.forward)r[rr) rrrrrrrr5r*rr;r@rrr7rris"rcseZdZdeddffdd ZddZe      dd ejdBd ejdBd e dBd e dBd e de dBde e BfddZ Z S)BeitForSemanticSegmentationrBr$Nc st||j|_t|dd|_t|jjdkrtdt t j |j |j dddt |j t t j |j |j ddd|_t t j |j |j ddd|_t |_t jddd|_t||_|jrft|nd|_|dS)NFrQrzBeitForSemanticSegmentation requires config.out_indices to be a list of 4 integers, specifying which features to use from the backbone. One can use [3, 5, 7, 11] in case of a base-sized architecture.r[r)r4r5rRr:r*rrB out_indicesrrrConvTranspose2drErhGELUfpn1fpn2rfpn3 MaxPool2dfpn4r~ decode_headuse_auxiliary_headrauxiliary_headrArr7rrr5s*     z$BeitForSemanticSegmentation.__init__c Cstjj||jddddd}|dur"tjj||jddddd}t|jjd}|||}|}|durA|||} ||jj| 7}|S)NrrFr]) ignore_index)rrgrhr(rrBsemantic_loss_ignore_indexauxiliary_loss_weight) r6rYauxiliary_logitsrVupsampled_logitsupsampled_auxiliary_logitsr\ main_lossrXauxiliary_lossrrr compute_losss  z(BeitForSemanticSegmentation.compute_lossFrrrVrrrqrc s|dur|njj}|dur|njj}|dur"jjdkr"tdj||d||d}|r1|jn|d} fddt| D} |jdjj jj fd d| D} j j j jg} tt| D] } | | | | | | <qg| } d}jdur| }d}|dur| ||}|s|r| f|dd}n | f|d d}|dur|f|S|St|| |r|jnd|jd S) a labels (`torch.LongTensor` of shape `(batch_size, height, width)`, *optional*): Ground truth semantic segmentation maps for computing the loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels > 1`, a classification loss is computed (Cross-Entropy). Examples: ```python >>> from transformers import AutoImageProcessor, BeitForSemanticSegmentation >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> image_processor = AutoImageProcessor.from_pretrained("microsoft/beit-base-finetuned-ade-640-640") >>> model = BeitForSemanticSegmentation.from_pretrained("microsoft/beit-base-finetuned-ade-640-640") >>> inputs = image_processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> # logits are of shape (batch_size, num_labels, height, width) >>> logits = outputs.logits ```Nrz/The number of labels should be greater than oneTr_cs$g|]\}}|djjvr|qSr%)rBr)ridxfeaturer<rrrs$z7BeitForSemanticSegmentation.forward..rcs<g|]}|ddddddfddddqS)Nrrr[rZ)rfrer )r{patch_resolutionrrrs0r[rW)rBrGrrRrr*r9r$r(rMrKrrrrrrrrrrr#)r6rrrVrrrqrrHrrfeaturesopsrrYrrXr0r)r{rr6rr;sP$      z#BeitForSemanticSegmentation.forwardra)rrrrr5rrr*rrrrr;r@rrr7rrs2  rzM BEiT backbone, to be used with frameworks like DETR and MaskFormer. c sZeZdZfddZddZe   d dededBdedBd edBd ef d d Z Z S) BeitBackbonec stfddtjdD|_t|_t|jjj d|_ j rot |j jdkr3tdj}ttj||dddtj|jd ttj||ddd|_ttj||ddd|_t|_tjddd|_|dS) Ncsg|]}jqSr)rE)rrxrBrrr?sz)BeitBackbone.__init__..rrrzBeitBackbone requires config.out_indices to be a list of 4 integers, specifying which features to use from the backbone. One can use [3, 5, 7, 11] in case of a base-sized architecture.r[rr)r4r5rr num_featuresrArWr rJrr<add_fpnrrBrrrErrrrhbatch_norm_epsrrrrrrrrA)r6rBrEr7rrr5<s(    zBeitBackbone.__init__cCrBr3rCr<rrrrDYrEz!BeitBackbone.get_input_embeddingsNrrrrrr$cKs|dur|n|jj}|dur|n|jj}|dur|n|jj}|jd}||\}\}} |jdd} |j|d|| |d} |rB| jn| d} d} t|j | D].\}}||j vr||jj rw|ddddddf}| ddd}| |d|| }| |f7} qN|jjr|| d|| d|| d|| d g} t| } |s|r| f| dd}|S| f| dd}|St| |r| jnd| jd S) a Examples: ```python >>> from transformers import AutoImageProcessor, AutoBackbone >>> import torch >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> processor = AutoImageProcessor.from_pretrained("microsoft/beit-base-patch16-224") >>> model = AutoBackbone.from_pretrained( ... "microsoft/beit-base-patch16-224", out_features=["stage1", "stage2", "stage3", "stage4"] ... ) >>> inputs = processor(image, return_tensors="pt") >>> outputs = model(**inputs) >>> feature_maps = outputs.feature_maps >>> list(feature_maps[-1].shape) [1, 768, 14, 14] ```Nrr[T)rrrrrrrZr) feature_mapsr9r#)rBrGrrr(rWr<r9zip stage_names out_featuresreshape_hidden_statesrfrerrrrrrr r#)r6rrrrrrHr{rIryrzrrr9rstagersr0rrrr;\sT#        zBeitBackbone.forward)NNN) rrrr5rDrrrr r;r@rrr7rr6s$ r)r]rPrr:r)r)r F)Grcollections.abcrNr dataclassesrr*rrtorch.nnrrr- activationsrbackbone_utilsr modeling_layersr modeling_outputsr r r rrrmodeling_utilsr pytorch_utilsrutilsrrrconfiguration_beitr get_loggerrrrr>rr1Moduler2rArIrrrrrrrrrr r)r:r?rPr]rbrlrtr~rrr__all__rrrrs             _&T> ?SHG_;%%U; y