o eie@s6dZddlZddlZddlmZddlZddlmZddlmZ ddl m Z ddl m Z dd lmZmZmZmZdd lmZmZdd lmZdd lmZmZmZmZdd lmZmZddl m!Z!ddl"m#Z#e$e%Z&Gdddej'Z(Gdddej'Z)  dr%r#expand unsqueezetype_asr!rrFrMr'r*) r,rNrOrM batch_size num_channelsr2r3r1 seq_length mask_tokensmask cls_tokensr/r/r0forwardds   zViTEmbeddings.forwardF)NF)__name__ __module__ __qualname____doc__rboolrrTensorintrM BoolTensorrY __classcell__r/r/r-r0r+s +rcsBeZdZdZdeffdd Zd dejdedejfd d Z Z S) r$z This class turns `pixel_values` of shape `(batch_size, num_channels, height, width)` into the initial `hidden_states` (patch embeddings) of shape `(batch_size, seq_length, hidden_size)` to be consumed by a Transformer. rcst|j|j}}|j|j}}t|tjj r|n||f}t|tjj r)|n||f}|d|d|d|d}||_||_||_||_ t j ||||d|_ dS)Nrr) kernel_sizestride)rr image_sizer+rTr isinstance collectionsabcIterabler&rConv2d projection)r,rrfr+rTr r&r-r/r0rs  zViTPatchEmbeddings.__init__FrNrMr4c Cs|j\}}}}||jkrtd|jd|d|s?||jdks(||jdkr?td|d|d|jdd|jdd ||d dd }|S) NzoMake sure that the channel dimension of the pixel values match with the one set in the configuration. Expected z but got .rrzInput image size (*z) doesn't match model (z).r7)r>rT ValueErrorrfrlflatten transpose)r,rNrMrSrTr2r3r1r/r/r0rYs( zViTPatchEmbeddings.forwardrZ) r[r\r]r^rrrr`r_rYrcr/r/r-r0r$s$r$modulequerykeyvalueattention_maskscalingr*kwargsc Ks|dur |dd}t||dd|}|dur||}tjj|dd}tjj|||jd}t||} | dd } | |fS)Nr5r7rr<)ptrainingr) r9rmatmulrqrrCsoftmaxr*r| contiguous) rsrtrurvrwrxr*ry attn_weights attn_outputr/r/r0eager_attention_forwards  rcsBeZdZdeffdd ZdejdeejejffddZZ S)ViTSelfAttentionrcst|j|jdkrt|dstd|jd|jd||_|j|_t|j|j|_|j|j|_ |j |_ |jd|_ d|_ tj|j|j |jd|_tj|j|j |jd|_tj|j|j |jd|_dS) Nrembedding_sizezThe hidden size z4 is not a multiple of the number of attention heads rmrzF)bias)rrr num_attention_headshasattrrorraattention_head_size all_head_sizeattention_probs_dropout_prob dropout_probrx is_causalrLinearqkv_biasrtrurvr,rr-r/r0rs"   zViTSelfAttention.__init__ hidden_statesr4c Cs|jd}|d|j|jf}||j|dd}||j|dd}||j|dd}t |j j t }|||||d|j |j|jsHdn|jd\}} |dd|jf} || }|| fS)Nrr5rr7rr)rrxr*)r>rrrurErqrvrtr get_interfacer_attn_implementationrrrxr|rr9rrA) r,rrS new_shape key_layer value_layer query_layerattention_interface context_layerattention_probsnew_context_layer_shaper/r/r0rYs*   zViTSelfAttention.forward) r[r\r]rrrr`tuplerYrcr/r/r-r0rs(rcsBeZdZdZdeffdd Zdejdejdejfdd ZZ S) ViTSelfOutputz The residual connection is defined in ViTLayer instead of here (as is the case with other models), due to the layernorm applied before each block. rcs.tt|j|j|_t|j|_dSN) rrrrr denser(r)r*rr-r/r0r zViTSelfOutput.__init__r input_tensorr4cC||}||}|Srrr*r,rrr/r/r0rY  zViTSelfOutput.forward r[r\r]r^rrrr`rYrcr/r/r-r0rs$rc8eZdZdeffdd ZdejdejfddZZS) ViTAttentionrcs"tt||_t||_dSr)rrr attentionroutputrr-r/r0r s  zViTAttention.__init__rr4cCs||\}}|||}|Sr)rr)r,rself_attn_output_rr/r/r0rYs zViTAttention.forward r[r\r]rrrr`rYrcr/r/r-r0r rcr)ViTIntermediatercsDtt|j|j|_t|jt rt |j|_ dS|j|_ dSr) rrrrr intermediate_sizerrg hidden_actstrrintermediate_act_fnrr-r/r0rs   zViTIntermediate.__init__rr4cCrr)rr)r,rr/r/r0rYrzViTIntermediate.forwardrr/r/r-r0rsrcs>eZdZdeffdd ZdejdejdejfddZZS) ViTOutputrcs.tt|j|j|_t|j|_ dSr) rrrrrr rr(r)r*rr-r/r0r&rzViTOutput.__init__rrr4cCs ||}||}||}|Srrrr/r/r0rY+s  zViTOutput.forwardrr/r/r-r0r%s$rcs<eZdZdZdeffdd ZdejdejfddZZ S) ViTLayerz?This corresponds to the Block class in the timm implementation.rcsbt|j|_d|_t||_t||_t||_ t j |j |j d|_t j |j |j d|_dS)Nreps)rrchunk_size_feed_forward seq_len_dimrrr intermediaterrr LayerNormr layer_norm_epslayernorm_beforelayernorm_afterrr-r/r0r5s    zViTLayer.__init__rr4cCs@||}||}||}||}||}|||}|Sr)rrrrr)r,rhidden_states_normattention_output layer_outputr/r/r0rY?s     zViTLayer.forwardrr/r/r-r0r2s rcs6eZdZdeffdd ZdejdefddZZ S) ViTEncoderrcs:t|_tfddtjD|_d|_dS)Ncsg|]}tqSr/)r).0rrr/r0 Tsz'ViTEncoder.__init__..F) rrrr ModuleListrangenum_hidden_layerslayergradient_checkpointingrr-rr0rQs   zViTEncoder.__init__rr4cCs&t|jD]\}}||}qt|dS)N)last_hidden_state) enumeraterr)r,ri layer_moduler/r/r0rYWs  zViTEncoder.forward) r[r\r]rrrr`rrYrcr/r/r-r0rPsrc@sleZdZUeed<dZdZdZdZddgZ dZ dZ dZ dZ eedZed ejejBejBfd d Zd S) ViTPreTrainedModelrvitrN)imageTrr)r attentionsrscCst|tjtjfr#tj|jd|jjd|j dur!t |j dSdSt|tj r7t |j t |jdSt|t r_tj|jd|jjdtj|jd|jjd|jdurat |jdSdSdS)zInitialize the weightsrr)meanstdN)rgrrrkinit trunc_normal_weightrinitializer_rangerzeros_rones_rr'r!r#)r,rsr/r/r0 _init_weightsos     z ViTPreTrainedModel._init_weightsN)r[r\r]r__annotations__base_model_prefixmain_input_nameinput_modalitiessupports_gradient_checkpointing_no_split_modules_supports_sdpa_supports_flash_attn_supports_flex_attn_supports_attention_backendrr_can_record_outputsrno_gradrrrkrrr/r/r/r0r^s "rcseZdZddededeffdd Zdefd d Zee dd e dd e j d Bde j d Bded Bdeedef ddZZS)ViTModelTFradd_pooling_layerrcs\t|||_t||d|_t||_tj|j |j d|_ |r%t |nd|_ |dS)z add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer use_mask_token (`bool`, *optional*, defaults to `False`): Whether to use a mask token for masked image modeling. )rrN)rrrrr1rencoderrrr r layernorm ViTPoolerpooler post_init)r,rrrr-r/r0rs   zViTModel.__init__r4cCs|jjSr)r1r%)r,r/r/r0get_input_embeddingsszViTModel.get_input_embeddings)tie_last_hidden_statesNrNrOrMryc Ks|durtd|jjjjj}|j|kr||}|j|||d}||}|j}| |}|j dur8| |nd} t || dS)z bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, num_patches)`, *optional*): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). Nz You have to specify pixel_valuesrOrM)r pooler_output) ror1r%rlrdtypetorrrrr ) r,rNrOrMryexpected_dtypeembedding_outputencoder_outputssequence_output pooled_outputr/r/r0rYs     zViTModel.forward)TFNNN)r[r\r]rr_rr$rrrrrr`rbrrr rYrcr/r/r-r0rs(rcr)rrcs,tt|j|j|_t|j|_ dSr) rrrrr pooler_output_sizerr pooler_act activationrr-r/r0rs zViTPooler.__init__rr4cCs(|dddf}||}||}|S)Nr)rr)r,rfirst_token_tensorrr/r/r0rYs  zViTPooler.forwardrr/r/r-r0rrrac ViT Model with a decoder on top for masked image modeling, as proposed in [SimMIM](https://huggingface.co/papers/2111.09886). Note that we provide a script to pre-train this model on custom data in our [examples directory](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-pretraining). ) custom_introc sdeZdZdeffdd Zee   d dejdBdej dBde dBde e d e f d d ZZS) ViTForMaskedImageModelingrcsXt|t|ddd|_ttj|j|jd|j ddt |j|_ | dS)NFT)rrr7r) in_channels out_channelsrd) rrrrr Sequentialrkr encoder_striderT PixelShuffledecoderrrr-r/r0rs   z"ViTForMaskedImageModeling.__init__NrNrOrMryr4cKs@|dur|jj|jjkrtd|jjd|jjd|j|f||d|}|j}|ddddf}|j\}}} t|d} } | dd d || | | }| |} d} |dur|jj |jj}| d ||}| |jjd |jjd d}tjj|| d d }|||d |jj} t| | |j|jdS)a| bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, num_patches)`): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). Examples: ```python >>> from transformers import AutoImageProcessor, ViTForMaskedImageModeling >>> import torch >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> image_processor = AutoImageProcessor.from_pretrained("google/vit-base-patch16-224-in21k") >>> model = ViTForMaskedImageModeling.from_pretrained("google/vit-base-patch16-224-in21k") >>> num_patches = (model.config.image_size // model.config.patch_size) ** 2 >>> pixel_values = image_processor(images=image, return_tensors="pt").pixel_values >>> # create random boolean mask of shape (batch_size, num_patches) >>> bool_masked_pos = torch.randint(low=0, high=2, size=(1, num_patches)).bool() >>> outputs = model(pixel_values, bool_masked_pos=bool_masked_pos) >>> loss, reconstructed_pixel_values = outputs.loss, outputs.reconstruction >>> list(reconstructed_pixel_values.shape) [1, 3, 224, 224] ```NzWhen `bool_masked_pos` is provided, `patch_size` must be equal to `encoder_stride` to ensure that the reconstructed image has the same dimensions as the input. Got `patch_size` = z and `encoder_stride` = rmrrr6rr7r5none) reductiongh㈵>)lossreconstructionrr)rr+rrorrr>mathfloorrBrArrfrepeat_interleaverQrrrCl1_losssumrTr rr)r,rNrOrMryoutputsrrSsequence_lengthrTr2r3reconstructed_pixel_valuesmasked_im_lossr9rWreconstruction_lossr/r/r0rYsN'    z!ViTForMaskedImageModeling.forwardr)r[r\r]rrrrrr`rbr_rrr rYrcr/r/r-r0rs$ ra ViT Model transformer with an image classification head on top (a linear layer on top of the final hidden state of the [CLS] token) e.g. for ImageNet. Note that it's possible to fine-tune ViT on higher resolution images than the ones it has been trained on, by setting `interpolate_pos_encoding` to `True` in the forward of the model. This will interpolate the pre-trained position embeddings to the higher resolution. c sdeZdZdeffdd Zee   d dejdBdejdBde dBde e d e f d d Z ZS) ViTForImageClassificationrcsRt||j|_t|dd|_|jdkrt|j|jnt|_ | dS)NF)rr) rr num_labelsrrrrr Identity classifierrrr-r/r0rMs $ z"ViTForImageClassification.__init__NrNlabelsrMryr4c Kst|j|fd|i|}|j}|dddddf}||}d} |dur0|j|||jfi|} t| ||j|jdS)a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the image classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). rMNr)r logitsrr)rrr loss_functionrr rr) r,rNrrMryrrrrr r/r/r0rYYs& z!ViTForImageClassification.forwardr)r[r\r]rrrrrr`r_rrr rYrcr/r/r-r0r>s$ r)rrrr)Nrr):r^collections.abcrhr rrrrr activationsrmodeling_layersrmodeling_outputsrr r r modeling_utilsr r processing_utilsrutilsrrrr utils.genericrrutils.output_capturingrconfiguration_vitr get_loggerr[loggerModulerr$r`floatrrrrrrrrrrrrr__all__r/r/r/r0sv          X. 2  !; g 3