o i@shdZddlZddlmZddlmZmZmZddl Z ddl m Z ddl m Z ddl mZdd lmZmZmZmZdd lmZmZdd lmZdd lmZmZdd lmZmZmZm Z m!Z!ddl"m#Z#m$Z$ddl%m&Z&e 'e(Z)Gddde j*Z+Gddde j*Z, dAde j*de j-de j-de j-dee j-de.de.fddZ/Gdd d e j*Z0Gd!d"d"e j*Z1Gd#d$d$e j*Z2Gd%d&d&e j*Z3Gd'd(d(e j*Z4Gd)d*d*eZ5Gd+d,d,e j*Z6eGd-d.d.eZ7eGd/d0d0e7Z8Gd1d2d2e j*Z9ed3d4Gd5d6d6e7Z:ed7d4Gd8d9d9e7Z;eed:d4Gd;d<dd?d?e7Z=gd@Z>dS)BzPyTorch DeiT model.N) dataclass)CallableOptionalUnion)nn)ACT2FN)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPoolingImageClassifierOutputMaskedImageModelingOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack) find_pruneable_heads_and_indicesprune_linear_layer) ModelOutputTransformersKwargsauto_docstringlogging torch_int)can_return_tuplecheck_model_inputs) DeiTConfigc sxeZdZdZddededdffdd Zd ejd e d e dejfd d Z  ddejde ej dedejfddZ ZS)DeiTEmbeddingszv Construct the CLS token, distillation token, position and patch embeddings. Optionally, also the mask token. Fconfiguse_mask_tokenreturnNcstttdd|j|_ttdd|j|_|r*ttdd|jnd|_ t ||_ |j j }ttd|d|j|_ t|j|_|j|_dS)Nr)super__init__r Parametertorchzeros hidden_size cls_tokendistillation_token mask_tokenDeiTPatchEmbeddingspatch_embeddings num_patchesposition_embeddingsDropouthidden_dropout_probdropout patch_size)selfrrr, __class__c/home/ubuntu/veenaModal/venv/lib/python3.10/site-packages/transformers/models/deit/modeling_deit.pyr"0s    zDeiTEmbeddings.__init__ embeddingsheightwidthc Cs|jdd}|jjdd}tjs||kr||kr|jS|jddddf}|jddddf}|jd}||j} ||j} t|d} |d| | |}|dddd}t j j || | fdd d }|dddd dd|}tj ||fdd S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing and 2 class embeddings. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rr N?rrbicubicF)sizemode align_cornersdim)shaper-r$jit is_tracingr1rreshapepermuter functional interpolateviewcat) r2r7r8r9r, num_positionsclass_and_dist_pos_embedpatch_pos_embedrA new_height new_widthsqrt_num_positionsr5r5r6interpolate_pos_encoding<s(     z'DeiTEmbeddings.interpolate_pos_encoding pixel_valuesbool_masked_posrQcCs|j\}}}}||}|\}} }|dur1|j|| d} |d| } |d| | | }|j|dd} |j|dd} t j | | |fdd}|j }|rW| |||}||}| |}|S)Nr:?rr@)rBr+r=r)expand unsqueezetype_asr'r(r$rJr-rQr0)r2rRrSrQ_r8r9r7 batch_size seq_length mask_tokensmask cls_tokensdistillation_tokensposition_embeddingr5r5r6forwardds   zDeiTEmbeddings.forward)F)NF)__name__ __module__ __qualname____doc__rboolr"r$TensorintrQr BoolTensorr` __classcell__r5r5r3r6r+s +rcs6eZdZdZfddZdejdejfddZZS)r*z This class turns `pixel_values` of shape `(batch_size, num_channels, height, width)` into the initial `hidden_states` (patch embeddings) of shape `(batch_size, seq_length, hidden_size)` to be consumed by a Transformer. cst|j|j}}|j|j}}t|tjj r|n||f}t|tjj r)|n||f}|d|d|d|d}||_||_||_||_ t j ||||d|_ dS)Nrr) kernel_sizestride)r!r" image_sizer1 num_channelsr& isinstance collectionsabcIterabler,rConv2d projection)r2rrlr1rmr&r,r3r5r6r"s  zDeiTPatchEmbeddings.__init__rRrcCs<|j\}}}}||jkrtd||ddd}|S)NzeMake sure that the channel dimension of the pixel values match with the one set in the configuration.r r)rBrm ValueErrorrsflatten transpose)r2rRrYrmr8r9xr5r5r6r`s zDeiTPatchEmbeddings.forward) rarbrcrdr"r$rfr`rir5r5r3r6r*s r*modulequerykeyvalueattention_maskscalingr0c Ks|t||dd|}tjj|dtjd|j}tjj |||j d}|dur,||}t||} | dd } | |fS)Nr:)rAdtype)ptrainingrr ) r$matmulrvrrGsoftmaxfloat32torr0r contiguous) ryrzr{r|r}r~r0kwargs attn_weights attn_outputr5r5r6eager_attention_forwards  rc sPeZdZdeffdd Z d dejdeejdeejejffdd Z Z S) DeiTSelfAttentionrcst|j|jdkrt|dstd|jd|jd||_|j|_t|j|j|_|j|j|_ |j |_ |jd|_ d|_ tj|j|j |jd|_tj|j|j |jd|_tj|j|j |jd|_dS) Nrembedding_sizezThe hidden size z4 is not a multiple of the number of attention heads .gF)bias)r!r"r&num_attention_headshasattrrtrrgattention_head_size all_head_sizeattention_probs_dropout_prob dropout_probr~ is_causalrLinearqkv_biasrzr{r|r2rr3r5r6r"s"   zDeiTSelfAttention.__init__N hidden_states head_maskrc Cs|jd}|d|j|jf}||j|dd}||j|dd}||j|dd}t}|j j dkr?t |j j }|||||||j |j |jsNdn|jd\} } | dd|jf} | | } | | fS) Nrr:rr eagerrx)rr~r0r)rBrrr{rIrvr|rzrr_attn_implementationrrr~rrr=rrE) r2rrrY new_shape key_layer value_layer query_layerattention_interface context_layerattention_probsnew_context_layer_shaper5r5r6r`s*     zDeiTSelfAttention.forwardN) rarbrcrr"r$rfrtupler`rir5r5r3r6rsrcsBeZdZdZdeffdd Zdejdejdejfdd ZZ S) DeiTSelfOutputz The residual connection is defined in DeiTLayer instead of here (as is the case with other models), due to the layernorm applied before each block. rcs.tt|j|j|_t|j|_dSr) r!r"rrr&denser.r/r0rr3r5r6r" zDeiTSelfOutput.__init__r input_tensorrcC||}||}|Srrr0r2rrr5r5r6r`  zDeiTSelfOutput.forward) rarbrcrdrr"r$rfr`rir5r5r3r6rs$rcsVeZdZdeffdd ZdeefddZd dej d e ej d ej fd d Z Z S) DeiTAttentionrcs*tt||_t||_t|_dSr)r!r"r attentionroutputset pruned_headsrr3r5r6r" s    zDeiTAttention.__init__headscCst|dkrdSt||jj|jj|j\}}t|jj||j_t|jj||j_t|jj ||j_ t|j j |dd|j _ |jjt||j_|jj|jj|j_ |j ||_dS)Nrrr@)lenrrrrrrrzr{r|rrrunion)r2rindexr5r5r6 prune_headss zDeiTAttention.prune_headsNrrrcCs |||\}}|||}|Sr)rr)r2rrself_attn_outputrXrr5r5r6r`$s zDeiTAttention.forwardr) rarbrcrr"rrgrr$rfrr`rir5r5r3r6r s*rc8eZdZdeffdd ZdejdejfddZZS)DeiTIntermediatercsDtt|j|j|_t|jt rt |j|_ dS|j|_ dSr) r!r"rrr&intermediate_sizerrn hidden_actstrrintermediate_act_fnrr3r5r6r",s   zDeiTIntermediate.__init__rrcCrr)rr)r2rr5r5r6r`4rzDeiTIntermediate.forward rarbrcrr"r$rfr`rir5r5r3r6r+srcs>eZdZdeffdd ZdejdejdejfddZZS) DeiTOutputrcs.tt|j|j|_t|j|_ dSr) r!r"rrrr&rr.r/r0rr3r5r6r"<rzDeiTOutput.__init__rrrcCs ||}||}||}|Srrrr5r5r6r`As  zDeiTOutput.forwardrr5r5r3r6r;s$rcsHeZdZdZdeffdd Zd dejdeejdejfd d Z Z S) DeiTLayerz?This corresponds to the Block class in the timm implementation.rcsbt|j|_d|_t||_t||_t||_ t j |j |j d|_t j |j |j d|_dS)Nreps)r!r"chunk_size_feed_forward seq_len_dimrrr intermediaterrr LayerNormr&layer_norm_epslayernorm_beforelayernorm_afterrr3r5r6r"Ls    zDeiTLayer.__init__NrrrcCsB||}|||}||}||}||}|||}|Sr)rrrrr)r2rrhidden_states_normattention_output layer_outputr5r5r6r`Vs     zDeiTLayer.forwardr) rarbrcrdrr"r$rfrr`rir5r5r3r6rIs* rcsBeZdZdeffdd Zd dejdeejdefdd Z Z S) DeiTEncoderrcs:t|_tfddtjD|_d|_dS)Ncsg|]}tqSr5)r).0rXrr5r6 lsz(DeiTEncoder.__init__..F) r!r"rr ModuleListrangenum_hidden_layerslayergradient_checkpointingrr3rr6r"is   zDeiTEncoder.__init__NrrrcCs<t|jD]\}}|dur||nd}|||}qt|dS)N)last_hidden_state) enumeraterr )r2rri layer_modulelayer_head_maskr5r5r6r`os  zDeiTEncoder.forwardr) rarbrcrr"r$rfrr r`rir5r5r3r6rhs(rc@sdeZdZUeed<dZdZdZdgZdZ dZ dZ dZ e edZdeejejejfdd fd d Zd S) DeiTPreTrainedModelrdeitrRTr)r attentionsryrNcCst|tjtjfr0tjj|jjt j d|j j d|jj |j_|jdur.|jjdSdSt|tjrE|jj|jjddSt|tri|jj|jj|jj|jdurk|jjdSdSdS)zInitialize the weightsrx)meanstdNrT)rnrrrrinit trunc_normal_weightdatarr$rrinitializer_rangerrzero_rfill_rr'r-r(r))r2ryr5r5r6 _init_weightss(         z!DeiTPreTrainedModel._init_weights)rarbrcr__annotations__base_model_prefixmain_input_namesupports_gradient_checkpointing_no_split_modules_supports_sdpa_supports_flash_attn_supports_flex_attn_supports_attention_backendrr_can_record_outputsrrrrrrrr5r5r5r6rws &rcseZdZddedededdffdd Zdefd d Zd d Ze dde    dde e j de e jde e j dedeedef ddZZS) DeiTModelTFradd_pooling_layerrrNcs\t|||_t||d|_t||_tj|j |j d|_ |r%t |nd|_ |dS)z add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer use_mask_token (`bool`, *optional*, defaults to `False`): Whether to use a mask token for masked image modeling. )rrN)r!r"rrr7rencoderrrr&r layernorm DeiTPoolerpooler post_init)r2rrrr3r5r6r"s   zDeiTModel.__init__cCs|jjSr)r7r+)r2r5r5r6get_input_embeddingsszDeiTModel.get_input_embeddingscCs*|D]\}}|jj|j|qdS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrrr)r2heads_to_prunerrr5r5r6 _prune_headsszDeiTModel._prune_heads)tie_last_hidden_statesrRrSrrQrc Ks|durtd|||jj}|jjjjj}|j|kr!| |}|j|||d}|j ||d}|j } | | } |j durB| | nd} t| | dS)z bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, num_patches)`, *optional*): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). Nz You have to specify pixel_values)rSrQ)r)r pooler_output)rt get_head_maskrrr7r+rsrrrrrrrr ) r2rRrSrrQrexpected_dtypeembedding_outputencoder_outputssequence_output pooled_outputr5r5r6r`s"   zDeiTModel.forward)TFNNNF)rarbrcrrer"r*rrrrrr$rfrhrrr r`rir5r5r3r6rs. rcr)rrcs,tt|j|j|_t|j|_ dSr) r!r"rrr&pooler_output_sizerr pooler_act activationrr3r5r6r"s zDeiTPooler.__init__rrcCs(|dddf}||}||}|S)Nr)rr )r2rfirst_token_tensorrr5r5r6r`s  zDeiTPooler.forwardrr5r5r3r6rsrad DeiT Model with a decoder on top for masked image modeling, as proposed in [SimMIM](https://huggingface.co/papers/2111.09886). Note that we provide a script to pre-train this model on custom data in our [examples directory](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-pretraining). ) custom_introcspeZdZdeddffdd Zee    ddeej deej d eej d e d e e def d d ZZS)DeiTForMaskedImageModelingrrNcsXt|t|ddd|_ttj|j|jd|j ddt |j|_ | dS)NFT)rrr r) in_channels out_channelsrj) r!r"rrr Sequentialrrr&encoder_striderm PixelShuffledecoderrrr3r5r6r"s   z#DeiTForMaskedImageModeling.__init__FrRrSrrQrcKs |j|f|||d|}|j}|ddddf}|j\}} } t| d} } |ddd|| | | }||} d}|dur{|jj|jj }|d||}| |jj d |jj d d }t jj|| dd }|||d |jj}t|| |j|jd S) a; bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, num_patches)`): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). Examples: ```python >>> from transformers import AutoImageProcessor, DeiTForMaskedImageModeling >>> import torch >>> from PIL import Image >>> import requests >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> image_processor = AutoImageProcessor.from_pretrained("facebook/deit-base-distilled-patch16-224") >>> model = DeiTForMaskedImageModeling.from_pretrained("facebook/deit-base-distilled-patch16-224") >>> num_patches = (model.config.image_size // model.config.patch_size) ** 2 >>> pixel_values = image_processor(images=image, return_tensors="pt").pixel_values >>> # create random boolean mask of shape (batch_size, num_patches) >>> bool_masked_pos = torch.randint(low=0, high=2, size=(1, num_patches)).bool() >>> outputs = model(pixel_values, bool_masked_pos=bool_masked_pos) >>> loss, reconstructed_pixel_values = outputs.loss, outputs.reconstruction >>> list(reconstructed_pixel_values.shape) [1, 3, 224, 224] ```)rSrrQNrr:r;rr none) reductiongh㈵>)lossreconstructionrr)rrrBrgrFrErrrlr1repeat_interleaverVrrrGl1_losssumrmr rr)r2rRrSrrQroutputsrrYsequence_lengthrmr8r9reconstructed_pixel_valuesmasked_im_lossr=r\reconstruction_lossr5r5r6r`s@&    z"DeiTForMaskedImageModeling.forwardr)rarbrcrr"rrrr$rfrhrerrr r`rir5r5r3r6rs* rz DeiT Model transformer with an image classification head on top (a linear layer on top of the final hidden state of the [CLS] token) e.g. for ImageNet. cspeZdZdeddffdd Zee    ddeej deej d eej d e d e e de f d d ZZS)DeiTForImageClassificationrrNcsRt||j|_t|dd|_|jdkrt|j|jnt|_ | dSNF)rr) r!r" num_labelsrrrrr&Identity classifierrrr3r5r6r"ls $ z#DeiTForImageClassification.__init__FrRrlabelsrQrc Ksr|j|f||d|}|j}||dddddf}d} |dur/|j|||jfi|} t| ||j|jdS)aZ labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the image classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Examples: ```python >>> from transformers import AutoImageProcessor, DeiTForImageClassification >>> import torch >>> from PIL import Image >>> import requests >>> torch.manual_seed(3) # doctest: +IGNORE_RESULT >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> # note: we are loading a DeiTForImageClassificationWithTeacher from the hub here, >>> # so the head will be randomly initialized, hence the predictions will be random >>> image_processor = AutoImageProcessor.from_pretrained("facebook/deit-base-distilled-patch16-224") >>> model = DeiTForImageClassification.from_pretrained("facebook/deit-base-distilled-patch16-224") >>> inputs = image_processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> logits = outputs.logits >>> # model predicts one of the 1000 ImageNet classes >>> predicted_class_idx = logits.argmax(-1).item() >>> print("Predicted class:", model.config.id2label[predicted_class_idx]) Predicted class: Polaroid camera, Polaroid Land camera ```rrQNr)rlogitsrr)rrr% loss_functionrr rr) r2rRrr&rQrrrr(rr5r5r6r`xs&*z"DeiTForImageClassification.forwardr)rarbrcrr"rrrr$rfrerrr r`rir5r5r3r6r!es* r!zC Output type of [`DeiTForImageClassificationWithTeacher`]. c@steZdZUdZdZeejed<dZ eejed<dZ eejed<dZ ee ejed<dZ ee ejed<dS)+DeiTForImageClassificationWithTeacherOutputaj logits (`torch.FloatTensor` of shape `(batch_size, config.num_labels)`): Prediction scores as the average of the cls_logits and distillation logits. cls_logits (`torch.FloatTensor` of shape `(batch_size, config.num_labels)`): Prediction scores of the classification head (i.e. the linear layer on top of the final hidden state of the class token). distillation_logits (`torch.FloatTensor` of shape `(batch_size, config.num_labels)`): Prediction scores of the distillation head (i.e. the linear layer on top of the final hidden state of the distillation token). Nr( cls_logitsdistillation_logitsrr)rarbrcrdr(rr$ FloatTensorrr+r,rrrr5r5r5r6r*s  r*a DeiT Model transformer with image classification heads on top (a linear layer on top of the final hidden state of the [CLS] token and a linear layer on top of the final hidden state of the distillation token) e.g. for ImageNet. .. warning:: This model supports inference-only. Fine-tuning with distillation (i.e. with a teacher) is not yet supported. c sdeZdZdeddffdd Zee   d deej deej d e d e e de f d d ZZS)%DeiTForImageClassificationWithTeacherrrNcsvt||j|_t|dd|_|jdkrt|j|jnt|_ |jdkr0t|j|jnt|_ | dSr") r!r"r#rrrrr&r$cls_classifierdistillation_classifierrrr3r5r6r"s    z.DeiTForImageClassificationWithTeacher.__init__FrRrrQrc Ksx|j|f||d|}|j}||dddddf}||dddddf}||d} t| |||j|jdS)Nr'rrr )r(r+r,rr)rrr/r0r*rr) r2rRrrQrrrr+r,r(r5r5r6r`s&  z-DeiTForImageClassificationWithTeacher.forward)NNF)rarbrcrr"rrrr$rfrerrr*r`rir5r5r3r6r.s$ r.)r!r.rrr)rx)?rdcollections.abcro dataclassesrtypingrrrr$r activationsrmodeling_layersr modeling_outputsr r r r modeling_utilsrrprocessing_utilsr pytorch_utilsrrutilsrrrrr utils.genericrrconfiguration_deitr get_loggerraloggerModulerr*rffloatrrrrrrrrrrrrr!r*r.__all__r5r5r5r6s       Y( 5 $M `O 3