o eiHv@svdZddlZddlmZddlmZddlZddlmZddlm Z ddl m Z dd l mZdd lmZmZmZmZdd lmZmZdd lmZdd lmZmZmZmZmZddlm Z m!Z!ddl"m#Z#ddl$m%Z%e&e'Z(Gdddej)Z*Gdddej)Z+  dCdej)dej,dej,dej,dej,dBde-dBde-deefdd Z.Gd!d"d"ej)Z/Gd#d$d$ej)Z0Gd%d&d&ej)Z1Gd'd(d(ej)Z2Gd)d*d*ej)Z3Gd+d,d,eZ4Gd-d.d.ej)Z5eGd/d0d0eZ6eGd1d2d2e6Z7Gd3d4d4ej)Z8ed5d6Gd7d8d8e6Z9ed9d6Gd:d;d;e6Z:eedd>eZ;ed?d6Gd@dAdAe6Z)r@r)r;r'expand unsqueezetype_asr%r&r"rHr+rOr.)r0rPrQrO_r6r7r5 batch_size seq_length mask_tokensmask cls_tokensdistillation_tokensposition_embeddingr3r3r4forwardds   zDeiTEmbeddings.forward)FNF)__name__ __module__ __qualname____doc__rboolr r"TensorintrO BoolTensorr] __classcell__r3r3r1r4r+s +rcs6eZdZdZfddZdejdejfddZZS)r(z This class turns `pixel_values` of shape `(batch_size, num_channels, height, width)` into the initial `hidden_states` (patch embeddings) of shape `(batch_size, seq_length, hidden_size)` to be consumed by a Transformer. cst|j|j}}|j|j}}t|tjj r|n||f}t|tjj r)|n||f}|d|d|d|d}||_||_||_||_ t j ||||d|_ dS)Nrr) kernel_sizestride)rr image_sizer/ num_channelsr$ isinstance collectionsabcIterabler*rConv2d projection)r0rrjr/rkr$r*r1r3r4r s  zDeiTPatchEmbeddings.__init__rPrcCs<|j\}}}}||jkrtd||ddd}|S)NzeMake sure that the channel dimension of the pixel values match with the one set in the configuration.rr)r@rk ValueErrorrqflatten transpose)r0rPrVrkr6r7xr3r3r4r]s zDeiTPatchEmbeddings.forward) r_r`rarbr r"rdr]rgr3r3r1r4r(s r(modulequerykeyvalueattention_maskscalingr.kwargsc Ks|dur |dd}t||dd|}|dur||}tjj|dd}tjj|||jd}t||} | dd } | |fS)Nr8rrr>)ptrainingr) r;r"matmulrtrrEsoftmaxr.r contiguous) rwrxryrzr{r|r.r} attn_weights attn_outputr3r3r4eager_attention_forwards  rcsBeZdZdeffdd ZdejdeejejffddZZ S)DeiTSelfAttentionrcst|j|jdkrt|dstd|jd|jd||_|j|_t|j|j|_|j|j|_ |j |_ |jd|_ d|_ tj|j|j |jd|_tj|j|j |jd|_tj|j|j |jd|_dS) Nrembedding_sizezThe hidden size z4 is not a multiple of the number of attention heads .r~F)bias)rr r$num_attention_headshasattrrrrreattention_head_size all_head_sizeattention_probs_dropout_prob dropout_probr| is_causalrLinearqkv_biasrxryrzr0rr1r3r4r s"   zDeiTSelfAttention.__init__ hidden_statesrc Cs|jd}|d|j|jf}||j|dd}||j|dd}||j|dd}t |j j t }|||||d|j |j|jsHdn|jd\}} |dd|jf} || }|| fS)Nrr8rrrv)rr|r.)r@rrryrGrtrzrxr get_interfacer_attn_implementationrrr|rrr;rrC) r0rrV new_shape key_layer value_layer query_layerattention_interface context_layerattention_probsnew_context_layer_shaper3r3r4r]s*   zDeiTSelfAttention.forward) r_r`rarr r"rdtupler]rgr3r3r1r4rs(rcsBeZdZdZdeffdd Zdejdejdejfdd ZZ S) DeiTSelfOutputz The residual connection is defined in DeiTLayer instead of here (as is the case with other models), due to the layernorm applied before each block. rcs.tt|j|j|_t|j|_dSN) rr rrr$denser,r-r.rr1r3r4r  zDeiTSelfOutput.__init__r input_tensorrcC||}||}|Srrr.r0rrr3r3r4r]  zDeiTSelfOutput.forward r_r`rarbrr r"rdr]rgr3r3r1r4rs$rc8eZdZdeffdd ZdejdejfddZZS) DeiTAttentionrcs"tt||_t||_dSr)rr r attentionroutputrr1r3r4r s  zDeiTAttention.__init__rrcCs||\}}|||}|Sr)rr)r0rself_attn_outputrUrr3r3r4r] s zDeiTAttention.forward r_r`rarr r"rdr]rgr3r3r1r4rrcr)DeiTIntermediatercsDtt|j|j|_t|jt rt |j|_ dS|j|_ dSr) rr rrr$intermediate_sizerrl hidden_actstrrintermediate_act_fnrr1r3r4r s   zDeiTIntermediate.__init__rrcCrr)rr)r0rr3r3r4r]rzDeiTIntermediate.forwardrr3r3r1r4rsrcs>eZdZdeffdd ZdejdejdejfddZZS) DeiTOutputrcs.tt|j|j|_t|j|_ dSr) rr rrrr$rr,r-r.rr1r3r4r %rzDeiTOutput.__init__rrrcCs ||}||}||}|Srrrr3r3r4r]*s  zDeiTOutput.forwardrr3r3r1r4r$s$rcs<eZdZdZdeffdd ZdejdejfddZZ S) DeiTLayerz?This corresponds to the Block class in the timm implementation.rcsbt|j|_d|_t||_t||_t||_ t j |j |j d|_t j |j |j d|_dS)Nreps)rr chunk_size_feed_forward seq_len_dimrrr intermediaterrr LayerNormr$layer_norm_epslayernorm_beforelayernorm_afterrr1r3r4r 5s    zDeiTLayer.__init__rrcCs@||}||}||}||}||}|||}|Sr)rrrrr)r0rhidden_states_normattention_output layer_outputr3r3r4r]?s     zDeiTLayer.forwardrr3r3r1r4r2s rcs6eZdZdeffdd ZdejdefddZZ S) DeiTEncoderrcs:t|_tfddtjD|_d|_dS)Ncsg|]}tqSr3)r).0rUrr3r4 Usz(DeiTEncoder.__init__..F) rr rr ModuleListrangenum_hidden_layerslayergradient_checkpointingrr1rr4r Rs   zDeiTEncoder.__init__rrcCs&t|jD]\}}||}qt|dS)N)last_hidden_state) enumeraterr )r0ri layer_moduler3r3r4r]Xs  zDeiTEncoder.forward) r_r`rarr r"rdr r]rgr3r3r1r4rQsrc@sneZdZUeed<dZdZdZdZdgZ dZ dZ dZ dZ eedZedejejBejBd d fd d Zd S) DeiTPreTrainedModelrdeitrP)imageTr)r attentionsrwrNcCst|tjtjfr#tj|jd|jjd|j dur!t |j dSdSt|tj r7t |j t |jdSt|t r[t |jt |jt |j|jdur]t |jdSdSdS)zInitialize the weightsrv)meanstdN)rlrrrpinit trunc_normal_weightrinitializer_rangerzeros_rones_rr%r+r&r')r0rwr3r3r4 _init_weightsps         z!DeiTPreTrainedModel._init_weights)r_r`rar__annotations__base_model_prefixmain_input_nameinput_modalitiessupports_gradient_checkpointing_no_split_modules_supports_sdpa_supports_flash_attn_supports_flex_attn_supports_attention_backendrr_can_record_outputsr"no_gradrrrprrr3r3r3r4r_s &rcseZdZddedededdffdd Zdefd d Zee dd e   dd e j dBde j dBdedeedef ddZZS) DeiTModelTFradd_pooling_layerrrNcs\t|||_t||d|_t||_tj|j |j d|_ |r%t |nd|_ |dS)z add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer use_mask_token (`bool`, *optional*, defaults to `False`): Whether to use a mask token for masked image modeling. )rrN)rr rrr5rencoderrrr$r layernorm DeiTPoolerpooler post_init)r0rrrr1r3r4r s   zDeiTModel.__init__cCs|jjSr)r5r))r0r3r3r4get_input_embeddingsszDeiTModel.get_input_embeddings)tie_last_hidden_statesrPrQrOr}c Ks|durtd|jjjjj}|j|kr||}|j|||d}||}|j}| |}|j dur8| |nd} t || dS)z bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, num_patches)`, *optional*): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). Nz You have to specify pixel_valuesrQrO)r pooler_output) rrr5r)rqrdtypetorrrrr ) r0rPrQrOr}expected_dtypeembedding_outputencoder_outputssequence_output pooled_outputr3r3r4r]s     zDeiTModel.forward)TFNNF)r_r`rarrcr r(rrrrr"rdrfrrr r]rgr3r3r1r4rs( rcr)rrcs,tt|j|j|_t|j|_ dSr) rr rrr$pooler_output_sizerr pooler_act activationrr1r3r4r s zDeiTPooler.__init__rrcCs(|dddf}||}||}|S)Nr)rr)r0rfirst_token_tensorrr3r3r4r]s  zDeiTPooler.forwardrr3r3r1r4rrrad DeiT Model with a decoder on top for masked image modeling, as proposed in [SimMIM](https://huggingface.co/papers/2111.09886). Note that we provide a script to pre-train this model on custom data in our [examples directory](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-pretraining). ) custom_introc sdeZdZdeddffdd Zee   d dejdBdej dBd e d e e de f d d ZZS)DeiTForMaskedImageModelingrrNcsXt|t|ddd|_ttj|j|jd|j ddt |j|_ | dS)NFT)rrrr) in_channels out_channelsrh) rr rrr Sequentialrpr$encoder_striderk PixelShuffledecoderrrr1r3r4r s   z#DeiTForMaskedImageModeling.__init__FrPrQrOr}cKs|j|f||d|}|j}|ddddf}|j\}}} t|d} } |ddd|| | | }||} d} |durz|jj|jj }|d||}| |jj d |jj d d }t jj|| dd }|||d |jj} t| | |j|jd S) a bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, num_patches)`): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). Examples: ```python >>> from transformers import AutoImageProcessor, DeiTForMaskedImageModeling >>> import torch >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> image_processor = AutoImageProcessor.from_pretrained("facebook/deit-base-distilled-patch16-224") >>> model = DeiTForMaskedImageModeling.from_pretrained("facebook/deit-base-distilled-patch16-224") >>> num_patches = (model.config.image_size // model.config.patch_size) ** 2 >>> pixel_values = image_processor(images=image, return_tensors="pt").pixel_values >>> # create random boolean mask of shape (batch_size, num_patches) >>> bool_masked_pos = torch.randint(low=0, high=2, size=(1, num_patches)).bool() >>> outputs = model(pixel_values, bool_masked_pos=bool_masked_pos) >>> loss, reconstructed_pixel_values = outputs.loss, outputs.reconstruction >>> list(reconstructed_pixel_values.shape) [1, 3, 224, 224] ```rNrr8r9rrnone) reductiongh㈵>)lossreconstructionrr)rrr@rerDrCr rrjr/repeat_interleaverSrrrEl1_losssumrkr rr)r0rPrQrOr}outputsrrVsequence_lengthrkr6r7reconstructed_pixel_valuesmasked_im_lossr;rYreconstruction_lossr3r3r4r]s>'    z"DeiTForMaskedImageModeling.forwardr)r_r`rarr rrr"rdrfrcrrr r]rgr3r3r1r4rs$ rz DeiT Model transformer with an image classification head on top (a linear layer on top of the final hidden state of the [CLS] token) e.g. for ImageNet. c sdeZdZdeddffdd Zee   d dejdBdejdBd e d e e de f d d Z ZS)DeiTForImageClassificationrrNcsRt||j|_t|dd|_|jdkrt|j|jnt|_ | dSNF)rr) rr num_labelsrrrrr$Identity classifierrrr1r3r4r Cs $ z#DeiTForImageClassification.__init__FrPlabelsrOr}c Ksp|j|fd|i|}|j}||dddddf}d}|dur.|j|||jfi|}t|||j|jdS)a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the image classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Examples: ```python >>> from transformers import AutoImageProcessor, DeiTForImageClassification >>> import torch >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> torch.manual_seed(3) # doctest: +IGNORE_RESULT >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> # note: we are loading a DeiTForImageClassificationWithTeacher from the hub here, >>> # so the head will be randomly initialized, hence the predictions will be random >>> image_processor = AutoImageProcessor.from_pretrained("facebook/deit-base-distilled-patch16-224") >>> model = DeiTForImageClassification.from_pretrained("facebook/deit-base-distilled-patch16-224") >>> inputs = image_processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> logits = outputs.logits >>> # model predicts one of the 1000 ImageNet classes >>> predicted_class_idx = logits.argmax(-1).item() >>> print("Predicted class:", model.config.id2label[predicted_class_idx]) Predicted class: Polaroid camera, Polaroid Land camera ```rONr)r logitsrr)rrr loss_functionrr rr) r0rPrrOr}rrrr r3r3r4r]Os$+z"DeiTForImageClassification.forwardr)r_r`rarr rrr"rdrcrrr r]rgr3r3r1r4r<s$ rzC Output type of [`DeiTForImageClassificationWithTeacher`]. c@steZdZUdZdZejdBed<dZejdBed<dZ ejdBed<dZ e ejdBed<dZ e ejdBed<dS)+DeiTForImageClassificationWithTeacherOutputaj logits (`torch.FloatTensor` of shape `(batch_size, config.num_labels)`): Prediction scores as the average of the cls_logits and distillation logits. cls_logits (`torch.FloatTensor` of shape `(batch_size, config.num_labels)`): Prediction scores of the classification head (i.e. the linear layer on top of the final hidden state of the class token). distillation_logits (`torch.FloatTensor` of shape `(batch_size, config.num_labels)`): Prediction scores of the distillation head (i.e. the linear layer on top of the final hidden state of the distillation token). Nr cls_logitsdistillation_logitsrr) r_r`rarbrr" FloatTensorrr r!rrrr3r3r3r4rs  ra DeiT Model transformer with image classification heads on top (a linear layer on top of the final hidden state of the [CLS] token and a linear layer on top of the final hidden state of the distillation token) e.g. for ImageNet. .. warning:: This model supports inference-only. Fine-tuning with distillation (i.e. with a teacher) is not yet supported. c sXeZdZdeddffdd Zee  d dejdBde d e e de fd d Z ZS) %DeiTForImageClassificationWithTeacherrrNcsvt||j|_t|dd|_|jdkrt|j|jnt|_ |jdkr0t|j|jnt|_ | dSr) rr rrrrrr$rcls_classifierdistillation_classifierrrr1r3r4r s    z.DeiTForImageClassificationWithTeacher.__init__FrPrOr}c Ksv|j|fd|i|}|j}||dddddf}||dddddf}||d}t||||j|jdS)NrOrrr)rr r!rr)rrr$r%rrr) r0rPrOr}rrr r!rr3r3r4r]s$ z-DeiTForImageClassificationWithTeacher.forwardr^)r_r`rarr rrr"rdrcrrrr]rgr3r3r1r4r#s r#)rr#rrr)Nrv)>rbcollections.abcrmr dataclassesrr"rrr activationsrmodeling_layersrmodeling_outputsr r r r modeling_utilsr rprocessing_utilsrutilsrrrrr utils.genericrrutils.output_capturingrconfiguration_deitr get_loggerr_loggerModulerr(rdfloatrrrrrrrrrrrrrrr#__all__r3r3r3r4s           Y' 3 "> `O 1