o i@sdZddlmZddlZddlZddlmZddlZ ddl m Z ddl m Z mZmZmZddlmZmZmZmZmZmZdd lmZmZdd lmZmZmZmZm Z m!Z!d d l"m#Z#e $e%Z&d Z'dZ(gdZ)dZ*dZ+eGdddeZ,Gdddej-j.Z/Gdddej-j.Z0Gdddej-j.Z1Gdddej-j.Z2Gdddej-j.Z3Gdddej-j.Z4Gdd d ej-j.Z5Gd!d"d"ej-j.Z6Gd#d$d$ej-j.Z7eGd%d&d&ej-j.Z8Gd'd(d(eZ9d)Z:d*Z;ed+e:Gd,d-d-e9ZGd2d3d3ej-j.Z?ed4e:Gd5d6d6e9Z@ed7e:Gd8d9d9e9eZAed:e:Gd;d<dzTensorFlow DeiT model.) annotationsN) dataclass)get_tf_activation)TFBaseModelOutputTFBaseModelOutputWithPoolingTFImageClassifierOutputTFMaskedImageModelingOutput)TFPreTrainedModelTFSequenceClassificationLossget_initializerkeraskeras_serializable unpack_inputs) shape_liststable_softmax) ModelOutputadd_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardloggingreplace_return_docstrings) DeiTConfigrz(facebook/deit-base-distilled-patch16-224)riztabby, tabby catc@sNeZdZUdZdZded<dZded<dZded<dZded<dZ ded <dS) -TFDeiTForImageClassificationWithTeacherOutputa Output type of [`DeiTForImageClassificationWithTeacher`]. Args: logits (`tf.Tensor` of shape `(batch_size, config.num_labels)`): Prediction scores as the average of the cls_logits and distillation logits. cls_logits (`tf.Tensor` of shape `(batch_size, config.num_labels)`): Prediction scores of the classification head (i.e. the linear layer on top of the final hidden state of the class token). distillation_logits (`tf.Tensor` of shape `(batch_size, config.num_labels)`): Prediction scores of the distillation head (i.e. the linear layer on top of the final hidden state of the distillation token). hidden_states (`tuple(tf.Tensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `tf.Tensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (`tuple(tf.Tensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `tf.Tensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Ntf.Tensor | Nonelogits cls_logitsdistillation_logitsztuple[tf.Tensor] | None hidden_states attentions) __name__ __module__ __qualname____doc__r__annotations__rrr r!r'r']/home/ubuntu/.local/lib/python3.10/site-packages/transformers/models/deit/modeling_tf_deit.pyrBs     rcsJeZdZdZddfd d Zdd d ZdddZ  d d!ddZZS)"TFDeiTEmbeddingszv Construct the CLS token, distillation token, position and patch embeddings. Optionally, also the mask token. Fconfigruse_mask_tokenboolreturnNonec sDtjdi|||_||_t|dd|_tjj|j dd|_ dS)Npatch_embeddings)r*namedropoutr0r') super__init__r*r+TFDeiTPatchEmbeddingsr/r layersDropouthidden_dropout_probr1)selfr*r+kwargs __class__r'r(r4fs zTFDeiTEmbeddings.__init__NcCsV|jdd|jjftjddd|_|jdd|jjftjddd|_d|_|j r<|jdd|jjftjddd|_|j j }|jd|d|jjftjddd|_ |j rYdSd|_ t|d ddurt|j j|j dWdn1s|wYt|d ddurt|jj|jdWddS1swYdSdS) NrT cls_token)shape initializer trainabler0distillation_token mask_tokenposition_embeddingsr/r1) add_weightr* hidden_sizer initializerszerosr=rArBr+r/ num_patchesrDbuiltgetattrtf name_scoper0buildr1)r9 input_shaperIr'r'r(rNmsN   "zTFDeiTEmbeddings.build embeddings tf.Tensorheightintwidthc CsF|jdd}|jjdd}||kr||kr|jS|jdddddf}|jdddddf}|jddddddf}|jd} ||jj} ||jj} | d| d} } t|dtt|tt|| f}tj j |t| t| fdd}tj |gdd }t|dd| f}tj tj |dd tj |dd |gdd S) NrrCrg?bicubic)sizemethodrrCrrpermaxis)r>rDr* patch_sizerLreshaperSmathsqrtimageresize transposeconcat expand_dims) r9rPrRrTrI num_positionsclass_pos_embeddist_pos_embedpatch_pos_embeddimh0w0r'r'r(interpolate_pos_encodings(    z)TFDeiTEmbeddings.interpolate_pos_encoding pixel_valuesbool_masked_posrtrainingrncCs|j\}}}}||}t|\} } }|dur:t|j| | dg} tj|dd} tj| | jd} |d| | | }tj |j | dd} tj |j | dd}tj | ||fdd}|j }|rb||||}||}|j||d}|S) NrrUr\dtypeg?r)repeatsr]rq)r>r/rrLtilerBrfcastrsrepeatr=rArerDrnr1)r9rorprqrn_rRrTrP batch_size seq_length mask_tokensmask cls_tokensdistillation_tokensposition_embeddingr'r'r(calls" zTFDeiTEmbeddings.callF)r*rr+r,r-r.N)rPrQrRrSrTrSr-rQ)NFF) rorQrprrqr,rnr,r-rQ) r"r#r$r%r4rNrnr __classcell__r'r'r;r(r)as  'r)cs6eZdZdZdfdd Zdd d Zdd dZZS)r5z This class turns `pixel_values` of shape `(batch_size, num_channels, height, width)` into the initial `hidden_states` (patch embeddings) of shape `(batch_size, seq_length, hidden_size)` to be consumed by a Transformer. r*rr-r.c stjdi||j|j}}|j|j}}t|tjj r |n||f}t|tjj r-|n||f}|d|d|d|d}||_||_||_||_ t j j |||dd|_dS)Nrr projection) kernel_sizestridesr0r')r3r4 image_sizer^ num_channelsrF isinstance collectionsabcIterablerIr r6Conv2Dr)r9r*r:rr^rrFrIr;r'r(r4s  zTFDeiTPatchEmbeddings.__init__rorQcCs^t|\}}}}tr||jkrtd||}t|\}}}}t|||||f}|S)NzeMake sure that the channel dimension of the pixel values match with the one set in the configuration.)rrLexecuting_eagerlyr ValueErrorrr_)r9rorzrRrTrxr'r'r(rs zTFDeiTPatchEmbeddings.callNcCsn|jrdSd|_t|dddur5t|jj|jddd|jgWddS1s.wYdSdS)NTr)rJrKrLrMrr0rNrr9rOr'r'r(rN"zTFDeiTPatchEmbeddings.buildr*rr-r.)rorQr-rQrr"r#r$r%r4rrNrr'r'r;r(r5s   r5cs@eZdZdfdd Zdd d Z ddddZdddZZS)TFDeiTSelfAttentionr*rc stjd i||j|jdkrtd|jd|jd|j|_t|j|j|_|j|j|_t |j|_ t j j |jt|jdd|_t j j |jt|jdd|_t j j |jt|jdd|_t j j|jd |_||_dS) NrzThe hidden size (z6) is not a multiple of the number of attention heads ()queryunitskernel_initializerr0keyvaluerater')r3r4rFnum_attention_headsrrSattention_head_size all_head_sizer`rasqrt_att_head_sizer r6Denser initializer_rangerrrr7attention_probs_dropout_probr1r*r9r*r:r;r'r(r4s,  zTFDeiTSelfAttention.__init__tensorrQrzrSr-cCs,tj||d|j|jfd}tj|gddS)NrUrr>rrCrrrZ)rLr_rrrd)r9rrzr'r'r(transpose_for_scoressz(TFDeiTSelfAttention.transpose_for_scoresFr head_maskoutput_attentionsr,rqtuple[tf.Tensor]cCst|d}|j|d}|j|d}|j|d}|||} |||} |||} tj| | dd} tj|j| j d} t | | } t | dd}|j ||d}|durXt ||}t|| }tj|gd d }tj||d|jfd }|rz||f}|S|f}|S) NrinputsT) transpose_brrrU)rr]rrqrrZr)rrrrrrLmatmulrwrrsdividerr1multiplyrdr_r)r9r rrrqrzmixed_query_layermixed_key_layermixed_value_layer query_layer key_layer value_layerattention_scoresdkattention_probsattention_outputoutputsr'r'r(rs*            zTFDeiTSelfAttention.callNcCs|jrdSd|_t|dddur2t|jj|jdd|jjgWdn1s-wYt|dddur\t|j j|j dd|jjgWdn1sWwYt|dddurt|j j|j dd|jjgWddS1swYdSdS)NTrrr) rJrKrLrMrr0rNr*rFrrrr'r'r(rNGs "zTFDeiTSelfAttention.buildr*r)rrQrzrSr-rQr r rQrrQrr,rqr,r-rr)r"r#r$r4rrrNrr'r'r;r(rs   )rcs8eZdZdZdfdd Zddd dZdddZZS)TFDeiTSelfOutputz The residual connection is defined in TFDeiTLayer instead of here (as is the case with other models), due to the layernorm applied before each block. r*rc Jtjdi|tjj|jt|jdd|_tjj |j d|_ ||_ dSNdenserrr' r3r4r r6rrFr rrr7r8r1r*rr;r'r(r4]  zTFDeiTSelfOutput.__init__Fr rQ input_tensorrqr,r-cCs|j|d}|j||d}|SNrrrr1r9r rrqr'r'r(rfs zTFDeiTSelfOutput.callNcCn|jrdSd|_t|dddur5t|jj|jdd|jjgWddS1s.wYdSdSNTr rJrKrLrMrr0rNr*rFrr'r'r(rNlrzTFDeiTSelfOutput.buildrrr rQrrQrqr,r-rQrrr'r'r;r(rWs  rcs>eZdZdfdd ZddZ ddddZdddZZS)TFDeiTAttentionr*rc s2tjdi|t|dd|_t|dd|_dS)N attentionr2outputr')r3r4rself_attentionr dense_outputrr;r'r(r4wszTFDeiTAttention.__init__cCtrNotImplementedError)r9headsr'r'r( prune_heads}szTFDeiTAttention.prune_headsFrrQrrr,rqr-rcCs<|j||||d}|j|d||d}|f|dd}|S)Nr rrrqrr rrqr)rr)r9rrrrq self_outputsrrr'r'r(rs zTFDeiTAttention.callNcC|jrdSd|_t|dddur-t|jj|jdWdn1s(wYt|dddurUt|jj|jdWddS1sNwYdSdS)NTrr)rJrKrLrMrr0rNrrr'r'r(rN"zTFDeiTAttention.buildrr) rrQrrQrr,rqr,r-rr)r"r#r$r4rrrNrr'r'r;r(rvs  rc2eZdZd fdd Zddd Zdd d ZZS)TFDeiTIntermediater*rc sZtjdi|tjj|jt|jdd|_t |j t r$t |j |_ n|j |_ ||_dS)Nrrr')r3r4r r6rintermediate_sizer rrr hidden_actstrrintermediate_act_fnr*rr;r'r(r4s  zTFDeiTIntermediate.__init__r rQr-cCs|j|d}||}|S)Nr)rr)r9r r'r'r(rs  zTFDeiTIntermediate.callNcCrrrrr'r'r(rNrzTFDeiTIntermediate.buildrr rQr-rQrr"r#r$r4rrNrr'r'r;r(rs rcs4eZdZdfdd Zddd d ZdddZZS) TFDeiTOutputr*rc rrrrr;r'r(r4rzTFDeiTOutput.__init__Fr rQrrqr,r-cCs&|j|d}|j||d}||}|Srrrr'r'r(rs zTFDeiTOutput.callNcCrr) rJrKrLrMrr0rNr*rrr'r'r(rNrzTFDeiTOutput.buildrrrrrr'r'r;r(rs rcs:eZdZdZdfdd Z ddddZdddZZS) TFDeiTLayerz?This corresponds to the Block class in the timm implementation.r*rc sntjdi|t|dd|_t|dd|_t|dd|_tj j |j dd|_ tj j |j dd|_ ||_dS) Nrr2 intermediaterlayernorm_beforeepsilonr0layernorm_afterr')r3r4rrrrr deit_outputr r6LayerNormalizationlayer_norm_epsrrr*rr;r'r(r4s zTFDeiTLayer.__init__Fr rQrrr,rqr-rc Csn|j|j||d|||d}|d}||}|j||d}|j||d}|j|||d}|f|dd} | S)Nr)rrrrqr)r rqrr)rrrrr) r9r rrrqattention_outputsr layer_outputintermediate_outputrr'r'r(rs zTFDeiTLayer.callNcCs|jrdSd|_t|dddur-t|jj|jdWdn1s(wYt|dddurRt|jj|jdWdn1sMwYt|dddurwt|jj|jdWdn1srwYt|dddurt|j j|j dd|j j gWdn1swYt|dddurt|j j|j dd|j j gWddS1swYdSdS)NTrrrrr) rJrKrLrMrr0rNrrrr*rFrrr'r'r(rNs0"zTFDeiTLayer.buildrrrrrr'r'r;r(rs   rcs6eZdZdfdd Z ddddZdddZZS) TFDeiTEncoderr*rc s0tjdi|fddtjD|_dS)Ncsg|] }td|dqS)zlayer_._r2)r).0ir*r'r( sz*TFDeiTEncoder.__init__..r')r3r4rangenum_hidden_layerslayerrr;rr(r4szTFDeiTEncoder.__init__Fr rQrrr,output_hidden_states return_dictrqr-$TFBaseModelOutput | tuple[tf.Tensor]c Cs|rdnd}|r dnd}t|jD]"\} } |r||f}| ||| ||d} | d}|r3|| df}q|r;||f}|sItdd|||fDSt|||dS)Nr'rrrcss|] }|dur|VqdSrr')rvr'r'r( @sz%TFDeiTEncoder.call..)last_hidden_stater r!) enumeratertupler) r9r rrrrrqall_hidden_statesall_attentionsr layer_module layer_outputsr'r'r(r s,   zTFDeiTEncoder.callNc Csj|jrdSd|_t|dddur1|jD]}t|j |dWdn1s+wYqdSdS)NTr)rJrKrrLrMr0rN)r9rOrr'r'r(rNFs  zTFDeiTEncoder.buildrr)r rQrrQrr,rr,rr,rqr,r-rrrr'r'r;r(rs  &rcsjeZdZeZ d#d$fd d Zd%d dZddZddZe        d&d'dd Z d(d!d"Z Z S))TFDeiTMainLayerTFr*radd_pooling_layerr,r+r-r.c sjtjdi|||_t||dd|_t|dd|_tjj |j dd|_ |r0t |dd|_ dSd|_ dS) NrP)r+r0encoderr2 layernormrpoolerr')r3r4r*r)rPrr r r6rrr  TFDeiTPoolerr r9r*r r+r:r;r'r(r4Ts  zTFDeiTMainLayer.__init__r5cCs|jjSr)rPr/)r9r'r'r(get_input_embeddings`sz$TFDeiTMainLayer.get_input_embeddingscCr)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel r)r9heads_to_pruner'r'r( _prune_headscszTFDeiTMainLayer._prune_headscCs|durtdg|jj}|Sr)rr*r)r9rr'r'r( get_head_maskjszTFDeiTMainLayer.get_head_maskNrorrprr bool | Nonerrrnrq4TFBaseModelOutputWithPooling | tuple[tf.Tensor, ...]c Cs|dur|n|jj}|dur|n|jj}|dur|n|jj}|dur&tdt|d}||}|j||||d} |j | |||||d} | d} |j | |d} |j dur\|j | |dnd} |ss| durh| | fn| f} | | ddSt | | | j | jdS) Nz You have to specify pixel_valuesrY)rprqrn)rrrrrqrrur)r pooler_outputr r!)r*rruse_return_dictrrLrdrrPr r r rr r!)r9rorprrrrrnrqembedding_outputencoder_outputssequence_output pooled_output head_outputsr'r'r(rrsD   zTFDeiTMainLayer.callcCsL|jrdSd|_t|dddur-t|jj|jdWdn1s(wYt|dddurRt|jj|jdWdn1sMwYt|dddur|t|jj|jdd|j j gWdn1swwYt|dddurt|j j|j dWddS1swYdSdS)NTrPr r r ) rJrKrLrMrPr0rNr r r*rFr rr'r'r(rNs("zTFDeiTMainLayer.buildTFr*rr r,r+r,r-r.)r-r5NNNNNNFF)rorrprrrrrrrrrrnr,rqr,r-rr) r"r#r$r config_classr4rrrrrrNrr'r'r;r(r Ps$ =r c@seZdZdZeZdZdZdS)TFDeiTPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. deitroN)r"r#r$r%rr base_model_prefixmain_input_namer'r'r'r(r!s r!aR This model is a TensorFlow [keras.layers.Layer](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Layer). Use it as a regular TensorFlow Module and refer to the TensorFlow documentation for all matter related to general usage and behavior. Parameters: config ([`DeiTConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a Args: pixel_values (`tf.Tensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Pixel values can be obtained using [`AutoImageProcessor`]. See [`DeiTImageProcessor.__call__`] for details. head_mask (`tf.Tensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. interpolate_pos_encoding (`bool`, *optional*, defaults to `False`): Whether to interpolate the pre-trained position encodings. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. z^The bare DeiT Model transformer outputting raw hidden-states without any specific head on top.c sfeZdZ ddfd d Zeeeeee e d e d         d d!ddZ d"ddZ ZS)# TFDeiTModelTFr*rr r,r+r-r.c s*tj|fi|t|||dd|_dS)Nr"r r+r0)r3r4r r"rr;r'r(r4s zTFDeiTModel.__init__vision) checkpoint output_typer modalityexpected_outputNrorrprrrrrrnrq$tuple | TFBaseModelOutputWithPoolingc Cs|j||||||||d} | S)N)rorprrrrrnrq)r") r9rorprrrrrnrqrr'r'r(rs zTFDeiTModel.callcCsd|jrdSd|_t|dddur0t|jj|jdWddS1s)wYdSdS)NTr")rJrKrLrMr"r0rNrr'r'r(rN s"zTFDeiTModel.buildrrr)rorrprrrrrrrrrrnr,rqr,r-r,r)r"r#r$r4rrDEIT_INPUTS_DOCSTRINGr_CHECKPOINT_FOR_DOCr_CONFIG_FOR_DOC_EXPECTED_OUTPUT_SHAPErrNrr'r'r;r(r%s,  r%cr)rr*rc s<tjdi|tjj|jt|j|jdd|_ ||_ dS)Nr)rr activationr0r') r3r4r r6rpooler_output_sizer r pooler_actrr*rr;r'r(r4+s zTFDeiTPooler.__init__r rQr-cCs |dddf}|j|d}|S)Nrr)r)r9r first_token_tensorrr'r'r(r6s zTFDeiTPooler.callNcCrrrrr'r'r(rN>rzTFDeiTPooler.buildrrrrr'r'r;r(r*s rcs,eZdZdZd fdd Zd d d ZZS)TFDeitPixelShufflez0TF layer implementation of torch.nn.PixelShuffleupscale_factorrSr-r.c s<tjdi|t|tr|dkrtd|||_dS)NrCz1upscale_factor must be an integer value >= 2 got r')r3r4rrSrr6)r9r6r:r;r'r(r4Js zTFDeitPixelShuffle.__init__rrQcs~|}t|\}}}}|jdt|tfddtDg}tj|t||dgdd}tjj ||jdd}|S) NrCcs&g|]}tD]}||qqSr')r)rrjblock_size_squared output_depthr'r(rZs&z+TFDeitPixelShuffle.call..rrU)paramsindices batch_dimsNHWC) block_size data_format) rr6rSrLconstantrgatherrvnndepth_to_space)r9rr rzrynum_input_channels permutationr'r8r(rPs  zTFDeitPixelShuffle.call)r6rSr-r.)rrQr-rQ)r"r#r$r%r4rrr'r'r;r(r5Gsr5cs4eZdZdfdd Zddd d ZdddZZS) TFDeitDecoderr*rr-r.c sLtjdi|tjj|jd|jddd|_t|jdd|_ ||_ dS)NrCr0)filtersrr01r2r') r3r4r r6rencoder_striderconv2dr5 pixel_shuffler*rr;r'r(r4bs  zTFDeitDecoder.__init__FrrQrqr,cCs|}||}||}|Sr)rLrM)r9rrqr r'r'r(rjs  zTFDeitDecoder.callNcCs|jrdSd|_t|dddur3t|jj|jddd|jjgWdn1s.wYt|dddur[t|j j|j dWddS1sTwYdSdS)NTrLrM) rJrKrLrMrLr0rNr*rFrMrr'r'r(rNps"zTFDeitDecoder.buildrr)rrQrqr,r-rQrrr'r'r;r(rGas rGz~DeiT Model with a decoder on top for masked image modeling, as proposed in [SimMIM](https://huggingface.co/papers/2111.09886).cs\eZdZdfdd Zeeeeee d       ddddZ dddZ Z S)TFDeiTForMaskedImageModelingr*rr-r.cs0t|t|dddd|_t|dd|_dS)NFTr"r&decoderr2)r3r4r r"rGrOr9r*r;r'r(r4s z%TFDeiTForMaskedImageModeling.__init__r)r NFrorrprrrrrrnr,rq#tuple | TFMaskedImageModelingOutputc  Cs|dur|n|jj}|j||||||||d} | d} | ddddf} t| \} } } t| d}}t| | ||| f} |j| |d}t|d}d}|dur|jj |jj }t|d||f}t ||jj d}t ||jj d }t |d}t |tj}tjt|d t|d }t |d}t||}t|d |jj}||}t|d }|s|f| dd}|dur|f|S|St||| j| jd S)a bool_masked_pos (`tf.Tensor` of type bool and shape `(batch_size, num_patches)`): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). Returns: Examples: ```python >>> from transformers import AutoImageProcessor, TFDeiTForMaskedImageModeling >>> import tensorflow as tf >>> from PIL import Image >>> import requests >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> image_processor = AutoImageProcessor.from_pretrained("facebook/deit-base-distilled-patch16-224") >>> model = TFDeiTForMaskedImageModeling.from_pretrained("facebook/deit-base-distilled-patch16-224") >>> num_patches = (model.config.image_size // model.config.patch_size) ** 2 >>> pixel_values = image_processor(images=image, return_tensors="tf").pixel_values >>> # create random boolean mask of shape (batch_size, num_patches) >>> bool_masked_pos = tf.cast(tf.random.uniform((1, num_patches), minval=0, maxval=2, dtype=tf.int32), tf.bool) >>> outputs = model(pixel_values, bool_masked_pos=bool_masked_pos) >>> loss, reconstructed_pixel_values = outputs.loss, outputs.reconstruction >>> list(reconstructed_pixel_values.shape) [1, 3, 224, 224] ```N)rprrrrrnrqrrrUg?ru)rrrrCrC)rrCrrgh㈵>)r)lossreconstructionr r!)r*rr"rrSrLr_rOrdrr^rxrfrwfloat32r lossesmean_absolute_error reduce_sumrr r r!)r9rorprrrrrnrqrrrzsequence_lengthrrRrTreconstructed_pixel_valuesmasked_im_lossrWr}reconstruction_loss total_lossnum_masked_pixelsrr'r'r(rsX+       z!TFDeiTForMaskedImageModeling.callcCr)NTr"rO)rJrKrLrMr"r0rNrOrr'r'r(rNrz"TFDeiTForMaskedImageModeling.buildrr)rorrprrrrrrrrrrnr,rqr,r-rRr) r"r#r$r4rrr-rr r/rrNrr'r'r;r(rN|s crNz DeiT Model transformer with an image classification head on top (a linear layer on top of the final hidden state of the [CLS] token) e.g. for ImageNet. cs\eZdZdfdd Zeeeeee d        ddddZ dddZ Z S)TFDeiTForImageClassificationr*rcs\t||j|_t|ddd|_|jdkr tjj|jddntjjddd|_ ||_ dS)NFr"r r0r classifierr2linear) r3r4 num_labelsr r"r r6r Activationrar*rPr;r'r(r4s   z%TFDeiTForImageClassification.__init__rQNFrorrlabelsrrrrrnr,rqr-#tf.Tensor | TFImageClassifierOutputc  Cs|dur|n|jj}|j|||||||d} | d} || dddddf} |dur.dn||| } |sJ| f| dd} | durH| f| S| St| | | j| jdS)a labels (`tf.Tensor` of shape `(batch_size,)`, *optional*): Labels for computing the image classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Returns: Examples: ```python >>> from transformers import AutoImageProcessor, TFDeiTForImageClassification >>> import tensorflow as tf >>> from PIL import Image >>> import requests >>> keras.utils.set_random_seed(3) # doctest: +IGNORE_RESULT >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> # note: we are loading a TFDeiTForImageClassificationWithTeacher from the hub here, >>> # so the head will be randomly initialized, hence the predictions will be random >>> image_processor = AutoImageProcessor.from_pretrained("facebook/deit-base-distilled-patch16-224") >>> model = TFDeiTForImageClassification.from_pretrained("facebook/deit-base-distilled-patch16-224") >>> inputs = image_processor(images=image, return_tensors="tf") >>> outputs = model(**inputs) >>> logits = outputs.logits >>> # model predicts one of the 1000 ImageNet classes >>> predicted_class_idx = tf.math.argmax(logits, axis=-1)[0] >>> print("Predicted class:", model.config.id2label[int(predicted_class_idx)]) Predicted class: little blue heron, Egretta caerulea ```Nrrrrrnrqrr)rSrr r!)r*rr"rahf_compute_lossrr r!)r9rorrerrrrnrqrrrrSrr'r'r(rs,/ z!TFDeiTForImageClassification.callcCs|jrdSd|_t|dddur-t|jj|jdWdn1s(wYt|dddurZt|jj|jdd|jj gWddS1sSwYdSdS)NTr"ra) rJrKrLrMr"r0rNrar*rFrr'r'r(rN]s"z"TFDeiTForImageClassification.buildrr)rorrrrerrrrrrrrnr,rqr,r-rfr) r"r#r$r4rrr-rrr/rrNrr'r'r;r(r_s Jr_a DeiT Model transformer with image classification heads on top (a linear layer on top of the final hidden state of the [CLS] token and a linear layer on top of the final hidden state of the distillation token) e.g. for ImageNet. .. warning:: This model supports inference-only. Fine-tuning with distillation (i.e. with a teacher) is not yet supported. cs^eZdZdfdd Zeeeeee e e d      ddddZ dddZ ZS)'TFDeiTForImageClassificationWithTeacherr*rr-r.cst||j|_t|ddd|_|jdkr tjj|jddntjjddd|_ |jdkr7tjj|jddntjjddd|_ ||_ dS) NFr"r`rcls_classifierr2rbdistillation_classifier) r3r4rcr r"r r6rrdrjrkr*rPr;r'r(r4vs    z0TFDeiTForImageClassificationWithTeacher.__init__)r(r)r r+NFrorrrrrrrnr,rq5tuple | TFDeiTForImageClassificationWithTeacherOutputc Cs|dur|n|jj}|j|||||||d}|d} || dddddf} || dddddf} | | d} |sK| | | f|dd} | St| | | |j|jdS)NrgrrrC)rrrr r!)r*rr"rjrkrr r!)r9rorrrrrnrqrrrrrrr'r'r(rs0  z,TFDeiTForImageClassificationWithTeacher.callcCs |jrdSd|_t|dddur-t|jj|jdWdn1s(wYt|dddurWt|jj|jdd|jj gWdn1sRwYt|dddurt|j j|j dd|jj gWddS1s}wYdSdS)NTr"rjrk) rJrKrLrMr"r0rNrjr*rFrkrr'r'r(rNs "z-TFDeiTForImageClassificationWithTeacher.buildr)NNNNNFF)rorrrrrrrrrrnr,rqr,r-rlr)r"r#r$r4rrr-r_IMAGE_CLASS_CHECKPOINTrr/_IMAGE_CLASS_EXPECTED_OUTPUTrrNrr'r'r;r(riis& *ri)r_rirNr%r!)Dr% __future__rcollections.abcrr` dataclassesr tensorflowrLactivations_tfrmodeling_tf_outputsrrrr modeling_tf_utilsr r r r rrtf_utilsrrutilsrrrrrrconfiguration_deitr get_loggerr"loggerr/r.r0rmrnrr6Layerr)r5rrrrrrrr r!DEIT_START_DOCSTRINGr-r%rr5rGrNr_ri__all__r'r'r'r(st       m.[(D6r  4yh  U