o wiҫ@sdZddlZddlmZddlmZddlmZm Z m Z ddl Z ddl Z ddlZ ddl mZddlmZmZmZdd lmZdd lmZdd lmZmZdd lmZmZdd lmZm Z ddl!m"Z"m#Z#m$Z$ddl%m&Z&m'Z'ddl(m)Z)e$*e+Z,ee#ddGddde"Z-ee#ddGddde"Z.ddZ/Gdddej0Z1Gdddej0Z2 dDd ej0d!e j3d"e j3d#e j3d$e e j3d%e4d&e4fd'd(Z5Gd)d*d*ej0Z6Gd+d,d,ej0Z7Gd-d.d.ej0Z8Gd/d0d0ej0Z9Gd1d2d2ej0Z:Gd3d4d4eZ;Gd5d6d6ej0ZGd;d<dd?d?e=Z@e#d@dGdAdBdBe=ZAgdCZBdS)Ez,PyTorch VideoMAE (masked autoencoder) model.N)deepcopy) dataclass)CallableOptionalUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)GradientCheckpointingLayer)BaseModelOutputImageClassifierOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel) find_pruneable_heads_and_indicesprune_linear_layer) ModelOutputauto_docstringlogging)IMAGENET_DEFAULT_MEANIMAGENET_DEFAULT_STD)VideoMAEConfigz[ Class for VideoMAEDecoder's outputs, with potential hidden states and attentions. ) custom_introc@sPeZdZUdZdZeejed<dZ ee ejed<dZ ee ejed<dS)VideoMAEDecoderOutputz logits (`torch.FloatTensor` of shape `(batch_size, patch_size ** 2 * num_channels)`): Pixel reconstruction logits. Nlogits hidden_states attentions) __name__ __module__ __qualname____doc__rrtorch FloatTensor__annotations__rtuplerr(r(k/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/transformers/models/videomae/modeling_videomae.pyr-s rzb Class for VideoMAEForPreTraining's outputs, with potential hidden states and attentions. c@sbeZdZUdZdZeejed<dZ eejed<dZ ee ejed<dZ ee ejed<dS)VideoMAEForPreTrainingOutputz loss (`torch.FloatTensor` of shape `(1,)`): Pixel reconstruction loss. logits (`torch.FloatTensor` of shape `(batch_size, patch_size ** 2 * num_channels)`): Pixel reconstruction logits. Nlossrrr) r r!r"r#r+rr$r%r&rrr'rr(r(r(r)r*>s r*csfddtfddt|D}t|dddddf|dddddf<t|dddddf|dddddf<t|dS) z Sinusoid position encoding tablecsfddtDS)Nc s(g|]}tdd|dqS)i')nppower).0hid_j)d_hidpositionr(r) Ys(zOget_sinusoid_encoding_table..get_position_angle_vec..)ranger2)r1r5r)get_position_angle_vecXsz;get_sinusoid_encoding_table..get_position_angle_veccsg|]}|qSr(r()r/pos_i)r6r(r)r3[z/get_sinusoid_encoding_table..Nrr,r)r-arrayr4sincosr$r% unsqueeze) n_positionr1sinusoid_tabler()r1r6r)get_sinusoid_encoding_tableTs ..r?c(eZdZdZfddZddZZS)VideoMAEEmbeddingsz7 Construct the patch and position embeddings. cs8tt||_|jj|_t|j|j|_||_dSN) super__init__VideoMAEPatchEmbeddingspatch_embeddings num_patchesr? hidden_sizeposition_embeddingsconfigselfrJ __class__r(r)rDhs    zVideoMAEEmbeddings.__init__cCsZ||}||j|j|jdd}|dur+|j\}}}||}||d|}|S)NTdevicecopy)rFrIdetachtype_astorPshapereshape)rL pixel_valuesbool_masked_pos embeddings batch_size_ num_channelsr(r(r)forwardqs   zVideoMAEEmbeddings.forwardr r!r"r#rDr^ __classcell__r(r(rMr)rAbs  rAcr@)rEaw Video to Patch Embedding. This module turns a batch of videos of shape (batch_size, num_frames, num_channels, height, width) into a tensor of shape (batch_size, seq_len, hidden_size) to be consumed by a Transformer encoder. The seq_len (the number of patches) equals (number of frames // tubelet_size) * (height // patch_size) * (width // patch_size). c st|j}|j}|j}|j}|j}|j}t|t j j r |n||f}t|t j j r-|n||f}||_||_t ||_|d|d|d|d||j}||_||_ tj|||j|d|df|j|d|dfd|_dS)Nrr) in_channels out_channels kernel_sizestride)rCrD image_size patch_sizer]rH num_frames tubelet_size isinstance collectionsabcIterableintrGrConv3d projection) rLrJrerfr]rHrgrhrGrMr(r)rDs,  ( z VideoMAEPatchEmbeddings.__init__c Cs|j\}}}}}||jkrtd||jdks||jdkr6td|d|d|jdd|jdd |dddd d }||ddd}|S) NzeMake sure that the channel dimension of the pixel values match with the one set in the configuration.rrzInput image size (*z) doesn't match model (z).r,r )rVr] ValueErrorrepermuteroflatten transpose)rLrXr[rgr]heightwidthrZr(r(r)r^s (zVideoMAEPatchEmbeddings.forwardr_r(r(rMr)rEs rEmodulequerykeyvalueattention_maskscalingdropoutc Ks|t||dd|}tjj|dtjd|j}tjj |||j d}|dur,||}t||} | dd } | |fS)NrR)dimdtype)ptrainingrr,) r$matmulrur functionalsoftmaxfloat32rUrrr contiguous) ryrzr{r|r}r~rkwargs attn_weights attn_outputr(r(r)eager_attention_forwards  rc sveZdZdeddffdd ZdejdejfddZ dd eejd e de e ejejfe ejffd d Z Z S)VideoMAESelfAttentionrJreturnNcs t|j|jdkrt|dstd|jd|jd||_|j|_t|j|j|_|j|j|_ |j |_ |jd|_ d|_ tj|j|j dd|_tj|j|j dd|_tj|j|j dd|_|jr}tt|j |_tt|j |_dSd|_d|_dS) Nrembedding_sizezThe hidden size z4 is not a multiple of the number of attention heads .gFbias)rCrDrHnum_attention_headshasattrrrrJrmattention_head_size all_head_sizeattention_probs_dropout_prob dropout_probr~ is_causalrLinearrzr{r|qkv_bias Parameterr$zerosq_biasv_biasrKrMr(r)rDs,    zVideoMAESelfAttention.__init__xcCs6|dd|j|jf}||}|ddddS)NrRrr,rr )sizerrviewrs)rLr new_x_shaper(r(r)transpose_for_scoress z*VideoMAESelfAttention.transpose_for_scoresF head_maskoutput_attentionsc Cs"|jdur tj|jddnd}tjj||jj|d}tjj||j j|jd}tjj||j j|jd}| |}| |} | |} t } |j jdkr]|j jdkrW|rWtdnt|j j} | || || ||j|j|jsldn|jd\} } | dd |jf}| |} |r| | f}|S| f}|S) NF) requires_grad)inputweightreagersdpaz`torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to eager attention. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.rx)rr~rr)rr$ zeros_likerrrlinearr{rr|rzrrrJ_attn_implementationlogger warning_oncerrr~rrrrrW)rLrrrk_biaskeysvaluesqueries key_layer value_layer query_layerattention_interface context_layerattention_probsnew_context_layer_shapeoutputsr(r(r)r^s<        zVideoMAESelfAttention.forwardNF)r r!r"rrDr$Tensorrrboolrr'r^r`r(r(rMr)rsrcsFeZdZdZdeddffdd Zdejdejdejfd d ZZ S) VideoMAESelfOutputz The residual connection is defined in VideoMAELayer instead of here (as is the case with other models), due to the layernorm applied before each block. rJrNcs.tt|j|j|_t|j|_dSrB) rCrDrrrHdenseDropouthidden_dropout_probrrKrMr(r)rD' zVideoMAESelfOutput.__init__r input_tensorcC||}||}|SrBrrrLrrr(r(r)r^,  zVideoMAESelfOutput.forward) r r!r"r#rrDr$rr^r`r(r(rMr)r!s$rc s~eZdZdeddffdd ZdeeddfddZ  dd ej d e ej d e de e ej ej fe ej ffd dZZS)VideoMAEAttentionrJrNcs*tt||_t||_t|_dSrB)rCrDr attentionroutputset pruned_headsrKrMr(r)rD5s    zVideoMAEAttention.__init__headscCst|dkrdSt||jj|jj|j\}}t|jj||j_t|jj||j_t|jj ||j_ t|j j |dd|j _ |jjt||j_|jj|jj|j_ |j ||_dS)Nrrr)lenrrrrrrrzr{r|rrrunion)rLrindexr(r(r) prune_heads;s zVideoMAEAttention.prune_headsFrrrcCs4||||}||d|}|f|dd}|S)Nrr)rr)rLrrr self_outputsattention_outputrr(r(r)r^MszVideoMAEAttention.forwardr)r r!r"rrDrrmrr$rrrrr'r^r`r(r(rMr)r4srcs<eZdZdeddffdd ZdejdejfddZZS) VideoMAEIntermediaterJrNcsDtt|j|j|_t|jt rt |j|_ dS|j|_ dSrB) rCrDrrrHintermediate_sizerri hidden_actstrr intermediate_act_fnrKrMr(r)rD]s   zVideoMAEIntermediate.__init__rcCrrB)rr)rLrr(r(r)r^erzVideoMAEIntermediate.forward r r!r"rrDr$rr^r`r(r(rMr)r\srcsBeZdZdeddffdd Zdejdejdejfdd ZZS) VideoMAEOutputrJrNcs.tt|j|j|_t|j|_ dSrB) rCrDrrrrHrrrrrKrMr(r)rDnrzVideoMAEOutput.__init__rrcCs ||}||}||}|SrBrrr(r(r)r^ss  zVideoMAEOutput.forwardrr(r(rMr)rms$rc sleZdZdZdeddffdd Z  d dejd eejd e de e ejejfe ejffd d Z Z S) VideoMAELayerz?This corresponds to the Block class in the timm implementation.rJrNcsbt|j|_d|_t||_t||_t||_ t j |j |j d|_t j |j |j d|_dS)Nreps)rCrDchunk_size_feed_forward seq_len_dimrrr intermediaterrr LayerNormrHlayer_norm_epslayernorm_beforelayernorm_afterrKrMr(r)rDs    zVideoMAELayer.__init__FrrrcCs`|j||||d}|d}|dd}||}||}||}|||}|f|}|S)N)rrr)rrrrr)rLrrrself_attention_outputsrr layer_outputr(r(r)r^s     zVideoMAELayer.forwardr)r r!r"r#rrDr$rrrrr'r^r`r(r(rMr)r}s rcsbeZdZdeddffdd Z    ddejd eejd ed ed ede e e ff d dZ Z S)VideoMAEEncoderrJrNcs:t|_tfddtjD|_d|_dS)Ncg|]}tqSr(rr/r\rJr(r)r3r8z,VideoMAEEncoder.__init__..F) rCrDrJr ModuleListr4num_hidden_layerslayergradient_checkpointingrKrMrr)rDs   zVideoMAEEncoder.__init__FTrrroutput_hidden_states return_dictc Cs|rdnd}|r dnd}t|jD](\}} |r||f}|dur$||nd} | || |} | d}|r9|| df}q|rA||f}|sOtdd|||fDSt|||dS)Nr(rrcs|] }|dur|VqdSrBr(r/vr(r(r) z*VideoMAEEncoder.forward..last_hidden_staterr) enumeraterr'r) rLrrrrrall_hidden_statesall_self_attentionsi layer_modulelayer_head_mask layer_outputsr(r(r)r^s(     zVideoMAEEncoder.forward)NFFT)r r!r"rrDr$rrrrr'rr^r`r(r(rMr)rs&  rc@s4eZdZeZdZdZdZdZdZ dZ dZ ddZ dS)VideoMAEPreTrainedModelvideomaerXTcCstt|tjtjfr#|jjjd|jjd|j dur!|j j dSdSt|tj r8|j j |jj ddSdS)zInitialize the weightsrx)meanstdNg?) rirrrnrdatanormal_rJinitializer_rangerzero_rfill_)rLryr(r(r) _init_weightss   z%VideoMAEPreTrainedModel._init_weightsN) r r!r"r config_classbase_model_prefixmain_input_namesupports_gradient_checkpointing_supports_sdpa_supports_flash_attn_2_supports_flex_attn_supports_attention_backendr r(r(r(r)rs rcseZdZfddZddZddZe     ddejd e ej d e ej d e e d e e d e e de eeffddZZS) VideoMAEModelcsTt|||_t||_t||_|jrd|_n t j |j |j d|_| dS)Nr)rCrDrJrArZrencoderuse_mean_pooling layernormrrrHr post_initrKrMr(r)rDs    zVideoMAEModel.__init__cCs|jjSrB)rZrF)rLr(r(r)get_input_embeddingssz"VideoMAEModel.get_input_embeddingscCs*|D]\}}|jj|j|qdS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrrr)rLheads_to_prunerrr(r(r) _prune_headsszVideoMAEModel._prune_headsNrXrYrrrrrc Cs|dur|n|jj}|dur|n|jj}|dur|n|jj}|||jj}|||}|j|||||d}|d} |jdurD|| } |sO| f|ddSt | |j |j dS)a bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, sequence_length)`, *optional*): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). Each video in the batch must have the same number of masked patches. If `None`, then all patches are considered. Sequence length is `(num_frames // tubelet_size) * (image_size // patch_size) ** 2`. Examples: ```python >>> import av >>> import numpy as np >>> from transformers import AutoImageProcessor, VideoMAEModel >>> from huggingface_hub import hf_hub_download >>> np.random.seed(0) >>> def read_video_pyav(container, indices): ... ''' ... Decode the video with PyAV decoder. ... Args: ... container (`av.container.input.InputContainer`): PyAV container. ... indices (`list[int]`): List of frame indices to decode. ... Returns: ... result (np.ndarray): np array of decoded frames of shape (num_frames, height, width, 3). ... ''' ... frames = [] ... container.seek(0) ... start_index = indices[0] ... end_index = indices[-1] ... for i, frame in enumerate(container.decode(video=0)): ... if i > end_index: ... break ... if i >= start_index and i in indices: ... frames.append(frame) ... return np.stack([x.to_ndarray(format="rgb24") for x in frames]) >>> def sample_frame_indices(clip_len, frame_sample_rate, seg_len): ... ''' ... Sample a given number of frame indices from the video. ... Args: ... clip_len (`int`): Total number of frames to sample. ... frame_sample_rate (`int`): Sample every n-th frame. ... seg_len (`int`): Maximum allowed index of sample's last frame. ... Returns: ... indices (`list[int]`): List of sampled frame indices ... ''' ... converted_len = int(clip_len * frame_sample_rate) ... end_idx = np.random.randint(converted_len, seg_len) ... start_idx = end_idx - converted_len ... indices = np.linspace(start_idx, end_idx, num=clip_len) ... indices = np.clip(indices, start_idx, end_idx - 1).astype(np.int64) ... return indices >>> # video clip consists of 300 frames (10 seconds at 30 FPS) >>> file_path = hf_hub_download( ... repo_id="nielsr/video-demo", filename="eating_spaghetti.mp4", repo_type="dataset" ... ) >>> container = av.open(file_path) >>> # sample 16 frames >>> indices = sample_frame_indices(clip_len=16, frame_sample_rate=1, seg_len=container.streams.video[0].frames) >>> video = read_video_pyav(container, indices) >>> image_processor = AutoImageProcessor.from_pretrained("MCG-NJU/videomae-base") >>> model = VideoMAEModel.from_pretrained("MCG-NJU/videomae-base") >>> # prepare video for the model >>> inputs = image_processor(list(video), return_tensors="pt") >>> # forward pass >>> outputs = model(**inputs) >>> last_hidden_states = outputs.last_hidden_state >>> list(last_hidden_states.shape) [1, 1568, 768] ```Nrrrrrrr) rJrruse_return_dict get_head_maskrrZrrrrr) rLrXrYrrrrembedding_outputencoder_outputssequence_outputr(r(r)r^s.Y   zVideoMAEModel.forward)NNNNN)r r!r"rDrrrr$r%r BoolTensorrrrr'rr^r`r(r(rMr)rs2  rcs,eZdZfddZ   dddZZS)VideoMAEDecodercst|j|j|jd}t||j_|j_ |j _ |j _ tfddt|jD|_t|j|_|dkrFt|j|nt|_d|_||_dS)Nr,crr(rrdecoder_configr(r)r3r8z,VideoMAEDecoder.__init__..rF)rCrDr]rhrfrdecoder_hidden_sizerHdecoder_num_hidden_layersrdecoder_num_attention_headsrdecoder_intermediate_sizerrrr4decoder_layersrnormrIdentityheadrrJ)rLrJrGdecoder_num_labelsrMr'r)rDs  zVideoMAEDecoder.__init__FTc Cs|rdnd}|r dnd}t|jD]\}} |r||f}| |d|d} | d}|r0|| df}q|r8||f}|dkrG|dd| df}||}||} |s_tdd| ||fDSt| ||dS)Nr()rrrrcsrrBr(rr(r(r)rrz*VideoMAEDecoder.forward..)rrr)rr-r.r0r'r) rLrreturn_token_numrrrrrrrrrr(r(r)r^s&     zVideoMAEDecoder.forward)FFT)r r!r"rDr^r`r(r(rMr)r&s  r&zb The VideoMAE Model transformer with the decoder on top for self-supervised pre-training. csneZdZfddZe    d dejdejdeej dee dee d ee d e e e ffd d ZZS)VideoMAEForPreTrainingcs~t|||_t||_tj|j|jdd|_ t t dd|j|_ t|jjj|j|_t||jjjd|_|dS)NFrr)rG)rCrDrJrrrrrHr)encoder_to_decoderrr$r mask_tokenr?rZrGrIr&decoderrrKrMr(r)rDs    zVideoMAEForPreTraining.__init__NrXrYrrrrrc# Csj|dur|n|jj}|j||||||d}|d}||}|j\} } } |dur,td|j| dd|} | j |j dd} | | | d| } | | | d| }t j|| |j|gdd }|||jd}|j}d}t  |jjd kr|}n2|j }|j}t tj ||d ddddddf}t tj ||d ddddddf}|||}|j\} }} }}|jj|jj}}|jjr*|| |||| ||||||}|ddd d dddd }|| |||||||||| }||jddd|jdddd d}|| |||||||||| }nB|jjd kr5td|| |||| ||||||}|ddd d dddd }|| |||||||||| }|j\} }} || | d| } Wdn 1swYt!}!|!|| }|s|f|dd}"|dur|f|"S|"St"|||j#|j$dS)a bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, sequence_length)`): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). Each video in the batch must have the same number of masked patches. Sequence length is `(num_frames // tubelet_size) * (image_size // patch_size) ** 2`. Examples: ```python >>> from transformers import AutoImageProcessor, VideoMAEForPreTraining >>> import numpy as np >>> import torch >>> num_frames = 16 >>> video = list(np.random.randint(0, 256, (num_frames, 3, 224, 224))) >>> image_processor = AutoImageProcessor.from_pretrained("MCG-NJU/videomae-base") >>> model = VideoMAEForPreTraining.from_pretrained("MCG-NJU/videomae-base") >>> pixel_values = image_processor(video, return_tensors="pt").pixel_values >>> num_patches_per_frame = (model.config.image_size // model.config.patch_size) ** 2 >>> seq_length = (num_frames // model.config.tubelet_size) * num_patches_per_frame >>> bool_masked_pos = torch.randint(0, 2, (1, seq_length)).bool() >>> outputs = model(pixel_values, bool_masked_pos=bool_masked_pos) >>> loss = outputs.loss ```N)rYrrrrrz!One must provided a boolean mask rRTrOrrr )rPrrqr,r)rkeepdim)runbiasedr:gư>zQCan't unnormalize non-RGB images. Consider setting config.norm_pix_loss to False.r+rrr)%rJr rr4rVrrrIexpandrTrSrUrPrWr$catr5r6rno_gradr]r as_tensorrrrhrf norm_pix_lossrrsrrvarsqrtr r*rr)#rLrXrYrrrrrr$r[seq_lenr]expanded_position_embeddingspos_emb_visible pos_emb_maskx_fulldecoder_outputsrr+framesrPrrrtimervrwrhrf frames_norm videos_patchr\labelsloss_fctrr(r(r)r^s%    &&      J zVideoMAEForPreTraining.forward)NNNN)r r!r"rDrr$r%r%rrrrr'r*r^r`r(r(rMr)r3s,  r3z VideoMAE Model transformer with a video classification head on top (a linear layer on top of the average pooled hidden states of all tokens) e.g. for ImageNet. cszeZdZfddZe      d deejdeejdeejdeedeed eed e e e ffd d Z Z S)VideoMAEForVideoClassificationcsft||j|_t||_|jrt|jnd|_ |jdkr(t |j|jnt |_ | dS)Nr)rCrD num_labelsrrrrrrHfc_normrr/ classifierrrKrMr(r)rD}s  $ z'VideoMAEForVideoClassification.__init__NrXrrNrrrrc Cs|dur|n|jj}|j|||||d}|d}|jdur&||d}n|dddf}||} d} |dur|jjdurc|jdkrId|j_n|jdkr_|jt j ksZ|jt j kr_d|j_nd|j_|jjdkrt } |jdkr{| | | } n+| | |} n%|jjdkrt} | | d|j|d} n|jjdkrt} | | |} |s| f|dd} | dur| f| S| St| | |j|jd S) a! labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the image classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Examples: ```python >>> import av >>> import torch >>> import numpy as np >>> from transformers import AutoImageProcessor, VideoMAEForVideoClassification >>> from huggingface_hub import hf_hub_download >>> np.random.seed(0) >>> def read_video_pyav(container, indices): ... ''' ... Decode the video with PyAV decoder. ... Args: ... container (`av.container.input.InputContainer`): PyAV container. ... indices (`list[int]`): List of frame indices to decode. ... Returns: ... result (np.ndarray): np array of decoded frames of shape (num_frames, height, width, 3). ... ''' ... frames = [] ... container.seek(0) ... start_index = indices[0] ... end_index = indices[-1] ... for i, frame in enumerate(container.decode(video=0)): ... if i > end_index: ... break ... if i >= start_index and i in indices: ... frames.append(frame) ... return np.stack([x.to_ndarray(format="rgb24") for x in frames]) >>> def sample_frame_indices(clip_len, frame_sample_rate, seg_len): ... ''' ... Sample a given number of frame indices from the video. ... Args: ... clip_len (`int`): Total number of frames to sample. ... frame_sample_rate (`int`): Sample every n-th frame. ... seg_len (`int`): Maximum allowed index of sample's last frame. ... Returns: ... indices (`list[int]`): List of sampled frame indices ... ''' ... converted_len = int(clip_len * frame_sample_rate) ... end_idx = np.random.randint(converted_len, seg_len) ... start_idx = end_idx - converted_len ... indices = np.linspace(start_idx, end_idx, num=clip_len) ... indices = np.clip(indices, start_idx, end_idx - 1).astype(np.int64) ... return indices >>> # video clip consists of 300 frames (10 seconds at 30 FPS) >>> file_path = hf_hub_download( ... repo_id="nielsr/video-demo", filename="eating_spaghetti.mp4", repo_type="dataset" ... ) >>> container = av.open(file_path) >>> # sample 16 frames >>> indices = sample_frame_indices(clip_len=16, frame_sample_rate=1, seg_len=container.streams.video[0].frames) >>> video = read_video_pyav(container, indices) >>> image_processor = AutoImageProcessor.from_pretrained("MCG-NJU/videomae-base-finetuned-kinetics") >>> model = VideoMAEForVideoClassification.from_pretrained("MCG-NJU/videomae-base-finetuned-kinetics") >>> inputs = image_processor(list(video), return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**inputs) ... logits = outputs.logits >>> # model predicts one of the 400 Kinetics-400 classes >>> predicted_label = logits.argmax(-1).item() >>> print(model.config.id2label[predicted_label]) eating spaghetti ```Nrrr regressionsingle_label_classificationmulti_label_classificationrRr<)rJr rrRrrS problem_typerQrr$longrmr squeezer rrrrr) rLrXrrNrrrrr$rr+rOrr(r(r)r^sR\     "       z&VideoMAEForVideoClassification.forward)NNNNNN)r r!r"rDrrr$rrrr'rr^r`r(r(rMr)rPvs0   rP)r3rrrP)rx)Cr#collections.abcrjrQr dataclassesrtypingrrrnumpyr-r$torch.utils.checkpointrtorch.nnrr r activationsr modeling_layersr modeling_outputsrrmodeling_utilsrr pytorch_utilsrrutilsrrrutils.constantsrrconfiguration_videomaer get_loggerr rrr*r?ModulerArErfloatrrrrrrrrrrr&r3rP__all__r(r(r(r)s        != J(++<2 !