o ۷iە@sdZddlZddlmZddlmZddlmZm Z ddl Z ddl Z ddl m Z ddlmZdd lmZdd lmZdd lmZmZdd lmZmZdd lmZddlmZmZddlm Z m!Z!m"Z"m#Z#ddl$m%Z%m&Z&ddl'm(Z(m)Z)ddl*m+Z+e#,e-Z.ee"ddGddde Z/ee"ddGddde Z0ddZ1Gddde j2Z3Gdd d e j2Z4 !dFd"e j2d#e j5d$e j5d%e j5d&e e j5d'e6d(e6fd)d*Z7Gd+d,d,e j2Z8Gd-d.d.e j2Z9Gd/d0d0e j2Z:Gd1d2d2e j2Z;Gd3d4d4e j2Ze"Gd9d:d:eZ?e"Gd;d<dd>e j2ZAe"d?dGd@dAdAe?ZBe"dBdGdCdDdDe?ZCgdEZDdS)Gz,PyTorch VideoMAE (masked autoencoder) model.N)deepcopy) dataclass)CallableOptional)nn)MSELoss)ACT2FN)GradientCheckpointingLayer)BaseModelOutputImageClassifierOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack) find_pruneable_heads_and_indicesprune_linear_layer) ModelOutputTransformersKwargsauto_docstringlogging)IMAGENET_DEFAULT_MEANIMAGENET_DEFAULT_STD)can_return_tuplecheck_model_inputs)VideoMAEConfigz[ Class for VideoMAEDecoder's outputs, with potential hidden states and attentions. ) custom_introc@sPeZdZUdZdZeejed<dZ ee ejed<dZ ee ejed<dS)VideoMAEDecoderOutputz logits (`torch.FloatTensor` of shape `(batch_size, patch_size ** 2 * num_channels)`): Pixel reconstruction logits. Nlogits hidden_states attentions) __name__ __module__ __qualname____doc__rrtorch FloatTensor__annotations__rtupler r)r)d/home/ubuntu/vllm_env/lib/python3.10/site-packages/transformers/models/videomae/modeling_videomae.pyr*s rzb Class for VideoMAEForPreTraining's outputs, with potential hidden states and attentions. c@sbeZdZUdZdZeejed<dZ eejed<dZ ee ejed<dZ ee ejed<dS)VideoMAEForPreTrainingOutputz loss (`torch.FloatTensor` of shape `(1,)`): Pixel reconstruction loss. logits (`torch.FloatTensor` of shape `(batch_size, patch_size ** 2 * num_channels)`): Pixel reconstruction logits. Nlossrrr ) r!r"r#r$r,rr%r&r'rrr(r r)r)r)r*r+;s r+csfddtfddt|D}t|dddddf|dddddf<t|dddddf|dddddf<t|dS) z Sinusoid position encoding tablecsfddtDS)Nc s(g|]}tdd|dqS)i')nppower).0hid_j)d_hidpositionr)r* Vs(zOget_sinusoid_encoding_table..get_position_angle_vec..)ranger3)r2r6r*get_position_angle_vecUsz;get_sinusoid_encoding_table..get_position_angle_veccsg|]}|qSr)r))r0pos_i)r7r)r*r4Xz/get_sinusoid_encoding_table..Nrr-r)r.arrayr5sincosr%r& unsqueeze) n_positionr2sinusoid_tabler))r2r7r*get_sinusoid_encoding_tableQs ..r@c(eZdZdZfddZddZZS)VideoMAEEmbeddingsz7 Construct the patch and position embeddings. cs8tt||_|jj|_t|j|j|_||_dSN) super__init__VideoMAEPatchEmbeddingspatch_embeddings num_patchesr@ hidden_sizeposition_embeddingsconfigselfrK __class__r)r*rEes    zVideoMAEEmbeddings.__init__cCsZ||}||j|j|jdd}|dur+|j\}}}||}||d|}|S)NTdevicecopy)rGrJdetachtype_astorQshapereshape)rM pixel_valuesbool_masked_pos embeddings batch_size_ num_channelsr)r)r*forwardns   zVideoMAEEmbeddings.forwardr!r"r#r$rEr_ __classcell__r)r)rNr*rB_s  rBcrA)rFaw Video to Patch Embedding. This module turns a batch of videos of shape (batch_size, num_frames, num_channels, height, width) into a tensor of shape (batch_size, seq_len, hidden_size) to be consumed by a Transformer encoder. The seq_len (the number of patches) equals (number of frames // tubelet_size) * (height // patch_size) * (width // patch_size). c st|j}|j}|j}|j}|j}|j}t|t j j r |n||f}t|t j j r-|n||f}||_||_t ||_|d|d|d|d||j}||_||_ tj|||j|d|df|j|d|dfd|_dS)Nrr) in_channels out_channels kernel_sizestride)rDrE image_size patch_sizer^rI num_frames tubelet_size isinstance collectionsabcIterableintrHrConv3d projection) rMrKrfrgr^rIrhrirHrNr)r*rEs,  ( z VideoMAEPatchEmbeddings.__init__c Cs|j\}}}}}||jkrtd||jdks||jdkr6td|d|d|jdd|jdd |dddd d }||ddd}|S) NzeMake sure that the channel dimension of the pixel values match with the one set in the configuration.rrzInput image size (*z) doesn't match model (z).r-r)rWr^ ValueErrorrfpermuterpflatten transpose)rMrYr\rhr^heightwidthr[r)r)r*r_s (zVideoMAEPatchEmbeddings.forwardr`r)r)rNr*rFs rFmodulequerykeyvalueattention_maskscalingdropoutc Ks|t||dd|}tjj|dtjd|j}tjj |||j d}|dur,||}t||} | dd } | |fS)NrS)dimdtype)ptrainingrr-) r%matmulrvr functionalsoftmaxfloat32rVrrr contiguous) rzr{r|r}r~rrkwargs attn_weights attn_outputr)r)r*eager_attention_forwards  rcsLeZdZdeddffdd Zd deejdeejejffddZ Z S) VideoMAESelfAttentionrKreturnNcs t|j|jdkrt|dstd|jd|jd||_|j|_t|j|j|_|j|j|_ |j |_ |jd|_ d|_ tj|j|j dd|_tj|j|j dd|_tj|j|j dd|_|jr}tt|j |_tt|j |_dSd|_d|_dS) Nrembedding_sizezThe hidden size z4 is not a multiple of the number of attention heads .gFbias)rDrErInum_attention_headshasattrrsrKrnattention_head_size all_head_sizeattention_probs_dropout_prob dropout_probr is_causalrLinearr{r|r}qkv_bias Parameterr%zerosq_biasv_biasrLrNr)r*rEs,    zVideoMAESelfAttention.__init__ head_maskc Cs6|j\}}}|jdurtj|jddnd}tjj||jj |d}tjj||j j |jd}tjj||j j |jd} | |d|j |jdd} | |d|j |jdd} | |d|j |jdd} t} |jjdkrpt|jj} | || | | ||j|j|jsdn|jd \}}|dd |jf}||}||fS) NF) requires_grad)inputweightrrSrr-eagerry)rrrr)rWrr% zeros_likerrrlinearr|rr}r{viewrrrvrrK_attn_implementationr rrrrsizerrX)rMrrr\ seq_lengthr]k_biaskeysvaluesqueries key_layer value_layer query_layerattention_interface context_layerattention_probsnew_context_layer_shaper)r)r*r_s0     zVideoMAESelfAttention.forwardrC) r!r"r#rrErr%Tensorr(r_rar)r)rNr*rs.rcsBeZdZdZdeffdd Zdejdejdejfdd ZZ S) VideoMAESelfOutputz The residual connection is defined in VideoMAELayer instead of here (as is the case with other models), due to the layernorm applied before each block. rKcs.tt|j|j|_t|j|_dSrC) rDrErrrIdenseDropouthidden_dropout_probrrLrNr)r*rE zVideoMAESelfOutput.__init__r input_tensorrcC||}||}|SrCrrrMrrr)r)r*r_  zVideoMAESelfOutput.forward) r!r"r#r$rrEr%rr_rar)r)rNr*rs$rcsVeZdZdeffdd ZdeefddZd dej d e ej d ej fd d Z Z S)VideoMAEAttentionrKcs*tt||_t||_t|_dSrC)rDrEr attentionroutputset pruned_headsrLrNr)r*rE$s    zVideoMAEAttention.__init__headscCst|dkrdSt||jj|jj|j\}}t|jj||j_t|jj||j_t|jj ||j_ t|j j |dd|j _ |jjt||j_|jj|jj|j_ |j ||_dS)Nrrr)lenrrrrrrr{r|r}rrrunion)rMrindexr)r)r* prune_heads*s zVideoMAEAttention.prune_headsNrrrcCs |||\}}|||}|SrC)rr)rMrrself_attn_outputr]rr)r)r*r_<s zVideoMAEAttention.forwardrC) r!r"r#rrErrnrr%rrr_rar)r)rNr*r#s*rcs8eZdZdeffdd ZdejdejfddZZS)VideoMAEIntermediaterKcsDtt|j|j|_t|jt rt |j|_ dS|j|_ dSrC) rDrErrrIintermediate_sizerrj hidden_actstrr intermediate_act_fnrLrNr)r*rEDs   zVideoMAEIntermediate.__init__rrcCrrC)rr)rMrr)r)r*r_LrzVideoMAEIntermediate.forward r!r"r#rrEr%rr_rar)r)rNr*rCsrcs>eZdZdeffdd ZdejdejdejfddZZS) VideoMAEOutputrKcs.tt|j|j|_t|j|_ dSrC) rDrErrrrIrrrrrLrNr)r*rETrzVideoMAEOutput.__init__rrrcCs ||}||}||}|SrCrrr)r)r*r_Ys  zVideoMAEOutput.forwardrr)r)rNr*rSs$rcsHeZdZdZdeffdd Zd dejdeejdejfd d Z Z S) VideoMAELayerz?This corresponds to the Block class in the timm implementation.rKcsbt|j|_d|_t||_t||_t||_ t j |j |j d|_t j |j |j d|_dS)Nreps)rDrEchunk_size_feed_forward seq_len_dimrrr intermediaterrr LayerNormrIlayer_norm_epslayernorm_beforelayernorm_afterrLrNr)r*rEds    zVideoMAELayer.__init__NrrrcCsB||}|||}||}||}||}|||}|SrC)rrrrr)rMrrhidden_states_normattention_output layer_outputr)r)r*r_ns     zVideoMAELayer.forwardrC) r!r"r#r$rrEr%rrr_rar)r)rNr*ras* rcsBeZdZdeffdd Zd dejdeejdefdd Z Z S) VideoMAEEncoderrKcs:t|_tfddtjD|_d|_dS)Ncg|]}tqSr)rr0r]rKr)r*r4r9z,VideoMAEEncoder.__init__..F) rDrErKr ModuleListr5num_hidden_layerslayergradient_checkpointingrLrNrr*rEs   zVideoMAEEncoder.__init__NrrrcCs<t|jD]\}}|dur||nd}|||}qt|dS)Nlast_hidden_state) enumeraterr )rMrri layer_modulelayer_head_maskr)r)r*r_s  zVideoMAEEncoder.forwardrC) r!r"r#rrEr%rrr r_rar)r)rNr*rs(rc@sLeZdZUeed<dZdZdZddgZdZ dZ dZ dZ e edZdd Zd S) VideoMAEPreTrainedModelrKvideomaerYTrBr)rr cCstt|tjtjfr#|jjjd|jjd|j dur!|j j dSdSt|tj r8|j j |jj ddSdS)zInitialize the weightsry)meanstdNg?) rjrrrordatanormal_rKinitializer_rangerzero_rfill_)rMrzr)r)r* _init_weightss   z%VideoMAEPreTrainedModel._init_weightsN)r!r"r#rr'base_model_prefixmain_input_namesupports_gradient_checkpointing_no_split_modules_supports_sdpa_supports_flash_attn_supports_flex_attn_supports_attention_backendrr_can_record_outputsrr)r)r)r*rs  rc speZdZfddZddZddZedde dd ej d e ej d e ej d e edef ddZZS) VideoMAEModelcsTt|||_t||_t||_|jrd|_n t j |j |j d|_| dS)Nr)rDrErKrBr[rencoderuse_mean_pooling layernormrrrIr post_initrLrNr)r*rEs    zVideoMAEModel.__init__cCs|jjSrC)r[rG)rMr)r)r*get_input_embeddingssz"VideoMAEModel.get_input_embeddingscCs*|D]\}}|jj|j|qdS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrrr)rMheads_to_prunerrr)r)r* _prune_headsszVideoMAEModel._prune_headsF)tie_last_hidden_statesNrYrZrrrcKsN|||jj}|||}|j||d}|j}|jdur"||}t|dS)a bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, sequence_length)`, *optional*): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). Each video in the batch must have the same number of masked patches. If `None`, then all patches are considered. Sequence length is `(num_frames // tubelet_size) * (image_size // patch_size) ** 2`. Examples: ```python >>> import av >>> import numpy as np >>> from transformers import AutoImageProcessor, VideoMAEModel >>> from huggingface_hub import hf_hub_download >>> np.random.seed(0) >>> def read_video_pyav(container, indices): ... ''' ... Decode the video with PyAV decoder. ... Args: ... container (`av.container.input.InputContainer`): PyAV container. ... indices (`list[int]`): List of frame indices to decode. ... Returns: ... result (np.ndarray): np array of decoded frames of shape (num_frames, height, width, 3). ... ''' ... frames = [] ... container.seek(0) ... start_index = indices[0] ... end_index = indices[-1] ... for i, frame in enumerate(container.decode(video=0)): ... if i > end_index: ... break ... if i >= start_index and i in indices: ... frames.append(frame) ... return np.stack([x.to_ndarray(format="rgb24") for x in frames]) >>> def sample_frame_indices(clip_len, frame_sample_rate, seg_len): ... ''' ... Sample a given number of frame indices from the video. ... Args: ... clip_len (`int`): Total number of frames to sample. ... frame_sample_rate (`int`): Sample every n-th frame. ... seg_len (`int`): Maximum allowed index of sample's last frame. ... Returns: ... indices (`list[int]`): List of sampled frame indices ... ''' ... converted_len = int(clip_len * frame_sample_rate) ... end_idx = np.random.randint(converted_len, seg_len) ... start_idx = end_idx - converted_len ... indices = np.linspace(start_idx, end_idx, num=clip_len) ... indices = np.clip(indices, start_idx, end_idx - 1).astype(np.int64) ... return indices >>> # video clip consists of 300 frames (10 seconds at 30 FPS) >>> file_path = hf_hub_download( ... repo_id="nielsr/video-demo", filename="eating_spaghetti.mp4", repo_type="dataset" ... ) >>> container = av.open(file_path) >>> # sample 16 frames >>> indices = sample_frame_indices(clip_len=16, frame_sample_rate=1, seg_len=container.streams.video[0].frames) >>> video = read_video_pyav(container, indices) >>> image_processor = AutoImageProcessor.from_pretrained("MCG-NJU/videomae-base") >>> model = VideoMAEModel.from_pretrained("MCG-NJU/videomae-base") >>> # prepare video for the model >>> inputs = image_processor(list(video), return_tensors="pt") >>> # forward pass >>> outputs = model(**inputs) >>> last_hidden_states = outputs.last_hidden_state >>> list(last_hidden_states.shape) [1, 1568, 768] ```rNr) get_head_maskrKrr[rrrr )rMrYrZrrembedding_outputencoder_outputssequence_outputr)r)r*r_s^    zVideoMAEModel.forward)NN)r!r"r#rEr r rrr%r&r BoolTensorrrrr r_rar)r)rNr*rs& rcs6eZdZdeffdd ZdejdefddZZ S)VideoMAEDecoderrKcst|j|j|jd}t||j_|j_ |j _ |j _ tfddt|jD|_t|j|_|dkrFt|j|nt|_d|_|_dS)Nr-crr)rrdecoder_configr)r*r4>r9z,VideoMAEDecoder.__init__..rF)rDrEr^rirgrdecoder_hidden_sizerIdecoder_num_hidden_layersrdecoder_num_attention_headsrdecoder_intermediate_sizerrrr5decoder_layersrnormrIdentityheadrrK)rMrKdecoder_num_labelsrNrr*rE3s  zVideoMAEDecoder.__init__rreturn_token_numcCsT|jD]}||dd}q|dkr|dd| df}||}||}t|dS)Nrr)r)rrrr)rMrr!rrr)r)r*r_Is    zVideoMAEDecoder.forward) r!r"r#rrEr%rrnr_rar)r)rNr*r2srzb The VideoMAE Model transformer with the decoder on top for self-supervised pre-training. c sTeZdZfddZee d dejdejde ej de e de f d d ZZS) VideoMAEForPreTrainingcstt|||_t||_tj|j|jdd|_ t t dd|j|_ t|jjj|j|_t||_|dS)NFrr)rDrErKrrrrrIrencoder_to_decoderrr%r mask_tokenr@r[rHrJrdecoderr rLrNr)r*rE^s     zVideoMAEForPreTraining.__init__NrYrZrrrc Ks&|j|f||d|}|j}||}|j\}}} |dur"td|j|dd|} | j |j dd} | | |d| } | | |d| } t j || |j| gdd} || | jd}|j}d}t  |jjd kru|}n2|j }|j}t tj ||d ddddddf}t tj ||d ddddddf}|||}|j\}}} }}|jj|jj}}|jjr |||||| ||||||}|d dd d dddd }|||||||||||| }||jddd|jdddd d}|||||||||||| }nB|jjd kr+td|||||| ||||||}|d dd d dddd }|||||||||||| }|j\}}} || |d| }Wdn 1s|wYt!}|||}t"|||j#|j$dS)a bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, sequence_length)`): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). Each video in the batch must have the same number of masked patches. Sequence length is `(num_frames // tubelet_size) * (image_size // patch_size) ** 2`. Examples: ```python >>> from transformers import AutoImageProcessor, VideoMAEForPreTraining >>> import numpy as np >>> import torch >>> num_frames = 16 >>> video = list(np.random.randint(0, 256, (num_frames, 3, 224, 224))) >>> image_processor = AutoImageProcessor.from_pretrained("MCG-NJU/videomae-base") >>> model = VideoMAEForPreTraining.from_pretrained("MCG-NJU/videomae-base") >>> pixel_values = image_processor(video, return_tensors="pt").pixel_values >>> num_patches_per_frame = (model.config.image_size // model.config.patch_size) ** 2 >>> seq_length = (num_frames // model.config.tubelet_size) * num_patches_per_frame >>> bool_masked_pos = torch.randint(0, 2, (1, seq_length)).bool() >>> outputs = model(pixel_values, bool_masked_pos=bool_masked_pos) >>> loss = outputs.loss ```)rZrNz!One must provided a boolean mask rSTrPrrr)rQrrrrr-r)rkeepdim)runbiasedr)gư>zQCan't unnormalize non-RGB images. Consider setting config.norm_pix_loss to False.r,rrr )%rrr#rWrsrJexpandrUrTrVrQrXr%catr$r%rno_gradrKr^r as_tensorrrrirg norm_pix_lossrrtrrvarsqrtrr+rr )rMrYrZrroutputsrr\r]r^expanded_position_embeddingspos_emb_visible pos_emb_maskx_fulldecoder_outputsrr,framesrQrrrtimerwrxrirg frames_norm videos_patchlabelsloss_fctr)r)r*r_os$    &&      J zVideoMAEForPreTraining.forwardrC)r!r"r#rErrr%r&rrrrrr+r_rar)r)rNr*r"Xs  r"z VideoMAE Model transformer with a video classification head on top (a linear layer on top of the average pooled hidden states of all tokens) e.g. for ImageNet. c s`eZdZfddZee   d deejdeejdeejde e de f d d Z Z S) VideoMAEForVideoClassificationcsft||j|_t||_|jrt|jnd|_ |jdkr(t |j|jnt |_ | dS)Nr)rDrE num_labelsrrrrrrIfc_normrr classifierr rLrNr)r*rE s  $ z'VideoMAEForVideoClassification.__init__NrYrr=rrc Ks|j|fd|i|}|j}|jdur|d}||}n|dddf}||}d} |dur=|j|||jfi|} t| ||j|j dS)a! labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the image classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Examples: ```python >>> import av >>> import torch >>> import numpy as np >>> from transformers import AutoImageProcessor, VideoMAEForVideoClassification >>> from huggingface_hub import hf_hub_download >>> np.random.seed(0) >>> def read_video_pyav(container, indices): ... ''' ... Decode the video with PyAV decoder. ... Args: ... container (`av.container.input.InputContainer`): PyAV container. ... indices (`list[int]`): List of frame indices to decode. ... Returns: ... result (np.ndarray): np array of decoded frames of shape (num_frames, height, width, 3). ... ''' ... frames = [] ... container.seek(0) ... start_index = indices[0] ... end_index = indices[-1] ... for i, frame in enumerate(container.decode(video=0)): ... if i > end_index: ... break ... if i >= start_index and i in indices: ... frames.append(frame) ... return np.stack([x.to_ndarray(format="rgb24") for x in frames]) >>> def sample_frame_indices(clip_len, frame_sample_rate, seg_len): ... ''' ... Sample a given number of frame indices from the video. ... Args: ... clip_len (`int`): Total number of frames to sample. ... frame_sample_rate (`int`): Sample every n-th frame. ... seg_len (`int`): Maximum allowed index of sample's last frame. ... Returns: ... indices (`list[int]`): List of sampled frame indices ... ''' ... converted_len = int(clip_len * frame_sample_rate) ... end_idx = np.random.randint(converted_len, seg_len) ... start_idx = end_idx - converted_len ... indices = np.linspace(start_idx, end_idx, num=clip_len) ... indices = np.clip(indices, start_idx, end_idx - 1).astype(np.int64) ... return indices >>> # video clip consists of 300 frames (10 seconds at 30 FPS) >>> file_path = hf_hub_download( ... repo_id="nielsr/video-demo", filename="eating_spaghetti.mp4", repo_type="dataset" ... ) >>> container = av.open(file_path) >>> # sample 16 frames >>> indices = sample_frame_indices(clip_len=16, frame_sample_rate=1, seg_len=container.streams.video[0].frames) >>> video = read_video_pyav(container, indices) >>> image_processor = AutoImageProcessor.from_pretrained("MCG-NJU/videomae-base-finetuned-kinetics") >>> model = VideoMAEForVideoClassification.from_pretrained("MCG-NJU/videomae-base-finetuned-kinetics") >>> inputs = image_processor(list(video), return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**inputs) ... logits = outputs.logits >>> # model predicts one of the 400 Kinetics-400 classes >>> predicted_label = logits.argmax(-1).item() >>> print(model.config.id2label[predicted_label]) eating spaghetti ```rNrrr+) rrrArrB loss_functionrKr rr ) rMrYrr=rr3rrrr,r)r)r*r_s \    z&VideoMAEForVideoClassification.forward)NNN)r!r"r#rErrrr%rrrr r_rar)r)rNr*r?s$  r?)r"rrr?)ry)Er$collections.abcrkrRr dataclassesrtypingrrnumpyr.r%rtorch.nnr activationsr modeling_layersr modeling_outputsr r modeling_utilsr rprocessing_utilsr pytorch_utilsrrutilsrrrrutils.constantsrr utils.genericrrconfiguration_videomaer get_loggerr!loggerrr+r@ModulerBrFrfloatrrrrrrrrrrrr"r?__all__r)r)r)r*s          != = &'