o ۷ip@s4dZddlZddlZddlmZddlmZddlmZmZm Z ddl Z ddl m Z ddl m Z dd lmZdd lmZmZdd lmZdd lmZmZdd lmZmZmZmZmZddlmZm Z m!Z!m"Z"m#Z#e$e%Z&dZ'dZ(dZ)e e#e!e"fZ*eeddGdddeZ+eeddGdddeZ,eeddGdddeZ-Gddde j.Z/Gddde j.Z0Gd d!d!e j.Z1Gd"d#d#e j.Z2Gd$d%d%e j.Z3Gd&d'd'e j.Z4Gd(d)d)e j.Z5Gd*d+d+e j.Z6Gd,d-d-eZ7Gd.d/d/e j.Z8Gd0d1d1e j.Z9eGd2d3d3eZ:eGd4d5d5e:Z;eGd6d7d7e:ZGdd?d?e j.Z@Gd@dAdAe j.ZAedBdGdCdDdDe:ZBGdEdFdFe j.ZCGdGdHdHe j.ZDGdIdJdJe j.ZEGdKdLdLe j.ZFedMdGdNdOdOe:ZGgdPZHdS)QzPyTorch FLAVA model.N) OrderedDict) dataclass)AnyOptionalUnion)nn)ACT2FN)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPooling)PreTrainedModel) find_pruneable_heads_and_indicesprune_linear_layer) ModelOutputauto_docstringfilter_out_non_signature_kwargslogging torch_int) FlavaConfigFlavaImageCodebookConfigFlavaImageConfigFlavaMultimodalConfigFlavaTextConfigzfacebook/flava-image-codebookg$(~k@a Output from FlavaModel containing embeddings and outputs from individual encoders. Note that `image_embeddings` and `text_embeddigns` returned are similar to pooled output returned from a transformer. If you want embeddings for contrastive loss or retrieval use a FLAVA model's `image_projection` and `text_projection` layers on `image_embeddings` and `text_embeddings` respectively. ) custom_introc@seZdZUdZdZeejed<dZ ee ed<dZ eejed<dZ ee ed<dZ eejed<dZee ed<d eefd d ZdS) FlavaModelOutputa image_embeddings (`torch.FloatTensor` of shape `(batch_size, output_dim)`, *optional*, returned when `pixel_values` are present): The image embeddings which are basically the pooled output of [`FlavaImageModel`]. image_output (`BaseModelOutputWithPooling`, *optional*, returned when `pixel_values` are present): The output of the [`FlavaImageModel`]. text_embeddings (`torch.FloatTensor` of shape `(batch_size, output_dim)`, *optional*, returned when `input_ids` are present): The text embeddings which are basically the pooled output of [`FlavaTextModel`]. text_output (`BaseModelOutputWithPooling`, *optional*, returned when `input_ids` are present): The output of the [`FlavaTextModel`]. multimodal_embeddings (`torch.FloatTensor` of shape `(batch_size, output_dim)`, *optional*, returned when `input_ids` and `pixel_values` are present and `skip_multimodal_encoder` is `None` or `False`): The multimodal embeddings which are basically the pooled output of [`FlavaTextModel`]. multimodal_output (`BaseModelOutputWithPooling`, returned when `input_ids` and `pixel_values` are present and `skip_multimodal_encoder` is `None` or `False`): The output of the [`FlavaMultimodalModel`]. Nimage_embeddings image_outputtext_embeddings text_outputmultimodal_embeddingsmultimodal_outputreturncstfddDS)Nc3s.|]}|dvr |nt|VqdS))r rr"Ngetattrto_tuple.0kself^/home/ubuntu/vllm_env/lib/python3.10/site-packages/transformers/models/flava/modeling_flava.py Us  z,FlavaModelOutput.to_tuple..tuplekeysr*r,r*r-r&Ts zFlavaModelOutput.to_tuple)__name__ __module__ __qualname____doc__rrtorch FloatTensor__annotations__rr rr r!r"r0rr&r,r,r,r-r3s  rz@ Class representing pretraining losses from FLAVA model c@seZdZUdZdZeejed<dZ eejed<dZ eejed<dZ eejed<dZ eejed<dZ eejed<d efd d ZdS) FlavaLossesa mim (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `mim_labels` and `pixel_values` are present, `input_ids_masked` is absent and `mim_weight` > 0.): Masked Image Modeling loss as used in BeIT calculated only for unimodal image data. mlm (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `mlm_labels` and `input_ids_masked` are present, `pixel_values` is absent and `mlm_weight` > 0.): Masked Language Modeling loss as used in BERT calculated only for unimodal text data. itm (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `itm_labels`, `input_ids_masked`, `pixel_values` are present and `itm_weight` > 0.): Image Text Matching (ITM) loss calculated for paired image-text data. Note that ITM loss is calculated on masked pairs in FLAVA. global_contrastive (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `input_ids` and `pixel_values` are present and `global_contrastive_weight` > 0.): Contrastive loss for image-text similarity similar to CLIP but calculated globally for paired image-text data. This is calculated on unmasked images and texts. mmm_image (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `mim_labels`, `pixel_values` and `input_ids_masked` are present and `mmm_image_weight` > 0.): Masked Multimodal Modeling loss's image component calculated on paired image-text data. mmm_text (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `mlm_labels`, `pixel_values` and `input_ids_masked` are present and `mmm_text_weight` > 0.): Masked Multimodal Modeling loss's text component calculated on paired image-text data. Nmimmlmitmglobal_contrastive mmm_imagemmm_textr#cCs(d}|D] }|durd}|Sq|S)NTF)values)r+all_nonevr,r,r-rAzs zFlavaLosses.all_none)r2r3r4r5r:rr6r7r8r;r<r=r>r?boolrAr,r,r,r-r9[s r9a Output from FlavaForPreTraining containing embeddings, and outputs from individual encoders. Note that `image_embeddings` and `text_embeddings` returned are similar to pooled output returned from a transformer. If you want embeddings for contrastive loss or retrieval use a FLAVA model's `image_projection` and `text_projection` layers on `image_embeddings` and `text_embeddings` respectively. c@seZdZUdZdZeejed<dZ e ed<dZ eejed<dZ ee ed<dZeejed<dZee ed<dZeejed <dZee ed <dZeejed <dZee ed <dZeejed <dZee ed<dZeejed<dZee ed<dZeejed<dZeejed<dZeejed<dZeejed<dZeejed<dZeejed<dZeejed<dee fddZ!dS)FlavaForPreTrainingOutputay loss (`torch.FloatTensor`, *optional*, returned when `return_loss` is True): Total loss calculated for this model. loss_info (`FlavaLosses`): Detailed info for FLAVA Pretraining losses. Check `FlavaLosses` class description for the information on the keys. image_embeddings (`torch.FloatTensor` of shape `(batch_size, output_dim)`, *optional*, returned when `pixel_values` are present): The image embeddings which are basically the pooled output of [`FlavaImageModel`]. image_output (`BaseModelOutputWithPooling`, *optional*, returned when `pixel_values` are present): The output of the [`FlavaImageModel`]. text_embeddings (`torch.FloatTensor` of shape `(batch_size, output_dim)`, *optional*, returned when `input_ids` are present): The text embeddings which are basically the pooled output of [`FlavaTextModel`]. text_output (`BaseModelOutputWithPooling`, *optional*, returned when `input_ids` are present): The output of the [`FlavaTextModel`]. multimodal_embeddings (`torch.FloatTensor` of shape `(batch_size, output_dim)`, *optional*, returned when `input_ids` and `pixel_values` are present and `skip_unmasked_multimodal_encoder` is `None` or `False`): The multimodal embeddings which are basically the pooled output of [`FlavaTextModel`]. multimodal_output (`BaseModelOutputWithPooling`, returned when `input_ids` and `pixel_values` are present and `skip_unmasked_multimodal_encoder` is `None` or `False`): The output of the [`FlavaMultimodalModel`]. image_masked_embeddings (`torch.FloatTensor` of shape `(batch_size, output_dim)`, *optional*, returned when `pixel_values` are present): The image embeddings which are basically the pooled output of [`FlavaImageModel`]. Uses `bool_masked_pos` to create masked images. image_masked_output (`BaseModelOutputWithPooling`, *optional*, returned when `pixel_values` are present): The output of the [`FlavaImageModel`]. Uses `bool_masked_pos` to create masked images. text_masked_embeddings (`torch.FloatTensor` of shape `(batch_size, output_dim)`, *optional*, returned when `input_ids_masked` are present): The text embeddings which are basically the pooled output of [`FlavaTextModel`]. text_masked_output (`BaseModelOutputWithPooling`, *optional*, returned when `input_ids_masked` are present): The output of the [`FlavaTextModel`]. multimodal_masked_embeddings (`torch.FloatTensor` of shape `(batch_size, output_dim)`, *optional*, returned when `input_ids` and `pixel_values` are present): The multimodal embeddings which are basically the pooled output of [`FlavaTextModel`]. multimodal_masked_output (`BaseModelOutputWithPooling`, *optional*, returned when `input_ids_masked` and `pixel_values` are present): The output of the [`FlavaMultimodalModel`]. mim_logits (`torch.FloatTensor` of shape `(batch_size, num_image_patches, image_vocab_size)` or of shape `(total_masked_patches, image_vocab_size)` , *optional*, returned when `pixel_values` are present and `input_ids_masked` are not): The logits for MIM unimodal loss. Uses `book_masked_pos` to get masked patches. The flattened output is returned when `bool_masked_pos` has some of the patches masked. mlm_logits (`torch.FloatTensor` of shape `(batch_size, text_seq_length, text_vocab_size)` or of shape `(total_masked_seq_length, text_vocab_size)`, *optional*, returned when `input_ids_masked` are present and `pixel_values` are not): The logits for MLM unimodal loss. The flattened output is returned when `input_ids_masked` has some of the tokens masked. itm_logits (`torch.FloatTensor` of shape `(batch_size, 2)`, *optional*, returned when `input_ids_masked` and `pixel_values` are present): The logits for ITM loss. Note that ITM loss is calculated on masked pairs in FLAVA. contrastive_logits_per_image (`torch.FloatTensor` of shape `(image_batch_size, text_batch_size)`): The scaled dot product scores between `image_embeddings` and `text_embeddings` but passed through FLAVA's `image_projection` and `text_projection` layers respectively. This represents the image-text similarity scores. This is calculated on unmasked images and texts. contrastive_logits_per_text (`torch.FloatTensor` of shape `(text_batch_size, image_batch_size)`): The scaled dot product scores between `text_embeddings` and `image_embeddings` but passed through FLAVA's `text_projection` and `image_projection` layers respectively. This is calculated on unmasked images and texts. mmm_image_logits (`torch.FloatTensor` of shape `(batch_size, num_image_patches, image_vocab_size)` or of shape`(total_masked_patches, image_vocab_size)`, *optional*, returned when `pixel_values` and `input_ids_masked` are present): The logits for MMM image multimodal loss. Uses `book_masked_pos` to get masked patches. The flattened output is returned when `bool_masked_pos` has some of the patches masked. mmm_text_logits (`torch.FloatTensor` of shape `(batch_size, text_seq_length, text_vocab_size)` or of shape `(`(total_masked_seq_length, text_vocab_size)`), *optional*, returned when `pixel_values` and `input_ids_masked` are present): The logits for MMM text multimodal loss. The flattened output is returned when `input_ids_masked` has some of the tokens masked. Nloss loss_inforrrr r!r"image_masked_embeddingsimage_masked_outputtext_masked_embeddingstext_masked_outputmultimodal_masked_embeddingsmultimodal_masked_output mim_logits mlm_logits itm_logitscontrastive_logits_per_imagecontrastive_logits_per_textmmm_image_logitsmmm_text_logitsr#cs$gdtfddDS)N)r rr"rJrHrLc3s.|]}|vr |nt|VqdSNr$r'r+transformer_outputsr,r-r.s,z5FlavaForPreTrainingOutput.to_tuple..r/r*r,rUr-r&sz"FlavaForPreTrainingOutput.to_tuple)"r2r3r4r5rErr6r7r8rFr9rrr rr r!r"rGrHrIrJrKrLrMrNrOrPrQrRrSr0rr&r,r,r,r-rDs0  7 rDc sxeZdZdZddededdffdd Zd ejd e d e dejfd d Z  ddejde ej dedejfddZ ZS)FlavaImageEmbeddingszb Construct the CLS token, position and patch embeddings. Optionally, also the mask token. Fconfiguse_mask_tokenr#Ncst|p |j}ttdd|j|_|r#ttdd|jnd|_t |j |j |j |jd|_ |j j}ttd|d|j|_t|j|_|j |_ ||_dS)Nr) image_size patch_size num_channels embed_dim)super__init__ mask_tokenr Parameterr6zeros hidden_size cls_tokenPatchEmbeddingsrZr[r\patch_embeddings num_patchesposition_embeddingsDropouthidden_dropout_probdropoutrX)r+rXrYrg __class__r,r-r_s    zFlavaImageEmbeddings.__init__ embeddingsheightwidthc Cs|jdd}|jjdd}tjs||kr||kr|jS|jddddf}|jddddf}|jd}||j} ||j} t|d} |d| | |}|dddd}t j j || | fdd d }|dddd dd|}tj ||fdd S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rNg?rrbicubicF)sizemode align_cornersdim)shaperhr6jit is_tracingr[rreshapepermuter functional interpolateviewcat) r+rnrorprg num_positionsclass_pos_embedpatch_pos_embedrx new_height new_widthsqrt_num_positionsr,r,r-interpolate_pos_encodings(     z-FlavaImageEmbeddings.interpolate_pos_encoding pixel_valuesbool_masked_posrcCs|j\}}}}|j||d}|\}} } |durB|j|| d} |dkr0||dd}|d| } |d| | | }|j |dd} t j | |fdd}|r_|| |||}n||j }||}|S)N)rrqrr?rrw)ryrfrtr`expandrxr unsqueezetype_asrdr6rrrhrk)r+rrr batch_sizer\rorprnseq_len_ mask_tokensmask cls_tokensr,r,r-forward)s    zFlavaImageEmbeddings.forwardF)NF)r2r3r4r5rrCr_r6Tensorintrr BoolTensorr __classcell__r,r,rlr-rWs+rWc sheZdZdZ    ddedeeeeeffded effd d Zdd ej de dej fddZ Z S)rez# Image to Patch Embedding. rrZr[r\r]cstt|tjjs||f}t|tjjs||f}|d|d|d|d}||_||_||_t j ||||d|_ dS)Nrr) kernel_sizestride) r^r_ isinstance collectionsabcIterablerZr[rgrConv2d projection)r+rZr[r\r]rgrlr,r-r_Rs  zPatchEmbeddings.__init__Frrr#c Csx|j\}}}}|s.||jdks||jdkr.td|d|d|jdd|jdd ||ddd}|S)NrrzInput image size (*z) doesn't match model (z).rr)ryrZ ValueErrorrflatten transpose)r+rrrr\rorpxr,r,r-reszPatchEmbeddings.forward)rrrrr) r2r3r4r5rrr0r_r6rrCrrr,r,rlr-reMs $recsPeZdZdZfddZ   d deejdeejdeejfdd ZZ S) FlavaTextEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.csttj|j|j|jd|_t|j|j|_ t|j |j|_ tj |j|j d|_ t|j|_t|dd|_|jdt|jddd|jd tj|jtjd dddS) N) padding_idxepsposition_embedding_typeabsolute position_ids)rrqF) persistenttoken_type_ids)dtype)r^r_r Embedding vocab_sizerc pad_token_idword_embeddingsmax_position_embeddingsrhtype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsrirjrkr%rregister_bufferr6arangerrbrrtlongr+rXrlr,r-r_ts  zFlavaTextEmbeddings.__init__N input_idsrrc Cs|}|d}|dur|jddd|f}|durAt|dr6|jddd|f}||d|}|}n tj|tj|jjd}| |}| |} || } |j dkr]| |} | | 7} | | } || } | S)Nrrr)rdevicer)rtrhasattrrrr6rbrrrrrrhrrk) r+rrr input_shape seq_lengthbuffered_token_type_ids buffered_token_type_ids_expanded inputs_embedsrrnrhr,r,r-rs&       zFlavaTextEmbeddings.forwardNNN) r2r3r4r5r_rr6rrrr,r,rlr-rqs rcsteZdZdeddffdd Z   d dejdeejd eejd ede e ejejfe ejff d d Z Z S)FlavaSelfAttentionrXr#Ncst|j|jdkrt|dstd|jd|jd|j|_t|j|j|_|j|j|_t j |j|j|j d|_ t j |j|j|j d|_ t j |j|j|j d|_t |j|_dS)Nrembedding_sizezThe hidden size z4 is not a multiple of the number of attention heads .bias)r^r_rcnum_attention_headsrrrattention_head_size all_head_sizerLinearqkv_biasquerykeyvalueriattention_probs_dropout_probrkrrlr,r-r_s  zFlavaSelfAttention.__init__F hidden_statesattention_mask head_maskoutput_attentionscCs,|j\}}}|||d|j|jdd}|||d|j|jdd} |||d|j|jdd} t || dd} | t |j} |durS| |} t j j| dd} || } |durh| |} t | | } | dddd} | dd|jf}| j|} |r| | f}|S| f}|S)Nrqrrrrwrr)ryrrrrrrrr6matmulmathsqrtrr~softmaxrkr} contiguousrtr)r+rrrrrrr query_layer key_layer value_layerattention_scoresattention_probs context_layernew_context_layer_shapeoutputsr,r,r-rs:     zFlavaSelfAttention.forwardNNF) r2r3r4FlavaPossibleConfigsr_r6rrrCrr0rrr,r,rlr-rs rcsFeZdZdZdeddffdd Zdejdejdejfd d ZZ S) FlavaSelfOutputz The residual connection is defined in FlavaLayer (same as ViTLayer) instead of here (as is the case with other models), due to the layernorm applied before each block. rXr#Ncs.tt|j|j|_t|j|_dSrT) r^r_rrrcdenserirjrkrrlr,r-r_ zFlavaSelfOutput.__init__r input_tensorcC||}||}|SrTrrkr+rrr,r,r-r  zFlavaSelfOutput.forward) r2r3r4r5rr_r6rrrr,r,rlr-rs$rcseZdZdeddffdd ZdeeddfddZ   dd ej d e ej d e ej d e de e ej ej fe ej ff ddZZS)FlavaAttentionrXr#Ncs*tt||_t||_t|_dSrT)r^r_r attentionroutputset pruned_headsrrlr,r-r_s    zFlavaAttention.__init__headscCst|dkrdSt||jj|jj|j\}}t|jj||j_t|jj||j_t|jj ||j_ t|j j |dd|j _ |jjt||j_|jj|jj|j_ |j ||_dS)Nrrrw)lenrrrrrrrrrrrrunion)r+rindexr,r,r- prune_heads s zFlavaAttention.prune_headsFrrrrcCs8|j||||d}||d|}|f|dd}|SN)rrrrr)rr)r+rrrr self_outputsattention_outputrr,r,r-rs zFlavaAttention.forwardr)r2r3r4rr_rrrr6rrrCrr0rrr,r,rlr-rs"rcs<eZdZdeddffdd ZdejdejfddZZS) FlavaIntermediaterXr#NcsDtt|j|j|_t|jt rt |j|_ dS|j|_ dSrT) r^r_rrrcintermediate_sizerr hidden_actstrr intermediate_act_fnrrlr,r-r_0s   zFlavaIntermediate.__init__rcCrrT)rrr+rr,r,r-r9rzFlavaIntermediate.forward r2r3r4rr_r6rrrr,r,rlr-r/s rcsBeZdZdeddffdd Zdejdejdejfdd ZZS) FlavaOutputrXr#Ncs.tt|j|j|_t|j|_ dSrT) r^r_rrrrcrrirjrkrrlr,r-r_ArzFlavaOutput.__init__rrcCs ||}||}||}|SrTrrr,r,r-rGs  zFlavaOutput.forwardrr,r,rlr-r @s$r csxeZdZdZdeddffdd Z   ddejd eejd eejd e de e ejejfe ejff d d Z Z S) FlavaLayerz?This corresponds to the Block class in the timm implementation.rXr#Ncsbt|j|_d|_t||_t||_t||_ t j |j |j d|_t j |j |j d|_dS)Nrr)r^r_chunk_size_feed_forward seq_len_dimrrr intermediater rrrrcrlayernorm_beforelayernorm_afterrrlr,r-r_Ss    zFlavaLayer.__init__Frrrrc Csb|j|||||d}|d}|dd}||}||}||}|||}|f|}|Sr)rrrr r) r+rrrrself_attention_outputsrr layer_outputr,r,r-r_s     zFlavaLayer.forwardr)r2r3r4r5rr_r6rrrCrr0rrr,r,rlr-r Ps"r csneZdZdeddffdd Z     ddejd eejd eejd ed ed ede e e ffddZ Z S) FlavaEncoderrXr#Ncs:t|_tfddtjD|_d|_dS)Ncsg|]}tqSr,)r r(rrXr,r- sz)FlavaEncoder.__init__..F) r^r_rXr ModuleListrangenum_hidden_layerslayergradient_checkpointingrrlrr-r_s   zFlavaEncoder.__init__FTrrrroutput_hidden_states return_dictc Cs|rdnd}|r dnd}t|jD])\} } |r||f}|dur$|| nd} | ||| |} | d}|r:|| df}q|rB||f}|sPtdd|||fDSt|||dS)Nr,rrcss|] }|dur|VqdSrTr,)r(rBr,r,r-r.z'FlavaEncoder.forward..)last_hidden_stater attentions) enumeraterr0r ) r+rrrrrrall_hidden_statesall_self_attentionsi layer_modulelayer_head_mask layer_outputsr,r,r-rs$   zFlavaEncoder.forward)NNFFT)r2r3r4rr_r6rrrCrr0r rrr,r,rlr-r~s,  rcs2eZdZdeffdd ZdejfddZZS) FlavaPoolerrXcs*tt|j|j|_t|_dSrT)r^r_rrrcrTanh activationrrlr,r-r_s zFlavaPooler.__init__rcCs(|dddf}||}||}|SNr)rr))r+rfirst_token_tensor pooled_outputr,r,r-rs  zFlavaPooler.forwardrr,r,rlr-r'sr'c@s@eZdZUeed<dZdZdeej ej ej fddfddZ dS) FlavaPreTrainedModelrXflavaTmoduler#NcCsXt|tjtjfr#|jjjd|jjd|j dur!|j j dSdSt|tj rF|jjjd|jjd|j durD|jj|j  dSdSt|tj r[|j j |jjddSt|trh|j j dSt|tr|jj |jj |jdur|jj dSdSt|tr|jr|jj dSdSt|tr|jj|jjdSdS)zInitialize the weightsg)meanstdNr)rrrrweightdatanormal_rXinitializer_rangerzero_rrrfill_FlavaMaskedPredictionHeadrWrdrhr`FlavaMultimodalModel use_cls_token FlavaModel logit_scalelogit_scale_init_value)r+r/r,r,r- _init_weightss8            z"FlavaPreTrainedModel._init_weights) r2r3r4rr8base_model_prefixsupports_gradient_checkpointingrrrrrr>r,r,r,r-r-s &r-cseZdZUeed<dZdZddedeffdd Zde j fd d Z d e j fd d Z de eeefddfddZe        ddeejdeejdeedeejdeejdeedeedeedeeeffddZZS)FlavaImageModelrXzflava.image_modelrTadd_pooling_layercXt|||_t||_t||_tj|j |j d|_ |r#t |nd|_ |dSv add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer rN)r^r_rXrWrnrencoderrrrcr layernormr'pooler post_initr+rXrBrlr,r-r_s    zFlavaImageModel.__init__r#cC|jjSrTrnrfr*r,r,r-get_input_embeddingsz$FlavaImageModel.get_input_embeddingsrcC ||j_dSrTrLr+rr,r,r-set_input_embeddings z$FlavaImageModel.set_input_embeddingsheads_to_pruneNcC*|D]\}}|jj|j|qdSz Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel NitemsrFrrrr+rSrrr,r,r- _prune_headszFlavaImageModel._prune_headsrrrrrrrc Cs|dur|n|jj}|dur|n|jj}|dur|n|jj}|dur&td|||jj}|j|||d} |j| |||||d} | d} | | } |j durT| | nd} |sb| | f| ddSt | | | j | j dS)z bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, image_num_patches)`): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). Nz You have to specify pixel_values)rrrrrrrrrr pooler_outputrr)rXrruse_return_dictr get_head_maskrrnrFrGrHr rr) r+rrrrrrrrembedding_outputencoder_outputssequence_outputr,r,r,r-rs: zFlavaImageModel.forwardTNNNNNNNN)r2r3r4rr8r?main_input_namerCr_rModulerMrQdictrlistrYrrr6rrrr0r rrr,r,rlr-rAsH   rAcseZdZUeed<dZddedeffdd Zdefdd Z d e j fd d Z d e eeefddfddZe        ddeejdeejdeejdeejdeejdeedeedeedeeeffddZZS)FlavaTextModelrXzflava.text_modelTrBcrCrD)r^r_rXrrnrrFrrrcrrGr'rHrIrJrlr,r-r_As    zFlavaTextModel.__init__r#cCrKrTrnrr*r,r,r-rMQrNz#FlavaTextModel.get_input_embeddingsrcCrOrTrjrPr,r,r-rQTrRz#FlavaTextModel.set_input_embeddingsrSNcCrTrUrVrXr,r,r-rYWrZzFlavaTextModel._prune_headsrrrrrrrrc Cs|dur|n|jj}|dur|n|jj}|dur|n|jj}|dur&td|} |dur6tj| |jd}| ||jj }| || |j} |j |||d} |j | | ||||d} | d} || } |jdurl|| nd}|sz| |f| ddSt| || j| jdS) a input_ids (`torch.LongTensor` of shape `(batch_size, text_seq_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) token_type_ids (`torch.LongTensor` of shape `(batch_size, text_seq_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) NzYou have to specify input_idsr)rrrr[rrr\)rXrrr^rrtr6onesrr_rget_extended_attention_maskrnrFrGrHr rr)r+rrrrrrrrrextended_attention_maskr`rarbr,r,r,r-r_sJ zFlavaTextModel.forwardrcrd)r2r3r4rr8r?rCr_rerMrrfrQrgrrhrYrrr6rrr0r rrr,r,rlr-ri;sF   ricseZdZUeed<dZdZddeffdd Zdee e e fdd fd d Z e dde jd ee jd ee jdeedeedeedeeeffddZZS)r9rXzflava.multimodal_modelrTcsvt|||_|jj|_|jrttdd|j|_ t ||_ tj |j|j d|_|r2t|nd|_|dS)rErrN)r^r_rXr:rrar6rbrcrdrrFrrrGr'rHrIrJrlr,r-r_s    zFlavaMultimodalModel.__init__rSr#NcCrTrUrVrXr,r,r-rYrZz!FlavaMultimodalModel._prune_headsrrrrrcCs&|dur|n|jj}|dur|n|jj}|dur|n|jj}|\}}} |jr=|j|dd} tj | |fdd}|d7}|durKtj ||f|j d}| ||jj }||||f|j } |j|| ||||d} | d} || } |jdur{|| nd}|s| |f| ddSt| || j| jdS) z hidden_states (`torch.FloatTensor` of shape `(batch_size, image_num_patches + text_seq_len, hidden_size)`): The concatenated hidden states of unimodal encoders. Nrqrrwrkr[rr\)rXrrr^rtr:rdrr6rrlrr_rrmrFrGrHr rr)r+rrrrrrrrrrrnrarbr,r,r,r-rsD  zFlavaMultimodalModel.forwardrc)NNNNN)r2r3r4rr8r?rer_rgrrhrYrr6rrrCrr0r rrr,r,rlr-r9s6  r9csVeZdZUeed<deffdd Zee   ddej de ej de ej de ej d ej f d d Z ee    dd ej d e ej de ede ej de ej d ej f ddZe           dde ejd e ej de ej de ej d e ej de ejde ej de ede edede ed eeeffddZZS)r;rXcs0t|t|jtstdt|jdt|jts(tdt|jdt|j t s;tddt|j d|j}|j}|j }|j |_ |j |_ |j |_|j |_t||_t||_t||_t|j|j |_t|j |j |_tt|jj|_t|j|j|_ t|j |j|_!|"dS)NzLconfig.text_config is expected to be of type FlavaTextConfig but is of type rzNconfig.image_config is expected to be of type FlavaImageConfig but is of type zMconfig.multimodal_config is expected to be of type FlavaMultimodalConfig but z is of type )#r^r_r text_configr TypeErrortype image_configrmultimodal_configrprojection_dimrctext_hidden_sizeimage_hidden_sizemm_hidden_sizeri text_modelrA image_modelr9multimodal_modelrrimage_projectiontext_projectionrar6tensorrXr=r<image_to_mm_projectiontext_to_mm_projectionrI)r+rXrorrrsrlr,r-r_sF        zFlavaModel.__init__Nrrrrr#cCs&|j||||d}|j}||}|S)a input_ids (`torch.LongTensor` of shape `(batch_size, text_seq_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) token_type_ids (`torch.LongTensor` of shape `(batch_size, text_seq_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) Returns: text_features (`torch.FloatTensor` of shape `(batch_size, output_dim`): The text embeddings obtained by applying the projection layer to the pooled output of [`FlavaTextModel`]. Examples: ```python >>> import torch >>> from transformers import AutoProcessor, FlavaModel >>> model = FlavaModel.from_pretrained("{0}") >>> processor = AutoProcessor.from_pretrained("{0}") >>> inputs = processor( ... text=["a photo of a cat", "a photo of a dog"], max_length=77, padding="max_length", return_tensors="pt" ... ) >>> with torch.inference_mode(): ... text_features = model.get_text_features(**inputs) ``` )rrrr)rxrr|)r+rrrr text_outputsr, text_featuresr,r,r-get_text_features;s) zFlavaModel.get_text_featuresrrrrc Cs(|j|||||d}|j}||}|S)a bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, image_num_patches)`): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). Returns: image_features (`torch.FloatTensor` of shape `(batch_size, output_dim`): The image embeddings obtained by applying the projection layer to the pooled output of [`FlavaImageModel`]. Examples: ```python >>> import torch >>> from transformers import AutoProcessor, FlavaModel >>> from transformers.image_utils import load_image >>> model = FlavaModel.from_pretrained("{0}") >>> processor = AutoProcessor.from_pretrained("{0}") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = load_image(url) >>> inputs = processor(images=image, return_tensors="pt") >>> with torch.inference_mode(): ... image_features = model.get_image_features(**inputs) ``` )rrrrr)ryrr{) r+rrrrr image_outputsr,image_featuresr,r,r-get_image_featuresos% zFlavaModel.get_image_featuresTimage_attention_maskskip_multimodal_encoderrrrc  Csz| dur| n|jj} | stdd} d} d}d}|dur7|j|||| | | d}|d|d} } || d}d}d}d}d}|dur_|j||||| | | d}|d|d}}||d}d}d}|dur|dur|s|dur|j\}}}|jj r|d7}t j |||j d }t j ||gdd }nd}t j ||gdd }|j||| d }|d}| s| |||||fSt| |||||d S) a input_ids (`torch.LongTensor` of shape `(batch_size, image_num_patches + text_seq_len)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) token_type_ids (`torch.LongTensor` of shape `(batch_size, image_num_patches + text_seq_len)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, image_num_patches)`): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). image_attention_mask (`torch.Tensor` of shape `(batch_size, image_num_patches)`, *optional*): Mask to avoid performing attention on padding pixel values for image inputs. Mask values selected in `[0, 1]`: - 1 for pixel values that are real (i.e., **not masked**), - 0 for pixel values that are padding (i.e., **masked**). skip_multimodal_encoder (*bool*, *optional*): Skip any calculations for multimodal encoder. Useful if multimodal encoding is not going to be used. Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, FlavaModel >>> model = FlavaModel.from_pretrained("facebook/flava-full") >>> processor = AutoProcessor.from_pretrained("facebook/flava-full") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(text=["a photo of a cat"], images=image, return_tensors="pt", padding=True) >>> outputs = model(**inputs) >>> image_embeddings = outputs.image_embeddings >>> text_embeddings = outputs.text_embeddings >>> multimodal_embeddings = outputs.multimodal_embeddings >>> outputs.image_embeddings.shape torch.Size([1, 197, 768]) >>> text_embeddings.shape torch.Size([1, 7, 768]) >>> multimodal_embeddings.shape torch.Size([1, 205, 768]) ``` NzRFLAVA model requires hidden states to work. Please set `output_hidden_states=True`)rrrrrrrrrrq)rrrrrrrrrkrw)rr)rrrr r!r")rXrrryr~rxrryrzr:r6rlrrr)r+rrrrrrrrrrrr image_statesimage_mm_projectionrr text_statestext_mm_projectionr r!r"rrrattention_mask_imageattention_multimodalmultimodal_inputr,r,r-rsC   zFlavaModel.forwardr)NNNN) NNNNNNNNNTN)r2r3r4rr8r_rrr6rrr7rrrCr LongTensorrr0r rrr,r,rlr-r; s +2/     r;cs<eZdZdedeffdd ZdejdejfddZZS) FlavaImageCodebookResPathin_sizeout_sizec st|d}t}t|d<tj||ddd|d<t|d<tj||ddd|d<t|d <tj||ddd|d <t|d <tj||dd d|d <t||_dS)Nrelu_1rrrpaddingconv_1relu_2conv_2relu_3conv_3relu_4rconv_4)r^r_rrReLUr Sequentialpath)r+rrkwargshid_sizerrlr,r-r_0s     z"FlavaImageCodebookResPath.__init__rr#cC ||SrT)rr+rr,r,r-r@ z!FlavaImageCodebookResPath.forward r2r3r4rr_r6rrrr,r,rlr-r/srcs@eZdZdededeffdd Zdejdejfdd ZZS) FlavaImageCodebookBlockrr num_layersc sPtd|d|_||krtj||ddd|_nt|_t|||_dS)Nrrrrr) r^r_ post_gainrrid_pathIdentityrres_path)r+rrrrrlr,r-r_Es  z FlavaImageCodebookBlock.__init__rr#cCs|||j||SrT)rrrrr,r,r-rQszFlavaImageCodebookBlock.forwardrr,r,rlr-rDs rc sJeZdZd dededededef fdd Zd ejd ejfd d ZZ S)FlavaImageCodebookLayerGroupT num_blocksrrruse_poolcstt}t|D]!}|dkr t||||d|d<q t||||d|d<q |r8tjdd|d<t||_dS)Nrblock_rrr)rpool) r^r_rrrr MaxPool2drgroup)r+rrrrrblocksr#rlr,r-r_Vs  z%FlavaImageCodebookLayerGroup.__init__rr#cCrrT)rrr,r,r-rdrz$FlavaImageCodebookLayerGroup.forwardrc) r2r3r4rrCr_r6rrrr,r,rlr-rUs$ra" The FLAVA's image codebook model inspired from DALL-E's original encoder. Outputs raw hidden states and can be used to generate image tokens for an image based on DALL-E's vocab. Used to generate labels for MIM. Use `get_codebook_indices` to get image tokens for an image. cs~eZdZUdZeed<dZdZdedeffdd Z de j de j fd d Z de j de j fd d Z de jde j fd dZZS)FlavaImageCodebookrXrFrc sdt|||_|j|_|j|_|j|_|j|_|j|_|j|j}t}t |d<t j d|j|jddd|d<t}t j |jd|jddd|d <t |j|d|jd|j|d <t |j|d|jd |j|d <t |j|d |jd |j|d<t |j|d |jd|jdd|d<t ||d<t ||_||jjr|D]}d|_qdSdS)Nrelurrrconvrinputgroup_1rrgroup_2rgroup_3F)rgroup_4r)r^r_rX num_groupsinput_channelsnum_blocks_per_grouprcrrrrrrrrrIfreeze parameters requires_grad)r+rXrr output_blocksrparamrlr,r-r_vsB      zFlavaImageCodebook.__init__r#cCs*dtdtd||}tj|ddS)Na) Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Codebook pixel values can be obtained using [`AutoImageProcessor`] by passing `return_codebook_pixels=True`. See [`FlavaImageProcessor.__call__`] for details. Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoImageProcessor, FlavaImageCodebook >>> model = FlavaImageCodebook.from_pretrained("E") >>> image_processor = AutoImageProcessor.from_pretrained("a") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = image_processor([image], return_codebook_pixels=True, return_tensors="pt") >>> inputs = dict(pixel_values=inputs.codebook_pixel_values) >>> outputs = model.get_codebook_indices(**inputs) ``` r)axis)_CHECKPOINT_FOR_CODEBOOK_DOCrr6argmaxr+rz_logitsr,r,r-get_codebook_indicess   z'FlavaImageCodebook.get_codebook_indicescCs||}tjdd|S)Nrrw)rrSoftmaxrr,r,r-get_codebook_probss z%FlavaImageCodebook.get_codebook_probscCshdtdtdt|jdkrtd|jd|jd|jkr/td|jdd |j||S) Na* Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Codebook pixel values can be obtained using [`AutoImageProcessor`] by passing `return_codebook_pixels=True`. See [`FlavaImageProcessor.__call__`] for details. Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoImageProcessor, FlavaImageCodebook >>> model = FlavaImageCodebook.from_pretrained("ra") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = image_processor([image], return_codebook_pixels=True, return_tensors="pt") >>> inputs = dict(pixel_values=inputs.codebook_pixel_values) >>> outputs = model(**inputs) >>> print(outputs.shape) (1, 196) ``` rz input shape z is not 4drz input has z channels but model built for )rrryrrr)r+rr,r,r-rs  zFlavaImageCodebook.forward)r2r3r4r?rr8rer@rr_r6rrrr7rrr,r,rlr-ris ,rc$eZdZfddZddZZS)FlavaPredictionHeadTransformcsVtt|j|j|_t|jtrt |j|_ n|j|_ tj |j|j d|_ dS)Nr) r^r_rrrcrrrrr transform_act_fnrrrrlr,r-r_s  z%FlavaPredictionHeadTransform.__init__cCs"||}||}||}|SrT)rrrrr,r,r-rs   z$FlavaPredictionHeadTransform.forwardr2r3r4r_rrr,r,rlr-rs  rcs.eZdZdfdd ZddZddZZS) r8Ncsbt||_t||_tj|j|jdd|_ t t |j|_ |dur*||j _|j |j _ dS)NFr)r^r_rXr transformrrrcrdecoderrar6rbrr2)r+rXr2rlr,r-r_s  z"FlavaMaskedPredictionHead.__init__cCs|j|j_dSrT)rrr*r,r,r- _tie_weightssz&FlavaMaskedPredictionHead._tie_weightscCrrT)rrrr,r,r-r  z!FlavaMaskedPredictionHead.forwardrT)r2r3r4r_rrrr,r,rlr-r8s r8cr) FlavaITMHeadcs.t||_t||_t|jd|_dS)Nrr) r^r_rXr'rHrrrcseq_relationshiprrlr,r-r_ s  zFlavaITMHead.__init__cCrrT)rHrrr,r,r-rrzFlavaITMHead.forwardrr,r,rlr-r s rcr)FlavaGlobalContrastiveHeadcst||_|j|_dSrT)r^r_rXglobal_backprop_contrastiverrlr,r-r_s  z#FlavaGlobalContrastiveHead.__init__c s2t|}tjrtjs!tjdjd}g}g}nQd}tj} |j r?tjj j }tjj j }n$fddt | D}fddt | D}tj |tj ||tjtj|jd}t|}t|}t|dd|} t|dd|} | | |fS)Nrrkcg|]}tqSr,r6 zeros_liker)rr,r-r/z6FlavaGlobalContrastiveHead.forward..crr,rr)rr,r-r0rr)r6exp distributed is_availableis_initializedrrtrget_world_sizerrr~ all_gatherrget_rankrrr) r+rrr< temperaturelabelsimage_embeddings_alltext_embeddings_alllocal_batch_size world_sizelogits_per_imagelogits_per_textr,)rrr-rs,      z"FlavaGlobalContrastiveHead.forwardrr,r,rlr-rs rzk The FLAVA model for pretraining which outputs losses, embeddings, logits and transformer outputs. c(s&eZdZgdZddedeejffdd Zde j fdd Z e                d d ee j d ee j d ee jdee jdee j dee j dee j dee j dee j deedee j dee j dee j deededeedeedeee j eff$ddZZS)!FlavaForPreTraining)zmmm_text_head.decoder.biaszmmm_image_head.decoder.biaszmlm_head.decoder.biaszmim_head.decoder.biasNrXimage_codebookcst|t||_||_|jdur|jrt|j|_t|j |_ t|j |_ t ||_t|j |_t|j |_t||_|j j|_|j j|_|j|_|j|_|j|_|j|_|j|_|j|_|j|_|j|_|dS)z image_codebook ([`nn.Module`]): If passed, the image codebook will be set to this. Otherwise, it will be initialized using the image_codebook_config defined in the config first as the first parameter. N)r^r_r;r.r init_codebookrimage_codebook_configr8rrmim_headromlm_headritm_headmmm_image_head mmm_text_headrglobal_contrastive_headrimage_vocab_sizetext_vocab_size mlm_weight mim_weightglobal_contrastive_weightce_ignore_index itm_weightmmm_image_weightmmm_text_weight skip_unmasked_multimodal_encoderrI)r+rXrrlr,r-r_Os,            zFlavaForPreTraining.__init__rcCs"|dkr||dd}|S)Nrrrrq)rxrrtrr,r,r- _resize_to_2drs z!FlavaForPreTraining._resize_to_2dTrinput_ids_maskedrcodebook_pixel_valuesrrrrrr mlm_labels mim_labels itm_labelsrrr return_lossr#c6Cs|dur|n|jj}|dur|n|jj}| dur| n|j} |dur,|dur,td|}|j|||||| | ||dd }|j||||| |||dd }d}|j}|j}|j}|j}|j }d}}}}}}} d}!}"}#}$d}%}&}'|dus~|dur| dur|r|j durt d|durt d|j |} |jdkr |dur |dur |}(| dur|| } ||}|j| |d<|(dd| d  dddf}(| |j})| |)}*|(|)ddf}(||(}!|rtj|!d |j|*d }||j9}n||(}!|jdkrj|durj|durj|}+| dure|| } |+dd| d  dddf}+| |j})| |)},|+|)ddf}+||+}"|rdtj|"d |j|,d }||j9}n||+}"|jdkr|dur||}%| dur| d}-t|- |-|-!dg}|rtj|%| } | |j9} |dur||}| dur| |} | dur| |} ||}|dur-|j"dkr-|}(|d d }.|(ddd d |.ddf}(| dur(|| } ||}|j| |d<| |j})| |)}*|(|)ddf}(|#|(}$|r'tj|$d |j|*d }||j"9}n|#|(}$|dur|j$dkr|}+|+dd|d  dddf}+| dur|| } | |j})| |)},|+|)ddf}+|%|+}#|rtj|#d |j|,d }||j$9}n|%|+}#|dur |dur |j&dkr |j'|dddddf}/tjj(|/d d }/|j)|dddddf}0tjj(|0d d }0|jj*j+,t-t.|/|0|/|jj*\}&}'}1|dur|&|}&|'|}'|1|}1|r tj|&|1}2tj|'|1}3|2|3d }||j&9}t0||| |||d }4|r&|41s&t2dd|43D}|s||j4dur5|j45nd||j6durB|j65nd|j |j7durP|j75nd||j4dur]|j45nd||j6durj|j65nd||j7durw|j75nd|!|"|%|&|&|$|#f}5|r|41s||4f|5}5t8dd|5DSt9d&id|d|4d|d|j4d|d|j6d|j d|j7d|d|j4d|d|j6d|d|j7d|!d |"d!|%d"|&d#|'d$|$d%|#S)'a input_ids (`torch.LongTensor` of shape `(batch_size, text_seq_len)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) input_ids_masked (`torch.LongTensor` of shape `(batch_size, text_seq_len)`): Indices of input sequence tokens in the vocabulary. These ones are the masked version of the original task to be used with MLM. Indices can be obtained using [`AutoTokenizer`] along with [`DataCollatorForMaskedLanguageModeling`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) codebook_pixel_values (`torch.FloatTensor` of shape `(batch_size, num_image_patches, patch_size, patch_size, 3)`, *optional*): Pixel values for image patches that are used to compute the image codebook labels for masked image modeling. token_type_ids (`torch.LongTensor` of shape `(batch_size, text_seq_len)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, image_num_patches)`): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). image_attention_mask (`torch.FloatTensor` of shape `(batch_size, image_num_patches)`, *optional*): Mask to avoid performing attention on padding token indices specifically for images. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) skip_unmasked_multimodal_encoder (*bool*, *optional*): Skip any calculations for multimodal encoder for unmasked inputs. FLAVA pretraining doesn't need unmasked multimodal embeddings or outputs as of now. mlm_labels (`torch.LongTensor` of shape `(batch_size, text_seq_len)`, *optional*): Labels for computing the left-to-right language and multimodal masked modeling loss (next word prediction). Indices should be in `[-100, 0, ..., text_config.vocab_size - 1]` (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., text_config.vocab_size - 1]`. mim_labels (`torch.LongTensor` of shape `(batch_size, image_num_patches)`, *optional*): Labels for computing the image and multimodal masked modeling loss. Indices should be in `[-100, 0, ..., image_config.vocab_size - 1]`. Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., image_config.vocab_size - 1]`. If not passed, they are generated automatically using the image codebook assigned to the model. By default, it uses [`FlavaImageCodebook`]. See [`FlavaImageCodebook`] to understand how to generate mim_labels. itm_labels (`torch.LongTensor` of shape `(batch_size, 1)`, *optional*): Labels for computing the image-text matching loss. 0 means the pairs don't match and 1 means they match. The pairs with 0 will be skipped for calculation of MMM and global contrastive losses as well. return_loss (`bool`, *optional*, default to None): Whether to return calculated loss or not. Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import FlavaForPreTraining, AutoProcessor >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> model = FlavaForPreTraining.from_pretrained("facebook/flava-full") >>> processor = AutoProcessor.from_pretrained("facebook/flava-full") >>> text = ["a photo of a cat"] >>> inputs = processor( ... images=[image], ... text=text, ... return_masks=True, ... return_codebook_pixels=True, ... padding=True, ... max_length=77, ... return_tensors="pt", ... ) >>> output = model(**inputs) ``` Nz`input_ids_masked` isn't passed which means MLM loss won't be calculated correctlySetting it to `input_ids` so that model can work. Please pass it if this is unintentional. This is usually OKAY if you are doing inference on unmasked text...T) rrrrrrrrrr) rrrrrrrrrz`return_loss` is set to True but the image codebook is not initialized and no `mim_labels` have been passed. Reinstantiate the model with `init_codebook` set to True or pass in your custom `mim_labels`z`codebook_pixel_value` are required to generate `mim_labels` if loss is expected. Call `AutoProcessor` with `return_codebook_pixels` set to Truerrrqrrrw)r:r;r<r=r>r?css |] }|dur |ndVqdSr*r,)r(rEr,r,r-r.sz.FlavaForPreTraining.forward..css|] }|dur|VqdSrTr,)r(rr,r,r-r.rrErFrrrr r!r"rGrHrIrJrKrLrMrNrOrPrQrRrSr,):rXr^r rloggerwarningr.rrr!r RuntimeErrorrrrrrnertrrr~ cross_entropyrrrrrrrr6whereanynewrrrrrr| normalizer{r<r3clamp_LOGIT_SCALE_CLAMP_MINLOGIT_SCALE_CLAMP_MAXrr9rAsumr@rr&r r"r0rD)6r+rrrrrrrrrrrrr rrrr  flava_outputflava_masked_outputpos_maskrrrGrIrK total_lossmim_lossmlm_loss mmm_text_lossmmm_image_lossgc_lossitm_lossrMrNrSrRrOrrsequence_for_image masked_tokensmim_labels_filteredsequence_for_textmlm_labels_filtered pos_pairs end_indextext_embeddingimage_embedding gc_labels gc_loss_image gc_loss_text flava_lossesrr,r,r-rws_         "       "                  "                   zFlavaForPreTraining.forwardrT)NNNNNNNNNNNNNNTNN)r2r3r4_tied_weights_keysrrrrfr_r6rrrrr7rCrr0rDrrr,r,rlr-rAsv#     r)rrrAr;r9r-ri)Ir5rrr dataclassesrtypingrrrr6r activationsr modeling_layersr modeling_outputsr r modeling_utilsr pytorch_utilsrrutilsrrrrrconfiguration_flavarrrrr get_loggerr2r rrrrrr9rDrfrWrerrrrrr r rr'r-rArir9r;rrrrrr8rrr__all__r,r,r,r-s        " \d$9I*.*"`p_$u ( #