o wiq@s0dZddlZddlZddlmZddlmZddlmZmZm Z ddl Z ddl Z ddl m Z ddl mZdd lmZdd lmZmZdd lmZmZmZdd lmZmZmZmZd dlmZmZm Z m!Z!m"Z"e#e$Z%dZ&dZ'dZ(e e"e e!fZ)eeddGdddeZ*eeddGdddeZ+eeddGdddeZ,Gddde j-Z.Gddde j-Z/Gdd d e j-Z0Gd!d"d"e j-Z1Gd#d$d$e j-Z2Gd%d&d&e j-Z3Gd'd(d(e j-Z4Gd)d*d*e j-Z5Gd+d,d,eZ6Gd-d.d.e j-Z7Gd/d0d0e j-Z8eGd1d2d2eZ9eGd3d4d4e9Z:eGd5d6d6e9Z;eGd7d8d8e9ZGd=d>d>e j-Z?Gd?d@d@e j-Z@edAdGdBdCdCe9ZAGdDdEdEe j-ZBGdFdGdGe j-ZCGdHdIdIe j-ZDGdJdKdKe j-ZEedLdGdMdNdNe9ZFgdOZGdS)PzPyTorch FLAVA model.N) OrderedDict) dataclass)AnyOptionalUnion)nn)ACT2FN)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPooling)PreTrainedModel find_pruneable_heads_and_indicesprune_linear_layer) ModelOutputauto_docstringlogging torch_int) FlavaConfigFlavaImageCodebookConfigFlavaImageConfigFlavaMultimodalConfigFlavaTextConfigzfacebook/flava-image-codebookg$(~k@a Output from FlavaModel containing embeddings and outputs from individual encoders. Note that `image_embeddings` and `text_embeddigns` returned are similar to pooled output returned from a transformer. If you want embeddings for contrastive loss or retrieval use a FLAVA model's `image_projection` and `text_projection` layers on `image_embeddings` and `text_embeddings` respectively. ) custom_introc@seZdZUdZdZeejed<dZ ee ed<dZ eejed<dZ ee ed<dZ eejed<dZee ed<d eefd d ZdS) FlavaModelOutputa image_embeddings (`torch.FloatTensor` of shape `(batch_size, output_dim)`, *optional*, returned when `pixel_values` are present): The image embeddings which are basically the pooled output of [`FlavaImageModel`]. image_output (`BaseModelOutputWithPooling`, *optional*, returned when `pixel_values` are present): The output of the [`FlavaImageModel`]. text_embeddings (`torch.FloatTensor` of shape `(batch_size, output_dim)`, *optional*, returned when `input_ids` are present): The text embeddings which are basically the pooled output of [`FlavaTextModel`]. text_output (`BaseModelOutputWithPooling`, *optional*, returned when `input_ids` are present): The output of the [`FlavaTextModel`]. multimodal_embeddings (`torch.FloatTensor` of shape `(batch_size, output_dim)`, *optional*, returned when `input_ids` and `pixel_values` are present and `skip_multimodal_encoder` is `None` or `False`): The multimodal embeddings which are basically the pooled output of [`FlavaTextModel`]. multimodal_output (`BaseModelOutputWithPooling`, returned when `input_ids` and `pixel_values` are present and `skip_multimodal_encoder` is `None` or `False`): The output of the [`FlavaMultimodalModel`]. Nimage_embeddings image_outputtext_embeddings text_outputmultimodal_embeddingsmultimodal_outputreturncstfddDS)Nc3s.|]}|dvr |nt|VqdS))rrr!Ngetattrto_tuple.0kselfe/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/transformers/models/flava/modeling_flava.py Us  z,FlavaModelOutput.to_tuple..tuplekeysr)r+r)r,r%Ts zFlavaModelOutput.to_tuple)__name__ __module__ __qualname____doc__rrtorch FloatTensor__annotations__rr rrr r!r/rr%r+r+r+r,r3s  rz@ Class representing pretraining losses from FLAVA model c@seZdZUdZdZeejed<dZ eejed<dZ eejed<dZ eejed<dZ eejed<dZ eejed<d efd d ZdS) FlavaLossesa mim (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `mim_labels` and `pixel_values` are present, `input_ids_masked` is absent and `mim_weight` > 0.): Masked Image Modeling loss as used in BeIT calculated only for unimodal image data. mlm (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `mlm_labels` and `input_ids_masked` are present, `pixel_values` is absent and `mlm_weight` > 0.): Masked Language Modeling loss as used in BERT calculated only for unimodal text data. itm (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `itm_labels`, `input_ids_masked`, `pixel_values` are present and `itm_weight` > 0.): Image Text Matching (ITM) loss calculated for paired image-text data. Note that ITM loss is calculated on masked pairs in FLAVA. global_contrastive (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `input_ids` and `pixel_values` are present and `global_contrastive_weight` > 0.): Contrastive loss for image-text similarity similar to CLIP but calculated globally for paired image-text data. This is calculated on unmasked images and texts. mmm_image (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `mim_labels`, `pixel_values` and `input_ids_masked` are present and `mmm_image_weight` > 0.): Masked Multimodal Modeling loss's image component calculated on paired image-text data. mmm_text (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `mlm_labels`, `pixel_values` and `input_ids_masked` are present and `mmm_text_weight` > 0.): Masked Multimodal Modeling loss's text component calculated on paired image-text data. Nmimmlmitmglobal_contrastive mmm_imagemmm_textr"cCs(d}|D] }|durd}|Sq|S)NTF)values)r*all_nonevr+r+r,r@zs zFlavaLosses.all_none)r1r2r3r4r9rr5r6r7r:r;r<r=r>boolr@r+r+r+r,r8[s r8a Output from FlavaForPreTraining containing embeddings, and outputs from individual encoders. Note that `image_embeddings` and `text_embeddings` returned are similar to pooled output returned from a transformer. If you want embeddings for contrastive loss or retrieval use a FLAVA model's `image_projection` and `text_projection` layers on `image_embeddings` and `text_embeddings` respectively. c@seZdZUdZdZeejed<dZ e ed<dZ eejed<dZ ee ed<dZeejed<dZee ed<dZeejed <dZee ed <dZeejed <dZee ed <dZeejed <dZee ed<dZeejed<dZee ed<dZeejed<dZeejed<dZeejed<dZeejed<dZeejed<dZeejed<dZeejed<dee fddZ!dS)FlavaForPreTrainingOutputay loss (`torch.FloatTensor`, *optional*, returned when `return_loss` is True): Total loss calculated for this model. loss_info (`FlavaLosses`): Detailed info for FLAVA Pretraining losses. Check `FlavaLosses` class description for the information on the keys. image_embeddings (`torch.FloatTensor` of shape `(batch_size, output_dim)`, *optional*, returned when `pixel_values` are present): The image embeddings which are basically the pooled output of [`FlavaImageModel`]. image_output (`BaseModelOutputWithPooling`, *optional*, returned when `pixel_values` are present): The output of the [`FlavaImageModel`]. text_embeddings (`torch.FloatTensor` of shape `(batch_size, output_dim)`, *optional*, returned when `input_ids` are present): The text embeddings which are basically the pooled output of [`FlavaTextModel`]. text_output (`BaseModelOutputWithPooling`, *optional*, returned when `input_ids` are present): The output of the [`FlavaTextModel`]. multimodal_embeddings (`torch.FloatTensor` of shape `(batch_size, output_dim)`, *optional*, returned when `input_ids` and `pixel_values` are present and `skip_unmasked_multimodal_encoder` is `None` or `False`): The multimodal embeddings which are basically the pooled output of [`FlavaTextModel`]. multimodal_output (`BaseModelOutputWithPooling`, returned when `input_ids` and `pixel_values` are present and `skip_unmasked_multimodal_encoder` is `None` or `False`): The output of the [`FlavaMultimodalModel`]. image_masked_embeddings (`torch.FloatTensor` of shape `(batch_size, output_dim)`, *optional*, returned when `pixel_values` are present): The image embeddings which are basically the pooled output of [`FlavaImageModel`]. Uses `bool_masked_pos` to create masked images. image_masked_output (`BaseModelOutputWithPooling`, *optional*, returned when `pixel_values` are present): The output of the [`FlavaImageModel`]. Uses `bool_masked_pos` to create masked images. text_masked_embeddings (`torch.FloatTensor` of shape `(batch_size, output_dim)`, *optional*, returned when `input_ids_masked` are present): The text embeddings which are basically the pooled output of [`FlavaTextModel`]. text_masked_output (`BaseModelOutputWithPooling`, *optional*, returned when `input_ids_masked` are present): The output of the [`FlavaTextModel`]. multimodal_masked_embeddings (`torch.FloatTensor` of shape `(batch_size, output_dim)`, *optional*, returned when `input_ids` and `pixel_values` are present): The multimodal embeddings which are basically the pooled output of [`FlavaTextModel`]. multimodal_masked_output (`BaseModelOutputWithPooling`, *optional*, returned when `input_ids_masked` and `pixel_values` are present): The output of the [`FlavaMultimodalModel`]. mim_logits (`torch.FloatTensor` of shape `(batch_size, num_image_patches, image_vocab_size)` or of shape `(total_masked_patches, image_vocab_size)` , *optional*, returned when `pixel_values` are present and `input_ids_masked` are not): The logits for MIM unimodal loss. Uses `book_masked_pos` to get masked patches. The flattened output is returned when `bool_masked_pos` has some of the patches masked. mlm_logits (`torch.FloatTensor` of shape `(batch_size, text_seq_length, text_vocab_size)` or of shape `(total_masked_seq_length, text_vocab_size)`, *optional*, returned when `input_ids_masked` are present and `pixel_values` are not): The logits for MLM unimodal loss. The flattened output is returned when `input_ids_masked` has some of the tokens masked. itm_logits (`torch.FloatTensor` of shape `(batch_size, 2)`, *optional*, returned when `input_ids_masked` and `pixel_values` are present): The logits for ITM loss. Note that ITM loss is calculated on masked pairs in FLAVA. contrastive_logits_per_image (`torch.FloatTensor` of shape `(image_batch_size, text_batch_size)`): The scaled dot product scores between `image_embeddings` and `text_embeddings` but passed through FLAVA's `image_projection` and `text_projection` layers respectively. This represents the image-text similarity scores. This is calculated on unmasked images and texts. contrastive_logits_per_text (`torch.FloatTensor` of shape `(text_batch_size, image_batch_size)`): The scaled dot product scores between `text_embeddings` and `image_embeddings` but passed through FLAVA's `text_projection` and `image_projection` layers respectively. This is calculated on unmasked images and texts. mmm_image_logits (`torch.FloatTensor` of shape `(batch_size, num_image_patches, image_vocab_size)` or of shape`(total_masked_patches, image_vocab_size)`, *optional*, returned when `pixel_values` and `input_ids_masked` are present): The logits for MMM image multimodal loss. Uses `book_masked_pos` to get masked patches. The flattened output is returned when `bool_masked_pos` has some of the patches masked. mmm_text_logits (`torch.FloatTensor` of shape `(batch_size, text_seq_length, text_vocab_size)` or of shape `(`(total_masked_seq_length, text_vocab_size)`), *optional*, returned when `pixel_values` and `input_ids_masked` are present): The logits for MMM text multimodal loss. The flattened output is returned when `input_ids_masked` has some of the tokens masked. Nloss loss_inforrrrr r!image_masked_embeddingsimage_masked_outputtext_masked_embeddingstext_masked_outputmultimodal_masked_embeddingsmultimodal_masked_output mim_logits mlm_logits itm_logitscontrastive_logits_per_imagecontrastive_logits_per_textmmm_image_logitsmmm_text_logitsr"cs$gdtfddDS)N)rrr!rIrGrKc3s.|]}|vr |nt|VqdSNr#r&r*transformer_outputsr+r,r-s,z5FlavaForPreTrainingOutput.to_tuple..r.r)r+rTr,r%sz"FlavaForPreTrainingOutput.to_tuple)"r1r2r3r4rDrr5r6r7rEr8rrr rrr r!rFrGrHrIrJrKrLrMrNrOrPrQrRr/rr%r+r+r+r,rCs0  7 rCc sxeZdZdZddededdffdd Zd ejd e d e dejfd d Z  ddejde ej dedejfddZ ZS)FlavaImageEmbeddingszb Construct the CLS token, position and patch embeddings. Optionally, also the mask token. Fconfiguse_mask_tokenr"Ncst|p |j}ttdd|j|_|r#ttdd|jnd|_t |j |j |j |jd|_ |j j}ttd|d|j|_t|j|_|j |_ ||_dS)Nr) image_size patch_size num_channels embed_dim)super__init__ mask_tokenr Parameterr5zeros hidden_size cls_tokenPatchEmbeddingsrYrZr[patch_embeddings num_patchesposition_embeddingsDropouthidden_dropout_probdropoutrW)r*rWrXrf __class__r+r,r^s    zFlavaImageEmbeddings.__init__ embeddingsheightwidthc Cs|jdd}|jjdd}tjs||kr||kr|jS|jddddf}|jddddf}|jd}||j} ||j} t|d} |d| | |}|dddd}t j j || | fdd d }|dddd dd|}tj ||fdd S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rNg?rrbicubicF)sizemode align_cornersdim)shapergr5jit is_tracingrZrreshapepermuter functional interpolateviewcat) r*rmrnrorf num_positionsclass_pos_embedpatch_pos_embedrw new_height new_widthsqrt_num_positionsr+r+r,interpolate_pos_encodings(     z-FlavaImageEmbeddings.interpolate_pos_encoding pixel_valuesbool_masked_posrcCs|j\}}}}|j||d}|\}} } |durB|j|| d} |dkr0||dd}|d| } |d| | | }|j |dd} t j | |fdd}|r_|| |||}n||j }||}|S)N)rrprr?rrv)rxrersr_expandrwr unsqueezetype_asrcr5rrrgrj)r*rrr batch_sizer[rnrormseq_len_ mask_tokensmask cls_tokensr+r+r,forward)s    zFlavaImageEmbeddings.forwardF)NF)r1r2r3r4rrBr^r5Tensorintrr BoolTensorr __classcell__r+r+rkr,rVs+rVc sheZdZdZ    ddedeeeeeffded effd d Zdd ej de dej fddZ Z S)rdz# Image to Patch Embedding. rrYrZr[r\cstt|tjjs||f}t|tjjs||f}|d|d|d|d}||_||_||_t j ||||d|_ dS)Nrr) kernel_sizestride) r]r^ isinstance collectionsabcIterablerYrZrfrConv2d projection)r*rYrZr[r\rfrkr+r,r^Rs  zPatchEmbeddings.__init__Frrr"c Csx|j\}}}}|s.||jdks||jdkr.td|d|d|jdd|jdd ||ddd}|S)NrrzInput image size (*z) doesn't match model (z).rq)rxrY ValueErrorrflatten transpose)r*rrrr[rnroxr+r+r,reszPatchEmbeddings.forward)rrrrr) r1r2r3r4rrr/r^r5rrBrrr+r+rkr,rdMs $rdcsPeZdZdZfddZ   d deejdeejdeejfdd ZZ S) FlavaTextEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.csttj|j|j|jd|_t|j|j|_ t|j |j|_ tj |j|j d|_ t|j|_t|dd|_|jdt|jddd|jd tj|jtjd dddS) N) padding_idxepsposition_embedding_typeabsolute position_ids)rrpF) persistenttoken_type_ids)dtype)r]r^r Embedding vocab_sizerb pad_token_idword_embeddingsmax_position_embeddingsrgtype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsrhrirjr$rregister_bufferr5arangerrarrslongr*rWrkr+r,r^ts  zFlavaTextEmbeddings.__init__N input_idsrrc Cs|}|d}|dur|jddd|f}|durAt|dr6|jddd|f}||d|}|}n tj|tj|jjd}| |}| |} || } |j dkr]| |} | | 7} | | } || } | S)Nrrr)rdevicer)rsrhasattrrrr5rarrrrrrgrrj) r*rrr input_shape seq_lengthbuffered_token_type_ids buffered_token_type_ids_expanded inputs_embedsrrmrgr+r+r,rs&       zFlavaTextEmbeddings.forward)NNN) r1r2r3r4r^rr5rrrr+r+rkr,rqs rcseZdZdeddffdd ZdejdejfddZ   dd ejd eejd eejd e de e ejejfe ejff ddZ Z S)FlavaSelfAttentionrWr"Ncst|j|jdkrt|dstd|jd|jd|j|_t|j|j|_|j|j|_t j |j|j|j d|_ t j |j|j|j d|_ t j |j|j|j d|_t |j|_dS)Nrembedding_sizezThe hidden size z4 is not a multiple of the number of attention heads .bias)r]r^rbnum_attention_headsrrrattention_head_size all_head_sizerLinearqkv_biasquerykeyvaluerhattention_probs_dropout_probrjrrkr+r,r^s  zFlavaSelfAttention.__init__rcCs6|dd|j|jf}|j|}|ddddS)Nrprrqrr)rsrrrr|)r*r new_x_shaper+r+r,transpose_for_scoress z'FlavaSelfAttention.transpose_for_scoresF hidden_statesattention_mask head_maskoutput_attentionscCs||}|||}|||}||}t||dd} | t|j } |dur4| |} t j j | dd} | | } |durI| |} t| |} | dddd} | dd|jf} | j| } |rr| | f} | S| f} | S)Nrprvrrqrr)rrrrr5matmulrmathsqrtrrr}softmaxrjr| contiguousrsrr)r*rrrrmixed_query_layer key_layer value_layer query_layerattention_scoresattention_probs context_layernew_context_layer_shapeoutputsr+r+r,rs(      zFlavaSelfAttention.forwardNNF)r1r2r3FlavaPossibleConfigsr^r5rrrrBrr/rrr+r+rkr,rs"rcsFeZdZdZdeddffdd Zdejdejdejfd d ZZ S) FlavaSelfOutputz The residual connection is defined in FlavaLayer (same as ViTLayer) instead of here (as is the case with other models), due to the layernorm applied before each block. rWr"Ncs.tt|j|j|_t|j|_dSrS) r]r^rrrbdenserhrirjrrkr+r,r^ zFlavaSelfOutput.__init__r input_tensorcC||}||}|SrSrrjr*rrr+r+r,r  zFlavaSelfOutput.forward) r1r2r3r4rr^r5rrrr+r+rkr,rs$rcseZdZdeddffdd ZdeeddfddZ   dd ej d e ej d e ej d e de e ej ej fe ej ff ddZZS)FlavaAttentionrWr"Ncs*tt||_t||_t|_dSrS)r]r^r attentionroutputset pruned_headsrrkr+r,r^s    zFlavaAttention.__init__headscCst|dkrdSt||jj|jj|j\}}t|jj||j_t|jj||j_t|jj ||j_ t|j j |dd|j _ |jjt||j_|jj|jj|j_ |j ||_dS)Nrrrv)lenrrrrrrrrrrrrunion)r*rindexr+r+r, prune_headss zFlavaAttention.prune_headsFrrrrcCs8|j||||d}||d|}|f|dd}|SN)rrrrr)rr)r*rrrr self_outputsattention_outputrr+r+r,rs zFlavaAttention.forwardr)r1r2r3rr^rrrr5rrrBrr/rrr+r+rkr,rs"rcs<eZdZdeddffdd ZdejdejfddZZS) FlavaIntermediaterWr"NcsDtt|j|j|_t|jt rt |j|_ dS|j|_ dSrS) r]r^rrrbintermediate_sizerr hidden_actstrr intermediate_act_fnrrkr+r,r^*s   zFlavaIntermediate.__init__rcCrrS)rrr*rr+r+r,r3rzFlavaIntermediate.forward r1r2r3rr^r5rrrr+r+rkr,r)s rcsBeZdZdeddffdd Zdejdejdejfdd ZZS) FlavaOutputrWr"Ncs.tt|j|j|_t|j|_ dSrS) r]r^rrrrbrrhrirjrrkr+r,r^;rzFlavaOutput.__init__rrcCs ||}||}||}|SrSrrr+r+r,rAs  zFlavaOutput.forwardr r+r+rkr,r :s$r csxeZdZdZdeddffdd Z   ddejd eejd eejd e de e ejejfe ejff d d Z Z S) FlavaLayerz?This corresponds to the Block class in the timm implementation.rWr"Ncsbt|j|_d|_t||_t||_t||_ t j |j |j d|_t j |j |j d|_dS)Nrr)r]r^chunk_size_feed_forward seq_len_dimrrr intermediater rrrrbrlayernorm_beforelayernorm_afterrrkr+r,r^Ms    zFlavaLayer.__init__Frrrrc Csb|j|||||d}|d}|dd}||}||}||}|||}|f|}|Sr)rrrrr) r*rrrrself_attention_outputsrr layer_outputr+r+r,rYs     zFlavaLayer.forwardr)r1r2r3r4rr^r5rrrBrr/rrr+r+rkr,r Js"r csneZdZdeddffdd Z     ddejd eejd eejd ed ed ede e e ffddZ Z S) FlavaEncoderrWr"Ncs:t|_tfddtjD|_d|_dS)Ncsg|]}tqSr+)r r'rrWr+r, |sz)FlavaEncoder.__init__..F) r]r^rWr ModuleListrangenum_hidden_layerslayergradient_checkpointingrrkrr,r^ys   zFlavaEncoder.__init__FTrrrroutput_hidden_states return_dictc Cs|rdnd}|r dnd}t|jD])\} } |r||f}|dur$|| nd} | ||| |} | d}|r:|| df}q|rB||f}|sPtdd|||fDSt|||dS)Nr+rrcss|] }|dur|VqdSrSr+)r'rAr+r+r,r-z'FlavaEncoder.forward..)last_hidden_stater attentions) enumeraterr/r ) r*rrrrrrall_hidden_statesall_self_attentionsi layer_modulelayer_head_mask layer_outputsr+r+r,rs$   zFlavaEncoder.forward)NNFFT)r1r2r3rr^r5rrrBrr/r rrr+r+rkr,rxs,  rcs2eZdZdeffdd ZdejfddZZS) FlavaPoolerrWcs*tt|j|j|_t|_dSrS)r]r^rrrbrTanh activationrrkr+r,r^s zFlavaPooler.__init__rcCs(|dddf}||}||}|SNr)rr*)r*rfirst_token_tensor pooled_outputr+r+r,rs  zFlavaPooler.forwardr r+r+rkr,r(sr(c@s:eZdZeZdZdZdeej ej ej fddfddZ dS)FlavaPreTrainedModelflavaTmoduler"NcCsXt|tjtjfr#|jjjd|jjd|j dur!|j j dSdSt|tj rF|jjjd|jjd|j durD|jj|j  dSdSt|tj r[|j j |jjddSt|trh|j j dSt|tr|jj |jj |jdur|jj dSdSt|tr|jr|jj dSdSt|tr|jj|jjdSdS)zInitialize the weightsg)meanstdNr)rrrrweightdatanormal_rWinitializer_rangerzero_rrrfill_FlavaMaskedPredictionHeadrVrcrgr_FlavaMultimodalModel use_cls_token FlavaModel logit_scalelogit_scale_init_value)r*r0r+r+r, _init_weightss8            z"FlavaPreTrainedModel._init_weights) r1r2r3r config_classbase_model_prefixsupports_gradient_checkpointingrrrrrr?r+r+r+r,r.s &r.cseZdZeZdZdZddedeffdd Zde j fd d Z d e j fd d Z de eeefddfddZe        ddeejdeejdeedeejdeejdeedeedeedeeeffddZZS)FlavaImageModelzflava.image_modelrTrWadd_pooling_layercXt|||_t||_t||_tj|j |j d|_ |r#t |nd|_ |dSv add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer rN)r]r^rWrVrmrencoderrrrbr layernormr(pooler post_initr*rWrDrkr+r,r^s    zFlavaImageModel.__init__r"cC|jjSrSrmrer)r+r+r,get_input_embeddingsz$FlavaImageModel.get_input_embeddingsrcC ||j_dSrSrNr*rr+r+r,set_input_embeddings z$FlavaImageModel.set_input_embeddingsheads_to_pruneNcC*|D]\}}|jj|j|qdSz Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel NitemsrHrrrr*rUrrr+r+r, _prune_headszFlavaImageModel._prune_headsrrrrrrrc Cs|dur|n|jj}|dur|n|jj}|dur|n|jj}|dur&td|||jj}|j|||d} |j| |||||d} | d} | | } |j durT| | nd} |sb| | f| ddSt | | | j | j dS)z bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, image_num_patches)`): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). Nz You have to specify pixel_values)rrrrrrrrrr pooler_outputrr )rWrruse_return_dictr get_head_maskrrmrHrIrJr rr ) r*rrrrrrrrembedding_outputencoder_outputssequence_outputr-r+r+r,rs: zFlavaImageModel.forwardTNNNNNNNN)r1r2r3rr@rAmain_input_namerBr^rModulerOrSdictrlistr[rrr5rrrr/r rrr+r+rkr,rCsH  rCcseZdZeZdZddedeffdd Zdefdd Z d e j fd d Z d e eeefddfddZe        ddeejdeejdeejdeejdeejdeedeedeedeeeffddZZS)FlavaTextModelzflava.text_modelTrWrDcrErF)r]r^rWrrmrrHrrrbrrIr(rJrKrLrkr+r,r^;s    zFlavaTextModel.__init__r"cCrMrSrmrr)r+r+r,rOKrPz#FlavaTextModel.get_input_embeddingsrcCrQrSrlrRr+r+r,rSNrTz#FlavaTextModel.set_input_embeddingsrUNcCrVrWrXrZr+r+r,r[Qr\zFlavaTextModel._prune_headsrrrrrrrrc Cs|dur|n|jj}|dur|n|jj}|dur|n|jj}|dur&td|} |dur6tj| |jd}| ||jj }| || |j} |j |||d} |j | | ||||d} | d} || } |jdurl|| nd}|sz| |f| ddSt| || j| jdS) a input_ids (`torch.LongTensor` of shape `(batch_size, text_seq_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) token_type_ids (`torch.LongTensor` of shape `(batch_size, text_seq_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) NzYou have to specify input_idsr)rrrr]rrr^)rWrrr`rrsr5onesrrarget_extended_attention_maskrmrHrIrJr rr )r*rrrrrrrrrextended_attention_maskrbrcrdr-r+r+r,rYsJ zFlavaTextModel.forwardrerf)r1r2r3rr@rArBr^rdrOrrhrSrirrjr[rrr5rrr/r rrr+r+rkr,rk5sF  rkcseZdZeZdZdZddeffdd Zdee e e fdd fd d Z e dde jd ee jd ee jdeedeedeedeeeffddZZS)r:zflava.multimodal_modelrTrWcsvt|||_|jj|_|jrttdd|j|_ t ||_ tj |j|j d|_|r2t|nd|_|dS)rGrrN)r]r^rWr;rr`r5rarbrcrrHrrrIr(rJrKrLrkr+r,r^s    zFlavaMultimodalModel.__init__rUr"NcCrVrWrXrZr+r+r,r[r\z!FlavaMultimodalModel._prune_headsrrrrrcCs&|dur|n|jj}|dur|n|jj}|dur|n|jj}|\}}} |jr=|j|dd} tj | |fdd}|d7}|durKtj ||f|j d}| ||jj }||||f|j } |j|| ||||d} | d} || } |jdur{|| nd}|s| |f| ddSt| || j| jdS) z hidden_states (`torch.FloatTensor` of shape `(batch_size, image_num_patches + text_seq_len, hidden_size)`): The concatenated hidden states of unimodal encoders. Nrprrvrmr]rr^)rWrrr`rsr;rcrr5rrnrrarrorHrIrJr rr )r*rrrrrrrrrrrprcrdr-r+r+r,rsD  zFlavaMultimodalModel.forwardre)NNNNN)r1r2r3rr@rArgr^rirrjr[rr5rrrBrr/r rrr+r+rkr,r:s6 r:cseZdZeZdeffdd Ze       ddeej deej deej deej d ee d ee d ee d ej fd dZ e        ddeej deej dee deej deej d ee d ee d ee d ej fddZe           ddeejdeej deej deej deej deejdeej dee d ee d e d ee d eeeffddZZS)r<rWcs0t|t|jtstdt|jdt|jts(tdt|jdt|j t s;tddt|j d|j}|j}|j }|j |_ |j |_ |j |_|j |_t||_t||_t||_t|j|j |_t|j |j |_tt|jj|_t|j|j|_ t|j |j|_!|"dS)NzLconfig.text_config is expected to be of type FlavaTextConfig but is of type rzNconfig.image_config is expected to be of type FlavaImageConfig but is of type zMconfig.multimodal_config is expected to be of type FlavaMultimodalConfig but z is of type )#r]r^r text_configr TypeErrortype image_configrmultimodal_configrprojection_dimrbtext_hidden_sizeimage_hidden_sizemm_hidden_sizerk text_modelrC image_modelr:multimodal_modelrrimage_projectiontext_projectionr`r5tensorrWr>r=image_to_mm_projectiontext_to_mm_projectionrK)r*rWrqrtrurkr+r,r^ sF        zFlavaModel.__init__Nrrrrrrrr"c Cs.|j|||||||d}|d} || } | S)a input_ids (`torch.LongTensor` of shape `(batch_size, text_seq_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) token_type_ids (`torch.LongTensor` of shape `(batch_size, text_seq_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) Returns: text_features (`torch.FloatTensor` of shape `(batch_size, output_dim`): The text embeddings obtained by applying the projection layer to the pooled output of [`FlavaTextModel`]. Examples: ```python >>> from transformers import AutoProcessor, FlavaModel >>> model = FlavaModel.from_pretrained("{0}") >>> processor = AutoProcessor.from_pretrained("{0}") >>> inputs = processor( ... text=["a photo of a cat", "a photo of a dog"], max_length=77, padding="max_length", return_tensors="pt" ... ) >>> text_features = model.get_text_features(**inputs) ``` )rrrrrrrr)rzr~) r*rrrrrrr text_outputsr- text_featuresr+r+r,get_text_features5s) zFlavaModel.get_text_featuresrrrrc Cs0|j||||||||d} | d} || } | S)a bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, image_num_patches)`): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). Returns: image_features (`torch.FloatTensor` of shape `(batch_size, output_dim`): The image embeddings obtained by applying the projection layer to the pooled output of [`FlavaImageModel`]. Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, FlavaModel >>> model = FlavaModel.from_pretrained("{0}") >>> processor = AutoProcessor.from_pretrained("{0}") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> image_features = model.get_image_features(**inputs) ``` )rrrrrrrrr)r{r}) r*rrrrrrrr image_outputsr-image_featuresr+r+r,get_image_featuresms& zFlavaModel.get_image_featuresTimage_attention_maskskip_multimodal_encoderc  Csz| dur| n|jj} | stdd} d} d}d}|dur7|j|||| | | d}|d|d} } || d}d}d}d}d}|dur_|j||||| | | d}|d|d}}||d}d}d}|dur|dur|s|dur|j\}}}|jj r|d7}t j |||j d }t j ||gdd }nd}t j ||gdd }|j||| d }|d}| s| |||||fSt| |||||d S) a input_ids (`torch.LongTensor` of shape `(batch_size, image_num_patches + text_seq_len)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) token_type_ids (`torch.LongTensor` of shape `(batch_size, image_num_patches + text_seq_len)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, image_num_patches)`): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). image_attention_mask (`torch.Tensor` of shape `(batch_size, image_num_patches)`, *optional*): Mask to avoid performing attention on padding pixel values for image inputs. Mask values selected in `[0, 1]`: - 1 for pixel values that are real (i.e., **not masked**), - 0 for pixel values that are padding (i.e., **masked**). skip_multimodal_encoder (*bool*, *optional*): Skip any calculations for multimodal encoder. Useful if multimodal encoding is not going to be used. Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, FlavaModel >>> model = FlavaModel.from_pretrained("facebook/flava-full") >>> processor = AutoProcessor.from_pretrained("facebook/flava-full") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(text=["a photo of a cat"], images=image, return_tensors="pt", padding=True) >>> outputs = model(**inputs) >>> image_embeddings = outputs.image_embeddings >>> text_embeddings = outputs.text_embeddings >>> multimodal_embeddings = outputs.multimodal_embeddings >>> outputs.image_embeddings.shape torch.Size([1, 197, 768]) >>> text_embeddings.shape torch.Size([1, 7, 768]) >>> multimodal_embeddings.shape torch.Size([1, 205, 768]) ``` NzRFLAVA model requires hidden states to work. Please set `output_hidden_states=True`)rrrrrrrrqrp)rrrrrrrrrmrv)rr)rrrrr r!)rWrrr{rrzrrxr|r;r5rnrrr)r*rrrrrrrrrrrr image_statesimage_mm_projectionrr text_statestext_mm_projectionrr r!rrrattention_mask_imageattention_multimodalmultimodal_inputr+r+r,rsC   zFlavaModel.forward)NNNNNNNrf) NNNNNNNNNTN)r1r2r3rr@r^rrr5rrBr6rrr LongTensorrr/r rrr+r+rkr,r<s+ 7  5     r<cs<eZdZdedeffdd ZdejdejfddZZS) FlavaImageCodebookResPathin_sizeout_sizec st|d}t}t|d<tj||ddd|d<t|d<tj||ddd|d<t|d <tj||ddd|d <t|d <tj||dd d|d <t||_dS)Nrelu_1rrrpaddingconv_1relu_2conv_2relu_3conv_3relu_4rconv_4)r]r^rrReLUr Sequentialpath)r*rrkwargshid_sizerrkr+r,r^3s     z"FlavaImageCodebookResPath.__init__rr"cC ||SrS)rr*rr+r+r,rC z!FlavaImageCodebookResPath.forward r1r2r3rr^r5rrrr+r+rkr,r2srcs@eZdZdededeffdd Zdejdejfdd ZZS) FlavaImageCodebookBlockrr num_layersc sPtd|d|_||krtj||ddd|_nt|_t|||_dS)Nrrqrr) r]r^ post_gainrrid_pathIdentityrres_path)r*rrrrrkr+r,r^Hs  z FlavaImageCodebookBlock.__init__rr"cCs|||j||SrS)rrrrr+r+r,rTszFlavaImageCodebookBlock.forwardrr+r+rkr,rGs rc sJeZdZd dededededef fdd Zd ejd ejfd d ZZ S)FlavaImageCodebookLayerGroupT num_blocksrrruse_poolcstt}t|D]!}|dkr t||||d|d<q t||||d|d<q |r8tjdd|d<t||_dS)Nrblock_rrq)rpool) r]r^rrrr MaxPool2drgroup)r*rrrrrblocksr$rkr+r,r^Ys  z%FlavaImageCodebookLayerGroup.__init__rr"cCrrS)rrr+r+r,rgrz$FlavaImageCodebookLayerGroup.forwardre) r1r2r3rrBr^r5rrrr+r+rkr,rXs$ra" The FLAVA's image codebook model inspired from DALL-E's original encoder. Outputs raw hidden states and can be used to generate image tokens for an image based on DALL-E's vocab. Used to generate labels for MIM. Use `get_codebook_indices` to get image tokens for an image. csxeZdZdZeZdZdZdedeffdd Z de j de j fd d Z de j de j fd d Z de jde j fd dZZS)FlavaImageCodebookrFrWrc sdt|||_|j|_|j|_|j|_|j|_|j|_|j|j}t}t |d<t j d|j|jddd|d<t}t j |jd|jddd|d <t |j|d|jd|j|d <t |j|d|jd |j|d <t |j|d |jd |j|d<t |j|d |jd|jdd|d<t ||d<t ||_||jjr|D]}d|_qdSdS)Nrelurrrconvrinputgroup_1rqgroup_2rgroup_3F)rgroup_4r)r]r^rW num_groupsinput_channelsnum_blocks_per_grouprbrrrrrrrrrKfreeze parameters requires_grad)r*rWrr output_blocksrparamrkr+r,r^ysB      zFlavaImageCodebook.__init__r"cCs*dtdtd||}tj|ddS)Na) Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Codebook pixel values can be obtained using [`AutoImageProcessor`] by passing `return_codebook_pixels=True`. See [`FlavaImageProcessor.__call__`] for details. Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoImageProcessor, FlavaImageCodebook >>> model = FlavaImageCodebook.from_pretrained("E") >>> image_processor = AutoImageProcessor.from_pretrained("a") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = image_processor([image], return_codebook_pixels=True, return_tensors="pt") >>> inputs = dict(pixel_values=inputs.codebook_pixel_values) >>> outputs = model.get_codebook_indices(**inputs) ``` r)axis)_CHECKPOINT_FOR_CODEBOOK_DOCrr5argmaxr*rz_logitsr+r+r,get_codebook_indicess   z'FlavaImageCodebook.get_codebook_indicescCs||}tjdd|S)Nrrv)rrSoftmaxrr+r+r,get_codebook_probss z%FlavaImageCodebook.get_codebook_probscCshdtdtdt|jdkrtd|jd|jd|jkr/td|jdd |j||S) Na* Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Codebook pixel values can be obtained using [`AutoImageProcessor`] by passing `return_codebook_pixels=True`. See [`FlavaImageProcessor.__call__`] for details. Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoImageProcessor, FlavaImageCodebook >>> model = FlavaImageCodebook.from_pretrained("ra") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = image_processor([image], return_codebook_pixels=True, return_tensors="pt") >>> inputs = dict(pixel_values=inputs.codebook_pixel_values) >>> outputs = model(**inputs) >>> print(outputs.shape) (1, 196) ``` rz input shape z is not 4drz input has z channels but model built for )rrrxrrr)r*rr+r+r,rs  zFlavaImageCodebook.forward)r1r2r3rArr@rgrBrr^r5rrrr6rrr+r+rkr,rls,rc$eZdZfddZddZZS)FlavaPredictionHeadTransformcsVtt|j|j|_t|jtrt |j|_ n|j|_ tj |j|j d|_ dS)Nr) r]r^rrrbrrrrr transform_act_fnrrrrkr+r,r^s  z%FlavaPredictionHeadTransform.__init__cCs"||}||}||}|SrS)rrrrr+r+r,rs   z$FlavaPredictionHeadTransform.forwardr1r2r3r^rrr+r+rkr,rs  rcs.eZdZdfdd ZddZddZZS) r9Ncsbt||_t||_tj|j|jdd|_ t t |j|_ |dur*||j _|j |j _ dS)NFr)r]r^rWr transformrrrbrdecoderr`r5rarr3)r*rWr3rkr+r,r^s  z"FlavaMaskedPredictionHead.__init__cCs|j|j_dSrS)rrr)r+r+r, _tie_weightssz&FlavaMaskedPredictionHead._tie_weightscCrrS)rrrr+r+r,r   z!FlavaMaskedPredictionHead.forwardrS)r1r2r3r^rrrr+r+rkr,r9s r9cr) FlavaITMHeadcs.t||_t||_t|jd|_dS)Nrq) r]r^rWr(rJrrrbseq_relationshiprrkr+r,r^s  zFlavaITMHead.__init__cCrrS)rJrrr+r+r,rrzFlavaITMHead.forwardrr+r+rkr,rs rcr)FlavaGlobalContrastiveHeadcst||_|j|_dSrS)r]r^rWglobal_backprop_contrastiverrkr+r,r^s  z#FlavaGlobalContrastiveHead.__init__c s2t|}tjrtjs!tjdjd}g}g}nQd}tj} |j r?tjj j }tjj j }n$fddt | D}fddt | D}tj |tj ||tjtj|jd}t|}t|}t|dd|} t|dd|} | | |fS)Nrrmcg|]}tqSr+r5 zeros_liker)rr+r,r2z6FlavaGlobalContrastiveHead.forward..crr+rr)rr+r,r3rr)r5exp distributed is_availableis_initializedrrsrget_world_sizerrr} all_gatherrget_rankrrr) r*rrr= temperaturelabelsimage_embeddings_alltext_embeddings_alllocal_batch_size world_sizelogits_per_imagelogits_per_textr+)rrr,r"s,      z"FlavaGlobalContrastiveHead.forwardrr+r+rkr,rs rzk The FLAVA model for pretraining which outputs losses, embeddings, logits and transformer outputs. c(s&eZdZgdZddedeejffdd Zde j fdd Z e                d d ee j d ee j d ee jdee jdee j dee j dee j dee j dee j deedee j dee j dee j deededeedeedeee j eff$ddZZS)!FlavaForPreTraining)zmmm_text_head.decoder.biaszmmm_image_head.decoder.biaszmlm_head.decoder.biaszmim_head.decoder.biasNrWimage_codebookcst|t||_||_|jdur|jrt|j|_t|j |_ t|j |_ t ||_t|j |_t|j |_t||_|j j|_|j j|_|j|_|j|_|j|_|j|_|j|_|j|_|j|_|j|_|dS)z image_codebook ([`nn.Module`]): If passed, the image codebook will be set to this. Otherwise, it will be initialized using the image_codebook_config defined in the config first as the first parameter. N)r]r^r<r/r init_codebookrimage_codebook_configr9rtmim_headrqmlm_headritm_headmmm_image_head mmm_text_headrglobal_contrastive_headrimage_vocab_sizetext_vocab_size mlm_weight mim_weightglobal_contrastive_weightce_ignore_index itm_weightmmm_image_weightmmm_text_weight skip_unmasked_multimodal_encoderrK)r*rWrrkr+r,r^Rs,            zFlavaForPreTraining.__init__rcCs"|dkr||dd}|S)Nrqrrp)rwrrsrr+r+r, _resize_to_2dus z!FlavaForPreTraining._resize_to_2dTrinput_ids_maskedrcodebook_pixel_valuesrrrrrr mlm_labels mim_labels itm_labelsrrr return_lossr"c6Cs|dur|n|jj}|dur|n|jj}| dur| n|j} |dur,|dur,td|}|j|||||| | ||dd }|j||||| |||dd }d}|j}|j}|j}|j}|j }d}}}}}}} d}!}"}#}$d}%}&}'|dus~|dur| dur|r|j durt d|durt d|j |} |jdkr |dur |dur |}(| dur|| } ||}|j| |d<|(dd| d  dddf}(| |j})| |)}*|(|)ddf}(||(}!|rtj|!d |j|*d }||j9}n||(}!|jdkrj|durj|durj|}+| dure|| } |+dd| d  dddf}+| |j})| |)},|+|)ddf}+||+}"|rdtj|"d |j|,d }||j9}n||+}"|jdkr|dur||}%| dur| d}-t|- |-|-!dg}|rtj|%| } | |j9} |dur||}| dur| |} | dur| |} ||}|dur-|j"dkr-|}(|d d }.|(ddd d |.ddf}(| dur(|| } ||}|j| |d<| |j})| |)}*|(|)ddf}(|#|(}$|r'tj|$d |j|*d }||j"9}n|#|(}$|dur|j$dkr|}+|+dd|d  dddf}+| dur|| } | |j})| |)},|+|)ddf}+|%|+}#|rtj|#d |j|,d }||j$9}n|%|+}#|dur |dur |j&dkr |j'|dddddf}/tjj(|/d d }/|j)|dddddf}0tjj(|0d d }0|jj*j+,t-t.|/|0|/|jj*\}&}'}1|dur|&|}&|'|}'|1|}1|r tj|&|1}2tj|'|1}3|2|3d }||j&9}t0||| |||d }4|r&|41s&t2dd|43D}|s||j4dur5|j45nd||j6durB|j65nd|j |j7durP|j75nd||j4dur]|j45nd||j6durj|j65nd||j7durw|j75nd|!|"|%|&|&|$|#f}5|r|41s||4f|5}5t8dd|5DSt9d&id|d|4d|d|j4d|d|j6d|j d|j7d|d|j4d|d|j6d|d|j7d|!d |"d!|%d"|&d#|'d$|$d%|#S)'a input_ids (`torch.LongTensor` of shape `(batch_size, text_seq_len)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) input_ids_masked (`torch.LongTensor` of shape `(batch_size, text_seq_len)`): Indices of input sequence tokens in the vocabulary. These ones are the masked version of the original task to be used with MLM. Indices can be obtained using [`AutoTokenizer`] along with [`DataCollatorForMaskedLanguageModeling`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) codebook_pixel_values (`torch.FloatTensor` of shape `(batch_size, num_image_patches, patch_size, patch_size, 3)`, *optional*): Pixel values for image patches that are used to compute the image codebook labels for masked image modeling. token_type_ids (`torch.LongTensor` of shape `(batch_size, text_seq_len)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, image_num_patches)`): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). image_attention_mask (`torch.FloatTensor` of shape `(batch_size, image_num_patches)`, *optional*): Mask to avoid performing attention on padding token indices specifically for images. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) skip_unmasked_multimodal_encoder (*bool*, *optional*): Skip any calculations for multimodal encoder for unmasked inputs. FLAVA pretraining doesn't need unmasked multimodal embeddings or outputs as of now. mlm_labels (`torch.LongTensor` of shape `(batch_size, text_seq_len)`, *optional*): Labels for computing the left-to-right language and multimodal masked modeling loss (next word prediction). Indices should be in `[-100, 0, ..., text_config.vocab_size - 1]` (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., text_config.vocab_size - 1]`. mim_labels (`torch.LongTensor` of shape `(batch_size, image_num_patches)`, *optional*): Labels for computing the image and multimodal masked modeling loss. Indices should be in `[-100, 0, ..., image_config.vocab_size - 1]`. Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., image_config.vocab_size - 1]`. If not passed, they are generated automatically using the image codebook assigned to the model. By default, it uses [`FlavaImageCodebook`]. See [`FlavaImageCodebook`] to understand how to generate mim_labels. itm_labels (`torch.LongTensor` of shape `(batch_size, 1)`, *optional*): Labels for computing the image-text matching loss. 0 means the pairs don't match and 1 means they match. The pairs with 0 will be skipped for calculation of MMM and global contrastive losses as well. return_loss (`bool`, *optional*, default to None): Whether to return calculated loss or not. Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import FlavaForPreTraining, AutoProcessor >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> model = FlavaForPreTraining.from_pretrained("facebook/flava-full") >>> processor = AutoProcessor.from_pretrained("facebook/flava-full") >>> text = ["a photo of a cat"] >>> inputs = processor( ... images=[image], ... text=text, ... return_masks=True, ... return_codebook_pixels=True, ... padding=True, ... max_length=77, ... return_tensors="pt", ... ) >>> output = model(**inputs) ``` Nz`input_ids_masked` isn't passed which means MLM loss won't be calculated correctlySetting it to `input_ids` so that model can work. Please pass it if this is unintentional. This is usually OKAY if you are doing inference on unmasked text...T) rrrrrrrrrr) rrrrrrrrrz`return_loss` is set to True but the image codebook is not initialized and no `mim_labels` have been passed. Reinstantiate the model with `init_codebook` set to True or pass in your custom `mim_labels`z`codebook_pixel_value` are required to generate `mim_labels` if loss is expected. Call `AutoProcessor` with `return_codebook_pixels` set to Truerrrprqrv)r9r:r;r<r=r>css |] }|dur |ndVqdSr+r+)r'rDr+r+r,r-sz.FlavaForPreTraining.forward..css|] }|dur|VqdSrSr+)r'rr+r+r,r-rrDrErrrrr r!rFrGrHrIrJrKrLrMrNrOrPrQrRr+):rWr`r rloggerwarningr/rrr r RuntimeErrorrrrrrnersrrr} cross_entropyrrrrrrrr5whereanynewrrrrrr~ normalizer}r=r4clamp_LOGIT_SCALE_CLAMP_MINLOGIT_SCALE_CLAMP_MAXrr8r@sumr?rr%rr!r/rC)6r*rrrrrrrrrrr r r rrrr  flava_outputflava_masked_outputpos_maskrrrFrHrJ total_lossmim_lossmlm_loss mmm_text_lossmmm_image_lossgc_lossitm_lossrLrMrRrQrNrrsequence_for_image masked_tokensmim_labels_filteredsequence_for_textmlm_labels_filtered pos_pairs end_indextext_embeddingimage_embedding gc_labels gc_loss_image gc_loss_text flava_lossesrr+r+r,rzs_         "       "                  "                   zFlavaForPreTraining.forwardrS)NNNNNNNNNNNNNNTNN)r1r2r3_tied_weights_keysrrrrhr^r5rrrrr6rBrr/rCrrr+r+rkr,rDsv#     r)rrrCr<r:r.rk)Hr4rrr dataclassesrtypingrrrr5torch.utils.checkpointr activationsr modeling_layersr modeling_outputsr r modeling_utilsr rrutilsrrrrconfiguration_flavarrrrr get_loggerr1r rrrrrr8rCrhrVrdrrrrrr r rr(r.rCrkr:r<rrrrrr9rrr__all__r+r+r+r,s       " \d$9C*.*"`p_-u ( #