o i@sdZddlZddlZddlmZddlmZmZddl Z ddl m Z ddl m Z ddl mZdd lmZdd lmZmZmZmZmZmZdd lmZdd lmZmZmZdd lmZm Z ddl!m"Z"e #e$Z%eeddGdddeZ&Gddde j'Z(Gddde j'Z)Gddde j'Z*Gddde j'Z+Gddde j'Z,Gddde j'Z-Gd d!d!e j'Z.Gd"d#d#e j'Z/Gd$d%d%eZ0Gd&d'd'e j'Z1eGd(d)d)eZ2eGd*d+d+e2Z3Gd,d-d-e j'Z4ed.dGd/d0d0e2Z5Gd1d2d2e j'Z6Gd3d4d4e j'Z7ed5dGd6d7d7e2Z8ed8dGd9d:d:e2Z9ed;dGdd?d?e2Z;gd@Zjs z/ViltEmbeddings.visual_embed..ij)indexingdeviceF)as_tuplec$g|]}dddf|kqSNrr'rMu) valid_idxr'r(rS$cr[r\r'r]) non_valid_idxr'r(rSr`cSg|]}|dqSrrCrMvr'r'r(rScSrbrcrCrdr'r'r(rSrfcsg|]}|qSr'r'rdmax_image_lengthr'r(rST) replacement)&r4 projectionweightshapefloatrrJrLlongsumr= image_size patch_sizer6 transposeviewr"catzipflattenstackrarangetorYexpand isinstanceintmaxminnonzerounique enumerate multinomialonesappendr2r<)r> pixel_values pixel_maskrh_phpwxx_maskx_hx_w batch_size num_channels patch_dim pos_embed patch_indexeffective_resolution unique_rows valid_row_idxnon_valid_row_idx valid_numsnon_valid_numspad_numsselectirenvp valid_choice pad_choice cls_tokensr')rPrhrarQr_rRr( visual_embed]sx  $$$0  &     (.,..8 &zViltEmbeddings.visual_embedrc  Cs|j|||d} |dur|j|||jjd\}} } n|d} |dur%d}| |tj|tj| j d} ||tj | |tj| j d}tj | |gdd} tj || gdd} | | fS)N) input_idstoken_type_ids inputs_embedsrgrdtyperYrE) r.rr=rhrwr9r" zeros_likerorY full_likeru)r>rattention_maskrrrr image_embedsimage_token_type_idx text_embeds image_masksr embeddingsmasksr'r'r(forwards&   zViltEmbeddings.forward)rA)r)rrr r!r,rr __classcell__r'r'r?r(r)Es   ar)cs*eZdZdZfddZdddZZS)r-zGConstruct the embeddings from word, position and token_type embeddings.csttj|j|j|jd|_t|j|j|_ t|j |j|_ tj |j|j d|_ t|j|_t|dd|_|jdt|jddd|jd tj|jtjd dddS) N) padding_idxepsposition_embedding_typeabsolute position_ids)rrUF) persistentrr)r+r,rr7 vocab_sizer1 pad_token_idword_embeddingsmax_position_embeddingsr6type_vocab_sizer9 LayerNormlayer_norm_epsr:r;r<getattrrregister_bufferr"ryr{r0rrDror>r=r?r'r(r,s  zTextEmbeddings.__init__Nc Cs|dur |}n|dd}|d}|dur$|jddd|f}|durNt|drC|jddd|f}||d|}|}n tj|tj|jjd}|durW| |}| |} || } |j dkrn| |} | | 7} | | } || } | S)NrUrrrrr)rDrhasattrrr{r"r0rorYrr9rr6rr<) r>rrrr input_shape seq_lengthbuffered_token_type_ids buffered_token_type_ids_expandedr9rr6r'r'r(rs,        zTextEmbeddings.forward)NNNNrrr r!r,rrr'r'r?r(r-s r-cs(eZdZdZfddZddZZS)r3z# Image to Patch Embedding. cst|j|j}}|j|j}}t|tjj r|n||f}t|tjj r)|n||f}|d|d|d|d}||_||_||_||_ t j ||||d|_ dS)Nrr) kernel_sizestride)r+r,rqrrrr1r| collectionsabcIterabler5rConv2drk)r>r=rqrrrr1r5r?r'r(r,s  zViltPatchEmbeddings.__init__cCs@|j\}}}}||jkrtd|jjj}||j|d}|S)NzeMake sure that the channel dimension of the pixel values match with the one set in the configuration.r)rmr ValueErrorrkrlrrz)r>rrrrPrR target_dtyperr'r'r(r,s  zViltPatchEmbeddings.forwardrr'r'r?r(r3s r3cs&eZdZfddZdddZZS)ViltSelfAttentioncst|j|jdkrt|dstd|jd|jd|j|_t|j|j|_|j|j|_t j |j|j|j d|_ t j |j|j|j d|_ t j |j|j|j d|_t |j|_dS)Nrembedding_sizezThe hidden size z4 is not a multiple of the number of attention heads .bias)r+r,r1num_attention_headsrrr}attention_head_size all_head_sizerLinearqkv_biasquerykeyvaluer:attention_probs_dropout_probr<rr?r'r(r,8s  zViltSelfAttention.__init__NFcCs,|j\}}}|||d|j|jdd}|||d|j|jdd} |||d|j|jdd} t || dd} | t |j} |durS| |} t j dd| } || } |durh| |} t | | } | dddd} | dd|jf}| j|} |r| | f}|S| f}|S)NrUrrBrTrErr)rmrrtrrrsrrr"matmulmathsqrtrSoftmaxr<permute contiguousrDr)r>rr head_maskoutput_attentionsrrr query_layer key_layer value_layerattention_scoresattention_probs context_layernew_context_layer_shapeoutputsr'r'r(rJs:     zViltSelfAttention.forwardNNFrrr r,rrr'r'r?r(r7s rcsBeZdZdZdeffdd Zdejdejdejfdd ZZ S) ViltSelfOutputz The residual connection is defined in ViltLayer instead of here (as is the case with other models), due to the layernorm applied before each block. r=cs.tt|j|j|_t|j|_dSN) r+r,rrr1denser:r;r<rr?r'r(r, zViltSelfOutput.__init__r input_tensorreturncC||}||}|Srrr<r>rrr'r'r(r  zViltSelfOutput.forward) rrr r!rr,r"Tensorrrr'r'r?r(rzs$rcs.eZdZfddZddZd ddZZS) ViltAttentioncs*tt||_t||_t|_dSr)r+r,r attentionroutputset pruned_headsrr?r'r(r,s    zViltAttention.__init__cCst|dkrdSt||jj|jj|j\}}t|jj||j_t|jj||j_t|jj ||j_ t|j j |dd|j _ |jjt||j_|jj|jj|j_ |j ||_dS)NrrrE)lenrrrrrrrrrrrrunion)r>headsindexr'r'r( prune_headss zViltAttention.prune_headsNFcCs6|||||}||d|}|f|dd}|S)Nrr)rr)r>rrrr self_outputsattention_outputrr'r'r(rszViltAttention.forwardr)rrr r,rrrr'r'r?r(rs rcs8eZdZdeffdd ZdejdejfddZZS)ViltIntermediater=csDtt|j|j|_t|jt rt |j|_ dS|j|_ dSr) r+r,rrr1intermediate_sizerr| hidden_actstrrintermediate_act_fnrr?r'r(r,s   zViltIntermediate.__init__rrcCrr)rrr>rr'r'r(rrzViltIntermediate.forward rrr rr,r"rrrr'r'r?r(r sr cs>eZdZdeffdd ZdejdejdejfddZZS) ViltOutputr=cs.tt|j|j|_t|j|_ dSr) r+r,rrr r1rr:r;r<rr?r'r(r,rzViltOutput.__init__rrrcCs ||}||}||}|Srrrr'r'r(rs  zViltOutput.forwardrr'r'r?r(rs$rcs*eZdZdZfddZdddZZS) ViltLayerz?This corresponds to the Block class in the timm implementation.csbt|j|_d|_t||_t||_t||_ t j |j |j d|_t j |j |j d|_dS)Nrr)r+r,chunk_size_feed_forward seq_len_dimrrr  intermediaterrrrr1rlayernorm_beforelayernorm_afterrr?r'r(r,s    zViltLayer.__init__NFc Csj|j|||||d}|d}|dd}|||j}||}||}|||}|f|}|S)N)rrr)rrrzrYrrr) r>rrrrself_attention_outputsr r layer_outputr'r'r(rs     zViltLayer.forwardrrr'r'r?r(rs  rcs0eZdZfddZ     dddZZS) ViltEncodercs:t|_tfddtjD|_d|_dS)Ncsg|]}tqSr')r)rMrr=r'r(rSriz(ViltEncoder.__init__..F) r+r,r=r ModuleListrangenum_hidden_layerslayergradient_checkpointingrr?rr(r,s   zViltEncoder.__init__NFTc Cs|rdnd}|r dnd}t|jD])\} } |r||f}|dur$|| nd} | ||| |} | d}|r:|| df}q|rB||f}|sPtdd|||fDSt|||dS)Nr'rrcss|] }|dur|VqdSrr'rdr'r'r( sz&ViltEncoder.forward..)last_hidden_staterr)rrr&r ) r>rrrroutput_hidden_states return_dictall_hidden_statesall_self_attentionsr layer_modulelayer_head_mask layer_outputsr'r'r(rs(   zViltEncoder.forward)NNFFTrr'r'r?r(rs  rc@s.eZdZUeed<dZdZddgZddZdS) ViltPreTrainedModelr=viltTr)rcCst|tjtjfr#|jjjd|jjd|j dur!|j j dSdSt|tj rF|jjjd|jjd|j durD|jj|j  dSdSt|tj r[|j j |jjddSdS)zInitialize the weightsg)meanstdNg?)r|rrrrldatanormal_r=initializer_rangerzero_r7rrfill_)r>moduler'r'r( _init_weights$s     z!ViltPreTrainedModel._init_weightsN) rrr rr$base_model_prefixsupports_gradient_checkpointing_no_split_modulesr4r'r'r'r(r*s  r*cseZdZdfdd ZddZddZdd Ze dd ee j d ee j d ee j dee j dee j dee j dee j dee j dee dee dee dee deeee j ffddZZS) ViltModelTcsXt|||_t||_t||_tj|j |j d|_ |r#t |nd|_ |dS)zv add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer rN)r+r,r=r)rrencoderrrr1r layernorm ViltPoolerpooler post_init)r>r=add_pooling_layerr?r'r(r,7s    zViltModel.__init__cCs |jjjSrrr.rr>r'r'r(get_input_embeddingsHs zViltModel.get_input_embeddingscCs||jj_dSrr?)r>rr'r'r(set_input_embeddingsKzViltModel.set_input_embeddingscCs*|D]\}}|jj|j|qdS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsr9rrr)r>heads_to_prunerrr'r'r( _prune_headsNszViltModel._prune_headsNrrrrrrrrrrr#r$rc  Cs| dur| n|jj} | dur| n|jj} | dur| n|jj} |dur*|dur*td|dur9||||} n|durF|dd} ntd| \}}|durU|jn|j}|duretj ||f|d}|durq|durqtd|dur}|dur}td|dur|j dn|j d}||krtd |durtj ||jj |jj f|d}| ||jj }|j|||||||| d \}}||| }|j|||| | | d }|d}||}|jdur||nd}| s||f|d dSt|||j|jd S)ak image_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*): Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings. image_token_type_idx (`int`, *optional*): - The token type ids for images. Examples: ```python >>> from transformers import ViltProcessor, ViltModel >>> from PIL import Image >>> import requests >>> # prepare image and text >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> text = "hello world" >>> processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-mlm") >>> model = ViltModel.from_pretrained("dandelin/vilt-b32-mlm") >>> inputs = processor(image, text, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_states = outputs.last_hidden_state ```NzDYou cannot specify both input_ids and inputs_embeds at the same timerUz5You have to specify either input_ids or inputs_embedsrXzFYou cannot specify both pixel_values and image_embeds at the same timez7You have to specify either pixel_values or image_embedsrzAThe text inputs and image inputs need to have the same batch size)r)rrrr#r$r)r" pooler_outputrr)r=rr#use_return_dictr%warn_if_padding_and_no_attention_maskrDrYr"rrmrq get_head_maskrrget_extended_attention_maskr9r:r<r rr)r>rrrrrrrrrrr#r$rtext_batch_sizerrYimage_batch_sizeembedding_outputextended_attention_maskencoder_outputssequence_output pooled_outputr'r'r(rVsp*    zViltModel.forward)T NNNNNNNNNNNN)rrr r,rArBrFrrr" LongTensorr#r}boolrr r&rrr'r'r?r(r85sZ     r8c$eZdZfddZddZZS)r;cs*tt|j|j|_t|_dSr)r+r,rrr1rTanh activationrr?r'r(r,s zViltPooler.__init__cCs(|dddf}||}||}|Sr\)rrX)r>rfirst_token_tensorrRr'r'r(rs  zViltPooler.forwardrr'r'r?r(r;s r;zU ViLT Model with a language modeling head on top as done during pretraining. cseZdZddgZfddZddZddZe dd ee j d ee j d ee j d ee j dee j dee j dee j dee j dee j dee dee dee de eee j ffddZZS)ViltForMaskedLMzmlm_score.decoder.weightzmlm_score.decoder.biascs,t|t||_t||_|dSr)r+r,r8r+ ViltMLMHead mlm_scorer=rr?r'r(r,s    zViltForMaskedLM.__init__cCs|jjSr)r\decoderr@r'r'r(get_output_embeddingssz%ViltForMaskedLM.get_output_embeddingscCs||j_|j|j_dSr)r\r]r)r>new_embeddingsr'r'r(set_output_embeddingssz%ViltForMaskedLM.set_output_embeddingsNrrrrrrrrlabelsrr#r$rc  Cs| dur| n|jj} |j||||||||| | | d } | dd\}}|dur+|jdn|jd}|ddd|f|dd|df}}||}d}| durgt}| |j} ||d|jj | d}| s}|f| dd}|dur{|f|S|St ||| j | j dS)a/ image_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*): Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings. labels (*torch.LongTensor* of shape *(batch_size, sequence_length)*, *optional*): Labels for computing the masked language modeling loss. Indices should be in *[-100, 0, ..., config.vocab_size]* (see *input_ids* docstring) Tokens with indices set to *-100* are ignored (masked), the loss is only computed for the tokens with labels in *[0, ..., config.vocab_size]* Examples: ```python >>> from transformers import ViltProcessor, ViltForMaskedLM >>> import requests >>> from PIL import Image >>> import re >>> import torch >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> text = "a bunch of [MASK] laying on a [MASK]." >>> processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-mlm") >>> model = ViltForMaskedLM.from_pretrained("dandelin/vilt-b32-mlm") >>> # prepare inputs >>> encoding = processor(image, text, return_tensors="pt") >>> # forward pass >>> outputs = model(**encoding) >>> tl = len(re.findall("\[MASK\]", text)) >>> inferred_token = [text] >>> # gradually fill in the MASK tokens, one by one >>> with torch.no_grad(): ... for i in range(tl): ... encoded = processor.tokenizer(inferred_token) ... input_ids = torch.tensor(encoded.input_ids) ... encoded = encoded["input_ids"][0][1:-1] ... outputs = model(input_ids=input_ids, pixel_values=encoding.pixel_values) ... mlm_logits = outputs.logits[0] # shape (seq_len, vocab_size) ... # only take into account text features (minus CLS and SEP token) ... mlm_logits = mlm_logits[1 : input_ids.shape[1] - 1, :] ... mlm_values, mlm_ids = mlm_logits.softmax(dim=-1).max(dim=-1) ... # only take into account text ... mlm_values[torch.tensor(encoded) != 103] = 0 ... select = mlm_values.argmax().item() ... encoded[select] = mlm_ids[select].item() ... inferred_token = [processor.decode(encoded)] >>> selected_token = "" >>> encoded = processor.tokenizer(inferred_token) >>> output = processor.decode(encoded.input_ids[0], skip_special_tokens=True) >>> print(output) a bunch of cats laying on a couch. ```N rrrrrrrrr#r$rBrrUrrrr) r=rHr+rmr\rrzrYrtrr rr)r>rrrrrrrrrarr#r$rrQrR text_seq_len text_featuresr mlm_logitsmasked_lm_lossloss_fctrr'r'r(rs@I*  zViltForMaskedLM.forwardrS)rrr _tied_weights_keysr,r^r`rrr"rTr#rUrr r&rrr'r'r?r(rZsZ       rZcrV)ViltPredictionHeadTransformcsVtt|j|j|_t|jtrt |j|_ n|j|_ tj |j|j d|_ dS)Nr) r+r,rrr1rr|r r rtransform_act_fnrrrr?r'r(r,is  z$ViltPredictionHeadTransform.__init__cCs"||}||}||}|Sr)rrkrrr'r'r(rrs   z#ViltPredictionHeadTransform.forwardrr'r'r?r(rjhs  rjcs.eZdZdfdd ZddZddZZS) r[Ncsbt||_t||_tj|j|jdd|_ t t |j|_ |dur*||j _|j |j _ dS)NFr)r+r,r=rj transformrrr1rr]r/r"r0rrl)r>r=rlr?r'r(r,zs  zViltMLMHead.__init__cCs|j|j_dSr)rr]r@r'r'r( _tie_weightsrCzViltMLMHead._tie_weightscCrr)rlr])r>rr'r'r(rrzViltMLMHead.forwardr)rrr r,rmrrr'r'r?r(r[ys r[z Vilt Model transformer with a classifier head on top (a linear layer on top of the final hidden state of the [CLS] token) for visual question answering, e.g. for VQAv2. ceZdZfddZe            ddeejdeejdeejdeejdeejd eejd eejd eejd eejd ee dee dee de e e ejffddZ ZS)ViltForQuestionAnsweringc slt||j|_t||_tt|j|jdt |jdt t|jd|j|_ | dS)NrB) r+r, num_labelsr8r+r Sequentialrr1rGELU classifierr=rr?r'r(r,s   z!ViltForQuestionAnswering.__init__Nrrrrrrrrrarr#r$rc  Cs| dur| n|jj} |j||||||||| | | d } | r| jn| d}||}d}| dur@| |j} tj || | j d}| sV|f| dd}|durT|f|S|St ||| j | j dS)aX image_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*): Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings. labels (`torch.FloatTensor` of shape `(batch_size, num_labels)`, *optional*): Labels for computing the visual question answering loss. This tensor must be either a one-hot encoding of all answers that are applicable for a given example in the batch, or a soft encoding indicating which answers are applicable, where 1.0 is the highest score. Examples: ```python >>> from transformers import ViltProcessor, ViltForQuestionAnswering >>> import requests >>> from PIL import Image >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> text = "How many cats are there?" >>> processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-finetuned-vqa") >>> model = ViltForQuestionAnswering.from_pretrained("dandelin/vilt-b32-finetuned-vqa") >>> # prepare inputs >>> encoding = processor(image, text, return_tensors="pt") >>> # forward pass >>> outputs = model(**encoding) >>> logits = outputs.logits >>> idx = logits.argmax(-1).item() >>> print("Predicted answer:", model.config.id2label[idx]) Predicted answer: 2 ```NrbrrBrc)r=rHr+rGrsrzrYrrJ binary_cross_entropy_with_logitsrmrrr)r>rrrrrrrrrarr#r$rrGrrrr'r'r(rs:1  z ViltForQuestionAnswering.forwardrSrrr r,rrr"rTr#rUrrr&rrr'r'r?r(rosT      roz Vilt Model transformer with a classifier head on top (a linear layer on top of the final hidden state of the [CLS] token) for image-to-text or text-to-image retrieval, e.g. MSCOCO and F30K. crn)ViltForImageAndTextRetrievalcs2t|t||_t|jd|_|dSr*) r+r,r8r+rrr1 rank_outputr=rr?r'r(r,s   z%ViltForImageAndTextRetrieval.__init__Nrrrrrrrrrarr#r$rc  Cs| dur| n|jj} d} | durtd|j||||||||| | | d }| r)|jn|d}||}| sH|f|dd}| durF| f|S|St| ||j|jdS)ad image_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*): Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels are currently not supported. Examples: ```python >>> from transformers import ViltProcessor, ViltForImageAndTextRetrieval >>> import requests >>> from PIL import Image >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> texts = ["An image of two cats chilling on a couch", "A football player scoring a goal"] >>> processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-finetuned-coco") >>> model = ViltForImageAndTextRetrieval.from_pretrained("dandelin/vilt-b32-finetuned-coco") >>> # forward pass >>> scores = dict() >>> for text in texts: ... # prepare inputs ... encoding = processor(image, text, return_tensors="pt") ... outputs = model(**encoding) ... scores[text] = outputs.logits[0, :].item() ```NzTraining is not yet supported.rbrrBrc) r=rHNotImplementedErrorr+rGrwrrr)r>rrrrrrrrrarr#r$rrrGrrr'r'r(rs8- z$ViltForImageAndTextRetrieval.forwardrSrur'r'r?r(rvsT       rvzq Vilt Model transformer with a classifier head on top for natural language visual reasoning, e.g. NLVR2. crn)"ViltForImagesAndTextClassificationc svt||j|_t||_|j}tt|j ||j |t |j |t t|j ||j|_ | dSr)r+r,rpr8r+ num_imagesrrqrr1rrrrsr=)r>r=rzr?r'r(r,hs   z+ViltForImagesAndTextClassification.__init__Nrrrrrrrrrarr#r$rc CsD| dur| n|jj} | dur| n|jj} | dur| n|jj} |dur,|jdkr,|d}|dur:|jdkr:|d}|durC|jdnd} | durT|durR|jdnd} | |jjkr^tdg}| rdgnd}| rjgnd}t | D]l}|j ||||dur|dd|ddddddfnd|dur|dd|ddddfnd|||dur|dd|ddddfnd|d| | | d }| r|j n|d}| || r| |j | r| |jqptj|dd}||}d}| durt}| |j} ||d|j| d}| s|||f}|dur|f|S|St||||d S) a3 image_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*): Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Binary classification labels. Examples: ```python >>> from transformers import ViltProcessor, ViltForImagesAndTextClassification >>> import requests >>> from PIL import Image >>> image1 = Image.open(requests.get("https://lil.nlp.cornell.edu/nlvr/exs/ex0_0.jpg", stream=True).raw) >>> image2 = Image.open(requests.get("https://lil.nlp.cornell.edu/nlvr/exs/ex0_1.jpg", stream=True).raw) >>> text = "The left image contains twice the number of dogs as the right image." >>> processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-finetuned-nlvr2") >>> model = ViltForImagesAndTextClassification.from_pretrained("dandelin/vilt-b32-finetuned-nlvr2") >>> # prepare inputs >>> encoding = processor([image1, image2], text, return_tensors="pt") >>> # forward pass >>> outputs = model(input_ids=encoding.input_ids, pixel_values=encoding.pixel_values.unsqueeze(0)) >>> logits = outputs.logits >>> idx = logits.argmax(-1).item() >>> print("Predicted answer:", model.config.id2label[idx]) Predicted answer: True ```Nrrz\Make sure to match the number of images in the model with the number of images in the input.) rrrrrrrrrr#r$rUrErc)r=rr#rHndim unsqueezermrzrrr+rGrrrr"rursrrzrYrtrpr)r>rrrrrrrrrarr#r$rzpooler_outputsrrrrrGrRrrrhrr'r'r(rzsp/      ,&&       z*ViltForImagesAndTextClassification.forwardrS)rrr r,rrr"rTr#rUrrr&rrr'r'r?r(rybsT      rycrn)ViltForTokenClassificationcsNt||j|_t|dd|_t|j|_t |j |j|_ | dS)NF)r>) r+r,rpr8r+rr:r;r<rr1rsr=rr?r'r(r,s  z#ViltForTokenClassification.__init__Nrrrrrrrrrarr#r$rc  Cs| dur| n|jj} |j||||||||| | | d } | d}|dur'|jdn|jd}||}||ddd|f}d}| durZt}| |j} || d|j | d}| sp|f| dd}|durn|f|S|St ||| j | j dS)a/ image_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*): Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings. labels (`torch.LongTensor` of shape `(batch_size, text_sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. NrbrrrUrBrc)r=rHr+rmr<rsrrzrYrtrprrr)r>rrrrrrrrrarr#r$rrQtext_input_sizerrrhrr'r'r(rs@  z"ViltForTokenClassification.forwardrS)rrr r,rrr"rTr#rUrrr&rrr'r'r?r(rsT       r)rvryrrZrorr8r*)=r!collections.abcrr dataclassesrtypingrrr"rtorch.nnr activationsrmodeling_layersr modeling_outputsr r r r rrmodeling_utilsr pytorch_utilsrrrutilsrrconfiguration_viltr get_loggerrloggerrModuler)r-r3rrrr rrrr*r8r;rZrjr[rorvryr__all__r'r'r'r(sz         9C# &,j] O