o wi@sdZddlZddlZddlmZddlmZmZddl Z ddl Z ddl m Z ddl m Z ddlmZdd lmZdd lmZmZmZmZmZmZdd lmZdd lmZmZmZdd lm Z m!Z!ddl"m#Z#e!$e%Z&ee ddGdddeZ'Gddde j(Z)Gddde j(Z*Gddde j(Z+Gddde j(Z,Gddde j(Z-Gddde j(Z.Gd d!d!e j(Z/Gd"d#d#e j(Z0Gd$d%d%eZ1Gd&d'd'e j(Z2e Gd(d)d)eZ3e Gd*d+d+e3Z4Gd,d-d-e j(Z5e d.dGd/d0d0e3Z6Gd1d2d2e j(Z7Gd3d4d4e j(Z8e d5dGd6d7d7e3Z9e d8dGd9d:d:e3Z:e d;dGdd?d?e3Zks z/ViltEmbeddings.visual_embed..ij)indexingdeviceF)as_tuplec$g|]}dddf|kqSNrr'rMu) valid_idxr'r(rS$cr[r\r'r]) non_valid_idxr'r(rSr`cSg|]}|dqSrrCrMvr'r'r(rScSrbrcrCrdr'r'r(rSrfcsg|]}|qSr'r'rdmax_image_lengthr'r(rST) replacement)&r4 projectionweightshapefloatrrJrLlongsumr= image_size patch_sizer6 transposeviewr"catzipflattenstackrarangetorYexpand isinstanceintmaxminnonzerounique enumerate multinomialonesappendr2r<)r> pixel_values pixel_maskrh_phpwxx_maskx_hx_w batch_size num_channels patch_dim pos_embed patch_indexeffective_resolution unique_rows valid_row_idxnon_valid_row_idx valid_numsnon_valid_numspad_numsselectirenvp valid_choice pad_choice cls_tokensr')rPrhrarQr_rRr( visual_embed^sx  $$$0  &     (.,..8 &zViltEmbeddings.visual_embedrc  Cs|j|||d} |dur|j|||jjd\}} } n|d} |dur%d}| |tj|tj| j d} ||tj | |tj| j d}tj | |gdd} tj || gdd} | | fS)N) input_idstoken_type_ids inputs_embedsrgrdtyperYrE) r.rr=rhrwr9r" zeros_likerorY full_likeru)r>rattention_maskrrrr image_embedsimage_token_type_idx text_embeds image_masksr embeddingsmasksr'r'r(forwards&   zViltEmbeddings.forward)rA)r)rrr r!r,rr __classcell__r'r'r?r(r)Fs   ar)cs*eZdZdZfddZdddZZS)r-zGConstruct the embeddings from word, position and token_type embeddings.csttj|j|j|jd|_t|j|j|_ t|j |j|_ tj |j|j d|_ t|j|_t|dd|_|jdt|jddd|jd tj|jtjd dddS) N) padding_idxepsposition_embedding_typeabsolute position_ids)rrUF) persistentrr)r+r,rr7 vocab_sizer1 pad_token_idword_embeddingsmax_position_embeddingsr6type_vocab_sizer9 LayerNormlayer_norm_epsr:r;r<getattrrregister_bufferr"ryr{r0rrDror>r=r?r'r(r,s  zTextEmbeddings.__init__Nc Cs|dur |}n|dd}|d}|dur$|jddd|f}|durNt|drC|jddd|f}||d|}|}n tj|tj|jjd}|durW| |}| |} || } |j dkrn| |} | | 7} | | } || } | S)NrUrrrrr)rDrhasattrrr{r"r0rorYrr9rr6rr<) r>rrrr input_shape seq_lengthbuffered_token_type_ids buffered_token_type_ids_expandedr9rr6r'r'r(rs,        zTextEmbeddings.forward)NNNNrrr r!r,rrr'r'r?r(r-s r-cs(eZdZdZfddZddZZS)r3z# Image to Patch Embedding. cst|j|j}}|j|j}}t|tjj r|n||f}t|tjj r)|n||f}|d|d|d|d}||_||_||_||_ t j ||||d|_ dS)Nrr) kernel_sizestride)r+r,rqrrrr1r| collectionsabcIterabler5rConv2drk)r>r=rqrrrr1r5r?r'r(r,s  zViltPatchEmbeddings.__init__cCs@|j\}}}}||jkrtd|jjj}||j|d}|S)NzeMake sure that the channel dimension of the pixel values match with the one set in the configuration.r)rmr ValueErrorrkrlrrz)r>rrrrPrR target_dtyperr'r'r(r-s  zViltPatchEmbeddings.forwardrr'r'r?r(r3s r3c.eZdZfddZddZd ddZZS) ViltSelfAttentioncst|j|jdkrt|dstd|jd|jd|j|_t|j|j|_|j|j|_t j |j|j|j d|_ t j |j|j|j d|_ t j |j|j|j d|_t |j|_dS)Nrembedding_sizezThe hidden size z4 is not a multiple of the number of attention heads .bias)r+r,r1num_attention_headsrrr}attention_head_size all_head_sizerLinearqkv_biasquerykeyvaluer:attention_probs_dropout_probr<rr?r'r(r,9s  zViltSelfAttention.__init__cCs6|dd|j|jf}|j|}|ddddS)NrUrrBrr)rDrrrtpermute)r>r new_x_shaper'r'r(transpose_for_scoresKs z&ViltSelfAttention.transpose_for_scoresNFcCs||}|||}|||}||}t||dd} | t|j } |dur4| |} t j dd| } | | } |durI| |} t| |} | dddd} | dd|jf} | j| } |rr| | f} | S| f} | S)NrUrTrErrBrr)rrrrr"matmulrsmathsqrtrrSoftmaxr<r contiguousrDrrt)r>rr head_maskoutput_attentionsmixed_query_layer key_layer value_layer query_layerattention_scoresattention_probs context_layernew_context_layer_shapeoutputsr'r'r(rPs(      zViltSelfAttention.forwardNNF)rrr r,rrrr'r'r?r(r8s rcsFeZdZdZdeddffdd Zdejdejdejfd d ZZ S) ViltSelfOutputz The residual connection is defined in ViltLayer instead of here (as is the case with other models), due to the layernorm applied before each block. r=returnNcs.tt|j|j|_t|j|_dSN) r+r,rrr1denser:r;r<rr?r'r(r,{ zViltSelfOutput.__init__r input_tensorcC||}||}|Srrr<r>rrr'r'r(r  zViltSelfOutput.forward) rrr r!rr,r"Tensorrrr'r'r?r(rus$rcr) ViltAttentioncs*tt||_t||_t|_dSr)r+r,r attentionroutputset pruned_headsrr?r'r(r,s    zViltAttention.__init__cCst|dkrdSt||jj|jj|j\}}t|jj||j_t|jj||j_t|jj ||j_ t|j j |dd|j _ |jjt||j_|jj|jj|j_ |j ||_dS)NrrrE)lenrrrrrrrrrrrrunion)r>headsindexr'r'r( prune_headss zViltAttention.prune_headsNFcCs6|||||}||d|}|f|dd}|S)Nrr)rr)r>rrrr self_outputsattention_outputrr'r'r(rszViltAttention.forwardr)rrr r,r rrr'r'r?r(rs rcs<eZdZdeddffdd ZdejdejfddZZS) ViltIntermediater=rNcsDtt|j|j|_t|jt rt |j|_ dS|j|_ dSr) r+r,rrr1intermediate_sizerr| hidden_actstrrintermediate_act_fnrr?r'r(r,s   zViltIntermediate.__init__rcCrr)rrr>rr'r'r(rrzViltIntermediate.forward rrr rr,r"rrrr'r'r?r(r sr csBeZdZdeddffdd Zdejdejdejfdd ZZS) ViltOutputr=rNcs.tt|j|j|_t|j|_ dSr) r+r,rrrr1rr:r;r<rr?r'r(r,rzViltOutput.__init__rrcCs ||}||}||}|Srrrr'r'r(rs  zViltOutput.forwardrr'r'r?r(rs$rcs*eZdZdZfddZdddZZS) ViltLayerz?This corresponds to the Block class in the timm implementation.csbt|j|_d|_t||_t||_t||_ t j |j |j d|_t j |j |j d|_dS)Nrr)r+r,chunk_size_feed_forward seq_len_dimrrr  intermediaterrrrr1rlayernorm_beforelayernorm_afterrr?r'r(r,s    zViltLayer.__init__NFc Csj|j|||||d}|d}|dd}|||j}||}||}|||}|f|}|S)N)rrr)rrrzrYrrr) r>rrrrself_attention_outputsr r layer_outputr'r'r(rs     zViltLayer.forwardrrr'r'r?r(rs  rcs0eZdZfddZ     dddZZS) ViltEncodercs:t|_tfddtjD|_d|_dS)Ncsg|]}tqSr')r)rMrr=r'r(rSriz(ViltEncoder.__init__..F) r+r,r=r ModuleListrangenum_hidden_layerslayergradient_checkpointingrr?rr(r,s   zViltEncoder.__init__NFTc Cs|rdnd}|r dnd}t|jD])\} } |r||f}|dur$|| nd} | ||| |} | d}|r:|| df}q|rB||f}|sPtdd|||fDSt|||dS)Nr'rrcss|] }|dur|VqdSrr'rdr'r'r( sz&ViltEncoder.forward..)last_hidden_staterr)rr"r&r ) r>rrrroutput_hidden_states return_dictall_hidden_statesall_self_attentionsr layer_modulelayer_head_mask layer_outputsr'r'r(rs(   zViltEncoder.forward)NNFFTrrr r,rrr'r'r?r(rs  rc@s(eZdZeZdZdZddgZddZdS)ViltPreTrainedModelviltTr)rcCst|tjtjfr#|jjjd|jjd|j dur!|j j dSdSt|tj rF|jjjd|jjd|j durD|jj|j  dSdSt|tj r[|j j |jjddSdS)zInitialize the weightsg)meanstdNg?)r|rrrrldatanormal_r=initializer_rangerzero_r7rrfill_)r>moduler'r'r( _init_weights#s     z!ViltPreTrainedModel._init_weightsN) rrr r config_classbase_model_prefixsupports_gradient_checkpointing_no_split_modulesr8r'r'r'r(r.s  r.cseZdZdfdd ZddZddZdd Ze dd ee j d ee j d ee j dee j dee j dee j dee j dee j dee dee dee dee deeee j ffddZZS) ViltModelTcsXt|||_t||_t||_tj|j |j d|_ |r#t |nd|_ |dS)zv add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer rN)r+r,r=r)rrencoderrrr1r layernorm ViltPoolerpooler post_init)r>r=add_pooling_layerr?r'r(r,6s    zViltModel.__init__cCs |jjjSrrr.rr>r'r'r(get_input_embeddingsGs zViltModel.get_input_embeddingscCs||jj_dSrrD)r>rr'r'r(set_input_embeddingsJzViltModel.set_input_embeddingscCs*|D]\}}|jj|j|qdS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsr>r"rr )r>heads_to_pruner"rr'r'r( _prune_headsMszViltModel._prune_headsNrrrrrrrrrrr&r'rc  Cs| dur| n|jj} | dur| n|jj} | dur| n|jj} |dur*|dur*td|dur9||||} n|durF|dd} ntd| \}}|durU|jn|j}|duretj ||f|d}|durq|durqtd|dur}|dur}td|dur|j dn|j d}||krtd |durtj ||jj |jj f|d}| ||jj }|j|||||||| d \}}||| }|j|||| | | d }|d}||}|jdur||nd}| s||f|d dSt|||j|jd S)ak image_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*): Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings. image_token_type_idx (`int`, *optional*): - The token type ids for images. Examples: ```python >>> from transformers import ViltProcessor, ViltModel >>> from PIL import Image >>> import requests >>> # prepare image and text >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> text = "hello world" >>> processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-mlm") >>> model = ViltModel.from_pretrained("dandelin/vilt-b32-mlm") >>> inputs = processor(image, text, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_states = outputs.last_hidden_state ```NzDYou cannot specify both input_ids and inputs_embeds at the same timerUz5You have to specify either input_ids or inputs_embedsrXzFYou cannot specify both pixel_values and image_embeds at the same timez7You have to specify either pixel_values or image_embedsrzAThe text inputs and image inputs need to have the same batch size)r)rrrr&r'r)r% pooler_outputrr)r=rr&use_return_dictr%warn_if_padding_and_no_attention_maskrDrYr"rrmrq get_head_maskr!rget_extended_attention_maskr>r?rAr rr)r>rrrrrrrrrrr&r'rtext_batch_sizerrYimage_batch_sizeembedding_outputextended_attention_maskencoder_outputssequence_output pooled_outputr'r'r(rUsp*    zViltModel.forward)T NNNNNNNNNNNN)rrr r,rFrGrKrrr" LongTensorr#r}boolrr r&rrr'r'r?r(r=4sZ     r=c$eZdZfddZddZZS)r@cs*tt|j|j|_t|_dSr)r+r,rrr1rTanh activationrr?r'r(r,s zViltPooler.__init__cCs(|dddf}||}||}|Sr\)rr])r>rfirst_token_tensorrWr'r'r(rs  zViltPooler.forwardr-r'r'r?r(r@s r@zU ViLT Model with a language modeling head on top as done during pretraining. cseZdZddgZfddZddZddZe dd ee j d ee j d ee j d ee j dee j dee j dee j dee j dee j dee dee dee de eee j ffddZZS)ViltForMaskedLMzmlm_score.decoder.weightzmlm_score.decoder.biascs,t|t||_t||_|dSr)r+r,r=r/ ViltMLMHead mlm_scorerBrr?r'r(r,s    zViltForMaskedLM.__init__cCs|jjSr)radecoderrEr'r'r(get_output_embeddingssz%ViltForMaskedLM.get_output_embeddingscCs||j_|j|j_dSr)rarbr)r>new_embeddingsr'r'r(set_output_embeddingssz%ViltForMaskedLM.set_output_embeddingsNrrrrrrrrlabelsrr&r'rc  Cs| dur| n|jj} |j||||||||| | | d } | dd\}}|dur+|jdn|jd}|ddd|f|dd|df}}||}d}| durgt}| |j} ||d|jj | d}| s}|f| dd}|dur{|f|S|St ||| j | j dS)a/ image_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*): Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings. labels (*torch.LongTensor* of shape *(batch_size, sequence_length)*, *optional*): Labels for computing the masked language modeling loss. Indices should be in *[-100, 0, ..., config.vocab_size]* (see *input_ids* docstring) Tokens with indices set to *-100* are ignored (masked), the loss is only computed for the tokens with labels in *[0, ..., config.vocab_size]* Examples: ```python >>> from transformers import ViltProcessor, ViltForMaskedLM >>> import requests >>> from PIL import Image >>> import re >>> import torch >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> text = "a bunch of [MASK] laying on a [MASK]." >>> processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-mlm") >>> model = ViltForMaskedLM.from_pretrained("dandelin/vilt-b32-mlm") >>> # prepare inputs >>> encoding = processor(image, text, return_tensors="pt") >>> # forward pass >>> outputs = model(**encoding) >>> tl = len(re.findall("\[MASK\]", text)) >>> inferred_token = [text] >>> # gradually fill in the MASK tokens, one by one >>> with torch.no_grad(): ... for i in range(tl): ... encoded = processor.tokenizer(inferred_token) ... input_ids = torch.tensor(encoded.input_ids) ... encoded = encoded["input_ids"][0][1:-1] ... outputs = model(input_ids=input_ids, pixel_values=encoding.pixel_values) ... mlm_logits = outputs.logits[0] # shape (seq_len, vocab_size) ... # only take into account text features (minus CLS and SEP token) ... mlm_logits = mlm_logits[1 : input_ids.shape[1] - 1, :] ... mlm_values, mlm_ids = mlm_logits.softmax(dim=-1).max(dim=-1) ... # only take into account text ... mlm_values[torch.tensor(encoded) != 103] = 0 ... select = mlm_values.argmax().item() ... encoded[select] = mlm_ids[select].item() ... inferred_token = [processor.decode(encoded)] >>> selected_token = "" >>> encoded = processor.tokenizer(inferred_token) >>> output = processor.decode(encoded.input_ids[0], skip_special_tokens=True) >>> print(output) a bunch of cats laying on a couch. ```N rrrrrrrrr&r'rBrrUrrrr) r=rMr/rmrarrzrYrtrr rr)r>rrrrrrrrrfrr&r'rrVrW text_seq_len text_featuresr mlm_logitsmasked_lm_lossloss_fctrr'r'r(rs@I*  zViltForMaskedLM.forwardrX)rrr _tied_weights_keysr,rcrerrr"rYr#rZrr r&rrr'r'r?r(r_sZ       r_cr[)ViltPredictionHeadTransformcsVtt|j|j|_t|jtrt |j|_ n|j|_ tj |j|j d|_ dS)Nr) r+r,rrr1rr|rrrtransform_act_fnrrrr?r'r(r,hs  z$ViltPredictionHeadTransform.__init__cCs"||}||}||}|Sr)rrprrr'r'r(rqs   z#ViltPredictionHeadTransform.forwardr-r'r'r?r(rogs  rocs.eZdZdfdd ZddZddZZS) r`Ncsbt||_t||_tj|j|jdd|_ t t |j|_ |dur*||j _|j |j _ dS)NFr)r+r,r=ro transformrrr1rrbr/r"r0rrl)r>r=rlr?r'r(r,ys  zViltMLMHead.__init__cCs|j|j_dSr)rrbrEr'r'r( _tie_weightsrHzViltMLMHead._tie_weightscCrr)rqrb)r>rr'r'r(rs  zViltMLMHead.forwardr)rrr r,rrrrr'r'r?r(r`xs r`z Vilt Model transformer with a classifier head on top (a linear layer on top of the final hidden state of the [CLS] token) for visual question answering, e.g. for VQAv2. ceZdZfddZe            ddeejdeejdeejdeejdeejd eejd eejd eejd eejd ee dee dee de e e ejffddZ ZS)ViltForQuestionAnsweringc slt||j|_t||_tt|j|jdt |jdt t|jd|j|_ | dS)NrB) r+r, num_labelsr=r/r Sequentialrr1rGELU classifierrBrr?r'r(r,s   z!ViltForQuestionAnswering.__init__Nrrrrrrrrrfrr&r'rc  Cs| dur| n|jj} |j||||||||| | | d } | r| jn| d}||}d}| dur@| |j} tj || | j d}| sV|f| dd}|durT|f|S|St ||| j | j dS)aX image_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*): Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings. labels (`torch.FloatTensor` of shape `(batch_size, num_labels)`, *optional*): Labels for computing the visual question answering loss. This tensor must be either a one-hot encoding of all answers that are applicable for a given example in the batch, or a soft encoding indicating which answers are applicable, where 1.0 is the highest score. Examples: ```python >>> from transformers import ViltProcessor, ViltForQuestionAnswering >>> import requests >>> from PIL import Image >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> text = "How many cats are there?" >>> processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-finetuned-vqa") >>> model = ViltForQuestionAnswering.from_pretrained("dandelin/vilt-b32-finetuned-vqa") >>> # prepare inputs >>> encoding = processor(image, text, return_tensors="pt") >>> # forward pass >>> outputs = model(**encoding) >>> logits = outputs.logits >>> idx = logits.argmax(-1).item() >>> print("Predicted answer:", model.config.id2label[idx]) Predicted answer: 2 ```NrgrrBrh)r=rMr/rLrxrzrYrrJ binary_cross_entropy_with_logitsrmrrr)r>rrrrrrrrrfrr&r'rrLrrrr'r'r(rs:1  z ViltForQuestionAnswering.forwardrXrrr r,rrr"rYr#rZrrr&rrr'r'r?r(rtsT      rtz Vilt Model transformer with a classifier head on top (a linear layer on top of the final hidden state of the [CLS] token) for image-to-text or text-to-image retrieval, e.g. MSCOCO and F30K. crs)ViltForImageAndTextRetrievalcs2t|t||_t|jd|_|dSr*) r+r,r=r/rrr1 rank_outputrBrr?r'r(r,s   z%ViltForImageAndTextRetrieval.__init__Nrrrrrrrrrfrr&r'rc  Cs| dur| n|jj} d} | durtd|j||||||||| | | d }| r)|jn|d}||}| sH|f|dd}| durF| f|S|St| ||j|jdS)ad image_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*): Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels are currently not supported. Examples: ```python >>> from transformers import ViltProcessor, ViltForImageAndTextRetrieval >>> import requests >>> from PIL import Image >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> texts = ["An image of two cats chilling on a couch", "A football player scoring a goal"] >>> processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-finetuned-coco") >>> model = ViltForImageAndTextRetrieval.from_pretrained("dandelin/vilt-b32-finetuned-coco") >>> # forward pass >>> scores = dict() >>> for text in texts: ... # prepare inputs ... encoding = processor(image, text, return_tensors="pt") ... outputs = model(**encoding) ... scores[text] = outputs.logits[0, :].item() ```NzTraining is not yet supported.rgrrBrh) r=rMNotImplementedErrorr/rLr|rrr)r>rrrrrrrrrfrr&r'rrrLrrr'r'r(rs8- z$ViltForImageAndTextRetrieval.forwardrXrzr'r'r?r(r{sT       r{zq Vilt Model transformer with a classifier head on top for natural language visual reasoning, e.g. NLVR2. crs)"ViltForImagesAndTextClassificationc svt||j|_t||_|j}tt|j ||j |t |j |t t|j ||j|_ | dSr)r+r,rur=r/ num_imagesrrvrr1rrwrxrB)r>r=rr?r'r(r,gs   z+ViltForImagesAndTextClassification.__init__Nrrrrrrrrrfrr&r'rc CsD| dur| n|jj} | dur| n|jj} | dur| n|jj} |dur,|jdkr,|d}|dur:|jdkr:|d}|durC|jdnd} | durT|durR|jdnd} | |jjkr^tdg}| rdgnd}| rjgnd}t | D]l}|j ||||dur|dd|ddddddfnd|dur|dd|ddddfnd|||dur|dd|ddddfnd|d| | | d }| r|j n|d}| || r| |j | r| |jqptj|dd}||}d}| durt}| |j} ||d|j| d}| s|||f}|dur|f|S|St||||d S) a3 image_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*): Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Binary classification labels. Examples: ```python >>> from transformers import ViltProcessor, ViltForImagesAndTextClassification >>> import requests >>> from PIL import Image >>> image1 = Image.open(requests.get("https://lil.nlp.cornell.edu/nlvr/exs/ex0_0.jpg", stream=True).raw) >>> image2 = Image.open(requests.get("https://lil.nlp.cornell.edu/nlvr/exs/ex0_1.jpg", stream=True).raw) >>> text = "The left image contains twice the number of dogs as the right image." >>> processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-finetuned-nlvr2") >>> model = ViltForImagesAndTextClassification.from_pretrained("dandelin/vilt-b32-finetuned-nlvr2") >>> # prepare inputs >>> encoding = processor([image1, image2], text, return_tensors="pt") >>> # forward pass >>> outputs = model(input_ids=encoding.input_ids, pixel_values=encoding.pixel_values.unsqueeze(0)) >>> logits = outputs.logits >>> idx = logits.argmax(-1).item() >>> print("Predicted answer:", model.config.id2label[idx]) Predicted answer: True ```Nrrz\Make sure to match the number of images in the model with the number of images in the input.) rrrrrrrrrr&r'rUrErh)r=rr&rMndim unsqueezermrrr r/rLrrrr"rurxrrzrYrtrur)r>rrrrrrrrrfrr&r'rpooler_outputsrrrrrLrWrrrmrr'r'r(rysp/      ,&&       z*ViltForImagesAndTextClassification.forwardrX)rrr r,rrr"rYr#rZrrr&rrr'r'r?r(r~asT      r~crs)ViltForTokenClassificationcsNt||j|_t|dd|_t|j|_t |j |j|_ | dS)NF)rC) r+r,rur=r/rr:r;r<rr1rxrBrr?r'r(r,s  z#ViltForTokenClassification.__init__Nrrrrrrrrrfrr&r'rc  Cs| dur| n|jj} |j||||||||| | | d } | d}|dur'|jdn|jd}||}||ddd|f}d}| durZt}| |j} || d|j | d}| sp|f| dd}|durn|f|S|St ||| j | j dS)a/ image_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*): Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings. labels (`torch.LongTensor` of shape `(batch_size, text_sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. NrgrrrUrBrh)r=rMr/rmr<rxrrzrYrtrurrr)r>rrrrrrrrrfrr&r'rrVtext_input_sizerrrmrr'r'r(rs@  z"ViltForTokenClassification.forwardrX)rrr r,rrr"rYr#rZrrr&rrr'r'r?r(rsT       r)r{r~rr_rtrr=r.)>r!collections.abcrr dataclassesrtypingrrr"torch.utils.checkpointrtorch.nnr activationsrmodeling_layersr modeling_outputsr r r r rrmodeling_utilsr pytorch_utilsrrrutilsrrconfiguration_viltr get_loggerrloggerrModuler)r-r3rrrr rrrr.r=r@r_ror`rtr{r~r__all__r'r'r'r(s|         9=#&,j] O