o ei\@sldZddlZddlZddlmZddlZddlmZddlm Z ddl m Z ddl mZdd lmZdd lmZmZmZmZmZmZdd lmZdd lmZmZd dlmZee Z!eeddGdddeZ"Gdddej#Z$Gdddej#Z%Gdddej#Z&Gdddej#Z'Gdddej#Z(Gdddej#Z)Gdd d ej#Z*Gd!d"d"ej#Z+Gd#d$d$eZ,Gd%d&d&ej#Z-eGd'd(d(eZ.eGd)d*d*e.Z/Gd+d,d,ej#Z0ed-dGd.d/d/e.Z1Gd0d1d1ej#Z2Gd2d3d3ej#Z3ed4dGd5d6d6e.Z4ed7dGd8d9d9e.Z5ed:dGd;d<dd>e.Z7gd?Z8dS)@zPyTorch ViLT model.N) dataclass)nn)CrossEntropyLoss)initialization)ACT2FN)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPoolingMaskedLMOutput ModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel)auto_docstringlogging) ViltConfigzF Class for outputs of [`ViltForImagesAndTextClassification`]. ) custom_introc@sjeZdZUdZdZejdBed<dZejdBed<dZ e e ejdBed<dZ e e ejdBed<dS)(ViltForImagesAndTextClassificationOutputa7 loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Classification (or regression if config.num_labels==1) loss. logits (`torch.FloatTensor` of shape `(batch_size, config.num_labels)`): Classification (or regression if config.num_labels==1) scores (before SoftMax). hidden_states (`list[tuple(torch.FloatTensor)]`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): List of tuples of `torch.FloatTensor` (one for each image-text pair, each tuple containing the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. Nlosslogits hidden_states attentions) __name__ __module__ __qualname____doc__rtorch FloatTensor__annotations__rrlisttuplerr#r#d/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/vilt/modeling_vilt.pyr+s  rcs6eZdZdZfddZd ddZ d dd ZZS) ViltEmbeddingsz Construct the text and patch embeddings. Text embeddings are equivalent to BERT embeddings. Patch embeddings are equivalent to ViT embeddings. cstt||_ttdd|j|_ t ||_ |j j }ttd|d|j|_ t|j|j|_t|j|_||_dSNr)super__init__TextEmbeddingstext_embeddingsr Parameterrzeros hidden_size cls_tokenViltPatchEmbeddingspatch_embeddings num_patchesposition_embeddings Embeddingmodality_type_vocab_sizetoken_type_embeddingsDropouthidden_dropout_probdropoutconfig)selfr9r1 __class__r#r$r(Ls    zViltEmbeddings.__init__cs|jjjj\}}}}||}|dddddddf}tjj||jd|jdfd}|dddfj dddddf} |dddfj dddddf} |j\} } |j j |j j } |j ddddddfddd| | | tjfddt| | Ddd}|ddd}|ddd}tjtjt|jd t|jd d d d dj|jd }|ddddddddf}||jd|jdd d d }|dd}|d}dksdustts| | }|n | | }t||jddd|jdddddf}fdd|D}fdd|D}dd|D}dd|D}fdd|D}g}t t|||D]B\}\}}}|dkrvt!t"|}|#|||qTtj!t"||dd}|#tj|||||gddqTtj|dd}||dddf|dddff| d | }||dddf|dddff| d }||dddf|dddff| d d}||dddf|dddff| d | }|j$| d d }tj||fdd}tj|j dddddfdddddf| d d |fdd}||}|%|}tjt"|jdd||gdd}|||fffS)Nrsizerrdimc sBg|]\}}tjtjj||fdddd|d|fqS)bilinearT)r@mode align_cornersr)r functionalpad interpolate).0hw)height spatial_poswidthr#r$ hs z/ViltEmbeddings.visual_embed..ij)indexingdeviceF)as_tuplec$g|]}dddf|kqSNrr#rIu) valid_idxr#r$rO$crWrXr#rY) non_valid_idxr#r$rOr\cSg|]}|dqSrr?rIvr#r#r$rOcSr^r_r?r`r#r#r$rOrbcsg|]}|qSr#r#r`max_image_lengthr#r$rOT) replacement)&r0 projectionweightshapefloatrrFrHlongsumr9 image_size patch_sizer2 transposeviewrcatzipflattenstackmeshgridarangetorUexpand isinstanceintmaxminnonzerounique enumerate multinomialonesappendr.r8)r: pixel_values pixel_maskrd_phpwxx_maskx_hx_w batch_size num_channels patch_dim pos_embed patch_indexeffective_resolution unique_rows valid_row_idxnon_valid_row_idx valid_numsnon_valid_numspad_numsselectiranvp valid_choice pad_choice cls_tokensr#)rLrdr]rMr[rNr$ visual_embed[sx  $$$0  (     (.,..8 &zViltEmbeddings.visual_embedrc  Cs|j|||d} |dur|j|||jjd\}} } n|d} |dur%d}| |tj|tj| j d} ||tj | |tj| j d}tj | |gdd} tj || gdd} | | fS)N) input_idstoken_type_ids inputs_embedsrcrdtyperUrA) r*rr9rdrsr5r zeros_likerkrU full_likerq)r:rattention_maskrrrr image_embedsimage_token_type_idx text_embeds image_masksr embeddingsmasksr#r#r$forwards&   zViltEmbeddings.forward)r=)r)rrrrr(rr __classcell__r#r#r;r$r%Cs   ar%cs*eZdZdZfddZdddZZS)r)zGConstruct the embeddings from word, position and token_type embeddings.csttj|j|j|jd|_t|j|j|_ t|j |j|_ tj |j|j d|_ t|j|_|jdt|jddd|jdtj|jtjddddS) N) padding_idxeps position_idsrrQF) persistentrr)r'r(rr3 vocab_sizer- pad_token_idword_embeddingsmax_position_embeddingsr2type_vocab_sizer5 LayerNormlayer_norm_epsr6r7r8register_bufferrrvrxr,rr@rkr:r9r;r#r$r(s  zTextEmbeddings.__init__Nc Cs|dur |}n|dd}|d}|dur$|jddd|f}|durNt|drC|jddd|f}||d|}|}n tj|tj|jjd}|durW| |}| |} || } | |} | | 7} | | } | | } | S)NrQrrrr)r@rhasattrrrxrr,rkrUrr5r2rr8) r:rrrr input_shape seq_lengthbuffered_token_type_ids buffered_token_type_ids_expandedr5rr2r#r#r$rs*       zTextEmbeddings.forward)NNNNrrrrr(rrr#r#r;r$r)s r)cs(eZdZdZfddZddZZS)r/z# Image to Patch Embedding. cst|j|j}}|j|j}}t|tjj r|n||f}t|tjj r)|n||f}|d|d|d|d}||_||_||_||_ t j ||||d|_ dS)Nrr) kernel_sizestride)r'r(rmrnrr-ry collectionsabcIterabler1rConv2drg)r:r9rmrnrr-r1r;r#r$r(s  zViltPatchEmbeddings.__init__cCs@|j\}}}}||jkrtd|jjj}||j|d}|S)NzeMake sure that the channel dimension of the pixel values match with the one set in the configuration.r)rir ValueErrorrgrhrrw)r:rrrrLrN target_dtyperr#r#r$r's  zViltPatchEmbeddings.forwardrr#r#r;r$r/s r/c&eZdZfddZdddZZS)ViltSelfAttentioncst|j|jdkrt|dstd|jd|jd|j|_t|j|j|_|j|j|_t j |j|j|j d|_ t j |j|j|j d|_ t j |j|j|j d|_t |j|_dS)Nrembedding_sizezThe hidden size z4 is not a multiple of the number of attention heads .)bias)r'r(r-num_attention_headsrrrzattention_head_size all_head_sizerLinearqkv_biasquerykeyvaluer6attention_probs_dropout_probr8rr;r#r$r(3s  zViltSelfAttention.__init__NFcCs|j\}}}|||d|j|jdd}|||d|j|jdd}|||d|j|jdd} t ||dd} | t |j} |durS| |} t j dd| } || } t | | } | dddd} | dd|jf} | j| } |r| | f}|S| f}|S)NrQrr>rPrArr)rirrprrrorrrmatmulmathsqrtrSoftmaxr8permute contiguousr@r)r:rroutput_attentionsrrr query_layer key_layer value_layerattention_scoresattention_probs context_layernew_context_layer_shapeoutputsr#r#r$rEs6     zViltSelfAttention.forwardNFrrrr(rrr#r#r;r$r2s rcsBeZdZdZdeffdd Zdejdejdejfdd ZZ S) ViltSelfOutputz The residual connection is defined in ViltLayer instead of here (as is the case with other models), due to the layernorm applied before each block. r9cs.tt|j|j|_t|j|_dSN) r'r(rrr-denser6r7r8rr;r#r$r(w zViltSelfOutput.__init__r input_tensorreturncC||}||}|Srrr8r:rrr#r#r$r|  zViltSelfOutput.forward) rrrrrr(rTensorrrr#r#r;r$rqs$rcr) ViltAttentioncs"tt||_t||_dSr)r'r(r attentionroutputrr;r#r$r(s  zViltAttention.__init__NFcCs4||||}||d|}|f|dd}|S)Nrr)rr)r:rrr self_outputsattention_outputrr#r#r$rszViltAttention.forwardrrr#r#r;r$rs rcs8eZdZdeffdd ZdejdejfddZZS)ViltIntermediater9csDtt|j|j|_t|jt rt |j|_ dS|j|_ dSr) r'r(rrr-intermediate_sizerry hidden_actstrrintermediate_act_fnrr;r#r$r(s   zViltIntermediate.__init__rrcCrr)rrr:rr#r#r$rrzViltIntermediate.forward rrrrr(rrrrr#r#r;r$rsrcs>eZdZdeffdd ZdejdejdejfddZZS) ViltOutputr9cs.tt|j|j|_t|j|_ dSr) r'r(rrrr-rr6r7r8rr;r#r$r(rzViltOutput.__init__rrrcCs ||}||}||}|Srrrr#r#r$rs  zViltOutput.forwardrr#r#r;r$rs$rcs*eZdZdZfddZdddZZS) ViltLayerz?This corresponds to the Block class in the timm implementation.csbt|j|_d|_t||_t||_t||_ t j |j |j d|_t j |j |j d|_dS)Nrr)r'r(chunk_size_feed_forward seq_len_dimrrr intermediaterrrrr-rlayernorm_beforelayernorm_afterrr;r#r$r(s    zViltLayer.__init__NFcCsh|j||||d}|d}|dd}|||j}||}||}|||}|f|}|S)N)rrr)rr rwrUr rr)r:rrrself_attention_outputsrr layer_outputr#r#r$rs     zViltLayer.forwardrrr#r#r;r$rs  rcs.eZdZfddZ    dddZZS) ViltEncodercs:t|_tfddtjD|_d|_dS)Ncsg|]}tqSr#)r)rIrr9r#r$rOrez(ViltEncoder.__init__..F) r'r(r9r ModuleListrangenum_hidden_layerslayergradient_checkpointingrr;rr$r(s   zViltEncoder.__init__NFTc Cs|rdnd}|r dnd}t|jD]\}} |r||f}| |||} | d}|r/|| df}q|r7||f}|sEtdd|||fDSt|||dS)Nr#rrcss|] }|dur|VqdSrr#r`r#r#r$ sz&ViltEncoder.forward..)last_hidden_staterr)rrr"r ) r:rrroutput_hidden_states return_dictall_hidden_statesall_self_attentionsr layer_module layer_outputsr#r#r$rs&     zViltEncoder.forward)NFFTrr#r#r;r$r s  r cs:eZdZUeed<dZdZdZddgZfddZ Z S) ViltPreTrainedModelr9vilt)imagetextTr%rcsLt|t|tr$t|jt|jj d dt |j dSdS)NrQr) r' _init_weightsryr)initcopy_rrrvrirxzeros_r)r:moduler;r#r$r s  "z!ViltPreTrainedModel._init_weights) rrrrr base_model_prefixinput_modalitiessupports_gradient_checkpointing_no_split_modulesr rr#r#r;r$rs rcseZdZdfdd ZddZddZe           dd ejdBd ej dBd ejdBd ej dBd ejdBdej dBdej dBde dBde dBde dBde dBde e ej BfddZZS) ViltModelTcsXt|||_t||_t||_tj|j |j d|_ |r#t |nd|_ |dS)zv add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer rN)r'r(r9r%rr encoderrrr-r layernorm ViltPoolerpooler post_init)r:r9add_pooling_layerr;r#r$r(s    zViltModel.__init__cCs |jjjSrrr*rr:r#r#r$get_input_embeddingss zViltModel.get_input_embeddingscCs||jj_dSrr0)r:rr#r#r$set_input_embeddings"szViltModel.set_input_embeddingsNrrrrrrrrrrrrc  Ks| dur| n|jj} | dur| n|jj} | dur| n|jj} |dur*|dur*td|dur9||||} n|durF|dd} ntd| \}}|durU|jn|j}|duretj ||f|d}|durq|durqtd|dur}|dur}td|dur|j dn|j d}||krtd |durtj ||jj |jj f|d}|j ||||||||d \}}| || }|j||| | | d }|d}||}|jdur||nd}| s||f|d dSt|||j|jd S)a image_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*): Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings. image_token_type_idx (`int`, *optional*): - The token type ids for images. Examples: ```python >>> from transformers import ViltProcessor, ViltModel >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> # prepare image and text >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> text = "hello world" >>> processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-mlm") >>> model = ViltModel.from_pretrained("dandelin/vilt-b32-mlm") >>> inputs = processor(image, text, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_states = outputs.last_hidden_state ```NzDYou cannot specify both input_ids and inputs_embeds at the same timerQz5You have to specify either input_ids or inputs_embedsrTzFYou cannot specify both pixel_values and image_embeds at the same timez7You have to specify either pixel_values or image_embedsrzAThe text inputs and image inputs need to have the same batch size)r)rrrrr)r pooler_outputrr)r9rruse_return_dictr%warn_if_padding_and_no_attention_maskr@rUrrrirmrget_extended_attention_maskr*r+r-r rr)r:rrrrrrrrrrrkwargsrtext_batch_sizerrUimage_batch_sizeembedding_outputextended_attention_maskencoder_outputssequence_output pooled_outputr#r#r$r%sl,    zViltModel.forward)T NNNNNNNNNNN)rrrr(r2r3rr LongTensorrrzboolr r"rrr#r#r;r$r) sR     r)c$eZdZfddZddZZS)r,cs*tt|j|j|_t|_dSr)r'r(rrr-rTanh activationrr;r#r$r(s zViltPooler.__init__cCs(|dddf}||}||}|SrX)rrE)r:rfirst_token_tensorr?r#r#r$rs  zViltPooler.forwardrr#r#r;r$r,s r,zU ViLT Model with a language modeling head on top as done during pretraining. cseZdZddiZfddZddZddZe dd ej d Bd ej d Bd ej d Bd ej d Bdej d Bdej d Bdej d Bdej d Bde d Bde d Bde d Bde e ej BfddZZS)ViltForMaskedLMzmlm_score.decoder.weightz6vilt.embeddings.text_embeddings.word_embeddings.weightcs,t|t||_t||_|dSr)r'r(r)r ViltMLMHead mlm_scorer.rr;r#r$r(s    zViltForMaskedLM.__init__cCs|jjSr)rIdecoderr1r#r#r$get_output_embeddingssz%ViltForMaskedLM.get_output_embeddingscCs||j_|j|j_dSr)rIrJr)r:new_embeddingsr#r#r$set_output_embeddingssz%ViltForMaskedLM.set_output_embeddingsNrrrrrrrlabelsrrrrc  Ks | dur| n|jj} |j|||||||| | | d } | dd\}}|dur*|jdn|jd}|ddd|f|dd|df}}||}d}|durft}||j}||d|jj |d}| s||f| dd}|durz|f|S|St ||| j | j dS)a image_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*): Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings. labels (*torch.LongTensor* of shape *(batch_size, sequence_length)*, *optional*): Labels for computing the masked language modeling loss. Indices should be in *[-100, 0, ..., config.vocab_size]* (see *input_ids* docstring) Tokens with indices set to *-100* are ignored (masked), the loss is only computed for the tokens with labels in *[0, ..., config.vocab_size]* Examples: ```python >>> from transformers import ViltProcessor, ViltForMaskedLM >>> import httpx >>> from io import BytesIO >>> from PIL import Image >>> import re >>> import torch >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> text = "a bunch of [MASK] laying on a [MASK]." >>> processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-mlm") >>> model = ViltForMaskedLM.from_pretrained("dandelin/vilt-b32-mlm") >>> # prepare inputs >>> encoding = processor(image, text, return_tensors="pt") >>> # forward pass >>> outputs = model(**encoding) >>> tl = len(re.findall("\[MASK\]", text)) >>> inferred_token = [text] >>> # gradually fill in the MASK tokens, one by one >>> with torch.no_grad(): ... for i in range(tl): ... encoded = processor.tokenizer(inferred_token) ... input_ids = torch.tensor(encoded.input_ids) ... encoded = encoded["input_ids"][0][1:-1] ... outputs = model(input_ids=input_ids, pixel_values=encoding.pixel_values) ... mlm_logits = outputs.logits[0] # shape (seq_len, vocab_size) ... # only take into account text features (minus CLS and SEP token) ... mlm_logits = mlm_logits[1 : input_ids.shape[1] - 1, :] ... mlm_values, mlm_ids = mlm_logits.softmax(dim=-1).max(dim=-1) ... # only take into account text ... mlm_values[torch.tensor(encoded) != 103] = 0 ... select = mlm_values.argmax().item() ... encoded[select] = mlm_ids[select].item() ... inferred_token = [processor.decode(encoded)] >>> selected_token = "" >>> encoded = processor.tokenizer(inferred_token) >>> output = processor.decode(encoded.input_ids[0], skip_special_tokens=True) >>> print(output) a bunch of cats laying on a couch. ```N rrrrrrrrrr>rrQrrrr) r9r5rrirIrrwrUrprr rr)r:rrrrrrrrNrrrr8rr>r? text_seq_len text_featuresr mlm_logitsmasked_lm_lossloss_fctrr#r#r$rs>K *  zViltForMaskedLM.forwardr@)rrr_tied_weights_keysr(rKrMrrrArrBr r"rrr#r#r;r$rGsV       rGcrC)ViltPredictionHeadTransformcsVtt|j|j|_t|jtrt |j|_ n|j|_ tj |j|j d|_ dS)Nr) r'r(rrr-rryrrrtransform_act_fnrrrr;r#r$r(5s  z$ViltPredictionHeadTransform.__init__cCs"||}||}||}|Sr)rrXrrr#r#r$r>s   z#ViltPredictionHeadTransform.forwardrr#r#r;r$rW4s  rWcrC)rHcs0t||_t||_t|j|j|_ dSr) r'r(r9rW transformrrr-rrJrr;r#r$r(Fs  zViltMLMHead.__init__cCrr)rYrJ)r:rr#r#r$rLrzViltMLMHead.forwardrr#r#r;r$rHEs rHz Vilt Model transformer with a classifier head on top (a linear layer on top of the final hidden state of the [CLS] token) for visual question answering, e.g. for VQAv2. ceZdZfddZe           ddejdBdejdBdejdBdejdBdejdBd ejdBd ejdBd ejdBd edBd edBdedBde e ejBfddZ Z S)ViltForQuestionAnsweringc slt||j|_t||_tt|j|jdt |jdt t|jd|j|_ | dS)Nr>) r'r( num_labelsr)rr Sequentialrr-rGELU classifierr.rr;r#r$r(Ys   z!ViltForQuestionAnswering.__init__NrrrrrrrrNrrrrc  Ks| dur| n|jj} |j|||||||| | | d } | r| jn| d}||}d}|dur?||j}tj |||j d}| sU|f| dd}|durS|f|S|St ||| j | j dS)a image_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*): Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings. labels (`torch.FloatTensor` of shape `(batch_size, num_labels)`, *optional*): Labels for computing the visual question answering loss. This tensor must be either a one-hot encoding of all answers that are applicable for a given example in the batch, or a soft encoding indicating which answers are applicable, where 1.0 is the highest score. Examples: ```python >>> from transformers import ViltProcessor, ViltForQuestionAnswering >>> import httpx >>> from io import BytesIO >>> from PIL import Image >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> text = "How many cats are there?" >>> processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-finetuned-vqa") >>> model = ViltForQuestionAnswering.from_pretrained("dandelin/vilt-b32-finetuned-vqa") >>> # prepare inputs >>> encoding = processor(image, text, return_tensors="pt") >>> # forward pass >>> outputs = model(**encoding) >>> logits = outputs.logits >>> idx = logits.argmax(-1).item() >>> print("Predicted answer:", model.config.id2label[idx]) Predicted answer: 2 ```NrOrr>rP)r9r5rr4r_rwrUrrF binary_cross_entropy_with_logitsrir rr)r:rrrrrrrrNrrrr8rr4rrrr#r#r$rjs83  z ViltForQuestionAnswering.forwardr@ rrrr(rrrArrBr r"rrr#r#r;r$r[RsN      r[z Vilt Model transformer with a classifier head on top (a linear layer on top of the final hidden state of the [CLS] token) for image-to-text or text-to-image retrieval, e.g. MSCOCO and F30K. crZ)ViltForImageAndTextRetrievalcs2t|t||_t|jd|_|dSr&) r'r(r)rrrr- rank_outputr.rr;r#r$r(s   z%ViltForImageAndTextRetrieval.__init__NrrrrrrrrNrrrrc  Ks| dur| n|jj} d} |durtd|j|||||||| | | d }| r(|jn|d}||}| sG|f|dd}| durE| f|S|St| ||j|jdS)a image_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*): Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels are currently not supported. Examples: ```python >>> from transformers import ViltProcessor, ViltForImageAndTextRetrieval >>> import httpx >>> from io import BytesIO >>> from PIL import Image >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> texts = ["An image of two cats chilling on a couch", "A football player scoring a goal"] >>> processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-finetuned-coco") >>> model = ViltForImageAndTextRetrieval.from_pretrained("dandelin/vilt-b32-finetuned-coco") >>> # forward pass >>> scores = dict() >>> for text in texts: ... # prepare inputs ... encoding = processor(image, text, return_tensors="pt") ... outputs = model(**encoding) ... scores[text] = outputs.logits[0, :].item() ```NzTraining is not yet supported.rOrr>rP) r9r5NotImplementedErrorrr4rcr rr)r:rrrrrrrrNrrrr8rrr4rrr#r#r$rs6/ z$ViltForImageAndTextRetrieval.forwardr@rar#r#r;r$rbsN       rbzq Vilt Model transformer with a classifier head on top for natural language visual reasoning, e.g. NLVR2. crZ)"ViltForImagesAndTextClassificationc svt||j|_t||_|j}tt|j ||j |t |j |t t|j ||j|_ | dSr)r'r(r\r)r num_imagesrr]rr-rr^r_r.)r:r9rfr;r#r$r(-s   z+ViltForImagesAndTextClassification.__init__NrrrrrrrrNrrrrc KsB| dur| n|jj} | dur| n|jj} | dur| n|jj} |dur,|jdkr,|d}|dur:|jdkr:|d}|durC|jdnd} | durT|durR|jdnd} | |jjkr^tdg}| rdgnd}| rjgnd}t | D]k}|j ||||dur|dd|ddddddfnd|dur|dd|ddddfnd||dur|dd|ddddfnd|d| | | d }| r|j n|d}| || r| |j | r| |jqptj|dd}||}d}|durt}||j}||d|j|d}| s|||f}|dur|f|S|St||||d S) a image_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*): Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Binary classification labels. Examples: ```python >>> from transformers import ViltProcessor, ViltForImagesAndTextClassification >>> import httpx >>> from io import BytesIO >>> from PIL import Image >>> url_1 = "https://lil.nlp.cornell.edu/nlvr/exs/ex0_0.jpg" >>> with httpx.stream("GET", url_1) as response: ... image_1 = Image.open(BytesIO(response.read())) >>> url_2 = "https://lil.nlp.cornell.edu/nlvr/exs/ex0_1.jpg" >>> with httpx.stream("GET", url_2) as response: ... image_2 = Image.open(BytesIO(response.read())) >>> text = "The left image contains twice the number of dogs as the right image." >>> processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-finetuned-nlvr2") >>> model = ViltForImagesAndTextClassification.from_pretrained("dandelin/vilt-b32-finetuned-nlvr2") >>> # prepare inputs >>> encoding = processor([image_1, image_2], text, return_tensors="pt") >>> # forward pass >>> outputs = model(input_ids=encoding.input_ids, pixel_values=encoding.pixel_values.unsqueeze(0)) >>> logits = outputs.logits >>> idx = logits.argmax(-1).item() >>> print("Predicted answer:", model.config.id2label[idx]) Predicted answer: True ```Nrrz\Make sure to match the number of images in the model with the number of images in the input.) rrrrrrrrrrrQrArP)r9rrr5ndim unsqueezerirfrrrr4rrrrrqr_rrwrUrpr\r)r:rrrrrrrrNrrrr8rfpooler_outputsrrrrr4r?rrrUrr#r#r$r?sn6      ,&&       z*ViltForImagesAndTextClassification.forwardr@) rrrr(rrrArrBrr"rrr#r#r;r$re'sN      recrZ)ViltForTokenClassificationcsNt||j|_t|dd|_t|j|_t |j |j|_ | dS)NF)r/) r'r(r\r)rrr6r7r8rr-r_r.rr;r#r$r(s  z#ViltForTokenClassification.__init__NrrrrrrrrNrrrrc  Ks| dur| n|jj} |j|||||||| | | d } | d}|dur&|jdn|jd}||}||ddd|f}d}|durYt}||j}|| d|j | d}| so|f| dd}|durm|f|S|St ||| j | j dS)a/ image_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*): Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings. labels (`torch.LongTensor` of shape `(batch_size, text_sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. NrOrrrQr>rP)r9r5rrir8r_rrwrUrpr\rrr)r:rrrrrrrrNrrrr8rr>text_input_sizerrrUrr#r#r$rs>   z"ViltForTokenClassification.forwardr@) rrrr(rrrArrBrr"rrr#r#r;r$rksN       rk)rbrerkrGr[rr)r)9rcollections.abcrr dataclassesrrrtorch.nnrrr! activationsrmodeling_layersrmodeling_outputsr r r r r rmodeling_utilsrutilsrrconfiguration_viltr get_loggerrloggerrModuler%r)r/rrrrrrr rr)r,rGrWrHr[rbrerk__all__r#r#r#r$sx           6? %)   k^ N