o eiD@sdZddlZddlmZddlmZddlmZddlmZm Z ddl m Z dd l m Z dd lmZdd lmZmZmZmZmZd d lmZeeZeGddde ZeddGdddeZeddGdddeeZgdZdS)zPyTorch Fuyu model.N)nn)Cache)GenerationMixin)BaseModelOutputWithPoolingCausalLMOutputWithPast)PreTrainedModel) AutoModel)Unpack)TransformersKwargsauto_docstringcan_return_tupleloggingtorch_compilable_check) FuyuConfigc@s:eZdZUeed<dZdZdZdZdZ dZ dZ gZ dZ dS)FuyuPreTrainedModelconfigfuyu)imagetextTpast_key_valuesN)__name__ __module__ __qualname__r__annotations__base_model_prefixinput_modalitiessupports_gradient_checkpointing_supports_attention_backend_supports_flash_attn_supports_sdpa_supports_flex_attn_no_split_modules_skip_keys_device_placementr%r%d/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/fuyu/modeling_fuyu.pyr s rzt The Fuyu model which consists of a vision backbone and a language model, without a language modeling head. ) custom_introcs6eZdZddiZdeffdd ZddZdd Zd ej d e ej d ej d ej fddZ e e dejdeed eeBfddZdejdejdejfddZe           d%dejdBdej dBdej dBdej dBdejdBdedBdejdBdedBd edBd!edBd"edBd eeBfd#d$ZZS)& FuyuModelzlanguage_model.modellanguage_modelrcs\t||j|_|jj|_t|j|_t |j |j |j |j |_d|_|dS)NF)super__init__ pad_token_id padding_idx text_config vocab_sizer from_configr)rLinear patch_size num_channels hidden_sizevision_embed_tokensgradient_checkpointing post_initselfr __class__r%r&r+6s   zFuyuModel.__init__cC |jSN)r)get_input_embeddingsr9r%r%r&r>C zFuyuModel.get_input_embeddingscC|j|dSr=)r)set_input_embeddingsr9valuer%r%r&rBFzFuyuModel.set_input_embeddingsword_embeddingscontinuous_embeddingsimage_patch_input_indicesreturnc Cs|jdt|kstdt|d|jd|}t|jdD]A}tj||dkddd}|||}|jd||jdkrVtd||jd|jd|d ||||j|||f<q#|S) aThis function places the continuous_embeddings into the word_embeddings at the locations indicated by image_patch_input_indices. Different batch elements can have different numbers of continuous embeddings. Args: word_embeddings (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Tensor of word embeddings. continuous_embeddings (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`): Tensor of continuous embeddings. The length of the list is the batch size. Each entry is shape [num_image_embeddings, hidden], and num_image_embeddings needs to match the number of non-negative indices in image_patch_input_indices for that batch element. image_patch_input_indices (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Tensor of indices of the image patches in the input_ids tensor. rz7Batch sizes must match! Got len(continuous_embeddings)=z and word_embeddings.shape[0]=T)as_tuplezGNumber of continuous embeddings continuous_embeddings[batch_idx].shape=zA does not match number of continuous token ids src_indices.shape=z in batch element .) shapelen ValueErrorclonerangetorchnonzerotodevice)r9rFrGrHoutput_embeddings batch_idx dst_indices src_indicesr%r%r&gather_continuous_embeddingsIs(  z&FuyuModel.gather_continuous_embeddings pixel_valueskwargscKs||}t|dS)z pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, image_size, image_size)`): The tensors corresponding to the input images. )last_hidden_state)r5r)r9rZr[patch_embeddingsr%r%r&get_image_featuresus zFuyuModel.get_image_features input_ids inputs_embedsimage_featurescCs|dur||tj|jjtj|jdk}|d}n||jjk}|}|j d|j d}| d | |j}t |||kd|d||S)z Obtains multimodal placeholder mask from `input_ids` or `inputs_embeds`, and checks that the placeholder token count is equal to the length of multimodal features. If the lengths are different, an error is raised. NdtyperTrrz6Image features and image tokens do not match, tokens: z , features: )r>rQtensorrimage_token_idlongrTallsumrL unsqueeze expand_asrSrnumel)r9r_r`raspecial_image_maskn_image_tokensn_image_featuresr%r%r&get_placeholder_masks  zFuyuModel.get_placeholder_maskN image_patchesimage_patches_indicesattention_mask position_idsr use_cacheoutput_attentionsoutput_hidden_states return_dictc  Ksl| dur| n|jj} | dur| n|jj} |dur|n|jj}| dur$| n|jj} |dur4|dur4td|dur>|j\} }n|durI|j\} }}ntd|durv|durX|jn|j}|durc|nd}t j |||t j |d}| d}|dur|j |}|dur|j|ddj}||j|j}|j|||d}|||}|j d ||||| | || d | }|S) a image_patches (`torch.FloatTensor` of shape `(batch_size, num_total_patches, patch_size_ x patch_size x num_channels)`, *optional*): Image patches to be used as continuous embeddings. The patches are flattened and then projected to the hidden size of the model. image_patches_indices (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Tensor of indices of the image patches in the input_ids tensor. NzDYou cannot specify both input_ids and inputs_embeds at the same timez4You have to specify either input_is or inputs_embedsrrbT)rx)r`ra)r`rsrtrrvrwrurxr%)rrvrwruuse_return_dictrNrLrTget_seq_lengthrQarangergrjr)r>r^r\rSrcrpmasked_scatter)r9r_rqrrrsrtrr`rurvrwrxr[ batch_size seq_length_rTpast_key_values_lengthr]rmoutputsr%r%r&forwardsR     zFuyuModel.forward) NNNNNNNNNNN)rrr_checkpoint_conversion_mappingrr+r>rBrQTensorlistrYr r FloatTensorr r tuplerr^ LongTensorrprboolrr __classcell__r%r%r:r&r(.s  ,       r(zz Fuyu Model with a language modeling head on top for causal language model conditioned on image patches and text. c seZdZddddZddiZdeffdd Zd d Zd d Ze e             d#de j dBde j dBde j dBde j dBde j dBdedBde jdBdedBde j dBdedBdedBdedBdedBdeeBfddZ       d$fd!d" ZZS)%FuyuForCausalLMzmodel.language_modelzmodel.vision_embed_tokenslm_head)z^language_model.modelz^vision_embed_tokensz^language_model.lm_headzlm_head.weightz(model.language_model.embed_tokens.weightrcs<t|t||_tj|jj|jjdd|_ | dS)NF)bias) r*r+r(modelrr1r.r4r/rr7r8r:r%r&r+s   zFuyuForCausalLM.__init__cCr<r=)rr>r?r%r%r&r>r@z$FuyuForCausalLM.get_input_embeddingscCrAr=)rrBrCr%r%r&rBrEz$FuyuForCausalLM.set_input_embeddingsNrr_rqrrrsrtrr`rulabelsrvrwrxlogits_to_keeprIc Ks| dur| n|jj} | dur| n|jj} |dur|n|jj}| dur$| n|jj} |j|||||||| | |dd }|d}t| trGt| dn| }| |dd|ddf}d}| durl|j d|| |jj j d|}t |||j|j|jdS)a image_patches (`torch.FloatTensor` of shape `(batch_size, num_total_patches, patch_size_ x patch_size x num_channels)`, *optional*): Image patches to be used as continuous embeddings. The patches are flattened and then projected to the hidden size of the model. image_patches_indices (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Tensor of indices of the image patches in the input_ids tensor. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.text_config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.text_config.vocab_size]`. Examples: ```python >>> from transformers import FuyuProcessor, FuyuForCausalLM >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> processor = FuyuProcessor.from_pretrained("adept/fuyu-8b") >>> model = FuyuForCausalLM.from_pretrained("adept/fuyu-8b") >>> url = "https://huggingface.co/datasets/hf-internal-testing/fixtures-captioning/resolve/main/bus.png" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> prompt = "Generate a coco-style caption.\n" >>> inputs = processor(images=image, text=prompt, return_tensors="pt") >>> outputs = model(**inputs) >>> generated_ids = model.generate(**inputs, max_new_tokens=7) >>> generation_text = processor.batch_decode(generated_ids[:, -7:], skip_special_tokens=True) >>> print(generation_text[0]) A blue bus parked on the side of a road. ```NT) r_rqrrr`rsrtrrvrwrurxr)logitsrr/)lossrr hidden_states attentionsr%)rrvrwruryr isinstanceintslicer loss_functionr.r/rrrr)r9r_rqrrrsrtrr`rurrvrwrxrr[rr slice_indicesrrr%r%r&rsF8 zFuyuForCausalLM.forwardFc sHtj|f|||||||d| } |s"| ddr"d| d<d| d<| S)N)rrsr`rqrrcache_positionis_first_iterationruTrrrq)r*prepare_inputs_for_generationget) r9r_rrsr`rqrrrrr[ model_inputsr:r%r&r_s"  z-FuyuForCausalLM.prepare_inputs_for_generation) NNNNNNNNNNNNr)NNNNNNF)rrrr_tied_weights_keysrr+r>rBr r rQrrrrrrrrrrrr%r%r:r&rsz     cr)rrr()__doc__rQr cache_utilsr generationrmodeling_outputsrrmodeling_utilsrmodels.auto.modeling_autor processing_utilsr utilsr r r rrconfiguration_fuyur get_loggerrloggerrr(r__all__r%r%r%r&s2          2