o wi}l@sdZddlmZddlmZmZddlZddlZddlmZddl m Z m Z m Z ddl mZdd lmZdd lmZdd lmZdd lmZdd lmZmZmZmZmZmZddlmZddl m!Z!e"e#Z$eeddGdddeZ%eeddGdddeZ&Gdddej'Z(eGdddeZ)eddGddde)Z*Gd d!d!eeZ+ed"dGd#d$d$e)eZ,gd%Z-dS)&zPyTorch PaliGemmamodel.) dataclass)OptionalUnionN)nn)Cache HybridCache StaticCache)GenerationMixin)FlashAttentionKwargs)BaseModelOutputWithPast)PreTrainedModel)Unpack) LossKwargs ModelOutputauto_docstringcan_return_tupleis_torchdynamo_compilinglogging) AutoModel)PaliGemmaConfigzN Base class for Paligemma outputs, with hidden states and attentions. ) custom_introc@s$eZdZUdZdZeejed<dS)PaligemmaModelOutputWithPasta past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder and after projecting the last hidden state. Nimage_hidden_states) __name__ __module__ __qualname____doc__rrtorch FloatTensor__annotations__r#r#m/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/transformers/models/paligemma/modeling_paligemma.pyr&s  rzU Base class for PaliGemma causal language model (or autoregressive) outputs. c@seZdZUdZdZeejed<dZ eejed<dZ ee e eje fed<dZeeejed<dZeeejed<dZeejed<dS) PaliGemmaCausalLMOutputWithPasta{ loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.text_config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder after projecting last hidden state. Nlosslogitspast_key_values hidden_states attentionsr)rrrrr&rr r!r"r'r(rlistrr)tupler*rr#r#r#r$r%<s r%cs*eZdZdeffdd ZddZZS)PaliGemmaMultiModalProjectorconfigcs(ttj|jj|jjdd|_dS)NTbias)super__init__rLinear vision_config hidden_sizeprojection_dimlinearselfr. __class__r#r$r2\s z%PaliGemmaMultiModalProjector.__init__cCs||}|SN)r7)r9image_featuresr)r#r#r$forward`s z$PaliGemmaMultiModalProjector.forward)rrrrr2r> __classcell__r#r#r:r$r-[sr-c@sFeZdZeZdZdZdgZdZdZ dZ dZ dZ dZ dZdZddZdS)PaliGemmaPreTrainedModelTr-r(cCsVt|jd|jj}t|tjr'|jjj d|d|j dur)|j j dSdSdS)Ninitializer_range)meanstd) getattrr.get_text_configrB isinstancerr3weightdatanormal_r0zero_)r9modulerEr#r#r$ _init_weightsus  z&PaliGemmaPreTrainedModel._init_weightsN)rrrr config_classbase_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_cache_class_supports_quantized_cache_supports_static_cache_supports_flash_attn_2_supports_sdpa_supports_flex_attn_supports_attention_backendrNr#r#r#r$r@fs r@z{ The Base Paligemma model which consists of a vision backbone and a language model withou language modeling head., c#s<eZdZddiZdZdeffdd ZddZd d Zd d Z d dZ     d&de e fddZ dejfddZee             d'dejdejde ejde ejde eeejefde ejde ejde ejde ejde e de e d e e d!e e d"eed#eeeffd$d%ZZS)(PaliGemmaModelzlanguage_model.modellanguage_modelFr.csjt|tj|jd|_t||_|jj |_ tj|jd}||_ |j j dur,|j j nd|_ | dS)N)r.)r1r2r from_configr4 vision_towerr-multi_modal_projector text_config vocab_sizer\r. pad_token_id post_init)r9r.r\r:r#r$r2s    zPaliGemmaModel.__init__cC |jSr<)r\get_input_embeddingsr9r#r#r$rf z#PaliGemmaModel.get_input_embeddingscC|j|dSr<)r\set_input_embeddingsr9valuer#r#r$rjz#PaliGemmaModel.set_input_embeddingscC ||_dSr<r\r9decoderr#r#r$ set_decoderrhzPaliGemmaModel.set_decodercC|jSr<rorgr#r#r$ get_decoderzPaliGemmaModel.get_decoderN is_trainingcCs|jjjdkr|durd|vr|SdS|dur|n|j}t|t}t|jj }|dur.|}|j dd\} } |r>| } nt|t rH| } nt|tj rS|j dn|d| d} |durg|dkrg|Stj| | f||j|jd} | dkr|rtj| dd } n d| ddd| f<| tj| |jd |ddk9} | ddddddf| ddd} |dur>| } |j d} |r|durtd | ddddddd| f|ddddddf| jdkd| ddddddd| f<| ddddddd| f|ddddddf| j}|dk}| ddddddd| f||| ddddddd| f<| S) Nflash_attention_2rCrr]rr fill_valuedtypedevicediagonalr|z/Token type ids must be provided during training)r.ra_attn_implementationtrainingrHr r finfor{minshapeget_max_cache_shaperTensordimfullr|triuarangereshapeexpandclone ValueError masked_fillto)r9attention_masktoken_type_idsr(cache_position input_tensorrvusing_static_cache min_dtypeinputs_lead_dimsequence_length target_length causal_mask mask_length padding_maskr#r#r$_update_causal_masksX        $   $ @  z"PaliGemmaModel._update_causal_mask pixel_valuescCs0||}|j}||}||jjjd}|S)a Obtains image last hidden states from the vision tower and apply multimodal projection. Args: pixel_values (`torch.FloatTensor]` of shape `(batch_size, channels, height, width)`) The tensors corresponding to the input images. Returns: image_features (`torch.Tensor`): Image feature tensor of shape `(num_images, image_length, embed_dim)`). g?)r_last_hidden_stater`r.rar5)r9r image_outputsselected_image_featurer=r#r#r$get_image_featuress  z!PaliGemmaModel.get_image_features input_idsr position_idsr(rr inputs_embedslabels use_cacheoutput_attentionsoutput_hidden_states return_dictkwargsreturnc Ks:|du|duAr td| dur| n|jj} | dur| n|jj} | dur&| n|jj} |duo1| du}|durL|jj|jkrL||jjk}|}d||<n|}|durX||}|durt|durd| nd}t j |||j d|j d}|dur|dd}|dur||}|dur||t j|jjt j|j dk}n||jjkd}|||j }ts|||kr|jddjddd}td |d |j d|j dd ||j |j}|||}|||||||}|jd||||| | | d |d |}t|j|j|j|j|dur|dSddS)i labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.text_config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.text_config.vocab_size]`. Example: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, PaliGemmaForConditionalGeneration >>> model = PaliGemmaForConditionalGeneration.from_pretrained("google/paligemma2-3b-mix-224") >>> processor = AutoProcessor.from_pretrained("google/paligemma2-3b-mix-224") >>> prompt = "Where is the cat standing?" >>> url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, text=prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(**inputs,) >>> processor.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Where is the cat standing?\nsnow" ```Nz:You must specify exactly one of input_ids or inputs_embedsrrr)r{r|r])rzVNumber of images does not match number of special image tokens in the input text. Got z image tokens in the text but z tokens from image embeddings.T) rrr(rrrrrr)rr(r)r*rr#) rr.rruse_return_dictimage_token_idrbrrfget_seq_lengthr rrr| unsqueezertensorlong expand_asrrnumelsumr{masked_scatterrr\rrr(r)r*)r9rrrrr(rrrrrrrrrrvspecial_image_mask llm_input_idspast_seen_tokensr=image_tokens_in_textroutputsr#r#r$r>s/         zPaliGemmaModel.forward)NNNNN) NNNNNNNNNNNNN)rrr_checkpoint_conversion_mappingaccepts_loss_kwargsrr2rfrjrrrtrboolrr r!rrr LongTensorrrr+rrr r,rr>r?r#r#r:r$r[s~  D      r[c@s eZdZdS)KwargsForCausalLMN)rrrr#r#r#r$rus rz| The Base Paligemma model which consists of a vision backbone and a language model without language modeling head., c%seZdZdddddZdgZdeffdd Zd d Zd d ZddZ ddZ ddZ ddZ ddZ eddZeddZeddZee              d;d ejd!ejd"eejd#eejd$eeeejefd%eejd&eejd'eejd(eejd)eed*eed+eed,eed-eeejfd.eed/ee e!ff d0d1Z"        2  d<fd3d4 Z#e$d"ejd5ed6ed7ej%d&ejd8ef d9d:Z&Z'S)=!PaliGemmaForConditionalGenerationzmodel.language_modelzmodel.vision_towerzmodel.multi_modal_projectorlm_head)z^language_model.modelz ^vision_towerz^multi_modal_projectorz^language_model.lm_headzlm_head.weightr.cs<t|t||_tj|jj|jjdd|_ | dS)NFr/) r1r2r[modelrr3rar5rbrrdr8r:r#r$r2s   z*PaliGemmaForConditionalGeneration.__init__cCrer<)rrfrgr#r#r$rfrhz6PaliGemmaForConditionalGeneration.get_input_embeddingscCrir<)rrjrkr#r#r$rjrmz6PaliGemmaForConditionalGeneration.set_input_embeddingscCrsr<rrgr#r#r$get_output_embeddingsruz7PaliGemmaForConditionalGeneration.get_output_embeddingscCrnr<r)r9new_embeddingsr#r#r$set_output_embeddingsrhz7PaliGemmaForConditionalGeneration.set_output_embeddingscCrir<)rrrrpr#r#r$rrrmz-PaliGemmaForConditionalGeneration.set_decodercCrer<)rrtrgr#r#r$rtrhz-PaliGemmaForConditionalGeneration.get_decodercCs |j|Sr<)rr)r9rr#r#r$rs z4PaliGemmaForConditionalGeneration.get_image_featurescC|jjSr<)rr\rgr#r#r$r\z0PaliGemmaForConditionalGeneration.language_modelcCrr<)rr_rgr#r#r$r_rz.PaliGemmaForConditionalGeneration.vision_towercCrr<)rr`rgr#r#r$r`rz7PaliGemmaForConditionalGeneration.multi_modal_projectorNrrrrrr(rrrrrrrrlogits_to_keeprrcKs| dur| n|jj} | dur| n|jj} | dur| n|jj} |jd|||||||| | | | d|d |}|d}t|trCt| dn|}||dd|ddf}d}| durh|j d|| |jj j d|}t |||j |j|j|jdS)rNT) rrrrrr(rrrrrrrr)r'rrb)r&r'r(r)r*rr#)r.rrrrrHintslicer loss_functionrarbr%r(r)r*r)r9rrrrr(rrrrrrrrrrrr) slice_indicesr'r&r#r#r$r>sN/ z)PaliGemmaForConditionalGeneration.forwardTc  stj|f|||||| | |d| } | ddur"| dd7<|ddkr,|| d<|duo3| du}|ddkrVt|trV|durE|n|}|j||||||}|| d<| S)N)r(rrrrrrrrrrrr)r1prepare_inputs_for_generationgetrHrrr)r9rr(rrrrrrrrrr model_inputsrvrrr:r#r$rs4   z?PaliGemmaForConditionalGeneration.prepare_inputs_for_generationrrr{ batch_sizec KsD|dur|dkr|}|St|j}tj||f|||jd}|dkr+tj|dd}|tj||jd|ddk9}|ddddddf |ddd}|dur| }|j d} |ddddddd| f|ddddddf |j} | dk} |ddddddd| f | ||ddddddd| f<|S) aM Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)`, or if the input `attention_mask` is already 4D, do nothing. Args: attention_mask (`torch.Tensor`): A 2D attention mask of shape `(batch_size, key_value_length)` or a 4D attention mask of shape `(batch_size, 1, query_length, key_value_length)`. sequence_length (`int`): The sequence length being processed. target_length (`int`): The target length: when generating with static cache, the mask should be as long as the static cache, to account for the 0 padding, the part of the cache that is not filled yet. dtype (`torch.dtype`): The dtype to use for the 4D attention mask. cache_position (`torch.Tensor`): Indices depicting the position of the input sequence tokens in the sequence. batch_size (`torch.Tensor`): Batch size. Nrxryrr}rr]r)rr rrrr|rrrrrrrr) rrrr{rrrrrrrr#r#r$5_prepare_4d_causal_attention_mask_with_cache_position6s,  $ 6  zWPaliGemmaForConditionalGeneration._prepare_4d_causal_attention_mask_with_cache_position)NNNNNNNNNNNNNr) NNNNNNNTNN)(rrrr_tied_weights_keysrr2rfrjrrrrrtrpropertyr\r_r`rrr rr!rrrr+rrrrrr,r%r>r staticmethodr{rr?r#r#r:r$rxs          [.r)rr@r[).r dataclassesrtypingrrr torch.utils.checkpointr cache_utilsrrr generationr modeling_flash_attention_utilsr modeling_outputsr modeling_utilsr processing_utilsrutilsrrrrrrautorconfiguration_paligemmar get_loggerrloggerrr%Moduler-r@r[rr__all__r#r#r#r$sR            q s