o ij@sbdZddlmZddlmZmZddlZddlmZddlm Z m Z ddl m Z dd l mZdd lmZdd lmZdd lmZdd lmZmZmZmZmZddlmZddlmZee Z!eeddGdddeZ"eeddGdddeZ#Gdddej$Z%eGdddeZ&eddGddde&Z'eddGd d!d!e&e Z(gd"Z)dS)#zPyTorch PaliGemmamodel.) dataclass)OptionalUnionN)nn)Cache StaticCache)GenerationMixin)FlashAttentionKwargs)BaseModelOutputWithPast)PreTrainedModel)Unpack) ModelOutputTransformersKwargsauto_docstringcan_return_tuplelogging) AutoModel)PaliGemmaConfigzN Base class for Paligemma outputs, with hidden states and attentions. ) custom_introc@s$eZdZUdZdZeejed<dS)PaligemmaModelOutputWithPasta image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder and after projecting the last hidden state. Nimage_hidden_states) __name__ __module__ __qualname____doc__rrtorch FloatTensor__annotations__r!r!m/home/ubuntu/veenaModal/venv/lib/python3.10/site-packages/transformers/models/paligemma/modeling_paligemma.pyr+s rzU Base class for PaliGemma causal language model (or autoregressive) outputs. c@seZdZUdZdZeejed<dZ eejed<dZ ee ed<dZ ee ejed<dZee ejed<dZeejed<dS) PaliGemmaCausalLMOutputWithPasta8 loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.text_config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder after projecting last hidden state. Nlosslogitspast_key_values hidden_states attentionsr)rrrrr$rrrr r%r&rr'tupler(rr!r!r!r"r#;s r#cs*eZdZdeffdd ZddZZS)PaliGemmaMultiModalProjectorconfigcs(ttj|jj|jjdd|_dS)NTbias)super__init__rLinear vision_config hidden_sizeprojection_dimlinearselfr+ __class__r!r"r/Zs z%PaliGemmaMultiModalProjector.__init__cCs||}|SN)r4)r6image_featuresr'r!r!r"forward^s z$PaliGemmaMultiModalProjector.forward)rrrrr/r; __classcell__r!r!r7r"r*Ysr*c@sDeZdZUeed<dZdZdgZdZdZ dZ dZ dZ dZ ddZd S) PaliGemmaPreTrainedModelr+Tr*r&FcCsVt|jd|jj}t|tjr'|jjj d|d|j dur)|j j dSdSdS)Ninitializer_range)meanstd) getattrr+get_text_configr? isinstancerr0weightdatanormal_r-zero_)r6modulerBr!r!r" _init_weightsrs  z&PaliGemmaPreTrainedModel._init_weightsN)rrrrr base_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_can_compile_fullgraph_supports_flash_attn_supports_sdpa_supports_flex_attn_supports_attention_backendrKr!r!r!r"r=ds  r=z| The Base Paligemma model which consists of a vision backbone and a language model without language modeling head., c#sReZdZddiZdZdeffdd ZddZd d Zd d Z d dZ     d)de e fddZ dejfddZdejdejdejfddZee             d*de ejde ejde ejde ejde ede ejde ejde ejd e ejd!e e d"e e d#e e d$e e d%eed&eeeffd'd(ZZS)+PaliGemmaModelzlanguage_model.modellanguage_modelFr+cs~t|tj|jd|_t||_|jj |_ tj|jd}||_ |j j dur,|j j nd|_ |j jp7|j|_|dS)N)r+)r.r/r from_configr1 vision_towerr*multi_modal_projector text_config vocab_sizerVr+ pad_token_idrDdtypetext_config_dtype post_init)r6r+rVr7r!r"r/s    zPaliGemmaModel.__init__cC |jSr9)rVget_input_embeddingsr6r!r!r"rb z#PaliGemmaModel.get_input_embeddingscC|j|dSr9)rVset_input_embeddingsr6valuer!r!r"rfz#PaliGemmaModel.set_input_embeddingscCs ||_dSr9rVr6decoderr!r!r" set_decoderrdzPaliGemmaModel.set_decodercCs|jSr9rjrcr!r!r" get_decoderszPaliGemmaModel.get_decoderN is_trainingcCsl|jjjdkr|durd|vr|SdS|dur|n|j}t|t}t|jj }|dur.|}|j dd\} } |r>| } nt|tj rI|j dn|d| d} |dur]| dkr]|Stj| | f||j|jd} | dkr|rxtj| dd } n d| ddd| f<| tj| |jd |ddk9} | ddddddf| ddd} |dur4| } |j d} |r|durtd | ddddddd| f|ddddddf| jdkd| ddddddd| f<| ddddddd| f|ddddddf| j}|dk}| ddddddd| f||| ddddddd| f<| S) Nflash_attention_2r@rrWrr fill_valuer^devicediagonalrtz/Token type ids must be provided during training)r+r[_attn_implementationtrainingrErrfinfor_minshapeget_max_cache_shapeTensordimfullrttriuarangereshapeexpandclone ValueError masked_fillto)r6attention_masktoken_type_idsr&cache_position input_tensorrousing_static_cache min_dtypeinputs_lead_dimsequence_length target_length causal_mask mask_length padding_maskr!r!r"_update_causal_masksZ      $   $ @  z"PaliGemmaModel._update_causal_mask pixel_valuescCs0||}|j}||}||jjjd}|S)a Obtains image last hidden states from the vision tower and apply multimodal projection. Args: pixel_values (`torch.FloatTensor]` of shape `(batch_size, channels, height, width)`) The tensors corresponding to the input images. Returns: image_features (`torch.Tensor`): Image feature tensor of shape `(num_images, image_length, embed_dim)`). g?)rYlast_hidden_staterZr+r[r2)r6r image_outputsselected_image_featurer:r!r!r"get_image_featuress  z!PaliGemmaModel.get_image_features input_ids inputs_embedsr:cCs|dur||tj|jjtj|jdk}|d}n||jjk}|}| d | |j}|j d|j d}|| | krPtd|d||S)z Obtains multimodal placeholder mask from `input_ids` or `inputs_embeds`, and checks that the placeholder token count is equal to the length of multimodal features. If the lengths are different, an error is raised. N)r^rtrWrrz6Image features and image tokens do not match: tokens: z , features )rbrtensorr+image_token_idlongrtallsum unsqueeze expand_asrr|numelr)r6rrr:special_image_maskn_image_tokensn_image_featuresr!r!r"get_placeholder_masks  z#PaliGemmaModel.get_placeholder_maskr position_idsr&rrlabels use_cacheoutput_attentionsoutput_hidden_states return_dictkwargsreturnc Ks|du|duAr td| dur| n|jj} | dur| n|jj} | dur&| n|jj} |duo1| du}|durL|jj|jkrL||jjk}|}d||<n|}|durX||}|durt|durd| nd}t j |||j d|j d}|dur|dd}|dur||}||j |j}|j|||d}|||}|||||||}|jd ||||| | | d|d |}t|j|j|j|j|dur|d Sdd S) i labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.text_config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.text_config.vocab_size]`. Example: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, PaliGemmaForConditionalGeneration >>> model = PaliGemmaForConditionalGeneration.from_pretrained("google/paligemma2-3b-mix-224") >>> processor = AutoProcessor.from_pretrained("google/paligemma2-3b-mix-224") >>> prompt = "Where is the cat standing?" >>> url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, text=prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(**inputs,) >>> processor.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Where is the cat standing?\nsnow" ```Nz:You must specify exactly one of input_ids or inputs_embedsrrrw)rr:T) rrr&rrrrrr)rr&r'r(rr!)rr+rruse_return_dictrr\rrbget_seq_lengthrrr|rtrrrr^rmasked_scatterrrVrrr&r'r()r6rrrrr&rrrrrrrrrror llm_input_idspast_seen_tokensr:routputsr!r!r"r;sj/         zPaliGemmaModel.forward)NNNNN) NNNNNNNNNNNNN)rrr_checkpoint_conversion_mappingaccepts_loss_kwargsrr/rbrfrmrnrboolrrrr LongTensorrrrr~rr r rr)rr;r<r!r!r7r"rU}s E       rUc%seZdZdddddZdgZdeffdd Zd d Zd d ZddZ ddZ ddZ e ddZ e ddZe ddZee              d7deejdeejdeejdeejd eed!eejd"eejd#eejd$eejd%eed&eed'eed(eed)eeejfd*eed+eeeff d,d-Z        .  d8fd/d0 Z e!dejd1ed2ed3ej"d"ejd4ef d5d6Z#Z$S)9!PaliGemmaForConditionalGenerationzmodel.language_modelzmodel.vision_towerzmodel.multi_modal_projectorlm_head)z^language_model.modelz ^vision_towerz^multi_modal_projectorz^language_model.lm_headzlm_head.weightr+cs<t|t||_tj|jj|jjdd|_ | dS)NFr,) r.r/rUmodelrr0r[r2r\rr`r5r7r!r"r/s   z*PaliGemmaForConditionalGeneration.__init__cCrar9)rrbrcr!r!r"rbrdz6PaliGemmaForConditionalGeneration.get_input_embeddingscCrer9)rrfrgr!r!r"rfriz6PaliGemmaForConditionalGeneration.set_input_embeddingscCrer9)rrmrkr!r!r"rmriz-PaliGemmaForConditionalGeneration.set_decodercCrar9)rrnrcr!r!r"rnrdz-PaliGemmaForConditionalGeneration.get_decodercCs |j|Sr9)rr)r6rr!r!r"rs z4PaliGemmaForConditionalGeneration.get_image_featurescC|jjSr9)rrVrcr!r!r"rVz0PaliGemmaForConditionalGeneration.language_modelcCrr9)rrYrcr!r!r"rYrz.PaliGemmaForConditionalGeneration.vision_towercCrr9)rrZrcr!r!r"rZrz7PaliGemmaForConditionalGeneration.multi_modal_projectorNrrrrrr&rrrrrrrrlogits_to_keeprrcKs| dur| n|jj} | dur| n|jj} | dur| n|jj} |jd|||||||| | | | d|d |}|d}t|trCt| dn|}||dd|ddf}d}| durh|j d|| |jj j d|}t |||j |j|j|jdS)rNT) rrrrrr&rrrrrrrr)r%rr\)r$r%r&r'r(rr!)r+rrrrrEintslicer loss_functionr[r\r#r&r'r(r)r6rrrrr&rrrrrrrrrrrr' slice_indicesr%r$r!r!r"r;sN/ z)PaliGemmaForConditionalGeneration.forwardTc  stj|f|||||| | |d| } | ddur"| dd7<|ddkr,|| d<|duo3| du}t|to=t|j}|ddkr]|r]|durL|n|}|j||||||}|| d<| S)N)r&rrrrrrrrrrrr) r.prepare_inputs_for_generationgetrErany is_slidingrr)r6rr&rrrrrrrrrr model_inputsrois_static_hybrid_cacherrr7r!r"r s6   z?PaliGemmaForConditionalGeneration.prepare_inputs_for_generationrrr^ batch_sizec KsD|dur|dkr|}|St|j}tj||f|||jd}|dkr+tj|dd}|tj||jd|ddk9}|ddddddf |ddd}|dur| }|j d} |ddddddd| f|ddddddf |j} | dk} |ddddddd| f | ||ddddddd| f<|S) aM Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)`, or if the input `attention_mask` is already 4D, do nothing. Args: attention_mask (`torch.Tensor`): A 2D attention mask of shape `(batch_size, key_value_length)` or a 4D attention mask of shape `(batch_size, 1, query_length, key_value_length)`. sequence_length (`int`): The sequence length being processed. target_length (`int`): The target length: when generating with static cache, the mask should be as long as the static cache, to account for the 0 padding, the part of the cache that is not filled yet. dtype (`torch.dtype`): The dtype to use for the 4D attention mask. cache_position (`torch.Tensor`): Indices depicting the position of the input sequence tokens in the sequence. batch_size (`torch.Tensor`): Batch size. NrqrrrrurwrWr)rrrzr{rrtrrrrrr|rr) rrrr^rrrrrrrr!r!r"5_prepare_4d_causal_attention_mask_with_cache_position8s,  $ 6  zWPaliGemmaForConditionalGeneration._prepare_4d_causal_attention_mask_with_cache_position)NNNNNNNNNNNNNr) NNNNNNNTNN)%rrrr_tied_weights_keysrr/rbrfrmrnrpropertyrVrYrZrrrrrrr~rrrrr rr)r#r;r staticmethodr^rr<r!r!r7r"rs          [/r)rr=rU)*r dataclassesrtypingrrrr cache_utilsrr generationr modeling_flash_attention_utilsr modeling_outputsr modeling_utilsr processing_utilsr utilsrrrrrautorconfiguration_paligemmar get_loggerrloggerrr#Moduler*r=rUr__all__r!r!r!r"sN            ~ n