o eij@s"dZddlmZddlmZddlZddlmZddlmZddl m Z dd l m Z dd l mZdd lmZdd lmZmZdd lmZddlmZddlmZmZmZmZmZmZddlm Z ddl!m"Z"ddl#m$Z$e%e&Z'eeddGdddeZ(eeddGdddeZ)Gdddej*Z+dej,dBdej,dBd edBfd!d"Z-e d#d$d%d&   ' d:d(e d%ej,d)ej,dBd*ej,d+edBd,ej,dBdej,dBd-ej.dBd.e/dBd/e/dBd e0fd0d1Z1eGd2d3d3eZ2ed4dGd5d6d6e2Z3ed4dGd7d8d8e2e Z4gd9Z5dS);zPyTorch PaliGemmamodel.)Callable) dataclassN)nn)Cache)PreTrainedConfig)GenerationMixin)create_masks_for_generate)FlashAttentionKwargs)BaseModelOutputWithPastBaseModelOutputWithPooling)PreTrainedModel)Unpack) ModelOutputTransformersKwargsauto_docstringcan_return_tupleloggingtorch_compilable_check)deprecate_kwarg) AutoModel)PaliGemmaConfigzN Base class for Paligemma outputs, with hidden states and attentions.  custom_introc@s$eZdZUdZdZejdBed<dS)PaligemmaModelOutputWithPasta image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder and after projecting the last hidden state. Nimage_hidden_states)__name__ __module__ __qualname____doc__rtorch FloatTensor__annotations__r%r%n/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/paligemma/modeling_paligemma.pyr.s rzU Base class for PaliGemma causal language model (or autoregressive) outputs. c@seZdZUdZdZejdBed<dZejdBed<dZ e dBed<dZ e ejdBed<dZ e ejdBed<dZejdBed<dS) PaliGemmaCausalLMOutputWithPasta8 loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.text_config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder after projecting last hidden state. Nlosslogitspast_key_values hidden_states attentionsr)rrr r!r(r"r#r$r)r*rr+tupler,rr%r%r%r&r'>s r'cs*eZdZdeffdd ZddZZS)PaliGemmaMultiModalProjectorconfigcs(ttj|jj|jjdd|_dS)NTbias)super__init__rLinear vision_config hidden_sizeprojection_dimlinearselfr/ __class__r%r&r3]s z%PaliGemmaMultiModalProjector.__init__cCs||}|SN)r8)r:image_featuresr+r%r%r&forwardas z$PaliGemmaMultiModalProjector.forward)rrr rr3r? __classcell__r%r%r;r&r.\sr.token_type_idsimage_group_idsreturnc s4durdSdtdtdtdtdtf fdd }|S) z This function adds the correct offsets to the `q_idx` and `kv_idx` as the torch API can only accept lengths, not start and end indices. N batch_idxhead_idxq_idxkv_idxrCc st|jdk|d}t|jdk|d}||f}t|jdk|d}||f}t|jdk|d}||f}t|jdk|d}||f} t|jdk| d} |dk|dk@} || k} | | @S)Nrr)r"whereshape) rDrErFrG safe_q_idx safe_kv_idxtoken_type_ids_at_q_idxtoken_type_ids_at_kv_idximage_group_ids_at_q_idximage_group_ids_at_kv_idxis_image_blocksame_image_blockrBrAr%r& inner_maskss    z0token_type_ids_mask_function..inner_mask)intbool)rArBrTr%rSr&token_type_ids_mask_functiongs $rW input_embeds5.6.0 inputs_embedsversionnew_nameFr/attention_maskcache_positionr* position_ids pixel_values is_trainingis_first_iterationc Ks*|r |dur td||||||d} | r| n |dup$|j p$|du} | s-| ddsI|dur6d|}ntdt|dddddf}|dur| r|dk|j } t j j | d dd dddd f} | | @}tj |dd d}t| |t|d }t||j || d <tdi| S)a" Overwrites the base `create_masks_for_generate` with `token_type_ids` masking to create the causal mask mapping for all kinds of forward passes. Paligemma uses a bidirectional mask on the prompt tokens. Uses `pixel_values` as an optional input to disambiguate edge cases. Nz;`token_type_ids` is required as a model input when training)r/rZr^r_r*r` use_cacheTrzIt is a prefill stage but The `token_type_ids` is not provided. We recommend passing `token_type_ids` to the model to prevent bad attention masking.r)rr)valuerH)dimor_mask_functionr%) ValueErrorget_text_configis_initializedgetlogger warning_oncer" ones_liketodevicer functionalpadcumsumrUrI full_likerWr )r/rZr^r_r*r`rArarbrckwargs mask_kwargsis_imageis_previous_imagenew_image_startrBr%r%r&create_causal_mask_mappings<   "  rzc@s@eZdZUeed<dZdZdZdgZdZ dZ dZ dZ dZ dZdS) PaliGemmaPreTrainedModelr/model)imagetextTr.r*FN)rrr rr$base_model_prefixinput_modalitiessupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_can_compile_fullgraph_supports_flash_attn_supports_sdpa_supports_flex_attn_supports_attention_backendr%r%r%r&r{s r{z| The Base Paligemma model which consists of a vision backbone and a language model without language modeling head., c"s>eZdZddiZdZdeffdd ZddZd d Ze e d d d e j de edeeBfddZde jde j de j fddZe e             d$de jdBd e j dBde jdBde jdBdedBde jdBde jdBde j dBde jdBdedBdedBd edBd!edBde edeeBfd"d#ZZS)%PaliGemmaModelzlanguage_model.modellanguage_modelFr/csdt|tj|jd|_t||_|jj |_ tj|jd}||_ |j j p*|j |_|dS)N)r/)r2r3r from_configr5 vision_towerr.multi_modal_projector text_config vocab_sizerr/ridtypetext_config_dtype post_init)r:r/rr;r%r&r3s    zPaliGemmaModel.__init__cC |jSr=)rget_input_embeddingsr:r%r%r&r z#PaliGemmaModel.get_input_embeddingscC|j|dSr=)rset_input_embeddingsr:rer%r%r&rz#PaliGemmaModel.set_input_embeddingszWObtains image last hidden states from the vision tower and apply multimodal projection.rrarurCcKsB|j|fddi|}|j}||}||jjjd}||_|S)N return_dictTg?)rlast_hidden_staterr/rr6 pooler_output)r:raru image_outputsselected_image_featurer>r%r%r&get_image_featuress  z!PaliGemmaModel.get_image_features input_idsrZr>cCs|dur||tj|jjtj|jdk}|d}n||jjk}|}|j d|j d}| d | |j}t |||kd|d||S)z Obtains multimodal placeholder mask from `input_ids` or `inputs_embeds`, and checks that the placeholder token count is equal to the length of multimodal features. If the lengths are different, an error is raised. N)rrprHrrz6Image features and image tokens do not match, tokens: z , features: )rr"tensorr/image_token_idlongrpallsumrJ unsqueeze expand_asrornumel)r:rrZr>special_image_maskn_image_tokensn_image_featuresr%r%r&get_placeholder_masks  z#PaliGemmaModel.get_placeholder_maskNr^r`r*rAr_labelsrdoutput_attentionsoutput_hidden_statesrc Ks|du|duAr td| dur| n|jj} | dur| n|jj} | dur&| n|jj} |durD|jj|jkrD||jjk}|}d||<n|}|durP||}|durl|dur\| nd}t j |||j d|j d}|durw|dd}|dur|j|ddj}||j |j}|j|||d}|||}t|}tst|j||||||||jd }|jd ||||| | | d|d |}t|j|j|j|j|dur|d Sdd S)  labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.text_config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.text_config.vocab_size]`. Example: ```python >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> from transformers import AutoProcessor, PaliGemmaForConditionalGeneration >>> model = PaliGemmaForConditionalGeneration.from_pretrained("google/paligemma2-3b-mix-224") >>> processor = AutoProcessor.from_pretrained("google/paligemma2-3b-mix-224") >>> prompt = "Where is the cat standing?" >>> url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> inputs = processor(images=image, text=prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(**inputs,) >>> processor.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Where is the cat standing?\nsnow" ```Nz:You must specify exactly one of input_ids or inputs_embedsrr)rpT)r)rZr>)rb) r^r`r*rZrdrrrr_)rr*r+r,rr%)rhr/rruse_return_dictrrclonerget_seq_lengthr"arangerJrprrrrorrmasked_scatter isinstancedictrztrainingrrrr*r+r,)r:rrar^r`r*rAr_rZrrdrrrrur llm_input_idspast_seen_tokensr>causal_mask_mappingoutputsr%r%r&r?.sz1        zPaliGemmaModel.forward) NNNNNNNNNNNNN)rrr _checkpoint_conversion_mappingaccepts_loss_kwargsrr3rrrrr"r#rrr-r r LongTensorrTensorrrVr rr?r@r%r%r;r&rs        rc$seZdZdddddZddiZdeffd d Zd d Zd dZe de j de e fddZee              d/de jdBde j dBde jdBde jdBdedBde jdBde jdBde j dBde jdBdedBdedBdedBd edBd!ee jBde e d"eeBf d#d$Z        %   &d0fd'd( Zeed)d*dd+  &d1dede jde jdBde jdedBde jdBde jdBd,edBd"efd-d.ZZS)2!PaliGemmaForConditionalGenerationzmodel.language_modelzmodel.vision_towerzmodel.multi_modal_projectorlm_head)z^language_model.modelz ^vision_towerz^multi_modal_projectorz^language_model.lm_headzlm_head.weightz(model.language_model.embed_tokens.weightr/cs<t|t||_tj|jj|jjdd|_ | dS)NFr0) r2r3rr|rr4rr6rrrr9r;r%r&r3s   z*PaliGemmaForConditionalGeneration.__init__cCrr=)r|rrr%r%r&rrz6PaliGemmaForConditionalGeneration.get_input_embeddingscCrr=)r|rrr%r%r&rrz6PaliGemmaForConditionalGeneration.set_input_embeddingsrarucKs|jj|fi|Sr=)r|r)r:rarur%r%r&rsz4PaliGemmaForConditionalGeneration.get_image_featuresNrrr^r`r*rAr_rZrrdrrrlogits_to_keeprCcKs| dur| n|jj} | dur| n|jj} | dur| n|jj} |jd|||||||| | | | d|d |}|d}t|trCt| dn|}||dd|ddf}d}| durh|j d|| |jj j d|}t |||j |j|j|jdS)rNT) rrarAr^r`r*rZrdrrrrr_r)r)rr)r(r)r*r+r,rr%)r/rrrr|rrUslicer loss_functionrrr'r*r+r,r)r:rrar^r`r*rAr_rZrrdrrrrrurr+ slice_indicesr)r(r%r%r&r?sN1 z)PaliGemmaForConditionalGeneration.forwardTFc  sZtj|f|||||| | || d | }|ddur#|dd7<| s'| s+||d<|S)N) r*rZr^r`r_rdrrArcr`rra)r2prepare_inputs_for_generationrk)r:rr*rZr_r`rar^rArdrrrcru model_inputsr;r%r&r#s( z?PaliGemmaForConditionalGeneration.prepare_inputs_for_generationrXrYr[rcc Ks.t|||||||fd|idd|DS)NrccSsi|] \}}|dkr||qS)rar%).0kvr%r%r& fszOPaliGemmaForConditionalGeneration.create_masks_for_generate..)rzitems) r/rZr^r_r*r`rArcrur%r%r&r Os z;PaliGemmaForConditionalGeneration.create_masks_for_generate)NNNNNNNNNNNNNr) NNNNNNNTNNF)NF)rrr r_tied_weights_keysrr3rrrr"r#rrrrrrrrVrUr-r'r?r staticmethodrrrr r@r%r%r;r&rs     ],  r)rr{r)NNFN)6r!collections.abcr dataclassesrr"r cache_utilsrconfiguration_utilsr generationr masking_utilsr modeling_flash_attention_utilsr modeling_outputsr r modeling_utilsr processing_utilsrutilsrrrrrrutils.deprecationrautorconfiguration_paligemmar get_loggerrrlrr'Moduler.rrWr#rVrrzr{rr__all__r%r%r%r&s                  (   J< =