o eiR@sLddlmZddlZddlmZddlmZddlmZddlm Z ddl m Z m Z m Z dd lmZdd lmZdd lmZmZmZdd lmZd dlmZddlmZeeddGddde ZeeddGddde ZGdddejZeGdddeZ eddGddde Z!eddGd d!d!e e Z"gd"Z#dS)#) dataclassN)nn)ACT2FN)Cache)GenerationMixin)BaseModelOutputWithPastBaseModelOutputWithPooling ModelOutput)PreTrainedModel)Unpack)TransformersKwargsauto_docstringtorch_compilable_check)can_return_tuple) AutoModel)VipLlavaConfigzM Base class for VipLlava outputs, with hidden states and attentions.  custom_introc@s$eZdZUdZdZejdBed<dS)VipLlavaModelOutputWithPasta past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder and after projecting the last hidden state. Nimage_hidden_states)__name__ __module__ __qualname____doc__rtorch FloatTensor__annotations__r r l/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/vipllava/modeling_vipllava.pyr&s  rzT Base class for VipLlava causal language model (or autoregressive) outputs. c@seZdZUdZdZejdBed<dZejdBed<dZ e dBed<dZ e ejdBed<dZ e ejdBed<dZejdBed<dS) VipLlavaCausalLMOutputWithPasta4 loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder and after projecting the last hidden state. Nlosslogitspast_key_values hidden_states attentionsr)rrrrr#rrrr$r%rr&tupler'rr r r r!r";s r"cs*eZdZdeffdd ZddZZS)VipLlavaMultiModalProjectorconfigcstt|jtr dnt|j}tj||jj |j d|_ tj ||jj |j j dd|_t|j|_tj |j j |j j dd|_dS)Nr)epsTbias)super__init__ isinstancevision_feature_layersintlenr LayerNorm vision_config hidden_sizeprojector_layernorm_epsprojector_layernormLinear text_configlinear_1rprojector_hidden_actactlinear_2)selfr*num_feature_layers __class__r r!r/Zs   z$VipLlavaMultiModalProjector.__init__cCs,||}||}||}||}|SN)r8r;r=r>)r?r&r r r!forwardis    z#VipLlavaMultiModalProjector.forward)rrrrr/rD __classcell__r r rAr!r)Ysr)c@s:eZdZUeed<dZdZdZdZdZ dZ dZ dZ dZ dS)VipLlavaPreTrainedModelr*model)imagetextTr%N)rrrrrbase_model_prefixinput_modalitiessupports_gradient_checkpointing_skip_keys_device_placement_supports_flash_attn_supports_sdpa_can_compile_fullgraph_supports_flex_attn_supports_attention_backendr r r r!rFqs rFzx The VipLlava model which consists of a vision backbone and a language model, without a language modeling head. csFeZdZddiZdeffdd ZddZdd Zee d d  d"d e j de e e Bd Bded BdeedeeBf ddZde jde j de j fddZe d#de jd Bd e j d Bde jd Bde jd Bded Bde j d Bde e e Bd Bded Bded Bded Bded Bde jd BdeeBfd d!ZZS)$ VipLlavaModel^language_model.modellanguage_modelr*cs>t|t|j|_t||_t|j|_ | dSrC) r.r/r from_configr5 vision_towerr)multi_modal_projectorr:rU post_initr?r*rAr r!r/s   zVipLlavaModel.__init__cC |jSrC)rUget_input_embeddingsr?r r r!r\ z"VipLlavaModel.get_input_embeddingscC|j|dSrC)rUset_input_embeddingsr?valuer r r!r`z"VipLlavaModel.set_input_embeddingszWObtains image last hidden states from the vision tower and apply multimodal projection.rN pixel_valuesr1output_hidden_stateskwargsreturnc s|dur|n|jj}|j|fddd|t|tr)j|ddddf}nfdd|D}tj|dd}||}|_ S) \ pixel_values (`torch.FloatTensor]` of shape `(batch_size, channels, height, width)`): The tensors corresponding to the input images. vision_feature_layers (`Union[int, list[int]]`, *optional*): The vision feature layer, or the list of indexes of the layers to select the vision feature. NT)re return_dictrcs&g|]}j|ddddfqS)Nr)r&).0index image_outputsr r! s&z4VipLlavaModel.get_image_features..)dim) r*r1rWr0r2r&rcatrX pooler_output)r?rdr1rerfimage_featuresr rlr!get_image_featuress"  z VipLlavaModel.get_image_features input_ids inputs_embedsrscCs|dur||tj|jjtj|jdk}|d}n||jjk}|}|j d|j d}| d | |j}t |||kd|d||S)z Obtains multimodal placeholder mask from `input_ids` or `inputs_embeds`, and checks that the placeholder token count is equal to the length of multimodal features. If the lengths are different, an error is raised. N)dtypedevicerorrz6Image features and image tokens do not match, tokens: z , features: )r\rtensorr*image_token_idlongrxallsumshape unsqueeze expand_astornumel)r?rurvrsspecial_image_maskn_image_tokensn_image_featuresr r r!get_placeholder_masks  z"VipLlavaModel.get_placeholder_maskattention_mask position_idsr% use_cacheoutput_attentionsricache_positionc  Ks| dur| n|jj} | dur| n|jj} | dur| n|jj} |dur$|n|jj}|du|duAr4td|dur>||}|dura|j||ddj}| |j |j }|j |||d}| ||}|jd|||||| | d| d | }t|j|j|j|j|dur|ndd}| r|S|S) z vision_feature_layers (`Union[int, list[int]]`, *optional*): The vision feature layer, or the list of indexes of the layers to select the vision feature. Nz:You must specify exactly one of input_ids or inputs_embedsT)rdr1ri)rvrs) rrr%rvrrrerir)last_hidden_stater%r&r'rr )r*rreuse_return_dictr1 ValueErrorr\rtrrrrxrwrmasked_scatterrUrrr%r&r'to_tuple)r?rurdrrr%rvr1rrrerir lm_kwargsrsroutputsoutputr r r!rDsT    zVipLlavaModel.forward)NN) NNNNNNNNNNNN)rrr_checkpoint_conversion_mappingrr/r\r`rrrrr2listboolr r r(r rt LongTensorrTensorrrrDrEr r rAr!rSs%      rSzV The VIPLLAVA model which consists of a vision backbone and a language model. c"sleZdZdddddZddiZdeffd d Zd d Zd dZde j fddZ e d*de jdeeeBdBdeedeeBfddZee              d+de jdBde jdBde jdBde jdBdedBde jdBdeeeBdBde jdBdedBd edBd!edBd"edBd#e jdBd$ee jBdeeBfd%d&Z       'd,fd(d) ZZS)- VipLlavaForConditionalGenerationzmodel.language_modelzmodel.vision_towerzmodel.multi_modal_projectorlm_head)rTz ^vision_towerz^multi_modal_projectorz^language_model.lm_headzlm_head.weightz(model.language_model.embed_tokens.weightr*cs<t|t||_tj|jj|jjdd|_ | dS)NFr,) r.r/rSrGrr9r:r6 vocab_sizerrYrZrAr r!r/.s   z)VipLlavaForConditionalGeneration.__init__cCr[rC)rGr\r]r r r!r\4r^z5VipLlavaForConditionalGeneration.get_input_embeddingscCr_rC)rGr`rar r r!r`7rcz5VipLlavaForConditionalGeneration.set_input_embeddingsrgcCs|jSrC)rr]r r r!get_output_embeddings:sz6VipLlavaForConditionalGeneration.get_output_embeddingsNrdr1rfcKs|jjd||d|S)rh)rdr1Nr )rGrt)r?rdr1rfr r r!rt=s z3VipLlavaForConditionalGeneration.get_image_featuresrrurrr%rvlabelsrrrerirlogits_to_keepcKs| dur| n|jj} | dur| n|jj} | dur| n|jj} |dur$|n|jj}|jd||||||| || | d| d |}|d}t|trLt| dn|}| |dd|ddf}d}|durm|j |||jj j d}t |||j|j|j|jdS)a vision_feature_layers (`Union[int, list[int]]`, *optional*): The vision feature layer, or the list of indexes of the layers to select the vision feature. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> import torch >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> from transformers import AutoProcessor, VipLlavaForConditionalGeneration >>> model = VipLlavaForConditionalGeneration.from_pretrained("llava-hf/vip-llava-7b-hf", device_map="auto", dtype=torch.float16) >>> processor = AutoProcessor.from_pretrained("llava-hf/vip-llava-7b-hf") >>> prompt = "A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions.###Human: \n{}###Assistant:" >>> question = "Can you please describe this image?" >>> prompt = prompt.format(question) >>> url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/compel-neg.png" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> inputs = processor(text=text, images=image, return_tensors="pt").to(0, torch.float16) >>> # Generate >>> generate_ids = model.generate(**inputs, max_new_tokens=20) >>> processor.decode(generate_ids[0][len(inputs["input_ids"][0]):], skip_special_tokens=True) The image features a brown and white cat sitting on a green surface, with a red ball in its ```NT) rurdrrr%rvrr1rrerirr)r$rr)r#r$r%r&r'rr )r*rrerr1rGr0r2slicer loss_functionr:rr"r%r&r'r)r?rurdrrr%rvr1rrrrerirrrrr& slice_indicesr$r#r r r!rDOsH8 z(VipLlavaForConditionalGeneration.forwardFc s>tj|f||||||d| } |s| dds|| d<| S)N)r%rvrrris_first_iterationrTrd)r.prepare_inputs_for_generationget) r?rur%rvrdrrrrrf model_inputsrAr r!rs z>VipLlavaForConditionalGeneration.prepare_inputs_for_generationrC)NNNNNNNNNNNNNr)NNNNNNF)rrrr_tied_weights_keysrr/r\r`rModulerrrrr2rr r r(r rtrrrrrr"rDrrEr r rAr!r s      dr)rSrrF)$ dataclassesrrr activationsr cache_utilsr generationrmodeling_outputsrr r modeling_utilsr processing_utilsr utilsr rr utils.genericrautorconfiguration_vipllavarrr"rr)rFrSr__all__r r r r!sJ           1