o ei3@sddlZddlmZddlmZmZmZmZmZddlm Z ddl m Z ddl m Z ddlmZdd lmZmZmZdd lmZd d lmZeeZGd ddeZGdddeZGdddejZGdddeZGdddeZ GdddeZ!gdZ"dS)N)nn)LlavaCausalLMOutputWithPastLlavaForConditionalGeneration LlavaModelLlavaModelOutputWithPastLlavaPreTrainedModel)ACT2FN)Cache)BaseModelOutputWithPooling)Unpack)TransformersKwargsauto_docstringlogging)can_return_tuple)VipLlavaConfigc@ eZdZdS)VipLlavaModelOutputWithPastN__name__ __module__ __qualname__rrk/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/vipllava/modular_vipllava.pyr&rc@r)VipLlavaCausalLMOutputWithPastNrrrrrr*rrcs*eZdZdeffdd ZddZZS)VipLlavaMultiModalProjectorconfigcstt|jtr dnt|j}tj||jj |j d|_ tj ||jj |j j dd|_t|j|_tj |j j |j j dd|_dS)Nr)epsT)bias)super__init__ isinstancevision_feature_layersintlenr LayerNorm vision_config hidden_sizeprojector_layernorm_epsprojector_layernormLinear text_configlinear_1r projector_hidden_actactlinear_2)selfrnum_feature_layers __class__rrr"/s   z$VipLlavaMultiModalProjector.__init__cCs,||}||}||}||}|SN)r+r.r0r1)r2 hidden_statesrrrforward>s    z#VipLlavaMultiModalProjector.forward)rrrrr"r8 __classcell__rrr4rr.src@r)VipLlavaPreTrainedModelNrrrrrr:Frr:c@seZdZeedd  ddejdeeeBdBde dBde e de e Bf d d Ze            dd ejdBdejdBd ejdBd ejdBdedBdejdBdeeeBdBde dBde dBde dBde dBdejdBde eBfddZdS) VipLlavaModelzWObtains image last hidden states from the vision tower and apply multimodal projection.) custom_introN pixel_valuesr$output_hidden_stateskwargsreturnc s|dur|n|jj}|j|fddd|t|tr)j|ddddf}nfdd|D}tj|dd}||}|_ S) \ pixel_values (`torch.FloatTensor]` of shape `(batch_size, channels, height, width)`): The tensors corresponding to the input images. vision_feature_layers (`Union[int, list[int]]`, *optional*): The vision feature layer, or the list of indexes of the layers to select the vision feature. NT)r> return_dictrcs&g|]}j|ddddfqS)Nr)r7).0index image_outputsrr ms&z4VipLlavaModel.get_image_features..)dim) rr$ vision_towerr#r%r7torchcatmulti_modal_projector pooler_output)r2r=r$r>r?image_featuresrrErget_image_featuresKs"  z VipLlavaModel.get_image_features input_idsattention_mask position_idspast_key_values inputs_embeds use_cacheoutput_attentionsrBcache_positionc  Ks| dur| n|jj} | dur| n|jj} | dur| n|jj} |dur$|n|jj}|du|duAr4td|dur>||}|dura|j||ddj}| |j |j }|j |||d}| ||}|jd|||||| | d| d | }t|j|j|j|j|dur|ndd}| r|S|S) z vision_feature_layers (`Union[int, list[int]]`, *optional*): The vision feature layer, or the list of indexes of the layers to select the vision feature. Nz:You must specify exactly one of input_ids or inputs_embedsT)r=r$rB)rUrO) rRrSrTrUrVrWr>rBrX)last_hidden_staterTr7 attentionsimage_hidden_statesr)rrWr>use_return_dictr$ ValueErrorget_input_embeddingsrPrNtodevicedtypeget_placeholder_maskmasked_scatterlanguage_modelrrYrTr7rZto_tuple)r2rQr=rRrSrTrUr$rVrWr>rBrX lm_kwargsrOspecial_image_maskoutputsoutputrrrr8tsT    zVipLlavaModel.forward)NN) NNNNNNNNNNNN)rrrrrrK FloatTensorr%listboolr r tupler rP LongTensorTensorr rr8rrrrr;Jst%     r;c @seZdZe ddejdeeeBdBdee de e BfddZ              dd ej dBdejdBd ejdBd ej dBd edBd ejdBdeeeBdBdej dBdedBdedBdedBdedBdej dBdeejBde eBfddZdS) VipLlavaForConditionalGenerationNr=r$r?r@cKs|jjd||d|S)rA)r=r$Nr)modelrP)r2r=r$r?rrrrPs z3VipLlavaForConditionalGeneration.get_image_featuresrrQrRrSrTrUlabelsrVrWr>rBrXlogits_to_keepcKs| dur| n|jj} | dur| n|jj} | dur| n|jj} |dur$|n|jj}|jd||||||| || | d| d |}|d}t|trLt| dn|}| |dd|ddf}d}|durm|j |||jj j d}t |||j|j|j|jdS)a vision_feature_layers (`Union[int, list[int]]`, *optional*): The vision feature layer, or the list of indexes of the layers to select the vision feature. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> import torch >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> from transformers import AutoProcessor, VipLlavaForConditionalGeneration >>> model = VipLlavaForConditionalGeneration.from_pretrained("llava-hf/vip-llava-7b-hf", device_map="auto", dtype=torch.float16) >>> processor = AutoProcessor.from_pretrained("llava-hf/vip-llava-7b-hf") >>> prompt = "A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions.###Human: \n{}###Assistant:" >>> question = "Can you please describe this image?" >>> prompt = prompt.format(question) >>> url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/compel-neg.png" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> inputs = processor(text=text, images=image, return_tensors="pt").to(0, torch.float16) >>> # Generate >>> generate_ids = model.generate(**inputs, max_new_tokens=20) >>> processor.decode(generate_ids[0][len(inputs["input_ids"][0]):], skip_special_tokens=True) The image features a brown and white cat sitting on a green surface, with a red ball in its ```NT) rQr=rRrSrTrUrVr$rWr>rBrXr)logitsrr vocab_size)lossrtrTr7rZr[r)rrWr>r\r$rqr#r%slicelm_head loss_functionr-rurrTr7rZr[)r2rQr=rRrSrTrUr$rrrVrWr>rBrXrsrfrhr7 slice_indicesrtrvrrrr8sH6 z(VipLlavaForConditionalGeneration.forwardr6)NNNNNNNNNNNNNr)rrrrrKrjr%rkr r rmr rPrnror rlrr8rrrrrpsr      rp)r;rpr:)#rKr(transformers.models.llava.modeling_llavarrrrr activationsr cache_utilsr modeling_outputsr processing_utilsr utilsr rr utils.genericrconfiguration_vipllavar get_loggerrloggerrrModulerr:r;rp__all__rrrrs$        p u