o i@sddlZddlmZddlmZmZmZddlZddl m Z ddl m Z ddl mZddlmZddlmZdd lmZdd lmZmZmZdd lmZmZdd lmZdd lmZm Z m!Z!m"Z"m#Z#ddl$m%Z%ddl&m'Z'ddl(m)Z)m*Z*edGddde j+Z, dFde j+dej-dej-dej-deej-de.de.fddZ/Gd d!d!e j+Z0ee!d"d#Gd$d%d%eZ1Gd&d'd'e j+Z2Gd(d)d)e j+Z3Gd*d+d+e j+Z4e j5e,d,Z6Gd-d.d.eZ7Gd/d0d0e j+Z8e!Gd1d2d2eZ9e!Gd3d4d4e9Z:e!Gd5d6d6eZ;Gd7d8d8e j+Zd>e;Z>ee!d?d#Gd@dAdAeZ?e!dBd#GdCdDdDe;eZ@gdEZAdS)GN) dataclass)CallableOptionalUnion)ACT2FN)Cache)GenerationMixin)use_kernel_forward_from_hub)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPastBaseModelOutputWithPooling)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack) ModelOutputTransformersKwargsauto_docstringcan_return_tuple torch_int)check_model_inputs) AutoModel)InternVLConfigInternVLVisionConfigRMSNormcs.eZdZdfdd ZddZddZZS) InternVLVisionRMSNormư>cs&ttt||_||_dS)zD InternVLVisionRMSNorm is equivalent to T5LayerNorm N)super__init__nn Parametertorchonesweightvariance_epsilon)self hidden_sizeeps __class__b/home/ubuntu/.local/lib/python3.10/site-packages/transformers/models/internvl/modeling_internvl.pyr!.s  zInternVLVisionRMSNorm.__init__cCsJ|j}|tj}|djddd}|t||j}|j||S)NrT)keepdim) dtypetor$float32powmeanrsqrtr'r&)r( hidden_states input_dtypevariancer-r-r.forward6s  zInternVLVisionRMSNorm.forwardcCst|jjd|jS)Nz, eps=)tupler&shaper'r(r-r-r. extra_repr=sz InternVLVisionRMSNorm.extra_repr)r)__name__ __module__ __qualname__r!r:r> __classcell__r-r-r+r.r,srmodulequerykeyvalueattention_maskscalingdropoutc Ks|}|} t||dd|} |dur+|ddddddd|jdf} | | } tjj| dd} tjj| ||jd} t| | } | dd } | | fS)Nrrr/dim)ptrainingr) r$matmul transposer<r" functionalsoftmaxrJrO contiguous) rDrErFrGrHrIrJkwargs key_states value_states attn_weights causal_mask attn_outputr-r-r.eager_attention_forwardAs & r[csLeZdZdZdeffdd Z d dejdeejde e fd d Z Z S) InternVLVisionAttentionz+Attention Class for InternVL Vision EncoderconfigcsBt||_|j|_|j|_|j|j|_|j|j|jkr-td|jd|jd|jd|_ |j |_ |j }|j }d|_ tj|j|j|j|jd|_tj|j|j|j|jd|_tj|j|j|j|jd|_t|j|j|_|dkrt|nt|_ |rt|jnt|_|rt|j|_dSt|_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).gFbiasr)r r!r]r) embed_dimnum_attention_heads num_headshead_dim ValueErrorscaleattention_dropoutprojection_dropout use_qk_norm is_causalr"Linearattention_biasq_projk_projv_projprojection_layerDropoutIdentityrq_normk_norm)r(r] proj_dropoutqk_normr+r-r.r!_s.   "z InternVLVisionAttention.__init__Nr7rHrUcKs|\}}}||}||}||} ||}||}||||j|j dd}||||j|j dd}| |||j|j dd} t } |j j dkrXt|j j } | |||| |f|jsddn|j|jdd|\} } | |||j} || } || } | | fS)NrreagerrCF)rJrIri)sizerlrmrnrrrsreshaperbrcrQviewr[r]_attn_implementationrrOrfrer`rorg)r(r7rHrU batch_sizeseq_len_ query_statesrVrWattention_interfacerZrXoutputr-r-r.r:{s:          zInternVLVisionAttention.forwardN) r?r@rA__doc__rr!r$Tensorrrrr:rBr-r-r+r.r\\sr\z7 Class for outputs of [`InternVLVisionModel`]. ) custom_introc@seZdZdZdS)$InternVLVisionModelOutputWithPoolingaF pooler_output (`torch.FloatTensor` of shape `(batch_size, hidden_size)`): Average of the last layer hidden states of the patch tokens (excluding the *[CLS]* token) if *config.use_mean_pooling* is set to True. If set to False, then the final hidden state of the *[CLS]* token will be returned. N)r?r@rArr-r-r-r.rsrcs6eZdZdZfddZdejdejfddZZS)InternVLVisionPatchEmbeddingsz This class turns `pixel_values` of shape `(batch_size, num_channels, height, width)` into the initial `hidden_states` (patch embeddings) of shape `(batch_size, seq_length, hidden_size)` to be consumed by a Transformer. cst|j|j}}|j|j}}|d|d|d|d}|d|d|d|df}||_||_||_||_||_tj ||||d|_ dS)Nrr) kernel_sizestride) r r! image_size patch_size num_channelsr) num_patches patch_shaper"Conv2d projection)r(r]rrrr)rrr+r-r.r!s   z&InternVLVisionPatchEmbeddings.__init__ pixel_valuesreturnc Cs^|j\}}}}||jkrtd||}|jd|jd}}|ddd}|||ffS)NzeMake sure that the channel dimension of the pixel values match with the one set in the configuration.rrr)r<rrdrflattenrQ) r(rr{rheightwidth embeddings patch_height patch_widthr-r-r.r:s   z%InternVLVisionPatchEmbeddings.forward) r?r@rArr!r$rr:rBr-r-r+r.rs rcsleZdZdZdeddffdd Zdejded edejfd d Z dd ejd e ej dejfddZ Z S)InternVLVisionEmbeddingszc Construct the CLS token, position and patch embeddings. Optionally, also the mask token. r]rNcstttdd|j|_|jr!ttdd|j|_ nd|_ t ||_ |j |_ t |jtjjr8|jn|j|jf|_|j j}|jrUttd|d|j|_nd|_t|j|_dS)Nr)r r!r"r#r$zerosr) cls_tokenuse_mask_token mask_tokenrpatch_embeddingsr isinstancer collectionsabcIterabler use_absolute_position_embeddingsposition_embeddingsrphidden_dropout_probrJ)r(r]rr+r-r.r!s   z!InternVLVisionEmbeddings.__init__rrrc Cs|jdd}|jjdd}tjs||kr||kr|jS|jddddf}|jddddf}|jd}||jd} ||jd} t|d} |d| | |}|dddd}t j j || | fdd d }|dddd dd|}tj ||fdd S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rNr/r?rrbicubicF)rwmode align_cornersrL)r<rr$jit is_tracingrrrxpermuter"rR interpolaterycat) r(rrrr num_positionsclass_pos_embedpatch_pos_embedrM new_height new_widthsqrt_num_positionsr-r-r.interpolate_pos_encodings(   z1InternVLVisionEmbeddings.interpolate_pos_encodingrbool_masked_poscCs|j\}}}}||\}\}}|\} } }|dur5|j| | d} |d| } |d| | | }|j| dd} tj | |fdd}|j durT|| |||}| |}|||ffS)Nr/rrL) r<rrwrexpand unsqueezetype_asrr$rrrrJ)r(rrr}rrrrrr{r| mask_tokensw cls_tokensr-r-r.r:s   z InternVLVisionEmbeddings.forwardr)r?r@rArrr!r$rintrr BoolTensorr:rBr-r-r+r.rs+rcs2eZdZfddZdejdejfddZZS)InternVLVisionMLPcsDt||_t|j|_t|j|j |_ t|j |j|_ dSr) r r!r]r hidden_act activation_fnr"rjr)intermediate_sizefc1fc2r(r]r+r-r.r!9s  zInternVLVisionMLP.__init__r7rcCs"||}||}||}|Sr)rrr)r(r7r-r-r.r:@s   zInternVLVisionMLP.forward)r?r@rAr!r$rr:rBr-r-r+r.r8s r) layer_normrms_normcsXeZdZdZdeddffdd Zdejdee eje ejejfffdd Z Z S) InternVLVisionLayerz?This corresponds to the Block class in the timm implementation.r]rNcst|j|_d|_t||_t||_t|j |j |j d|_ t|j |j |j d|_ |j}tj|t|j dd|_tj|t|j dd|_t|j|_dS)Nrr*T) requires_grad)r r!chunk_size_feed_forward seq_len_dimr\ attentionrmlpNORM2FN norm_typer)layer_norm_epslayernorm_beforelayernorm_afterlayer_scale_init_valuer"r#r$r%lambda_1lambda_2rprrJ)r(r] init_valuesr+r-r.r!Ms   zInternVLVisionLayer.__init__r7cCsd|||\}}|j|}||}||}||}||}|jdur,|j|}||}|Sr)rrrrrrJr)r(r7attention_outputr} layer_outputr-r-r.r:\s      zInternVLVisionLayer.forward) r?r@rArrr!r$rrr;r:rBr-r-r+r.rJsrcsBeZdZdeddffdd Zdejdeee ffddZ Z S) InternVLVisionEncoderr]rNcs:t|_tfddtjD|_d|_dS)Ncsg|]}tqSr-)r).0ir]r-r. |sz2InternVLVisionEncoder.__init__..F) r r!r]r" ModuleListrangenum_hidden_layerslayergradient_checkpointingrr+rr.r!ys   zInternVLVisionEncoder.__init__r7cCs|jD]}||}qt|dS)N)last_hidden_state)rr )r(r7 layer_moduler-r-r.r:s  zInternVLVisionEncoder.forward) r?r@rArr!r$rrr;r r:rBr-r-r+r.rxs rcsReZdZUeed<dZdZdZdgZdZ dZ dZ dZ e edZfddZZS) InternVLVisionPreTrainedModelr]internvl_visionrTr)r7 attentionscst|t|tr+|jj|jdur|jj|jdur)|jjdSdSt|t rD|j j |j j |jj |j j dSdS)zInitialize the weightsN)r _init_weightsrrrdatazero_rrrrfill_r]rr)r(rDr+r-r.rs       z+InternVLVisionPreTrainedModel._init_weights)r?r@rAr__annotations__base_model_prefixmain_input_namesupports_gradient_checkpointing_no_split_modules_supports_sdpa_supports_flash_attn_supports_flex_attn_supports_attention_backendrr\_can_record_outputsrrBr-r-r+r.rs rc sfeZdZdeddffdd ZddZedd e dd ej d e ej de e effd d ZZS)InternVLVisionModelr]rNcsTt|||_t||_t||_|jrt ntj |j |j d|_ |dS)Nr)r r!r]rrrencoderuse_mean_poolingr"rq LayerNormr)r layernorm post_initrr+r-r.r!s    zInternVLVisionModel.__init__cC|jjSr)rrr=r-r-r.get_input_embeddingssz(InternVLVisionModel.get_input_embeddingsF)tie_last_hidden_statesrrcCs@|j||d\}}||}|d}||}t||j|jdS)z bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, num_patches)`, *optional*): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). )rr)rr7r)rrrrr7r)r(rrembedding_outputr}encoder_outputssequence_outputr-r-r.r:s  zInternVLVisionModel.forwardr)r?r@rArr!rrrr$rrrrr;rr:rBr-r-r+r.rs rc@s6eZdZUeed<dZdZdZdZdZ dZ dZ dZ dS)InternVLPreTrainedModelr]Tpast_key_valuesN) r?r@rArrrr_skip_keys_device_placementrr_can_compile_fullgraphrrr-r-r-r.rs rcs*eZdZdeffdd ZddZZS)InternVLMultiModalProjectorr]csztt|jjtd|jd|_t |jjtd|jd|j j|_ t |j |_t |j j|j j|_dS)Nrr)r r!r"r vision_configr)rdownsample_ratiorrj text_configlinear_1rprojector_hidden_actactlinear_2rr+r-r.r!s " z$InternVLMultiModalProjector.__init__cCs,||}||}||}||}|Sr)rr r r)r(image_featuresr7r-r-r.r:s    z#InternVLMultiModalProjector.forward)r?r@rArr!r:rBr-r-r+r.rs rzM Base class for InternVL outputs, with hidden states and attentions. c@s$eZdZUdZdZeejed<dS)InternVLModelOutputWithPasta past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder and after projecting the last hidden state. Nimage_hidden_states) r?r@rArrrr$ FloatTensorrr-r-r-r.rs  rzx The InternVL model which consists of a vision backbone and a language model, without a language modeling head. csHeZdZddiZdeffdd ZddZdd Zd d Zd d Z  d&de j de e eeefde efddZde jde j de j fddZee         d'de e jde e j de e jde e jde ede e j de e eeefde ede e jdeede eeffdd Zd(d"e jd#efd$d%ZZS)) InternVLModelzlanguage_model.modellanguage_modelr]cs>t|t|j|_t||_t|j|_ | dSr) r r!r from_configr vision_towerrmulti_modal_projectorr rrrr+r-r.r!s   zInternVLModel.__init__cC |jSr)rrr=r-r-r.r z"InternVLModel.get_input_embeddingscC|j|dSr)rset_input_embeddingsr(rGr-r-r.rz"InternVLModel.set_input_embeddingscCs ||_dSrrr(decoderr-r-r. set_decoder!rzInternVLModel.set_decodercC|jSrrr=r-r-r. get_decoder$zInternVLModel.get_decoderNrvision_feature_layervision_feature_select_strategyc Ks|dur|n|jj}|dur|n|jj}|j|jd}|jj}|dkr+|j|dj}n |j|dj |}|dkrE|ddddddf}|j d}t |d}|j d} | | ||d}|j ||d }| | d|j d}||}|S) a% Obtains image last hidden states from the vision tower and apply multimodal projection. Args: pixel_values (`torch.FloatTensor]` of shape `(batch_size, channels, height, width)`) The tensors corresponding to the input images. vision_feature_layer (`int` or `list[int]`): Layer index or list of layer indices to extract features from. Returns: vision_features (`torch.Tensor`): Image feature tensor of shape `(num_images, image_length, embed_dim)`. N)r1r/)rdefaultrrr) scale_factor)r]r%r&r2r1r rr vision_modelr7r<rrx pixel_shuffler) r(rr%r&rUr vision_featureschannels feature_sizer{r-r-r.get_image_features's*    z InternVLModel.get_image_features input_ids inputs_embedsrcCs|dur||tj|jjtj|jdk}|d}n||jjk}|}| d | |j}|j d|j d}|| | krPtd|d||S)z Obtains multimodal placeholder mask from `input_ids` or `inputs_embeds`, and checks that the placeholder token count is equal to the length of multimodal features. If the lengths are different, an error is raised. N)r1devicer/rrz6Image features and image tokens do not match: tokens: z , features )rr$tensorr]image_token_idlongr1allsumr expand_asr2r<numelrd)r(r/r0rspecial_image_maskn_image_tokensn_image_featuresr-r-r.get_placeholder_mask]s  z"InternVLModel.get_placeholder_maskrH position_idsrcache_positionrUrc Ks|dur|n|jj}|dur|n|jj}|du|duAr td|dur*||}|durL|j|||d} | |j|j} |j ||| d} | | | }|j d||||| d| } t | j | j| j| j|durk| dSddS)Nz:You must specify exactly one of input_ids or inputs_embedsrr%r&)r0r)rHr=rr0r>)rrr7rrr-)r]r%r&rdrr.r2r1r1r<masked_scatterrrrrr7r)r(r/rrHr=rr0r%r&r>rUrr9outputsr-r-r.r:usN    zInternVLModel.forwardrr+r(c Cs|\}}}}||dks||dkrtd|||t||t||}|dddd}||t||t||t||d}|dddd}|S)a&Perform pixel shuffle downsampling on vision features. Args: vision_features (`torch.Tensor`): Input tensor of shape (batch_size, width, height, channels). scale_factor (`float`, *optional*, defaults to `0.5`): Factor by which to downsample. Default is 0.5, which halves the dimensions. Returns: vision_features (`torch.Tensor`): Downsampled tensor of shape (batch_size, height*scale_factor, width*scale_factor, channels/(scale_factor^2)). rzKHeight and width must be divisible by scale_factor for proper downsampling.rrr)rwrdryrrrT)r(r+r(r{rrr,r-r-r.r*s $zInternVLModel.pixel_shuffleNN) NNNNNNNNN)r) r?r@rA_checkpoint_conversion_mappingrr!rrr!r#r$rrrrliststrr. LongTensorr<rrrrrrr;rr:floatr*rBr-r-r+r.r st 6     9rzT Base class for InternVL causal language model (or autoregressive) outputs. c@seZdZUdZdZeejed<dZ eejed<dZ ee ed<dZ ee ejed<dZee ejed<dZeejed<dS) InternVLCausalLMOutputWithPasta4 loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder and after projecting the last hidden state. Nlosslogitsrr7rr)r?r@rArrIrr$rrrJrrr7r;rrr-r-r-r.rHs rHzV The INTERNVL model which consists of a vision backbone and a language model. c!seZdZdddddZdgZdeffdd Zd d Zd d Zde j fddZ ddZ ddZ  d0dejdeeeeefdeefddZeddZeddZedd Zee           ! d1d"eejdeejd#eejd$eejd%eed&eejdeeeeefdeed'eejd(eejd)eeejfd*eejd+ee dee!e"ffd,d-Z#      d2fd.d/ Z$Z%S)3 InternVLForConditionalGenerationzmodel.language_modelzmodel.vision_towerzmodel.multi_modal_projectorlm_head)z^language_model.modelz ^vision_towerz^multi_modal_projectorz^language_model.lm_headzlm_head.weightr]cs<t|t||_tj|jj|jjdd|_ | dS)NFr^) r r!rmodelr"rjr r) vocab_sizerLrrr+r-r.r!s   z)InternVLForConditionalGeneration.__init__cCrr)rMrr=r-r-r.rrz5InternVLForConditionalGeneration.get_input_embeddingscCrr)rMrrr-r-r.r rz5InternVLForConditionalGeneration.set_input_embeddingsrcCr"r)rLr=r-r-r.get_output_embeddings r$z6InternVLForConditionalGeneration.get_output_embeddingscCrr)rMr!rr-r-r.r!rz,InternVLForConditionalGeneration.set_decodercCrr)rMr#r=r-r-r.r#rz,InternVLForConditionalGeneration.get_decoderNrr%r&cKs|jjd|||d|S)Nr?r-)rMr.)r(rr%r&rUr-r-r.r.sz3InternVLForConditionalGeneration.get_image_featurescCrr)rMrr=r-r-r.r$z/InternVLForConditionalGeneration.language_modelcCrr)rMrr=r-r-r.r(rPz-InternVLForConditionalGeneration.vision_towercCrr)rMrr=r-r-r.r,rPz6InternVLForConditionalGeneration.multi_modal_projectorrr/rHr=rr0labelsr>logits_to_keep image_sizesrUc  Ks|dur|n|jj}|dur|n|jj}|jd||||||||| | d | }|d}t| tr6t| dn| }||dd|ddf}d}| dur[|jd|| |jj j d| }t |||j |j |j|jdS)ac Example: ```python >>> import torch >>> from transformers import AutoProcessor, AutoModelForImageTextToText >>> torch_device = "cuda" >>> processor = AutoProcessor.from_pretrained("OpenGVLab/InternVL3-1B-hf") >>> model = AutoModelForImageTextToText.from_pretrained( ... "OpenGVLab/InternVL3-1B-hf", dtype=torch.bfloat16, device_map=torch_device ... ) >>> messages = [ ... { ... "role": "user", ... "content": [ ... { ... "type": "image", ... "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg", ... }, ... { ... "type": "image", ... "url": "https://thumbs.dreamstime.com/b/golden-gate-bridge-san-francisco-purple-flowers-california-echium-candicans-36805947.jpg", ... }, ... {"type": "text", "text": "These images depict two different landmarks. Can you identify them?"}, ... ], ... }, ... ] >>> inputs = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt").to(torch_device) >>> generate_ids = model.generate(**inputs, max_new_tokens=200) >>> print(processor.decode(generate_ids[0, inputs["input_ids"].shape[1] :], skip_special_tokens=True)) The images depict the Statue of Liberty and the Golden Gate Bridge. ```N) r/rrHr=rr0r%r&r>rSr)rJrQrN)rIrJrr7rrr-)r]r%r&rMrrslicerL loss_functionr rNrHrr7rr)r(r/rrHr=rr0r%r&rQr>rRrSrUrAr7 slice_indicesrJrIr-r-r.r:0sL6  z(InternVLForConditionalGeneration.forwardc  s8tj|f|||||d|} |ddkr|| d<| S)N)rr0rHr>rRrr)r prepare_inputs_for_generation) r(r/rr0rrHr>rRrU model_inputsr+r-r.rWs  z>InternVLForConditionalGeneration.prepare_inputs_for_generationrB) NNNNNNNNNNrN)NNNNNN)&r?r@rArC_tied_weights_keysrr!rrr"ModulerOr!r#r$rrrrrDrEr.propertyrrrrrrFrrrrr;rHr:rWrBr-r-r+r.rKs          arK)rrrrrK)rC)Bcollections.abcr dataclassesrtypingrrrr$torch.nnr" activationsr cache_utilsr generationr integrationsr modeling_layersr modeling_outputsr r rmodeling_utilsrrprocessing_utilsrutilsrrrrr utils.genericrautorconfiguration_internvlrrrZrrrGr[r\rrrrrrrrrrrrrrrHrK__all__r-r-r-r.s           I &^ .*E 9