o ei@sddlZddlmZddlmZddlZddlmZddlm Z ddl m Z ddl mZddlmZdd lmZdd lmZdd lmZmZmZdd lmZmZdd lmZddlmZm Z m!Z!m"Z"m#Z#ddl$m%Z%m&Z&ddl'm(Z(ddl)m*Z*ddl+m,Z,m-Z-edGdddej.Z/ dHdej.dej0dej0dej0dej0dBde1de1fd d!Z2Gd"d#d#ej.Z3ee!d$d%Gd&d'd'eZ4Gd(d)d)ej.Z5Gd*d+d+ej.Z6Gd,d-d-ej.Z7ej8e/d.Z9Gd/d0d0eZ:Gd1d2d2ej.Z;e!Gd3d4d4eZGd9d:d:ej.Z?ee!d;d%Gdd%Gd?d@d@e>ZAee!dAd%GdBdCdCeZBe!dDd%GdEdFdFe>eZCgdGZDdS)IN)Callable) dataclass)initialization)ACT2FN)Cache)GenerationMixin)use_kernel_forward_from_hub)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPastBaseModelOutputWithPooling)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack) ModelOutputTransformersKwargsauto_docstringtorch_compilable_check torch_int)can_return_tuplemerge_with_config_defaults)capture_outputs) AutoModel)InternVLConfigInternVLVisionConfigRMSNormcsFeZdZd deddffdd Zdejdejfdd Zd d ZZ S) InternVLVisionRMSNormư>epsreturnNcs&ttt||_||_dS)zD InternVLVisionRMSNorm is equivalent to T5LayerNorm N)super__init__nn Parametertorchonesweightvariance_epsilon)self hidden_sizer! __class__l/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/internvl/modeling_internvl.pyr$/s  zInternVLVisionRMSNorm.__init__ hidden_statescCsJ|j}|tj}|djddd}|t||j}|j||S)NrT)keepdim) dtypetor'float32powmeanrsqrtr*r))r+r1 input_dtypevariancer/r/r0forward7s  zInternVLVisionRMSNorm.forwardcCst|jjd|jS)Nz, eps=)tupler)shaper*r+r/r/r0 extra_repr>sz InternVLVisionRMSNorm.extra_repr)r ) __name__ __module__ __qualname__floatr$r'Tensorr<r@ __classcell__r/r/r-r0r-srmodulequerykeyvalueattention_maskscalingdropoutc Ksx|}|} t||dd|} |dur| |} tjj| dd} tjj| ||jd} t| | } | dd} | | fS)Nrrr2dim)ptrainingr) r'matmul transposer% functionalsoftmaxrNrR contiguous) rHrIrJrKrLrMrNkwargs key_states value_states attn_weights attn_outputr/r/r0eager_attention_forwardBs  r]csLeZdZdZdeffdd Z d dejdejdBdee fd d Z Z S) InternVLVisionAttentionz+Attention Class for InternVL Vision EncoderconfigcsBt||_|j|_|j|_|j|j|_|j|j|jkr-td|jd|jd|jd|_ |j |_ |j }|j }d|_ tj|j|j|j|jd|_tj|j|j|j|jd|_tj|j|j|j|jd|_t|j|j|_|dkrt|nt|_ |rt|jnt|_|rt|j|_dSt|_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).gFbiasr)r#r$r_r, embed_dimnum_attention_heads num_headshead_dim ValueErrorscaleattention_dropoutprojection_dropout use_qk_norm is_causalr%Linearattention_biasq_projk_projv_projprojection_layerDropoutIdentityrq_normk_norm)r+r_ proj_dropoutqk_normr-r/r0r$_s.   "z InternVLVisionAttention.__init__Nr1rLrXcKs|\}}}||}||}||} ||}||}||||j|j dd}||||j|j dd}| |||j|j dd} t |j jt} | |||| |f|js^dn|j|jdd|\} } | |||j} || } || } | | fS)NrrrGF)rNrMrk)sizernrorprtrureshaperdrerTviewr get_interfacer__attn_implementationr]rRrhrgrbrqri)r+r1rLrX batch_sizeseq_len_ query_statesrYrZattention_interfacer\r[outputr/r/r0r<{s:        zInternVLVisionAttention.forwardN) rArBrC__doc__rr$r'rErrr<rFr/r/r-r0r^\sr^z7 Class for outputs of [`InternVLVisionModel`].  custom_introc@seZdZdZdS)$InternVLVisionModelOutputWithPoolingaF pooler_output (`torch.FloatTensor` of shape `(batch_size, hidden_size)`): Average of the last layer hidden states of the patch tokens (excluding the *[CLS]* token) if *config.use_mean_pooling* is set to True. If set to False, then the final hidden state of the *[CLS]* token will be returned. N)rArBrCrr/r/r/r0rsrcs6eZdZdZfddZdejdejfddZZS)InternVLVisionPatchEmbeddingsz This class turns `pixel_values` of shape `(batch_size, num_channels, height, width)` into the initial `hidden_states` (patch embeddings) of shape `(batch_size, seq_length, hidden_size)` to be consumed by a Transformer. cst|j|j}}|j|j}}|d|d|d|d}|d|d|d|df}||_||_||_||_||_tj ||||d|_ dS)Nrr) kernel_sizestride) r#r$ image_size patch_size num_channelsr, num_patches patch_shaper%Conv2d projection)r+r_rrrr,rrr-r/r0r$s   z&InternVLVisionPatchEmbeddings.__init__ pixel_valuesr"cCsL|j\}}}}||jkrtd|||jjj}|ddd}|S)NzeMake sure that the channel dimension of the pixel values match with the one set in the configuration.rr) r>rrfrr5r)r4flattenrT)r+rr}rheightwidth embeddingsr/r/r0r<s z%InternVLVisionPatchEmbeddings.forward) rArBrCrr$r'rEr<rFr/r/r-r0rs rcsleZdZdZdeddffdd Zdejded edejfd d Z dd ejd ej dBdejfddZ Z S)InternVLVisionEmbeddingszc Construct the CLS token, position and patch embeddings. Optionally, also the mask token. r_r"Ncstttdd|j|_|jr!ttdd|j|_ nd|_ t ||_ |j |_ t |jtjjr8|jn|j|jf|_|j j}|jrUttd|d|j|_nd|_t|j|_dS)Nr)r#r$r%r&r'zerosr, cls_tokenuse_mask_token mask_tokenrpatch_embeddingsr isinstancer collectionsabcIterabler use_absolute_position_embeddingsposition_embeddingsrrhidden_dropout_probrN)r+r_rr-r/r0r$s   z!InternVLVisionEmbeddings.__init__rrrc Cs|jdd}|jjdd}tjs||kr||kr|jS|jddddf}|jddddf}|jd}||jd} ||jd} t|d} |d| | |}|dddd}t j j || | fdd d }|dddd dd|}tj ||fdd S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rNr2r?rrbicubicF)rxmode align_cornersrO)r>rr'jit is_tracingrrrypermuter%rU interpolaterzcat) r+rrrr num_positionsclass_pos_embedpatch_pos_embedrP new_height new_widthsqrt_num_positionsr/r/r0interpolate_pos_encodings(   z1InternVLVisionEmbeddings.interpolate_pos_encodingrbool_masked_posc Cs|j\}}}}||}|\}}}|dur1|j||d} |d| } |d| | | }|j|dd} tj | |fdd}|j durP|| |||}| |}|S)Nr2rrO) r>rrxrexpand unsqueezetype_asrr'rrrrN) r+rrrrrrr}r~ mask_tokensw cls_tokensr/r/r0r<s   z InternVLVisionEmbeddings.forwardr) rArBrCrrr$r'rEintr BoolTensorr<rFr/r/r-r0rs+rcs2eZdZfddZdejdejfddZZS)InternVLVisionMLPcsDt||_t|j|_t|j|j |_ t|j |j|_ dSr) r#r$r_r hidden_act activation_fnr%rlr,intermediate_sizefc1fc2r+r_r-r/r0r$8s  zInternVLVisionMLP.__init__r1r"cCs"||}||}||}|Sr)rrr)r+r1r/r/r0r<?s   zInternVLVisionMLP.forward)rArBrCr$r'rEr<rFr/r/r-r0r7s r) layer_normrms_normcsTeZdZdZdeddffdd ZdejdeejeejejfBfdd Z Z S) InternVLVisionLayerz?This corresponds to the Block class in the timm implementation.r_r"Ncst|j|_d|_t||_t||_t|j |j |j d|_ t|j |j |j d|_ |j}tj|t|j dd|_tj|t|j dd|_t|j|_dS)Nrr!T) requires_grad)r#r$chunk_size_feed_forward seq_len_dimr^ attentionrmlpNORM2FN norm_typer,layer_norm_epslayernorm_beforelayernorm_afterlayer_scale_init_valuer%r&r'r(lambda_1lambda_2rrrrN)r+r_ init_valuesr-r/r0r$Ls   zInternVLVisionLayer.__init__r1cCsd|||\}}|j|}||}||}||}||}|jdur,|j|}||}|Sr)rrrrrrNr)r+r1attention_outputr layer_outputr/r/r0r<[s      zInternVLVisionLayer.forward) rArBrCrrr$r'rEr=r<rFr/r/r-r0rIsrcs>eZdZdeddffdd ZdejdeeBfddZ Z S) InternVLVisionEncoderr_r"Ncs:t|_tfddtjD|_d|_dS)Ncsg|]}tqSr/)r).0ir_r/r0 {sz2InternVLVisionEncoder.__init__..F) r#r$r_r% ModuleListrangenum_hidden_layerslayergradient_checkpointingrr-rr0r$xs   zInternVLVisionEncoder.__init__r1cCs|jD]}||}qt|dS)N)last_hidden_state)rr )r+r1 layer_moduler/r/r0r<~s  zInternVLVisionEncoder.forward) rArBrCrr$r'rEr=r r<rFr/r/r-r0rwsrcs^eZdZUeed<dZdZdZdZdgZ dZ dZ dZ dZ eedZefdd ZZS) InternVLVisionPreTrainedModelr_internvl_visionr)imagevideoTr)r1 attentionscst|t|tr+t|j|jdurt|j|jdur)t|jdSdSt|t rDt |j |j j t |j|j j dSdS)zInitialize the weightsN)r# _init_weightsrrinitzeros_rrrr constant_rr_rr)r+rHr-r/r0rs       z+InternVLVisionPreTrainedModel._init_weights)rArBrCr__annotations__base_model_prefixmain_input_nameinput_modalitiessupports_gradient_checkpointing_no_split_modules_supports_sdpa_supports_flash_attn_supports_flex_attn_supports_attention_backendrr^_can_record_outputsr'no_gradrrFr/r/r-r0rs rc sfeZdZdeddffdd ZddZeedd e dd e j d e j dBde e Bfd d ZZS)InternVLVisionModelr_r"NcsTt|||_t||_t||_|jrt ntj |j |j d|_ |dS)Nr)r#r$r_rrrencoderuse_mean_poolingr%rs LayerNormr,r layernorm post_initrr-r/r0r$s    zInternVLVisionModel.__init__cCs|jjSr)rrr?r/r/r0get_input_embeddingssz(InternVLVisionModel.get_input_embeddingsF)tie_last_hidden_statesrrcKs<|j||d}||}|d}||}t||j|jdS)z bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, num_patches)`, *optional*): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). )rr)rr1r)rrrrr1r)r+rrrXembedding_outputencoder_outputssequence_outputr/r/r0r<s  zInternVLVisionModel.forwardr)rArBrCrr$rrrrr'rErr=rr<rFr/r/r-r0rsrc@s:eZdZUeed<dZdZdZdZdZ dZ dZ dZ dZ dS)InternVLPreTrainedModelr_model)rtextrTpast_key_valuesN)rArBrCrrrrr_skip_keys_device_placementrr_can_compile_fullgraphrrr/r/r/r0rs rcs*eZdZdeffdd ZddZZS)InternVLMultiModalProjectorr_csztt|jjtd|jd|_t |jjtd|jd|j j|_ t |j |_t |j j|j j|_dS)Nrr)r#r$r%r vision_configr,rdownsample_ratiorrl text_configlinear_1rprojector_hidden_actactlinear_2rr-r/r0r$s " z$InternVLMultiModalProjector.__init__cCs,||}||}||}||}|Sr)rrrr)r+image_featuresr1r/r/r0r<s    z#InternVLMultiModalProjector.forward)rArBrCrr$r<rFr/r/r-r0r s r zM Base class for InternVL outputs, with hidden states and attentions. c@s$eZdZUdZdZejdBed<dS)InternVLModelOutputWithPasta past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder and after projecting the last hidden state. Nimage_hidden_states)rArBrCrrr' FloatTensorrr/r/r/r0rs  rzx The InternVL model which consists of a vision backbone and a language model, without a language modeling head. csNeZdZddiZdeffdd ZddZdd Zee e d d  d$d e j de ee Bd Bded BdeedeeBf ddZde jde j de j fddZee d%de jd Bd e j d Bde jd Bde jd Bded Bde j d Bde ee Bd Bded Bde jd BdeedeeBfddZd&d e jd!efd"d#ZZS)' InternVLModel^language_model.modellanguage_modelr_cs>t|t|j|_t||_t|j|_ | dSr) r#r$r from_configr  vision_towerr multi_modal_projectorr rrrr-r/r0r$s   zInternVLModel.__init__cC |jSr)rrr?r/r/r0r z"InternVLModel.get_input_embeddingscC|j|dSr)rset_input_embeddingsr+rKr/r/r0r!z"InternVLModel.set_input_embeddingszWObtains image last hidden states from the vision tower and apply multimodal projection.rNrvision_feature_layervision_feature_select_strategyrXr"c Ks|j|jd}|jj}|dkrd|d<|jd |dd|}|dkr&|j}n|j|}|dkr<|ddddddf}|jd}t|d } |jd } | | | | d}|j ||d }| | d|jd}| |}||_ |S) a! pixel_values (`torch.FloatTensor]` of shape `(batch_size, channels, height, width)`) The tensors corresponding to the input images. vision_feature_layer (`int` or `list[int]`): Layer index or list of layer indices to extract features from. )r4r2Toutput_hidden_states)r return_dictdefaultNrrr) scale_factorr/) r5r4r_r rrr1r>rry pixel_shuffler pooler_output) r+rr"r#rXr vision_outputsvision_featureschannels feature_sizer}r/r/r0get_image_features$s&     z InternVLModel.get_image_features input_ids inputs_embedsrcCs|dur||tj|jjtj|jdk}|d}n||jjk}|}|j d|j d}| d | |j}t |||kd|d||S)z Obtains multimodal placeholder mask from `input_ids` or `inputs_embeds`, and checks that the placeholder token count is equal to the length of multimodal features. If the lengths are different, an error is raised. N)r4devicer2rrz6Image features and image tokens do not match, tokens: z , features: )rr'tensorr_image_token_idlongr1allsumr>r expand_asr5rnumel)r+r/r0rspecial_image_maskn_image_tokensn_image_featuresr/r/r0get_placeholder_maskWs  z"InternVLModel.get_placeholder_maskrL position_idsrcache_positionc Ks|du|duAr td|dur||}|dur:|j|||ddj} | |j|j} |j||| d} || | }|j d||||| d| } t | j | j | j | j|durY| dSddS)Nz:You must specify exactly one of input_ids or inputs_embedsT)rr"r#r%)r0r)rLr=rr0r>)rrr1rrr/)rfrr.r)r5r1r4r<masked_scatterrrrrr1r)r+r/rrLr=rr0r"r#r>rXrr9outputsr/r/r0r<osH    zInternVLModel.forwardrr+r'c Cs|\}}}}||dks||dkrtd|||t||t||}|dddd}||t||t||t||d}|dddd}|S)a&Perform pixel shuffle downsampling on vision features. Args: vision_features (`torch.Tensor`): Input tensor of shape (batch_size, width, height, channels). scale_factor (`float`, *optional*, defaults to `0.5`): Factor by which to downsample. Default is 0.5, which halves the dimensions. Returns: vision_features (`torch.Tensor`): Downsampled tensor of shape (batch_size, height*scale_factor, width*scale_factor, channels/(scale_factor^2)). rzKHeight and width must be divisible by scale_factor for proper downsampling.rrr)rxrfrzrrrW)r+r+r'r}rrr,r/r/r0r(s $zInternVLModel.pixel_shuffleNN) NNNNNNNNN)r)rArBrC_checkpoint_conversion_mappingrr$rrrrrr'rrliststrrrr=r r. LongTensorr<rErrr<rDr(rFr/r/r-r0r s.     1rzT Base class for InternVL causal language model (or autoregressive) outputs. c@seZdZUdZdZejdBed<dZejdBed<dZ e dBed<dZ e ejdBed<dZ e ejdBed<dZejdBed<dS) InternVLCausalLMOutputWithPasta4 loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder and after projecting the last hidden state. Nlosslogitsrr1rr)rArBrCrrGr'rrrHrrr1r=rrr/r/r/r0rFs rFzV The INTERNVL model which consists of a vision backbone and a language model. c sleZdZdddddZddiZdeffd d Zd d Zd dZde j fddZ e  d(de jdeeeBdBdedBdeedeeBf ddZee            d)de jdBde jdBde jdBde jdBdedBde jdBdeeeBdBdedBde jdBd e jdBd!ee jBd"e jdBdeedeeBfd#d$Z       %d*fd&d' ZZS)+ InternVLForConditionalGenerationzmodel.language_modelzmodel.vision_towerzmodel.multi_modal_projectorlm_head)rz ^vision_towerz^multi_modal_projectorz^language_model.lm_headzlm_head.weightz(model.language_model.embed_tokens.weightr_cs<t|t||_tj|jj|jjdd|_ | dS)NFr`) r#r$rrr%rlr r, vocab_sizerJrrr-r/r0r$s   z)InternVLForConditionalGeneration.__init__cCrr)rrr?r/r/r0rrz5InternVLForConditionalGeneration.get_input_embeddingscCrr)rrr r/r/r0rr!z5InternVLForConditionalGeneration.set_input_embeddingsr"cCs|jSr)rJr?r/r/r0get_output_embeddingssz6InternVLForConditionalGeneration.get_output_embeddingsNrr"r#rXcKs|jjd|||d|S)N)rr"r#r/)rr.)r+rr"r#rXr/r/r0r.sz3InternVLForConditionalGeneration.get_image_featuresrr/rLr=rr0labelsr>logits_to_keep image_sizesc  Ks|jd||||||||| | d | }|d}t| tr"t| dn| }||dd|ddf}d}| durG|jd|| |jjjd| }t |||j |j |j |j dS)ac Example: ```python >>> import torch >>> from transformers import AutoProcessor, AutoModelForImageTextToText >>> torch_device = "cuda" >>> processor = AutoProcessor.from_pretrained("OpenGVLab/InternVL3-1B-hf") >>> model = AutoModelForImageTextToText.from_pretrained( ... "OpenGVLab/InternVL3-1B-hf", dtype=torch.bfloat16, device_map=torch_device ... ) >>> messages = [ ... { ... "role": "user", ... "content": [ ... { ... "type": "image", ... "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg", ... }, ... { ... "type": "image", ... "url": "https://thumbs.dreamstime.com/b/golden-gate-bridge-san-francisco-purple-flowers-california-echium-candicans-36805947.jpg", ... }, ... {"type": "text", "text": "These images depict two different landmarks. Can you identify them?"}, ... ], ... }, ... ] >>> inputs = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt").to(torch_device) >>> generate_ids = model.generate(**inputs, max_new_tokens=200) >>> print(processor.decode(generate_ids[0, inputs["input_ids"].shape[1] :], skip_special_tokens=True)) The images depict the Statue of Liberty and the Golden Gate Bridge. ```) r/rrLr=rr0r"r#r>rOrN)rHrMrK)rGrHrr1rrr/)rrrslicerJ loss_functionr_r rKrFrr1rr)r+r/rrLr=rr0r"r#rMr>rNrOrXr@r1 slice_indicesrHrGr/r/r0r<s@5  z(InternVLForConditionalGeneration.forwardFc s>tj|f||||||d| } |s| dds|| d<| S)N)rr0rLr>rNis_first_iteration use_cacheTr)r#prepare_inputs_for_generationget) r+r/rr0rrLr>rNrSrX model_inputsr-r/r0rUgs z>InternVLForConditionalGeneration.prepare_inputs_for_generationrA) NNNNNNNNNNrN)NNNNNNF)rArBrCrB_tied_weights_keysrr$rrr%ModulerLrr'rrrCrDrrr=r r.rrErErrFr<rUrFr/r/r-r0rIs      XrI)rrrrrI)rG)Ecollections.abcrr dataclassesrr'torch.nnr%rr activationsr cache_utilsr generationr integrationsr modeling_layersr modeling_outputsr r r modeling_utilsrrprocessing_utilsrutilsrrrrr utils.genericrrutils.output_capturingrautorconfiguration_internvlrrrYrrErDr]r^rrrrrrrrrrrr rrrFrI__all__r/r/r/r0s             I %^ . )6 "