o wi@sddlZddlmZddlmZmZmZddlZddl m Z ddl m Z ddl mZddlmZddlmZdd lmZdd lmZmZmZdd lmZmZdd lmZdd lmZm Z m!Z!m"Z"m#Z#m$Z$ddl%m&Z&ddl'm(Z(m)Z)edGddde j*Z+ dGde j*dej,dej,dej,deej,de-de-fddZ.Gdd d e j*Z/e!Gd!d"d"eZ0ee!d#d$Gd%d&d&eZ1Gd'd(d(e j*Z2Gd)d*d*e j*Z3Gd+d,d,e j*Z4e j5e+d-Z6Gd.d/d/eZ7Gd0d1d1e j*Z8e!Gd2d3d3e0Z9e!Gd4d5d5eZ:Gd6d7d7e j*Z;ee!d8d$Gd9d:d:eZd$Gd?d@d@e Z>GdAdBdBeeZ?e!dCd$GdDdEdEe:eZ@gdFZAdS)HN) dataclass)CallableOptionalUnion)ACT2FN)GenerationMixin)use_kernel_forward_from_hub)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPastBaseModelOutputWithPooling)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack) LossKwargs ModelOutputauto_docstringcan_return_tupleis_torchdynamo_compiling torch_int) AutoModel)InternVLConfigInternVLVisionConfigRMSNormcs.eZdZdfdd ZddZddZZS) InternVLVisionRMSNormư>cs&ttt||_||_dS)zD InternVLVisionRMSNorm is equivalent to T5LayerNorm N)super__init__nn Parametertorchonesweightvariance_epsilon)self hidden_sizeeps __class__k/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/transformers/models/internvl/modeling_internvl.pyr!4s  zInternVLVisionRMSNorm.__init__cCsJ|j}|tj}|djddd}|t||j}|j||S)NrT)keepdim) dtypetor$float32powmeanrsqrtr'r&)r( hidden_states input_dtypevariancer-r-r.forward<s  zInternVLVisionRMSNorm.forwardcCst|jjd|jS)Nz, eps=)tupler&shaper'r(r-r-r. extra_reprCsz InternVLVisionRMSNorm.extra_repr)r)__name__ __module__ __qualname__r!r:r> __classcell__r-r-r+r.r2srmodulequerykeyvalueattention_maskscalingdropoutc Ks|}|} t||dd|} |dur+|ddddddd|jdf} | | } tjj| dd} tjj| ||jd} t| | } | dd } | | fS)Nrrr/dim)ptrainingr) r$matmul transposer<r" functionalsoftmaxrJrO contiguous) rDrErFrGrHrIrJkwargs key_states value_states attn_weights causal_mask attn_outputr-r-r.eager_attention_forwardGs & r[c sXeZdZdZdeffdd Z  d dejdeejdeejd e e fd d Z Z S) InternVLVisionAttentionz+Attention Class for InternVL Vision EncoderconfigcsBt||_|j|_|j|_|j|j|_|j|j|jkr-td|jd|jd|jd|_ |j |_ |j }|j }d|_ tj|j|j|j|jd|_tj|j|j|j|jd|_tj|j|j|j|jd|_t|j|j|_|dkrt|nt|_ |rt|jnt|_|rt|j|_dSt|_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).gFbiasr)r r!r]r) embed_dimnum_attention_heads num_headshead_dim ValueErrorscaleattention_dropoutprojection_dropout use_qk_norm is_causalr"Linearattention_biasq_projk_projv_projprojection_layerDropoutIdentityrq_normk_norm)r(r] proj_dropoutqk_normr+r-r.r!es.   "z InternVLVisionAttention.__init__Nr7rHoutput_attentionsrUcKs"|\}}}||}||} ||} ||}|| } ||||j|j dd}| |||j|j dd} | |||j|j dd} t } |j j dkrXt|j j } | ||| | |f|jsddn|j|jdd|\} } | |||j} || }||}|r|| f}|S|df}|S)NrreagerrCF)rJrIri)sizerlrmrnrrrsreshaperbrcrQviewr[r]_attn_implementationrrOrfrer`rorg)r(r7rHrvrU batch_sizeseq_len_ query_statesrVrWattention_interfacerZrXoutputoutputsr-r-r.r:s@           zInternVLVisionAttention.forwardNN) r?r@rA__doc__rr!r$Tensorrrr r:rBr-r-r+r.r\bsr\c@s:eZdZeZdZdZdZdgZdZ dZ dZ dZ ddZ dS)InternVLVisionPreTrainedModelinternvl_vision pixel_valuesTInternVLVisionLayercCs:t|tjtjtjfr%|jjjd|jj d|j dur#|j j dSdSt|tj rH|jjjd|jj d|j durF|jj|j  dSdSt|tjr]|j j |jjddSt|tr|jj |jdurs|jj |jdur|jj dSdSt|tr|jj|jj|jj|jjdSdS)zInitialize the weightsrCr5stdN?) isinstancer"rjConv2dConvTranspose2dr&datanormal_r]initializer_ranger_zero_ Embedding padding_idx LayerNormfill_InternVLVisionEmbeddings cls_token mask_tokenposition_embeddingsrlambda_1layer_scale_init_valuelambda_2)r(rDr-r-r. _init_weightss0           z+InternVLVisionPreTrainedModel._init_weightsN)r?r@rAr config_classbase_model_prefixmain_input_namesupports_gradient_checkpointing_no_split_modules_supports_sdpa_supports_flash_attn_2_supports_flex_attn_supports_attention_backendrr-r-r-r.rs rz7 Class for outputs of [`InternVLVisionModel`]. ) custom_introc@seZdZdZdS)$InternVLVisionModelOutputWithPoolingaF pooler_output (`torch.FloatTensor` of shape `(batch_size, hidden_size)`): Average of the last layer hidden states of the patch tokens (excluding the *[CLS]* token) if *config.use_mean_pooling* is set to True. If set to False, then the final hidden state of the *[CLS]* token will be returned. N)r?r@rArr-r-r-r.rsrcs6eZdZdZfddZdejdejfddZZS)InternVLVisionPatchEmbeddingsz This class turns `pixel_values` of shape `(batch_size, num_channels, height, width)` into the initial `hidden_states` (patch embeddings) of shape `(batch_size, seq_length, hidden_size)` to be consumed by a Transformer. cst|j|j}}|j|j}}|d|d|d|d}|d|d|d|df}||_||_||_||_||_tj ||||d|_ dS)Nrr) kernel_sizestride) r r! image_size patch_size num_channelsr) num_patches patch_shaper"r projection)r(r]rrrr)rrr+r-r.r!s   z&InternVLVisionPatchEmbeddings.__init__rreturnc Cs^|j\}}}}||jkrtd||}|jd|jd}}|ddd}|||ffS)NzeMake sure that the channel dimension of the pixel values match with the one set in the configuration.rrr)r<rrdrflattenrQ) r(rr|rheightwidth embeddings patch_height patch_widthr-r-r.r:s   z%InternVLVisionPatchEmbeddings.forward) r?r@rArr!r$rr:rBr-r-r+r.rs rcsleZdZdZdeddffdd Zdejded edejfd d Z dd ejd e ej dejfddZ Z S)rzc Construct the CLS token, position and patch embeddings. Optionally, also the mask token. r]rNcstttdd|j|_|jr!ttdd|j|_ nd|_ t ||_ |j |_ t |jtjjr8|jn|j|jf|_|j j}|jrUttd|d|j|_nd|_t|j|_dS)Nr)r r!r"r#r$zerosr)ruse_mask_tokenrrpatch_embeddingsrrr collectionsabcIterabler use_absolute_position_embeddingsrrphidden_dropout_probrJ)r(r]rr+r-r.r!s   z!InternVLVisionEmbeddings.__init__rrrc Cs|jdd}|jjdd}tjs||kr||kr|jS|jddddf}|jddddf}|jd}||jd} ||jd} t|d} |d| | |}|dddd}t j j || | fdd d }|dddd dd|}tj ||fdd S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rNr/r?rrbicubicF)rxmode align_cornersrL)r<rr$jit is_tracingrrrypermuter"rR interpolaterzcat) r(rrrr num_positionsclass_pos_embedpatch_pos_embedrM new_height new_widthsqrt_num_positionsr-r-r.interpolate_pos_encoding$s(   z1InternVLVisionEmbeddings.interpolate_pos_encodingrbool_masked_poscCs|j\}}}}||\}\}}|\} } }|dur5|j| | d} |d| } |d| | | }|j| dd} tj | |fdd}|j durT|| |||}| |}|||ffS)Nr/rrL) r<rrxrexpand unsqueezetype_asrr$rrrrJ)r(rrr~rrrrrr|r} mask_tokensw cls_tokensr-r-r.r:Ls   z InternVLVisionEmbeddings.forwardN)r?r@rArrr!r$rintrr BoolTensorr:rBr-r-r+r.rs+rcs2eZdZfddZdejdejfddZZS)InternVLVisionMLPcsDt||_t|j|_t|j|j |_ t|j |j|_ dSr) r r!r]r hidden_act activation_fnr"rjr)intermediate_sizefc1fc2r(r]r+r-r.r!gs  zInternVLVisionMLP.__init__r7rcCs"||}||}||}|Sr)rrr)r(r7r-r-r.r:ns   zInternVLVisionMLP.forward)r?r@rAr!r$rr:rBr-r-r+r.rfs r) layer_normrms_normc s`eZdZdZdeddffdd Z d dejd ede e eje ejejfffd d Z Z S) rz?This corresponds to the Block class in the timm implementation.r]rNcst|j|_d|_t||_t||_t|j |j |j d|_ t|j |j |j d|_ |j}tj|t|j dd|_tj|t|j dd|_t|j|_dS)Nrr*T) requires_grad)r r!chunk_size_feed_forward seq_len_dimr\ attentionrmlpNORM2FN norm_typer)layer_norm_epslayernorm_beforelayernorm_afterrr"r#r$r%rrrprrJ)r(r] init_valuesr+r-r.r!{s   zInternVLVisionLayer.__init__Fr7rvcCsl|j|||d\}}|j|}||}||}||}||}|jdur.|j|}||}||fS)N)rv)rrrrrrJr)r(r7rvattention_outputattention_weights layer_outputr-r-r.r:s      zInternVLVisionLayer.forward)F) r?r@rArrr!r$rboolrr;r:rBr-r-r+r.rxsrc sTeZdZdeddffdd Ze  d dejded ede e e ffd d Z Z S) InternVLVisionEncoderr]rNcs:t|_tfddtjD|_d|_dS)Ncsg|]}tqSr-)r).0ir]r-r. sz2InternVLVisionEncoder.__init__..F) r r!r]r" ModuleListrangenum_hidden_layerslayergradient_checkpointingrr+rr.r!s   zInternVLVisionEncoder.__init__Fr7rvoutput_hidden_statesc Csz|rdnd}|r dnd}t|jD]\}}|r||f}|||}|d}|r.||df}q|r6||f}t|||dS)Nr-rrlast_hidden_stater7 attentions) enumeraterr ) r(r7rvrall_hidden_statesall_self_attentionsr layer_module layer_outputsr-r-r.r:s"     zInternVLVisionEncoder.forward)FF)r?r@rArr!rr$rrrr;r r:rBr-r-r+r.rs rcsteZdZdeddffdd ZddZee   ddej d e ej d e e d e e de eeff d d ZZS)InternVLVisionModelr]rNcsTt|||_t||_t||_|jrt ntj |j |j d|_ |dS)Nr)r r!r]rrrencoderuse_mean_poolingr"rqrr)r layernorm post_initrr+r-r.r!s    zInternVLVisionModel.__init__cC|jjSr)rrr=r-r-r.get_input_embeddingsz(InternVLVisionModel.get_input_embeddingsrrrvrc Csn|dur|n|jj}|dur|n|jj}|j||d\}}|j|||d}|d}||}t||j|jdS)z bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, num_patches)`, *optional*): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). N)r)rvrrr) r]rvrrr r rr7r) r(rrrvrembedding_outputr~encoder_outputssequence_outputr-r-r.r:s   zInternVLVisionModel.forward)NNN)r?r@rArr!rrrr$rrrrrr;rr:rBr-r-r+r.r s& r c@s@eZdZeZdZdZdZdZdZ dZ dZ dZ dZ dZddZdS)InternVLPreTrainedModelTpast_key_valuescCst|jd|jj}t|tjr)|jjj d|d|j dur'|j j dSdSt|tj r>|j j |jj ddSdS)NrrCrr)getattrr]get_text_configrrr"rjr&rrr_rrr)r(rDrr-r-r.rs    z%InternVLPreTrainedModel._init_weightsN)r?r@rArrrr_skip_keys_device_placement_supports_cache_classrr_supports_quantized_cache_supports_static_cacherrrr-r-r-r.rs rcs*eZdZdeffdd ZddZZS)InternVLMultiModalProjectorr]csztt|jjtd|jd|_t |jjtd|jd|j j|_ t |j |_t |j j|j j|_dS)Nrr)r r!r"r vision_configr)rdownsample_ratiorrj text_configlinear_1rprojector_hidden_actactlinear_2rr+r-r.r!s " z$InternVLMultiModalProjector.__init__cCs,||}||}||}||}|Sr)rr!r#r$)r(image_featuresr7r-r-r.r:(s    z#InternVLMultiModalProjector.forward)r?r@rArr!r:rBr-r-r+r.rs rzM Base class for InternVL outputs, with hidden states and attentions. c@s$eZdZUdZdZeejed<dS)InternVLModelOutputWithPasta past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder and after projecting the last hidden state. Nimage_hidden_states) r?r@rArr'rr$ FloatTensor__annotations__r-r-r-r.r&0s  r&zx The InternVL model which consists of a vision backbone and a language model, without a language modeling head. c#sReZdZddiZdeffdd ZddZdd Zd d Zd d Z  d'de j de e eeefde efddZee             d(de jde j de e jde e jde ee j de e j de e eeefde ede ede ede ede ede e jdeede eeffd d!Zd)d#e jd$efd%d&ZZS)* InternVLModelzlanguage_model.modellanguage_modelr]cs>t|t|j|_t||_t|j|_ | dSr) r r!r from_configr vision_towerrmulti_modal_projectorr r+r rr+r-r.r!Ns   zInternVLModel.__init__cC |jSr)r+rr=r-r-r.rV z"InternVLModel.get_input_embeddingscC|j|dSr)r+set_input_embeddingsr(rGr-r-r.r2Yz"InternVLModel.set_input_embeddingscC ||_dSrr+r(decoderr-r-r. set_decoder\r0zInternVLModel.set_decodercC|jSrr6r=r-r-r. get_decoder_zInternVLModel.get_decoderNrvision_feature_layervision_feature_select_strategyc Ks|dur|n|jj}|dur|n|jj}|jj}|dkr$|j|dj}n |j|dj|}|dkr>|ddddddf}|jd}t |d}|jd} | | ||d}|j ||d}| | d|jd}| |}|S) a% Obtains image last hidden states from the vision tower and apply multimodal projection. Args: pixel_values (`torch.FloatTensor]` of shape `(batch_size, channels, height, width)`) The tensors corresponding to the input images. vision_feature_layer (`int` or `list[int]`): Layer index or list of layer indices to extract features from. Returns: vision_features (`torch.Tensor`): Image feature tensor of shape `(num_images, image_length, embed_dim)`. Nr/)rdefaultrrr) scale_factor) r]r=r>rr-r vision_modelr7r<rry pixel_shuffler.) r(rr=r>rUrvision_featureschannels feature_sizer|r-r-r.get_image_featuresbs(    z InternVLModel.get_image_features input_idsrH position_idsr inputs_embeds use_cachervr return_dictcache_positionrUrc Ks| dur| n|jj} | dur| n|jj} | dur| n|jj} |dur$|n|jj}|dur.|n|jj}|du|duAr>td|durH||}|dur|j|||d}|durw||t j |jj t j |j dk}|jddjddd}n||jj kd}|||j }||jj k}ts|||kr||jj k}|jd|jd}td|d |||j |j}|||}|jd ||||| | | d | d |}t|j|j|j|j|dur|d Sdd S)Nz:You must specify exactly one of input_ids or inputs_embedsrr=r>)r1devicerrLrr/z6Image features and image tokens do not match: tokens: z , features T) rHrHrrIrJrvrrKrL)rrr7rr'r-)r]rvruse_return_dictr=r>rdrrFr$tensorimage_token_idlongrNsumr expand_asr2rnumelr<r1masked_scatterr+r&rrr7r)r(rGrrHrHrrIr=r>rJrvrrKrLrUr%special_image_maskn_image_tokensn_image_featuresrr-r-r.r:st     zInternVLModel.forwardrrCr@c Cs|\}}}}||dks||dkrtd|||t||t||}|dddd}||t||t||t||d}|dddd}|S)a&Perform pixel shuffle downsampling on vision features. Args: vision_features (`torch.Tensor`): Input tensor of shape (batch_size, width, height, channels). scale_factor (`float`, *optional*, defaults to `0.5`): Factor by which to downsample. Default is 0.5, which halves the dimensions. Returns: vision_features (`torch.Tensor`): Downsampled tensor of shape (batch_size, height*scale_factor, width*scale_factor, channels/(scale_factor^2)). rzKHeight and width must be divisible by scale_factor for proper downsampling.rrr)rxrdrzrrrT)r(rCr@r|rrrDr-r-r.rBs $zInternVLModel.pixel_shuffler) NNNNNNNNNNNNN)r)r?r@rA_checkpoint_conversion_mappingrr!rr2r9r;r$r(rrrliststrrFrr LongTensorrrrr r;r&r:floatrBrBr-r-r+r.r*Fs~ 5       Tr*zT Base class for InternVL causal language model (or autoregressive) outputs. c@seZdZUdZdZeejed<dZ eejed<dZ ee ejed<dZ ee ejed<dZee ejed<dZeejed<dS) InternVLCausalLMOutputWithPastaw loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder and after projecting the last hidden state. Nlosslogitsrr7rr')r?r@rArr`rr$r(r)rarr[r7r;rr'r-r-r-r.r_s r_c@s eZdZdS)KwargsForCausalLMN)r?r@rAr-r-r-r.rb0s rbzV The INTERNVL model which consists of a vision backbone and a language model. c)seZdZdddddZdgZdeffdd Zd d Zd d Zde j fddZ ddZ ddZ ddZ  d6dejdeeeeefdeefddZeddZedd Zed!d"Zee               # d7d$ejdejd%eejd&eejd'eeejd(eejdeeeeefdeed)eejd*eed+eed,eed-eed.eejd/eeejfd0eejd1e e!dee"e#ff$d2d3Z$      d8fd4d5 Z%Z&S)9 InternVLForConditionalGenerationzmodel.language_modelzmodel.vision_towerzmodel.multi_modal_projectorlm_head)z^language_model.modelz ^vision_towerz^multi_modal_projectorz^language_model.lm_headzlm_head.weightr]cs<t|t||_tj|jj|jjdd|_ | dS)NFr^) r r!r*modelr"rjr r) vocab_sizerdr rr+r-r.r!As   z)InternVLForConditionalGeneration.__init__cCr/r)rerr=r-r-r.rGr0z5InternVLForConditionalGeneration.get_input_embeddingscCr1r)rer2r3r-r-r.r2Jr4z5InternVLForConditionalGeneration.set_input_embeddingsrcCr:rrdr=r-r-r.get_output_embeddingsMr<z6InternVLForConditionalGeneration.get_output_embeddingscCr5rrg)r(new_embeddingsr-r-r.set_output_embeddingsPr0z6InternVLForConditionalGeneration.set_output_embeddingscCr1r)rer9r7r-r-r.r9Sr4z,InternVLForConditionalGeneration.set_decodercCrr)rer;r=r-r-r.r;Vrz,InternVLForConditionalGeneration.get_decoderNrr=r>cKs|jjd|||d|S)NrMr-)rerF)r(rr=r>rUr-r-r.rFYsz3InternVLForConditionalGeneration.get_image_featurescCrr)rer+r=r-r-r.r+hz/InternVLForConditionalGeneration.language_modelcCrr)rer-r=r-r-r.r-lrkz-InternVLForConditionalGeneration.vision_towercCrr)rer.r=r-r-r.r.prkz6InternVLForConditionalGeneration.multi_modal_projectorrrGrHrHrrIlabelsrJrvrrKrLlogits_to_keep image_sizesrUcKs| dur| n|jj} | dur| n|jj} | dur| n|jj} |dur$|n|jj}|dur.|n|jj}|jd||||||||| | | d||d|}|d}t|trXt | dn|}| |dd|ddf}d}| dur}|j d|| |jj j d|}t|||j|j|j|jdS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> import torch >>> from transformers import AutoProcessor, AutoModelForImageTextToText >>> torch_device = "cuda" >>> processor = AutoProcessor.from_pretrained("OpenGVLab/InternVL3-1B-hf") >>> model = AutoModelForImageTextToText.from_pretrained( ... "OpenGVLab/InternVL3-1B-hf", torch_dtype=torch.bfloat16, device_map=torch_device ... ) >>> messages = [ ... { ... "role": "user", ... "content": [ ... { ... "type": "image", ... "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg", ... }, ... { ... "type": "image", ... "url": "https://thumbs.dreamstime.com/b/golden-gate-bridge-san-francisco-purple-flowers-california-echium-candicans-36805947.jpg", ... }, ... {"type": "text", "text": "These images depict two different landmarks. Can you identify them?"}, ... ], ... }, ... ] >>> inputs = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt").to(torch_device) >>> generate_ids = model.generate(**inputs, max_new_tokens=200) >>> print(processor.decode(generate_ids[0, inputs["input_ids"].shape[1] :], skip_special_tokens=True)) The images depict the Statue of Liberty and the Golden Gate Bridge. ```NT)rGrrHrHrrIr=r>rJrvrrKrLrnr)rarlrf)r`rarr7rr'r-)r]rvrrOr=r>rerrslicerd loss_functionr rfr_rr7rr')r(rGrrHrHrrIr=r>rlrJrvrrKrLrmrnrUrr7 slice_indicesrar`r-r-r.r:ts\> z(InternVLForConditionalGeneration.forwardc  s8tj|f|||||d|} |ddkr|| d<| S)N)rrIrHrLrmrr)r prepare_inputs_for_generation) r(rGrrIrrHrLrmrU model_inputsr+r-r.rrs  z>InternVLForConditionalGeneration.prepare_inputs_for_generationr)NNNNNNNNNNNNNNrN)NNNNNN)'r?r@rArZ_tied_weights_keysrr!rr2r"Modulerhrjr9r;r$r(rrrr[r\rFpropertyr+r-r.rrr]rrrrbr;r_r:rrrBr-r-r+r.rc3s            src)rr rr*rc)rC)Bcollections.abcr dataclassesrtypingrrrr$torch.nnr" activationsr generationr integrationsr modeling_flash_attention_utilsr modeling_layersr modeling_outputsr r rmodeling_utilsrrprocessing_utilsrutilsrrrrrrautorconfiguration_internvlrrrurrr^r[r\rrrrrrrrrr rrr&r*r_rbrc__all__r-r-r-r.s           K% &^ 0&5G N