o eiL@sZddlZddlmZddlmZmZddlmZddlmZddl m Z ddl m Z dd l mZmZmZmZmZd d lmZmZd d lmZd d lmZd dlmZmZmZmZmZe e!Z"GdddeZ#GdddeZ$GdddeZ%GdddeZ&GdddeZ'GdddeZ(GdddeZ)GdddeZ*Gdd d eZ+gd!Z,dS)"N)nn)Cache DynamicCache)GenerationConfig)FlashAttentionKwargs)BaseModelOutputWithPooling)Unpack)TransformersKwargsauto_docstringcan_return_tupleloggingtorch_compilable_check)Idefics3ConfigIdefics3VisionConfig)Idefics3ImageProcessor)Idefics3ImageProcessorFast)Idefics3BaseModelOutputWithPast Idefics3ForConditionalGeneration Idefics3ModelIdefics3PreTrainedModelIdefics3VisionTransformerc@eZdZdZdZdS)SmolVLMVisionConfiga This is the configuration class to store the configuration of a [`SmolVLMVisionModel`]. It is used to instantiate a SmolVLM vision encoder according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the SigLIP checkpoint [google/siglip-so400m-patch14-384](https://huggingface.co/google/siglip-so400m-patch14-384) used in SmolVLM [HuggingFaceTB/SmolVLM2-2.2B-Instruct](https://huggingface.co/HuggingFaceTB/SmolVLM2-2.2B-Instruct). Configuration objects inherit from [`PreTrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PreTrainedConfig`] for more information. Args: hidden_size (`int`, *optional*, defaults to 1152): Dimensionality of the encoder layers and the pooler layer. intermediate_size (`int`, *optional*, defaults to 3072): Dimensionality of the "intermediate" (i.e., feed-forward) layer in the Transformer encoder. num_hidden_layers (`int`, *optional*, defaults to 12): Number of hidden layers in the Transformer encoder. num_attention_heads (`int`, *optional*, defaults to 16): Number of attention heads for each attention layer in the Transformer encoder. num_channels (`int`, *optional*, defaults to 3): Number of channels in the input images. image_size (`int`, *optional*, defaults to 224): The size (resolution) of each image. patch_size (`int`, *optional*, defaults to 32): The size (resolution) of each patch. hidden_act (`str` or `function`, *optional*, defaults to `"gelu_pytorch_tanh"`): The non-linear activation function (function or string) in the encoder and pooler. If string, `"gelu"`, `"relu"`, `"selu"` and `"gelu_new"` `"quick_gelu"` are supported. layer_norm_eps (`float`, *optional*, defaults to 1e-06): The epsilon used by the layer normalization layers. attention_dropout (`float`, *optional*, defaults to 0.0): The dropout ratio for the attention probabilities. initializer_range (`float`, *optional*, defaults to 0.02): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. Example: ```python >>> from transformers.models.smolvlm.modeling_smolvlm import SmolVLMVisionTransformer >>> from transformers.models.smolvlm.configuration_smolvlm import SmolVLMVisionConfig >>> # Initializing a SmolVLMVisionConfig with google/siglip-so400m-patch14-384 style configuration >>> configuration = SmolVLMVisionConfig() >>> # Initializing a SmolVLMVisionTransformer (with random weights) from the google/siglip-so400m-patch14-384 style configuration >>> model = SmolVLMVisionTransformer(configuration) >>> # Accessing the model configuration >>> configuration = model.config ```smolvlm_visionN__name__ __module__ __qualname____doc__ model_typer"r"i/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/smolvlm/modular_smolvlm.pyr(s3rc@ eZdZdS)SmolVLMPreTrainedModelNrrrr"r"r"r#r%_r%c@r$)SmolVLMVisionTransformerNr&r"r"r"r#r(cr'r(c@r) SmolVLMConfiga This is the configuration class to store the configuration of a [`SmolVLMModel`]. It is used to instantiate a SmolVLM model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the model of the SmolVLM [HuggingFaceTB/SmolVLM2-2.2B-Instruct](https://huggingface.co/HuggingFaceTB/SmolVLM2-2.2B-Instruct) architecture. Configuration objects inherit from [`PreTrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PreTrainedConfig`] for more information. Args: use_cache (`bool`, *optional*, defaults to `True`): Whether or not the model should cache the key/value pairs of the attention mechanism. Only relevant if `config.is_decoder=True`. image_token_id (`int`, *optional*, defaults to 128257): The id of the "image" token. tie_word_embeddings (`bool`, *optional*, defaults to `False`): Whether or not to tie the word embeddings with the token embeddings. vision_config (`IdeficsVisionConfig` or `dict`, *optional*, defaults to `IdeficsVisionConfig`): Custom vision config or dict for the vision tower text_config (`PreTrainedConfig` or `dict`, *optional*, defaults to `LlamaConfig`): Custom text config or dict for the text model scale_factor (`int`, *optional*, defaults to 2): The scale factor for the image encoder. pad_token_id (`int`, *optional*, defaults to 128002): The id of the padding token. Example: ```python >>> from transformers import SmolVLMModel, SmolVLMConfig >>> # Initializing configuration >>> configuration = SmolVLMConfig() >>> # Initializing a model from the configuration >>> model = SmolVLMModel(configuration) >>> # Accessing the model configuration >>> configuration = model.config ```smolvlmNrr"r"r"r#r)gs%r)c@r$)SmolVLMImageProcessorNr&r"r"r"r#r+r'r+c@r$)SmolVLMImageProcessorFastNr&r"r"r"r#r,r'r,c@r$)SmolVLMBaseModelOutputWithPastNr&r"r"r"r#r-r'r-c"@s$eZdZdZdejdejdejfddZee dd dd ej d ejd Bd e e d e eBfddZee dd ddejd Bdejd Bdejd Bded Bdej d Bd ej d Bd ejd Bdej d Bded Bded Bded Bded Bdejd Bd e ed e eBfddZd S) SmolVLMModelz A subclass of Idefics3Model. We do *not* remove or block the call to inputs_merger in forward. Instead, we override inputs_merger here with custom logic. input_ids inputs_embedsimage_hidden_statescCs|j\}}}|dur!||tj|jjtj|jdk}|d}n||jjk}|jdd}t t ||dkd||}tj j j |jddddd} | dd } |jd d} | d|} | d|} | d| }t|}|||| |ddf||<t|d ||}|S) Ndtypedevice).rdimrzCAt least one sample has tokens not divisible by patch_size.)r5r)value)shapeget_input_embeddingstorchtensorconfigimage_token_idlongr4sumrallr functionalpadcumsum unsqueeze zeros_likewhere)selfr/r0r1_ patch_size image_masknum_image_tokensblocks_per_sampleoffsets block_offsetrow_cum chunk_idx local_idx block_idx image_embeds merged_embedsr"r"r# inputs_mergers.         zSmolVLMModel.inputs_mergerzVEncodes images into continuous embeddings that can be forwarded to the language model.) custom_introN pixel_valuespixel_attention_maskkwargsreturnc sPj\}}}}}j|jdj||gjddRjdd} dkjdd| k} | dt| O<| |durZtj fd d d Dtj j d }n|j||g|jddR}|| }|j j j} |jd| | d } | jd| | d } | jdddk } |jd| dd|}|j}||}||_|S)a4 pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, image_size, image_size)`): The tensors corresponding to the input images. pixel_attention_mask (`torch.LongTensor`, *optional*): The attention mask indicating padded regions in the image. )r3rNr5g)r9r6rcsg|]}j|qSr")r:).0irYr"r# sz3SmolVLMModel.get_image_features..)rrr)sizer3r4) dimensionrcstep)r9r]T)rYpatch_attention_mask return_dictr")r:tor3viewnumelrAr<any contiguousonesboolr4r> vision_configrKunfold vision_modellast_hidden_state connector pooler_output)rIrYrZr[ batch_size num_images num_channelsheightwidthnb_values_per_imagereal_images_indsrKpatches_subgridrf image_outputsr1image_featuresr"rar#get_image_featuress8      zSmolVLMModel.get_image_featuresa Inputs fed to the model can have an arbitrary number of images. To account for this, pixel_values fed to the model have image padding -> (batch_size, max_num_images, 3, max_heights, max_widths) where max_num_images is the maximum number of images among the batch_size samples in the batch. Padding images are not needed beyond padding the pixel_values at the entrance of the model. For efficiency, we only pass through the vision_model's forward the real images by discarding the padding images i.e. pixel_values of size (image_batch_size, 3, height, width) where image_batch_size would be 7 when num_images_per_sample=[1, 3, 1, 2] and max_num_images would be 3. attention_mask position_idspast_key_values use_cacheoutput_attentionsoutput_hidden_statesrgcache_positionc Ks| dur| n|jj} | dur| n|jj} | dur| n|jj} | dur$| n|jj} |jr8|jjr8| r8t dd} |durB|j \}}n|durM|j \}}}nt d| r]|dur]t |jd}|durl|j ||j}|durx|durxt d|dur|j||ddj}||j}n |dur|j|j|jd}|dur|j|||d }|jd ||||| | | d| d |}t|j|j|j|j|d S) NzZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...Fz5You have to specify either input_ids or inputs_embeds)r>zMYou cannot specify both pixel_values and image_hidden_states at the same timeT)rgr2)r/r0r1) r0rrrrrrrgr)rrr hidden_states attentionsr1r")r>rrruse_return_dicttraining text_modelgradient_checkpointinglogger warning_oncer: ValueErrorrr;rhr4rrtr3rWr-rrrrr)rIr/rrrr0rYrZr1rrrrgrr[ru seq_lengthrJoutputsr"r"r#forwardsp     zSmolVLMModel.forward)N) NNNNNNNNNNNNN)rrrr r< LongTensorTensorrWr r FloatTensorr r tuplerrr BoolTensorrnrr-rr"r"r"r#r.s !4      r.cs0eZdZddiZfddZfddZZS)SmolVLMForConditionalGenerationzlm_head.weightz$model.text_model.embed_tokens.weightcsLt|t||_t||jj_tj |j j |j j dd|_ |dS)NF)bias)super__init__r.modelrfrom_model_configrgeneration_configrLinear text_config hidden_size vocab_sizelm_head post_init)rIr> __class__r"r#ras   z(SmolVLMForConditionalGeneration.__init__c stjdi|dS)a pixel_attention_mask (`torch.Tensor` of shape `(batch_size, image_size, image_size)`, *optional*): Mask to avoid performing attention on padding pixel indices. image_hidden_states (`torch.FloatTensor` of shape `(batch_size, num_channels, image_size, image_size)`): The hidden states of the image encoder after modality projection. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or `model.image_token_id`. Tokens with indices set to `model.image_token_id` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> import httpx >>> from io import BytesIO >>> import torch >>> from PIL import Image >>> from io import BytesIO >>> from transformers import AutoProcessor, AutoModelForImageTextToText >>> from transformers.image_utils import load_image >>> # Note that passing the image urls (instead of the actual pil images) to the processor is also possible >>> image1 = load_image("https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg") >>> image2 = load_image("https://cdn.britannica.com/59/94459-050-DBA42467/Skyline-Chicago.jpg") >>> image3 = load_image("https://cdn.britannica.com/68/170868-050-8DDE8263/Golden-Gate-Bridge-San-Francisco.jpg") >>> processor = AutoProcessor.from_pretrained("HuggingFaceTB/SmolVLM2-2.2B-Instruct") >>> model = AutoModelForImageTextToText.from_pretrained("HuggingFaceTB/SmolVLM2-2.2B-Instruct", dtype=torch.bfloat16, device_map="auto") >>> # Create inputs >>> messages = [ ... { ... "role": "user", ... "content": [ ... {"type": "video", "path": path/to/video}, ... {"type": "text", "text": "What is happening in this video?"}, ... ] ... } ... ] >>> inputs = processor.apply_chat_template([messages], add_generation_prompt=True) >>> # Generate >>> generated_ids = model.generate(**inputs, max_new_tokens=256) >>> generated_texts = processor.batch_decode(generated_ids, skip_special_tokens=True) >>> print(generated_texts) ```Nr")rr)rI super_kwargsrr"r#rhs2z'SmolVLMForConditionalGeneration.forward)rrr_tied_weights_keysrr __classcell__r"r"rr#r^s r)rr)r+r,rr%r.r()-r<r cache_utilsrr generationrmodeling_flash_attention_utilsrmodeling_outputsrprocessing_utilsr utilsr r r r ridefics3.configuration_idefics3rr"idefics3.image_processing_idefics3r'idefics3.image_processing_idefics3_fastridefics3.modeling_idefics3rrrrr get_loggerrrrr%r(r)r+r,r-r.r__all__r"r"r"r#s0        7)C ?