o eiGO@srddlmZddlZddlmZddlmZddlm Z ddl m Z m Z ddl mZmZddlmZdd lmZmZmZd d lmZmZmZmZmZmZmZd d lmZd d l m!Z!m"Z"e#e$Z%Gddde Z&GdddeZ'Gddde!Z(GdddeZ)GdddeZ*GdddeZ+GdddeZ,eGdddeZ-GdddeZ.Gd d!d!eZ/gd"Z0dS)#)CallableN)initialization)PreTrainedConfig)BaseModelOutputBaseModelOutputWithPooling)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringlogging)CLIPMLP CLIPAttention CLIPEncoderCLIPEncoderLayerCLIPVisionEmbeddingsCLIPVisionModelCLIPVisionTransformer)eager_attention_forward)VisionRotaryEmbeddingapply_rotary_pos_emb_visioncsDeZdZdZdZdZ        dfdd ZZS)MLCDVisionConfiga This is the configuration class to store the configuration of a [`MLCDVisionModel`]. It is used to instantiate a MLCD vision encoder according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the vision encoder of the MLCD [DeepGlint-AI/mlcd-vit-bigG-patch14-336](https://huggingface.co/DeepGlint-AI/mlcd-vit-bigG-patch14-336) architecture. Configuration objects inherit from [`PreTrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PreTrainedConfig`] for more information. Args: hidden_size (`int`, *optional*, defaults to 1664): Dimensionality of the encoder layers and the pooler layer. intermediate_size (`int`, *optional*, defaults to 8192): Dimensionality of the "intermediate" (i.e., feed-forward) layer in the Transformer encoder. projection_dim (`int`, *optional*, defaults to 1024): Dimensionality of text and vision projection layers. num_hidden_layers (`int`, *optional*, defaults to 48): Number of hidden layers in the Transformer encoder. num_attention_heads (`int`, *optional*, defaults to 16): Number of attention heads for each attention layer in the Transformer encoder. num_channels (`int`, *optional*, defaults to 3): The number of input channels. image_size (`int`, *optional*, defaults to 336): The size (resolution) of each image. patch_size (`int`, *optional*, defaults to 14): The size (resolution) of each patch. hidden_act (`str` or `function`, *optional*, defaults to `"gelu"`): The non-linear activation function (function or string) in the encoder and pooler. If string, `"gelu"`, `"relu"`, `"selu"` and `"gelu_new"` `"quick_gelu"` are supported. layer_norm_eps (`float`, *optional*, defaults to 1e-05): The epsilon used by the layer normalization layers. attention_dropout (`float`, *optional*, defaults to 0.0): The dropout ratio for the attention probabilities. initializer_range (`float`, *optional*, defaults to 0.02): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. initializer_factor (`float`, *optional*, defaults to 1.0): A factor for initializing all weight matrices (should be kept to 1, used internally for initialization testing). Example: ```python >>> from transformers import MLCDVisionConfig, MLCDVisionModel >>> # Initializing a MLCDVisionConfig with DeepGlint-AI/mlcd-vit-bigG-patch14-336 style configuration >>> configuration = MLCDVisionConfig() >>> # Initializing a MLCDVisionModel (with random weights) from the DeepGlint-AI/mlcd-vit-bigG-patch14-336 style configuration >>> model = MLCDVisionModel(configuration) >>> # Accessing the model configuration >>> configuration = model.config ```mlcd_vision_model vision_config 0rPgeluh㈵>{Gz??c sdtjdi|||_||_||_||_||_||_||_||_ | |_ | |_ | |_ | |_ | |_dS)N)super__init__ hidden_sizeintermediate_sizenum_hidden_layersnum_attention_headsnum_key_value_groups num_channels patch_size image_sizeinitializer_rangeinitializer_factorattention_dropoutlayer_norm_eps hidden_act)selfr+r,r-r.r/r0r2r1r7r6r5r3r4kwargs __class__r(c/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/mlcd/modular_mlcd.pyr*cs zMLCDVisionConfig.__init__) rrrrr rr!r"r#r$r%r&r')__name__ __module__ __qualname____doc__ model_typebase_config_keyr* __classcell__r(r(r:r<r)s$6rc@s eZdZdS)MLCDMLPN)r=r>r?r(r(r(r<rDsrDc@s$eZdZdededejfddZdS)MLCDRotaryEmbeddingnum_patches_heightnum_patches_widthreturnc Cstj||jjddd|}tj||jjdd|d}tj||gdd}t||}tj||jj|jj d}t ||j}||d} | S)a} Calculate the Rotary Position Embedding (RoPE) for MLCDVisionModel based on the grid size. Args: num_patches_height (int): Number of patches in the height dimension. num_patches_width (int): Number of patches in the width dimension. Returns: torch.Tensor: Rotary positional embeddings for the given grid size. )devicer rdim)rIdtype) torcharangeinv_freqrI unsqueezeexpandstackflattenmaxrMouter) r8rFrGhpos_idswpos_idspos_ids max_grid_sizeseqrotary_pos_emb_fullrotary_pos_embr(r(r<forwards  zMLCDRotaryEmbedding.forwardN)r=r>r?intrNTensorr^r(r(r(r<rEsrEcs8eZdZdeffdd ZdejdejfddZZ S)MLCDVisionEmbeddingsconfigcst||`dSN)r)r*position_embeddingr8rbr:r(r<r*s zMLCDVisionEmbeddings.__init__ pixel_valuesrHcCs^|jd}|jjj}||j|d}|ddd}|j|dd}t j ||gdd}|S)NrrMrr rJrK) shapepatch_embeddingweightrMtorT transposeclass_embeddingrRrNcat)r8rf batch_size target_dtype patch_embeds class_embeds embeddingsr(r(r<r^s  zMLCDVisionEmbeddings.forward) r=r>r?rr*rN FloatTensorr`r^rCr(r(r:r<rasracspeZdZdZdeffdd Z d dejdeejejfdejdBd e e d eejejdBff d d Z Z S) MLCDAttentionzMulti-headed attention with RoPE. Refer to papers: - Attention is all you need: https://huggingface.co/papers/1706.03762 - RoFormer: Enhanced Transformer with Rotary Position Embedding: https://huggingface.co/papers/2104.09864 rbcst||j|_d|_dS)NF)r)r*r/ is_causalrer:r(r<r*s  zMLCDAttention.__init__N hidden_statesposition_embeddingsattention_maskr9rHcKsb|jdd\}}|||||j|jf}|||||j|jf}|||||j|jf} |dd} |dd} t ||| | \}}| dddd }| dddd }| dddd } t |jjt} | |||| |f|jsdn|j|j|jd|\} }| dddd } | ||d} || } | ddd } | |fS)NrJrr rrr%)dropoutscalingrv)rhq_projreshape num_headshead_dimk_projv_projrQfloatrpermute contiguousr get_interfacerb_attn_implementationrtrainingrzscalervviewout_proj)r8rwrxryr9ro seq_length query_states key_states value_statescossinattention_interface attn_output attn_weightsr(r(r<r^s>   zMLCDAttention.forwardrc) r=r>r?r@rr*rNr`tupler r r^rCr(r(r:r<rus ruc sbeZdZdeffdd Z d dejdeejejfdejdBdee d eej f d d Z Z S) MLCDEncoderLayerrbcst|t||_dSrc)r)r*ru self_attnrer:r(r<r*s zMLCDEncoderLayer.__init__Nrwrxryr9rHcKsV|}||}|jd|||d|\}}||}|}||}||}||}|S)a Args: hidden_states (`torch.FloatTensor`): Input to the layer of shape `(batch, seq_len, embed_dim)`. Represents the hidden states from the previous layer or the input embeddings. position_embeddings (`tuple[torch.Tensor, torch.Tensor]`): A tuple of two tensors, each of shape `(batch, seq_len, embed_dim)`. Represents absolute positional embeddings for the query and key in the attention mechanism. attention_mask (`torch.FloatTensor`): Attention mask of shape `(batch, 1, q_len, k_v_seq_len)` where padding elements are indicated by very large negative values. )rwrxryNr() layer_norm1r layer_norm2mlp)r8rwrxryr9residual_r(r(r<r^s    zMLCDEncoderLayer.forwardrc) r=r>r?rr*rNr`rr r rtr^rCr(r(r:r<rsrc sdeZdZdZdeffdd Z d dejdeej ej fdej dBd e e d ee Bf d d Z ZS) MLCDEncoderz Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`MLCDEncoderLayer`]. Args: config: MLCDVisionConfig rbcst|dS)z3Overwrite dummy `MLCDConfig` to `MLCDVisionConfig`.N)r)r*rer:r(r<r*,szMLCDEncoder.__init__N inputs_embedsrxryr9rHcKs.|}|jD] }||||fi|}qt|dS)a= Args: inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. position_embeddings (`tuple[torch.Tensor, torch.Tensor]`): A tuple of two tensors, each of shape `(batch, seq_len, embed_dim)`. Represents absolute positional embeddings for the query and key in the attention mechanism. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) )last_hidden_state)layersr)r8rrxryr9rw encoder_layerr(r(r<r^0s zMLCDEncoder.forwardrc)r=r>r?r@rr*rNrtrr`r r rr^rCr(r(r:r<r#src@sLeZdZUeed<dZdZdZdZdZ dZ dZ e e dZeddZdS) MLCDPreTrainedModelrbmlcdTF)rw attentionscCsL|jj}t|tr;|jj}tj|jd|jd|dtj|jj |jj |dt |j t |j jdddSt|tr|jj}|jdd|jjd|}|jd|}tj|jj |dtj|jj |dtj|jj |dtj|jj |ddSt|tr|jj}|jjdd|jjd|}d|jjd|}tj|jj |dtj|jj |ddSt|tr|jj}|jj|jjdd|}tj|jd|ddSt|tjrt |j!t"|j dSt|tj#r|j!durt |j!dSt|t$r$d |j%t jd |j&dt j'd |j&}t |j(|dSdS) zInitialize the weightsr%g)meanstd)rrJ)r rJrNr'rrg))rbr4 isinstancerainitnormal_rm embed_dimrirjr3copy_ position_idsrNrOrhrRrur-r|rrrrDr+fc1fc2MLCDVisionTransformerr. class_pos_embnn LayerNormzeros_biasones_LinearrEthetarLrrP)r8modulefactor in_proj_std out_proj_stdfc_std pos_emb_stdrPr(r(r< _init_weightscsB &       &z!MLCDPreTrainedModel._init_weightsN)r=r>r?r__annotations__base_model_prefixsupports_gradient_checkpointingaccepts_loss_kwargs_supports_flash_attn_supports_sdpa_supports_flex_attn_supports_attention_backendrru_can_record_outputsrNno_gradrr(r(r(r<rTs rcsJeZdZdeffdd Z d dejdBdeede e Bfdd Z Z S) rrbcsFt|t|j|jd|_tt d|j|jd|_ dS)Nrr ) r)r*rEr+r.vision_rotary_embeddingr ParameterrNrandnrrer:r(r<r*s $zMLCDVisionTransformer.__init__Nrfr9rHc Ks|durtd|jd|jj}|jd|jj}|||}||jj}tj |j|gdd}tj ||fdd}| | f}| |}| |}|jd||d|} | d} | dddddf} || } t| | dS) Nz You have to specify pixel_valuesrJrrK)rrx)r pooler_outputr() ValueErrorrhrbr1rrkrrIrNrnrrrs pre_layrnormencoderpost_layernormr) r8rfr9rFrGr]embrxrwencoder_outputsr pooled_outputr(r(r<r^s0    zMLCDVisionTransformer.forwardrc) r=r>r?rr*rNrtr r rrr^rCr(r(r:r<rsrc@s4eZdZ ddejdBdeedeeBfddZ dS)MLCDVisionModelNrfr9rHcKs|jdd|i|S)a Example: ```python >>> import httpx >>> from io import BytesIO >>> from PIL import Image >>> from transformers import AutoProcessor, MLCDVisionModel >>> model = MLCDVisionModel.from_pretrained("DeepGlint-AI/mlcd-vit-bigG-patch14-448") >>> processor = AutoProcessor.from_pretrained("DeepGlint-AI/mlcd-vit-bigG-patch14-448") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> inputs = processor(images=image, return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**inputs, output_attentions=True) >>> features = outputs.last_hidden_state >>> print(f"Extracted features shape: {features.shape}") >>> print(f"Number of attention layers: {len(outputs.attentions)}") >>> print(f"Attention shape: {outputs.attentions[0].shape}") ```rfNr() vision_model)r8rfr9r(r(r<r^s zMLCDVisionModel.forwardrc) r=r>r?rNrtr r rrr^r(r(r(r<rsr)rrr)1collections.abcrrNtorch.nnrrrconfiguration_utilsrmodeling_outputsrrmodeling_utilsrr processing_utilsr utilsr r r clip.modeling_cliprrrrrrrllama.modeling_llamarqwen2_vl.modeling_qwen2_vlrr get_loggerr=loggerrrDrErarurrrrr__all__r(r(r(r<s2     $  \"<*13) $