o ij@sddlmZmZmZddlZddlmZddlmZddl m Z ddl m Z ddl mZmZddlmZmZdd lmZdd lmZmZmZd d lmZGd ddejZGdddejZGdddejZ d9dejdejdejdejdeejde de deefddZ!ddZ"d ejd!e#d"ejfd#d$Z$d%ejd&ejd'ejd(ejd"e%ejejff d)d*Z&Gd+d,d,ejZ'Gd-d.d.e Z(Gd/d0d0ejZ)Gd1d2d2ejZ*eGd3d4d4eZ+ed5d6Gd7d8d8e+Z,d4d8gZ-dS):)CallableOptionalUnionN)ACT2FN)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPooling)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstring torch_int)MLCDVisionConfigcs2eZdZfddZdejdejfddZZS)MLCDMLPcsDt||_t|j|_t|j|j |_ t|j |j|_ dSN) super__init__configr hidden_act activation_fnnnLinear hidden_sizeintermediate_sizefc1fc2selfr __class__Z/home/ubuntu/.local/lib/python3.10/site-packages/transformers/models/mlcd/modeling_mlcd.pyr%s  zMLCDMLP.__init__ hidden_statesreturncCs"||}||}||}|Sr)rrr)r!r&r$r$r%forward,s   zMLCDMLP.forward)__name__ __module__ __qualname__rtorchTensorr( __classcell__r$r$r"r%r$s rcsPeZdZUejed<d dededdffdd Zd ed edejfd d Z Z S)MLCDRotaryEmbeddinginv_freq@dimthetar'Ncs>td|tjd|dtjd|}|jd|dddS)N?rdtyper0F persistent)rrr,arangefloatregister_buffer)r!r2r3r0r"r$r%r6s  zMLCDRotaryEmbedding.__init__num_patches_heightnum_patches_widthc Cstj||jjddd|}tj||jjdd|d}tj||gdd}t||}tj||jj|jj d}t ||j}||d} | S)a} Calculate the Rotary Position Embedding (RoPE) for MLCDVisionModel based on the grid size. Args: num_patches_height (int): Number of patches in the height dimension. num_patches_width (int): Number of patches in the width dimension. Returns: torch.Tensor: Rotary positional embeddings for the given grid size. )devicerrr2)r?r7) r,r:r0r? unsqueezeexpandstackflattenmaxr7outer) r!r=r>hpos_idswpos_idspos_ids max_grid_sizeseqrotary_pos_emb_fullrotary_pos_embr$r$r%r(;s  zMLCDRotaryEmbedding.forward)r1) r)r*r+r,r-__annotations__intr;rr(r.r$r$r"r%r/3s  r/csVeZdZdeffdd Zdejdededejfdd Zd ej dejfd d Z Z S) MLCDVisionEmbeddingsrcst||_|j|_|j|_|j|_tt |j|_ tj |j |j|j|jdd|_|j|jd|_|jd|_|jdt |jddddS)NF) in_channels out_channels kernel_sizestridebiasr5r position_ids)rr@r8)rrrr embed_dim image_size patch_sizer Parameterr,randnclass_embeddingConv2d num_channelspatch_embedding num_patches num_positionsr<r:rCr r"r$r%r]s  "zMLCDVisionEmbeddings.__init__ embeddingsheightwidthr'c Cs|jdd}|jjd}|jdd}tjs(||kr(||kr(||jS|ddddf}|ddddf}|jd} ||j} ||j} t |d} | d| | | }| dddd}t j j|| | fdd d }| dddddd| }tj||fdd S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rrNr@g?rr5bicubicF)sizemode align_cornersrA)shapeposition_embeddingweightrBr,jit is_tracingrWrZrreshapepermuter functional interpolateviewcat) r!rcrdrerarkrbclass_pos_embedpatch_pos_embedr2 new_height new_widthsqrt_num_positionsr$r$r%interpolate_pos_encodingrs*      z-MLCDVisionEmbeddings.interpolate_pos_encoding pixel_valuescCs^|jd}|jjj}||j|d}|ddd}|j|dd}t j ||gdd}|S)Nrr6r5rr@rA) rjr`rlr7torE transposer]rCr,rt)r!r{ batch_size target_dtype patch_embeds class_embedsrcr$r$r%r(s  zMLCDVisionEmbeddings.forward) r)r*r+rrr,r-rPrz FloatTensorr(r.r$r$r"r%rQ\s)rQmodulequerykeyvalueattention_maskscalingdropoutkwargsc Kst||j}t||j} t||dd|} |dur3|ddddddd|jdf} | | } tjj| dtj d |j } tjj | ||j d} t| | } | dd} | | fS)Nr5rr@)r2r7)ptrainingr) repeat_kvnum_key_value_groupsr,matmulr}rjrrqsoftmaxfloat32r|r7rr contiguous) rrrrrrrr key_states value_states attn_weights causal_mask attn_outputr$r$r%eager_attention_forwards & rcCsH|dd|jddf}|d|jdddf}tj| |fddS)z*Rotates half the hidden dims of the input..Nr@r5rA)rjr,rt)xx1x2r$r$r% rotate_halfsrr&n_repr'cCs^|j\}}}}|dkr |S|dddddddddf|||||}||||||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) rN)rjrCro)r&rbatchnum_key_value_headsslenhead_dimr$r$r%rs 0rqkcossincCs|j}|j}||}}|d|d}}||t||}||t||}||}||}||fS)Nr)r7r;rBrr|)rrrr orig_q_dtype orig_k_dtypeq_embedk_embedr$r$r%apply_rotary_pos_emb_visions  rcspeZdZdZdeffdd Z d dejdeejejfde ejd e e d eeje ejff d d Z Z S) MLCDAttentionzMulti-headed attention with RoPE. Refer to papers: - Attention is all you need: https://huggingface.co/papers/1706.03762 - RoFormer: Enhanced Transformer with Rotary Position Embedding: https://huggingface.co/papers/2104.09864 rcst||_|j|_|j|_|j|j|_|j|j|jkr-td|jd|jd|jd|_ |j |_ d|_ t |j|j|_t |j|j|_t |j|j|_t |j|j|_|j|_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).F)rrrrrXnum_attention_heads num_headsr ValueErrorscaleattention_dropoutr is_causalrrk_projv_projq_projout_projrr r"r$r%rs&    zMLCDAttention.__init__Nr&position_embeddingsrrr'cKsn|jdd\}}|||||j|jf}|||||j|jf}|||||j|jf} |dd} |dd} t ||| | \}}| dddd }| dddd }| dddd } t } |j jdkrzt|j j} | |||| |f|jsdn|j|j|jd |\} }| dddd } | ||d} || } | ddd } | |fS) z#Input shape: Batch x Time x ChannelNr@rrr5reagerr)rrr)rjrrorrrrrBr;rrprrr_attn_implementationr rrrrrsr)r!r&rrrr~ seq_length query_statesrrrrattention_interfacerrr$r$r%r(s>     zMLCDAttention.forwardr)r)r*r+__doc__rrr,r-tuplerr rr(r.r$r$r"r%rsrc sdeZdZdeffdd Z  d dejdeejejfdeejd ee d eej f d d Z Z S)MLCDEncoderLayerrcsRt|j|_t||_tj|j|jd|_ t ||_ tj|j|jd|_ dS)Neps) rrrrXr self_attnr LayerNormlayer_norm_eps layer_norm1rmlp layer_norm2r r"r$r%r1s   zMLCDEncoderLayer.__init__NFr&rroutput_attentionsr'cCsd|}||}|j||||d\}}||}|}||}||}||}|f}|r0||f7}|S)a Args: hidden_states (`torch.FloatTensor`): Input to the layer of shape `(batch, seq_len, embed_dim)`. Represents the hidden states from the previous layer or the input embeddings. position_embeddings (`tuple[torch.Tensor, torch.Tensor]`): A tuple of two tensors, each of shape `(batch, seq_len, embed_dim)`. Represents absolute positional embeddings for the query and key in the attention mechanism. attention_mask (`torch.FloatTensor`): Attention mask of shape `(batch, 1, q_len, k_v_seq_len)` where padding elements are indicated by very large negative values. output_attentions (`bool`, *optional*, defaults to `False`): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. r&rrr)rrrr)r!r&rrrresidualroutputsr$r$r%r(9s"     zMLCDEncoderLayer.forward)NF) r)r*r+rrr,r-rrboolrr(r.r$r$r"r%r0s rcs~eZdZdZdeffdd Z    ddejdeej ej fde ej d e e d e e d e e d e ee ffd dZZS) MLCDEncoderz Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`MLCDEncoderLayer`]. Args: config: MLCDVisionConfig rcs:t|_tfddtjD|_d|_dS)z3Overwrite dummy `MLCDConfig` to `MLCDVisionConfig`.csg|]}tqSr$)r).0_rr$r% ssz(MLCDEncoder.__init__..FN) rrrr ModuleListrangenum_hidden_layerslayersgradient_checkpointingr r"rr%ros   zMLCDEncoder.__init__N inputs_embedsrrroutput_hidden_states return_dictr'c Cs|dur|n|jj}|dur|n|jj}|dur|n|jj}|r"dnd}|r(dnd}|} t|jD] \} } |r<|| f}| | |||d} | d} |rQ|| df}q1|rY|| f}|sgtdd| ||fDSt| ||dS) aj Args: inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. position_embeddings (`tuple[torch.Tensor, torch.Tensor]`): A tuple of two tensors, each of shape `(batch, seq_len, embed_dim)`. Represents absolute positional embeddings for the query and key in the attention mechanism. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. Nr$rrrcss|] }|dur|VqdSrr$)rvr$r$r% sz&MLCDEncoder.forward..)last_hidden_stater& attentions)rruse_return_dictr enumeraterrr ) r!rrrrrrencoder_statesall_attentionsr&idx encoder_layer layer_outputsr$r$r%r(vs:"    zMLCDEncoder.forwardNNNN)r)r*r+rrrr,rrr-rrrr r(r.r$r$r"r%rfs,  rcsheZdZdeffdd Ze    d deejdee dee dee d e e e ff d d Z ZS) MLCDVisionTransformerrcst||_|j}t||_tj||jd|_ t ||_ tj||jd|_ t |j|jd|_ttd|j|jd|_dS)Nrr5r)rrrrrQrcrrr pre_layrnormrencoderpost_layernormr/rvision_rotary_embeddingr[r,r\ class_pos_emb)r!rrXr"r$r%rs   $zMLCDVisionTransformer.__init__Nr{rrrr'cCs<|dur|n|jj}|dur|n|jj}|dur|n|jj}|dur&td|jd|jj}|jd|jj}|||}||j j }t j |j |gdd}t j ||fdd}| |f} ||} || } |j| | |||d} | d} | dddddf} || } |s| | f| ddSt| | | j| jdS) Nz You have to specify pixel_valuesrr@rrA)rrrrrr)r pooler_outputr&r)rrrrrrjrZrr|rr?r,rtrrrcrrrr r&r)r!r{rrrr=r>rNembrr&encoder_outputsr pooled_outputr$r$r%r(sB     zMLCDVisionTransformer.forwardr)r)r*r+rrrrr,rrrrr r(r.r$r$r"r%rs$  rc@s.eZdZUeed<dZdZdZdZddZ dS)MLCDPreTrainedModelrmlcdTcCs|jj}t|tr,|jj}tjj|jd|jd|dtjj|j j |jj |ddSt|t ru|jj}|jdd|jj d|}|jd|}tjj|jj |dtjj|jj |dtjj|jj |dtjj|jj |ddSt|tr|jj}|jjdd|jj d|}d|jjd|}tjj|jj |dtjj|jj |ddSt|tr|jj}|jj|jjdd|}tjj|jd|ddSt|tjr|jj|j jddSt|tjr|jdur|jjdSdSdS)zInitialize the weightsrr)meanstd)rr5r4N)rinitializer_factor isinstancerQrinitnormal_r]rXr`rlinitializer_rangerrrrrrrrrrrrrrrVdatazero_fill_r)r!rfactor in_proj_std out_proj_stdfc_std pos_emb_stdr$r$r% _init_weightss:        z!MLCDPreTrainedModel._init_weightsN) r)r*r+rrObase_model_prefixsupports_gradient_checkpointing_supports_flash_attn_supports_sdparr$r$r$r%rs  rzN The vision model from M_L_C_D without any head or projection on top. ) custom_introcseZdZUeed<dZdgZdeffdd Zdej fddZ e dde e jd e ed e ed e edeeeff d dZZS)MLCDVisionModelrr{rcs"t|t||_|dSr)rrr vision_model post_initr r"r$r%r.s   zMLCDVisionModel.__init__r'cCs |jjjSr)r rcr`)r!r$r$r%get_input_embeddings4s z$MLCDVisionModel.get_input_embeddingsNrrrcCsN|dur|n|jj}|dur|n|jj}|dur|n|jj}|j||||dS)a Example: ```python >>> import requests >>> from PIL import Image >>> from transformers import AutoProcessor, MLCDVisionModel >>> model = MLCDVisionModel.from_pretrained("DeepGlint-AI/mlcd-vit-bigG-patch14-448") >>> processor = AutoProcessor.from_pretrained("DeepGlint-AI/mlcd-vit-bigG-patch14-448") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**inputs, output_attentions=True) >>> features = outputs.last_hidden_state >>> print(f"Extracted features shape: {features.shape}") >>> print(f"Number of attention layers: {len(outputs.attentions)}") >>> print(f"Attention shape: {outputs.attentions[0].shape}") ```N)r{rrr)rrrrr )r!r{rrrr$r$r%r(7szMLCDVisionModel.forwardr)r)r*r+rrOmain_input_name_no_split_modulesrrModuler rrr,rrrrr r(r.r$r$r"r%r $s,  r )r).typingrrrr,torch.nnr activationsrmodeling_flash_attention_utilsrmodeling_layersrmodeling_outputsr r modeling_utilsr r processing_utilsr utilsrrrconfiguration_mlcdrrrr/rQr-r;rrrPrrrrrrrrr __all__r$r$r$r%sp      )S   M6V@' :