o eii`@sddlmZddlZddlmZddlmZddlm Z ddl m Z ddl m Z mZddlmZmZdd lmZdd lmZmZmZdd lmZdd lmZd dlmZGdddejZGdddejZ GdddejZ! d;dejdej"dej"dej"dej"dBde#de#deefddZ$d d!Z%d"ej"d#e&d$ej"fd%d&Z'd'ej"d(ej"d)ej"d*ej"d$e(ej"ej"ff d+d,Z)Gd-d.d.ejZ*Gd/d0d0e Z+Gd1d2d2ejZ,eGd3d4d4eZ-Gd5d6d6e-Z.ed7d8Gd9d:d:e-Z/d4d:gZ0dS)<)CallableN)initialization)ACT2FN)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPooling)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstring torch_int)merge_with_config_defaults)capture_outputs)MLCDVisionConfigcs2eZdZfddZdejdejfddZZS)MLCDMLPcsDt||_t|j|_t|j|j |_ t|j |j|_ dSN) super__init__configr hidden_act activation_fnnnLinear hidden_sizeintermediate_sizefc1fc2selfr __class__d/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/mlcd/modeling_mlcd.pyr&s  zMLCDMLP.__init__ hidden_statesreturncCs"||}||}||}|Sr)rrr)r!r&r$r$r%forward-s   zMLCDMLP.forward)__name__ __module__ __qualname__rtorchTensorr( __classcell__r$r$r"r%r%s rcsPeZdZUejed<d dededdffdd Zd ed edejfd d Z Z S)MLCDRotaryEmbeddinginv_freq@dimthetar'NcsJt||_||_d|tjd|dtjd|}|jd|dddS)N?rdtyper0F persistent)rrr2r3r,arangefloatregister_buffer)r!r2r3r0r"r$r%r7s  zMLCDRotaryEmbedding.__init__num_patches_heightnum_patches_widthc Cstj||jjddd|}tj||jjdd|d}tj||gdd}t||}tj||jj|jj d}t ||j}||d} | S)a} Calculate the Rotary Position Embedding (RoPE) for MLCDVisionModel based on the grid size. Args: num_patches_height (int): Number of patches in the height dimension. num_patches_width (int): Number of patches in the width dimension. Returns: torch.Tensor: Rotary positional embeddings for the given grid size. )devicerrr2)r?r7) r,r:r0r? unsqueezeexpandstackflattenmaxr7outer) r!r=r>hpos_idswpos_idspos_ids max_grid_sizeseqrotary_pos_emb_fullrotary_pos_embr$r$r%r(>s  zMLCDRotaryEmbedding.forward)r1) r)r*r+r,r-__annotations__intr;rr(r.r$r$r"r%r/4s  r/csVeZdZdeffdd Zdejdededejfdd Zd ej dejfd d Z Z S) MLCDVisionEmbeddingsrcst||_|j|_|j|_|j|_tt |j|_ tj |j |j|j|jdd|_|j|jd|_|jd|_|jdt |jddddS)NF) in_channels out_channels kernel_sizestridebiasr5r position_idsrr@r8)rrrr embed_dim image_size patch_sizer Parameterr,randnclass_embeddingConv2d num_channelspatch_embedding num_patches num_positionsr<r:rCr r"r$r%r`s  "zMLCDVisionEmbeddings.__init__ embeddingsheightwidthr'c Cs|jdd}|jjd}|jdd}tjs(||kr(||kr(||jS|ddddf}|ddddf}|jd} ||j} ||j} t |d} | d| | | }| dddd}t j j|| | fdd d }| dddddd| }tj||fdd S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rrNr@g?rr5bicubicF)sizemode align_cornersrA)shapeposition_embeddingweightrBr,jit is_tracingrWr[rreshapepermuter functional interpolateviewcat) r!rdrerfrbrlrcclass_pos_embedpatch_pos_embedr2 new_height new_widthsqrt_num_positionsr$r$r%interpolate_pos_encodingus*      z-MLCDVisionEmbeddings.interpolate_pos_encoding pixel_valuescCs^|jd}|jjj}||j|d}|ddd}|j|dd}t j ||gdd}|S)Nrr6r5rr@rA) rkrarmr7torE transposer^rCr,ru)r!r| batch_size target_dtype patch_embeds class_embedsrdr$r$r%r(s  zMLCDVisionEmbeddings.forward) r)r*r+rrr,r-rPr{ FloatTensorr(r.r$r$r"r%rQ_s)rQmodulequerykeyvalueattention_maskscalingdropoutkwargsc Kst||j}t||j} t||dd|} |dur | |} tjj| dtjd |j } tjj | ||j d} t| | } | dd } | | fS)Nr5rr@)r2r7)ptrainingr) repeat_kvnum_key_value_groupsr,matmulr~rrrsoftmaxfloat32r}r7rr contiguous) rrrrrrrr key_states value_states attn_weights attn_outputr$r$r%eager_attention_forwards  rcCsH|dd|jddf}|d|jdddf}tj| |fddS)z*Rotates half the hidden dims of the input..Nr@r5rA)rkr,ru)xx1x2r$r$r% rotate_halfsrr&n_repr'cCs^|j\}}}}|dkr |S|dddddddddf|||||}||||||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) rN)rkrCrp)r&rbatchnum_key_value_headsslenhead_dimr$r$r%rs 0rqkcossincCs|j}|j}||}}|d|d}}||t||}||t||}||}||}||fS)N)r7r;rBrr})rrrr orig_q_dtype orig_k_dtypeq_embedk_embedr$r$r%apply_rotary_pos_emb_visions  rcspeZdZdZdeffdd Z d dejdeejejfdejdBd e e d eejejdBff d d Z Z S) MLCDAttentionzMulti-headed attention with RoPE. Refer to papers: - Attention is all you need: https://huggingface.co/papers/1706.03762 - RoFormer: Enhanced Transformer with Rotary Position Embedding: https://huggingface.co/papers/2104.09864 rcst||_|j|_|j|_|j|j|_|j|j|jkr-td|jd|jd|jd|_ |j |_ d|_ t |j|j|_t |j|j|_t |j|j|_t |j|j|_|j|_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).F)rrrrrYnum_attention_heads num_headsr ValueErrorscaleattention_dropoutr is_causalrrk_projv_projq_projout_projrr r"r$r%rs&    zMLCDAttention.__init__Nr&position_embeddingsrrr'cKsb|jdd\}}|||||j|jf}|||||j|jf}|||||j|jf} |dd} |dd} t ||| | \}}| dddd }| dddd }| dddd } t |jjt} | |||| |f|jsdn|j|j|jd|\} }| dddd } | ||d} || } | ddd } | |fS) z#Input shape: Batch x Time x ChannelNr@rrr5rr)rrr)rkrrprrrrrBr;rrqrr get_interfacer_attn_implementationrrrrrrtr)r!r&rrrr seq_length query_statesrrrrattention_interfacerrr$r$r%r(s>   zMLCDAttention.forwardr) r)r*r+__doc__rrr,r-tupler r r(r.r$r$r"r%rsrc sbeZdZdeffdd Z d dejdeejejfdejdBdee d eej f d d Z Z S) MLCDEncoderLayerrcsRt|j|_t||_tj|j|jd|_ t ||_ tj|j|jd|_ dS)Neps) rrrrYr self_attnr LayerNormlayer_norm_eps layer_norm1rmlp layer_norm2r r"r$r%r3s   zMLCDEncoderLayer.__init__Nr&rrrr'cKsV|}||}|jd|||d|\}}||}|}||}||}||}|S)a Args: hidden_states (`torch.FloatTensor`): Input to the layer of shape `(batch, seq_len, embed_dim)`. Represents the hidden states from the previous layer or the input embeddings. position_embeddings (`tuple[torch.Tensor, torch.Tensor]`): A tuple of two tensors, each of shape `(batch, seq_len, embed_dim)`. Represents absolute positional embeddings for the query and key in the attention mechanism. attention_mask (`torch.FloatTensor`): Attention mask of shape `(batch, 1, q_len, k_v_seq_len)` where padding elements are indicated by very large negative values. )r&rrNr$)rrrr)r!r&rrrresidual_r$r$r%r(;s    zMLCDEncoderLayer.forwardr) r)r*r+rrr,r-rr r rr(r.r$r$r"r%r2s rc sdeZdZdZdeffdd Z d dejdeej ej fdej dBd e e d ee Bf d d Z ZS) MLCDEncoderz Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`MLCDEncoderLayer`]. Args: config: MLCDVisionConfig rcs:t|_tfddtjD|_d|_dS)z3Overwrite dummy `MLCDConfig` to `MLCDVisionConfig`.csg|]}tqSr$)r).0rrr$r% msz(MLCDEncoder.__init__..FN) rrrr ModuleListrangenum_hidden_layerslayersgradient_checkpointingr r"rr%ris   zMLCDEncoder.__init__N inputs_embedsrrrr'cKs.|}|jD] }||||fi|}qt|dS)a= Args: inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. position_embeddings (`tuple[torch.Tensor, torch.Tensor]`): A tuple of two tensors, each of shape `(batch, seq_len, embed_dim)`. Represents absolute positional embeddings for the query and key in the attention mechanism. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) )last_hidden_state)rr)r!rrrrr& encoder_layerr$r$r%r(ps zMLCDEncoder.forwardr)r)r*r+rrrr,rrr-r r rr(r.r$r$r"r%r`s rc@sLeZdZUeed<dZdZdZdZdZ dZ dZ e e dZeddZdS) MLCDPreTrainedModelrmlcdTF)r& attentionscCsL|jj}t|tr;|jj}tj|jd|jd|dtj|jj |jj |dt |j t |j jdddSt|tr|jj}|jdd|jjd|}|jd|}tj|jj |dtj|jj |dtj|jj |dtj|jj |ddSt|tr|jj}|jjdd|jjd|}d|jjd|}tj|jj |dtj|jj |ddSt|tr|jj}|jj|jjdd|}tj|jd|ddSt|tjrt |j!t"|j dSt|tj#r|j!durt |j!dSt|t$r$d |j%t jd |j&dt j'd |j&}t |j(|dSdS) zInitialize the weightsrr)meanstd)rr@rXr5Nr4rr6))rinitializer_factor isinstancerQinitnormal_r^rYrarminitializer_rangecopy_rWr,r:rkrCrrrrrrrrrrMLCDVisionTransformerr class_pos_embrrzeros_rVones_rr/r3r2r;r0)r!rfactor in_proj_std out_proj_stdfc_std pos_emb_stdr0r$r$r% _init_weightssB &       &z!MLCDPreTrainedModel._init_weightsN)r)r*r+rrObase_model_prefixsupports_gradient_checkpointingaccepts_loss_kwargs_supports_flash_attn_supports_sdpa_supports_flex_attn_supports_attention_backendrr_can_record_outputsr,no_gradrr$r$r$r%rs rc steZdZUeed<dZdZdgZdeffdd Ze e dde dde j d Bd eed eeBfd d ZZS)rrr|imagercst|||_|j}t||_tj||jd|_ t ||_ tj||jd|_ t |j|jd|_ttd|j|jd|_|dS)Nrr5r)rrrrrQrdrrr pre_layrnormrencoderpost_layernormr/rvision_rotary_embeddingr\r,r]r post_init)r!rrYr"r$r%rs     zMLCDVisionTransformer.__init__F)tie_last_hidden_statesNrr'c Ks|durtd|jd|jj}|jd|jj}|||}||jj}tj |j|gdd}tj ||fdd}| | f}| |}| |}|jd||d|} | d} | dddddf} || } t| | dS) Nz You have to specify pixel_valuesrr@rrA)rr)r pooler_outputr$)rrkrr[rr}rr?r,rurrrdrrrr) r!r|rr=r>rNembrr&encoder_outputsr pooled_outputr$r$r%r(s0    zMLCDVisionTransformer.forwardr)r)r*r+rrOmain_input_nameinput_modalities_no_split_modulesrrrr r,rr r rrr(r.r$r$r"r%rs"  rzN The vision model from M_L_C_D without any head or projection on top. ) custom_introc sveZdZUeed<dZdZdgZdeffdd Zde j fdd Z e dde jd Bd eedeeBfd d ZZS)MLCDVisionModelrr|rrcs"t|t||_|dSr)rrr vision_modelrr r"r$r%r s   zMLCDVisionModel.__init__r'cCs |jjjSr)rrdra)r!r$r$r%get_input_embeddingss z$MLCDVisionModel.get_input_embeddingsNrcKs|jdd|i|S)a Example: ```python >>> import httpx >>> from io import BytesIO >>> from PIL import Image >>> from transformers import AutoProcessor, MLCDVisionModel >>> model = MLCDVisionModel.from_pretrained("DeepGlint-AI/mlcd-vit-bigG-patch14-448") >>> processor = AutoProcessor.from_pretrained("DeepGlint-AI/mlcd-vit-bigG-patch14-448") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> inputs = processor(images=image, return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**inputs, output_attentions=True) >>> features = outputs.last_hidden_state >>> print(f"Extracted features shape: {features.shape}") >>> print(f"Number of attention layers: {len(outputs.attentions)}") >>> print(f"Attention shape: {outputs.attentions[0].shape}") ```r|Nr$)r)r!r|rr$r$r%r(s zMLCDVisionModel.forwardr)r)r*r+rrOrrrrrModulerr r,rr r rrr(r.r$r$r"r%rs r)r)1collections.abcrr,torch.nnrrr activationsrmodeling_layersrmodeling_outputsrrmodeling_utilsr r processing_utilsr utilsr r r utils.genericrutils.output_capturingrconfiguration_mlcdrr rr/rQr-r;rrrPrrrrrrrrr__all__r$r$r$r%st         +S   M.439 3