o
i @ s d dl mZmZ d dlZd dlmZ d dlmZ ddlm Z m
Z
ddl m Z ddl
mZmZmZ dd lmZmZmZmZ G d
d d eZeG d d
d
eZG dd deeZeddG dd deeZg dZdS ) )OptionalUnionN) IJepaConfig )BaseModelOutputWithPoolingImageClassifierOutput)Unpack)TransformersKwargsauto_docstring torch_int )
ViTEmbeddingsViTForImageClassificationViTModelViTPreTrainedModelc st e Zd Zddededdf fdd
Zdejd ed
edejfd d Z dd
ejde
ej dedejfddZ Z
S )IJepaEmbeddingsFconfiguse_mask_tokenreturnNc s6 t || | `| jj}ttd||j | _
d S )N ) super__init__ cls_tokenpatch_embeddings num_patchesnn Parametertorchrandn hidden_sizeposition_embeddings)selfr r r __class__ d/home/ubuntu/veenaModal/venv/lib/python3.10/site-packages/transformers/models/ijepa/modular_ijepa.pyr s zIJepaEmbeddings.__init__
embeddingsheightwidthc C s |j d }| jj d }tj s||kr||kr| jS | j}|j d }|| j }|| j } t|d }
|d|
|
|}|dddd}t j
j ||| fddd }|dddd dd|}|S )
a
This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution
images. This method is also adapted to support torch.jit tracing.
Adapted from:
- https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and
- https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211
r g ?r r r bicubicF)sizemode
align_corners)
shaper r jit
is_tracing
patch_sizer reshapepermuter
functional interpolateview) r! r& r' r( r
num_positionspatch_pos_embeddim
new_height new_widthsqrt_num_positionsr$ r$ r% interpolate_pos_encoding s&
z(IJepaEmbeddings.interpolate_pos_encoding pixel_valuesbool_masked_posr= c C s |j \}}}}| j||d}|d ur1|j d } | j|| d}
|d|
} |d| |
| }|r=|| ||| }n|| j }| |}|S )N)r= r r) ?) r. r
mask_tokenexpand unsqueezetype_asr= r dropout) r! r> r? r=
batch_size_r' r( r&
seq_length mask_tokensmaskr$ r$ r% forward= s
zIJepaEmbeddings.forward)F)NF)__name__
__module__ __qualname__r boolr r Tensorintr= r
BoolTensorrK
__classcell__r$ r$ r" r% r s *r c @ s. e Zd Zdeejejejf ddfddZdS )IJepaPreTrainedModelmoduler Nc C s t |tjtjfr0tjj|jjt j
d| j j d|jj
|j_|jdur.|jj dS dS t |tjrE|jj |jjd dS t |trotjj|jjt j
d| j j d|jj
|j_|jdurq|jj dS dS dS )zInitialize the weightsg )meanstdNr@ )
isinstancer LinearConv2dinit
trunc_normal_weightdatator float32r initializer_rangedtypebiaszero_ LayerNormfill_r r rA )r! rU r$ r$ r%
_init_weightsZ s0
z"IJepaPreTrainedModel._init_weights) rL rM rN r r rY rZ re rg r$ r$ r$ r% rT X s &rT c s, e Zd Zddededef fdd
Z ZS )
IJepaModelFr add_pooling_layerr c s$ t | || _t||d| _dS )z
add_pooling_layer (bool, *optional*, defaults to `True`):
Whether to add a pooling layer
use_mask_token (`bool`, *optional*, defaults to `False`):
Whether to use a mask token for masked image modeling.
)r N)r r r r r& )r! r ri r r" r$ r% r r s zIJepaModel.__init__)FF)rL rM rN r rO r rS r$ r$ r" r% rh q s $rh a
IJepa Model transformer with an image classification head on top (a linear layer on top of the final hidden states)
e.g. for ImageNet.
Note that it's possible to fine-tune IJepa on higher resolution images than the ones it has been trained on, by
setting `interpolate_pos_encoding` to `True` in the forward of the model. This will interpolate the pre-trained
position embeddings to the higher resolution.
) custom_introc
sh e Zd Zdef fdd Z d
deej deej deej dee d e e
d
e f d d Z Z
S )IJepaForImageClassificationr c s& t | t|dd| _| d S )NF)ri )r r rh ijepa post_init)r! r r" r$ r% r s z$IJepaForImageClassification.__init__Nr> head_masklabelsr= kwargsr c
K sh | j |f||d|}|j}| |jdd}d} |dur*| j||| jfi |} t| ||j|jdS )a
labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*):
Labels for computing the image classification/regression loss. Indices should be in `[0, ...,
config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If
`config.num_labels > 1` a classification loss is computed (Cross-Entropy).
)rn r= r )r9 N)losslogits
hidden_states
attentions) rl last_hidden_state
classifierrV
loss_functionr r rs rt )
r! r> rn ro r= rp outputssequence_outputrr rq r$ r$ r% rK s& z#IJepaForImageClassification.forward)NNNN)rL rM rN r r r r rP rO r r r rK rS r$ r$ r" r% rk ~ s& rk )rT rh rk )typingr r r torch.nnr -transformers.models.ijepa.configuration_ijepar modeling_outputsr r processing_utilsr utilsr r
r vit.modeling_vitr
r r r r rT rh rk __all__r$ r$ r$ r% s" J
*