o
ei @ s d dl Z d dlmZ d dlmZ ddlmZ ddlm Z m
Z
ddl m Z ddl
mZmZmZ dd lmZmZmZmZ G d
d d eZeG d d
d
eZG dd deeZeddG dd deeZg dZdS ) N) IJepaConfig )initialization)BaseModelOutputWithPoolingImageClassifierOutput)Unpack)TransformersKwargsauto_docstring torch_int )
ViTEmbeddingsViTForImageClassificationViTModelViTPreTrainedModelc st e Zd Zddededdf fdd
Zdejd ed
edejfd d Z dd
ejdej
dB dedejfddZ Z S )IJepaEmbeddingsFconfiguse_mask_tokenreturnNc s6 t || | `| jj}ttd||j | _
d S )N ) super__init__ cls_tokenpatch_embeddings num_patchesnn Parametertorchrandn hidden_sizeposition_embeddings)selfr r r __class__ e/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/ijepa/modular_ijepa.pyr s zIJepaEmbeddings.__init__
embeddingsheightwidthc C s |j d }| jj d }tj s||kr||kr| jS | j}|j d }|| j }|| j } t|d }
|d|
|
|}|dddd}t j
j ||| fddd }|dddd dd|}|S )
a
This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution
images. This method is also adapted to support torch.jit tracing.
Adapted from:
- https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and
- https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211
r g ?r r r bicubicF)sizemode
align_corners)
shaper r jit
is_tracing
patch_sizer
reshapepermuter
functional interpolateview) r r% r& r' r
num_positionspatch_pos_embeddim
new_height new_widthsqrt_num_positionsr# r# r$ interpolate_pos_encoding s&
z(IJepaEmbeddings.interpolate_pos_encoding pixel_valuesbool_masked_posr< c C s |j \}}}}| j||d}|d ur1|j d } | j|| d}
|d|
} |d| |
| }|r=|| ||| }n|| j }| |}|S )N)r< r r( g ?) r- r
mask_tokenexpand unsqueezetype_asr< r dropout) r r= r> r<
batch_size_r& r' r%
seq_length mask_tokensmaskr# r# r$ forward< s
zIJepaEmbeddings.forward)F)NF)
__name__
__module__ __qualname__r boolr r Tensorintr<
BoolTensorrI
__classcell__r# r# r! r$ r
s *r c @ s4 e Zd Ze dejejB ejB ddfddZ dS )IJepaPreTrainedModelmoduler Nc C s t |tjtjfr#tj|jd| jjd |j dur!t
|j dS dS t |tj r7t
|j t |j dS t |t
rTtj|jd| jjd |jdurVt
|j dS dS dS )zInitialize the weightsg )meanstdN)
isinstancer LinearConv2dinit
trunc_normal_weightr initializer_rangebiaszeros_ LayerNormones_r r r? )r rS r# r# r$
_init_weightsY s
z"IJepaPreTrainedModel._init_weights)
rJ rK rL r no_gradr rW rX r_ ra r# r# r# r$ rR W s &rR c s, e Zd Zddededef fdd
Z ZS )
IJepaModelFr add_pooling_layerr c s$ t | || _t||d| _dS )z
add_pooling_layer (bool, *optional*, defaults to `True`):
Whether to add a pooling layer
use_mask_token (`bool`, *optional*, defaults to `False`):
Whether to use a mask token for masked image modeling.
)r N)r r r r r% )r r rd r r! r# r$ r j s zIJepaModel.__init__)FF)rJ rK rL r rM r rQ r# r# r! r$ rc i s $rc a
IJepa Model transformer with an image classification head on top (a linear layer on top of the final hidden states)
e.g. for ImageNet.
Note that it's possible to fine-tune IJepa on higher resolution images than the ones it has been trained on, by
setting `interpolate_pos_encoding` to `True` in the forward of the model. This will interpolate the pre-trained
position embeddings to the higher resolution.
) custom_introc s\ e Zd Zdef fdd Z d dejdB dejdB dedB dee d e
f
d
d Z Z S )
IJepaForImageClassificationr c s& t | t|dd| _| d S )NF)rd )r r rc ijepa post_init)r r r! r# r$ r s z$IJepaForImageClassification.__init__Nr= labelsr< kwargsr c K sf | j |fd|i|}|j}| |jdd}d}|dur)| j||| jfi |}t|||j|jdS )a
labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*):
Labels for computing the image classification/regression loss. Indices should be in `[0, ...,
config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If
`config.num_labels > 1` a classification loss is computed (Cross-Entropy).
r< r )r8 N)losslogits
hidden_states
attentions) rg last_hidden_state
classifierrT
loss_functionr r rm rn ) r r= ri r< rj outputssequence_outputrl rk r# r# r$ rI s$ z#IJepaForImageClassification.forward)NNN)
rJ rK rL r r r rN rM r r r rI rQ r# r# r! r$ rf v s rf )rR rc rf )r torch.nnr -transformers.models.ijepa.configuration_ijepar r rY modeling_outputsr r processing_utilsr utilsr r r
vit.modeling_vitr r
r r r rR rc rf __all__r# r# r# r$ s" J
(