o
wi)% @ s d dl mZmZ d dlZd dlmZ d dlmZmZmZ d dl m
Z
ddl m Z ddl
mZ ddlmZmZ d d
lmZmZmZ G d d d eZeG d
d deZG dd deeZeddG dd deeZg dZdS ) )OptionalUnionN)BCEWithLogitsLossCrossEntropyLossMSELoss) IJepaConfig )ImageClassifierOutput)PreTrainedModel)auto_docstring torch_int )
ViTEmbeddingsViTForImageClassificationViTModelc st e Zd Zddededdf fdd
Zdejd ed
edejfd d Z dd
ejde
ej dedejfddZ Z
S )IJepaEmbeddingsFconfiguse_mask_tokenreturnNc s6 t || | `| jj}ttd||j | _
d S )N ) super__init__ cls_tokenpatch_embeddings num_patchesnn Parametertorchrandn hidden_sizeposition_embeddings)selfr r r __class__ d/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/transformers/models/ijepa/modular_ijepa.pyr s zIJepaEmbeddings.__init__
embeddingsheightwidthc C s |j d }| jj d }tj s||kr||kr| jS | j}|j d }|| j }|| j } t|d }
|d|
|
|}|dddd}t j
j ||| fddd }|dddd dd|}|S )
a
This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution
images. This method is also adapted to support torch.jit tracing.
Adapted from:
- https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and
- https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211
r g ?r r r
bicubicF)sizemode
align_corners)
shaper r jit
is_tracing
patch_sizer reshapepermuter
functional interpolateview) r! r& r' r( r
num_positionspatch_pos_embeddim
new_height new_widthsqrt_num_positionsr$ r$ r% interpolate_pos_encoding s&
z(IJepaEmbeddings.interpolate_pos_encoding pixel_valuesbool_masked_posr= c C s |j \}}}}| j||d}|d ur1|j d } | j|| d}
|d|
} |d| |
| }|r=|| ||| }n|| j }| |}|S )N)r= r r) ?) r. r
mask_tokenexpand unsqueezetype_asr= r dropout) r! r> r? r=
batch_size_r' r( r&
seq_length mask_tokensmaskr$ r$ r% forward> s
zIJepaEmbeddings.forward)F)NF)__name__
__module__ __qualname__r boolr r Tensorintr= r
BoolTensorrK
__classcell__r$ r$ r" r% r s *r c @ sV e Zd ZeZdZdZdZddgZdZ dZ
dZ dZ de
ejejejf ddfd d
ZdS ) IJepaPreTrainedModelijepar> Tr
IJepaLayermoduler Nc C s t |tjtjfr0tjj|jjt j
d| j j d|jj
|j_|jdur.|jj dS dS t |tjrE|jj |jjd dS t |trotjj|jjt j
d| j j d|jj
|j_|jdurq|jj dS dS dS )zInitialize the weightsg )meanstdNr@ )
isinstancer LinearConv2dinit
trunc_normal_weightdatator float32r initializer_rangedtypebiaszero_ LayerNormfill_r r rA )r! rW r$ r$ r%
_init_weightse s0
z"IJepaPreTrainedModel._init_weights)rL rM rN r config_classbase_model_prefixmain_input_name supports_gradient_checkpointing_no_split_modules_supports_sdpa_supports_flash_attn_2_supports_flex_attn_supports_attention_backendr r r[ r\ rg ri r$ r$ r$ r% rT Y s &rT c s, e Zd Zddededef fdd
Z ZS )
IJepaModelFr add_pooling_layerr c s$ t | || _t||d| _dS )z
add_pooling_layer (bool, *optional*, defaults to `True`):
Whether to add a pooling layer
use_mask_token (`bool`, *optional*, defaults to `False`):
Whether to use a mask token for masked image modeling.
)r N)r r r r r& )r! r rt r r" r$ r% r } s zIJepaModel.__init__)FF)rL rM rN r rO r rS r$ r$ r" r% rs | s $rs a
IJepa Model transformer with an image classification head on top (a linear layer on top of the final hidden states)
e.g. for ImageNet.
Note that it's possible to fine-tune IJepa on higher resolution images than the ones it has been trained on, by
setting `interpolate_pos_encoding` to `True` in the forward of the model. This will interpolate the pre-trained
position embeddings to the higher resolution.
) custom_introc s e Zd Zdef fdd Z ddeej deej deej dee d ee d
ee d ee d e e
e f fd
dZ Z
S )IJepaForImageClassificationr c s& t | t|dd| _| d S )NF)rt )r r rs rU post_init)r! r r" r$ r% r s z$IJepaForImageClassification.__init__Nr> head_masklabelsoutput_attentionsoutput_hidden_statesr= return_dictr c C sv |dur|n| j j}| j||||||d}|d } | | jdd}
d} |dur||
j}| j jdu rX| jdkr>d| j _n| jdkrT|j t
j ksO|j t
j krTd| j _nd| j _| j jdkrvt
} | jdkrp| |
| } n+| |
|} n%| j jdkrt } | |
d | j|d } n| j jdkrt } | |
|} |s|
f|dd }
| dur| f|
S |
S t| |
|j|jd
S ) a
labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*):
Labels for computing the image classification/regression loss. Indices should be in `[0, ...,
config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If
`config.num_labels > 1` a classification loss is computed (Cross-Entropy).
N)rx rz r{ r= r| r r )r9
regressionsingle_label_classificationmulti_label_classificationr) )losslogits
hidden_states
attentions)r use_return_dictrU
classifierrX ra device problem_type
num_labelsrd r longrQ r squeezer r6 r r r r )r! r> rx ry rz r{ r= r| outputssequence_outputr r loss_fctoutputr$ r$ r% rK sP
"
z#IJepaForImageClassification.forward)NNNNNNN)rL rM rN r r r r rP rO r tupler rK rS r$ r$ r" r% rv s4
rv )rT rs rv )typingr r r torch.nnr r r r -transformers.models.ijepa.configuration_ijepar modeling_outputsr modeling_utilsr
utilsr r vit.modeling_vitr r r r rT rs rv __all__r$ r$ r$ r% s$ J"
J