o ۷i@sdZddlmZddlmZmZmZmZddlZddlm Z ddl m Z ddl m Z mZdd lmZdd lmZmZmZdd lmZmZdd lmZmZmZmZmZmZd dlm Z m!Z!m"Z"e#e$Z%dej&dej&fddZ'dej&dej&fddZ(dej&dej&fddZ)eeddGdddeZ*eeddGdddeZ+eeGd d!d!eZ,Gd"d#d#e j-Z.Gd$d%d%e j-Z/ & 'dPd(e j-d)ej&d*ej&d+ej&d,eej&d-e0d.e0d/e1fd0d1Z2Gd2d3d3e j-Z3Gd4d5d5e j-Z4Gd6d7d7eZ5eGd8d9d9eZ6Gd:d;d;e j-Z7GddGd?d@d@e6Z9GdAdBdBe j-Z:edCdGdDdEdEe6Z;eGdFdGdGe6ZedLdGdMdNdNe6Z?gdOZ@dS)QzPyTorch CLIP model.) dataclass)AnyCallableOptionalUnionN)nn)ACT2FN) _create_4d_causal_attention_mask_prepare_4d_attention_mask)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPoolingImageClassifierOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel) ModelOutputauto_docstringcan_return_tuplefilter_out_non_signature_kwargslogging torch_int) CLIPConfigCLIPTextConfigCLIPVisionConfiglogitsreturncCstj|tjt||jdS)Ndevice)r functional cross_entropytorcharangelenr)rr%\/home/ubuntu/vllm_env/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.pycontrastive_loss%sr' similaritycCs t|}t|}||dS)Ng@)r't)r( caption_loss image_lossr%r%r& clip_loss)s  r,tensorcCs,t|d}tj|ddd}t|d}|S)z This method is equivalent to tensor.norm(p=2, dim=-1, keepdim=True) and used to make model `executorch` exportable. See issue https://github.com/pytorch/executorch/issues/3566 T)dimkeepdim?)r"powsum)r- square_tensor sum_tensor normed_tensorr%r%r&_get_vector_norm/s  r8z} Base class for vision model's outputs that also contains image embeddings of the pooling of the last hidden states. ) custom_introc@jeZdZUdZdZeejed<dZ eejed<dZ ee ejdfed<dZ ee ejdfed<dS)CLIPVisionModelOutputz image_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim)` *optional* returned when model is initialized with `with_projection=True`): The image embeddings obtained by applying the projection layer to the pooler_output. N image_embedslast_hidden_state. hidden_states attentions) __name__ __module__ __qualname____doc__r<rr" FloatTensor__annotations__r=r>tupler?r%r%r%r&r;: r;ze Base class for text model's outputs that also contains a pooling of the last hidden states. c@r:)CLIPTextModelOutputz text_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim)` *optional* returned when model is initialized with `with_projection=True`): The text embeddings obtained by applying the projection layer to the pooler_output. N text_embedsr=.r>r?) r@rArBrCrIrr"rDrEr=r>rFr?r%r%r%r&rHLrGrHc@seZdZUdZdZeejed<dZ eejed<dZ eejed<dZ eejed<dZ eejed<dZ eed<dZeed <d eefd d ZdS) CLIPOutputa loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `return_loss` is `True`): Contrastive loss for image-text similarity. logits_per_image (`torch.FloatTensor` of shape `(image_batch_size, text_batch_size)`): The scaled dot product scores between `image_embeds` and `text_embeds`. This represents the image-text similarity scores. logits_per_text (`torch.FloatTensor` of shape `(text_batch_size, image_batch_size)`): The scaled dot product scores between `text_embeds` and `image_embeds`. This represents the text-image similarity scores. text_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim`): The text embeddings obtained by applying the projection layer to the pooled output of [`CLIPTextModel`]. image_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim`): The image embeddings obtained by applying the projection layer to the pooled output of [`CLIPVisionModel`]. text_model_output (`BaseModelOutputWithPooling`): The output of the [`CLIPTextModel`]. vision_model_output (`BaseModelOutputWithPooling`): The output of the [`CLIPVisionModel`]. Nlosslogits_per_imagelogits_per_textrIr<text_model_outputvision_model_outputrcstfddDS)Nc3s.|]}|dvr |nt|VqdS))rNrON)getattrto_tuple).0kselfr%r& }s  z&CLIPOutput.to_tuple..)rFkeysrTr%rTr&rQ|s zCLIPOutput.to_tuple)r@rArBrCrKrr"rDrErLrMrIr<rNrrOrFrrQr%r%r%r&rJ^s   rJcsXeZdZdeffdd Zdejdededejfdd Zdd ej dejfd d Z Z S)CLIPVisionEmbeddingsconfigcst||_|j|_|j|_|j|_tt |j|_ tj |j |j|j|jdd|_|j|jd|_|jd|_t|j|j|_|jdt |jddddS)NF) in_channels out_channels kernel_sizestridebiasr.r position_idsrr/ persistent)super__init__rY hidden_size embed_dim image_size patch_sizer Parameterr"randnclass_embeddingConv2d num_channelspatch_embedding num_patches num_positions Embeddingposition_embeddingregister_bufferr#expandrUrY __class__r%r&rds"  "zCLIPVisionEmbeddings.__init__ embeddingsheightwidthrc Cs|jdd}|jjd}|jdd}tjs(||kr(||kr(||jS|ddddf}|ddddf}|jd} ||j} ||j} t |d} | d| | | }| dddd}t j j|| | fdd d }| dddddd| }tj||fdd S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rrNr/r2rr.bicubicF)sizemode align_cornersr0)shaperrweight unsqueezer"jit is_tracingr_rhrreshapepermuterr interpolateviewcat) rUrxryrzrorrrpclass_pos_embedpatch_pos_embedr0 new_height new_widthsqrt_num_positionsr%r%r&interpolate_pos_encodings*      z-CLIPVisionEmbeddings.interpolate_pos_encodingF pixel_valuesc Cs|j\}}}}|s&||jks||jkr&td|d|d|jd|jd |jjj}||j|d}|ddd}|j |dd} t j | |gdd } |r[| | | ||} | S| ||j} | S) NzInput image size (*z) doesn't match model ().)dtyper.rr/r)rrg ValueErrorrnrrtoflatten transposerkrtr"rrrrr_) rUrr batch_size_ryrz target_dtype patch_embeds class_embedsrxr%r%r&forwards  zCLIPVisionEmbeddings.forwardF) r@rArBrrdr"TensorintrrDr __classcell__r%r%rvr&rXs )rXc sXeZdZdeffdd Z   d deejdeejdeejdej fd d Z Z S) CLIPTextEmbeddingsrYcsRt|j}t|j||_t|j||_|j dt |j ddddS)Nr_r`Fra) rcrdrerrq vocab_sizetoken_embeddingmax_position_embeddingsrrrsr"r#rtrUrYrfrvr%r&rds  zCLIPTextEmbeddings.__init__N input_idsr_ inputs_embedsrcCs|dur |jdn|jd}|jjjd}||kr#td|d||dur2|jddd|f}|dur;||}||}||}|S)Nr/rzRSequence length must be less than max_position_embeddings (got `sequence length`: z and max_position_embeddings: )rrrrrr_r)rUrr_r seq_lengthmax_position_embeddingposition_embeddingsrxr%r%r&rs"  zCLIPTextEmbeddings.forward)NNN) r@rArBrrdrr" LongTensorrDrrrr%r%rvr&rsrTmodulequerykeyvalueattention_maskscalingdropoutoutput_attentionsc Kst||dd|} |dur| |} tjj| dtjd|j} tjj | ||j d} t| |} | dd } |s>d} | | fS)Nr/r)r0r)ptrainingrr.) r"matmulrrr softmaxfloat32rrrr contiguous) rrrrrrrrkwargs attn_weights attn_outputr%r%r&eager_attention_forwards  rcsveZdZdZdeeefffdd Z   ddej de ej d e ej d e e d e ej e ej ff d d Z ZS) CLIPAttentionz=Multi-headed attention from 'Attention Is All You Need' paperrYcst||_|j|_|j|_|j|j|_|j|j|jkr-td|jd|jd|jd|_ |j |_ d|_ t |j|j|_t |j|j|_t |j|j|_t |j|j|_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: rF)rcrdrYrerfnum_attention_heads num_headshead_dimrscaleattention_dropoutr is_causalrLineark_projv_projq_projout_projrurvr%r&rds$   zCLIPAttention.__init__NFr>rcausal_attention_maskrrc Cs,|j\}}}||}||} ||} |||d|jdd}| ||d|jdd} | ||d|jdd} |jjdkrH|du|_ n|durU|durU||}n|dur[|}t } |jjdkrit |jj} | ||| | ||j |j |j sxdn|j|d \} } | |||} || } |sd} | | fS) z#Input shape: Batch x Time x Channelr/rr.flash_attention_2Neagerr)rrrr)rrrrrrrrY_attn_implementationrrrrrrrrr)rUr>rrrrrrfqueriesrWvaluesattention_interfacerrr%r%r&r-s@          zCLIPAttention.forward)NNF)r@rArBrCrrrrdr"rrboolrFrrr%r%rvr&rs"rcs2eZdZfddZdejdejfddZZS)CLIPMLPcsDt||_t|j|_t|j|j |_ t|j |j|_ dSN) rcrdrYr hidden_act activation_fnrrreintermediate_sizefc1fc2rurvr%r&rdbs  zCLIPMLP.__init__r>rcCs"||}||}||}|Sr)rrr)rUr>r%r%r&ris   zCLIPMLP.forward)r@rArBrdr"rrrr%r%rvr&ras rc s\eZdZdeeefffdd Z d dejdejdejde e d e ej f d d Z ZS) CLIPEncoderLayerrYcsRt|j|_t||_tj|j|jd|_ t ||_ tj|j|jd|_ dSN)eps) rcrdrerfr self_attnr LayerNormlayer_norm_eps layer_norm1rmlp layer_norm2rurvr%r&rdqs   zCLIPEncoderLayer.__init__Fr>rrrrcCsd|}||}|j||||d\}}||}|}||}||}||}|f}|r0||f7}|S)aI Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. `(config.encoder_attention_heads,)`. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. )r>rrr)rrrr)rUr>rrrresidualroutputsr%r%r&rys"     zCLIPEncoderLayer.forwardr)r@rArBrrrrdr"rrrrFrDrrr%r%rvr&rps rc@s6eZdZUeed<dZdZdZdZdZ dZ ddZ dS)CLIPPreTrainedModelrYclipTcCs|jj}t|tr#|jjjjd|dd|jjjjd|ddn)t|t rX|jj}t j j|j d|j d|dt j j|jj|jj|dt j j|jj|jj|dnt|tr|jj}|j dd|jjd|}|j d|}t j j|jj|dt j j|jj|dt j j|jj|dt j j|jj|dnt|tr|jj}|jjdd|jjd|}d|jjd|}t j j|jj|dt j j|jj|dntt|trt j j|jj|jd|jjdt j j|jj|jd|jjdnKt|trt j j|jj|jjd|jjdn2t|t r3t j j|jj|jjd|jjdnt|t!rLt j j|j"j|jj#jd|jjdt|t j$r`|j%j&|jj'dt|t j(ru|j%durw|j%j&dSdSdS) zInitialize the weightsrg{Gz?)meanstdr)rr.g?N))rYinitializer_factor isinstancerrrdatanormal_rrrXrinitrkrfrninitializer_rangernum_hidden_layersrrrrrrerr CLIPModeltext_projectiontext_embed_dimvisual_projectionvision_embed_dimCLIPVisionModelWithProjectionCLIPTextModelWithProjectionCLIPForImageClassification classifier vision_configrr^zero_fill_r)rUrfactor in_proj_std out_proj_stdfc_stdr%r%r& _init_weightssj          z!CLIPPreTrainedModel._init_weightsN) r@rArBrrEbase_model_prefixsupports_gradient_checkpointing_supports_sdpa_supports_flash_attn_supports_flex_attn_supports_attention_backendrr%r%r%r&rs  rc sbeZdZdZdeffdd Z    d deejdeejdee d ee d e f d d Z Z S) CLIPEncoderz Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`CLIPEncoderLayer`]. Args: config: CLIPConfig rYcs:t|_tfddtjD|_d|_dS)Ncsg|]}tqSr%)r)rRrrYr%r& sz(CLIPEncoder.__init__..F) rcrdrYr ModuleListrangerlayersgradient_checkpointingrurvr r&rds   zCLIPEncoder.__init__Nrrroutput_hidden_statesrc Cs|dur|n|jj}|dur|n|jj}|rdnd}|rdnd}|}t|jD] \} } |r2||f}| ||||d} | d}|rG|| df}q'|rO||f}t|||dS)a Args: inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) causal_attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Causal mask for the text model. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. Nr%)rrr)r=r>r?)rYrr enumeraterr ) rUrrrrrencoder_statesall_attentionsr>idx encoder_layer layer_outputsr%r%r&rs4%    zCLIPEncoder.forwardNNNN) r@rArBrCrrdrr"rrr rrr%r%rvr&r s$ r csneZdZdeffdd Ze     d deejdeejdeejdee d ee d e f d d Z Z S)CLIPTextTransformerrYcsHt||_|j}t||_t||_tj ||j d|_ |j |_ dSr) rcrdrYrerrxr encoderrrrfinal_layer_norm eos_token_idrrvr%r&rd<s    zCLIPTextTransformer.__init__Nrrr_rrrc CsF|dur|n|jj}|dur|n|jj}|durtd|}|d|d}|j||d}t||j|j d}|durH|jj dkrHt ||j}|j |||||d} | j } || } |jdkrz| tj| jd| j d|jtj| j d jdd f} n| tj| jd| j d|jtj| j d |jkjdd f} t| | | j| jd S) NzYou have to specify input_idsr/)rr_rr)rrrrrr.r)rrrr= pooler_outputr>r?)rYrrrr|rrxr rrrr rr=rrr"r#rrrargmaxrr>r?) rUrrr_rr input_shaper>rencoder_outputsr= pooled_outputr%r%r&rGsT      zCLIPTextTransformer.forwardNNNNN) r@rArBrrdrrr"rrrrrr%r%rvr&r;s* rzI The text model from CLIP without any head or projection on top. cseZdZUeed<ddgZdZdeffdd Zdej fdd Z d d Z e e dd eejdeejdeejdeedeedef ddZZS) CLIPTextModelrYrrFc"t|t||_|dSr)rcrdr text_model post_initrurvr%r&rd   zCLIPTextModel.__init__rcC |jjjSrr&rxrrTr%r%r&get_input_embeddings z"CLIPTextModel.get_input_embeddingscC||jj_dSrr*rUrr%r%r&set_input_embeddingsz"CLIPTextModel.set_input_embeddingsNrrr_rrcCs|j|||||dS)a9 Examples: ```python >>> from transformers import AutoTokenizer, CLIPTextModel >>> model = CLIPTextModel.from_pretrained("openai/clip-vit-base-patch32") >>> tokenizer = AutoTokenizer.from_pretrained("openai/clip-vit-base-patch32") >>> inputs = tokenizer(["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state >>> pooled_output = outputs.pooler_output # pooled (EOS token) states ```rrr_rr)r&)rUrrr_rrr%r%r&rszCLIPTextModel.forwardr#)r@rArBrrE_no_split_modulesrrdrModuler+r/rrrr"rrrrrr%r%rvr&r$s6 r$c s`eZdZdeffdd Ze    d deejdee dee d ee d e f d d Z Z S)CLIPVisionTransformerrYcsRt||_|j}t||_tj||jd|_ t ||_ tj||jd|_ dSr) rcrdrYrerXrxrrr pre_layrnormr rpost_layernormrrvr%r&rds   zCLIPVisionTransformer.__init__NFrrrrrc Cs|dur|n|jj}|dur|n|jj}|durtd|j||d}||}|j|||d}|j}|dddddf}||}t |||j |j dS)Nz You have to specify pixel_values)r)rrrrr) rYrrrrxr5rr=r6rr>r?) rUrrrrr>r!r=r"r%r%r&rs*  zCLIPVisionTransformer.forwardNNNF) r@rArBrrdrrr"rDrrrrr%r%rvr&r4s$ r4zK The vision model from CLIP without any head or projection on top. c seZdZUeed<dZdgZdeffdd Zdej fddZ e e dde ejd e ed e ed edef ddZZS)CLIPVisionModelrYrrcr%r)rcrdr4 vision_modelr'rurvr%r&rdr(zCLIPVisionModel.__init__rcCr)rr9rxrnrTr%r%r&r+ r,z$CLIPVisionModel.get_input_embeddingsNFrrrcCs|j||||dS)a Example: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, CLIPVisionModel >>> model = CLIPVisionModel.from_pretrained("openai/clip-vit-base-patch32") >>> processor = AutoProcessor.from_pretrained("openai/clip-vit-base-patch32") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state >>> pooled_output = outputs.pooler_output # pooled CLS states ```rrrr)r9)rUrrrrr%r%r&rs zCLIPVisionModel.forwardr7)r@rArBrrEmain_input_namer2rdrr3r+rrrr"rDrrrrr%r%rvr&r8s. r8cseZdZUeed<gdZdZdeffdd Zee  dde j de e j d e e j d e j fd d Zee dd e j ded e j fddZee        dde e jd e e j de e j d e e jde ede ede eded efddZZS)rrY)rrrXFcst|t|jtstdt|jdt|jts(tdt|jd|j}|j}|j |_ |j |_ |j |_ t |}|j|_t|}|j|_tj|j |j dd|_tj|j |j dd|_tt|jj|_|dS)NzKconfig.text_config is expected to be of type CLIPTextConfig but is of type .zOconfig.vision_config is expected to be of type CLIPVisionConfig but is of type Fr^)rcrdr text_configr TypeErrortyperrprojection_dimrerrr$ _from_configr&r8r9rrrrrir"r-rYlogit_scale_init_value logit_scaler')rUrYr?rr&r9rvr%r&rd:s4      zCLIPModel.__init__Nrrr_rcCs$|j|||d}|j}||}|S)a Returns: text_features (`torch.FloatTensor` of shape `(batch_size, output_dim`): The text embeddings obtained by applying the projection layer to the pooled output of [`CLIPTextModel`]. Examples: ```python >>> import torch >>> from transformers import AutoTokenizer, CLIPModel >>> model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32") >>> tokenizer = AutoTokenizer.from_pretrained("openai/clip-vit-base-patch32") >>> inputs = tokenizer(["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt") >>> with torch.inference_mode(): ... text_features = model.get_text_features(**inputs) ```)rrr_)r&rr)rUrrr_ text_outputsr" text_featuresr%r%r&get_text_features]s zCLIPModel.get_text_featuresrrcCs"|j||d}|j}||}|S)ai Returns: image_features (`torch.FloatTensor` of shape `(batch_size, output_dim`): The image embeddings obtained by applying the projection layer to the pooled output of [`CLIPVisionModel`]. Examples: ```python >>> import torch >>> from transformers import AutoProcessor, CLIPModel >>> from transformers.image_utils import load_image >>> model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32") >>> processor = AutoProcessor.from_pretrained("openai/clip-vit-base-patch32") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = load_image(url) >>> inputs = processor(images=image, return_tensors="pt") >>> with torch.inference_mode(): ... image_features = model.get_image_features(**inputs) ```)rr)r9rr)rUrrvision_outputsr"image_featuresr%r%r&get_image_featuress zCLIPModel.get_image_features return_lossrrc  Cs|dur|n|jj}|dur|n|jj}|j||||d} |j|||||d} | j} || } | j} || } | t| } | t| } t | | | j } | |j | j } | }d}|rft| }t||| | | | | dS)a return_loss (`bool`, *optional*): Whether or not to return the contrastive loss. Examples: ```python >>> import torch >>> from transformers import AutoProcessor, CLIPModel >>> from transformers.image_utils import load_image >>> model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32") >>> processor = AutoProcessor.from_pretrained("openai/clip-vit-base-patch32") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = load_image(url) >>> inputs = processor( ... text=["a photo of a cat", "a photo of a dog"], images=image, return_tensors="pt", padding=True ... ) >>> with torch.inference_mode(): ... outputs = model(**inputs) >>> logits_per_image = outputs.logits_per_image # this is the image-text similarity score >>> probs = logits_per_image.softmax(dim=1) # we can take the softmax to get the label probabilities ```Nr;r1)rKrLrMrIr<rNrO)rYrrr9r&rrrr8r"rr)rrrEexpr,rJ)rUrrrr_rLrrrrIrFr<rIrMrLrKr%r%r&rsJ(    zCLIPModel.forward)NNr)NNNNNNNF)r@rArBrrEr2rrdrrr"rrrDrHrrKrrrJrrr%r%rvr&r4sr ##%  rcseZdZUeed<dZddgZdeffdd Zdej fdd Z d d Z e e dd eejdeejdeejdeedeedef ddZZS)rrYFrrc@t|t|}|j|_tj|j|jdd|_ | dSNFr>) rcrdr$rCr&rrrerBrr')rUrYr&rvr%r&rd    z$CLIPTextModelWithProjection.__init__rcCr)rr*rTr%r%r&r+r,z0CLIPTextModelWithProjection.get_input_embeddingscCr-rr*r.r%r%r&r/r0z0CLIPTextModelWithProjection.set_input_embeddingsNrrr_rrc Cs:|j|||||d}|j}||}t||j|j|jdS)a@ Examples: ```python >>> import torch >>> from transformers import AutoTokenizer, CLIPTextModelWithProjection >>> model = CLIPTextModelWithProjection.from_pretrained("openai/clip-vit-base-patch32") >>> tokenizer = AutoTokenizer.from_pretrained("openai/clip-vit-base-patch32") >>> inputs = tokenizer(["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt") >>> with torch.inference_mode(): ... outputs = model(**inputs) >>> text_embeds = outputs.text_embeds ```r1)rIr=r>r?)r&rrrHr=r>r?) rUrrr_rrrFr"rIr%r%r&rs z#CLIPTextModelWithProjection.forwardr#)r@rArBrrErr2rdrr3r+r/rrrr"rrrHrrr%r%rvr&rs6  rc s~eZdZUeed<dZdeffdd ZdejfddZ e e    dde e jd e ed e ed edef d dZZS)rrYrcrNrO) rcrdr8rCr9rrrerBrr'rUrYr9rvr%r&rdNrPz&CLIPVisionModelWithProjection.__init__rcCr)rr:rTr%r%r&r+Yr,z2CLIPVisionModelWithProjection.get_input_embeddingsNFrrrcCs8|j||||d}|j}||}t||j|j|jdS)a Examples: ```python >>> import torch >>> from transformers import AutoProcessor, CLIPVisionModelWithProjection >>> from transformers.image_utils import load_image >>> model = CLIPVisionModelWithProjection.from_pretrained("openai/clip-vit-base-patch32") >>> processor = AutoProcessor.from_pretrained("openai/clip-vit-base-patch32") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = load_image(url) >>> inputs = processor(images=image, return_tensors="pt") >>> with torch.inference_mode(): ... outputs = model(**inputs) >>> image_embeds = outputs.image_embeds ```r;)r<r=r>r?)r9rrr;r=r>r?)rUrrrrrIr"r<r%r%r&r\s z%CLIPVisionModelWithProjection.forwardr7)r@rArBrrEr<rdrr3r+rrrr"rDrr;rrr%r%rvr&rIs,  rz CLIP vision encoder with an image classification head on top (a linear layer on top of the pooled final hidden states of the patch tokens) e.g. for ImageNet. c sneZdZdZdeddffdd Zee    d dee j dee j dee d ee de f d d Z ZS) rrrYrNcsZt||j|_t|j}|j|_|jdkr"t|jj |jnt |_ | dS)Nr) rcrd num_labelsr8rCrr9rrreIdentityrr'rQrvr%r&rds  " z#CLIPForImageClassification.__init__labelsrrc Cs|dur|n|jj}|dur|n|jj}|j|||d}|j}tj|ddddddfdd}||}d}|durD||||j}t |||j |j dS)a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the image classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). N)rrrr)rKrr>r?) rYrrr9r=r"rr loss_functionrr>r?) rUrrTrrrsequence_outputrrKr%r%r&rs($ z"CLIPForImageClassification.forwardr)r@rArBr<rrdrrrr"rrrrrr%r%rvr&rs(r)rrr$rr8rr)rT)ArC dataclassesrtypingrrrrr"r activationsr modeling_attn_mask_utilsr r modeling_layersr modeling_outputsr rrmodeling_utilsrrutilsrrrrrrconfiguration_cliprrr get_loggerr@loggerrr'r,r8r;rHrJr3rXrfloatrrrrrrr rr$r4r8rrrr__all__r%r%r%r&s         #S/ K2BVV504PDA ?