o wi@sdZddlmZddlmZmZmZmZddlZddlm Z ddl m Z m Z m Z ddlmZdd lmZmZdd lmZdd lmZmZmZdd lmZmZdd lmZmZmZm Z m!Z!ddl"m#Z#m$Z$m%Z%e &e'Z(dej)dej)fddZ*dej)dej)fddZ+dej)dej)fddZ,eeddGdddeZ-eeddGdd d eZ.eeGd!d"d"eZ/Gd#d$d$e j0Z1Gd%d&d&e j0Z2 ' (dQd)e j0d*ej)d+ej)d,ej)d-eej)d.e3d/e3d0e4fd1d2Z5Gd3d4d4e j0Z6Gd5d6d6e j0Z7Gd7d8d8eZ8eGd9d:d:eZ9Gd;d<dd>e j0Z;ed?dGd@dAdAe9ZeGdGdHdHe9Z?eGdIdJdJe9Z@eGdKdLdLe9ZAedMdGdNdOdOe9ZBgdPZCdS)RzPyTorch CLIP model.) dataclass)AnyCallableOptionalUnionN)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN) _create_4d_causal_attention_mask_prepare_4d_attention_mask)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPoolingImageClassifierOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel) ModelOutputauto_docstringcan_return_tuplelogging torch_int) CLIPConfigCLIPTextConfigCLIPVisionConfiglogitsreturncCstj|tjt||jdS)Ndevice)r functional cross_entropytorcharangelenr!)rr'c/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.pycontrastive_loss&sr) similaritycCs t|}t|}||dS)Ng@)r)t)r* caption_loss image_lossr'r'r( clip_loss*s  r.tensorcCs,t|d}tj|ddd}t|d}|S)z This method is equivalent to tensor.norm(p=2, dim=-1, keepdim=True) and used to make model `executorch` exportable. See issue https://github.com/pytorch/executorch/issues/3566 T)dimkeepdim?)r$powsum)r/ square_tensor sum_tensor normed_tensorr'r'r(_get_vector_norm0s  r:z} Base class for vision model's outputs that also contains image embeddings of the pooling of the last hidden states. ) custom_introc@jeZdZUdZdZeejed<dZ eejed<dZ ee ejdfed<dZ ee ejdfed<dS)CLIPVisionModelOutputz image_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim)` *optional* returned when model is initialized with `with_projection=True`): The image embeddings obtained by applying the projection layer to the pooler_output. N image_embedslast_hidden_state. hidden_states attentions) __name__ __module__ __qualname____doc__r>rr$ FloatTensor__annotations__r?r@tuplerAr'r'r'r(r=; r=ze Base class for text model's outputs that also contains a pooling of the last hidden states. c@r<)CLIPTextModelOutputz text_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim)` *optional* returned when model is initialized with `with_projection=True`): The text embeddings obtained by applying the projection layer to the pooler_output. N text_embedsr?.r@rA) rBrCrDrErKrr$rFrGr?r@rHrAr'r'r'r(rJMrIrJc@seZdZUdZdZeejed<dZ eejed<dZ eejed<dZ eejed<dZ eejed<dZ eed<dZeed <d eefd d ZdS) CLIPOutputa loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `return_loss` is `True`): Contrastive loss for image-text similarity. logits_per_image (`torch.FloatTensor` of shape `(image_batch_size, text_batch_size)`): The scaled dot product scores between `image_embeds` and `text_embeds`. This represents the image-text similarity scores. logits_per_text (`torch.FloatTensor` of shape `(text_batch_size, image_batch_size)`): The scaled dot product scores between `text_embeds` and `image_embeds`. This represents the text-image similarity scores. text_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim`): The text embeddings obtained by applying the projection layer to the pooled output of [`CLIPTextModel`]. image_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim`): The image embeddings obtained by applying the projection layer to the pooled output of [`CLIPVisionModel`]. text_model_output (`BaseModelOutputWithPooling`): The output of the [`CLIPTextModel`]. vision_model_output (`BaseModelOutputWithPooling`): The output of the [`CLIPVisionModel`]. Nlosslogits_per_imagelogits_per_textrKr>text_model_outputvision_model_outputrcstfddDS)Nc3s.|]}|dvr |nt|VqdS))rPrQN)getattrto_tuple).0kselfr'r( ~s  z&CLIPOutput.to_tuple..)rHkeysrVr'rVr(rS}s zCLIPOutput.to_tuple)rBrCrDrErMrr$rFrGrNrOrKr>rPrrQrHrrSr'r'r'r(rL_s   rLcsXeZdZdeffdd Zdejdededejfdd Zdd ej dejfd d Z Z S)CLIPVisionEmbeddingsconfigcst||_|j|_|j|_|j|_tt |j|_ tj |j |j|j|jdd|_|j|jd|_|jd|_t|j|j|_|jdt |jddddS)NF) in_channels out_channels kernel_sizestridebiasr0r position_idsrr1 persistent)super__init__r[ hidden_size embed_dim image_size patch_sizer Parameterr$randnclass_embeddingConv2d num_channelspatch_embedding num_patches num_positions Embeddingposition_embeddingregister_bufferr%expandrWr[ __class__r'r(rfs"  "zCLIPVisionEmbeddings.__init__ embeddingsheightwidthrc Cs|jdd}|jjd}|jdd}tjs(||kr(||kr(||jS|ddddf}|ddddf}|jd} ||j} ||j} t |d} | d| | | }| dddd}t j j|| | fdd d }| dddddd| }tj||fdd S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rrNr1r4r r0bicubicF)sizemode align_cornersr2)shapertweight unsqueezer$jit is_tracingrarjrreshapepermuterr" interpolateviewcat) rWrzr{r|rqrtrrclass_pos_embedpatch_pos_embedr2 new_height new_widthsqrt_num_positionsr'r'r(interpolate_pos_encodings*      z-CLIPVisionEmbeddings.interpolate_pos_encodingF pixel_valuesc Cs|j\}}}}|s&||jks||jkr&td|d|d|jd|jd |jjj}||j|d}|ddd}|j |dd} t j | |gdd } |r[| | | ||} | S| ||j} | S) NzInput image size (*z) doesn't match model ().)dtyper0rr1r)rri ValueErrorrprrtoflatten transposermrvr$rrrtra) rWrr batch_size_r{r| target_dtype patch_embeds class_embedsrzr'r'r(forwards  zCLIPVisionEmbeddings.forwardF) rBrCrDrrfr$TensorintrrFr __classcell__r'r'rxr(rZs )rZc sXeZdZdeffdd Z   d deejdeejdeejdej fd d Z Z S) CLIPTextEmbeddingsr[csRt|j}t|j||_t|j||_|j dt |j ddddS)NrarbFrc) rerfrgrrs vocab_sizetoken_embeddingmax_position_embeddingsrtrur$r%rvrWr[rhrxr'r(rfs  zCLIPTextEmbeddings.__init__N input_idsra inputs_embedsrcCs|dur |jdn|jd}|jjjd}||kr#td|d||dur2|jddd|f}|dur;||}||}||}|S)Nr1rzRSequence length must be less than max_position_embeddings (got `sequence length`: z and max_position_embeddings: )rrtrrrar)rWrrar seq_lengthmax_position_embeddingposition_embeddingsrzr'r'r(rs"  zCLIPTextEmbeddings.forward)NNN) rBrCrDrrfrr$ LongTensorrFrrrr'r'rxr(rsrTmodulequerykeyvalueattention_maskscalingdropoutoutput_attentionsc Kst||dd|} |dur| |} tjj| dtjd|j} tjj | ||j d} t| |} | dd } |s>d} | | fS)Nr1r)r2r)ptrainingrr0) r$matmulrrr"softmaxfloat32rrrr contiguous) rrrrrrrrkwargs attn_weights attn_outputr'r'r(eager_attention_forwards  rcsveZdZdZdeeefffdd Z   ddej de ej d e ej d e e d e ej e ej ff d d Z ZS) CLIPAttentionz=Multi-headed attention from 'Attention Is All You Need' paperr[cst||_|j|_|j|_|j|j|_|j|j|jkr-td|jd|jd|jd|_ |j |_ d|_ t |j|j|_t |j|j|_t |j|j|_t |j|j|_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: rF)rerfr[rgrhnum_attention_heads num_headshead_dimrscaleattention_dropoutr is_causalrLineark_projv_projq_projout_projrwrxr'r(rfs$   zCLIPAttention.__init__NFr@rcausal_attention_maskrrc CsH|j\}}}||}||} ||} |||d|jdd}| ||d|jdd} | ||d|jdd} |jjdkrH|du|_ n|durU|durU||}n|dur[|}t } |jjdkrw|jjdkrq|rqt dnt |jj} | ||| | ||j |j|jsd n|j|d \} } | |||} || } |sd} | | fS) z#Input shape: Batch x Time x Channelr1rr0flash_attention_2Neagersdpaz`torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to eager attention. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.r)rrrr)rrrrrrrr[_attn_implementationrrlogger warning_oncerrrrrrr)rWr@rrrrrrhqueriesrYvaluesattention_interfacerrr'r'r(r.sH          zCLIPAttention.forward)NNF)rBrCrDrErrrrfr$rrboolrHrrr'r'rxr(rs"rcs2eZdZfddZdejdejfddZZS)CLIPMLPcsDt||_t|j|_t|j|j |_ t|j |j|_ dSN) rerfr[r hidden_act activation_fnrrrgintermediate_sizefc1fc2rwrxr'r(rfis  zCLIPMLP.__init__r@rcCs"||}||}||}|Sr)rrr)rWr@r'r'r(rps   zCLIPMLP.forward)rBrCrDrfr$rrrr'r'rxr(rhs rc s\eZdZdeeefffdd Z d dejdejdejde e d e ej f d d Z ZS) CLIPEncoderLayerr[csRt|j|_t||_tj|j|jd|_ t ||_ tj|j|jd|_ dSNeps) rerfrgrhr self_attnr LayerNormlayer_norm_eps layer_norm1rmlp layer_norm2rwrxr'r(rfxs   zCLIPEncoderLayer.__init__Fr@rrrrcCsd|}||}|j||||d\}}||}|}||}||}||}|f}|r0||f7}|S)aI Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. `(config.encoder_attention_heads,)`. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. )r@rrr)rrrr)rWr@rrrresidualroutputsr'r'r(rs"     zCLIPEncoderLayer.forwardr)rBrCrDrrrrfr$rrrrHrFrrr'r'rxr(rws rc@s0eZdZeZdZdZdZdZdZ dZ ddZ dS)CLIPPreTrainedModelclipTcCs|jj}t|tr#|jjjjd|dd|jjjjd|ddn)t|t rX|jj}t j j|j d|j d|dt j j|jj|jj|dt j j|jj|jj|dnt|tr|jj}|j dd|jjd|}|j d|}t j j|jj|dt j j|jj|dt j j|jj|dt j j|jj|dnt|tr|jj}|jjdd|jjd|}d|jjd|}t j j|jj|dt j j|jj|dntt|trt j j|jj|jd|jjdt j j|jj|jd|jjdnKt|trt j j|jj|jjd|jjdn2t|t r3t j j|jj|jjd|jjdnt|t!rLt j j|j"j|jj#jd|jjdt|t j$r`|j%j&|jj'dt|t j(ru|j%durw|j%j&dSdSdS) zInitialize the weightsrg{Gz?)meanstdr)rr0g?N))r[initializer_factor isinstancerrrdatanormal_rtrZrinitrmrhrpinitializer_rangernum_hidden_layersrrrrrrgrr CLIPModeltext_projectiontext_embed_dimvisual_projectionvision_embed_dimCLIPVisionModelWithProjectionCLIPTextModelWithProjectionCLIPForImageClassification classifier vision_configrr`zero_fill_r)rWrfactor in_proj_std out_proj_stdfc_stdr'r'r( _init_weightssj          z!CLIPPreTrainedModel._init_weightsN) rBrCrDr config_classbase_model_prefixsupports_gradient_checkpointing_supports_sdpa_supports_flash_attn_2_supports_flex_attn_supports_attention_backendr r'r'r'r(rs rc sfeZdZdZdeffdd Ze    d deej deej dee d ee d e f d d Z Z S) CLIPEncoderz Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`CLIPEncoderLayer`]. Args: config: CLIPConfig r[cs:t|_tfddtjD|_d|_dS)Ncsg|]}tqSr')r)rTrr[r'r( sz(CLIPEncoder.__init__..F) rerfr[r ModuleListrangerlayersgradient_checkpointingrwrxrr(rfs   zCLIPEncoder.__init__Nrrroutput_hidden_statesrc Cs|dur|n|jj}|dur|n|jj}|rdnd}|rdnd}|}t|jD] \} } |r2||f}| ||||d} | d}|rG|| df}q'|rO||f}t|||dS)a Args: inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) causal_attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Causal mask for the text model. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. Nr')rrr)r?r@rA)r[rr enumeraterr) rWrrrrrencoder_statesall_attentionsr@idx encoder_layer layer_outputsr'r'r(rs4&    zCLIPEncoder.forwardNNNN)rBrCrDrErrfrrr$rrrrrr'r'rxr(rs&rcsreZdZdeffdd Zee     d deej deej deej dee d ee d e f d d Z Z S)CLIPTextTransformerr[csTt||_|j}t||_t||_tj ||j d|_ |j |_ |j dk|_dS)Nrr)rerfr[rgrrzrencoderrrrfinal_layer_norm eos_token_idr_use_flash_attention_2rrxr'r(rfDs   zCLIPTextTransformer.__init__Nrrrarrrc Cs@|dur|n|jj}|dur|n|jj}|durtd|}|d|d}|j||d}t||j|j d}|durE|j sEt ||j}|j |||||d} | j } || } |jdkrw| tj| jd| j d|jtj| j djdd f} n| tj| jd| j d|jtj| j d|jkjdd f} t| | | j| jd S) NzYou have to specify input_idsr1)rrar )rrrrrr0r)rr!rr? pooler_outputr@rA)r[rrrr~rrzr rr!r$rr!r?r"r#r$r%rrrargmaxrr@rA) rWrrrarr input_shaper@rencoder_outputsr? pooled_outputr'r'r(rRsT      zCLIPTextTransformer.forwardNNNNN)rBrCrDrrfrrrr$rrrrrr'r'rxr(r Cs,r zI The text model from CLIP without any head or projection on top. ceZdZeZddgZdeffdd ZdejfddZ d d Z e e dd e ejd e ejde ejde ede edef ddZZS) CLIPTextModelrrr[c"t|t||_|dSr)rerfr  text_model post_initrwrxr'r(rf   zCLIPTextModel.__init__rcC |jjjSrr/rzrrVr'r'r(get_input_embeddings z"CLIPTextModel.get_input_embeddingscC||jj_dSrr3rWrr'r'r(set_input_embeddingsz"CLIPTextModel.set_input_embeddingsNrrrarrcCs|j|||||dS)a9 Examples: ```python >>> from transformers import AutoTokenizer, CLIPTextModel >>> model = CLIPTextModel.from_pretrained("openai/clip-vit-base-patch32") >>> tokenizer = AutoTokenizer.from_pretrained("openai/clip-vit-base-patch32") >>> inputs = tokenizer(["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state >>> pooled_output = outputs.pooler_output # pooled (EOS token) states ```rrrarr)r/)rWrrrarrr'r'r(rszCLIPTextModel.forwardr+)rBrCrDrr _no_split_modulesrfrModuler4r8rrrr$rrrrrr'r'rxr(r-s4r-c sdeZdZdeffdd Zee    d deej dee dee d ee d e f d d Z Z S)CLIPVisionTransformerr[csRt||_|j}t||_tj||jd|_ t ||_ tj||jd|_ dSr) rerfr[rgrZrzrrr pre_layrnormrr!post_layernormrrxr'r(rfs   zCLIPVisionTransformer.__init__NFrrrrrc Cs|dur|n|jj}|dur|n|jj}|durtd|j||d}||}|j|||d}|j}|dddddf}||}t |||j |j dS)Nz You have to specify pixel_values)r)rrrrr%) r[rrrrzr>r!r?r?rr@rA) rWrrrrr@r)r?r*r'r'r(rs*   zCLIPVisionTransformer.forwardNNNF)rBrCrDrrfrrrr$rFrrrrr'r'rxr(r=s& r=zK The vision model from CLIP without any head or projection on top. c s~eZdZeZdZdgZdeffdd Zdej fddZ e e dde ejd e ed e ed edef ddZZS)CLIPVisionModelrrr[cr.r)rerfr= vision_modelr0rwrxr'r(rfr1zCLIPVisionModel.__init__rcCr2rrBrzrprVr'r'r(r4r5z$CLIPVisionModel.get_input_embeddingsNFrrrcCs|j||||dS)a Example: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, CLIPVisionModel >>> model = CLIPVisionModel.from_pretrained("openai/clip-vit-base-patch32") >>> processor = AutoProcessor.from_pretrained("openai/clip-vit-base-patch32") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state >>> pooled_output = outputs.pooler_output # pooled CLS states ```rrrr)rB)rWrrrrr'r'r(rs zCLIPVisionModel.forwardr@)rBrCrDrr main_input_namer;rfrr<r4rrrr$rFrrrrr'r'rxr(rAs.rAcs$eZdZeZgdZdeffdd Ze     ddee j dee j dee j d ee d ee d e j f d d Z e    ddee j d ee d ee de d e j f ddZee        ddee jdee j dee j dee jdee d ee d ee de d efddZZS)r)rrrZr[cst|t|jtstdt|jdt|jts(tdt|jd|j}|j}|j |_ |j |_ |j |_ t |}|j|_t|}|j|_tj|j |j dd|_tj|j |j dd|_tt|jj|_|dS)NzKconfig.text_config is expected to be of type CLIPTextConfig but is of type .zOconfig.vision_config is expected to be of type CLIPVisionConfig but is of type Fr`)rerfr text_configr TypeErrortyperrprojection_dimrgrrr- _from_configr/rArBrrrrrkr$r/r[logit_scale_init_value logit_scaler0)rWr[rHrr/rBrxr'r(rfEs4      zCLIPModel.__init__Nrrrarrrc CsP|dur|n|jj}|dur|n|jj}|j|||||d}|j}||}|S)a Returns: text_features (`torch.FloatTensor` of shape `(batch_size, output_dim`): The text embeddings obtained by applying the projection layer to the pooled output of [`CLIPTextModel`]. Examples: ```python >>> from transformers import AutoTokenizer, CLIPModel >>> model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32") >>> tokenizer = AutoTokenizer.from_pretrained("openai/clip-vit-base-patch32") >>> inputs = tokenizer(["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt") >>> text_features = model.get_text_features(**inputs) ```Nr:)r[rrr/r&r) rWrrrarr text_outputsr* text_featuresr'r'r(get_text_featureshs zCLIPModel.get_text_featuresFrrcCsN|dur|n|jj}|dur|n|jj}|j||||d}|j}||}|S)aD Returns: image_features (`torch.FloatTensor` of shape `(batch_size, output_dim`): The image embeddings obtained by applying the projection layer to the pooled output of [`CLIPVisionModel`]. Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, CLIPModel >>> model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32") >>> processor = AutoProcessor.from_pretrained("openai/clip-vit-base-patch32") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> image_features = model.get_image_features(**inputs) ```NrD)r[rrrBr&r)rWrrrrvision_outputsr*image_featuresr'r'r(get_image_featuress zCLIPModel.get_image_features return_lossc  Cs|dur|n|jj}|dur|n|jj}|j||||d} |j|||||d} | j} || } | j} || } | t| } | t| } t | | | j } | |j | j } | }d}|rft| }t||| | | | | dS)a return_loss (`bool`, *optional*): Whether or not to return the contrastive loss. Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, CLIPModel >>> model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32") >>> processor = AutoProcessor.from_pretrained("openai/clip-vit-base-patch32") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor( ... text=["a photo of a cat", "a photo of a dog"], images=image, return_tensors="pt", padding=True ... ) >>> outputs = model(**inputs) >>> logits_per_image = outputs.logits_per_image # this is the image-text similarity score >>> probs = logits_per_image.softmax(dim=1) # we can take the softmax to get the label probabilities ```NrDr:)rMrNrOrKr>rPrQ)r[rrrBr/r&rrr:r$rr+rr!rNexpr.rL)rWrrrrarUrrrrRrOr>rKrOrNrMr'r'r(rsJ'    zCLIPModel.forwardr+r@)NNNNNNNF)rBrCrDrr r;rfrrr$rrrFrQrTrrrLrrr'r'rxr(r@s# + /  rcr,)rrrr[c@t|t|}|j|_tj|j|jdd|_ | dSNFrG) rerfr-rLr/rrrgrKrr0)rWr[r/rxr'r(rf$   z$CLIPTextModelWithProjection.__init__rcCr2rr3rVr'r'r(r4/r5z0CLIPTextModelWithProjection.get_input_embeddingscCr6rr3r7r'r'r(r82r9z0CLIPTextModelWithProjection.set_input_embeddingsNrrrarrc Cs:|j|||||d}|j}||}t||j|j|jdS)a Examples: ```python >>> from transformers import AutoTokenizer, CLIPTextModelWithProjection >>> model = CLIPTextModelWithProjection.from_pretrained("openai/clip-vit-base-patch32") >>> tokenizer = AutoTokenizer.from_pretrained("openai/clip-vit-base-patch32") >>> inputs = tokenizer(["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt") >>> outputs = model(**inputs) >>> text_embeds = outputs.text_embeds ```r:)rKr?r@rA)r/r&rrJr?r@rA) rWrrrarrrOr*rKr'r'r(r5s z#CLIPTextModelWithProjection.forwardr+)rBrCrDrr r;rfrr<r4r8rrrr$rrrJrrr'r'rxr(rs4 rc sxeZdZeZdZdeffdd ZdejfddZ e e    dde e jd e ed e ed edef d dZZS)rrr[crWrX) rerfrArLrBrrrgrKrr0rWr[rBrxr'r(rferYz&CLIPVisionModelWithProjection.__init__rcCr2rrCrVr'r'r(r4pr5z2CLIPVisionModelWithProjection.get_input_embeddingsNFrrrcCs8|j||||d}|j}||}t||j|j|jdS)a Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, CLIPVisionModelWithProjection >>> model = CLIPVisionModelWithProjection.from_pretrained("openai/clip-vit-base-patch32") >>> processor = AutoProcessor.from_pretrained("openai/clip-vit-base-patch32") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> image_embeds = outputs.image_embeds ```rD)r>r?r@rA)rBr&rr=r?r@rA)rWrrrrrRr*r>r'r'r(rss z%CLIPVisionModelWithProjection.forwardr@)rBrCrDrr rErfrr<r4rrrr$rFrr=rrr'r'rxr(r`s, rz CLIP vision encoder with an image classification head on top (a linear layer on top of the pooled final hidden states of the patch tokens) e.g. for ImageNet. c sneZdZdZdeddffdd Zee    d dee j dee j dee d ee de f d d Z ZS) rrr[rNcsZt||j|_t|j}|j|_|jdkr"t|jj |jnt |_ | dS)Nr) rerf num_labelsrArLrrBrrrgIdentityrr0rZrxr'r(rfs  " z#CLIPForImageClassification.__init__labelsrrc Csr|dur|n|jj}|dur|n|jj}|j|||d}|j}tj|ddddddfdd}||}d}|dur||j }|jj durl|j dkrRd|j_ n|j dkrh|j tj ksc|j tjkrhd|j_ nd|j_ |jj dkrt} |j dkr| ||}n+| ||}n%|jj dkrt} | |d|j |d}n|jj dkrt} | ||}t|||j|jd S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the image classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). N)rrrr regressionsingle_label_classificationmulti_label_classificationr1)rMrr@rA)r[rrrBr?r$rrrr! problem_typer[rlongrr squeezer rrrr@rA) rWrr]rrrsequence_outputrrMloss_fctr'r'r(rsJ$     "       z"CLIPForImageClassification.forwardr)rBrCrDrErrfrrrr$rrrrrr'r'rxr(rs(r)rrr-rrArr)rT)DrE dataclassesrtypingrrrrr$rtorch.nnrr r activationsr modeling_attn_mask_utilsr rmodeling_layersrmodeling_outputsrrrmodeling_utilsrrutilsrrrrrconfiguration_cliprrr get_loggerrBrrr)r.r:r=rJrLr<rZrfloatrrrrrrrr r-r=rArrrr__all__r'r'r'r(s        #S/ Q2BWZ414^A@ T