o ei@sdZddlZddlmZddlmZddlmZddlZddl m Z ddl m Z ddlmZdd lmZdd lmZmZmZmZdd lmZmZdd lmZdd lmZddlmZm Z m!Z!m"Z"m#Z#m$Z$ddl%m&Z&m'Z'm(Z(e#)e*Z+dej,dej,fddZ-dej,dej,fddZ.ee!GdddeZ/Gddde j0Z1Gddde j0Z2Gddde j0Z3d e2iZ4Gd!d"d"e j0Z5Gd#d$d$e j0Z6Gd%d&d&e j0Z7Gd'd(d(eZ8Gd)d*d*e j0Z9Gd+d,d,e j0Z: -dPd.e j0d/ej,d0ej,d1ej,d2ej,dBd3e;d4e;fd5d6ZGd;d<dd>e j0Z@Gd?d@d@e j0ZAe!GdAdBdBeZBGdCdDdDe j0ZCGdEdFdFeBZDe!dGdHGdIdJdJeBZEGdKdLdLeBZFGdMdNdNeBZGgdOZHdS)QzPyTorch AltCLIP model.N)Callable) dataclass)Any)initialization)ACT2FN)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPooling,BaseModelOutputWithPoolingAndCrossAttentions'BaseModelOutputWithPoolingAndProjection)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)apply_chunking_to_forward) ModelOutputTransformersKwargsauto_docstringcan_return_tuplelogging torch_int) AltCLIPConfigAltCLIPTextConfigAltCLIPVisionConfiglogitsreturncCstj|tjt||jdS)Ndevice)nn functional cross_entropytorcharangelenr)rr%j/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/altclip/modeling_altclip.pycontrastive_loss-sr' similaritycCs t|}t|}||dS)Ng@)r't)r( caption_loss image_lossr%r%r& clip_loss1s  r,c@seZdZUdZdZejdBed<dZejdBed<dZ ejdBed<dZ ejdBed<dZ ejdBed<dZ e ed<dZe ed <d eefd d ZdS) AltCLIPOutputa loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `return_loss` is `True`): Contrastive loss for image-text similarity. logits_per_image (`torch.FloatTensor` of shape `(image_batch_size, text_batch_size)`): The scaled dot product scores between `image_embeds` and `text_embeds`. This represents the image-text similarity scores. logits_per_text (`torch.FloatTensor` of shape `(text_batch_size, image_batch_size)`): The scaled dot product scores between `text_embeds` and `image_embeds`. This represents the text-image similarity scores. text_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim`): The text embeddings obtained by applying the projection layer to the pooled output of [`AltCLIPTextModel`]. image_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim`): The image embeddings obtained by applying the projection layer to the pooled output of [`AltCLIPVisionModel`]. text_model_output (`BaseModelOutputWithPooling`): The output of the [`AltCLIPTextModel`]. vision_model_output (`BaseModelOutputWithPooling`): The output of the [`AltCLIPVisionModel`]. Nlosslogits_per_imagelogits_per_text text_embeds image_embedstext_model_outputvision_model_outputrcstfddDS)Nc3s.|]}|dvr |nt|VqdS))r3r4N)getattrto_tuple).0kselfr%r& Ws  z)AltCLIPOutput.to_tuple..)tuplekeysr9r%r9r&r6Vs zAltCLIPOutput.to_tuple)__name__ __module__ __qualname____doc__r.r" FloatTensor__annotations__r/r0r1r2r3r r4r<rr6r%r%r%r&r-7s   r-c seZdZdZfddZ     ddejdBdejdBdejdBd ejdBd ed ej f d d Z e ddZ e dddZ ZS)AltRobertaEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.csttj|j|j|jd|_t|j|j|_ tj |j|j d|_ t |j |_|jdt|jddd|jdtj|jtjddd|j|_tj|j|j|jd|_dS) N) padding_idxeps position_idsrF persistenttoken_type_idsdtype)super__init__r Embedding vocab_size hidden_size pad_token_idword_embeddingstype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutregister_bufferr"r#max_position_embeddingsexpandzerosrHsizelongrEposition_embeddingsr:config __class__r%r&rQas   zAltRobertaEmbeddings.__init__Nr input_idsrMrH inputs_embedspast_key_values_lengthrc Cs|dur|dur|||j|}n|||j}|dur!|}n|dd}|\}}|durZt|drO|j|jdd} tj | d|d} | ||}n tj |tj |j j d}|durc||}||} || } ||} | | } || } || } | S)NrJrMrr)dimindexrOr)"create_position_ids_from_input_idsrE&create_position_ids_from_inputs_embedsrbhasattrrMr`shaper"gatherrarcrHrrVrXrdrYr]) r:rirMrHrjrk input_shape batch_size seq_lengthbuffered_token_type_idsrX embeddingsrdr%r%r&forwardus2       zAltRobertaEmbeddings.forwardcCsJ|dd}|d}tj|d||dtj|jd}|d|S)z We are provided embeddings directly. We cannot infer which are padded so just generate sequential position ids. Args: inputs_embeds: torch.Tensor Returns: torch.Tensor NrJrrnr)rbr"r#rcr unsqueezer`)rjrErtsequence_lengthrHr%r%r&rps  z;AltRobertaEmbeddings.create_position_ids_from_inputs_embedscCs6||}tj|dd|||}||S)a Replace non-padding symbols with their position numbers. Position numbers begin at padding_idx+1. Padding symbols are ignored. This is modified from fairseq's `utils.make_positions`. Args: x: torch.Tensor x: Returns: torch.Tensor rrl)neintr"cumsumtype_asrc)rirErkmaskincremental_indicesr%r%r&ros  z7AltRobertaEmbeddings.create_position_ids_from_input_ids)NNNNr)r)r>r?r@rArQr" LongTensorrBr~Tensorry staticmethodrpro __classcell__r%r%rgr&rD^s2  0 rDc NeZdZfddZ  d dejdejdBdedBdeejfd d Z Z S) AltRobertaSelfAttentioncst|j|jdkrt|dstd|jd|jd|j|_t|j|j|_|j|j|_t |j|j|_ t |j|j|_ t |j|j|_ t |j|_dS)Nrembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ())rPrQrTnum_attention_headsrq ValueErrorr~attention_head_size all_head_sizerLinearquerykeyvaluer[attention_probs_dropout_probr]rergr%r&rQs  z AltRobertaSelfAttention.__init__NF hidden_statesattention_maskoutput_attentionsrcCs|jdd}g|d|jR}|||dd}|||dd}|||dd}t||dd} | t |j} |durO| |} t j j | dd} || } t| |} | dddd} | dd|jf} | | } |r| | f} | S| f} | S)NrJrr|rr)rrrrview transposerrr"matmulmathsqrtrr softmaxr]permute contiguousrbr)r:rrrrt hidden_shape query_layer key_layer value_layerattention_scoresattention_probs context_layernew_context_layer_shapeoutputsr%r%r&rys&    zAltRobertaSelfAttention.forwardNF r>r?r@rQr"rrBboolr<ryrr%r%rgr&rs rc8eZdZfddZdejdejdejfddZZS)AltRobertaSelfOutputcsBtt|j|j|_tj|j|jd|_t|j |_ dSNrF) rPrQrrrTdenserYrZr[r\r]rergr%r&rQ zAltRobertaSelfOutput.__init__r input_tensorrcC&||}||}|||}|SNrr]rYr:rrr%r%r&ry   zAltRobertaSelfOutput.forwardr>r?r@rQr"rryrr%r%rgr&r $reagerc r) AltRobertaAttentioncs(tt|j||_t||_dSr)rPrQ"ALT_ROBERTA_SELF_ATTENTION_CLASSES_attn_implementationr:routputrergr%r&rQs zAltRobertaAttention.__init__NFrrrrcCs6|j|||d}||d|}|f|dd}|SN)rrrr)r:r)r:rrr self_outputsattention_outputrr%r%r&ryszAltRobertaAttention.forwardrrr%r%rgr&rs rc2eZdZfddZdejdejfddZZS)AltRobertaIntermediatecsDtt|j|j|_t|jt rt |j|_ dS|j|_ dSr) rPrQrrrTintermediate_sizer isinstance hidden_actstrrintermediate_act_fnrergr%r&rQ.s   zAltRobertaIntermediate.__init__rrcCs||}||}|Sr)rrr:rr%r%r&ry6s  zAltRobertaIntermediate.forwardrr%r%rgr&r-s rcr)AltRobertaOutputcsBtt|j|j|_tj|j|jd|_t |j |_ dSr) rPrQrrrrTrrYrZr[r\r]rergr%r&rQ>rzAltRobertaOutput.__init__rrrcCrrrrr%r%r&ryDrzAltRobertaOutput.forwardrr%r%rgr&r=rrc s^eZdZfddZ  ddejdejdBdedBdee d e ejf d d Z d d Z Z S)AltRobertaLayercs:t|j|_d|_t||_t||_t||_ dS)Nr) rPrQchunk_size_feed_forward seq_len_dimr attentionr intermediaterrrergr%r&rQMs   zAltRobertaLayer.__init__NFrrrkwargsrc KsN|j|f||d|}|d}|dd}t|j|j|j|}|f|}|Sr)rrfeed_forward_chunkrr) r:rrrrself_attention_outputsrr layer_outputr%r%r&ryUs  zAltRobertaLayer.forwardcCs||}|||}|Sr)rr)r:rintermediate_outputrr%r%r&rls  z"AltRobertaLayer.feed_forward_chunkr)r>r?r@rQr"rrBrrrr<ryrrr%r%rgr&rLs    rcsreZdZfddZe    ddejdejdBdedBd edBd edBd e e d e eje Bfd dZ ZS)AltRobertaEncoderc:t|_tfddtjD|_d|_dS)Ncg|]}tqSr%)r)r7irfr%r& wz.AltRobertaEncoder.__init__..F) rPrQrfr ModuleListrangenum_hidden_layerslayergradient_checkpointingrergrr&rQt   zAltRobertaEncoder.__init__NFTrrroutput_hidden_states return_dictrrc Ks|rdnd}|r dnd}t|jD]"\} } |r||f}| |||fi|} | d}|r3|| df}q|r;||f}t|||dS)Nr%rrlast_hidden_stater attentions) enumeraterr ) r:rrrrrrall_hidden_statesall_self_attentionsr layer_module layer_outputsr%r%r&ryzs.   zAltRobertaEncoder.forward)NFFT)r>r?r@rQrr"rrBrrrr<r ryrr%r%rgr&rss,  rcr)AltRobertaPoolercs*tt|j|j|_t|_dSr)rPrQrrrTrTanh activationrergr%r&rQs zAltRobertaPooler.__init__rrcCs(|dddf}||}||}|S)Nr)rr)r:rfirst_token_tensor pooled_outputr%r%r&rys  zAltRobertaPooler.forwardrr%r%rgr&rs rmodulerrrrscalingr]c Ks|t||dd|}|dur||}tjj|dtjd|j}tjj |||j d}t||} | dd } | |fS)NrJr)rlrO)ptrainingrr) r"rrrr rfloat32torOr]rr) rrrrrrr]r attn_weights attn_outputr%r%r&eager_attention_forwards  rcsdeZdZdZfddZ  d dejdejdBdedBd ee d e ejejdBff d d Z Z S)AltCLIPAttentionz=Multi-headed attention from 'Attention Is All You Need' papercst||_|j|_|j|_|j|j|_|j|j|jkr-td|jd|jd|jd|_ |j |_ d|_ t |j|j|_t |j|j|_t |j|j|_t |j|j|_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: ).F)rPrQrfrT embed_dimr num_headshead_dimrscaleattention_dropoutr] is_causalrrk_projv_projq_projout_projrergr%r&rQs$   zAltCLIPAttention.__init__NFrrrrrcKs|j\}}}||}||} ||} ||||j|jdd}| |||j|jdd} | |||j|jdd} t |j j t } | ||| | |f|j |jsUdn|jd|\} } | |||} || } |sqd} | | fS)z#Input shape: Batch x Time x Channelrrr)rr]N)rrrrrrrrrr get_interfacerfrrrrr]reshaperr)r:rrrrrurvrqueriesr=valuesattention_interfacerrr%r%r&rys6     zAltCLIPAttention.forwardr) r>r?r@rArQr"rrrrr<ryrr%r%rgr&rs  rcr) AltCLIPMLPcsDt||_t|j|_t|j|j |_ t|j |j|_ dSr) rPrQrfrr activation_fnrrrTrfc1fc2rergr%r&rQ s  zAltCLIPMLP.__init__rrcCs"||}||}||}|Sr)rrrrr%r%r&rys   zAltCLIPMLP.forwardrr%r%rgr&r  s r c sVeZdZdeffdd Z d dejdejdedBd ee d e ej f d d Z Z S)AltCLIPEncoderLayerrfcsRt|j|_t||_tj|j|jd|_ t ||_ tj|j|jd|_ dSr) rPrQrTrr self_attnrrYrZ layer_norm1r mlp layer_norm2rergr%r&rQs   zAltCLIPEncoderLayer.__init__FrrrNrrcKsj|}||}|jd|||d|\}}||}|}||}||}||}|f}|r3||f7}|S)aI Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. `(config.encoder_attention_heads,)`. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. )rrrNr%)rrrr)r:rrrrresidualrrr%r%r&ry!s$     zAltCLIPEncoderLayer.forwardF)r>r?r@rrQr"rrrrr<rBryrr%r%rgr&rs rcspeZdZdZdeffdd Ze    ddejdBde dBde dBd e dBd e e d e e Bf d d ZZS)AltCLIPEncoderz Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`AltCLIPEncoderLayer`]. Args: config: AltCLIPConfig rfcr)Ncrr%)r)r7_rr%r&rVrz+AltCLIPEncoder.__init__..F) rPrQrfrrrrlayersrrergrr&rQSrzAltCLIPEncoder.__init__Nrrrrrrc Ks|dur|n|jj}|dur|n|jj}|dur|n|jj}|r"dnd}|r(dnd}|} t|jD]#\} } |r<|| f}| | |fd|i|} | d} |rT|| df}q1|r\|| f}t| ||dS)ad Args: inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. Nr%rrrr)rfrruse_return_dictrrr ) r:rjrrrrrencoder_statesall_attentionsridx encoder_layerrr%r%r&ryYs6     zAltCLIPEncoder.forward)NNNN)r>r?r@rArrQrr"rrrrr<r ryrr%r%rgr&rJs*rcsXeZdZdeffdd Zdejdededejfdd Zdd ej dejfd d Z Z S)AltCLIPVisionEmbeddingsrfcst||_|j|_|j|_|j|_tt |j|_ tj |j |j|j|jdd|_|j|jd|_|jd|_t|j|j|_|jdt |jddddS)NF) in_channels out_channels kernel_sizestridebiasrrrHrIrK)rPrQrfrTr image_size patch_sizer Parameterr"randnclass_embeddingConv2d num_channelspatch_embedding num_patches num_positionsrRposition_embeddingr^r#r`rergr%r&rQs"  "z AltCLIPVisionEmbeddings.__init__rxheightwidthrc Cs|jdd}|jjd}|jdd}tjs(||kr(||kr(||jS|ddddf}|ddddf}|jd} ||j} ||j} t |d} | d| | | }| dddd}t j j|| | fdd d }| dddddd| }tj||fdd S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rrNrJg?rrbicubicF)rbmode align_cornersr|)rrr0weightrzr"jit is_tracingrHr'rr rrr interpolatercat) r:rxr1r2r.r0r/class_pos_embedpatch_pos_embedrl new_height new_widthsqrt_num_positionsr%r%r&interpolate_pos_encodings*      z0AltCLIPVisionEmbeddings.interpolate_pos_encodingF pixel_valuesc Cs|j\}}}}|s&||jks||jkr&td|d|d|jd|jd |jjj}||j|d}|ddd}|j |dd} t j | |gdd } |r[| | | ||} | S| ||j} | S) NzInput image size (*z) doesn't match model (rrNrrrJr|)rrr&rr-r6rOrflattenrr*r`r"r:r@r0rH) r:rAr@rurr1r2 target_dtype patch_embeds class_embedsrxr%r%r&rys  zAltCLIPVisionEmbeddings.forwardr) r>r?r@rrQr"rr~r@rBryrr%r%rgr&r s )r c@s6eZdZUeed<dZdZdZgZe ddZ dS)AltCLIPPreTrainedModelrfaltclip)imagetextTcCs|jj}t|trE|jj}tj|jd|jd|dtj|jj |jj |dtj|j j |jj |dt |j t|jddSt|tr|jj}|jdd|jjd|}|jd|}tj|jj |dtj|jj |dtj|jj |dtj|jj |ddSt|tr|jj}|jjdd|jjd|}d|jjd|}tj|jj |dtj|jj |ddSt|trtj|jj |jd|jjdtj|jj |j d|jjddSt|t!j"rt#|j$t%|j dSt|t!j&rtj|j d|jjd|j$durt#|j$dSdSt|t!j'rMtj|j d|jjd|j(durIt)|j dd sKt#|j |j(dSdSdSt|t*rlt |j t|j j+d dt#|j,dSdS) zInitialize the weightsrr)meanstd)rLrIrN_is_hf_initializedFrJ)-rfinitializer_factorrr initnormal_r*rr-r6initializer_ranger0copy_rHr"r#r/r`rrrrrrr rTrr AltCLIPModeltext_projectiontext_embed_dimvisual_projectionvision_embed_dimrrYzeros_r%ones_rrRrEr5rDrrrM)r:rfactor in_proj_std out_proj_stdfc_stdr%r%r& _init_weightss^          "z$AltCLIPPreTrainedModel._init_weightsN) r>r?r@rrCbase_model_prefixinput_modalitiessupports_gradient_checkpointing_no_split_moduler"no_gradr^r%r%r%r&rGs rGcsreZdZdeffdd Zee     ddejdBde dBde dBd e dBd e dBd e e Bf d d Z Z S)AltCLIPVisionTransformerrfcsRt||_|j}t||_tj||jd|_ t ||_ tj||jd|_ dSr) rPrQrfrTr rxrrYrZ pre_layrnormrencoderpost_layernorm)r:rfrrgr%r&rQ)s   z!AltCLIPVisionTransformer.__init__NFrArrrr@rc Cs|dur|n|jj}|dur|n|jj}|dur|n|jj}|dur&td|j||d}||}|j|||dd}|d}|dddddf} || } t || |j |j dS)Nz You have to specify pixel_values)r@T)rjrrrrr pooler_outputrr) rfrrrrrxrerfrgr rr) r:rArrrr@rencoder_outputsrrr%r%r&ry3s.   z AltCLIPVisionTransformer.forward)NNNNF)r>r?r@rrQrrr"rBrr<r ryrr%r%rgr&rd(s, rdcseZdZUeed<dZdZdeffdd Zdej fddZ e dde j d Bd ed Bd ed Bd eded BdeedeeBfddZZS)AltCLIPVisionModelrfrA)rIcs"t|t||_|dSr)rPrQrd vision_model post_initrergr%r&rQas   zAltCLIPVisionModel.__init__rcC |jjjSr)rlrxr-r9r%r%r&get_input_embeddingsg z'AltCLIPVisionModel.get_input_embeddingsNFrrr@rrcKs(|dur|n|jj}|j|||||dS)a Examples: ```python >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> from transformers import AutoProcessor, AltCLIPVisionModel >>> model = AltCLIPVisionModel.from_pretrained("BAAI/AltCLIP") >>> processor = AutoProcessor.from_pretrained("BAAI/AltCLIP") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> inputs = processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state >>> pooled_output = outputs.pooler_output # pooled CLS states ```NrArrr@r)rfrrl)r:rArrr@rrr%r%r&ryjs zAltCLIPVisionModel.forward)NNNFN)r>r?r@rrCmain_input_namer`rQrModulerorr"rBrrrr<r ryrr%r%rgr&rk\s6 rkaE The model behaves as an encoder following the architecture described in *Attention is all you need*_ by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin. .. _*Attention is all you need*: https://huggingface.co/papers/1706.03762 ) custom_introcseZdZUeed<dfdd ZddZddZe dd e j d Bd e j d Bd e j d Bd e j d Bde j d Bde d Bde d Bde d Bde e j e BfddZZS)AltRobertaModelrfTcsDt|||_t||_t||_|rt|nd|_| dS)zv add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer N) rPrQrfrDrxrrfrpoolerrm)r:rfadd_pooling_layerrgr%r&rQs    zAltRobertaModel.__init__cCs|jjSrrxrVr9r%r%r&rosz$AltRobertaModel.get_input_embeddingscCs ||j_dSrrxr:rr%r%r&set_input_embeddings z$AltRobertaModel.set_input_embeddingsNrirrMrHrjrrrrc Ks|dur|n|jj}|dur|n|jj}|dur|n|jj}|dur*|dur*td|dur9||||} n|durF|dd} ntd| \} } |durU|jn|j} |duretj | | f| d}|durt |j dr|j j ddd| f}| | | }|}n tj| tj| d}||| }|j ||||d}|j||||dd }|d }|jdur||nd}t|||j|jd S) NzDYou cannot specify both input_ids and inputs_embeds at the same timerJz5You have to specify either input_ids or inputs_embedsrrMrn)rirHrMrjT)rrrrrrh)rfrrrr%warn_if_padding_and_no_attention_maskrbrr"onesrqrxrMr`rarcget_extended_attention_maskrfrvr rr)r:rirrMrHrjrrrrrtrurvrrw buffered_token_type_ids_expandedextended_attention_maskembedding_outputrjsequence_outputrr%r%r&rysX     zAltRobertaModel.forward)TNNNNNNNN)r>r?r@rrCrQrorzrr"rrr<r ryrr%r%rgr&rusB    rucseZdZUeed<dZfddZdejfddZ dej dd fd d Z dd e d Bdej ffd d Z ee ddejd Bdejd Bdejd Bdejd Bdejd Bded Bded Bded BdeedeeBfddZZS)AltCLIPTextModelrf)rJcsLt|t|dd|_t|j|j|_tj |j|j d|_ | dS)NF)rwrF) rPrQrurobertarrrT project_dimtransformationrYrZpre_LNrmrergr%r&rQs  zAltCLIPTextModel.__init__rcCrnrrrxrVr9r%r%r&ro rpz%AltCLIPTextModel.get_input_embeddingsrNcCs||jj_dSrrryr%r%r&rzsz%AltCLIPTextModel.set_input_embeddingsnew_num_tokenscs t|Sr)rPresize_token_embeddings)r:rrgr%r&rr{z(AltCLIPTextModel.resize_token_embeddingsrirrMrHrjrrrrc  Ksn|dur|n|jj}|j|||||||dd} | d} || } || } | dddf} t| | | j| jdS)a+ Examples: ```python >>> from transformers import AutoProcessor, AltCLIPTextModel >>> model = AltCLIPTextModel.from_pretrained("BAAI/AltCLIP") >>> processor = AutoProcessor.from_pretrained("BAAI/AltCLIP") >>> texts = ["it's a cat", "it's a dog"] >>> inputs = processor(text=texts, padding=True, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state >>> pooled_output = outputs.pooler_output # pooled CLS states ```NT)rirrMrHrjrrrrrh)rfrrrrr rr)r:rirrMrHrjrrrrrrprojection_staterir%r%r&rys*   zAltCLIPTextModel.forwardrr)r>r?r@rrCr`rQrrsrorRrzr~rrrr"rrrrr<r ryrr%r%rgr&rsL     rcs4eZdZUeed<deffdd Zee   ddej dej dBdej dBdej dBd e e d e e Bf d d Zee ddejded e e d e e BfddZe         ddejdBdejdBdej dBdejdBdej dBdedBdedBdedBdededBd e e d e eBfddZZS)rSrfcst|t|jtstdt|jdt|jts(tdt|jd|j}|j}|j |_ |j |_ |j |_ |j |_t||_t||_tj|j|j dd|_tj|j |j dd|_tt|jj|_|dS)NzRconfig.vision_config is expected to be of type AltCLIPVisionConfig but is of type .zNconfig.text_config is expected to be of type AltCLIPTextConfig but is of type F)r%)rPrQr vision_configr TypeErrortype text_configrrprojection_dimrrUrTrWr text_modelrdrlrrrVrTr(r"tensorrflogit_scale_init_value logit_scalerm)r:rfrrrgr%r&rQVs2      zAltCLIPModel.__init__NrirrHrMrrcKs2|jd||||dd|}|j}|||_|S)a Examples: ```python >>> import torch >>> from transformers import AutoProcessor, AltCLIPModel >>> model = AltCLIPModel.from_pretrained("BAAI/AltCLIP") >>> processor = AutoProcessor.from_pretrained("BAAI/AltCLIP") >>> inputs = processor(text=["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt") >>> with torch.inference_mode(): ... text_features = model.get_text_features(**inputs) ```T)rirrHrMrNr%)rrirT)r:rirrHrMr text_outputsrr%r%r&get_text_featuresws zAltCLIPModel.get_text_featuresFrAr@cKs.|jd||dd|}|j}|||_|S)ao Examples: ```python >>> import torch >>> from transformers import AutoProcessor, AltCLIPModel >>> from transformers.image_utils import load_image >>> model = AltCLIPModel.from_pretrained("BAAI/AltCLIP") >>> processor = AutoProcessor.from_pretrained("BAAI/AltCLIP") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = load_image(url) >>> inputs = processor(images=image, return_tensors="pt") >>> with torch.inference_mode(): ... image_features = model.get_image_features(**inputs) ```T)rAr@rNr%)rlrirV)r:rAr@rvision_outputsrr%r%r&get_image_featuress zAltCLIPModel.get_image_features return_lossrrrc  Ks(|dur|n|jj}|dur|n|jj}| dur| n|jj} |j||||||| d} |j|||| | d} | d}||}| d}||}||jdddd}||jdddd}|j }t || |}|j}d}|rtt|}| s||||| | f}|dur|f|S|St|||||| | d S) a return_loss (`bool`, *optional*): Whether or not to return the contrastive loss. Examples: ```python >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> from transformers import AutoProcessor, AltCLIPModel >>> model = AltCLIPModel.from_pretrained("BAAI/AltCLIP") >>> processor = AutoProcessor.from_pretrained("BAAI/AltCLIP") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> inputs = processor( ... text=["a photo of a cat", "a photo of a dog"], images=image, return_tensors="pt", padding=True ... ) >>> outputs = model(**inputs) >>> logits_per_image = outputs.logits_per_image # this is the image-text similarity score >>> probs = logits_per_image.softmax(dim=1) # we can take the softmax to get the label probabilities ```N)rirrMrHrrrrqrrrJT)rrlkeepdim)r.r/r0r1r2r3r4)rfrrrrrlrVrTnormrexpr"rr)Tr,r-)r:rirArrHrMrrrr@rrrrr2r1rr0r/r.rr%r%r&rysX(    zAltCLIPModel.forward)NNNr) NNNNNNNNFN)r>r?r@rrCrQrrr"rrrr<r rrBrrrr-ryrr%r%rgr&rSSs !##     rS)rGrkrrS)r)IrArcollections.abcr dataclassesrtypingrr"torch.nnrrrO activationsrmodeling_layersrmodeling_outputsr r r r modeling_utilsr rprocessing_utilsr pytorch_utilsrutilsrrrrrrconfiguration_altcliprrr get_loggerr>loggerrr'r,r-rsrDrrrrrrrrrfloatrrr rrr rGrdrkrurrS__all__r%r%r%r&s            $j;'. A2QS949 bS Q