o wie@sdZddlZddlZddlmZddlmZmZmZm Z ddl Z ddl Z ddl m Z ddl mZddlmZmZdd lmZdd lmZmZdd lmZmZdd lmZmZmZmZd dlm Z m!Z!m"Z"e#e$Z%de j&de j&fddZ'de j&de j&fddZ(eeGdddeZ)eeGdddeZ*eeGdddeZ+Gddde j,Z-Gddde j,Z. dGd!e j,d"e j&d#e j&d$e j&d%ee j&d&e/d'e/fd(d)Z0Gd*d+d+e j,Z1Gd,d-d-e j,Z2Gd.d/d/eZ3eGd0d1d1eZ4Gd2d3d3e j,Z5Gd4d5d5e j,Z6Gd6d7d7e4Z7Gd8d9d9e j,Z8Gd:d;d;e4Z9eGdd?d?e j,Z;Gd@dAdAe4ZdS)HzPyTorch CLIPSeg model.N) dataclass)AnyCallableOptionalUnion)nn)ACT2FN) _create_4d_causal_attention_mask_prepare_4d_attention_mask)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPooling)ALL_ATTENTION_FUNCTIONSPreTrainedModel) ModelOutputauto_docstringlogging torch_int) CLIPSegConfigCLIPSegTextConfigCLIPSegVisionConfiglogitsreturncCstj|tjt||jdS)Ndevice)r functional cross_entropytorcharangelenr)rr"i/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/transformers/models/clipseg/modeling_clipseg.pycontrastive_loss(sr$ similaritycCs t|}t|}||dS)Ng@)r$t)r% caption_loss image_lossr"r"r# clipseg_loss-s  r)c@seZdZUdZdZeejed<dZ eejed<dZ eejed<dZ eejed<dZ eejed<dZ eed<dZeed <d eefd d ZdS) CLIPSegOutputa loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `return_loss` is `True`): Contrastive loss for image-text similarity. logits_per_image (`torch.FloatTensor` of shape `(image_batch_size, text_batch_size)`): The scaled dot product scores between `image_embeds` and `text_embeds`. This represents the image-text similarity scores. logits_per_text (`torch.FloatTensor` of shape `(text_batch_size, image_batch_size)`): The scaled dot product scores between `text_embeds` and `image_embeds`. This represents the text-image similarity scores. text_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim`): The text embeddings obtained by applying the projection layer to the pooled output of [`CLIPSegTextModel`]. image_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim`): The image embeddings obtained by applying the projection layer to the pooled output of [`CLIPSegVisionModel`]. text_model_output (`BaseModelOutputWithPooling`): The output of the [`CLIPSegTextModel`]. vision_model_output (`BaseModelOutputWithPooling`): The output of the [`CLIPSegVisionModel`]. Nlosslogits_per_imagelogits_per_text text_embeds image_embedstext_model_outputvision_model_outputrctfddDS)Nc3.|]}|dvr |nt|VqdS))r0r1Ngetattrto_tuple.0kselfr"r# S  z)CLIPSegOutput.to_tuple..tuplekeysr:r"r:r#r6R zCLIPSegOutput.to_tuple)__name__ __module__ __qualname____doc__r+rr FloatTensor__annotations__r,r-r.r/r0rr1r?rr6r"r"r"r#r*3s   r*c@sPeZdZUdZdZeejed<dZ ee ejed<dZ ee ejed<dS)CLIPSegDecoderOutputz| logits (`torch.FloatTensor` of shape `(batch_size, height, width)`): Classification scores for each pixel. Nr hidden_states attentions) rBrCrDrErrrrFrGrIr?rJr"r"r"r#rHYs rHc@seZdZUdZdZeejed<dZ eejed<dZ eejed<dZ eejed<dZ e ed<dZeed<d eefd d ZdS) CLIPSegImageSegmentationOutputa loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Binary cross entropy loss for segmentation. logits (`torch.FloatTensor` of shape `(batch_size, height, width)`): Classification scores for each pixel. conditional_embeddings (`torch.FloatTensor` of shape `(batch_size, projection_dim)`): Conditional embeddings used for segmentation. pooled_output (`torch.FloatTensor` of shape `(batch_size, embed_dim)`): Pooled output of the [`CLIPSegVisionModel`]. vision_model_output (`BaseModelOutputWithPooling`): The output of the [`CLIPSegVisionModel`]. decoder_output (`CLIPSegDecoderOutput`): The output of the [`CLIPSegDecoder`]. Nr+rconditional_embeddings pooled_outputr1decoder_outputrcr2)Nc3r3))r1rNNr4r7r:r"r#r<r=z:CLIPSegImageSegmentationOutput.to_tuple..r>r:r"r:r#r6rAz'CLIPSegImageSegmentationOutput.to_tuple)rBrCrDrEr+rrrFrGrrLrMr1rrNrHr?rr6r"r"r"r#rKfs   rKcsXeZdZdeffdd Zdejdededejfdd Zdd ej dejfd d Z Z S)CLIPSegVisionEmbeddingsconfigcst||_|j|_|j|_|j|_tt |j|_ tj |j |j|j|jdd|_|j|jd|_|jd|_t|j|j|_|jdt |jddddS)NF) in_channels out_channels kernel_sizestridebiasr position_idsr persistent)super__init__rP hidden_size embed_dim image_size patch_sizer Parameterrrandnclass_embeddingConv2d num_channelspatch_embedding num_patches num_positions Embeddingposition_embeddingregister_bufferr expandr;rP __class__r"r#r]s"  "z CLIPSegVisionEmbeddings.__init__ embeddingsheightwidthrc Cs|jdd}|jjd}|jdd}tjs(||kr(||kr(||jS|ddddf}|ddddf}|jd} ||j} ||j} t |d} | d| | | }| dddd}t j j|| | fdd d }| dddddd| }tj||fdd S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rrNrYg?rrVbicubicF)sizemode align_cornersdim)shaperkweight unsqueezerjit is_tracingrWrarreshapepermuterr interpolateviewcat) r;rqrrrsrhrkriclass_pos_embedpatch_pos_embedry new_height new_widthsqrt_num_positionsr"r"r#interpolate_pos_encodings*      z0CLIPSegVisionEmbeddings.interpolate_pos_encodingT pixel_valuesc Cs|j\}}}}|s&||jks||jkr&td|d|d|jd|jd ||}|ddd}|j|dd}tj ||gdd} |rR| | | ||} | S| | |j } | S) NzInput image size (*z) doesn't match model ().rVrrYrx) rzr` ValueErrorrgflatten transposerdrmrrrrkrW) r;rr batch_size_rrrs patch_embeds class_embedsrqr"r"r#forwards  zCLIPSegVisionEmbeddings.forward)T) rBrCrDrr]rTensorintrrFr __classcell__r"r"ror#rOs )rOc sXeZdZdeffdd Z   d deejdeejdeejdej fd d Z Z S) CLIPSegTextEmbeddingsrPcsRt|j}t|j||_t|j||_|j dt |j ddddS)NrWrXFrZ) r\r]r^rrj vocab_sizetoken_embeddingmax_position_embeddingsrkrlrr rmr;rPr_ror"r#r]s  zCLIPSegTextEmbeddings.__init__N input_idsrW inputs_embedsrcCs|dur |jdn|jd}|jjjd}||kr#td|d||dur2|jddd|f}|dur;||}||}||}|S)NrYrzRSequence length must be less than max_position_embeddings (got `sequence length`: z and max_position_embeddings: )rzrkr{rrWr)r;rrWr seq_lengthmax_position_embeddingposition_embeddingsrqr"r"r#rs"  zCLIPSegTextEmbeddings.forward)NNN) rBrCrDrr]rr LongTensorrFrrrr"r"ror#rsrmodulequerykeyvalueattention_maskscalingdropoutc Ks|t||dd|}|dur||}tjj|dtjd|j}tjj |||j d}t||} | dd } | |fS)NrYr)rydtype)ptrainingrrV) rmatmulrrrsoftmaxfloat32torrr contiguous) rrrrrrrkwargs attn_weights attn_outputr"r"r#eager_attention_forwards  rcsveZdZdZdeeefffdd Z   ddej de ej d e ej d e e d e ej e ej ff d d Z ZS)CLIPSegAttentionz=Multi-headed attention from 'Attention Is All You Need' paperrPcst||_|j|_|j|_|j|j|_|j|j|jkr-td|jd|jd|jd|_ |j |_ d|_ t |j|j|_t |j|j|_t |j|j|_t |j|j|_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: rF)r\r]rPr^r_num_attention_heads num_headshead_dimrscaleattention_dropoutr is_causalrLineark_projv_projq_projout_projrnror"r#r]s$   zCLIPSegAttention.__init__NFrIrcausal_attention_maskoutput_attentionsrc CsL|j\}}}||}||} ||} ||||j|jdd}| |||j|jdd} | |||j|jdd} |jj dkrY|durR|durR||}n |durX|}n|du|_ t } |jj dkrz|jj dkrt|rtt dnt|jj } | ||| | ||j |j|jsdn|jd \} } | |||} || } |sd} | | fS) z#Input shape: Batch x Time x ChannelrrVflash_attention_2Neagersdpaz`torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to eager attention. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.r)rrr)rzrrrrrrrrP_attn_implementationrrlogger warning_oncerrrrrrr)r;rIrrrrrr_queriesr@valuesattention_interfacerrr"r"r#r0sH          zCLIPSegAttention.forward)NNF)rBrCrDrErrrr]rrrboolr?rrr"r"ror#rs"rcs2eZdZfddZdejdejfddZZS) CLIPSegMLPcsDt||_t|j|_t|j|j |_ t|j |j|_ dSN) r\r]rPr hidden_act activation_fnrrr^intermediate_sizefc1fc2rnror"r#r]ks  zCLIPSegMLP.__init__rIrcCs"||}||}||}|Sr)rrr)r;rIr"r"r#rrs   zCLIPSegMLP.forward)rBrCrDr]rrrrr"r"ror#rjs rc sTeZdZdeffdd Z d dejdejdejdeed e ej f d d Z Z S) CLIPSegEncoderLayerrPcRt|j|_t||_tj|j|jd|_ t ||_ tj|j|jd|_ dSN)eps r\r]r^r_r self_attnr LayerNormlayer_norm_eps layer_norm1rmlp layer_norm2rnror"r#r]{   zCLIPSegEncoderLayer.__init__FrIrrrrcCsd|}||}|j||||d\}}||}|}||}||}||}|f}|r0||f7}|SaI Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. `(config.encoder_attention_heads,)`. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. )rIrrr)rrrrr;rIrrrresidualroutputsr"r"r#rs"     zCLIPSegEncoderLayer.forwardF) rBrCrDrr]rrrrr?rFrrr"r"ror#rzs rc@s eZdZeZdZdZddZdS)CLIPSegPreTrainedModelclipTcCsV|jj}t|tr"|jjjjd|dd|jjjjd|ddnt|t rW|jj}t j j|j d|j d|dt j j|jj|jj|dt j j|jj|jj|dnt|tr|jj}|j dd|jjd|}|j d|}t j j|jj|dt j j|jj|dt j j|jj|dt j j|jj|dn_t|tr|jj}|jjdd|jjd|}d|jjd|}t j j|jj|dt j j|jj|dn't|trt j j|jj|jd|jjdt j j|jj|jd|jjdt|t jr|j j!|jj"dt|t j#r'|j dur)|j j!dSdSdS) zInitialize the weightsrg{Gz?)meanstdr)rrVg?N)$rPinitializer_factor isinstancerrr{datanormal_rkrOrinitrdr_rginitializer_rangernum_hidden_layersrrrrrr^rr CLIPSegModeltext_projectiontext_embed_dimvisual_projectionvision_embed_dimrrUzero_fill_r)r;rfactor in_proj_std out_proj_stdfc_stdr"r"r# _init_weightssL       z$CLIPSegPreTrainedModel._init_weightsN)rBrCrDr config_classbase_model_prefixsupports_gradient_checkpointingrr"r"r"r#rs  rcsteZdZdZdeffdd Z     ddeejdeejdee d ee d ee d e e e ff d d Z ZS)CLIPSegEncoderz Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`CLIPSegEncoderLayer`]. Args: config: CLIPSegConfig rPcs:t|_tfddtjD|_d|_dS)Ncg|]}tqSr")rr8rrPr"r# z+CLIPSegEncoder.__init__..F) r\r]rPr ModuleListrangerlayersgradient_checkpointingrnrorr#r]s   zCLIPSegEncoder.__init__Nrrroutput_hidden_states return_dictrc Cs|dur|n|jj}|dur|n|jj}|dur|n|jj}|r"dnd}|r(dnd}|} t|jD] \} } |r<|| f}| | |||d} | d} |rQ|| df}q1|rY|| f}|sgtdd| ||fDSt| ||dS) a Args: inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) causal_attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Causal mask for the text model. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. Nr")rrrcs|] }|dur|VqdSrr"r8vr"r"r#r<.z)CLIPSegEncoder.forward..)last_hidden_staterIrJ)rPrr use_return_dict enumerater r?r ) r;rrrrr rencoder_statesall_attentionsrIidx encoder_layer layer_outputsr"r"r#rs6&    zCLIPSegEncoder.forwardNNNNN)rBrCrDrErr]rrrrrr?r rrr"r"ror#rs*  rcseZdZdeffdd Ze      ddeejdeejdeejdee d ee d ee d e e e ffd d Z ZS)CLIPSegTextTransformerrPcsHt||_|j}t||_t||_tj ||j d|_ |j |_ dSr) r\r]rPr^rrqrencoderrrrfinal_layer_norm eos_token_idrror"r#r]5s    zCLIPSegTextTransformer.__init__NrrrWrr rrc Csj|dur|n|jj}|dur|n|jj}|dur|n|jj}|dur&td|}|d|d}|j||d}t||j |j d} |durLt ||j }|j ||| |||d} | d} | | } |jdkr| tj| jd| j d|jtj| j djdd f} n| tj| jd| j d|jtj| j d|jkjdd f} |s| | f| d dSt| | | j| jd S) NzYou have to specify input_idsrY)rrWr)rrrrr rrrV)rrrxrr pooler_outputrIrJ)rPrr rrrurrqr rrr rrrrr rzrrargmaxrrIrJ) r;rrrWrr r input_shaperIrencoder_outputsrrMr"r"r#r@s\      zCLIPSegTextTransformer.forwardNNNNNN)rBrCrDrr]rrrrrrr?rrrr"r"ror#r4s0  rcseZdZeZddgZdeffdd ZdejfddZ d d Z e dd e e jd e e jde e jde ede ede edeeeffddZZS)CLIPSegTextModelrrrPc"t|t||_|dSr)r\r]r text_model post_initrnror"r#r]   zCLIPSegTextModel.__init__rcC |jjjSrr(rqrr:r"r"r#get_input_embeddings z%CLIPSegTextModel.get_input_embeddingscCs||jj_dSrr,)r;rr"r"r#set_input_embeddingssz%CLIPSegTextModel.set_input_embeddingsNrrrWrr rcCs|j||||||dS)a; Examples: ```python >>> from transformers import AutoTokenizer, CLIPSegTextModel >>> tokenizer = AutoTokenizer.from_pretrained("CIDAS/clipseg-rd64-refined") >>> model = CLIPSegTextModel.from_pretrained("CIDAS/clipseg-rd64-refined") >>> inputs = tokenizer(["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state >>> pooled_output = outputs.pooler_output # pooled (EOS token) states ```rrrWrr r)r()r;rrrWrr rr"r"r#rszCLIPSegTextModel.forwardr%)rBrCrDrr_no_split_modulesr]rModuler-r/rrrrrrr?rrrr"r"ror#r&s8 r&cspeZdZdeffdd Ze    ddeejdee dee d ee d ee d e e e ff d d Z ZS)CLIPSegVisionTransformerrPcsRt||_|j}t||_tj||jd|_ t ||_ tj||jd|_ dSr) r\r]rPr^rOrqrrr pre_layrnormrrpost_layernormrror"r#r]s   z!CLIPSegVisionTransformer.__init__NTrrr rrrc Cs|dur|n|jj}|dur|n|jj}|dur|n|jj}|j||d}||}|j||||d}|d}|dddddf} || } |sS|| f|ddSt|| |j |j dS)N)r)rrr rrrr ) rPrr rrqr4rr5rrIrJ) r;rrr rrrIr$rrMr"r"r#rs.   z CLIPSegVisionTransformer.forward)NNNT)rBrCrDrr]rrrrFrrr?rrrr"r"ror#r3s(  r3cseZdZeZdZdeffdd ZdejfddZ e    dde e j d e ed e ed e ed e edeeeff ddZZS)CLIPSegVisionModelrrPcr'r)r\r]r3 vision_modelr)rnror"r#r]r*zCLIPSegVisionModel.__init__rcCr+r)r7rqrgr:r"r"r#r-r.z'CLIPSegVisionModel.get_input_embeddingsNTrr rrcCs|j|||||dS)a Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, CLIPSegVisionModel >>> processor = AutoProcessor.from_pretrained("CIDAS/clipseg-rd64-refined") >>> model = CLIPSegVisionModel.from_pretrained("CIDAS/clipseg-rd64-refined") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state >>> pooled_output = outputs.pooler_output # pooled CLS states ```rrr rr)r7)r;rrr rrr"r"r#rszCLIPSegVisionModel.forwardNNNTN)rBrCrDrrmain_input_namer]rr2r-rrrrFrrr?rrrr"r"ror#r6s0 r6cs>eZdZeZdeffdd Ze      ddeej deej deej dee d ee d ee d ej fd d Z e     ddeej dee d ee de d ee d ej f ddZ e         ddeejdeej deej deejdee dee d ee de d ee d eeeffddZZS)rrPcst|t|jtstdt|jdt|jts(tdt|jd|j}|j}|j |_ |j |_ |j |_ t ||_t||_tj|j |j dd|_tj|j |j dd|_tt|jj|_|dS)NzNconfig.text_config is expected to be of type CLIPSegTextConfig but is of type .zRconfig.vision_config is expected to be of type CLIPSegVisionConfig but is of type F)rU)r\r]r text_configr TypeErrortype vision_configrprojection_dimr^rrrr(r3r7rrrrrbrtensorrPlogit_scale_init_value logit_scaler))r;rPr<r?ror"r#r].s0      zCLIPSegModel.__init__NrrrWrr rrc Csh|dur|n|jj}|dur|n|jj}|dur|n|jj}|j||||||d}|d}||} | S)a Returns: text_features (`torch.FloatTensor` of shape `(batch_size, output_dim`): The text embeddings obtained by applying the projection layer to the pooled output of [`CLIPSegTextModel`]. Examples: ```python >>> from transformers import AutoTokenizer, CLIPSegModel >>> tokenizer = AutoTokenizer.from_pretrained("CIDAS/clipseg-rd64-refined") >>> model = CLIPSegModel.from_pretrained("CIDAS/clipseg-rd64-refined") >>> inputs = tokenizer(["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt") >>> text_features = model.get_text_features(**inputs) ```Nr0r)rPrr rr(r) r;rrrWrr r text_outputsrM text_featuresr"r"r#get_text_featuresNs zCLIPSegModel.get_text_featuresTrrc Csf|dur|n|jj}|dur|n|jj}|dur|n|jj}|j|||||d}|d}||}|S)aI Returns: image_features (`torch.FloatTensor` of shape `(batch_size, output_dim`): The image embeddings obtained by applying the projection layer to the pooled output of [`CLIPSegVisionModel`]. Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, CLIPSegModel >>> processor = AutoProcessor.from_pretrained("CIDAS/clipseg-rd64-refined") >>> model = CLIPSegModel.from_pretrained("CIDAS/clipseg-rd64-refined") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> image_features = model.get_image_features(**inputs) ```Nr8r)rPrr rr7r) r;rrr rrvision_outputsrMimage_featuresr"r"r#get_image_features}s  zCLIPSegModel.get_image_features return_lossc  Cs(|dur|n|jj}|dur|n|jj}| dur| n|jj} |j||||| d} |j|||||| d} | d} || } | d} || } | | jdddd} | | jdddd} |j }t | | |}| }d}|rtt|}| s||| | | | f}|dur|f|S|St|||| | | | d S) a return_loss (`bool`, *optional*): Whether or not to return the contrastive loss. Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, CLIPSegModel >>> processor = AutoProcessor.from_pretrained("CIDAS/clipseg-rd64-refined") >>> model = CLIPSegModel.from_pretrained("CIDAS/clipseg-rd64-refined") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor( ... text=["a photo of a cat", "a photo of a dog"], images=image, return_tensors="pt", padding=True ... ) >>> outputs = model(**inputs) >>> logits_per_image = outputs.logits_per_image # this is the image-text similarity score >>> probs = logits_per_image.softmax(dim=1) # we can take the softmax to get the label probabilities ```Nr8r0rrVrYT)rrykeepdim)r+r,r-r.r/r0r1)rPrr rr7r(rrnormrCexprrr&r)r*)r;rrrrWrJrr rrrGrDr/r.rCr-r,r+outputr"r"r#rsV'   zCLIPSegModel.forwardr%r9) NNNNNNNTN)rBrCrDrrr]rrrrrrFrFrIrrr?r*rrr"r"ror#r*s  . 2   rc sXeZdZdZdeffdd Z d dejdejdejd ee d e ej f d d Z Z S)CLIPSegDecoderLayerz CLIPSeg decoder layer, which is identical to `CLIPSegEncoderLayer`, except that normalization is applied after self-attention/MLP, rather than before. rPcrrrrnror"r#r]rzCLIPSegDecoderLayer.__init__FrIrrrrcCsd|}|j||||d\}}||}||}|}||}||}||}|f}|r0||f7}|Sr)rrrrrr"r"r#rs"     zCLIPSegDecoderLayer.forwardr)rBrCrDrErr]rrrrr?rFrrr"r"ror#rOs rOc s\eZdZdeffdd Z   d deejdejdee d ee d ee f d d Z Z S)CLIPSegDecoderrPc sXtj|_tjj|_tjj|_j r`j j dj j df}t tj jjdddttjjjd|d|ddttjjdd|d|dd|_ntjjdj j j j d|_tj}tfd d t|D|_tj j_j_j_d _tfd d ttjD|_dS) Nrr)rSpaddingrVr)rSrT)rTcsg|] }tjjjqSr")rrr?r^ reduce_dimrrr"r#rjsz+CLIPSegDecoder.__init__..relucrr")rOr)decoder_configr"r#rrr)r\r]conditional_layerrrr@rSfilm_mulfilm_add"use_complex_transposed_convolutionr?ra SequentialreReLUConvTranspose2dtransposed_convolutionr!extract_layersr r reducescopydeepcopyr^decoder_num_attention_headsrdecoder_intermediate_sizerrr )r;rPtransposed_kernelsdepthro)rPrUr#r]JsB   (zCLIPSegDecoder.__init__NTrIrLrr rcCsp|rdnd}|r dnd}|ddd}d} tt||j|jD]O\} \} } } | dur1| | | } n| | } | |jkrR||| ddd||} | ddd} | | dd|d}|d} |re|| f7}|rn||df7}q| ddddddfddd} tt | j d}|j d}| || j d||} | | d}|stdd|||fDSt|||d S) Nr"rYrrrV)rrrcsrrr"rr"r"r#r<rz)CLIPSegDecoder.forward..)rrIrJ)rzipr r_rVrWrrXrmathsqrtrzrr]squeezer?rH)r;rIrLrr rall_hidden_statesr activationsrNi activationlayerreducerrurrr"r"r#rtsD  "  $ zCLIPSegDecoder.forward)NNT) rBrCrDrr]r?rrrrrrr"r"ror#rPIs .rPzn CLIPSeg model with a Transformer-based decoder on top for zero-shot and one-shot image segmentation. ) custom_introcseZdZeZdeffdd Z     ddeedeej deej deej d eej f d d Z e          ddeej d eej d eej deej deej deej deej deedeededeedeeeffddZZS)CLIPSegForImageSegmentationrPcs:t|||_t||_|j|_t||_|dSr) r\r]rPrrr^rPdecoderr)rnror"r#r]s    z$CLIPSegForImageSegmentation.__init__NrrrrWconditional_pixel_valuescCs|dur.t||krtdt|jj|||d}Wd|S1s'wY|S|durYt||kr 1` a classification loss is computed (Cross-Entropy). Examples: ```python >>> from transformers import AutoProcessor, CLIPSegForImageSegmentation >>> from PIL import Image >>> import requests >>> processor = AutoProcessor.from_pretrained("CIDAS/clipseg-rd64-refined") >>> model = CLIPSegForImageSegmentation.from_pretrained("CIDAS/clipseg-rd64-refined") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> texts = ["a cat", "a remote", "a blanket"] >>> inputs = processor(text=texts, images=[image] * len(texts), padding=True, return_tensors="pt") >>> outputs = model(**inputs) >>> logits = outputs.logits >>> print(logits.shape) torch.Size([3, 352, 352]) ```NTr8rrVcsg|]}|dqS)rr")r8rlrIr"r#rsz7CLIPSegForImageSegmentation.forward..r rr)rrrrWrszWMake sure to pass as many conditional embeddings as there are query images in the batchzcMake sure that the feature dimension of the conditional embeddings matches `config.projection_dim`.)rr r)r+rrLrMr1rN)rPrrrtrr7rrIr^rrr!rJrurzrr@rrrrrrBCEWithLogitsLossrK)r;rrrsrLrrWrvrr rrrGrMrkdecoder_outputsrr+loss_fnrNr"rwr#rs~/    z#CLIPSegForImageSegmentation.forwardr) NNNNNNNNNTN)rBrCrDrrr]rrrrrurrFrrrr?r*rrr"r"ror#rqsp      rq)rrr&r6rq)r)?rEr`rg dataclassesrtypingrrrrrtorch.utils.checkpointrrkr modeling_attn_mask_utilsr r modeling_layersr modeling_outputsr rmodeling_utilsrrutilsrrrrconfiguration_clipsegrrr get_loggerrBrrr$r)r*rHrKr2rOrfloatrrrrrrrr&r3r6rrOrPrq__all__r"r"r"r#s     # T0 Q20W[443f9d .