o i@sdZddlZddlZddlmZddlmZmZmZm Z ddl Z ddl m Z ddl m Z ddlmZmZdd lmZdd lmZmZdd lmZmZdd lmZmZmZmZmZmZd dl m!Z!m"Z"m#Z#e$e%Z&de j'de j'fddZ(de j'de j'fddZ)eeGdddeZ*eeGdddeZ+eeGdddeZ,Gddde j-Z.Gddde j-Z/ dGd!e j-d"e j'd#e j'd$e j'd%ee j'd&e0d'e0fd(d)Z1Gd*d+d+e j-Z2Gd,d-d-e j-Z3Gd.d/d/eZ4eGd0d1d1eZ5Gd2d3d3e j-Z6Gd4d5d5e j-Z7Gd6d7d7e5Z8Gd8d9d9e j-Z9Gd:d;d;e5Z:eGdd?d?e j-ZgdFZ?dS)HzPyTorch CLIPSeg model.N) dataclass)AnyCallableOptionalUnion)nn)ACT2FN) _create_4d_causal_attention_mask_prepare_4d_attention_mask)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPooling)ALL_ATTENTION_FUNCTIONSPreTrainedModel) ModelOutputauto_docstringcan_return_tuplefilter_out_non_signature_kwargslogging torch_int) CLIPSegConfigCLIPSegTextConfigCLIPSegVisionConfiglogitsreturncCstj|tjt||jdS)Ndevice)r functional cross_entropytorcharangelenr)rr$`/home/ubuntu/.local/lib/python3.10/site-packages/transformers/models/clipseg/modeling_clipseg.pycontrastive_loss'sr& similaritycCs t|}t|}||dS)Ng@)r&t)r' caption_loss image_lossr$r$r% clipseg_loss,s  r+c@seZdZUdZdZeejed<dZ eejed<dZ eejed<dZ eejed<dZ eejed<dZ eed<dZeed <d eefd d ZdS) CLIPSegOutputa loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `return_loss` is `True`): Contrastive loss for image-text similarity. logits_per_image (`torch.FloatTensor` of shape `(image_batch_size, text_batch_size)`): The scaled dot product scores between `image_embeds` and `text_embeds`. This represents the image-text similarity scores. logits_per_text (`torch.FloatTensor` of shape `(text_batch_size, image_batch_size)`): The scaled dot product scores between `text_embeds` and `image_embeds`. This represents the text-image similarity scores. text_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim`): The text embeddings obtained by applying the projection layer to the pooled output of [`CLIPSegTextModel`]. image_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim`): The image embeddings obtained by applying the projection layer to the pooled output of [`CLIPSegVisionModel`]. text_model_output (`BaseModelOutputWithPooling`): The output of the [`CLIPSegTextModel`]. vision_model_output (`BaseModelOutputWithPooling`): The output of the [`CLIPSegVisionModel`]. Nlosslogits_per_imagelogits_per_text text_embeds image_embedstext_model_outputvision_model_outputrctfddDS)Nc3.|]}|dvr |nt|VqdS))r2r3Ngetattrto_tuple.0kselfr$r% R  z)CLIPSegOutput.to_tuple..tuplekeysr<r$r<r%r8Q zCLIPSegOutput.to_tuple)__name__ __module__ __qualname____doc__r-rr! FloatTensor__annotations__r.r/r0r1r2rr3rArr8r$r$r$r%r,2s   r,c@sPeZdZUdZdZeejed<dZ ee ejed<dZ ee ejed<dS)CLIPSegDecoderOutputz| logits (`torch.FloatTensor` of shape `(batch_size, height, width)`): Classification scores for each pixel. Nr hidden_states attentions) rDrErFrGrrr!rHrIrKrArLr$r$r$r%rJXs rJc@seZdZUdZdZeejed<dZ eejed<dZ eejed<dZ eejed<dZ e ed<dZeed<d eefd d ZdS) CLIPSegImageSegmentationOutputa loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Binary cross entropy loss for segmentation. logits (`torch.FloatTensor` of shape `(batch_size, height, width)`): Classification scores for each pixel. conditional_embeddings (`torch.FloatTensor` of shape `(batch_size, projection_dim)`): Conditional embeddings used for segmentation. pooled_output (`torch.FloatTensor` of shape `(batch_size, embed_dim)`): Pooled output of the [`CLIPSegVisionModel`]. vision_model_output (`BaseModelOutputWithPooling`): The output of the [`CLIPSegVisionModel`]. decoder_output (`CLIPSegDecoderOutput`): The output of the [`CLIPSegDecoder`]. Nr-rconditional_embeddings pooled_outputr3decoder_outputrcr4)Nc3r5))r3rPNr6r9r<r$r%r>r?z:CLIPSegImageSegmentationOutput.to_tuple..r@r<r$r<r%r8~rCz'CLIPSegImageSegmentationOutput.to_tuple)rDrErFrGr-rr!rHrIrrNrOr3rrPrJrArr8r$r$r$r%rMes   rMcsXeZdZdeffdd Zdejdededejfdd Zdd ej dejfd d Z Z S)CLIPSegVisionEmbeddingsconfigcst||_|j|_|j|_|j|_tt |j|_ tj |j |j|j|jdd|_|j|jd|_|jd|_t|j|j|_|jdt |jddddS)NF) in_channels out_channels kernel_sizestridebiasr position_idsr persistent)super__init__rR hidden_size embed_dim image_size patch_sizer Parameterr!randnclass_embeddingConv2d num_channelspatch_embedding num_patches num_positions Embeddingposition_embeddingregister_bufferr"expandr=rR __class__r$r%r_s"  "z CLIPSegVisionEmbeddings.__init__ embeddingsheightwidthrc Cs|jdd}|jjd}|jdd}tjs(||kr(||kr(||jS|ddddf}|ddddf}|jd} ||j} ||j} t |d} | d| | | }| dddd}t j j|| | fdd d }| dddddd| }tj||fdd S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rrNr[g?rrXbicubicF)sizemode align_cornersdim)shapermweight unsqueezer!jit is_tracingrYrcrreshapepermuterr interpolateviewcat) r=rsrtrurjrmrkclass_pos_embedpatch_pos_embedr{ new_height new_widthsqrt_num_positionsr$r$r%interpolate_pos_encodings*      z0CLIPSegVisionEmbeddings.interpolate_pos_encodingT pixel_valuesc Cs|j\}}}}|s&||jks||jkr&td|d|d|jd|jd ||}|ddd}|j|dd}tj ||gdd} |rR| | | ||} | S| | |j } | S) NzInput image size (*z) doesn't match model ().rXrr[rz) r|rb ValueErrorriflatten transposerfror!rrrmrY) r=rr batch_size_rtru patch_embeds class_embedsrsr$r$r%forwards  zCLIPSegVisionEmbeddings.forwardT) rDrErFrr_r!TensorintrrHr __classcell__r$r$rqr%rQs )rQc sXeZdZdeffdd Z   d deejdeejdeejdej fd d Z Z S) CLIPSegTextEmbeddingsrRcsRt|j}t|j||_t|j||_|j dt |j ddddS)NrYrZFr\) r^r_r`rrl vocab_sizetoken_embeddingmax_position_embeddingsrmrnr!r"ror=rRrarqr$r%r_s  zCLIPSegTextEmbeddings.__init__N input_idsrY inputs_embedsrcCs|dur |jdn|jd}|jjjd}||kr#td|d||dur2|jddd|f}|dur;||}||}||}|S)Nr[rzRSequence length must be less than max_position_embeddings (got `sequence length`: z and max_position_embeddings: )r|rmr}rrYr)r=rrYr seq_lengthmax_position_embeddingposition_embeddingsrsr$r$r%rs"  zCLIPSegTextEmbeddings.forward)NNN) rDrErFrr_rr! LongTensorrHrrrr$r$rqr%rsrmodulequerykeyvalueattention_maskscalingdropoutc Ks|t||dd|}|dur||}tjj|dtjd|j}tjj |||j d}t||} | dd } | |fS)Nr[r)r{dtype)ptrainingrrX) r!matmulrrrsoftmaxfloat32torrr contiguous) rrrrrrrkwargs attn_weights attn_outputr$r$r%eager_attention_forwards  rcsveZdZdZdeeefffdd Z   ddej de ej d e ej d e e d e ej e ej ff d d Z ZS)CLIPSegAttentionz=Multi-headed attention from 'Attention Is All You Need' paperrRcst||_|j|_|j|_|j|j|_|j|j|jkr-td|jd|jd|jd|_ |j |_ d|_ t |j|j|_t |j|j|_t |j|j|_t |j|j|_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: rF)r^r_rRr`ranum_attention_heads num_headshead_dimrscaleattention_dropoutr is_causalrLineark_projv_projq_projout_projrprqr$r%r_s$   zCLIPSegAttention.__init__NFrKrcausal_attention_maskoutput_attentionsrc Cs0|j\}}}||}||} ||} ||||j|jdd}| |||j|jdd} | |||j|jdd} |jj dkrY|durR|durR||}n |durX|}n|du|_ t } |jj dkrlt |jj } | ||| | ||j |j |js{dn|jd\} } | |||} || } |sd} | | fS)z#Input shape: Batch x Time x ChannelrrXflash_attention_2Neagerr)rrr)r|rrrrrrrrR_attn_implementationrrrrrrrrr)r=rKrrrrrraqueriesrBvaluesattention_interfacerrr$r$r%r0s@          zCLIPSegAttention.forward)NNF)rDrErFrGrrrr_r!rrboolrArrr$r$rqr%rs"rcs2eZdZfddZdejdejfddZZS) CLIPSegMLPcsDt||_t|j|_t|j|j |_ t|j |j|_ dSN) r^r_rRr hidden_act activation_fnrrr`intermediate_sizefc1fc2rprqr$r%r_es  zCLIPSegMLP.__init__rKrcCs"||}||}||}|Sr)rrr)r=rKr$r$r%rls   zCLIPSegMLP.forward)rDrErFr_r!rrrr$r$rqr%rds rc sTeZdZdeffdd Z d dejdejdejdeed e ej f d d Z Z S) CLIPSegEncoderLayerrRcRt|j|_t||_tj|j|jd|_ t ||_ tj|j|jd|_ dSN)eps r^r_r`rar self_attnr LayerNormlayer_norm_eps layer_norm1rmlp layer_norm2rprqr$r%r_u   zCLIPSegEncoderLayer.__init__FrKrrrrcCsd|}||}|j||||d\}}||}|}||}||}||}|f}|r0||f7}|SaI Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. `(config.encoder_attention_heads,)`. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. )rKrrr)rrrrr=rKrrrresidualroutputsr$r$r%r}s"     zCLIPSegEncoderLayer.forwardF) rDrErFrr_r!rrrrArHrrr$r$rqr%rts rc@s&eZdZUeed<dZdZddZdS)CLIPSegPreTrainedModelrRclipTcCsV|jj}t|tr"|jjjjd|dd|jjjjd|ddnt|t rW|jj}t j j|j d|j d|dt j j|jj|jj|dt j j|jj|jj|dnt|tr|jj}|j dd|jjd|}|j d|}t j j|jj|dt j j|jj|dt j j|jj|dt j j|jj|dn_t|tr|jj}|jjdd|jjd|}d|jjd|}t j j|jj|dt j j|jj|dn't|trt j j|jj|jd|jjdt j j|jj|jd|jjdt|t jr|j j!|jj"dt|t j#r'|j dur)|j j!dSdSdS) zInitialize the weightsrg{Gz?)meanstdr)rrXg?N)$rRinitializer_factor isinstancerrr}datanormal_rmrQrinitrfrariinitializer_rangernum_hidden_layersrrrrrr`rr CLIPSegModeltext_projectiontext_embed_dimvisual_projectionvision_embed_dimrrWzero_fill_r)r=rfactor in_proj_std out_proj_stdfc_stdr$r$r% _init_weightssL       z$CLIPSegPreTrainedModel._init_weightsN)rDrErFrrIbase_model_prefixsupports_gradient_checkpointingrr$r$r$r%rs  rcsxeZdZdZdeffdd Ze     ddeej deej dee d ee d ee d e e e ff d d ZZS)CLIPSegEncoderz Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`CLIPSegEncoderLayer`]. Args: config: CLIPSegConfig rRcs:t|_tfddtjD|_d|_dS)Ncg|]}tqSr$)rr:rrRr$r% z+CLIPSegEncoder.__init__..F) r^r_rRr ModuleListrangerlayersgradient_checkpointingrprqrr%r_s   zCLIPSegEncoder.__init__Nrrroutput_hidden_states return_dictrc Cs|dur|n|jj}|dur|n|jj}|dur|n|jj}|r"dnd}|r(dnd}|} t|jD] \} } |r<|| f}| | |||d} | d} |rQ|| df}q1|rY|| f}t| ||dS)a Args: inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) causal_attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Causal mask for the text model. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. Nr$)rrr)last_hidden_staterKrL)rRrr use_return_dict enumerater r ) r=rrrrr r encoder_statesall_attentionsrKidx encoder_layer layer_outputsr$r$r%rs2'    zCLIPSegEncoder.forwardNNNNN)rDrErFrGrr_rrr!rrrrAr rrr$r$rqr%rs, rcseZdZdeffdd Ze      ddeejdeejdeejdee d ee d ee d e e e ffd d Z ZS)CLIPSegTextTransformerrRcsHt||_|j}t||_t||_tj ||j d|_ |j |_ dSr) r^r_rRr`rrsrencoderrrrfinal_layer_norm eos_token_idrrqr$r%r_.s    zCLIPSegTextTransformer.__init__NrrrYrr r rc Csj|dur|n|jj}|dur|n|jj}|dur|n|jj}|dur&td|}|d|d}|j||d}t||j |j d} |durLt ||j }|j ||| |||d} | d} | | } |jdkr| tj| jd| j d|jtj| j djdd f} n| tj| jd| j d|jtj| j d|jkjdd f} |s| | f| d dSt| | | j| jd S) NzYou have to specify input_idsr[)rrYr)rrrrr r rrX)rrrzrr pooler_outputrKrL)rRrr rrrwrrsr rrr rrrr!r"r|rrargmaxrrKrL) r=rrrYrr r  input_shaperKrencoder_outputsrrOr$r$r%r9s\      zCLIPSegTextTransformer.forwardNNNNNN)rDrErFrr_rrr!rrrrArrrr$r$rqr%r-s0  rcseZdZUeed<ddgZdeffdd ZdejfddZ d d Z e dd e e jd e e jde e jde ede ede edeeeffddZZS)CLIPSegTextModelrRrrc"t|t||_|dSr)r^r_r text_model post_initrprqr$r%r_   zCLIPSegTextModel.__init__rcC |jjjSrr#rsrr<r$r$r%get_input_embeddings z%CLIPSegTextModel.get_input_embeddingscCs||jj_dSrr')r=rr$r$r%set_input_embeddingssz%CLIPSegTextModel.set_input_embeddingsNrrrYrr r cCs|j||||||dS)a; Examples: ```python >>> from transformers import AutoTokenizer, CLIPSegTextModel >>> tokenizer = AutoTokenizer.from_pretrained("CIDAS/clipseg-rd64-refined") >>> model = CLIPSegTextModel.from_pretrained("CIDAS/clipseg-rd64-refined") >>> inputs = tokenizer(["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state >>> pooled_output = outputs.pooler_output # pooled (EOS token) states ```rrrYrr r )r#)r=rrrYrr r r$r$r%rszCLIPSegTextModel.forwardr )rDrErFrrI_no_split_modulesr_rModuler(r*rrr!rrrrArrrr$r$rqr%r!s8  r!cspeZdZdeffdd Ze    ddeejdee dee d ee d ee d e e e ff d d Z ZS)CLIPSegVisionTransformerrRcsRt||_|j}t||_tj||jd|_ t ||_ tj||jd|_ dSr) r^r_rRr`rQrsrrr pre_layrnormrrpost_layernormrrqr$r%r_s   z!CLIPSegVisionTransformer.__init__NTrrr r rrc Cs|dur|n|jj}|dur|n|jj}|dur|n|jj}|j||d}||}|j||||d}|d}|dddddf} || } |sS|| f|ddSt|| |j |j dS)N)r)rrr r rrr) rRrr rrsr/rr0rrKrL) r=rrr r rrKrrrOr$r$r%rs.   z CLIPSegVisionTransformer.forward)NNNT)rDrErFrr_rrr!rHrrrArrrr$r$rqr%r.s(  r.cseZdZUeed<dZdeffdd ZdejfddZ e    dde e j d e ed e ed e ed e edeeeff ddZZS)CLIPSegVisionModelrRrcr"r)r^r_r. vision_modelr$rprqr$r%r_r%zCLIPSegVisionModel.__init__rcCr&r)r2rsrir<r$r$r%r(r)z'CLIPSegVisionModel.get_input_embeddingsNTrr rr cCs|j|||||dS)a Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, CLIPSegVisionModel >>> processor = AutoProcessor.from_pretrained("CIDAS/clipseg-rd64-refined") >>> model = CLIPSegVisionModel.from_pretrained("CIDAS/clipseg-rd64-refined") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state >>> pooled_output = outputs.pooler_output # pooled CLS states ```rrr rr )r2)r=rrr rr r$r$r%rszCLIPSegVisionModel.forward)NNNTN)rDrErFrrImain_input_namer_rr-r(rrr!rHrrrArrrr$r$rqr%r1s0  r1cseZdZUeed<deffdd Zee  ddej de ej de ej dej fd d Z ee dd ej d e dej fddZe        dde ejd e ej de ej de ejde e de e de e d e de e deeeffddZZS)rrRcst|t|jtstdt|jdt|jts(tdt|jd|j}|j}|j |_ |j |_ |j |_ |j |_ |j |_ t||_t||_tj|j |j dd|_tj|j |j dd|_tt|jj|_|dS)NzNconfig.text_config is expected to be of type CLIPSegTextConfig but is of type .zRconfig.vision_config is expected to be of type CLIPSegVisionConfig but is of type F)rW)r^r_r text_configr TypeErrortype vision_configrrprojection_dimr`rrrr#r.r2rrrrrdr!tensorrRlogit_scale_init_value logit_scaler$)r=rRr6r9rqr$r%r_'s4      zCLIPSegModel.__init__NrrrYrcCs$|j|||d}|j}||}|S)a Returns: text_features (`torch.FloatTensor` of shape `(batch_size, output_dim`): The text embeddings obtained by applying the projection layer to the pooled output of [`CLIPSegTextModel`]. Examples: ```python >>> import torch >>> from transformers import AutoTokenizer, CLIPSegModel >>> tokenizer = AutoTokenizer.from_pretrained("CIDAS/clipseg-rd64-refined") >>> model = CLIPSegModel.from_pretrained("CIDAS/clipseg-rd64-refined") >>> inputs = tokenizer(["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt") >>> with torch.inference_mode(): ... text_features = model.get_text_features(**inputs) ```)rrrY)r#rr)r=rrrY text_outputsrO text_featuresr$r$r%get_text_featuresKs zCLIPSegModel.get_text_featuresTrrcCs"|j||d}|j}||}|S)an Returns: image_features (`torch.FloatTensor` of shape `(batch_size, output_dim`): The image embeddings obtained by applying the projection layer to the pooled output of [`CLIPSegVisionModel`]. Examples: ```python >>> import torch >>> from transformers import AutoProcessor, CLIPSegModel >>> from transformers.image_utils import load_image >>> processor = AutoProcessor.from_pretrained("CIDAS/clipseg-rd64-refined") >>> model = CLIPSegModel.from_pretrained("CIDAS/clipseg-rd64-refined") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = load_image(url) >>> inputs = processor(images=image, return_tensors="pt") >>> with torch.inference_mode(): ... image_features = model.get_image_features(**inputs) ```)rr)r2rr)r=rrvision_outputsrOimage_featuresr$r$r%get_image_featuresos zCLIPSegModel.get_image_features return_lossrr r c  Cs(|dur|n|jj}|dur|n|jj}| dur| n|jj} |j||||| d} |j|||||| d} | d} || } | d} || } | | jdddd} | | jdddd} |j }t | | |}| }d}|rtt|}| s||| | | | f}|dur|f|S|St|||| | | | d S) a return_loss (`bool`, *optional*): Whether or not to return the contrastive loss. Examples: ```python >>> import torch >>> from transformers import AutoProcessor, CLIPSegModel >>> from transformers.image_utils import load_image >>> processor = AutoProcessor.from_pretrained("CIDAS/clipseg-rd64-refined") >>> model = CLIPSegModel.from_pretrained("CIDAS/clipseg-rd64-refined") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = load_image(url) >>> inputs = processor( ... text=["a photo of a cat", "a photo of a dog"], images=image, return_tensors="pt", padding=True ... ) >>> with torch.inference_mode(): ... outputs = model(**inputs) >>> logits_per_image = outputs.logits_per_image # this is the image-text similarity score >>> probs = logits_per_image.softmax(dim=1) # we can take the softmax to get the label probabilities ```Nr3r+rrXr[T)rr{keepdim)r-r.r/r0r1r2r3)rRrr rr2r#rrnormr=expr!rr(r+r,)r=rrrrYrDrr rr rAr>r1r0r=r/r.r-outputr$r$r%rsV(   zCLIPSegModel.forward)NNr) NNNNNNNTN)rDrErFrrIr_rrr!rrrHr@rrCrrrAr,rrr$r$rqr%r#sr $"%   rc sXeZdZdZdeffdd Z d dejdejdejd ee d e ej f d d Z Z S)CLIPSegDecoderLayerz CLIPSeg decoder layer, which is identical to `CLIPSegEncoderLayer`, except that normalization is applied after self-attention/MLP, rather than before. rRcrrrrprqr$r%r_rzCLIPSegDecoderLayer.__init__FrKrrrrcCsd|}|j||||d\}}||}||}|}||}||}||}|f}|r0||f7}|Sr)rrrrrr$r$r%rs"     zCLIPSegDecoderLayer.forwardr)rDrErFrGrr_r!rrrrArHrrr$r$rqr%rIs rIc s\eZdZdeffdd Z   d deejdejdee d ee d ee f d d Z Z S)CLIPSegDecoderrRc sXtj|_tjj|_tjj|_j r`j j dj j df}t tj jjdddttjjjd|d|ddttjjdd|d|dd|_ntjjdj j j j d|_tj}tfd d t|D|_tj j_j_j_d _tfd d ttjD|_dS) Nrr)rUpaddingrXr)rUrV)rVcsg|] }tjjjqSr$)rrr9r` reduce_dimrrr$r%rQsz+CLIPSegDecoder.__init__..relucrr$)rIr)decoder_configr$r%rYr)r^r_conditional_layerrrr:rMfilm_mulfilm_add"use_complex_transposed_convolutionr9rc SequentialrgReLUConvTranspose2dtransposed_convolutionr#extract_layersrr reducescopydeepcopyr`decoder_num_attention_headsrdecoder_intermediate_sizerrr )r=rRtransposed_kernelsdepthrq)rRrOr%r_1sB   (zCLIPSegDecoder.__init__NTrKrNrr r cCsp|rdnd}|r dnd}|ddd}d} tt||j|jD]O\} \} } } | dur1| | | } n| | } | |jkrR||| ddd||} | ddd} | | dd|d}|d} |re|| f7}|rn||df7}q| ddddddfddd} tt | j d}|j d}| || j d||} | | d}|stdd|||fDSt|||d S) Nr$r[rrrX)rrrcss|] }|dur|VqdSrr$)r:vr$r$r%r>sz)CLIPSegDecoder.forward..)rrKrL)rzipr rYrPrQrrRrmathsqrtr|rrWsqueezerArJ)r=rKrNrr r all_hidden_statesr activationsrHi activationlayerreducerrwrrr$r$r%r[sD  "  $ zCLIPSegDecoder.forward)NNT) rDrErFrr_rAr!rrrrrr$r$rqr%rJ0s .rJzn CLIPSeg model with a Transformer-based decoder on top for zero-shot and one-shot image segmentation. ) custom_introcseZdZUeed<deffdd Z     ddeedeej deej deej d eej f d d Z e          ddeej d eej d eej deej deej deej deej deedeededeedeeeffddZZS)CLIPSegForImageSegmentationrRcs:t|||_t||_|j|_t||_|dSr) r^r_rRrrrXrJdecoderr$rprqr$r%r_s    z$CLIPSegForImageSegmentation.__init__NrrrrYconditional_pixel_valuescCs|dur.t||krtdt|jj|||d}Wd|S1s'wY|S|durYt||kr 1` a classification loss is computed (Cross-Entropy). Examples: ```python >>> import torch >>> from transformers import AutoProcessor, CLIPSegForImageSegmentation >>> from transformers.image_utils import load_image >>> processor = AutoProcessor.from_pretrained("CIDAS/clipseg-rd64-refined") >>> model = CLIPSegForImageSegmentation.from_pretrained("CIDAS/clipseg-rd64-refined") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = load_image(url) >>> texts = ["a cat", "a remote", "a blanket"] >>> inputs = processor(text=texts, images=[image] * len(texts), padding=True, return_tensors="pt") >>> with torch.inference_mode(): ... outputs = model(**inputs) >>> logits = outputs.logits >>> print(logits.shape) torch.Size([3, 352, 352]) ```NTr3rrXcsg|]}|dqS)rr$)r:rgrKr$r%rsz7CLIPSegForImageSegmentation.forward..rrr)rrrrYrnzWMake sure to pass as many conditional embeddings as there are query images in the batchzcMake sure that the feature dimension of the conditional embeddings matches `config.projection_dim`.)rr r )r-rrNrOr3rP)rRrr!rorr2rrKrXrrrrLrpr|rr:rmrrrrBCEWithLogitsLossrM)r=rrrnrNrrYrqrr rr rArOrfdecoder_outputsrr-loss_fnrHr$rrr%rs~1    z#CLIPSegForImageSegmentation.forwardr) NNNNNNNNNTN)rDrErFrrIr_rrr!rrprrHrrrrAr,rrr$r$rqr%rlsp       rl)rrr!r1rl)r)@rGrZrb dataclassesrtypingrrrrr!rrfr modeling_attn_mask_utilsr r modeling_layersr modeling_outputsr rmodeling_utilsrrutilsrrrrrrconfiguration_clipsegrrr get_loggerrDloggerrr&r+r,rJrMr-rQrfloatrrrrrrrr!r.r1rrIrJrl__all__r$r$r$r%s      # T0 K20V[443T9d 0