o oi@@sddlmZddlmZmZddlZddlmZddlm Z ddl m Z m Z ddl mZmZddlmZmZmZdd lmZmZmZdd lmZdd lmZdd lmZGd ddZdS)) annotations)AnyOptionalN)LongestMaxSize)AugmentationSequential)PromptsSegmentationResults)Sam SamConfig)Tensorpadtensor) KORNIA_CHECKKORNIA_CHECK_IS_TENSORKORNIA_CHECK_SHAPE) normalize)Boxes) KeypointscseZdZdZ   dJdKfd d ZdLdMddZedLdNddZdOddZ dPd d!Z dQd#d$Z dd%dRd+d,Z    dSdTd1d2Z e     3 3dUdVd8d9ZdWd:d;ZddXdHdIZZS)YVisualPrompteraWAllow the user to run multiple query with multiple prompts for a model. At the moment, we just support the SAM model. The model is loaded based on the given config. For default the images are transformed to have their long side with size of the `image_encoder.img_size`. This Prompter class ensure to transform the images and the prompts before prediction. Also, the image is passed automatically for the method `preprocess_image`, which is responsible for normalize the image and pad it to have the right size for the SAM model :math:`(\text{image_encoder.img_size}, \text{image_encoder.img_size})`. For default the image is normalized by the mean and standard deviation of the SAM dataset values. Args: config: A model config to generate the model. Now just the SAM model is supported. device: The desired device to use the model. dtype: The desired dtype to use the model. Example: >>> # prompter = VisualPrompter() # Will load the vit h for default >>> # You can load a custom SAM type for modifying the config >>> prompter = VisualPrompter(SamConfig('vit_b')) >>> image = torch.rand(3, 25, 30) >>> prompter.set_image(image) >>> boxes = Boxes( ... torch.tensor( ... [[[[0, 0], [0, 10], [10, 0], [10, 10]]]], ... device=prompter.device, ... dtype=torch.float32 ... ), ... mode='xyxy' ... ) >>> prediction = prompter.predict(boxes=boxes) >>> prediction.logits.shape torch.Size([1, 3, 256, 256]) NconfigOptional[SamConfig]deviceOptional[torch.device]dtypeOptional[torch.dtype]returnNonecst|durtddd}t|tr=t||_t|jjj ddf}t gd||dd|_ t gd ||dd|_ nt |jj||d|_t|d di|_||_||_d|_d|_d|_|dS) Nvit_hT) model_type pretrainedg?)p)g33333^@gR]@gRY@)rr)g(\2M@g(\L@gL@ same_on_batch)super__init__r isinstancer from_configmodelr image_encoderimg_sizer pixel_mean pixel_stdNotImplementedErrortor transformsrr_original_image_size_input_image_size_input_encoder_size reset_image)selfrrrr. __class__R/home/ubuntu/.local/lib/python3.10/site-packages/kornia/contrib/visual_prompter.pyr$Gs"     zVisualPrompter.__init__xr meanOptional[Tensor]stdcCst|trt|trt|||}nt|jtr%t|jtr%t||j|j}|jjj}||jd}||jd}t |d|d|f}|S)aaNormalize and pad a tensor. For normalize the tensor: will prioritize the `mean` and `std` passed as argument, if None will use the default Sam Dataset values. For pad the tensor: Will pad the tensor into the right and bottom to match with the size of `self.model.image_encoder.img_size` Args: x: The image to be preprocessed mean: Mean for each channel. std: Standard deviations for each channel. Returns: The image preprocessed (normalized if has mean and str available and padded to encoder size) r) r%r rr*r+r'r(r)shaper )r3r8r9r;encoder_im_sizepad_hpad_wr6r6r7preprocess_imagecs zVisualPrompter.preprocess_imageimagecCst|gd||jd|jdf|_|j|dgd}|jj|_|jd|jdf|_||||}|jd|jdf|_ |j ||_ d|_ dS)aESet the embeddings from the given image with `image_decoder` of the model. Prepare the given image with the selected transforms and the preprocess method. Args: image: RGB image. Normally images with range of [0-1], the model preprocess normalize the pixel values with the mean and std defined in its initialization. Expected to be into a float32 dtype. Shape :math:`(3, H, W)`. mean: mean value of dataset for normalization. std: standard deviation of dataset for normalization. )3HWr<r=input data_keysTN)rr2r>r/r._params _tfs_paramsr0rBr1r'r(image_embeddings is_image_set)r3rCr9r;r6r6r7 set_images  zVisualPrompter.set_image keypointsKeypoints | TensorlabelsrcCsRt|jgdt|jddgt|jd|jdkdt|tr't|}|S)z:Validate the keypoints shape and ensure to be a Keypoints.)KN2rRrSrz8The keypoints and labels should have the same batch size)rdatarr>r%r r from_tensor)r3rOrQr6r6r7_valid_keypointss   zVisualPrompter._valid_keypointsboxesBoxes | TensorrcCsPt|trt|jddgt|dd}|jdkr|}|St|jdddd}|S)zAValidate the boxes shape and ensure to be a Boxes into xyxy mode.rR4xyxymode)r%r rrUrr] to_tensor)r3rX boxes_xyxyr6r6r7 _valid_boxess   zVisualPrompter._valid_boxesmaskscCst|gd|S)zValidate the input masks shape.)rR1256rc)r)r3rar6r6r7 _valid_masksszVisualPrompter._valid_masksrHpromptsTensor | Boxes | KeypointsrIOptional[list[str]]%dict[str, Tensor | Boxes | Keypoints]csJ|j|||jd|durg}tttfsgfddt|DS)N)rIparamscsi|] \}}||qSr6r6).0idxkeytransformed_promptsr6r7 sz5VisualPrompter._transform_prompts..)r.rKr%listtuple enumerate)r3rIrer6rmr7_transform_promptss z!VisualPrompter._transform_promptsOptional[Keypoints | Tensor]keypoints_labelsOptional[Boxes | Tensor]rc Csg}g}t|ttfr t|tr |||}|d||t|ttfr6|||d||t|tr@|||j|d|i}d|vrft|dtrf|d }t |ret |re||f} nd} d|vrt|dtr|dj dd} t | r| } nd} t | | |dS)zMValidate and preprocess the given prompts to be aligned with the input image.rO bbox_xyxyrINr[r\pointsrXra) r%rr rWappendrr`rdrsr^rr) r3rOrurXrarI to_transformrU kpts_tensorry_bboxbboxr6r6r7preprocess_promptss4         z!VisualPrompter.preprocess_promptsTmultimask_outputbooloutput_original_sizerc Cst|jd|||||}|jj|j|j|jd\}} ~|jj|j |jj || |d\} } t | | } |rLt |j trLt |jtrL| |j |j|j| S)asPredict masks for the given image based on the input prompts. Args: keypoints: Point prompts to the model. Each point is in (X,Y) in pixels. Shape :math:`(K, N, 2)`. Where `N` is the number of points and `K` the number of prompts. keypoints_labels: Labels for the point prompts. 1 indicates a foreground point and 0 indicates a background point. Shape :math:`(K, N)`. Where `N` is the number of points, and `K` the number of prompts. boxes: A box prompt to the model. If a tensor, should be in a xyxy mode. Shape :math:`(K, 4)` masks: A low resolution mask input to the model, typically coming from a previous prediction iteration. Has shape :math:`(K, 1, H, W)`, where for SAM, H=W=256. multimask_output: If true, the model will return three masks. For ambiguous input prompts (such as a single click), this will often produce better masks than a single prediction. If only a single mask is needed, the model's predicted quality score can be used to select the best mask. For non-ambiguous prompts, such as multiple input prompts, multimask_output=False can give better results. output_original_size: If true, the logits of `SegmentationResults` will be post-process to match the original input image size. Returns: A prediction with the logits and scores (IoU of each predicted mask) zKAn image must be set with `self.set_image(...)` before `predict` be called!rx)rLimage_pesparse_prompt_embeddingsdense_prompt_embeddingsr)rrMrr'prompt_encoderryrXra mask_decoderrL get_dense_perr%r0rqr/original_res_logitsr1) r3rOrurXrarrresparse_embeddingsdense_embeddingslogitsscoresresultsr6r6r7predicts, !      zVisualPrompter.predictcCs6d|_d|_d|_d|_t|dr|`d|_d|_dS)NrLF)rKr/r0r1hasattrrLrM)r3r6r6r7r22s  zVisualPrompter.reset_imageFinductor fullgraphdynamicbackendr]optionsdisablerrrstrr] Optional[str]rOptional[dict[Any, Any]]rc Csdtj|jj||||||d|j_tj|jj||||||d|j_tj|jj||||||d|j_dS)uFApply `torch.compile(...)`/dynamo API into the VisualPrompter API. .. note:: For more information about the dynamo API check the official docs https://pytorch.org/docs/stable/generated/torch.compile.html Args: fullgraph: Whether it is ok to break model into several subgraphs dynamic: Use dynamic shape tracing backend: backend to be used mode: Can be either “default”, “reduce-overhead” or “max-autotune” options: A dictionary of options to pass to the backend. disable: Turn torch.compile() into a no-op for testing Example: >>> # prompter = VisualPrompter() >>> # prompter.compile() # You should have torch >= 2.0.0 installed >>> # Use the prompter methods ... rN)torchcompiler'r(rr)r3rrrr]rrr6r6r7r>s6'   zVisualPrompter.compile)NNN)rrrrrrrr)NN)r8r r9r:r;r:rr )rCr r9r:r;r:rr)rOrPrQr rr)rXrYrr)rar rr )rerfrIrgrrh)NNNN) rOrtrur:rXrvrar:rr)NNNNTT)rOrtrur:rXrvrar:rrrrrr)rr)rrrrrrr]rrrrrrr)__name__ __module__ __qualname____doc__r$rBrno_gradrNrWr`rdrsrrr2r __classcell__r6r6r4r7r#sF%    * >r) __future__rtypingrrrkornia.augmentationr%kornia.augmentation.container.augmentrkornia.contrib.modelsrrkornia.contrib.models.samr r kornia.corer r r kornia.core.checkrrrkornia.enhancerkornia.geometry.boxesrkornia.geometry.keypointsrrr6r6r6r7s