o oiT8@sdZddlmZddlZddlmZddlmZddlm Z m Z ddl Z ddl m Z ddlmZdd lmZdd lmZdd lmZdd lmZdd lmZddlmZddlmZddlmZm Z GdddeZ!eGdddZ"Gdddee"Z#d ddZ$dS)!aPBased from the original code from Meta Platforms, Inc. and affiliates. https://github.com/facebookresearch/segment- anything/blob/3518c86b78b3bc9cf4fbe3d18e682fad1c79dc51/segment_anything/build_sam.py https://github.com/facebookresearch/segment- anything/blob/3518c86b78b3bc9cf4fbe3d18e682fad1c79dc51/segment_anything/modeling/sam.py ) annotationsN) dataclass)Enum)AnyOptional)SegmentationResults) ModelBase) LayerNorm)ImageEncoderViT) MaskDecoder) PromptEncoder)TwoWayTransformer)TinyViT)Tensor) KORNIA_CHECKKORNIA_CHECK_SHAPEc@s eZdZdZdZdZdZdZdS) SamModelTypezMap the SAM model types.rN)__name__ __module__ __qualname____doc__vit_hvit_lvit_b mobile_samrrS/home/ubuntu/.local/lib/python3.10/site-packages/kornia/contrib/models/sam/model.pyr0s rc@sfeZdZUdZdZded<dZded<dZded <dZd ed <dZ d ed <dZ d ed <dZ ded<dS) SamConfigaEncapsulate the Config to build a SAM model. Args: model_type: the available models are: - 0, 'vit_h' or :func:`kornia.contrib.sam.SamModelType.vit_h` - 1, 'vit_l' or :func:`kornia.contrib.sam.SamModelType.vit_l` - 2, 'vit_b' or :func:`kornia.contrib.sam.SamModelType.vit_b` - 3, 'mobile_sam', or :func:`kornia.contrib.sam.SamModelType.mobile_sam` checkpoint: URL or a path for a file with the weights of the model encoder_embed_dim: Patch embedding dimension. encoder_depth: Depth of ViT. encoder_num_heads: Number of attention heads in each ViT block. encoder_global_attn_indexes: Encoder indexes for blocks using global attention. Nz"Optional[str | int | SamModelType] model_typez Optional[str] checkpointFbool pretrainedz Optional[int]encoder_embed_dim encoder_depthencoder_num_headszOptional[tuple[int, ...]]encoder_global_attn_indexes) rrrrr!__annotations__r"r$r%r&r'r(rrrrr 9s       r csZeZdZUdZded<dfd d Zed ddZed!ddZe d"ddZ Z S)#Samgfloatmask_threshold image_encoderImageEncoderViT | TinyViTprompt_encoderr mask_decoderr returnNonecs t||_||_||_dS)aSAM predicts object masks from an image and input prompts. Args: image_encoder: The backbone used to encode the image into image embeddings that allow for efficient mask prediction. prompt_encoder: Encodes various types of input prompts. mask_decoder: Predicts masks from the image embeddings and encoded prompts. N)super__init__r-r/r0)selfr-r/r0 __class__rrr4Zs  z Sam.__init__namestrcCs$|dvr tt|Std|)a'Build/load the SAM model based on it's name. Args: name: The name of the SAM model. Valid names are: - 'vit_b' - 'vit_l' - 'vit_h' - 'mobile_sam' Returns: The respective SAM model )rrrrzInvalid SAM model name: )r* from_configr ValueError)r8rrr from_nameksz Sam.from_nameconfigr c Cs|j}t|tr t|}nt|tr!tjtjtjtjd}||}|tjkr/t ddddd}ny|tjkr=t dddd d}nk|tjkrKt d d dd d}n]|tjkrd }d}d}||}t t j d|ddt |||f||fddtdtd|ddd|dd dd}n)t|jtrt|jtrt|jtrt|jtrt |j|j|j|jd}ntd|j}|jr|durtjdtjdtjdtjdi|}ntjdd d!|r|||S)"aBuild/load the SAM model based on it's config. Args: config: The SamConfig data structure. If the model_type is available, build from it, otherwise will use the parameters set. Returns: The respective SAM model Example: >>> from kornia.contrib.models.sam import SamConfig >>> sam_model = Sam.from_config(SamConfig('vit_b')) )rrrri )r )r%r&r'r()r?rAi )rF5mT)img_sizer embed_dimimage_embedding_sizeinput_image_size mask_in_chansrrr@depth embedding_dimmlp_dim num_headsnum_multimask_outputs transformertransformer_dimiou_head_depthiou_head_hidden_dimr-r/r0zKUnexpected config. The model_type should be provide or the encoder configs.NzDhttps://dl.fbaipublicfiles.com/segment_anything/sam_vit_b_01ec64.pthzDhttps://dl.fbaipublicfiles.com/segment_anything/sam_vit_l_0b3195.pthzDhttps://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pthzmhttps://github.com/ChaoningZhang/MobileSAM/raw/a509aac54fdd7af59f843135f2f7cee307283c88/weights/mobile_sam.ptz2checkpoint is not None. pretrained=True is ignoredr) stacklevel)r! isinstanceintrr9rrrr _build_samr*rr:r r r r%r&r'r(rXNotImplementedErrorr"r$warningswarnload_checkpoint) r=r! _map_sam_typemodelprompt_embed_dim image_sizevit_patch_sizerPr"rrrr:s            zSam.from_configimagesrbatched_promptslist[dict[str, Any]]multimask_outputr#list[SegmentationResults]c Cst|gdt|jdt|kd||}g}t||D]5\}}|j|dd|dd|ddd\}} |j|d |j || |d \} } | t | | |j q|S) a Predicts masks end-to-end from provided images and prompts. This method expects that the images have already been pre-processed, at least been normalized, resized and padded to be compatible with the `self.image_encoder`. .. note:: For each image :math:`(3, H, W)`, it is possible to input a batch (:math:`K`) of :math:`N` prompts, the results are batched by the number of prompts batch. So given a prompt with :math:`K=5`, and :math:`N=10`, the results will look like :math:`5xCxHxW` where :math:`C` is determined by multimask_output. And within each of these masks :math:`(5xC)`, it should be possible to find :math:`N` instances if the model succeed. Args: images: The image as a torch tensor in :math:`(B, 3, H, W)` format, already transformed for input to the model. batched_prompts: A list over the batch of images (list length should be :math:`B`), each a dictionary with the following keys. If it does not have the respective prompt, it should not be included in this dictionary. The options are: - "points": tuple of (Tensor, Tensor) within the coordinate keypoints and their respective labels. the tuple should look like (keypoints, labels), where: - The keypoints (a tensor) are a batched point prompts for this image, with shape :math:`(K, N, 2)`. Already transformed to the input frame of the model. - The labels (a tensor) are a batched labels for point prompts, with shape :math:`(K, N)`. Where 1 indicates a foreground point and 0 indicates a background point. - "boxes": (Tensor) Batched box inputs, with shape :math:`(K, 4)`. Already transformed to the input frame of the model. - "mask_inputs": (Tensor) Batched mask inputs to the model, in the form :math:`(K, 1, H, W)`. multimask_output: Whether the model should predict multiple disambiguating masks, or return a single mask. Returns: A list over input images, where each element is as SegmentationResults the following. - logits: Low resolution logits with shape :math:`(K, C, H, W)`. Can be passed as mask input to subsequent iterations of prediction. Where :math:`K` is the number of input prompts, :math:`C` is determined by multimask_output, and :math:`H=W=256` are the model output size. - scores: The model's predictions of mask quality (iou prediction), in shape BxC. )B3HWrzCThe number of images (`B`) should match with the length of prompts!pointsNboxes mask_inputs)rvrwmasks)N.)image_embeddingsimage_pesparse_prompt_embeddingsdense_prompt_embeddingsrp) rrshapelenr-zipr/getr0 get_dense_peappendrr,) r5rmrnrprzoutputs prompt_recordcurr_embeddingsparse_embeddingsdense_embeddingslow_res_logitsiou_predictionsrrrforwards,,       z Sam.forward)r-r.r/r r0r r1r2)r8r9r1r*)r=r r1r*)rmrrnrorpr#r1rq) rrrr,r)r4 staticmethodr<r:torchno_gradr __classcell__rrr6rr*Ws   lr*r%rbr&r'r(tuple[int, ...]r1cCsld}d}d}||}tt|||dt||dd|d|d t|||f||fddtd td |d d d |d dddS)NrKrBrDT) rUrOrM mlp_ratio norm_layerrX patch_sizeqkv_bias use_rel_posglobal_attn_indexes window_size out_chansrNrrrSr@rTrYr_)r*r r r r r )r%r&r'r(rjrkrlrPrrrrc8sBrc) r%rbr&rbr'rbr(rr1r*)%r __future__rre dataclassesrenumrtypingrrrkornia.contrib.modelsrkornia.contrib.models.baser-kornia.contrib.models.sam.architecture.commonr 4kornia.contrib.models.sam.architecture.image_encoderr 3kornia.contrib.models.sam.architecture.mask_decoderr 5kornia.contrib.models.sam.architecture.prompt_encoderr 2kornia.contrib.models.sam.architecture.transformerr kornia.contrib.models.tiny_vitr kornia.corerkornia.core.checkrrrr r*rcrrrrs.             b