o iY@sdZddlmZddlmZmZddlZddlm Z ddl m Z ddl m Z dd lmZmZmZdd lmZeeZerBddlZed d Gd dde ZdgZdS)z Processor class for SAM2. deepcopy)OptionalUnionN) ImageInput)ProcessorMixin) BatchEncoding) TensorTypeis_torch_availablelogging)requires)torch)backendscseZdZdZdgZdZd1deedeffdd Z       d2d ee d ee d ee e e e e e e jfd ee e e e ee jfdee e e e e e jfdee e e e e jfdee eefdefddZ d3deddddfddZd4ddZd5ddZd6ddZd d!Zd"d#Z d5d$e e jeje fd%ed&ed'ed(eede f d)d*Zd7d+d,Z - . - - d8d/d0ZZS)9 Sam2Processora Constructs a SAM2 processor which wraps a SAM2 image processor and an 2D points & Bounding boxes processor into a single processor. [`Sam2Processor`] offers all the functionalities of [`Sam2ImageProcessorFast`] and [`Sam2VideoProcessor`]. See the docstring of [`~Sam2ImageProcessorFast.__call__`] and [`~Sam2VideoProcessor.__call__`] for more information. Args: image_processor (`Sam2ImageProcessorFast`): An instance of [`Sam2ImageProcessorFast`]. target_size (`int`, *optional*): The target size (target_size, target_size) to which the image will be resized. point_pad_value (`int`, *optional*, defaults to -10): The value used for padding input points. image_processorSam2ImageProcessorFastN target_sizepoint_pad_valuec s>tj|fi|||_|dur||_dS|jjd|_dS)Nheight)super__init__rrsizer)selfrrrkwargs __class__e/home/ubuntu/veenaModal/venv/lib/python3.10/site-packages/transformers/models/sam2/processing_sam2.pyr:s$zSam2Processor.__init__imagessegmentation_maps input_points input_labels input_boxesoriginal_sizesreturn_tensorsreturnc s:|dur|j|f||d|} n|dur*t|tjr!|}td|i|d} ntd| d}|durHt|dkrHt|t|krHtd|dusU|dusU|dur|j |dd d d d } |j |d ddd} |j |d dddd } | dur| | dd } | dur| | dd }| dur| | dd | dur| dur| |krtd| durt| d krt fdd| Drtd| dur| | | d g}tj |tjd}|j||dd| d|i| dur| | |}tj |tjd}| d|i| durtj | tjd}|j||dd| d|i| S)a This method uses [`Sam2ImageProcessorFast.__call__`] method to prepare image(s) for the model. It also prepares 2D points and bounding boxes for the model if they are provided. Args: images (`ImageInput`, *optional*): The image(s) to process. segmentation_maps (`ImageInput`, *optional*): The segmentation maps to process. input_points (`list[list[list[list[float]]]]`, `torch.Tensor`, *optional*): The points to add to the frame. input_labels (`list[list[list[int]]]`, `torch.Tensor`, *optional*): The labels for the points. input_boxes (`list[list[list[float]]]`, `torch.Tensor`, *optional*): The bounding boxes to add to the frame. original_sizes (`list[list[float]]`, `torch.Tensor`, *optional*): The original sizes of the images. return_tensors (`str` or `TensorType`, *optional*): The type of tensors to return. **kwargs: Additional keyword arguments to pass to the image processor. Returns: A [`BatchEncoding`] with the following fields: - `pixel_values` (`torch.Tensor`): The processed image(s). - `original_sizes` (`list[list[float]]`): The original sizes of the images. - `reshaped_input_sizes` (`torch.Tensor`): The reshaped input sizes of the images. - `labels` (`torch.Tensor`): The processed segmentation maps (if provided). - `input_points` (`torch.Tensor`): The processed points. - `input_labels` (`torch.Tensor`): The processed labels. - `input_boxes` (`torch.Tensor`): The processed bounding boxes. N)r!r&r%) tensor_typez0Either images or original_sizes must be providedz{original_sizes must be of length 1 or len(images). If you are passing a single image, you must pass a single original_size.pointsz;[image level, object level, point level, point coordinates])expected_depth input_nameexpected_formatexpected_coord_sizerlabelsz([image level, object level, point level])r-r.r/boxesz)[image level, box level, box coordinates]zbInput points and labels have inconsistent dimensions. Please ensure they have the same dimensions.c3s |] }t|dkVqdSr)N)len).0 img_boxesboxes_max_dimsrr sz)Sam2Processor.__call__..zInput boxes have inconsistent dimensions that would require padding, but boxes cannot be padded due to model limitations. Please ensure all images have the same number of boxes.)dtypeT)preserve_paddingr"r#is_bounding_boxr$)r isinstancerTensorcputolistr ValueErrorr4_validate_single_input_get_nested_dimensionsany_pad_nested_listtensorfloat32_normalize_tensor_coordinatesupdateint64)rr r!r"r#r$r%r&rencoding_image_processorprocessed_pointsprocessed_labelsprocessed_boxespoints_max_dimslabels_max_dims padded_points final_points padded_labels final_labels final_boxesrr7r__call__?s+  $   zSam2Processor.__call__Fcoordsz torch.Tensorc Csl|\}}||}}t|}|r|ddd}|d|||d<|d|||d<|r4|dd}|S)a  Expects a numpy array of length 2 in the final dimension. Requires the original image size in (H, W) format. Args: target_size (`int`): The target size of the image. coords (`torch.Tensor`): The coordinates to be normalized. original_size (`tuple`): The original size of the image. is_bounding_box (`bool`, *optional*, defaults to `False`): Whether the coordinates are bounding boxes. r,).r).r)r*)rfloatreshape) rrrX original_sizer=old_hold_wnew_hnew_wrrr_normalize_coordinatess   z$Sam2Processor._normalize_coordinatesrcs|durdSt|tjr*dkst|jdkr|Sfdd|DSt|tjrLdks=t|jdkrA|Sfdd|DSt|t rbkrW|Sfdd|DSt|t t frk|St dt |)aS Recursively convert various input formats (tensors, numpy arrays, lists) to nested lists. Args: data: Input data in any format expected_depth: Expected nesting depth current_depth: Current depth in recursion Returns: Nested list representation of the data Nr,cg|] }|dqSr)_convert_to_nested_listr5item current_depthr-rrr z9Sam2Processor._convert_to_nested_list..crbrcrdrfrhrrrjrkcrbrcrdrfrhrrrjrkzUnsupported data type: )r>rr?r4shapenumpyrAnpndarraylistintrZrBtype)rdatar-rirrhrres"     z%Sam2Processor._convert_to_nested_listcCs|durg}t|ts |St|dkr|t|n t|dt||d<t|dkr_|D]0}t|tr^||}t|D]\}}|dt|krP||q>t||d|||d<q>q.|S)a` Get the maximum dimensions at each level of nesting. Args: nested_list (`list`): Nested list structure. max_dims (`list`, *optional*): Current maximum dimensions (for recursion). Returns: `list`: A list of maximum dimensions for each nesting level. Nrr))r>rpr4appendmaxrD enumerate)r nested_listmax_dimsrgsub_dimsidimrrrrDs"       z$Sam2Processor._get_nested_dimensionsc sV|dur|j}|t|kr|St|ts|g}t|}||}|t|dkr2||g||nQ|dkrg|t|dkrM||dd}|||n |g||d|fddt||Dn||dd}||||fddt|D|t|dkrtt|D]}t||tr|||||d|||<q|S)a Recursively pad a nested list to match target dimensions. Args: nested_list (`list`): Nested list to pad. target_dims (`list`): Target dimensions for each level. current_level (`int`, *optional*, defaults to 0): Current nesting level. pad_value (`int`, *optional*): Value to use for padding. Returns: `list`: The padded nested list. Nr)rr,cg|]}tqSrrr5_templaterrrj[z2Sam2Processor._pad_nested_list..cr|rrr}rrrrj`r)rr4r>rpextend_create_empty_nested_structurerangerF) rrw target_dims current_level pad_value current_sizer template_dimsrzrrrrF,s2  " zSam2Processor._pad_nested_listcs8tdkr gdSfddtdDS)a Create an empty nested structure with given dimensions filled with pad_value. Args: dims (`list`): The dimensions of the nested structure. pad_value (`int`): The value to fill the structure with. r)rcs g|] }ddqSr3)rr}dimsrrrrrjws z@Sam2Processor._create_empty_nested_structure..)r4r)rrrrrrrjs z,Sam2Processor._create_empty_nested_structurecCsLt|trt|dkr dSd||dSt|tjtjfr$t|jSdS)z Get the nesting level of a list structure. Args: input_list (`list`): The list to get the nesting level of. rr)) r>rpr4_get_nesting_levelrnrorr?rl)r input_listrrrrys   z Sam2Processor._get_nesting_levelrsr-r.r/r0c Cs|durdSt|tjtjfrH|j|kr&td|d|d|d|jd |durB|jd|krBtd|d|d|jdd |||St|t rm| |}||krgtd|d |d |d|d |||SdS) a Validate a single input by ensuring proper nesting and raising an error if the input is not valid. Args: data (`torch.Tensor`, `np.ndarray`, or `list`): Input data to process. expected_depth (`int`): Expected nesting depth. input_name (`str`): Name of the input for error messages. expected_format (`str`): The expected format of the input. expected_coord_size (`int`, *optional*): Expected coordinate size (2 for points, 4 for boxes, None for labels). . NzInput z must be a tensor/array with z, dimensions. The expected nesting format is z. Got z dimensions.rYz as the last dimension, got .z must be a nested list with z( levels. The expected nesting format is z levels.) r>rr?rnrondimrBrlrerpr)rrsr-r.r/r0rirrrrCs*     z$Sam2Processor._validate_single_inputc Cs|r||jk}|jddd}tt|D]>}||jdkrR|t|kr'||n|d}|j|j||||d} |rN||} t| ||| ||||<q| ||<qdS)a Helper method to normalize coordinates in a tensor across multiple images. Args: tensor (`torch.Tensor`): Input tensor with coordinates. original_sizes (`list`): Original image sizes. is_bounding_box (`bool`, *optional*, defaults to `False`): Whether coordinates are bounding boxes. preserve_padding (`bool`, *optional*, defaults to `False`): Whether to preserve padding values (for points). rYT)r{keepdimrr<N) rallrr4rlrarrwhere expand_as) rrGr%r=r;mask coord_maskimg_idxr\normalized_coordsimg_maskrrrrIs"  z+Sam2Processor._normalize_tensor_coordinatesTc Ks |jj|||||||fi|S)a- Remove padding and upscale masks to the original image size. Args: masks (`Union[List[torch.Tensor], List[np.ndarray]]`): Batched masks from the mask_decoder in (batch_size, num_channels, height, width) format. original_sizes (`Union[torch.Tensor, List[Tuple[int,int]]]`): The original sizes of each image before it was resized to the model's expected input shape, in (height, width) format. mask_threshold (`float`, *optional*, defaults to 0.0): Threshold for binarization and post-processing operations. binarize (`bool`, *optional*, defaults to `True`): Whether to binarize the masks. max_hole_area (`float`, *optional*, defaults to 0.0): The maximum area of a hole to fill. max_sprinkle_area (`float`, *optional*, defaults to 0.0): The maximum area of a sprinkle to fill. apply_non_overlapping_constraints (`bool`, *optional*, defaults to `False`): Whether to apply non-overlapping constraints to the masks. Returns: (`torch.Tensor`): Batched masks in batch_size, num_channels, height, width) format, where (height, width) is given by original_size. )rpost_process_masks) rmasksr%mask_thresholdbinarize max_hole_areamax_sprinkle_area!apply_non_overlapping_constraintsrrrrrs#z Sam2Processor.post_process_masks)Nr)NNNNNNN)F)r)N)rN)FF)rTrrF)__name__ __module__ __qualname____doc__ attributesimage_processor_classrrqrrrrprZrr?strr r rWrarerDrFrrrnrorCrIr __classcell__rrrrr%s    & %> 2'r)rcopyrtypingrrrmrn image_utilsrprocessing_utilsrtokenization_utils_baser utilsr r r utils.import_utilsr get_loggerrloggerrr__all__rrrrs$       k