o eil@sdZddlmZddlZddlmZddlmZddl m Z m Z m Z ddl mZmZmZmZdd lmZeeZer@ddlZd d Zd d ZddZddZddZddZddZeddeGdddeZdgZ dS)z Processor class for SAM3. deepcopyN) ImageInput)ProcessorMixin) BatchEncodingPreTokenizedInput TextInput) TensorTypeauto_docstringis_torch_availablelogging)requirescCsL|d\}}}}|d||d||d||d|g}tj|ddSN?dimunbindtorchstackxx_cy_cwhbrf/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/sam3/processing_sam3.pybox_cxcywh_to_xyxy#s,r!cCs<|d\}}}}|d||d|||g}tj|ddSrrrrrr box_cxcywh_to_xywh)r"cCs4|d\}}}}||||||g}tj|ddSNrrrryrrrrrr box_xywh_to_xyxy/r'cCs<|d\}}}}|d||d|||g}tj|ddSrrr%rrr box_xywh_to_cxcywh5r#r)cCs4|d\}}}}||||||g}tj|ddSr$r)rr&XYrrrr box_xyxy_to_xywh;r(r,cCsD|d\}}}}||d||d||||g}tj|ddS)Nrrr)rx0y0x1y1rrrr box_xyxy_to_cxcywhAs$r2cCs"|d\}}}}||||S)z Batched version of box area. Boxes should be in [x0, y0, x1, y1] format. Inputs: - boxes: Tensor of shape (..., 4) Returns: - areas: Tensor of shape (...,) r)r)boxesr.r/r0r1rrr box_areaGs r4)r)backendscsxeZdZ d4dedBdeffdd Ze       d5dedBdeeBe eBe eBdBd edBd e e e e e j BdBd e e e ee j BdBd e e e e j BdBd e eBdBdefddZd6d7ddZd8ddZddZd9ddZd:ddZdd Zd!d"Z d9d#e j ejBe Bd$ed%e d&e d'edBde f d(d)Zd;d*d+Zdd2d3ZZS)? Sam3ProcessorN target_sizepoint_pad_valuec s@tj||fi|||_|dur||_dS|jjd|_dS)z target_size (`int`, *optional*): The target size (target_size, target_size) to which the image will be resized. point_pad_value (`int`, *optional*, defaults to -10): The value used for padding input boxes. Nheight)super__init__r9image_processorsizer8)selfr= tokenizerr8r9kwargs __class__rr r<Xs $zSam3Processor.__init__imagestextsegmentation_maps input_boxesinput_boxes_labelsoriginal_sizesreturn_tensorsreturncKsd} |dur|j|f||d|} n!|dur,t|tjr#|}td|i|d} n|dur4td|||}|durS|j ||ddd} | durQ| | n| } |dur| d}|j |d d d d d } |j |dddd} | dur{| | dd} | dur| | dd}| dur| dur| |krtd| dur| | | d g}tj|tjd}|j||dddt|}| d|i| dur| | |}tj|tjd}| d|i| S)a} images (`ImageInput`, *optional*): The image(s) to process. text (`str`, `list[str]`, `list[list[str]]`, *optional*): The text to process. segmentation_maps (`ImageInput`, *optional*): The segmentation maps to process. input_boxes (`list[list[list[float]]]`, `torch.Tensor`, *optional*): The bounding boxes to process. input_boxes_labels (`list[list[int]]`, `torch.Tensor`, *optional*): The labels for the bounding boxes. original_sizes (`list[list[float]]`, `torch.Tensor`, *optional*): The original sizes of the images. Returns: A [`BatchEncoding`] with the following fields: - `pixel_values` (`torch.Tensor`): The processed image(s). - `original_sizes` (`list[list[float]]`): The original sizes of the images. - `labels` (`torch.Tensor`): The processed segmentation maps (if provided). - `input_boxes_labels` (`torch.Tensor`): The processed labels for the bounding boxes. - `input_boxes` (`torch.Tensor`): The processed bounding boxes. N)rFrJrI) tensor_typezKEither images or original_sizes must be provided if input_boxes is not None max_length )rJpaddingrMrr3z)[image level, box level, box coordinates])expected_depth input_nameexpected_formatexpected_coord_sizer-labelsz[image level, box level])rQrRrSzaInput boxes and labels have inconsistent dimensions. Please ensure they have the same dimensions.)dtypeT)is_bounding_boxpreserve_paddingrGrH)r= isinstancerTensorcputolistr ValueError_resolve_text_promptsr@update_validate_single_input_get_nested_dimensions_pad_nested_listtensorfloat32_normalize_tensor_coordinatesr2int64)r?rDrErFrGrHrIrJrAencoding text_inputsprocessed_boxesprocessed_boxes_labelsboxes_max_dimsboxes_labels_max_dims padded_boxes final_boxespadded_boxes_labelsfinal_boxes_labelsrrr __call__esv"     zSam3Processor.__call__Fcoords torch.TensorcCsZ|\}}t|}|r|ddd}|d||d<|d||d<|r+|dd}|S)a  Expects a numpy array of length 2 in the final dimension. Requires the original image size in (H, W) format. Args: target_size (`int`): The target size of the image. coords (`torch.Tensor`): The coordinates to be normalized. original_size (`tuple`): The original size of the image. is_bounding_box (`bool`, *optional*, defaults to `False`): Whether the coordinates are bounding boxes. rr-).r).rP)rfloatreshape)r?rr original_sizerWold_hold_wrrr _normalize_coordinatess  z$Sam3Processor._normalize_coordinatesrcs|durdSt|tjr*dkst|jdkr|Sfdd|DSt|tjrLdks=t|jdkrA|Sfdd|DSt|t rbkrW|Sfdd|DSt|t t frk|St dt |)a Recursively convert various input formats (tensors, numpy arrays, lists) to nested lists. Preserves None values within lists. Args: data: Input data in any format (may be None or contain None values) expected_depth: Expected nesting depth current_depth: Current depth in recursion Returns: Nested list representation of the data (or None) Nr-cg|] }|dqSrt_convert_to_nested_list.0item current_depthrQr?rr z9Sam3Processor._convert_to_nested_list..cr{r|r}rrrr rrcs*g|]}|dur|dndqS)Nrtr}rrrr r szUnsupported data type: )rYrrZlenshapenumpyr\npndarraylistintrur]type)r?datarQrrrr r~s&     z%Sam3Processor._convert_to_nested_listcCs|dur |rdSdSt|ttfs|St|}|r0t|t|kr0tdt|dt|dt|D]\}}|durH|rH||durHd||<q4|S)zQ Resolve text prompts by setting defaults based on prompt types. NvisualzEThe number of text prompts must match the number of input boxes. Got z text prompts and z input boxes.)rYrtuplerr] enumerate)r?rErGi text_valuerrr r^s$ z#Sam3Processor._resolve_text_promptscCs|durg}t|ts |St|dkr|t|n t|dt||d<t|dkrd|D]5}|dur5q.t|trc||}t|D]\}}|dt|krU||qCt||d|||d<qCq.|S)a Get the maximum dimensions at each level of nesting, skipping None values. Args: nested_list (`list`): Nested list structure (may contain None values). max_dims (`list`, *optional*): Current maximum dimensions (for recursion). Returns: `list`: A list of maximum dimensions for each nesting level. Nrrt)rYrrappendmaxrar)r? nested_listmax_dimsrsub_dimsrrrrr ra.s&       z$Sam3Processor._get_nested_dimensionsc s|dur|j}|t|kr|St|ts|g}t|}||}|t|dkr2||g||nQ|dkrg|t|dkrM||dd}|||n |g||d|fddt||Dn||dd}||||fddt|D|t|dkrtt|D].}||dur||dd}|||||<qt||tr|||||d|||<q|S)a3 Recursively pad a nested list to match target dimensions. Replaces None values with padded structures. Args: nested_list (`list`): Nested list to pad (may contain None values). target_dims (`list`): Target dimensions for each level. current_level (`int`, *optional*, defaults to 0): Current nesting level. pad_value (`int`, *optional*): Value to use for padding. Returns: `list`: The padded nested list. Nrtrr-cg|]}tqSrrr_templaterr rz2Sam3Processor._pad_nested_list..crrrrrrr rr)r9rrYrextend_create_empty_nested_structurerangerb) r?r target_dims current_level pad_value current_sizer8 template_dimsrrrr rbVs8  "  zSam3Processor._pad_nested_listcs8tdkr gdSfddtdDS)a Create an empty nested structure with given dimensions filled with pad_value. Args: dims (`list`): The dimensions of the nested structure. pad_value (`int`): The value to fill the structure with. rtrcs g|] }ddqS)rtN)rrdimsrr?rr rs z@Sam3Processor._create_empty_nested_structure..)rr)r?rrrrr rs z,Sam3Processor._create_empty_nested_structurecCsbt|tr!t|dkr dS|D]}|durd||SqdSt|tjtjfr/t|jSdS)z Get the nesting level of a list structure, skipping None values. Args: input_list (`list`): The list to get the nesting level of. rrtN) rYrr_get_nesting_levelrrrrZr)r? input_listrrrr rs   z Sam3Processor._get_nesting_levelrrQrRrSrTc Cs|durdSt|tjtjfrH|j|kr&td|d|d|d|jd |durB|jd|krBtd|d|d|jdd |||St|t rm| |}||krgtd|d |d |d|d |||SdS) a Validate a single input by ensuring proper nesting and raising an error if the input is not valid. Args: data (`torch.Tensor`, `np.ndarray`, or `list`): Input data to process. expected_depth (`int`): Expected nesting depth. input_name (`str`): Name of the input for error messages. expected_format (`str`): The expected format of the input. expected_coord_size (`int`, *optional*): Expected coordinate size (4 for boxes, None for labels). . NzInput z must be a tensor/array with z, dimensions. The expected nesting format is z. Got z dimensions.rz as the last dimension, got .z must be a nested list with z( levels. The expected nesting format is z levels.) rYrrZrrndimr]rr~rr)r?rrQrRrSrTrrrr r`s*     z$Sam3Processor._validate_single_inputc Cs|r||jk}|jddd}tt|D]<}||jdkrP|t|kr'||n|d}|j||||d} |rL||} t| ||| ||||<q| ||<qdS)a Helper method to normalize coordinates in a tensor across multiple images. Args: tensor (`torch.Tensor`): Input tensor with coordinates. original_sizes (`list`): Original image sizes. is_bounding_box (`bool`, *optional*, defaults to `False`): Whether coordinates are bounding boxes. preserve_padding (`bool`, *optional*, defaults to `False`): Whether to preserve padding values (for boxes). rT)rkeepdimr)rWN) r9allrrrrzrwhere expand_as) r?rcrIrWrXmask coord_maskimg_idxrwnormalized_coordsimg_maskrrr res"   z+Sam3Processor._normalize_tensor_coordinatesrcC|j|||S)a Converts the output of [`Sam3Model`] into semantic segmentation maps. Args: outputs ([`Sam3ImageSegmentationOutput`]): Raw outputs of the model containing semantic_seg. target_sizes (`list[tuple]` of length `batch_size`, *optional*): List of tuples corresponding to the requested final size (height, width) of each prediction. If unset, predictions will not be resized. threshold (`float`, *optional*, defaults to 0.5): Threshold for binarizing the semantic segmentation masks. Returns: semantic_segmentation: `list[torch.Tensor]` of length `batch_size`, where each item is a semantic segmentation map of shape (height, width) corresponding to the target_sizes entry (if `target_sizes` is specified). Each entry is a binary mask (0 or 1). )r="post_process_semantic_segmentation)r?outputs target_sizes thresholdrrr rsz0Sam3Processor.post_process_semantic_segmentation333333?cCr)a Converts the raw output of [`Sam3Model`] into final bounding boxes in (top_left_x, top_left_y, bottom_right_x, bottom_right_y) format. This is a convenience wrapper around the image processor method. Args: outputs ([`Sam3ImageSegmentationOutput`]): Raw outputs of the model containing pred_boxes, pred_logits, and optionally presence_logits. threshold (`float`, *optional*, defaults to 0.3): Score threshold to keep object detection predictions. target_sizes (`list[tuple[int, int]]`, *optional*): List of tuples (`tuple[int, int]`) containing the target size `(height, width)` of each image in the batch. If unset, predictions will not be resized. Returns: `list[dict]`: A list of dictionaries, each dictionary containing the following keys: - **scores** (`torch.Tensor`): The confidence scores for each predicted box on the image. - **boxes** (`torch.Tensor`): Image bounding boxes in (top_left_x, top_left_y, bottom_right_x, bottom_right_y) format. Example: ```python >>> from transformers import AutoModel, AutoProcessor >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> model = AutoModel.from_pretrained("facebook/sam3-base") >>> processor = AutoProcessor.from_pretrained("facebook/sam3-base") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> inputs = processor(images=image, text="cat", return_tensors="pt") >>> outputs = model(**inputs) >>> # Post-process to get bounding boxes >>> results = processor.post_process_object_detection(outputs, threshold=0.3, target_sizes=[image.size[::-1]]) >>> boxes = results[0]["boxes"] >>> scores = results[0]["scores"] ``` )r=post_process_object_detection)r?rrrrrr r&s+z+Sam3Processor.post_process_object_detectioncCs|j||||S)ay Converts the raw output of [`Sam3Model`] into instance segmentation predictions with bounding boxes and masks. This is a convenience wrapper around the image processor method. Args: outputs ([`Sam3ImageSegmentationOutput`]): Raw outputs of the model containing pred_boxes, pred_logits, pred_masks, and optionally presence_logits. threshold (`float`, *optional*, defaults to 0.3): Score threshold to keep instance predictions. mask_threshold (`float`, *optional*, defaults to 0.5): Threshold for binarizing the predicted masks. target_sizes (`list[tuple[int, int]]`, *optional*): List of tuples (`tuple[int, int]`) containing the target size `(height, width)` of each image in the batch. If unset, predictions will not be resized. Returns: `list[dict]`: A list of dictionaries, each dictionary containing the following keys: - **scores** (`torch.Tensor`): The confidence scores for each predicted instance on the image. - **boxes** (`torch.Tensor`): Image bounding boxes in (top_left_x, top_left_y, bottom_right_x, bottom_right_y) format. - **masks** (`torch.Tensor`): Binary segmentation masks for each instance, shape (num_instances, height, width). Example: ```python >>> from transformers import AutoModel, AutoProcessor >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> model = AutoModel.from_pretrained("facebook/sam3-base") >>> processor = AutoProcessor.from_pretrained("facebook/sam3-base") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> inputs = processor(images=image, text="cat", return_tensors="pt") >>> outputs = model(**inputs) >>> # Post-process to get instance segmentation >>> results = processor.post_process_instance_segmentation( ... outputs, threshold=0.3, target_sizes=[image.size[::-1]] ... ) >>> masks = results[0]["masks"] >>> boxes = results[0]["boxes"] >>> scores = results[0]["scores"] ``` )r="post_process_instance_segmentation)r?rrmask_thresholdrrrr rSs9z0Sam3Processor.post_process_instance_segmentation)Nr7)NNNNNNN)F)rrrsrKrs)r)N)rN)FF)Nr)rN)rrN) __name__ __module__ __qualname__rr<r rr rrrurrZstrr rrqrzr~r^rarbrrrrr`rerrr __classcell__rrrBr r6Usx   h *  (B 2 # 0r6)!__doc__copyrrr image_utilsrprocessing_utilsrtokenization_utils_baserrr utilsr r r r utils.import_utilsr get_loggerrloggerrr!r"r'r)r,r2r4r6__all__rrrr s4      >