o iU@sfdZddlmZmZmZddlZddlmZm Z m Z ddl m Z m Z ddlmZmZmZmZmZmZmZmZmZmZmZmZmZmZddlmZmZm Z dd l!m"Z"erYddl#Z#erad d l$m%Z%erpddl&Z&dd l&m'Z'm(Z(e)e*Z+ dd ej,deee-effddZ. dd edeee-efdefddZ/defddZ0e"ddGdddeZ1dgZ2dS)z%Image processor class for SuperPoint.) TYPE_CHECKINGOptionalUnionN)BaseImageProcessor BatchFeature get_size_dict)resizeto_channel_dimension_format)ChannelDimension ImageInput ImageTypePILImageResamplingget_image_typeinfer_channel_dimension_format is_pil_imageis_scaled_imageis_torch_availableis_valid_imageis_vision_availableto_numpy_array valid_imagesvalidate_preprocess_arguments) TensorTypeloggingrequires_backends)requires)KeypointMatchingOutput)Image ImageDrawimageinput_data_formatcCs|tjkr$|jddkrdSt|d|dko#t|d|dkS|tjkrH|jddkr2dSt|d|d koGt|d |d kSdS) NrrTr.r...r.r.r&)r FIRSTshapenpallLAST)r!r"r0l/home/ubuntu/.local/lib/python3.10/site-packages/transformers/models/superglue/image_processing_superglue.py is_grayscale8s , ,r2returncCsttdgt|tjrZt||dr|S|tjkr7|dd|dd|dd}tj|gd d d }|S|tj krX|d d|d d|dd}tj|gd dd }|St|t j j sc|S| d}|S)ao Converts an image to grayscale format using the NTSC formula. Only support numpy and PIL Image. TODO support torch and tensorflow grayscale conversion This function is supposed to return a 1-channel image, but it returns a 3-channel image with the same value in each channel, because of an issue that is discussed in : https://github.com/huggingface/transformers/pull/25786#issuecomment-1730176446 Args: image (Image): The image to convert. input_data_format (`ChannelDimension` or `str`, *optional*): The channel dimension format for the input image. visionr"r#gŏ1w-!?r$gbX9?r%gv/?rr)axisr(r)r*r'L) rconvert_to_grayscale isinstancer-ndarrayr2r r+stackr/PILrconvert)r!r" gray_imager0r0r1r8Gs    $ $ r8imagescshd}ddt|tr0t|dkrtfdd|Dr|Stfdd|Dr0dd |DSt|) N)z-Input images must be a one of the following :z - A pair of PIL images.z - A pair of 3D arrays.z! - A list of pairs of PIL images.z - A list of pairs of 3D arrays.cSs,t|pt|ot|tjkot|jdkS)z$images is a PIL Image or a 3D array.r)rrrr r<lenr,)r!r0r0r1_is_valid_imagevs"z8validate_and_format_image_pairs.._is_valid_imager&c3|]}|VqdSNr0.0r!rAr0r1 }z2validate_and_format_image_pairs..c3s<|]}t|tot|dkotfdd|DVqdS)r&c3rBrCr0rDrFr0r1rGrHz..N)r9listr@r.)rE image_pairrFr0r1rGs  cSsg|] }|D]}|qqSr0r0)rErJr!r0r0r1 sz3validate_and_format_image_pairs..)r9rIr@r. ValueError)r? error_messager0rFr1validate_and_format_image_pairsms " rN)torch)backendscseZdZdZdgZddejdddfdedee e e fded ed e d ed dffd d Z  d%dejde e e fdeee efdeee effddZdddddddejdf deedee e e fdeed eed ee d eedeee efdedeee efd efddZ d&dddeeeefde d ee e ejffddZdedee e ejfd ed fd!d"Zd#d$ZZS)'SuperGlueImageProcessorap Constructs a SuperGlue image processor. Args: do_resize (`bool`, *optional*, defaults to `True`): Controls whether to resize the image's (height, width) dimensions to the specified `size`. Can be overridden by `do_resize` in the `preprocess` method. size (`dict[str, int]` *optional*, defaults to `{"height": 480, "width": 640}`): Resolution of the output image after `resize` is applied. Only has an effect if `do_resize` is set to `True`. Can be overridden by `size` in the `preprocess` method. resample (`PILImageResampling`, *optional*, defaults to `Resampling.BILINEAR`): Resampling filter to use if resizing the image. Can be overridden by `resample` in the `preprocess` method. do_rescale (`bool`, *optional*, defaults to `True`): Whether to rescale the image by the specified scale `rescale_factor`. Can be overridden by `do_rescale` in the `preprocess` method. rescale_factor (`int` or `float`, *optional*, defaults to `1/255`): Scale factor to use if rescaling the image. Can be overridden by `rescale_factor` in the `preprocess` method. do_grayscale (`bool`, *optional*, defaults to `True`): Whether to convert the image to grayscale. Can be overridden by `do_grayscale` in the `preprocess` method. pixel_valuesTNgp? do_resizesizeresample do_rescalerescale_factor do_grayscaler3c s\tjdi||dur|nddd}t|dd}||_||_||_||_||_||_dS)Nii)heightwidthFdefault_to_squarer0) super__init__rrSrTrUrVrWrX)selfrSrTrUrVrWrXkwargs __class__r0r1r^s   z SuperGlueImageProcessor.__init__r! data_formatr"cKs0t|dd}t|f|d|df||d|S)aL Resize an image. Args: image (`np.ndarray`): Image to resize. size (`dict[str, int]`): Dictionary of the form `{"height": int, "width": int}`, specifying the size of the output image. data_format (`ChannelDimension` or `str`, *optional*): The channel dimension format of the output image. If not provided, it will be inferred from the input image. Can be one of: - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format. - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format. - `"none"` or `ChannelDimension.NONE`: image in (height, width) format. input_data_format (`ChannelDimension` or `str`, *optional*): The channel dimension format for the input image. If unset, the channel dimension format is inferred from the input image. Can be one of: - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format. - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format. - `"none"` or `ChannelDimension.NONE`: image in (height, width) format. Fr[rYrZ)rTrcr")rr )r_r!rTrcr"r`r0r0r1r s zSuperGlueImageProcessor.resizereturn_tensorsc  sp|dur|n|j}|dur|n|j}|dur|n|j}|dur!|n|j}|dur*|n|j}|dur3|n|j}t|dd}t|}t|sHt dt |||||ddd|D}t |dre|ret d | durot|d} g|D]+} |r|j| ||| d } |r|j| || d } |rt| | d } t| | | d } | qsfddtdtdD} d| i}t||dS)a Preprocess an image or batch of images. Args: images (`ImageInput`): Image pairs to preprocess. Expects either a list of 2 images or a list of list of 2 images list with pixel values ranging from 0 to 255. If passing in images with pixel values between 0 and 1, set `do_rescale=False`. do_resize (`bool`, *optional*, defaults to `self.do_resize`): Whether to resize the image. size (`dict[str, int]`, *optional*, defaults to `self.size`): Size of the output image after `resize` has been applied. If `size["shortest_edge"]` >= 384, the image is resized to `(size["shortest_edge"], size["shortest_edge"])`. Otherwise, the smaller edge of the image will be matched to `int(size["shortest_edge"]/ crop_pct)`, after which the image is cropped to `(size["shortest_edge"], size["shortest_edge"])`. Only has an effect if `do_resize` is set to `True`. resample (`PILImageResampling`, *optional*, defaults to `self.resample`): Resampling filter to use if resizing the image. This can be one of `PILImageResampling`, filters. Only has an effect if `do_resize` is set to `True`. do_rescale (`bool`, *optional*, defaults to `self.do_rescale`): Whether to rescale the image values between [0 - 1]. rescale_factor (`float`, *optional*, defaults to `self.rescale_factor`): Rescale factor to rescale the image by if `do_rescale` is set to `True`. do_grayscale (`bool`, *optional*, defaults to `self.do_grayscale`): Whether to convert the image to grayscale. return_tensors (`str` or `TensorType`, *optional*): The type of tensors to return. Can be one of: - Unset: Return a list of `np.ndarray`. - `TensorType.TENSORFLOW` or `'tf'`: Return a batch of type `tf.Tensor`. - `TensorType.PYTORCH` or `'pt'`: Return a batch of type `torch.Tensor`. - `TensorType.NUMPY` or `'np'`: Return a batch of type `np.ndarray`. - `TensorType.JAX` or `'jax'`: Return a batch of type `jax.numpy.ndarray`. data_format (`ChannelDimension` or `str`, *optional*, defaults to `ChannelDimension.FIRST`): The channel dimension format for the output image. Can be one of: - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format. - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format. - Unset: Use the channel dimension format of the input image. input_data_format (`ChannelDimension` or `str`, *optional*): The channel dimension format for the input image. If unset, the channel dimension format is inferred from the input image. Can be one of: - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format. - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format. - `"none"` or `ChannelDimension.NONE`: image in (height, width) format. NFr[zkInvalid image type. Must be of type PIL.Image.Image, numpy.ndarray, torch.Tensor, tf.Tensor or jax.ndarray.)rSrTrUrVrWcSg|]}t|qSr0rrDr0r0r1rK5z6SuperGlueImageProcessor.preprocess..rzIt looks like you are trying to rescale already rescaled images. If the input images have pixel values between 0 and 1, set `do_rescale=False` to avoid rescaling them again.)r!rTrUr")r!scaler"r5)input_channel_dimcg|] }||dqSr&r0rEi all_imagesr0r1rKPr&rR)data tensor_type)rSrUrVrWrXrTrrNrrLrrlogger warning_oncerr rescaler8r appendranger@r)r_r?rSrTrUrVrWrXrdrcr"r`r! image_pairsrqr0rnr1 preprocesssN:      z"SuperGlueImageProcessor.preprocessoutputsr target_sizes thresholdcCsp|jjdt|krtdtdd|Dstdt|tr*tj||jj d}n|jddks8|jddkr.zTEach element of target_sizes must contain the size (h, w) of each image of the batch)devicerr&r'N) keypoints0 keypoints1matching_scores)maskr,r@rLr.r9rIrOtensorr keypointscloneflipreshapetoint32zipmatchesr logical_andrv)r_r{r|r}image_pair_sizesrresults mask_pairkeypoints_pairrscoresmask0mask1rrmatches0scores0 valid_matchesmatched_keypoints0matched_keypoints1rr0r0r1post_process_keypoint_matchingVsF   &     z6SuperGlueImageProcessor.post_process_keypoint_matchingr?keypoint_matching_outputz Image.ImagecstddDfddtdtdD}g}t||D]\}}|djdd\}}|djdd\} } tjt|| || dftjd } |d| d|d|f<|d| d| |df<t | } t | } |d  d\}}|d  d\}}t|||||d D]D\}}}}}||}| j|||||f|dd | j|d|d|d|dfdd| j||d|d||d|dfddq|| q!|S)a Plots the image pairs side by side with the detected keypoints as well as the matching between them. Args: images (`ImageInput`): Image pairs to plot. Same as `SuperGlueImageProcessor.preprocess`. Expects either a list of 2 images or a list of list of 2 images list with pixel values ranging from 0 to 255. keypoint_matching_output (List[Dict[str, torch.Tensor]]]): A post processed keypoint matching output Returns: `List[PIL.Image.Image]`: A list of PIL images, each containing the image pairs side by side with the detected keypoints as well as the matching between them. cSrer0rfrDr0r0r1rKrgzGSuperGlueImageProcessor.visualize_keypoint_matching..crjrkr0rlr?r0r1rKrprr&Nrr)dtyperrr)fillrZblack)r)rNrwr@rr,r-zerosmaxuint8r fromarrayr Drawunbind _get_colorlineellipserv)r_r?rrxrrJ pair_outputheight0width0height1width1 plot_imageplot_image_pildraw keypoints0_x keypoints0_y keypoints1_x keypoints1_y keypoint0_x keypoint0_y keypoint1_x keypoint1_ymatching_scorecolorr0rr1visualize_keypoint_matchings<    &" z3SuperGlueImageProcessor.visualize_keypoint_matchingcCs*tdd|}td|}d}|||fS)zMaps a score to a color.rr)int)r_scorergbr0r0r1rs  z"SuperGlueImageProcessor._get_color)NN)rz) __name__ __module__ __qualname____doc__model_input_namesrBILINEARboolrdictstrrfloatr^r-r:rr r r+rrryrItuplerOTensorrr rr __classcell__r0r0rar1rQs   *    z E 7rQrC)3rtypingrrrnumpyr-image_processing_utilsrrrimage_transformsr r image_utilsr r r rrrrrrrrrrrutilsrrrutils.import_utilsrrOmodeling_supergluerr<rr get_loggerrrsr:rr2r8rNrQ__all__r0r0r0r1sH @     & R