o i a@sbddlZddlmZmZddlZddlZddlmZm Z m Z ddl m Z m Z ddlmZmZmZmZmZmZmZmZmZmZmZmZmZddlmZmZmZm Z ddl!m"Z"d d l#m$Z$ergdd l%m&Z&m'Z'ernddl%Z%e(e)Z* dd ej+d eee,effddZ- dd ed eee,efdefddZ.defddZ/e"ddGdddeZ0dgZ1dS)N)OptionalUnion)BaseImageProcessor BatchFeature get_size_dict)resizeto_channel_dimension_format) ChannelDimension ImageInput ImageTypePILImageResamplingget_image_typeinfer_channel_dimension_format is_pil_imageis_scaled_imageis_valid_imageis_vision_availableto_numpy_array valid_imagesvalidate_preprocess_arguments) TensorTypeis_matplotlib_availableloggingrequires_backends)requires)LightGlueKeypointMatchingOutput)Image ImageDrawimageinput_data_formatcCs|tjkr$|jddkrdSt|d|dko#t|d|dkS|tjkrH|jddkr2dSt|d|d koGt|d |d kSdS) NrrTr.r...r.r.r%)r FIRSTshapenpallLAST)r r!r/l/home/ubuntu/.local/lib/python3.10/site-packages/transformers/models/lightglue/image_processing_lightglue.py is_grayscale9s , ,r1returncCsttdgt|tjrZt||dr|S|tjkr7|dd|dd|dd}tj|gd d d }|S|tj krX|d d|d d|dd}tj|gd dd }|St|t j j sc|S| d}|S)ao Converts an image to grayscale format using the NTSC formula. Only support numpy and PIL Image. TODO support torch and tensorflow grayscale conversion This function is supposed to return a 1-channel image, but it returns a 3-channel image with the same value in each channel, because of an issue that is discussed in : https://github.com/huggingface/transformers/pull/25786#issuecomment-1730176446 Args: image (Image): The image to convert. input_data_format (`ChannelDimension` or `str`, *optional*): The channel dimension format for the input image. visionr!r"gŏ1w-!?r#gbX9?r$gv/?rr)axisr'r(r)r&L) rconvert_to_grayscale isinstancer,ndarrayr1r r*stackr.PILrconvert)r r! gray_imager/r/r0r7Gs    $ $ r7imagescshd}ddt|tr0t|dkrtfdd|Dr|Stfdd|Dr0dd |DSt|) N)z-Input images must be a one of the following :z - A pair of PIL images.z - A pair of 3D arrays.z! - A list of pairs of PIL images.z - A list of pairs of 3D arrays.cSs,t|pt|ot|tjkot|jdkS)z$images is a PIL Image or a 3D array.r)rrrr r;lenr+)r r/r/r0_is_valid_imagevs"z8validate_and_format_image_pairs.._is_valid_imager%c3|]}|VqdSNr/.0r r@r/r0 }z2validate_and_format_image_pairs..c3s<|]}t|tot|dkotfdd|DVqdS)r%c3rArBr/rCrEr/r0rFrGz..N)r8listr?r-)rD image_pairrEr/r0rFs  cSsg|] }|D]}|qqSr/r/)rDrIr r/r/r0 sz3validate_and_format_image_pairs..)r8rHr?r- ValueError)r> error_messager/rEr0validate_and_format_image_pairsms " rM)torch)backendscseZdZdZdgZddejdddfdedee e e fded ed e d ed dffd d Z  d&dejde e e fdeee efdeee effddZdddddddejdf deedee e e fdeed eed ee d eedeee efdedeee efd efddZ d'dedeeeefde d ee e ejffddZdedee e ejfd edfd d!Zd"d#Zdedefd$d%Z Z!S)(LightGlueImageProcessorap Constructs a LightGlue image processor. Args: do_resize (`bool`, *optional*, defaults to `True`): Controls whether to resize the image's (height, width) dimensions to the specified `size`. Can be overridden by `do_resize` in the `preprocess` method. size (`dict[str, int]` *optional*, defaults to `{"height": 480, "width": 640}`): Resolution of the output image after `resize` is applied. Only has an effect if `do_resize` is set to `True`. Can be overridden by `size` in the `preprocess` method. resample (`PILImageResampling`, *optional*, defaults to `Resampling.BILINEAR`): Resampling filter to use if resizing the image. Can be overridden by `resample` in the `preprocess` method. do_rescale (`bool`, *optional*, defaults to `True`): Whether to rescale the image by the specified scale `rescale_factor`. Can be overridden by `do_rescale` in the `preprocess` method. rescale_factor (`int` or `float`, *optional*, defaults to `1/255`): Scale factor to use if rescaling the image. Can be overridden by `rescale_factor` in the `preprocess` method. do_grayscale (`bool`, *optional*, defaults to `True`): Whether to convert the image to grayscale. Can be overridden by `do_grayscale` in the `preprocess` method. pixel_valuesTNgp? do_resizesizeresample do_rescalerescale_factor do_grayscaler2c s\tjdi||dur|nddd}t|dd}||_||_||_||_||_||_dS)Nii)heightwidthFdefault_to_squarer/) super__init__rrRrSrTrUrVrW)selfrRrSrTrUrVrWkwargs __class__r/r0r]s   z LightGlueImageProcessor.__init__r data_formatr!cKs0t|dd}t|f|d|df||d|S)aL Resize an image. Args: image (`np.ndarray`): Image to resize. size (`dict[str, int]`): Dictionary of the form `{"height": int, "width": int}`, specifying the size of the output image. data_format (`ChannelDimension` or `str`, *optional*): The channel dimension format of the output image. If not provided, it will be inferred from the input image. Can be one of: - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format. - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format. - `"none"` or `ChannelDimension.NONE`: image in (height, width) format. input_data_format (`ChannelDimension` or `str`, *optional*): The channel dimension format for the input image. If unset, the channel dimension format is inferred from the input image. Can be one of: - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format. - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format. - `"none"` or `ChannelDimension.NONE`: image in (height, width) format. FrZrXrY)rSrbr!)rr)r^r rSrbr!r_r/r/r0rs zLightGlueImageProcessor.resizereturn_tensorsc  sp|dur|n|j}|dur|n|j}|dur|n|j}|dur!|n|j}|dur*|n|j}|dur3|n|j}t|dd}t|}t|sHt dt |||||ddd|D}t |dre|ret d | durot|d} g|D]+} |r|j| ||| d } |r|j| || d } |rt| | d } t| | | d } | qsfddtdtdD} d| i}t||dS)a Preprocess an image or batch of images. Args: images (`ImageInput`): Image pairs to preprocess. Expects either a list of 2 images or a list of list of 2 images list with pixel values ranging from 0 to 255. If passing in images with pixel values between 0 and 1, set `do_rescale=False`. do_resize (`bool`, *optional*, defaults to `self.do_resize`): Whether to resize the image. size (`dict[str, int]`, *optional*, defaults to `self.size`): Size of the output image after `resize` has been applied. If `size["shortest_edge"]` >= 384, the image is resized to `(size["shortest_edge"], size["shortest_edge"])`. Otherwise, the smaller edge of the image will be matched to `int(size["shortest_edge"]/ crop_pct)`, after which the image is cropped to `(size["shortest_edge"], size["shortest_edge"])`. Only has an effect if `do_resize` is set to `True`. resample (`PILImageResampling`, *optional*, defaults to `self.resample`): Resampling filter to use if resizing the image. This can be one of `PILImageResampling`, filters. Only has an effect if `do_resize` is set to `True`. do_rescale (`bool`, *optional*, defaults to `self.do_rescale`): Whether to rescale the image values between [0 - 1]. rescale_factor (`float`, *optional*, defaults to `self.rescale_factor`): Rescale factor to rescale the image by if `do_rescale` is set to `True`. do_grayscale (`bool`, *optional*, defaults to `self.do_grayscale`): Whether to convert the image to grayscale. return_tensors (`str` or `TensorType`, *optional*): The type of tensors to return. Can be one of: - Unset: Return a list of `np.ndarray`. - `TensorType.TENSORFLOW` or `'tf'`: Return a batch of type `tf.Tensor`. - `TensorType.PYTORCH` or `'pt'`: Return a batch of type `torch.Tensor`. - `TensorType.NUMPY` or `'np'`: Return a batch of type `np.ndarray`. - `TensorType.JAX` or `'jax'`: Return a batch of type `jax.numpy.ndarray`. data_format (`ChannelDimension` or `str`, *optional*, defaults to `ChannelDimension.FIRST`): The channel dimension format for the output image. Can be one of: - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format. - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format. - Unset: Use the channel dimension format of the input image. input_data_format (`ChannelDimension` or `str`, *optional*): The channel dimension format for the input image. If unset, the channel dimension format is inferred from the input image. Can be one of: - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format. - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format. - `"none"` or `ChannelDimension.NONE`: image in (height, width) format. NFrZzkInvalid image type. Must be of type PIL.Image.Image, numpy.ndarray, torch.Tensor, tf.Tensor or jax.ndarray.)rRrSrTrUrVcSg|]}t|qSr/rrCr/r/r0rJ4z6LightGlueImageProcessor.preprocess..rzIt looks like you are trying to rescale already rescaled images. If the input images have pixel values between 0 and 1, set `do_rescale=False` to avoid rescaling them again.)r rSrTr!)r scaler!r4)input_channel_dimcg|] }||dqSr%r/rDi all_imagesr/r0rJOr%rQ)data tensor_type)rRrTrUrVrWrSrrMrrKrrlogger warning_oncerrrescaler7r appendranger?r)r^r>rRrSrTrUrVrWrcrbr!r_r image_pairsrpr/rmr0 preprocesssN:      z"LightGlueImageProcessor.preprocessoutputs target_sizes thresholdcCsp|jjdt|krtdtdd|Dstdt|tr*tj||jj d}n|jddks8|jddkr.zTEach element of target_sizes must contain the size (h, w) of each image of the batch)devicerr%r&N) keypoints0 keypoints1matching_scores)maskr+r?rKr-r8rHrNtensorr~ keypointscloneflipreshapetoint32zipmatchesr logical_andru)r^rzr{r|image_pair_sizesrresults mask_pairkeypoints_pairrscoresmask0mask1rrmatches0scores0 valid_matchesmatched_keypoints0matched_keypoints1rr/r/r0post_process_keypoint_matchingUsF   &     z6LightGlueImageProcessor.post_process_keypoint_matchingr>keypoint_matching_outputz Image.ImagecstddDfddtdtdD}g}t||D]\}}|djdd\}}|djdd\} } tjt|| || dftjd } |d| d|d|f<|d| d| |df<t | } t | } |d  d\}}|d  d\}}t|||||d D]D\}}}}}||}| j|||||f|dd | j|d|d|d|dfdd| j||d|d||d|dfddq|| q!|S)a Plots the image pairs side by side with the detected keypoints as well as the matching between them. Args: images (`ImageInput`): Image pairs to plot. Same as `LightGlueImageProcessor.preprocess`. Expects either a list of 2 images or a list of list of 2 images list with pixel values ranging from 0 to 255. keypoint_matching_output (List[Dict[str, torch.Tensor]]]): A post processed keypoint matching output Returns: `List[PIL.Image.Image]`: A list of PIL images, each containing the image pairs side by side with the detected keypoints as well as the matching between them. cSrdr/rerCr/r/r0rJrfzGLightGlueImageProcessor.visualize_keypoint_matching..crirjr/rkr>r/r0rJrorr%Nrr)dtyperrr)fillrYblack)r)rMrvr?rr+r,zerosmaxuint8r fromarrayrDrawunbind _get_colorlineellipseru)r^r>rrwrrI pair_outputheight0width0height1width1 plot_imageplot_image_pildraw keypoints0_x keypoints0_y keypoints1_x keypoints1_y keypoint0_x keypoint0_y keypoint1_x keypoint1_ymatching_scorecolorr/rr0visualize_keypoint_matchings<    &" z3LightGlueImageProcessor.visualize_keypoint_matchingcCs*tdd|}td|}d}|||fS)zMaps a score to a color.rr)int)r^scorergbr/r/r0rs  z"LightGlueImageProcessor._get_colorc stdttrddlm}ntdtddDfddtdt dD}t ||D]\}}|dj dd\}}|d j dd\} } t t|| || d f} |dd | d|d|f<|d d | d| |df<|| |d |d d \} } |dd \}}t | | |||dD]2\}}}}}|j|||g||g|d|ddd|j||ddd|j|||dddq|q3dS)a Plots the image pairs side by side with the detected keypoints as well as the matching between them. Requires matplotlib to be installed. .. deprecated:: `plot_keypoint_matching` is deprecated and will be removed in a future version. Use `visualize_keypoint_matching` instead. Args: images (`ImageInput`): Image pairs to plot. Same as `LightGlueImageProcessor.preprocess`. Expects either a list of 2 images or a list of list of 2 images list with pixel values ranging from 0 to 255. keypoint_matching_output ([`LightGlueKeypointMatchingOutput`]): Raw outputs of the model. zx`plot_keypoint_matching` is deprecated and will be removed in transformers v. Use `visualize_keypoint_matching` instead.rNz@Please install matplotlib to use `plot_keypoint_matching` methodcSrdr/rerCr/r/r0rJrfzBLightGlueImageProcessor.plot_keypoint_matching..crirjr/rkrr/r0rJror%rrgo@offrrrRdYlGng?g?)ralpha linewidthr)cs)warningswarn FutureWarningrmatplotlib.pyplotpyplot ImportErrorrMrvr?rr+r,rrimshowr5rplotget_cmapitemscattershow)r^r>rpltrwrIrrrrrrrrrrrrrrrr/rr0plot_keypoint_matchingsD    z.LightGlueImageProcessor.plot_keypoint_matching)NN)ry)"__name__ __module__ __qualname____doc__model_input_namesr BILINEARboolrdictstrrfloatr]r,r9rr rr*rrrxrrHtuplerNTensorrr rrr __classcell__r/r/r`r0rPs   *    z D 6rPrB)2rtypingrrnumpyr,rNimage_processing_utilsrrrimage_transformsrr image_utilsr r r r rrrrrrrrrutilsrrrrutils.import_utilsrmodeling_lightgluerr;rr get_loggerrrrr9rr1r7rMrP__all__r/r/r/r0sH<     &