o iT@sPdZddlmZmZddlZddlmZmZddl m Z m Z m Z ddl mZmZddlmZmZmZmZmZmZmZmZmZmZmZmZdd lmZmZm Z erUddl!Z!erjddl"Z"dd l"m#Z#m$Z$d d l%m&Z&e'e(Z) dd ej*deee+effddZ, dd edeee+efdefddZ-defddZ.Gddde Z/dgZ0dS)z%Image processor class for SuperPoint.)OptionalUnionN)is_torch_availableis_vision_available)BaseImageProcessor BatchFeature get_size_dict)resizeto_channel_dimension_format) ChannelDimension ImageInput ImageTypePILImageResamplingget_image_typeinfer_channel_dimension_format is_pil_imageis_scaled_imageis_valid_imageto_numpy_array valid_imagesvalidate_preprocess_arguments) TensorTypeloggingrequires_backends)Image ImageDraw)KeypointMatchingOutputimageinput_data_formatcCs|tjkr$|jddkrdSt|d|dko#t|d|dkS|tjkrH|jddkr2dSt|d|d koGt|d |d kSdS) NrrTr.r...r.r.r$)r FIRSTshapenpallLAST)rr r.v/home/ubuntu/.local/lib/python3.10/site-packages/transformers/models/efficientloftr/image_processing_efficientloftr.py is_grayscale5s , ,r0returncCsttdgt|tjrZt||dr|S|tjkr7|dd|dd|dd}tj|gd d d }|S|tj krX|d d|d d|dd}tj|gd dd }|St|t j j sc|S| d}|S)ao Converts an image to grayscale format using the NTSC formula. Only support numpy and PIL Image. TODO support torch and tensorflow grayscale conversion This function is supposed to return a 1-channel image, but it returns a 3-channel image with the same value in each channel, because of an issue that is discussed in : https://github.com/huggingface/transformers/pull/25786#issuecomment-1730176446 Args: image (Image): The image to convert. input_data_format (`ChannelDimension` or `str`, *optional*): The channel dimension format for the input image. visionr r!gŏ1w-!?r"gbX9?r#gv/?rr)axisr&r'r(r%L) rconvert_to_grayscale isinstancer+ndarrayr0r r)stackr-PILrconvert)rr gray_imager.r.r/r6Ds    $ $ r6imagescshd}ddt|tr0t|dkrtfdd|Dr|Stfdd|Dr0dd |DSt|) N)z-Input images must be a one of the following :z - A pair of PIL images.z - A pair of 3D arrays.z! - A list of pairs of PIL images.z - A list of pairs of 3D arrays.cSs,t|pt|ot|tjkot|jdkS)z$images is a PIL Image or a 3D array.r)rrrrr:lenr*)rr.r.r/_is_valid_imagets"z8validate_and_format_image_pairs.._is_valid_imager$c3|]}|VqdSNr..0rr?r.r/ {z2validate_and_format_image_pairs..c3s<|]}t|tot|dkotfdd|DVqdS)r$c3r@rAr.rBrDr.r/rErFz..N)r7listr>r,)rC image_pairrDr.r/rE}s  cSsg|] }|D]}|qqSr.r.)rCrHrr.r.r/ sz3validate_and_format_image_pairs..)r7rGr>r, ValueError)r= error_messager.rDr/validate_and_format_image_pairsks " rLcseZdZdZdgZddejdddfdedee e e fded ed e d ed dffd d Z  d%dejde e e fdeee efdeee effddZdddddddejdf deedee e e fdeed eed ee d eedeee efdedeee efd efddZ d&dddeeeefde d ee e ejffddZdedee e ejfd ed fd!d"Zd#d$ZZS)'EfficientLoFTRImageProcessorau Constructs a EfficientLoFTR image processor. Args: do_resize (`bool`, *optional*, defaults to `True`): Controls whether to resize the image's (height, width) dimensions to the specified `size`. Can be overridden by `do_resize` in the `preprocess` method. size (`Dict[str, int]` *optional*, defaults to `{"height": 480, "width": 640}`): Resolution of the output image after `resize` is applied. Only has an effect if `do_resize` is set to `True`. Can be overridden by `size` in the `preprocess` method. resample (`PILImageResampling`, *optional*, defaults to `Resampling.BILINEAR`): Resampling filter to use if resizing the image. Can be overridden by `resample` in the `preprocess` method. do_rescale (`bool`, *optional*, defaults to `True`): Whether to rescale the image by the specified scale `rescale_factor`. Can be overridden by `do_rescale` in the `preprocess` method. rescale_factor (`int` or `float`, *optional*, defaults to `1/255`): Scale factor to use if rescaling the image. Can be overridden by `rescale_factor` in the `preprocess` method. do_grayscale (`bool`, *optional*, defaults to `True`): Whether to convert the image to grayscale. Can be overridden by `do_grayscale` in the `preprocess` method. pixel_valuesTNgp? do_resizesizeresample do_rescalerescale_factor do_grayscaler1c s\tjdi||dur|nddd}t|dd}||_||_||_||_||_||_dS)Nii)heightwidthFdefault_to_squarer.) super__init__r rOrPrQrRrSrT)selfrOrPrQrRrSrTkwargs __class__r.r/rZs   z%EfficientLoFTRImageProcessor.__init__r data_formatr cKs0t|dd}t|f|d|df||d|S)aL Resize an image. Args: image (`np.ndarray`): Image to resize. size (`dict[str, int]`): Dictionary of the form `{"height": int, "width": int}`, specifying the size of the output image. data_format (`ChannelDimension` or `str`, *optional*): The channel dimension format of the output image. If not provided, it will be inferred from the input image. Can be one of: - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format. - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format. - `"none"` or `ChannelDimension.NONE`: image in (height, width) format. input_data_format (`ChannelDimension` or `str`, *optional*): The channel dimension format for the input image. If unset, the channel dimension format is inferred from the input image. Can be one of: - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format. - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format. - `"none"` or `ChannelDimension.NONE`: image in (height, width) format. FrWrUrV)rPr_r )r r )r[rrPr_r r\r.r.r/r s z#EfficientLoFTRImageProcessor.resizereturn_tensorsc  sp|dur|n|j}|dur|n|j}|dur|n|j}|dur!|n|j}|dur*|n|j}|dur3|n|j}t|dd}t|}t|sHt dt |||||ddd|D}t |dre|ret d | durot|d} g|D]+} |r|j| ||| d } |r|j| || d } |rt| | d } t| | | d } | qsfddtdtdD} d| i}t||dS)a Preprocess an image or batch of images. Args: images (`ImageInput`): Image pairs to preprocess. Expects either a list of 2 images or a list of list of 2 images list with pixel values ranging from 0 to 255. If passing in images with pixel values between 0 and 1, set `do_rescale=False`. do_resize (`bool`, *optional*, defaults to `self.do_resize`): Whether to resize the image. size (`dict[str, int]`, *optional*, defaults to `self.size`): Size of the output image after `resize` has been applied. If `size["shortest_edge"]` >= 384, the image is resized to `(size["shortest_edge"], size["shortest_edge"])`. Otherwise, the smaller edge of the image will be matched to `int(size["shortest_edge"]/ crop_pct)`, after which the image is cropped to `(size["shortest_edge"], size["shortest_edge"])`. Only has an effect if `do_resize` is set to `True`. resample (`PILImageResampling`, *optional*, defaults to `self.resample`): Resampling filter to use if resizing the image. This can be one of `PILImageResampling`, filters. Only has an effect if `do_resize` is set to `True`. do_rescale (`bool`, *optional*, defaults to `self.do_rescale`): Whether to rescale the image values between [0 - 1]. rescale_factor (`float`, *optional*, defaults to `self.rescale_factor`): Rescale factor to rescale the image by if `do_rescale` is set to `True`. do_grayscale (`bool`, *optional*, defaults to `self.do_grayscale`): Whether to convert the image to grayscale. return_tensors (`str` or `TensorType`, *optional*): The type of tensors to return. Can be one of: - Unset: Return a list of `np.ndarray`. - `TensorType.TENSORFLOW` or `'tf'`: Return a batch of type `tf.Tensor`. - `TensorType.PYTORCH` or `'pt'`: Return a batch of type `torch.Tensor`. - `TensorType.NUMPY` or `'np'`: Return a batch of type `np.ndarray`. - `TensorType.JAX` or `'jax'`: Return a batch of type `jax.numpy.ndarray`. data_format (`ChannelDimension` or `str`, *optional*, defaults to `ChannelDimension.FIRST`): The channel dimension format for the output image. Can be one of: - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format. - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format. - Unset: Use the channel dimension format of the input image. input_data_format (`ChannelDimension` or `str`, *optional*): The channel dimension format for the input image. If unset, the channel dimension format is inferred from the input image. Can be one of: - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format. - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format. - `"none"` or `ChannelDimension.NONE`: image in (height, width) format. NFrWzkInvalid image type. Must be of type PIL.Image.Image, numpy.ndarray, torch.Tensor, tf.Tensor or jax.ndarray.)rOrPrQrRrScSg|]}t|qSr.rrBr.r.r/rI3z;EfficientLoFTRImageProcessor.preprocess..rzIt looks like you are trying to rescale already rescaled images. If the input images have pixel values between 0 and 1, set `do_rescale=False` to avoid rescaling them again.)rrPrQr )rscaler r3)input_channel_dimcg|] }||dqSr$r.rCi all_imagesr.r/rINr$rN)data tensor_type)rOrQrRrSrTrPr rLrrJrrlogger warning_oncerr rescaler6r appendranger>r)r[r=rOrPrQrRrSrTr`r_r r\r image_pairsrmr.rjr/ preprocesssN:      z'EfficientLoFTRImageProcessor.preprocessoutputsr target_sizes thresholdcCs&|jjdt|krtdtdd|Dstdt|tr*tj||jj d}n|jddks8|jddkr)rC target_sizer.r.r/rElszNEfficientLoFTRImageProcessor.post_process_keypoint_matching..zTEach element of target_sizes must contain the size (h, w) of each image of the batch)devicerr$r%) keypoints0 keypoints1matching_scores)matchesr*r>rJr,r7rGtorchtensorr{ keypointscloneflipreshapetoint32zipr~ logical_andrr)r[rwrxryimage_pair_sizesrresultskeypoints_pairrscores valid_matchesmatched_keypoints0matched_keypoints1r~r.r.r/post_process_keypoint_matchingTs6   z;EfficientLoFTRImageProcessor.post_process_keypoint_matchingr=keypoint_matching_outputz Image.ImagecstddDfddtdtdD}g}t||D]\}}|djdd\}}|djdd\} } tjt|| || dftjd } |d| d|d|f<|d| d| |df<t | } t | } |d  d\}}|d  d\}}t|||||d D]D\}}}}}||}| j|||||f|dd | j|d|d|d|dfdd| j||d|d||d|dfddq|| q!|S)a Plots the image pairs side by side with the detected keypoints as well as the matching between them. Args: images (`ImageInput`): Image pairs to plot. Same as `EfficientLoFTRImageProcessor.preprocess`. Expects either a list of 2 images or a list of list of 2 images list with pixel values ranging from 0 to 255. keypoint_matching_output (List[Dict[str, torch.Tensor]]]): A post processed keypoint matching output Returns: `List[PIL.Image.Image]`: A list of PIL images, each containing the image pairs side by side with the detected keypoints as well as the matching between them. cSrar.rbrBr.r.r/rIrczLEfficientLoFTRImageProcessor.visualize_keypoint_matching..crfrgr.rhr=r.r/rIrlrr$Nrr)dtyper|r}r~)fillrVblack)r)rLrsr>rr*r+zerosmaxuint8r fromarrayrDrawunbind _get_colorlineellipserr)r[r=rrtrrH pair_outputheight0width0height1width1 plot_imageplot_image_pildraw keypoints0_x keypoints0_y keypoints1_x keypoints1_y keypoint0_x keypoint0_y keypoint1_x keypoint1_ymatching_scorecolorr.rr/visualize_keypoint_matchings<    &" z8EfficientLoFTRImageProcessor.visualize_keypoint_matchingcCs*tdd|}td|}d}|||fS)zMaps a score to a color.rr)int)r[scorergbr.r.r/rs  z'EfficientLoFTRImageProcessor._get_color)NN)rv) __name__ __module__ __qualname____doc__model_input_namesrBILINEARboolrdictstrrfloatrZr+r8rr r r)rrrurGtuplerTensorrr rr __classcell__r.r.r]r/rMs   +    z ; 6rMrA)1rtypingrrnumpyr+rrimage_processing_utilsrrr image_transformsr r image_utilsr r rrrrrrrrrrutilsrrrrr:rrmodeling_efficientloftrr get_loggerrror8rr0r6rLrM__all__r.r.r.r/sD 8    ' H