o iWM@sdZddlZddlZddlmZmZddlZddlm Z ddl m Z m Z ddl mZmZmZmZddlmZmZmZmZmZmZmZdd lmZmZmZmZdd lm Z ereddl!Z!dd l"m#Z#m$Z$m%Z%erlddl&Z&e'e(Z)d Z*d dZ+         d'de,de-de,de,de-de-de-de-dee.dee,de#j#fddZ/ d(d ej0d!e,d"eee,e1ffd#d$Z2Gd%d&d&e Z3d&gZ4dS))z%Image processor class for Pix2Struct.N)OptionalUnion)hf_hub_download)BaseImageProcessor BatchFeature)convert_to_rgb normalizeto_channel_dimension_format to_pil_image)ChannelDimension ImageInputget_image_sizeinfer_channel_dimension_formatmake_flat_list_of_imagesto_numpy_array valid_images) TensorTypeis_torch_availableis_vision_availablelogging)requires_backends)Image ImageDraw ImageFontzybelkada/fontscCsttdg|d}tjjj|||f||fd}||d|d||d}| ddddd|d||d||d||}|dS) a Utility function to extract patches from a given image tensor. Returns a tensor of shape (1, `rows`, `columns`, `num_channels`x `patch_height` x `patch_width`). Args: image_tensor (torch.Tensor): The image tensor to extract patches from. patch_height (int): The height of the patches to extract. patch_width (int): The width of the patches to extract. torchr)strider) rtorch_extract_patches unsqueezernn functionalunfoldreshapesizepermute) image_tensor patch_height patch_widthpatchesr-n/home/ubuntu/.local/lib/python3.10/site-packages/transformers/models/pix2struct/image_processing_pix2struct.pyr!4s    r!$blackwhitetext text_size text_colorbackground_color left_padding right_padding top_paddingbottom_padding font_bytes font_pathreturnc Csttdtjdd} | j|d} d| } |dur$| dur$t|} n | dur+| } ntt d} t j | d|d } t td d |}|d | | \}}}}|||}|||}td ||f|}t |}|j||f| || d |S)a Render text. This script is entirely adapted from the original script that can be found here: https://github.com/google-research/pix2struct/blob/main/pix2struct/preprocessing/preprocessing_utils.py Args: text (`str`, *optional*, defaults to ): Text to render. text_size (`int`, *optional*, defaults to 36): Size of the text. text_color (`str`, *optional*, defaults to `"black"`): Color of the text. background_color (`str`, *optional*, defaults to `"white"`): Color of the background. left_padding (`int`, *optional*, defaults to 5): Padding on the left. right_padding (`int`, *optional*, defaults to 5): Padding on the right. top_padding (`int`, *optional*, defaults to 5): Padding on the top. bottom_padding (`int`, *optional*, defaults to 5): Padding on the bottom. font_bytes (`bytes`, *optional*): Bytes of the font to use. If `None`, the default font will be used. font_path (`str`, *optional*): Path to the font to use. If `None`, the default font will be used. visionP)width)r3 Nz Arial.TTFzUTF-8)encodingr'RGB)rrrr)xyr3fillfont)r render_texttextwrap TextWrapperwrapjoinioBytesIOrDEFAULT_FONT_PATHrtruetyperDrawrnewtextbboxr3)r3r4r5r6r7r8r9r:r;r<wrapperlines wrapped_textrG temp_draw_ text_width text_height image_width image_heightimagedrawr-r-r.rHOs$ &        rHr]headerinput_data_formatc Ksttdt||d}t|fi|}t|j|j}t|j||j}t|j||j}t d|||fd}| | ||fd| | ||fd|ft |}t |tjkrbt|tj}|S)a Renders the input text as a header on the input image. Args: image (`np.ndarray`): The image to render the header on. header (`str`): The header text. data_format (`Union[ChannelDimension, str]`, *optional*): The data format of the image. Can be either "ChannelDimension.channels_first" or "ChannelDimension.channels_last". Returns: `np.ndarray`: The image with the header rendered. r>)r`rCr1rDr)r render_headerr rHmaxr@intheightrrRpasteresizerrr LASTr ) r]r_r`kwargs header_image new_width new_heightnew_header_height new_imager-r-r.ras   racsBeZdZdZddgZ     dded ed eeee fd e d ed df fdd Z dde j d e d edee eefd e j f ddZ  dde j dee eefdee eefd e j fddZddddddejdfdedeedeed eed ee d eeee fdee eefdedee eefd efddZZS)Pix2StructImageProcessoraj Constructs a Pix2Struct image processor. Args: do_convert_rgb (`bool`, *optional*, defaults to `True`): Whether to convert the image to RGB. do_normalize (`bool`, *optional*, defaults to `True`): Whether to normalize the image. Can be overridden by the `do_normalize` parameter in the `preprocess` method. According to Pix2Struct paper and code, the image is normalized with its own mean and standard deviation. patch_size (`dict[str, int]`, *optional*, defaults to `{"height": 16, "width": 16}`): The patch size to use for the image. According to Pix2Struct paper and code, the patch size is 16x16. max_patches (`int`, *optional*, defaults to 2048): The maximum number of patches to extract from the image as per the [Pix2Struct paper](https://huggingface.co/papers/2210.03347). is_vqa (`bool`, *optional*, defaults to `False`): Whether or not the image processor is for the VQA task. If `True` and `header_text` is passed in, text is rendered onto the input images. flattened_patchesattention_maskTNFdo_convert_rgb do_normalize patch_size max_patchesis_vqar=c sFtjdi||dur|nddd|_||_||_||_||_dS)N)rdr@r-)super__init__rtrsrrrurv)selfrrrsrtrurvrh __class__r-r.rys   z!Pix2StructImageProcessor.__init__r]r`c Kst|jdt|tj|}t|}|d|d}}t|tj\}} t ||||| } t t t | |||d} t t t | | ||d} t | |d} t | |d}tj jj|d| |fdddd d}t|||}|j}|d}|d }|d }||||g}t||dgd|||dg}t|d|g|d||dg}|d7}|d7}|tj}|tj}t|||gd }tj j|ddd|||g}t|}|S) a Extract flattened patches from an image. Args: image (`np.ndarray`): Image to extract flattened patches from. max_patches (`int`): Maximum number of patches to extract. patch_size (`dict`): Dictionary containing the patch height and width. Returns: result (`np.ndarray`): A sequence of `max_patches` flattened patches. rrdr@rrbilinearFT)r'mode align_corners antialiasr rr)rextract_flattened_patchesr r FIRSTr from_numpyrmathsqrtrbminfloorr#r$ interpolater"squeezer!shaper&arangerepeattofloat32catpadfloatr)rzr]rurtr`rhr*r+r\r[scalenum_feasible_rowsnum_feasible_colsresized_height resized_widthr, patches_shaperowscolumnsdepthrow_idscol_idsresultr-r-r.rsF   **  $z2Pix2StructImageProcessor.extract_flattened_patches data_formatcKsb|jtjkr |tj}t|}t|}t|dt t |j }t |f||||d|S)a Normalize an image. image = (image - image_mean) / image_std. The image std is to mimic the tensorflow implementation of the `per_image_standardization`: https://www.tensorflow.org/api_docs/python/tf/image/per_image_standardization Args: image (`np.ndarray`): Image to normalize. data_format (`str` or `ChannelDimension`, *optional*): The channel dimension format for the output image. If unset, the channel dimension format of the input image is used. input_data_format (`str` or `ChannelDimension`, *optional*): The channel dimension format of the input image. If not provided, it will be inferred. g?)meanstdrr`) dtypenpuint8astyperrrrbrrprodrr )rzr]rr`rhrradjusted_stddevr-r-r.r 5s    z"Pix2StructImageProcessor.normalizeimages header_textreturn_tensorsc  sb|dur|nj}|dur|nj}durnjdur!njj} | ddur2tdt|}t|s>td|rGdd|D}dd|D}durXt |d| rdurbtd | d d| d dt t rzgt |fd dt|D}|rfd d|D}fdd|D}dd|D} t|| d|d} | S)a Preprocess an image or batch of images. The processor first computes the maximum possible number of aspect-ratio preserving patches of size `patch_size` that can be extracted from the image. It then pads the image with zeros to make the image respect the constraint of `max_patches`. Before extracting the patches the images are standardized following the tensorflow implementation of `per_image_standardization` (https://www.tensorflow.org/api_docs/python/tf/image/per_image_standardization). Args: images (`ImageInput`): Image to preprocess. Expects a single or batch of images. header_text (`Union[list[str], str]`, *optional*): Text to render as a header. Only has an effect if `image_processor.is_vqa` is `True`. do_convert_rgb (`bool`, *optional*, defaults to `self.do_convert_rgb`): Whether to convert the image to RGB. do_normalize (`bool`, *optional*, defaults to `self.do_normalize`): Whether to normalize the image. max_patches (`int`, *optional*, defaults to `self.max_patches`): Maximum number of patches to extract. patch_size (`dict`, *optional*, defaults to `self.patch_size`): Dictionary containing the patch height and width. return_tensors (`str` or `TensorType`, *optional*): The type of tensors to return. Can be one of: - Unset: Return a list of `np.ndarray`. - `TensorType.TENSORFLOW` or `'tf'`: Return a batch of type `tf.Tensor`. - `TensorType.PYTORCH` or `'pt'`: Return a batch of type `torch.Tensor`. - `TensorType.NUMPY` or `'np'`: Return a batch of type `np.ndarray`. - `TensorType.JAX` or `'jax'`: Return a batch of type `jax.numpy.ndarray`. data_format (`ChannelDimension` or `str`, *optional*, defaults to `ChannelDimension.FIRST`): The channel dimension format for the output image. Can be one of: - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format. - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format. - Unset: Use the channel dimension format of the input image. input_data_format (`ChannelDimension` or `str`, *optional*): The channel dimension format for the input image. If unset, the channel dimension format is inferred from the input image. Can be one of: - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format. - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format. - `"none"` or `ChannelDimension.NONE`: image in (height, width) format. Nrz8data_format is not an accepted input as the outputs are zkInvalid image type. Must be of type PIL.Image.Image, numpy.ndarray, torch.Tensor, tf.Tensor or jax.ndarray.cSg|]}t|qSr-)r.0r]r-r-r. z7Pix2StructImageProcessor.preprocess..cSrr-)rrr-r-r.rrrz.A header text must be provided for VQA models.r;r<cs$g|]\}}t||dqS))r;r<)ra)rir])r;r<rr-r.rscsg|] }j|dqS))r]r`)r r)r`rzr-r.rscsg|] }j|dqS))r]rurtr`)rr)r`rurtrzr-r.rs cSs$g|]}|jdddktjqS)r)axisr)sumrrrrr-r-r.rs$)rorp)data tensor_type)rsrrrtrurvget ValueErrorrrrpop isinstancestrlen enumerater)rzrrrrrsrurtrrr`rhrvattention_masksencoded_outputsr-)r;r<rr`rurtrzr. preprocess\sJ5     z#Pix2StructImageProcessor.preprocess)TTNrqFN)NN)__name__ __module__ __qualname____doc__model_input_namesboolrdictrrcryrndarrayrr rr rr rr __classcell__r-r-r{r.rns T *   rn) r/r0r1r2r2r2r2NNr)5rrMrtypingrrnumpyrhuggingface_hubrimage_processing_utilsrrimage_transformsrr r r image_utilsr r rrrrrutilsrrrrutils.import_utilsrrIPILrrrr get_loggerrloggerrOr!rrcbytesrHrChildProcessErrorrarn__all__r-r-r-r.s~ $     E *