o i^@sdZddlZddlmZmZddlZddlmZm Z ddl m Z m Z m Z ddlmZmZmZmZmZmZmZmZmZmZmZmZddlmZmZdd lmZe e!Z" dde#de#de#de#de#de#de#fddZ$GdddeZ%dgZ&dS)z#Image processor class for GLM-4.1V.N)OptionalUnion)BaseImageProcessor BatchFeature)convert_to_rgbresizeto_channel_dimension_format) OPENAI_CLIP_MEANOPENAI_CLIP_STDChannelDimension ImageInputPILImageResamplingget_image_sizeinfer_channel_dimension_formatis_scaled_imagemake_flat_list_of_imagesto_numpy_array valid_imagesvalidate_preprocess_arguments) TensorTypelogging) VideoInput1 num_framesheightwidthtemporal_factorfactor min_pixels max_pixelsc Csl||krtd|d|||ks||kr#td|d|d|t||t||dkr>tdt||t||t|||}t|||}t|||} | |||krt||||} t|t|| ||}t|t|| ||}||fS| |||krt||||} t|| ||}t|| ||}||fS)Nzt:z% must be larger than temporal_factor:zheight:z or width:z must be larger than factor:z4absolute aspect ratio must be smaller than 200, got ) ValueErrormaxminroundmathsqrtfloorceil) rrrr r!r"r#h_barw_bart_barbetar1d/home/ubuntu/.local/lib/python3.10/site-packages/transformers/models/glm4v/image_processing_glm4v.py smart_resize1s* r3c&sreZdZdZddgZddejdddddddddf d ed ee e e fd ed ed e e e fdedee e ee fdee e ee fdede de de ddffdd Zddddddddddddejdfde eefd eed ee e e fd eed eed ee deedee e ee fdee e ee fdee dee dee deedeedee e effddZddddddddddddddejdfdedeed eed ee e e fd eed eed ee deedee e ee fdee e ee fdee dee dee deedee e efdeedee e eff"dd Zd%d!e d"e fd#d$ZZS)&Glm4vImageProcessora Constructs a GLM-4V image processor that dynamically resizes images based on the original images. Args: do_resize (`bool`, *optional*, defaults to `True`): Whether to resize the image's (height, width) dimensions. size (`Dict[str, int]` *optional*, defaults to `{"shortest_edge": 112 * 112, "longest_edge": 28 * 28 * 15000}`): Size of the image's `(height, width)` dimensions after resizing. Can be overridden by the `size` parameter in the `preprocess` method. Available options are: - `{"height": int, "width": int}`: The image will be resized to the exact size `(height, width)`. Do NOT keep the aspect ratio. - `{"shortest_edge": int, "longest_edge": int}`: The image will be resized to a maximum size respecting the aspect ratio and keeping the shortest edge less or equal to `shortest_edge` and the longest edge less or equal to `longest_edge`. - `{"max_height": int, "max_width": int}`: The image will be resized to the maximum size respecting the aspect ratio and keeping the height less or equal to `max_height` and the width less or equal to `max_width`. resample (`PILImageResampling`, *optional*, defaults to `Resampling.BICUBIC`): Resampling filter to use when resizing the image. do_rescale (`bool`, *optional*, defaults to `True`): Whether to rescale the image by the specified scale `rescale_factor`. rescale_factor (`int` or `float`, *optional*, defaults to `1/255`): Scale factor to use if rescaling the image. do_normalize (`bool`, *optional*, defaults to `True`): Whether to normalize the image. image_mean (`float` or `List[float]`, *optional*, defaults to `[0.48145466, 0.4578275, 0.40821073]`): Mean to use if normalizing the image. This is a float or list of floats for each channel in the image. image_std (`float` or `List[float]`, *optional*, defaults to `[0.26862954, 0.26130258, 0.27577711]`): Standard deviation to use if normalizing the image. This is a float or list of floats for each channel in the image. do_convert_rgb (`bool`, *optional*, defaults to `True`): Whether to convert the image to RGB. patch_size (`int`, *optional*, defaults to 14): The spatial patch size of the vision encoder. temporal_patch_size (`int`, *optional*, defaults to 2): The temporal patch size of the vision encoder. merge_size (`int`, *optional*, defaults to 2): The merge size of the vision encoder to llm encoder. pixel_valuesimage_grid_thwTNgp?r do_resizesizeresample do_rescalerescale_factor do_normalize image_mean image_stddo_convert_rgb patch_sizetemporal_patch_size merge_sizereturnc  stjdi| |durd|vsd|vrtd|dur"ddd}||_||_||_||_||_||_|dur:|nt |_ |durC|nt |_ | |_ | |_| |_| |_dS)N shortest_edge longest_edge:size must contain 'shortest_edge' and 'longest_edge' keys.rqrErFr1)super__init__r%r9r8r:r;r<r=r r>r r?rArBrCr@)selfr8r9r:r;r<r=r>r?r@rArBrCkwargs __class__r1r2rK|s"  zGlm4vImageProcessor.__init__images data_formatinput_data_formatc Cst|}| r dd|D}dd|D}|r!t|dr!td|dur+t|d}t|d|d\}}||}}g}|D]B}|r_t| ||| | | |d|d d \}}t|||f||d }|ri|j|||d }|rt|j ||| |d }t |||d}| |q>t |}|tjkr|dddd}|jd| dkrt j|dt j| |jd| dd}t j||gdd}|jd}|jd| }|| || }}||| ||| | | || | | }|ddddddddd }|||||| | | }||||ffS)a~ Preprocess an image or batch of images. Copy of the `preprocess` method from `CLIPImageProcessor`. Args: images (`ImageInput`): Image or batch of images to preprocess. Expects pixel values ranging from 0 to 255. If pixel values range from 0 to 1, set `do_rescale=False`. vision_info (`List[Dict]`, *optional*): Optional list of dictionaries containing additional information about vision inputs. do_resize (`bool`, *optional*, defaults to `self.do_resize`): Whether to resize the image. size (`Dict[str, int]`, *optional*, defaults to `self.size`): Size of the image after resizing. `shortest_edge` and `longest_edge` keys must be present. resample (`PILImageResampling`, *optional*, defaults to `self.resample`): Resampling filter to use if resizing the image. This can be one of the `PILImageResampling` enums. do_rescale (`bool`, *optional*, defaults to `self.do_rescale`): Whether to rescale the image. rescale_factor (`float`, *optional*, defaults to `self.rescale_factor`): Scale factor to use if rescaling the image. do_normalize (`bool`, *optional*, defaults to `self.do_normalize`): Whether to normalize the image. image_mean (`float` or `List[float]`, *optional*, defaults to `self.image_mean`): Mean to use if normalizing the image. Can be a float or a list of floats corresponding to the number of channels in the image. image_std (`float` or `List[float]`, *optional*, defaults to `self.image_std`): Standard deviation to use if normalizing the image. Can be a float or a list of floats corresponding to the number of channels in the image. patch_size (`int`, *optional*, defaults to `self.patch_size`): The spatial patch size of the vision encoder. temporal_patch_size (`int`, *optional*, defaults to `self.temporal_patch_size`): The temporal patch size of the vision encoder. merge_size (`int`, *optional*, defaults to `self.merge_size`): The merge size of the vision encoder to llm encoder. do_convert_rgb (`bool`, *optional*, defaults to `self.do_convert_rgb`): Whether to convert the image to RGB. data_format (`ChannelDimension`, *optional*, defaults to `ChannelDimension.FIRST`): The channel dimension format for the output image. Can be one of: - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format. - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format. - Unset: Use the channel dimension format of the input image. input_data_format (`ChannelDimension` or `str`, *optional*): The channel dimension format for the input image. Can be one of: - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format. - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format. - `"none"` or `ChannelDimension.NONE`: image in (height, width) format. - `"none"` or `ChannelDimension.NONE`: image in (height, width) format. cSg|]}t|qSr1)r.0imager1r1r2 z3Glm4vImageProcessor._preprocess..cSrSr1)rrTr1r1r2rWrXrzIt looks like you are trying to rescale already rescaled images. If the input images have pixel values between 0 and 1, set `do_rescale=False` to avoid rescaling them again.N) channel_dimrErF)rrrr r!r"r#)r9r:rR)scalerR)rVmeanstdrR)input_channel_dimrr)axis)rrlogger warning_oncerrr3rrescale normalizer appendnparrayr LAST transposeshaperepeatnewaxis concatenatereshape)rLrPr8r9r:r;r<r=r>r?rArBrCr@rQrRrrresized_height resized_widthprocessed_imagesrVpatchesrepeatschannelgrid_tgrid_hgrid_wflatten_patchesr1r1r2 _preprocesss|=         zGlm4vImageProcessor._preprocessvideosreturn_tensorscCs|dur|n|j}|durd|vsd|vrtd|dur"ddd}|dur(|n|j}|dur1|n|j}|dur:|n|j}|durC|n|j}|durL|n|j}| durU| n|j} | dur^| n|j} | durg| n|j } | durp| n|j } | dury| n|j } |dur|n|j }|dur| |}t|}|durt|stdt||| | |||d i}|durgg}}|D]"}|j|||||||| | | | | |||d \}}||||qt|}t|}|||d t||d S) a Args: images (`ImageInput`): Image to preprocess. Expects a single or batch of images with pixel values ranging from 0 to 255. If passing in images with pixel values between 0 and 1, set `do_rescale=False`. videos (`VideoInput`): Video to preprocess. Expects a single or batch of videos with pixel values ranging from 0 to 255. If passing in videos with pixel values between 0 and 1, set `do_rescale=False`. do_resize (`bool`, *optional*, defaults to `self.do_resize`): Whether to resize the image. size (`Dict[str, int]`, *optional*, defaults to `self.size`): Size of the image after resizing. Shortest edge of the image is resized to size["shortest_edge"], with the longest edge resized to keep the input aspect ratio. resample (`int`, *optional*, defaults to `self.resample`): Resampling filter to use if resizing the image. This can be one of the enum `PILImageResampling`. Only has an effect if `do_resize` is set to `True`. do_rescale (`bool`, *optional*, defaults to `self.do_rescale`): Whether to rescale the image. rescale_factor (`float`, *optional*, defaults to `self.rescale_factor`): Rescale factor to rescale the image by if `do_rescale` is set to `True`. do_normalize (`bool`, *optional*, defaults to `self.do_normalize`): Whether to normalize the image. image_mean (`float` or `List[float]`, *optional*, defaults to `self.image_mean`): Image mean to use for normalization. Only has an effect if `do_normalize` is set to `True`. image_std (`float` or `List[float]`, *optional*, defaults to `self.image_std`): Image standard deviation to use for normalization. Only has an effect if `do_normalize` is set to `True`. The max pixels of the image to resize the image. patch_size (`int`, *optional*, defaults to `self.patch_size`): The spatial patch size of the vision encoder. temporal_patch_size (`int`, *optional*, defaults to `self.temporal_patch_size`): The temporal patch size of the vision encoder. merge_size (`int`, *optional*, defaults to `self.merge_size`): The merge size of the vision encoder to llm encoder. do_convert_rgb (`bool`, *optional*, defaults to `self.do_convert_rgb`): Whether to convert the image to RGB. return_tensors (`str` or `TensorType`, *optional*): The type of tensors to return. Can be one of: - Unset: Return a list of `np.ndarray`. - `TensorType.TENSORFLOW` or `'tf'`: Return a batch of type `tf.Tensor`. - `TensorType.PYTORCH` or `'pt'`: Return a batch of type `torch.Tensor`. - `TensorType.NUMPY` or `'np'`: Return a batch of type `np.ndarray`. - `TensorType.JAX` or `'jax'`: Return a batch of type `jax.numpy.ndarray`. data_format (`ChannelDimension` or `str`, *optional*, defaults to `ChannelDimension.FIRST`): The channel dimension format for the output image. Can be one of: - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format. - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format. - Unset: Use the channel dimension format of the input image. input_data_format (`ChannelDimension` or `str`, *optional*): The channel dimension format for the input image. If unset, the channel dimension format is inferred from the input image. Can be one of: - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format. - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format. - `"none"` or `ChannelDimension.NONE`: image in (height, width) format. NrErFrGrrHrIzkInvalid image type. Must be of type PIL.Image.Image, numpy.ndarray, torch.Tensor, tf.Tensor or jax.ndarray.)r<r=r>r?r8r9r:)r8r9r:r;r<r=r>r?rArBrCrQr@rR)r5r6)data tensor_type)r9r%r8r:r;r<r=r>r?rArBrCr@ fetch_imagesrrrr~extendrjrkrlupdater)rLrPrr8r9r:r;r<r=r>r?rArBrCr@rrQrRrr5vision_grid_thwsrVrwr6r1r1r2 preprocess(svM         zGlm4vImageProcessor.preprocessrrc Csv|d|j}|d|j}|dddd}||}t|j||||d|d|jd \}} ||| |} } | | S) a A utility that returns number of image patches for a given image size. Args: height (`int`): Height of the input image. width (`int`): Width of the input image. images_kwargs (`dict`, *optional*) Any kwargs to override defaults of the image processor. Returns: `int`: Number of image patches per image. rArCr9rrHrIrErF)rrrr!r"r#r )getrArCr3rB) rLrr images_kwargsrArCr9r!rtrur{r|r1r1r2get_number_of_image_patchess  z/Glm4vImageProcessor.get_number_of_image_patches)N)__name__ __module__ __qualname____doc__model_input_namesrBICUBICboolrdictstrintrfloatlistrKr FIRSTr rr~rrr __classcell__r1r1rNr2r4Rs'      '              r4)rrrr)'rr)typingrrnumpyrkimage_processing_utilsrrimage_transformsrrr image_utilsr r r r rrrrrrrrutilsrr video_utilsr get_loggerrrfrr3r4__all__r1r1r1r2sD8   !