o i{@sdZddlZddlmZmZddlZddlmZm Z m Z ddl m Z m Z mZddlmZmZmZmZmZmZmZmZmZmZddlmZmZmZmZmZm Z m!Z!erWddl"Z"e #e$Z%d ee&e&ee&eefd e&e&efd d Z'Gd dde Z(GdddeZ)dgZ*dS)zImage processor class for Fuyu.N)OptionalUnion)BaseImageProcessor BatchFeature get_size_dict)padresizeto_channel_dimension_format) ChannelDimension ImageInputPILImageResamplingget_image_sizeinfer_channel_dimension_formatis_scaled_imageis_valid_imagemake_list_of_imagesto_numpy_arrayvalidate_preprocess_arguments) TensorTypefilter_out_non_signature_kwargsis_torch_availableis_torch_deviceis_torch_dtypeloggingrequires_backendsimagesreturncCsPt|r|ggSt|trtdd|Dr|St|tr$dd|DStd)Ncss|]}t|tVqdSN) isinstancelist.0imager$b/home/ubuntu/.local/lib/python3.10/site-packages/transformers/models/fuyu/image_processing_fuyu.py @sz.make_list_of_list_of_images..cSg|]}t|qSr$)rr!r$r$r% Dz/make_list_of_list_of_images..zHimages must be a list of list of images or a list of images or an image.)rrr all ValueError)rr$r$r%make_list_of_list_of_images:s r,c@s6eZdZdZd deeeeffddZd dd Z dS) FuyuBatchFeaturez BatchFeature class for Fuyu image processor and processor. The outputs dictionary from the processors contains a mix of tensors and lists of tensors. N tensor_typecs|dur|S|j|d\fddfdd|D]3\}t|tr._convert_tensorcs*z|WSdkrtdtd)Noverflowing_valueszKUnable to create tensor returning overflowing values of different lengths. zUnable to create tensor, you should probably activate padding with 'padding=True' to have batched tensors with the same length.)r+r/)r3keyr$r%_safe_convert_tensorcs zAFuyuBatchFeature.convert_to_tensors.._safe_convert_tensorrcsg|] }fdd|DqS)cg|]}|qSr$r$r"r0r6r$r%r(rr)zBFuyuBatchFeature.convert_to_tensors...r$)r"elemsr9r$r%r(rsz7FuyuBatchFeature.convert_to_tensors..cr7r$r$r8r9r$r%r(ur))_get_is_as_tensor_fnsitemsrr )selfr.valuer$)r3r6r1r2r5r%convert_to_tensorsPs   z#FuyuBatchFeature.convert_to_tensorsrrc s"t|dgddli}ddur?tdkr?d}t|r$nt|ts2t|s2t|tr5|n t dt|dfdd| D]?\}}t|t rtt|dt rtg}|D]}| fd d |Dq`|||<qLt|t rfd d |D||<qL|||<qL||_ |S) a  Send all values to device by calling `v.to(*args, **kwargs)` (PyTorch only). This should support casting in different `dtypes` and sending the `BatchFeature` to a different `device`. Args: args (`Tuple`): Will be passed to the `to(...)` function of the tensors. kwargs (`Dict`, *optional*): Will be passed to the `to(...)` function of the tensors. Returns: [`BatchFeature`]: The same instance after modification. torchrNdevicez*Attempting to cast a BatchFeature to type z. This is not supported.cs2|r |jiSdur|jdS|S)N)rA)is_floating_pointtor/)argsrAkwargsr@r$r%_tos  z FuyuBatchFeature.to.._tocr7r$r$r8rFr$r%r(r)z'FuyuBatchFeature.to..cr7r$r$r8rGr$r%r(r))rr@getlenrrstrrintr+r<r appenddata) r=rDrEnew_dataargkvnew_vr:r$)rFrDrArEr@r%rC{s.     zFuyuBatchFeature.tor)rr) __name__ __module__ __qualname____doc__rrrJrr?rCr$r$r$r%r-Is+r-c!seZdZdZgdZddejdddddddddf d ed ee e e fd ed ed e de dede e ee fde e ee fdede dee e e fffdd Zejddfdejd e e e fd edee e efdee e efdejf ddZ    d4dejd e e e fde de dee e efdee e efdejfdd Zeddddddddddddejddfd eed ee e e fd eed eed ee dee deedee dee deedee dee e e fdee e efdee e efd!eefd"d#Zd5d$e d%e dee e e fde fd&d'Zd5dd(dee e e fdd(fd)d*Z d5d+d(d,d(d-d(d.d(d/e d0e d1edee e e fdefd2d3ZZS)6FuyuImageProcessora This class should handle the image processing part before the main FuyuForCausalLM. In particular, it should handle: - Processing Images: Taking a batch of images as input. If the images are variable-sized, it resizes them based on the desired patch dimensions. The image output is always img_h, img_w of (1080, 1920) Then, it patches up these images using the patchify_image function. - Creating Image Input IDs: For each patch, a placeholder ID is given to identify where these patches belong in a token sequence. For variable-sized images, each line of patches is terminated with a newline ID. - Image Patch Indices: For each image patch, the code maintains an index where these patches should be inserted in a token stream. Args: do_resize (`bool`, *optional*, defaults to `True`): Whether to resize the image to `size`. size (`dict[str, int]`, *optional*, defaults to `{"height": 1080, "width": 1920}`): Dictionary in the format `{"height": int, "width": int}` specifying the size of the output image. resample (`PILImageResampling`, *optional*, defaults to `Resampling.BILINEAR`): `PILImageResampling` filter to use when resizing the image e.g. `PILImageResampling.BILINEAR`. do_pad (`bool`, *optional*, defaults to `True`): Whether to pad the image to `size`. padding_value (`float`, *optional*, defaults to 1.0): The value to pad the image with. padding_mode (`str`, *optional*, defaults to `"constant"`): The padding mode to use when padding the image. do_normalize (`bool`, *optional*, defaults to `True`): Whether to normalize the image. image_mean (`float`, *optional*, defaults to 0.5): The mean to use when normalizing the image. image_std (`float`, *optional*, defaults to 0.5): The standard deviation to use when normalizing the image. do_rescale (`bool`, *optional*, defaults to `True`): Whether to rescale the image. rescale_factor (`float`, *optional*, defaults to `1 / 255`): The factor to use when rescaling the image. patch_size (`dict[str, int]`, *optional*, defaults to `{"height": 30, "width": 30}`): Dictionary in the format `{"height": int, "width": int}` specifying the size of the patches. rimage_input_ids image_patchesimage_patch_indices_per_batch#image_patch_indices_per_subsequenceTN?constantg?gp? do_resizesizeresampledo_pad padding_value padding_mode do_normalize image_mean image_std do_rescalerescale_factor patch_sizec  stjdi| ||_|dur|nddd|_||_||_||_||_||_||_ | |_ | |_ | |_ | dur<| |_ dSddd|_ dS)Ni8i)heightwidthr$)super__init__r_r`rarbrcrdrerfrgrhrirj)r=r_r`rarbrcrdrerfrgrhrirjrE __class__r$r%ros"zFuyuImageProcessor.__init__r# data_formatinput_data_formatrcKst||\}}|d|d} } || kr|| kr|S| |} | |} t| | } t|| }t|| }td|||f|||d|}|S)a Resize an image to `(size["height"], size["width"])`. Args: image (`np.ndarray`): Image to resize. size (`dict[str, int]`): Dictionary in the format `{"height": int, "width": int}` specifying the size of the output image. resample (`PILImageResampling`, *optional*, defaults to `PILImageResampling.BILINEAR`): `PILImageResampling` filter to use when resizing the image e.g. `PILImageResampling.BILINEAR`. data_format (`ChannelDimension` or `str`, *optional*): The channel dimension format for the output image. If unset, the channel dimension format of the input image is used. Can be one of: - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format. - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format. - `"none"` or `ChannelDimension.NONE`: image in (height, width) format. input_data_format (`ChannelDimension` or `str`, *optional*): The channel dimension format for the input image. If unset, the channel dimension format is inferred from the input image. Can be one of: - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format. - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format. - `"none"` or `ChannelDimension.NONE`: image in (height, width) format. Returns: `np.ndarray`: The resized image. rkrl)r#r`rarrrsNr$)rminrKr )r=r#r`rarrrsrE image_height image_width target_height target_widthheight_scale_factorwidth_scale_factoroptimal_scale_factor new_height new_width scaled_imager$r$r%r s&#   zFuyuImageProcessor.resizemodeconstant_valuescCs\t||\}}|d|d} } d} d} | |} | |}t|| | f| |ff||||d}|S)a Pad an image to `(size["height"], size["width"])`. Args: image (`np.ndarray`): Image to pad. size (`dict[str, int]`): Dictionary in the format `{"height": int, "width": int}` specifying the size of the output image. data_format (`ChannelDimension` or `str`, *optional*): The data format of the output image. If unset, the same format as the input image is used. input_data_format (`ChannelDimension` or `str`, *optional*): The channel dimension format of the input image. If not provided, it will be inferred. rkrlr)paddingrrrrrs)rr)r=r#r`rrrrrsrurvrwrx padding_top padding_leftpadding_bottom padding_right padded_imager$r$r% pad_imageDszFuyuImageProcessor.pad_imagereturn_tensorsc s|dur|nj}durnj|dur|nj}|dur!|nj}| dur*| nj} dur3nj|dur<|nj}durEnjdurNnjdurWnj dur`nj | duri| nj} durrnj| dur{| nj } t |t rtdd|Drtdt|}t| |||ddd|D}dd|Dd d }| rt|rtd durt|fd d|D}t|r߇fd d|D}fd d|D}dd|D}dd|D}ddt||D}|rfdd|D}| rfdd|D}|r-fdd|D}dur<fdd|D}||||d}t||dS)a Utility function to preprocess the images and extract necessary information about original formats. Args: images (`ImageInput`): Images to preprocess. Expects a single image, a list or images or a list of lists of images. Pixel values range from 0 to 255, or between 0 and 1 if `do_rescale` is `False`. do_resize (`bool`, *optional*, defaults to `self.do_resize`): Whether to resize the image to `size`. size (`dict[str, int]`, *optional*, defaults to `self.size`): Dictionary in the format `{"height": int, "width": int}` specifying the size of the output image. resample (`PILImageResampling`, *optional*, defaults to `self.resample`): `PILImageResampling` filter to use when resizing the image e.g. `PILImageResampling.BILINEAR`. do_pad (`bool`, *optional*, defaults to `self.do_pad`): Whether to pad the image to `size`. padding_value (`float`, *optional*, defaults to `self.padding_value`): The value to pad the image with. padding_mode (`str`, *optional*, defaults to `self.padding_mode`): The padding mode to use when padding the image. do_normalize (`bool`, *optional*, defaults to `self.do_normalize`): Whether to normalize the image. image_mean (`float`, *optional*, defaults to `self.image_mean`): The mean to use when normalizing the image. image_std (`float`, *optional*, defaults to `self.image_std`): The standard deviation to use when normalizing the image. do_rescale (`bool`, *optional*, defaults to `self.do_rescale`): Whether to rescale the image. rescale_factor (`float`, *optional*, defaults to `self.rescale_factor`): The factor to use when rescaling the image. patch_size (`dict[str, int]`, *optional*, defaults to `self.patch_size`): Dictionary in the format `{"height": int, "width": int}` specifying the size of the patches. return_tensors (`str` or `TensorType`, *optional*): The type of tensors to return. Can be one of: - Unset: Return a list of `np.ndarray`. - `TensorType.TENSORFLOW` or `'tf'`: Return a batch of type `tf.Tensor`. - `TensorType.PYTORCH` or `'pt'`: Return a batch of type `torch.Tensor`. - `TensorType.NUMPY` or `'np'`: Return a batch of type `np.ndarray`. - `TensorType.JAX` or `'jax'`: Return a batch of type `jax.numpy.ndarray`. data_format (`ChannelDimension` or `str`, *optional*, defaults to `ChannelDimension.FIRST`): The channel dimension format of the output image. Can be one of: - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format. - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format. input_data_format (`ChannelDimension` or `str`, *optional*): The channel dimension format for the input image. If unset, the channel dimension format is inferred from the input image. Can be one of: - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format. - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format. Ncss&|]}t|tot|dkVqdS)N)rr rIr8r$r$r%r&s$z0FuyuImageProcessor.preprocess..z:Multiple images for a single sample are not yet supported.)rhrirerfrgr_r`racSsg|] }dd|DqS)cSr'r$)rr!r$r$r%r(r)..r$r"rr$r$r%r(sz1FuyuImageProcessor.preprocess..cSsg|]}|r|qSr$r$rr$r$r%r(r)rzIt looks like you are trying to rescale already rescaled images. If the input images have pixel values between 0 and 1, set `do_rescale=False` to avoid rescaling them again.c g|] }|rt|ddqSr) channel_dimrrrsr$r%r(s c"g|] }fdd|DqS)csg|] }j|dqS))r`rs)r r!rsr=r`r$r%r(rr$rrr$r%r(crrrrrr$r%r(s cSg|]}|dgqSrr$r" image_sizer$r$r%r(cSr)r$rr$r$r%r(rcSs"g|] \}}|d|dgqSrr$)r" original_size resized_sizer$r$r%r(scs&g|]}fdd|DqS)c s g|] }j|dqS))r`rrrs)rr!rsrdrcr=r`r$r%r(srr$rrr$r%r(s  cr)csg|] }j|dqS))scalers)rescaler!rsrir=r$r%r(rrr$rrr$r%r(rcs$g|]}fdd|DqS)csg|] }j|dqS))meanstdrs) normalizer!rfrgrsr=r$r%r( srr$rrr$r%r( s cs g|] }fdd|DqS)csg|]}t|qSr$)r r!rrrsr$r%r(srr$rrr$r%r(s)rimage_unpadded_heightsimage_unpadded_widthsimage_scale_factors)rMr.)r_r`rarbrhrirerfrgrcrdrjrr anyr+r,rrlogger warning_oncerrzipr-)r=rr_r`rarbrcrdrerfrgrhrirjrrrsr batch_imagesfirst_image_in_listoriginal_image_sizes image_sizesrrrrMr$) rrrfrgrsrdrcrir=r`r% preprocessjsF       zFuyuImageProcessor.preprocessrurvc Cs|dur|n|j}|jd|jd}}||dkr$td|d|||dkr4td|d|||}||}||}|S)a Calculate number of patches required to encode an image. Args: image_height (`int`): Height of the image. image_width (`int`): Width of the image. patch_size (`dict[str, int]`, *optional*, defaults to `self.patch_size`): Dictionary in the format `{"height": int, "width": int}` specifying the size of the patches. Nrkrlrz image_height=z must be divisible by z image_width=)rjr+) r=rurvrj patch_height patch_widthnum_patches_per_dim_hnum_patches_per_dim_w num_patchesr$r$r%get_num_patches s   z"FuyuImageProcessor.get_num_patchesz torch.Tensorc Cst|dg|dur |n|j}|d|d}}|j\}}}}|d||}|d||} | } | ||d||} | dddd d } | |d|||} | S) a| Convert an image into a tensor of patches. Args: image (`torch.Tensor`): Image to convert. Shape: [batch, channels, height, width] patch_size (`dict[str, int]`, *optional*, defaults to `self.patch_size`): Dictionary in the format `{"height": int, "width": int}` specifying the size of the patches. r@Nrkrlrrrr)rrjshapeunfold contiguousviewpermutereshape) r=r#rjrr batch_sizechannels_unfolded_along_heightpatchesr$r$r%patchify_image9s z!FuyuImageProcessor.patchify_image image_input image_presentimage_unpadded_himage_unpadded_wimage_placeholder_idimage_newline_idvariable_sizedc ' Cst|dg|dur |n|j}|d|d} } g} g} g} t|jdD]}g}g}t|jdD]}|||fr|||f}|jd|jd}}|rt|t|||f| | }t|t|||f| | }|ddd|d|f}||}}|j||d}tj |g|tj |j d }|j | dd d}||jdksJ|r|d || }tj |jddg|tj |j d }tj||gdd }|d }| |g||||q2|tjgtj |j d q2| || |q%g}g}| D][}d}g}g} |D]F}!|!|k}"t|"}tj|tj|!j d |!}#t|!d }$t|!d }%tj|"d dd}&|#||$|&<|#|%|&<||$| |%||7}q |||| qt| | | ||ddS)aProcess images for model input. In particular, variable-sized images are handled here. Args: image_input (`torch.Tensor` of shape [batch_size, subsequence_size, num_channels, height, width]): Tensor of images padded to model input size. image_present (`torch.Tensor` of shape [batch_size, subsequence_size, num_images]): Tensor of 1s and 0s indicating whether an image is present. image_unpadded_h (`torch.Tensor` of shape [batch_size, subsequence_size]): Tensor of unpadded image heights. image_unpadded_w (`torch.Tensor` of shape [batch_size, subsequence_size]): Tensor of unpadded image widths. image_placeholder_id (int): The id of the image placeholder token. Comes from an associated tokenizer. image_newline_id (int): The id of the image newline token. Comes from an associated tokenizer. variable_sized (bool): Whether to process images as variable-sized. patch_size (`dict[str, int]`, *optional*, defaults to `self.patch_size`): Size of the patches. r@Nrkrlrrr)rurv)dtyperA)r#r)dimT)as_tuplerX)rM)rrjrangerrtmathceilrr@fullint32rAr unsqueezesqueezercatrLtensor count_nonzeroarangeint64type_as full_likenonzeror-)'r=rrrrrrrrjrrrbatch_image_patchesbatch_image_input_ids batch_indexrYrZ subseq_indexr#rurvnew_hnew_wrtensor_of_image_idsr newline_idsr[r\sample_image_input_ids index_offsetper_batch_indicesper_subsequence_indicessubseq_image_input_ids patches_maskindicesindices_in_stream_per_batch!indices_in_stream_per_subsequence patches_indsr$r$r%preprocess_with_tokenizer_infoSs                    z1FuyuImageProcessor.preprocess_with_tokenizer_info)r^r]NNr)rSrTrUrVmodel_input_namesr BILINEARboolrdictrJrKfloatrr ronpndarrayr r rrFIRSTrrrrr-r __classcell__r$r$rpr%rWs"-      "  >  &      (6$#  rW)+rVrtypingrrnumpyrimage_processing_utilsrrrimage_transformsrr r image_utilsr r r rrrrrrrutilsrrrrrrrr@ get_loggerrSrr r,r-rW__all__r$r$r$r%s.0$     m &