o wiXB@s&dZddlmZmZddlZddlmZddlm Z m Z m Z ddl m Z mZmZmZmZmZmZddlmZmZmZdd lmZeeZd Zd d ed Ddd edDZGdddeZ Gddde Z!GdddeddZ"de#fddZ$ddZ%ddZ&dd Z'Gd!d"d"eZ(d"gZ)dS)#z Processor class for PaliGemma. )OptionalUnionN) BatchFeature) ImageInputis_valid_imagemake_flat_list_of_images) ImagesKwargsMultiModalDataProcessingKwargsProcessorMixin TextKwargsUnpack!_validate_images_text_input_order) AddedTokenPreTokenizedInput TextInput)loggingzcCg|] }d|ddqS)z4>.0irro/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/transformers/models/paligemma/processing_paligemma.py )ricCr)z3rrrrrrr)rc@s.eZdZUeeeeeeeefed<dS)PaliGemmaTextKwargssuffixN) __name__ __module__ __qualname__rrrrlist__annotations__rrrrr,s $rc@seZdZUeeed<dS)PaliGemmaImagesKwargsdo_convert_rgbN)r r!r"rboolr$rrrrr%0s r%c@s2eZdZUeed<eed<dddddidZdS) PaliGemmaProcessorKwargs text_kwargs images_kwargsF)paddingreturn_mm_token_type_ids data_formatchannels_first)r)r*N)r r!r"rr$r% _defaultsrrrrr(4s  r(F)totalreturncCst|to |dS)Nhttp) isinstancestr startswith)valrrris_urlCsr7cCst|pt|SN)r7relemrrris_image_or_image_urlHsr;cCst|tpt|Sr8)r3r4r;r9rrr_is_str_or_imageLsr<cCs|||||dS)aZ Builds a string from the input prompt and image tokens. For example, for the call: build_string_from_input( prompt="Prefix str" bos_token="", image_seq_len=3, image_token="", ) The output will be: "Initial str" Args: prompt (`list[Union[str, ImageInput]]`): The input prompt. bos_token (`str`): The beginning of sentence token. image_seq_len (`int`): The length of the image sequence. image_token (`str`): The image token. num_images (`int`): Number of images in the prompt.  rprompt bos_token image_seq_len image_token num_imagesrrrbuild_string_from_inputPsrDc seZdZdZddgZdZdZ   dfdd Z    dd ed e e e e e e e fd e ed efd dZdddZddZddZeddZZS)PaliGemmaProcessora Constructs a PaliGemma processor which wraps a PaliGemma image processor and a PaliGemma tokenizer into a single processor. [`PaliGemmaProcessor`] offers all the functionalities of [`SiglipImageProcessor`] and [`GemmaTokenizerFast`]. See the [`~PaliGemmaProcessor.__call__`] and [`~PaliGemmaProcessor.decode`] for more information. Args: image_processor ([`SiglipImageProcessor`], *optional*): The image processor is a required input. tokenizer ([`GemmaTokenizerFast`], *optional*): The tokenizer is a required input. chat_template (`str`, *optional*): A Jinja template which will be used to convert lists of messages in a chat into a tokenizable string. image_processor tokenizer)SiglipImageProcessorSiglipImageProcessorFast)GemmaTokenizerGemmaTokenizerFastNc s|durtd|durtdt|dstd|j|_t|ds=ttddd}d |gi}|||t|_t|_n|j|_|j|_| t d|_ d|_ t j|||d dS) Nz)You need to specify an `image_processor`.z"You need to specify a `tokenizer`.image_seq_lengthz;Image processor is missing an `image_seq_length` attribute.rBFT) normalizedspecialadditional_special_tokens) chat_template) ValueErrorhasattrrLr IMAGE_TOKENadd_special_tokensconvert_tokens_to_idsimage_token_idrB add_tokens EXTRA_TOKENS add_bos_token add_eos_tokensuper__init__)selfrFrGrPkwargsrB tokens_to_add __class__rrr\zs&      zPaliGemmaProcessor.__init__imagestextr^r1c sFt||\}}jtfdjji|}|ddd}|dur"dnd}|dur,td|dur7tdd }t |r?|g}n t |t rKt |d rK |dur |dur t d d |Dst d t |t rt |t rt|t|krtdt|dt|dt|r|gg}n1t |t tfrt|d rdd|D}nt |t tfrt |d t tfrt|d d stdfddt||D} t|}n=g} |D]1} | ttj} | t} | dkr| ttnd }| d|jj| |d} | | qdd| D} |durt |r|g}|dur'fdd|D}j|fi|dd}|ddd}|ddd}j| f||d|d}j| |dgdi|d|i}|rt|d}d |t|d!d k<|d"|i|rt|d}t|d}d#||jk<| |d$<t!||d%S)&ah Main method to prepare for the model one or several sequences(s) and image(s). This method forwards the `text` and `kwargs` arguments to GemmaTokenizerFast's [`~GemmaTokenizerFast.__call__`] if `text` is not `None` to encode the text. To prepare the image(s), this method forwards the `images` and `kwrags` arguments to SiglipImageProcessor's [`~SiglipImageProcessor.__call__`] if `images` is not `None`. Please refer to the docstring of the above two methods for more information. The usage for PaliGemma fine-tuning preparation is slightly different than usual. suffix passed are suffixes to the prompt in `text`, and will be placed after the prompt. This is because attention is handled differently for the prefix and the suffix. For instance, ```python image = PIL_cow_image prompt = "answer en Where is the cow standing?" suffix = "on the beach" inputs = processor(text=prompt, images=image, suffix=suffix) ``` Here `inputs` will contain the `input_ids` and `token_type_ids` that follow ```python inputs["input_ids"][:, 256:] # tensor([[ 2, 6006, 603, 573, 13910, 9980, 235336, 108, 477, 573, 8318]]) inputs["token_type_ids"][:, 256:] tensor([[0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1]]) ``` Meaning the last three tokens are of "label" ("suffix") type while the other ones are of "prefix" type. Args: images (`PIL.Image.Image`, `np.ndarray`, `torch.Tensor`, `list[PIL.Image.Image]`, `list[np.ndarray]`, `list[torch.Tensor]`): The image or batch of images to be prepared. Each image can be a PIL image, NumPy array or PyTorch tensor. In case of a NumPy array/PyTorch tensor, each image should be of shape (C, H, W), where C is a number of channels, H and W are image height and width. text (`str`, `list[str]`, `list[list[str]]`): The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set `is_split_into_words=True` (to lift the ambiguity with a batch of sequences). return_tensors (`str` or [`~utils.TensorType`], *optional*): If set, will return tensors of a particular framework. Acceptable values are: - `'tf'`: Return TensorFlow `tf.constant` objects. - `'pt'`: Return PyTorch `torch.Tensor` objects. - `'np'`: Return NumPy `np.ndarray` objects. - `'jax'`: Return JAX `jnp.ndarray` objects. suffix (`str`, `list[str]`, `list[list[str]]`): The suffixes or batch of suffixes to be encoded. Only necessary for finetuning. See https://github.com/google-research/big_vision/blob/main/big_vision/configs/proj/paligemma/README.md for more information. If your prompt is " What is on the image", the suffix corresponds to the expected prediction "a cow sitting on a bench". Returns: [`BatchFeature`]: A [`BatchFeature`] with the following fields: - **input_ids** -- List of token ids to be fed to a model. Returned when `text` is not `None`. If `suffix` is provided, the `input_ids` will also contain the suffix input ids. - **attention_mask** -- List of indices specifying which tokens should be attended to by the model (when `return_attention_mask=True` or if *"attention_mask"* is in `self.model_input_names` and if `text` is not `None`). - **pixel_values** -- Pixel values to be fed to a model. Returned when `images` is not `None`. - **labels** -- Labels compatible with training if `suffix` is not None tokenizer_init_kwargsr)rNTFzF`images` are expected as arguments to a `PaliGemmaProcessor` instance.z]You are using PaliGemma without a text prefix. It will perform as a picture-captioning model.rcss|]}t|vVqdSr8)rSrsamplerrr sz.PaliGemmaProcessor.__call__..aLYou are passing both `text` and `images` to `PaliGemmaProcessor`. The processor expects special image tokens in the text, as many tokens as there are images per each text. It is recommended to add `` tokens in the very beginning of your text. For this call, we will infer how many images each text has and add special tokens.z Received z images for zK prompts. Each prompt should be associated with an image or list of images.cSsg|]}|gqSrr)rimagerrrrsz/PaliGemmaProcessor.__call__..zAimages must be an image, list of images or list of list of imagesc s:g|]\}}t|jjjtt|trt|nddqS)r>)rDrGr@rLrSr3r#len)rr? image_listr]rrrscSsg|]}|dqS)r=rrfrrrr$scsg|]}|jjqSr)rG eos_token)rsfxrmrrr)sr* pixel_valuesreturn_tensorsr,) text_pairreturn_token_type_idsri) modalities input_idsitoken_type_idslabelsrjmm_token_type_ids)data tensor_type)"r _merge_kwargsr(rG init_kwargspoprQlogger warning_oncer<r3r#anywarningrkrtupleziprreplacerSrLrfindr@appendrF_check_special_mm_tokensnparrayupdate zeros_likerVtolistr)r]rbrcaudiovideosr^ output_kwargsrrt input_stringsexpanded_samplesrgexpanded_samplebos_rfind_index bos_indexrqrrr,inputs return_datarx array_idsryrrmr__call__sB         zPaliGemmaProcessor.__call__cKsHi}|dur|jgt|}dgt|}|||dtdi|S)aB Computes the number of placeholder tokens needed for multimodal inputs with the given sizes. Args: image_sizes (list[list[str]], *optional*): The input sizes formatted as (height, width) per each image. Returns: dict[str, list[int]]: A dictionary mapping each modality ("image", "video", "audio") to a list containing the number of placeholder tokens required. If the model doesn't accept a certain modality or no input sizes are provided, the dict value is set to an empty list. Nrj)num_image_tokensnum_image_patchesr)rLrkrr )r] image_sizesr^ vision_datarrrrr_get_num_multimodal_tokensEs  z-PaliGemmaProcessor._get_num_multimodal_tokenscO|jj|i|S)z This method forwards all its arguments to GemmaTokenizerFast's [`~PreTrainedTokenizer.batch_decode`]. Please refer to the docstring of this method for more information. )rG batch_decoder]argsr^rrrrYzPaliGemmaProcessor.batch_decodecOr)z This method forwards all its arguments to GemmaTokenizerFast's [`~PreTrainedTokenizer.decode`]. Please refer to the docstring of this method for more information. )rGdecoderrrrrarzPaliGemmaProcessor.decodecCs"|jj}|jj}tt||Sr8)rGmodel_input_namesrFr#dictfromkeys)r]tokenizer_input_namesimage_processor_input_namesrrrrhsz$PaliGemmaProcessor.model_input_names)NNN)NNNNr8)r r!r"__doc__ attributesimage_processor_classtokenizer_classr\rrrrr#rr(rrrrrpropertyr __classcell__rrr`rrEfs8"  ,rE)*rtypingrrnumpyrfeature_extraction_utilsr image_utilsrrrprocessing_utilsr r r r r rrtokenization_utils_baserrrutilsr get_loggerr rrSrangerXrr%r(r'r7r;r<rDrE__all__rrrrs, $  $