o ei-@sdZddlZddlmZddlmZmZddlm Z m Z m Z m Z m Z ddlmZmZmZddlmZmZeeZd Zd d ed Dd d edDZGddde ZGddde ddZdefddZddZddZ ddZ!eGddde Z"dgZ#dS) z Processor class for PaliGemma. N) BatchFeature) ImageInputis_valid_image)MultiModalDataProcessingKwargsProcessorMixin TextKwargsUnpack) AddedTokenPreTokenizedInput TextInput)auto_docstringloggingzcCg|] }d|ddqS)z4>.0irrp/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/paligemma/processing_paligemma.py $ricCr)z3rrrrrrr$rc@s2eZdZUdZeeBeeBeeBdBed<dS)PaliGemmaTextKwargsa suffix (`str`, `list[str]`, `list[list[str]]`): The suffixes or batch of suffixes to be encoded. Only necessary for finetuning. See https://github.com/google-research/big_vision/blob/main/big_vision/configs/proj/paligemma/README.md for more information. If your prompt is " What is on the image", the suffix corresponds to the expected prediction "a cow sitting on a bench". Nsuffix)__name__ __module__ __qualname____doc__r r list__annotations__rrrrr's $rc@s*eZdZUeed<dddddidZdS)PaliGemmaProcessorKwargs text_kwargsF)paddingreturn_mm_token_type_ids data_formatchannels_first)r# images_kwargsN)rrrrr! _defaultsrrrrr"1s  r"F)totalreturncCst|to |dS)Nhttp) isinstancestr startswith)valrrris_url?sr1cCst|pt|SN)r1relemrrris_image_or_image_urlDsr5cCst|tpt|Sr2)r-r.r5r3rrr_is_str_or_imageHsr6cCs|||||dS)aZ Builds a string from the input prompt and image tokens. For example, for the call: build_string_from_input( prompt="Prefix str" bos_token="", image_seq_len=3, image_token="", ) The output will be: "Initial str" Args: prompt (`list[Union[str, ImageInput]]`): The input prompt. bos_token (`str`): The beginning of sentence token. image_seq_len (`int`): The length of the image sequence. image_token (`str`): The image token. num_images (`int`): Number of images in the prompt.  rprompt bos_token image_seq_len image_token num_imagesrrrbuild_string_from_inputLsr>c szeZdZ   dfdd Ze  ddedBdeeBeeBeeBde e de fdd Z dd d Z ed d ZZS)PaliGemmaProcessorNc st|ds td|j|_t|ds-ttddd}d|gi}|||t|_t|_n|j|_|j|_| t d|_ d|_ t j|||ddS) Nimage_seq_lengthz;Image processor is missing an `image_seq_length` attribute.r<FT) normalizedspecialadditional_special_tokens) chat_template)hasattr ValueErrorr@r IMAGE_TOKENadd_special_tokensconvert_tokens_to_idsimage_token_idr< add_tokens EXTRA_TOKENS add_bos_token add_eos_tokensuper__init__)selfimage_processor tokenizerrDkwargsr< tokens_to_add __class__rrrPds      zPaliGemmaProcessor.__init__imagestextrTr+c s jtfdjji|}|ddd}d}|durtd|dur*tdd}t|r2|g}n t |t r>t|d r> |dur|durt d d |Dst d t |t rut |t rut |t |krutd t |dt |dt|r~|gg}n1t |t tfrt|d rdd|D}nt |t tfrt |d t tfrt|d d stdfddt||D}n=g}|D]1} | ttj} | t} | dkr| t tnd } | d| jj| | d} || qdd|D}|durt|r|g}|durfdd|D}j|fi|dd} |ddd}|ddd}j|f||d|d}j||dgdi|d| i}|rlt|d}d|t|d d k<|d!|i|rt|d}t|d}d"||jk<||d#<t||d$S)%a Returns: [`BatchFeature`]: A [`BatchFeature`] with the following fields: - **input_ids** -- List of token ids to be fed to a model. Returned when `text` is not `None`. If `suffix` is provided, the `input_ids` will also contain the suffix input ids. - **attention_mask** -- List of indices specifying which tokens should be attended to by the model (when `return_attention_mask=True` or if *"attention_mask"* is in `self.model_input_names` and if `text` is not `None`). - **pixel_values** -- Pixel values to be fed to a model. Returned when `images` is not `None`. - **labels** -- Labels compatible with training if `suffix` is not None tokenizer_init_kwargsr#rNTzF`images` are expected as arguments to a `PaliGemmaProcessor` instance.z]You are using PaliGemma without a text prefix. It will perform as a picture-captioning model.rcss|]}t|vVqdSr2)rGrsamplerrr sz.PaliGemmaProcessor.__call__..aLYou are passing both `text` and `images` to `PaliGemmaProcessor`. The processor expects special image tokens in the text, as many tokens as there are images per each text. It is recommended to add `` tokens in the very beginning of your text. For this call, we will infer how many images each text has and add special tokens.z Received z images for zK prompts. Each prompt should be associated with an image or list of images.cSsg|]}|gqSrr)rimagerrrrsz/PaliGemmaProcessor.__call__..zAimages must be an image, list of images or list of list of imagesc s:g|]\}}t|jjjtt|trt|nddqS)r8)r>rSr:r@rGr-r len)rr9 image_listrQrrrscSsg|]}|dqS)r7rr\rrrrscsg|]}|jjqSr)rS eos_token)rsfxrcrrrsr( pixel_valuesreturn_tensorsr%) text_pairreturn_token_type_idsr_) modalities input_idsitoken_type_idslabelsr`mm_token_type_ids)data tensor_type) _merge_kwargsr"rS init_kwargspoprFlogger warning_oncer6r-r anywarningrartuplezipreplacerGr@rfindr:appendrR_check_special_mm_tokensnparrayupdate zeros_likerJtolistr)rQrXrYrT output_kwargsrrj input_stringsexpanded_samplesr]expanded_samplebos_rfind_index bos_indexrgrhr%inputs return_datarn array_idsrorrcr__call__s         zPaliGemmaProcessor.__call__cKsHi}|dur|jgt|}dgt|}|||dtdi|S)a Computes the number of placeholder tokens needed for multimodal inputs with the given sizes. Args: image_sizes (list[list[str]], *optional*): The input sizes formatted as (height, width) per each image. Returns: `MultiModalData`: A `MultiModalData` object holding number of tokens per each of the provided input modalities, along with other useful data. Nr`)num_image_tokensnum_image_patchesr)r@rarr)rQ image_sizesrT vision_datarrrrr_get_num_multimodal_tokenss  z-PaliGemmaProcessor._get_num_multimodal_tokenscCs$|jjddg}|jj}t||S)Nrmrn)rSmodel_input_namesrRr )rQtokenizer_input_namesimage_processor_input_namesrrrr s z$PaliGemmaProcessor.model_input_names)NNN)NNr2)rrrrPrrr r r r r"rrrpropertyr __classcell__rrrVrr?bs( zr?)$rnumpyrfeature_extraction_utilsr image_utilsrrprocessing_utilsrrrr r tokenization_utils_baser r r utilsrr get_loggerrrurGrangerLrr"boolr1r5r6r>r?__all__rrrrs(   $  2