o ei=2@sddlmZmZddlmZmZmZddlmZddl m Z m Z ddl m Z mZddlmZmZddlmZmZer?dd lZeeZGd d d e d d ZGdddeZdgZd S))OptionalUnion) IMAGE_TOKENPaliGemmaProcessorbuild_string_from_input) BatchFeature) ImageInputmake_flat_list_of_images)ProcessingKwargsUnpack)PreTokenizedInput TextInput)is_torch_availableloggingNc@s&eZdZddidddddidZd S) ColPaliProcessorKwargspaddinglongestchannels_firstT) data_formatdo_convert_rgbreturn_tensorspt) text_kwargs images_kwargs common_kwargsN)__name__ __module__ __qualname__ _defaultsr r i/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/colpali/modular_colpali.pyr!s rF)totalc seZdZ     d dedeffdd Zedefd d Z  d!d edBd ee Be eBe e Bd e e de fddZ d"d edBd e e de fddZd ee eBd e e de fddZ   d#dede dfdede dfdededdedefddf ddZZS)$ColPaliProcessorNDescribe the image. Question: visual_prompt_prefix query_prefixcs"||_||_tj|||ddS)a! visual_prompt_prefix (`str`, *optional*, defaults to `"Describe the image."`): A string that gets tokenized and prepended to the image tokens. query_prefix (`str`, *optional*, defaults to `"Question: "`): A prefix to be used for the query. )image_processor tokenizer chat_templateN)r&r'super__init__)selfr(r)r*r&r' __class__r r!r,/szColPaliProcessor.__init__returncCs|jjS)z Return the query augmentation token. Query augmentation buffers are used as reasoning buffers during inference. )r) pad_tokenr-r r r!query_augmentation_tokenAsz)ColPaliProcessor.query_augmentation_tokenimagestextkwargsc s jtfdjji|}|ddd}d}|dur#|dur#td|dur/|dur/td|durj|}t|}j gt |}dd |D}fd d t ||D}j|fi|d d } |d d ddurz|dd j 7<j|fd|i|d} i| d | i} |r| d| ddkd} | d| it| dS|durt|tr|g}nt|trt|dtstd|durЈjd}g} |D]}jjj||d}| |q|d d d|dd <j| fd|i|d}|SdS)a Returns: [`BatchFeature`]: A [`BatchFeature`] with the following fields: - **input_ids** -- List of token ids to be fed to a model. - **attention_mask** -- List of indices specifying which tokens should be attended to by the model (when `return_attention_mask=True` or if *"attention_mask"* is in `self.model_input_names` and if `text` is not `None`). - **pixel_values** -- Pixel values to be fed to a model. Returned when `images` is not `None`. tokenizer_init_kwargsrsuffixNTz&Either text or images must be providedz5Only one of text or images can be processed at a timecSsg|]}|dqS)RGB)convert).0imager r r! lsz-ColPaliProcessor.__call__..c s:g|]\}}t|jjjtt|trt|nddqS))prompt bos_token image_seq_len image_token num_images)rr)r@image_seq_lengthr isinstancelistlen)r;r? image_listr2r r!r=nsr pixel_values max_lengthreturn_token_type_ids input_idstoken_type_idsrilabels)dataz*Text must be a string or a list of strings  2) _merge_kwargsrr) init_kwargspop ValueErrorr( fetch_imagesr r&rGzipgetrD masked_fillupdaterrEstrrFr3r@r'append)r-r4r5r6 output_kwargsr8rK texts_doc input_stringsrIinputs return_datarN texts_queryquery batch_queryr r2r!__call__Jsr        zColPaliProcessor.__call__cK|jdd|i|S)a Prepare for the model one or several image(s). This method is a wrapper around the `__call__` method of the ColPaliProcessor's [`ColPaliProcessor.__call__`]. This method forwards the `images` and `kwargs` arguments to the image processor. Args: images (`PIL.Image.Image`, `np.ndarray`, `torch.Tensor`, `list[PIL.Image.Image]`, `list[np.ndarray]`, `list[torch.Tensor]`): The image or batch of images to be prepared. Each image can be a PIL image, NumPy array or PyTorch tensor. In case of a NumPy array/PyTorch tensor, each image should be of shape (C, H, W), where C is a number of channels, H and W are image height and width. return_tensors (`str` or [`~utils.TensorType`], *optional*): If set, will return tensors of a particular framework. Acceptable values are: - `'pt'`: Return PyTorch `torch.Tensor` objects. - `'np'`: Return NumPy `np.ndarray` objects. Returns: [`BatchFeature`]: A [`BatchFeature`] with the following fields: - **input_ids** -- List of token ids to be fed to a model. - **attention_mask** -- List of indices specifying which tokens should be attended to by the model (when `return_attention_mask=True` or if *"attention_mask"* is in `self.model_input_names` and if `text` is not `None`). - **pixel_values** -- Pixel values to be fed to a model. Returned when `images` is not `None`. r4Nr rf)r-r4r6r r r!process_imagesszColPaliProcessor.process_imagescKrg)ag Prepare for the model one or several texts. This method is a wrapper around the `__call__` method of the ColPaliProcessor's [`ColPaliProcessor.__call__`]. This method forwards the `text` and `kwargs` arguments to the tokenizer. Args: text (`str`, `list[str]`, `list[list[str]]`): The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set `is_split_into_words=True` (to lift the ambiguity with a batch of sequences). return_tensors (`str` or [`~utils.TensorType`], *optional*): If set, will return tensors of a particular framework. Acceptable values are: - `'pt'`: Return PyTorch `torch.Tensor` objects. - `'np'`: Return NumPy `np.ndarray` objects. Returns: [`BatchFeature`]: A [`BatchFeature`] with the following fields: - **input_ids** -- List of token ids to be fed to a model. - **attention_mask** -- List of indices specifying which tokens should be attended to by the model (when `return_attention_mask=True` or if *"attention_mask"* is in `self.model_input_names` and if `text` is not `None`). r5Nr rh)r-r5r6r r r!process_queriessz ColPaliProcessor.process_queriescpuquery_embeddingsz torch.Tensorpassage_embeddings batch_size output_dtypez torch.dtype output_devicez torch.devicec Cs@t|dkr tdt|dkrtd|dj|djkr"td|dj|djkr0td|dur9|dj}g}tdt||D]U}g}tjjjj ||||ddd} tdt||D]'} tjjjj || | |ddd} | t d | | j d d dj d d q`| tj|d d ||qCtj|dd S)aZ Compute the late-interaction/MaxSim score (ColBERT-like) for the given multi-vector query embeddings (`qs`) and passage embeddings (`ps`). For ColPali, a passage is the image of a document page. Because the embedding tensors are multi-vector and can thus have different shapes, they should be fed as: (1) a list of tensors, where the i-th tensor is of shape (sequence_length_i, embedding_dim) (2) a single tensor of shape (n_passages, max_sequence_length, embedding_dim) -> usually obtained by padding the list of tensors. Args: query_embeddings (`Union[torch.Tensor, list[torch.Tensor]`): Query embeddings. passage_embeddings (`Union[torch.Tensor, list[torch.Tensor]`): Passage embeddings. batch_size (`int`, *optional*, defaults to 128): Batch size for computing scores. output_dtype (`torch.dtype`, *optional*, defaults to `torch.float32`): The dtype of the output tensor. If `None`, the dtype of the input embeddings is used. output_device (`torch.device` or `str`, *optional*, defaults to "cpu"): The device of the output tensor. Returns: `torch.Tensor`: A tensor of shape `(n_queries, n_passages)` containing the scores. The score tensor is saved on the "cpu" device. rzNo queries providedzNo passages providedz/Queries and passages must be on the same devicez-Queries and passages must have the same dtypeNT) batch_first padding_valuez bnd,csd->bcnsr)dimr>)rGrVdevicedtyperangetorchnnutilsrnn pad_sequencer]einsummaxsumcatto) r-rmrnrorprqscoresi batch_scores batch_queriesjbatch_passagesr r r!score_retrievals2      "z ColPaliProcessor.score_retrieval)NNNr$r%)NN)N)rkNrl)rrrr\r,propertyr3r rr rFr rrrfrirjrintrr __classcell__r r r.r!r#.sn  \ !  $ r#)typingrr2transformers.models.paligemma.processing_paligemmarrrfeature_extraction_utilsr image_utilsr r processing_utilsr r tokenization_utils_baser rr{rrry get_loggerrloggerrr#__all__r r r r!s   z