o eil@@sddlmZmZddlmZddlmZmZddlm Z m Z m Z m Z ddl mZmZmZddlmZmZer;ddlZGd d d e d d Zd ZddedDddedDZddZeGddde ZdgZdS))OptionalUnion) BatchFeature) ImageInputmake_flat_list_of_images)MultiModalDataProcessingKwargsProcessorMixinUnpack) AddedTokenPreTokenizedInput TextInput)auto_docstringis_torch_availableNc@s&eZdZddidddddidZd S) ColPaliProcessorKwargspaddinglongestchannels_firstT) data_formatdo_convert_rgbreturn_tensorspt) text_kwargs images_kwargs common_kwargsN)__name__ __module__ __qualname__ _defaultsr r l/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/colpali/processing_colpali.pyr#s rF)totalzcCg|] }d|ddqS)z4>r .0ir r r! 1r(icCr#)z3r$r r%r r r!r(1r)cCs|||||dS)aZ Builds a string from the input prompt and image tokens. For example, for the call: build_string_from_input( prompt="Prefix str" bos_token="", image_seq_len=3, image_token="", ) The output will be: "Initial str" Args: prompt (`list[Union[str, ImageInput]]`): The input prompt. bos_token (`str`): The beginning of sentence token. image_seq_len (`int`): The length of the image sequence. image_token (`str`): The image token. num_images (`int`): Number of images in the prompt.  r prompt bos_token image_seq_len image_token num_imagesr r r!build_string_from_input4sr2c s,eZdZ     d$dedeffdd Ze  d%dedBd eeBe eBe eBd e e d e fd d Z d&ddZeddZed efddZ d&dedBd e e d e fddZd ee eBd e e d e fddZ   d'dede dfdede dfdededd ed!efd df d"d#ZZS)(ColPaliProcessorNDescribe the image. Question: visual_prompt_prefix query_prefixcs||_||_t|dstd|j|_t|ds3ttddd}d|gi}|||t|_ t|_ n|j |_ |j |_ | t d|_ d|_tj|||dd S) a! visual_prompt_prefix (`str`, *optional*, defaults to `"Describe the image."`): A string that gets tokenized and prepended to the image tokens. query_prefix (`str`, *optional*, defaults to `"Question: "`): A prefix to be used for the query. image_seq_lengthz;Image processor is missing an `image_seq_length` attribute.r0FT) normalizedspecialadditional_special_tokens) chat_templateN)r6r7hasattr ValueErrorr8r IMAGE_TOKENadd_special_tokensconvert_tokens_to_idsimage_token_idr0 add_tokens EXTRA_TOKENS add_bos_token add_eos_tokensuper__init__)selfimage_processor tokenizerr<r6r7r0 tokens_to_add __class__r r!rHLs"      zColPaliProcessor.__init__imagestextkwargsreturnc s jtfdjji|}|ddd}d}|dur#|dur#td|dur/|dur/td|durj|}t|}j gt |}dd |D}fd d t ||D}j|fi|d d } |d d ddurz|dd j 7<j|fd|i|d} i| d | i} |r| d| ddkd} | d| it| dS|durt|tr|g}nt|trt|dtstd|durЈjd}g} |D]}jjj||d}| |q|d d d|dd <j| fd|i|d}|SdS)a Returns: [`BatchFeature`]: A [`BatchFeature`] with the following fields: - **input_ids** -- List of token ids to be fed to a model. - **attention_mask** -- List of indices specifying which tokens should be attended to by the model (when `return_attention_mask=True` or if *"attention_mask"* is in `self.model_input_names` and if `text` is not `None`). - **pixel_values** -- Pixel values to be fed to a model. Returned when `images` is not `None`. tokenizer_init_kwargsrsuffixNTz&Either text or images must be providedz5Only one of text or images can be processed at a timecSsg|]}|dqS)RGB)convert)r&imager r r!r(sz-ColPaliProcessor.__call__..c s:g|]\}}t|jjjtt|trt|nddqS)r,)r2rKr.r8r? isinstancelistlen)r&r- image_listrIr r!r(sr pixel_values max_lengthreturn_token_type_ids input_idstoken_type_idsrilabels)dataz*Text must be a string or a list of strings r+2) _merge_kwargsrrK init_kwargspopr>rJ fetch_imagesrr6r[zipgetr8 masked_fillupdaterrYstrrZquery_augmentation_tokenr.r7append)rIrOrPrQ output_kwargsrTr` texts_doc input_stringsr^inputs return_datarc texts_queryquery batch_queryr r]r!__call__qsr        zColPaliProcessor.__call__cKsHi}|dur|jgt|}dgt|}|||dtdi|S)a Computes the number of placeholder tokens needed for multimodal inputs with the given sizes. Args: image_sizes (list[list[str]], *optional*): The input sizes formatted as (height, width) per each image. Returns: `MultiModalData`: A `MultiModalData` object holding number of tokens per each of the provided input modalities, along with other useful data. NrX)num_image_tokensnum_image_patchesr )r8r[rnr)rI image_sizesrQ vision_datar{r|r r r!_get_num_multimodal_tokenss  z+ColPaliProcessor._get_num_multimodal_tokenscCs$|jjddg}|jj}t||S)Nrbrc)rKmodel_input_namesrJrZ)rItokenizer_input_namesimage_processor_input_namesr r r!rs z"ColPaliProcessor.model_input_namescCs|jjS)z Return the query augmentation token. Query augmentation buffers are used as reasoning buffers during inference. )rK pad_tokenr]r r r!rpsz)ColPaliProcessor.query_augmentation_tokencK|jdd|i|S)a Prepare for the model one or several image(s). This method is a wrapper around the `__call__` method of the ColPaliProcessor's [`ColPaliProcessor.__call__`]. This method forwards the `images` and `kwargs` arguments to the image processor. Args: images (`PIL.Image.Image`, `np.ndarray`, `torch.Tensor`, `list[PIL.Image.Image]`, `list[np.ndarray]`, `list[torch.Tensor]`): The image or batch of images to be prepared. Each image can be a PIL image, NumPy array or PyTorch tensor. In case of a NumPy array/PyTorch tensor, each image should be of shape (C, H, W), where C is a number of channels, H and W are image height and width. return_tensors (`str` or [`~utils.TensorType`], *optional*): If set, will return tensors of a particular framework. Acceptable values are: - `'pt'`: Return PyTorch `torch.Tensor` objects. - `'np'`: Return NumPy `np.ndarray` objects. Returns: [`BatchFeature`]: A [`BatchFeature`] with the following fields: - **input_ids** -- List of token ids to be fed to a model. - **attention_mask** -- List of indices specifying which tokens should be attended to by the model (when `return_attention_mask=True` or if *"attention_mask"* is in `self.model_input_names` and if `text` is not `None`). - **pixel_values** -- Pixel values to be fed to a model. Returned when `images` is not `None`. rONr rz)rIrOrQr r r!process_imagesszColPaliProcessor.process_imagescKr)ag Prepare for the model one or several texts. This method is a wrapper around the `__call__` method of the ColPaliProcessor's [`ColPaliProcessor.__call__`]. This method forwards the `text` and `kwargs` arguments to the tokenizer. Args: text (`str`, `list[str]`, `list[list[str]]`): The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set `is_split_into_words=True` (to lift the ambiguity with a batch of sequences). return_tensors (`str` or [`~utils.TensorType`], *optional*): If set, will return tensors of a particular framework. Acceptable values are: - `'pt'`: Return PyTorch `torch.Tensor` objects. - `'np'`: Return NumPy `np.ndarray` objects. Returns: [`BatchFeature`]: A [`BatchFeature`] with the following fields: - **input_ids** -- List of token ids to be fed to a model. - **attention_mask** -- List of indices specifying which tokens should be attended to by the model (when `return_attention_mask=True` or if *"attention_mask"* is in `self.model_input_names` and if `text` is not `None`). rPNr r)rIrPrQr r r!process_queriessz ColPaliProcessor.process_queriesr*cpuquery_embeddingsz torch.Tensorpassage_embeddings batch_size output_dtypez torch.dtype output_devicez torch.devicec Cs@t|dkr tdt|dkrtd|dj|djkr"td|dj|djkr0td|dur9|dj}g}tdt||D]U}g}tjjjj ||||ddd} tdt||D]'} tjjjj || | |ddd} | t d | | j d d dj d d q`| tj|d d ||qCtj|dd S)aZ Compute the late-interaction/MaxSim score (ColBERT-like) for the given multi-vector query embeddings (`qs`) and passage embeddings (`ps`). For ColPali, a passage is the image of a document page. Because the embedding tensors are multi-vector and can thus have different shapes, they should be fed as: (1) a list of tensors, where the i-th tensor is of shape (sequence_length_i, embedding_dim) (2) a single tensor of shape (n_passages, max_sequence_length, embedding_dim) -> usually obtained by padding the list of tensors. Args: query_embeddings (`Union[torch.Tensor, list[torch.Tensor]`): Query embeddings. passage_embeddings (`Union[torch.Tensor, list[torch.Tensor]`): Passage embeddings. batch_size (`int`, *optional*, defaults to 128): Batch size for computing scores. output_dtype (`torch.dtype`, *optional*, defaults to `torch.float32`): The dtype of the output tensor. If `None`, the dtype of the input embeddings is used. output_device (`torch.device` or `str`, *optional*, defaults to "cpu"): The device of the output tensor. Returns: `torch.Tensor`: A tensor of shape `(n_queries, n_passages)` containing the scores. The score tensor is saved on the "cpu" device. rzNo queries providedzNo passages providedz/Queries and passages must be on the same devicez-Queries and passages must have the same dtypeNT) batch_first padding_valuez bnd,csd->bcnsr)dimrX)r[r>devicedtyperangetorchnnutilsrnn pad_sequencerqeinsummaxsumcatto) rIrrrrrscoresr' batch_scores batch_queriesjbatch_passagesr r r!score_retrieval.s2      "z ColPaliProcessor.score_retrieval)NNNr4r5)NN)N)r*Nr)rrrrorHrrrr rZr rrrzrpropertyrrprrrintrr __classcell__r r rMr!r3Jsv% Z   !  $ r3)typingrrfeature_extraction_utilsr image_utilsrrprocessing_utilsrr r r tokenization_utils_baser r rrrrrrr?rrDr2r3__all__r r r r!s"  $ &