o ei{A@sddlmZmZddlmZddlmZmZddlm Z m Z m Z m Z ddl mZmZddlmZmZer9ddlZGd d d e d d ZeGd dde ZdgZdS))OptionalUnion) BatchFeature) ImageInputis_valid_image)MultiModalDataProcessingKwargsProcessorMixinUnpack)PreTokenizedInput TextInput)auto_docstringis_torch_availableNc@s&eZdZddidddddidZd S) ColQwen2ProcessorKwargspaddinglongestchannels_firstT) data_formatdo_convert_rgbreturn_tensorspt) text_kwargs images_kwargs common_kwargsN)__name__ __module__ __qualname__ _defaultsrrn/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/colqwen2/processing_colqwen2.pyr"s rF)totalc s4eZdZ     d"dedBdedBffdd Ze  d#dedBdeeBe eBe eBde e d e fd d Z d$d d ZeddZed efddZ d$dedBde e d e fddZdee eBde e d e fddZ   d%dede dfdede dfdededdedefd df d d!ZZS)&ColQwen2ProcessorNvisual_prompt_prefix query_prefixc sftj|||dt|dsdn|j|_t|dsdn|j|_|dur%d}||_|dur.d}||_dS) ar visual_prompt_prefix (`str`, *optional*, defaults to `"<|im_start|>user\n<|vision_start|><|image_pad|><|vision_end|>Describe the image.<|im_end|><|endoftext|>"`): A string that gets tokenized and prepended to the image tokens. query_prefix (`str`, *optional*, defaults to `"Query: "`): A prefix to be used for the query. ) chat_template image_tokenz <|image_pad|> video_tokenz <|video_pad|>Nzf<|im_start|>user <|vision_start|><|image_pad|><|vision_end|>Describe the image.<|im_end|><|endoftext|>zQuery: )super__init__hasattrr&r'r#r$)selfimage_processor tokenizerr%r#r$kwargs __class__rr r)1s zColQwen2Processor.__init__imagestextr.returncKs|jtfd|jji|}|ddd}|du}|dur%|dur%td|dur1|dur1td|durt|r>|g}n$t|trJt|drJnt|tr^t|dtr^t|ddsbtd|j gt |}|j dd |i|d }|d } | dur|j j d } d} t t |D]3} |j|| vr|| |jd | | | d|| <| d7} |j|| vs|| d |j|| <q|j|fddi|d} ti| |d}|d dddf|d ddd f}tt|d|}tjjjj|dd|d<|r|d|ddkd}|d|i|S|durht|tr)|g}nt|tr7t|dts;td|durE|jd}g}|D]}|j||}||qI|j|fddi|d}|SdS)a Returns: [`BatchFeature`]: A [`BatchFeature`] with the following fields: - **input_ids** -- List of token ids to be fed to a model. - **attention_mask** -- List of indices specifying which tokens should be attended to by the model (when `return_attention_mask=True` or if *"attention_mask"* is in `self.model_input_names` and if `text` is not `None`). - **pixel_values** -- Pixel values to be fed to a model. Returned when `images` is not `None`. tokenizer_init_kwargsrsuffixNz&Either text or images must be providedz5Only one of text or images can be processed at a timerzAimages must be an image, list of images or list of list of imagesr1rimage_grid_thwz<|placeholder|>return_token_type_idsF)data pixel_valuesT) batch_first input_idstoken_type_idsilabelsz*Text must be a string or a list of strings r) _merge_kwargsrr- init_kwargspop ValueErrorr isinstancelistr#lenr, merge_sizeranger&replaceprodrtorchsplittolistnnutilsrnn pad_sequence masked_fillupdatestrquery_augmentation_tokenr$append)r+r1r2r. output_kwargsr5r9 texts_doc image_inputsr6 merge_lengthindexi text_inputs return_dataoffsetsr;r? texts_queryqueryaugmented_query batch_queryrrr __call__Ls ( (      zColQwen2Processor.__call__c s|i}|dur7tjdi|ddpjjfdd|D}fdd|D}|||dtdi|S) a Computes the number of placeholder tokens needed for multimodal inputs with the given sizes. Args: image_sizes (`list[list[int]]`, *optional*): The input sizes formatted as (height, width) per each image. Returns: `MultiModalData`: A `MultiModalData` object holding number of tokens per each of the provided input modalities, along with other useful data. NrrHcs"g|] }jjg|RqSr)r,get_number_of_image_patches).0 image_size)rr+rr sz@ColQwen2Processor._get_num_multimodal_tokens..csg|]}|dqS)r7r)rg num_patches)rHrr ris)num_image_tokensnum_image_patchesr)rrgetrTr,rHr)r+ image_sizesr. vision_datarlrkr)rrHr+r _get_num_multimodal_tokenss   z,ColQwen2Processor._get_num_multimodal_tokenscCs&|jj}|jj}dd|D}||S)NcSsg|]}|dvr|qS))pixel_values_videosvideo_grid_thwr)rgnamerrr risz7ColQwen2Processor.model_input_names..)r-model_input_namesr,)r+tokenizer_input_namesimage_processor_input_namesrrr rts z#ColQwen2Processor.model_input_namescCs|jjS)z Return the query augmentation token. Query augmentation buffers are used as reasoning buffers during inference. )r- pad_token)r+rrr rVsz*ColQwen2Processor.query_augmentation_tokencK|jdd|i|S)a Prepare for the model one or several image(s). This method is a wrapper around the `__call__` method of the ColQwen2Processor's [`ColQwen2Processor.__call__`]. This method forwards the `images` and `kwargs` arguments to the image processor. Args: images (`PIL.Image.Image`, `np.ndarray`, `torch.Tensor`, `list[PIL.Image.Image]`, `list[np.ndarray]`, `list[torch.Tensor]`): The image or batch of images to be prepared. Each image can be a PIL image, NumPy array or PyTorch tensor. In case of a NumPy array/PyTorch tensor, each image should be of shape (C, H, W), where C is a number of channels, H and W are image height and width. return_tensors (`str` or [`~utils.TensorType`], *optional*): If set, will return tensors of a particular framework. Acceptable values are: - `'pt'`: Return PyTorch `torch.Tensor` objects. - `'np'`: Return NumPy `np.ndarray` objects. Returns: [`BatchFeature`]: A [`BatchFeature`] with the following fields: - **input_ids** -- List of token ids to be fed to a model. - **attention_mask** -- List of indices specifying which tokens should be attended to by the model (when `return_attention_mask=True` or if *"attention_mask"* is in `self.model_input_names` and if `text` is not `None`). - **pixel_values** -- Pixel values to be fed to a model. Returned when `images` is not `None`. r1Nrre)r+r1r.rrr process_imagessz ColQwen2Processor.process_imagescKrx)ai Prepare for the model one or several texts. This method is a wrapper around the `__call__` method of the ColQwen2Processor's [`ColQwen2Processor.__call__`]. This method forwards the `text` and `kwargs` arguments to the tokenizer. Args: text (`str`, `list[str]`, `list[list[str]]`): The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set `is_split_into_words=True` (to lift the ambiguity with a batch of sequences). return_tensors (`str` or [`~utils.TensorType`], *optional*): If set, will return tensors of a particular framework. Acceptable values are: - `'pt'`: Return PyTorch `torch.Tensor` objects. - `'np'`: Return NumPy `np.ndarray` objects. Returns: [`BatchFeature`]: A [`BatchFeature`] with the following fields: - **input_ids** -- List of token ids to be fed to a model. - **attention_mask** -- List of indices specifying which tokens should be attended to by the model (when `return_attention_mask=True` or if *"attention_mask"* is in `self.model_input_names` and if `text` is not `None`). r2Nrry)r+r2r.rrr process_queriessz!ColQwen2Processor.process_queriescpuquery_embeddingsz torch.Tensorpassage_embeddings batch_size output_dtypez torch.dtype output_devicez torch.devicec Cs@t|dkr tdt|dkrtd|dj|djkr"td|dj|djkr0td|dur9|dj}g}tdt||D]U}g}tjjjj ||||ddd} tdt||D]'} tjjjj || | |ddd} | t d | | j d d dj d d q`| tj|d d ||qCtj|dd S)a[ Compute the late-interaction/MaxSim score (ColBERT-like) for the given multi-vector query embeddings (`qs`) and passage embeddings (`ps`). For ColQwen2, a passage is the image of a document page. Because the embedding tensors are multi-vector and can thus have different shapes, they should be fed as: (1) a list of tensors, where the i-th tensor is of shape (sequence_length_i, embedding_dim) (2) a single tensor of shape (n_passages, max_sequence_length, embedding_dim) -> usually obtained by padding the list of tensors. Args: query_embeddings (`Union[torch.Tensor, list[torch.Tensor]`): Query embeddings. passage_embeddings (`Union[torch.Tensor, list[torch.Tensor]`): Passage embeddings. batch_size (`int`, *optional*, defaults to 128): Batch size for computing scores. output_dtype (`torch.dtype`, *optional*, defaults to `torch.float32`): The dtype of the output tensor. If `None`, the dtype of the input embeddings is used. output_device (`torch.device` or `str`, *optional*, defaults to "cpu"): The device of the output tensor. Returns: `torch.Tensor`: A tensor of shape `(n_queries, n_passages)` containing the scores. The score tensor is saved on the "cpu" device. rzNo queries providedzNo passages providedz/Queries and passages must be on the same devicez-Queries and passages must have the same dtypeNT)r< padding_valuez bnd,csd->bcnsr)dimr7r8)rGrDdevicedtyperIrLrOrPrQrRrWeinsummaxsumcatto) r+r~rrrrscoresr] batch_scores batch_queriesjbatch_passagesrrr score_retrieval%s2      "z!ColQwen2Processor.score_retrieval)NNNNN)NN)N)r|Nr})rrrrUr)rrr r rFr rrrerppropertyrtrVrzr{rintrr __classcell__rrr/r r"/sv h    !  $ r")typingrrfeature_extraction_utilsr image_utilsrrprocessing_utilsrr r r tokenization_utils_baser r rPrrrLrr"__all__rrrr s   8