o ei$%@sdZddlZddlmZddlmZmZddlm Z m Z m Z m Z ddl mZmZddlmZmZmZer;d d lmZeeZGd d d e d dZdefddZddZeGddde ZdgZdS)z Processor class for Pixtral. N) BatchFeature) ImageInputis_valid_image)MultiModalDataProcessingKwargsProcessorMixinUnpack)PreTokenizedInput TextInput)auto_docstringis_vision_availablelogging)get_resize_output_image_sizec@s eZdZdddddidZdS)PixtralProcessorKwargsF)paddingreturn_mm_token_type_idsreturn_tensorspt) text_kwargs common_kwargsN)__name__ __module__ __qualname__ _defaultsrrl/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/pixtral/processing_pixtral.pyr's rF)totalreturncCst|to |dS)Nhttp) isinstancestr startswith)valrrris_url4sr%cCst|pt|SN)r%r)elemrrris_image_or_image_url9sr(c seZdZ        ddedeffd d Ze  dd edBd eeBe eBe eBd e e de fddZ dddZeddZZS)PixtralProcessorNr[IMG] [IMG_BREAK] [IMG_END] patch_sizespatial_merge_sizec  s~||_||_||_||j|_||_||_||j|_||j|_||j|_|j|j|jg|_ t j |||ddS)a patch_size (`int`, *optional*, defaults to 16): Patch size from the vision tower. spatial_merge_size (`int`, *optional*, defaults to 1): The downsampling factor for the spatial merge operation. image_token (`str`, *optional*, defaults to `"[IMG]"`): Special token used to denote image location. image_break_token (`str`, *optional*, defaults to `"[IMG_BREAK]"`): Special token used to denote the end of a line of pixels in an image. image_end_token (`str`, *optional*, defaults to `"[IMG_END]"`): Special token used to denote the end of an image input. ) chat_templateN) r.r/ image_tokenconvert_tokens_to_idsimage_token_idimage_break_tokenimage_end_tokenimage_break_token_idimage_end_token_id image_idssuper__init__) selfimage_processor tokenizerr.r/r0r1r4r5kwargs __class__rrr:?szPixtralProcessor.__init__imagestextr>rcKs*|jtfdt|jdii|}|j|j}|dur,||dd<|j|fi|d}ni}t|tr7|g}nt|t sGt|dtsGt d|}| ddurt |d }g}g} |D]^} |j | vrt|\} } | |} | |}|j g||jgg| }d d |D}|j|d <d |}| || |j dd} |j | vscd| vr| d}| d|d} d| vs|| q\|ddd}|ddd}|ddd|j|fi|dddi}|j||dgd|r t|d}t|d}d|t||j<||d<ti|||dS)a? Returns: [`BatchFeature`]: A [`BatchFeature`] with the following fields: - **input_ids** -- List of token ids to be fed to a model. Returned when `text` is not `None`. - **attention_mask** -- List of indices specifying which tokens should be attended to by the model (when `return_attention_mask=True` or if *"attention_mask"* is in `self.model_input_names` and if `text` is not `None`). - **pixel_values** -- Pixel values to be fed to a model. Returned when `images` is not `None`. tokenizer_init_kwargs init_kwargsN images_kwargsr.rzAInvalid input text. Please provide a string, or a list of strings pixel_values image_sizescSsg|] }|D]}|qqSrr).0sublistitemrrr sz-PixtralProcessor.__call__..z rrrrFreturn_token_type_idsimage) modalities input_idsmm_token_type_ids)data tensor_type) _merge_kwargsrgetattrr=r.r/r<r!r"list TypeErrorgetiterr1nextr4r5joinappendreplacepop_check_special_mm_tokensnparray zeros_likeisinr8tolistr)r;rArBr> output_kwargsr. image_inputsprompt_stringsrGreplace_stringssampleheightwidthnum_height_tokensnum_width_tokensreplace_tokens replace_strrr text_inputs array_idsrRrrr__call__csh              zPixtralProcessor.__call__cKsi}|durbtjdi}|||ddp|jj}|j|j}g}|D],\}} tt || df|d|df||fd\} } | |} | |} | | d| q&dgt |}|||dt d i|S) a Computes the number of placeholder tokens needed for multimodal inputs with the given sizes. Args: image_sizes (`list[list[int]]`, *optional*): The input sizes formatted as (height, width) per each image. Returns: `MultiModalData`: A `MultiModalData` object holding number of tokens per each of the provided input modalities, along with other useful data. NrEsizer longest_edge)rtr.r)num_image_tokensnum_image_patchesr)rrrYupdater<rtr.r/rrazerosr]lenr)r;rGr> vision_datarErtr.rvrkrlresized_height resized_widthrmrnrwrrr_get_num_multimodal_tokenss&     z+PixtralProcessor._get_num_multimodal_tokenscCs|jj}|jj}||dgS)NrG)r=model_input_namesr<)r;tokenizer_input_namesimage_processor_input_namesrrrrsz"PixtralProcessor.model_input_names)NNr*rNr+r,r-)NNr&)rrrintr:r rr r rWr rrrsr~propertyr __classcell__rrr?rr)=s:$ O$r))__doc__numpyrafeature_extraction_utilsr image_utilsrrprocessing_utilsrrrr tokenization_utils_baser r utilsr r rimage_processing_pixtralr get_loggerrloggerrboolr%r(r)__all__rrrrs"      !