o eiW0@sddlZddlmZddlmZddlmZmZm Z m Z ddl m Z m Z ddlmZmZddlmZeeZGd d d ed d ZeGd dde ZdgZdS)N) BatchFeature) ImageInput)MultiModalDataProcessingKwargsProcessorMixinUnpack)PreTokenizedInput TextInput)auto_docstringlogging) VideoInputc@s"eZdZddddddidZdS)Glm4vProcessorKwargsF)paddingreturn_token_type_idsreturn_mm_token_type_idsreturn_metadataT) text_kwargs videos_kwargsN)__name__ __module__ __qualname__ _defaultsrrh/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/glm4v/processing_glm4v.pyr"s rF)totalc seZdZdfdd Ze   ddedBdeeBeeBeeBde dBde e de f d d Z dd d Z dddZddZZS)Glm4vProcessorNc st|dsdn|j|_t|dsdn|j|_t|ddr|jn||j|_t|ddr/|jn||j|_tj||||ddS)N image_tokenz <|image|> video_tokenz <|video|>image_token_idvideo_token_id) chat_template) hasattrrrgetattrrconvert_tokens_to_idsr super__init__)selfimage_processor tokenizervideo_processorr!kwargs __class__rrr&/s    zGlm4vProcessor.__init__imagestextvideosr+returncKs|jtfd|jji|}|dur"|jdd|i|d}|d}ni}d}|durJ|jdd|i|d}|dsA|d } n|d } |d } ni}d} t|t sV|g}| }|dur|jj d } d } t t |D]5} |j|| vr|| | }|| |jd |d|| <| d7} |j|| vsu|| d |j|| <ql| dur|jj d } d }t t |D]} |j|| vrs| |d }d}| |}|jdurtd|jdurdn|j|_|jddd }g}t d t |D] }|||q|d|}t ||kr||r|dnd t ||kst |D]}||}||}||7}q|| |j|d|| <| || | |d }t |D]}|j|| vrf|| |jd |d|| <qN|d7}|j|| vs|| d |j|| <q|ddd}|ddd}|j|fi|d}|j||ddgd|rt|d}t|d}d|||jk<||d<ti||||dS)a Returns: [`BatchFeature`]: A [`BatchFeature`] with the following fields: - **input_ids** -- List of token ids to be fed to a model. Returned when `text` is not `None`. - **attention_mask** -- List of indices specifying which tokens should be attended to by the model (when `return_attention_mask=True` or if *"attention_mask"* is in `self.model_input_names` and if `text` is not `None`). - **pixel_values** -- Pixel values to be fed to a model. Returned when `images` is not `None`. - **pixel_values_videos** -- Pixel values of videos to be fed to a model. Returned when `videos` is not `None`. - **image_grid_thw** -- List of image 3D grid in LLM. Returned when `images` is not `None`. - **video_grid_thw** -- List of video 3D grid in LLM. Returned when `videos` is not `None`. tokenizer_init_kwargsNr. images_kwargsimage_grid_thwr0rrvideo_metadatavideo_grid_thwrz<|placeholder|>aSmolVLM requires frame timestamps to construct prompts, but the `fps` of the input video could not be inferred. Probably `video_metadata` was missing from inputs and you passed pre-sampled frames. Defaulting to `fps=24`. Please provide `video_metadata` for more accurate results.rreturn_tensorsrFimagevideo) modalities input_idsmm_token_type_ids)data tensor_typer) _merge_kwargsrr) init_kwargsr(r*getpop isinstancelistcopy merge_sizerangelenrprodreplacerfpslogger warning_once timestampsappendreplace_frame_token_id_check_special_mm_tokensnparray zeros_likertolistr)r'r.r/r0r+ output_kwargs image_inputsr4 videos_inputsr5r6 merge_lengthindexinum_image_tokens video_index num_framesvideo_structuremetadatarSunique_timestampsidxselected_timestamps frame_idx timestamp_secframe_structurer<r text_inputs array_idsrArrr__call__>s               % zGlm4vProcessor.__call__c  si}|dur7tjdi|ddpjjfdd|D}fdd|D}|||d|dur^tjdi|fd d|D}fd d|D}||d <td i|S) aK Computes the number of placeholder tokens needed for multimodal inputs with the given sizes. Args: image_sizes (`list[list[int]]`, *optional*): The input sizes formatted as (height, width) per each image. video_sizes (`list[list[int]]`, *optional*): The input sizes formatted as (num_frames, height, width) per each video. Returns: `MultiModalData`: A `MultiModalData` object holding number of tokens per each of the provided input modalities, along with other useful data. Nr3rKcs"g|] }jjg|RqSr)r(get_number_of_image_patches).0 image_size)r3r'rr z=Glm4vProcessor._get_num_multimodal_tokens..cg|]}|dqSr7rrp num_patchesrKrrrr)ranum_image_patchesrcs"g|] }jjg|RqSr)r*get_number_of_video_patches)rp video_size)r'rrrrrrscrtrurrvrxrrrrrynum_video_tokensr)rrrFupdater(rKr) r' image_sizes video_sizesr+ vision_datarzranum_video_patchesr}r)r3rKr'rr_get_num_multimodal_tokenss&     z)Glm4vProcessor._get_num_multimodal_tokensTFcKs|jj|f||d|S)a Post-process the output of the model to decode the text. Args: generated_outputs (`torch.Tensor` or `np.ndarray`): The output of the model `generate` function. The output is expected to be a tensor of shape `(batch_size, sequence_length)` or `(sequence_length,)`. skip_special_tokens (`bool`, *optional*, defaults to `True`): Whether or not to remove special tokens in the output. Argument passed to the tokenizer's `batch_decode` method. clean_up_tokenization_spaces (`bool`, *optional*, defaults to `False`): Whether or not to clean up the tokenization spaces. Argument passed to the tokenizer's `batch_decode` method. **kwargs: Additional arguments to be passed to the tokenizer's `batch_decode method`. Returns: `list[str]`: The decoded text. )skip_special_tokensclean_up_tokenization_spaces)r) batch_decode)r'generated_outputsrrr+rrrpost_process_image_text_to_textsz.Glm4vProcessor.post_process_image_text_to_textcCsd|jdt|S)Nz<|begin_of_image|>z<|end_of_image|>)rint)r'rjrrrrUsz%Glm4vProcessor.replace_frame_token_id)NNNN)NNN)NN)TF)rrrr&r rr r rIr rrrrnrrrU __classcell__rrr,rr-s* p' r)numpyrWfeature_extraction_utilsr image_utilsrprocessing_utilsrrrrtokenization_utils_baser r utilsr r video_utilsr get_loggerrrQrr__all__rrrrs      G