o i@sddlZddlZddlZddlmZddlmZddlmZm Z m Z m Z ddl Z ddlmZddlmZmZddlmZdd lmZmZmZdd lmZmZdd lmZmZmZm Z m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)dd l*m+Z+dd l,m-Z-ddl.m/Z/m0Z0m1Z1m2Z2m3Z3m4Z4m5Z5m6Z6m7Z7e&rddl8Z8e(rddl9m:Z;e)dZ?e!de?e-ddGdddeZ@e"e@jAe@_Ae@jAjBdure@jAjBjCdddde@jA_BdSdS)N)deepcopy)partial)AnyCallableOptionalUnion)custom_object_save) BatchFeature get_size_dict)BaseImageProcessorFast)ChannelDimensionSizeDictvalidate_kwargs)Unpack VideosKwargs) IMAGE_PROCESSOR_NAMEPROCESSOR_NAMEVIDEO_PROCESSOR_NAME TensorTypeadd_start_docstrings copy_func download_urlis_offline_mode is_remote_urlis_torch_availableis_torchcodec_availableis_torchvision_v2_availablelogging cached_file)requires) VideoInput VideoMetadatagroup_videos_by_shapeis_valid_video load_videomake_batched_metadatamake_batched_videosreorder_videosto_channel_dimension_format) functionala Args: do_resize (`bool`, *optional*, defaults to `self.do_resize`): Whether to resize the video's (height, width) dimensions to the specified `size`. Can be overridden by the `do_resize` parameter in the `preprocess` method. size (`dict`, *optional*, defaults to `self.size`): Size of the output video after resizing. Can be overridden by the `size` parameter in the `preprocess` method. size_divisor (`int`, *optional*, defaults to `self.size_divisor`): The size by which to make sure both the height and width can be divided. default_to_square (`bool`, *optional*, defaults to `self.default_to_square`): Whether to default to a square video when resizing, if size is an int. resample (`PILImageResampling`, *optional*, defaults to `self.resample`): Resampling filter to use if resizing the video. Only has an effect if `do_resize` is set to `True`. Can be overridden by the `resample` parameter in the `preprocess` method. do_center_crop (`bool`, *optional*, defaults to `self.do_center_crop`): Whether to center crop the video to the specified `crop_size`. Can be overridden by `do_center_crop` in the `preprocess` method. crop_size (`dict[str, int]` *optional*, defaults to `self.crop_size`): Size of the output video after applying `center_crop`. Can be overridden by `crop_size` in the `preprocess` method. do_rescale (`bool`, *optional*, defaults to `self.do_rescale`): Whether to rescale the video by the specified scale `rescale_factor`. Can be overridden by the `do_rescale` parameter in the `preprocess` method. rescale_factor (`int` or `float`, *optional*, defaults to `self.rescale_factor`): Scale factor to use if rescaling the video. Only has an effect if `do_rescale` is set to `True`. Can be overridden by the `rescale_factor` parameter in the `preprocess` method. do_normalize (`bool`, *optional*, defaults to `self.do_normalize`): Whether to normalize the video. Can be overridden by the `do_normalize` parameter in the `preprocess` method. Can be overridden by the `do_normalize` parameter in the `preprocess` method. image_mean (`float` or `list[float]`, *optional*, defaults to `self.image_mean`): Mean to use if normalizing the video. This is a float or list of floats the length of the number of channels in the video. Can be overridden by the `image_mean` parameter in the `preprocess` method. Can be overridden by the `image_mean` parameter in the `preprocess` method. image_std (`float` or `list[float]`, *optional*, defaults to `self.image_std`): Standard deviation to use if normalizing the video. This is a float or list of floats the length of the number of channels in the video. Can be overridden by the `image_std` parameter in the `preprocess` method. Can be overridden by the `image_std` parameter in the `preprocess` method. do_convert_rgb (`bool`, *optional*, defaults to `self.image_std`): Whether to convert the video to RGB. video_metadata (`VideoMetadata`, *optional*): Metadata of the video containing information about total duration, fps and total number of frames. do_sample_frames (`int`, *optional*, defaults to `self.do_sample_frames`): Whether to sample frames from the video before processing or to process the whole video. num_frames (`int`, *optional*, defaults to `self.num_frames`): Maximum number of frames to sample when `do_sample_frames=True`. fps (`int` or `float`, *optional*, defaults to `self.fps`): Target frames to sample per second when `do_sample_frames=True`. return_tensors (`str` or `TensorType`, *optional*): Returns stacked tensors if set to `pt, otherwise returns a list of tensors. data_format (`ChannelDimension` or `str`, *optional*, defaults to `ChannelDimension.FIRST`): The channel dimension format for the output video. Can be one of: - `"channels_first"` or `ChannelDimension.FIRST`: video in (num_channels, height, width) format. - `"channels_last"` or `ChannelDimension.LAST`: video in (height, width, num_channels) format. - Unset: Use the channel dimension format of the input video. input_data_format (`ChannelDimension` or `str`, *optional*): The channel dimension format for the input video. If unset, the channel dimension format is inferred from the input video. Can be one of: - `"channels_first"` or `ChannelDimension.FIRST`: video in (num_channels, height, width) format. - `"channels_last"` or `ChannelDimension.LAST`: video in (height, width, num_channels) format. - `"none"` or `ChannelDimension.NONE`: video in (height, width) format. device (`torch.device`, *optional*): The device to process the videos on. If unset, the device is inferred from the input videos. return_metadata (`bool`, *optional*): Whether to return video metadata or not. z!Constructs a base VideoProcessor.)vision torchvision)backendscs.eZdZdZdZdZdZdZdZdZ dZ dZ dZ dZ dZdZdZdZdZdZdZdZeZdgZdeeddffdd Zdefd d Zd d defddZ  dTdede e!de e"e!e#ffddZ$  dTdede"ee%fde e&de e'de(d f ddZ)  dTdede e"e*e+fde e*de(d fddZ,e-e.dedeedefdd Z/ dUde(d d!e&d"e&d#e0d$e d%d&e&d'e0d(e&d)e#d*e&d+e e"e#e(e#fd,e e"e#e(e#fd-e e"e*e1fdefd.d/Z2e3     0dVd1e"e*e4j5fd2e e"e*e4j5fd3e&d4e&d5e e"e*e&fd6e*f d7d8Z6dWd9e"e*e4j5fd:e&fd;d<Z7e3d1e"e*e4j5fde8e%e*e9fe%e*e9fffd=d>Z:e3d?e%e*e9ffd@dAZ;de%e*e9ffdBdCZdIdJZ?e3dKe"e*e4j5ffdLdMZ@e3dXdOdPZAdUdQe"e*e(e*e(e(e*ffdRdSZBZCS)YBaseVideoProcessorNTgp?Fpixel_values_videoskwargsreturnc s&t|dd|_|D](\}}zt|||Wqty8}ztd|d|d||d}~ww|d|j }|durOt ||d|j dnd|_ |d|j }|durct |dd nd|_ t |jj|_|jD]}||durt||||qrt||tt||dqrdS) Nprocessor_classz Can't set z with value z for sizedefault_to_square)r4r5 crop_size) param_name)super__init__pop_processor_classitemssetattrAttributeErrorloggererrorr4r r5r6list valid_kwargs__annotations__keysmodel_valid_processing_keysgetrgetattr)selfr1keyvalueerrr4r6 __class__W/home/ubuntu/.local/lib/python3.10/site-packages/transformers/video_processing_utils.pyr9s.  zBaseVideoProcessor.__init__cKs|j|fi|SN) preprocess)rHvideosr1rNrNrO__call__szBaseVideoProcessor.__call__videoz torch.TensorcCst|}|jddks|ddddddfdks|S|ddddddfd}d|ddddddfd|ddddddf|dddddddf}|S)z Converts a video to RGB format. Args: video (`"torch.Tensor"`): The video to convert. Returns: `torch.Tensor`: The converted video. .Ngo@r)Fgrayscale_to_rgbshapeany)rHrTalpharNrNrOconvert_to_rgbs .Tz!BaseVideoProcessor.convert_to_rgbmetadata num_framesfpscKs|dur |dur td|dur|n|j}|dur|n|j}|j}|dur?|dur?|dus2|jdur6tdt||j|}||krNtd|d|d|dur_td|||}|Std|}|S)a% Default sampling function which uniformly samples the desired number of frames between 0 and total number of frames. If `fps` is passed along with metadata, `fps` frames per second are sampled uniformty. Arguments `num_frames` and `fps` are mutually exclusive. Args: metadata (`VideoMetadata`): Metadata of the video containing information about total duration, fps and total number of frames. num_frames (`int`, *optional*): Maximum number of frames to sample. Defaults to `self.num_frames`. fps (`int` or `float`, *optional*): Target frames to sample per second. Defaults to `self.fps`. Returns: np.ndarray: Indices to sample video frames. Nzc`num_frames`, `fps`, and `sample_indices_fn` are mutually exclusive arguments, please use only one!zAsked to sample `fps` frames per second but no video metadata was provided which is required when sampling with `fps`. Please pass in `VideoMetadata` object or use a fixed `num_frames` per input videoz(Video can't be sampled. The `num_frames=z` exceeds `total_num_frames=z`. r) ValueErrorr_r`total_num_framesinttorcharange)rHr^r_r`r1rbindicesrNrNrO sample_framess,z BaseVideoProcessor.sample_framesrRvideo_metadatado_sample_framessample_indices_fnc Cst|}t||d}t|dr<|r:szKBaseVideoProcessor._decode_and_sample_videos...rdim)rdstack)rlimagesrNrNrOrn9sz@BaseVideoProcessor._decode_and_sample_videos..zUSampling frames from a list of images is not supported! Set `do_sample_frames=False`.rj) r(r'r%zipframes_indicesappend isinstancerA fetch_imagesra fetch_videos) rHrRrhrirjsampled_videossampled_metadatarTr^rfrNrNrO_decode_and_sample_videoss2    z,BaseVideoProcessor._decode_and_sample_videosinput_data_formatdevicecCsVg}|D]$}t|tjrt|tj|}t|}|dur#| |}| |q|S)z: Prepare the input videos for processing. N) rwnpndarrayr*r FIRSTrd from_numpy contiguoustorv)rHrRr}r~processed_videosrTrNrNrO_prepare_input_videosFs    z(BaseVideoProcessor._prepare_input_videosc Kst|t|jjdgd|jjD] }||t||dq|d}|d}|d}|d}|rAt|j fi|nd}|j ||||d\}}|j |||d}|j d i|}|j d i||d |d } |jd d |i|} | r|| d<| S) Nreturn_tensors)captured_kwargsvalid_processor_keysr}rir~rh)rhrirj)rRr}r~ data_formatreturn_metadatarRrN)rrDrArBrC setdefaultrGr:rrgr|r_further_process_kwargs_validate_preprocess_kwargs _preprocess) rHrRr1 kwarg_namer}rir~rhrjrpreprocessed_videosrNrNrOrQ]s4        zBaseVideoProcessor.preprocessdo_convert_rgb do_resizer4 interpolationzF.InterpolationModedo_center_cropr6 do_rescalerescale_factor do_normalize image_mean image_stdrc Kst|\}}i}|D]\}}|r||}|r!|j|||d}|||<q t||}t|\}}i}|D]\}}|rC|||}|||| | | | }|||<q7t||}| r`tj|ddn|}t d|i| dS)N)r4rrror0)data tensor_type) r$r<r]resizer) center_croprescale_and_normalizerdrqr )rHrRrrr4rrr6rrrrrrr1grouped_videosgrouped_videos_indexresized_videos_groupedrZstacked_videosresized_videosprocessed_videos_groupedrrNrNrOrs*         zBaseVideoProcessor._preprocessmainpretrained_model_name_or_path cache_dirforce_downloadlocal_files_onlytokenrevisionc Ks||d<||d<||d<||d<|dd}|dur*tdt|dur(td|}|dur2||d <|j|fi|\} }|j| fi|S) a Instantiate a type of [`~video_processing_utils.VideoProcessorBase`] from an video processor. Args: pretrained_model_name_or_path (`str` or `os.PathLike`): This can be either: - a string, the *model id* of a pretrained video hosted inside a model repo on huggingface.co. - a path to a *directory* containing a video processor file saved using the [`~video_processing_utils.VideoProcessorBase.save_pretrained`] method, e.g., `./my_model_directory/`. - a path or url to a saved video processor JSON *file*, e.g., `./my_model_directory/video_preprocessor_config.json`. cache_dir (`str` or `os.PathLike`, *optional*): Path to a directory in which a downloaded pretrained model video processor should be cached if the standard cache should not be used. force_download (`bool`, *optional*, defaults to `False`): Whether or not to force to (re-)download the video processor files and override the cached versions if they exist. resume_download: Deprecated and ignored. All downloads are now resumed by default when possible. Will be removed in v5 of Transformers. proxies (`dict[str, str]`, *optional*): A dictionary of proxy servers to use by protocol or endpoint, e.g., `{'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}.` The proxies are used on each request. token (`str` or `bool`, *optional*): The token to use as HTTP bearer authorization for remote files. If `True`, or not specified, will use the token generated when running `hf auth login` (stored in `~/.huggingface`). revision (`str`, *optional*, defaults to `"main"`): The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so `revision` can be any identifier allowed by git. To test a pull request you made on the Hub, you can pass `revision="refs/pr/"`. return_unused_kwargs (`bool`, *optional*, defaults to `False`): If `False`, then this function returns just the final video processor object. If `True`, then this functions returns a `Tuple(video_processor, unused_kwargs)` where *unused_kwargs* is a dictionary consisting of the key/value pairs whose keys are not video processor attributes: i.e., the part of `kwargs` which has not been used to update `video_processor` and is otherwise ignored. subfolder (`str`, *optional*, defaults to `""`): In case the relevant files are located inside a subfolder of the model repo on huggingface.co, you can specify the folder name here. kwargs (`dict[str, Any]`, *optional*): The values in kwargs of any keys which are video processor attributes will be used to override the loaded values. Behavior concerning key/value pairs whose keys are *not* video processor attributes is controlled by the `return_unused_kwargs` keyword parameter. Returns: A video processor of type [`~video_processing_utils.ImagVideoProcessorBase`]. Examples: ```python # We can't instantiate directly the base class *VideoProcessorBase* so let's show the examples on a # derived class: *LlavaOnevisionVideoProcessor* video_processor = LlavaOnevisionVideoProcessor.from_pretrained( "llava-hf/llava-onevision-qwen2-0.5b-ov-hf" ) # Download video_processing_config from huggingface.co and cache. video_processor = LlavaOnevisionVideoProcessor.from_pretrained( "./test/saved_model/" ) # E.g. video processor (or model) was saved using *save_pretrained('./test/saved_model/')* video_processor = LlavaOnevisionVideoProcessor.from_pretrained("./test/saved_model/video_preprocessor_config.json") video_processor = LlavaOnevisionVideoProcessor.from_pretrained( "llava-hf/llava-onevision-qwen2-0.5b-ov-hf", do_normalize=False, foo=False ) assert video_processor.do_normalize is False video_processor, unused_kwargs = LlavaOnevisionVideoProcessor.from_pretrained( "llava-hf/llava-onevision-qwen2-0.5b-ov-hf", do_normalize=False, foo=False, return_unused_kwargs=True ) assert video_processor.do_normalize is False assert unused_kwargs == {"foo": False} ```rrrruse_auth_tokenNrThe `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.V`token` and `use_auth_token` are both specified. Please set only the argument `token`.r)r:warningswarn FutureWarningraget_video_processor_dict from_dict) clsrrrrrrr1rvideo_processor_dictrNrNrOfrom_pretraineds&Z z"BaseVideoProcessor.from_pretrainedsave_directory push_to_hubc Ks|dd}|durtdt|ddurtd||d<tj|r-t d|dtj |dd |rW|d d}|d | tjj d }|j |fi|}||}|jdurct|||d tj|t}||td||r|j|||||dd|gS)aq Save an video processor object to the directory `save_directory`, so that it can be re-loaded using the [`~video_processing_utils.VideoProcessorBase.from_pretrained`] class method. Args: save_directory (`str` or `os.PathLike`): Directory where the video processor JSON file will be saved (will be created if it does not exist). push_to_hub (`bool`, *optional*, defaults to `False`): Whether or not to push your model to the Hugging Face model hub after saving it. You can specify the repository you want to push to with `repo_id` (will default to the name of `save_directory` in your namespace). kwargs (`dict[str, Any]`, *optional*): Additional key word arguments passed along to the [`~utils.PushToHubMixin.push_to_hub`] method. rNrrrzProvided path (z#) should be a directory, not a fileT)exist_okcommit_messagerepo_id)configzVideo processor saved in )rr)r:rrrrFraospathisfileAssertionErrormakedirssplitsep _create_repo_get_files_timestamps _auto_classr joinr to_json_filer?info_upload_modified_files) rHrrr1rrrfiles_timestampsoutput_video_processor_filerNrNrOsave_pretrained(sB      z"BaseVideoProcessor.save_pretrainedc sD|dd|dd|dd|dd|dd |dd}|d d|d d|d d |d d}|dd}|durVtdt durTtd| d|d |durc| d<trosotddtt j }t j r}d}nFt r}t}n;t}z f ddtttfD} | d}Wntytytdddtdwz(t|ddd } | } Wdn1swYt| } | d | } Wntjytd!|d"w|rtd#|| |fStd#|d$|| |fS)%a From a `pretrained_model_name_or_path`, resolve to a dictionary of parameters, to be used for instantiating a video processor of type [`~video_processing_utils.VideoProcessorBase`] using `from_dict`. Parameters: pretrained_model_name_or_path (`str` or `os.PathLike`): The identifier of the pre-trained checkpoint from which we want the dictionary of parameters. subfolder (`str`, *optional*, defaults to `""`): In case the relevant files are located inside a subfolder of the model repo on huggingface.co, you can specify the folder name here. Returns: `tuple[Dict, Dict]`: The dictionary(ies) that will be used to instantiate the video processor object. rNrFresume_downloadproxiesrrrr subfolder_from_pipeline _from_autorrvideo processor) file_typefrom_auto_classusing_pipelinez+Offline mode: forcing local_files_only=TrueTcs8g|]}t| dd durqS)F) filenamerrrrrr user_agentrr%_raise_exceptions_for_missing_entriesNr)rlr rrrrr resolved_filerrrrrrNrOrns(z?BaseVideoProcessor.get_video_processor_dict..rz Can't load video processor for 'z'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'z2' is the correct path to a directory containing a z filerutf-8encodingvideo_processorz"It looks like the config file at 'z' is not a valid JSON file.zloading configuration file z from cache at )r:rrrrarr?rstrrrisdirrrrrrrOSError ExceptionopenreadjsonloadsrFJSONDecodeError) rrr1r from_pipelineris_localresolved_video_processor_filevideo_processor_fileresolved_video_processor_filesreadertextrrNrrOres                        z+BaseVideoProcessor.get_video_processor_dictrcKs|}|dd}d|vrd|vr|d|d<d|vr(d|vr(|d|d<|di|}g}|D]\}}t||rIt|||||q5|D]}||dqLtd||rc||fS|S)a Instantiates a type of [`~video_processing_utils.VideoProcessorBase`] from a Python dictionary of parameters. Args: video_processor_dict (`dict[str, Any]`): Dictionary that will be used to instantiate the video processor object. Such a dictionary can be retrieved from a pretrained checkpoint by leveraging the [`~video_processing_utils.VideoProcessorBase.to_dict`] method. kwargs (`dict[str, Any]`): Additional parameters from which to initialize the video processor object. Returns: [`~video_processing_utils.VideoProcessorBase`]: The video processor object instantiated from those parameters. return_unused_kwargsFr4r6NzVideo processor rN)copyr:r<hasattrr=rvr?r)rrr1rr to_removerIrJrNrNrOrs&    zBaseVideoProcessor.from_dictcCs2t|j}|dd|dd|jj|d<|S)z Serializes this instance to a Python dictionary. Returns: `dict[str, Any]`: Dictionary of all the attributes that make up this video processor instance. rEN_valid_kwargs_namesvideo_processor_type)r__dict__r:rM__name__)rHoutputrNrNrOto_dicts    zBaseVideoProcessor.to_dictcCsb|}|D]\}}t|tjr|||<q|dd}|dur'||d<tj|ddddS)z Serializes this instance to a JSON string. Returns: `str`: String containing all the attributes that make up this feature_extractor instance in JSON format. r;Nr3T)indent sort_keys ) rr<rwrrtolistr:rdumps)rH dictionaryrIrJr;rNrNrOto_json_strings   z!BaseVideoProcessor.to_json_stringjson_file_pathcCsBt|ddd}||WddS1swYdS)z Save this instance to a JSON file. Args: json_file_path (`str` or `os.PathLike`): Path to the JSON file in which this image_processor instance's parameters will be saved. wrrN)rwriter)rHrwriterrNrNrOr+s"zBaseVideoProcessor.to_json_filecCs|jjd|S)N )rMrr)rHrNrNrO__repr__6szBaseVideoProcessor.__repr__ json_filecCsNt|ddd }|}Wdn1swYt|}|di|S)a Instantiates a video processor of type [`~video_processing_utils.VideoProcessorBase`] from the path to a JSON file of parameters. Args: json_file (`str` or `os.PathLike`): Path to the JSON file containing the parameters. Returns: A video processor of type [`~video_processing_utils.VideoProcessorBase`]: The video_processor object instantiated from that JSON file. rrrNrN)rrrr)rr rrrrNrNrOfrom_json_file9s   z!BaseVideoProcessor.from_json_fileAutoVideoProcessorcCsDt|ts|j}ddlmm}t||st|d||_dS)a  Register this class with a given auto class. This should only be used for custom video processors as the ones in the library are already mapped with `AutoVideoProcessor `. This API is experimental and may have some slight breaking changes in the next releases. Args: auto_class (`str` or `type`, *optional*, defaults to `"AutoVideoProcessor "`): The auto class to register this new video processor with. rNz is not a valid auto class.) rwrrtransformers.models.automodelsautorrar)r auto_class auto_modulerNrNrOregister_for_auto_classLs   z*BaseVideoProcessor.register_for_auto_classvideo_url_or_urlscsLd}ts tdd}t|trttfdd|DSt||dS)z Convert a single or a list of urls into the corresponding `np.array` objects. If a single url is passed, the return value will be a single object. If a list is passed a list of objects is returned. torchcodecz`torchcodec` is not installed and cannot be used to decode the video by default. Falling back to `torchvision`. Note that `torchvision` decoding is deprecated and will be removed in future versions. r-csg|] }j|dqS)rs)ry)rlxrjrHrNrOrnvsz3BaseVideoProcessor.fetch_videos..)backendrj)rrrrwrArtr&)rHrrjrrNrrOryfs zBaseVideoProcessor.fetch_videos)NNrP)NFFNr)F)r)Dr __module__ __qualname__rresamplerrr4 size_divisorr5r6rrrrrrrir`r_rhrrrBmodel_input_namesrr9r rSr"r]r#rrcrfloatrgdictboolrrAr|rr rrBASE_VIDEO_PROCESSOR_DOCSTRINGrQrrr classmethodrPathLikerrtuplerrrrrrr r rry __classcell__rNrNrLrOr/s  9  +  6      .  q=  u,  .r/rrzvideo processor file)object object_class object_files)Drrrrr functoolsrtypingrrrrnumpyrdynamic_module_utilsr image_processing_utilsr r image_processing_utils_fastr image_utilsr rrprocessing_utilsrrutilsrrrrrrrrrrrrr utils.hubr utils.import_utilsr! video_utilsr"r#r$r%r&r'r(r)r*rdtorchvision.transforms.v2r+rX get_loggerrr?r"r/r__doc__formatrNrNrNrOsN    <  ,   D j