o ϯiG@sddlZddlZddlZddlZddlmZddlmZddlm Z m Z m Z m Z m Z mZddlZddlZddlmZddlmZddlmZmZmZmZmZddlmZmZdd lm Z zddl!Z!Wn e"ymd Z#Ynwd Z#d d ddddZ$d ddddddddddddd Z%d dd d!Z&Gd"d#d#eZ'e'j(e$e'j)e%e'j*e&iZ+eGd$d%d%eZ, d;d&e-d'e.d(e e/d)e.fd*d+Z0d,e-d-e-d.e-d/e-d)e ee-e-ff d0d1Z1Gd2d3d3ej2j3j4Z5Gd4d5d5ej2j3j4Z6d if an overlap is found, or None otherwise. N)maxmin)rQrRrSrToverlap_start_timeoverlap_stop_timer:r:r; _get_overlap_for_time_range_pairts   rYc @seZdZdZ  ddedededeeeeefgefde d df d d Z e d eee fd eee effddZd eeeffddZd efddZdeeefd eeeffddZdS)DomsevFrameDatasetz Egocentric video classification frame-based dataset for `DoMSEV `_ This dataset handles the loading, decoding, and configurable sampling for the image frames. NFvideo_data_manifest_file_pathvideo_info_file_pathlabels_file_path transformmultithreaded_iorLcCsp|sJ|sJ|s Jt|||}t|tddd}t|||_||_|j|_ t | |_ t |j dS)a/ Args: video_data_manifest_file_path (str): The path to a json file outlining the available video data for the associated videos. File must be a csv (w/header) with columns: ``{[f.name for f in dataclass_fields(EncodedVideoInfo)]}`` To generate this file from a directory of video frames, see helper functions in module: ``pytorchvideo.data.domsev.utils`` video_info_file_path (str): Path or URI to manifest with basic metadata of each video. File must be a csv (w/header) with columns: ``{[f.name for f in dataclass_fields(VideoInfo)]}`` labels_file_path (str): Path or URI to manifest with temporal annotations for each video. File must be a csv (w/header) with columns: ``{[f.name for f in dataclass_fields(LabelData)]}`` dataset_type (VideoDatasetType): The data format in which dataset video data is stored (e.g. video frames, encoded video etc). transform (Optional[Callable[[Dict[str, Any]], Any]]): This callable is evaluated on the clip output before the clip is returned. It can be used for user-defined preprocessing and augmentations to the clips. The clip output format is described in __next__(). multithreaded_io (bool): Boolean to control whether io operations are performed across multiple threads. r=T list_per_keyN)r _load_imagesrr<rZ_assign_labels_to_frames_labels_per_frame_user_transform_transform_frame _transformlistvalues_framesrandomshuffle)selfr[r\r]r^r_ frames_dict video_labelsr:r:r;__init__s"( zDomsevFrameDataset.__init__rnrocCsPi}|D]\}}||j}|D]}|j|jkr$|j|jkr$|j||<qq|S)a8 Args: frames_dict: The mapping of for all the frames in the dataset. video_labels: The list of temporal labels for each video Also unpacks one label per frame. Also converts them to class IDs and then a tensor. )itemsr= frame_numberr@rArB)rnrolabels_per_frameframe_id image_infolabels_in_videolabelr:r:r;rcs    z+DomsevFrameDataset._assign_labels_to_framescCsB|j|}|j|j}t|j}|j||d}|jr||}|S)a Samples an image frame associated to the given index. Args: index (int): index for the image frame Returns: An image frame with the following format if transform is None. .. code-block:: text {{ 'frame_id': , 'image': , 'label': , }} )rtimagerw)rjrdrt_load_image_from_pathframe_file_pathrg)rmindexframelabel_in_frame image_data frame_datar:r:r; __getitem__s    zDomsevFrameDataset.__getitem__cC t|jS)zK Returns: The number of frames in the dataset. )lenrjrmr:r:r;__len__ zDomsevFrameDataset.__len__r|cC8|D]}||durtg||<q|jr||}|S)a< Transforms a given image frame, according to some pre-defined transforms and an optional user transform function (self._user_transform). Args: clip (Dict[str, Any]): The clip that will be transformed. Returns: (Dict[str, Any]) The transformed clip. Ntorchtensorre)rmr|keyr:r:r;rf  z#DomsevFrameDataset._transform_frame)NF)r4r5r6rDrErrrrboolrp staticmethodr rr<rcrrHrrfr:r:r:r;rZs4  B  "&rZc@seZdZdZejddddfdedededeeee feee e fge e fd ed e d eeeeefgefd eee e ge e fd eddfddZdeeeffddZde fddZdeeefdeeeffddZdS)DomsevVideoDataseta3 Egocentric classification video clip-based dataset for `DoMSEV `_ stored as an encoded video (with frame-level labels). This dataset handles the loading, decoding, and configurable clip sampling for the videos. rNFr[r\r] clip_sampler dataset_typeframes_per_secondr^ frame_filterr_rLc Csj|sJ|sJ|s Jt||| ||_t|tddd|_||j|j|_||_||_|j |_ ||_ dS)ao Args: video_data_manifest_file_path (str): The path to a json file outlining the available video data for the associated videos. File must be a csv (w/header) with columns: ``{[f.name for f in dataclass_fields(EncodedVideoInfo)]}`` To generate this file from a directory of video frames, see helper functions in module: ``pytorchvideo.data.domsev.utils`` video_info_file_path (str): Path or URI to manifest with basic metadata of each video. File must be a csv (w/header) with columns: ``{[f.name for f in dataclass_fields(VideoInfo)]}`` labels_file_path (str): Path or URI to manifest with annotations for each video. File must be a csv (w/header) with columns: ``{[f.name for f in dataclass_fields(LabelData)]}`` clip_sampler (Callable[[Dict[str, Video], Dict[str, List[LabelData]]], List[VideoClipInfo]]): Defines how clips should be sampled from each video. See the clip sampling documentation for more information. dataset_type (VideoDatasetType): The data format in which dataset video data is stored (e.g. video frames, encoded video etc). frames_per_second (int): The FPS of the stored videos. (NOTE: this is variable and may be different than the original FPS reported on the DoMSEV dataset website -- it depends on the preprocessed subsampling and frame extraction). transform (Optional[Callable[[Dict[str, Any]], Any]]): This callable is evaluated on the clip output before the clip is returned. It can be used for user-defined preprocessing and augmentations to the clips. The clip output format is described in __next__(). frame_filter (Optional[Callable[[List[int]], List[int]]]): This callable is evaluated on the set of available frame indices to be included in a sampled clip. This can be used to subselect frames within a clip to be loaded. multithreaded_io (bool): Boolean to control whether io operations are performed across multiple threads. r=Tr`N) r _load_videos_videosrr<_labels_per_video_clips_frames_per_secondre_transform_cliprg _frame_filter) rmr[r\r]rrrr^rr_r:r:r;rp6s(=  zDomsevVideoDataset.__init__cs|j|}|j|j}g|D].}t|j|j|j|j}|dur=|\}}t||j}t||j} t|| D]} |q5qfddtt D} t | } d|ji|j |j|j|j| |j|jd} |jrq|| } | S)a/ Samples a video clip associated to the given index. Args: index (int): index for the video clip. Returns: A video clip with the following format if transform is None. .. code-block:: text {{ 'video_id': , 'video': , 'audio': , 'labels': , 'start_time': , 'stop_time': }} Ncsg|]}|jqSr:)rB).0ilabels_in_clipr:r; sz2DomsevVideoDataset.__getitem__..r=)labelsr>r?)rrr=rYr>r?rPrrangeappendrrrrget_cliprg)rmr{cliprv label_dataoverlap_periodrWrXoverlap_start_frameoverlap_stop_frame_ label_idslabel_ids_tensor clip_datar:rr;rsD     zDomsevVideoDataset.__getitem__cCr)zP Returns: The number of video clips in the dataset. )rrrr:r:r;rrzDomsevVideoDataset.__len__rcCr)a; Transforms a given video clip, according to some pre-defined transforms and an optional user transform function (self._user_transform). Args: clip (Dict[str, Any]): The clip that will be transformed. Returns: (Dict[str, Any]) The transformed clip. Nr)rmrrr:r:r;rrz"DomsevVideoDataset._transform_clip)r4r5r6rDrFramerErrrrr<rrHrrrrprrrr:r:r:r;r,sB"     YD&rr. image_path num_retriesc Cststdd}t|D]I}t|d }t|tj}t j |t j d}t |t j }Wdn1s7wY|durD|}ntd|d|dtdq |duratd |t|}|S) a Loads the given image path using PathManager and decodes it as an RGB image. Args: image_path (str): the path to the image. num_retries (int): number of times to retry image reading to handle transient error. Returns: A PIL Image of the image RGB data with shape: (channel, height, width). The frames are of type np.uint8 and in the range [0 - 255]. Raises an exception if unable to load images. zVopencv2 is required to use FrameVideo. Please install with 'pip install opencv-python'Nrb)flagszReading attempt /z failed.gư>zFailed to load image from {})_HAS_CV2 ImportErrorrr opennp frombufferreaduint8cv2imdecode IMREAD_COLORcvtColor COLOR_BGR2RGBloggingwarningtimesleep Exceptionformatr fromarray) rrimg_arrrfimg_strimg_bgrimg_rgb pil_imager:r:r;rys(    ry)T)r.)9rrMrkr dataclassesrenumrtypingrrrrrr numpyrriopath.common.file_ior PILr (pytorchvideo.data.dataset_manifest_utilsr r rrrpytorchvideo.data.utilsrrpytorchvideo.data.videorrrrUSER_ENVIRONMENT_MAPUSER_ACTIVITY_MAPUSER_ATTENTION_MAPr3r7r8r9LABEL_TYPE_2_MAPr<rGrHrrPrYutilsdataDatasetrZrrEryr:r:r:r;s           %D