o
ϯin/ @ s d dl mZ d dlmZ d dlmZmZmZm Z m
Z
d dl Z d dl m
Z
mZmZmZmZ d dlmZmZ d dlmZ G dd d eZG d
d d eZdS ) )fields)Enum)AnyCallableDictListOptionalN)EncodedVideoInfo
VideoClipInfoVideoDatasetTypeVideoFrameInfo VideoInfo)
ActionDataEpicKitchenDataset)Videoc @ s e Zd ZdZdS ) ClipSampling N)__name__
__module__ __qualname__Random r r ^/home/ubuntu/.local/lib/python3.10/site-packages/pytorchvideo/data/epic_kitchen_forecasting.pyr s r c sT e Zd ZdZejejdddddddf deded ed
ed ed e d
e de
de e
de
de e
eef gef def fdd
Zede e
eef ge
eef f de
de
de
de e
eef ge
eef f f
ddZede
d e d
e de
de ee
gee
f f
ddZed
ed e d
e de
de
de e
eef e
eee f gee f f ddZ ZS ) EpicKitchenForecastingz
Action forecasting video data set for EpicKitchen-55 Dataset.
This dataset handles the loading, decoding, and clip sampling for the videos.
g @g $@r NTvideo_info_file_pathactions_file_path video_data_manifest_file_path
clip_sampling dataset_typeseconds_per_clipclip_time_stridenum_input_clipsframes_per_clipnum_forecast_actions transformmultithreaded_ioc
s ddd t tD ddd t tD ddd t tD dd d t tD d
t|||||
}
| d ur@t| |||nd }t| |
| |} t j
||||| ||
| d d S ) Nz
Args:
video_info_file_path (str):
Path or URI to manifest with basic metadata of each video.
File must be a csv (w/header) with columns:
c S g | ]}|j qS r name.0fr r r
3 z3EpicKitchenForecasting.__init__..z
actions_file_path (str):
Path or URI to manifest with action annotations for each video.
File must ber a csv (w/header) with columns:
c S r& r r' r) r r r r, 8 r- a
video_data_manifest_file_path (str):
The path to a json file outlining the available video data for the
associated videos. File must be a csv (w/header) with columns either:
For Frame Videos:
c S r& r r' r) r r r r, ? r- z6
For Encoded Videos:
c S r& r r' r) r r r r, B r- a
To generate this file from a directory of video frames, see helper
functions in Module: pytorchvideo.data.epic_kitchen.utils
clip_sampling (ClipSampling):
The type of sampling to perform to perform on the videos of the dataset.
dataset_type (VideoDatasetType): The dataformat in which dataset
video data is store (e.g. video frames, encoded video etc).
seconds_per_clip (float): The length of each sampled subclip in seconds.
clip_time_stride (float): The time difference in seconds between the start of
each input subclip.
num_input_clips (int): The number of subclips to be included in the input
video data.
frames_per_clip (Optional[int]): The number of frames per clip to sample.
If None, all frames in the clip will be included.
num_forecast_actions (int): The number of actions to be included in the
action vector.
transform (Callable[[Dict[str, Any]], Any]):
This callable is evaluated on the clip output before the clip is returned.
It can be used for user-defined preprocessing and augmentations to the clips.
The clip input is a dictionary with the following format:
{
'video_id': ,
'video': ,
'audio': ,
'label': ,
'start_time': ,
'stop_time':
}
If transform is None, the raw clip output in the above format is
returned unmodified.
multithreaded_io (bool):
Boolean to control whether parllelizable io operations are performed across
multiple threads.
)r r r r r$ frame_filter clip_samplerr% ) dataclass_fieldsr
r r r r _define_clip_structure_generator_frame_filter_generator_transform_generatorsuper__init__)selfr r r r r r r r! r" r# r$ r% define_clip_structure_fnr. __class__r r r5 sH
B
z EpicKitchenForecasting.__init__returnc s0 dt ttf dt ttf f fdd }|S )a]
Args:
transform (Callable[[Dict[str, Any]], Dict[str, Any]]): A function that performs
any operation on a clip before it is returned in the default transform function.
num_forecast_actions: (int) The number of actions to be included in the
action vector.
frames_per_clip (int): The number of frames per clip to sample.
num_input_clips (int): The number of subclips to be included in the video data.
Returns:
A function that performs any operation on a clip and returns the transformed clip.
clipr: c s t fddtt d d D sJ d fdd d D d }| d< d d ks8J t fd dtD }| d< D ]} | d u r\tg |< qMrc S )
Nc 3 s0 | ]} d | j d |d j kV qdS )actionsr N
start_timer* ir; r r s
zVEpicKitchenForecasting._transform_generator..transform_clip..r< r zActions must be sortedc s g | ] }|j d kr|qS ) stop_timer= )r* arA r r r, s zWEpicKitchenForecasting._transform_generator..transform_clip..videoc s< g | ]} d dd| |d ddddf qS )rE Nr r r? )r; r" r r r, s &)allrangelensizetorchstacktensor)r; next_k_actionsclip_video_tensorkeyr" r# r! r$ rA r transform_clip s2
zCEpicKitchenForecasting._transform_generator..transform_clip)r strr )r$ r# r" r! rQ r rP r r3 s , z+EpicKitchenForecasting._transform_generatorc sB |d | dt t dt t f fdd }|S )a
Args:
frames_per_clip (int): The number of frames per clip to sample.
seconds_per_clip (float): The length of each sampled subclip in seconds.
clip_time_stride (float): The time difference in seconds between the start of
each input subclip.
num_input_clips (int): The number of subclips to be included in the video data.
Returns:
A function that takes in a list of frame indicies and outputs a subsampled list.
r
frame_indicesr: c sv t | }| }t| }t tD ]}t| | }tD ] } ||| q#q fddt| D S )Nc s g | ]
\}}| v r|qS r r )r* r@ xselected_framesr r r, s zXEpicKitchenForecasting._frame_filter_generator..frame_filter..)rH intsetrG add enumerate)rS "num_available_frames_for_all_clipsavailable_frames_per_secondintra_clip_sampling_strider@ clip_start_indexjr desired_frames_per_secondr" r! time_window_lengthrU r r. s"
zDEpicKitchenForecasting._frame_filter_generator..frame_filter)r rW )r" r r r! r. r r` r r2 s &z.EpicKitchenForecasting._frame_filter_generatorc sj | t jkstdt j d| d||d | dtttf dtttt f dtt f fdd }|S )
a
Args:
clip_sampling (ClipSampling):
The type of sampling to perform to perform on the videos of the dataset.
seconds_per_clip (float): The length of each sampled clip in seconds.
clip_time_stride: The time difference in seconds between the start of
each input subclip.
num_input_clips (int): The number of subclips to be included in the video data.
num_forecast_actions (int): The number of actions to be included in the
action vector.
Returns:
A function that takes a dictionary of videos and outputs a list of sampled
clips.
zOnly z is implemented. z not implemented.r videos
video_actionsr: c s g }| D ]G\}}t|d d D ]8\}}d}t|d t|D ](}|| j|jkr1|d7 }| krK|j dkrI|t||j |j nq#qq|S )Nr r )itemsrZ rG rH r> rC appendr
) rc rd candidate_sample_clipsvideo_idr< r@ actionnumber_valid_actionsr_ r# rb r r define_clip_structure s* zVEpicKitchenForecasting._define_clip_structure_generator..define_clip_structure) r r NotImplementedErrorr rR r r r r
)r r r r! r# rm r rl r r1 s
z7EpicKitchenForecasting._define_clip_structure_generator)r r r __doc__r r r FramerR floatrW r r r r boolr5 staticmethodr3 r r2 r r r
r1
__classcell__r r r8 r r s
o 3 )&r ) dataclassesr r0 enumr typingr r r r r rJ (pytorchvideo.data.dataset_manifest_utilsr r
r r r
pytorchvideo.data.epic_kitchenr r pytorchvideo.data.videor r r r r r r s