o ॵip3@sNddlZddlZddlZddlZddlmZddlmZddlm Z ddl Z ddl Z ddl Z ddlmmZddlmmZddlmZddlmZddlmZddlmZdd lmZmZdd lm Z d d l!m"Z"d d l#m$Z$  dddZ%ddZ&ddZ' dddZ(dddZ)Gddde*Z+e$j,ej-ej.dGddde"Z/dS)N)exists)TemporaryDirectory)urlparse) VideoReader)Compose) http_get_file) Preprocessors)FieldsModeKeys) type_assert) Preprocessor) PREPROCESSORScCst|}|jdvrt|jrt|||}n-t"}tj}t |||ddt j ||}t|||}Wdn1s=wY|durN|} t ||} n |j j} t ||j j} g} t|dD]} t| D]} | jd| | | || qhqbtj| ddS)aw simple interface to load video frames from file Args: cfg (Config): The global config object. video_path (str): video file path num_spatial_crops_override (int): the spatial crops per clip num_temporal_views_override (int): the temporal clips per video Returns: data (Tensor): the normalized video clips for model inputs )fileN)url local_dir file_namecookiesrr dim)rschemerpath _decode_videoruuiduuid4hexrosjoinkinetics400_tranformTESTNUM_SPATIAL_CROPSrangesize transformsset_spatial_indexappendtorchstack)cfg video_pathnum_spatial_crops_overridenum_temporal_views_override url_parseddatatemporary_cache_dir random_strtemp_file_pathnum_spatial_crops transform data_listijr7R/home/ubuntu/.local/lib/python3.10/site-packages/modelscope/preprocessors/video.py ReadVideoDatas<     r9cCsHt|jj|jjg|jj|d}t|tj|jj|jjddg}t |S)aI Configs the transform for the kinetics-400 dataset. We apply controlled spatial cropping and normalization. Args: cfg (Config): The global config object. num_spatial_crops (int): the spatial crops per clip Returns: transform_function (Compose): the transform function for input clips )short_side_range crop_sizer2T)meanstdinplace) KineticsResizedCropDATA TEST_SCALETEST_CROP_SIZEr$ ToTensorVideoNormalizeVideoMEANSTDr)r)r2 resize_videostd_transform_listr7r7r8rEs rc Cs|dkrtd|dg}|S||||} t|| d} |dkr'| d} n |t| |d} |r;| | |} n| | d} t| | |}t|d|d}|S)a Generates the frame index list using interval based sampling. Args: vid_length (int): the length of the whole video (valid selection range). vid_fps (int): the original video fps target_fps (int): the normalized video fps clip_idx (int): -1 for random temporal sampling, and positive values for sampling specific clip from the video num_clips (int): the total clips to be sampled from each video. combined with clip_idx, the sampled video is the "clip_idx-th" video from "num_clips" videos. num_frames (int): number of frames in each sampled clips. interval (int): the interval to sample each frame. minus_interval (bool): control the end index Returns: index (tensor): the sampled frame indexes r r) randomrandintmaxmathfloorr'linspaceclamplong) vid_lengthvid_fps target_fpsclip_idx num_clips num_framesintervalminus_intervalindex clip_lengthmax_idx start_idxend_idxr7r7r8_interval_based_sampling[s  r_c sttsJ|dur|}n|jj}g}t|D]1}tt||jj|||jj |jj |jj }d}t tjfdd|Ddd}||qt |}~ |S)a Decodes the video given the numpy frames. Args: cfg (Config): The global config object. frames_list (list): all frames for a video, the frames should be numpy array. vid_fps (int): the fps of this video. num_temporal_views_override (int): the temporal clips per video Returns: frames (Tensor): video tensor data Ncsg|]}|qSr7r7).0rZ frames_listr7r8 sz-_decode_video_frames_list..r)axis) isinstancelistr NUM_ENSEMBLE_VIEWSr"r_lenr@ TARGET_FPSNUM_INPUT_FRAMES SAMPLING_RATEMINUS_INTERVALr' from_numpynpr(tolistr&) r)rbrSr,num_clips_per_video frame_listrUlist_framesvrr7rar8_decode_video_frames_lists0    ruc Cst|}|dur |}n|jj}g}t|D]U}tt|||jj|||jj |jj |jj }d}| drYt d|dd} t|t | |g}|| jdd}n t||}||qt |}~|S)aK Decodes the video given the numpy frames. Args: cfg (Config): The global config object. path (str): video file path. num_temporal_views_override (int): the temporal clips per video Returns: frames (Tensor): video tensor data Nz.avir)rr rgr"r_rh get_avg_fpsr@rirjrkrlendswithr'arangedlpack from_dlpack get_batchcat to_dlpackcloneshaper&r() r)rr,rtrprqrUrrrs append_listr7r7r8rsJ       rc@s<eZdZdZ dddZddZddZd d Zd d Zd S)r?aAPerform resize and crop for kinetics-400 dataset Args: short_side_range (list): The length of short side range. In inference, this shoudle be [256, 256] crop_size (int): The cropped size for frames. num_spatial_crops (int): The number of the cropped spatial regions in each video. r cCs d|_||_t||_||_dS)N)idxr:intr;r2)selfr:r;r2r7r7r8__init__s  zKineticsResizedCrop.__init__c Csv|j\}}}}|jd}||kr)t|}t|||}tjjj|||fdd}nt|}t|||}tjjj|||fdd}t||j} t||j} |jdkr]| d} | d} nH|jdkr|j dkr}||krr| d} d} n3||kr|d} | d} n(|j dkr| d} | d} n|j dkr||kr| d} | } n ||kr| } | d} |dddd| | |j| | |jfS)zPerform controlled crop for video tensor. Args: clip (Tensor): the video data, the shape is [T, C, H, W] rbilinearr#moder rIN) rr:rr'nn functional interpolater;r2r) rclip_ clip_height clip_widthlengthnew_clip_heightnew_clip_widthnew_clipx_maxy_maxxyr7r7r8_get_controlled_cropsL          ,z(KineticsResizedCrop._get_controlled_cropcCs|j\}}}}t||}t||}ttj|j}t|||}||kr*|} |} n|} |} tjj j || | fdd} t| |j } t| |j } ttd| }ttd| }| dddd|||j |||j fS)Nrrr) rminrLrrJuniformr:r'rrrr;)rrrrr short_side long_sidenew_short_side new_long_siderrrrrrrr7r7r8_get_random_crops$   ,z$KineticsResizedCrop._get_random_cropcCs ||_dS)zSet the spatial cropping index for controlled cropping.. Args: idx (int): the spatial index. The value should be in [0, 1, 2], means [left, center, right], respectively. N)r)rrr7r7r8r%6s z%KineticsResizedCrop.set_spatial_indexcCs ||SN)r)rrr7r7r8__call__=s zKineticsResizedCrop.__call__N)r ) __name__ __module__ __qualname____doc__rrrr%rr7r7r7r8r?s  , r?) module_namecs>eZdZfddZddZddZeeeddZZ S) "MovieSceneSegmentationPreprocessorcsvtj|i||dd|_|tjd|_|tjd|_|dd|_ ddl m }||j|_ ||j|_ dS)z7 movie scene segmentation preprocessor is_trainTN num_keyframerr ) get_transform)superrpoprr TRAINpreprocessor_train_cfgEVALpreprocessor_test_cfgrmovie_scene_segmentationrtrain_transformtest_transform)rargskwargsr __class__r7r8rEs  z+MovieSceneSegmentationPreprocessor.__init__cC d|_dS)NTrrr7r7r8trainTz(MovieSceneSegmentationPreprocessor.traincCr)NFrrr7r7r8evalXrz'MovieSceneSegmentationPreprocessor.evalcCs>|jr|j}n|j}tj||dd}|d|jddd}|S)Nrrrr)rrrr'r(viewr)rresultsr$r7r7r8r\s z+MovieSceneSegmentationPreprocessor.__call__) rrrrrrr objectr __classcell__r7r7rr8rAs  r)NNr)0rMrrJros.pathrtempfiler urllib.parsernumpyrnr'torch.utils.datatorch.utils.dlpackutilsrz(torchvision.transforms._transforms_videor$_transforms_videodecordrtorchvision.transformsrmodelscope.hub.file_downloadrmodelscope.metainformodelscope.utils.constantr r modelscope.utils.type_assertr baser builderrr9rr_rurrr?register_modulecv%movie_scene_segmentation_preprocessorrr7r7r7r8sB           ,/ +.a