o ϯi.@sddlZddlZddlZddlZddlmZddlZddlmZddl m Z ddl m Z ddlmZddlm Z ddlmZdZdd Zd d Zd d ZddZddZ       d&ddZddZd'ddZGdd d ejZGd!d"d"Z  # d(d$d%ZdS))N)Image) transforms)ConstantClipsPerVideoSampler) EncodedVideo)SimpleTokenizer cCs tddS)N imagebindz bpe/bpe_simple_vocab_16e6.txt.gz) pkg_resourcesresource_filenamer r B/home/ubuntu/.local/lib/python3.10/site-packages/imagebind/data.pyreturn_bpe_pathsr c Cs||8}tjjj|d|dd|ddtd }|dd}|d}||}t||d kr4t d |||dkrFt j j j|d|fd dd }n|dkrT|ddd|f}|d}|S) NTFhanning) htk_compatsample_frequency use_energy window_type num_mel_binsdither frame_length frame_shiftrg?zhLarge gap between audio n_frames(%d) and target_length (%d). Is the audio_target_length setting correct?constant)modevalue)mean torchaudio compliancekaldifbankDEFAULT_AUDIO_FRAME_SHIFT_MS transposesizeabsloggingwarningtorchnn functionalpad unsqueeze)waveform sample_rater target_lengthr!n_framespr r r waveform2melspec s6   r2cCs>g}d}d}|s|||dd\}}}}}|||f|r|S)NFr) annotation)append) clip_samplerdurationall_clips_timepoints is_last_clipendstart_r r r get_clip_timepointsGsr<c Cs|durdSg}ttjdtjjdtdttjdddg}|D]+}t|d}t | d}Wdn1s>wY|| |}| |q$t j|dd S) N) interpolationg3<4'?gwg M?gy{ ?gB91?gwt.?g U?rstdrbRGBrdim)rComposeResizeInterpolationModeBICUBIC CenterCropToTensor Normalizeopenrconverttor4r(stack) image_pathsdevice image_outputsdata_transform image_pathfopenimager r r load_and_transform_vision_dataRs(  rYcs>|durdSttdfdd|D}tj|dd}|S)N)bpe_pathcs g|] }|dqS)r)r,rP).0trS tokenizerr r qs z+load_and_transform_text..rrE)rr r(cat)textrStokensr r]r load_and_transform_textms  rc>On-F"@c s|durdSg} t||d} |D]f} t| \} } || kr&tjj| | |d} t| | d|}g}|D]$}| ddt|d|t|d|f}t||||}| |q4t j ||dfdd|D}t j |dd}| |qt j | ddS) N clip_durationclips_per_video) orig_freqnew_freqrrrAcsg|] }|qSr )rP)r[acrS normalizer r r_sz1load_and_transform_audio_data..rE)rrloadr*resampler<r$intr2r4rrMr(rQ) audio_pathsrSrr/r.rlrmrrB audio_outputsr5 audio_pathr-srr7 all_clipsclip_timepoints waveform_clipwaveform_melspecr rqr load_and_transform_audio_datavsB    r~cCs\|}|ddddgf||ddddgf<|ddddgf||ddddgf<|S)a Perform crop on the bounding boxes given the offsets. Args: boxes (ndarray or None): bounding boxes to perform crop. The dimension is `num boxes` x 4. x_offset (int): cropping offset in the x axis. y_offset (int): cropping offset in the y axis. Returns: cropped_boxes (ndarray or None): the cropped boxes with dimension of `num boxes` x 4. Nrrgrrh)copy)boxesx_offsety_offset cropped_boxesr r r crop_boxess ((rc Cs`|dvsJt|j}|dkr|d}|jd}|jd}|durJ||kr2|t|||}}n t||||}}tjjj|||fddd}tt ||d}tt ||d} ||krt|dkrkd}n|dkrs||}n|dkr{d} n|dkr||} |dddd|||| | |f} |durt || |nd} |dkr| d} | | fS) a Perform uniform spatial sampling on the images and corresponding boxes. Args: images (tensor): images to perform uniform crop. The dimension is `num frames` x `channel` x `height` x `width`. size (int): size of height and weight to crop the images. spatial_idx (int): 0, 1, or 2 for left, center, and right crop if width is larger than height. Or 0, 1, or 2 for top, center, and bottom crop if height is larger than width. boxes (ndarray or None): optional. Corresponding boxes to images. Dimension is `num boxes` x 4. scale_size (int): optinal. If not None, resize the images to scale_size before performing any crop. Returns: cropped (tensor): images with dimension of `num frames` x `channel` x `size` x `size`. cropped_boxes (ndarray or None): the cropped boxes with dimension of `num boxes` x 4. rrrgrhrrgNbilinearF)r$r align_corners) lenshaper,rur(r)r* interpolatemathceilrsqueeze) imagesr$ spatial_idxr scale_sizendimheightwidthrrcroppedrr r r uniform_cropsB     ( rcs4eZdZdZd dedeffdd Zdd ZZS) SpatialCropaK Convert the video into 3 smaller clips spatially. Must be used after the temporal crops to get spatial crops, and should be used with -2 in the spatial crop at the slowfast augmentation stage (so full frames are passed in here). Will return a larger list with the 3x spatial crops as well. r=rh crop_size num_cropscsNt||_|dkrgd|_g|_dS|dkr#dg|_g|_dStd)NrhrrzNothing else supported yet)super__init__r crops_to_extflipped_crops_to_extNotImplementedError)selfrr __class__r r rs    zSpatialCrop.__init__cCst|ts Jdtdd|DsJdg}|D]0}|jD]}|t||j|dq|js2qtj |}|jD]}|t||j|dq;q|S)z Args: videos: A list of C, T, H, W videos. Returns: videos: A list with 3x the number of elements. Each video converted to C, T, H', W' by spatial cropping. z-Must be a list of videos after temporal cropscSsg|]}|jdkqS))r)r[videor r r r_sz'SpatialCrop.forward..zMust be (C,T,H,W)r) isinstancelistallrr4rrrrr*hflip)rvideosresrr flipped_videor r r forward s   zSpatialCrop.forward)r=rh)__name__ __module__ __qualname____doc__rurr __classcell__r r rr rs rc@seZdZdddZddZdS)NormalizeVideoFcCs||_||_||_dS)N)rrBinplace)rrrBrr r r r"s zNormalizeVideo.__init__cCsn|js|}tj|j|j|jd}tj|j|j|jd}||dddddf |dddddf|S)N)dtyperS) rcloner( as_tensorrrrSrBsub_div_)rcliprrBr r r __call__'s 0zNormalizeVideo.__call__N)F)rrrrrr r r r r!s  rcs|durdSg}ttdtdddgt||d}tj|d}|D][}tj|fddd d |i} t || j } g} | D]#} | | d | d } | durRt d || d}|d}| |q>fdd| D} tddd| } tj| d d} | | q$tj|d d|S)Nr=r?r@rArk) num_samplesdecordF)decoder decode_audior.rrz No clip foundrgo@csg|]}|qSr r )r[rvideo_transformr r r__sz1load_and_transform_video_data..rh)rrE)rrG pv_transformsShortSideScalerrUniformTemporalSubsampler from_pathr<r6get_clip ValueErrorr4rr(rQrP) video_pathsrSrlrmr. video_outputsr5 frame_sampler video_pathrr7 all_videor{r video_clipr rr load_and_transform_video_data0sL      r)rdrerfrgrhrirj)NN)rgrrf) r&rr r(torch.nnr)rPILr pytorchvideorrpytorchvideo.data.clip_samplingrpytorchvideo.data.encoded_videor torchvision)imagebind.models.multimodal_preprocessorsrr"r r2r<rYrcr~rrModulerrrr r r r sD       '   2 ;+