o i6@s(ddlZddlZddlmZddlmZmZddlZddl Z ddl m Z ddl m Z ddZ dd Zd e jfd d Zd dZGdddZddde dfddZddde dfddZdde dfddZddZddZdeeje jffd d!Zdeeje jfd"eeeeeffd#d$ZdS)%N)Thread)TupleUnion)Image)tqdmcCstjrKtjdjdk}tjdjdk}|s!tjdtddtddtj d ddD}|d krCtjd tj d tdd|d kpI| }nd }d}d }|||fS)NrzSFlash Attention is disabled as it requires a GPU with Ampere (8.0) CUDA capability.category stacklevelcss|]}t|VqdSN)int).0vrS/home/ubuntu/.local/lib/python3.10/site-packages/torchao/_models/sam2/utils/misc.py sz$get_sdpa_settings...)r r zYou are using PyTorch zw without Flash Attention v2 support. Consider upgrading to PyTorch 2.2+ for Flash Attention v2 (which could be faster).TF) torchcuda is_availableget_device_propertiesmajorwarningswarn UserWarningtuple __version__split)old_gpuuse_flash_attnpytorch_versionmath_kernel_onrrrget_sdpa_settingss* "  r$cCs"ddlm}||tjS)a Get the connected components (8-connectivity) of binary masks of shape (N, 1, H, W). Inputs: - mask: A binary mask tensor of shape (N, 1, H, W), where 1 is foreground and 0 is background. Outputs: - labels: A tensor of shape (N, 1, H, W) containing the connected component labels for foreground pixels and 0 for background pixels. - counts: A tensor of shape (N, 1, H, W) containing the area of the connected components for foreground pixels and 0 for background pixels. r)_C)sam2r%get_connected_componnetstoruint8 contiguous)maskr%rrrget_connected_components0s r,maskscCs|j\}}}}|j}tj||tjd}tj||tjd}tj||dd\}} |d|d||}| d|d||} tjt||| ddd\} }tj t||d ddd\} }tjt|| | ddd\} }tj t|| d ddd\} }tj | | | | fdd}|S) z compute bounding box given an input mask Inputs: - masks: [B, 1, H, W] masks, dtype=torch.Tensor Returns: - box_coords: [B, 1, 4], contains (x, y) coordinates of top left and bottom right box corners, dtype=torch.Tensor )devicedtypexy)indexing)NN.dim) shaper.rarangeint32meshgridexpandminwhereflattenmaxstack)r-B_hwr.xsysgrid_xsgrid_ysmin_xsmax_xsmin_ysmax_ys bbox_coordsrrr mask_to_boxCs """"rNcCsxt|}t|d||f}|jtjkr|d}n td|jd|t | ddd}|j \}}|||fS)NRGBo@zUnknown image dtype: z on r rr2) ropennparrayconvertresizer/r) RuntimeErrorr from_numpypermutesize)img_path image_sizeimg_pilimg_npimg video_width video_heightrrr_load_img_as_tensor]s     rac@s(eZdZdZddZddZddZdS) AsyncVideoFrameLoaderzZ A list of video frames to be load asynchronously without blocking session start. csx|_|_|_|_|_dgt|_d_d_d_ |_ dfdd}t |dd_ j dS)Nrc sZztttjddD]}|q WdSty,}z |_WYd}~dSd}~ww)Nframe loading (JPEG)desc)rrangelenimages __getitem__ Exception exception)neselfrr _load_framess z4AsyncVideoFrameLoader.__init__.._load_framesT)targetdaemon) img_pathsr[offload_video_to_cpuimg_meanimg_stdrgrhrkr`r_compute_devicerirthreadstart)rorsr[rtrurvrwrprrnr__init__ns   zAsyncVideoFrameLoader.__init__cCs|jdur td|j|j|}|dur|St|j||j\}}}||_||_||j8}||j }|j s=|j |j dd}||j|<|S)NzFailure in frame loading threadT) non_blocking) rkrVrhrarsr[r`r_rurvrtr(rw)roindexr^r`r_rrrris        z!AsyncVideoFrameLoader.__getitem__cCs t|jSr )rgrhrnrrr__len__s zAsyncVideoFrameLoader.__len__N)__name__ __module__ __qualname____doc__rzrir}rrrrrbis & rb)g ףp= ?gv/?gCl?)gZd;O?gy&1?g?Frc Csvt|t}t|t}|otj|ddv} |s| r$t||||||dS|r7tj|r7t|||||||dSt d)z Load the video frames from video_path. The frames are resized to image_size as in the model and are loaded to GPU if offload_video_to_cpu=False. This is used by the demo. r4)z.mp4z.MP4) video_pathr[rtrurvrw)rr[rtrurvasync_loading_framesrwz;Only MP4 video and JPEG folder are supported at this moment) isinstancebytesstrospathsplitext!load_video_frames_from_video_fileisdir!load_video_frames_from_jpg_imagesNotImplementedError) rr[rtrurvrrwis_bytesis_str is_mp4_pathrrrload_video_framess2  rcsXt|trtj|r|ntdddtD}|jdddt|}|dkr3t dfd d|D} t j |t j d d d d d f}t j |t j d d d d d f}|rlt | |||||} | | j| jfSt j|d ||t j d } tt| d dD]\} } t| |\| | <}}q|s| |} ||}||}| |8} | |} | ||fS)aX Load the video frames from a directory of JPEG files (".jpg" format). The frames are resized to image_size x image_size and are loaded to GPU if `offload_video_to_cpu` is `False` and to CPU if `offload_video_to_cpu` is `True`. You can load a frame asynchronously by setting `async_loading_frames` to `True`. akOnly JPEG frames are supported at this moment. For video files, you may use ffmpeg (https://ffmpeg.org/) to extract frames into a folder of JPEG files, such as ``` ffmpeg -i .mp4 -q:v 2 -start_number 0 /'%05d.jpg' ``` where `-q:v` generates high-quality JPEG frames and `-start_number 0` asks ffmpeg to start the JPEG file from 00000.jpg.cSs$g|]}tj|ddvr|qS)r4)z.jpgz.jpegz.JPGz.JPEG)rrr)rprrr s z5load_video_frames_from_jpg_images..cSsttj|dS)Nr)rrrr)rrrrsz3load_video_frames_from_jpg_images..)keyrzno images found in csg|] }tj|qSr)rrjoin)r frame_name jpg_folderrrrsr/Nrcrd)rrrrrrlistdirsortrgrVrtensorfloat32rbr`r_zeros enumeraterrar()rr[rtrurvrrw frame_names num_framesrs lazy_imagesrhrlrZr`r_rrrrsF     rc Csddl}tj|tjdddddf}tj|tjdddddf}|jd||j\}}} g} |j|||dD] } | | dddq=tj | dd d } |sf| |} | |}| |}| |8} | |} | ||fS) z(Load the video frames from a video file.rNrr)widthheightr r2r5rP)decordrrrbridge set_bridge VideoReadernextr7appendrXr@floatr() rr[rtrurvrwrr`r_rBrhframerrrrs       rc Cs|dksJd|}zt|dk\}}|dk||k@}t|d|}W|StyD}ztj|dtdd|}WYd}~|Sd}~ww)zY A post processor to fill small holes in mask scores with area under `max_area`. rzmax_area must be positiveg?a* Skipping the post-processing step due to the error above. You can still use SAM 2 and it's OK to ignore the error above, although some post-processing functionality may be limited (which doesn't affect the results in most cases; see https://github.com/facebookresearch/sam2/blob/main/INSTALL.md).r r N)r,rr=rjrrr)r+max_area input_masklabelsareasis_holermrrrfill_holes_in_mask_scores9s" rcCsJ|dur ||}}ntj|d|gdd}tj|d|gdd}||dS)zDAdd new points and labels to previous point inputs (add at the end).N point_coordsr2r5 point_labels)rr)rcat)old_point_inputs new_points new_labelspointsrrrr concat_pointsVs   rimagecCsht|tjr |jddSt|tjr|j\}}}||fSt|tr+|j\}}||fStdt |)Nr z;Only support np.ndarray, torch.Tensoror PIL Image, but got ) rrRndarrayr7rTensorrrYrtype)rrBrCrDrrrget_image_sizeas      rcrop_boxcCsj|\}}}}t|tjr|||||ddfSt|tjr,|dd||||fStdt|)NzAExpected image to be of type np.ndarray or torch.Tensor, but got )rrRrrr ValueErrorr)rrx0y0x1y1rrr crop_imageps   r)rr threadingrtypingrrnumpyrRrPILrrr$r,rrNrarbr.rrrrrrrrrrrrrsF    H - G