o QiMR@sddlZddlZddlZddlZddlZddlZddlZddlZddlm Z ddl m Z ddl m Z ddlmZmZmZmZmZmZddlZddlZddlZddlZddlmZddlmZddlm Z mZdd lm Z d Z!d Z"d Z#d Z$dZ%dZ&dZ'd Z(d Z)dZ*dZ+e,e-ej./ddZ0e1e2Z3de,de,de,fddZ4de,de,de,fddZ5de,de,de,fddZ6dLde,de,de,dee,d ee,dee,e,ff d!d"Z7d#ejdejfd$d%Z8dMd'ee9ee9ejffd(e,dejfd)d*Z:d'ee9efd+e,d,ee,e-fde,fd-d.Z;d'ee9efdeejfd1d2Z?d'ee9efd+e,d,e-dee,e,e,ffd3d4Z@d'ee9efdeejfd7d8ZBd'ee9efdeejde9fd?d@ZG A AdNd'ee9efd(e,dBe>dCe>deejdCe>d(e,deeeejeeeejtdtdt||t||t|t||}t|t||}|||krot|||}t |||}t |||}||fS|||krt|||}t |||}t |||}||fS)a+ Rescales the image so that the following conditions are met: 1. Both dimensions (height and width) are divisible by 'factor'. 2. The total number of pixels is within the range ['min_pixels', 'max_pixels']. 3. The aspect ratio of the image is maintained as closely as possible. NrzDThe max_pixels of image must be greater than or equal to min_pixels.z+absolute aspect ratio must be smaller than z, got ) IMAGE_MAX_TOKEN_NUMIMAGE_MIN_TOKEN_NUMmaxmin MAX_RATIO ValueErrorrr sqrtr%r")r&r'rr(r)h_barw_barbetarrr smart_resize8s&  r4 pil_imagecCs>|jdkrtd|jd}|j||dd|S|dS)NRGBARGB)r8r8)mask)moder newsizepastesplitconvert)r5white_backgroundrrrto_rgbTs  rBeleimage_patch_sizec Csd|vr |d}n|d}d}t|t}t|tjr|}n|ds(|dratj|dd)}|t|j }t t |}Wdn1sLwYWdn1s[wYnG|drpt |dd}n8|d rd |vr| d d \}}t|} t| }t t |}Wdn1swYnt |}|durtd |t|}d |vrd|vrt|d |d|d\} } n$|j\} } |dt|d}|dt|d}t| | |||d\} } || | f}|S)Nimage image_urlhttp://https://T)streamfile://z data:imagezbase64,zRUnrecognized image input, support local path, http url, base64 and PIL.Image, got resized_height resized_widthrr(rr)rr(r))intSPATIAL_MERGE_SIZE isinstancer startswithrequestsgetraise_for_statusrcontentcopydeepcopyopenr?base64 b64decoder/rBr4r=r+r*resize)rDrErF image_obj patch_factorresponsebio_ base64_datadatarNrOr'r&r(r)rrr fetch_image]s`           rg total_frames video_fpscCsd|vr d|vr Jdd|vrt|dt}nA|dt}t|dtt}t|dtt|t}|||}||krIt d|d|dttt ||||}t|t}t|kra||ksot d td |d |d |S) aacalculate the number of frames for video used for model inputs. Args: ele (dict): a dict contains the configuration of video. support either `fps` or `nframes`: - nframes: the number of frames to extract for model inputs. - fps: the fps to extract frames for model inputs. - min_frames: the minimum number of frames of the video, only used when fps is provided. - max_frames: the maximum number of frames of the video, only used when fps is provided. total_frames (int): the original total number of frames of the video. video_fps (int | float): the original fps of the video. Raises: ValueError: nframes should in interval [FRAME_FACTOR, total_frames]. Returns: int: the number of frames for video used for model inputs. fpsnframesz%Only accept either `fps` or `nframes` min_frames max_frameszsmart_nframes: nframes[z] > total_frames[]znframes should in interval [z, z ], but got .) r FRAME_FACTORrWFPSr"FPS_MIN_FRAMESr%r-FPS_MAX_FRAMESloggerwarningr,r/)rDrhrirkrjrlrmrrr smart_nframess   rvc Cs|d}ttjtdkr&d|vsd|vrtdd|vr&|dd}t}tj|| d d | d dd d d\}}}| d|d}}t d|d|d|dt|dd t |||d}td|d|} |t|d|} || }t|| |dd} || | fS)aread video using torchvision.io.read_video Args: ele (dict): a dict contains the configuration of video. support keys: - video: the path of video. support "file://", "http://", "https://" and local path. - video_start: the start time of video. - video_end: the end time of video. Returns: torch.Tensor: the video tensor with shape (T, C, H, W). videoz0.19.0rHrIzVtorchvision < 0.19.0 does not support http/https video path, please upgrade to 0.19.0.rKrLN video_start video_endsecTCHW) start_ptsend_ptspts_unit output_formatrriztorchvision: video_path=, total_frames= , video_fps=, time=.3fsrhrirMư> torchvisionrjframes_indicestotal_num_frames video_backend)r parser __version__warningswarntimer read_videorWr=rtinforvtorchlinspacerlongr,dict) rD video_pathstrwaudiorrhrirkidx sample_fpsvideo_metadatarrr_read_video_torchvisions6     . rcCddl}|jdduS)Nrdecordimportlib.utilutil find_spec importlibrrris_decord_availablerc Cs`|dkrtd|dkrtd|dd}|dd}|dur+|dur+d|d|fS||}|durCtdt||}t||}nd}|dur`tdt||}t||} t| |d} n|d} || krtd |d |durs|ndd | d |dur|n|d |d d|d|dtd|d| d|d|d|d|d || | |dfS)a Calculate the start and end frame indices based on the given time range. Args: ele (dict): A dictionary containing optional 'video_start' and 'video_end' keys (in seconds). total_frames (int): Total number of frames in the video. video_fps (float): Frames per second of the video. Returns: tuple: A tuple containing (start_frame, end_frame, frame_count). Raises: ValueError: If input parameters are invalid or the time range is inconsistent. rz#video_fps must be a positive numberz'total_frames must be a positive integerrxNrzrMryz Invalid time range: Start frame z (at zs) exceeds end frame zs). Video duration: z.2fzs (z frames @ zfps)z)calculate video frame range: start_frame=z , end_frame=rz from video_start=z , video_end=rr) r/rWr,r-r r!r$rtr) rDrhrirxrz max_durationvideo_start_clamped start_framevideo_end_clamped end_framerrrcalculate_video_frame_rangesD  0rc Csddl}|d}t}||}t||}}t|||\}}}t|||d} t|||  } | |  } t| dddd} td|d |d |d t|d d | t|d|} t|| |dd} | | | fS)aread video using decord.VideoReader Args: ele (dict): a dict contains the configuration of video. support keys: - video: the path of video. support "file://", "http://", "https://" and local path. - video_start: the start time of video. - video_end: the end time of video. Returns: torch.Tensor: the video tensor with shape (T, C, H, W). rNrwrr9rMrzdecord: video_path=rrrrrrrr)rr VideoReaderlen get_avg_fpsrrvrrrrtolist get_batchasnumpytensorpermutertrr,r)rDrrrvrrhrirrrkrrwrrrrr_read_video_decord$s.  . rcCr)Nr torchcodecrrrrris_torchcodec_availableLrrc Csddlm}ttjdd}td||d}t}|||d}|j j }|j j }t |||\}} }t |||d} t|| | } | t|d |} |j| d j} td |d |d |dt|dd t|| |dd}| || fS)aread video using torchcodec.decoders.VideoDecoder Args: ele (dict): a dict contains the configuration of video. support keys: - video: the path of video. support "file://", "http://", "https://" and local path. - video_start: the start time of video. - video_end: the end time of video. Returns: torch.Tensor: the video tensor with shape (T, C, H, W). r) VideoDecoderTORCHCODEC_NUM_THREADSrzset TORCHCODEC_NUM_THREADS: rw)num_ffmpeg_threadsrr)indicesztorchcodec: video_path=rrrrrrr)torchcodec.decodersrrRosenvironrWrtrrmetadata average_fps num_framesrrvrrrrrr, get_frames_atrfr)rDrrrrdecoderrirhrrrkrrrwrrrr_read_video_torchcodecRs2   . r)rrrFORCE_QWENVL_VIDEO_READERrM)maxsizecCsDtdurt}ntr d}ntrd}nd}td|dtjd|S)Nrrrzqwen-vl-utils using z to read video.)file)rrrprintsysstderr)video_reader_backendrrrget_video_reader_backendsrFreturn_video_sample_fpsreturn_video_metadatac s|tt}t}t|dtrLt}z t||\}}} WntyK} zt d|d| td|\}}} WYd} ~ nd} ~ wwt|dt t fsWJ|  dd ddttt|d} t| dfdd|dD} d d| D} Wdn1swYtt| t}t| |kr| | d g|t| |d d } td d| D} d| }t|ddtt|D|| |d}|j\}}}}|d|}|dtd}tt|||tt|d}|d|}||krt d|d|dt||}d|vr;d|vr;t|d|dd\}}n t||||d\}}tj j!|||gt"j#dd$}|r\||fn|}|re|| fS|S)Nrwzvideo_reader_backend z) error, use torchvision as default, msg: rtype) max_workerscs"g|] }td|iqS)rF)submitrg).0 video_elementexecutor image_factor process_inforr szfetch_video..cSsg|]}|qSr)result)rfuturerrrrsrrc Ss&g|]}tt|dddqS)rrrM)r from_numpynparray transpose)rrFrrrrsraw_fpscSsg|]}|qSrr)rirrrrs)rjrrr( total_pixelsg?g?r)zThe given max_pixels[z] exceeds limit[z].rNrOrPrQT) interpolation antialias)%rSVIDEO_MIN_TOKEN_NUMVIDEO_MAX_TOKEN_NUMrTstrrVIDEO_READER_BACKENDS ExceptionrtrulisttuplerZpopr-MAX_NUM_WORKERS_FETCH_VIDEOrrr"rpextendrWrstackrrangeshaperr,rRr4r functionalr_rBICUBICfloat)rDrErrVIDEO_FRAME_MIN_PIXELSVIDEO_FRAME_MAX_PIXELSrrwrrerfutures image_listrkrrdr&r'r(rr)max_pixels_supposedrNrO final_videorrr fetch_videos                 r conversationscCs~g}t|dtr |g}|D].}|D])}t|dtr;|dD]}d|vs5d|vs5d|vs5|dddvr:||qqq|S) NrrYrFrGrwrtext)rFrGrw)rTrrrWappend)r vision_infos conversationmessagerDrrrextract_vision_infos   rreturn_video_kwargsc Cst|}g}g}g}|D]0}d|vsd|vr |t||dq d|vr9t|d||d\} } || || q tdt|dkrEd}t|dkrMd}d d i} |sZ| d |i|ra||| fS||fS) NrFrG)rErwT)rrErz,image, image_url or video should in content.rdo_sample_framesFrj)rrrgrr/rupdate) rrrrEr image_inputs video_inputsvideo_sample_fps_list vision_info video_inputvideo_sample_fps video_kwargsrrrprocess_vision_infos0     r)NN)rC)rCFF)FFrC)Kr]rZloggingr rrrrconcurrent.futuresr functoolsrr rtypingrrrrr r numpyrrVrr packagingr PILr rtorchvision.transformsrr.rSr+r*rrrqrprrrsrrRrrrWr getLogger__name__rtrr"r%r4rBrrgrvTensorrboolrrrrrrgetenvrrrrrrrrrs        4, 3   (   ,   :   (   +   BP&: