o -i-k@sJddlZddlmZddlmZddlmZmZmZddl Z ddl mZ er*ddl Z ddlmZddlmZeeZde jdeeefd e jfd d Zde jd ed e jfd dZde jded e jfddZGdddZeZedGdddeZedGdddeZedGdddeZ edGdddeZ!dS) N)abstractmethod)BytesIO) TYPE_CHECKINGAnycast) init_logger)ExtensionManagerframessizereturnc Csd|j\}}}}|\}}tj||||f|jd}ddl}t|D]\} } || ||f} | || <q|S)Ndtyper)shapenpemptyr cv2 enumerateresize) r r num_frames_channels new_height new_widthresized_framesriframe resized_framerR/home/ubuntu/veenaModal/venv/lib/python3.10/site-packages/vllm/multimodal/video.py resize_videos r size_factorcCs4|j\}}}}t||}t||}t|||fSN)rintr)r r rheightwidthrrrrrrescale_video_size#s  r%rcCs<|jd}|dkr |Stjd|d|td}||df}|S)Nrr .)rrlinspacer")r r total_frames frame_indicessampled_framesrrrsample_frames_from_video+s  r,c@seZdZee ddededeej e e e fffddZ ededeed e eefd edef d d Zed ddeed edeej eee eefffddZedeedededeej eeeffddZdS) VideoLoaderr&datarr cKstr!)NotImplementedErrorclsr.rkwargsrrr load_bytes6szVideoLoader.load_bytesidx failed_framesnext_target_mapr)cCs$|sdS|d}|||}||kS)z;Check if current frame can recover the oldest failed frame.Fr)get)r4r5r6r) oldest_failedlimitrrr_can_use_for_recovery=s  z!VideoLoader._can_use_for_recoverycapzcv2.VideoCapturer*cCsddl}t||j}t||j}|dkr|dks&Jd|d|t|}|r0|dnd}i}tt|dD] } || d||| <q<|||d<g} g} g} i} d}t|dD]o}||v}|}|sy|rxt d|| |q_t || ||}|s|r|\}}|r|dur|jdkr|||j}| || ||d7}|r| d}|| |<t d||||q_|rt d || |q_| D]}t d |q| rt| }n tjd||d ftjd }|| | fS) a Read frames with dynamic window forward-scan recovery. When a target frame fails to load, the next successfully grabbed frame (before the next target frame) will be used to recover it. Args: cap: OpenCV VideoCapture object frame_indices: Sorted list of target frame indices to load total_frames: Total number of frames in the video Returns: Tuple of (frames_array, valid_frame_indices, recovered_map) - frames_array: Array of loaded frames - valid_frame_indices: List of frame indices that were loaded - recovered_map: Dict mapping recovered_idx -> source_idx rNz Invalid video frame size: width=z , height=r&r'z-Failed to grab frame %d during video loading.z-Recovered frame %d using frame %d (delay: %d)z1Failed to retrieve frame %d during video loading.z/Frame %d could not be recovered (end of video).r )rr"r7CAP_PROP_FRAME_WIDTHCAP_PROP_FRAME_HEIGHTsetrangelengrabloggerwarningappendr-r:retriever cvtColor COLOR_BGR2RGBpopinforstackruint8)r;r*r)rr$r# frame_idx_set max_frame_idxr6k frames_listvalid_frame_indicesfailed_frames_idx recovered_maprr4is_target_frameok can_recoverretr rgb_frame recovered_idx failed_idxr rrr_read_frames_with_recoveryKs         z&VideoLoader._read_frames_with_recoverynum_expected_framesrNcCsddl}t||j}t||j}tj|||dftjd}d}g} t|dD]8} | } | s=| |vrrrrLr@rBrCrDrFrGrHrErA)r;r*r\rNrr$r#r rrQr4rUrWrvalid_num_framesrrr _read_framessF   zVideoLoader._read_framesNr&)__name__ __module__ __qualname__ classmethodrbytesr"tuplenptNDArraydictstrrr3 staticmethodlistboolr:r[r?r^rrrrr-5sV    jr-identityc @s:eZdZdZe d dedededeeeffddZ d S) IdentityVideoLoaderaJIdentityVideoLoader returns raw video bytes without decoding. This allows the model processor to handle video decoding and is required for models like Kimi-K2.5 that need custom video chunk splitting. NOTE: This is temporary for Kimi-K2.5 testing. Remember to change back to opencv before release if needed. r&r.rr2r cKs|dfSr!rr0rrrr3szIdentityVideoLoader.load_bytesNr_) r`rarb__doc__rcrdr"rrer3rrrrrns  rnopencvc@sVeZdZddZe    ddededed ed ed ee j e e e fff d d ZdS)OpenCVVideoBackendcCjddlm}d}|D]&}||sq ||s.||\}}}|dks-|dkr.|dkr.q |}|S|SNrr'cv2.videoio_registryvideoio_registrygetStreamBufferedBackends hasBackendisBackendBuiltIn%getStreamBufferedBackendPluginVersionselfvrapi_prefbackendrabiapirrrget_cv2_video_api    z$OpenCVVideoBackend.get_cv2_video_apir&,Fr.rfps max_durationframe_recoveryr cKsLddl}|}|t||g} | stdt| |j} | |j } | dkr0| | nd} | } |dkr=t || } |dkrKt | t | |} t d| } | | kr\ttd| }ntjd| d| td}|}|r|| || \}}}t|}|rtdt|nt|}|| || t |\}}}| | | d||| kd}||fS) a Load video frames from bytes. Args: data: Raw video bytes num_frames: Target number of frames to sample (-1 for all) fps: Target FPS for sampling (-1 for original) max_duration: Maximum duration (unused in base backend) frame_recovery: Enable forward-scan recovery for failed frames Returns: Tuple of (frames_array, metadata_dict) rNCould not open video streamr'r 7Frame recovery: %d frames recovered using forward scan.rptotal_num_framesrduration video_backendframes_indicesdo_sample_frames)rr VideoCapturerisOpened ValueErrorr"r7CAP_PROP_FRAME_COUNT CAP_PROP_FPSminmathfloormaxrkr@rr(tolistr[rArCrJr?r^)r1r.rrrrr2rrr;total_frames_num original_fpsrnum_frames_to_sample frame_idxuniform_sampled_framesr rQrSr]rMmetadatarrrr3sV         zOpenCVVideoBackend.load_bytesN)r&r&rF)r`rarbrrcrdr"rlrerfrgrhrirr3rrrrrqs(rqopencv_dynamicc@sNeZdZe    ddedededed ed eej e e e fff d d Z d S)OpenCVDynamicVideoBackendr&rtrFr.rrrrr c s|ddl}|}|t||g} | stdt| |j} | |j dkr0| nd} | d| p?t d} | |kr]tt | } t fddt| D} n&t|}|| krntt| } ntjd| |dd}t fd d|D} |r|| | | \}}}t|}|rtd t|nt| }|| |t| | d\}}}| | d |d d }||fS)a Load video frames with dynamic sampling based on duration. Args: data: Raw video bytes num_frames: Not used in dynamic backend fps: Target FPS for sampling (default: 2) max_duration: Maximum video duration to process (default: 300s) frame_recovery: Enable forward-scan recovery for failed frames Returns: Tuple of (frames_array, metadata_dict) rNrr'c s(h|]}ttt|qSrrr"rceil).0rrrNrrr sz7OpenCVDynamicVideoBackend.load_bytes..T)endpointc s$h|]}ttt|qSrr)rt)rNrrrrsrrFr)rrrrrrr"r7rrroundrrsortedr@rkrr(r[rArCrJr?r^)r1r.rrrrr2rrr;rrnframe_indices_list num_samplestarget_secondsr rQrSr]frame_indices_setrrrrr3ksb       z$OpenCVDynamicVideoBackend.load_bytesN)r&rtrF)r`rarbrcrdr"rlrerfrgrhrirr3rrrrris&rmolmo2c@szeZdZddZe d"dedededeefdd Zeded ed ed e d eededBf ddZ ededBd ed edede edBe j ff ddZe d#ded ed e dedBd eedBde j f ddZedededed e dededede j fddZe    d$ded e dBdededede e j ee efff ddZe d%dedede e j ee efffd d!ZdS)&Molmo2VideoBackendcCrrrsrur|rrrrrz$Molmo2VideoBackend.get_cv2_video_api @ video_fps sampling_fpsmax_fpsr cCst|}t|}t|}|durtd|dks|dkr'td|d|d||dkr8td|d|d g}t||d |D]}||krK|S||dkrX|t|qB|S) a Return the subset of `video_fps` factors that remain multiples of `sampling_fps`. Examples: >>> get_candidate_target_fps(video_fps=6, sampling_fps=2) [2, 6] >>> get_candidate_target_fps(video_fps=5, sampling_fps=1) [1, 5] >>> get_candidate_target_fps(video_fps=2, sampling_fps=2) [2] >>> get_candidate_target_fps(video_fps=5, sampling_fps=2) Traceback (most recent call last): ... ValueError: sampling_fps=2 must divide video_fps=5 to produce consistent frame steps. Nzsampling_fps must be providedrz1video_fps and sampling_fps must be positive (got z, )z sampling_fps=z must divide video_fps=.r')r"rr@rEfloat)r1rrr candidates candidaterrrget_candidate_target_fpss4  z+Molmo2VideoBackend.get_candidate_target_fps max_framesr)frame_sample_modecandidate_target_fpsNc Csd}d}|D]8}tt||d} t|| } |dkr+d|vr&| |kr&|S|}| }q|| ks1J| |kr6q| |kr>|}| }q|S)z] Get the target fps that best spans the videoand has the most frames sampled rNr'uniform)rr") r1rrr)rrnum_frames_sampledselected_target_fps target_fps step_sizenum_frames_sampled_at_fpsrrrget_target_fps s(   z!Molmo2VideoBackend.get_target_fpsrcCs^|durtjd||dtd}ntt||d}td||}t||kr+|d|}||fS)NrF)rr r')rr(r"rarangerA)r1rr)rrr*rrrrget_frame_times_and_chosen_fps1s   z1Molmo2VideoBackend.get_frame_times_and_chosen_fpsrc Ks|dkr1|dus J|d}|ddD] }|||krn|}qtd||} | | |k} | S|dkr{|durn|d|} | |krPtjd||dtjd} | Stjd|d|d} tj| |ggdd } t| |kslJ| Stjd||dtjd} | St|) Nrrr'uniform_last_frameT)numrr stopstepaxis)rrr(float64 concatenaterAr/) r1rrrrrr2r candidate_fpstimesrrrr sample_timesDs:      zMolmo2VideoBackend.sample_timesrrc CsT|dkrp|durp|dkrt|t}|S||d|kr1tjd|dt||ddt}|Stjd|dt||d} t| d |dkrVtj| |dggdd } t| t}|d |ksfJt | |ksnJ|S|dkrtjd|dt||ddt}|S|d kr| ||} | ||||| } | | |||\} }|St |) Nrrtr'rT)rrrrr&rr)rrastyper"r(rrrrrArrrr/) r1rrrrrrrindices float_indicesrrrrrr_sample_framesnsl /,   z!Molmo2VideoBackend._sample_framesr&rtr.c Ksddl}|}|t||g} | stdt| |j} | |j } | dkr0| | nd} |durdt t d| } t | }| | || t| \}}}|| k}| | | d|d}|s`||d<||fS|| | | ||||} | | t | t| | d\}}}| | | d|dd}||fS) Nrrrp)rrrrrrr'Fr)rrrrrrr"r7rrrkr@r?r^rrrrA)r1r.rrrrr2rrr;rrrrrMr r]rQrrrrrload_bytes_opencvsb       z$Molmo2VideoBackend.load_bytes_opencvcKsXttdB|dd}tt|dd}tt|dd}|j|||||fi|}|S)Nrrrtr)rrirIr"r)r1r.rr2rrroutrrrr3szMolmo2VideoBackend.load_bytes)rr!)Nr&rtrtr_)r`rarbrrcrrkrr"rirrerfrgrrrrdrhrrr3rrrrrs 0 '   ) = Cr)"rabcriortypingrrrnumpyr numpy.typingrfr vllm.loggerrvllm.utils.registryrr`rCrgrer"rrr%r,r-VIDEO_LOADER_REGISTRYregisterrnrqrrrrrrs2     " 7e]