o پiy@sddlZddlZddlmZmZddlmZddlZddlmZddl m Z ddl m Z ddl mZddlmZd d lmZmZejd d Gd dde Zejd d GdddeZed d Gddde Zed d dGdddZGdddejjeZdS)N)ABCabstractmethod) dataclass)PretrainedConfig)MultimodalDataItem)EmbeddingResult)BaseMultimodalProcessor)logger)compute_retention_mask%replace_offsets_with_tokens_per_frameT)kw_onlyc@s$eZdZUeeeeefed<dS) EVSDataItem thw_gridsN)__name__ __module__ __qualname__listtupleint__annotations__rrX/home/ubuntu/.local/lib/python3.10/site-packages/sglang/srt/multimodal/evs/evs_module.pyr s rc@s eZdZUejed<ddZdS)VideoEVSDataItempre_chunked_input_idscCs|sJdS)N)is_videoselfrrr __post_init__)szVideoEVSDataItem.__post_init__N)rrrtorchTensorrrrrrrr%s  rc@sdeZdZUdZeeed<dejdee eefde dedede ejee eefff d d Z d S) EVSEmbeddingResultam Embedding result that includes per-frame token counts after EVS pruning. After pruning, each frame retains a different number of tokens based on its dissimilarity to the previous frame. This metadata is needed downstream to adjust the input_ids placeholder spans to match the actual embedding sizes. Attributes: embedding: The pruned video embeddings tensor. num_tokens_per_frame: Actual retained token count for each frame. For example, [256, 180, 195, 256] means frame 0 kept all 256 tokens (first frame is never pruned), while frames 1-2 were pruned. num_tokens_per_frame input_idsoffsetsitemextend_prefix_lenextend_seq_lenreturnc Cst||ksJt|tsJdt||j}|j}t||j||d}tj ||j |j d}t ||}||||}t||ksOJdt|d|||fS)NzExpected VideoEVSDataItem, got )rr"frame_offsets_inclusivefiller_token_id)dtypedevicez3Input ids length changed after redistribution, got z != )len isinstancertyper pad_valuer r"rtensorr+r,rget_mm_items_offset) rr#r$r%r&r'rr*input_ids_listrrr'redistribute_pruned_frames_placeholders?s2   z:EVSEmbeddingResult.redistribute_pruned_frames_placeholdersN) rrr__doc__rrrrr rrr4rrrrr!-s  r!)frozenr c@s*eZdZUeed<dZeed<ddZdS) EVSConfigvideo_pruning_rater spatial_merge_sizecCs(|jdkr |jdksJd|jdS)Ng?zLVideo pruning rate must be between 0.0 and 1.0, got self.video_pruning_rate=)r8rrrrrfs zEVSConfig.__post_init__N)rrrfloatrr9rrrrrrr7as   r7cseZdZdZeededefddZede e de j fddZ ded ejd ejdd ffd d Zde e defddZZS)EVSa^ Base class for video models that support EVS pruning. Subclass this alongside your model class and implement the static `create_evs_config`. On initialization, if video_pruning_rate > 0, this mixin replaces the model's get_video_feature() method with a wrapper that applies EVS pruning. Example: See `NemotronH_Nano_VL_V2` configr(cCtzJExtract EVS parameters from model config. Must be implemented by subclass.NotImplementedError)r=rrrcreate_evs_configwszEVS.create_evs_configitemscCr>r?r@)rrCrrrget_video_feature}szEVS.get_video_featureargskwargsNcsvt|jj}|j|_|||_|jjdk|_ |j r0t d|d|jd|j |_dSt d|ddS)Nr:z[EVS] enabled for z []z[EVS] requested on model z) but is disabled for pruning_rate == 0.0.) super__init__ __class__rrDoriginal_get_video_featurerB evs_configr8 evs_enabledr info evs_video)rr=rErF model_namerJrrrIs    z EVS.__init__cs@td|jjd|jdt|dksJdt||d}t|ts.Jd||jj}|jj | |g}|j dkrG| dd}|j d ksQJ|j g}g}fd d |j D}t|||j d d D]*\}} t|| |d} || } || | d} | | djdd} || qjt|}t||dS)a Apply EVS pruning to video embeddings. Args: items: List containing a single VideoEVSDataItem with video features. Returns: EVSEmbeddingResult with pruned embeddings and actual token counts per frame. z[EVS] beginning for model z [evs_config=self.evs_config=rGr zExpected 1 item, got rz3Expected VideoEVSDataItem with modality VIDEO, got cs&g|]\}}}|||dqS)rSr).0thwmergerr s&z!EVS.evs_video..T)strict)video_size_thwr9q)dim) embeddingr")r debugrJrrLr-r.rr8r9rKndimflattenrzipsplitr appendreshapesumtolistextendrcatr!)rrCr%r]videos_featuresfinal_embeddingsr"sizes single_videor\retention_mask preserved num_framestokens_per_framefinal_embeddings_tensorrrXrrOsR       z EVS.evs_video)rrrr5 staticmethodrrr7rBrrrr rDtypingAnyrIr!rO __classcell__rrrQrr<ls" r<) dataclassesrvabcrrrr transformersr"sglang.srt.managers.schedule_batchr%sglang.srt.mem_cache.multimodal_cacher/sglang.srt.multimodal.processors.base_processorr sglang.utilsr evs_corer r rrr!r7nnModuler<rrrrs(         3