o ۾iSB@sdZddlZddlmZmZmZddlmZddlm Z m Z m Z ddl Z ddl m Z ddlmZddlmZdd lmZdd lmZdd lmZdd lmZmZmZdd lmZmZmZddl m!Z!ddl"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)ddl*m+Z+m,Z,ddl-m.Z.m/Z/m0Z0m1Z1m2Z2m3Z3ddl4m5Z5ddl6m7Z7ddl8m9Z9ddl:m;Z;ddlZ>ddl?m@Z@mAZAmBZBmCZCeeDZEeGdddZFGddde=ZGGdddeZHGdd d e0ZIGd!d"d"e.eIZJGd#d$d$e/eIZKe!jLeKeIeJd%Gd&d'd'e jMeeeZNdS)(z Kimi-K2.5 Model Implementation for vLLM. Kimi-K2.5 extends Kimi-K2 with vision support This module defines: - KimiK25ProcessingInfo/KimiK25MultiModalProcessor: Processing logic - KimiK25ForConditionalGeneration: Main model class N)IterableMappingSequence) dataclass) AnnotatedAnyLiteral)nn) BatchFeature)ProcessorMixin) VllmConfig)BaseDummyOptions) init_logger)SupportsMultiModal SupportsPP SupportsQuant)KimiK25MultiModalProjectorMoonViT3dPretrainedModelvision_tower_forward)MULTIMODAL_REGISTRY)MultiModalDataDictMultiModalFieldConfigMultiModalKwargsItems NestedTensors VisionChunkVisionChunkImageVisionChunkVideo)MultiModalDataItemsVisionChunkProcessorItems)BaseDummyInputsBuilderBaseMultiModalProcessorBaseProcessingInfoInputProcessingContextPromptReplacement PromptUpdate)current_platform)IntermediateTensors) KimiK25Config)cached_get_image_processor) TensorSchema TensorShape)AutoWeightsLoader WeightsMapperinit_vllm_registered_model maybe_prefixc@s&eZdZUdZeed<dZeed<dS)MaxImageTokenMetai widthheightN)__name__ __module__ __qualname__r1int__annotations__r2r8r8W/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/kimi_k25.pyr0Is r0c@s`eZdZUdZdZeded<eej e ej Be ddddfed<eej e ddfed<d S) KimiK25MediaPixelInputsz Media input schema for K2-VL model. Dimensions: - np: Number of patches (flattened from all media items) - ps: Patch size - nm: Number of media items pixel_valuestypenppsnm grid_thwsN) r3r4r5__doc__r<rr7rtorchTensorlistr*r8r8r8r9r:Os   r:cs^eZdZdgZdZ d dedBffdd Z d deedBdeee Bd e fd d Z Z S)MoonshotKimiVAutoProcessor tokenizer AutoTokenizerNmedia_token_idcs*t|||_||_|jdusJdSN)super__init__media_processorrI)selfrMrGrI __class__r8r9rLgs z#MoonshotKimiVAutoProcessor.__init__ vision_chunkstextreturnc  si}t|tr j|n|}|durIt|tsJj|}fdd|D}g}|D]}|jkrA|jg| dq-| |q-|}t dt |gi|dS)a Args: vision_chunks: List of VisionChunk items to be processed. For image: VisionChunkImage with type='image', image=PIL.Image For video_chunk: VisionChunkVideo with type='video_chunk', video_chunk=list[PIL.Image] text: The token ids to be fed to a model (required). Returns: [`BatchFeature`]: A [`BatchFeature`] with the following fields: - **input_ids** -- list of token ids to be fed to a model. - **pixel_values** -- Pixel values to be fed to a model. Returned when `vision_chunks` is not `None`. - **grid_thws** -- list of image 3D grid in LLM. Returned when `vision_chunks` is not `None`. Ncsg|]}j|qSr8)rMmedia_tokens_calculator).0chunkrNr8r9 s z7MoonshotKimiVAutoProcessor.__call__..r input_ids)data) isinstancestrrGencoderErM preprocessrIextendpopappendr rCtensor) rNrQrRkwargs mm_inputsrYnum_tokens_per_chunk new_input_idstokenr8rWr9__call__ps,     z#MoonshotKimiVAutoProcessor.__call__)NNNrJ) r3r4r5 attributestokenizer_classr6rLrErr\r rh __classcell__r8r8rOr9rFcs   rFcsTeZdZdZdeddffdd ZddZd d Zdee e dBffd d Z Z S) KimiK25ProcessingInfozProcessing information for Kimi-K2.5 model. Provides configuration and utilities for processing both images and video-chunks. ctxrSNcs^t|||_|jj|_t|jjj dd}||_ t |j | |jd|_ |j j|_dS)NT)trust_remote_code)rMrGrI)rKrL get_hf_config hf_configmedia_placeholder_token_idrIr(rm model_configmodelrMrF get_tokenizer hf_processorrT)rNrmrMrOr8r9rLs    zKimiK25ProcessingInfo.__init__cCs|jSrJ)rurWr8r8r9get_hf_processorsz&KimiK25ProcessingInfo.get_hf_processorcCs |jtSrJ)rmror'rWr8r8r9ros z#KimiK25ProcessingInfo.get_hf_configcCsddiSN vision_chunkr8rWr8r8r9get_supported_mm_limitssz-KimiK25ProcessingInfo.get_supported_mm_limits) r3r4r5rBr"rLrvrorr\r6ryrkr8r8rOr9rls "rlc s~eZdZdZdeddffdd Zdeeefdefdd Z d d Z dd edeeefd eee fdBde fddZ ZS)KimiK25DummyInputsBuilderz2Builds dummy inputs for Kimi-K2.5 model profiling.inforSNcs&t||jj|_|jjj|_dSrJ)rKrLr{rIrMnum_frames_per_chunkframe_per_chunk)rNr{rOr8r9rLs  z"KimiK25DummyInputsBuilder.__init__ mm_countscCs|dd}d|S)Nrxrz <|media_pad|>)get)rNr~ num_mediar8r8r9get_dummy_texts z(KimiK25DummyInputsBuilder.get_dummy_textcCsn|jtjtj|jd}td|d}|j|}td|jtjtjdddd}|j|}||kr4|gS|gS)N)r2r1 num_images video_chunk)r<rimager+r)r<r) _get_dummy_imagesr0r2r1r}rr{rTr)rN dummy_videosvideo_chunk_dummy_itemvideo_chunk_num_tokensimage_dummy_itemimage_num_tokensr8r8r9get_dummy_mm_itemss2 z,KimiK25DummyInputsBuilder.get_dummy_mm_itemsseq_len mm_optionscCs|}d|iSrw)r)rNrr~r dummy_itemsr8r8r9get_dummy_mm_datasz+KimiK25DummyInputsBuilder.get_dummy_mm_datarJ)r3r4r5rBrlrLrr\r6rrr rrrkr8r8rOr9rzs! rzc @sdeZdZdZdedeeefdeeeffddZ de deee fde de efd d Zd d Zd S)KimiK25MultiModalProcessorz]Multi-modal processor for Kimi-K2.5. Handles both image and video-chunk modalities. hf_inputshf_processor_mm_kwargsrScCs6|dtd}|d}ttd|tddS)aIndicates how to slice media input into multiple items. pixel_values: [N, 3, patch_size, patch_size], all patches collected from B medias grid_thws: [B,3], each item: [N_t, N_h ,N_w], indicates the grid size in time/height/width direction for current item. by multiplying [N_t, N_h ,N_w], we get the number of patches for each media item, thus we can slice pixel_values by pixel_values[start:start + N_t*N_h*N_w] to get patches of one item. rA)rr>rx)r;rA)rrCemptyproddictrflat_from_sizesbatched)rNrrrA grid_sizesr8r8r9_get_mm_fields_configs z0KimiK25MultiModalProcessor._get_mm_fields_configmm_items out_mm_kwargscs8j}|jdtffdd }tdg|dgS)Nitem_idxcs(dtf}j||}g|Srw) get_itemsrr{rT)rmedianum_media_tokenrIrrNr8r9get_replacements zGKimiK25MultiModalProcessor._get_prompt_updates..get_replacementrx)modalitytarget replacement)r{rorqr6r#)rNrrrrprr8rr9_get_prompt_updatess z.KimiK25MultiModalProcessor._get_prompt_updatescCs|jj|SrJ)r{rMsplit_video_chunks)rNvideor8r8r9r,sz-KimiK25MultiModalProcessor.split_video_chunksN)r3r4r5rBr rr\objectrrrrrrr$rrr8r8r8r9rs&     r)r{ dummy_inputsc seZdZdZdZedddddZeded e d ed Bfd d Z d'de ded d ffdd Z de d ed BfddZded eejfddZde d ed BfddZ d(dejdejded Bdejd Bde d ef dd Zd!ejd ejfd"d#Zd$eeeejffd%d&ZZS))KimiK25ForConditionalGenerationzKimi-K2.5 model for conditional generation. Supports both image and video-chunk modalities. Video-chunks are temporal segments (typically 4 frames) that are processed with temporal pooling. Tzlanguage_model.model.layers.zmm_projector.linear_1zmm_projector.linear_2)zlanguage_model.layers.zmm_projector.proj.0zmm_projector.proj.2)orig_to_new_prefixrirSNcCs&|dkrdS|dkr dStd|)Nrz?<|media_begin|>image<|media_content|><|media_pad|><|media_end|>rz<|kimi_k25_video_placeholder|>zUnsupported modality: ) ValueError)clsrrr8r8r9get_placeholder_strLs z3KimiK25ForConditionalGeneration.get_placeholder_str vllm_configprefixcsHt|j}|j}||_|j}|jjdk|_|j j |_ t |_ ||d6t|jt|dd|_|jj|j |jd|_t|j|jt|dd|_|jj|j |jd|_Wdn1sawY||_t|}|jjj |j_||t||j t|dd gd |_Wdn1swY|jj|_|jj|_dS) NrZrx vision_tower)r)devicedtype mm_projector)configuse_data_parallelrlanguage_modelDeepseekV2ForCausalLM)rrpr architectures)rKrLrrrpr quant_configmultimodal_configmm_encoder_tp_moder text_config hidden_sizer%current_devicer_mark_tower_modelr vision_configr/rtorrrcopydeepcopy_mark_language_modelr.rmake_empty_intermediate_tensorsrqmedia_placeholder)rNrrrrrrsub_vllm_configrOr8r9rLWsR        z(KimiK25ForConditionalGeneration.__init__rccKs|dd}|dd}|durdSt|trtj|dd}t|jdks,t|jdkrB|j|jd|jdg|jddR}t|j j }| |}t|tj s^Jd t||d |jd }|jdkrs|ddks{Jd |jtd||d S) Nr;rAr)dimr>r+z%expect grid_thws to be a tensor, get rz unexpected shape for grid_thws: )r<r;rA)r`r[rErCcatlenshapereshapenextr parametersrrrDr<ndimsizer:)rNrcr;rA target_dtyper8r8r9_parse_and_validate_media_inputs4       z?KimiK25ForConditionalGeneration._parse_and_validate_media_input media_inputcCs$t|j|d|d|j|jd}|S)Nr;rA)rr)rrrr)rNrmedia_featuresr8r8r9_process_media_inputsz4KimiK25ForConditionalGeneration._process_media_inputcKs*|jdi|}|durdS||}|S)Nr8)rr)rNrcrvision_embeddingsr8r8r9embed_multimodals  z0KimiK25ForConditionalGeneration.embed_multimodalrY positionsintermediate_tensors inputs_embedscKs"|durd}|j||||d}|S)N)rYrrr)r)rNrYrrrrc hidden_statesr8r8r9forwardsz'KimiK25ForConditionalGeneration.forwardrcKs|j|}|SrJ)rcompute_logits)rNrrclogitsr8r8r9rs z.KimiK25ForConditionalGeneration.compute_logitsweightscCst|}|j||jdS)N)mapper)r, load_weightshf_to_vllm_mapper)rNrloaderr8r8r9rsz,KimiK25ForConditionalGeneration.load_weights)r)NN)r3r4r5rBsupports_encoder_tp_datar-r classmethodr\r6rr rLrr:rrErCrDrrrr&rrrtuplerrkr8r8rOr9r0s^  5 "   $r)OrBrcollections.abcrrr dataclassesrtypingrrrrCr transformersr transformers.processing_utilsr vllm.configr vllm.config.multimodalr vllm.loggerr%vllm.model_executor.models.interfacesrrr'vllm.model_executor.models.kimi_k25_vitrrrvllm.multimodalrvllm.multimodal.inputsrrrrrrrvllm.multimodal.parserrvllm.multimodal.processingrr r!r"r#r$vllm.platformsr% vllm.sequencer&vllm.transformers_utils.configsr'!vllm.transformers_utils.processorr(vllm.utils.tensor_schemar)r*utilsr,r-r.r/r3loggerr0r:rFrlrzrregister_processorModulerr8r8r8r9sP         $     ?!49