o
پi @ sd d dl mZmZ d dlZd dlmZmZ d dlmZ d dl m
Z
d dl m Z m
Z
G dd de ZdS ) )ListUnionN)ModalityMultimodalDataItem)MiniCPMO)MiniCPMV)BaseMultimodalProcessorMultimodalSpecialTokensc sR e Zd ZeegZdZ fddZdee e
e f dee e
e f fddZ Z
S )MiniCPMMultimodalProcessorTc s t j|||g|R i | | jj}t|dd | _t|dd | _t|dd | _t|dd | _t|dd | _ t|dd | _
t|dd | _ t dd d
| j d
|| _d S ) Nslice_start_id slice_end_idaudio_start_id audio_end_id im_start_id im_end_idunk_idz(./)z()z()) image_token audio_token video_tokenimage_token_id)super__init__
_processor tokenizergetattrr r r
r r r im_token_idr build mm_tokens)self hf_config server_argsr argskwargsr __class__ \/home/ubuntu/.local/lib/python3.10/site-packages/sglang/srt/multimodal/processors/minicpm.pyr s" z#MiniCPMMultimodalProcessor.__init__
image_data
audio_datac
sN | j |||| jd}|d u rd S | j|j|j|jd}|d }|d } t|tjt fs5t
dt | t| tjt fsFt
dt | t |t | kr\t
dt | dt | g }
g } t
|| D ].\} }
t | t |
krt
d t | d
t |
t
| |
D ]\}}|
|g7 }
| |g7 } qqe|
}g }|d }| j|| j| jd }| j|| j| jd }|| t|}t |d
krt||d| itjd}||g7 }d|v r|d d urt |d d
kr| jd ur| jd ur| j|| j| jd }nd }t|d gd|d i|tjd}||g7 }|| | j| j| j | j| j| j| jd S )N)promptr( r' multimodal_tokens)
input_textimagesaudios pixel_values tgt_sizesz*Incorrect type of pixel values. Got type: z*Incorrect type of target sizes. Got type: z#Inconsistent batch lengths, found: z vs. z Inconsistent N lengths, found: z vs input_ids)r0 mm_start_id mm_end_idr tgt_size)featureoffsetsmodel_specific_datamodalityaudio_featuresaudio_feature_lens)r4 r6 r5 r7 ) mm_itemsr0 r
r r r r r r ) load_mm_datar process_mm_datar+ r, r-
isinstancetorchTensorlist
ValueErrortypelenzipflattenget_mm_items_offset_by_pairr r r r extendsortedr r IMAGEr
r AUDIOtolistr )r r' r( r+ request_objr" base_outputresr. r/ pixel_values_flattgt_sizes_flatpixel_btgt_bpixel_ntgt_nitemsr0
image_offsets
slice_offsetsitem
audio_offsetsr% r% r& process_mm_data_async% s
z0MiniCPMMultimodalProcessor.process_mm_data_async)__name__
__module__ __qualname__r r models support_dynamic_frame_expansionr r r strbytesrZ
__classcell__r% r% r# r& r
s r
)typingr r r> "sglang.srt.managers.schedule_batchr r sglang.srt.models.minicpmor sglang.srt.models.minicpmvr /sglang.srt.multimodal.processors.base_processorr r r
r% r% r% r& s