o پi@s^ddlZddlZddlZddlmZddlmZddlmZm Z e e Z GdddeZ dS)N)Modality)MiDashengLMModel)BaseMultimodalProcessorMultimodalSpecialTokenscs:eZdZdZegZfddZ d ddZddZZ S) MiDashengLMMultimodalProcessorz:Multimodal processor for MiDashengLM audio-language model.cstj|||g|Ri|d|_td|_|jj}|d|_ |d|_ |d|_ t |j|j|j d ||_|jtjtjdd|jvrT|jddSdS) Nz#<|audio_bos|><|AUDIO|><|audio_eos|>z.<\|audio_bos\|>(?:<\|AUDIO\|>)+<\|audio_eos\|>z <|audio_bos|>z <|AUDIO|>z <|audio_eos|>) audio_tokenaudio_token_regexaudio_token_id) input_values audio_lengthr )super__init__ AUDIO_TOKENrecompileAUDIO_TOKEN_REGEX _processor tokenizerconvert_tokens_to_idsaudio_start_idr audio_end_idrbuild mm_tokensATTR_NAME_TO_MODALITYupdaterAUDIO FEATURE_NAMESappend)self hf_config server_argsrargskwargsr __class__`/home/ubuntu/.local/lib/python3.10/site-packages/sglang/srt/multimodal/processors/midashenglm.pyr s0    z'MiDashengLMMultimodalProcessor.__init__Nc Ks|r||d<|r ||d<|r||d<i|d<|ddd|j}|jd |gddd |}t|jd dsFd D]}||vrE||||<q7|S)zGOverride to use correct audio parameter name for MiDashengLM processor.imagesvideosaudio audio_kwargs truncationFTpt)textpaddingreturn_tensorskeep_mm_feature_on_device)r Nr%) setdefaultr__call__getattrr cpu) r input_textr'r(audiosr" processorresult feature_namer%r%r&process_mm_data2s,z.MiDashengLMMultimodalProcessor.process_mm_datac stdtdtd|dutd|td|r6|j|s6|j|}td|j|||jd}|durJtddS|||j\}}}td t|td t | td |j td |j d |j d|jtd||j kt|D]S\}} td|d| jtd|dt| ddtd|dt| ddtd|dt| dt| dr| jdurtd|d| jj qd|vr"t|dkr"|d} t| tjr| dkr | n| d} | |d_td| dn.d|vrPt|dkrP|d} | jd kr=| j d!n| j d} | |d_td| d"|||j |j |jd#} td$t| d%d&| S)'aProcess audio data for MiDashengLM model. Args: audio_data: Audio input data input_text: Text prompt **kwargs: Additional arguments Returns: Dictionary containing processed multimodal data zP================================================================================zprocess_mm_data_async calledzaudio_data is not None: Nz input_text: zAuto-prepended audio token)prompt audio_datamultimodal_tokenszbase_output is Nonezmm_items count: z ret keys: zinput_ids shape: zaudio_token_id=z, audio_start_id=z, audio_end_id=z&Count of audio_token_id in input_ids: zmm_item[z ] modality: z ] pad_value: pad_valuezNOT SETz ] offsets: offsetsz] has feature: featurez] feature shape: r rzSet audio_length=z" (from processor, mel frame count)r z (fallback, waveform length))mm_items input_idsrr rz Returning rDz mm_items)loggerinforsearchr load_mm_datarprocess_and_combine_mm_datalenlistkeysshaper rrsumitem enumeratemodalityr3hasattrr@ isinstancetorchTensornumelr ndimtolist) rr<r5r" base_outputrDrEretirPr r r8r%r%r&process_mm_data_asyncNs            z4MiDashengLMMultimodalProcessor.process_mm_data_async)NNN) __name__ __module__ __qualname____doc__rmodelsr r:r] __classcell__r%r%r#r&rs  r)loggingrrU"sglang.srt.managers.schedule_batchrsglang.srt.models.midashenglmr/sglang.srt.multimodal.processors.base_processorrr getLoggerr^rFrr%r%r%r&s