o ir@sUdZddlZddlZddlmZmZmZddlmZddl m Z m Z m Z m Z ddlZddlmZddlmZddlmZmZdd lmZdd lmZdd lmZmZdd lmZdd lm Z ddl!m"Z"m#Z#ddl$m%Z%ddl&m'Z'ddl(m)Z)ddl*m+Z+ddl,m-Z-m.Z.m/Z/m0Z0ddl1m2Z2m3Z3ddl4m5Z5m6Z6m7Z7m8Z8m9Z9ddl:m;Z;ddlm?Z?m@Z@ddlAmBZBmCZCmDZDmEZEddlFmGZGmHZHmIZImJZJmKZKdZLdZMGddde?ZNGd d!d!e?ZOeNeOBZPe eQd"<Gd#d$d$e7ZRGd%d&d&e5eRZSGd'd(d(e6eRZTGd)d*d*ejUZVGd+d,d,ejUZWGd-d.d.ejUeZXGd/d0d0eZYe+jZeTeReSd1Gd2d3d3ejUeDeEeCZ[d4ej\e]ej\Be]e]ej\Bd5ej\fd6d7Z^dS)8zPyTorch Ultravox model.N)IterableMappingSequence)SimpleNamespace) AnnotatedAnyLiteral TypeAlias)nn) functional) BatchFeatureProcessorMixin)ModuleUtilsMixin)WhisperFeatureExtractor)WhisperEncoderWhisperEncoderLayer) VllmConfig)BaseDummyOptions) MulAndSilu get_act_fn)RMSNorm)DefaultModelLoader)MultiModelKeys)MULTIMODAL_REGISTRY)MultiModalDataDictMultiModalFieldConfigMultiModalKwargsItems NestedTensors)MultiModalDataItemsMultiModalDataParser)BaseDummyInputsBuilderBaseMultiModalProcessorBaseProcessingInfoPromptReplacement PromptUpdate)IntermediateTensors)UltravoxConfig) TensorSchema TensorShape)MultiModalEmbeddings SupportsLoRASupportsMultiModal SupportsPP)AutoWeightsLoader WeightsMapper flatten_bninit_vllm_registered_model maybe_prefix <|audio|>c@seZdZUdZeded<eeje ejBe e ejBe dddfed<eeje dfed< eeje dfed < eeje d fed <d S) UltravoxAudioFeatureInputszz Dimensions: - b: batch size - n: number of chunks - t: Time frames (M) - nmb: Number of mel bins audio_featurestypebnnmbtdatalens token_lenn num_chunksN __name__ __module__ __qualname____doc__r__annotations__rtorchTensorlistr(rIrIY/home/ubuntu/vllm_env/lib/python3.10/site-packages/vllm/model_executor/models/ultravox.pyr5Cs   r5c@sDeZdZUdZeded<eeje ejBe ddddfed<d S) UltravoxAudioEmbeddingInputszx Dimensions: - b: batch size - na: number of audios - afs: audio feature size - hs: hidden size audio_embedsr7bnaafshsr;Nr@rIrIrIrJrK[s   rKUltravoxAudioInputsc@s`eZdZdedefddZdedefddZddZde fd d Z de e e d Bffd d Z d S)UltravoxProcessingInfokwargsreturncKs.|jjj}|jjdi|}t|_|j|_|SNrI)ctx model_config hf_configget_hf_processor_AUDIO_PLACEHOLDER_OVERRIDEaudio_token_replacementaudio_token_indexaudio_replacement_token_id)selfrSconfig hf_processorrIrIrJrYps z'UltravoxProcessingInfo.get_hf_processorcKs<|jdi|}|j}t|tr|S|j}t|tsJ|SrU)rYaudio_processor isinstancerfeature_extractor)r^rSr`rarcrIrIrJget_feature_extractor|s z,UltravoxProcessingInfo.get_feature_extractorcCs |}t|j||dS)N) target_srtarget_channelsexpected_hidden_size)rdr sampling_rateget_target_channels_get_expected_hidden_size)r^rcrIrIrJget_data_parsers z&UltravoxProcessingInfo.get_data_parsercCsdS)z8Return target audio channels for Ultravox models (mono).r)rIr^rIrIrJrisz*UltravoxProcessingInfo.get_target_channelsNcCsddiS)NaudiorIrlrIrIrJget_supported_mm_limitssz.UltravoxProcessingInfo.get_supported_mm_limits)rArBrCobjectr rYrrdrkintrirstrrnrIrIrIrJrRos    rRc @sXeZdZdeeefdefddZ d dedeeefdeeefdBdefdd Z dS) UltravoxDummyInputsBuilder mm_countsrTcCs|dd}d|S)Nrmrr3)get)r^rs num_audiosrIrIrJget_dummy_texts z)UltravoxDummyInputsBuilder.get_dummy_textNseq_len mm_optionsc CsP|j}|j}|j|t}|dd}|r|dnd}d|j|||diS)Nrmr)lengthru overrides)infordrh chunk_length_MAX_ENCODER_BATCH_SIZErt_get_dummy_audios) r^rwrsrxrcrh audio_lenruaudio_overridesrIrIrJget_dummy_mm_datas   z,UltravoxDummyInputsBuilder.get_dummy_mm_dataN) rArBrCrrqrprvrrrrIrIrIrJrrs  rrc seZdZdedeeefdeeefdeeefdef fdd Zded eeefdeeeffd d Z d e d eee fd e de efddZZS)UltravoxMultiModalProcessorpromptmm_data mm_kwargs tok_kwargsrTc s|dgs|jj|dd}||}tt|gdddSt|}|dg}t|t s0J|jj di|}tdi||j dd}tdi|d|i}|d d|d d|d dt j ||||d } | d | d<| S)NaudiosF)add_special_tokens) input_idspt) tensor_typeT)rhinclude_audio_num_chunksrpadding truncation)rrrr audio_valuesr6rI)rtr{ get_tokenizerencode_apply_hf_processor_tokens_onlyr dictpoprbrHrdrhsuper_call_hf_processor) r^rrrr prompt_idsrrcitem_processor_dataoutput __class__rIrJrs8        z.UltravoxMultiModalProcessor._call_hf_processor hf_inputshf_processor_mm_kwargscCsH|dtd}ttd|td|td|tdtddS)Naudio_num_chunksrrm)r6audio_token_len audio_lensrrL)rtrFzerosrrflat_from_sizesbatched)r^rrr?rIrIrJ_get_mm_fields_configs   z1UltravoxMultiModalProcessor._get_mm_fields_configmm_items out_mm_kwargscs|jjd i|}|j|dtd}tj|dtjdt tj dgtjdgdt ffdd }t dd |d gS) Nrr)dimdtyperitem_idxcs6|}|d}d||}gt|S)Nr)r)sumrp)rstartendrchunks_start_idx out_mm_datareplacement_idrIrJget_replacement_ultravox s zQUltravoxMultiModalProcessor._get_prompt_updates..get_replacement_ultravoxrmr3)modalitytarget replacementrI) r{rYr]get_datartrFrcumsumint32cattensorrpr#)r^rrrr`r?rrIrrJ_get_prompt_updatess"z/UltravoxMultiModalProcessor._get_prompt_updates)rArBrCrqrror rrrrrrrr$r __classcell__rIrIrrJrs8   +    rcs>eZdZdZd deffdd Zdejdejfdd ZZ S) StackAudioFrameszk Stack the audio embedding frames to reduce the sequence length by a factor of `stack_factor`.  stack_factorcst||_dSr)r__init__r)r^rrrIrJrs  zStackAudioFrames.__init__rLrTcCsh|j\}}}||jd|j|j}t|ddd||f}|j\}}}||||j||j}|S)Nr)r)shaperFpadview)r^rLBTCT_padrIrIrJforward#s  zStackAudioFrames.forward)r) rArBrCrDrprrFrGrrrIrIrrJrsrc>eZdZdeffdd ZdejdejdejfddZZS) UltravoxFeedForwardProjectorr_cst|j|_t|j|_|jj|j}t||_ t j ||jdd|_ |j}|j dkr6t|_|d}nt|j |_|jj}t j ||dd|_|jrXt||_t |_dSt |_t||_dS)NF)biasswiglu)rr hidden_size hidden_dimrr_pad_and_stack audio_configrln_prer Linearlinear_1 projector_actractr text_configlinear_2projector_ln_midln_midIdentityln_post)r^r_dim_indim_middim_outrrIrJr/s$        z%UltravoxFeedForwardProjector.__init__r6rrTcCsJ||}||}||}||}||}||}||}|Sr)rrrrrrr)r^r6r hidden_statesrIrIrJrJs       z$UltravoxFeedForwardProjector.forward rArBrCr&rrFrGrrrIrIrrJr.srcr) UltravoxTransformerProjectorr_csttdd|_t|j|_|jj|j}t |jt ||_ t |j|_t jj|_t fddt|jD|_t j|_t j|jj|_dS)NF) is_decodercsg|]}tqSrI)r).0_projector_audio_configrIrJ jsz9UltravoxTransformerProjector.__init__..)rrrr_rrrrrcopydeepcopyrrr rd_model linear_in Embeddingmax_source_positionsembed_positions ModuleListrangenum_projector_layerslayersrr linear_out)r^r_rrrrJrXs(         z%UltravoxTransformerProjector.__init__r6rrTc Cs||}|jd}tj||jddddf|dddf}|||j|j}||}| |}| tj| d|jd}||}i}dt |jdjjvrXd|d<|jD]} | |fd|i|} | d}q[||}||}|S)Nr)devicelayer_head_maskrattention_mask)rrrFarangerltget_extended_attention_maskrrrrsizeinspect signaturerr parametersrr) r^r6rmax_len_stackedrextended_attention_maskr positionsrSlayer layer_outputsrIrIrJrus>          z$UltravoxTransformerProjector.forwardrrIrIrrJrWsrcsheZdZdZdZfddZeddZdej dBd ej fd d Z dd ej dej dBfd dZ Z S)ModifiedWhisperEncodera Encoder portion of OpenAI's Whisper model. This implementation is a slightly modified version of HF Transformers' Whisper Encoder, with only a few fixes: 1. base_model_prefix updated to allow for doing `.from_pretrained` directly on the encoder 2. allow less than 30 second of audio padding to be passed in: - relaxed ValueError check for `input_features` length to be less than or equal to `expected_seq_length` instead of strictly equal - embed_pos is now sliced to match the length of `inputs_embeds` Original: https://github.com/huggingface/transformers/blob/main/src/transformers/models/whisper/modeling_whisper.py See commentary: https://github.com/huggingface/transformers/issues/25744 z model.encodercstj|i|d|j_dS)NF)rrr_r)r^argsrSrrIrJrs zModifiedWhisperEncoder.__init__cCs |jj|jjd|jjdS)Nr)r_rconv1strideconv2rlrIrIrJmax_context_lengths   z)ModifiedWhisperEncoder.max_context_lengthrNrcCs`|durdS||}|jd}tj||jddddf|dd}|j|d|jd}|S)aH Create attention mask based on audio lengths to mask out padding tokens For each sample in batch: - Convert raw audio length to feature length after convolutions - Create bool mask: True for valid positions and False for padding - Convert to attention mask format expected by transformer layers (1.0 for positions to attend to, large negative for positions to ignore) This masking ensures consistent behavior between training and inference by preventing the model from attending to padding tokens in both cases Nr)rr) _get_feat_extract_output_lengthsrrFrrrrrr)r^rraudio_feature_len max_seq_lenrrIrIrJget_attention_mask_by_audio_lens    z6ModifiedWhisperEncoder.get_attention_mask_by_audio_leninput_featuresc Cs|j}|jd|krtd|d|jdd|dtj||}tj||}|ddd}|j j d| d }||}tjj ||j |j d }|||}i}d t|jdjjvrfd|d <|jD]} | ||fi|} | d}qi||}|S) Nr z7Whisper expects the mel input features to be of length z or less, but found z-. Make sure to pad the input mel features to .rrr))ptrainingr)r r ValueErrorr r gelur r permuterweightrdropoutrrrrrrr layer_norm) r^rrexpected_seq_length inputs_embeds embed_posrrrS encoder_layerrrIrIrJrsB     zModifiedWhisperEncoder.forwardr) rArBrCrDbase_model_prefixrpropertyr rFrGrrrrIrIrrJrs"   r)r{ dummy_inputsc seZdZgdddgdZeddidZeded ed ed Bfd d Z ddde deffddZ d e fddZ dejdejdejd ejfddZded ed BfddZded eeejdfBfd d!Zded efd"d#Z d7d d$d%d&ejd'ed Bd(ejd Bd)ed ejf fd*d+Z d8d&ejd Bd,ejd-ejd Bd.ejd Bd ejeBf d/d0Zd1ejd ejfd2d3Zd4eeeejfd eefd5d6Z Z!S)9 UltravoxModel)q_projk_projv_proj gate_projup_proj)qkv_proj gate_up_projzaudio_tower.model.encoder. audio_tower.)orig_to_new_prefixrirTNcCs|drdStd)Nrmr3z Only audio modality is supported) startswithr)clsrr.rIrIrJget_placeholder_strs z!UltravoxModel.get_placeholder_str)prefix vllm_configr3cs&t|jj}|jj}||_||_|jsJg|_|jdur-|j t j |jddd|j dur?|j t j |j ddd| |dt|j|_|jdkrWt||_nt||_Wdn1sfwY||t||jt|dd|_Wdn1swY|jj|_dS)Nr,) model_or_pathrevisionr3language_model.rmrlanguage_model)r4rXr3)rrrWrXmultimodal_configr_multi_modal_configsecondary_weightsaudio_model_idappendrSource text_model_id_mark_tower_modelrr audio_towerrrmulti_modal_projectorr_mark_language_modelr1wrapped_model_configr2r8make_empty_intermediate_tensors)r^r4r3r_r9rrIrJrsN          zUltravoxModel.__init__cCstjddddS)z< Get the module prefix in multimodal models r7zmulti_modal_projector.r,)r8 connector tower_model)rfrom_string_fieldrlrIrIrJget_mm_mappingLs zUltravoxModel.get_mm_mappingrrrc Cs||jj}|d}g}td|tD]-}t|t|}|||||||} | |jj} || |||} || qt j |dd}|S)Nr)r) torArrrr}minrBr=rFr) r^rrrr6 batch_sizeaudio_embeddingsrrbatch_featuresbatch_embeddingsrIrIrJ_audio_features_to_embeddingsVs   z+UltravoxModel._audio_features_to_embeddingsrScKs|dd}|dd}|dd}|dd}|dd}|dur(|dur(dS|dur5td||||dS|dur?td|dStd) Nr6rLrrr)r7r;r<r=r?)r7r;z This line should be unreachable.)rr5rKAssertionError)r^rSr6rLrrrrIrIrJ_parse_and_validate_audio_inputss$      z-UltravoxModel._parse_and_validate_audio_input audio_input.c Cs|ddkr |dSt|d}|d}|d}||||}|jd}tj||jd|jdd }||dddfk}||} d d ||d D} | | S) Nr7rLr;r<r=r)rrr cSsg|]}|qSrI)ritem)r chunk_lensrIrIrJrs z6UltravoxModel._process_audio_input..r?) pad_and_concat_to_dim3rPrrFrrexpandsplittolist) r^rSr6rr embeddingsmax_lenindicesmaskflattened_embeddings embed_lensrIrIrJ_process_audio_inputs$     z"UltravoxModel._process_audio_inputcKs*|jdi|}|durgS||}|SrU)rRr`)r^rSrSrMrIrIrJembed_multimodals  zUltravoxModel.embed_multimodalT) is_multimodalhandle_oov_mm_tokenrmultimodal_embeddingsrbrccs0|dus|durt|Stj||||dS)N)rdrbrc)rembed_input_ids)r^rrdrbrcrrIrJres zUltravoxModel.embed_input_idsrintermediate_tensorsrcKs8|durd}|j}t|dr|j}|j||||d}|S)aRun forward pass for Ultravox One key thing to understand is the `input_ids` already accounts for the positions of the to-be-inserted audio embeddings. The to-be-inserted audio has a size that is essentially 6.25 tokens per second of audio. This way, the `positions` and `attn_metadata` are consistent with the `input_ids`. Args: input_ids: Flattened (concatenated) input_ids corresponding to a batch. positions: Position indices for the input tokens. intermediate_tensors: Intermediate tensors from prior forward pass. inputs_embeds: Optional tensor of input embeddings. Nr8)r)r8hasattrmodel)r^rrrfrrSr8rrIrIrJrs zUltravoxModel.forwardrcCs |j|Sr)r8compute_logits)r^rrIrIrJris zUltravoxModel.compute_logitsweightscCst|dgd}|j||jdS)Nr,)ignore_unexpected_prefixes)mapper)r. load_weightshf_to_vllm_mapper)r^rjloaderrIrIrJrmszUltravoxModel.load_weightsr)NN)"rArBrCpacked_modules_mappingr/rn classmethodrqrpr1rrrrIrFrGrProrQrRrtupler`r*raboolrer%rrirsetrmrrIrIrrJr$sz0    &  &,r$featuresrTcslt|tjr|jdkrt|}|Sdd|D}tdd|Ddd|D}fdd|D}t|S)a Pad and concatenate a list of tensors. output: Tensor of shape [B, C, M] where M is the maximum length of the input tensors, B is the sum of the batch sizes of the input tensors. C must be the same for all input tensors. cSsg|]}t|qSrI)rVrfrIrIrJrsz*pad_and_concat_to_dim3..css|]}|jdVqdS)r N)rrwrIrIrJ sz)pad_and_concat_to_dim3..cSs(g|]}|jdg|jddRqS)r rN)rrrwrIrIrJrs(c s&g|]}t|d|jdfqS)rr )rrrrwr[rIrJrs&)rbrFrGndimr0maxr)rurIrzrJrVs  rV)_rDrrcollections.abcrrrtypesrtypingrrrr rFr torch.nnr r transformersr r transformers.modeling_utilsrtransformers.models.whisperr,transformers.models.whisper.modeling_whisperrr vllm.configrvllm.config.multimodalr%vllm.model_executor.layers.activationrr$vllm.model_executor.layers.layernormr vllm.model_executor.model_loaderr)vllm.model_executor.models.module_mappingrvllm.multimodalrvllm.multimodal.inputsrrrrvllm.multimodal.parserrvllm.multimodal.processingr r!r"r#r$ vllm.sequencer%(vllm.transformers_utils.configs.ultravoxr&vllm.utils.tensor_schemar'r( interfacesr*r+r,r-utilsr.r/r0r1r2rZr}r5rKrQrErRrrrModulerrrrregister_processorr$rGrHrVrIrIrIrJsj              *c)Fir