o -i< @sddlZddlZddlmZmZmZddlmZddlm Z m Z m Z ddl Z ddlZddlmZddlmZmZmZddlmZddlmZdd lmZdd lmZmZmZmZdd lm Z dd l!m"Z"dd l#m$Z$ddl%m&Z&ddl'm(Z(ddl)m*Z*ddl+m,Z,ddl-m.Z.m/Z/m0Z0ddl1m2Z2ddl3m4Z4ddl5m6Z6ddl7m8Z8ddl9m:Z:ddl;mZ>m?Z?m@Z@ddlAmBZBmCZCddlDmEZEmFZFmGZGmHZHmIZIddlJmKZKddlLmMZMddlNmOZOmPZPddlQmRZRdd lSmTZTd!d"lUmVZVmWZWmXZXd!d#lYmZZZm[Z[m\Z\m]Z]m^Z^e&e_Z`Gd$d%d%ejaZbGd&d'd'eOZcGd(d)d)e,ZdGd*d+d+ejeZfGd,d-d-ejgZhGd.d/d/ehZiGd0d1d1ejgZjGd2d3d3ejgZkGd4d5d5ejgZlGd6d7d7ejgZmedd8d9d:Gd;d<dd>ejgZoGd?d@d@eFZpGdAdBdBeEepZqGdCdDdDeGepZre?r4) kernel_sizepaddingrX)striderrrct|ddSN.layersrrv)rrwrrGrH z)WhisperEncoder.__init__..rrwz\Only sinusoidal or learned position embeddings are supported for non-causal models, but got ))r[rhrrrr=rpos_embed_type num_mel_binsmax_source_positionsscale_embeddingmathsqrt embed_scaler Conv1dconv1conv2r total_strider;encoder_layers start_layer end_layerlayersr layer_normrDrFrr2rPfloat32rno_grad Embeddingembed_positionsricopy_r shape)r^rrvrrromaybe_fp32_init_ctxrarrHrhsL    PzWhisperEncoder.__init__rMrWcCsg}d}|D]9}tj||}tj||}|dd}||jjd|dddf |j }| ||j dk}q|rHt |}nt j|dd}|jD]}||}qR||}|S)NFrrXrr)r functionalgelurr transposerrisizetodtypeappendndimrPcatstackrr)r^rMrinput_is_batchedfeaturesembeds encoder_layerrGrGrHr\s$ "      zWhisperEncoder.forward) rArBrCrrrrhrPrQrOr\rcrGrGrarHrs1rr) input_ids positions)dynamic_arg_dimscs\eZdZdddedeffddZdejdejd Bfd d Zd ejd ejfddZ Z S)WhisperDecoderrnrwrrvcstjj}|j|_|j|_|j|_|j |_ |j r"t |j nd|_t|j|j |j|_t|j|j |_t|jfdd|dd\|_|_|_t|j |_dS)Nrcrr)rrwrrGrHrrz)WhisperDecoder.__init__..rrw)r[rhrrdecoder_layerdrop layerdrop pad_token_id padding_idxmax_target_positionsrrrrrrr r vocab_size embed_tokensrdrr;decoder_layersrrrrr)r^rrvrrarrHrhs&   zWhisperDecoder.__init__rrNcCsB||}||}||}|jD]}|||d}q||}|S)N)r)embed_input_idsrrr)r^rrr inputs_embedsr decoder_layerrGrGrHr\&s    zWhisperDecoder.forwardrrWcCs ||Srg)r)r^rrGrGrHr9rkzWhisperDecoder.embed_input_ids) rArBrCrrrhrPrQr\rrcrGrGrarHr s rcseZdZdddedeffddZdejdBd ejd eejd ejfd d Z dejeejBdBd ejdBfddZ de e eejfd e efddZZS) WhisperModelrnrwrrvcs6tt||dd|_t||dd|_dS)Nz.encoderrz.decoder)r[rhrencoderrdecoder)r^rrvrarGrHrh>s    zWhisperModel.__init__rNrencoder_outputsrWcCs.t|r tj|ddnd}|j|||d}|S)Nrr)rrr)lenrPrr )r^rrr  enc_statesdecoder_outputsrGrGrHr\GszWhisperModel.forwardrMcCs|durdS||Srg)r )r^rMrGrGrHget_encoder_outputsUs z WhisperModel.get_encoder_outputsweightsc Csgd}t|}t}|D]M\}}|D](\}}} ||vrq|||}|dr/||vr/q||} | j} | | || n|drH||vrHq||} t| dt} | | |||q|S)N)).self_attn.qkv_projz.self_attn.q_projr)rz.self_attn.k_projr)rz.self_attn.v_projr).encoder_attn.kv_projz.encoder_attn.k_projr)rz.encoder_attn.v_projrz.bias weight_loader) dictnamed_parameterssetreplaceendswithrrr!add) r^rstacked_params_mapping params_dict loaded_paramsname loaded_weight param_name weight_nameshard_idparamrrGrGrH load_weights]s*       zWhisperModel.load_weights)rArBrCrrrhrPrQrOr\rrtuplerr$rcrGrGrarHr =s"   ,r c@steZdZdefddZedefddZdee e dBffddZ d e de fd d Zde fd d Zde fddZdS)WhisperProcessingInforWcCs |jtSrg)ctx get_hf_configr r^rGrGrHr(s z#WhisperProcessingInfo.get_hf_configcCdS)NTrGr)rGrGrHskip_prompt_length_checkz.WhisperProcessingInfo.skip_prompt_length_checkNcCsddiS)Naudior4rGr)rGrGrHget_supported_mm_limitssz-WhisperProcessingInfo.get_supported_mm_limitskwargscKs(|jdi|}|j}t|tsJ|S)NrG)get_hf_processorfeature_extractor isinstancer )r^r/ hf_processorr1rGrGrHget_feature_extractorsz+WhisperProcessingInfo.get_feature_extractorcCr*)z7Return target audio channels for Whisper models (mono).r4rGr)rGrGrHget_target_channelsr,z)WhisperProcessingInfo.get_target_channelscCs |jSrg)r(rr)rGrGrHget_num_audio_tokensrkz*WhisperProcessingInfo.get_num_audio_tokens)rArBrCr r(propertyrr+rrrlr.objectr r4r5r6rGrGrGrHr&sr&c @sXeZdZdeeefdefddZ d dedeeefdeeefdBdefdd Z dS) WhisperDummyInputsBuilder mm_countsrWcCs|dd}d|S)Nr-rz<|startoftranscript|>)get)r^r: num_audiosrGrGrHget_dummy_texts z(WhisperDummyInputsBuilder.get_dummy_textNseq_len mm_optionsc CsL|j}|j}|j|}|dd}|r|dnd}d|j|||diS)Nr-r)lengthr< overrides)infor4 sampling_rate chunk_lengthr;_get_dummy_audios) r^r>r:r?r1rC audio_lenr<audio_overridesrGrGrHget_dummy_mm_datas   z+WhisperDummyInputsBuilder.get_dummy_mm_datarg) rArBrCrrrlr=rr$rHrGrGrGrHr9s  r9c seZdZdefddZdeeeBdedeeeBfddZ dede ee fde ee fd e ee fde f fd d Z d e d e ee fde eeffddZded e ee fdedeefddZZS)WhisperMultiModalProcessorrWcCs|j}t|j|jdS)N) target_srtarget_channels)rBr4r(rCr5)r^r1rGrGrH_get_data_parsers z+WhisperMultiModalProcessor._get_data_parserpromptmm_datacCsdgS)NrrG)r^rMrNrGrGrHcreate_encoder_prompts z0WhisperMultiModalProcessor.create_encoder_prompt mm_kwargs tok_kwargscsl|r|jjdi|}t|dd}tdi|d|ji}tj||||d}d|vr4|d|d<|S)Naudios)r-rC)rMrNrPrQlabelsrrG)rBr4rpoprCr[_call_hf_processor)r^rMrNrPrQr1processed_outputsrarGrHrUs"z-WhisperMultiModalProcessor._call_hf_processor hf_inputshf_processor_mm_kwargscCsttddS)Nr-rM)rr%batched)r^rWrXrGrGrH_get_mm_fields_configsz0WhisperMultiModalProcessor._get_mm_fields_configmm_items out_mm_kwargscCs"|j}tddgdg|dgS)Nr-r)modalitytarget replacement)rBr6r,)r^r\rXr] num_tokensrGrGrH_get_prompt_updatess z.WhisperMultiModalProcessor._get_prompt_updates)rArBrCr(rLrrOrlr$rOrr8r rUr%r[r'r&rr-rbrcrGrGrarHrIsH           rI)rB dummy_inputscseZdZgdddgdZeddddZd Zd ZeZ e d e d Bd e d Bffd d Z e de jdeded e d Bdedde de d Bd efddZe de ded e d BfddZe dede d efddZe dededed ed Bfdd Zd!d"d#ed$e ffd%d&Z d>d'ejd(ejd)eejd Bd ejfd*d+Zd,ed e fd-d.Z! d>d d/d0d'ejd1e d Bd2ejd Bd3e"d ejf d4d5Z#d,ed e$fd6d7Z%d8ejd ejfd9d:Z&d;e'e(e ejfd e)e fdz<|startoftranscript|><|z|>z<|z|><|notimestamps|>)encoder_promptdecoder_prompt)r sample_raterr) rkr-rrlrfrmrprqrMrGrGrHget_generation_prompts     z5WhisperForConditionalGeneration.get_generation_promptr^icCs|drdStd)Nr-z Only audio modality is supported) startswithr)rkr^rwrGrGrHget_placeholder_str>s z3WhisperForConditionalGeneration.get_placeholder_strcCst|}t|jj|jjdS)N)max_audio_clip_sru)r.rr1rDrC)rkrrm processorrGrGrHget_speech_to_text_configEs z9WhisperForConditionalGeneration.get_speech_to_text_configaudio_duration_scCs0t|}|jj}|dusJt||j|Srg)r.r1 hop_lengthrceilru)rkr}rlrr{r~rGrGrHr6Ps z4WhisperForConditionalGeneration.get_num_audio_tokensrnrwrrvcst|jj}|j}||_|jj|_|j|tdt idt ||d|_ Wdn1s0wYt |j |j|t|dd|_|j|j jj|_t|dd}t|j |d|_dS) Nr-)language_targets tower_targetsrproj_out)rurv logit_scaler)scale)r[rhrrrurr_mark_composite_modelrrr modelr rrr<r tie_weightsr rrrlogits_processor)r^rrvrrurrarGrHrh`s*   z(WhisperForConditionalGeneration.__init__rrr cKs |durg}|j|||d}|S)N)rrr )r)r^rrr r/rrGrGrHr\xsz'WhisperForConditionalGeneration.forwardr/cKs,|jdi|}|j|d}|jddS)NrMrrrG)_parse_and_validate_audio_inputrrunbind)r^r/ audio_input enc_outputrGrGrHembed_multimodals z0WhisperForConditionalGeneration.embed_multimodalF) is_multimodalhandle_oov_mm_tokenmultimodal_embeddingsrrcCs|jj|Srg)rr r)r^rrrrrGrGrHrs z/WhisperForConditionalGeneration.embed_input_idsc s0|dd}|durtfdd|}t|dS)NrMcs |jSrg)rr)xr)rGrHrs zQWhisperForConditionalGeneration._parse_and_validate_audio_input..rY)rTr/rI)r^r/rMrGr)rHrs  z?WhisperForConditionalGeneration._parse_and_validate_audio_inputrcCs||j|}|Srg)rr)r^rlogitsrGrGrHcompute_logitssz.WhisperForConditionalGeneration.compute_logitsrcCs(t|dgd}t|d}|j||jdS)Nz proj_out.) skip_prefixesz.k_proj.weight)mapper)r8_create_fake_bias_for_k_projr$hf_to_vllm_mapper)r^rloaderrGrGrHr$s z,WhisperForConditionalGeneration.load_weightsrg),rArBrCpacked_modules_mappingr9rsupports_transcription_onlysupports_segment_timestampr"supported_languages classmethodrrjnpndarrayrrrrrvrlryr|floatr6rrhrPrQrOr\r8r5rrrrIrrrr%rr$rcrGrGrarHrds             ,rdrfake_bias_key_namerWccsZ|D]'\}}||r%t|d}|dd}||f||fgEdH||fVqdS)z Create full zeros bias for k_proj weight in self-attn and x-attn layers. So that the bias for k_proj in qkv_proj can be initialized with zeros. rrirqN)rrPzerosrr)rrrrirq bias_namerGrGrHrs    r)yenumrcollections.abcrrr contextlibrtypingrrrnumpyrrPr transformersr r r ,transformers.models.whisper.modeling_whisperr vllm.attention.layerrvllm.compilation.decoratorsr vllm.configrrrrvllm.config.multimodalrvllm.distributedrvllm.inputs.datar vllm.loggerr%vllm.model_executor.layers.activationr4vllm.model_executor.layers.attention.cross_attentionr9vllm.model_executor.layers.attention.mm_encoder_attentionr!vllm.model_executor.layers.linearrrr+vllm.model_executor.layers.logits_processorr'vllm.model_executor.layers.quantizationr3vllm.model_executor.layers.vocab_parallel_embeddingr -vllm.model_executor.model_loader.weight_utilsr!(vllm.model_executor.models.whisper_utilsr"vllm.multimodalr#vllm.multimodal.inputsr$r%r&vllm.multimodal.parser'r(vllm.multimodal.processingr)r*r+r,r-!vllm.transformers_utils.processorr.vllm.utils.jsontreer/vllm.utils.tensor_schemar0r1vllm.utils.torch_utilsr2vllm.v1.attention.backendr3 interfacesr5r6r7utilsr8r9r:r;r<rArhEnumr=rIrSrrdModulermrrrrrrr r&r9rIregister_processorrdr%rrQrrGrGrGrHs                      mB .;P0DB 5