o ۷i/@sHddlZddlmZmZddlZddlmZddlmZddlm Z ddl m Z ddl m Z mZmZdd lmZdd lmZmZmZdd lmZd d lmZmZd dlmZmZmZmZddl m!Z!GdddeZ"GdddeZ#GdddeZ$eddGdddeZ%Gdddej&Z'eddGddde$e Z(gd Z)dS)!N)OptionalUnion)nn)ACT2FN)Cache)GenerationMixin)BaseModelOutputBaseModelOutputWithPastCausalLMOutputWithPast)Unpack)TransformersKwargsauto_docstringcan_return_tuple)check_model_inputs) AutoModelAutoModelForCausalLM)Qwen2AudioAttentionQwen2AudioEncoderQwen2AudioEncoderLayerQwen2AudioPreTrainedModel) VoxtralConfigc@ eZdZdS)VoxtralAttentionN__name__ __module__ __qualname__r r a/home/ubuntu/vllm_env/lib/python3.10/site-packages/transformers/models/voxtral/modular_voxtral.pyr'rc@r)VoxtralEncoderLayerNrr r r r!r#+r"r#c@s$eZdZdZdZdZdZdZdZdS)VoxtralPreTrainedModelTN)rrr_supports_flex_attn_supports_cache_class_supports_attention_backend_can_compile_fullgraph_no_split_modulesr r r r!r$/sr$z: The Voxtral encoder, which is a Whisper encoder. ) custom_introc@s2eZdZeedZe ddeefddZ dS)VoxtralEncoder) attentions hidden_statesNkwargsc Ks|jj|jjd|jjd}|jd|kr(td|d|jdd|d|j|jjj |jjj d}t j ||}t j ||}|ddd }|jj}|||j }t j j||j|jd }t|jD]\}} | ||d d } | d}qj||}t|d S)a Args: input_features (`torch.LongTensor` of shape `(batch_size, feature_size, sequence_length)`): Float values of mel features extracted from the raw speech waveform. Raw speech waveform can be obtained by loading a `.flac` or `.wav` audio file into an array of type `list[float]` or a `numpy.ndarray`, *e.g.* via the soundfile library (`pip install soundfile`). To prepare the array into `input_features`, the [`AutoFeatureExtractor`] should be used for extracting the mel features, padding and conversion into a tensor of type `torch.FloatTensor`. See [`~WhisperFeatureExtractor.__call__`] attention_mask (`torch.Tensor`)`, *optional*): Voxtral does not support masking of the `input_features`, this argument is preserved for compatibility, but it is not used. By default the silence in the input log mel spectrogram are ignored. rz:Qwen2Audio expects the mel input features to be of length z , but found z-. Make sure to pad the input mel features to .)dtypedevicerr)ptrainingN)attention_masklayer_head_mask)last_hidden_state)configmax_source_positionsconv1strideconv2shape ValueErrortoweightr1r2r functionalgelupermuteembed_positionsdropoutr4 enumeratelayers layer_normr ) selfinput_featuresr5r.expected_seq_length inputs_embeds embed_posr-idx encoder_layer layer_outputsr r r!forwardDs.   zVoxtralEncoder.forwardN) rrrrr#_can_record_outputsrr r rQr r r r!r+9sr+cs*eZdZdeffdd ZddZZS)VoxtralMultiModalProjectorr8csNttj|jj|jjdd|_t |j |_ tj|jj|jjdd|_ dS)NF)bias) super__init__rLinear audio_configintermediate_size text_config hidden_sizelinear_1rprojector_hidden_actactlinear_2rIr8 __class__r r!rWvs  z#VoxtralMultiModalProjector.__init__cCs"||}||}||}|SrR)r]r_r`)rIaudio_featuresr-r r r!rQ|s   z"VoxtralMultiModalProjector.forward)rrrrrWrQ __classcell__r r rbr!rTusrTzs The Voxtral model, which consists of Whisper encoder, a multi-modal projector and a LLama language model. cs4eZdZdgZddiZddgdgfiZdgZfddZd d Zd d Z d dZ ddZ ddZ ddZ dejfddZdejfddZee          d+deejdeejdeejdeejdeed eejd!eejd"eed#eejd$eeejfd%eed&efd'd(Zfd)d*ZZ S),VoxtralForConditionalGenerationzlm_head.weightlm_head colwise_repr-logitsrDcsHt||jj|_t|j|_t|j|_ t ||_ | dSrR) rVrWr[ vocab_sizer from_configrY audio_towerrlanguage_modelrTmulti_modal_projector post_initrarbr r!rWs    z(VoxtralForConditionalGeneration.__init__cC |jSrR)rmget_input_embeddingsrIr r r!rq z4VoxtralForConditionalGeneration.get_input_embeddingscC|j|dSrR)rmset_input_embeddings)rIvaluer r r!ruz4VoxtralForConditionalGeneration.set_input_embeddingscCrprR)rmget_output_embeddingsrrr r r!rxrsz5VoxtralForConditionalGeneration.get_output_embeddingscCrtrR)rmset_output_embeddings)rInew_embeddingsr r r!ryrwz5VoxtralForConditionalGeneration.set_output_embeddingscCrtrR)rm set_decoder)rIdecoderr r r!r{rwz+VoxtralForConditionalGeneration.set_decodercCrprR)rm get_decoderrrr r r!r}rsz+VoxtralForConditionalGeneration.get_decoderrJcCs0||}|j}|d|jjj}||}|S)a This method is used to get the audio embeddings from input features (a log mel spectrogram), meaning inferring the audio encoder and the multi-modal projector. Args: input_features (`torch.FloatTensor`): Float values of mel features extracted from the raw speech waveform. Raw speech waveform can be obtained by loading a `.flac` or `.wav` audio file into an array of type `list[float]` or a `numpy.ndarray`, *e.g.* via the soundfile library (`pip install soundfile`). To prepare the array into `input_features`, the [`AutoFeatureExtractor`] should be used for extracting the mel features, padding and conversion into a tensor of type `torch.FloatTensor`. See [`~WhisperFeatureExtractor.__call__`] Returns: `torch.FloatTensor`: The audio embeddings. r/)rlr7reshaper8rYrZrn)rIrJ audio_outputsaudio_hidden_states audio_embedsr r r!get_audio_featuress  z2VoxtralForConditionalGeneration.get_audio_featurescCstdt||S)NzUThe method `get_audio_embeds` is deprecated. Please use `get_audio_features` instead.)warningswarn FutureWarningr)rIrJr r r!get_audio_embedss z0VoxtralForConditionalGeneration.get_audio_embedsNr input_idsr5 position_idspast_key_valuesrLlabels use_cachecache_positionlogits_to_keepr.returnc  Ks|dur ||}|dur.|dur.||} ||jjkd} || |j| |j}|jd||||||| | d| }|S)aj Example: ```python >>> from transformers import VoxtralForConditionalGeneration, AutoProcessor >>> import torch >>> device = "cuda" if torch.cuda.is_available() else "cpu" >>> repo_id = "mistralai/Voxtral-Mini-3B-2507" >>> processor = AutoProcessor.from_pretrained(repo_id) >>> model = VoxtralForConditionalGeneration.from_pretrained(repo_id, dtype=torch.bfloat16, device_map=device) >>> conversation = [ { "role": "user", "content": [ { "type": "audio", "url": "https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/dude_where_is_my_car.wav", }, {"type": "text", "text": "What can you tell me about this audio?"}, ], } ] >>> inputs = processor.apply_chat_template(conversation) >>> inputs = inputs.to(device, dtype=torch.bfloat16) >>> outputs = model.generate(**inputs, max_new_tokens=30) >>> processor.batch_decode(outputs[:, inputs.input_ids.shape[1]:], skip_special_tokens=True) ["This audio is a humorous conversation between two friends, likely in English, where one of them is trying to figure out what the other's tattoo says."] ```Nr/)r5rrrLrrrrr ) rqrr8audio_token_id unsqueezemasked_scatterr?r2rm)rIrrJr5rrrLrrrrr.raudio_token_maskoutputsr r r!rQs*1    z'VoxtralForConditionalGeneration.forwardcsH|dd}|d}tj|i|}|dur"|ddkr"||d<|S)NrJrr)popgetrVprepare_inputs_for_generation)rIargsr.rJr model_inputsrbr r!rs  z=VoxtralForConditionalGeneration.prepare_inputs_for_generation) NNNNNNNNNr)!rrr_tied_weights_keys_tp_plan_pp_plan_keep_in_fp32_modules_strictrWrqrurxryr{r}torch FloatTensorrrrrr LongTensorTensorrboolrintr r r rQrrer r rbr!rfsh       Hrf)r$r+rf)*rtypingrrrr activationsr cache_utilsr generationrmodeling_outputsr r r processing_utilsr utilsr rr utils.genericrautorr qwen2_audio.modeling_qwen2_audiorrrrconfiguration_voxtralrrr#r$r+ModulerTrf__all__r r r r!s8        7