o ei(U@sddlZddlmZddlZddlmZddlmZddlmZddl m Z ddl m Z dd l mZmZmZdd lmZmZdd lmZdd lmZmZmZmZdd lmZddlmZddlm Z m!Z!ddl"m#Z#m$Z$e%e&Z'  d-dej(dej)dej)dej)dej)dBde*dBde*fddZ+Gdddej(Z,Gdd d e Z-eGd!d"d"eZ.ed#d$Gd%d&d&e.Z/Gd'd(d(ej(Z0ed)d$Gd*d+d+e.e Z1gd,Z2dS).N)Callable)nn)ACT2FN)Cache)GenerationMixin)GradientCheckpointingLayer)BaseModelOutputWithPastBaseModelOutputWithPoolingCausalLMOutputWithPast)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tuplelogging)merge_with_config_defaults)capture_outputs) AutoModelAutoModelForCausalLM) VoxtralConfigVoxtralEncoderConfigmodulequerykeyvalueattention_maskscalingdropoutc Ks|dur |dd}t||dd|}|dur||}tjj|dd}tjj|||jd}t||} | dd } | |fS)Nrr)dimptrainingr) sizetorchmatmul transposer functionalsoftmaxr"r( contiguous) rrrrr r!r"kwargs attn_weights attn_outputr3j/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/voxtral/modeling_voxtral.pyeager_attention_forward-s  r5cseZdZdZ      ddededed ed ed ed edBd edBffdd Zde j dedefddZ  dde j de j dBdede e j e j dBe e j dBffddZ ZS)VoxtralAttentionz=Multi-headed attention from 'Attention Is All You Need' paperrFTN embed_dim num_headsr" is_decoderbias is_causal layer_idxconfigc st||_||_||_|||_||_|j||jkr*td|jd|d|jd|_||_ ||_ |durG|rGt d|j jd||_tj||dd|_tj|||d|_tj|||d|_tj|||d|_dS) Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).r$zInstantiating a decoder z without passing `layer_idx` is not recommended and will to errors during the forward call, if caching is used. Please make sure to provide a `layer_idx` when creating this class.Fr:)super__init__r7r8r"head_dimr= ValueErrorr!r9r;logger warning_once __class____name__r<rLineark_projv_projq_projout_proj) selfr7r8r"r9r:r;r<r=rEr3r4r@Js0     zVoxtralAttention.__init__tensorseq_lenbszcCs ||||j|jddS)Nrr)viewr8rAr,r/)rLrNrOrPr3r3r4_shapers zVoxtralAttention._shape hidden_statesr output_attentionsreturncKs|\}}}||||j||}|||d|} |||d|} t|jj t } | ||| | |f|j ssz+VoxtralEncoder.__init__..)r)r?r@r"encoder_layerdrop layerdroprl num_mel_binsmax_source_positionsscale_embeddingmathsqrt embed_scalerConv1dconv1conv2 Embeddingembed_positionsrequires_grad_ ModuleListrangeencoder_layerslayersrp layer_norm AvgPool1d avg_poolergradient_checkpointing post_init)rLr=r7rMrr4r@s    zVoxtralEncoder.__init__cCs|D]}d|_qd|_dSra) parameters requires_grad_requires_grad)rLparamr3r3r4_freeze_parameterss  z!VoxtralEncoder._freeze_parametersrUcCs|jSNrrLr3r3r4get_input_embeddings sz#VoxtralEncoder.get_input_embeddingsrcCs ||_dSrrrLrr3r3r4set_input_embeddings# z#VoxtralEncoder.set_input_embeddingsNr0c Ks|jj|jjd|jjd}|jd|kr(td|d|jdd|d|j|jjj |jjj d}t j ||}t j ||}|ddd }|jj}|||j }t j j||j|jd }t|jD]\}} | ||d } | d}qj||}t|d S) a Args: input_features (`torch.LongTensor` of shape `(batch_size, feature_size, sequence_length)`): Float values of mel features extracted from the raw speech waveform. Raw speech waveform can be obtained by loading a `.flac` or `.wav` audio file into an array of type `list[float]` or a `numpy.ndarray`, *e.g.* via the soundfile library (`pip install soundfile`). To prepare the array into `input_features`, the [`AutoFeatureExtractor`] should be used for extracting the mel features, padding and conversion into a tensor of type `torch.FloatTensor`. See [`~WhisperFeatureExtractor.__call__`] attention_mask (`torch.Tensor`)`, *optional*): Voxtral does not support masking of the `input_features`, this argument is preserved for compatibility, but it is not used. By default the silence in the input log mel spectrogram are ignored. rr#z7Voxtral expects the mel input features to be of length z , but found z-. Make sure to pad the input mel features to .)r|devicerrr&)r )last_hidden_state)r=rrrrshaperBtoweightr|rrr-gelupermuterr"r( enumeraterrr ) rLrr r0expected_seq_length inputs_embeds embed_posrSidx encoder_layer layer_outputsr3r3r4r`&s,   zVoxtralEncoder.forward input_lengthscCs(|ddd}|ddd}||fS)zs Computes the output length of the convolutional layers and the output length of the audio encoder rrr3)rLroutput_lengthsr3r3r4 _get_feat_extract_output_lengthsWsz/VoxtralEncoder._get_feat_extract_output_lengthsr)rFrbrcrdrrmain_input_namerrr6rk_can_record_outputsr@rrModulerrrrrrrir r`r* LongTensorrrjr3r3rMr4rs,  /rcs*eZdZdeffdd ZddZZS)VoxtralMultiModalProjectorr=csNttj|jj|jjdd|_t |j |_ tj|jj|jjdd|_ dS)NFr>) r?r@rrG audio_configintermediate_size text_config hidden_sizelinear_1rprojector_hidden_actactlinear_2ryrMr3r4r@as  z#VoxtralMultiModalProjector.__init__cCs"||}||}||}|Sr)rrr)rLaudio_featuresrSr3r3r4r`gs   z"VoxtralMultiModalProjector.forward)rFrbrcrr@r`rjr3r3rMr4r`srzs The Voxtral model, which consists of Whisper encoder, a multi-modal projector and a LLama language model. cs eZdZdgZfddZddZddZdd Zd d Zd d Z ddZ e e ddde jdeedeeBfddZe e          d&de jdBde jdBde jdBde jdBdedBde jdBde jdBdedBd e jdBd!ee jBdeedefd"d#Zfd$d%ZZS)'VoxtralForConditionalGenerationrcsHt||jj|_t|j|_t|j|_ t ||_ | dSr) r?r@r vocab_sizer from_configr audio_towerrlanguage_modelrmulti_modal_projectorrryrMr3r4r@vs    z(VoxtralForConditionalGeneration.__init__cC |jSr)rrrr3r3r4rrz4VoxtralForConditionalGeneration.get_input_embeddingscC|j|dSr)rrrr3r3r4rz4VoxtralForConditionalGeneration.set_input_embeddingscCrr)rget_output_embeddingsrr3r3r4rrz5VoxtralForConditionalGeneration.get_output_embeddingscCrr)rset_output_embeddings)rLnew_embeddingsr3r3r4rrz5VoxtralForConditionalGeneration.set_output_embeddingscCrr)r set_decoder)rLdecoderr3r3r4rrz+VoxtralForConditionalGeneration.set_decodercCrr)r get_decoderrr3r3r4rrz+VoxtralForConditionalGeneration.get_decoderzThis method is used to get the audio embeddings from input features (a log mel spectrogram), meaning inferring the audio encoder and the multi-modal projector.rrr0rUcKsB|j|fddi|}|j}|d|jjj}||}||_|S)aa input_features (`torch.FloatTensor`): Float values of mel features extracted from the raw speech waveform. Raw speech waveform can be obtained by loading a `.flac` or `.wav` audio file into an array of type `list[float]` or a `numpy.ndarray`, *e.g.* via the soundfile library (`pip install soundfile`). To prepare the array into `input_features`, the [`AutoFeatureExtractor`] should be used for extracting the mel features, padding and conversion into a tensor of type `torch.FloatTensor`. See [`~WhisperFeatureExtractor.__call__`] return_dictTr#)rrrYr=rrr pooler_output)rLrr0 audio_outputsaudio_hidden_states audio_embedsr3r3r4get_audio_featuress  z2VoxtralForConditionalGeneration.get_audio_featuresNr input_idsr position_idsrrlabels use_cachecache_positionlogits_to_keepc  Ks|dur ||}|dur1|dur1|j|ddj} ||jjkd} || |j| |j}|j d||||||| | d| }|S)aj Example: ```python >>> from transformers import VoxtralForConditionalGeneration, AutoProcessor >>> import torch >>> device = "cuda" if torch.cuda.is_available() else "cpu" >>> repo_id = "mistralai/Voxtral-Mini-3B-2507" >>> processor = AutoProcessor.from_pretrained(repo_id) >>> model = VoxtralForConditionalGeneration.from_pretrained(repo_id, dtype=torch.bfloat16, device_map=device) >>> conversation = [ { "role": "user", "content": [ { "type": "audio", "url": "https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/dude_where_is_my_car.wav", }, {"type": "text", "text": "What can you tell me about this audio?"}, ], } ] >>> inputs = processor.apply_chat_template(conversation) >>> inputs = inputs.to(device, dtype=torch.bfloat16) >>> outputs = model.generate(**inputs, max_new_tokens=30) >>> processor.batch_decode(outputs[:, inputs.input_ids.shape[1]:], skip_special_tokens=True) ["This audio is a humorous conversation between two friends, likely in English, where one of them is trying to figure out what the other's tattoo says."] ```NT)rr#)r rrrrrrrr3) rrrr=audio_token_id unsqueezemasked_scatterrrr)rLrrr rrrrrrrr0raudio_token_maskoutputsr3r3r4r`s*1   z'VoxtralForConditionalGeneration.forwardcsF|dd}|dd}tj|i|}|s|dds!||d<|S)Nris_first_iterationFrT)popgetr?prepare_inputs_for_generation)rLargsr0rr model_inputsrMr3r4rs  z=VoxtralForConditionalGeneration.prepare_inputs_for_generation) NNNNNNNNNr)rFrbrc_keep_in_fp32_modules_strictr@rrrrrrrrr* FloatTensorrrrir rrrhrrgrer r`rrjr3r3rMr4rnst       Hr)rrr)Nr)3rcollections.abcrr*r activationsr cache_utilsr generationrmodeling_layersrmodeling_outputsr r r modeling_utilsr r processing_utilsrutilsrrrr utils.genericrutils.output_capturingrautorrconfiguration_voxtralrr get_loggerrFrCrrhrfr5r6rkrrrr__all__r3r3r3r4sb            X;q