o ei3@sJddlZddlmZddlmZddlmZddlmZddlm Z m Z ddl m Z dd l mZmZmZmZdd lmZdd lmZd d lmZmZd dlmZmZd dlmZmZddlm Z e!e"Z#GdddeZ$GdddeZ%GdddeZ&eddGdddeZ'GdddeZ(eddGdd d eZ)gd!Z*dS)"N)nn)ACT2FN)Cache)create_bidirectional_mask)BaseModelOutputWithPoolingCausalLMOutputWithPast)Unpack)TransformersKwargsauto_docstringcan_return_tuplelogging)merge_with_config_defaults)capture_outputs)Qwen2AudioEncoderQwen2AudioPreTrainedModel)VoxtralForConditionalGenerationVoxtralMultiModalProjector)WhisperAttentionWhisperEncoderLayer)AudioFlamingo3Configc@ eZdZdS)AudioFlamingo3AttentionN__name__ __module__ __qualname__rrw/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/audioflamingo3/modular_audioflamingo3.pyr'rc@r)AudioFlamingo3EncoderLayerNrrrrr r"+r!r"c@r)AudioFlamingo3PreTrainedModelNrrrrr r#/r!r#zT The audio model from AudioFlamingo3 without any head or projection on top.  custom_introc @sHeZdZdZeedZee d de j de j dBde e BfddZ dS) AudioFlamingo3EncoderzY AudioFlamingo3 encoder: Whisper encoder, average pool (time/2), then LayerNorm. ) hidden_states attentionsNinput_featuresinput_features_maskreturnc Ks|jdddd}|d}|ddd}tj||jd|dddfk}tj||}tj| |}| ddd}||j j }tjj ||j |jd}t|j||d}|jD]} |joitg|jk} | ss| ||d}q]| ddd}|| ddd}||}t|d S) ap Args: input_features (`torch.FloatTensor` of shape `(batch_size, feature_size, sequence_length)`): Log-Mel features extracted from raw audio. Use the processor/feature extractor to compute and pad these features from waveform input. input_features_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding feature indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. rrdeviceNr)ptraining)config inputs_embedsattention_mask)last_hidden_state)shapesumtorcharanger.r functionalgeluconv1conv2permuteembed_positionsweightdropoutr0rr1layersrand layerdrop avg_pooler layer_normr) selfr)r*kwargsseq_leninput_features_lengthsr2r'r3layerdroprrr forwardBs2     zAudioFlamingo3Encoder.forwardN)rrr__doc__r"r_can_record_outputsrrr7TensortuplerrLrrrr r&3sr&cs&eZdZdZdeffdd ZZS)!AudioFlamingo3MultiModalProjectorz Audio adaptor (small MLP) that projects AudioFlamingo3Encoder features to the LLM embedding space so they can replace `` tokens. r1csRttj|jj|jj|jd|_t |j |_ tj|jj|jj|jd|_ dS)N)bias) super__init__rLinear audio_config hidden_size text_configprojector_biaslinear_1rprojector_hidden_actactlinear_2rFr1 __class__rr rUs   z*AudioFlamingo3MultiModalProjector.__init__)rrrrNrrU __classcell__rrr`r rRzsrRz The AudioFlamingo3 model which consists of a fine-tuned Whisper encoder, a multi-modal projector and a Qwen2 language model. cseZdZdZdZdZfddZeeddde j de j de e d eeBfd d Zee           dd e jdBde j dBde j dBde j dBde jdBdedBde j dBde jdBdedBde jdBdee j Bde e d efddZfddZZS)&AudioFlamingo3ForConditionalGenerationNcst|dSrM)rTrUr_r`rr rUsz/AudioFlamingo3ForConditionalGeneration.__init__zThis method is used to get the audio embeddings from input features (a log mel spectrogram), meaning inferring the audio encoder and the multi-modal projector.r$r)r*rGr+cKs|j|f|dd|}||j}|dddd}tj|jd|jddddf|dddfk}|||j}||_ |S)a  input_features (`torch.FloatTensor`): Float values of mel features extracted from the raw speech waveform. Raw speech waveform can be obtained by loading a `.flac` or `.wav` audio file into an array of type `list[float]` or a `numpy.ndarray`, *e.g.* via the soundfile library (`pip install soundfile`). To prepare the array into `input_features`, the [`AutoFeatureExtractor`] should be used for extracting the mel features, padding and conversion into a tensor of type `torch.FloatTensor`. See [`~WhisperFeatureExtractor.__call__`] input_features_mask (`torch.Tensor` of shape `(batch_size, feature_sequence_length)`): Mask to avoid performing attention on padded feature indices. T)r* return_dictr,rrr-N) audio_towermulti_modal_projectorr4r6r7r8r5r.to pooler_output)rFr)r*rG audio_output audio_embeds post_lengths valid_maskrrr get_audio_featuress 2z9AudioFlamingo3ForConditionalGeneration.get_audio_featuresr input_idsr3 position_idspast_key_valuesr2labels use_cachecache_positionlogits_to_keepc  Ks|dur ||}|dur2|dur2|j||ddj} ||jjkd}|||j| |j}|j d|||||| | | d| }|S)a+ input_features_mask (`torch.Tensor` of shape `(batch_size, feature_sequence_length)`): Mask to avoid performing attention on padding feature indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AudioFlamingo3ForConditionalGeneration, AutoProcessor >>> model_id = "nvidia/audio-flamingo-3-hf" >>> processor = AutoProcessor.from_pretrained(model_id) >>> model = AudioFlamingo3ForConditionalGeneration.from_pretrained(model_id, device_map="auto") >>> conversations = [ >>> [ >>> { >>> "role": "user", >>> "content": [ >>> {"type": "text", "text": "Transcribe the input speech."}, >>> { >>> "type": "audio", >>> "path": "https://huggingface.co/datasets/nvidia/AudioSkills/resolve/main/assets/t_837b89f2-26aa-4ee2-bdf6-f73f0dd59b26.wav", >>> }, >>> ], >>> } >>> ], >>> [ >>> { >>> "role": "user", >>> "content": [ >>> { >>> "type": "text", >>> "text": "This track feels really peaceful and introspective. What elements make it feel so calming and meditative?", >>> }, >>> {"type": "audio", "path": "https://huggingface.co/datasets/nvidia/AudioSkills/resolve/main/assets/FPSbCAANfbJLVSwD.mp3"}, >>> ], >>> } >>> ], >>> ] >>> inputs = processor.apply_chat_template( >>> conversations, >>> tokenize=True, >>> add_generation_prompt=True, >>> return_dict=True, >>> ).to(model.device) >>> outputs = model.generate(**inputs, max_new_tokens=500) >>> decoded_outputs = processor.batch_decode( >>> outputs[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True >>> ) >>> print(decoded_outputs) ["The spoken content of the audio is...", "The track's calming and meditative feel can be attributed to..."] ```NT)rdr,)r2r3rorprqrrrsrtr) get_input_embeddingsrmrhr1audio_token_id unsqueezemasked_scatterrgr.language_model)rFrnr)r*r3rorpr2rqrrrsrtrGrjaudio_token_maskoutputsrrr rLs*P   z.AudioFlamingo3ForConditionalGeneration.forwardcsl|dd}|dd}|d}tj|i|}|dur4|ddkr4|dur,||d<|dur4||d<|S)Nr)r*rsr)popgetrTprepare_inputs_for_generation)rFargsrGr)r*rs model_inputsr`rr r~$s   zDAudioFlamingo3ForConditionalGeneration.prepare_inputs_for_generation) NNNNNNNNNNr)rrr_tp_plan_pp_plan_keep_in_fp32_modules_strictrUr r r7 FloatTensorrPr r rQrrm LongTensorrboolintrrLr~rbrrr`r rcsv      grc)rcr#r&)+r7r activationsr cache_utilsr masking_utilsrmodeling_outputsrrprocessing_utilsr utilsr r r r utils.genericrutils.output_capturingr qwen2_audio.modeling_qwen2_audiorrvoxtral.modeling_voxtralrrwhisper.modeling_whisperrrconfiguration_audioflamingo3r get_loggerrloggerrr"r#r&rRrc__all__rrrr s:         B (