o zi @sddlZGdddZdS)Nc@sjeZdZdZdeejrdndfdedefddZ d ej fd d Z d efd d Z d ej fddZ dS) WhisperMixinFzopenai/whisper-base.encudacpupretrained_model_name_or_pathdevicecCsHddlm}ddlm}||_|||_|||j|_d|_dS)Nr)WhisperForConditionalGeneration)WhisperProcessorT) transformersrrwhisper_devicefrom_pretrainedwhisper_processorto whisper_modelis_initialized)selfrrrrrK/home/ubuntu/.local/lib/python3.10/site-packages/audiotools/core/whisper.py setup_whispers   zWhisperMixin.setup_whisperreturncCsddl}|js |||j}t||jj j j dddddf }| |j||jj j ddj}Wd|S1sEwY|S)zPreprocess audio signal as per the whisper model's training config. Returns ------- torch.Tensor The prepinput features of the audio signal. Shape: (1, channels, seq_len) rNpt) sampling_ratereturn_tensors)torchrrr rlistcloneresampler feature_extractorr audio_datanumpyinference_modeinput_features)rrsignal raw_speechr rrrget_whisper_featuress.    z!WhisperMixin.get_whisper_featurescCsl|js||}t||j}|jj|d}Wdn1s'wY|j |}|dS)zGet the transcript of the audio signal using the whisper model. Returns ------- str The transcript of the audio signal, including special tokens such as <|startoftranscript|> and <|endoftext|>. )inputsNr) rrr#rrr r rgenerater batch_decode)rr generated_ids transcriptionrrrget_whisper_transcript8s    z#WhisperMixin.get_whisper_transcriptcCsnddl}|js ||}|j}|||j}||}Wd|j S1s/wY|j S)zGet the last hidden state embeddings of the audio signal using the whisper model. Returns ------- torch.Tensor The Whisper embeddings of the audio signal. Shape: (1, seq_len, hidden_size) rN) rrrr#r get_encoderrr r last_hidden_state)rrr encoder embeddingsrrrget_whisper_embeddingsMs     z#WhisperMixin.get_whisper_embeddingsN)__name__ __module__ __qualname__rrrr is_availablestrrTensorr#r)r.rrrrrs  r)rrrrrrs