o %ݫiX@sxdZddlmZddlZddlZddlmZddlm Z ddl m Z dZ dZ dZd Zee Ze eZGd d d e ZdS) aThis lobe enables the integration of huggingface pretrained whisper model. Transformer from HuggingFace needs to be installed: https://huggingface.co/transformers/installation.html Authors * Adel Moumen 2022, 2024 * Titouan Parcollet 2022 * Luca Della Libera 2022 * Ha Nguyen 2023 )cached_propertyN)HFTransformersInterface) get_logger>icseZdZdZ        dBfdd Zdd ZdCd d Zd d Z dDdefddZ e dfdefddZ ddZ  dEddZ eddZeddZedd Zed!efd"d#Zed!efd$d%Zed!efd&d'Zed!efd(d)Zed!efd*d+Zed!efd,d-Zed!efd.d/Zed!efd0d1Zed!efd2d3Zed!efd4d5Zd6d7Zd8d9Zd:d;Zedd?Z e!"d@dAZ#Z$S)FWhispera This lobe enables the integration of HuggingFace pretrained Whisper model. Source paper whisper: https://cdn.openai.com/papers/whisper.pdf Transformer from HuggingFace needs to be installed: https://huggingface.co/transformers/installation.html Some part of the code also cis adapted from the official OpenAI repository: https://github.com/openai/whisper The model can be finetuned. It will download automatically the model from HuggingFace or use a local path. Arguments --------- source : str HuggingFace hub name: e.g "openai/whisper-tiny" save_path : str Path (dir) of the downloaded model. sampling_rate : int (default: 16000) Sampling rate of the audio signal. encoder_only : bool (default: False) If True, the forward function outputs the hidden states from the last transformer layer of the encoder. If False, one step of the decoder is performed and returned. freeze : bool (default: False) If True, the model is frozen. freeze_encoder : bool (default: False) If True, the encoder is frozen. output_attentions : bool (default: False) If ``True``, the forward function outputs the attention weights. By default, it is ``False`` because flash attention requires having ``output_attentions=False``. In case ``output_attentions`` is ``True``, a from-scratch attention implementation is being used, which can make the code slower and can increase the VRAM memory usage. output_all_hiddens: bool (default: False) If True, the forward function outputs the hidden states from all transformer layers of the encoder. For example whisper-base has 6 transformer layers and the output is of shape (7, B, T, C), where the output of the CNN output is added to the beginning. If False, the forward function outputs the hidden states only from the last transformer layer of the encoder. language: str (default: "en") Language token to use for the decoder. task: str (default: "transcribe") Task token to use for the decoder. It must be one of the following: - "transcribe" - "translate" Example ------- >>> model_hub = "openai/whisper-tiny" >>> save_path = "savedir" >>> sampling_rate = 16000 >>> model = Whisper(model_hub, save_path, sampling_rate) >>> tokens = torch.tensor([[1, 1]]) * model.model.config.decoder_start_token_id >>> inputs = torch.randn([1, 93680]) >>> outputs = model(inputs, tokens) rFN transcribec sbtj|||d||_||_||_||_||_| |_| |_|r>d|_ |j j |j ` d|j _ ddl } | tjn|j|dd|jrV|jpLd} |j j| |jd|j|||d|jj|_|jj|_|jj|_|jj} | jd|jjkr}| j } | jd|jjksJ|!dtj"| tj#d |j$s|jrt%&d |j j'(D] } d | _)qdSdSdS) N)source save_pathfreezer<|startoftranscript|>) bos_tokenen)languagetask) sampling_rate _mel_filtersdtypezVspeechbrain.lobes.models.huggingface_transformers.whisper - whisper encoder is frozen.F)*super__init__r encoder_onlyfreeze_encoderoutput_attentionsoutput_all_hiddensrr tokenizermodeldecodercpugccollecttorchcuda empty_cacheload_tokenizeris_multilingualset_prefix_tokensload_feature_extractorfeature_extractorn_fft_n_fft hop_length _hop_length n_samples _n_samples mel_filtersshape feature_sizeTregister_buffer as_tensorfloat32r loggerwarningencoder parameters requires_grad)selfr r rrr rrrrrr r0param __class__m/home/ubuntu/.local/lib/python3.10/site-packages/speechbrain/lobes/models/huggingface_transformers/whisper.pyrZs\        zWhisper.__init__cCs*td||D]}d|_q dS)z Freezes parameters of a model. Arguments --------- model : from AutoModel.from_config Valid HuggingFace transformers model object. z^speechbrain.lobes.models.huggingface_transformers.whisper - whisper encoder-decoder is frozen.FN)r7r8trainr:r;)r<rr=r@r@rA freeze_models  zWhisper.freeze_modelcsNfdd}jr$t |WdS1swYdS|S)a1Perform mel transformation and one step of the whisper (encoder-decoder). Arguments --------- wav : torch.Tensor A batch of audio signals to transform to features. decoder_input_ids : torch.Tensor Input tokens for the decoder. This can be language, task, etc. Please refer to the whisper paper for more details or go to the seq2seq2.py file in SpeechBrain to see how to generate the tokens with Greedy Search and/or Beam Search. Returns ------- out_encoder : torch.Tensor The output of the encoder model. decoder_logits : torch.Tensor The output of the decoder model. decoder_attn : torch.Tensor The attention values of the decoder model. csX}|}jr|Sjr|d\}}}n |\}}}|||fS)zForward pass of the model)_get_melforward_encoderrrforward_decoder)mel out_encoderdecoder_logits decoder_attn_decoder_input_idsr<wavr@rA_forwards    z!Whisper.forward.._forwardN)r r"no_grad)r<rOrNrPr@rMrAforwards  $zWhisper.forwardcCs||}||}|S)ag Compute the mel spectrogram features from the input audio waveform. Arguments --------- wav : torch.Tensor A batch of audio signals to compute mel spectrogram features from. Returns ------- torch.Tensor Mel spectrogram features computed from the input audio waveform. ) pad_or_trimlog_mel_spectrogram)r<rOmelsr@r@rArEs  zWhisper._get_melrpaddingc Cs|dkr tj|d|f}tj|j|jd}tj||j|j|dd}|dddf d}|j }||}tj |d d  }t ||d }|d d }|S) a*Compute the Mel spectrogram of a batch of input waveforms. Reference: adapted from https://github.com/openai/whisper/blob/eff383b27b783e280c089475852ba83f20f64998/whisper/audio.py#L92 Arguments --------- audio : torch.Tensor A batch of audio waveforms in 16 kHz. padding : int The number of samples to append to the end of the audio tensor. Returns ------- log_spec : torch.Tensor A tensor that contains the batch of Mel spectrograms. rdeviceT)windowreturn_complex.NrDg|=)ming @g@)nn functionalpadr" hann_windowr+rXstftr-absrclamplog10maximummax) r<audiorVrYra magnitudesfiltersmel_speclog_specr@r@rArTs" zWhisper.log_mel_spectrogramrDlengthcCs~|j||kr|j|tj||jdd}|j||kr=dg|j}d||j|f||<tj|dd|dddD}|S) aMPad or trim the Mel spectrograms as expected by the encoder. Reference: adapted from https://github.com/openai/whisper/blob/eff383b27b783e280c089475852ba83f20f64998/whisper/audio.py#L52 Arguments --------- array : torch.Tensor A tensor that contains the batch of Mel spectrograms. length : int Input tensor will be coerced to `length` number of samples. axis : int The axis along which to pad. Returns ------- array : torch.Tensor The padded tensor. rW)dimindex)rrrcSsg|] }|D]}|qqSr@r@).0sizesr_r@r@rA Csz'Whisper.pad_or_trim..NrD) r1 index_selectr"arangerXndimr]r^r_)r<arrayrlaxis pad_widthsr@r@rArS"s  zWhisper.pad_or_trimcCs*|jj||jd}|jrt|jS|jS)aTakes an input mel and return its corresponding encoder states. Returns the last hidden state of the encoder or all hidden states if output_all_hiddens is True. Arguments --------- mel : torch.Tensor (signal) A batch of audio mel to transform to features. Returns ------- torch.Tensor The last hidden state of the encoder or all hidden states if output_all_hiddens is True. )output_hidden_states)rr9rr"stack hidden_stateslast_hidden_state)r<rHencoder_statesr@r@rArFHs  zWhisper.forward_encoderTc Cs|dur|dddfd}|jj||||j|d}|jr:|jd}|j|jd|jdg|jddR}nd}|j}|t |jjj j |j dd}|||jfS)a|Perform one step of the whisper decoder. Arguments --------- encoder_states : torch.Tensor A batch of encoder_states features (mel + whisper feature extractor). decoder_input_ids : torch.Tensor Input tokens for the decoder. This can be language, task, etc. Please refer to the whisper paper for more details or go to the seq2seq2.py file in SpeechBrain to see how to generate the tokens with Greedy Search and/or Beam Search. use_cache : bool If True, keys and values are returned as output for KV caching. past_key_values : torch.Tensor (default: None) If not None, the past key values are used for KV caching and avoid recomputing the attention weights. Returns ------- logits : torch.Tensor The logits of the decoder. attn : torch.Tensor | None If ``output_attentions`` is True, the attention weights are returned. Otherwise, ``None`` is returned. past_key_values : torch.Tensor The past key values of the decoder. NrD)encoder_hidden_states input_idspast_key_valuesr use_cacherr[) unsqueezerrr attentionsviewr1r{r" transpose embed_tokensweighttorfloatr) r<r|rNrr output_statesattnxlogitsr@r@rArG`s,! . zWhisper.forward_decodercCsVddlm}t|}|j|jj}g}|D]}||d||qt |S)z@Returns the list of tokens corresponding to the language tokens.r LANGUAGESr) 0transformers.models.whisper.tokenization_whisperrlistkeysrconvert_tokens_to_idsrappendrntuple)r<rlangs bos_token_idresultlangr@r@rAall_language_tokenss  zWhisper.all_language_tokenscCs ddlm}t|}t|S)zHReturns the list of language codes corresponding to the language tokens.rr)rrrrr)r<rrr@r@rAall_language_codess  zWhisper.all_language_codescCstd}|d7}td}tdd|DsJ|jjdddd |jjd ddd h}|t|D](}|jj|dd|jjd |ddfD]}t|d ksS||vrZ||d qGq3tt |S) u Returns the list of tokens to suppress in order to avoid any speaker tags or non-speech annotations, to prevent sampling texts that are not actually spoken in the audio, e.g. - ♪♪♪ - ( SPEAKING FOREIGN LANGUAGE ) - [DAVID] Hey there, keeping basic punctuations like commas, periods, question marks, exclamation points, etc. Taken from: openai/whisper GitHub u#"#()*+/:;<=>@[\]^_`{|}~「」『』uK<< >> <<< >>> -- --- -( -[ (' (" (( )) ((( ))) [[ ]] {{ }} ♪♪ ♪♪♪u♩♪♫♬♭♮♯css,|]}dt|kodknVqdS)i@&i&N)ord)rocr@r@rA s*z,Whisper.non_speech_tokens..z -F)add_special_tokensrz ' r) rsplitsetallrencodelenaddrsorted)r<symbols miscellaneousrsymboltokensr@r@rAnon_speech_tokenss   zWhisper.non_speech_tokensreturncC |jdS)zIReturns the token id corresponding to the value of the `transcribe` fieldz<|transcribe|>rrr<r@r@rAr  zWhisper.transcribecCr)zHReturns the token id corresponding to the value of the `translate` fieldz <|translate|>rrr@r@rA translaterzWhisper.translatecCr)zBReturns the token id corresponding to the value of the `bos` fieldr rrr@r@rAbosrz Whisper.boscCr)zBReturns the token id corresponding to the value of the `eos` fieldz <|endoftext|>rrr@r@rAeosrz Whisper.eoscCr)zEReturns the token id corresponding to the value of the `bos_lm` fieldz <|startoflm|>rrr@r@rAbos_lmrzWhisper.bos_lmcCr)zGReturns the token id corresponding to the value of the `bos_prev` fieldz<|startofprev|>rrr@r@rAbos_prevrzWhisper.bos_prevcCr)zLReturns the token id corresponding to the value of the `no_timestamps` fieldz<|notimestamps|>rrr@r@rA no_timestampsrzWhisper.no_timestampscCr)zNReturns the token id corresponding to the value of the `timestamp_begin` fieldz<|0.00|>rrr@r@rAtimestamp_beginrzWhisper.timestamp_begincCs |jdS)zHReturns the token id corresponding to the value of the `no_speech` fieldr)rrr@r@rA no_speechs zWhisper.no_speechcCs|jdur td||jS)zGReturns the token id corresponding to the value of the `language` fieldNz6This tokenizer does not have language token configured)r ValueErrorto_language_tokenrr@r@rAlanguage_tokens  zWhisper.language_tokencCs0|jjd|dd}|r|Std|d)aReturns the token id corresponding to the given language. Arguments --------- language : str The language to convert to a token. Returns ------- token The token id corresponding to the given language. Raises ------ KeyError If the language is not found in the tokenizer. z<|z|>Nz Language z not found in tokenizer.)rrgetKeyError)r<rtokenr@r@rArs  zWhisper.to_language_tokencC||_|jj|jddS)zSet the language token to the given language. Arguments --------- language : str The language to set the token to. )rN)rrr')r<rr@r@rAset_language_token*zWhisper.set_language_tokencCr)zSet the task token to the given task. Arguments --------- task : str The task to set the token to. )rN)rrr')r<rr@r@rAset_task5rzWhisper.set_taskcCs |jjdkS)z;Returns True if the model is multilingual, False otherwise.i)config vocab_sizerr@r@rAr&@rzWhisper.is_multilingualcCstt|jjS)z&Returns the list of tokens to suppress)rrrsuppress_tokensrr@r@rAget_suppress_tokensEszWhisper.get_suppress_tokensc sjjdur td|jd}j|j}tj gg| |j } ||ddddf}tj |jdtjd}d|tj<tj |dd|f<|jdd}|jddfdd t|D}||fS) aDetect the language of the given mel spectrogram features. Arguments --------- mel : torch.Tensor Mel spectrogram features to detect the language of. Returns ------- language_tokens : torch.Tensor of shape (batch_size,) ids of the most probable language tokens, which appears after the startoftranscript token. language_probs : List[Dict[str, float]] list of dictionaries containing the probability distribution over all languages. Raises ------ ValueError If the model doesn't have language tokens. NzCThis model doesn't have language tokens so it can't perform lang idrrDrF)rmcs*g|]fddtjjDqS)cs"i|] \}}||fqSr@)item)rojr)ilanguage_token_probsr@rA rsz6Whisper.detect_language...)ziprr)rorr<)rrArqqs z+Whisper.detect_language..)rrrr1rr9r{r"tensorrrrXrGonesboolrrnpinfargmaxsoftmaxrrange) r<rH batch_size enc_statesrNrmasklanguage_tokenslanguage_probsr@rrAdetect_languageJs&     zWhisper.detect_language)rFFFFFNr )N)r)TN)%__name__ __module__ __qualname____doc__rrCrRrEintrT N_SAMPLESrSrFrGrrrrr rrrrrrrrrrrrr&rr"rQr __classcell__r@r@r>rAr!st<M . *& =   &   r)r functoolsrnumpyrr"torch.nnr]=speechbrain.lobes.models.huggingface_transformers.huggingfacerspeechbrain.utils.loggerr SAMPLE_RATEN_FFT HOP_LENGTH CHUNK_LENGTHrrr7rr@r@r@rAs