o i\A@sdZddlmZmZddlZddlmZddlm Z m Z m Z ddl m Z ddlmZdd lmZmZer9ddlZeeZGd d d e Zd gZdS) z% Feature extractor class for Whisper )OptionalUnionN)is_torch_available)mel_filter_bank spectrogramwindow_function)SequenceFeatureExtractor) BatchFeature) TensorTypeloggingcs>eZdZdZdgZ        d(fd d Zd ejd edejfddZ d)dejd edejfddZ e d*de ejde ejde de ejfddZ          d+deeje e e eje e e fdedeed eeeefd!eed"eedeed#eed$eed eed%eedefd&d'ZZS),WhisperFeatureExtractora Constructs a Whisper feature extractor. This feature extractor inherits from [`~feature_extraction_sequence_utils.SequenceFeatureExtractor`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. This class extracts mel-filter bank features from raw speech using a custom numpy implementation of the `Short Time Fourier Transform` which should match pytorch's `torch.stft` equivalent. Args: feature_size (`int`, *optional*, defaults to 80): The feature dimension of the extracted features. sampling_rate (`int`, *optional*, defaults to 16000): The sampling rate at which the audio files should be digitalized expressed in hertz (Hz). hop_length (`int`, *optional*, defaults to 160): Length of the overlapping windows for the STFT used to obtain the Mel Frequency coefficients. chunk_length (`int`, *optional*, defaults to 30): The maximum number of chunks of `sampling_rate` samples used to trim and pad longer or shorter audio sequences. n_fft (`int`, *optional*, defaults to 400): Size of the Fourier transform. padding_value (`float`, *optional*, defaults to 0.0): Padding value used to pad the audio. Should correspond to silences. dither (`float`, *optional*, defaults to 0.0): Adds dithering. In other words, adds a small Gaussian noise to each frame. E.g. use 0.0001 to add dithering with a normal distribution centered around 0.0 with standard deviation 0.0001 (assuming [-1,+1] range of raw_speech). The value 0.0 means no dithering. Dithering has similar effect as `spectrogram(mel_floor=...)`. It reduces the high log_mel_fbank values for signals with hard-zero sections, when VAD cutoff is present in the signal. input_featuresP>Fc sttjd||||d| ||_||_||_|||_|j||_||_||_t d|d|dd|ddd|_ dS) N) feature_size sampling_rate padding_valuereturn_attention_maskrg@@slaney)num_frequency_binsnum_mel_filters min_frequency max_frequencyrnorm mel_scale) super__init__n_fft hop_length chunk_length n_samples nb_max_framesrditherr mel_filters) selfrrr&r'r%rr*rkwargs __class__r"j/home/ubuntu/.local/lib/python3.10/site-packages/transformers/models/whisper/feature_extraction_whisper.pyr$Hs0     z WhisperFeatureExtractor.__init__waveform_batchdevicereturnc Cs|dkr td|dg}|D]5}t|t|jd|j|jd|j|jdd}|dddd f}t|| d }|d d }| |qt |}|S) z Compute the log-mel spectrogram of the provided audio, gives similar results to Whisper's original torch implementation with 1e-5 tolerance. cpuz Got device `z` for feature extraction, but feature extraction on CUDA accelerator devices requires torch, which is not installed. Either set `device='cpu'`, or install torch according to the official instructions: https://pytorch.org/get-started/locally/hanng@log10) frame_lengthr&powerr*r+log_melN @@) ValueErrorrrr%r&r*r+npmaximummaxappendarray)r,r1r2log_spec_batchwaveformlog_specr"r"r0_np_extract_fbank_featuresls,      z2WhisperFeatureExtractor._np_extract_fbank_featuresr4rDc Cs*t||tj}tj|j|d}|jdkr'||jtj|j|j |j d7}tj ||j|j |dd}|dddf d }t|j|tj}|j|}tj|d d }|d krw|jd dd d jddd d } t|| d}n t||d}|dd}|dkr|}|S)z Compute the log-mel spectrogram of the audio using PyTorch's GPU-accelerated STFT implementation with batching, yielding results similar to cpu computing with 1e-5 tolerance. )r2r)dtyper2T)windowreturn_complex.Nr:rg|=)min)dimkeepdimrrr;r<r4)torch from_numpytofloat32 hann_windowr%r*randnshaperGr2stftr&absr+Tclampr6rKr@r?detachr4numpy) r,rDr2rHrT magnitudesr+mel_specrEmax_valr"r"r0_torch_extract_fbank_featuress"       z5WhisperFeatureExtractor._torch_extract_fbank_features input_valuesattention_maskrcCs|durEt|tj}g}t||dD]-\}}||d|t|d|d}||jdkr=|||d<| |q|Sdd|D}|S)z[ Every array in the list is normalized to have zero mean and unit variance Nr:Hz>rcSs*g|]}||t|dqS)r`)meanr>sqrtvar).0xr"r"r0 s*zCWhisperFeatureExtractor.zero_mean_unit_var_norm..) r>rBint32zipsumrarbrcrSrA)r^r_rnormed_input_valuesvectorlength normed_slicer"r"r0zero_mean_unit_var_norms.  z/WhisperFeatureExtractor.zero_mean_unit_var_normTN max_length raw_speech truncationpad_to_multiple_ofreturn_tensorsrpaddingr do_normalizereturn_token_timestampsc  s|dur|jkrtdjjdjdjd|d n tdjjdt|tjo6t |j d k} | rGt |j d krGtd | pZt|t t foZt|d tjt t f}|red d|D}n&|svt|tjsvtj |tjd}nt|tjr|jttjur|tj}|st |gjg}td|i}j|||r|nj|||p| d}| rɈj|d|djd|d<tj|dd d|d<|dd d d }trوjnj}||d | }t|d t rdd|D|d<n||d<|r$|dddddjf}|dj d jd kr |ddddf}||d<| dur?tdjjdfdd|D|d<|durI| |}|S)a Main method to featurize and prepare for the model one or several sequence(s). Implementation uses PyTorch for the STFT computation if available, otherwise a slower NumPy based one. Args: raw_speech (`np.ndarray`, `list[float]`, `list[np.ndarray]`, `list[list[float]]`): The sequence or batch of sequences to be padded. Each sequence can be a numpy array, a list of float values, a list of numpy arrays or a list of list of float values. Must be mono channel audio, not stereo, i.e. single float per timestep. truncation (`bool`, *optional*, default to `True`): Activates truncation to cut input sequences longer than *max_length* to *max_length*. pad_to_multiple_of (`int`, *optional*, defaults to None): If set will pad the sequence to a multiple of the provided value. This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability `>= 7.5` (Volta), or on TPUs which benefit from having sequence lengths be a multiple of 128. return_attention_mask (`bool`, *optional*): Whether to return the attention mask. If left to the default, will return the attention mask according to the specific feature_extractor's default. [What are attention masks?](../glossary#attention-mask) For Whisper models, `attention_mask` should always be passed for batched inference, to avoid subtle bugs. return_tensors (`str` or [`~utils.TensorType`], *optional*): If set, will return tensors instead of list of python integers. Acceptable values are: - `'tf'`: Return TensorFlow `tf.constant` objects. - `'pt'`: Return PyTorch `torch.Tensor` objects. - `'np'`: Return Numpy `np.ndarray` objects. sampling_rate (`int`, *optional*): The sampling rate at which the `raw_speech` input was sampled. It is strongly recommended to pass `sampling_rate` at the forward call to prevent silent errors and allow automatic speech recognition pipeline. padding_value (`float`, *optional*, defaults to 0.0): The value that is used to fill the padding values / vectors. do_normalize (`bool`, *optional*, defaults to `False`): Whether or not to zero-mean unit-variance normalize the input. Normalizing can help to significantly improve the performance of the model. device (`str`, *optional*, defaults to `'cpu'`): Specifies the device for computation of the log-mel spectrogram of audio signals in the `_torch_extract_fbank_features` method. (e.g., "cpu", "cuda") return_token_timestamps (`bool`, *optional*, defaults to `None`): Deprecated. Use `return_attention_mask` instead from which the number of frames can be inferred. Whether or not to return the number of frames of the input raw_speech. These num_frames can be used by the model to compute word level timestamps. Nz3The model corresponding to this feature extractor: z& was trained using a sampling rate of zI. Please make sure that the provided `raw_speech` input was sampled with z and not .zDIt is strongly recommended to pass the `sampling_rate` argument to `zN()`. Failing to do so can result in silent errors that might be hard to debug.rrz2Only mono-channel audio is supported for input to rcSs g|] }tj|gtjdjqSrG)r>asarrayrPrV)rdspeechr"r"r0rfs z4WhisperFeatureExtractor.__call__..ryr)rtrorqrrrr_)r_r)axiscSsg|] }tj|tjdqSrx)r>rzrP)rdfeaturer"r"r0rfBsr:z,`return_token_timestamps` is deprecated for z~ and will be removed in Transformers v5. Use `return_attention_mask` instead, as the number of frames can be inferred from it.csg|] }t|jqSr")lenr&)rd raw_speech_ir,r"r0rfVs num_frames)!rr=r/__name__loggerwarning isinstancer>ndarrayr~rSlisttuplerzrPrGfloat64astyperVr padr(rnrstackget transposerr]rFr& warning_onceconvert_to_tensors)r,rprqrrrsrrtrorrur2rvr-is_batched_numpy is_batchedbatched_speech padded_inputsrextract_fbank_featuresrescaled_attention_maskr"rr0__call__sD  "        z WhisperFeatureExtractor.__call__)rrrrrrrF)r4)r) TNNNroNNNr4N)r __module__ __qualname____doc__model_input_namesr$r>rstrrFr] staticmethodrfloatrnrboolrintr r r __classcell__r"r"r.r0r $sx!$ "    r )rtypingrrrYr>r audio_utilsrrr!feature_extraction_sequence_utilsr feature_extraction_utilsr utilsr r rM get_loggerrrr __all__r"r"r"r0s     <