o iE@sdZddlZddlmZmZmZddlZddlm Z m Z m Z m Z ddl mZddlmZddlmZmZmZeeZGd d d eZd gZdS) z%Feature extractor class for SpeechT5.N)AnyOptionalUnion)mel_filter_bankoptimal_fft_length spectrogramwindow_function)SequenceFeatureExtractor) BatchFeature)PaddingStrategy TensorTypeloggingcseZdZdZddgZ         d7dedededededededededededededeffdd Z e d8de e j de e j ded!e e j fd"d#Zd$e j d!e j fd%d&Z ' '  '  ' ' ' 'd9d(eee j e ee e j e e efd)eee j e ee e j e e efd*eeeefd+eed,ed-eedeed.eeeefdeed!efd/d0Z   '  ' ' 'd:d1ee j e ee e j e e efd2ed*eeeefd+eed,ed-eedeed.eeeefd!efd3d4Zd!eeefffd5d6 ZZS);SpeechT5FeatureExtractora Constructs a SpeechT5 feature extractor. This class can pre-process a raw speech signal by (optionally) normalizing to zero-mean unit-variance, for use by the SpeechT5 speech encoder prenet. This class can also extract log-mel filter bank features from raw speech, for use by the SpeechT5 speech decoder prenet. This feature extractor inherits from [`~feature_extraction_sequence_utils.SequenceFeatureExtractor`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: feature_size (`int`, *optional*, defaults to 1): The feature dimension of the extracted features. sampling_rate (`int`, *optional*, defaults to 16000): The sampling rate at which the audio files should be digitalized expressed in hertz (Hz). padding_value (`float`, *optional*, defaults to 0.0): The value that is used to fill the padding values. do_normalize (`bool`, *optional*, defaults to `False`): Whether or not to zero-mean unit-variance normalize the input. Normalizing can help to significantly improve the performance for some models. num_mel_bins (`int`, *optional*, defaults to 80): The number of mel-frequency bins in the extracted spectrogram features. hop_length (`int`, *optional*, defaults to 16): Number of ms between windows. Otherwise referred to as "shift" in many papers. win_length (`int`, *optional*, defaults to 64): Number of ms per window. win_function (`str`, *optional*, defaults to `"hann_window"`): Name for the window function used for windowing, must be accessible via `torch.{win_function}` frame_signal_scale (`float`, *optional*, defaults to 1.0): Constant multiplied in creating the frames before applying DFT. This argument is deprecated. fmin (`float`, *optional*, defaults to 80): Minimum mel frequency in Hz. fmax (`float`, *optional*, defaults to 7600): Maximum mel frequency in Hz. mel_floor (`float`, *optional*, defaults to 1e-10): Minimum value of mel frequency banks. reduction_factor (`int`, *optional*, defaults to 2): Spectrogram length reduction factor. This argument is deprecated. return_attention_mask (`bool`, *optional*, defaults to `True`): Whether or not [`~SpeechT5FeatureExtractor.__call__`] should return `attention_mask`. input_valuesattention_mask>FP@ hann_window?绽|=T feature_size sampling_rate padding_value do_normalize num_mel_bins hop_length win_length win_functionframe_signal_scalefminfmax mel_floorreduction_factorreturn_attention_maskc stjd |||d|||_||_||_||_||_||_| |_| |_ | |_ | |_ | |_ ||d|_ ||d|_t|j |_|jdd|_t|j |jdd|_t|j|j|j |j |jddd|_| d krotd t| d kr{td tdSdS)N)rrrirrT) window_lengthnameperiodicslaney)num_frequency_binsnum_mel_filters min_frequency max_frequencyrnorm mel_scalerzeThe argument `frame_signal_scale` is deprecated and will be removed in version 4.30.0 of Transformersg@zcThe argument `reduction_factor` is deprecated and will be removed in version 4.30.0 of Transformers)super__init__r r*r!r"r#r$r%r&r'r(r) sample_size sample_stridern_fftn_freqsr windowrr mel_filterswarningswarn FutureWarning)selfrrrr r!r"r#r$r%r&r'r(r)r*kwargs __class__r5l/home/ubuntu/.local/lib/python3.10/site-packages/transformers/models/speecht5/feature_extraction_speecht5.pyr7NsJ  z!SpeechT5FeatureExtractor.__init__returncCs|durEt|tj}g}t||dD]-\}}||d|t|d|d}||jdkr=|||d<| |q|Sdd|D}|S)z[ Every array in the list is normalized to have zero mean and unit variance NHz>rcSs*g|]}||t|dqS)rH)meannpsqrtvar).0xr5r5rE s*zDSpeechT5FeatureExtractor.zero_mean_unit_var_norm..) rJarrayint32zipsumrIrKrLshapeappend)rrrnormed_input_valuesvectorlength normed_slicer5r5rEzero_mean_unit_var_norms.  z0SpeechT5FeatureExtractor.zero_mean_unit_var_norm one_waveformc Cs*t||j|j|j|j|j|jdd}|jS)zZ Extracts log-mel filterbank features for one waveform array (unbatched). log10)r< frame_lengthr" fft_lengthr=r(log_mel)rr<r8r9r:r=r(T)rAr[ log_mel_specr5r5rE_extract_mel_featuress z.SpeechT5FeatureExtractor._extract_mel_featuresNaudio audio_targetpadding max_length truncationpad_to_multiple_ofreturn_tensorsc  Ks|dur |dur td| dur)| |jkr(td|d|jd|jd| d n td|jjd |durI|j|d ||||||fi| } nd} |durx|j|d ||||||fi| } | dure| S| d | d <| d} | durx| | d<| S)aA Main method to featurize and prepare for the model one or several sequence(s). Pass in a value for `audio` to extract waveform features. Pass in a value for `audio_target` to extract log-mel spectrogram features. Args: audio (`np.ndarray`, `list[float]`, `list[np.ndarray]`, `list[list[float]]`, *optional*): The sequence or batch of sequences to be processed. Each sequence can be a numpy array, a list of float values, a list of numpy arrays or a list of list of float values. This outputs waveform features. Must be mono channel audio, not stereo, i.e. single float per timestep. audio_target (`np.ndarray`, `list[float]`, `list[np.ndarray]`, `list[list[float]]`, *optional*): The sequence or batch of sequences to be processed as targets. Each sequence can be a numpy array, a list of float values, a list of numpy arrays or a list of list of float values. This outputs log-mel spectrogram features. padding (`bool`, `str` or [`~utils.PaddingStrategy`], *optional*, defaults to `False`): Select a strategy to pad the returned sequences (according to the model's padding side and padding index) among: - `True` or `'longest'`: Pad to the longest sequence in the batch (or no padding if only a single sequence if provided). - `'max_length'`: Pad to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. - `False` or `'do_not_pad'` (default): No padding (i.e., can output a batch with sequences of different lengths). max_length (`int`, *optional*): Maximum length of the returned list and optionally padding length (see above). truncation (`bool`): Activates truncation to cut input sequences longer than *max_length* to *max_length*. pad_to_multiple_of (`int`, *optional*): If set will pad the sequence to a multiple of the provided value. This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability `>= 7.5` (Volta), or on TPUs which benefit from having sequence lengths be a multiple of 128. return_attention_mask (`bool`, *optional*): Whether to return the attention mask. If left to the default, will return the attention mask according to the specific feature_extractor's default. [What are attention masks?](../glossary#attention-mask) return_tensors (`str` or [`~utils.TensorType`], *optional*): If set, will return tensors instead of list of python integers. Acceptable values are: - `'tf'`: Return TensorFlow `tf.constant` objects. - `'pt'`: Return PyTorch `torch.Tensor` objects. - `'np'`: Return Numpy `np.ndarray` objects. sampling_rate (`int`, *optional*): The sampling rate at which the `audio` or `audio_target` input was sampled. It is strongly recommended to pass `sampling_rate` at the forward call to prevent silent errors. Nz9You must provide either `audio` or `audio_target` values.z3The model corresponding to this feature extractor: z& was trained using a sampling rate of zB. Please make sure that the provided audio input was sampled with z and not .zDIt is strongly recommended to pass the `sampling_rate` argument to `zN()`. Failing to do so can result in silent errors that might be hard to debug.FTrlabelsrdecoder_attention_mask) ValueErrorrloggerwarningrD__name___process_audioget)rArcrdrerfrgrhr*rirrBinputs inputs_targetrlr5r5rE__call__sf?       z!SpeechT5FeatureExtractor.__call__speech is_targetc  sPt|tjo t|jdk} | rt|jdkrtd| p0t|ttfo0t|dtjttf} | r;dd|D}n&| sLt|tjsLtj|tj d}nt|tjra|j t tj ura| tj }| sf|g}j } |rfdd|D} td | i}j_ ntd |i}j|f|||||d | }| _ |d }t|dtjsd d|D|d <n:t|tjst|dtjr|dj t tj urd d|D|d <nt|tjr|j t tj ur| tj |d <|d }|durdd|D|d <|sjrj||dtjur |nd}j|d |jd|d <|dur&||}|S)Nrrz2Only mono-channel audio is supported for input to rcSg|] }tj|tjdqSdtyperJasarrayfloat32)rMrvr5r5rErO=z;SpeechT5FeatureExtractor._process_audio..rzcsg|]}|qSr5)rb)rMwaveformrAr5rErOLsr)rerfrgrhr*cSrxryr|rMrPr5r5rErOarcSsg|]}|tjqSr5)astyperJr~rr5r5rErOgsrcSrxry)rJr}rQrr5r5rErOnr)rf)rr) isinstancerJndarraylenrTrmlisttupler}r~r{float64rrr r!padrrr _get_padding_strategiesr DO_NOT_PADrZrconvert_to_tensors)rArvrwrerfrgrhr*rirBis_batched_numpy is_batchedfeature_size_hackfeaturesencoded_inputs padded_inputsrrr5rrErq)sr "          z'SpeechT5FeatureExtractor._process_audiocs.t}gd}|D] }||vr||=q |S)N)r<r=r8r9r:r;)r6to_dict)rAoutputnamesr,rCr5rErs z SpeechT5FeatureExtractor.to_dict)rrrFrrrrrrrrrT)r) NNFNFNNNN)FFNFNNN)rp __module__ __qualname____doc__model_input_namesintfloatboolstrr7 staticmethodrrJrrZrbrrr r r rurqdictrr __classcell__r5r5rCrErs,     <  &&    x"   "Wr)rr>typingrrrnumpyrJ audio_utilsrrrr !feature_extraction_sequence_utilsr feature_extraction_utilsr utilsr r r get_loggerrprnr__all__r5r5r5rEs    o