o eiEw@s>dZddlmZddlmZddlmZddlZddlm Z ddl m Z ddl m Z dd lmZmZmZmZmZmZdd lmZmZmZmZmZdd lmZdd lmZGd ddejj Z!Gdddejj Z"Gdddejj Z#de$de$fddZ%eGdddZ&Gdddejj Z'Gdddejj Z(d!dd Z)dS)"zBasic feature pipelines. Authors * Mirco Ravanelli 2020 * Peter Plantinga 2020 * Sarthak Yadav 2020 * Sylvain de Langen 2024 ) dataclass)partial)OptionalN) GaborConv1d)PCEN)GaussianLowpassPooling)DCTSTFT ContextWindowDeltas Filterbankspectral_magnitude)PERIODIC_NEIGHBORScompute_autocorr_features compute_gnecompute_periodic_featurescompute_spectral_features)fwd_default_precision)FilterPropertiescsbeZdZdZ          dfdd ZeejdddZde fddZ Z S)Fbanka Generate features for input to the speech pipeline. Arguments --------- deltas : bool (default: False) Whether or not to append derivatives and second derivatives to the features. context : bool (default: False) Whether or not to append forward and backward contexts to the features. requires_grad : bool (default: False) Whether to allow parameters (i.e. fbank centers and spreads) to update during training. sample_rate : int (default: 160000) Sampling rate for the input waveforms. f_min : int (default: 0) Lowest frequency for the Mel filters. f_max : int (default: None) Highest frequency for the Mel filters. Note that if f_max is not specified it will be set to sample_rate // 2. n_fft : int (default: 400) Number of samples to use in each stft. n_mels : int (default: 40) Number of Mel filters. filter_shape : str (default: triangular) Shape of the filters ('triangular', 'rectangular', 'gaussian'). param_change_factor : float (default: 1.0) If freeze=False, this parameter affects the speed at which the filter parameters (i.e., central_freqs and bands) can be changed. When high (e.g., param_change_factor=1) the filters change a lot during training. When low (e.g. param_change_factor=0.1) the filter parameters are more stable during training. param_rand_factor : float (default: 0.0) This parameter can be used to randomly change the filter parameters (i.e, central frequencies and bands) during training. It is thus a sort of regularization. param_rand_factor=0 does not affect, while param_rand_factor=0.15 allows random variations within +-15% of the standard values of the filter parameters (e.g., if the central freq is 100 Hz, we can randomly change it from 85 Hz to 115 Hz). left_frames : int (default: 5) Number of frames of left context to add. right_frames : int (default: 5) Number of frames of right context to add. win_length : float (default: 25) Length (in ms) of the sliding window used to compute the STFT. hop_length : float (default: 10) Length (in ms) of the hop of the sliding window used to compute the STFT. Example ------- >>> import torch >>> inputs = torch.randn([10, 16000]) >>> feature_maker = Fbank() >>> feats = feature_maker(inputs) >>> feats.shape torch.Size([10, 101, 40]) F>rN( triangular? c szt||_||_||_|dur|d}t||||d|_t|||||| | | | d |_t |d|_ t | | d|_ dS)N sample_raten_fft win_length hop_length r!r"n_melsf_minf_maxfreeze filter_shapeparam_change_factorparam_rand_factor input_size left_frames right_frames) super__init__deltascontext requires_gradr compute_STFTr compute_fbanksr compute_deltasr context_window)selfr4r5r6r!r'r(r"r&r*r+r,r0r1r#r$ __class__X/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/speechbrain/lobes/features.pyr3bs8   zFbank.__init__ cast_inputscCs^||}t|}||}|jr%||}||}tj|||gdd}|jr-||}|S)aReturns a set of features generated from the input waveforms. Arguments --------- wav : torch.Tensor A batch of audio signals to transform to features. Returns ------- fbanks : torch.Tensor rdim) r7r r8r4r9torchcatr5r:)r;wavr magfbanksdelta1delta2r>r>r?forwards     z Fbank.forwardreturncCs |jSN)r7get_filter_propertiesr;r>r>r?rNs zFbank.get_filter_properties)FFFrrNrrrrrrrrr) __name__ __module__ __qualname____doc__r3rrDfloat32rKrrN __classcell__r>r>r<r?r&s*= 1 rcsVeZdZdZ          dfdd ZeejdddZZ S)MFCCaW Generate features for input to the speech pipeline. Arguments --------- deltas : bool (default: True) Whether or not to append derivatives and second derivatives to the features. context : bool (default: True) Whether or not to append forward and backward contexts to the features. requires_grad : bool (default: False) Whether to allow parameters (i.e. fbank centers and spreads) to update during training. sample_rate : int (default: 16000) Sampling rate for the input waveforms. f_min : int (default: 0) Lowest frequency for the Mel filters. f_max : int (default: None) Highest frequency for the Mel filters. Note that if f_max is not specified it will be set to sample_rate // 2. n_fft : int (default: 400) Number of samples to use in each stft. n_mels : int (default: 23) Number of filters to use for creating filterbank. n_mfcc : int (default: 20) Number of output coefficients filter_shape : str (default 'triangular') Shape of the filters ('triangular', 'rectangular', 'gaussian'). param_change_factor: bool (default 1.0) If freeze=False, this parameter affects the speed at which the filter parameters (i.e., central_freqs and bands) can be changed. When high (e.g., param_change_factor=1) the filters change a lot during training. When low (e.g. param_change_factor=0.1) the filter parameters are more stable during training. param_rand_factor: float (default 0.0) This parameter can be used to randomly change the filter parameters (i.e, central frequencies and bands) during training. It is thus a sort of regularization. param_rand_factor=0 does not affect, while param_rand_factor=0.15 allows random variations within +-15% of the standard values of the filter parameters (e.g., if the central freq is 100 Hz, we can randomly change it from 85 Hz to 115 Hz). left_frames : int (default 5) Number of frames of left context to add. right_frames : int (default 5) Number of frames of right context to add. win_length : float (default: 25) Length (in ms) of the sliding window used to compute the STFT. hop_length : float (default: 10) Length (in ms) of the hop of the sliding window used to compute the STFT. Example ------- >>> import torch >>> inputs = torch.randn([10, 16000]) >>> feature_maker = MFCC() >>> feats = feature_maker(inputs) >>> feats.shape torch.Size([10, 101, 660]) TFrrNrrrrrrrc st||_||_||_|dur|d}t||||d|_t|||||| | | | d |_t || d|_ t | d|_ t | |d|_dS)Nrr r%r.n_outr-r/)r2r3r4r5r6r r7r r8r compute_dctr r9r r:)r;r4r5r6r!r'r(r"r&n_mfccr*r+r,r0r1r#r$r<r>r?r3s:   z MFCC.__init__r@cCsh||}t|}||}||}|jr*||}||}tj|||gdd}|jr2| |}|S)aReturns a set of mfccs generated from the input waveforms. Arguments --------- wav : torch.Tensor A batch of audio signals to transform to features. Returns ------- mfccs : torch.Tensor rrB) r7r r8r[r4r9rDrEr5r:)r;rFr rGrHmfccsrIrJr>r>r?rK"s      z MFCC.forward)TTFrrNrrWrXrrrrrrr) rPrQrRrSr3rrDrTrKrUr>r>r<r?rVs*? 4rVcsleZdZdZ            dd ed ed effd d Zeej dddZ ddZ ddZ Z S)Leafa This class implements the LEAF audio frontend from Neil Zeghidour, Olivier Teboul, F{'e}lix de Chaumont Quitry & Marco Tagliasacchi, "LEAF: A LEARNABLE FRONTEND FOR AUDIO CLASSIFICATION", in Proc. of ICLR 2021 (https://arxiv.org/abs/2101.08596) Arguments --------- out_channels : int It is the number of output channels. window_len: float length of filter window in milliseconds window_stride : float Stride factor of the filters in milliseconds sample_rate : int, Sampling rate of the input signals. It is only used for sinc_conv. input_shape : tuple Expected shape of the inputs. in_channels : int Expected number of input channels. min_freq : float Lowest possible frequency (in Hz) for a filter max_freq : float Highest possible frequency (in Hz) for a filter use_pcen: bool If True (default), a per-channel energy normalization layer is used learnable_pcen: bool: If True (default), the per-channel energy normalization layer is learnable use_legacy_complex: bool If False, torch.complex64 data type is used for gabor impulse responses If True, computation is performed on two real-valued torch.Tensors skip_transpose: bool If False, uses batch x time x channel convention of speechbrain. If True, uses batch x channel x time convention. n_fft: int Number of FFT bins Example ------- >>> inp_tensor = torch.rand([10, 8000]) >>> leaf = Leaf( ... out_channels=40, window_len=25., window_stride=10., in_channels=1 ... ) >>> out_tensor = leaf(inp_tensor) >>> out_tensor.shape torch.Size([10, 50, 40]) 9@$@rNN@TF window_len window_strider!cst||_t||dd}t||d}|dur&|dur&td|dur/||}td|||ddd| |||| dd |_t|j||dd |_ | r]t |jd d d d | ddd|_ nd|_ | |_ dS)Niz.Must provide one of input_shape or in_channelsrsameFT) out_channels in_channels kernel_sizestridepaddingbiasr"r!min_freqmax_frequse_legacy_complexskip_transpose)rhrirjrpgQ?g{Gz?@g-q=)alpha smooth_coefdeltafloor trainableper_channel_smooth_coefrp) r2r3rgint ValueError_check_input_shaper complex_convrpoolingr compressionrp)r;rgrcrdr! input_shaperhrmrnuse_pcenlearnable_pcenrorpr" window_sizer<r>r?r3msR    z Leaf.__init__r@cCs|js |dd}|jdk}|r|d}||}||}||}t|tj d|j d}|j r8| |}|jsA|dd}|S)a Returns the learned LEAF features Arguments --------- x : torch.Tensor of shape (batch, time, 1) or (batch, time) batch of input signals. 2d or 3d tensors are expected. Returns ------- outputs : torch.Tensor rergh㈵>device) rp transposendim unsqueezer{_squared_modulus_activationr|rDmaximumtensorrr})r;xroutputsr>r>r?rKs         z Leaf.forwardcCs8|dd}dtjjj|dddd}|dd}|S)Nrerrq)rirjrrDnn functional avg_pool1d)r;routputr>r>r?rs    z Leaf._squared_modulus_activationcCs<t|dkr d}|St|dkrd}|Stdtt|)z@Checks the input shape and returns the number of input channels.rrez"Leaf expects 2d or 3d inputs. Got )lenrystr)r;shaperhr>r>r?rzs  zLeaf._check_input_shape) r_r`rNNraNTTFFrb)rPrQrRrSfloatrxr3rrDrTrKrrzrUr>r>r<r?r^<s23 ? !r^torLcCs,|dksJ||dkr|S||||S)zMIf `x` cannot evenly divide `to`, round it up to the next value that can.rr>)rrr>r>r? upalign_values  rc@s eZdZUdZeejed<dS)StreamingFeatureWrapperContextzQStreaming metadata for the feature extractor. Holds some past context frames. left_contextN)rPrQrRrSrrDTensor__annotations__r>r>r>r?rs rcseZdZdZdejjdeffdd Zde fddZ de fd d Z d e de fd d Z dej dedej fddZdefddZdefddZZS)StreamingFeatureWrapperaWraps an arbitrary filter so that it can be used in a streaming fashion (i.e. on a per-chunk basis), by remembering context and making "clever" use of padding. Arguments --------- module : torch.nn.Module The filter to wrap; e.g. a module list that constitutes a sequential feature extraction pipeline. The module is assumed to pad its inputs, e.g. the output of a convolution with a stride of 1 would end up with the same frame count as the input. properties : FilterProperties The effective filter properties of the provided module. This is used to determine padding and caching. module propertiescs>t||_||_|jjrtd|jjdkrtddS)Nz5Causal streaming feature wrapper is not yet supportedrez7Dilation not yet supported in streaming feature wrapper)r2r3rrcausalrydilation)r;rrr<r>r?r3s  z StreamingFeatureWrapper.__init__rLcCst|jjdd|jjS)zComputes the number of padding/context frames that need to be injected at the past and future of the input signal in the forward pass. rer)rrrrjrOr>r>r?get_required_paddingsz,StreamingFeatureWrapper.get_required_paddingcCs||jjS)zdComputes the exact number of produced frames (along the time dimension) per input pad frame.)rrrjrOr>r>r?get_output_count_per_pad_frame'sz6StreamingFeatureWrapper.get_output_count_per_pad_frameframes_per_chunkcCst|||S)auGet the recommended number of zero chunks to inject at the end of an input stream depending on the filter properties of the extractor. The number of injected chunks is chosen to ensure that the filter has output frames centered on the last input frames. See also :meth:`~StreamingFeatureWrapper.forward`. Arguments --------- frames_per_chunk : int The number of frames per chunk, i.e. the size of the time dimension passed to :meth:`~StreamingFeatureWrapper.forward`. Returns ------- Recommended number of chunks. )rr)r;rr>r>r?!get_recommended_final_chunk_count-s z9StreamingFeatureWrapper.get_recommended_final_chunk_countchunkr5cOs|}|}|jdurtjj||ddf}n t|j|fd}|dd| ddf|_|j|g|Ri|}|dd|| df}|S)a~Forward pass for the streaming feature wrapper. For the first chunk, 0-padding is inserted at the past of the input. For any chunk (including the first), some future frames get truncated and cached to be inserted as left context for the next chunk in time. For further explanations, see the comments in the code. Note that due to how the padding is implemented, you may want to call this with a chunk worth full of zeros (potentially more for filters with large windows) at the end of your input so that the final frames have a chance to get processed by the filter. See :meth:`~StreamingFeatureWrapper.get_recommended_final_chunk_count`. This is not really an issue when processing endless streams, but when processing files, it could otherwise result in truncated outputs. Arguments --------- chunk : torch.Tensor Chunk of input of shape [batch size, time]; typically a raw waveform. Normally, in a chunkwise streaming scenario, `time = (stride-1) * chunk_size` where `chunk_size` is the desired **output** frame count. context : StreamingFeatureWrapperContext Mutable streaming context object; should be reused for subsequent calls in the same streaming session. *extra_args : tuple **extra_kwargs : dict Args to be passed to he module. Returns ------- torch.Tensor Processed chunk of shape [batch size, output frames]. This shape is equivalent to the shape of `module(chunk)`. Nrrre.) rrrrDrrpadrEr)r;rr5 extra_args extra_kwargs feat_pad_sizenum_outputs_per_padfeatsr>r>r?rKEs,   zStreamingFeatureWrapper.forwardcCs|jSrM)rrOr>r>r?rNsz-StreamingFeatureWrapper.get_filter_propertiescCstdSrM)rrOr>r>r?make_streaming_contextsz.StreamingFeatureWrapper.make_streaming_context)rPrQrRrSrDrModulerr3rxrrrrrrKrNrrUr>r>r<r?rs  UrcspeZdZdZ        dd ed ededededededededeffdd Zdej fddZ Z S) VocalFeaturesaEstimates the vocal characteristics of a signal in four categories of features: * Autocorrelation-based * Period-based (jitter/shimmer) * Spectrum-based * MFCCs Arguments --------- min_f0_Hz: int The minimum allowed fundamental frequency, to reduce octave errors. Default is 80 Hz, based on human voice standard frequency range. max_f0_Hz: int The maximum allowed fundamental frequency, to reduce octave errors. Default is 300 Hz, based on human voice standard frequency range. step_size: float The time between analysis windows (in seconds). window_size: float The size of the analysis window (in seconds). Must be long enough to contain at least 4 periods at the minimum frequency. sample_rate: int The number of samples in a second. log_scores: bool Whether to represent the jitter/shimmer/hnr/gne on a log scale, as these features are typically close to zero. eps: float The minimum value before log transformation, default of 1e-3 results in a maximum value of 30 dB. sma_neighbors: int Number of frames to average -- default 3 n_mels: int (default: 23) Number of filters to use for creating filterbank. n_mfcc: int (default: 4) Number of output coefficients Example ------- >>> audio = torch.rand(1, 16000) >>> feature_maker = VocalFeatures() >>> vocal_features = feature_maker(audio) >>> vocal_features.shape torch.Size([1, 96, 17]) P,{Gz?皙?rTMbP?rrW min_f0_Hz max_f0_Hz step_sizerr! log_scoreseps sma_neighborsr&r\c stt|||_t|||_t|||_t|||_||_||_||_ ||_ |jt |jks=Jdt dt ||j| d|_ t| | d|_tt||d|_dS)NzNeed at least z periods in a window)r!r"r&rY) frame_lenhop_len)r2r3rx step_sampleswindow_samplesmax_lagmin_lagr!rrrrr r8rr[rr) r;rrrrr!rrrr&r\r<r>r?r3s*   zVocalFeatures.__init__audioc Cs|dks Jd|jd|j|jd}t||j|j\}}|j|}d|}t||\}}| ||j} | d| dkrL| ddd| df} |j r}d|j |j d}d|j |j d}d|j |j d}dd| j |j d} tj|j|jd } ttj|| ddd} t| } ||| } tj||||| fdd }tj|| | fdd }|jdkrt|d|jd }|S) aCompute voice features. Arguments --------- audio: torch.Tensor The audio signal to be converted to voice features. Returns ------- features: torch.Tensor A [batch, frame, 13+n_mfcc] tensor with the following features per-frame. * autocorr_f0: A per-frame estimate of the f0 in Hz. * autocorr_hnr: harmonicity-to-noise ratio for each frame. * periodic_jitter: Average deviation in period length. * periodic_shimmer: Average deviation in amplitude per period. * gne: The glottal-to-noise-excitation ratio. * spectral_centroid: "center-of-mass" for spectral frames. * spectral_spread: avg distance from centroid for spectral frames. * spectral_skew: asymmetry of spectrum about the centroid. * spectral_kurtosis: tailedness of spectrum. * spectral_entropy: The peakiness of the spectrum. * spectral_flatness: The ratio of geometric mean to arithmetic mean. * spectral_crest: The ratio of spectral maximum to arithmetic mean. * spectral_flux: The 2-normed diff between successive spectral values. * mfcc_{0-n_mfcc}: The mel cepstral coefficients. rz4Expected audio to be 2-dimensional, [batch, samples]r) dimensionsizestepreNi)minrrB)rCn)rCunfoldrrrrrr!rrrrclamprlog10rD hann_windowrabsfftrfftviewrr[r8stackrErmoving_average)r;rframes harmonicity best_lagsf0hnrjittershimmergnehannspectrumspectral_featuresr]featuresr>r>r?rKs:     zVocalFeatures.forward) rrrrrTrrrWr) rPrQrRrSrxrboolr3rDrrKrUr>r>r<r?rsD-   'rrercCs8||d}|d}tjjj|||ddd}||dS)a=Computes moving average on a given dimension. Arguments --------- features: torch.Tensor The feature tensor to smooth out. dim: int The time dimension (for smoothing). n: int The number of points in the moving average Returns ------- smoothed_features: torch.Tensor The features after the moving average is applied. Example ------- >>> feats = torch.tensor([[0., 1., 0., 1., 0., 1., 0.]]) >>> moving_average(feats) tensor([[0.5000, 0.3333, 0.6667, 0.3333, 0.6667, 0.3333, 0.5000]]) rrreF)rirkrjcount_include_padr)rrCrrr>r>r?r@s   r)rer)*rS dataclassesr functoolsrtypingrrDspeechbrain.nnet.CNNrspeechbrain.nnet.normalizationrspeechbrain.nnet.poolingrspeechbrain.processing.featuresrr r r r r %speechbrain.processing.vocal_featuresrrrrrspeechbrain.utils.autocastr!speechbrain.utils.filter_analysisrrrrrVr^rxrrrrrr>r>r>r?s4          )  &