o Si@sdZddlZddlZddlmZmZmZddlZddl Z ddl m Z z)ddl m Z de jde jfdd Zde jde jfd d Zde jde jfd d ZWn*eyude jde jfdd Zde jde jfdd Zde jde jfdd ZYnwddlmZmZGddde jZGddde jZGdddeZGdddeZGdddeZGdddeZde jded ed!ede jf d"d#Z  $dJde jd eded%ee jd!edee je jff d&d'Z de jd(e!de jfd)d*Z"   +dKd,ed-ed.ed/e!d0ee!d1ede jfd2d3Z#dee$fd4d5Z%d6Z&d7Z'd8Z(d9Z)d:Z*dLdZ+d?d@Z,dAdBZ-dedefdCdDZ.dEedFedGe!d/e!d0e!dee je jff dHdIZ/dS)MuK Copyright 2019 Johns Hopkins University (Author: Jesus Villalba) 2021 Johns Hopkins University (Author: Piotr Żelasko) Apache 2.0 (http://www.apache.org/licenses/LICENSE-2.0) This whole module is authored and contributed by Jesus Villalba, with minor changes by Piotr Żelasko to make it more consistent with Lhotse. It contains a PyTorch implementation of feature extractors that is very close to Kaldi's -- notably, it differs in that the preemphasis and DC offset removal are applied in the time, rather than frequency domain. This should not significantly affect any results, as confirmed by Jesus. This implementation works well with autograd and batching, and can be used neural network layers. Update January 2022: These modules now expose a new API function called "online_inference" that may be used to compute the features when the audio is streaming. The implementation is stateless, and passes the waveform remainders back to the user to feed them to the modules once new data becomes available. The implementation is compatible with JIT scripting via TorchScript. N)ListOptionalTuple)nn)rfftxreturncCs t|ddSNdim) torch_rfftrrP/home/ubuntu/.local/lib/python3.10/site-packages/lhotse/features/kaldi/layers.py_rfft# rcCs |dS)Nabsrrrr_pow_spectrogram&rrcC|SNrrrrr _spectrogram)rcCstj|ddddS)NFT) normalizedonesided)torchrrrrrr.scCs|ddSNrr )powsumrrrrr1scCs|ddSr)r r!sqrtrrrrr4s)EPSILONSecondscseZdZdZdddddddd d edd f d ed ed edeedede de de dede dededdffdd Z ddZ ddZ dejdeejeejffdd Zd!ejdeejeejffd"d#Zejj d'd!ejd$eejdeeejeejfejffd%d&ZZS)(Wav2Wina  Apply standard Kaldi preprocessing (dithering, removing DC offset, pre-emphasis, etc.) on the input waveforms and partition them into overlapping frames (of audio samples). Note: no feature extraction happens in here, the output is still a time-domain signal. Example:: >>> x = torch.randn(1, 16000, dtype=torch.float32) >>> x.shape torch.Size([1, 16000]) >>> t = Wav2Win() >>> t(x).shape torch.Size([1, 100, 400]) The input is a tensor of shape ``(batch_size, num_samples)``. The output is a tensor of shape ``(batch_size, num_frames, window_length)``. When ``return_log_energy==True``, returns a tuple where the second element is a log-energy tensor of shape ``(batch_size, num_frames)``. >皙?{Gz?NT ףp= ?poveyF sampling_rate frame_length frame_shift pad_lengthremove_dc_offset preemph_coeff window_typedither snip_edges energy_floor raw_energyreturn_log_energyrc st||_||_||_||_||_||_||_| |_ | |_ | |_ | |_ | r-t dtt||} | |_tt|||_tjt| |ddd|_|durU| n||_|j| ksgJd|d| dS)Nz|Setting snip_edges=True is generally incompatible with Lhotse -- you might experience mismatched duration/num_frames errors.)r2F requires_gradzpad_length (or fft_length) = z cannot be smaller than N = )super__init__r,r-r.r0r1r2r3r4r5r6r7warningswarnintmathfloor_length_shiftr Parametercreate_frame_window_windowr/)selfr,r-r.r/r0r1r2r3r4r5r6r7N __class__rrr;Ps4   zWav2Win.__init__cCrr)__str__rFrrr__repr__~rzWav2Win.__repr__cCsBd|jj|j|j|j|j|j|j|j |j |j |j |j |j }|S)Nz{}(sampling_rate={}, frame_length={}, frame_shift={}, pad_length={}, remove_dc_offset={}, preemph_coeff={}, window_type={} dither={}, snip_edges={}, energy_floor={}, raw_energy={}, return_log_energy={}))formatrI__name__r,r-r.r/r0r1r2r3r4r5r6r7)rFsrrrrJs$zWav2Win.__str__ x_stridedcCs|jrtj|ddd}||}d}|jr|jrt||j}|jdkr>tjj j |ddd}||j|ddddddf}||j }|j |j krb|j |j }tjj j |d d |gd dd d }|jrn|jsnt||j}||fS) NrT)r keepdimr+)rr replicate)moder rrconstantrSvalue)r0rmeanr7r6_get_log_energyr5r1r functionalpadrEr/rA unsqueezesqueeze)rFrPmu log_energyx_offsetrZrrr_forward_strideds.   $     zWav2Win._forward_stridedrcCsH|jdkrtj|j|jd}||j|}t||j|j|j}| |S)Nr+device) r3rrandnshaperb_get_strided_batchrArBr4r`)rFrnrPrrrforwards  zWav2Win.forwardcontextcCs`|jdkrtj|j|jd}||j|}t||j|j||jd\}}| |\}}||f|fS)z The same as the ``forward()`` method, except it accepts an extra argument with the remainder waveform from the previous call of ``online_inference()``, and returns a tuple of ``((frames, log_energy), remainder)``. r+ra) window_length window_shiftprev_remainderr4) r3rrcrdrb_get_strided_batch_streamingrArBr4r`)rFrrhrfrP remainderr^rrronline_inferences   zWav2Win.online_inferencer)rN __module__ __qualname____doc__r#r>r$rboolfloatstrr;rLrJrTensorrr`rgjitexportrn __classcell__rrrHrr%;st     . $& r%csZeZdZdZddddddddd eddf d ed ed ed ededede dedededededdffdd Z e defddZ e defddZ e defddZe defd d!Ze defd"d#Ze de fd$d%Ze defd&d'Zd(ejd)eejdejfd*d+Zd,ejdejfd-d.Zejj d2d,ejd/eejdeejejffd0d1ZZS)3Wav2FFTad Apply standard Kaldi preprocessing (dithering, removing DC offset, pre-emphasis, etc.) on the input waveforms and compute their Short-Time Fourier Transform (STFT). The output is a complex-valued tensor. Example:: >>> x = torch.randn(1, 16000, dtype=torch.float32) >>> x.shape torch.Size([1, 16000]) >>> t = Wav2FFT() >>> t(x).shape torch.Size([1, 100, 257]) The input is a tensor of shape ``(batch_size, num_samples)``. The output is a tensor of shape ``(batch_size, num_frames, num_fft_bins)`` with dtype ``torch.complex64``. r&r'r(Tr)r*r+Fr,r-r.round_to_power_of_twor0r1r2r3r4r5r6 use_energyrNc s\t| |_tt||} |rt| n| |_t||||j||||| | | | d |_ dS)N) r/r0r1r2r3r4r5r6r7) r:r;r{r>r?r@next_power_of_2 fft_lengthr%wav2win)rFr,r-r.rzr0r1r2r3r4r5r6r{rGrHrrr;s$  zWav2FFT.__init__cC|jjSr)r~r,rKrrrr,zWav2FFT.sampling_ratecCrr)r~r-rKrrrr-rzWav2FFT.frame_lengthcCrr)r~r.rKrrrr.!rzWav2FFT.frame_shiftcCrr)r~r0rKrrrr0%rzWav2FFT.remove_dc_offsetcCrr)r~r1rKrrrr1)rzWav2FFT.preemph_coeffcCrr)r~r2rKrrrr2-rzWav2FFT.window_typecCrr)r~r3rKrrrr31rzWav2FFT.ditherrPlog_ecCs0t|}|jr|dur||dddddf<|SNr)rr{)rFrPrXrrrr`5szWav2FFT._forward_stridedrcCs||\}}|j||dS)NrPr)r~r`)rFrrPrrrrrgBszWav2FFT.forwardrhcCs*|jj||d\\}}}|j||d|fS)N)rhr)r~rnr`)rFrrhrPrrmrrrrnFszWav2FFT.online_inferencer)rNrorprqr#r>r$rrrsrtr;propertyr,r-r.r0r1r2r3rrurr`rgrvrwrrnrxrrrHrrys     "  ryceZdZdZddddddddd eddd f d ed ed ed ededede dededededededdffdd Z de j de e j de j fddZZS)Wav2Speca Apply standard Kaldi preprocessing (dithering, removing DC offset, pre-emphasis, etc.) on the input waveforms and compute their Short-Time Fourier Transform (STFT). The STFT is transformed either to a magnitude spectrum (``use_fft_mag=True``) or a power spectrum (``use_fft_mag=False``). Example:: >>> x = torch.randn(1, 16000, dtype=torch.float32) >>> x.shape torch.Size([1, 16000]) >>> t = Wav2Spec() >>> t(x).shape torch.Size([1, 100, 257]) The input is a tensor of shape ``(batch_size, num_samples)``. The output is a tensor of shape ``(batch_size, num_frames, num_fft_bins)``. r&r'r(Tr)r*r+Fr,r-r.rzr0r1r2r3r4r5r6r{ use_fft_magrNcBtj||||||||| | | | d | |_| rt|_dSt|_dSN rzr0r1r2r3r4r5r6r{r:r;rr_to_specrrFr,r-r.rzr0r1r2r3r4r5r6r{rrHrrr;d$  zWav2Spec.__init__rPrcCs:t|}||}|jr|dur||dddddf<|Sr)rrr{rFrPrrpow_specrrrr`s  zWav2Spec._forward_stridedrNrorprqr#r>r$rrrsrtr;rrurr`rxrrrHrrPf     $rcr) Wav2LogSpeca Apply standard Kaldi preprocessing (dithering, removing DC offset, pre-emphasis, etc.) on the input waveforms and compute their Short-Time Fourier Transform (STFT). The STFT is transformed either to a log-magnitude spectrum (``use_fft_mag=True``) or a log-power spectrum (``use_fft_mag=False``). Example:: >>> x = torch.randn(1, 16000, dtype=torch.float32) >>> x.shape torch.Size([1, 16000]) >>> t = Wav2LogSpec() >>> t(x).shape torch.Size([1, 100, 257]) The input is a tensor of shape ``(batch_size, num_samples)``. The output is a tensor of shape ``(batch_size, num_frames, num_fft_bins)``. r&r'r(Tr)r*r+Fr,r-r.rzr0r1r2r3r4r5r6r{rrNcrrrrrHrrr;rzWav2LogSpec.__init__rPrcCsFt|}||}|d}|jr!|dur!||dddddf<|S)NV瞯>> x = torch.randn(1, 16000, dtype=torch.float32) >>> x.shape torch.Size([1, 16000]) >>> t = Wav2LogFilterBank() >>> t(x).shape torch.Size([1, 100, 80]) The input is a tensor of shape ``(batch_size, num_samples)``. The output is a tensor of shape ``(batch_size, num_frames, num_filters)``. r&r'r(Tr)r*r+F4@yPr,r-r.rzr0r1r2r3r4r5r6r{rlow_freq high_freq num_filters norm_filterstorchaudio_compatible_mel_scalecstj||||||||| | | | d | |_||_||_||_||_tjt t t j j dd|_| r7t|_nt|_|rUt||j|||d\}}t jjj|ddddj}n t||j||||d }tj|dd|_dS NrFr8)num_binswindow_length_padded sample_freqrr)rrrTrrU)rr}r,rrr)r:r;rrrrrrrCrtensorfinforseps_epsrrr get_mel_banksr}rYrZTcreate_mel_scale_fb)rFr,r-r.rzr0r1r2r3r4r5r6r{rrrrrrfb_rHrrr;sV zWav2LogFilterBank.__init__rPrrcCs\t|}||}t||j}t||j}|jr,|dur,tj | d|fdd}|Sr ) rrrmatmulrmaxrrr{catr[rrrrr`5s z"Wav2LogFilterBank._forward_stridedrrrrHrrs     Grc)seZdZdZddddddddd edd d d d d d d ddfdedededededede dedededededededededed ed!ed"ef(fd#d$ Z e d%d&Z e d'd(Z d)ejd*eejd+ejfd,d-ZZS).Wav2MFCCa Apply standard Kaldi preprocessing (dithering, removing DC offset, pre-emphasis, etc.) on the input waveforms and compute their Mel-Frequency Cepstral Coefficients (MFCC). Example:: >>> x = torch.randn(1, 16000, dtype=torch.float32) >>> x.shape torch.Size([1, 16000]) >>> t = Wav2MFCC() >>> t(x).shape torch.Size([1, 100, 13]) The input is a tensor of shape ``(batch_size, num_samples)``. The output is a tensor of shape ``(batch_size, num_frames, num_ceps)``. r&r'r(Tr)r*r+Frr r,r-r.rzr0r1r2r3r4r5r6r{rrrrrnum_cepscepstral_lifterrcstj||||||||| | | | d | |_||_||_||_||_||_||_t j t t t jjdd|_| r=t|_nt|_|r[t||j|||d\}}t j jj|ddddj}n t||j||||d }t j |dd|_t j ||j|jdd|_t j ||j|jdd|_dSr)r:r;rrrrrrrrrCrrrrsrrrrrrr}rYrZrrrmake_dct_matrix_dct make_lifter_lifter)rFr,r-r.rzr0r1r2r3r4r5r6r{rrrrrrrrrrrHrrr;Wsf  zWav2MFCC.__init__c Cs:|dkrdSdd|ttjtj|td|S)zMakes the liftering function Args: N: Number of cepstral coefficients. Q: Liftering parameter Returns: Liftering vector. rr?dtype)rsinr?piarangeget_default_dtype)rGQrrrrs   zWav2MFCC.make_liftercCs~tt|d}tt|}ttjt||d|}|dddfdtd9<|tdt|9}|S)Nrrr?@)rrrsr[cosr?rr")rrrfkdctrrrrs"zWav2MFCC.make_dct_matrixrPrrcCsvt|}||}t||j}t||j}t||j}|j dkr*||j 9}|j r9|dur9||dddf<|Sr) rrrrrrrrrrrr{)rFrPrrrmfccrrrr`s   zWav2MFCC._forward_strided)rNrorprqr#r>r$rrrsrtr; staticmethodrrrrurr`rxrrrHrrEs     R   rwaveformrirjr4cCs.|dksJ|d}|d}|r&||krtdSd|||}nV||d|}|d||}||}t||d} || } t|ddd| fd} | dkrht|dd| dfd} n tjd|j|jd} tj | || fdd }| d|| d| df} |||g}| || S) aGiven a waveform (2D tensor of size ``(batch_size, num_samples)``, it returns a 2D tensor ``(batch_size, num_frames, window_length)`` representing how the window is shifted along the waveform. Each row is a frame. Args: waveform (torch.Tensor): Tensor of size ``(batch_size, num_samples)`` window_size (int): Frame length window_shift (int): Frame shift snip_edges (bool): If True, end effects will be handled by outputting only frames that completely fit in the file, and the number of frames depends on the frame_length. If False, the number of frames depends only on the frame_shift, and we reflect the data at the ends. Returns: torch.Tensor: 3D tensor of size (m, ``window_size``) where each row is a frame rrr )rrrrNr)rrbr ) r sizeremptyr>flipzerosrrbrstride as_strided)rrirjr4 batch_size num_samples num_framesnew_num_samplesnpad npad_left npad_rightpad_left pad_rightstridessizesrrrres.       reFrkcCsH||ksJ|dksJ|d}|dur9|s8t||d}t|ddd|fd}tj||fdd}n|dksAJ|d|ksJJtj||fdd}|d}|rq||krht|ddf|fSd|||} n ||} || |} |dd| |df} |d||d|df} || |g} || | | fS)a A variant of _get_strided_batch that creates short frames of a batch of audio signals in a way suitable for streaming. It accepts a waveform, window size parameters, and an optional buffer of previously unused samples. It returns a pair of waveform windows tensor, and unused part of the waveform to be passed as ``prev_remainder`` in the next call to this function. Example usage:: >>> # get the first buffer of audio and make frames >>> waveform = get_incoming_audio_from_mic() >>> frames, remainder = _get_strided_batch_streaming( ... waveform, ... window_shift=160, ... window_length=200, ... ) >>> >>> process(frames) # do sth with the frames >>> >>> # get the next buffer and use previous remainder to make frames >>> waveform = get_incoming_audio_from_mic() >>> frames, remainder = _get_strided_batch_streaming( ... waveform, ... window_shift=160, ... window_length=200, ... prev_remainder=prev_remainder, ... ) :param waveform: A waveform tensor of shape ``(batch_size, num_samples)``. :param window_shift: The shift between frames measured in the number of samples. :param window_length: The number of samples in each window (frame). :param prev_remainder: An optional waveform tensor of shape ``(batch_size, num_samples)``. Can be ``None`` which indicates the start of a recording. :param snip_edges (bool): If True, end effects will be handled by outputting only frames that completely fit in the file, and the number of frames depends on the frame_length. If False, the number of frames depends only on the frame_shift, and we reflect the data at the ends. :return: a pair of tensors with shapes ``(batch_size, num_frames, window_length)`` and ``(batch_size, remainder_len)``. rrNrrr r ) r rr>rrrrrr)rrjrirkr4rrrrrwindow_remainderrmrrrrrrls4 /     rlr5cCsB|ddd}|dkrt|tjt||jd}|S)zF Returns the log energy of size (m) for a strided_input (m,*) rr rr+r)r r!rrrrr?r)rr5r^rrrrX[srXTrr}r,rrrcCs@|dus|dkr |d}|dkr|d|}t|}t|}t|||d}ttd||} tjt|dd|ftjd} t|D]J} || } || d} || d}tt|dD]/}| |}| |krn|krnq^|| kr|| | | | || f<q^|||| | || f<q^qD|r| tj| ddd} t | S)NrrrrT)axiskeepdims) lin2melnplinspacerr>float32ranger!r from_numpy)rr}r,rrr mel_low_freq mel_high_freqmelfcmelsBrleft_mel center_mel right_meljmel_jrrrris0      rcCstttttgSr)HAMMINGHANNINGPOVEY RECTANGULARBLACKMANrrrravailable_windowssrhamminghanningr* rectangularblackmanzG?r2cCs|tkr tj|ddS|tkrtj|ddddS|tkr&tj|dddS|tkr3tj|t dS|t kr_dt j |}tj |t d}|d t||d |td||Std |) z6Returns a window function with the given type and sizeF)periodicgHzG?gq= ףp?)ralphabetag333333?rrrzInvalid window type: )rr hann_windowrhamming_windowrr ronesrrr?rrr Exception) window_sizer2blackman_coeffawindow_functionrrrrDs$rDcCsdtd|dS)N@r)rrrrrrrrcCsdt|ddS)Nrrr)rexprrrrmel2linrr cCs|dkrdSd|dS)z Returns the smallest power of 2 that is greater than x. Original source: TorchAudio (torchaudio/compliance/kaldi.py) rrr) bit_lengthrrrrr|sr|rrrcCsP|dksJd|ddksJ|d}d|}|dkr ||7}d|kr*|krrrrerlrsrXrrtrrrrrrrDrr r|rrrrrs    )mEGi 4 T %