o wiu@sddlZddlZddlmZmZddlmZddlmZm Z ddl Z ddl m Z ddl mZmZddlmZmZddlmZmZdd lmZmZmZdd lmZmZmZmZmZm Z dd l!m"Z"dd l#m$Z$dd l%m&Z&m'Z'zddl(Z(ddl)Z(ddl*Z(e +e(j,Z-e +dZ.dZ/Wn e0ydZ/YnwgdZ1GdddeeZ2Gddde2eZ3Gddde2Z4GdddeZ5GdddeZ6GdddeZ7eGdddZ8eGd d!d!Z9eGd"d#d#Z:eGd$d%d%Z;eGd&d'd'Z{Gz?{Gz?Nr! per_feature ףp= ?@rTaddp>h㈵>F@slaneyuse_torchaudiocs||_|r|rt|d|r|rt|d|r"t||j}|r+t||j}t|||s7t}nt}|did|jd|d|d|d|d|d | d | d | d | d | d|d|d|d|d|d|d|d|d|d|d|d|d|d||_dS)NK received both window_size and n_window_size. Only one should be specified.O received both window_stride and n_window_stride. Only one should be specified. sample_rate n_window_sizen_window_stridewindow normalizen_fftpreemphnfiltlowfreqhighfreqloglog_zero_guard_typelog_zero_guard_valueditherpad_toframe_splicing exact_pad pad_value mag_powerrngnb_augmentation_prob nb_max_freqmel_normstft_exact_pad stft_convr')r_ ValueErrorintr+r,r r featurizer)r8rz window_size window_strider{r|r}r~rrfeaturesrrrrrrrrrrrrrrrwrrrfeaturizer_classr9r'r;r,s      z*AudioToMelSpectrogramPreprocessor.__init__} max_batchmax_dim min_lengthcCs>|jj}tj||g|d}tj|||g|d}||d<||fS)N)sizedevice)lowhighrrr) filter_banksrr/randnrandint)r8rrrdevsignalslengthsr'r'r; input_example#s z/AudioToMelSpectrogramPreprocessor.input_examplecCs |||SrG)rrHr'r'r;r?+s z.AudioToMelSpectrogramPreprocessor.get_featurescCs|jjSrG)rrrar'r'r;r.sz.AudioToMelSpectrogramPreprocessor.filter_banks)rhrirjNNr!rkNrlrmrNTrnrorprqrrFrrsNrtruFrvFF)rrr)rIrJrKrLstrrQ classmethodrVpropertyrbrgboolr,rrr?rrNr'r'r9r;rosXE  Mrcs~eZdZdZeddZeddZdefddZe d efd d Z            dfdd Z ddZ Z S)ra!Preprocessor that converts wavs to MFCCs. Uses torchaudio.transforms.MFCC. Args: sample_rate: The sample rate of the audio. Defaults to 16000. window_size: Size of window for fft in seconds. Used to calculate the win_length arg for mel spectrogram. Defaults to 0.02 window_stride: Stride of window for fft in seconds. Used to caculate the hop_length arg for mel spect. Defaults to 0.01 n_window_size: Size of window for fft in samples Defaults to None. Use one of window_size or n_window_size. n_window_stride: Stride of window for fft in samples Defaults to None. Use one of window_stride or n_window_stride. window: Windowing function for fft. can be one of ['hann', 'hamming', 'blackman', 'bartlett', 'none', 'null']. Defaults to 'hann' n_fft: Length of FT window. If None, it uses the smallest power of 2 that is larger than n_window_size. Defaults to None lowfreq (int): Lower bound on mel basis in Hz. Defaults to 0 highfreq (int): Lower bound on mel basis in Hz. Defaults to None n_mels: Number of mel filterbanks. Defaults to 64 n_mfcc: Number of coefficients to retain Defaults to 64 dct_type: Type of discrete cosine transform to use norm: Type of norm to use log: Whether to use log-mel spectrograms instead of db-scaled. Defaults to True. cCrXrYr^rar'r'r;rbXsz#AudioToMFCCPreprocessor.input_typescCrc+Returns definitions of module output ports.rdrZrf)rrr`rrar'r'r;rg` z$AudioToMFCCPreprocessor.output_typesrOcCrFrGr'rPr'r'r;rQhrRzAudioToMFCCPreprocessor.save_torScCrFrGr'rTr'r'r;rVkrWz$AudioToMFCCPreprocessor.restore_fromrhrirjNr!rtrmorthoTcs||_tstdtd|r|rt|d|r$|r$t|d|r-t||j}|r6t||j}t||i}||d<| |d<| |d<|pVdt t ||d <||d <||d <|j |d}|durttd |d ||d<tjj|j| | | ||d|_dS)Nz:Could not import torchaudio. Some features might not work.ztorchaudio is not installed but is necessary for AudioToMFCCPreprocessor. We recommend you try building it from source for the PyTorch version you have.rxryf_minf_maxn_melsrrr-r.z/Window argument for AudioProcessor is invalid: z,.For no window function, use 'ones' or None. window_fn)rzn_mfccdct_typenormlog_mels melkwargs)r_HAVE_TORCHAUDIOrerrorModuleNotFoundErrorrrr+r,mathceillog2r4get torchaudio transformsMFCCr)r8rzrrr{r|r}rrrrrrrr mel_kwargsrr9r'r;r,osL   z AudioToMFCCPreprocessor.__init__cCs4||}t|tj|jjtjd}||fS)Nr()rr/rr@r7r.long)r8rArBrseq_lenr'r'r;r?s "z$AudioToMFCCPreprocessor.get_features)rhrirjNNr!NrtNrmrmrrT)rIrJrKrLrrbrgrrQrrVr,r?rNr'r'r9r;r3s2$  FrcsheZdZdZeddZeddZ       ddedeffdd Ze ddZ Z S)ra Performs time and freq cuts in one of two ways. SpecAugment zeroes out vertical and horizontal sections as described in SpecAugment (https://arxiv.org/abs/1904.08779). Arguments for use with SpecAugment are `freq_masks`, `time_masks`, `freq_width`, and `time_width`. SpecCutout zeroes out rectangulars as described in Cutout (https://arxiv.org/abs/1708.04552). Arguments for use with Cutout are `rect_masks`, `rect_freq`, and `rect_time`. Args: freq_masks (int): how many frequency segments should be cut. Defaults to 0. time_masks (int): how many time segments should be cut Defaults to 0. freq_width (int): maximum number of frequencies to be cut in one segment. Defaults to 10. time_width (int): maximum number of time steps to be cut in one segment Defaults to 10. rect_masks (int): how many rectangular masks should be cut Defaults to 0. rect_freq (int): maximum size of cut rectangles along the frequency dimension Defaults to 5. rect_time (int): maximum size of cut rectangles along the time dimension Defaults to 25. use_numba_spec_augment: use numba code for Spectrogram augmentation use_vectorized_spec_augment: use vectorized code for Spectrogram augmentation cCrcz)Returns definitions of module input typesrdrZ input_specrBrrr`rrar'r'r;rbrz#SpectrogramAugmentation.input_typescCdtdtiSz*Returns definitions of module output typesaugmented_specrdrrrar'r'r;rgz$SpectrogramAugmentation.output_typesr NrtTFuse_vectorized_spec_augmentuse_numba_spec_augmentc st|dkrt||||d|_ndd|_||dkr+t|||||| | d|_ndd|_| rIttrIt dt |||||| d|_ dSd|_ dS) Nr) rect_masks rect_time rect_freqrcS|SrGr'rr'r'r;z2SpectrogramAugmentation.__init__..) freq_masks time_masks freq_width time_widthr mask_valueuse_vectorized_codecSrrGr'rr'r'r;rrz+Numba CUDA SpecAugment kernel is being used)rrrrrr) r+r,r spec_cutoutr spec_augmentrnumba_cuda_is_supportedrrinforspec_augment_numba) r8rrrrrrrrrrrr9r'r;r,s>      z SpectrogramAugmentation.__init__cCsD|j|d}|jdurt||r|j||d}|S|j||d}|S)Nrr)rrr r)r8rrBrr'r'r;rEs zSpectrogramAugmentation.forward) rrrrrrrNrtTF) rIrJrKrLrrbrgrr,rrErNr'r'r9r;rs0!    5rc sbeZdZdZeddZeddZ    dd ed ed ed effd d Z e ddZ Z S)raO Zeroes out fixed size time patches of the spectrogram. All samples in batch are guaranteed to have the same amount of masked time steps. Optionally also performs frequency masking in the same way as SpecAugment. Args: patch_size (int): up to how many time steps does one patch consist of. Defaults to 48. mask_patches (float): how many patches should be masked in each sample. if >= 1., interpreted as number of patches (after converting to int) if <1., interpreted as fraction of total tokens to be masked (number of patches is rounded up) Defaults to 10. freq_masks (int): how many frequency segments should be cut. Defaults to 0. freq_width (int): maximum number of frequencies to be cut in a segment. Defaults to 0. cCrcrrrar'r'r;rb>rz#MaskedPatchAugmentation.input_typescCrrrrar'r'r;rgFrz$MaskedPatchAugmentation.output_types0$@r patch_size mask_patchesrrcsjt||_|dkrt||_n|dkr||_d|_ntd|dkr0t|d|dd|_dSd|_dS)Nrrrzmask_patches cannot be negative)rrrr) r+r,rrr_mask_fractionrr r)r8rrrrr9r'r;r,Ks   z MaskedPatchAugmentation.__init__c Cs|}t|}|jdur"t||j}||jt||jdk}n|j}||j|kr1||j}t|jdD]+}||}t||j} t | |} | D]} d||dd| |j| d|jf<qMq8|j durp|j ||d}|S)Nrrtrrr) r/minrrrrrangeshaperandomsampler) r8rrBrmin_len len_fractionridxcur_lenpatchesmasked_patchesmpr'r'r;rEfs$    ( zMaskedPatchAugmentation.forward)rrrr) rIrJrKrLrrbrgrfloatr,rrErNr'r'r9r;r,s*  rcsneZdZdZfddZeeddZe ddZ e dd Z d e fd d Z ed e fddZZS)rz Pad or Crop the incoming Spectrogram to a certain shape. Args: audio_length (int): the final number of timesteps that is required. The signal will be either padded or cropped temporally to this size. cs*tt|||_|jdkrtddS)Nrzaudio_length must be non-negative. If using a dataclass with OmegaConf, please call OmegaConf.to_object(cfg) to call appropriate __post_init__ methods.)r+rr, audio_lengthr)r8rr9r'r;r,s z)CropOrPadSpectrogramAugmentation.__init__c Cs|}|jd}|j}|jd}||krHg}tjd||d|gd}t|D]\} }||| | ddd|||fq&tj|dd}~n%||d} ||d} ||ddkr`| d7} tjjj || | gddd}|d|}||fS) Nrrr)rrr)dimrconstant)r<value) rrr/r enumerateappendcatnn functionalpad) r8rArBimage num_imagesr image_len cutout_imagesoffsetrpad_left pad_rightr'r'r;rEs$  *   z(CropOrPadSpectrogramAugmentation.forwardcCrc)rrdrZr]rrar'r'r;rbrz,CropOrPadSpectrogramAugmentation.input_typescCrcrrrar'r'r;rgrz-CropOrPadSpectrogramAugmentation.output_typesrOcCrFrGr'rPr'r'r;rQrRz(CropOrPadSpectrogramAugmentation.save_torScCrFrGr'rTr'r'r;rVrWz-CropOrPadSpectrogramAugmentation.restore_from)rIrJrKrLr,rr/rMrErrbrgrrQrrVrNr'r'r9r;rs     rc@s~eZdZUdZeed<dZeed<dZe ed<dZ e ed<d Z e eed <d Z e eed <d Zeed <dZeed<d Ze eed<dZe ed<dZeed<dZeed<d Ze eed<dZeed<dZeed<dZe ed<dZe ed<d Zeed!<d"Zeed#<d$Zeed%<dZeed&<d'Ze ed(<d Ze eed)<d*Z e ed+<d,Z!eed-<d$Z"eed.<d/Z#eed0<d$Z$eed1<d$Z%eed2<d S)3'AudioToMelSpectrogramPreprocessorConfigz>nemo.collections.asr.modules.AudioToMelSpectrogramPreprocessor_target_rhrzrirrjrNr{r|r!r}rkr~rrlrrmrrrrTrrnrrorrprrqrrrrFrrrsrrrtrrurrwrvrrr)&rIrJrKrr__annotations__rzrrrrr{rr|r}r~rrrrrrrrrrrrrrrrrrrwrrrr'r'r'r;rs<                       rc@seZdZUdZeed<dZeed<dZe ed<dZ e ed<d Z e eed <d Z e eed <d Zeed <d Ze eed<dZe e ed<d Ze e ed<dZeed<dZeed<dZeed<dZeed<dZeed<d S)AudioToMFCCPreprocessorConfigz4nemo.collections.asr.modules.AudioToMFCCPreprocessorrrhrzrirrjrNr{r|r!r}rrtrrrmrrrrrrTr)rIrJrKrrrrzrrrrr{rr|r}rrrrrrrrrr'r'r'r;rs         rc@seZdZUdZeed<dZeed<dZeed<dZ eed<dZ e e ed<dZ eed<dZeed <dZeed <dZeed <d Ze e ed <dZeed<dZeed<d S)SpectrogramAugmentationConfigz4nemo.collections.asr.modules.SpectrogramAugmentationrrrrrrrrrrNrFrTr)rIrJrKrrrrrrrrrrrrrrrrrrrr'r'r'r;rs         rc@s"eZdZUeed<dZeed<dS)&CropOrPadSpectrogramAugmentationConfigrz=nemo.collections.asr.modules.CropOrPadSpectrogramAugmentationrN)rIrJrKrrrrr'r'r'r;rs rc@sJeZdZUdZeed<dZeed<dZeed<dZ eed<dZ e ed <d S) MaskedPatchAugmentationConfigrrrrrrrz4nemo.collections.asr.modules.MaskedPatchAugmentationrN) rIrJrKrrrrrrrrrr'r'r'r;rs    r)=rrabcrr dataclassesrtypingrrr/ packagingr-nemo.collections.asr.parts.numba.spec_augmentrr 1nemo.collections.asr.parts.preprocessing.featuresr r 4nemo.collections.asr.parts.submodules.spectr_augmentr r nemo.core.classesrrrnemo.core.neural_typesrrrrrrnemo.core.utilsrnemo.core.utils.numba_utilsr nemo.utilsrrrtorchaudio.functionaltorchaudio.transformsparse __version__TORCHAUDIO_VERSIONTORCHAUDIO_VERSION_MINrr__all__r rrrrrrrrrrr'r'r'r;sX         2E qXN