o wiS@s ddlZddlmZmZddlmZddlmZmZm Z m Z m Z m Z ddl Z ddlZddlZddlmZddlmZddlmZmZmZddlmZeGd d d eZd eeefd ed ededef ddZde e ed eeefd ed edef ddZde edejd eeefd ed eddf ddZ d)deeefde ed eeefd ed ede e e!e!fddfddZ"d eeefde!de!de e e!e!ffddZ#deeefde ed e eeefddfd!d"Z$Gd#d$d$eZ%Gd%d&d&eZ&Gd'd(d(eZ'dS)*N)ABCabstractmethod)Path)AnyDictListOptionalTupleUnion)Tensor)!AudioToMelSpectrogramPreprocessor)get_audio_filepathsnormalize_volume stack_tensors) experimentalc @seZdZeddeeefdedededdf dd Z edeeefdededeee ffd d Z ed e eee fdeee ffd dZ dS) FeaturizerTmanifest_entry audio_dir feature_dir overwritereturnNcCdS)a\ Save feature value to disk for given manifest entry. Args: manifest_entry: Manifest entry dictionary. audio_dir: base directory where audio is stored. feature_dir: base directory where features will be stored. overwrite: whether to overwrite features if they already exist. N)selfrrrrrrn/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/nemo/collections/tts/parts/preprocessing/features.pysave!zFeaturizer.savecCr)aZ Read saved feature value for given manifest entry. Args: manifest_entry: Manifest entry dictionary. audio_dir: base directory where audio is stored. feature_dir: base directory where features were stored by save(). Returns: Dictionary of feature names to Tensors Nr)rrrrrrrload-rzFeaturizer.load train_batchcCr)zK Combine list/batch of features into a feature dictionary. Nr)rrrrr collate_fn;rzFeaturizer.collate_fnT)__name__ __module__ __qualname__rrstrrrboolrr rrrrrrrrs* , ,rrrr feature_namercCs&t||d\}}|||d}|S)z Get the absolute path for the feature file corresponding to the input manifest entry Example: audio_filepath "/speaker1/audio1.wav" becomes feature_filepath "//speaker1/audio1.npy" rrz.npy)r with_suffix)rrrr&_audio_filepath_relfeature_filepathrrr_get_feature_filepathBs r, feature_namescCs6|D]}|dur qt||||d}|sdSqdS)Nrrrr&FT)r,exists)r-rrrr&r+rrr_features_existsPsr0featurescCsB|durdSt||||d}|jjdddtjt||ddS)zA If feature_name is provided, save feature as .npy file. Nr.T)exist_okparents)filearr)r,parentmkdirnprr$)r&r1rrrr+rrr _save_feature^s r9 feature_dictindicesc Csv|durdSt||||d}t|}|r+tj|dd}||d|d}t|}nt|}t|} | ||<dS)zU If feature_name is provided, load feature into feature_dict from .npy file. Nr.r) mmap_moder)r,r$r8rcopytorch from_numpy) r:r&rrrr;r+ feature_mmap feature_arrayfeature_tensorrrr _load_featurers     rE sample_rate hop_lengthcCsPd|vrdS|d}|d}tjj|||d}d|tjj|||d}||fS)Noffsetduration)srrGr>)librosacoretime_to_frames)rrFrGrHrIstart_iend_irrr_get_frame_indicessrPrcCsX|durdSg}|D] }||}||q tdd|D}t||gd}|||<dS)NcSsg|]}|jdqS)r)shape).0frrr sz$_collate_feature..)max_lens)appendmaxr)r:r&rfeature_tensorsexamplerDmax_lenstacked_featuresrrr_collate_features  r\c@seZdZ         d(d ed edededededededededeeeefdedd fddZ de ee fde de jfddZd)de ee fde de d edd f d!d"Zde ee fde de de eeffd#d$Zd%ee eefde eeffd&d'Zd S)*MelSpectrogramFeaturizermel_spec"VPr@Tadd?Nr&rFmel_dim win_lengthrGlowfreqhighfreqloglog_zero_guard_typelog_zero_guard_valuemel_norm volume_normrc Cs||_||_||_||_| |_tdid|d|ddd|d|ddd dd |d |d |d dd|d| d| d| dddddd|_dS)NrFr1pad_tor> n_window_sizen_window_stride window_sizeF window_striden_fftrhri mag_powerrerjrkrlrm normalizepreemphditherr)r&rFrgrGrnr preprocessor) rr&rFrfrgrGrhrirjrkrlrmrnrrr__init__sT      z!MelSpectrogramFeaturizer.__init__rrc Cst||d\}}tj||jd\}}|jrt|}tj|tj ddftj d}tj|j dgtj d}|j ||d\}}|d}|} | S)a( Computes mel spectrogram for the input manifest entry. Args: manifest_entry: Manifest entry dictionary. audio_dir: base directory where audio is store Returns: [spec_dim, T_spec] float tensor containing spectrogram features. r'rJN)dtyper) input_signallength)r rKrrFrnrr@tensorr8newaxisfloat32rQint32rzdetachnumpy) rrraudio_filepath_absr)audio audio_tensoraudio_len_tensor spec_tensor spec_arrayrrrcompute_mel_specs  z)MelSpectrogramFeaturizer.compute_mel_specrrcCB|st|jg|||drdS|j||d}t|j||||ddSN)r-rrrr')r&r1rrr)r0r&rr9)rrrrrspecrrrr zMelSpectrogramFeaturizer.savecCs0i}t||j|jd}t||j||||d|SN)rrFrG)r:r&rrrr;)rPrFrGrEr&rrrrr:r;rrrrs zMelSpectrogramFeaturizer.loadrcCi}t||j|d|SN)r:r&rr\r&rrr:rrrrz#MelSpectrogramFeaturizer.collate_fn) r^r_r`rarbrrcTrdreNTr )r!r"r#r$intr%floatrr r{rrrr8ndarrayrrr rrrrrrrr]sX      *(**r]c @seZdZddededdfddZdeeefd ede j fd d Z ddeeefd ed ede ddf ddZ deeefd ed edeeeffddZdeeeefdeeeffddZdS)EnergyFeaturizerenergyspec_featurizerr&rNcCs||_||_dSN)r&r)rrr&rrrr{s zEnergyFeaturizer.__init__rrcCs$|jj||d}tjj|dd}|S)a Computes energy for the input manifest entry. Args: manifest_entry: Manifest entry dictionary. audio_dir: base directory where audio is store Returns: [T_spec] float tensor containing energy features. r'raxis)rrr8linalgnorm)rrrrrrrrcompute_energys zEnergyFeaturizer.compute_energyTrrcCrr)r0r&rr9)rrrrrrrrrr/rzEnergyFeaturizer.savecCs4i}t||jj|jjd}t||j||||d|Sr)rPrrFrGrEr&rrrrrAszEnergyFeaturizer.loadrcCrrrrrrrrRrzEnergyFeaturizer.collate_fn)rr )r!r"r#r]r$r{rrrr8rrr%rr rrrrrrrrs  (**rc@s$eZdZdZddddddeded d d d f d eedeedeedededededede dee deddfddZ de ee fdedeejejejffddZd(de ee fdeded e ddf d!d"Zde ee fdedede eeffd#d$Zd%ee eefde eeffd&d'ZdS))PitchFeaturizera  Class for computing pitch features. Args: pitch_name: Optional directory name to save pitch features under. If None, then pitch will not be saved. voiced_mask_name: Optional directory name to save voiced mask under. If None, then voiced mask will not be saved. voiced_prob_name: Optional directory name to save voiced probabilities under. If None, then voiced probabilities will not be saved. sample_rate: Sample rate to use when loading audio. win_length: Audio frame length to use for pitch computation. hop_length: Audio hop length to use for pitch computation. pitch_fmin: Minimum pitch value to compute. Defaults to librosa.note_to_hz('C2') = 65.41 Hz. pitch_fmax: Maximum pitch value to compute. Defaults to librosa.note_to_hz('C7') = 2093.00 Hz. Setting this to a lower value will speed up computation, but may lose some pitch information. volume_norm: Whether to apply volume normalization to the audio. batch_seconds: Optional float, if provided then long audio files will have their pitch computed after splitting them into segments batch_seconds seconds long, to avoid running out of memory. batch_padding: If batch_seconds is provided, then this determines how many audio frames will be padded on both sides of each segment to ensure that the pitch values at the boundary are correct. If batch_seconds is not provided then this parameter is ignored. pitch voiced_maskNr_rarbC2C7Tg>@ pitch_namevoiced_mask_namevoiced_prob_namerFrgrG pitch_fmin pitch_fmaxrn batch_seconds batch_paddingrc Cs||_||_||_||_||_||_| |_||_||_| rG| dus#Jt | |} t t | |j|_ |j|j |_ | |_|j|j|_dSd|_ d|_dSr)rrrrFrgrGrnrrrmathceil batch_frames batch_samplesbatch_padding_framesbatch_padding_samplesr) rrrrrFrgrGrrrnrrrrrrr{qs$   zPitchFeaturizer.__init__rrc Cst||d\}}tj||jd\}}|jrt|}|jr$|jd|jkr9tj||j |j |j|j |j dd\}}}nt t|jd|j} g} g} g} t| D]} | |j}| d|j}| dkrg||j8}| | dkrr||j7}|||}tj||j |j |j|j |j dd\}}}| dkr||jd}||jd}||jd}| | dkr|d|j}|d|j}|d|j}| || || |qPtj| dd}tj| dd}tj| dd}|tj}|tj}|||fS) a Computes pitch and optional voiced mask for the input manifest entry. Args: manifest_entry: Manifest entry dictionary. audio_dir: base directory where audio is store Returns: pitch: [T_spec] float tensor containing pitch for each audio frame. voiced_mask: [T_spec] bool tensor indicating whether each audio frame is voiced. voiced_prob: [T_spec] float array with [0, 1] probability that each audio frame is voiced. r'r|rry)fminfmaxrJ frame_lengthrGfill_nar>Nr)r rKrrFrnrrrQpyinrrrgrGrr8rrangerrrrV concatenateastyper)rrrrr)rrr voiced_prob num_chunks pitch_listvoiced_mask_listvoiced_prob_listirNrO audio_chunkpitch_i voiced_mask_i voiced_prob_irrr compute_pitchsh                zPitchFeaturizer.compute_pitchrrcCsx|st|j|j|jg|||drdS|j||d\}}}t|j||||dt|j||||dt|j||||ddSr)r0rrrrr9)rrrrrrrrrrrrs: zPitchFeaturizer.savecCs\i}t||j|jd}t||j||||dt||j||||dt||j||||d|Sr)rPrFrGrErrrrrrrrs: zPitchFeaturizer.loadrcCs8i}t||j|dt||j|dt||j|d|Sr)r\rrrrrrrr$s zPitchFeaturizer.collate_fnr )r!r"r#__doc__rK note_to_hzrr$rr%rr{rrrr r8rrrr rrrrrrrrXs`     #  (Q* *rr)(rabcrrpathlibrtypingrrrrr r rKrr8r@r nemo.collections.asr.modulesr 2nemo.collections.tts.parts.utils.tts_dataset_utilsr rrnemo.utils.decoratorsrrr$r,r%r0rr9rrErPr\r]rrrrrrs     "          .  p?