o }oiMe@s@ddlZddlmZddlZddlmmZddlm Z ddl m Z ddl m Z mZmZddlmZmZmZddlmZddlmZmZdd lmZdd lmZmZmZdd lm Z dd l!m"Z"m#Z#dd l$m%Z%m&Z&ddl'm(Z(ddl)m*Z*m+Z+ddl,m-Z-m.Z.dZ/zddl0Z0Wn e1ydZ/YnwGdddee Z2dS)N)Path) instantiate) WandbLogger) DictConfig OmegaConf open_dict)DiscriminatorLossFeatureMatchingLoss GeneratorLoss)Vocoder)MultiPeriodDiscriminatorMultiScaleDiscriminator)LoggingCallback)get_batch_sizeget_num_workersplot_spectrogram_to_numpy) Exportable)PretrainedModelInfo typecheck) AudioSignalMelSpectrogramType) NeuralType)compute_max_stepsprepare_lr_scheduler)logging model_utilsTFcsNeZdZdZdIdeddffdd Zedd Zed d Z d d Z dJddZ e ddZ e dedeidedeiddKddZddZdLddZd d!Zd"d#Zd$d%Zd&d'Zd(d)ZdMd,ed-efd.d/Zd0d1Zd2d3Zd4d5Zd6d7Ze dNd9d:Z!dOfd;d< Z"d=d>Z#ed?d@Z$edAdBZ%dPdEdFZ&dGdHZ'Z(S)Q HifiGanModelzp HiFi-GAN model (https://arxiv.org/abs/2010.05646) that is used to generate audio from mel spectrogram. NcfgtrainerTrainercs t|}t|}|jjj|_tj||dt |j |_ t |j ddd|_ t |j |_ td|vr5|jndd|_td|vrB|jndd|_t|_t|_t|_|dd|_|jj j|_d|_d|_|jro|jjj |_|d d|_!|d d|_"d|_#d|_$dS) N)rrT)highfreq use_gradsdebugF)r"l1_loss_factor- log_audio log_config)%r#convert_model_config_to_dict_configmaybe_update_config_versiontrain_dsdataset_target_ds_classsuper__init__r preprocessoraudio_to_melspec_precessortrg_melspec_fn generatorr r"mpdr msdr feature_lossrdiscriminator_lossr generator_lossget l1_factor_cfg sample_rate stft_bias input_as_mel _train_dlload_precomputed_melr%r&lr_schedule_intervalautomatic_optimization)selfrr __class__W/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/tts/models/hifigan.pyr.0s,        zHifiGanModel.__init__c Cs|d|jvr |jdSd|jvrtdd|jvr!|jj|jjSt|jj|jj|jjt |jt |j j t |j |j jdS)N max_steps max_epochsz)Must specify 'max_steps' or 'max_epochs'.steps_per_epoch)rHaccumulate_grad_batcheslimit_train_batches num_workers num_samples batch_size drop_last)r:r8 ValueErrorrHrIrrrJrKrlenr>r*rrOrBrErErFrGRs     zHifiGanModel.max_stepscCs |dur|S|dur||SdSNrErG warmup_steps warmup_ratiorErErFget_warmup_stepsgs zHifiGanModel.get_warmup_stepsc Cs|jj}t|d|dd}t|d|j}t |j |j }t ||d}t ||d}|dur=||gS|j }|j||dd|ddd}t|d||d<|rf||d<|ddt|dt|||jd } t|||jd } | d |_||g| | gfS) NFschedT)paramsrUrVrTrG) optimizerscheduler_configtrain_dataloaderinterval)r:optimcopyr set_structpopr2 parameters itertoolschainr4r3rrGrWr8rr>r@) rB optim_config sched_config gen_params disc_paramsoptim_goptim_drGrU scheduler_g scheduler_drErErFconfigure_optimizersqs<             z!HifiGanModel.configure_optimizersstepcCs>|}|dur|j|kr|\}}||dSdSdSrS) lr_schedulersr@rn)rBr] schedulerssch1sch2rErErF update_lrs  zHifiGanModel.update_lrcC |j|dSz^ Runs the generator, for inputs and outputs see input_types, and output_types xr2rBspecrErErFforwards zHifiGanModel.forwardrz)BCTaudio)r|r~) input_types output_types torch.tensorreturncCs||ddS)Nrz)squeezeryrErErFconvert_spectrogram_to_audiosz)HifiGanModel.convert_spectrogram_to_audioc Cs||\}}}}|||\}}|d}|j|d}||d|\} }|\} } | |j||d\} } }}|j | | d\}}}|j ||d\}}}}|j ||d\}}}||}| || | t | |}|j||d\}} }}|j ||d\}}}}|j||d}|j||d}|j| d\}}|j|d\}}||||||j}| || ||||||||||j| jddd }|j|d d d |jd |d d d ddS)Nrrv)yy_hat)disc_real_outputsdisc_generated_outputs)fmap_rfmap_g) disc_outputsrlr) g_loss_fm_mpd g_loss_fm_msdg_loss_gen_mpdg_loss_gen_msdg_loss d_loss_mpd d_loss_msdd_loss global_steprT)on_step sync_dist g_l1_lossF)prog_barloggerr)_process_batchr1 unsqueezer2r optimizers zero_gradr3detachr6r4manual_backwardrnFl1_lossr5r7r9rsr param_groupslog_dictlog)rBbatch batch_idxr audio_len audio_mel_ audio_trg_mel audio_predaudio_pred_melrirjmpd_score_real mpd_score_gen loss_disc_mpdmsd_score_real msd_score_gen loss_disc_msdloss_dloss_mel fmap_mpd_real fmap_mpd_gen fmap_msd_real fmap_msd_gen loss_fm_mpd loss_fm_msd loss_gen_mpd loss_gen_msdloss_gmetricsrErErF training_stepsV          zHifiGanModel.training_stepcCs|ddS)Nepoch)rsrRrErErFon_train_epoch_endszHifiGanModel.on_train_epoch_endc Cs||\}}}}||d}|jr|||\}} ||d|\} } t|| } |jd| iddd|jr=|dkr?t|j t rAt rC| ||d} || |\}} g}g}t td|jdD]}|tj||d||fjd||jd tj||dd||fjd d ||jd tj| |d||fjd ||jd g7}|tjt||ddd||fjd |dtjt| |ddd||fjd|dtjt||ddd||fjd|dg7}|jr0|tjt||ddd||fjd|dg7}qf|j j||ddSdSdSdSdS)Nrrval_lossT)on_epochrrz real audio )captionr;float32zgenerated audio zdenoised audio z input mel )rz output mel z denoised mel zgt mel )rspecs)rr=r0rrrrr% isinstancerr HAVE_WANDB _bias_denoiserangeminshapewandbAudiodatacpunumpyr;astypeImager experimentr)rBrrrrr audio_mel_lenrgt_mel gt_mel_lenrrr pred_denoisedpred_denoised_melclipsrirErErFvalidation_stepsl  &$&&&&zHifiGanModel.validation_stepcCsv|jr|\}}}|jdg|jd}||||fS|jdkr)|d}|d}n|\}}|||\}}||||fS)Nrr8nemo.collections.tts.data.vocoder_dataset.VocoderDatasetr audio_lens)r=rr,r8r0)rBrrrrrrErErFr(s      zHifiGanModel._process_batchc Csdd}dd}|jdus|jjd|jdkr@|tj||jdd}||\|_}|jdddddfdddddf|_||\}}||jdd |j}t|d }|||d } | S) NcSs`tj|dddddd}t|}|d|d}}t|d|d}t||}||fS) NrT)n_fft hop_length win_lengthreturn_complex).r).r)torchstftr view_as_realsqrtatan2)rwcomprealimagmagsphaserErErFr8s   z(HifiGanModel._bias_denoise..stftcSsBtj|t||t|gdd}tjt|dddd}|S)N)dimrr)rrr)rstackcossinistftview_as_complex)rrrrwrErErFr@s&z)HifiGanModel._bias_denoise..istftr)devicerdenoise_strengthg{Gzd?gr) r<rr zeros_likerrr8clampr) rBrmelrr audio_biasr audio_mags audio_phaseaudio_denoisedrErErFr7s ,  zHifiGanModel._bias_denoisecCsDt|j}|j|jj|jjd}tjj j |f|j |d|j}|S)N) world_size) collate_fnsampler) rr* get_samplerdataloader_paramsrNrrrutilsr DataLoaderr)rBrr*r data_loaderrErErF_setup_train_dataloaderRs z$HifiGanModel._setup_train_dataloadercCs,t|j}tjjj|fd|ji|j}|S)Nr)rr*rr rr rr )rBrr*r rErErF_setup_test_dataloaderZs z#HifiGanModel._setup_test_dataloaderTtrainshuffle_should_benamecCsd|vs t|jtstd|d|vst|jts"td||rad|jvrPtd|d|dt|d d |j_Wdn1sJwYn#|jjs`t d |d |d n|ss|jjrst d |d |d t |j}t j j j|fd|ji|jS)Nr*zNo dataset for r zNo dataloader_params for shufflez"Shuffle should be set to True for z's zE dataloader but was not found in its config. Manually setting to TrueTzThe z dataloader for z has shuffle set to False!!!z has shuffle set to True!!!r)rr*rrPr rwarningrrerrorrrr rr r)rBrrrr*rErErF__setup_dataloader_from_config_s(    z+HifiGanModel.__setup_dataloader_from_configcCs*|jdkr |||_dS|||_dS)Nr)r,r r>+_HifiGanModel__setup_dataloader_from_configrBrrErErFsetup_training_datats z HifiGanModel.setup_training_datacCs0|jdkr |||_dS|j|ddd|_dS)NrF validation)rr)r,r_validation_dlrrrErErFsetup_validation_datazs z"HifiGanModel.setup_validation_datacCsdSrSrErrErErFsetup_test_dataszHifiGanModel.setup_test_datac Cs|jsgS|jjj}|dkrtd|||j}t|jj}|jjr+t|jjnd}t |||jj |jj ||j j |jj|jjd}|gS)Nrz6Sample logging only supported for VocoderDataset, got ) generatorsr  log_epochsepoch_frequency output_dirloggerslog_tensorboard log_wandb)r&r*r+rPrrrlog_dirrrrrrr!r"r#)rBsample_ds_classr rr$ log_callbackrErErFconfigure_callbackss&    z HifiGanModel.configure_callbacksOptional[Dict[str, str]]cCsg}tddd|d}||tddd|d}||tdd d |d}||td d d |d}||tddd|d}||tddd|d}||tddd|d}||tddd|d}||tddd|d}|||S)Ntts_en_hifiganzehttps://api.ngc.nvidia.com/v2/models/nvidia/nemo/tts_hifigan/versions/1.0.0rc1/files/tts_hifigan.nemozThis model is trained on LJSpeech audio sampled at 22050Hz and mel spectrograms generated from Tacotron2, TalkNet, and FastPitch. This model has been tested on generating female English voices with an American accent.)pretrained_model_namelocation descriptionclass_tts_en_lj_hifigan_ft_mixerttszzhttps://api.ngc.nvidia.com/v2/models/nvidia/nemo/tts_en_lj_hifigan/versions/1.6.0/files/tts_en_lj_hifigan_ft_mixertts.nemozThis model is trained on LJSpeech audio sampled at 22050Hz and mel spectrograms generated from Mixer-TTS. This model has been tested on generating female English voices with an American accent.tts_en_lj_hifigan_ft_mixerttsxz{https://api.ngc.nvidia.com/v2/models/nvidia/nemo/tts_en_lj_hifigan/versions/1.6.0/files/tts_en_lj_hifigan_ft_mixerttsx.nemozThis model is trained on LJSpeech audio sampled at 22050Hz and mel spectrograms generated from Mixer-TTS-X. This model has been tested on generating female English voices with an American accent.#tts_en_hifitts_hifigan_ft_fastpitchzhttps://api.ngc.nvidia.com/v2/models/nvidia/nemo/tts_en_multispeaker_fastpitchhifigan/versions/1.10.0/files/tts_en_hifitts_hifigan_ft_fastpitch.nemozThis model is trained on HiFiTTS audio sampled at 44100Hz and mel spectrograms generated from FastPitch. This model has been tested on generating male and female English voices with an American accent.1tts_de_hifigan_singleSpeaker_thorstenNeutral_2102zhttps://api.ngc.nvidia.com/v2/models/nvidia/nemo/tts_de_fastpitchhifigan/versions/1.15.0/files/tts_de_hifigan_thorstens2102.nemoaThis model is finetuned from the HiFiGAN pretrained checkpoint `tts_en_lj_hifigan_ft_mixerttsx` by the mel-spectrograms generated from the FastPitch checkpoint `tts_de_fastpitch_singleSpeaker_thorstenNeutral_2102`. This model has been tested on generating male German neutral voices.1tts_de_hifigan_singleSpeaker_thorstenNeutral_2210zhttps://api.ngc.nvidia.com/v2/models/nvidia/nemo/tts_de_fastpitchhifigan/versions/1.15.0/files/tts_de_hifigan_thorstens2210.nemoaThis model is finetuned from the HiFiGAN pretrained checkpoint `tts_en_lj_hifigan_ft_mixerttsx` by the mel-spectrograms generated from the FastPitch checkpoint `tts_de_fastpitch_singleSpeaker_thorstenNeutral_2210`. This model has been tested on generating male German neutral voices..tts_de_hui_hifigan_ft_fastpitch_multispeaker_5zhttps://api.ngc.nvidia.com/v2/models/nvidia/nemo/tts_de_fastpitch_multispeaker_5/versions/1.11.0/files/tts_de_hui_hifigan_ft_fastpitch_multispeaker_5.nemoaThis model is finetuned from the HiFiGAN pretrained checkpoint `tts_en_hifitts_hifigan_ft_fastpitch` by the mel-spectrograms generated from the FastPitch checkpoint `tts_de_fastpitch_multispeaker_5`. This model has been tested on generating male and female German voices.(tts_es_hifigan_ft_fastpitch_multispeakerzhttps://api.ngc.nvidia.com/v2/models/nvidia/nemo/tts_es_multispeaker_fastpitchhifigan/versions/1.15.0/files/tts_es_hifigan_ft_fastpitch_multispeaker.nemoa2This model is trained on the audio from 6 crowdsourced Latin American Spanish OpenSLR datasets and finetuned on the mel-spectrograms generated from the FastPitch checkpoint `tts_es_fastpitch_multispeaker`. This model has been tested on generating male and female Spanish voices with Latin American accents.tts_zh_hifigan_sfspeechzhttps://api.ngc.nvidia.com/v2/models/nvidia/nemo/tts_zh_fastpitch_hifigan_sfspeech/versions/1.15.0/files/tts_zh_hifigan_sfspeech.nemoaThis model is finetuned from the HiFiGAN pretrained checkpoint `tts_en_lj_hifigan_ft_mixerttsx` by the mel-spectrograms generated from the FastPitch checkpoint `tts_zh_fastpitch_sfspeech`. This model has been tested on generating female Mandarin Chinese voices.)rappend)clslist_of_modelsmodelrErErFlist_available_modelss         z"HifiGanModel.list_available_modelsc si}t|jdd}|D]9\}}|}d|vrD|d}t|dkrDt|d} | |d| |} d| dd|dd}|||<qtj||d dS) Nr2resblock_kernel_sizes resblocks.rzgenerator.resblocks.)strict)rQritemssplitintjoinr-load_state_dict) rB state_dictr@new_state_dict num_resblockskvnew_kpartslayer new_layerrCrErFrEs    zHifiGanModel.load_state_dictcKs4|jdurz|jWdStyYdSwdSrS)r2remove_weight_normrP)rBkwargsrErErF_prepare_for_exports  z HifiGanModel._prepare_for_exportcCsdtdtiS)Nrz)r|Dr~)rrrRrErErFrs zHifiGanModel.input_typescCsdtdt|jiS)Nr)r|Sr~)rrr;rRrErErFrszHifiGanModel.output_typesrrcCs:t|}tj||jdd|f|j|jd}d|ifS)zs Generates input examples for tracing etc. Returns: A tuple of input examples. r/nfilt)rdtyperz)nextrbrrandnrrrU)rB max_batchmax_dimparrrErErF input_example"s $ zHifiGanModel.input_examplecCrtrurxryrErErFforward_for_export,s zHifiGanModel.forward_for_exportrS)rn)rzrrr)rN)Tr)rr()T)rr))__name__ __module__ __qualname____doc__rr.propertyrG staticmethodrWrmrsrr{rrrrrrrrrr rboolstrrrrrr' classmethodr:rErQrrr[r\ __classcell__rErErCrFr+sL"   )   8> b    r)3rcpathlibrrtorch.nn.functionalnn functionalr hydra.utilsrlightning.pytorch.loggers.wandbr omegaconfrrr*nemo.collections.tts.losses.hifigan_lossesrr r nemo.collections.tts.models.baser ,nemo.collections.tts.modules.hifigan_modulesr r *nemo.collections.tts.parts.utils.callbacksr(nemo.collections.tts.parts.utils.helpersrrrnemo.core.classesrnemo.core.classes.commonrrnemo.core.neural_types.elementsrr"nemo.core.neural_types.neural_typernemo.core.optim.lr_schedulerrr nemo.utilsrrrrModuleNotFoundErrorrrErErErFs2