o :is%@sdZddlZddlZddlZddlZddlZddlmZddl m Z ddl Z ddl ZddlZddlZddlmZddlmZddlmZddlmZdd lmZdd lmZmZmZmZdd l m!Z!dd l"m#Z#m$Z$dd l%m&Z&ddl'm(Z(m)Z)m*Z*m+Z+m,Z,m-Z-m.Z.ddl/m0Z0dZ1dddZ2ddZ3dAde e4fddZ5       dBde4de4d e4d!e4d"ej6j7d#ej6j7d$ed%e&d&ej8d'e9d(e:d)e:d*e:d+e:d,e:d-e9f d.d/Z;        0 1dCde4de4d e4d!e4d"ej6j7d#ej6j7d$ed%e&d&ej8d'e9d(e:d)e:d*e:d+e:d,e:d-e9d2e:d3ee?d;d<Z@eAd=kr[d>ZBejCeBejDd?d@e@dSdS)Ea This script generates speech with our pre-trained ZipVoice or ZipVoice-Distill models. If no local model is specified, Required files will be automatically downloaded from HuggingFace. Usage: Note: If you having trouble connecting to HuggingFace, try switching endpoint to mirror site: export HF_ENDPOINT=https://hf-mirror.com (1) Inference of a single sentence: python3 -m zipvoice.bin.infer_zipvoice --model-name zipvoice --prompt-wav prompt.wav --prompt-text "I am a prompt." --text "I am a sentence." --res-wav-path result.wav (2) Inference of a list of sentences: python3 -m zipvoice.bin.infer_zipvoice --model-name zipvoice --test-list test.tsv --res-dir results `--model-name` can be `zipvoice` or `zipvoice_distill`, which are the models before and after distillation, respectively. Each line of `test.tsv` is in the format of `{wav_name} {prompt_transcription} {prompt_wav} {text}`. (3) Inference with TensorRT: python3 -m zipvoice.bin.infer_zipvoice --model-name zipvoice_distill --prompt-wav prompt.wav --prompt-text "I am a prompt." --text "I am a sentence." --res-wav-path result.wav --trt-engine-path models/zipvoice_distill_onnx_trt/fm_decoder.fp16.plan N)Path)Optional)hf_hub_download)fix_random_seed)Vocos)ZipVoice)ZipVoiceDistill)EmiliaTokenizerEspeakTokenizerLibriTTSTokenizerSimpleTokenizer)load_checkpoint) AttributeDictstr2bool) VocosFbank)add_punctuationbatchify_tokenschunk_tokens_punctuationcross_fade_concatload_prompt_wavremove_silencerms_norm)load_trtzk2-fsa/ZipVoicezipvoicezipvoice_distillrrcCstjtjd}|jdtdddgdd|jdtddd |jd td d d |jd tddd |jdtdgddd|jdtddd |jdtddd |jdtddd |jdtddd |jdtddd |jdtdd d |jd!td"d#d |jd$tdd%d |jd&tdd'd |jd(td)d*d |jd+td,d-d |jd.td/d0d |jd1td)d2d |jd3td4d5d |jd6td7d8d |jd9td:d;d |jdd |jd?td:d@d |jdAtddBd |S)CN)formatter_classz --model-namerrzThe model used for inference)typedefaultchoiceshelpz --model-dirzThe model directory that contains model checkpoint, configuration file model.json, and tokens file tokens.txt. Will download pre-trained checkpoint from huggingface if not specified.)rrr z--checkpoint-namezmodel.ptzThe name of model checkpoint.z--vocoder-pathz\The vocoder checkpoint. Will download pre-trained vocoder from huggingface if not specified.z --tokenizeremilia)r!librittsespeaksimplezTokenizer type.z--langzen-usz~Language identifier, used when tokenizer type is espeak. seehttps://github.com/rhasspy/espeak-ng/blob/master/docs/languages.mdz --test-listzThe list of prompt speech, prompt_transcription, and text to synthesizein the format of '{wav_name} {prompt_transcription} {prompt_wav} {text}'.z --prompt-wavzThe prompt wav to mimicz --prompt-textz#The transcription of the prompt wavz--textzThe text to synthesizez --res-dirresultsz^ Path name of the generated wavs dir, used when test-list is not None z--res-wav-pathz result.wavzZ Path name of the generated wav path, used when test-list is None z--guidance-scalez7The scale of classifier-free guidance during inference.z --num-stepzThe number of sampling steps.z --feat-scale皙?z!The scale factor of fbank featurez--speed?z;Control speech speed, 1.0 means normal, >1.0 means speed upz --t-shift?z(Shift t to smaller ones if t_shift < 1.0z --target-rmszHTarget speech normalization rms value, set to 0 to disable normalizationz--seediz Random seedz --num-threadz,Number of threads to use for PyTorch on CPU.z--raw-evaluationFztWhether to use the 'raw' evaluation mode where provided prompts and text are fed to the model without pre-processingz--max-durationdzMaximum duration (seconds) in a single batch, including durations of the prompt and generated wavs. You can reduce it if it causes CUDA OOM.z--remove-long-silzqWhether to remove long silences in the middle of the generated speech (edge silences will be removed by default).z--trt-engine-pathz%The path to the TensorRT engine file.)argparseArgumentParserArgumentDefaultsHelpFormatter add_argumentstrfloatintr)parserr32/home/ubuntu/LuxTTS/zipvoice/bin/infer_zipvoice.py get_parserls,     r5vocos_local_pathcCsF|rt|d}tj|dddd}|||Std}|S)Nz /config.yamlz/pytorch_model.binTcpu) weights_only map_locationzcharactr/vocos-mel-24khz)r from_hparamstorchloadload_state_dictfrom_pretrained)r6vocoder state_dictr3r3r4 get_vocoder-s  rAr'r(r&] save_path prompt_text prompt_wavtextmodelr? tokenizerfeature_extractordevicenum_stepguidance_scalespeedt_shift target_rms feat_scale sampling_ratec$ Csbt||d}t|| \}}|j||d|}|d|}tj|dg|d}||g}||g}t j }|j ||||| | d| | d \}}}}| ddd|}t j }||ddd}t j |}||}t j |}|jd|}||} ||}!||}"||||| |!|"d }#|| kr||| }tj|||d |#S) a" Generate waveform of a text based on a given prompt waveform and its transcription, this function directly feed the prompt_text, prompt_wav and text to the model. It is not efficient and can have poor results for some inappropriate inputs. (e.g., prompt wav contains long silence, text to be generated is too long) This function can be used to evaluate the "raw" performance of the model. Args: save_path (str): Path to save the generated wav. prompt_text (str): Transcription of the prompt wav. prompt_wav (str): Path to the prompt wav file. text (str): Text to be synthesized into a waveform. model (torch.nn.Module): The model used for generation. vocoder (torch.nn.Module): The vocoder used to convert features to waveforms. tokenizer (EmiliaTokenizer): The tokenizer used to convert text to tokens. feature_extractor (VocosFbank): The feature extractor used to extract acoustic features. device (torch.device): The device on which computations are performed. num_step (int, optional): Number of steps for decoding. Defaults to 16. guidance_scale (float, optional): Scale for classifier-free guidance. Defaults to 1.0. speed (float, optional): Speed control. Defaults to 1.0. t_shift (float, optional): Time shift. Defaults to 0.5. target_rms (float, optional): Target RMS for waveform normalization. Defaults to 0.1. feat_scale (float, optional): Scale for features. Defaults to 0.1. sampling_rate (int, optional): Sampling rate for the waveform. Defaults to 24000. Returns: metrics (dict): Dictionary containing time and real-time factor metrics for processing. rRrr)rKpredict tokens prompt_tokensprompt_featuresprompt_features_lensrNrOdurationrLrMt t_no_vocoder t_vocoder wav_secondsrtfrtf_no_vocoder rtf_vocoder sample_rate)rrextractto unsqueezer;tensorsizetexts_to_token_idsdtdatetimenowsamplepermutedecodesqueezeclamp total_secondsshape torchaudiosaver7)$rDrErFrGrHr?rIrJrKrLrMrNrOrPrQrR prompt_rmsrYrZrWrXstart_t pred_featurespred_features_lenspred_prompt_featurespred_prompt_features_lensstart_vocoder_twavr_r`rarbrcrdremetricsr3r3r4 generate_sentence_raw_evaluation;sd 5      rr*F max_durationremove_long_silc8 s$t||d}t||ddd}t|| \}}|jd|}|dkr*td|dn |d kr7td |d |j||d|}|d |}t |}t |}| |gd }| |gd }|jd|t || }t d ||}t ||d}||}||g}t||||\}}g}tj}|D]A} |t | }!|t | dd}"tjt | f|d|d}#|j| |!|"|#| | d| | d \}$}%}&}'|$d dd|}$||$|%fqg}(tj})|D]>\}$}%g}*t|$d D]+}+||$|+dddd|%|+fddd},|| kr|,|| },|*|,q|(|*qtj|}-ddt ||(D}.t!|.dddfddtt D}/t"|/d|d}0t|0|| d d}0|)|}1tj|)}2|0jd|}3|-|3}4|1|3}5|2|3}6|-|1|2|3|4|5|6d}7t#j$||0%|d|7S)aQ Generate waveform of a text based on a given prompt waveform and its transcription, this function will do the following to improve the generation quality: 1. chunk the text according to punctuations. 2. process chunked texts in batches. 3. remove long silences in the prompt audio. 4. add punctuation to the end of prompt text and text if there is not. Args: save_path (str): Path to save the generated wav. prompt_text (str): Transcription of the prompt wav. prompt_wav (str): Path to the prompt wav file. text (str): Text to be synthesized into a waveform. model (torch.nn.Module): The model used for generation. vocoder (torch.nn.Module): The vocoder used to convert features to waveforms. tokenizer (EmiliaTokenizer): The tokenizer used to convert text to tokens. feature_extractor (VocosFbank): The feature extractor used to extract acoustic features. device (torch.device): The device on which computations are performed. num_step (int, optional): Number of steps for decoding. Defaults to 16. guidance_scale (float, optional): Scale for classifier-free guidance. Defaults to 1.0. speed (float, optional): Speed control. Defaults to 1.0. t_shift (float, optional): Time shift. Defaults to 0.5. target_rms (float, optional): Target RMS for waveform normalization. Defaults to 0.1. feat_scale (float, optional): Scale for features. Defaults to 0.1. sampling_rate (int, optional): Sampling rate for the waveform. Defaults to 24000. max_duration (float, optional): The maximum duration to process in each batch. Used to control memory consumption when generating long audios. remove_long_sil (bool, optional): Whether to remove long silences in the middle of the generated speech (edge silences will be removed by default). Returns: metrics (dict): Dictionary containing time and real-time factor metrics for processing. rSF) only_edge trail_silr]zGiven prompt wav is too long (z>s). Please provide a shorter one (1-3 seconds is recommended). zGiven prompt wav is long (zMs). It will lead to slower inference speed and possibly worse speech quality.r) max_tokensr)rTrUrVr\NcSsg|]\}}||fqSr3r3).0indexrr3r3r4 `s z%generate_sentence..cSs|dS)Nrr3)xr3r3r4csz#generate_sentence..)keycsg|]}|dqS)r)r3)risequential_indexed_chunked_wavsr3r4rds r&) fade_durationrgr^rf)&rrrrwloggingwarningrhrirjrtexts_to_tokenslenr1rtokens_to_token_idsrrnrorprepeatr;fullrlrqrrappendrangersrtruextendrvzipsortedrrxryr7)8rDrErFrGrHr?rIrJrKrLrMrNrOrPrQrRrrrzprompt_durationrY tokens_strprompt_tokens_strtoken_durationrchunked_tokens_strchunked_tokensrXtokens_batches chunked_indexchunked_featuresr{ batch_tokensbatch_prompt_tokensbatch_prompt_featuresbatch_prompt_features_lensr|r}r~r chunked_wavsr batch_wavrrr_indexed_chunked_wavssequential_chunked_wavs final_wavr`rarbrcrdrerr3rr4generate_sentences <            "         rres_dir test_listraw_evaluationc Csg}g}g}g}t|d }|}Wdn1swYt|D]\}}|d\}}}}|d|d}id|d|d|d|d |d |d |d |d |d|d|d| d| d| d| d| }|rxtd"i|}n td"i|||d}td|d|td|d|dd| |d| |d| |d| |dq%tdt |t |dtd t |t |dtd!t |t |ddS)#Nr /z.wavrDrErFrGrHr?rIrJrKrLrMrNrOrPrQrR)rrz [Sentence: z ] Saved to: z] RTF: rcz.4fr_r`rarbz Average RTF: zAverage RTF w/o vocoder: zAverage RTF vocoder: r3) open readlines enumeratestripsplitrrrinfornpsum) rrrHr?rIrJrKrLrMrNrOrPrQrRrrrtotal_ttotal_t_no_vocodertotal_t_vocodertotal_wav_secondsfrlinesrlinewav_namerErFrGrD common_paramsrr3r3r4 generate_lists        "rcCsdt}|}t|jt|jt}|t|t |j ddddddd}| |j i}| D]\}}t||durTt|||td|d|q8|jdu|joa|joa|jduAsiJd |jdurt|j|_|jst|jd |jd d fD]}|j|st|j|d q|j|j}|jd } |jd } td |j d|jd|jn.td|j dttt|j dd}ttt|j dd} ttt|j dd} |jdkrt | d} n'|jdkrt!| d} n|jdkrt"| |j#d} n |jdksJt$| d} | j%| j&d} t'| d } t()| } Wdn 1s:wY|j dkrQt*dHi| d| }n|j d ksYJt+dHi| d| }t,|-d!rtt.j/||nt,|-d"rt0||d#d$nt1d%|tj23rt4d&d'|_4ntj5j63rt4d(|_4nt4d)|_4td*|j4|7|j4}|8|j9rt:||j9t;|j<}|7|j4}|8| d+d,d-krt=}n t1d.| d+d,| d+d/|_>td0|jrQ|j?}t@jA|d#d1tBdHid2|j?d3|jd|d4|d5| d6|d7|j4d8|jCd9|jDd:|jEd;|jFd<|jGd=|jHd/|j>d>|jId?|jJd@|jKnZ|jIrYJdAtLdHidB|jMdC|jdD|jdE|jd|d4|d5| d6|d7|j4d8|jCd9|jDd:|jEd;|jFd<|jGd=|jHd/|j>d?|jJd@|jKtdF|jMtdGdS)INrBr')rLrMg@rzSetting z to default value: zuFor inference, please provide prompts and text with either '--test-list' or '--prompt-wav, --prompt-text and --text'.z does not existz model.jsonz tokens.txtzUsing z in local model dir z , checkpoint zUsing pretrained z model from the Huggingfacez /model.pt)filenamez /model.jsonz /tokens.txtr!) token_filer"r#)rlangr$) vocab_sizepad_idrrrHrz .safetensorsz.ptT)rrHstrictz%Unsupported model checkpoint format: cudarmpsr7zDevice: featurervocoszUnsupported feature type: rRzStart generating...)exist_okrrr?rIrJrKrLrMrNrOrPrQrrrz-Raw evaluation is only valid with --test-listrDrErFrGz Saved to: Doner3)Nr5 parse_argsr;set_num_threads num_threadset_num_interop_threadsrupdatevarsrseedget model_nameitemsgetattrsetattrrrrrFrErG model_dirris_dirFileNotFoundErrorcheckpoint_nameis_filerHUGGINGFACE_REPO MODEL_DIRrIr r r rr rrrjsonr<rrr/endswith safetensors load_modelr NotImplementedErrorr is_availablerKbackendsrrievaltrt_engine_pathrrA vocoder_pathrrRrosmakedirsrrLrMrNrOrPrQrrrr res_wav_path)r2argsparamsmodel_defaultsmodel_specific_defaultsparamvaluer model_ckpt model_configrrItokenizer_configfrHr?rJrr3r3r4mainsj                                     r__main__z?%(asctime)s %(levelname)s [%(filename)s:%(lineno)d] %(message)sT)formatlevelforce)N)rBr'r'r(r&r&rC) rBr'r'r(r&r&rCr*F) rBr'r'r(r&r&rCFr*F)E__doc__r+rornrrrpathlibrtypingrnumpyrsafetensors.torchrr;rxhuggingface_hubr lhotse.utilsrrrzipvoice.models.zipvoicer zipvoice.models.zipvoice_distillrzipvoice.tokenizer.tokenizerr r r r zipvoice.utils.checkpointr zipvoice.utils.commonrrzipvoice.utils.featurerzipvoice.utils.inferrrrrrrrzipvoice.utils.tensorrtrrrr5r/rAnnModulerKr1r0rboolrrinference_moder__name__ formatter basicConfigINFOr3r3r3r4sV-         $ B            X      L  /