o J…µis<ã@s€dZddlZddlZddlZddlmZddlmZddlm Z dd„Z dd „Z d d „Z ddd„Z dd„Zedkr>eƒdSdS)a RTF optimization v2: properly skip CFG + torch.compile with safe mode. Key insight: cfg_scale=1.0 still runs negative LM + doubles diffusion batch. We patch sample_speech_tokens to run single-path when cfg disabled. We also patch generate() to skip negative LM forward entirely. éN)Ú*VibeVoiceForConditionalGenerationInference)Ú AudioStreamer)ÚVibeVoiceProcessorcs(t ¡d‡fdd„ ƒ}ˆjˆ_|ˆ_dS)zEPatch model to truly skip CFG: no negative LM, single-path diffusion.Nçð?csŽˆjj ˆj¡ˆjjj}| |¡}tj|j dˆj j ||j d}ˆjjj D]}ˆjj|| |j d¡ |¡|d}ˆjj |||¡j}q&|S)Nr)ÚdeviceÚdtype)Ú condition)ÚmodelÚnoise_schedulerÚ set_timestepsÚddpm_inference_stepsÚprediction_headrÚtoÚtorchÚrandnÚshapeÚconfigÚacoustic_vae_dimrÚ timestepsÚrepeatÚstepÚ prev_sample)rÚ neg_conditionÚ cfg_scalerÚspeechÚtÚeps©r ©ú)/home/ubuntu/vibevoice/rtf_optimize_v2.pyÚsample_speech_tokens_no_cfgs  $z1patch_no_cfg..sample_speech_tokens_no_cfg)Nr)rÚno_gradÚsample_speech_tokensÚ_original_sample)r r rrrÚ patch_no_cfgs r$cCst|dƒr |j|_dSdS)Nr#)Úhasattrr#r"rrrrÚunpatch_no_cfg#s  ÿr&csœddl}ddlm‰ddlm‰ddlm}m‰m}ddlm ‰t |j dƒr*|j j nd}t  ¡      d ‡‡‡‡fd d „ ƒ}|j |_| ||¡|_ dS) z4Patch generate to skip all negative/CFG computation.rN)ÚVibeVoiceGenerationOutput)Ú VibeVoiceTokenizerStreamingCache)ÚGenerationConfigÚLogitsProcessorListÚStoppingCriteriaList)Ú!VibeVoiceTokenConstraintProcessorÚ __wrapped__TrcD sl| dd¡}| dd¡}| dd¡}| dd¡}| dd¡dur.|jjj|djd|d<|j|||fd d i|¤Ž\}}}}}ˆƒ}ˆƒ}|jd }|j}tj |tj |d }d}| d d¡} dd„t |ƒDƒ}!|jd}"|dj dd}#|j |j|j|jg}$t|dƒr|jdur|$ |j¡ˆ|$|d}%|durœˆƒ}| |%¡t|j|"t||"ƒƒ}&t |j|#||# ¡¡}'tj |tj |d }(t |&ƒD]¹})| ¡rÑn±|jd|jkrÜn¦|j|fi|¤Ž}*|ri}+| durö| j|d|+d<| dur|  |¡|+d<| dur|  |¡|+d<d}n |* dd¡},d|i}+|d i|*¤|+¤dd dddœ¤Ž}-|j|-|dd}|-jdd…ddd…fjd tj|d}.|||.ƒ}/tj|/dd}0|j|0|<tj||0dd…dfgdd}|0|jk  ¡r–|0|jkj!dd "d¡}1|1||1}2|2 #¡d kr–d ||2<|dur–| $|2¡|0|jkj!dd "d¡}3|3 #¡d kr³| %|3¡| %|3¡|)|'k}4tj!|4|@dd "d¡}5|5 #¡d krÝd ||5<d |(|5<|durÝ| $|5¡|j& '¡|0ƒ (d¡}6tj)||d||0|jk@}7|7 #¡d kr|-j*|7ddd…f}8| +|8¡ (d¡}9|9|j&j, |9j¡|j&j- |9j¡}:|j&j.j/|: |j&j.j¡||7 |j&j.j¡d dd};t0|7ƒD]\}<}=|= 1¡}>||>sS|!|> |;|<¡q=|dur`| 2|;|7¡|j&j3j4|;||7d ddj5}?|j& 6|9¡}@|j& 7|?¡}A|@|A|6|7<|6}qÇ|dur‹| $¡g}B|!D]}C|Cr |B tj|Cdd¡q|B d¡qˆ||r±|B|(dSd|(dS)!NÚ tokenizerÚparsed_scriptsÚall_speakers_listÚmax_length_timeséÚmax_new_tokensÚ input_idséÿÿÿÿÚreturn_processorsTr)rrÚverboseFcSsg|]}g‘qSrr)Ú.0Ú_rrrÚ QszGpatch_generate_skip_negative..fast_generate..Úattention_mask)ÚdimÚ bos_token_id)rÚspeech_tensorsÚ speech_masksÚspeech_input_maskÚ inputs_embedsé)Úlogits_to_keepÚ return_dictÚoutput_attentionsÚoutput_hidden_states)Úis_encoder_decoder)Úcopyrr)Úas_tuple)ÚcacheÚsample_indicesÚ use_cacheÚdebug)Ú sequencesÚspeech_outputsÚreach_max_step_sampler)8ÚpopÚgetrÚdecoder_configÚmax_position_embeddingsrÚ#_build_generate_config_model_kwargsrrÚzerosÚboolÚrangeÚsumÚspeech_start_idÚ speech_end_idÚspeech_diffusion_idÚ eos_token_idr%r=ÚappendÚminÚ max_lengthÚintÚlongÚallÚprepare_inputs_for_generationrÚ#_update_model_kwargs_for_generationÚlogitsÚfloat32ÚargmaxÚcatÚanyÚnonzeroÚsqueezeÚnumelÚendÚ set_to_zeror Úget_input_embeddingsÚ unsqueezeÚarangeÚlast_hidden_stater"Úspeech_scaling_factorÚspeech_bias_factorÚacoustic_tokenizerÚdecodeÚ enumerateÚitemÚputÚsemantic_tokenizerÚencodeÚmeanÚacoustic_connectorÚsemantic_connector)DÚselfÚinputsÚgeneration_configÚlogits_processorÚstopping_criteriaÚprefix_allowed_tokens_fnÚ synced_gpusÚassistant_modelÚaudio_streamerÚnegative_prompt_idsÚnegative_prompt_attention_maskr>r?r@Ú is_prefillÚ return_speechrÚ stop_check_fnÚ tqdm_classÚkwargsr.r/r0r1Ú model_kwargsr4Úacoustic_cacheÚsemantic_cacheÚ batch_sizerÚ finished_tagsrAr7Ú audio_chunksÚinitial_lengthÚinitial_length_per_sampleÚ valid_tokensÚtoken_constraintÚ max_stepsÚmax_step_per_samplerPrÚ model_inputsÚprefill_inputsr9ÚoutputsÚnext_token_logitsÚnext_token_scoresÚ next_tokensÚeos_idxÚnew_eosÚdiffusion_end_indicesÚmax_length_reachedÚnew_maxÚnext_inputs_embedsÚdiffusion_indicesÚpositive_conditionÚ speech_latentÚ scaled_latentÚ audio_chunkÚiÚ sample_idxÚidxÚsemantic_featuresÚacoustic_embedÚsemantic_embedÚ final_audioÚchunks©r*r'r,r(rrÚ fast_generate2s    ÿÿÿ   þ    þ   ÿÿ þÿ&         ÿ$ý €  þý     þþz3patch_generate_skip_negative..fast_generate)NNNNNNNNNNNNNTTrNN)ÚtypesÚ.vibevoice.modular.modeling_vibevoice_inferencer'Ú-vibevoice.modular.modular_vibevoice_tokenizerr(Útransformers.generationr)r*r+r,r%Úgenerater-rr!Ú_original_generateÚ MethodType)r r·r)r+Úoriginal_generater¶rrµrÚpatch_generate_skip_negative(s$   ø#r¿éçÍÌÌÌÌÌô?c sz| d¡s d|›}||g|ggdddd}| ¡D]\}}t |¡r*| d¡||<qtddd‰dg‰d g‰dg} ‡‡‡fd d „} tj| dd } |  ¡t  d ¡tj   d ¡|j |dtj   ¡t ¡| d <|jdi|¤d||jddiddˆddœ¤Ž} tj   ¡t ¡} | jdd| | d }ˆd r¢ˆd | d dnd}ˆd d}|d kr²||ntdƒ}||||dœS)NÚSpeakerz Speaker 1: TÚpt)ÚtextÚ voice_samplesÚpaddingÚreturn_tensorsÚreturn_attention_maskÚcudarB)r“Ú stop_signalrcsFˆ d¡D]}t ¡}ˆddur|ˆd<ˆd|jd7<qdS)Nrr5)Ú get_streamÚtimeÚ perf_counterr)Úchunkr©Ú first_chunkÚstreamerÚ total_samprrÚconsumerés  üzmeasure..consumer)ÚtargetÚdaemoné*)Ú num_stepsÚ do_sampleF)r3rr.r‚r7r‹rˆÚshow_progress_baré)Útimeoutièr5gp×@Úinf)Úttfb_msÚgen_sÚaudio_sÚrtfr)Ú startswithÚitemsrÚ is_tensorrrÚ threadingÚThreadÚstartÚ manual_seedrÉÚmanual_seed_allÚset_ddpm_inference_stepsÚ synchronizerÌrÍr»r.ÚjoinÚfloat)r Ú processorÚ voice_pathrÄÚ ddpm_stepsrrÚkÚvrærÓÚthržrnÚgenÚttfbÚdurràrrÏrÚmeasureÙsJ   þ €      ÿ  ý     röc Cs0d}d}d}t |¡}z tj|tjddd}Wntj|tjddd}Y| ¡g}tdƒ| d ¡t|||d d d }td ƒt||||d d d}|  d|f¡td|dd›d|dd›dƒtdƒt |ƒt |ƒt|||d d d }t||||d dd}|  d|f¡td|dd›d|dd›dƒtdƒtj |j jdd|j _t|||dd d }t|||dd d }t||||d dd}|  d|f¡td|dd›d|dd›dƒtd ƒtj |j jdd|j _t|||d!d d }t||||d dd}|  d"|f¡td|dd›d|dd›dƒtd#ƒt||||d$dd}|  d%|f¡td|dd›d|dd›dƒtd&d'›ƒtd(d)›d*d+d,›d*d-d.›d*d/d.›d*d0d.› ƒtd1›ƒ|D]-\}}|ddkrpd2nd3} t|d)›d*|dd4›d5|dd6›d7|d8d9›d:| d,› ƒqctd'›ƒdS);Nzdemo/voices/modi.wavzmicrosoft/VibeVoice-1.5Bu?Speaker 1: मेरे पà¥à¤¯à¤¾à¤°à¥‡ देशवासियों, आज मैं आपके साथ कà¥à¤› बहà¥à¤¤ ज़रूरी बातें करना चाहता हूà¤. हमारा देश à¤à¤• नये दौर में पà¥à¤°à¤µà¥‡à¤¶ कर रहा है.rÉÚflash_attention_2)Ú torch_dtypeÚ device_mapÚattn_implementationÚsdpaz Warming up...rÀzSpeaker 1: test.)rïz2[1] Baseline: original generate, cfg=1.3, 20 stepsrÁ)rïrzBaseline (cfg=1.3, 20 steps)z RTF=ràz.3fzx TTFB=rÝz.0fÚmszG [2] NO-CFG patched: skip negative LM + single-path diffusion, 20 stepsrzNO-CFG patched, 20 stepsz7 [3] NO-CFG + torch.compile(LM, mode=default), 20 stepsÚdefault)ÚmodezSpeaker 1: compile warmup.zSpeaker 1: compile warmup2.zNO-CFG + compile(LM), 20 stepsz- [4] NO-CFG + compile(LM+diffusion), 20 stepszSpeaker 1: diff compile warmup.z#NO-CFG + compile(LM+diff), 20 stepsz( [5] NO-CFG + compile(LM+diff), 10 stepsé zNO-CFG + compile all, 10 stepsÚ zZ==========================================================================================ÚConfigz<45Ú ÚRTFz>7ÚTTFBz>8ÚAudiozStream?zZ------------------------------------------------------------------------------------------ÚYESÚNOz>6.3fzx z>7.0fzms rßz>7.2fzs )rÚfrom_pretrainedrrÚbfloat16ÚevalÚprintrérör^r$r¿Úcompiler Úlanguage_modelr ) rîÚ model_pathrÄrír Úconfigsr9ÚrÚnameÚcanrrrÚmain sl  þþ """"".  >rÚ__main__)rÀrÁ)Ú__doc__rÌrärr¸rÚvibevoice.modular.streamerrÚ'vibevoice.processor.vibevoice_processorrr$r&r¿rörÚ__name__rrrrÚs     20G ÿ