o ꁱiȋ @sdZddlmZddlmZmZmZmZmZm Z m Z ddl m Z ddl Z ddl mZddlmZmZddlmZmZmZmZmZddlmZmZdd lmZdd lmZdd lm Z dd l!m"Z"d dl#m$Z$d dl%m&Z&ddl'm(Z(d dl)m*Z*d dl+m,Z,m-Z-d dl.m/Z/m0Z0m1Z1d dl2m3Z3m4Z4e"5e6Z7e8edrej9durgde_9dZ:dZ; d&dedeeeGdd d eZ?eGd!d"d"eZ@Gd#d$d$e/eZAeBe*eAgd%ZCdS)'a6 VibeVoice Streaming Inference Model (0.5B) This module implements the inference engine for real-time streaming TTS. Key features: - Window-based text/speech interleaving for streaming - Binary EOS classifier for end-of-speech detection - Classifier-free guidance for speech quality - Audio streaming support ) dataclass)AnyDictListOptionalTupleUnionCallable)tqdmN) AutoModelAutoModelForCausalLM)GenerationMixinGenerationConfigLogitsProcessorLogitsProcessorListStoppingCriteriaList)BaseModelOutputWithPast ModelOutput)modeling_utils)PreTrainedModel)FlashAttentionKwargs)logging) VibeVoiceTokenizerStreamingCache)VibeVoiceDiffusionHead)DPMSolverMultistepScheduler)VibeVoiceStreamingConfig)VibeVoiceTextTokenizerVibeVoiceTextTokenizerFast)!VibeVoiceStreamingPreTrainedModelVibeVoiceStreamingModelBinaryClassifier) AudioStreamerAsyncAudioStreamerALL_PARALLEL_STYLES)tpnonecolwiserowwiseoutputs model_kwargsnum_new_tokensreturncCsxt|d|d<|d}tj|||jd|fgdd|d<t|ddd|dd|d|dj|d<|S)a. Update model_kwargs after adding new tokens. Mainly for the case num_new_tokens > 1 (e.g. a whole text window): - past_key_values: take from current outputs - attention_mask: append num_new_tokens ones - cache_position: advance by creating a range for all new positions past_key_valuesattention_maskrdimcache_positionr)getattrtorchcatnew_onesshapearangetodevice)r+r,r-r0r=\/home/ubuntu/vibevoice-community/vibevoice/modular/modeling_vibevoice_streaming_inference.py#_update_model_kwargs_for_generation-s 8r?c@s eZdZUdZeejed<dS)VibeVoiceCausalLMOutputWithPastNlogits)__name__ __module__ __qualname__rArr6 FloatTensor__annotations__r=r=r=r>r@Hs r@c@sHeZdZUdZdZejed<dZe e ej ed<dZ e ej ed<dS)VibeVoiceGenerationOutputaC Output type for VibeVoice generation. Args: sequences (`torch.LongTensor` of shape `(batch_size, sequence_length)`): The generated sequences. speech_outputs (`List[torch.FloatTensor]`, *optional*): List of generated speech waveforms or latents for each speech segment. N sequencesspeech_outputsreach_max_step_sample)rBrCrD__doc__rHr6 LongTensorrFrIrrrErJ BoolTensorr=r=r=r>rGMs  rGc(seZdZdZfddZeddZeddZedd Zed d Z ed d Z eddZ ddZ ddZ ddZddZddZdOddZdOddZ           dPdejd eejd!eejd"eeeejd#eejd$eejd%eed&eed'eed(eed)eejd*eeeffd+d,Z             dQdejd eejd!eejd"eeeejd#eejd$eejd%eed&eed'eed(eed)eejd-eejd.eejd*eeeffd/d0Zd1d2Z dRd4d5Z!e"               6 7 dSd8eejd9ee#d:ee$d;ee%ded?d@eee)e*fdAeejdBeejdCeejdDeejdEeejdFeejdGedHe+dIee&gefd*eeje,ff$dJdKZ-e"dTdMdNZ.Z/S)U3VibeVoiceStreamingForConditionalGenerationInferenceaG VibeVoice Streaming model for conditional generation inference. This class handles the full streaming TTS pipeline: 1. Prefill text prompt with cached voice embeddings 2. Window-based text/speech interleaving 3. Diffusion-based speech token generation 4. Binary EOS detection for natural stopping cs:t|t||_t|jj|_|jj |_ | dSN) super__init__r modelr!decoder_config hidden_sizetts_eos_classifierdiffusion_head_configddpm_num_inference_stepsddpm_inference_steps post_init)selfconfig __class__r=r>rQhs    zr_wzCVibeVoiceStreamingForConditionalGenerationInference.noise_schedulercCr^rO)rRprediction_headr`r=r=r>rb{razCVibeVoiceStreamingForConditionalGenerationInference.prediction_headcCr^rO)rRspeech_scaling_factorr`r=r=r>rcrazIVibeVoiceStreamingForConditionalGenerationInference.speech_scaling_factorcCr^rO)rRspeech_bias_factorr`r=r=r>rdrazFVibeVoiceStreamingForConditionalGenerationInference.speech_bias_factorcCr^rO)rRacoustic_tokenizerr`r=r=r>rerazFVibeVoiceStreamingForConditionalGenerationInference.acoustic_tokenizercCr^rO)rRacoustic_connectorr`r=r=r>rfrazFVibeVoiceStreamingForConditionalGenerationInference.acoustic_connectorcCsFt|jdds dSt|drt|jjdr!|jjjj|j_dSdSdS)zY Tie the weights between the input embeddings and the output embeddings. tie_word_embeddingsFNlm_head embed_tokens)r5r[hasattrrRlanguage_modelriweightrhr`r=r=r> tie_weightss z?VibeVoiceStreamingForConditionalGenerationInference.tie_weightscCs |jSrO)rRget_input_embeddingsr`r=r=r>rns zHVibeVoiceStreamingForConditionalGenerationInference.get_input_embeddingscCs|j|dSrO)rRset_input_embeddings)rZvaluer=r=r>roszHVibeVoiceStreamingForConditionalGenerationInference.set_input_embeddingscCsdS)zR This model does not define an `lm_head` (vocabulary projection). Nr=r`r=r=r>get_output_embeddingsszIVibeVoiceStreamingForConditionalGenerationInference.get_output_embeddingscCtd)z No-op because there is no `lm_head`. Provided only to satisfy optional API calls. To enable, first create `self.lm_head` then allow assignment. zvOutput embeddings (lm_head) are not defined for this model. Create one before calling set_output_embeddings if needed. RuntimeError)rZnew_embeddingsr=r=r>set_output_embeddingsszIVibeVoiceStreamingForConditionalGenerationInference.set_output_embeddingsNcCs|j|dS)z@Set the speech tokenizers used for encoding and decoding speech.N)rRset_speech_tokenizers)rZrer=r=r>rwszIVibeVoiceStreamingForConditionalGenerationInference.set_speech_tokenizerscCs|p|jjj|_dSrO)r[rVrWrX)rZ num_stepsr=r=r>set_ddpm_inference_stepsszLVibeVoiceStreamingForConditionalGenerationInference.set_ddpm_inference_steps input_idsr0 position_idsr/ inputs_embedslabels use_cacheoutput_attentionsoutput_hidden_states return_dictr4r.c  Ks| dur| n|jj} |dur|j|}|jjd||||||| | | d | } | s.| dn| j}|dur9tdt| j|| j dS)aN Single pass of the base text LM. - Builds embeddings if `inputs_embeds` not provided. - Uses (and returns) `past_key_values` when `use_cache=True`. - No loss / no lm_head / no speech logic. Args: input_ids: (B, S) token ids. attention_mask: (B, S) mask. past_key_values: cache from previous steps. cache_position: positions for cached tokens. labels: unsupported (will raise). Returns: BaseModelOutputWithPast with `last_hidden_state` and `past_key_values`. N r|r0r{r/r~rrrr4r4Loss computation is not implemented in this version.)r/last_hidden_state attentionsr=) r[use_return_dictrRrnrkrNotImplementedErrorrr/r)rZrzr0r{r/r|r}r~rrrr4kwargsr+ hidden_statesr=r=r> forward_lms0   z>VibeVoiceStreamingForConditionalGenerationInference.forward_lmlm_last_hidden_statetts_text_masksc Ks| dur| n|jj} |dur|j|}|jd| jd}| |dd|dddf<||j| }|jjd||||||| | | d |}| sO|dn|j}| |dddddf}|durht dt ||j ||j dS) aV Single pass of the TTS LM. - Overwrites tail embeddings with `lm_last_hidden_state`. - Adds type embedding via `tts_text_masks` (1=text, 0=speech). - Predicts EOS from last hidden state (binary classifier). - No loss / no full acoustic decoding here. Args: input_ids: (B, S) token ids. attention_mask: (B, S) mask. lm_last_hidden_state: (B, K, H) hidden states to splice into the tail. tts_text_masks: (B, 1) mask marking current position as text(1)/speech(0). past_key_values: cache from previous TTS steps. cache_position: positions for cached tokens. labels: unsupported (will raise). Returns: VibeVoiceCausalLMOutputWithPast with `logits` (EOS), `last_hidden_state`, `past_key_values`. Nrrrr1r)rAr/rrr=)r[rrRrnr9tts_input_typeslongtts_language_modelrrUrr@r/r)rZrzr0r{r/r|r}r~rrrr4rrr start_idxr+rrAr=r=r>forward_tts_lms:%  zBVibeVoiceStreamingForConditionalGenerationInference.forward_tts_lmcOrr)a Unified forward is intentionally disabled. Reasons: 1. The inference pipeline is staged: base text LM, then TTS LM, plus streaming & diffusion handled in `generate`. 2. A monolithic call would hide required sequencing (prefill, window stepping, speech diffusion sampling). Use instead: - self.forward_lm(...) for a base text LM step (prefill or incremental). - self.forward_tts_lm(...) for a single TTS LM step (needs LM hidden states). - self.generate(...) for full streaming (text + speech + diffusion + audio assembly). Raises: RuntimeError: Always (by design). zWUnified forward is disabled. Use `forward_lm`, `forward_tts_lm`, or `generate` instead.rs)rZargsrr=r=r>forward@sz;VibeVoiceStreamingForConditionalGenerationInference.forwardFc Ks|durt|j|j|jd}ntdi||j|j|jd}|j|df|j|j|jd|\}}|j|_|j|_|j|_|||j|\}}}|j d} |j } |j |d| dd|_ |j |d<| |j } | j d} |dduow|jdu} |d duo|jdu}|j|| |||| d }|jd}|||d| || tj| | tjd |d <|D]\}}t|tjr|j | d||<q|r|j|| |dt|j |d }|j|td}||| ||fS||| fS)N) bos_token_id eos_token_id pad_token_idT)speech_start_id speech_end_idspeech_diffusion_idrr<r~r max_length min_length)generation_confighas_default_max_lengthhas_default_min_lengthmodel_input_name inputs_tensorinput_ids_length)r<dtyper4)rinput_ids_seq_lengthencoder_input_idsprefix_allowed_tokens_fnlogits_processorr<r,)rstopping_criteriar=)rrrr_prepare_generation_configrrr_prepare_model_inputsr9r<_prepare_special_tokensr~r;getrr_prepare_generated_length_prepare_cache_for_generationr6r:ritems isinstanceTensor_get_logits_processorr_get_stopping_criteriar)rZrinputs tokenizerreturn_processorsrr,rr batch_sizer<rzrrrmax_cache_lengthkvrrr=r=r>#_build_generate_config_model_kwargsTs          zWVibeVoiceStreamingForConditionalGenerationInference._build_generate_config_model_kwargsT?rrrrr synced_gpusassistant_modelraudio_streamernegative_prompt_idsnegative_prompt_attention_maskspeech_tensors speech_masksspeech_input_mask tts_text_ids return_speech cfg_scale stop_check_fncM Ks |dd}|d}|dd}|dd}|dd}||j}|dddur7|jjj|jd|d<|j |||fd d i|\}}}}}t j |d jd d f|t j |d jdt j |d jd d ft j |d jd|ddd}|j dd|fd di|\}}}|||ddd}|j dd|fd di|\}} }t j |d jd d f|t j |d jdt j |d jd d ft j |d jd|ddd}!|j dd|fd di|!\}"}#}$t}%|jd }&|&d ksJd|j}'t j|&t j|'d}(|dd})ddt|&D}*d }+t j|&t j|'d},|jd tkrtn|jd }-|d}.|d}/|d}0|d}1t|.||-d}t|/| |-d} |j|0|dd}|j|1|#dd}#|jd }2d }3d }4|dd rut|jd|2d|2d|jd |2dd!}5nd}5 |dur|r|)rtd"|2d |dur|n|(rt|5d#r|5d$n|dd|+t|+d tf}6|dd|+d t|+d%tfjd }7|+d 7}+|6jd d krt j||6gdd&}t j||6gdd&}|jd |jkr|)rtd'|jd(t j|&|'d)|(}8|8d krd |,|8<n%|2|6jd 7}2|4|6jd 7}4|5durG|5|6jd |5d|4d*|3d+|2d|jd |j|fi|}9|j d2i|9d ddd,}.t|.||7d}|j|fi| }:t !|ddddf|.j"d-};|j#d2i|:|;d ddd,}/|j|/| dd} t $d g}|1j"||?|d.'d }@|@|j(j)|@j|j(j*|@j}A|j(j+j,|A|j(j+j|%|<|j(j+jd dd/}Bt-|<D]\}C}D|D.}E|(|Es|*|E/|B|Cq|dur|0|B|<|j(1|@}Ft j|t !|ddddfgdd&}|jd |jkr9n|2d 7}2|3d 7}3|5dur^|5d |5d|4d*|3d+|2d|jd |j|fi| }:t 2|ddddf|Fd-};|j#d2i|:|;d ddd,}/|=t%d kr|7d krt|/| |7d} n|j|/| dd} t j|$t !|ddddfgdd&}$|j|$fi|#}Gt 2|$ddddf|Fd-}H|j#d2i|G|Hd ddd,}1|j|1|#dd}#t 3|4|/j"|tts_lm_input_idstts_lm_attention_maskall_prefilled_outputsmax_new_tokensr1rTrzrr)rr<d)rzr0rFz'Currently only supports batch size == 1verbosecSsg|]}gqSr=r=).0_r=r=r> szPVibeVoiceStreamingForConditionalGenerationInference.generate..lmtts_lmneg_lm neg_tts_lm)r-)is_encoder_decodershow_progress_barz Prefilled z tokens, current step (z / ))totaldescinitialleavez&Generation stopped externally at step set_descriptionzGeneration completer2z"Reached maximum generation length z , stopped it.rz text tokens, generated z speech tokens, current step ()rrr)rr)r)cachesample_indicesr~debugg?)rHrIrJr=)7popconvert_tokens_to_idsr;r<rr[rSmax_position_embeddingsr9rr6fullronesrzerosboolrangeTTS_TEXT_WINDOW_SIZEr?r rprintendallrjrr7r:numelupdateprepare_inputs_for_generationr ones_likerrrLTTS_SPEECH_WINDOW_SIZEsample_speech_tokens unsqueezerRrcrdredecode enumerateitemappendputrf zeros_likesigmoidrUanyrG)MrZrrrrrrrrrrrrrrrrrrrneg_text_input_idrrrr,rznegative_kwargsnegative_generation_confignegative_model_kwargsnegative_input_ids tts_lm_kwargstts_lm_generation_configtts_lm_model_kwargstts_lm_negative_kwargs!tts_lm_negative_generation_configtts_lm_negative_model_kwargstts_lm_negative_input_idsacoustic_cacherr< finished_tagsr audio_chunkstts_text_window_indexrJfirst_text_window_sizer+tts_lm_outputsnegative_outputstts_lm_negative_outputssteptotal_generated_speech_tokenstotal_prefilled_text_tokens progress_barcur_input_tts_text_idsnext_text_window_sizereached_samples model_inputstts_lm_model_inputstts_lm_additional_inputsdiffusion_indicescur_speech_indexpositive_conditionnegative_condition speech_latent scaled_latent audio_chunki sample_idxidxacoustic_embedtts_lm_negative_model_inputs!tts_lm_negative_additional_inputstts_eos_logitsfinal_audio_outputs sample_chunksconcatenated_audior=r=r>generates 6     &$    &$          * &   $     (  & (     z