o ꁱi"@sdZddlmZddlmZmZmZmZmZm Z ddl m Z ddl Z ddl Z ddl mZddlmmZddlmZddlmZmZddlmZddlmZmZmZdd lmZdd l m!Z!dd l"m#Z#dd l$m%Z%dd l&m'Z'ddl(m)Z)ddl*m+Z+ddl,m-Z-e'.e/Z0e1e!dre!j2durgde!_2Gdddej3Z4Gdddej3Z5Gddde#Z6Gddde6Z7e8e-e7gdZ9dS)aU VibeVoice Streaming Model Architecture (0.5B) This module implements the streaming-optimized version of VibeVoice for real-time TTS. Key differences from the multi-speaker model: - No semantic tokenizer (only acoustic) - Split language model architecture: lower layers for text, upper layers for TTS - Optimized for low-latency generation ) dataclass)DictListOptionalTupleUnionCallable)tqdmN) AutoModelAutoModelForCausalLM)ACT2FN)CausalLMOutputBaseModelOutputWithPast ModelOutput) LlamaRMSNorm)modeling_utils)PreTrainedModel)FlashAttentionKwargs)logging)VibeVoiceDiffusionHead)DPMSolverMultistepScheduler)VibeVoiceStreamingConfigALL_PARALLEL_STYLES)tpnonecolwiserowwisec(eZdZdZfddZddZZS)BinaryClassifierz?Binary classifier for end-of-speech detection in streaming TTS.cs.tt|t|||_t|d|_dS)Nr)superr__init__nnLinearfc1fc2)self hidden_size __class__R/home/ubuntu/vibevoice-community/vibevoice/modular/modeling_vibevoice_streaming.pyr!-szBinaryClassifier.__init__cCst||}||}|SN)torchrelur$r%)r&xr*r*r+forward2s zBinaryClassifier.forward__name__ __module__ __qualname____doc__r!r0 __classcell__r*r*r(r+r*s rcr)SpeechConnectorzQConnector module that projects speech latents to language model hidden dimension.cs8tt|||_t|dd|_t|||_dS)Ngư>)eps)r r!r"r#r$rnormr%)r& input_dim output_dimr(r*r+r!;s zSpeechConnector.__init__cKs"||}||}||}|Sr,)r$r9r%)r&featureskwargsr/r*r*r+r0As   zSpeechConnector.forwardr1r*r*r(r+r78s r7c@s@eZdZdZeZdZdZdZdZ dZ dZ dZ dZ dZddZdS)!VibeVoiceStreamingPreTrainedModelz*Base class for VibeVoice Streaming models.modelTpast_key_valuescCst|tr |dSt|jdrt|jjdr|jjj}nt|jdr1t|jjdr1|jjj}nd}t|tj rQ|j j j d|d|j durO|j j dSdSt|tjrf|j j d|j j dSdS)Nlanguage_model_configinitializer_rangedecoder_configg{Gz?g)meanstdg?) isinstancerinitialize_weightshasattrconfigrArBrCr"r#weightdatanormal_biaszero_ LayerNormfill_)r&modulerEr*r*r+ _init_weightsVs"      z/VibeVoiceStreamingPreTrainedModel._init_weightsN)r2r3r4r5r config_classbase_model_prefixsupports_gradient_checkpointing_skip_keys_device_placement_supports_cache_class_supports_flash_attn_2_supports_sdpa_supports_quantized_cache_supports_static_cache_supports_attention_backendrRr*r*r*r+r>Hs r>csBeZdZdZfddZddZddZd d d Zd d ZZ S)VibeVoiceStreamingModela8 VibeVoice Streaming Model for real-time TTS. The model uses a split architecture: - language_model: Lower transformer layers for text encoding - tts_language_model: Upper transformer layers for TTS generation This separation enables streaming text input and low-latency speech output. cs@t|t|dr!|jdur!t|jtrtt|j}n|j}ntj}t |j }t|dd|j }||_ t||_t|j_t |}|j |_ t||_tjd|j jd|_t|j||_t|j|j||_|dtt d|dtt dt|j!||_"t#|j!j$|j!j%|j!j&d |_'dS) N torch_dtypenum_hidden_layers)num_embeddings embedding_dimspeech_scaling_factornanspeech_bias_factor)num_train_timesteps beta_scheduleprediction_type)(r r!rHr^rFstrgetattrr-float32copydeepcopyrCtts_backbone_num_hidden_layersr_r from_configlanguage_modelr"Identityr9tts_language_model Embeddingr'tts_input_typesacoustic_tokenizer_configtoacoustic_tokenizerr7acoustic_vae_dimacoustic_connectorregister_buffertensorfloatdiffusion_head_configprediction_headrddpm_num_stepsddpm_beta_schedulerinoise_scheduler)r&rIdtype lm_configlm_backbone_num_hidden_layers tts_lm_configr(r*r+r!ws2        z VibeVoiceStreamingModel.__init__cCsLt|jdr |jjS|jjD]\}}|jdkr!t|j|SqJd)N embed_tokenszembed_tokens.weightFzshould not arrive here)rHrqrfullmapitems orig_namerk)r&nameattrr*r*r+get_input_embeddingss  z,VibeVoiceStreamingModel.get_input_embeddingscCs ||j_dSr,)rqr)r&valuer*r*r+set_input_embeddingss z,VibeVoiceStreamingModel.set_input_embeddingsNcCs"||_|jdur|jdSdS)z@Set the speech tokenizers used for encoding and decoding speech.N)rxeval)r&rxr*r*r+set_speech_tokenizerss z-VibeVoiceStreamingModel.set_speech_tokenizerscOstd)aq Intentionally not implemented. This streaming model is split into two explicit submodules: - `language_model` for plain text processing (lower layers). - `tts_language_model` for TTS-related upper layers. We deliberately avoid a unified `forward` to prevent accidental calls that mix responsibilities. To use the model: - Call `self.language_model(...)` for text embeddings / hidden states. - Call `self.tts_language_model(...)` for the TTS portion. - Use the dedicated inference class for combined generation logic. zVibeVoiceStreamingModel.forward is intentionally disabled. Use `model.language_model(...)` or `model.tts_language_model(...)` instead.) RuntimeError)r&argsr=r*r*r+r0szVibeVoiceStreamingModel.forwardr,) r2r3r4r5r!rrrr0r6r*r*r(r+r]ls / r])r>r]rr7):r5 dataclassesrtypingrrrrrrr rmr-torch.nnr"torch.nn.functional functionalFtorch.distributed distributeddisttransformers.models.autor r transformers.activationsr transformers.modeling_outputsr rr(transformers.models.llama.modeling_llamar transformersrtransformers.modeling_utilsr+transformers.modeling_flash_attention_utilsrtransformers.utilsr modular_vibevoice_diffusion_headrvibevoice.schedule.dpm_solverr!configuration_vibevoice_streamingr get_loggerr2loggerrHrModulerr7r>r]register__all__r*r*r*r+s:               $ e