o ꁱiH@sdZddlZddlZddlmZmZmZmZmZm Z ddl Z ddl Z ddl Z ddlZddlmZmZmZmZmZddlmZmZddlmZeeZGdd d Zd gZdS) z VibeVoice Streaming Processor This processor handles input preparation for the streaming 0.5B model, including text tokenization and cached voice prompt handling. N)ListOptionalUnionDictAnyTuple) BatchEncodingPaddingStrategyPreTokenizedInput TextInputTruncationStrategy) TensorTypelogging)AudioNormalizerc@seZdZdZd0ddZeddZd eee j ffd d Z d e fd dZ       d1deedeeeefdeeeefdeeeefdeedeeeefded e fddZ     d2deeeefdeeeefdeeeefdeedeeeefded e fddZ   d3deejdeeeefdeeeejfdeejd eeeff dd Zd!d"Zd#d$Z e!d%d&Z" '   (d4d)eej#ejeeej#ejffd*ed+eed,ed-ed ef d.d/Z$dS)5VibeVoiceStreamingProcessora" Constructs a VibeVoice Streaming processor which wraps a VibeVoice tokenizer and audio processor into a single processor. The streaming processor is designed for the 0.5B real-time model that uses pre-computed voice embeddings (.pt files) instead of live audio conditioning. Args: tokenizer (`VibeVoiceTextTokenizer` or `VibeVoiceTextTokenizerFast`): The tokenizer for text processing. audio_processor (`VibeVoiceTokenizerProcessor`): The audio processor for speech processing. speech_tok_compress_ratio (`int`, *optional*, defaults to 3200): The compression ratio for speech tokenization. db_normalize (`bool`, *optional*, defaults to True): Whether to apply decibel normalization to audio inputs. N TcKs2||_||_||_||_|rt|_dSd|_dS)N) tokenizeraudio_processorspeech_tok_compress_ratio db_normalizeraudio_normalizer)selfrrrrkwargsrU/home/ubuntu/vibevoice-community/vibevoice/processor/vibevoice_streaming_processor.py__init__*s z$VibeVoiceStreamingProcessor.__init__c Ksddl}ddl}ddlm}ddlm}ddlm}m}|j |d} d} |j | rFt | d } | | } Wdn1s@wYnMz%||dfi|} t | d } | | } Wdn1sewYWn'ty} ztd |d | td d d d} WYd} ~ nd} ~ ww| dd }| dd }| ddp|dd}td|d|vr|j|fi|}ntd|dd| vr| d}||dd|dd |dd|ddd}n|}|||||d S)!a Instantiate a VibeVoiceStreamingProcessor from a pretrained VibeVoice Streaming processor. Args: pretrained_model_name_or_path (`str` or `os.PathLike`): This can be either: - a string, the *model id* of a pretrained model - a path to a *directory* containing processor config Returns: [`VibeVoiceStreamingProcessor`]: The processor object instantiated from pretrained model. rN) cached_filer)VibeVoiceTokenizerProcessor)VibeVoiceTextTokenizerVibeVoiceTextTokenizerFastpreprocessor_config.jsonrz-Could not load preprocessor_config.json from z: zUsing default configurationrT)rrrrlanguage_model_pretrained_namezQwen/Qwen2.5-0.5BzLoading tokenizer from qwenzUnsupported tokenizer type for z. Supported types: Qwen.r sampling_rate]normalize_audio target_dB_FSepsư>)r%r'r(r*)rrrr)osjsontransformers.utilsrvibevoice_tokenizer_processorr2vibevoice.modular.modular_vibevoice_text_tokenizerrr pathjoinexistsopenload Exceptionloggerwarninggetpopinfolowerfrom_pretrained ValueError)clspretrained_model_name_or_pathrr,r-rrrr config_pathconfigf config_fileerrr#r audio_configrrrrr=1sr              z+VibeVoiceStreamingProcessor.from_pretrainedsave_directoryc Ksddl}ddl}|j|ddd|j|jdt|jddt|jd dt|jd d t|jd d dd}|j|d}t |d}|j ||ddWdn1sQwYt d|dS)a  Save a processor to a directory, so that it can be re-loaded using the [`~VibeVoiceStreamingProcessor.from_pretrained`] class method. Args: save_directory (`str` or `os.PathLike`): Directory where the processor will be saved. rNT)exist_okrrr%r&r'r(r)r*r+)feature_extractor_typer%r'r(r*)processor_classrrrr!w)indentz!Processor configuration saved in ) r,r-makedirsrrgetattrrr1r2r4dumpr7r;)rrGrr,r-processor_configrArCrrrsave_pretraineds$      z+VibeVoiceStreamingProcessor.save_pretrainedreturncCstd)z Note: This method is intentionally not implemented in the streaming processor. Use `process_input_with_cached_prompt` for streaming use cases. zsVibeVoiceStreamingProcessor.__call__ is not implemented. Use process_input_with_cached_prompt for streaming inputs.)NotImplementedErrorrrrr__call__sz$VibeVoiceStreamingProcessor.__call__Ftext cached_promptpadding truncation max_lengthreturn_tensorsreturn_attention_maskcKs|g} |g} d} g} t| | D]B\} }|jj| ddd}|ddd}|ddd}|jjg|}|jjg|}dg|}|||d|d }| |q|j| |||||d }|S) a Main method to process one text script based on cached prompt. The streaming model uses pre-computed voice embeddings (cached prompts) loaded from .pt files instead of processing audio on-the-fly. Args: text (`str`): The input text to process. cached_prompt (`Dict[str, Any]`, *optional*): The cached prompt to use for processing. It contains the kv cache of the voice prompt. Load this from a .pt file using torch.load(). padding (`bool`, `str` or `PaddingStrategy`, defaults to `True`): Whether to pad sequences to the same length truncation (`bool`, `str` or `TruncationStrategy`, defaults to `False`): Whether to truncate sequences max_length (`int`, *optional*): Maximum length of the returned sequences return_tensors (`str` or `TensorType`, *optional*): If set, will return tensors of a particular framework return_attention_mask (`bool`, defaults to `True`): Whether to return the attention mask Returns: `BatchEncoding`: A BatchEncoding with the following fields: - **input_ids** -- List of token id sequences or tensor - **attention_mask** -- List of attention masks or tensor - **tts_lm_input_ids** -- List of token id sequences or tensor used for TTS LM - **tts_lm_attention_mask** -- List of attention masks or tensor used for TTS LM - **tts_text_ids** -- List of token id sequences or tensor for TTS text input - **speech_tensors** -- Padded speech inputs (if voice_samples provided) - **speech_masks** -- Speech masks (if voice_samples provided) - **speech_input_mask** -- Boolean masks indicating speech token positions F )add_special_tokenslmlast_hidden_statertts_lmN) input_idstts_lm_input_ids tts_text_ids speech_inputsspeech_input_mask)rYrZr[r\r])ziprencodestripsizepad_idappend _batch_encode)rrWrXrYrZr[r\r]rtextscached_prompts is_batched all_encodings text_inputcached_prompt_input script_tokensinput_id_lengthtts_lm_input_id_lengthrcrdrgencodingbatch_encodingrrr process_input_with_cached_prompts6.   zz=VibeVoiceStreamingProcessor._batch_encode..cSr|)rdrr}rrrrrcSr|)rerr}rrrrrcSr|)rgrr}rrrrrcSg|] }dgt|qSrlenr~idsrrrrNcSrrrrrrrrrFrfTdtypercrdreattention_masktts_lm_attention_maskrg)r\padded_speechesspeech_tensors speech_masks)extendrtorchtensorlongboolprepare_speech_inputs)rr{rYrZr[r\r]input_ids_listtts_lm_input_ids_listtts_text_ids_listspeech_input_masks_listattention_maskstts_lm_attention_masksall_speech_inputs has_speechrry speech_dictrrrrnsP      z)VibeVoiceStreamingProcessor._batch_encoderfdevicerc s$|sdddSfdd|D}tdd|D}|djdkr.tjt||fdtjd }ntjt|||djd fdtjd }tjt|t|ftjd }t t ||D]\} \} } | || dt| f<d || d| f<qV||d} |d krt j |||pt jd| d<t j ||t j d| d<| S)aT Prepare speech inputs for model consumption. Args: speech_inputs: List of speech arrays return_tensors: Output tensor type device: Device to place tensors on dtype: Data type for tensors Returns: Dictionary with padded_speeches and speech_masks N)rrcs"g|] }t|jdjqS)r)mathceilshaperr~srUrrr^s"zEVibeVoiceStreamingProcessor.prepare_speech_inputs..css|]}|jdVqdS)rN)rrrrr _szDVibeVoiceStreamingProcessor.prepare_speech_inputs..rr) fill_valuerrTpt)rrrr)maxndimnpfullrfloat32rzerosbool_ enumeraterhrrr) rrfr\rrvae_tok_seqlensmax_speech_lengthrrispeechvae_tok_lengthresultrrUrrGs$ &z1VibeVoiceStreamingProcessor.prepare_speech_inputscO|jj|i|S)z This method forwards all its arguments to VibeVoiceTextTokenizer's [`~PreTrainedTokenizer.batch_decode`]. Please refer to the docstring of this method for more information. )r batch_decoderargsrrrrrxz(VibeVoiceStreamingProcessor.batch_decodecOr)z This method forwards all its arguments to VibeVoiceTextTokenizer's [`~PreTrainedTokenizer.decode`]. Please refer to the docstring of this method for more information. )rdecoderrrrrrz"VibeVoiceStreamingProcessor.decodecCs*|jj}|jj}tt||ddgS)zB Return the list of inputs accepted by the model. rfrg)rmodel_input_namesrlistdictfromkeys)rtokenizer_input_namesaudio_processor_input_namesrrrrsz-VibeVoiceStreamingProcessor.model_input_names output.wavaudio_audio output_pathr% normalize batch_prefixcCs|jj|||||dS)a Save audio data to a file. Args: audio (Union[torch.Tensor, np.ndarray, List[Union[torch.Tensor, np.ndarray]]]): The audio data to save. Can be a single tensor/array or a list of them. output_path (str, optional): Path to save the audio file. Defaults to "output.wav". sampling_rate (int, optional): Sampling rate for the audio. If None, uses the processor's default. normalize (bool, optional): Whether to normalize the audio before saving. Defaults to False. batch_prefix (str, optional): Prefix for batch audio files. Defaults to "audio_". Returns: str: The path to the saved audio file. )rr%rr)r save_audio)rrrr%rrrrrrsz&VibeVoiceStreamingProcessor.save_audio)NNrT)NNTFNNT)TFNNT)NNN)rNFr)%__name__ __module__ __qualname____doc__r classmethodr=rstrr,PathLikerRrrVrrrrr r intr rzrrnrndarrayrrrrrrpropertyrTensorrrrrrrs  U"    V   C  1   r)rrwarningstypingrrrrrrr,renumpyrr$transformers.tokenization_utils_baserr r r r r.r rr/r get_loggerrr7r__all__rrrrs$