o ꁱiE@sdZddlZddlZddlZddlmZmZmZmZm Z ddl Z ddl Z ddl mZddlmZeeZGdddZGdd d eZd dgZdS) z' Processor class for VibeVoice models. N)ListOptionalUnionDictAny)FeatureExtractionMixin)loggingc@sleZdZdZddedefddZdejd efd d Z ddejd e ed efddZ dejd ejfddZ d S)AudioNormalizerz Audio normalization class for VibeVoice tokenizer. This class provides audio normalization to ensure consistent input levels for the VibeVoice tokenizer while maintaining audio quality. ư> target_dB_FSepscCs||_||_dS)z Initialize the audio normalizer. Args: target_dB_FS (float): Target dB FS level for the audio. Default: -25 eps (float): Small value to avoid division by zero. Default: 1e-6 Nr r )selfr r rU/home/ubuntu/vibevoice-community/vibevoice/processor/vibevoice_tokenizer_processor.py__init__s zAudioNormalizer.__init__audioreturncCs>tt|d}d|jd||j}||}|||fS)z Adjust the audio to the target dB FS level. Args: audio (np.ndarray): Input audio signal Returns: tuple: (normalized_audio, rms, scalar)  )npsqrtmeanr r )rrrmsscalarnormalized_audiorrr tailor_dB_FS&s  zAudioNormalizer.tailor_dB_FSNrcCs<|durtt|}|dkr||j}nd}|||fS)a Avoid clipping by scaling down if necessary. Args: audio (np.ndarray): Input audio signal scalar (float, optional): Explicit scaling factor Returns: tuple: (normalized_audio, scalar) Ng?)rmaxabsr )rrrmax_valrrravoid_clipping5s    zAudioNormalizer.avoid_clippingcCs"||\}}}||\}}|S)z Normalize the audio by adjusting to target dB FS and avoiding clipping. Args: audio (np.ndarray): Input audio signal Returns: np.ndarray: Normalized audio signal )rr")rr_rrr__call__Is zAudioNormalizer.__call__)r r N) __name__ __module__ __qualname____doc__floatrrndarraytuplerrr"r$rrrrr s  r c seZdZdZdgZ    d)deded ed effd d Zd e j de j fddZ d e e j e efde j fddZ   d*d e ee j e ee e j e e ee efdeedeefddZdede j fddZ d+de ee j fdeede j fddZdeeeffddZ  ! "d,d e eje j e e eje j ffd#edeeded$ef d%d&Zd e j dede j fd'd(ZZS)-VibeVoiceTokenizerProcessora} Processor for VibeVoice acoustic tokenizer models. This processor handles audio preprocessing for VibeVoice models, including: - Audio format conversion (stereo to mono) - Optional audio normalization - Streaming support for infinite-length audio Args: sampling_rate (int, optional): Expected sampling rate. Defaults to 24000. normalize_audio (bool, optional): Whether to normalize audio. Defaults to True. target_dB_FS (float, optional): Target dB FS for normalization. Defaults to -25. eps (float, optional): Small value for numerical stability. Defaults to 1e-6. input_features]Tr r sampling_ratenormalize_audior r c sNtjdi|||_||_|jrt||d|_nd|_||||d|_dS)Nr)r0r1r r r)superrr0r1r normalizerfeature_extractor_dict)rr0r1r r kwargs __class__rrrls z$VibeVoiceTokenizerProcessor.__init__rrcCst|jdkr |St|jdkrL|jddkrtj|ddS|jddkr,tj|ddS|jddkr8|dS|jddkrD|dStd|jtd|j)z Convert stereo audio to mono if needed. Args: audio (np.ndarray): Input audio array Returns: np.ndarray: Mono audio array rr)axiszUnexpected audio shape: z%Audio should be 1D or 2D, got shape: )lenshaperrsqueeze ValueErrorrrrrr _ensure_monos   z(VibeVoiceTokenizerProcessor._ensure_monocCsRt|tjstj|tjd}n|tj}||}|jr'|jdur'||}|S)z Process a single audio array. Args: audio: Single audio input Returns: np.ndarray: Processed audio dtypeN) isinstancerr+arrayfloat32astyper?r1r3r>rrr_process_single_audios    z1VibeVoiceTokenizerProcessor._process_single_audioNreturn_tensorsc  s|durtd|dur|jkrtd|djdt|tr+|}d}n1t|trZt|dkr:tdt d d |DrOfd d |D}d }n t|dt j tf}nd}|rhfdd |D}n |g}|dkrt|dkrt |ddd}nBt dd |Dd}n4|dkrt|dkr|dt jt jddf}nt |ddt jddf}n t|dkr|dn|}d|i}|S)a Process audio for VibeVoice models. Args: audio: Audio input(s) to process. Can be: - str: Path to audio file - np.ndarray: Audio array - List[float]: Audio as list of floats - List[np.ndarray]: Batch of audio arrays - List[str]: Batch of audio file paths sampling_rate (int, optional): Sampling rate of the input audio return_tensors (str, optional): Return format ('pt' for PyTorch, 'np' for NumPy) Returns: dict: Processed audio inputs with keys: - input_features: Audio tensor(s) ready for the model NzAudio input is requiredzInput sampling rate (z') differs from expected sampling rate (z). Please resample your audio.FrzEmpty audio list providedcss|]}t|tVqdSr%)rBstr).0itemrrr sz7VibeVoiceTokenizerProcessor.__call__..cg|]}|qSr)_load_audio_from_path)rIpathrrr z8VibeVoiceTokenizerProcessor.__call__..TcrLr)rFrIarOrrrPrQptr8cSsg|]}t|qSr)torch from_numpyrRrrrrPrQrr)r=r0loggerwarningrBrHrMlistr:allrr+rFrUrV unsqueezestacknewaxis) rrr0rGr5 is_batchedprocessed_audior.outputsrrOrr$sD        z$VibeVoiceTokenizerProcessor.__call__ audio_pathcCstj|d}|dvrddl}|j||jdd\}}|S|dkrBtj|dd }t |tj r7| }nt |}|t jS|d krQt |}|t jStd |d ) z Load audio from file path. Args: audio_path (str): Path to audio file Returns: np.ndarray: Loaded audio array r8).wavz.mp3z.flacz.m4az.oggrNT)srmonoz.ptcpu) map_locationz.npyzUnsupported file format: zC. Supported formats: .wav, .mp3, .flac, .m4a, .ogg, .pt, .npy, .npz)osrNsplitextlowerlibrosaloadr0rUr<rBTensornumpyrrCrErDr=)rrafile_extrj audio_arrayrc audio_tensorrrrrMs*         z1VibeVoiceTokenizerProcessor._load_audio_from_pathaudio_path_or_array normalizecCsZt|tr ||}ntj|tjd}|j}|dur||_z ||}W||_|S||_w)a Convenience method to preprocess audio from file path or array. This method is kept for backward compatibility but __call__ is recommended. Args: audio_path_or_array: Path to audio file or numpy array normalize: Whether to normalize (overrides default setting) Returns: np.ndarray: Preprocessed audio array r@N)rBrHrMrrCrDr1rF)rrqrrrooriginal_normalize processedrrrpreprocess_audio7s   z,VibeVoiceTokenizerProcessor.preprocess_audiocCs|jS)zb Convert the object to a dict containing all attributes needed for serialization. )r4rOrrrto_dictZsz#VibeVoiceTokenizerProcessor.to_dict output.wavFaudio_ output_path batch_prefixcCs"|dur|j}zddl}Wn tytdwt|tjr*| }n+t|t j r3|}n"t|t rLt dd|DrIdd|D}n |}n tdt|g}t|t r|} tj| d d t|D]"\} } || |} tj| || d } || | ||| qi|St|jd kr|jd} | d kr|} tj| d d t| D]7} || }t|jd kr|jdd kr|d}|||}tj| || d } || |||| q|S|} || |} ||| ||||S|||} ||| ||||S)a; Save audio data to WAV file(s). Args: audio: Audio data to save. Can be: - torch.Tensor: PyTorch tensor with shape (B, C, T) or (B, T) or (T) - np.ndarray: NumPy array with shape (B, C, T) or (B, T) or (T) - List of tensors or arrays output_path: Path where to save the audio. If saving multiple files, this is treated as a directory and individual files will be saved inside. sampling_rate: Sampling rate for the saved audio. Defaults to the processor's rate. normalize: Whether to normalize audio before saving. batch_prefix: Prefix for batch files when saving multiple audios. Returns: List[str]: Paths to the saved audio files. NrzQsoundfile is required to save audio files. Install it with: pip install soundfilecss|] }t|tjVqdSr%)rBrUrlrRrrrrKsz9VibeVoiceTokenizerProcessor.save_audio..cSs g|] }|qSr)r*detachrermrRrrrrPs z:VibeVoiceTokenizerProcessor.save_audio..zUnsupported audio type: T)exist_okrbr8)r0 soundfile ImportErrorrBrUrlr*r{rermrr+rYrZr=typergmakedirs enumerate_prepare_audio_for_saverNjoinwriteappendr:r;ranger<)rrryr0rrrzsfaudio_np saved_paths output_diri audio_item file_path batch_size single_audiorrr save_audio`sh        '          z&VibeVoiceTokenizerProcessor.save_audiocCsLt|jdkr|jddkr|d}|r$t|}|dkr$||}|S)a4 Prepare audio for saving by ensuring it's the right shape and optionally normalizing. Args: audio: Audio data as numpy array normalize: Whether to normalize audio Returns: np.ndarray: Processed audio ready for saving r8r)r:r;r<rr r)rrrrr!rrrrs z3VibeVoiceTokenizerProcessor._prepare_audio_for_save)r/Tr r )NNNr%)rwNFrx)r&r'r(r)model_input_namesintboolr*rrr+r?rrrFrHrr$rMrurrrvrUrlrr __classcell__rrr6rr-[sl"* S+  #   "kr-)r)rgjsonwarningstypingrrrrrrmrrU%transformers.feature_extraction_utilsrtransformers.utilsr get_loggerr&rWr r-__all__rrrrs    H