hEdZddlZddlZddlZddlmZmZmZmZm Z ddl Z ddl Z ddl mZddlmZejeZGddZGdd eZd dgZdS) z' Processor class for VibeVoice models. N)ListOptionalUnionDictAny)FeatureExtractionMixin)loggingceZdZdZddedefdZdejdefd Z ddejd e edefd Z dejdejfd Z d S)AudioNormalizerz Audio normalization class for VibeVoice tokenizer. This class provides audio normalization to ensure consistent input levels for the VibeVoice tokenizer while maintaining audio quality. ư> target_dB_FSepsc"||_||_dS)z Initialize the audio normalizer. Args: target_dB_FS (float): Target dB FS level for the audio. Default: -25 eps (float): Small value to avoid division by zero. Default: 1e-6 Nrr)selfrrs Y/workspace/chatterbox-finetuning/src/vibevoice/processor/vibevoice_tokenizer_processor.py__init__zAudioNormalizer.__init__s)audioreturnctjtj|dz}d|jdz z||jzz }||z}|||fS)z Adjust the audio to the target dB FS level. Args: audio (np.ndarray): Input audio signal Returns: tuple: (normalized_audio, rms, scalar)  )npsqrtmeanrr)rrrmsscalarnormalized_audios r tailor_dB_FSzAudioNormalizer.tailor_dB_FS&sUgbgeQh''(()B./3>B 6>f,,rNr c|9tjtj|}|dkr ||jz}nd}||z |fS)a Avoid clipping by scaling down if necessary. Args: audio (np.ndarray): Input audio signal scalar (float, optional): Explicit scaling factor Returns: tuple: (normalized_audio, scalar) Ng?)rmaxabsr)rrr max_vals ravoid_clippingzAudioNormalizer.avoid_clipping5sK >fRVE]]++G}} 48+v~v%%rch||\}}}||\}}|S)z Normalize the audio by adjusting to target dB FS and avoiding clipping. Args: audio (np.ndarray): Input audio signal Returns: np.ndarray: Normalized audio signal )r"r')rr_s r__call__zAudioNormalizer.__call__Is9''.. q!&&u--q r)r r N) __name__ __module__ __qualname____doc__floatrrndarraytupler"rr'r*rrr r s  U u     -"* - - - - -&&BJ&&SX&&&&(bjRZrr c reZdZdZdgZ d deded ed effd Zd e j d e j fdZ d e e j e efd e j fdZ d!d e ee j e ee e j e e ee efdeedeefdZded e j fdZ d"de ee j fdeed e j fdZd eeeffdZ d#d e eje j e e eje j ffdedeededef dZd e j ded e j fdZxZS)$VibeVoiceTokenizerProcessora} Processor for VibeVoice acoustic tokenizer models. This processor handles audio preprocessing for VibeVoice models, including: - Audio format conversion (stereo to mono) - Optional audio normalization - Streaming support for infinite-length audio Args: sampling_rate (int, optional): Expected sampling rate. Defaults to 24000. normalize_audio (bool, optional): Whether to normalize audio. Defaults to True. target_dB_FS (float, optional): Target dB FS for normalization. Defaults to -25. eps (float, optional): Small value for numerical stability. Defaults to 1e-6. input_features]Tr r sampling_ratenormalize_audiorrc tjdi|||_||_|jrt |||_nd|_||||d|_dS)Nr)r8r9rrr3)superrr8r9r normalizerfeature_extractor_dict)rr8r9rrkwargs __class__s rrz$VibeVoiceTokenizerProcessor.__init__ls ""6"""*.   #-Q&& ==+++[^q(( ==+++$%M %M%MNNNRU[RRSS Src*t|tjs!tj|tj}n|tj}||}|jr|j||}|S)z Process a single audio array. Args: audio: Single audio input Returns: np.ndarray: Processed audio dtype) isinstancerr1arrayfloat32astyperHr9r<rGs r_process_single_audioz1VibeVoiceTokenizerProcessor._process_single_audios%,, -HU"*555EELL,,E!!%((   +DO$?OOE**E rNreturn_tensorsc |td|1|jkr&td|djdt |t r|}d}nt |trtt|dkrtdtd |Drfd |D}d }n*t |dtj tf}nd}|rfd |D}n |g}|d krt|dkrAtj|ddd}ntjd|Dd}n|dkrgt|dkr(|dtjtjddf}nItj|ddtjddf}nt|dkr|dn|}d|i}|S)a Process audio for VibeVoice models. Args: audio: Audio input(s) to process. Can be: - str: Path to audio file - np.ndarray: Audio array - List[float]: Audio as list of floats - List[np.ndarray]: Batch of audio arrays - List[str]: Batch of audio file paths sampling_rate (int, optional): Sampling rate of the input audio return_tensors (str, optional): Return format ('pt' for PyTorch, 'np' for NumPy) Returns: dict: Processed audio inputs with keys: - input_features: Audio tensor(s) ready for the model NzAudio input is requiredzInput sampling rate (z') differs from expected sampling rate (z). Please resample your audio.FrzEmpty audio list providedc3@K|]}t|tVdSr+)rLstr).0items r z7VibeVoiceTokenizerProcessor.__call__..s,;;T:dC((;;;;;;rc:g|]}|Sr3)_load_audio_from_path)rUpathrs r z8VibeVoiceTokenizerProcessor.__call__..s'LLLd33D99LLLrTc:g|]}|Sr3)rP)rUars rr[z8VibeVoiceTokenizerProcessor.__call__..s'LLLt99!<<LLLrptrAc6g|]}tj|Sr3)torch from_numpyrUr]s rr[z8VibeVoiceTokenizerProcessor.__call__..s#-[-[-[ae.>q.A.A-[-[-[rrr)rFr8loggerwarningrLrTrYlistrCallrr1rPr`ra unsqueezestacknewaxis) rrr8rQr> is_batchedprocessed_audior6outputss ` rr*z$VibeVoiceTokenizerProcessor.__call__s0 =677 7  $$:L)L)L NNU UU"&"4UUU    eS ! ! ..u55EJJ t $ $ 5zzQ !<===;;U;;;;; FLLLLeLLL! (a2:t2DEE J  BLLLLeLLLOO#99%@@AO T ! !?##q((!&!1/!2D!E!E!O!OPQ!R!R!\!\]^!_!_"'-[-[?-[-[-[!\!\!f!fgh!i!i t # #?##q((!0!3BJ AAA4M!N!#/!:!:111bj!!!;K!L363G3G13L3L_Q//RaN ^ r audio_pathc|tj|d}|dvr&ddl}|||jd\}}|S|dkrtj|d }t|tj r| }ntj |}|tjS|d kr3tj|}|tjSt!d |d ) z Load audio from file path. Args: audio_path (str): Path to audio file Returns: np.ndarray: Loaded audio array rA).wavz.mp3z.flacz.m4az.oggrNT)srmonoz.ptcpu) map_locationz.npyzUnsupported file format: zC. Supported formats: .wav, .mp3, .flac, .m4a, .ogg, .pt, .npy, .npz)osrZsplitextlowerlibrosaloadr8r`rErLTensornumpyrrMrOrNrF)rrmfile_extrw audio_arrayrp audio_tensors rrYz1VibeVoiceTokenizerProcessor._load_audio_from_pathsB7##J//288:: @ @ @ NNN%ll%+OK      :juEEEMMOOL, 55 5*0022  h|44 %%bj11 1   '*--K%%bj11 1UHUUU raudio_path_or_array normalizect|tr||}n tj|tj}|j}|||_ ||}||_n #||_wxYw|S)a Convenience method to preprocess audio from file path or array. This method is kept for backward compatibility but __call__ is recommended. Args: audio_path_or_array: Path to audio file or numpy array normalize: Whether to normalize (overrides default setting) Returns: np.ndarray: Preprocessed audio array rJ)rLrTrYrrMrNr9rP)rr~rr|original_normalize processeds rpreprocess_audioz,VibeVoiceTokenizerProcessor.preprocess_audio7s )3 / / J445HIIKK(#6bjIIIK"1  #,D  622;??I$6D #5D 5 5 5 5s A:: Bc|jS)zb Convert the object to a dict containing all attributes needed for serialization. )r=)rs rto_dictz#VibeVoiceTokenizerProcessor.to_dictZs **r output.wavFaudio_ output_path batch_prefixc||j} ddl}n#t$rtdwxYwt|tjrK| }nzt|tj r|}n]t|tr)td|Dr d|D}n"|}ntdt|g}t|tr|} t!j| dt%|D]l\} } || |} t j| || d } || | ||| mnt1|jd krB|jd} | d kr|} t!j| dt5| D]} || }t1|jd kr&|jdd kr|d}|||}t j| || d } || |||| n|} || |} ||| |||nB|||} ||| ||||S) a; Save audio data to WAV file(s). Args: audio: Audio data to save. Can be: - torch.Tensor: PyTorch tensor with shape (B, C, T) or (B, T) or (T) - np.ndarray: NumPy array with shape (B, C, T) or (B, T) or (T) - List of tensors or arrays output_path: Path where to save the audio. If saving multiple files, this is treated as a directory and individual files will be saved inside. sampling_rate: Sampling rate for the saved audio. Defaults to the processor's rate. normalize: Whether to normalize audio before saving. batch_prefix: Prefix for batch files when saving multiple audios. Returns: List[str]: Paths to the saved audio files. NrzQsoundfile is required to save audio files. Install it with: pip install soundfilec3JK|]}t|tjVdSr+)rLr`ryrbs rrWz9VibeVoiceTokenizerProcessor.save_audio..s.>>1:a..>>>>>>rcg|]L}|MSr3)r0detachrrrzrbs rr[z:VibeVoiceTokenizerProcessor.save_audio..sDLLLAGGII,,..2244::<<LLLrzUnsupported audio type: T)exist_okrorA)r8 soundfile ImportErrorrLr`ryr0rrrrzrr1rerfrFtypertmakedirs enumerate_prepare_audio_for_saverZjoinwriteappendrCrDrangerE)rrrr8rrsfaudio_np saved_paths output_diri audio_item file_path batch_size single_audios r save_audioz&VibeVoiceTokenizerProcessor.save_audio`s2   .M  " " " " "   9   eU\ * * G{{}}++--113399;;HH rz * * GHH t $ $ G>>>>>>> !LLeLLL EU EEFF F  h % %1 0$J K T 2 2 2 2"+8!4!4 . . :!99*iPP GLL 5Ma5M5M5MNN J >>>""9----  .8>""a''%^A. >>!,JK T::::#:.. 6 6'/{ |122Q66+1!499/;/C/CA/F/F '+'C'CLR['\'\ $&GLL =Ua=U=U=U$V$V L-HHH#**95555 6"*!1!1!3!3J!%!=!=j)!T!TJHH[*mDDD&&{3333"99(INN j-@@@"";///s*ct|jdkr&|jddkr|d}|r1tj|}|dkr||z }|S)a4 Prepare audio for saving by ensuring it's the right shape and optionally normalizing. Args: audio: Audio data as numpy array normalize: Whether to normalize audio Returns: np.ndarray: Processed audio ready for saving rAr)rCrDrErr%r$)rrrr&s rrz3VibeVoiceTokenizerProcessor._prepare_audio_for_savest u{  a  EKNa$7$7MM!$$E  (fUmm''))G{{ r)r7Tr r )NNNr+)rNFr)r,r-r.r/model_input_namesintboolr0rrr1rHrrrPrTrr*rYrrrrr`ryrr __classcell__)r?s@rr5r5[s  **# $!              6T"*TTTTT85T%[1H+Ibj6fj'+(, QQS"*d5k4 3CT$u+EVX\]`XaabQ }Q! QQQQf&& &&&&V%)  "3 ?3 D>      F+c3h++++('+$ iiU\2:tE%, :R4S/TTUii } i  i  iiiiVRZDRZrr5)r/rtjsonwarningstypingrrrrrrzrr`%transformers.feature_extraction_utilsrtransformers.utilsr get_loggerr,rcr r5__all__r3rrrs 33333333333333 HHHHHH&&&&&&  H % %DDDDDDDDPEEEEE"8EEEP )*; <r