o ꁱi<[@sdZddlZddlZddlZddlZddlmZmZmZm Z m Z m Z ddl Z ddlZddlmZddlmZmZddlmZmZz ddlmZd ZWneyZd Zed YnweeZd ZGd ddZ dgZ!dS)z+ Processor class for VibeVoice ASR models. N)ListOptionalUnionDictAnyTuple) BatchEncoding) TensorTypelogging)VibeVoiceTokenizerProcessorAudioNormalizer)load_audio_use_ffmpegTFzHaudio_utils not available, will fall back to soundfile for audio loadingzYYou are a helpful assistant that transcribes audio input into text output in JSON format.c@seZdZdZ     d*ddZdd Zed d Zd ee e j ffd dZ         d+de ee ejejeee ejejffde ede ee efdede edededede e defddZ    d,dee ejejfde ededede e dee eff ddZ    d-deee efdede edede e def dd Zd!d"Zd#d$Zd%e deee effd&d'Zed(d)Z dS).VibeVoiceASRProcessoraK Processor for VibeVoice ASR (Automatic Speech Recognition) models. This processor handles audio preprocessing and tokenization for ASR tasks, following the exact format used in training with proper chat templates. Args: tokenizer: The text tokenizer for processing text audio_processor: The audio processor for processing speech speech_tok_compress_ratio (int): Compression ratio for speech tokenization target_sample_rate (int): Target sample rate for audio normalize_audio (bool): Whether to normalize audio input N@]TcKsJ||_|p t||d|_||_||_||_|rt|_nd|_|dS)N) sampling_ratenormalize_audio) tokenizerr audio_processorspeech_tok_compress_ratiotarget_sample_raterr audio_normalizer_cache_special_tokens)selfrrrrrkwargsrO/home/ubuntu/vibevoice-community/vibevoice/processor/vibevoice_asr_processor.py__init__-s   zVibeVoiceASRProcessor.__init__cCst|jdr |jj|_n|jd|_t|jdr|jj|_n|jd|_t|jdr2|jj|_n|jd|_t|jdrF|jj|_d St|jdrS|jj|_d S|jd |_d S) z'Cache special token IDs for efficiency.speech_start_idz<|speech_start|> speech_end_idz<|speech_end|> speech_pad_idz<|speech_pad|>pad_id pad_token_idz <|endoftext|>N)hasattrrrconvert_tokens_to_idsr r!r"r#rrrrrGs        z+VibeVoiceASRProcessor._cache_special_tokensc Ksddl}ddlm}ddlm}tj|d}i}tj|r:t |d }| |}Wdn1s4wYnEz%||dfi|} t | d }| |}Wdn1sYwYWnt y~} zt d| t dWYd} ~ nd} ~ ww|d d } |d d } |d d} |ddp|dd}t d|d|vr|j|fi|}ntd|t| | |dd|ddd}|||| | | dS)a& Load processor from a pretrained model path. Args: pretrained_model_name_or_path: Path to the pretrained model **kwargs: Additional keyword arguments Returns: VibeVoiceASRProcessor: The loaded processor rN) cached_file)VibeVoiceASRTextTokenizerFastpreprocessor_config.jsonrz)Could not load preprocessor_config.json: zUsing default configurationri rrrTlanguage_model_pretrained_namezQwen/Qwen2.5-1.5BzLoading tokenizer from qwenzUnsupported tokenizer type for target_dB_FSepsư>)rrr-r/)rrrrr)jsontransformers.utilsr'2vibevoice.modular.modular_vibevoice_text_tokenizerr(ospathjoinexistsopenload Exceptionloggerwarninggetpopinfolowerfrom_pretrained ValueErrorr )clspretrained_model_name_or_pathrr1r'r( config_pathconfigf config_fileerrrr+rrrrrrA`sf             z%VibeVoiceASRProcessor.from_pretrainedsave_directorycKsddl}tj|ddd|j|j|jddd}tj|d }t|d }|j ||d d Wdn1s6wYt d |dS)z Save processor configuration to a directory. Args: save_directory: Directory to save the configuration **kwargs: Additional keyword arguments rNT)exist_okrr.r0)processor_classrrrr-r/r)w)indentz!Processor configuration saved in ) r1r4makedirsrrrr5r6r8dumpr;r?)rrJrr1processor_configrErGrrrsave_pretraineds z%VibeVoiceASRProcessor.save_pretrainedFaudiorreturn_tensorspadding max_length truncationadd_generation_prompt use_streaming context_inforeturnc Ksr|durtdt|trd} |} nd} |g} g} | D]}|j||||| d}| |q|j| ||||d}|S)a Process audio input for ASR model. Args: audio: Audio input(s). Can be: - str: Path to audio file - np.ndarray: Audio array - torch.Tensor: Audio tensor - List of the above for batch processing sampling_rate: Sampling rate of input audio return_tensors: Output format ('pt' for PyTorch, 'np' for NumPy) padding: Whether to pad batch inputs max_length: Maximum sequence length truncation: Whether to truncate long sequences add_generation_prompt: Whether to add generation prompt for inference use_streaming: Whether to use streaming mode (True by default, auto False if <60s) context_info: Optional context information (e.g., hotwords, metadata) to help transcription Returns: BatchEncoding with: - input_ids: Token IDs for the model - attention_mask: Attention mask - acoustic_input_mask: Mask indicating speech token positions - speech_tensors: Processed speech features - speech_masks: Valid speech masks - vae_tok_seqlens: Length of each speech segment in tokens Nz*Audio input is required for ASR processingTF)rrYrZr[)rVrWrXrU)rB isinstancelist_process_single_audioappend _batch_encode)rrTrrUrVrWrXrYrZr[r is_batched audio_list all_encodings audio_inputencodingbatch_encodingrrr__call__s2(  zVibeVoiceASRProcessor.__call__c szt|trktrBz t|dd\}}WnFtyA}z$td|ddl} | |\}}|j dkr7|j dd}WYd}~nd}~wwddl} | |\}}|j dkrX|j dd}|j krjddl } | j ||j d}n't|tjr|}|j dkr|}ntj|tjd }|j dkr|}|tj}jrjr|}t|j } |r| d krd}tt|j} jjd td gdd } j | }j!j"}j!j#}j!j$}gd}|r|%rd| dd|%dd&|}n d| ddd&|}d&|g|g| |gd|}jjd|d gdd }||}fdd|D}|||| dS)a Process a single audio input. Args: audio: Single audio input sampling_rate: Audio sampling rate add_generation_prompt: Whether to add generation prompt context_info: Optional context information (e.g., hotwords, metadata) to help transcription Returns: Dictionary with processed tokens and audio features F)resamplez2ffmpeg loading failed, falling back to soundfile: rNr )axis)orig_sr target_srdtypegN@system)rolecontent)tokenize) Start timeEnd time Speaker IDContentz This is a z.2fz! seconds audio, with extra info: z( Please transcribe it with these keys: z, z6 seconds audio, please transcribe it with these keys:  userTcsg|] }|jkr dndqS)r r)r!).0tokenr&rr {sz?VibeVoiceASRProcessor._process_single_audio..) input_idsacoustic_input_maskspeech vae_tok_len)'r]strHAS_FFMPEG_UTILSrr:warningswarn soundfilereadndimmeanrlibrosaritorchTensorcpunumpysqueezenparrayfloat32astyperrlenmathceilrrapply_chat_template SYSTEM_PROMPTencodeconvert_ids_to_tokensrr!r stripr6)rrTrrYrZr[ audio_arrayfile_srrIsfraudio_durationrsystem_prompt_text system_tokenssp_start_token sp_pad_token sp_end_token show_keys user_suffixuser_input_string user_tokens full_tokensr~rr&rr_ s                 $ z+VibeVoiceASRProcessor._process_single_audio encodingscCsdd|D}dd|D}dd|D}dd|D} |r|dur%|} n tdd|D} g} g} g} t||D]H\}}|rQt|| krQ|d| }|d| }| t|}|jg||}d g||}d g|d gt|}| || || |q9| }| }nd d|D} td d|D}tjt||ftjd }tjt|t| ftd }t t|| D]\}\}}|||dt|f<d||d|f<qt }|dkr t j |t j d |d<t j | t j d |d<t j |t jd |d<t j |t jd |d<t j |t jd |d<|St|d kr|n|d |d<t| d kr$| n| d |d<t|d kr3|n|d |d<t|d krB|n|d |d<t|d krQ|n|d |d<|S)a~ Combine multiple encodings into a batch. Args: encodings: List of encoded samples padding: Whether to pad sequences max_length: Maximum sequence length truncation: Whether to truncate return_tensors: Output format Returns: BatchEncoding with batched data cSg|]}|dqS)r}rrzencrrrr|z7VibeVoiceASRProcessor._batch_encode..cSr)r~rrrrrr|rcSr)rrrrrrr|rcSr)rrrrrrr|rNcs|]}t|VqdSNrrzidsrrr z6VibeVoiceASRProcessor._batch_encode..rr cSsg|] }dgt|qS)r rrrrrr|scsrrr)rzsrrrrrrmTptr}attention_maskr~speech_tensors speech_masks)maxziprr"r`rzerosrbool enumeraterrtensorlong)rrrVrWrXrUinput_ids_listacoustic_masks_list speech_list vae_tok_lens target_lengthpadded_input_idspadded_acoustic_masksattention_masksr} acoustic_maskpadding_length padded_idspadded_acousticrmax_speech_lengthpadded_speechesrirvae_lenrgrrrrasZ        z#VibeVoiceASRProcessor._batch_encodecO|jj|i|S)zi Decode batch of token IDs to text. Forwards to tokenizer's batch_decode method. )r batch_decoderargsrrrrrz"VibeVoiceASRProcessor.batch_decodecOr)zZ Decode token IDs to text. Forwards to tokenizer's decode method. )rdecoderrrrrrzVibeVoiceASRProcessor.decodetextc Cszd|vr|dd}|d|}|||}nH|d}|dkr)|d}|dkrad}|}t|t|D]!}||dvrE|d 7}q8||d vrY|d 8}|dkrY|d }nq8|||}n|}t|}t|trp|g}g}|D]-} t| tri} d d d d d d dd} | D]\} } | | vr| | | | <q| r| | qt|WStj y}zt d|t d|gWYd}~Sd}~wty}zt d|gWYd}~Sd}~ww)z Post-process the generated transcription text to extract structured data. Args: text: Generated text from the model Returns: List of dictionaries with transcription segments z```jsonz```[{rz[{r z]} start_timeend_time speaker_idr)rsStartrtEndruSpeakerrvz)Failed to parse JSON from transcription: z Raw text: Nz%Error post-processing transcription: )findrrangerr1loadsr]dictitemsr`JSONDecodeErrorr;r<debugr:)rr json_startjson_endjson_str bracket_countrresultcleaned_resultitem cleaned_item key_mappingkey mapped_keyrIrrrpost_process_transcriptionsn             z0VibeVoiceASRProcessor.post_process_transcriptioncCsgdS)z0Return the list of inputs accepted by the model.)r}rr~rrrr&rrrmodel_input_names7sz'VibeVoiceASRProcessor.model_input_names)NNrrT) NNNTNFTTN)NTTN)TNFN)!__name__ __module__ __qualname____doc__rr classmethodrArrr4PathLikerSrrndarrayrrrintr rrrhrrr_rarrrpropertyrrrrrrs  D(   M  | XMr)"rr4r1rrtypingrrrrrrrrr$transformers.tokenization_utils_baserr2r r vibevoice_tokenizer_processorr r audio_utilsrr ImportErrorr get_loggerrr;rr__all__rrrrs4      "