o 聱i @srdZddlmZmZmZddlmZddlmZddl m Z e e Z GdddeZGdd d e Zdd gZd S) z#Tokenization classes for vibevoice.)ListOptionalUnion)logging)Qwen2Tokenizer)Qwen2TokenizerFastcseZdZdZddgZ       dfd d Zd d Zed efddZ ed efddZ ed efddZ ed efddZ ed efddZ ZS)VibeVoiceTextTokenizera Construct a VibeVoice tokenizer. Based on the Qwen2 tokenizer with additional special tokens for speech. Args: vocab_file (`str`): Path to the vocabulary file. merges_file (`str`): Path to the merges file. errors (`str`, *optional*, defaults to `"replace"`): Paradigm to follow when decoding bytes to UTF-8. unk_token (`str`, *optional*, defaults to `"<|endoftext|>"`): The unknown token. bos_token (`str`, *optional*): The beginning of sequence token. Not used for vibevoice. eos_token (`str`, *optional*, defaults to `"<|endoftext|>"`): The end of sequence token. pad_token (`str`, *optional*, defaults to `"<|endoftext|>"`): The token used for padding. add_special_tokens (`bool`, *optional*, defaults to `True`): Whether or not to add special tokens when encoding. input_idsattention_maskreplace <|endoftext|>NFTc s2tjd||||||||| d | |dS)N) vocab_file merges_fileerrors unk_token bos_token eos_token pad_tokenadd_prefix_spaceadd_special_tokenssuper__init___add_vibevoice_special_tokens) selfr rrrrrrrrkwargs __class__r[/home/ubuntu/VibeVoice-finetuning/src/vibevoice/modular/modular_vibevoice_text_tokenizer.pyr%s   zVibeVoiceTextTokenizer.__init__cCsJdgdi}||}|d|_|d|_|d|_|d|_|S)&Add VibeVoice-specific special tokens.additional_special_tokens<|vision_start|><|vision_end|><|vision_pad|>r#r$r%r )rconvert_tokens_to_ids_speech_start_id_speech_end_id_speech_diffusion_id_eos_idrspecial_tokens num_addedrrrrBs     z4VibeVoiceTextTokenizer._add_vibevoice_special_tokensreturncC|jSz Id of the end of sequence token.r*rrrreos_idVzVibeVoiceTextTokenizer.eos_idcCr/zId of the speech start token.r'r2rrrspeech_start_id[r4z&VibeVoiceTextTokenizer.speech_start_idcCr/zId of the speech end token.r(r2rrr speech_end_id`r4z$VibeVoiceTextTokenizer.speech_end_idcCr/z!Id of the speech diffusion token.r)r2rrrspeech_diffusion_ider4z*VibeVoiceTextTokenizer.speech_diffusion_idcCsdS)4Id used for padding (returns -100 for loss masking).irr2rrrpad_idjszVibeVoiceTextTokenizer.pad_id)r r Nr r FT__name__ __module__ __qualname____doc__model_input_namesrrpropertyintr3r7r:r=r? __classcell__rrrrr s,rcseZdZdZddgZ        dfdd Zd d Zed efd d Z ed efddZ ed efddZ ed efddZ ed efddZ ZS)VibeVoiceTextTokenizerFasta Construct a "fast" VibeVoice tokenizer (backed by HuggingFace's *tokenizers* library). Based on the Qwen2 tokenizer with additional special tokens for speech. Args: vocab_file (`str`, *optional*): Path to the vocabulary file. merges_file (`str`, *optional*): Path to the merges file. tokenizer_file (`str`, *optional*): Path to [tokenizers](https://github.com/huggingface/tokenizers) file. unk_token (`str`, *optional*, defaults to `"<|endoftext|>"`): The unknown token. bos_token (`str`, *optional*): The beginning of sequence token. Not used for vibevoice. eos_token (`str`, *optional*, defaults to `"<|endoftext|>"`): The end of sequence token. pad_token (`str`, *optional*, defaults to `"<|endoftext|>"`): The token used for padding. r r Nr Fc s0tjd||||||||d| |dS)N)r rtokenizer_filerrrrrrr) rr rrJrrrrrrrrrrs   z#VibeVoiceTextTokenizerFast.__init__cCsRdgdi}||}|d|_|d|_|d|_|j|_|d|_|S)r r!r"r#r$r%z <|image_pad|>)rr&r'r(r) eos_token_idr*_pad_idr+rrrrs     z8VibeVoiceTextTokenizerFast._add_vibevoice_special_tokensr.cCr/r0r1r2rrrr3r4z!VibeVoiceTextTokenizerFast.eos_idcCr/r5r6r2rrrr7r4z*VibeVoiceTextTokenizerFast.speech_start_idcCr/r8r9r2rrrr:r4z(VibeVoiceTextTokenizerFast.speech_end_idcCr/r;r<r2rrrr=r4z.VibeVoiceTextTokenizerFast.speech_diffusion_idcCr/)r>)rLr2rrrr?r4z!VibeVoiceTextTokenizerFast.pad_id)NNNr Nr r Fr@rrrrrIps.rIN)rDtypingrrrtransformers.utilsr,transformers.models.qwen2.tokenization_qwen2r1transformers.models.qwen2.tokenization_qwen2_fastr get_loggerrAloggerrrI__all__rrrrs    dd