h dZddlmZmZmZddlmZddlmZddl m Z ej e Z GddeZGdd e Zdd gZd S) z#Tokenization classes for vibevoice.)ListOptionalUnion)logging)Qwen2Tokenizer)Qwen2TokenizerFastceZdZdZddgZ dfd Zd Zed efd Z ed efd Z ed efdZ ed efdZ ed efdZ xZS)VibeVoiceTextTokenizera Construct a VibeVoice tokenizer. Based on the Qwen2 tokenizer with additional special tokens for speech. Args: vocab_file (`str`): Path to the vocabulary file. merges_file (`str`): Path to the merges file. errors (`str`, *optional*, defaults to `"replace"`): Paradigm to follow when decoding bytes to UTF-8. unk_token (`str`, *optional*, defaults to `"<|endoftext|>"`): The unknown token. bos_token (`str`, *optional*): The beginning of sequence token. Not used for vibevoice. eos_token (`str`, *optional*, defaults to `"<|endoftext|>"`): The end of sequence token. pad_token (`str`, *optional*, defaults to `"<|endoftext|>"`): The token used for padding. add_special_tokens (`bool`, *optional*, defaults to `True`): Whether or not to add special tokens when encoding. input_idsattention_maskreplace <|endoftext|>NFTc vtjd||||||||| d | |dS)N) vocab_file merges_fileerrors unk_token bos_token eos_token pad_tokenadd_prefix_spaceadd_special_tokenssuper__init___add_vibevoice_special_tokens) selfrrrrrrrrrkwargs __class__s Z/workspace/chatterbox-finetuning/src/vibevoice/modular/modular_vibevoice_text_tokenizer.pyrzVibeVoiceTextTokenizer.__init__%si  !#-1   **,,,,,c dgdi}||}|d|_|d|_|d|_|d|_|S)&Add VibeVoice-specific special tokens.additional_special_tokens<|vision_start|><|vision_end|><|vision_pad|>r'r(r)r)rconvert_tokens_to_ids_speech_start_id_speech_end_id_speech_diffusion_id_eos_idrspecial_tokens num_addeds r!rz4VibeVoiceTextTokenizer._add_vibevoice_special_tokensBs (*** ++N;; !% : :;M N N"889IJJ$($>$>?O$P$P!11/BB r"returnc|jSz Id of the end of sequence token.r.rs r!eos_idzVibeVoiceTextTokenizer.eos_idV |r"c|jSzId of the speech start token.r+r6s r!speech_start_idz&VibeVoiceTextTokenizer.speech_start_id[ $$r"c|jSzId of the speech end token.r,r6s r! speech_end_idz$VibeVoiceTextTokenizer.speech_end_id` ""r"c|jSz!Id of the speech diffusion token.r-r6s r!speech_diffusion_idz*VibeVoiceTextTokenizer.speech_diffusion_ide ((r"cdS)4Id used for padding (returns -100 for loss masking).irr6s r!pad_idzVibeVoiceTextTokenizer.pad_idjs tr")r rNrrFT__name__ __module__ __qualname____doc__model_input_namesrrpropertyintr7r<rArFrJ __classcell__r s@r!r r s>,%&67 !!!------:(X%%%%X%#s###X#)S)))X)Xr"r ceZdZdZddgZ dfd ZdZed efd Z ed efd Z ed efd Z ed efd Z ed efdZ xZS)VibeVoiceTextTokenizerFasta Construct a "fast" VibeVoice tokenizer (backed by HuggingFace's *tokenizers* library). Based on the Qwen2 tokenizer with additional special tokens for speech. Args: vocab_file (`str`, *optional*): Path to the vocabulary file. merges_file (`str`, *optional*): Path to the merges file. tokenizer_file (`str`, *optional*): Path to [tokenizers](https://github.com/huggingface/tokenizers) file. unk_token (`str`, *optional*, defaults to `"<|endoftext|>"`): The unknown token. bos_token (`str`, *optional*): The beginning of sequence token. Not used for vibevoice. eos_token (`str`, *optional*, defaults to `"<|endoftext|>"`): The end of sequence token. pad_token (`str`, *optional*, defaults to `"<|endoftext|>"`): The token used for padding. r r NrFc ttjd||||||||d| |dS)N)rrtokenizer_filerrrrrrr) rrrrXrrrrrrr s r!rz#VibeVoiceTextTokenizerFast.__init__sf  !#)-   **,,,,,r"c$dgdi}||}|d|_|d|_|d|_|j|_|d|_|S)r$r%r&r'r(r)z <|image_pad|>)rr*r+r,r- eos_token_idr._pad_idr/s r!rz8VibeVoiceTextTokenizerFast._add_vibevoice_special_tokenss (*** ++N;; !% : :;M N N"889IJJ$($>$>?O$P$P!( 11/BB r"r2c|jSr4r5r6s r!r7z!VibeVoiceTextTokenizerFast.eos_idr8r"c|jSr:r;r6s r!r<z*VibeVoiceTextTokenizerFast.speech_start_idr=r"c|jSr?r@r6s r!rAz(VibeVoiceTextTokenizerFast.speech_end_idrBr"c|jSrDrEr6s r!rFz.VibeVoiceTextTokenizerFast.speech_diffusion_idrGr"c|jS)rI)r[r6s r!rJz!VibeVoiceTextTokenizerFast.pad_idr8r")NNNrNrrFrKrTs@r!rVrVpsA*%&67!!!------6,X%%%%X%#s###X#)S)))X)Xr"rVN)rOtypingrrrtransformers.utilsr,transformers.models.qwen2.tokenization_qwen2r1transformers.models.qwen2.tokenization_qwen2_fastr get_loggerrLloggerr rV__all__rr"r!rhs))((((((((((&&&&&&GGGGGGPPPPPP  H % %aaaaa^aaaH`````!3```H  r"