o i%@s@ddlmZddlZddlmZdefddZdefdd ZdS) )ListN)PreTrainedTokenizer tokenizercs.dd|jDGfddd}||S)uCreate a tokenizer wrapper that converts multi-character Chinese tokens to single characters. This function creates a wrapper around the provided tokenizer that automatically splits multi-character Chinese tokens into individual characters. This is useful for ensuring consistent tokenization of Chinese text. Args: tokenizer: The base tokenizer to wrap Returns: A CharTokenizerWrapper instance that handles multi-character Chinese tokens Example: >>> from transformers import LlamaTokenizerFast >>> tokenizer = LlamaTokenizerFast.from_pretrained("path/to/tokenizer") >>> wrapped_tokenizer = mask_multichar_chinese_tokens(tokenizer) >>> tokens = wrapped_tokenizer("你好世界") cSs.h|]}t|dkrtdd|Dr|qS)css(|]}d|ko dknVqdS)u一u鿿N).0crrF/home/ubuntu/.local/lib/python3.10/site-packages/voxcpm/model/utils.py s&z:mask_multichar_chinese_tokens...)lenall)rtokenrrr s z0mask_multichar_chinese_tokens..csReZdZdZdeddffdd Zdedeefdd Zdedee fd d Z dS) z;mask_multichar_chinese_tokens..CharTokenizerWrapperzWrapper class for tokenizers that handles multi-character Chinese tokens. This wrapper automatically splits multi-character Chinese tokens into individual characters while preserving the original tokenizer's interface. base_tokenizerreturnNcs||_|_dS)zInitialize the wrapper with a base tokenizer. Args: base_tokenizer: The tokenizer to wrap N)rmultichar_tokens)selfrrrr __init__&s zDmask_multichar_chinese_tokens..CharTokenizerWrapper.__init__textc[svt|tstdt||jj|fi|}g}|D]}|dd}||jvr3t|}| |q| |q|S)uiTokenize text and split multi-character Chinese tokens into single characters. Args: text: Input text to tokenize **kwargs: Additional arguments passed to the base tokenizer Returns: List of processed tokens with multi-character Chinese tokens split Example: >>> wrapper = CharTokenizerWrapper(tokenizer) >>> tokens = wrapper.tokenize("你好世界") >>> # Returns ["你", "好", "世", "界"] instead of ["你好", "世界"] zExpected string input, got u▁) isinstancestr TypeErrortypertokenizereplacerlistextendappend)rrkwargstokens processedr clean_tokencharsrrr r/s     zDmask_multichar_chinese_tokens..CharTokenizerWrapper.tokenizec [sRz|j|fi|}|j|}|WSty(}z tdt||d}~ww)a8Call the tokenizer and return token IDs. This method provides the same interface as the original tokenizer but with multi-character Chinese token handling. Args: text: Input text to tokenize **kwargs: Additional arguments passed to the base tokenizer Returns: List of token IDs Raises: TypeError: If input is not a string ValueError: If tokenization fails zTokenization failed: N)rrconvert_tokens_to_ids Exception ValueErrorr)rrr r!resulterrr __call__Qs zDmask_multichar_chinese_tokens..CharTokenizerWrapper.__call__) __name__ __module__ __qualname____doc__rrrrrintr*rrrr CharTokenizerWrappers  "r0)vocabkeys)rr0rrr mask_multichar_chinese_tokenss Jr3dtypecCsb|dkrtjS|dkrtjS|dkrtjS|dkrtjS|dkr#tjS|dkr*tjStd|)Nbfloat16bf16float16fp16float32fp32zUnsupported dtype: )torchr5r7r9r')r4rrr get_dtypelsr<)typingrr; transformersrr3rr<rrrr s  f