o iS@sddlmZddlmZddlmZddlmZddlmZddl m Z ddl m Z ddl mZ dd ed eeeeefdeeeeefdededededefddZd S))Path)Iterable)Union) AbsTokenizer) CharTokenizer)PhonemeTokenizer)SentencepiecesTokenizer) WordTokenizerNF token_typebpemodelnon_linguistic_symbolsremove_non_linguistic_symbols space_symbol delimiterg2p_typereturncCs|dkr|dur td|rtdt|S|dkr,|r'|dur't||ddSt|dS|d kr7t|||d S|d krCt||||d Std |)z*A helper function to instantiate TokenizerbpeNz*bpemodel is required if token_type = "bpe"zCremove_non_linguistic_symbols is not implemented for token_type=bpewordT)rr r)rchar)r rrphn)rr rrz2token_mode must be one of bpe, word, char or phn: ) ValueError RuntimeErrorrr rr)r r r rrrrrT/home/ubuntu/.local/lib/python3.10/site-packages/funasr/tokenizer/build_tokenizer.pybuild_tokenizer s<   r)NNFr NN)pathlibrtypingrrfunasr.tokenizer.abs_tokenizerrfunasr.tokenizer.char_tokenizerr"funasr.tokenizer.phoneme_tokenizerr(funasr.tokenizer.sentencepiece_tokenizerrfunasr.tokenizer.word_tokenizerr strboolrrrrrs>