o ´©i’ã@s$ddlmZe dd¡dd„ƒZdS)é)ÚtablesÚtokenizer_classesÚHuggingfaceTokenizercKs(zddlm}WnY| |¡}|S)Nr)Ú AutoTokenizer)Ú transformersrÚfrom_pretrained)Úinit_param_pathÚkwargsrÚ tokenizer©r úQ/home/ubuntu/.local/lib/python3.10/site-packages/funasr/tokenizer/hf_tokenizer.pyrs  N)Úfunasr.registerrÚregisterrr r r r Ús