o i !@s<ddlZddlmZddlZddlmZddlZddlmZddlm Z ddl m Z ddl m Z ddlmZdd lmZdd lmZdd lmZd e ed efddZdeded e ede ededede ede ededededededeede ede ef dd Zd ejfd!d"Zd&d#d$Zed%kredSdS)'N)Counter)Path)List)Optional)get_commandline_args)build_tokenizer) TextCleaner) g2p_classes)str2bool) str_or_nonefieldreturncCs|}zBd|vr5|jddd\}}|dkrd}n t|}|dkr'td|dkr0d}nt|}nt|}|d}|dkrEtdWntyTtd |w|dur`td|}|St|d|}|S) aConvert field string to slice Note that field string accepts 1-based integer. Examples: >>> field2slice("1-") slice(0, None, None) >>> field2slice("1-3") slice(0, 3, None) >>> field2slice("-3") slice(None, 3, None) -)maxsplitNrz1-based stringzmust be 1 or more valuez)Format error: e.g. '2-', '2-5', or '-5': )stripsplitint ValueError RuntimeErrorslice)r s1s2slicrL/home/ubuntu/.local/lib/python3.10/site-packages/funasr/bin/tokenize_text.py field2slices2      rinputoutput delimiter token_type space_symbolnon_linguistic_symbolsbpemodel log_levelwrite_vocabularyvocabulary_sizeremove_non_linguistic_symbolscutoff add_symbolcleanerg2pc stj|dd|dkrtj}n t|jddd}|dkr tj}nt|}|jjddd|jd dd}t |}t |||||| |d }t }|durMt |}|D]E}| }|durq||}||}|durld |}n||}||}||}| s|d |d qO|D] }||d 7<qqO| sdS| D]%}z |d\}}Wntytd|w|}||vr||=qttfddt|ddd}| dkr| t| krtd| |d| t| }| D]8}z |d\}}t|}Wntytd|w|}|dkr!t|d |}|||dfq|D] \}}||d q,t|}tdd|D}td|||dddS)Nz>%(asctime)s (%(module)s:%(lineno)d) %(levelname)s: %(message)s)levelformatrrzutf-8)encodingT)parentsexist_okw)r!r$r r"r#r(g2p_type  r:z Format error: e.g. ':0': cs |dkSNrrxr)rrs ztokenize..cSs |d Sr8rr9rrrr<s )keyrzvocabulary_size is too small: css |] \}}|dur|VqdSNr).0r3crrr sztokenize..z OOV rate = dz %)logging basicConfigsysstdinropenstdoutparentmkdirrrrrrstriprjoin text2tokenswriterrrlistfiltersorteditemslenrinsertsumvaluesinfo) rrr r r!r"r#r$r%r&r'r(r)r*r+r,finfoutp tokenizercounterlinetokenst symbol_and_idsymbolidxwords_and_countsr3r@ total_count invocab_countrr;rtokenizeAs            "rfcCsLtjdtjd}|jddddddd |jd d d d d|jddd dd|jdddd|jdddgddd|jddddd|jd d!d"d|jd#dd$d|jd%td&d'|jd(td)d*d+|jd,tgd-dd.d/|jd0ttdd1d/|d2}|jd3td)d4d+|jd5td6d7d+|jd8d6td9d:|jd;t gd|S)?NzTokenize texts) descriptionformatter_classz --log_levelcSs|Sr>)upperr9rrrr<szget_parser..INFO)CRITICALERRORWARNINGrjDEBUGNOTSETzThe verbose level of logging)typedefaultchoiceshelpz--inputz-iTz!Input text. - indicates sys.stdin)requiredrsz--outputz-oz#Output text. - indicates sys.stdoutz--fieldz-fz?The target columns of the input text as 1-based integer. e.g 2-)rsz --token_typez-tchar)rubpewordphnz Token type)rqrrrsz --delimiterz-dz The delimiter)rqrsz--space_symbolzzThe space symbolz --bpemodelzThe bpemodel file pathz--non_linguistic_symbolsz non_linguistic_symbols file path)rprsz--remove_non_linguistic_symbolsFz'Remove non-language-symbols from tokens)rprqrsz --cleaner)Ntacotronjaconv vietnamesekorean_cleanerzApply text cleaning)rprrrqrsz--g2pz&Specify g2p method if --token_type=phnzwrite_vocabulary mode relatedz--write_vocabularyz4Write tokens list instead of tokenized text per linez--vocabulary_sizerzVocabulary sizez--cutoffz0cut-off frequency used for write-vocabulary mode)rqrprsz --add_symbolappendzBAppend symbol e.g. --add_symbol ':0' --add_symbol ':1')rprqactionrs) argparseArgumentParserArgumentDefaultsHelpFormatter add_argumentr r r add_argument_grouprstr)parsergrouprrr get_parsers rcCs:tttjdt}||}t|}tdi|dS)N)filer)printrrEstderrr parse_argsvarsrf)cmdrargskwargsrrrmain s  r__main__r>)r collectionsrrCpathlibrrEtypingrrfunasr.utils.cli_utilsr funasr.tokenizer.build_tokenizerrfunasr.tokenizer.cleanerr"funasr.tokenizer.phoneme_tokenizerr funasr.utils.typesr r rrrboolrrfrrr__name__rrrrsh          .      z Q