o 5tit@s@ddlmZddlmZddlmZgdZGdddeZdS) ) lru_cache) BaseTokenizer)TokenizerRegexp))u㐀u䶵)u一u龥)u龦u龻)u豈u鶴)u侮u頻)u並u龎)u 0u⩭6)u⾀0u⾡d)u＀u￯)u⺀u⻿)u　u〿)u㇀u㇯)u⼀u⿟)u⿰u⿿)u㄀uㄯ)uㆠuㆿ)u︐u︟)u︰u﹏)u☀u⛿)u✀u➿)u㈀u㋿)u㌀u㏿c@sDeZdZddZddZeeddddZeddd d Zd S) TokenizerZhcCsdS)NzhselfrrU/home/ubuntu/.local/lib/python3.10/site-packages/sacrebleu/tokenizers/tokenizer_zh.py signatureJszTokenizerZh.signaturecCs t|_dS)N)r_post_tokenizerr rrr __init__Ms zTokenizerZh.__init__i)maxsizecCs.tD]\}}||kr|krdSqqdS)zu :param uchar: input char in unicode :return: whether the input char is a Chinese character. TF) _UCODE_RANGES)ucharstartendrrr _is_chinese_charPs zTokenizerZh._is_chinese_charcCsL|}d}|D]}||r|d7}||7}|d7}q||7}q||S)aXThe tokenization of Chinese text in this script contains two steps: separate each Chinese characters (by utf-8 encoding); tokenize the non Chinese part (following the `13a` i.e. mteval tokenizer). Author: Shujian Huang huangsj@nju.edu.cn :param line: input sentence :return: tokenized sentence  )striprr )r line line_in_charscharrrr __call__\s     zTokenizerZh.__call__N) __name__ __module__ __qualname__r r staticmethodrrrrrrr rHs  rN) functoolsrtokenizer_baser tokenizer_rerrrrrrr s '