o 5tiK@s|ddlZddlmZddlmZdedefddZdedefd d Zdedefd d Zdedefd dZ GdddeZ dS)N) lru_cache) BaseTokenizersentreturncCstdd|}tdd|}tdd|}tdd|}td d |}td d |}d|d}td d|}tdd|}tdd|}tdd|}tdd|}tdd|}|S)Nz\n-z\n z""z&&z<z([{-~[-` -&(-+:-@/]) \1 z's z 's z's$z 'sz([^0-9])([\.,])\1 \2 z([\.,])([^0-9])z \1 \2z ([0-9])(-)resubrrV/home/ubuntu/.local/lib/python3.10/site-packages/sacrebleu/tokenizers/tokenizer_ter.py_normalize_general_and_westerns rcCstdd|}tdd|}tdd|}tdd|}tdd|}tdd|}td d|}ttjd|}ttjd|}|S) Nz([\u4e00-\u9fff\u3400-\u4dbf])r z([\u31c0-\u31ef\u2e80-\u2eff])z+([\u3300-\u33ff\uf900-\ufaff\ufe30-\ufe4f])z([\u3200-\u3f22])z<(^|^[\u3040-\u309f])([\u3040-\u309f]+)(?=$|^[\u3040-\u309f])rz<(^|^[\u30a0-\u30ff])([\u30a0-\u30ff]+)(?=$|^[\u30a0-\u30ff])z<(^|^[\u31f0-\u31ff])([\u31f0-\u31ff]+)(?=$|^[\u31f0-\u31ff])rrTercomTokenizer ASIAN_PUNCTFULL_WIDTH_PUNCTrrrr_normalize_asian;s*rcCstdd|S)Nz[\.,\?:;!\"\(\)]rrrrrr _remove_punctcsrcCs$ttjd|}ttjd|}|S)Nrrrrrr_remove_asian_punctgsrc @s`eZdZdZdZdZ    ddedededefd d Zed d d e de fddZ ddZ dS)ra#Re-implementation of Tercom Tokenizer in Python 3. See src/ter/core/Normalizer.java in https://github.com/jhclark/tercom Note that Python doesn't support named Unicode blocks so the mapping for relevant blocks was taken from here: https://unicode-table.com/en/blocks/ z=([\u3001\u3002\u3008-\u3011\u3014-\u301f\uff61-\uff65\u30fb])z:([\uff0e\uff0c\uff1f\uff1a\uff1b\uff01\uff02\uff08\uff09])F normalizedno_punct asian_supportcase_sensitivecCs||_||_||_||_dS)aJInitialize the tokenizer. :param normalized: Enable character normalization. By default, normalizes a couple of things such as newlines being stripped, retrieving XML encoded characters, and fixing tokenization for punctuation. When 'asian_support' is enabled, also normalizes specific Asian (CJK) character sequences, i.e. split them down to the character level. :param no_punct: Remove punctuation. Can be used in conjunction with 'asian_support' to also remove typical punctuation markers in Asian languages (CJK). :param asian_support: Enable special treatment of Asian characters. This option only has an effect when 'normalized' and/or 'no_punct' is enabled. If 'normalized' is also enabled, then Asian (CJK) characters are split down to the character level. If 'no_punct' is enabled alongside 'asian_support', specific unicode ranges for CJK and full-width punctuations are also removed. :param case_sensitive: Enable case sensitivity, i.e., do not lower case data. N) _normalized _no_punct_asian_support_case_sensitive)selfrrrr rrr__init__zs zTercomTokenizer.__init__i)maxsizerrcCs\|sdS|js |}|jrt|}|jrt|}|jr't|}|jr't|}d | S)Nrr) r$lowerr!rr#rr"rrjoinsplit)r%rrrr__call__szTercomTokenizer.__call__cCsdS)Ntercomr)r%rrr signatureszTercomTokenizer.signatureN)FFFF) __name__ __module__ __qualname____doc__rrboolr&rstrr+r-rrrrrms(   r) r functoolsrtokenizer_baserr3rrrrrrrrrs  %(