o 5ti@s\ddlmZz ddlZddlZWn eydZYnwddlmZdZGdddeZ dS)) lru_cacheN) BaseTokenizerz Korean tokenization requires extra dependencies, but you do not have them installed. Please install them like so. pip install sacrebleu[ko] c@s.eZdZddZeddddZddZd S) TokenizerKoMecabcCsPtdurttttjd|_|j}|jdksJd|j dus&JdS)Nz -Owakatiic z5Please make sure to use the mecab-ko-dic for MeCab-ko) MeCab RuntimeError FAIL_MESSAGETagger mecab_ko_dic MECAB_ARGStaggerdictionary_infosizenext)selfdr[/home/ubuntu/.local/lib/python3.10/site-packages/sacrebleu/tokenizers/tokenizer_ko_mecab.py__init__s  zTokenizerKoMecab.__init__i)maxsizecCs|}|j|}|S)z Tokenizes an Korean input line using MeCab-ko morphological analyzer. :param line: a segment to tokenize :return: the tokenized line )stripr parse)rlinesentencerrr__call__!szTokenizerKoMecab.__call__cCs|jd}d|S)zU Returns the MeCab-ko parameters. :return: signature string z-KOz ko-mecab-)r version)r signaturerrrr-szTokenizerKoMecab.signatureN)__name__ __module__ __qualname__rrrrrrrrrs   r) functoolsrmecab_korr ImportErrortokenizer_baserrrrrrrs