o 5ti@s\ddlmZz ddlZddlZWn eydZYnwddlmZdZGdddeZdS)) lru_cacheN) BaseTokenizerz Japanese tokenization requires extra dependencies, but you do not have them installed. Please install them like so. pip install sacrebleu[ja] c@s.eZdZddZeddddZddZd S) TokenizerJaMecabcCsPtdurttttjd|_|j}|jdksJd|j dus&JdS)Nz -Owakatiiz4Please make sure to use the IPA dictionary for MeCab) MeCab RuntimeError FAIL_MESSAGETaggeripadic MECAB_ARGStaggerdictionary_infosizenext)selfdr[/home/ubuntu/.local/lib/python3.10/site-packages/sacrebleu/tokenizers/tokenizer_ja_mecab.py__init__s  zTokenizerJaMecab.__init__i)maxsizecCs|}|j|}|S)z Tokenizes an Japanese input line using MeCab morphological analyzer. :param line: a segment to tokenize :return: the tokenized line )stripr parse)rlinesentencerrr__call__!szTokenizerJaMecab.__call__cCs|jd}d|S)zR Returns the MeCab parameters. :return: signature string z-IPAz ja-mecab-)r version)r signaturerrrr-szTokenizerJaMecab.signatureN)__name__ __module__ __qualname__rrrrrrrrrs   r) functoolsrrr ImportErrortokenizer_baserrrrrrrs