o 5ti@sGdddZdS)c@s eZdZdZddZddZdS) BaseTokenizerz&A base dummy tokenizer to derive from.cCt)z[ Returns a signature for the tokenizer. :return: signature string NotImplementedError)selfrW/home/ubuntu/.local/lib/python3.10/site-packages/sacrebleu/tokenizers/tokenizer_base.py signatureszBaseTokenizer.signaturecCr)z Tokenizes an input line with the tokenizer. :param line: a segment to tokenize :return: the tokenized line r)rlinerrr__call__ szBaseTokenizer.__call__N)__name__ __module__ __qualname____doc__rr rrrrrs rN)rrrrrs