o 5t¾ipã@sJdZddlZddlmZddlmZGdd„dejƒZGdd„deƒZdS) zòLibrary containing Tokenizer definitions. The RougeScorer class can be instantiated with the tokenizers defined here. New tokenizers can be defined by creating a subclass of the Tokenizer abstract class and overriding the tokenize() method. éN)Úporter)Útokenizec@seZdZdZejdd„ƒZdS)Ú TokenizerzhAbstract base class for a tokenizer. Subclasses of Tokenizer must implement the tokenize() method. cCstdƒ‚)Nz)Tokenizer must override tokenize() method)ÚNotImplementedError©ÚselfÚtext©r úJ/home/ubuntu/.local/lib/python3.10/site-packages/rouge_score/tokenizers.pyr szTokenizer.tokenizeN)Ú__name__Ú __module__Ú __qualname__Ú__doc__ÚabcÚabstractmethodrr r r r rsrc@s"eZdZdZddd„Zdd„ZdS) ÚDefaultTokenizerz0Default tokenizer which tokenizes on whitespace.FcCs|r t ¡|_dSd|_dS)z°Constructor for DefaultTokenizer. Args: use_stemmer: boolean, indicating whether Porter stemmer should be used to strip word suffixes to improve matching. N)rÚ PorterStemmerÚ_stemmer)rÚ use_stemmerr r r Ú__init__(szDefaultTokenizer.__init__cCst ||j¡S)N)rrrr r r r1szDefaultTokenizer.tokenizeN)F)r r r rrrr r r r r%s  r) rrÚ nltk.stemrÚ rouge_scorerÚABCrrr r r r Ús