o ߥµi¥ã@s@dZddlmZmZmZddlmZe ¡ZGdd„deƒZ dS)zcTokenization classes for Space. mainly copied from :module:`~transformers.tokenization_xlm_roberta`é)ÚBasicTokenizerÚ BertTokenizerÚWordpieceTokenizer)Úloggerc@seZdZdZdS)ÚSpaceTokenizerzŽ This class overrides [`SpaceTokenizer`]. Please check the superclass for the appropriate documentation alongside usage examples. N)Ú__name__Ú __module__Ú __qualname__Ú__doc__©r r úh/home/ubuntu/.local/lib/python3.10/site-packages/modelscope/models/nlp/space/model/tokenization_space.pyrsrN) r Ú transformersrrrÚmodelscope.utilsrÚloggingÚ get_loggerrr r r r Ús