o ¾e¦i´ã@s:dZddlmZdddœZGdd„deƒZeZddgZd S) z$Tokenization classes for DistilBERT.é)Ú BertTokenizerz vocab.txtztokenizer.json)Ú vocab_fileÚtokenizer_filecs0eZdZddgZddœdef‡fdd„Z‡ZS)ÚDistilBertTokenizerÚ input_idsÚattention_maskT)Ú do_lower_casercstƒj|d|i|¤ŽdS)aË Construct a DistilBERT tokenizer (backed by HuggingFace's tokenizers library). Based on WordPiece. This tokenizer inherits from [`BertTokenizer`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: do_lower_case (`bool`, *optional*, defaults to `True`): Whether or not to lowercase the input when tokenizing. rN)ÚsuperÚ__init__)ÚselfrÚargsÚkwargs©Ú __class__©út/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/distilbert/tokenization_distilbert.pyr s zDistilBertTokenizer.__init__)Ú__name__Ú __module__Ú __qualname__Úmodel_input_namesÚboolr Ú __classcell__rrrrrs rÚDistilBertTokenizerFastN)Ú__doc__Úmodels.bert.tokenization_bertrÚVOCAB_FILES_NAMESrrÚ__all__rrrrÚs