o ߥi@sdZddlmZmZmZmZmZmZddlm Z ddl m Z ddl m Z mZddlmZddlmZeZdejiZdiiZd d d Zd d id d id ZGd dde ZdS)zTokenization classes for PoNet ) TYPE_CHECKINGAnyDictListOptionalUnion)PaddingStrategy) BertTokenizer) BatchEncoding EncodedInput) ModelFile) get_logger vocab_filei)z nlp_ponet_fill-mask_chinese-basez nlp_ponet_fill-mask_english-base do_lower_caseTc @sjeZdZdZeZeZeZ e Z de j ddfdeeeefefdeede deedeedef d d ZdS) PoNetTokenizera Construct an PoNet tokenizer. Based on BertTokenizer. This tokenizer inherits from :class:`~transformers.BertTokenizer` which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Refer to superclass :class:`~transformers.BertTokenizer` for usage examples and documentation concerning parameters. Nencoded_inputs max_lengthpadding_strategypad_to_multiple_ofreturn_attention_maskreturnc Cs|dur d|jv}||jd}|tjkrt|}|dur/|dur/||dkr/||d|}|tjko9t||k}|r|t|}|jdkr|rWdgt|dg||d<d|vrg|d|jg||d<d|vrv|ddg||d<d|vr|d|dd dg||d<||jg|||jd<|S|jd kr|rdg|dgt||d<d|vr|jg||d|d<d|vr|dd dg||d|d<d|vrdg||d|d<|jg||||jd<|Std t |j|r d|vr dgt||d<|S) a5 Pad encoded inputs (on left/right and up to predefined length or max length in the batch) Args: encoded_inputs: Dictionary of tokenized inputs (`List[int]`) or batch of tokenized inputs (`List[List[int]]`). max_length: maximum length of the returned list and optionally padding length (see below). Will truncate by taking into account the special tokens. padding_strategy: PaddingStrategy to use for padding. - PaddingStrategy.LONGEST Pad to the longest sequence in the batch - PaddingStrategy.MAX_LENGTH: Pad to the max length (default) - PaddingStrategy.DO_NOT_PAD: Do not pad The tokenizer padding sides are defined in self.padding_side: - 'left': pads on the left of the sequences - 'right': pads on the right of the sequences pad_to_multiple_of: (optional) Integer if set will pad the sequence to a multiple of the provided value. This is especially useful to enable the use of Tensor Core on NVIDIA hardware with compute capability >= 7.5 (Volta). return_attention_mask: (optional) Set to False to avoid returning attention mask (default: set to model specifics) Nattention_maskrrighttoken_type_idsspecial_tokens_mask segment_idsleftzInvalid padding strategy:) model_input_namesrLONGESTlen DO_NOT_PAD padding_sidepad_token_type_id pad_token_id ValueErrorstr) selfrrrrrrequired_inputneeds_to_be_padded differencer,\/home/ubuntu/.local/lib/python3.10/site-packages/modelscope/models/nlp/ponet/tokenization.py_pad@s          zPoNetTokenizer._pad)__name__ __module__ __qualname____doc__VOCAB_FILES_NAMESvocab_files_namesPRETRAINED_VOCAB_FILES_MAPpretrained_vocab_files_map&PRETRAINED_POSITIONAL_EMBEDDINGS_SIZESmax_model_input_sizesPRETRAINED_INIT_CONFIGURATIONpretrained_init_configurationrr"rrr'r r rintbooldictr.r,r,r,r-r0s. rN)r2typingrrrrrrtransformers.file_utilsr*transformers.models.bert.tokenization_bertr transformers.tokenization_utilsr r modelscope.utils.constantr modelscope.utils.loggerr logger VOCAB_FILEr3r5r7r9rr,r,r,r-s&