o ei, @sUdZddlmZmZddlmZeeZdZ dZ dZ dZ dZ d Zd Ze d e d e d ede dediZeeefed<ddeDZeeefed<GdddeZdgZdS)z Tokenization classes for CANINE.) AddedTokenPreTrainedTokenizer)loggingiiiiiiz[CLS]z[SEP]z[BOS]z[MASK]z[PAD]z [RESERVED]SPECIAL_CODEPOINTScCi|]\}}||qSr).0 codepointnamerrl/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/canine/tokenization_canine.py 4r SPECIAL_CODEPOINTS_BY_NAMEcseZdZdZgdZeeeeeeeeeeee ddffdd Z e de fdd Z d d Zd edeefd dZdede fddZde defddZddZZS)CanineTokenizera Construct a CANINE tokenizer (i.e. a character splitter). It turns text into a sequence of characters, and then converts each character into its Unicode code point. [`CanineTokenizer`] inherits from [`PreTrainedTokenizer`]. Refer to superclass [`PreTrainedTokenizer`] for usage examples and documentation concerning parameters. Args: model_max_length (`int`, *optional*, defaults to 2048): The maximum sentence length the model accepts. ) input_idsattention_masktoken_type_idsFic  s t|tr t|dddn|}t|trt|dddn|}t|tr(t|dddn|}t|tr6t|dddn|}t|trDt|dddn|}t|trRt|dddn|}i|_tD] \} } | |j| <q[dd|jD|_t|_t |j|_ t j d ||||||||dddd | dS) NF)lstriprstripTcSrrr)r r r rrr r bs z,CanineTokenizer.__init__.. all_zeroscls_sep) bos_token eos_token sep_token cls_token pad_token mask_tokenadd_prefix_spacemodel_max_lengthtoken_type_ids_pattern%token_type_ids_include_special_tokensspecial_tokens_patternr) isinstancestrr_special_codepointsritems_special_codepoint_stringsUNICODE_VOCAB_SIZE_unicode_vocab_sizelen_num_special_tokenssuper__init__) selfrrrrrrrrkwargsr r  __class__rr r-Gs:    zCanineTokenizer.__init__returncCs|jS)N)r))r.rrr vocab_sizexszCanineTokenizer.vocab_sizecCs$ddt|jD}||j|S)NcSsi|]}t||qSr)chr)r irrr r }rz-CanineTokenizer.get_vocab..)ranger3updateadded_tokens_encoder)r.vocabrrr get_vocab|s zCanineTokenizer.get_vocabtextcCst|S)z5Tokenize a string (i.e. perform character splitting).)list)r.r;rrr _tokenizeszCanineTokenizer._tokenizetokencCs*zt|WStytd|dw)zaConverts a token (i.e. a Unicode character) in an id (i.e. its integer Unicode code point value).zinvalid token: '')ord TypeError ValueError)r.r>rrr _convert_token_to_ids   z$CanineTokenizer._convert_token_to_idindexcCs:z|tvr t|WSt|WStytd|w)z Converts a Unicode code point (integer) in a token (str). In case it's a special code point, convert to human-readable format. z invalid id: )rr4rArB)r.rDrrr _convert_id_to_tokens   z$CanineTokenizer._convert_id_to_tokencCs d|S)N)join)r.tokensrrr convert_tokens_to_strings z(CanineTokenizer.convert_tokens_to_string)__name__ __module__ __qualname____doc__model_input_namesr4CLSSEPPADMASKr-propertyintr3r:r$r<r=rCrErI __classcell__rrr0r r7s& 1 rN)rMtokenization_pythonrrutilsr get_loggerrJloggerr(rQrOrPBOSrRRESERVEDrdictrTr$__annotations__r&rr__all__rrrr s*    " e