o ॵi`@s$ddlmZmZmZmZddlZddlZddlZddlZddl Z ddl Z ddl Z ddZ GdddeZ eeZddZd d ZGd d d eZGd ddeZGdddeZddZddZddZ zddlmZWn ey|ddZYnweddZddZGdddeZ dS) )absolute_importdivisionprint_functionunicode_literalsNc Cs8ddddddddd }|D] \}}|||}q|S) N-'zn'tz'mz don'tz'sz'vez're)z - z ' z n'tz 'mz do notz 'sz 'vez 're)itemsreplace)string replace_mpkvr`/home/ubuntu/.local/lib/python3.10/site-packages/modelscope/preprocessors/nlp/space/tokenizer.py clean_strings rc@s>eZdZgdfddZddZddZdd Zgfd d Zd S) TokenizerBertcsj|_|dkrsddd_|D]}|jvr&|dvr&dtjdj|<qdd jD_fd d |D}d _jtfd d|D7_t|jd_jD]}|jj vsiJd|dqYtjj _ dS|dkrddi_dd jD_fdd |D}t j |d}t j |d}t|||d_t|_tj_ dSt)Nrz [unused0]z [unused1])z[BOS]z[EOS])[PAD][UNK]z[unused]cSi|]\}}||qSrr.0r r rrr .z&Tokenizer.__init__..cg|] }j||qSrspec_convert_dictgetrtokselfrr 2sz&Tokenizer.__init__..rz[SEP]rz[CLS]z[MASK]c3s|] }|jvr|VqdSNspecial_tokensrxr!rr 7s  z%Tokenizer.__init__..) never_splitzspecial token 'z' is not in the vocabularyGPT2rzcSrrrrrrrrArcsg|] }|jvr|qSr)rrr!rrr#Es  z vocab.jsonz merges.txtr&)tokenizer_typerlenrspec_revert_dictr'tuple BertTokenizer _tokenizervocab vocab_sizeospathjoin GPT2Tokenizer num_specials ValueError)r" vocab_pathr'r-tokenr vocab_file merges_filerr!r__init__!sT      zTokenizer.__init__cCs |j|Sr%)r2tokenizer"textrrrr@Rs zTokenizer.tokenizecs`jdkrfdd|D}j|}|Sfdd|D}j|}fdd|D}|S)Nrcrrrrr!rrr#Wz3Tokenizer.convert_tokens_to_ids..crrrrr!rrr#[rCcsg|] }|jjqSrr9r4rir!rrr#])r-r2convert_tokens_to_ids)r"tokensidsrr!rrHUs   zTokenizer.convert_tokens_to_idscs`jdkrj|}fdd|D}|Sfdd|D}j|}fdd|D}|S)Nrcrrr/rrr!rrr#crCz3Tokenizer.convert_ids_to_tokens..csg|] }|jjqSrrDrEr!rrr#frGcrrrKrr!rrr#hrC)r-r2convert_ids_to_tokens)r"rJrIrr!rrL`s   zTokenizer.convert_ids_to_tokenscs|}tdkrtfdd|D}jdkr'd|dd}nd|}tfdd|Dd }t|}|S) Nrcsg|]}|vr|qSrrr) ignore_tokensrrr#oz$Tokenizer.decode..r z ##csg|]}jj|qSr)r2 byte_decoderrcr!rrr#t utf-8) rLr.setr-r7r bytearraydecoder)r"rJrMrIr r)rMr"rrXks     zTokenizer.decodeN)__name__ __module__ __qualname__r?r@rHrLrXrrrrrs 1  rcCsjt}d}t|ddd} |}|sn |}|||<|d7}qWd|S1s.wY|S)z*Loads a vocabulary file into a dictionary.rrrUencodingTN) collections OrderedDictopenreadlinestrip)r=r3indexreaderr<rrr load_vocabs  rgcCs|}|sgS|}|S)z@Runs basic whitespace cleaning and splitting on a piece of text.)rdsplit)rBrIrrrwhitespace_tokenizes ric@s:eZdZdZ    d ddZddZd d Zd d ZdS)r1z?Runs end-to-end tokenization: punctuation splitting + wordpieceTNr$cCstj|s td|t||_tdd|j D|_ ||_ |r,t ||d|_ t|jd|_|dur<||_dStd|_dS)aConstructs a BertTokenizer. Args: vocab_file: Path to a one-wordpiece-per-line vocabulary file do_lower_case: Whether to lower case the input Only has an effect when do_wordpiece_only=False do_basic_tokenize: Whether to do basic tokenization before wordpiece. max_len: An artificial maximum length to truncate tokenized sequences to; Effective maximum length is always the minimum of this value (if specified) and the underlying BERT model's sequence length. never_split: List of tokens which will never be split during tokenization. Only has an effect when do_wordpiece_only=False zCan't find a vocabulary file at path '{}'. To load the vocabulary from a Google pretrained model use `tokenizer = BertTokenizer.from_pretrained(PRETRAINED_MODEL_NAME)`cSsg|]\}}||fqSrr)rr rJrrrr#rTz*BertTokenizer.__init__.. do_lower_caser+)r3NmB)r5r6isfiler:formatrgr3r`rar ids_to_tokensdo_basic_tokenizeBasicTokenizerbasic_tokenizerWordpieceTokenizerwordpiece_tokenizerintmax_len)r"r=rkrvrpr+rrrr?s     zBertTokenizer.__init__cCsLg}|jr|j|D]}|j|D]}||qq |S|j|}|Sr%)rprrr@rtappend)r"rB split_tokensr< sub_tokenrrrr@s  zBertTokenizer.tokenizecCsHg}|D] }||j|qt||jkr"tdt||j|S)z7Converts a sequence of tokens into ids using the vocab.zToken indices sequence length is longer than the specified maximum sequence length for this BERT model ({} > {}). Running this sequence through BERT will result in indexing errors)rwr3r.rvloggerwarningrn)r"rIrJr<rrrrHs z#BertTokenizer.convert_tokens_to_idscCs"g}|D] }||j|q|S)z?Converts a sequence of ids in wordpiece tokens using the vocab.)rwro)r"rJrIrFrrrrLsz#BertTokenizer.convert_ids_to_tokens)TNTr$)rYrZr[__doc__r?r@rHrLrrrrr1s $ r1c@sNeZdZdZ  dddZddZdd Zd d Zd d ZddZ ddZ dS)rqzDRuns basic tokenization (punctuation splitting, lower casing, etc.).Tr$cCs||_||_dS)znConstructs a BasicTokenizer. Args: do_lower_case: Whether to lower case the input. Nrj)r"rkr+rrrr?s zBasicTokenizer.__init__cCsn||}||}t|}g}|D]}|jr%||jvr%|}||}|||qtd |}|S)zTokenizes a piece of text.rO) _clean_text_tokenize_chinese_charsrirkr+lower_run_strip_accentsextend_run_split_on_puncr7)r"rB orig_tokensrxr< output_tokensrrrr@s   zBasicTokenizer.tokenizecCsBtd|}g}|D]}t|}|dkrq ||q d|S)z$Strips accents from a piece of text.NFDMnrP) unicodedata normalizecategoryrwr7)r"rBoutputcharcatrrrrs    z!BasicTokenizer._run_strip_accentscCs||jvr|gSt|}d}d}g}|t|krC||}t|r)||gd}n|r0|gd}|d||d7}|t|ksdd|DS)z&Splits punctuation on a piece of text.rTFr_cSsg|]}d|qS)rP)r7r(rrrr#-z5BasicTokenizer._run_split_on_punc..)r+listr._is_punctuationrw)r"rBcharsrFstart_new_wordrrrrrrs$      z!BasicTokenizer._run_split_on_punccCsTg}|D] }t|}||r|d|||dq||qd|S)z)Adds whitespace around any CJK character.rOrP)ord_is_chinese_charrwr7r"rBrrcprrrr~/s      z&BasicTokenizer._tokenize_chinese_charscCs|dko|dk}|p|dko|dk}|p|dko|dk}|p%|dko%|dk}|p/|d ko/|d k}|p9|d ko9|d k}|pC|d koC|dk}|pM|dkoM|dk}|rRdSdS)z6Checks whether CP is the codepoint of a CJK character.iNii4iMiiߦii?i@ii iiiiiTFr)r"rtmprrrr<s zBasicTokenizer._is_chinese_charcCsXg}|D]"}t|}|dks|dkst|rqt|r!|dq||qd|S)zBPerforms invalid character removal and whitespace cleanup on text.rirOrP)r _is_control_is_whitespacerwr7rrrrr}Ss   zBasicTokenizer._clean_textN)Tr$) rYrZr[r|r?r@rrr~rr}rrrrrqs    rqc@s"eZdZdZd ddZddZdS) rszRuns WordPiece tokenization.rdcCs||_||_||_dSr%)r3 unk_tokenmax_input_chars_per_word)r"r3rrrrrr?ds zWordpieceTokenizer.__init__c Csg}t|D]m}t|}t||jkr||jqd}d}g}|t|kret|}d} ||krQd|||} |dkrAd| } | |jvrI| } n|d8}||ks0| durXd}n || |}|t|ks&|rn||jq||q|S)aTokenizes a piece of text into its word pieces. This uses a greedy longest-match-first algorithm to perform tokenization using the given vocabulary. For example: >>> input = "unaffable" >>> output = ["un", "##aff", "##able"] Args: text: A single token or whitespace separated tokens. This should have already been passed through `BasicTokenizer`. Returns: A list of wordpiece tokens. FrNrPz##r_T) rirr.rrwrr7r3r) r"rBrr<ris_badstart sub_tokensend cur_substrsubstrrrrr@is>       zWordpieceTokenizer.tokenizeN)rr)rYrZr[r|r?r@rrrrrsas  rscCs>|dks|dks|dks|dkrdSt|}|dkrdSdS)z1Checks whether `chars` is a whitespace character.rO   TZsF)rrrrrrrrs  rcCs8|dks |dks |dkrdSt|}|drdSdS)z.Checks whether `chars` is a control character.rrrFCT)rr startswithrrrrrs   rcCsxt|}|dko |dk}|p|dko|dk}|p|dko|dk}|p)|dko)|dk}|r.d St|}|d r:d Sd S) z2Checks whether `chars` is a punctuation character.!/:@[`{~TPF)rrrr)rrrrrrrrs  r) lru_cachecCsddS)NcSs|Sr%r)funcrrrszlru_cache..rrrrrrsrcstjddkr tnttttdtddtttdtddtttdtd d}|d d }d}td D]}||vrT|||d ||d7}q>fd d |D}tt ||S)a9 Returns list of utf-8 byte and a corresponding list of unicode strings. The reversible bpe codes work on unicode strings. This means you need a large # of unicode characters in your vocab if you want to avoid UNKs. When you're at something like a 10B token dataset you end up needing around 5K for decent coverage. This is a signficant percentage of your normal, say, 32K bpe vocab. To avoid that, we want lookup tables between utf-8 bytes and unicode strings. And avoids mapping to whitespace/control characters the bpe code barfs on. r!~r_¡¬®ÿNcsg|]}|qSrr)rn_chrrrr#sz$bytes_to_unicode..) sys version_infounichrchrrrangerrwdictzip)bscsrbrrrbytes_to_unicodes.        rcCs6t}|d}|ddD] }|||f|}q |S)zReturn set of symbol pairs in a word. Word is represented as tuple of symbols (symbols being variable-length strings). rr_N)rVadd)wordpairs prev_charrrrr get_pairss rc@sbeZdZdZ   dddZddZdd Zd d Zd d ZddZ dddZ ddZ ddZ dS)r8zF GPT-2 BPE tokenizer. Peculiarities: - Byte-level BPE r NcCs|dur|ntd|_tt|dd|_dd|jD|_||_t |_ dd|j D|_ t|dd ddd }d d |D}tt|tt||_i|_td |_i|_i|_||dS) NrlrUr]cSrrrrrrrrrz*GPT2Tokenizer.__init__..cSrrrrrrrrrrr_rcSsg|]}t|qSr)r0rh)rmergerrrr#rNz*GPT2Tokenizer.__init__..zJ's|'t|'re|'ve|'m|'ll|'d| ?\p{L}+| ?\p{N}+| ?[^\s\p{L}\p{N}]+|\s+(?!\S)|\s+)rurvjsonloadrbencoderrdecodererrorsr byte_encoderrQreadrhrrrr. bpe_rankscacherecompilepatr'special_tokens_decoderset_special_tokens)r"r=r>rr'rvbpe_data bpe_mergesrrrr?s zGPT2Tokenizer.__init__cCst|jt|jSr%)r.rr'r!rrr__len__)szGPT2Tokenizer.__len__cs\|s i_i_dStfddt|D_ddjD_tdjdS)z Add a list of additional tokens to the encoder. The additional tokens are indexed starting from the last index of the current vocabulary in the order of the `special_tokens` list. Nc3s&|]\}}|tj|fVqdSr%)r.r)rrFr r!rrr*5sz3GPT2Tokenizer.set_special_tokens..cSrrrrrrrr7rz4GPT2Tokenizer.set_special_tokens..zSpecial tokens {})r'rr enumeraterrzinforn)r"r'rr!rr,s  z GPT2Tokenizer.set_special_tokensc sX|jvr j|St|}t|}|s|S t|fddd}|jvr'ny|\}}g}d}|t|krz|||} |||| | }WntyZ|||dYn4w|||kr}|t|dkr}||d|kr}| |||d7}n | |||d7}|t|ks5t|}|}t|dkrnt|}qd |}|j|<|S) NTcsj|tdS)Ninf)rrfloat)pairr!rrrHsz#GPT2Tokenizer.bpe..)keyrr_rrO) rr0rminrr.rer Exceptionrwr7) r"r<rrbigramfirstsecondnew_wordrFjrr!rbpe=sV            zGPT2Tokenizer.bpecs^g}tj|D]#}dfdd|D}|dkrq |dd|dDq |S)z Tokenize a string. rPc3s,|]}t|jvrjt|VqdSr%)rr)rrr!rrr*ls  z)GPT2Tokenizer.tokenize..css|]}|VqdSr%r)r bpe_tokenrrrr*ps rO)rfindallrr7rrrh)r"rB bpe_tokensr<rr!rr@hs  zGPT2Tokenizer.tokenizecCsg}t|t}tjddkot|t}|s|r(||jvr!|j|S|j|dS|D]}||jvr:||j|q*||j|dq*t ||j krXt d t ||j |S)z9 Converts a sequence of tokens into ids using the vocab. rrzToken indices sequence length is longer than the specified maximum sequence length for this OpenAI GPT model ({} > {}). Running this sequence through the model will result in indexing errors) isinstancestrrrunicoder'rrrwr.rvrzr{rn)r"rIrJpython_version_3python_version_2r<rrrrHts$    z#GPT2Tokenizer.convert_tokens_to_idsFcCsBg}|D]}||jvr|s||j|q||j|q|S)z9Converts a sequence of ids in BPE tokens using the vocab.)rrwr)r"rJskip_special_tokensrIrFrrrrLs z#GPT2Tokenizer.convert_ids_to_tokenscCs|||Sr%)rHr@rArrrencodeszGPT2Tokenizer.encodecs>dfdd|D}tfdd|Djdjd}|S)NrPcg|]}j|qSr)r)rr<r!rrr#rz(GPT2Tokenizer.decode..crr)rQrRr!rrr#rrU)r)r7rWrXr)r"rIrBrr!rrXs zGPT2Tokenizer.decode)r NN)F) rYrZr[r|r?rrrr@rHrLrrXrrrrr8 s +  r8)! __future__rrrrr`loggingr5rrrregexrrobjectr getLoggerrYrzrgrir1rqrsrrr functoolsr ImportErrorrrr8rrrrs<i  Fv<