o ߥiF@sdZddlZddlZddlZddlmZddlZddlZddl Z eddZ eddZ dd Z d d Z d d ZGdddeZdddZdS)z CLIP Tokenizer.N) lru_cachecCstjtjtjtdS)Nzbpe_simple_vocab_16e6.txt.gz)ospathjoindirnameabspath__file__r r `/home/ubuntu/.local/lib/python3.10/site-packages/modelscope/models/multi_modal/gemm/tokenizer.py default_bpesr cCstttdtddtttdtddtttdtdd}|dd}d }td D]}||vrI|||d ||d7}q3d d |D}tt||S) a9 Returns list of utf-8 byte and a corresponding list of unicode strings. The reversible bpe codes work on unicode strings. This means you need a large # of unicode characters in your vocab if you want to avoid UNKs. When you're at something like a 10B token dataset you end up needing around 5K for decent coverage. This is a signficant percentage of your normal, say, 32K bpe vocab. To avoid that, we want lookup tables between utf-8 bytes and unicode strings. And avoids mapping to whitespace/control characters the bpe code barfs on. !~¡¬®ÿNrcSsg|]}t|qSr )chr).0nr r r 1z$bytes_to_unicode..)listrangeordappenddictzip)bscsrbr r r bytes_to_unicodes,       r"cCs6t}|d}|ddD] }|||f|}q |S)zReturn set of symbol pairs in a word. Word is represented as tuple of symbols (symbols being variable-length strings). rrN)setadd)wordpairs prev_charcharr r r get_pairs5s r)cCs"t|}tt|}|SN)ftfyfix_texthtmlunescapestriptextr r r basic_cleanAs r2cCstdd|}|}|S)Nz\s+ )resubr/r0r r r whitespace_cleanGsr6c@s8eZdZefdefddZddZddZdd Zd S) SimpleTokenizerbpe_pathcCst|_dd|jD|_t|dd}|dd}dd|D}t t }|d d|D}|D] }| d |q;| d d gtt|tt||_d d|jD|_tt|tt||_d d d|_tdtj|_dS)NcSi|]\}}||qSr r rkvr r r Qz,SimpleTokenizer.__init__..utf-8 ricSsg|]}t|qSr )tuplesplit)rmerger r r rTsz,SimpleTokenizer.__init__..cSsg|]}|dqS)r )rr<r r r rVr<|startoftext|> <|endoftext|>cSr9r r r:r r r r=[r>)rFrGz[<\|startoftext\|>|<\|endoftext\|>|'s|'t|'re|'ve|'m|'ll|'d|[\p{L}]+|[\p{N}]|[^\s\p{L}\p{N}]+)r" byte_encoderitems byte_decodergzipopenreaddecoderBrvaluesrrextendrrrlenencoderdecoder bpe_rankscacher4compile IGNORECASEpat)selfr8mergesvocabrCr r r __init__Os(  zSimpleTokenizer.__init__c s|jvr j|St|dd|ddf}t|}|s#|dSg} t|fddd}|jvr6n|\}}g}d} | t|krz||| } ||| | | } Wn tyw} z| | ||| dWYd} ~ n8d} ~ ww|| |kr| t|dkr|| d|kr| ||| d 7} n | || | d7} | t|ksDt|}|}t|dkrnt|}q&t|d krt |dd |}|j|<|S) NrDTcsj|tdS)Ninf)rTgetfloat)pairrYr r qsz%SimpleTokenizer.bpe..)keyrrdr3) rUrAr)minrTrQindexrP Exceptionrprintr) rYtokenr%r& error_listbigramfirstsecondnew_wordijerrr rbr bpees`               zSimpleTokenizer.bpecsng}tt|}tj|D]#}dfdd|dD}|fdd | dDq|S)NrEc3|]}j|VqdSr*)rH)rr!rbr r z)SimpleTokenizer.encode..r?c3rur*)rR)r bpe_tokenrbr r rvrwr3) r6r2lowerr4findallrXrencoderPrtrB)rYr1 bpe_tokensrkr rbr r{s zSimpleTokenizer.encodecsDdfdd|D}tfdd|Djddddd }|S) NrEcg|]}j|qSr )rS)rrkrbr r rr>z*SimpleTokenizer.decode..cr}r )rJ)rcrbr r rr>r?replace)errorsrDr3)r bytearrayrNr)rYtokensr1r rbr rNszSimpleTokenizer.decodeN) __name__ __module__ __qualname__r strr\rtr{rNr r r r r7Ms / r7MTcst|tr|g}jdjdfdd|D}tjt||tjd}t|D]0\}}t||krO|rC|d|}|d<n td||d |t |||dt|f<q,|S) a Returns the tokenized representation of given input string(s) Parameters ---------- texts : Union[str, List[str]] An input string or a list of input strings to tokenize context_length : int The context length to use; all CLIP models use 77 as the context length truncate: bool Whether to truncate the text in case its encoding is longer than the context length Returns ------- A two-dimensional tensor containing the resulting tokens, shape = [number of input strings, context_length]. We return LongTensor when torch version is <1.8.0, since older index_select requires indices to be long. rFrGcs"g|] }g|gqSr )r{)rr1 eot_token sot_token tokenizerr r rsz!clip_tokenize..)dtypeNr]zInput z is too long for context length ) isinstancerrRtorchzerosrQint enumerate RuntimeErrortensor)rtextscontext_lengthtruncate all_tokensresultrqrr rr clip_tokenizes$      r)rT)__doc__rKr-r functoolsrr+regexr4rr r"r)r2r6objectr7rr r r r s"    X