o ίi- @s`dZddlZddlZddlZddlZddlZddlmZmZddl m Z m Z m Z m Z ddlZddlZddlZdejd<dZedd Zed d Zd d ZddZddZddZddZddZdefddZddddZGdddeZ d-d!e ee efd"e!d#e!d$e!d%e d&e"f d'd(Z#d!e ee efd"e!d#e!d$e!d%e f d)d*Z$defd+d,Z%dS).zp CLIP tokenizer Copied from https://github.com/openai/CLIP. Originally MIT License, Copyright (c) 2021 OpenAI. N) lru_cachepartial)CallableListOptionalUnionfalseTOKENIZERS_PARALLELISMMcCstjtjtjtdS)Nzbpe_simple_vocab_16e6.txt.gz)ospathjoindirnameabspath__file__rrQ/home/ubuntu/.local/lib/python3.10/site-packages/core/vision_encoder/tokenizer.py default_bpesrcCstttdtddtttdtddtttdtdd}|dd}d }td D]}||vrI|||d ||d7}q3d d |D}tt||S) a: Returns list of utf-8 byte and a corresponding list of unicode strings. The reversible bpe codes work on unicode strings. This means you need a large # of unicode characters in your vocab if you want to avoid UNKs. When you're at something like a 10B token dataset you end up needing around 5K for decent coverage. This is a significant percentage of your normal, say, 32K bpe vocab. To avoid that, we want lookup tables between utf-8 bytes and unicode strings. And avoids mapping to whitespace/control characters the bpe code barfs on. !~¡¬®ÿNrcSsg|]}t|qSr)chr).0nrrr 6z$bytes_to_unicode..)listrangeordappenddictzip)bscsrbrrrbytes_to_unicodes     r*cCs6t}|d}|ddD] }|||f|}q |S)zReturn set of symbol pairs in a word. Word is represented as tuple of symbols (symbols being variable-length strings). rrN)setadd)wordpairs prev_charcharrrr get_pairs:s r1cCs"t|}tt|}|SN)ftfyfix_texthtmlunescapestriptextrrr basic_cleanFs r:cCstdd|}|}|S)N\s+ )resubr7r8rrrwhitespace_cleanLsr?cC tt|Sr2)canonicalize_textr:xrrr_clean_canonicalizeR rDcCstt|Sr2)r?r:lowerrBrrr _clean_lowerWsrGcCr@r2)r?r:rBrrr_clean_whitespace\rErHtypecCs4|dkrtS|dkr tS|dkrtSJd|d)N canonicalizerF whitespaceFzInvalid clean function (z).)rDrGrHrIrrr get_clean_fnasrM)keep_punctuation_exact_stringcCs`|dd}|r|dd||D}n |tddtj}|}t dd|}| S)aReturns canonicalized `text` (lowercase and punctuation removed). From: https://github.com/google-research/big_vision/blob/53f18caf27a9419231bbf08d3388b07671616d3d/big_vision/evaluators/proj/image_text/prompt_engineering.py#L94 Args: text: string to be canonicalized. keep_punctuation_exact_string: If provided, then this exact string kept. For example providing '{}' will keep any occurrences of '{}' (but will still remove '{' and '}' that appear separately). _r<css&|]}|tddtjVqdS)N) translatestr maketransstring punctuation)rpartrrr ys  z$canonicalize_text..rPr;) replacer splitrQrRrSrTrUrFr=r>r7)r9rNrrrrAls   rAc @seZdZededdfdedeeedeededef d d Z d d Z d dZ ddZ dde eeefdeedejfddZdS)SimpleTokenizerNrFrPbpe_pathadditional_special_tokenscontext_lengthcleanreduction_maskc st_ddjD_t|dd}|dd}dd|D}t t }|d d|D}|D] }| d |q;d d g} |rP| |7} | | tt|tt|_d djD_tt|tt|_dd| D_d | } t| dtj_tj_fdd| D_jd_jd_|_t|_ |rt!|_"dSd_"dS)NcSi|]\}}||qSrrrkvrrr z,SimpleTokenizer.__init__..utf-8 ricSsg|]}t|qSr)tuplerY)rmergerrrrsz,SimpleTokenizer.__init__..cSsg|]}|dqS)r)rrcrrrrr rPzz cSr`rrrarrrrdrecSsi|]}||qSrrrtrrrrd|z:|'s|'t|'re|'ve|'m|'ll|'d|[\p{L}]+|[\p{N}]|[^\s\p{L}\p{N}]+cg|]}j|qSrencoderrkselfrrrrer)#r* byte_encoderitems byte_decodergzipopenreaddecoderYr!valuesr$r extendr%r&r"lenrqdecoder bpe_rankscacher=compile IGNORECASEpat vocab_sizeall_special_ids sot_token_id eot_token_idr]rMclean_fnget_reduction_mask_fn reduction_fn) rsr[r\r]r^r_mergesvocabrispecial_tokensspecialrrrr__init__s@        zSimpleTokenizer.__init__c sj|jvr j|St|dd|ddf}t|}|s#|dS t|fddd}|jvr4nu|\}}g}d}|t|krz|||} |||| | }Wn|||dYn3|||kr|t|dkr||d|kr||||d 7}n ||||d7}|t|ksBt|}|}t|dkrnt|}q$d |}|j|<|S) NrjTcsj|tdS)Ninf)rgetfloat)pairrrrrrmz%SimpleTokenizer.bpe..)keyrrr<) rrhr1minrr}indexr|r$r ) rstokenr-r.bigramfirstsecondnew_wordijrrrrbpesH     ,     zSimpleTokenizer.bpecshg}|}tj|D]#}dfdd|dD}|fdd|dDq|S)NrPc3|]}j|VqdSr2)rt)rr)rrrrrWsz)SimpleTokenizer.encode..rfc3rr2rp)r bpe_tokenrrrrrWs  r<) rr=findallrr encoder|rrY)rsr9 bpe_tokensrrrrrrs  zSimpleTokenizer.encodecsDdfdd|D}tfdd|Djddddd }|S) NrPcror)r~)rrrrrrrrez*SimpleTokenizer.decode..cror)rv)rcrrrrrrerfrX)errorsrjr<)r bytearrayrzrX)rstokensr9rrrrrzszSimpleTokenizer.decodetextsreturncst|tr|g}|p j}|sJdjdur%j||jjjdSfdd|D}tjt ||tj d}t |D]"\}}t ||krR|d|}j|d<t |||dt |f<q=|S)aReturns the tokenized representation of given input string(s) Parameters ---------- texts : Union[str, List[str]] An input string or a list of input strings to tokenize context_length : int The context length to use; all CLIP models use 77 as the context length Returns ------- A two-dimensional tensor containing the resulting tokens, shape = [number of input strings, context_length] z!Please set a valid context lengthN)r]rr encode_fncs&g|]}jg|jgqSr)rrrrr9rrrrr sz,SimpleTokenizer.__call__..dtyper) isinstancerRr]rrrrtorchzerosr}long enumeratetensor)rsrr] all_tokensresultrrrrrr__call__s,        zSimpleTokenizer.__call__r2)__name__ __module__ __qualname__rDEFAULT_CONTEXT_LENGTHrRrrintrrrrzrr LongTensorrrrrrrZs8  ()  rZFrr]rrrshufflec sfdd|D}tjt||tjd}t|D]H\}} t| } t| } | |dkrH|d} tt| } | d| } |sB| } | | } | } |||df<| ||d| df<|||| df<q|S)Ncg|]}|qSrrrrrrr r z(random_mask_tokenize..rrrr)rrr}rrrrandpermmsort) rr]rrrrrrrr num_tokensnum_keepindicesrrrrandom_mask_tokenizes"    rc sfdd|D}tjt||tjd}t|D]7\}}t|} | |dkr:|d} td| | } || | | }|g||g}t|||dt|f<q|S)Ncrrrrrrrr<r z(simple_mask_tokenize..rrr)rrr}rrrandomrandintr) rr]rrrrrrrrr start_indexrrrsimple_mask_tokenize5s rcCs<|dvsJ|dkr tS|dkrtS|dkrttddSdS)zNChoose strategy for dropping (masking) tokens to achieve target context length)simplerrrrrT)rN)rrrrLrrrrLs r)F)&__doc__rwr5r rrT functoolsrrtypingrrrrr3regexr=renvironrrr*r1r:r?rDrGrHrRrMrAobjectrZrboolrrrrrrrsj