o ̳iC@sddlmZmZmZmZmZmZddlZddl m Z dZ Gddde Z dee effdd Zd eed fdeeeeffd d ZdedefddZdedeeeeffddZdS))AnyDictListMappingSetTupleN) BaseTokenizerzc @seZdZdZddededefddZd ed eefd d Z d eed efddZ dde ee fded e ee ffddZ ded efddZdS) CLIPTokenizerar Text tokenizer for CLIP. Based on the official implementation here: https://github.com/openai/CLIP/blob/main/clip/simple_tokenizer.py Args: path (str): the path to the CLIP merges file max_seq_len (int): the context length (all CLIP models use 77) truncate (bool): whether to truncate the text when longer than max_seq_len MTpath max_seq_lentruncatecCs||_||_t|_dd|jD|_t|}t|j}| dd|D| dd|D| ddgddt |D|_ d d|j D|_ d dt |D|_ td tj|_|j d|_|j d|_|j|_ddd |_dS) NcSi|]\}}||qSr.0kvrrT/home/ubuntu/.local/lib/python3.10/site-packages/torchtune/models/clip/_tokenizer.py !z*CLIPTokenizer.__init__..cSsg|]}|tqSr) WORD_BOUNDARY)rrrrr &z*CLIPTokenizer.__init__..cSsg|]}d|qS))join)rmergerrrr'r<|startoftext|> <|endoftext|>cSrrr)riwordrrrr*rcSrrrrrrrr+rcSrrr)rrrrrrr,rz[<\|startoftext\|>|<\|endoftext\|>|'s|'t|'re|'ve|'m|'ll|'d|[\p{L}]+|[\p{N}]|[^\s\p{L}\p{N}]+)rr)r r _bytes_to_unicode byte_encoderitems byte_decoder _load_mergeslistvaluesextend enumerateencoderdecoder bpe_ranksrecompile IGNORECASEpat sot_token eot_token pad_tokencache)selfr r r mergesvocabrrr__init__s,   zCLIPTokenizer.__init__textreturncst|}jg}tj|D],}dfdd|dD}|fdd | dDt |j kr=nq| jt |j kr^jsRJd|dj }j|d <|S) z Given a string, return the encoded list of token ids. Args: text (str): The text to encode. Returns: List[int]: The encoded list of token ids. rc3|]}j|VqdSN)r")rbr5rr Jsz'CLIPTokenizer.encode..utf-8c3r;r<)r*)r bpe_tokenr>rrr?Ks   zWTokenized text is larger than the maximum sequence length but truncate is set to False.N) _clean_textlowerr1r-findallr0rencoder(_bpesplitlenr appendr2r )r5r9tokenstokenrr>rrG<s$   zCLIPTokenizer.encoderLcs@dfdd|D}tfdd|DjdddtdS) z Given a list of token ids, return the decoded text, optionally including special tokens. Args: tokens (List[int]): The list of token ids to decode. Returns: str: The decoded text. rcg|]}j|qSr)r+)rrMr>rrrfrz(CLIPTokenizer.decode..crNr)r$)rcr>rrrhrr@replace)errorsrB)r bytearraydecoderPr)r5rLr9rr>rrS\s  zCLIPTokenizer.decodeFsample inferencecCs|d}|||d<|S)a] Tokenize the "text" field in the sample. Args: sample (Mapping[str, Any]): A sample with a "text" field containing a string to tokenize inference (bool): Unused by this tokenizer Returns: Mapping[str, Any]: The sample with added "tokens" field and the "messages" field removed. r9rL)poprG)r5rTrUr9rrr__call__ms zCLIPTokenizer.__call__rMc sv|jvr j|St|dkr|tSt|dd|dtf}t|} t|fddd}|jvr8nw|\}}g}d}|t|krz|||} |||| | }Wnt yk|||dYn4w|||kr|t|d kr||d |kr| |||d7}n | |||d 7}|t|ksFt|}t|d krnt|}q(d |}|j|<|S) z@ Performs byte-pair encoding on a single token. NrCTcsj|tdS)Ninf)r,getfloat)pairr>rrsz$CLIPTokenizer._bpe..)keyrrB) r4rJrtuple _get_pairsminr,indexr( ValueErrorrKr) r5rMr pairsbigramfirstsecondnew_wordrjrr>rrH~sH       ,    & zCLIPTokenizer._bpeN)r T)F)__name__ __module__ __qualname____doc__strintboolr8rrGrSrrrWrHrrrrr s      r r:cCstttdtddtttdtddtttdtdd}|dd}d }td D]}||vrI|||d ||d7}q3d d |D}tt||S) zQ Returns list of utf-8 byte and a corresponding list of unicode strings. !~r_¡¬®ÿNrcSsg|]}t|qSr)chr)rnrrrrrz%_bytes_to_unicode..)r&rangeordrKdictzip)bscsrzr=rrrr!s    r!r .cCs6t}|d}|ddD] }|||f|}q |S)z Return set of symbol pairs in a word. Word is represented as tuple of symbols (symbols being variable-length strings). rr_N)setadd)r re prev_charcharrrrras rar9cCs |ddS)zI Minimal version of CLIP's text cleaning via the `ftfy` package. u’')rP)r9rrrrDs rDr cCszg}t|dd+}t|D]\}}|}|dkr|ds |s!q |t|q Wd|S1s6wY|S)Nr@)encodingrz #version:)openr)strip startswithrKr`rI)r r6frlinerrrr%s r%)typingrrrrrrregexr-.torchtune.modules.transforms.tokenizers._utilsrrr rpror!rarDr%rrrrs  -& "