o eiC_ @s dZddlZddlZddlZddlZddlmZddlmZe e Z dddZ d d Z Gd d d eZ d ZdZededdddddf ZeddeejejBejBZedZeeejejBejBZedZd+ddZd,d d!ZGd"d#d#Zd$d%Zd&d'Zd-d)d*Z d gZ!dS).z!Tokenization classes for BERTweetN)PreTrainedTokenizer)logging vocab.txt bpe.codes) vocab_file merges_filecCs>t}|d}|ddD] }|||f|}q t|}|S)z Return set of symbol pairs in a word. Word is represented as tuple of symbols (symbols being variable-length strings). rN)setadd)wordpairs prev_charcharrp/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/bertweet/tokenization_bertweet.py get_pairs#src seZdZdZeZ        d%fdd Zed d Zd d Z ddZ ddZ ddZ ddZ ddZddZddZd&dededBdeed ffd!d"Zd#d$ZZS)'BertweetTokenizera Constructs a BERTweet tokenizer, using Byte-Pair-Encoding. This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: vocab_file (`str`): Path to the vocabulary file. merges_file (`str`): Path to the merges file. normalization (`bool`, *optional*, defaults to `False`): Whether or not to apply a normalization preprocess. bos_token (`str`, *optional*, defaults to `""`): The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token. When building a sequence using special tokens, this is not the token that is used for the beginning of sequence. The token used is the `cls_token`. eos_token (`str`, *optional*, defaults to `""`): The end of sequence token. When building a sequence using special tokens, this is not the token that is used for the end of sequence. The token used is the `sep_token`. sep_token (`str`, *optional*, defaults to `""`): The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for sequence classification or for a text and a question for question answering. It is also used as the last token of a sequence built with special tokens. cls_token (`str`, *optional*, defaults to `""`): The classifier token which is used when doing sequence classification (classification of the whole sequence instead of per-token classification). It is the first token of the sequence when built with special tokens. unk_token (`str`, *optional*, defaults to `""`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. pad_token (`str`, *optional*, defaults to `""`): The token used for padding, for example when batching sequences of different lengths. mask_token (`str`, *optional*, defaults to `""`): The token used for masking values. This is the token used when training this model with masked language modeling. This is the token which the model will try to predict. Fc  sZz ddlm} | |_Wntytdd|_Ynw||_||_i|_d|jt |<d|jt | <d|jt |<d|jt |<| |dd|j D|_ t |d d } | d dd }Wdn1snwYd d|D}tt|tt||_i|_||_t|_ddd|_tjd||||||| | dddd | dS)Nr)demojizezsemoji is not installed, thus not converting emoticons or emojis into text. Install emoji: pip3 install emoji==0.6.0r rcSsi|]\}}||qSrr).0kvrrr z.BertweetTokenizer.__init__..utf-8encoding cSs g|] }t|ddqS)Nr$)tuplesplit)rmergerrr s z.BertweetTokenizer.__init__..'z...)u’u… all_zerosTcls_double_sep) normalization bos_token eos_token sep_token cls_token unk_token pad_token mask_tokentoken_type_ids_pattern%token_type_ids_include_special_tokensspecial_tokens_patternr)emojir demojizer ImportErrorloggerwarningrrencoderstr add_from_fileitemsdecoderopenreadr&dictziprangelen bpe_rankscacher,TweetTokenizertweetPreprocessorspecial_punctssuper__init__)selfrrr,r-r.r/r0r1r2r3kwargsr merges_handlemerges __class__rrrMhsT       zBertweetTokenizer.__init__cCs t|jSN)rFr<rNrrr vocab_sizes zBertweetTokenizer.vocab_sizecCst|jfi|jSrT)rCr<added_tokens_encoderrUrrr get_vocabszBertweetTokenizer.get_vocabc s|jvr j|St|}tt|dd|ddg}t|}|s'|S t|fddd}|jvr8ny|\}}g}d}|t|krz|||} Wnty`| ||dYn?w| ||| | }|||kr|t|dkr||d|kr| |||d 7}n | |||d7}|t|ksFt|}|}t|dkrnt|}q(d |}|dd }|j|<|S) Nr$zTcsj|tdS)Ninf)rGgetfloat)pairrUrrsz'BertweetTokenizer.bpe..keyrr r@@ ) rHr%listrminrGrFindex ValueErrorextendappendjoin) rNtokenr r bigramfirstsecondnew_wordijrrUrbpesN  "   ,      zBertweetTokenizer.bpecCsH|jr||}g}td|}|D]}|t||dq|S)zTokenize a string.z\S+\n? )r,normalizeTweetrefindallrfrbrpr&)rNtext split_tokenswordsrirrr _tokenizes  zBertweetTokenizer._tokenizecsjD] }||j|}qj|}dfdd|D}|ddddddd d d d }|d ddddddddddd}|dddddddd }d|S)!z' Normalize a raw Tweet rqcsg|]}|qSr)normalizeToken)rrirUrrr(rz4BertweetTokenizer.normalizeTweet..zcannot zcan not zn't z n't zn 't zca n'tzcan'tzai n'tzain'tz'm z 'm z're z 're z's z 's z'll z 'll z'd z 'd z've z 've z p . m .z p.m.z p . m z p.m z a . m .z a.m.z a . m z a.m )rKreplacerJtokenizerhr&)rNtweetpuncttokens normTweetrrUrrrs.     z BertweetTokenizer.normalizeTweetcCsj|}|dr dS|ds|drdSt|dkr3||jvr'|j|S|jdur1||S|S|S)z- Normalize tokens in a Tweet @z@USERhttpwwwHTTPURLr N)lower startswithrFrKr8)rNrilowercased_tokenrrrrys      z BertweetTokenizer.normalizeTokencCs|j||j|jS)z0Converts a token (str) in an id using the vocab.)r<rZr1)rNrirrr_convert_token_to_idsz&BertweetTokenizer._convert_token_to_idcCs|j||jS)z=Converts an index (integer) in a token (str) using the vocab.)r@rZr1)rNrdrrr_convert_id_to_tokensz&BertweetTokenizer._convert_id_to_tokencCsd|dd}|S)z:Converts a sequence of tokens (string) in a single string.rqr`)rhrzstrip)rNr~ out_stringrrrconvert_tokens_to_string#sz*BertweetTokenizer.convert_tokens_to_stringNsave_directoryfilename_prefixreturn.c CsBtj|std|ddSt|di}|r|dnd}tj|||dd}t|d d d '}t |j d d dD]\}}|dkrS| |d|dq@Wdn1s^wYtj|||dd} t| d d d } | ddt |j dd dDWd|| fS1swY|| fS)zF Save the vocabulary and merges files to a directory. zVocabulary path (z) should be a directoryrvocab_files_names-rrrwr r!cS|dSNr rkvrrrr]=z3BertweetTokenizer.save_vocabulary..r^rqr#Nrrcss"|] \}}d|dVqdS)rqr#N)rh)r bpe_tokens token_indexrrr Es  z4BertweetTokenizer.save_vocabulary..cSrrrrrrrr]Gr)ospathisdirr:errorgetattrrhrZrAsortedr<r?write writelinesrG) rNrrrprefixrfritoken_id merge_filewriterrrrsave_vocabulary.s,    z!BertweetTokenizer.save_vocabularyc Cst|trCz!t|ddd}||WdWdS1swYWdSty4}z|d}~wtyBtd|dw|}|D]!}|}| d}|dkr\t d |d|}t |j |j |<qIdS) zi Loads a pre-existing dictionary from a text file and adds its symbols to this instance. rr r!NzIncorrect encoding detected in z, please rebuild the datasetrqr$z5Incorrect dictionary format, expected ' ') isinstancer=rAr>FileNotFoundError UnicodeError Exception readlinesrrfindrerFr<) rNrfdfnfelineslineTmplineidxr rrrr>Ls.      zBertweetTokenizer.add_from_file)FrrrrrrrrT)__name__ __module__ __qualname____doc__VOCAB_FILES_NAMESrrMpropertyrVrXrprxrrryrrrr=r%rr> __classcell__rrrRrr3s02@ , "$ rac (?: [<>]? [:;=8] # eyes [\-o\*\']? # optional nose [\)\]\(\[dDpP/\:\}\{@\|\\] # mouth | [\)\]\(\[dDpP/\:\}\{@\|\\] # mouth [\-o\*\']? # optional nose [:;=8] # eyes [<>]? | <3 # heart )u # Capture 1: entire matched URL (?: https?: # URL protocol and colon (?: /{1,3} # 1-3 slashes | # or [a-z0-9%] # Single letter or digit or '%' # (Trying not to match e.g. "URI::Escape") ) | # or # looks like domain name followed by a slash: [a-z0-9.\-]+[.] (?:[a-z]{2,13}) / ) (?: # One or more: [^\s()<>{}\[\]]+ # Run of non-space, non-()<>{}[] | # or \([^\s()]*?\([^\s()]+\)[^\s()]*?\) # balanced parens, one level deep: (...(...)...) | \([^\s]+?\) # balanced parens, non-recursive: (...) )+ (?: # End with: \([^\s()]*?\([^\s()]+\)[^\s()]*?\) # balanced parens, one level deep: (...(...)...) | \([^\s]+?\) # balanced parens, non-recursive: (...) | # or [^\s`!()\[\]{};:'".,<>?«»“”‘’] # not a space or one of these punct chars ) | # OR, the following to match naked domains: (?: (?\s]+>z [\-]+>|<[\-]+z (?:@[\w_]+)z(?:\#+[\w_]+[\w\'_\-]*[\w_]+)z#[\w.+-]+@[\w-]+\.(?:[\w-]\.?)+[\w-]a (?:[^\W\d_](?:[^\W\d_]|['\-_])+[^\W\d_]) # Words with apostrophes or dashes. | (?:[+\-]?\d+[,/.:-]\d+[+\-]?) # Numbers, including fractions, decimals. | (?:[\w_]+) # Words without apostrophes or dashes. | (?:\.(?:\s*\.){1,}) # Ellipsis dots. | (?:\S) # Everything else that isn't whitespace. z(%s)|z([^a-zA-Z0-9])\1{3,}z&(#?(x?))([^&;\s]+);strictcCs&|durd}t|tr|||S|S)Nr )rbytesdecode)rur"errorsrrr_str_to_unicodes   rrTr cs fdd}t|t||S)u Remove entities from text by converting them to their corresponding unicode character. Args: text: A unicode string or a byte string encoded in the given *encoding* (which defaults to 'utf-8'). keep (list): List of entity names which should not be replaced. This supports both numeric entities (`&#nnnn;` and `&#hhhh;`) and named entities (such as ` ` or `>`). remove_illegal (bool): If `True`, entities that can't be converted are removed. Otherwise, entities that can't be converted are kept "as is". Returns: A unicode string with the entities removed. See https://github.com/scrapy/w3lib/blob/master/w3lib/html.py Examples: ```python >>> from nltk.tokenize.casual import _replace_html_entities >>> _replace_html_entities(b"Price: £100") 'Price: \xa3100' >>> print(_replace_html_entities(b"Price: £100")) Price: £100 ```c s|d}|dr=z'|drt|d}nt|d}d|kr%dkr0nn t|fdWSWnty<d}Ynw|vrF|d Stjj|}|durbzt |WStt fyaYnwrfd S|d S) Nrr r cp1252rr) groupintrrrehtmlentitiesname2codepointrZchr OverflowError)match entity_bodynumberkeepremove_illegalrr_convert_entity;s,        z/_replace_html_entities.._convert_entity)ENT_REsubr)rurrr"rrrr_replace_html_entitiessrc@s"eZdZdZd ddZddZdS) rIa Examples: ```python >>> # Tokenizer for tweets. >>> from nltk.tokenize import TweetTokenizer >>> tknzr = TweetTokenizer() >>> s0 = "This is a cooool #dummysmiley: :-) :-P <3 and some arrows < > -> <--" >>> tknzr.tokenize(s0) ['This', 'is', 'a', 'cooool', '#dummysmiley', ':', ':-)', ':-P', '<3', 'and', 'some', 'arrows', '<', '>', '->', '<--'] >>> # Examples using *strip_handles* and *reduce_len parameters*: >>> tknzr = TweetTokenizer(strip_handles=True, reduce_len=True) >>> s1 = "@remy: This is waaaaayyyy too much for you!!!!!!" >>> tknzr.tokenize(s1) [':', 'This', 'is', 'waaayyy', 'too', 'much', 'for', 'you', '!', '!', '!'] ```TFcCs||_||_||_dSrT preserve_case reduce_len strip_handles)rNrrrrrrrMrs zTweetTokenizer.__init__cCsRt|}|jr t|}|jrt|}td|}t|}|j s'dd|D}|S)z Args: text: str Returns: list(str) A tokenized list of strings; concatenating this list returns the original string if `preserve_case=False` \1\1\1cSs"g|] }t|r |n|qSr) EMOTICON_REsearchr)rxrrrr(s"z+TweetTokenizer.tokenize..) rrremove_handlesrreduce_lengtheningHANG_RErWORD_RErtr)rNru safe_textrwrrrr{ws   zTweetTokenizer.tokenizeNTFF)rrrrrMr{rrrrrI^s  rIcCtd}|d|S)za Replace repeated character sequences of length 3 or greater with sequences of length 3. z (.)\1{2,}rregexcompilerrupatternrrrrs  rcCr)z4 Remove Twitter username handles from text. zv(?sN   >&.$0    @8