o eiJ/@sbdZddlZddlZddlmZddlmZeeZ dddZ d d Z Gd d d eZ d gZ dS) z Tokenization classes for BioGPT.N)PreTrainedTokenizer)loggingz vocab.jsonz merges.txt) vocab_file merges_filecCs6t}|d}|ddD] }|||f|}q |S)z Return set of symbol pairs in a word. word is represented as tuple of symbols (symbols being variable-length strings) rN)setadd)wordpairs prev_charcharrl/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/biogpt/tokenization_biogpt.py get_pairss rc seZdZdZeZddgZ     d.fdd Zed d Z d d Z ddZ ddZ ddZ d/ddZddZddZddZ d0deedeedBd eefd!d"Z d1deedeedBd#ed eeffd$d% Zd0d&ed'edBd eefd(d)Zd*d+Zd,d-ZZS)2BioGptTokenizera: Construct an FAIRSEQ Transformer tokenizer. Moses tokenization followed by Byte-Pair Encoding. This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: vocab_file (`str`): Path to the vocabulary file. merges_file (`str`): Merges file. unk_token (`str`, *optional*, defaults to `""`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. bos_token (`str`, *optional*, defaults to `""`): The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token. When building a sequence using special tokens, this is not the token that is used for the beginning of sequence. The token used is the `cls_token`. eos_token (`str`, *optional*, defaults to `""`): The end of sequence token. When building a sequence using special tokens, this is not the token that is used for the end of sequence. The token used is the `sep_token`. sep_token (`str`, *optional*, defaults to `""`): The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for sequence classification or for a text and a question for question answering. It is also used as the last token of a sequence built with special tokens. pad_token (`str`, *optional*, defaults to `""`): The token used for padding, for example when batching sequences of different lengths. input_idsattention_maskc  szddl} Wn tytdwd|_| |_i|_i|_ t|dd} t| |_ Wdn1s6wYdd|j D|_ t|dd} | ddd } Wdn1sbwYd d | D} tt| tt| |_i|_tjd |||||d |dS)NrzqYou need to install sacremoses to use BioGptTokenizer. See https://pypi.org/project/sacremoses/ for installation.enutf-8encodingcSsi|]\}}||qSrr).0kvrrr vsz,BioGptTokenizer.__init__.. cSs g|] }t|ddqS)N)tuplesplit)rmergerrr y z,BioGptTokenizer.__init__..) bos_token eos_token sep_token unk_token pad_tokenr) sacremoses ImportErrorlangsmcache_moses_tokenizercache_moses_detokenizeropenjsonloadencoderitemsdecoderreadr$dictziprangelen bpe_rankscachesuper__init__) selfrrr+r(r)r*r,kwargsr- vocab_handle merges_handlemerges __class__rrrAZs>   zBioGptTokenizer.__init__cCs t|jS)zReturns vocab size)r=r6rBrrr vocab_sizes zBioGptTokenizer.vocab_sizecCst|jfi|jSN)r:r6added_tokens_encoderrIrrr get_vocabszBioGptTokenizer.get_vocabcCs:||jvr|jj|d}||j|<|j|j|ddddS)Nr/TF)aggressive_dash_splits return_strescape)r1r0MosesTokenizertokenize)rBtextr/moses_tokenizerrrrmoses_tokenizes   zBioGptTokenizer.moses_tokenizecCs2||jvr|jj|d}||j|<|j||S)NrN)r2r0MosesDetokenizer detokenize)rBtokensr/moses_detokenizerrrrmoses_detokenizes  z BioGptTokenizer.moses_detokenizec s~t|dd|ddf}|jvrj|St|}|s#|dS t|fddd}|jvr4ny|\}}g}d}|t|krz|||} Wnty\|||dYn?w|||| | }|||kr|t|dkr||d|kr| |||d 7}n | |||d7}|t|ksBt|}|}t|dkrnt|}q$d |}|d krd }|j|<|S) Nr!Tcsj|tdS)Ninf)r>getfloat)pairrIrrsz%BioGptTokenizer.bpe..keyrrr" z z ) r#r?rminr>r=index ValueErrorextendappendjoin) rBtokenr r bigramfirstsecondnew_wordijrrIrbpesN     ,     zBioGptTokenizer.bpeFcCsL|r|}n|||j}g}|D]}|r#|t||dq|S)zReturns a tokenized string.rd)r$rVr/rhlistrr)rBrTbypass_tokenizer split_tokensrkrrr _tokenizes zBioGptTokenizer._tokenizecCs|j||j|jS)z0Converts a token (str) in an id using the vocab.)r6r^r+)rBrkrrr_convert_token_to_idsz$BioGptTokenizer._convert_token_to_idcCs|j||jS)z=Converts an index (integer) in a token (str) using the vocab.)r8r^r+)rBrfrrr_convert_id_to_tokensz$BioGptTokenizer._convert_id_to_tokencCs.dd|D}d|}|||j}|S)z:Converts a sequence of tokens (string) in a single string.cSs g|] }|ddddqS)rdr\)replace)rtrrrr&r'z.ry)rjr$r[r/)rBrYrTrrrconvert_tokens_to_stringsz(BioGptTokenizer.convert_tokens_to_stringN token_ids_0 token_ids_1returncCs,|dur |jg|S|jg}||||S)a Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. A BioGPT sequence has the following format: - single sequence: ` X ` - pair of sequences: ` A B ` Args: token_ids_0 (`List[int]`): List of IDs to which the special tokens will be added. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens. N) sep_token_id)rBr}r~seprrr build_inputs_with_special_tokenss z0BioGptTokenizer.build_inputs_with_special_tokensalready_has_special_tokenscsZ|r tj||ddS|dur#dgdgt|dgdgt|Sdgdgt|S)a Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. T)r}r~rNrr)r@get_special_tokens_maskr=)rBr}r~rrGrrrs(z'BioGptTokenizer.get_special_tokens_masksave_directoryfilename_prefixc CsLtj|std|ddStj||r|dndtd}tj||r,|dndtd}t|ddd }|t j |j d d d d dWdn1sTwYd}t|ddd 8}t |j dddD]!\}} || krtd|d| }|d|d|d7}qnWd||fS1swY||fS)NzVocabulary path (z) should be a directory-ryrrwrrr"TF)indent sort_keys ensure_asciir rcSs|dS)Nrr)kvrrrra*sz1BioGptTokenizer.save_vocabulary..rbzSaving vocabulary to zZ: BPE merge indices are not consecutive. Please check that the tokenizer is not corrupted!rdr)ospathisdirloggererrorrjVOCAB_FILES_NAMESr3writer4dumpsr6sortedr>r7warning) rBrrr merge_filefrfwriter bpe_tokens token_indexrrrsave_vocabularys6       zBioGptTokenizer.save_vocabularycCs|j}d|d<|S)Nr0)__dict__copy)rBstaterrr __getstate__6s zBioGptTokenizer.__getstate__cCs4||_zddl}Wn tytdw||_dS)NrznYou need to install sacremoses to use XLMTokenizer. See https://pypi.org/project/sacremoses/ for installation.)rr-r.r0)rBdr-rrr __setstate__;s   zBioGptTokenizer.__setstate__)rrrrr)FrK)NF)__name__ __module__ __qualname____doc__rvocab_files_namesmodel_input_namesrApropertyrJrMrVr[rrrvrwrxr|rsintrboolrstrr#rrr __classcell__rrrGrr,sR*,  ,     r)rr4rtokenization_pythonrutilsr get_loggerrrrrr__all__rrrrs