o ߥi'@svddlZddlmZmZmZmZddlmZddlm Z ddl m Z m Z ddl mZGdddZGd d d e ZdS) N)DictListOptionalUnion)SentencePieceProcessor)PreTrainedTokenizer) BatchEncoding EncodedInput)PaddingStrategyc @seZdZdefddZdefddZ  ddeded ed eefd d Z d eed efddZ deed efddZ ddZ ddZ dS) SPTokenizer model_pathcCstj|s J|t|d|_|j|_|j|_|j|_|j |_ |j|j ks4Jgd}i|_ i|_ |D]}|j|j |<||j |j<|jd7_q@dS)N) model_file)z[MASK][gMASK]z[sMASK]sopeop)ospathisfilersp_model vocab_sizen_wordsbos_ideos_idunk_idpad_idget_piece_sizespecial_tokensindex_special_tokens)selfr rtokenr!_/home/ubuntu/.local/lib/python3.10/site-packages/modelscope/models/nlp/chatglm2/tokenization.py__init__ s       zSPTokenizer.__init__scC |j|SN)rEncodeAsPieces)rr$r!r!r"tokenize  zSPTokenizer.tokenizeFboseosreturncCs@t|tusJ|j|}|r|jg|}|r||jg}|Sr&)typestrrencoderr)rr$r*r+tr!r!r"r/#s   zSPTokenizer.encoder0cCr%r&)rdecode)rr0r!r!r"r1/r)zSPTokenizer.decodetokenscCs|j|}|Sr&)r DecodePieces)rr2textr!r!r" decode_tokens2s zSPTokenizer.decode_tokenscCs ||jvr |j|S|j|Sz2 Converts a token (str) in an id using the vocab. )rr PieceToIdrr r!r!r"convert_token_to_id6s   zSPTokenizer.convert_token_to_idcCs6||jvs||j|j|jfvs|dkrdS|j|S)=Converts an index (integer) in a token (str) using the vocab.r)rrrrr IdToPiecerindexr!r!r"convert_id_to_token<s   zSPTokenizer.convert_id_to_tokenN)FF)__name__ __module__ __qualname__r.r#r(boolrintr/r1r5r9r?r!r!r!r"r s$   r c sDeZdZddiZgdZd1fdd ZddZed efd d Z ed d Z ed efddZ eddZ eddZ ddZddZddZddZdeed efddZd2d d!Zd"d#Zd2d$d%Z d2d&eed'eeed eefd(d)Zdejddfd*eeeefefd+eed,ed-eed.ee d e!f d/d0Z"Z#S)3ChatGLM2Tokenizer vocab_fileztokenizer.model) input_idsattention_mask position_idsleftc sJd|_||_t||_|jj|jj|jjd|_tj dd|i|dS)N GLMTokenizer)z padding_sider!) namerFr tokenizerrrrrsuperr#)rrFrNkwargs __class__r!r"r#Js zChatGLM2Tokenizer.__init__cCs@||jvr |j|S||jjvsJ|d|j|jj|S)Nz is not a special token for )rrPrOr8r!r!r" get_commandVs    zChatGLM2Tokenizer.get_commandr,cCdS)Nzr!rr!r!r" pad_token\zChatGLM2Tokenizer.pad_tokencC |dS)NrMrUrWr!r!r" pad_token_id` zChatGLM2Tokenizer.pad_token_idcCrV)Nzr!rWr!r!r" eos_tokendrYzChatGLM2Tokenizer.eos_tokencCrZ)NrLr[rWr!r!r" eos_token_idhr]zChatGLM2Tokenizer.eos_token_idcCs|jjSr&)rPrrWr!r!r"rlszChatGLM2Tokenizer.vocab_sizecs(fddtjD}|j|S)z Returns vocab as a dict csi|]}||qSr!)_convert_id_to_token).0irWr!r" rs z/ChatGLM2Tokenizer.get_vocab..)rangerupdateadded_tokens_encoder)rvocabr!rWr" get_vocabps  zChatGLM2Tokenizer.get_vocabcKr%r&)rPr()rr4rRr!r!r" _tokenizeyr)zChatGLM2Tokenizer._tokenizecCr%r6)rPr9r8r!r!r"_convert_token_to_id| z&ChatGLM2Tokenizer._convert_token_to_idcCr%)r:)rPr?r=r!r!r"r`rkz&ChatGLM2Tokenizer._convert_id_to_tokenr2cCr%r&)rPr5)rr2r!r!r"convert_tokens_to_stringr)z*ChatGLM2Tokenizer.convert_tokens_to_stringNcCstj|rtj||jd}n|}t|jd }|}Wdn1s(wYt|d}||Wd|fS1sDwY|fS)a Save the vocabulary and special tokens file to a directory. Args: save_directory (`str`): The directory in which to save the vocabulary. filename_prefix (`str`, *optional*): An optional prefix to add to the named of the saved files. Returns: `Tuple(str)`: Paths to the files saved. rFrbNwb) rrisdirjoinvocab_files_namesopenrFreadwrite)rsave_directoryfilename_prefixrFfin proto_strwriterr!r!r"save_vocabularys     z!ChatGLM2Tokenizer.save_vocabularycCs|d|dg}|S)Nrrr[)r prefix_tokensr!r!r"get_prefix_tokenssz#ChatGLM2Tokenizer.get_prefix_tokenscCsX|durg}d}t|D]\}\}}|d|d||7}q |dt|d|7}|S)Nr;u [Round {}] 问：{} 答：{} ru[Round {}] 问：{} 答：) enumerateformatlen)rqueryhistorypromptrb old_queryresponser!r!r" build_prompts zChatGLM2Tokenizer.build_prompt token_ids_0 token_ids_1cCs0|}||}|dur|||dg}|S)a Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. A BERT sequence has the following format: - single sequence: `[CLS] X [SEP]` - pair of sequences: `[CLS] A [SEP] B [SEP]` Args: token_ids_0 (`List[int]`): List of IDs to which the special tokens will be added. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens. NrL)r|rU)rrrr{r!r!r" build_inputs_with_special_tokenssz2ChatGLM2Tokenizer.build_inputs_with_special_tokensencoded_inputs max_lengthpadding_strategypad_to_multiple_ofreturn_attention_maskc Cs|jdksJ||jd}t|}|tjkrt|}|dur1|dur1||dkr1||d|}|tjko;t||k}d|vrGdg||d<d|vrStt||d<|r|t|} d|vrjdg| |d|d<d|vrydg| |d|d<|jg| |||jd<|S)a? Pad encoded inputs (on left/right and up to predefined length or max length in the batch) Args: encoded_inputs: Dictionary of tokenized inputs (`List[int]`) or batch of tokenized inputs (`List[List[int]]`). max_length: maximum length of the returned list and optionally padding length (see below). Will truncate by taking into account the special tokens. padding_strategy: PaddingStrategy to use for padding. - PaddingStrategy.LONGEST Pad to the longest sequence in the batch - PaddingStrategy.MAX_LENGTH: Pad to the max length (default) - PaddingStrategy.DO_NOT_PAD: Do not pad The tokenizer padding sides are defined in self.padding_side: - 'left': pads on the left of the sequences - 'right': pads on the right of the sequences pad_to_multiple_of: (optional) Integer if set will pad the sequence to a multiple of the provided value. This is especially useful to enable the use of Tensor Core on NVIDIA hardware with compute capability `>= 7.5` (Volta). return_attention_mask: (optional) Set to False to avoid returning attention mask (default: set to model specifics) rJrNrrHrI) rNmodel_input_namesrr LONGEST DO_NOT_PADlistrdr\) rrrrrrrequired_input seq_lengthneeds_to_be_padded differencer!r!r"_padsN      zChatGLM2Tokenizer._pad)rJr&)$r@rArBrqrr#rUpropertyr.rXr\r^r_rrhrirjr`rrlrzr|rrDrrr rrrr rrCdictr __classcell__r!r!rSr"rEEs`          rE)rtypingrrrr sentencepiecer transformersr$transformers.tokenization_utils_baserr transformers.utilsr r rEr!r!r!r"s   ;