o }oi -@szddlZddlmZmZddlmZmZmZddl m Z ddl m Z ddgZ d d d d Zd eefddZeGdddZ       d&dedeedeedeedeeeefdeedeedeefddZ              d'dedeedeedeedeedeeeefdeedeedeed eed!eed"eedeed#eefd$d%ZdS)(N)MISSING dataclass)DictListOptional)logging))get_huggingface_pretrained_lm_models_list get_tokenizerget_tokenizer_listzmegatron-bert-345m-uncasedzmegatron-bert-345m-casedzmegatron-gpt-345m)BertWordPieceLowerCaseBertWordPieceCaseGPT2BPETokenizerreturncCs2ttdd}|ttddgdt|S)z3 Returns all all supported tokenizer names F)include_externalT) sentencepiececharword)setr updatelist)srg/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/nlp/modules/common/tokenizer_utils.pyr sc@seZdZUdZeZeed<dZe eed<dZ e e ed<dZ e eed<dZ e eeefed<dZe eed <d Ze eed <dZe e ed <d Ze eed<d Ze eed<dS)TokenizerConfigz, Tokenizer Configuration Dataclass. libraryNtokenizer_model vocab_size vocab_filespecial_tokens bpe_dropoutg+?coveragetraining_sample_sizeFr2lsentencepiece_legacy)__name__ __module__ __qualname____doc__rrstr__annotations__rrrintrrrr!floatr"r#r$boolr%rrrrr)s  rFr tokenizer_namerr merges_fileruse_fastr! chat_templatec Csddl}ddlm} t||jj|jjfr| |}|dur!i} n|} d|vrQz ddlm } m } m } Wn t t fy@t dw|durM| |}| |}| |}|dkrlddlm}td t||||d |d S|d kr~dd lm}|||ddS|dkrddlm}|d d|i| S|dkrddlm}|d d|i| S|dkrddlm}|j||dStd|d|d|d| d| ddlm}|d |||d| ||d}|S)!a$ Args: tokenizer_name: sentencepiece or pretrained model from the hugging face list, for example: bert-base-cased To see the list of all HuggingFace pretrained models, use: nemo_nlp.modules.common.get_huggingface_pretrained_lm_models_list() tokenizer_model: tokenizer model file of sentencepiece special_tokens: dict of special tokens. For additional special tokens besides standard special tokens (bos, eos, pad, etc.), such as sentinel tokens for T5 (, , etc.), use key 'additional_special_tokens' vocab_file: path to vocab file use_fast: (only for HuggingFace AutoTokenizer) set to True to use fast HuggingFace tokenizer bpe_dropout: (experimental) BPE dropout tries to corrupt the standard segmentation procedure of BPE to help model better learn word compositionality and become robust to segmentation errors. It has empirically been shown to improve inference time BLEU scores. rN OmegaConfmegatron)get_megatron_merges_fileget_megatron_tokenizerget_megatron_vocab_filezMegatron-core was not found. Please see the NeMo README for installation instructions: https://github.com/NVIDIA/NeMo#megatron-gpt.rSentencePieceTokenizertokenizer_model: T model_pathrlegacyr2tiktokenTiktokenTokenizeradditional_special_tokens)rrr) WordTokenizerrr) CharTokenizerregexRegExTokenizer regex_filer>Getting HuggingFace AutoTokenizer with pretrained_model_name: z, vocab_file: z, merges_files: z, special_tokens_dict: z, and use_fast:  AutoTokenizerpretrained_model_namerr0)r1r2r) omegaconfr4 isinstance listconfig ListConfig dictconfig DictConfig to_container;nemo.collections.nlp.modules.common.megatron.megatron_utilsr6r7r8 ImportErrorModuleNotFoundError:nemo.collections.common.tokenizers.sentencepiece_tokenizerr:rinfor*5nemo.collections.common.tokenizers.tiktoken_tokenizerrA1nemo.collections.common.tokenizers.word_tokenizerrC1nemo.collections.common.tokenizers.char_tokenizerrD2nemo.collections.common.tokenizers.regex_tokenizerrGload_tokenizer=nemo.collections.common.tokenizers.huggingface.auto_tokenizerrL)r/rrr0rr1r!r2rOr4special_tokens_dictr6r7r8r:rArCrDrGrL tokenizerrrrr ;s         rr model_namer$r> delimitertrust_remote_codercCs>ddl}ddlm}t||jj|jjfr||}|dur!i}n|}|dkr9|dur9|dus5tj |s9t d|dkrdddl m }td||d&|||d ||| | d }| rb| |j_|S|d kr~dd lm}td ||||| | dS|dkrddlm}td||S|dkrddlm}td|j||dS|dkr|dkrdd lm}tdt|||| dS|tvrt|}td|d|d|t||||| dS|dkrddlm}||| dS|d krdd!lm}||d"S|d#kr| dusJdd$lm }|| St!d%)'a Args: model_name: if using a pretrained model from NeMo, HuggingFace, or Megatron tokenizer_model: tokenizer model file of sentencepiece special_tokens: dict of special tokens vocab_file: path to vocab file use_fast: (only for HuggingFace AutoTokenizer) set to True to use fast HuggingFace tokenizer bpe_dropout: (experimental) BPE dropout tries to corrupt the standard segmentation procedure of BPE to help model better learn word compositionality and become robust to segmentation errors. It has empirically been shown to improve inference time BLEU scores. r2l: Whether to return subword IDs from right to left rNr3z byte-levelz2No Tokenizer path provided or file does not exist! huggingfacerKrJrM)r1rer2rr9z"Getting SentencePiece with model: r<)ByteLevelTokenizerzUsing byte-level tokenizationrErFzUsing regex tokenizationrHr5GPTSentencePieceTokenizerr;)r=r>z6Getting Megatron tokenizer for pretrained model name: z, custom vocab file: z, and merges file: )r/rr0rr2tabular)TabularTokenizer)rdr?r@)rnull) NullTokenizerzCurrently we only support "huggingface", "sentencepiece", "megatron", "byte-level", "regex", "tabular","tiktoken", and "null" tokenizer libraries.r)"rOr4rPrQrRrSrTrUospathisfile ValueErrorr`rLrrZrbr2rYr:7nemo.collections.common.tokenizers.bytelevel_tokenizersrgr^rGr_megatron_tokenizer_model_mapr 4nemo.collections.common.tokenizers.tabular_tokenizerrjr[rA1nemo.collections.common.tokenizers.null_tokenizerrlNotImplementedError)rrcrrr0rr1r!r$r>rdrer2rrOr4rarLrbr:rgrGrjrArlrrrget_nmt_tokenizers                     rv)NNNNFr N)rNNNNNFr FFNFNN)os.pathrm dataclassesrrtypingrrr nemo.utilsrhuggingface.huggingface_utilsr __all__rrr*r rr.r-r r,rvrrrrs     j