o }oi@sBddlmZmZddlmZddlmZdgZGdddeZdS))ABCabstractmethod) OrderedDict)List TokenizerSpecc@seZdZdZeddZeddZeddZedd Zed d Z ed d Z de e fddZ ddZeddZeddZeddZeddZeddZeddZedd Zed!d"Zed#d$Zd%S)&rz: Inherit this class to implement a new tokenizer. cCdS)z$Converts text into a list of tokens.Nselftextrre/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/common/tokenizers/tokenizer_spec.pytext_to_tokenszTokenizerSpec.text_to_tokenscCr)z)Converts a list of tokens back into text.Nrr tokensrrr tokens_to_text rzTokenizerSpec.tokens_to_textcCr)z5Converts a list of tokens to their corresponding IDs.Nrrrrr tokens_to_ids%rzTokenizerSpec.tokens_to_idscCr)z,Converts a list of token IDs back to tokens.Nrr idsrrr ids_to_tokens*rzTokenizerSpec.ids_to_tokenscCr)z$Converts text directly to token IDs.Nrr rrr text_to_ids/rzTokenizerSpec.text_to_idscCr)z Converts token IDs back to text.Nrrrrr ids_to_text4rzTokenizerSpec.ids_to_textspecial_tokenscCtd)z0Adds special tokens (eos, pad, cls...) to vocab.To be implementedNotImplementedError)r rrrr add_special_tokens9z TokenizerSpec.add_special_tokenscOr)z*Appies chat template and tokenizes resultsrr)r argskwargsrrr apply_chat_template=rz!TokenizerSpec.apply_chat_templatecCs t|jS)zname of the class)type__name__r rrr nameAs zTokenizerSpec.namecCs"tdt|jdt|jiS)z6Property required for use with megatron-core datasets.class.)rr" __module__ __qualname__r$rrr unique_identifiersFs"z TokenizerSpec.unique_identifierscC$t|dr|jStt|jd)zGProperty alias to match MegatronTokenizer; returns cls_id if available.cls_idz# has no attribute 'cls' or 'cls_id')hasattrr,AttributeErrorr"r#r$rrr clsK zTokenizerSpec.clscCr+)zGProperty alias to match MegatronTokenizer; returns sep_id if available.sep_idz# has no attribute 'sep' or 'sep_id')r-r1r.r"r#r$rrr sepRr0zTokenizerSpec.sepcCr+)zGProperty alias to match MegatronTokenizer; returns pad_id if available.pad_idz# has no attribute 'pad' or 'pad_id')r-r3r.r"r#r$rrr padYr0zTokenizerSpec.padcCs4t|dr|jSt|dr|jStt|jd)zGProperty alias to match MegatronTokenizer; returns eod_id if available.eod_ideos_idz5 has no attribute 'eod', 'eod_id', 'eos', or 'eos_id')r-r5r6r.r"r#r$rrr eod`s  zTokenizerSpec.eodcCr+)zGProperty alias to match MegatronTokenizer; returns bos_id if available.bos_idz# has no attribute 'bos' or 'bos_id')r-r8r.r"r#r$rrr bosjr0zTokenizerSpec.boscCr+)zGProperty alias to match MegatronTokenizer; returns eos_id if available.r6z# has no attribute 'eos' or 'eos_id')r-r6r.r"r#r$rrr eosqr0zTokenizerSpec.eoscCr+)zHProperty alias to match MegatronTokenizer; returns mask_id if available.mask_idz% has no attribute 'mask' or 'mask_id')r-r;r.r"r#r$rrr maskxr0zTokenizerSpec.maskN)r#r(r)__doc__rr rrrrrrstrrr!propertyr%r*r/r2r4r7r9r:r<rrrr rsD              N) abcrr collectionsrtypingr__all__rrrrr s