o }oi@sNddlmZmZmZmZddlmZddgZGdddZGdddeZ dS))DictListOptionalUnion) TokenizerSpecByteLevelProcessorByteLevelTokenizerc@sNeZdZdZdeedefddZdedeefddZdedefd d Zd S) rzf A very basic tokenization and detokenization class for use with byte-level tokenization. tokensreturncCs d|S)z< Detokenize a list of tokens into a string.  )joinselfr rk/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/common/tokenizers/bytelevel_tokenizers.py detokenize zByteLevelProcessor.detokenizetextcCst|S)z: Tokenize a string into a list of tokens. )listrrrrrtokenize"szByteLevelProcessor.tokenizecCs|S)z% Normalize a string. rrrrr normalize(szByteLevelProcessor.normalizeN) __name__ __module__ __qualname____doc__rstrrrrrrrrrs c @seZdZdZ     d%deeeeefeefde de d e d e f d d Z d dZ ddZ ddZ ddZddZddZddZddZeddZedd Zed!d"Zed#d$ZdS)&rz A byte-level tokenizer that encodes text as UTF-8 bytes with user control over the EOS, BOS, and PAD tokens as well as the vocabulary size and a mapping of other special tokens to their IDs. Nrspecial_tokens vocab_size_eos_id_pad_id_bos_idcCs||_||_||_|j|j|j|j|j|ji|_|dur|n|t||_|j|_ |dur/in|}|D]}|j d8_ |j |j|<q3dd|j D|_ dS)aA byte-level tokenizer that encodes text as UTF-8 bytes. This tokenizer treats each byte as a token, with a default vocabulary size of 512 to accommodate UTF-8 byte values (0-255) plus special tokens. It can handle arbitrary text input by encoding it into bytes. Args: special_tokens: Dictionary or list of special tokens to add to the vocabulary. These tokens will be assigned IDs at the end of the vocabulary. Defaults to None. vocab_size: Size of the vocabulary, should be at least 256 to handle all byte values. Special tokens will be added after this size. Defaults to 512. _eos_id: ID to use for the end-of-sequence token. Defaults to 0. _pad_id: ID to use for the padding token. Defaults to 1. _bos_id: ID to use for the beginning-of-sequence token. Defaults to None. NrcSsi|]\}}||qSrr).0kvrrr `sz/ByteLevelTokenizer.__init__..) r!r"r#pad_idbos_ideos_idspecial_token_to_idlenr special_startitemsid_to_special_token)rrr r!r"r#tokrrr__init__5szByteLevelTokenizer.__init__cC ||S)z5 Convert a text to a list of tokens. ) text_to_idsrrrrtext_to_tokenscrz!ByteLevelTokenizer.text_to_tokenscCr2)z5 Convert a list of tokens to a text. ) ids_to_textr rrrtokens_to_textirz!ByteLevelTokenizer.tokens_to_textcCst|dS)z2 Convert a text to a list of IDs. utf-8)rencoderrrrr3oszByteLevelTokenizer.text_to_idscs(fdd|D}t|jdddS)z2 Convert a list of IDs to a text. csg|] }|jkr|qSr)r-)r$xrrr zsz2ByteLevelTokenizer.ids_to_text..r7ignore)errors)bytesdecoderstrip)ridsrr:rr5uszByteLevelTokenizer.ids_to_textcC2t|tr|g}g}|D] }|||q |S)z< Convert a list of tokens to a list of IDs. ) isinstancerappend token_to_id)rr rAtokenrrr tokens_to_ids} z ByteLevelTokenizer.tokens_to_idscCrB)z< Convert a list of IDs to a list of tokens. )rCintrD id_to_token)rrAr idrrr ids_to_tokensrHz ByteLevelTokenizer.ids_to_tokenscCs||jvr |j|S|S)z: Convert a token to its corresponding ID. )r+)rrFrrrrEs  zByteLevelTokenizer.token_to_idcCs||jkr|S|j|S)z; Convert an ID to its corresponding token. )r-r/)rrKrrrrJs  zByteLevelTokenizer.id_to_tokencC|jS)z% Get the padding ID. )r"r:rrrr(zByteLevelTokenizer.pad_idcCrM)z3 Get the beginning-of-sequence ID. )r#r:rrrr)rNzByteLevelTokenizer.bos_idcCrM)z- Get the end-of-sequence ID. )r!r:rrrr*rNzByteLevelTokenizer.eos_idcCsdS)z% Get the unknown ID. irr:rrrunk_idszByteLevelTokenizer.unk_id)NrrrN)rrrrrrrrrrIr1r4r6r3r5rGrLrErJpropertyr(r)r*rOrrrrr/sD .      N) typingrrrr1nemo.collections.common.tokenizers.tokenizer_specr__all__rrrrrrs