o wi_'@sddlZddlZddlmZmZmZmZddlZddl Z ddl Z ddl m Z ddl mZddlmZddlmZddgZGd ddeeZ      d#dedededededeedededededededeedeedededed ed!ef&d"dZdS)$N)DictListOptionalUnion)if_exist)ChatTemplateMixin) TokenizerSpec)loggingSentencePieceTokenizercreate_spt_modelc @sHeZdZdZ      d6dedeeeeefeefde d e d eef d d Z d dZ d7ddZ d7ddZ d7ddZddZddZddZddZgfdeeeefdeedeeeeffd d!Zd"d#Zed$d%Zed&d'Zed(d)Zed*d+Zed,d-Zed.d/Zed0d1Zed2d3Zed4d5ZdS)8r aSentencepiecetokenizer https://github.com/google/sentencepiece. Args: model_path: path to sentence piece tokenizer model. To create the model use create_spt_model() special_tokens: either list of special tokens or dictionary of token name to token value legacy: when set to True, the previous behavior of the SentecePiece wrapper will be restored, including the possibility to add special tokens inside wrapper. ignore_extra_whitespaces: whether to ignore extra whitespaces in the input text while encoding. Note: This is done for the current models tokenizers that don't handle extra whitespaces as by default tokenizer learned to ignore it. To check if the tokenizer by default ignores extra whitespaces refer to `self.removed_extra_spaces` attribute of the tokenizer. We added a parameter to process_asr_tokenizer.py for upcoming models to handle it inbuilt. NFT▁ model_pathspecial_tokenslegacyignore_extra_whitespaces chat_templatecCs||_|r tj|std|dt|_|j||j |_ |j |_ ||_ ||_ d|_i|_i|_||_||_|j||_|rT|j sOtd|||jd|jdk|_|d|d|dk|_dS) Nz model_path: z is invalidu☯z^Special tokens must be None when legacy is set to False. Provide special tokens at train time.zx yzx yxy)rospathexists ValueError sentencepieceSentencePieceProcessor tokenizerLoadget_piece_sizeoriginal_vocab_size vocab_sizerrextra_space_tokenspecial_token_to_idid_to_special_token&trim_spm_separator_after_special_token spm_separator piece_to_idspm_separator_idadd_special_tokensencode_as_piecesremoved_extra_spacestext_to_tokensspace_sensitive)selfr rrrrr"r#r,w/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/nemo/collections/common/tokenizers/sentencepiece_tokenizer.py__init__/s.      $zSentencePieceTokenizer.__init__c s`jrjstddjd|}jrg}d} i}jD]}z ||d|||<Wqty7Yqwt |dkr?nJt ||j d}|||}j |||}jrxt |dkrx|djvrxt |dkrx|djkrx|d|||||t |}q|j ||dnj |}jrjsttfdd |}|S) zConverts input text to a list of tokens. If legacy mode is enabled, handles special tokens separately. Args: text: The input string to tokenize. Returns: A list of string tokens. (?<= )(?= )|^ | $ rNkeycs |jkSN)r)rr+r,r-s z7SentencePieceTokenizer.text_to_tokens..)r(rresubrrr indexrlenmingetrr'r"r#popextendappendlistfilter) r+texttokenscur_idx st_indicestokennext_special_tokennext_start_idx text_tokensr,r6r-r)UsD           "  z%SentencePieceTokenizer.text_to_tokenscCs<t|tr |||St|tr||Stdt|)alConverts input text to a list of token IDs. Handles chat formatting or raw string tokenization depending on input type. Args: text: A string or list representing chat template inputs. sample_alpha: Optional float to enable subword sampling for data augmentation. Returns: A list of token IDs. z+Expected either str or list input, but got ) isinstancestr _text_to_idsrAapply_chat_templatertype)r+rC sample_alphar,r,r- text_to_idss   z"SentencePieceTokenizer.text_to_idsc Cs|jr|jstdd|jd|}|jrg}d} i}|jD]}z ||d|||<Wq!t y9Yq!wt |dkrAnMt ||j d}|||}|j |||} |jrzt |dkrz|d|jvrzt | dkrz| d|jkrz| d|| ||j||t |}q|jr|js||||d|S||j ||d|S|jr|js|||S|dur|j j|d|dd S|j |S) aInternal method to convert text to token IDs, handling optional sampling and special token logic. Args: text: Input string. sample_alpha: Optional alpha value for stochastic subword sampling. Returns: A list of token IDs. r/r0rr1Nr2r4Tenable_samplingalpha nbest_size)r(rr8r9rrstriprr r:rr;r<r=rencoder"r!r%r>r?r@_text_to_ids_extra_space encode_as_ids) r+rCrPidsrErFrGrHrIrJr,r,r-rMsN           "   z#SentencePieceTokenizer._text_to_idscCshg}i}|durd|dd}||jD]}|sq||j7}|jj|fi|}||ddq|S)aTokenizes text while preserving extra space tokens for legacy mode. Args: text: Input string. sample_alpha: Optional alpha value for subword sampling. Returns: A list of token IDs with preserved extra space markers. NTr4rR)splitrrrYr?)r+rCrPrZencoding_kwargspartpart_idsr,r,r-rXs   z/SentencePieceTokenizer._text_to_ids_extra_spacecCs&t|tjtjfr |}|j|S)zConverts a list of tokens back to the corresponding string. Args: tokens: A list of string tokens or a tensor/array of token IDs. Returns: The decoded string. )rKnpndarraytorchTensortolistr decode_pieces)r+rDr,r,r-tokens_to_texts  z%SentencePieceTokenizer.tokens_to_textcCst|tjtjfr |}|jrMd}d}t|D]$\}}||jvr<||j |||d7}||j|d7}|d}q||j ||d7}| S|j |S)zDecodes a list of token IDs into a string, handling special tokens if in legacy mode. Args: ids: A list or tensor/array of token IDs. Returns: The decoded string. rr0r1N) rKr_r`rarbrcr enumerater!r decode_idsstrip)r+rZrClast_iiidr,r,r- ids_to_text s   z"SentencePieceTokenizer.ids_to_textcCs&|jr ||jvr |j|S|j|S)zGets the ID corresponding to a token. Args: token: Token string. Returns: Token ID as an integer. )rr rr$)r+rGr,r,r- token_to_id%s  z"SentencePieceTokenizer.token_to_idcCsZt|tjtjfr |}g}|D]}||jkr!||j|q||j |q|S)zConverts a list of token IDs into corresponding token strings. Args: ids: A list or array/tensor of token IDs. Returns: List of string tokens. ) rKr_r`rarbrcrr@r!r id_to_piece)r+rZrDrlr,r,r- ids_to_tokens3s  z$SentencePieceTokenizer.ids_to_tokensrDtokens_to_skipreturncCs:t|tr|g}g}|D]}||vr|||q |S)a&Converts one or more tokens into their respective IDs, skipping any specified tokens. Args: tokens: A string or list of token strings. tokens_to_skip: List of tokens to ignore during conversion. Returns: A single ID or list of IDs. )rKrLr@rn)r+rDrqrZrGr,r,r- tokens_to_idsFs z$SentencePieceTokenizer.tokens_to_idscCsh|jstdt|trS|D]B}|j||jkr4||jvr4|j|j|<||j |j<|jd7_q|j||jkrP|j||j|<||j |j|<qdSt|t r| D]J\}}t ||||j||jkr||jvr|j|j|<||j |j<|jd7_q\|j||jkr|j||j|<||j |j|<q\dSt dtt|)a-Adds new special tokens to the tokenizer's vocabulary (only if legacy=True). Args: special_tokens: List or dict of special tokens to add. Raises: AttributeError: If not in legacy mode. ValueError: If the input is not a list or dictionary. zASpecial Token addition does not work when legacy is set to False.r1z/Expected special_tokens to be a list or a dict N)rAttributeErrorrKrArr$unk_idr rr!dictitemssetattrrrLrO)r+rrG token_namer,r,r-r&Xs8           z)SentencePieceTokenizer.add_special_tokenscC*|jr||jgd}|S|j}|S)z%Returns the ID for the padding token.r)rrs pad_tokenrpad_id)r+r|r,r,r-r|  zSentencePieceTokenizer.pad_idcCrz)z3Returns the ID for the beginning-of-sequence token.r)rrs bos_tokenrbos_id)r+rr,r,r-rr}zSentencePieceTokenizer.bos_idcCrz)z-Returns the ID for the end-of-sequence token.r)rrs eos_tokenreos_id)r+rr,r,r-rr}zSentencePieceTokenizer.eos_idcC |jr ||jgdStd)z=Returns the ID for the separator token (only in legacy mode).rVUse function token_to_id to retrieve special tokens other than unk, pad, bos, and eos.)rrs sep_token NameErrorr6r,r,r-sep_idzSentencePieceTokenizer.sep_idcCr)zBReturns the ID for the classification token (only in legacy mode).rr)rrs cls_tokenrr6r,r,r-cls_idrzSentencePieceTokenizer.cls_idcCr)z8Returns the ID for the mask token (only in legacy mode).rr)rrs mask_tokenrr6r,r,r-mask_idrzSentencePieceTokenizer.mask_idcCs |jS)z%Returns the ID for the unknown token.)rrur6r,r,r-rus zSentencePieceTokenizer.unk_idcs8t|j|j|j|j|j|jgfdd|jDS)zReturns a list of the additional special tokens (excluding bos, eos, pad, unk). Used to return sentinel tokens for e.g. T5. csg|] \}}|vr|qSr,r,).0kvrr,r- szHSentencePieceTokenizer.additional_special_tokens_ids..) setr~rr{rrrr rwr6r,rr-additional_special_tokens_idssz4SentencePieceTokenizer.additional_special_tokens_idscsBfddtjD}fddtjjD}||S)zHReturns the combined vocabulary list, including base and special tokens.csg|]}j|qSr,)rro)rrlr6r,r-rsz0SentencePieceTokenizer.vocab..csg|] }jj|qSr,)r!r)rrkr6r,r-rs)rangerrrr)r+ main_vocabrr,r6r-vocabs  zSentencePieceTokenizer.vocab)NFTNTr r5) __name__ __module__ __qualname____doc__rLrrrrboolr.r)rQrMrXrermrnrpintrsr&propertyr|rrrrrrurrr,r,r,r-r s^ & <  A6*         unigram?Fr4T data_filer sample_size do_lower_casetokenizer_type output_dircharacter_coveragetrain_extremely_large_corpusmax_sentencepiece_lengthboseospadcontrol_symbolsuser_defined_symbols byte_fallback split_digitssplit_by_whitespacesplit_by_unicode_scriptremove_extra_whitespacesc Cs|rtj|std|tj|}g}gd}|s"|d}t|dgr;td|d|d|dfStd |d |tj|d d d |d|d|d|d| }d}| sj|d8}|d7}| st|d8}|d7}| r}|d|7}| rd | }|d|7}|| 7}| rd | }|d|7}|| 7}|r|d7}|dkr|d|7}|r|d7}|dkr|d|7}|r|d7}|r|d 7}|s|d!7}|s|d"7}|s|d#7}t j |g}t |d$d%d&d'?}|D]4}|d(d}||vrq|d)r|dd*nd+|}t|dkr#||q||dqWd*n 1s6wY|||d}t |d,d&d'}|D] }||d-qOWd*n 1sfwY|d|fS).uCreates sentence piece tokenizer model from data file. Args: data_file: data file vocab_size: vocabulary size sample_size: maximum size of sentences the trainer loads do_lower_case: if text should be lower cased before tokenizer model is created character_coverage: float value between 0 and 1 (as a percentage). For languages with a vast charset, can be < 1.0, but for all other languages, it should be set as 1.0 output_dir: folder to save created tokenizer model. If not specified will store at data_file/../spt folder train_extremely_large_corpus: If training on huge datasets, pass this flag to allow SentencePiece to build the tokenizer. max_sentencepiece_length: Limits the maximum length of the SentencePiece subword that can be constructed. By default, no limit is placed. bos: when True, bos token "" is added to the vocabulary. eos: when True, eos token "" is added to the vocabulary. pad: when True, pad token "" is added to the vocabulary. control_symbols: control symbols to add to tokenizer, as defined by sentencepiece. These tokens get removed at decode time and are not encoded from the text - can only be added to the input programatically. user_defined_symbols: user symbols to add to tokenizer, as defined by sentencepiece. These tokens remain in the decoded text and are encoded automatically when present in the input text. byte_fallback: If , fallback to a byte sequence of the character. split_digits: If true, digits are split into individual tokens. split_by_whitespace: Whether to respect white space while creating subwords. If False, will learn merges across whitespace. split_by_unicode_script: Whether to include multiple Unicode scripts. Ex. is Arabic diacritics which are considered part of the letter (عِدَّةُ). remove_extra_whitespaces: Whether to remove leading, trailing, and duplicate internal whitespace. If true, will skip double spaces during encoding. z+data_file must be valid file path, but got )zzzzz/sptztokenizer.modelztokenizer model z/tokenizer.model already existsz/tokenizer.modelz /vocab.txtz Processing z and store at T)exist_okz--input=z --model_prefix=z/tokenizer --vocab_size=zE --shuffle_input_sentence=true --hard_vocab_limit=false --model_type=z --character_coverage=r1z --bos_id=-1z --eos_id=-1z --pad_id=,z --control_symbols=z --user_defined_symbols=z& --normalization_rule_name=nmt_nfkc_cfrz --input_sentence_size=z$ --train_extremely_large_corpus=truez --max_sentencepiece_length=z --byte_fallback=truez --split_digits=truez --split_by_whitespace=falsez --split_by_unicode_script=falsez! --remove_extra_whitespaces=falsez/tokenizer.vocabrutf8)encoding r Nz##w )rrrrdirnamerr infomakedirsjoinrSentencePieceTrainerTrainopenr[ startswithr;r@r?write) rrrrrrrrrrrrrrrrrrrdata_dirrrcmdr|control_string user_stringrDflinepiecerG vocab_filer,r,r-r s5       "   )rNrFr4FFFNNFFTTF)rr8typingrrrrnumpyr_rra#nemo.collections.common.parts.utilsr6nemo.collections.common.tokenizers.chat_template_mixinr1nemo.collections.common.tokenizers.tokenizer_specr nemo.utilsr __all__r rLrrfloatr r,r,r,r-s    9