o
}oia @ sT d dl Z d dlmZmZ d dlmZ dZe jdddd Ze jddd d
Z dS ) N)CanaryTokenizerSentencePieceTokenizer)create_spt_modela#
Example system message.
Example user message.
Example assistant message.
TEST
[INST]
[/INST]
<>
<>
User: Assistant:
user model
Instruct Output
<|
|>
<|en|> <|de|> <|fr|> <|es|> <|transcribe|> <|translate|> <|pnc|> <|nopnc|> <|startoftranscript|> <|endoftext|>
Feel free to add new tokens for your own tests!?
But know that if you do so, you may need to update the token IDs in the existing tests!
So, it might be a good idea to create a new tokenizer instead when adding new prompt formats.
session)scopec
C sL | d}|d }|t tt|dddt|dddd tt|d S ) N
bpe_tokenizerztext.txti FT)
vocab_size sample_size
do_lower_case
output_dirremove_extra_whitespacesboseosztokenizer.model)mktemp
write_textTOKENIZER_TRAIN_TEXTr strr )tmp_path_factorytmpdir text_path r g/home/ubuntu/.local/lib/python3.10/site-packages/tests/collections/common/prompt_formatters/conftest.pyr / s
r c C s* | d}tddg|}t|| ddS )N
spl_tokens
transcribeen)r r )
tokenizers)r r build_special_tokenizer)r r r r r r r canary_tokenizerA s
r )
pytest"nemo.collections.common.tokenizersr r :nemo.collections.common.tokenizers.sentencepiece_tokenizerr r fixturer r r r r r s