o }oi;@sddlZddlmZddlmZddlmZddlZddl m Z ddl m Z ddl mZddlmZmZmZd ed e d ed ed eef ddZ  ddedededed e d ed eeded efddZeGdddZdS)N) dataclass)Path)Optional) TokenizerSpec)create_sft_dataset)logging) create_histcreate_packing_strategyfill_packing_strategypath tokenizermax_seq_lengthseeddataset_kwargscs|si}|d}|rt|tst||d<|dd}|r#||j_td||||dd|t fddt t DS) a Tokenizes a dataset from the provided path using the specified tokenizer and prepares it for further processing. Args: path (Path): Path to the dataset file. tokenizer (TokenizerSpec): The tokenizer to use for tokenization. max_seq_length (int): Maximum sequence length for the tokens. seed (int): Random seed for shuffling the dataset (optional). Returns: np.ndarray: A NumPy array containing the tokenized data. tool_schemas chat_templateNT)r r seq_lengthris_testcsg|]}|qSr).0idatasetra/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/llm/gpt/data/packed_sequence.py Esz$tokenize_dataset..r) get isinstancestrjsondumpspopr rrnparrayrangelen)r r r rrtsrrrrtokenize_datasets$   r&first_fit_shuffle input_path output_pathoutput_metadata_pathpacked_sequence_sizepacking_algorithmc Cstd|t|||||} t| |\} } t| ||\} } t| | ||j}t|||durz%t |d}t |}t |t sFJdWdn1sPwYWn tyag}Ynw|| t |d}t ||Wdn1s}wYtd|dS)a Prepares a packed sequence dataset from a given input file and saves it to an output file. Args: input_path (Path): Path to the input dataset file. output_path (Path): Path to save the packed sequence data. packed_sequence_size (int): The maximum size for each packed sequence. tokenizer (TokenizerSpec): The tokenizer to use for tokenization. max_seq_length (int): Maximum sequence length for the tokens. seed (Optional[int]): Random seed for shuffling (optional). packing_algorithm (str): The algorithm used for packing sequences currently supports "first_fit_shuffle" and "first_fit_decreasing". Returns: None: Saves the packed sequence data to the specified output path. zPreparing packed sequence from Nrzinvalid packing_metadata_file!wz)Packed sequence is prepared and saved to )rinfor&rr r eos_idr!saveopenrloadrlistFileNotFoundErrorappenddump)r(r)r*r+r r rr,rr sequences histogram assignmentspacking_metadata output_datafpacking_metadata_filerrrprepare_packed_sequence_dataHs*      r?c@sneZdZUdZdZeed< dZeed< dZ eed< dZ eed< dZ eed< d Z e ed < d d ZdS) PackedSequenceSpecszPDefines the packed sequence specifications used for generating a packed dataset.r+Ntokenizer_model_namepacked_train_data_pathpacked_val_data_pathpacked_metadata_pathFpad_cu_seqlenscCs|jdur&t|j|_|jjdksJd|j|js&Jd|j|jdurLt|j|_|jjdks?Jd|j|jsNJd|jdSdS)Nz.npyz/packed training data file must be a .npy file: z*packed training data file does not exist: z1packed validation data file must be a .npy file: z,packed validation data file does not exist: )rCrsuffixexistsrD)selfrrr __post_init__s(        z!PackedSequenceSpecs.__post_init__)__name__ __module__ __qualname____doc__r+int__annotations__rBrrCrDrErFboolrJrrrrr@s        r@)rr'N)r dataclassesrpathlibrtypingrnumpyr!"nemo.collections.common.tokenizersr"nemo.collections.llm.gpt.data.corer nemo.utilsr!nemo.utils.sequence_packing_utilsrr r rOdictr&rr?r@rrrrsX       4 <