o }oi%@sddlZddlZddlmZmZmZmZmZm Z ddl m Z m Z ddl mZddlmZddlmZer@ddlmZddlmZGd d d eZdS) N) TYPE_CHECKINGAnyDictListOptionalUnion)Datasetconcatenate_datasets)FineTuningDataModule)get_dataset_root)logging) TokenizerSpec)PackedSequenceSpecsc2seZdZdZ               d0deeeefdeedeedeedeedede dedde de deee de de de de d e d!e d"e d#ed$d%ed&ed'ed(ee ee ff.fd)d* Z d1fd,d- Zd.d/ZZS)2CustomRetrievalDataModulez2Custom Retrieval Data Module loaded with json fileN{Gz?{Gz?custom_retrieval_datasetFTquestionpos_docneg_doc data_rootval_root test_root val_ratio test_ratiodataset_identifier seq_length tokenizerr micro_batch_sizeglobal_batch_sizerampup_batch_sizeforce_redownload delete_rawseedmemmap_workers num_workers pin_memorypersistent_workerspacked_sequence_specsr query_key pos_doc_key neg_doc_keydataset_kwargscsz| |_| |_|dusJdt|ts|g}|D]}tj|s(Jd|dq|dur?tj|s;Jd|dd|_n||_|durXtj|sTJd|dd|_n||_d|j|j|_ ||_ ||_ ||_ ||_ ||_||_d |d |j d |jd |jd }|j dur|d |j 7}|j dur|d|j 7}t|tjt|||| | | ||||||d dS)a$ Custom DataModule for Finetuning retrieval Dataset for Embedding model. Args: data_root (Union[str, List[str]]): The JSONL data file(s) used for training/validation/test. if val_root/test_root is not present, data_root will be split to training and val/test based on val_ratio/test_ratio. val_root (Optional[str]): The JSONL data file used for validation. If not provided, validation set will be split from data_root. test_root (Optional[str]): The JSONL data file used for test. If not provided, test set will be split from data_root. val_ratio (Optional[float]): The ratio of validation set when splitting from data_root. test_ratio (Optional[float]): The ratio of test set when splitting from data_root. dataset_identifier (str): Dataset identifier when saving the dataset to NEMO_HOME. seq_length (int, optional): The maximum sequence length for the input and output text. Defaults to 2048. tokenizer (Optional[TokenizerSpec], optional): The tokenizer to use for preprocessing the text. If not provided, a Megatron GPT2 BPE tokenizer will be used. micro_batch_size (int, optional): The micro batch size for training. Defaults to 4. global_batch_size (int, optional): The global batch size for training. Defaults to 8. rampup_batch_size (Optional[List[int]], optional): A list of batch sizes for ramping up during training. Defaults to None. seed (int, optional): The random seed for data shuffling. Defaults to 1234. memmap_workers (int, optional): The number of worker processes for loading data using TextMemMapDataset. Defaults to 1. num_workers (int, optional): The number of worker processes for data loading. Defaults to 8. pin_memory (bool, optional): Whether to pin memory during data loading for faster GPU training. Defaults to True. persistent_workers (bool, optional): Whether to keep data loading workers persistent across epochs. Defaults to False. dataset_kwargs (Optional[Dict[str, Any]], optional): Keyword arguments to pass into the GPTSFTDataset class Nz6RetrievalDataModule does not support packed sequences.z Data root z does not exist.zValidation root rz Test root rz data_root: z will be split to :z used for train/val/testz, separate validation root: z, separate test root: ) dataset_rootr!r"r#r$r%r(r)r*r+r,r1)r&r' isinstancelistospathexistsrr train_ratiorrr.r/r0unprocessed_rootr infosuper__init__r )selfrrrrrr r!r"r#r$r%r&r'r(r)r*r+r,r-r.r/r0r1 directorylog_info __class__[/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/llm/gpt/data/retrieval.pyr=!sb8     z"CustomRetrievalDataModule.__init__returncs&|jr|jr |tdS)z"Prepare data if not split already.N) train_pathr8r&_preprocess_and_split_datar< prepare_data)r>rArCrDrHsz&CustomRetrievalDataModule.prepare_datac Cstd|jjdi}g}|jD]}|tt t |dqt |}|j |j dkrB|j|j |j |jd}|d|d<n||d<i}|j dkrd|j dkrd|dj|j |j |j |jd}n!|j dkru|j dkru|d|d<n|j dkr|j dkr|d|d<|jdurtt t |jd|d <n|d|d <|jdurtt t |jd|d<n|d|d<td t|dtd t|d td t|d|D]m\}}|j|d } | j dddG} |D]<} t| |jtr | |jdn| |j} t| |jtr| |jn| |jg} | t| |j| | ddqWdn 1s?wYt|d| qdS)NzPreprocessing z! to jsonl format and splitting...rr) test_sizer(traintrainingtest validationztraining samples: zvalidation samples: ztest samples: z.jsonlwzutf-8)encoding)queryrr z split saved to )r r;rB__name__r:appendr from_listjsonloadopenr rrtrain_test_splitr(rrlenitemsr3r4r/r5r0writedumpsr.)r> save_splitstrain_datasetsdata_dir train_dataset split_datasetsplit_dataset2 split_namedataset output_fileforrrCrCrDrGsN      *($z4CustomRetrievalDataModule._preprocess_and_split_data)NNrrrrNrrNFTrrrTFNrrrN)rEN)rS __module__ __qualname____doc__rstrrrfloatintboolrrr=rHrG __classcell__rCrCrArDrs     nr)rVos.pathr6typingrrrrrrdatasetsrr *nemo.collections.llm.bert.data.fine_tuningr "nemo.collections.llm.gpt.data.corer nemo.utilsr "nemo.collections.common.tokenizersr -nemo.collections.llm.gpt.data.packed_sequencerrrCrCrCrDs