o }oi@sddlZddlZddlmZmZmZmZmZddlm Z m Z ddl m Z ddl mZddlmZddlmZer>ddlmZGd d d eeZdS) N) TYPE_CHECKINGAnyDictListOptional) DatasetDict load_dataset)get_dataset_root)FineTuningDataModule)IOMixin)logging) TokenizerSpecc seZdZdZ            d*d ed ed eddededeeededededededededee ee fffdd Z d+fdd Z dd Z d,d#ed$ed%efd&d'Zd(d)ZZS)-SpecterDataModuleaA data module for fine-tuning on the Specter dataset. This class inherits from the `FineTuningDataModule` class and is specifically designed for fine-tuning models on the SPECTER Datasets. It handles data download, preprocessing, splitting, and preparing the data in a format suitable for training, validation, and testing. Args: force_redownload (bool, optional): Whether to force re-download the dataset even if it exists locally. Defaults to False. delete_raw (bool, optional): Whether to delete the raw downloaded dataset after preprocessing. Defaults to True. See FineTuningDataModule for the other args NFTr dataset_root seq_length tokenizerr micro_batch_sizeglobal_batch_sizerampup_batch_sizeforce_redownload delete_rawseedmemmap_workers num_workers pin_memorypersistent_workersdataset_kwargscsD||_||_tj|durtdn||||||| | | | | |d dS)Nspecter) rrrrrrrrrrr r!)rrsuper__init__r )selfrrrrrrrrrrrrr r! __class__Z/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/llm/bert/data/specter.pyr$,s  zSpecterDataModule.__init__returncs0|jr|jr|}||tdS)z Prepare dataset for fine-tuning.N) train_pathexistsr_download_data_preprocess_and_split_datar# prepare_data)r%dsetr&r(r)r/Os zSpecterDataModule.prepare_datacCs:td|jjdtddt|j|jrddSddS)Nz Downloading z...zsentence-transformers/spectertripletr) cache_dir download_mode)r infor'__name__rstrrrr%r(r(r)r-Wsz SpecterDataModule._download_data皙?333333?r0 train_ratio val_ratioc Csdtd|jjdd||}i}|d}|j|||jd}|dj||||jd}|d|d<|d|d<|d|d<|D]G\} }|j| d } | j d d d "} |D]} | t | d | d| dgddq\Wdn1s~wYt| d| qF|j r|jD]} | rt| qd t| jvr| qdSdS)aPreprocesses and splits the downloaded dataset into training, validation, and test sets. Args: dset (DatasetDict): The downloaded dataset object. split_val_from_train (bool, optional): Whether to split the validation set from the training set. If False, the validation set is split from the test set. Defaults to True. val_proportion (float, optional): The proportion of the training or test set to be used for the validation split. Defaults to 0.05. zPreprocessing z! to jsonl format and splitting...rtrain) test_sizertesttraining validationz.jsonlwzutf-8)encodinganchorpositivenegative)querypos_docneg_doc Nz split saved to )r r4r'r5gettrain_test_splitritemsropenwritejsondumpsriterdiris_dirshutilrmtreer6nameunlink)r%r0r:r; test_ratio save_splitsdataset split_datasetsplit_dataset2 split_name output_filefopr(r(r)r.`s<     " z,SpecterDataModule._preprocess_and_split_datacCsdS)z?No need to reconfigure trainer.limit_val_batches for finetuningNr(r7r(r(r)reconfigure_limit_batchessz+SpecterDataModule.reconfigure_limit_batches)NrNrrNFTrrrTFN)r*N)r8r9)r5 __module__ __qualname____doc__r6intrrboolrrr$r/r-rfloatr.ra __classcell__r(r(r&r)rsb      # (r)rOrStypingrrrrrdatasetsrr#nemo.collections.llm.bert.data.corer *nemo.collections.llm.bert.data.fine_tuningr nemo.lightning.io.mixinr nemo.utilsr "nemo.collections.common.tokenizersr rr(r(r(r)s