o }oi-@sddlZddlZddlmZddlmZmZmZmZm Z m Z ddl m Z m Z ddlmZddlmZddlmZddlmZerLdd lmZdd lmZGd d d eeZdS) N)Path) TYPE_CHECKINGAnyDictListOptionalUnion) DatasetDict load_dataset)get_dataset_root)FineTuningDataModule)IOMixin)logging) TokenizerSpec)PackedSequenceSpecsc"seZdZdZ              d*d eeeefd ed ed dededee ede de dededede de deddee ee fffdd Z d+fdd Zd d!Z "d,d#ed$e d%efd&d'Zd(d)ZZS)-SquadDataModuleaA data module for fine-tuning on the Squad dataset. This class inherits from the `FineTuningDataModule` class and is specifically designed for fine-tuning models on the Stanford Question Answering Dataset (SQuAD). It handles data download, preprocessing, splitting, and preparing the data in a format suitable for training, validation, and testing. Args: dataset_root (Optional[Union[str, Path]]): The root directory containing the training, validation, and test data. Defaults to None, which by default downloads the data. force_redownload (bool, optional): Whether to force re-download the dataset even if it exists locally. Defaults to False. delete_raw (bool, optional): Whether to delete the raw downloaded dataset after preprocessing. Defaults to True. See FineTuningDataModule for the other args NFT dataset_root seq_length tokenizerrmicro_batch_sizeglobal_batch_sizerampup_batch_sizeforce_redownload delete_rawseedmemmap_workers num_workers pin_memorypersistent_workerspacked_sequence_specsrdataset_kwargscsF||_||_tj|dur|ntd|||||| | | | | ||d dS)Nsquad) rrrrrrrr r!r"r#r$r%)rrsuper__init__r )selfrrrrrrrrrr r!r"r#r$r% __class__W/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/llm/gpt/data/squad.pyr(2s" zSquadDataModule.__init__returncs0|jr|jr|}||tdS)N) train_pathexistsr_download_data_preprocess_and_split_datar' prepare_data)r)dsetr*r,r-r3Ws zSquadDataModule.prepare_datacCs8td|jjdtdt|j|jrddSddS)Nz Downloading z...r&r) cache_dir download_mode)rinfor+__name__r strrrr)r,r,r-r1^szSquadDataModule._download_data皙?r4split_val_from_trainval_proportionc Cstd|jjdi}|d}|d}|r2|j||jd}|d|d<|d|d<||d<n|j||jd}||d<|d|d<|d|d<|D]d\}} |j|d} | j d d d ?} | D]4} i} d | d d| dd| d<| ddd| d<|dkr| dd| d<| t | dqdWdn1swYt|d| qN|j r|jD]}|rt|qdt|jvr|qdSdS)aPreprocesses and splits the downloaded dataset into training, validation, and test sets. Args: dset (DatasetDict): The downloaded dataset object. split_val_from_train (bool, optional): Whether to split the validation set from the training set. If False, the validation set is split from the test set. Defaults to True. val_proportion (float, optional): The proportion of the training or test set to be used for the validation split. Defaults to 0.05. zPreprocessing z! to jsonl format and splitting...train validation) test_sizertrainingtestz.jsonlwzutf-8)encodingz Context: contextz Question: questionz Answer:inputanswerstextroutputoriginal_answers Nz split saved to )rr7r+r8gettrain_test_splitritemsropenwritejsondumpsriterdiris_dirshutilrmtreer9nameunlink)r)r4r<r= save_splits train_setval_set split_dataset split_namedataset output_filefexample json_linepr,r,r-r2fsH          z*SquadDataModule._preprocess_and_split_datacCsdS)zno opNr,r:r,r,r-reconfigure_limit_batchessz)SquadDataModule.reconfigure_limit_batches)NrNrrNFTrrrTFNN)r.N)Tr;)r8 __module__ __qualname____doc__rrr9rintrboolrrr(r3r1r floatr2re __classcell__r,r,r*r-rsv      %  4r)rRrVpathlibrtypingrrrrrrdatasetsr r "nemo.collections.llm.gpt.data.corer )nemo.collections.llm.gpt.data.fine_tuningr nemo.lightning.io.mixinr nemo.utilsr"nemo.collections.common.tokenizersr-nemo.collections.llm.gpt.data.packed_sequencerrr,r,r,r-s