o }oi!@sddlZddlZddlmZmZmZddlmZmZddl m Z ddl m Z ddl mZddlmZer:ddlmZGd d d e eZdS) N) TYPE_CHECKINGListOptional) DatasetDict load_dataset)get_dataset_root)FineTuningDataModule)IOMixin)logging) TokenizerSpeccseZdZdZ            d)d ed ed ededdededeeedededededededeffdd Z d*ddZ dd Z !d+d"e d#ed$e fd%d&Zd'd(ZZS),SquadDataModuleaA data module for fine-tuning on the Squad dataset. This class inherits from the `FineTuningDataModule` class and is specifically designed for fine-tuning models on the Stanford Question Answering Dataset (SQuAD). It handles data download, preprocessing, splitting, and preparing the data in a format suitable for training, validation, and testing. Args: force_redownload (bool, optional): Whether to force re-download the dataset even if it exists locally. Defaults to False. delete_raw (bool, optional): Whether to delete the raw downloaded dataset after preprocessing. Defaults to True. See FineTuningDataModule for the other args NFT dataset_root seq_lengthseq_length_dec tokenizerr micro_batch_sizeglobal_batch_sizerampup_batch_sizeforce_redownload delete_rawseedmemmap_workers num_workers pin_memorypersistent_workerscsD||_| |_tj|durtdn|||||||| | | | |d dS)Nsquad) rrrrrrrrrrrr )rrsuper__init__r)selfrrrrrrrrrrrrrr  __class__V/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/llm/t5/data/squad.pyr#+s  zSquadDataModule.__init__returncCs.|jr |js dS|}|j|dddS)NF)split_val_from_train) train_pathexistsr_download_data_preprocess_and_split_data)r$dsetr'r'r( prepare_dataNszSquadDataModule.prepare_datacCs8td|jjdtdt|j|jrddSddS)Nz Downloading z...r!r) cache_dir download_mode)r infor&__name__rstrrrr$r'r'r(r-VszSquadDataModule._download_data皙?r/r*val_proportionc Cstd|jjdi}|d}|d}|r2|j||jd}|d|d<|d|d<||d<n|j||jd}||d<|d|d<|d|d<|D]l\}} |j|d} | j d d d G} | D]<} i} d | d dd| ddd| d| d<| ddd| d<|dkr| dd| d<| t | dqdWdn1swYt|d| qN|j r|jD]}|rt|qdt|jvr|qdSdS)aPreprocesses and splits the downloaded dataset into training, validation, and test sets. Args: dset (DatasetDict): The downloaded dataset object. split_val_from_train (bool, optional): Whether to split the validation set from the training set. If False, the validation set is split from the test set. Defaults to True. val_proportion (float, optional): The proportion of the training or test set to be used for the validation split. Defaults to 0.05. zPreprocessing z! to jsonl format and splitting...train validation) test_sizertrainingtestz.jsonlwzutf-8)encodingzTitle: title z Paragraph: contextz Question: questioninputanswerstextroutputoriginal_answers Nz split saved to )r r3r&r4gettrain_test_splitritemsropenwritejsondumpsriterdiris_dirshutilrmtreer5nameunlink)r$r/r*r8 save_splits train_setval_set split_dataset split_namedataset output_filefexample json_linepr'r'r(r.^sd          z*SquadDataModule._preprocess_and_split_datacCsdS)Nr'r6r'r'r(reconfigure_limit_batchessz)SquadDataModule.reconfigure_limit_batches)Nr rNrrNFTrrrTF)r)N)Tr7)r4 __module__ __qualname____doc__r5intrrboolr#r0r-rfloatr.rb __classcell__r'r'r%r(r sp       #  s