o }oin@s(ddlZddlZddlmZddlmZddlmZddl Z ddl Z ddl m ZddlmZmZmZddlmZddlmZddlmZdd Zd d Zd d ZddZddZd ddZGdddejZ Gddde Z!Gddde Z"Gddde Z#GdddejZ$Gddde j%j&jZ'dS)!N)partial)Dict)Dataset DatasetDict load_dataset) DataLoader)HFDatasetPackedSequenceHelper)loggingcCs0d|vr |dd}d|vr|dd}|S)zremoves split from name Args: name (str): partition name (e.g. "train[:100]") Returns: str: return partition name without any selector (e.g. "train"). [r+)split)namer\/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/llm/gpt/data/hf_dataset.py clean_splits  rcCs gd}dd|D}i}|D]\}}||vsJ|D]}|||<qq|D]} | || <q't|trNt|tsAJdt|t|}||}|||<nt|try|} t d| |D]\} }|| }||dussJ|||<qcnnt|t rt d|t|t sJt t t|D]\} } || }||dusJ|| ||<qn;t|trt dt|t rJ|} d | vrtd d | vr| d d } || }||dusJ|||<ntd t|t|ksJ|tt dd|} | d ksJd| |S)a. Given a dataset (e.g. from datasets.load_dataset or datasets.Dataset.from_dict) it returns a dictionary containing the corresponding dataset splits. For example: $ ds = load_dataset("dataset-id") $ ans = make_dataset_splits(ds) # `ds` contains the following $ print(ds) > DatasetDict({ > train: Dataset({ > features: ['id', 'title', 'context', 'question', 'answers'], > num_rows: 87599 > }) > validation: Dataset({ > features: ['id', 'title', 'context', 'question', 'answers'], > num_rows: 10570 > }) > }) # In this case the value of `ans` (returned value) will be: $ print(ans) > { > "train": Dataset .. (with 87599 rows), > "val": Dataset .. (with 10570 rows), > } traintestvalcSsi|]}|dqSNr).0_splitrrr Nsz'make_dataset_splits..z)Expected split to be a string, but got {}z'HF dataset has the following splits: {}Nz%Loaded HF dataset will use {} splits.z%Loaded HF dataset has a single split.r z!Split concatenation not supportedr rz-Expected split name to be None, str or a listcSs|duSrrxrrrzz%make_dataset_splits..z7Expected at least one split to have been initialized {})items isinstancerstrformattyperrkeysr infolist enumeratemap ValueErrorr setsumvalues)datasetr split_aliasesvalid_split_namesdataset_splitsalias_to_split split_name_split_aliasesaliasr dataset_split_namesalias_split_nameinum_init_splitsrrrmake_dataset_splits/s^            r7cCstpttjdddkS)z-returns whether it runs on a dist-environment WORLD_SIZE0)distis_initializedintosenvirongetrrrr!has_dist_env_init_or_rank_env_varsrAcCs|jdkr |dS|S)aEnsures that the input tensor has at least two dimensions by adding an extra batch dimension if necessary. Parameters ---------- tensor : torch.Tensor The input tensor to be batchified. Returns ------- torch.Tensor The tensor with an extra dimension added if it was originally 1-dimensional. Otherwise, the tensor is returned as-is. r:r)ndim unsqueeze_)tensorrrrbatchifys  rEcsttfdd|S)aExtracts the value of the given key from each dictionary in a list of dictionaries. Parameters ---------- batch : List[dict] A list of dictionaries. key : str The key whose values are to be extracted from each dictionary. Returns ------- List A list of values associated with the specified key, in the same order as the dictionaries in the input batch. cs|Srrrkeyrrrrz(extract_key_from_dicts..)r$r&)batchrGrrFrextract_key_from_dictssrIcs6ttt||r||fdd|DS)aPads each list in a batch of lists to the same length with a specified token. Parameters ---------- batch : List[List[int]] A batch of sequences (e.g., token IDs), where each sequence is a list of integers. pad_token_id : int The token ID to use for padding shorter sequences. pad_seq_len_divisible : int The value to use for padding sequence length so that it is divisible by pad_seq_len_divisible. Returns ------- List[List[int]] A batch of sequences where each inner list has been padded with the pad token to match the length of the longest sequence in the batch. cs"g|] }|gt|qSr)len)ritemmax_len pad_token_idrr s"z$pad_within_micro..)maxr&rJrHrNpad_seq_len_divisiblerrLrpad_within_microsrSc seZdZdZddddddddddd gd d ggd df d$fdd ZeddZd%ddZd&ddZe ddZ e ddZ e ddZ ddZ ddZd d!Zd'd"d#ZZS)(HFDatasetDataModulea$ A PyTorch Lightning DataModule for loading and managing datasets from the `datasets` library. Args: path_or_dataset (str | Dataset | DatasetDict): The dataset name from HF or a preloaded dataset. split (str | list, optional): The dataset split(s) to load (e.g., "train" or ["train", "validation"]). Defaults to None. collate_fn (callable, optional): Custom function for batching data; defaults to a padding-based collation. Defaults to None. num_workers (int, optional): Number of workers for data loading. Defaults to 2. pin_memory (bool, optional): Whether to use pinned memory for faster GPU transfers. Defaults to True. persistent_workers (bool, optional): Whether to keep worker threads alive between epochs. Defaults to True. seq_length (int, optional): Maximum sequence length for tokenized inputs. Defaults to 1024. micro_batch_size (int, optional): Batch size per device. Defaults to 2. pad_token_id (int, optional): Token ID used for padding sequences. Defaults to 0. use_dist_sampler (bool, optional): Whether to enable distributed sampling. Defaults to False. train_aliases (list, optional): Alternative names for the training split. Defaults to ["train", "training"]. test_aliases (list, optional): Alternative names for the test split. Defaults to ["test", "testing"]. val_aliases (list, optional): Alternative names for the validation split. Defaults to ["val", "validation", "valid", "eval"]. **kwargs: Additional arguments passed to `datasets.load_dataset`. Raises: ValueError: If `path_or_dataset` is not a valid dataset type (str, Dataset, or DatasetDict). Examples: Load a single split (train) from a dataset: ```python data_module = HFDatasetDataModule("rajpurkar/squad", split="train") ``` Load multiple splits (train and validation): ```python data_module = HFDatasetDataModule("rajpurkar/squad", split=["train", "validation"]) ``` Use a preloaded dataset: ```python from datasets import load_dataset dataset = load_dataset("imdb") data_module = HFDatasetDataModule(dataset, split="train") ``` Notes: - If `use_dist_sampler` is not enabled, but a distributed environment is detected, HFDatasetDataModule will use a distributed-sampler automatically. - If no collation function is provided, a default function with padding using `pad_token_id` is applied. NTrFrtrainingrtesting)r validationvalidevalreturnc st| dus J| | | d}t|tr)td|t|fd|i|}nt|ts3t|t r>td||}n t dt |t |||_ |dur[fdd_n|_|_|_|_|_|_| _| __dS)Nrz3Loading HF dataset from {}, this may take a moment.r zUsing passed HF dataset {}zFExpected `path_or_dataset` to be str, Dataset, DatasetDict, but got {}csj|jdSN)rNrR) collate_fnrNrrRselfrrrsz.HFDatasetDataModule.__init__..)super__init__rrr r#r rrrr'r!r7r. _collate_fn num_workers pin_memorypersistent_workers seq_lengthmicro_batch_sizerNuse_dist_samplerrR)r`path_or_datasetr r^rdrerfrgrhrNri train_aliases test_aliases val_aliasesrRkwargsr,r+ __class__r_rrbs6     zHFDatasetDataModule.__init__cKst|}td||d|S)z wraps Dataset's from_dict method)rjr Nr)r from_dictrT) dataset_dictr rnr+rrrrq-s zHFDatasetDataModule.from_dictcfdddDS)Default batch collatorc 6i|]}|tttt||dkrndqS loss_maskrrEtorch LongTensorrSrIrrGrHrRrNrrr5 z2HFDatasetDataModule.collate_fn..rr"r`rHrNrRrr|rr^3s zHFDatasetDataModule.collate_fncs<|dusJ|durfdd}t|jjj|jdS)zDataloader creatorNcsj|jjdSr])r^rNrRrr`rrrFs z6HFDatasetDataModule._make_dataloader..)rdrerfr^ batch_size)rrdrerfrh)r`r+r^rrr_make_dataloaderBs  z$HFDatasetDataModule._make_dataloadercC |jdS)zReturns the training partitionrr.rrrrrS zHFDatasetDataModule.traincCr)z Returns the validation partitionrrrrrrrXrzHFDatasetDataModule.valcCr)zReturns the test partitionrrrrrrr]rzHFDatasetDataModule.testcC||j|jS)Returns the train dataloaderrrrcrrrrtrain_dataloaderbz$HFDatasetDataModule.train_dataloadercCr)!Returns the validation dataloaderrrrcrrrrval_dataloaderfrz"HFDatasetDataModule.val_dataloadercCr)Returns the test dataloaderrrrcrrrrtest_dataloaderjrz#HFDatasetDataModule.test_dataloadercKstt|tr |g}nt|trn|dur|j}ntd|D]}|j|dur7|j|j|fi||j|<qdS)zrMaps a function to all/selected splits Additional arguments can be passed down to dataset's map via kwargsNzsplit_names must None/str/list)rrr$r.r"r'r&)r`function split_namesrnr0rrrr&ns   zHFDatasetDataModule.map)r\NrNr)NN)__name__ __module__ __qualname____doc__rb staticmethodrqr^rpropertyrrrrrrr& __classcell__rrrorrTs@3;      rTcsbeZdZdZ ddedeffdd Zdfd d Zdfd d Zd dZ ddZ ddZ Z S)HFDatasetDataModulePackeda Inherits HFDatasetDataModule class and overrides methods for adding packing functionality. Args: path_or_dataset (str | Dataset | DatasetDict): The dataset name from HF or a preloaded dataset. packed_sequence_size (int): Specifies the number of tokens to pack. split_across_pack [Optional(bool)]: If the last sample in a pack does not fit in ``packed_sequence_size``, split the sample into the next pack, or move it entirely to the beginning of the next pack. For pre-training, typically this is set to True for general text completion. For fine-tuning, typically this is set to False to avoid truncating sentences in instruct tuning. Default is False. max_packs (int): Maximum number of packs. FNsplit_across_pack max_packsc s*tj|fi|||_||_||_dSr)rarbpacked_sequence_sizerr)r`rjrrrrnrorrrbs z"HFDatasetDataModulePacked.__init__rcs t|||S)z Creates the attn_mask and append it to the batch as its required in case of packed sequences. Then calls HFDatasetDataModule's collate_fn. )rar^rrorrr^s z$HFDatasetDataModulePacked.collate_fncs8|dusJt||}||j|j|j}t||S)zj Pack the sequences in the dataset and then call HFDatasetDataModule's _make_dataloader() N)rpackrrrrar)r`r+r r^packed_seq_helper_classrorrrs  z*HFDatasetDataModulePacked._make_dataloadercC||jd|jS)rrrrrrrrz*HFDatasetDataModulePacked.train_dataloadercCr)rrrrrrrrrz(HFDatasetDataModulePacked.val_dataloadercCr)rrrrrrrrrz)HFDatasetDataModulePacked.test_dataloader)FNrr) rrrrboolr=rbr^rrrrrrrrorrs  rcsHeZdZdZd fdd ZeddZeddZed d d ZZ S)HellaSwagHFDataModulezKA data module for handling the HellaSwag dataset using HFDatasetDataModule.Rowan/hellaswagcsD|j|_|j|_t|}tjt|d|dg|Ri|dS)NiLr) eos_token pad_tokeneos_idrNrrarbrpreprocess_dataset)r` tokenizer dataset_nameargsrnr+rorrrbs,zHellaSwagHFDataModule.__init__cCs2|}|dd}tdd|}|dd}|S)zEPreprocesses text data by removing unwanted characters and artifacts.z [title]z. z\[.*?\]z  )stripreplaceresub)textrrr preprocesss   z HellaSwagHFDataModule.preprocesscCsj|dd|d}t|dd|}dd|dD}t|d }||||d||d }|S) z_Processes a document from the HellaSwag dataset into a structured format suitable for training.ctx_arctx_bactivity_labelz: cSsg|]}t|qSr)rr)rendingrrrrOsz5HellaSwagHFDataModule.process_doc..endingslabel)querychoicesgoldr) capitalizerrr=)docctxrrrout_docrrr process_docs z!HellaSwagHFDataModule.process_doc*cCsRtd|tj}dd}t|||d}|j|ddgd}|j|d}|S) z5Preprocesses a dataset for training a language model.zPreprocessing dataset...cSs,||d|dd}dd|dD|d<|S)NrT) max_length truncationcSsg|] }|dddgqS)r:Nir)rrrrrrOszVHellaSwagHFDataModule.preprocess_dataset..preprocess_batch.. input_idslabelsr)rHrransrrrpreprocess_batchszBHellaSwagHFDataModule.preprocess_dataset..preprocess_batch)rrT)batched)rattention_maskrseed)printr&rrrselect_columnsshuffle)rrr+rr_preprocessing_functionrrrrs    z(HellaSwagHFDataModule.preprocess_dataset)r)r) rrrrrbrrrrrrrrorrs  rcs4eZdZdZfddZddZfddZZS)SquadHFDataModulea A data module for handling the SQuAD dataset using HFDatasetDataModule. This class is responsible for tokenizing and formatting the SQuAD dataset for training language models. It extends `HFDatasetDataModule` and implements a prompt-based formatting function suitable for causal language modeling. Attributes: tokenizer: A tokenizer instance used to convert text into token IDs. c s2tjdi|||_|jj|j_|jj|_dS)z Initializes the SquadHFDataModule. Args: tokenizer: A tokenizer instance for processing text data. **kwargs: Additional arguments passed to the parent class (`HFDatasetDataModule`). Nr)rarbrrrrrN)r`rrnrorrrb s zSquadHFDataModule.__init__cCsd|dd|ddd|ddd g}tt|jj|\}}t|jd d }t|jd d }t|d krI|d urI|d |krI|d |t|d kr^|d ur^|d |kr^||t ||dd ||d d d gt|ddgt|dS)a Formats a given example into a structured prompt for training. This method converts a dataset example (containing context, question, and answer) into a structured format, tokenizes it, and prepares input IDs and labels for training a language model. Args: example (dict): A dictionary containing the following keys: - 'context': The passage from which the question is derived. - 'question': The question about the passage. - 'answers': A dictionary with a 'text' key containing the answer(s). Returns: dict: A dictionary containing: - 'input_ids': Tokenized input sequence (excluding the last token). - 'labels': Tokenized output sequence (excluding the first token). - 'loss_mask': A mask indicating which tokens contribute to the loss. z Context: contextz Question: questionz Answer:ranswersrrbos_idNrr:)rrrw) rr$r&r text_to_idsgetattrrJinsertappenddict)r`exampleformatted_text context_ids answer_idsrrrrrformatting_prompts_funcs    z)SquadHFDataModule.formatting_prompts_funccs(t||j|jddgdddS)z Prepares the dataset for training and applies formatting. Args: stage (str): The stage of training. FrU)idtitlerrr)rrremove_columnsN)rasetupr&rr`stagerorrr<s  zSquadHFDataModule.setup)rrrrrbrrrrrrorrs   &rcseZdZdZ            d%d ed ed ed ededededededeffdd Zd&deddfddZde fddZ de fddZ de fddZ de fd d!Z ed'd#d$ZZS)(HFMockDataModulezMA PyTorch Lightning DataModule for generating mock data for testing purposes.rVN'TFrg vocab_sizerhnum_train_samplesnum_val_samplesnum_test_samplesrdrerfcreate_attention_maskcsvt||_||_||_||_||_||_| |_| |_ | |_ dd|_ ||_ |dur9||d|||_dSdS)NcSstj|ddS)Nr)rN)rr^rrrrrksz+HFMockDataModule.__init__..r:) rarbrgrhrrrrdrerfrr^r)r`rgrrhrampup_batch_sizerrrrdrerfr vocab_file merges_filerRrorrrbPs  zHFMockDataModule.__init__rr\cCsRt|jd|j|j|j|_t|jd|j|j|j|_t|jd|j|j|j|_ dS)rrrZrN) _MockGPTDatasetrrrgr _train_dsr_val_dsr_test_dsrrrrrps* zHFMockDataModule.setupcC ||jS)r)_create_dataloaderrrrrrr z!HFMockDataModule.train_dataloadercCr)r)rrrrrrrrzHFMockDataModule.val_dataloadercCr)r)rrrrrrrrz HFMockDataModule.test_dataloadercCst||j|j|j|j|jdS)z(creates the dataloader for given dataset)rrdrerfr^)rrhrdrerfr^)r`r+rrrrsz#HFMockDataModule._create_dataloaderrcrs)rtc rurvrxr{r|rrrr}z/HFMockDataModule.collate_fn..rr~rQrr|rr^s zHFMockDataModule.collate_fn)rrVrNrrrrTFFNNNrr)rrrrr=rrbrrrrrrrrr^rrrrorrMsX      rcsheZdZdZ  ddededededed ed d ffd d Zd efddZd e ee ffddZ Z S)rz?A mock dataset for generating random data for testing purposes.Frrr num_samplesrgrrr\Ncst||_||_||_||_||_||_|r2t tj |j|jftj dtj ddf |_tj |jtj d |_tj|jtjd |_dS)N)dtype)rarbr rgrlengthrrnptrilonesfloat32newaxistolistrrwarangeint64 position_ids)r`rr rrgrrrorrrbs  z_MockGPTDataset.__init__cCs|jSr)rrrrr__len__sz_MockGPTDataset.__len__cCsrtjj|j|d}|j|j|jgtjd}|j|j|jgtjd}|||j |j d}|j r7|j |d<|S)Nr)sizer)rrrwrr) rrandom default_rngrintegersrrgrrrwrrr)r`idxnp_genrrrHrrr __getitem__s z_MockGPTDataset.__getitem__)Fr) rrrrr=rrrbrrr$rrrrrorrs*rr)(r>r functoolsrtypingrlightning.pytorchpytorchplnumpyrrytorch.distributed distributedr;datasetsrrrtorch.utils.datar8nemo.collections.llm.gpt.data.hf_dataset_packed_sequencer nemo.utilsr rr7rArErIrSLightningDataModulerTrrrrutilsdatarrrrrs2       P ?>@Pc