o }oi$1@sddlZddlmZddlZddlmZddlm Z m Z m Z ddl m Z ddlmZddlmZddZdd Zd d Zd d ZddZddZGdddejZdS)N)Dataset DatasetDict load_dataset) DataLoader)DistributedSampler)loggingcCs0d|vr |dd}d|vr|dd}|S)zremoves split from name Args: name (str): partition name (e.g. "train[:100]") Returns: str: return partition name without any selector (e.g. "train"). [r+)split)namer [/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/vlm/hf/data/hf_dataset.py clean_splits  rcCs gd}dd|D}i}|D]\}}||vsJ|D]}|||<qq|D]} | || <q't|trNt|tsAJdt|t|}||}|||<nt|try|} t d| |D]\} }|| }||dussJ|||<qcnnt|t rt d|t|t sJt t t|D]\} } || }||dusJ|| ||<qn;t|trt dt|t rJ|} d | vrtd d | vr| d d } || }||dusJ|||<ntd t|t|ksJ|tt dd|} | d ksJd| |S)a. Given a dataset (e.g. from datasets.load_dataset or datasets.Dataset.from_dict) it returns a dictionary containing the corresponding dataset splits. For example: $ ds = load_dataset("dataset-id") $ ans = make_dataset_splits(ds) # `ds` contains the following $ print(ds) > DatasetDict({ > train: Dataset({ > features: ['id', 'title', 'context', 'question', 'answers'], > num_rows: 87599 > }) > validation: Dataset({ > features: ['id', 'title', 'context', 'question', 'answers'], > num_rows: 10570 > }) > }) # In this case the value of `ans` (returned value) will be: $ print(ans) > { > "train": Dataset .. (with 87599 rows), > "val": Dataset .. (with 10570 rows), > } traintestvalcSsi|]}|dqSNr ).0_splitr r r Jsz'make_dataset_splits..z)Expected split to be a string, but got {}z'HF dataset has the following splits: {}Nz%Loaded HF dataset will use {} splits.z%Loaded HF dataset has a single split.r z!Split concatenation not supportedrrz-Expected split name to be None, str or a listcSs|duSrr xr r r vz%make_dataset_splits..z7Expected at least one split to have been initialized {})items isinstancerstrformattyperrkeysrinfolist enumeratemap ValueErrorr setsumvalues)datasetr split_aliasesvalid_split_namesdataset_splitsalias_to_split split_name_split_aliasesaliasr dataset_split_namesalias_split_nameinum_init_splitsr r r make_dataset_splits+s^            r5cCstpttjdddkS)z-returns whether it runs on a dist-environment WORLD_SIZE0)distis_initializedintosenvirongetr r r r !has_dist_env_init_or_rank_env_var{sr?cCs|jdkr |dS|S)aEnsures that the input tensor has at least two dimensions by adding an extra batch dimension if necessary. Parameters ---------- tensor : torch.Tensor The input tensor to be batchified. Returns ------- torch.Tensor The tensor with an extra dimension added if it was originally 1-dimensional. Otherwise, the tensor is returned as-is. r8r)ndim unsqueeze_)tensorr r r batchifys  rCcsttfdd|S)aExtracts the value of the given key from each dictionary in a list of dictionaries. Parameters ---------- batch : List[dict] A list of dictionaries. key : str The key whose values are to be extracted from each dictionary. Returns ------- List A list of values associated with the specified key, in the same order as the dictionaries in the input batch. cs|Srr rkeyr r rrz(extract_key_from_dicts..)r"r$)batchrEr rDr extract_key_from_dictssrGcs"ttt|fdd|DS)aPads each list in a batch of lists to the same length with a specified token. Parameters ---------- batch : List[List[int]] A batch of sequences (e.g., token IDs), where each sequence is a list of integers. pad_token_id : int The token ID to use for padding shorter sequences. Returns ------- List[List[int]] A batch of sequences where each inner list has been padded with the pad token to match the length of the longest sequence in the batch. cs"g|] }|gt|qSr )len)ritemmax_len pad_token_idr r s"z$pad_within_micro..)maxr$rHrFrLr rJr pad_within_microsrPcseZdZdZdddddddddddd gd d ggd f d)fdd ZeddZed*ddZdefddZ ddZ d+ddZ e ddZ e ddZe dd Zd!d"Zd#d$Zd%d&Zd,d'd(ZZS)-HFDatasetDataModuleaHFDatasetDataModule wraps HF's load_dataset (datasets library) so that it can be used within NeMo. Users can select whether to use an mcore-sampler via use_mcore_sampler arg. Usage examples: - loading a single split (train) from a dataset llm.HFDatasetDataModule("rajpurkar/squad", split="train") - loading multiple splits (train, validation) from a dataset llm.HFDatasetDataModule("rajpurkar/squad", split=["train", "validation"]) NTirFrtrainingrtesting)r validationvalidevalreturnc std|vr|dd|vr|d| dusJ| | |d}t|tr;td|t|fd|i|}n!t|tsEt|t rRtdt||}n t dtt |t |||_ |durofdd _n|_|_|_|_|_|_| _| _| _dS) Nuse_mcore_samplermcore_dataloader_typerzLoading HF dataset from r zUsing passed HF dataset zDExpected `path_or_dataset` to be str, Dataset, DatasetDict, but got ctj|jdSN)rLrQ collate_fnrLrselfr r rsz.HFDatasetDataModule.__init__..)super__init__poprrrr!rrrr%rr5r, _collate_fn num_workers pin_memorypersistent_workers seq_lengthmicro_batch_sizeglobal_batch_sizerLuse_dist_sampler)r`path_or_datasetr r^rerfrgrhrirjrLrk train_aliases test_aliases val_aliaseskwargsr*r) __class__r_r rbs8       zHFDatasetDataModule.__init__cKst|}td||d|S)z#Creates a Dataset from a dictionary)rlr Nr )r from_dictrQ) dataset_dictr rpr)r r r rss zHFDatasetDataModule.from_dictcsfdddDS)zCollate for VLM datac s(i|]}|tttt|qSr )rCtorch LongTensorrPrG)rrErOr r r s z2HFDatasetDataModule.collate_fn..r)r rOr rOr r^s  zHFDatasetDataModule.collate_fnstagecCs(|jstrd|_tddSdSdS)zsetups samplerTz#Turning on distributed data samplerN)rkr?rr!)r`rwr r r setups zHFDatasetDataModule.setupcCs|jrt|SdS)zreturns the data samplerN)rkr)r`r)r r r get_data_samplersz$HFDatasetDataModule.get_data_samplerc sD|dusJ|durfdd}t|jjj|j|dS)zCreates a dataloaderNcr[r\r]rr_r r r^+sz8HFDatasetDataModule._make_dataloader..collate_fn)rerfrgr^ batch_sizesampler)rrerfrgriry)r`r)r^r r_r _make_dataloader%s  z$HFDatasetDataModule._make_dataloadercC |jdS)zTrain data splitrr,r_r r r r8 zHFDatasetDataModule.traincCr})zValidation data splitrr~r_r r r r=rzHFDatasetDataModule.valcCr})zTesting data splitrr~r_r r r rBrzHFDatasetDataModule.testcC||j|jS)z(Creates a dataloader for the train split)r|rrdr_r r r train_dataloaderGz$HFDatasetDataModule.train_dataloadercCr)z-Creates a dataloader for the validation split)r|rrdr_r r r val_dataloaderKrz"HFDatasetDataModule.val_dataloadercCr)z'Creates a dataloader for the test split)r|rrdr_r r r test_dataloaderOrz#HFDatasetDataModule.test_dataloadercKstt|tr |g}nt|trn|dur|j}ntd|D]}|j|dur7|j|j|fi||j|<qdS)zrMaps a function to all/selected splits Additional arguments can be passed down to dataset's map via kwargsNzsplit_names must None/str/list)rrr"r,r r%r$)r`function split_namesrpr.r r r r$Ss   zHFDatasetDataModule.map)rXN)rr)NN)__name__ __module__ __qualname____doc__rb staticmethodrsr^rrxryr|propertyrrrrrrr$ __classcell__r r rqr rQsF:      rQ)r<lightning.pytorchpytorchplrutorch.distributed distributedr9datasetsrrrtorch.utils.datartorch.utils.data.distributedr nemo.utilsrrr5r?rCrGrPLightningDataModulerQr r r r s     P