o }oi[/@sddlmZmZmZddlmZddlZddl Z ddl m Z m Z ddl mZddlmZmZddlmZmZddlmZddlmZdd lmZdd lmZdd lmZGd d d ejZ GdddeZ!dS))DictListOptionalN)EVAL_DATALOADERSTRAIN_DATALOADERS)data) DataLoaderDataset) AutoProcessorLlavaNextConfig) AutoTokenizer)get_number_of_features)IMAGE_TOKEN_INDEX)MegatronDataSampler)loggingcseZdZdZ             d#d ed eed ed ed eeededededededeffdd Zd$de ddfddZ de fddZ de fddZde fdd Zdefd!d"ZZS)%MockDataModulez A mock data module for LLaVA-Next training, validation, and testing. Provides datasets and data loaders for training, validation, and testing phases. Includes data sampling and preprocessing for multimodal tasks. N逖TF seq_lengthdecoder_seq_lengthmicro_batch_sizeglobal_batch_sizerampup_batch_sizenum_train_samplesnum_val_samplesnum_test_samples num_workers pin_memorypersistent_workerscst||_||_||_| |_| |_| |_| |_| |_ ||_ ||_ d}d}|dus/|dur;t dd}t|}|p@t||_|pF|j|_t|j|j|||d|_dS)a Initializes the mock data module with data sampling and preprocessing configurations. Args: seq_length (int): Maximum sequence length for tokens. decoder_seq_length (Optional[int]): Sequence length for the decoder. tokenizer: Tokenizer for text processing. image_processor: Processor for image preprocessing. micro_batch_size (int): Batch size per GPU. global_batch_size (int): Total batch size across GPUs. rampup_batch_size (Optional[List[int]]): Batch size ramp-up schedule. num_train_samples (int): Number of training samples. num_val_samples (int): Number of validation samples. num_test_samples (int): Number of testing samples. num_workers (int): Number of workers for data loading. pin_memory (bool): Whether to pin memory for data loaders. persistent_workers (bool): Whether to keep workers alive after the first iteration. NzYProcessor or tokenizer are not provided! Fall back to `llava-hf/llava-v1.6-vicuna-7b-hf`.z llava-hf/llava-v1.6-vicuna-7b-hf)seq_lendecoder_seq_lenrrr)super__init__rr#rrrrrr rrrwarningr from_pretrainedr tokenizerimage_processorr data_sampler)selfrrr(r)rrrrrrrrr model_name processor __class__]/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/vlm/llava_next/data/mock.pyr%(s8 "   zMockDataModule.__init__r!stagereturncCsRt|j|jd|j|j|_t|j|jd|j|j|_t|j|jd|j|j|_ dS)z Sets up the training, validation, and testing datasets. Args: stage (str): Stage of the setup ('train', 'valid', 'test'). trainvalidtestN) _MockLlavaNextDatasetr(r)rr _train_dsr_validation_dsr_test_ds)r+r2r0r0r1setupis zMockDataModule.setupcCt|ds |||jS)zz Creates the training data loader. Returns: TRAIN_DATALOADERS: Training data loader. r8)hasattrr;_create_dataloaderr8r+r0r0r1train_dataloaderz  zMockDataModule.train_dataloadercCr<)z} Creates the validation data loader. Returns: EVAL_DATALOADERS: Validation data loader. r9)r=r;r>r9r?r0r0r1val_dataloaderrAzMockDataModule.val_dataloadercCr<)zw Creates the testing data loader. Returns: TEST_DATALOADERS: Testing data loader. r:)r=r;r>r:r?r0r0r1test_dataloaderrAzMockDataModule.test_dataloadercKs"t|f|j|j|j|jd|S)a Creates a generic data loader for the given dataset. Args: dataset: The dataset for which the data loader is created. **kwargs: Additional arguments for the DataLoader. Returns: DataLoader: The created data loader. )rrr collate_fn)rrrr rD)r+datasetkwargsr0r0r1r>s z!MockDataModule._create_dataloader) rNNNrrNrrrrTF)r!)__name__ __module__ __qualname____doc__intrrboolr%strr;rr@rrBrCrr> __classcell__r0r0r.r1r sV       A   rc seZdZdZ ddededededdf fd d Zdefd d Zd edej fddZ de ee j ffddZddZddZZS)r7a A mock dataset for LLaVA-Next, generating synthetic multimodal data. Attributes: tokenizer: Tokenizer for text inputs. image_processor: Processor for image inputs. name (str): Name of the dataset ('train', 'valid', 'test'). num_samples (int): Number of samples in the dataset. seq_length (int): Sequence length for text tokens. seed (int): Random seed for reproducibility. *name num_samplesrseedr3Ncs^t||_||_|j|_|j}|d|d|_|_||_||_ ||_ ||_ t |_ dS)a Initializes the mock dataset with synthetic multimodal data. Args: tokenizer: Tokenizer for text inputs. image_processor: Processor for image inputs. name (str): Dataset name ('train', 'valid', 'test'). num_samples (int): Total number of samples in the dataset. seq_length (int): Sequence length for text tokens. seed (int): Random seed for data generation. heightwidthN)r$r%rPr vocab_size crop_size image_height image_widthlengthrRr(r)r hf_config)r+r(r)rPrQrrRrVr.r0r1r%s  z_MockLlavaNextDataset.__init__cCs|jS)zy Returns the length of the dataset. Returns: int: Number of samples in the dataset. )rYr?r0r0r1__len__sz_MockLlavaNextDataset.__len__idxcCs,tjj|j|d}|j|j|jgtjdS)z Generates synthetic text data. Args: idx (int): Index of the sample. Returns: np.ndarray: Synthetic text token IDs. rRsizedtype)nprandom default_rngrRintegersrUrint64)r+r\np_genr0r0r1 _get_texts z_MockLlavaNextDataset._get_textc CsXtjj|j|d}t|j|j|jdgtj d}t |j |j |j |j |j j|j jj}t|ddttg||ddgd}|}t|jd|j |j gtjd}|dd}|dd}t|}tj|tjd }tj|tj d } tjt|tjd } tj|j |j ggtjd } |jj|d d d d d} | jd} | |||| | | | dS)z Generates a synthetic multimodal sample. Args: idx (int): Index of the sample. Returns: Dict[str, torch.Tensor]: A dictionary containing synthetic tokens, images, and metadata. r]r^Nr)r`ptF)return_tensors do_rescale pixel_values)mediatokenslabels loss_mask position_ids image_sizesnum_media_tilesattention_mask)rarbrcrRtorch from_numpyrdrUrrer rWrXrZimage_grid_pinpoints vision_config patch_size concatenatetensorrclonefloat32lenonesfloatarangelongr) preprocessshape)r+r\rfrqnum_image_tokensrrimagesrrsrtrwru image_arrayrvr0r0r1 __getitem__s> ".    z!_MockLlavaNextDataset.__getitem__cCshtj|}|djdg|djddR|d<|djdg|djddR|d<|S)z A default implementation of a collation function. Users should override this method to define custom data loaders. rprkriNru)r dataloaderdefault_collate contiguousviewr)r+batchcollated_batchr0r0r1 _collate_fn&s ,&z!_MockLlavaNextDataset._collate_fncCs ||S)aMethod that user pass as functor to DataLoader. The method optionally performs neural type checking and add types to the outputs. Please note, subclasses of Dataset should not implement `input_types`. # Usage: dataloader = torch.utils.data.DataLoader( ...., collate_fn=dataset.collate_fn, .... ) Returns ------- Collated batch, with or without types. )r)r+rr0r0r1rD3s z _MockLlavaNextDataset.collate_fn)rO)rGrHrIrJrMrKr%r[randarrayrgrrxTensorrrrDrNr0r0r.r1r7s&$  / r7)"typingrrrlightning.pytorchpytorchplnumpyrarx!lightning.pytorch.utilities.typesrr torch.utilsrtorch.utils.datarr transformersr r =nemo.collections.common.tokenizers.huggingface.auto_tokenizerr +nemo.collections.vlm.llava_next.model.utilsr 0nemo.collections.vlm.neva.data.multimodal_tokensrnemo.lightning.pytorch.pluginsr nemo.utilsrLightningDataModulerr7r0r0r0r1s