o }oiE9@sddlmZmZmZddlmZddlZddl Z ddl m Z m Z ddl mZddlmZmZddlmZddlmZmZddlmZGd d d ejZGd d d eZdS) )DictListOptionalN)EVAL_DATALOADERSTRAIN_DATALOADERS)data) DataLoaderDataset) AutoProcessor) AudioToken ImageToken)MegatronDataSamplercseZdZdZ               d'd ed eed ededededeeededededededeffdd Zd(de ddfddZ de fdd Z de fd!d"Zde fd#d$Zdefd%d&ZZS))MockDataModulez A mock data module for AVLM training, validation, and testing. Provides datasets and data loaders for training, validation, and testing phases. Includes data sampling and preprocessing for multimodal tasks. N@逖TF seq_lengthdecoder_seq_lengthimage_embedding_tokensaudio_embedding_tokensmicro_batch_sizeglobal_batch_sizerampup_batch_sizenum_train_samplesnum_val_samplesnum_test_samples num_workers pin_memorypersistent_workerscst||_||_| |_| |_| |_||_||_||_ ||_ | |_ ||_ d}t |j|_||_||_||_t|j|j|| | d|_dS)a Initializes the mock data module with data sampling and preprocessing configurations. Args: seq_length (int): Maximum sequence length for tokens. decoder_seq_length (Optional[int]): Sequence length for the decoder. tokenizer: Tokenizer for text processing. image_processor: Processor for image preprocessing. audio_processor: Processor for audio preprocessing. image_embedding_tokens: Number of image embedding tokens for one image. audio_embedding_tokens: Number of audio embedding tokens for one audio. micro_batch_size (int): Batch size per GPU. global_batch_size (int): Total batch size across GPUs. rampup_batch_size (Optional[List[int]]): Batch size ramp-up schedule. num_train_samples (int): Number of training samples. num_val_samples (int): Number of validation samples. num_test_samples (int): Number of testing samples. num_workers (int): Number of workers for data loading. pin_memory (bool): Whether to pin memory for data loaders. persistent_workers (bool): Whether to keep workers alive after the first iteration. zllava-hf/llava-1.5-7b-hf)seq_lendecoder_seq_lenrrrN)super__init__rr#rrrrr r!rr tokenizerr from_pretrainedimage_processoraudio_processorrrr data_sampler)selfrrr&r(r)rrrrrrrrrr r! model_name __class__S/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/avlm/data/mock.pyr%%s0 ( zMockDataModule.__init__stagereturnc Cs|t|j|j|j|j|jd|j|jd|_t|j|j|j|j|jd|j |jd|_ t|j|j|j|j|jd|j |jd|_ dS)z Sets up the training, validation, and testing datasets. Args: stage (str): Stage of the setup ('train', 'valid', 'test'). train)r&r(r)rrname num_samplesrvalidtestN) _MockAVLMDatasetr&r(r)rrrr _train_dsr_validation_dsr_test_ds)r+r2r/r/r0setupfs<   zMockDataModule.setupcCt|ds |||jS)zz Creates the training data loader. Returns: TRAIN_DATALOADERS: Training data loader. r:)hasattrr=_create_dataloaderr:r+r/r/r0train_dataloader  zMockDataModule.train_dataloadercCr>)z} Creates the validation data loader. Returns: EVAL_DATALOADERS: Validation data loader. r;)r?r=r@r;rAr/r/r0val_dataloaderrCzMockDataModule.val_dataloadercCr>)zw Creates the testing data loader. Returns: TEST_DATALOADERS: Testing data loader. r<)r?r=r@r<rAr/r/r0test_dataloaderrCzMockDataModule.test_dataloadercKs"t|f|j|j|j|jd|S)a Creates a generic data loader for the given dataset. Args: dataset: The dataset for which the data loader is created. **kwargs: Additional arguments for the DataLoader. Returns: DataLoader: The created data loader. )rr r! collate_fn)rrr r!rF)r+datasetkwargsr/r/r0r@s z!MockDataModule._create_dataloader)rNNNNrrrrNrrrrTF)r1)__name__ __module__ __qualname____doc__intrrboolr%strr=rrBrrDrErr@ __classcell__r/r/r-r0rsd      A&   rc seZdZdZ ddededededdf fd d Zdefd d Zd edej fddZ de ee j ffddZddZddZZS)r9a A mock dataset for AVLM, generating synthetic multimodal data. Attributes: tokenizer: Tokenizer for text inputs. image_processor: Processor for image inputs. audio_processor: Processor for audio inputs. name (str): Name of the dataset ('train', 'valid', 'test'). num_samples (int): Number of samples in the dataset. seq_length (int): Sequence length for text tokens. seed (int): Random seed for reproducibility. *r5r6rseedr3Nc st||_||_|j|_|j} | d| d|_|_||_| |_ t j |jt j d|_ t j|jt jd|_||_||_||_||_||_dS)a Initializes the mock dataset with synthetic multimodal data. Args: tokenizer: Tokenizer for text inputs. image_processor: Processor for image inputs. audio_processor: Processor for audio inputs. name (str): Dataset name ('train', 'valid', 'test'). num_samples (int): Total number of samples in the dataset. seq_length (int): Sequence length for text tokens. seed (int): Random seed for data generation. heightwidthdtypeN)r$r%r5r vocab_size crop_size image_height image_widthlengthrRtorchonesfloat loss_maskarangeint64 position_idsr&r(r)rr) r+r&r(r)rrr5r6rrRrXr-r/r0r%s  z_MockAVLMDataset.__init__cCs|jS)zy Returns the length of the dataset. Returns: int: Number of samples in the dataset. )r[rAr/r/r0__len__sz_MockAVLMDataset.__len__idxcCs,tjj|j|d}|j|j|jgtjdS)z Generates synthetic text data. Args: idx (int): Index of the sample. Returns: np.ndarray: Synthetic text token IDs. rRsizerV)nprandom default_rngrRintegersrWrra)r+rdnp_genr/r/r0 _get_texts z_MockAVLMDataset._get_textc Cstjj|j|d}d}d}t|j|j|jdgtj d}dd|j gdd|j gg}t j ||dd|dd<t j ||dd|dd<dd|j gd d |j gg}tj ||dd|dd<tj ||dd|dd<|}|d d }|dd }tjt|tjd } |t|jd |j|jgtjdg} g} g} | D]} |jj| ddddd}| || |jdqt| } tj| tjd } tj||j|jggtjd }td}t|jdd||gd}t|jd||gd}|||j| |j| || ||d S)z Generates a synthetic multimodal sample. Args: idx (int): Index of the sample. Returns: Dict[str, torch.Tensor]: A dictionary containing synthetic tokens, images, and metadata. rerrfiriiNrUptF)return_tensors do_rescale pixel_valuesgKAgg?)rgi>) tokenslabelsr_attention_maskrbimages image_sizesnum_image_tilesaudios audio_lengths) rhrirjrRr\ from_numpyrkrWrrarr token_idrr cloner]lenlongrYrZfloat32r( preprocessappendshapestacktensorrMuniformr_rb)r+rdrl num_images num_audiosrvimages_tokens_indexaudios_tokens_indexrwrxryprocessed_imagesr{imageprocessed_imagerzaudio_max_lengthr|r}r/r/r0 __getitem__sX "    z_MockAVLMDataset.__getitem__cCstj|}|djdg|djddR|d<|d|d<|djdg|djddR|d<|djdg|djddR|d<|d|d<|S) z A default implementation of a collation function. Users should override this method to define custom data loaders. ryrprNr{rzr|r})r dataloaderdefault_collate contiguousviewrflatten)r+batchcollated_batchr/r/r0 _collate_fnRs ,&,z_MockAVLMDataset._collate_fncCs ||S)aMethod that user pass as functor to DataLoader. The method optionally performs neural type checking and add types to the outputs. Please note, subclasses of Dataset should not implement `input_types`. # Usage: dataloader = torch.utils.data.DataLoader( ...., collate_fn=dataset.collate_fn, .... ) Returns ------- Collated batch, with or without types. )r)r+rr/r/r0rFas z_MockAVLMDataset.collate_fn)rQ)rIrJrKrLrOrMr%rcrhndarrayrmrr\TensorrrrFrPr/r/r-r0r9s&   ,  @r9)typingrrrlightning.pytorchpytorchplnumpyrhr\!lightning.pytorch.utilities.typesrr torch.utilsrtorch.utils.datarr transformersr /nemo.collections.multimodal.data.energon.configr r nemo.lightning.pytorch.pluginsr LightningDataModulerr9r/r/r/r0s    &