o }oi@sddlmZddlZddlmZmZddlmZddl m Z ddl m Z ddl mZGdd d ZGd d d ejjjZGd d d ejZdS)N)EVAL_DATALOADERSTRAIN_DATALOADERS) DataLoader) DiTConfig)MegatronDataSampler) pos_id_3dc@s2eZdZdZddddddZddZd d Zd S) PosEmb3Dz?Generates and provides 3D positional embeddings for video data.`i)max_tmax_hmax_wcCs||_||_||_|dSN)r r r generate_pos_id)selfr r r rm/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/diffusion/data/diffusion_fake_datamodule.py__init__s zPosEmb3D.__init__c CsBtjttj|jddtj|jddtj|jdddd|_dS)zBGenerates the positional ID grid based on max_t, max_h, and max_w.cpu)device)dimN)torchstackmeshgridaranger r r gridrrrrr#s zPosEmb3D.generate_pos_idcCsl||jks||jks||jkr(t|j||_t|j||_t|j||_||jd|d|d|fS)a<Retrieves a subset of the positional IDs for the specified dimensions. Parameters: t (int): Number of time frames. h (int): Height dimension. w (int): Width dimension. Returns: torch.Tensor: The positional IDs tensor with shape (t, h, w, 3). N)r r r maxrr)rthwrrr get_pos_id_3d.s  zPosEmb3D.get_pos_id_3dN)__name__ __module__ __qualname____doc__rrr"rrrrr s  r c@s>eZdZdZ  dddZddZdd Zd d Zd d ZdS)DiTVideoLatentFakeDatasetz:A fake dataset for generating synthetic video latent data. c Cs4||_||_||_||_||_||_||_||_dSr)r max_height max_width patch_size in_channelstext_dim text_seqlen seq_length) rn_framesr r r,r-crossattn_emb_sizemax_text_seqlenr0rrrrDs  z"DiTVideoLatentFakeDataset.__init__cCsdS)z$Returns the total number of samples.irrrrr__len__Xsz!DiTVideoLatentFakeDataset.__len__c Cs|j}|j}|j}|j}|j}tj|j||dtjdd}tj |j |j tjd}t j |||||ddd} ||tj|jdgtjdtj|j gtjdtj|jdftjdtj|jdtjddS) zGenerates a single sample of data. Parameters: idx (int): Index of the data sample. Returns: dict: A dictionary containing video latent data and related information. )dtypeg?)rr r!rr)videot5_text_embeddings seq_len_q seq_len_kvpos_ids loss_mask)r r*r+r,r-ronesr0bfloat16randnr/r.rr"reshapetensorshapeint32squeezezeros) ridxrr r!pc video_latenttext_embeddingpos_embrrr __getitem__\s   z%DiTVideoLatentFakeDataset.__getitem__cCstjjj|S)zA default implementation of a collation function. Users should override this method to define custom data loaders. )rutilsdata dataloaderdefault_collaterbatchrrr _collate_fnxsz%DiTVideoLatentFakeDataset._collate_fncCs ||S)aMethod that user passes as a functor to DataLoader. The method optionally performs neural type checking and adds types to the outputs. Please note, subclasses of Dataset should not implement `input_types`. Usage: dataloader = torch.utils.data.DataLoader( ...., collate_fn=dataset.collate_fn, .... ) Returns: Collated batch, with or without types. )rTrRrrr collate_fns z$DiTVideoLatentFakeDataset.collate_fnN)r(r)) r#r$r%r&rr4rMrTrUrrrrr'As   r'cseZdZdZ       dded ed ed ed ed ededdffdd ZddeddfddZ de fddZ de fddZ defddZZS)VideoLatentFakeDataModulezIA LightningDataModule for generating fake video latent data for training.rTNF model_configr0micro_batch_sizeglobal_batch_size num_workers pin_memoryuse_train_split_for_valreturnc s>t||_||_||_||_||_t|j||d|_dS)N)seq_lenrZr[) superrr0rZr[r\rYr data_sampler) rrYr0rZr[r\r] task_encoderr^ __class__rrrs  z"VideoLatentFakeDataModule.__init__stagecCs2t|jj|jj|jj|jj|jj|jjd|_dS)zSets up the dataset for training and validation. Parameters: stage (str): Optional stage argument (unused). )r1r r r,r-r2N) r'rY max_frames max_img_h max_img_w patch_spatialr-r2 _train_ds)rrgrrrsetups zVideoLatentFakeDataModule.setupcCt|ds |||jS)z Returns the training DataLoader.rlhasattrrm_create_dataloaderrlrrrrtrain_dataloader  z*VideoLatentFakeDataModule.train_dataloadercCrn)z"Returns the validation DataLoader.rlrorrrrval_dataloaderrsz(VideoLatentFakeDataModule.val_dataloadercKst|f|jdd|jd|S)zCreates a DataLoader for the given dataset. Parameters: dataset (Dataset): The dataset to load. **kwargs: Additional arguments for DataLoader. Returns: DataLoader: The DataLoader instance. T)r\r]persistent_workersrU)rr\rU)rdatasetkwargsrrrrqs z,VideoLatentFakeDataModule._create_dataloader)rWrrXrTNF)rf)r#r$r%r&rintboolrstrrmrrrrrtrrq __classcell__rrrdrrVs<  rV)lightning.pytorchpytorchplr!lightning.pytorch.utilities.typesrrtorch.utils.datar'nemo.collections.diffusion.models.modelrnemo.lightning.pytorch.pluginsrdiffusion_taskencoderrr rNrODatasetr'LightningDataModulerVrrrrs     'R