o }oiuD@sddlmZmZmZmZmZddlZddlZddl Zddl m Z ddl m Z ddlmZddlmZddlmZddlmZmZmZGd d d ZdS) )AnyCallableDictOptionalTupleN) rearrange)parallel_state)Tensor) batch_mul)cat_outputs_cp)EDMSDE EDMSampler EDMScalingc @seZdZdZ       d7d d Zed d Zd dZdee e j fde de ee e j fe j ffddZde j de j dee e j ffddZdee e j fde j de j dee e j fde j de j f ddZde j fddZdefd d!Z " #d8ded$ed%edefd&d'Z "  # (d9ded$ed)edBd%ed*e de f d+d,Zd-e dede j fd.d/Zd:d0e j d1eede j fd2d3Zd;dee e fdee fd5d6ZdS)< EDMPipelinea EDMPipeline is a class that implements a diffusion model pipeline for video generation. It includes methods for initializing the pipeline, encoding and decoding video data, performing training steps, denoising, and generating samples. Attributes: p_mean: Mean for SDE process. p_std: Standard deviation for SDE process. sigma_max: Maximum noise level. sigma_min: Minimum noise level. _noise_generator: Generator for noise. _noise_level_generator: Generator for noise levels. sde: SDE process. sampler: Sampler for the diffusion model. scaling: Scaling for EDM. input_data_key: Key for input video data. input_image_key: Key for input image data. tensor_kwargs: Tensor keyword arguments. loss_reduce: Method for reducing loss. loss_scale: Scale factor for loss. aesthetic_finetuning: Aesthetic finetuning parameter. camera_sample_weight: Camera sample weight parameter. loss_mask_enabled: Flag for enabling loss mask. Methods: noise_level_generator: Returns the noise level generator. _initialize_generators: Initializes noise and noise-level generators. encode: Encodes input tensor using the video tokenizer. decode: Decodes latent tensor using video tokenizer. training_step: Performs a single training step for the diffusion model. denoise: Performs denoising on the input noise data, noise level, and condition. compute_loss_with_epsilon_and_sigma: Computes the loss for training. get_per_sigma_loss_weights: Returns loss weights per sigma noise level. get_condition_uncondition: Returns conditioning and unconditioning for classifier-free guidance. get_x0_fn_from_batch: Creates a function to generate denoised predictions with the sampler. generate_samples_from_batch: Generates samples based on input data batch. _normalize_video_databatch_inplace: Normalizes video data in-place on a CUDA device to [-1, 1]. draw_training_sigma_and_epsilon: Draws training noise (epsilon) and noise levels (sigma). random_dropout_input: Applies random dropout to the input tensor. get_data_and_condition: Retrieves data and conditioning for model input. N?P-C6*??c Cs||_||_||_||_||_||_||_||_d|_d|_ t |||||_ t |_ t||_d|_d|_dtjd|_d|_d|_dS)a_ Initializes the EDM pipeline with the given parameters. Args: net: The DiT model. vae: The Video Tokenizer (optional). p_mean (float): Mean for the SDE. p_std (float): Standard deviation for the SDE. sigma_max (float): Maximum sigma value for the SDE. sigma_min (float): Minimum sigma value for the SDE. sigma_data (float): Sigma value for EDM scaling. seed (int): Random seed for reproducibility. Attributes: vae: The Video Tokenizer. net: The DiT model. p_mean (float): Mean for the SDE. p_std (float): Standard deviation for the SDE. sigma_max (float): Maximum sigma value for the SDE. sigma_min (float): Minimum sigma value for the SDE. sigma_data (float): Sigma value for EDM scaling. seed (int): Random seed for reproducibility. _noise_generator: Placeholder for noise generator. _noise_level_generator: Placeholder for noise level generator. sde: Instance of EDMSDE initialized with p_mean, p_std, sigma_max, and sigma_min. sampler: Instance of EDMSampler. scaling: Instance of EDMScaling initialized with sigma_data. input_data_key (str): Key for input data. input_image_key (str): Key for input images. tensor_kwargs (dict): Tensor keyword arguments for device and dtype. loss_reduce (str): Method to reduce loss ('mean' or other). loss_scale (float): Scale factor for loss. Nvideo images_1024cuda)devicedtypemeanr)vaenetp_meanp_std sigma_max sigma_min sigma_dataseed_noise_generator_noise_level_generatorr sder samplerrscalinginput_data_keyinput_image_keytorchbfloat16 tensor_kwargs loss_reduce loss_scale) selfrrrrr r!r"r#r1g/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/diffusion/sampler/edm/edm_pipeline.py__init__Is$,  zEDMPipeline.__init__cCs|jS)z Generates noise levels for the EDM pipeline. Returns: Callable: A function or generator that produces noise levels. )r%)r0r1r1r2noise_level_generatorsz!EDMPipeline.noise_level_generatorcCsb|jdtjdd}|jdtjdd}tjdd|_|j|tj ||_ |j |j _ dS)a Initializes the random number generators for noise and noise level. This method sets up two generators: 1. A PyTorch generator for noise, seeded with a combination of the base seed and the data parallel rank. 2. A NumPy generator for noise levels, seeded similarly but without considering context parallel rank. Returns: None dT)with_context_parallelFr)rN) r#rget_data_parallel_rankr+ Generatorr$ manual_seednprandom default_rngr%r& _generator)r0 noise_seednoise_level_seedr1r1r2_initialize_generatorss   z"EDMPipeline._initialize_generators data_batch iterationreturnc Csf||\}}}|||\}}tr'|||||||\}} } || fS|||||||} | S)a Performs a single training step for the diffusion model. This method is responsible for executing one iteration of the model's training. It involves: 1. Adding noise to the input data using the SDE process. 2. Passing the noisy data through the network to generate predictions. 3. Computing the loss based on the difference between the predictions and the original data. Args: data_batch (dict): raw data batch draw from the training data loader. iteration (int): Current iteration number. Returns: A tuple with the output batch and the computed loss. )get_data_and_conditiondraw_training_sigma_and_epsilonsizeris_pipeline_last_stage#compute_loss_with_epsilon_and_sigma) r0rArBx0_from_data_batchx0 conditionsigmaepsilon output_batchpred_mseedm_loss net_outputr1r1r2 training_steps   zEDMPipeline.training_stepxtrLrKc Cs|jdi|j}|jdi|j}|j|d\}}}}|jdt|||d|}ts0|St||t||} t|| d|} | | fS)af Performs denoising on the input noise data, noise level, and condition Args: xt (torch.Tensor): The input noise data. sigma (torch.Tensor): The noise level. condition (dict[str, torch.Tensor]): conditional information Returns: Predicted clean data (x0) and noise (eps_pred). rL)x timestepsrNr1)tor-r(rr rrG) r0rSrLrKc_skipc_outc_inc_noiserQx0_predeps_predr1r1r2denoises zEDMPipeline.denoiserIrJrMc Cs|j||\}}|t||} trE|| ||\} } |j|d} || d} t| | }|| || || | d| |d}|| |fS|| ||} | S)ah Computes the loss for training. Args: data_batch: Batch of input data. x0_from_data_batch: Raw input tensor. x0: Latent tensor. condition: Conditional input data. epsilon: Noise tensor. sigma: Noise level tensor. Returns: The computed loss. rT)r\r])rJrSrLweights_per_sigmarK model_predmse_lossrP) r& marginal_probr rrGr^get_per_sigma_loss_weightsr contiguous)r0rArIrJrKrMrLrstdrSr\r]r`rOrPrNr1r1r2rHs&    z/EDMPipeline.compute_loss_with_epsilon_and_sigmacCs |d|jd||jdS)z Args: sigma (tensor): noise level Returns: loss weights per sigma noise level r_)r")r0rLr1r1r2rd"s z&EDMPipeline.get_per_sigma_loss_weightscCsl|j|dd\}}}d|vr(|d|d<|d|d<|j|dd\}}}||fS|j|dd\}}}||fS)zEReturns conditioning and unconditioning for classifier-free guidance.r dropout_rateneg_t5_text_embeddingst5_text_embeddingsneg_t5_text_mask t5_text_maskr)rD)r0rA_rK unconditionr1r1r2get_condition_uncondition,s  z%EDMPipeline.get_condition_uncondition?Fguidanceis_negative_promptcs8|\dtjdtjdtjffdd }|S)aF Creates a function to generate denoised predictions with the sampler. Args: data_batch: Batch of input data. guidance: Guidance scale factor. is_negative_prompt: Whether to use negative prompts. Returns: A callable to predict clean data (x0). noise_xrLrCcs4||\}}||\}}|||SN)r^)rsrLcond_x0rm uncond_x0rKrqr0rnr1r2x0_fnLsz/EDMPipeline.get_x0_fn_from_batch..x0_fn)ror+r )r0rArqrrrxr1rwr2get_x0_fn_from_batch9s&z EDMPipeline.get_x0_fn_from_batch# state_shape num_stepsc Cstdk}|jdur||j|||d}t|}|dt<tj|fi|jd|ji|j j }|j ||||j j d} |rPt } t | d| d} | S)a Generates samples based on input data batch. Args: data_batch: Batch of input data. guidance: Guidance scale factor. state_shape: Shape of the state. is_negative_prompt: Whether to use negative prompts. num_steps: Number of steps for sampling. solver_option: SDE Solver option. Returns: Generated samples from diffusion model. N)rr generator)r|r r_)seq_dimcp_group)rget_context_parallel_world_sizer$r@rylistr+randnr-r&r r'get_context_parallel_groupr ) r0rArqr{rrr| cp_enabledrx x_sigma_maxsamplesrr1r1r2generate_samples_from_batchSs  $z'EDMPipeline.generate_samples_from_batchx0_sizecCsX~|d}|jdur|tj|fi|jd|ji}|j|jdi|j|fS)a Draws training noise (epsilon) and noise levels (sigma). Args: x0_size: Shape of the input tensor. condition: Conditional input (unused). Returns: Noise level (sigma) and noise (epsilon). rNr~r1)r$r@r+rr-r&sample_trW)r0rrK batch_sizerMr1r1r2rE}s   z+EDMPipeline.draw_training_sigma_and_epsilon in_tensorrhcCs<|dur|n|j}ttd|t|jd||S)z Applies random dropout to the input tensor. Args: in_tensor: Input tensor. dropout_rate: Dropout probability (optional). Returns: Conditioning with random dropout applied. Nrr)rhr r+ bernoullionesshapetype_as)r0rrhr1r1r2random_dropout_inputs  "z EDMPipeline.random_dropout_input皙?cCsX|d|j}|}i}|D] \}}|dvr|||<q|j|d|d|d<|||fS)a Retrieves data and conditioning for model input. Args: data_batch: Batch of input data. dropout_rate: Dropout probability for conditioning. Returns: Raw data, latent data, and conditioning information. r)rrjrjrg crossattn_emb)r"itemsr)r0rArh raw_state latent_staterKkeyvaluer1r1r2rDs  z"EDMPipeline.get_data_and_condition)Nrrrrrr)rpF)rpNFrzrt)r)__name__ __module__ __qualname____doc__r3propertyr4r@dictstrr+r inttuplerRr^rHrdrrofloatboolrryrrrrErrrDr1r1r1r2r s+ C    $$"   4   * $r)typingrrrrrnumpyr:r+torch.distributedeinopsr megatron.corerr ,nemo.collections.diffusion.sampler.batch_opsr 3nemo.collections.diffusion.sampler.context_parallelr *nemo.collections.diffusion.sampler.edm.edmr r rrr1r1r1r2s