o }oim0@sddlmZmZmZmZmZmZddlZddl Z ddl Z ddl Z ddl Z ddl Z ddlmZddlmZddlmZddlmZddlmZddlmZdd lmZdd lmZmZdd l mZm Z dd l!m"Z#dd l$m%Z%ddl&m'Z'ddl(m)Z)ddl*m+Z+ddl,m-Z-ddl.m/Z/m0Z0m1Z1ddl2m3Z3Gddde0Z4GdddeZ5Gddde+Z6d%ddZ7Gdddee-Z8ej9j:e%j;dd ejd#krej9j?e%j;e=d$dSdS)&)AnyCallableDictOptionalSequenceTupleN) VAEGenerator)LPIPSWithDiscriminator) AutoencoderKL)parallel_state) ModelType)MegatronModule)TransformerConfig)DefaultTaskEncoder ImageSample)Tensornn) lightning)llm)DiffusionDataModule)pretrain)GPTModel)IOMixin)DataTMegatronLossReduction ReductionT)OptimizerModulec@sDeZdZdZdededeeeffddZde edefdd Z d S) AvgLossReductionz5Performs average loss reduction across micro-batches.batch forward_outreturncCs|}|d|ifS)z Forward pass for loss reduction. Args: batch: The batch of data. forward_out: The output tensor from forward computation. Returns: A tuple of (loss, reduction dictionary). avg)mean)selfrrlossr%\/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/diffusion/vae/train_vae.pyforward/s zAvgLossReduction.forwardlosses_reduced_per_micro_batchcCstdd|D}|S)z Reduce losses across multiple micro-batches by averaging them. Args: losses_reduced_per_micro_batch: A sequence of loss dictionaries. Returns: The averaged loss tensor. cSsg|]}|dqS)r!r%).0r$r%r%r& Gsz+AvgLossReduction.reduce..)torchstackr")r#r(lossesr%r%r&reduce=s zAvgLossReduction.reduceN) __name__ __module__ __qualname____doc__rrrrr'rr.r%r%r%r&r,srcs<eZdZdZd fdd ZddZdedd fd d ZZS) VAEz%Variational Autoencoder (VAE) module.Fcst||rtddd|_n tj|tjd|_tjddtjd}| }|j fdd | D}|jj |d d ~t d d dddddddd d dd |_ dS)a Initialize the VAE model. Args: config: Transformer configuration. pretrained_model_name_or_path: Path or name of the pretrained model. search_vae: Flag to indicate whether to search for a target VAE using AutoVAE. i)input_resolutioncompression_ratio) weight_dtypez(stabilityai/stable-diffusion-xl-base-1.0vae) subfolderr7cs2i|]\}}|vr||kr||qSr%)numel)r)kvvae_dictr%r& es2z VAE.__init__..F)strictiQggư>g??hinge) disc_start logvar_init kl_weightpixelloss_weightdisc_num_layersdisc_in_channels disc_factor disc_weightperceptual_weight use_actnormdisc_conditional disc_lossN)super__init__rr8r from_configr+bfloat16from_pretrained state_dictitemsload_state_dictr vae_loss)r#configpretrained_model_name_or_path search_vaesdxl_vaesd_dictpre_dict __class__r=r&rQNs4   z VAE.__init__cCsP|j|j}|}|j|j}|j|||d||jjjjd\}}|||fS)a Forward pass through the VAE. Args: target: Target images. global_step: Current global step. Returns: A tuple (aeloss, log_dict_ae, pred) containing the loss, log dictionary, and predictions. r)inputsreconstructions posteriors optimizer_idx global_step last_layer) r8encode latent_distsampledecoderXdecoderconv_outweight)r#targetre posteriorzpredaeloss log_dict_aer%r%r&r'xs   z VAE.forward input_tensorr NcCsdS)zk Set input tensor. Args: input_tensor: The input tensor to the model. Nr%)r#rtr%r%r&set_input_tensorszVAE.set_input_tensor)F) r/r0r1r2rQr'rru __classcell__r%r%r_r&r3Ks *r3c seZdZdZ  ddedeedeeej gej fffdd Z dd d Z de ee ffd d Zd dZddejfddZddejfddZedefddZedefddZdfdd ZZS)VAEModelzA GPTModel wrapper for the VAE.NrZoptimmodel_transformcs2||_tdddd}tj|_tj|||ddS)z Initialize the VAEModel. Args: pretrained_model_name_or_path: Path or name of the pretrained model. optim: Optional optimizer module. model_transform: Optional function to transform the model. ) num_layers hidden_sizenum_attention_heads)rxryN)rZrr encoder_or_decoder model_typerPrQ)r#rZrxryrYr_r%r&rQszVAEModel.__init__r cCs"t|dst|j|j|_dSdS)z/Configure the model by initializing the module.moduleN)hasattrr3rYrZrr#r%r%r&configure_models zVAEModel.configure_modelcCs$t|d}d|jjdtjddiS)z Perform a single data step to fetch a batch from the iterator. Args: dataloader_iter: The dataloader iterator. Returns: A dictionary with 'pixel_values' ready for the model. r pixel_valuescudaT)devicedtype non_blocking)nextimagetor+rS)r#dataloader_iterrr%r%r& data_steps zVAEModel.data_stepcOs|j|i|S)z Forward pass through the underlying module. Args: *args: Variable length argument list. **kwargs: Arbitrary keyword arguments. Returns: The result of forward pass of self.module. )r)r#argskwargsr%r%r&r's zVAEModel.forwardcCs2||d|j\}}}tjdkr|||S)z Perform a single training step. Args: batch: The input batch. batch_idx: Batch index. Returns: The loss tensor. rr)rer+ distributedget_ranklog_dict)r#r batch_idxr$rsrqr%r%r& training_steps  zVAEModel.training_stepc Cs||d|j\}}}tj|d|gdd}|ddd}d}t|krbtj|kr=ddt t D}nd}tjj |||t d |durb| |td d dt|Di|S) z Perform a single validation step. Args: batch: The input batch. batch_idx: Batch index. Returns: The loss tensor. rr)axisrBrzcSsg|]}dqSNr%)r)_r%r%r&r*sz,VAEModel.validation_step..N)groupz'Original (left), Reconstruction (right)cSs"g|] \}}ttj|qSr%)wandbImage torchvisionutils make_grid)r)rrr%r%r&r*s)rer+catcpuclampr get_data_parallel_src_rankrrrangeget_data_parallel_world_size gather_objectget_data_parallel_grouprrlog enumerate) r#rrr$rsrqr wandb_rank gather_listr%r%r&validation_steps(    zVAEModel.validation_stepcC|jst|_|jS)z/Returns the loss reduction method for training.)_training_loss_reductionrrr%r%r&training_loss_reductionz VAEModel.training_loss_reductioncCr)z1Returns the loss reduction method for validation.)_validation_loss_reductionrrr%r%r&validation_loss_reductionrz"VAEModel.validation_loss_reductioncs<t|jjdurt|ddrd|j_d|_dSdSdS)zz Hook to handle zero grad on validation model step. Used here to skip first validation on resume. N_restarting_skip_val_flagTF)rPon_validation_model_zero_gradtrainer ckpt_pathgetattrsanity_checkingrrr_r%r&rs  z&VAEModel.on_validation_model_zero_grad)NN)r Nr)r/r0r1r2strrrrrModulerQrrrrr'r+rrrpropertyrrrrrvr%r%r_r&rws*   'rwr4c Csh|jd|jd}}||}||}|d}||}|d}||} |d|||||| f} | S)z Crop the image so that both dimensions are divisible by the given divisor. Args: img: Image tensor. divisor: The divisor to use for cropping. Returns: The cropped image tensor. .)shape) imgdivisorhwdelta_hdelta_w delta_h_topdelta_h_bottom delta_w_left delta_w_right img_croppedr%r%r& crop_image!s rcs*eZdZdZdedeffdd ZZS)ImageTaskEncoderz7Image task encoder that crops and normalizes the image.rir cs,t|}t|jd|_|jd8_|S)z Encode a single image sample by cropping and shifting its values. Args: sample: An image sample. Returns: The transformed image sample. r4rB)rP encode_samplerr)r#rir_r%r&r?s zImageTaskEncoder.encode_sample)r/r0r1r2rrrvr%r%r_r&r<sr)rnr cCs~t}tjtdd|_tjtttddd|_tjtj j ddd|j _ d |j j _ d |j j _d |j_d |j_d |jjd _|S)ze Training factory function for VAE. Returns: A run.Partial recipe for training. z1nemo/collections/diffusion/vae/vae16x/config.json)rZ ) task_encoderglobal_batch_size num_workersdgeA) warmup_steps hold_stepsgh㈵>g{Gz?znemo_experiments/train_vaeir)rrunConfigrwmodelrrdatanl lr_schedulerWarmupHoldPolicySchedulerrxrYlr weight_decayrlog_dirrval_check_interval callbacksevery_n_train_steps)reciper%r%r& train_vaeOs$  r__main__)default_factory)r4)@typingrrrrrrnemo_runrr+torch.distributedtorch.utils.checkpointrrautovaercontperceptual_lossr diffusersr megatron.corer megatron.core.transformer.enumsr megatron.core.transformer.moduler ,megatron.core.transformer.transformer_configrmegatron.energonrrrrnemorrnemo.collectionsrsF               O