o }o™i$ã@s2ddlZddlmZddlTGdd„dejƒZdS)éN)Ú*csNeZdZdZ           d‡fdd„ Zdd d „Z dd d „Z‡ZS)ÚLPIPSWithDiscriminatora7 A perceptual loss module that combines LPIPS with an adversarial discriminator for improved reconstruction quality in variational autoencoders. This class calculates a combination of pixel-level, perceptual (LPIPS), KL, and adversarial losses for training a VAE model with a discriminator. ççð?éFÚhingec sŽtƒ ¡| dvs J‚||_||_tƒ ¡|_| |_t  t   d¡|¡|_ t ||| d t¡|_||_| dkr9tnt|_||_||_| |_dS)a Initializes the LPIPSWithDiscriminator module. Args: disc_start (int): Iteration at which to start discriminator updates. logvar_init (float): Initial value for the log variance parameter. kl_weight (float): Weight for the KL divergence term. pixelloss_weight (float): Weight for the pixel-level reconstruction loss. disc_num_layers (int): Number of layers in the discriminator. disc_in_channels (int): Number of input channels for the discriminator. disc_factor (float): Scaling factor for the discriminator loss. disc_weight (float): Weight applied to the discriminator gradient balancing. perceptual_weight (float): Weight for the LPIPS perceptual loss. use_actnorm (bool): Whether to use actnorm in the discriminator. disc_conditional (bool): Whether the discriminator is conditional on an additional input. disc_loss (str): Type of GAN loss to use ("hinge" or "vanilla"). )rÚvanillaé)Úinput_ncÚn_layersÚ use_actnormrN)ÚsuperÚ__init__Ú kl_weightÚ pixel_weightÚLPIPSÚevalÚperceptual_lossÚperceptual_weightÚnnÚ ParameterÚtorchÚonesÚlogvarÚNLayerDiscriminatorÚapplyÚ weights_initÚ discriminatorÚdiscriminator_iter_startÚ hinge_d_lossÚvanilla_d_lossÚ disc_lossÚ disc_factorÚdiscriminator_weightÚdisc_conditional) ÚselfÚ disc_startÚ logvar_initrÚpixelloss_weightÚdisc_num_layersÚdisc_in_channelsr"Ú disc_weightrr r$r!©Ú __class__©úf/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/diffusion/vae/contperceptual_loss.pyrs"  ÿþ zLPIPSWithDiscriminator.__init__NcCs¦|durtjj||ddd}tjj||ddd}ntjj||jdddd}tjj||jdddd}t |¡t |¡d}t |dd¡ ¡}||j}|S)u¡ Computes an adaptive weight that balances the reconstruction (NLL) and the adversarial (GAN) losses. This ensures stable training by adjusting the impact of the discriminator’s gradient on the generator. Args: nll_loss (torch.Tensor): The negative log-likelihood loss. g_loss (torch.Tensor): The generator (adversarial) loss. last_layer (torch.nn.Parameter, optional): Last layer parameters of the model for gradient-based calculations. If None, uses self.last_layer[0]. Returns: torch.Tensor: The computed adaptive weight for balancing the discriminator. NT)Ú retain_graphrg-Cëâ6?rgˆÃ@)rÚautogradÚgradÚ last_layerÚnormÚclampÚdetachr#)r%Únll_lossÚg_lossr3Ú nll_gradsÚg_gradsÚd_weightr.r.r/Úcalculate_adaptive_weightOs z0LPIPSWithDiscriminator.calculate_adaptive_weightc  Cs¶t | ¡| ¡¡} |jdkr!| | ¡| ¡¡} | |j| } | t |j¡|j} | } |dur6|| } t | ¡| jd} t | ¡| jd} |  ¡} t | ¡| jd} |dkró|durm|j reJ‚|  | ¡¡}n|j srJ‚|  tj | ¡|fdd¡}t  |¡ }|jdkr©z |j| ||d}Wnty¨|jr¡J‚t d¡}Ynwt d¡}t|j||jd}| |j| |||}| ¡ ¡  ¡|j ¡ ¡|  ¡  ¡|  ¡  ¡|  ¡  ¡| ¡t |¡| ¡  ¡dœ}||fS|dkrY|dur|  | ¡ ¡¡}|  | ¡ ¡¡}n |  tj | ¡ ¡|fdd¡}|  tj | ¡ ¡|fdd¡}t|j||jd}|| ||¡}| ¡ ¡  ¡| ¡  ¡| ¡  ¡d œ}||fSdS) aÅ Forward pass for computing the combined loss. Depending on the optimizer index, this either computes the generator loss (including pixel, perceptual, KL, and adversarial terms) or the discriminator loss. Args: inputs (torch.Tensor): Original inputs to reconstruct. reconstructions (torch.Tensor): Reconstructed outputs from the model. posteriors (object): Posteriors from the VAE model for KL computation. optimizer_idx (int): Indicates which optimizer is being updated (0 for generator, 1 for discriminator). global_step (int): Current training iteration step. last_layer (torch.nn.Parameter, optional): The last layer's parameters for adaptive weight calculation. cond (torch.Tensor, optional): Conditional input for the discriminator. weights (torch.Tensor, optional): Sample-wise weighting for the losses. Returns: (torch.Tensor, dict): A tuple of (loss, log_dict) where loss is the computed loss for the current optimizer and log_dict is a dictionary of intermediate values for logging and debugging. rNr )Údimr)r3)Ú threshold)Ú total_lossrÚkl_lossr7Úrec_lossr;r"r8)r!Ú logits_realÚ logits_fake)rÚabsÚ contiguousrrÚexprÚsumÚshapeÚklr$rÚcatÚmeanr"r<Ú RuntimeErrorÚtrainingÚtensorÚ adopt_weightrrÚcloner6Úitemr!)r%ÚinputsÚreconstructionsÚ posteriorsÚ optimizer_idxÚ global_stepr3ÚcondÚweightsrAÚp_lossr7Úweighted_nll_lossr@rCr8r;r"ÚlossÚlogrBÚd_lossr.r.r/Úforwardjsj       þ      ø      ýïzLPIPSWithDiscriminator.forward) rrrrrrrrFFr)N)NNN)Ú__name__Ú __module__Ú __qualname__Ú__doc__rr<r^Ú __classcell__r.r.r,r/rs" ó 2ÿr)rÚtorch.nnrÚ"taming.modules.losses.vqperceptualÚModulerr.r.r.r/Ús