o %ݫi#7@sdZddlmZddlZddlmZddlmZddlmZddl m Z Gdd d ej Z Gd d d e Z ed gd ZedgdZGddde ZdS)zoAutoencoder implementation. Can be used for Latent Diffusion or in isolation Authors * Artem Ploujnikov 2022 ) namedtupleN)nn) clean_padding) GlobalNorm)trim_asc@s*eZdZdZd ddZddZddZdS) AutoencoderaA standard interface for autoencoders Example ------- >>> import torch >>> from torch import nn >>> from speechbrain.nnet.linear import Linear >>> class SimpleAutoencoder(Autoencoder): ... def __init__(self): ... super().__init__() ... self.enc = Linear(n_neurons=16, input_size=128) ... self.dec = Linear(n_neurons=128, input_size=16) ... def encode(self, x, length=None): ... return self.enc(x) ... def decode(self, x, length=None): ... return self.dec(x) >>> autoencoder = SimpleAutoencoder() >>> x = torch.randn(4, 10, 128) >>> x_enc = autoencoder.encode(x) >>> x_enc.shape torch.Size([4, 10, 16]) >>> x_enc_fw = autoencoder(x) >>> x_enc_fw.shape torch.Size([4, 10, 16]) >>> x_rec = autoencoder.decode(x_enc) >>> x_rec.shape torch.Size([4, 10, 128]) NcCt)aConverts a sample from an original space (e.g. pixel or waveform) to a latent space Arguments --------- x: torch.Tensor the original data representation length: torch.Tensor a tensor of relative lengths NotImplementedErrorselfxlengthrQ/home/ubuntu/.local/lib/python3.10/site-packages/speechbrain/nnet/autoencoders.pyencode/s zAutoencoder.encodecCr)zDecodes the sample from a latent representation Arguments --------- latent: torch.Tensor the latent representation r r latentrrrdecode<szAutoencoder.decodecC ||S)zPerforms the forward pass Arguments --------- x: torch.Tensor the input tensor Returns ------- result: torch.Tensor the result )r)r r rrrforwardF zAutoencoder.forwardN)__name__ __module__ __qualname____doc__rrrrrrrrs   rcsVeZdZdZ       dfdd Zddd Zd d Zd d Z dddZZ S)VariationalAutoencodera A Variational Autoencoder (VAE) implementation. Paper reference: https://arxiv.org/abs/1312.6114 Arguments --------- encoder: torch.Module the encoder network decoder: torch.Module the decoder network mean: torch.Module the module that computes the mean log_var: torch.Module the module that computes the log variance len_dim: None the length dimension latent_padding: function the function to use when padding the latent variable mask_latent: bool where to apply the length mask to the latent representation mask_out: bool whether to apply the length mask to the output out_mask_value: float the mask value used for the output latent_mask_value: float the mask value used for the latent representation latent_stochastic: bool if true, the "latent" parameter of VariationalAutoencoderOutput will be the latent space sample if false, it will be the mean Example ------- The example below shows a very simple implementation of VAE, not suitable for actual experiments: >>> import torch >>> from torch import nn >>> from speechbrain.nnet.linear import Linear >>> vae_enc = Linear(n_neurons=16, input_size=128) >>> vae_dec = Linear(n_neurons=128, input_size=16) >>> vae_mean = Linear(n_neurons=16, input_size=16) >>> vae_log_var = Linear(n_neurons=16, input_size=16) >>> vae = VariationalAutoencoder( ... encoder=vae_enc, ... decoder=vae_dec, ... mean=vae_mean, ... log_var=vae_log_var, ... ) >>> x = torch.randn(4, 10, 128) `train_sample` encodes a single batch and then reconstructs it >>> vae_out = vae.train_sample(x) >>> vae_out.rec.shape torch.Size([4, 10, 128]) >>> vae_out.latent.shape torch.Size([4, 10, 16]) >>> vae_out.mean.shape torch.Size([4, 10, 16]) >>> vae_out.log_var.shape torch.Size([4, 10, 16]) >>> vae_out.latent_sample.shape torch.Size([4, 10, 16]) .encode() will return the mean corresponding to the sample provided >>> x_enc = vae.encode(x) >>> x_enc.shape torch.Size([4, 10, 16]) .reparameterize() performs the reparameterization trick >>> x_enc = vae.encoder(x) >>> mean = vae.mean(x_enc) >>> log_var = vae.log_var(x_enc) >>> x_repar = vae.reparameterize(mean, log_var) >>> x_repar.shape torch.Size([4, 10, 16]) NTc sPt||_||_||_||_||_||_||_||_ | |_ | |_ | |_ dSr) super__init__encoderdecodermeanlog_varlen_dimlatent_padding mask_latentmask_outout_mask_valuelatent_mask_valuelatent_stochastic) r r"r#r$r%r&r'r(r)r*r+r, __class__rrr!s  zVariationalAutoencoder.__init__cCs||}||S)aConverts a sample from an original space (e.g. pixel or waveform) to a latent space Arguments --------- x: torch.Tensor the original data representation length: torch.Tensor the length of the corresponding input samples (optional) Returns ------- latent: torch.Tensor the latent representation )r"r$)r r r encoder_outrrrrs  zVariationalAutoencoder.encodecCrzDecodes the sample from a latent representation Arguments --------- latent: torch.Tensor the latent representation Returns ------- result: torch.Tensor the decoded sample r#rrrrrrzVariationalAutoencoder.decodecCs t|}||td|S)aApplies the VAE reparameterization trick to get a latent space single latent space sample for decoding Arguments --------- mean: torch.Tensor the latent representation mean log_var: torch.Tensor the logarithm of the latent representation variance Returns ------- sample: torch.Tensor a latent space sample g?)torch randn_likeexp)r r$r%epsilonrrrreparameterizes z%VariationalAutoencoder.reparameterizec Cs|dur|j}|dur|j}||}||}||}|||}|jdur2|j||d\}} n|} |jrC|durCt|| |j |}| |} t | |} |j r\|dur\t| ||j |} |j rb|} n |j||d\} } t| | |||| S)aProvides a data sample for training the autoencoder Arguments --------- x: torch.Tensor the source data (in the sample space) length: None the length (optional). If provided, latents and outputs will be masked out_mask_value: float the mask value used for the output latent_mask_value: float the mask value used for the latent tensor Returns ------- result: VariationalAutoencoderOutput a named tuple with the following values rec: torch.Tensor the reconstruction latent: torch.Tensor the latent space sample mean: torch.Tensor the mean of the latent representation log_var: torch.Tensor the logarithm of the variance of the latent representation Nr)r*r+r"r$r%r6r'r(rr&rrr)r,VariationalAutoencoderOutput) r r rr*r+r/r$r% latent_sample latent_lengthx_recrrrr train_samples6           z#VariationalAutoencoder.train_sample)rNTTrrTrNNN) rrrrr!rrr6r< __classcell__rrr-rrVs[ rr8)recrr$r%r9r:AutoencoderOutput)r?rr:csNeZdZdZ       dfdd Zddd Zd d Z dd d ZZS)NormalizingAutoencoderanA classical (non-variational) autoencoder that does not use reparameterization but instead uses an ordinary normalization technique to constrain the latent space Arguments --------- encoder: torch.nn.Module the encoder to be used decoder: torch.nn.Module the decoder to be used latent_padding: function Function to use when padding the latent tensor norm: torch.nn.Module the normalization module len_dim: int The time dimension, which the length applies to. mask_out: bool whether to apply the length mask to the output mask_latent: bool where to apply the length mask to the latent representation out_mask_value: float the mask value used for the output latent_mask_value: float the mask value used for the latent tensor Examples -------- >>> import torch >>> from torch import nn >>> from speechbrain.nnet.linear import Linear >>> ae_enc = Linear(n_neurons=16, input_size=128) >>> ae_dec = Linear(n_neurons=128, input_size=16) >>> ae = NormalizingAutoencoder( ... encoder=ae_enc, ... decoder=ae_dec, ... ) >>> x = torch.randn(4, 10, 128) >>> x_enc = ae.encode(x) >>> x_enc.shape torch.Size([4, 10, 16]) >>> x_dec = ae.decode(x_enc) >>> x_dec.shape torch.Size([4, 10, 128]) NrTrc sRt||_||_||_|durt}||_||_||_||_ ||_ | |_ dSr) r r!r"r#r'rnormr&r)r(r*r+) r r"r#r'rBr&r)r(r*r+r-rrr!ws  zNormalizingAutoencoder.__init__cCs||}|j||d}|S)aConverts a sample from an original space (e.g. pixel or waveform) to a latent space Arguments --------- x: torch.Tensor the original data representation length: torch.Tensor The length of each sample in the input tensor. Returns ------- latent: torch.Tensor the latent representation )lengths)r"rBr rrrrs zNormalizingAutoencoder.encodecCrr0r1rrrrrrzNormalizingAutoencoder.decodecCs|dur|j}|dur|j}|j||d}|jdur$|j||d\}}n|}|jr5|dur5t|||j|}||}t||}|j rN|durNt|||j|}t |||S)aProvides a data sample for training the autoencoder Arguments --------- x: torch.Tensor the source data (in the sample space) length: torch.Tensor the length (optional). If provided, latents and outputs will be masked out_mask_value: float The value to use when masking the output. latent_mask_value: float The value to use when masking the latent tensor. Returns ------- result: AutoencoderOutput a named tuple with the following values rec: torch.Tensor the reconstruction latent: torch.Tensor the latent space sample Nr7) r*r+rr'r(rr&rrr)r@)r r rr*r+rr:r;rrrr<s"     z#NormalizingAutoencoder.train_sample)NNrTTrrrr=) rrrrr!rrr<r>rrr-rrAHs2 rA)r collectionsrr2rspeechbrain.dataio.dataiorspeechbrain.processing.featuresrspeechbrain.utils.data_utilsrModulerrr8r@rArrrrs$     Ei