{
  "id": "enwiki-001377688233",
  "slug": "variational-autoencoder",
  "title": "Variational autoencoder",
  "qid": "",
  "lead": "In machine learning, a variational autoencoder (VAE) is an artificial neural network architecture introduced by Diederik P. Kingma and Max Welling in 2013. It is part of the families of probabilistic graphical models and variational Bayesian methods. In addition to being seen as an autoencoder neural network architecture, variational autoencoders can also be studied within the mathematical formulation of variational Bayesian methods, connecting a neural encoder network to its decoder through a probabilistic latent space (for example, as a multivariate Gaussian distribution) that corresponds to the parameters of a variational distribution. Thus, the encoder maps each point (such as an image) from a large complex dataset into a distribution within the latent space, rather than to a single point in that space. The decoder has the opposite function, which is to map from the latent space to the input space, again according to a distribution (although in practice, noise is rarely added during the decoding stage). By mapping a point to a distribution instead of a single point, the network can avoid overfitting the training data. Both networks are typically trained together with the usage of the reparameterization trick, although the variance of the noise model can be learned separately.…",
  "body": "In machine learning, a variational autoencoder (VAE) is an artificial neural network architecture introduced by Diederik P. Kingma and Max Welling in 2013. It is part of the families of probabilistic graphical models and variational Bayesian methods. In addition to being seen as an autoencoder neural network architecture, variational autoencoders can also be studied within the mathematical formulation of variational Bayesian methods, connecting a neural encoder network to its decoder through a probabilistic latent space (for example, as a multivariate Gaussian distribution) that corresponds to the parameters of a variational distribution. Thus, the encoder maps each point (such as an image) from a large complex dataset into a distribution within the latent space, rather than to a single point in that space. The decoder has the opposite function, which is to map from the latent space to the input space, again according to a distribution (although in practice, noise is rarely added during the decoding stage). By mapping a point to a distribution instead of a single point, the network can avoid overfitting the training data. Both networks are typically trained together with the usage of the reparameterization trick, although the variance of the noise model can be learned separately. Although this type of model was initially designed for unsupervised learning, its effectiveness has been proven for semi-supervised learning and supervised learning. From the point of view of probabilistic modeling, one wants to maximize the likelihood of the data x by their chosen parameterized probability distribution p_{\\theta}(x) = p(x|\\theta) . This distribution is usually chosen to be a Gaussian N(x|\\mu,\\sigma) which is parameterized by \\mu and \\sigma respectively, and as a member of the exponential family it is easy to work with as a noise distribution. Simple distributions are easy enough to maximize, however distributions where a prior is assumed over the latents z results in intractable integrals. Let us find p_\\theta(x) via marginalizing over z . p_\\theta(x) = \\int_{z}p_\\theta({x,z}) \\, dz, where p_\\theta({x,z}) represents the joint distribution under p_\\theta of the observable data x and its latent representation or encoding z . According to the chain rule, the equation can be rewritten as p_\\theta(x) = \\int_{z}p_\\theta({x| z})p_\\theta(z) \\, dz In the vanilla variational autoencoder, z is usually taken to be a finite-dimensional vector of real numbers, and p_\\theta({x|z}) to be a Gaussian distribution. Then p_\\theta(x) is a mixture of Gaussian distributions. It is now possible to define the set of the relationships between the input data and its latent representation as Prior p_\\theta(z) Likelihood p_\\theta(x) Posterior p_\\theta(z|x) Unfortunately, the computation of p_\\theta(z|x) is expensive and in most cases intractable. To speed up the calculus to make it feasible, it is necessary to introduce a further function to approximate the posterior distribution as q_\\phi({z| x}) \\approx p_\\theta({z| x}) with \\phi defined as the set of real values that parametrize q . This is sometimes called amortized inference, since by \"investing\" in finding a good q_\\phi , one can later infer z from x quickly without doing any integrals. In this way, the problem is to find a good probabilistic autoencoder, in which the conditional likelihood distribution p_\\theta(x|z) is computed by the probabilistic decoder, and the approximated posterior distribution q_\\phi(z|x) is computed by the probabilistic encoder. Parametrize the encoder as E_\\phi , and the decoder as D_\\theta . D_{KL}(q_\\phi({z| x})\\parallel p_\\theta({z| x})) &= \\mathbb E_{z \\sim q_\\phi(\\cdot | x)} \\left[\\ln \\frac{q_\\phi(z|x)}{p_\\theta(z|x)}\\right]\\\\ &= \\mathbb E_{z \\sim q_\\phi(\\cdot | x)} \\left[\\ln \\frac{q_\\phi({z| x})p_\\theta(x)}{p_\\theta(x, z)}\\right]\\\\ &=\\ln p_\\theta(x) + \\mathbb E_{z \\sim q_\\phi(\\cdot | x)} \\left[\\ln \\frac{q_\\phi({z| x})}{p_\\theta(x, z)}\\right]. \\end{align} Now, define the evidence lower bound (ELBO): L_{\\theta,\\phi}(x) := \\mathbb E_{z \\sim q_\\phi(\\cdot | x)} \\left[\\ln \\frac{p_\\theta(x, z)}{q_\\phi({z| x})}\\right] = \\ln p_\\theta(x) - D_{KL}(q_\\phi({\\cdot| x})\\parallel p_\\theta({\\cdot | x})) Maximizing the ELBO \\theta^*,\\phi^* = \\underset{\\theta,\\phi}\\operatorname{arg max} \\, L_{\\theta,\\phi}(x) is equivalent to simultaneously maximizing \\ln p_\\theta(x) and minimizing D_{KL}(q_\\phi({z| x})\\parallel p_\\theta({z| x})) . That is, maximizing the log-likelihood of the observed data, and minimizing the divergence from the approximate posterior q_\\phi(\\cdot | x) to the exact posterior p_\\theta(\\cdot | x) . The form given is not very convenient for maximization, but the following, equivalent form, is: L_{\\theta,\\phi}(x) = \\mathbb E_{z \\sim q_\\phi(\\cdot | x)} \\left[\\ln p_\\theta(x|z)\\right] - D_{KL}(q_\\phi({\\cdot| x})\\parallel p_\\theta(\\cdot)) where \\ln p_\\theta(x|z) is implemented as -\\frac{1}{2}\\| x - D_\\theta(z)\\|^2_2 , since that is, up to an additive constant, what x|z \\sim \\mathcal N(D_\\theta(z), I) yields. That is, we model the distribution of x conditional on z to be a Gaussian distribution centered on D_\\theta(z) . The distribution of q_\\phi(z |x) and p_\\theta(z) are often also chosen to be Gaussians as z|x \\sim \\mathcal N(E_\\phi(x), \\sigma_\\phi(x)^2I) and z \\sim \\mathcal N(0, I) , with which we obtain by the formula for KL divergence of Gaussians: L_{\\theta,\\phi}(x) = -\\frac 12\\mathbb E_{z \\sim q_\\phi(\\cdot | x)} \\left[ \\|x - D_\\theta(z)\\|_2^2\\right] - \\frac 12 \\left( N\\sigma_\\phi(x)^2 + \\|E_\\phi(x)\\|_2^2 - 2N\\ln\\sigma_\\phi(x) \\right) + Const Here N is the dimension of z . For a more detailed derivation and more interpretations of ELBO and its maximization, see its main page. To efficiently search for \\theta^*,\\phi^* = \\underset{\\theta,\\phi}\\operatorname{arg max} \\, L_{\\theta,\\phi}(x) the typical method is gradient descent. It is straightforward to find \\nabla_\\theta \\mathbb E_{z \\sim q_\\phi(\\cdot | x)} \\left[\\ln \\frac{p_\\theta(x, z)}{q_\\phi({z| x})}\\right] = \\mathbb E_{z \\sim q_\\phi(\\cdot | x)} \\left[ \\nabla_\\theta \\ln \\frac{p_\\theta(x, z)}{q_\\phi({z| x})}\\right] However, \\nabla_\\phi \\mathbb E_{z \\sim q_\\phi(\\cdot | x)} \\left[\\ln \\frac{p_\\theta(x, z)}{q_\\phi({z| x})}\\right] does not allow one to put the \\nabla_\\phi inside the expectation, since \\phi appears in the probability distribution itself. The reparameterization trick (also known as stochastic backpropagation ) bypasses this difficulty. The most important example is when z \\sim q_\\phi(\\cdot | x) is normally distributed, as \\mathcal N(\\mu_\\phi(x), \\Sigma_\\phi(x)) . This can be reparametrized by letting \\boldsymbol{\\varepsilon} \\sim \\mathcal{N}(0, \\boldsymbol{I}) be a \"standard random number generator\", and construct z as z = \\mu_\\phi(x) + L_\\phi(x)\\epsilon . Here, L_\\phi(x) is obtained by the Cholesky decomposition: \\Sigma_\\phi(x) = L_\\phi(x)L_\\phi(x)^T Then we have \\nabla_\\phi \\mathbb E_{z \\sim q_\\phi(\\cdot | x)} \\left[\\ln \\frac{p_\\theta(x, z)}{q_\\phi({z| x})}\\right] = \\mathbb {E}_{\\epsilon}\\left[ \\nabla_\\phi \\ln {\\frac {p_{\\theta }(x, \\mu_\\phi(x) + L_\\phi(x)\\epsilon)}{q_{\\phi }(\\mu_\\phi(x) + L_\\phi(x)\\epsilon | x)}}\\right] and so we obtained an unbiased estimator of the gradient, allowing stochastic gradient descent. Since we reparametrized z , we need to find q_\\phi(z|x) . Let q_0 be the probability density function for \\epsilon , then \\ln q_\\phi(z | x) = \\ln q_0 (\\epsilon) - \\ln|\\det(\\partial_\\epsilon z)| where \\partial_\\epsilon z is the Jacobian matrix of z with respect to \\epsilon . Since z = \\mu_\\phi(x) + L_\\phi(x)\\epsilon , this is \\ln q_\\phi(z | x) = -\\frac 12 \\|\\epsilon\\|^2 - \\ln|\\det L_\\phi(x)| - \\frac n2 \\ln(2\\pi) == Statistical distance VAE variants== After the initial work of Diederik P. Kingma and Max Welling, several procedures were proposed to formulate in a more abstract way the operation of the VAE. In these approaches the loss function is composed of two parts: the usual reconstruction error part which seeks to ensure that the encoder-then-decoder mapping x \\mapsto D_\\theta(E_\\psi(x)) is as close to the identity map as possible; the sampling is done at run time from the empirical distribution \\mathbb{P}^{real} of objects available (e.g., for MNIST or IMAGENET this will be the empirical probability law of all images in the dataset). This gives the term: \\mathbb{E}_{x \\sim \\mathbb{P}^{real}} \\left[ \\|x - D_\\theta(E_\\phi(x))\\|_2^2\\right] . a variational part that ensures that, when the empirical distribution \\mathbb{P}^{real} is passed through the encoder E_\\phi , we recover the target distribution, denoted here \\mu(dz) that is usually taken to be a Multivariate normal distribution. We will denote E_\\phi \\sharp \\mathbb{P}^{real} this pushforward measure which in practice is just the empirical distribution obtained by passing all dataset objects through the encoder E_\\phi . In order to make sure that E_\\phi \\sharp \\mathbb{P}^{real} is close to the target \\mu(dz) , a Statistical distance d is invoked and the term d \\left( \\mu(dz), E_\\phi \\sharp \\mathbb{P}^{real} \\right)^2 is added to the loss. We obtain the final formula for the loss: L_{\\theta,\\phi} = \\mathbb{E}_{x \\sim \\mathbb{P}^{real}} \\left[ \\|x - D_\\theta(E_\\phi(x))\\|_2^2\\right] +d \\left( \\mu(dz), E_\\phi \\sharp \\mathbb{P}^{real} \\right)^2 The statistical distance d requires special properties, for instance it has to be possess a formula as expectation because the loss function will need to be optimized by stochastic optimization algorithms. Several distances can be chosen and this gave rise to several flavors of VAEs: the sliced Wasserstein distance used by S Kolouri, et al. in their VAE the energy distance implemented in the Radon Sobolev Variational Auto-Encoder the Maximum Mean Discrepancy distance used in the MMD-VAE the Wasserstein distance used in the WAEs kernel-based distances used in the Kernelized Variational Autoencoder (K-VAE)",
  "categories": [
    "106452236",
    "22905725",
    "22930083",
    "23603337",
    "24745122",
    "26220375",
    "27179372",
    "27424622",
    "27424661",
    "28269683",
    "43052210",
    "49989139",
    "68102209",
    "93333220"
  ],
  "license": "CC BY-SA 4.0",
  "source_uri": "https://en.wikipedia.org/w/index.php?oldid=1377688233",
  "content_hash": "",
  "created_at": null,
  "updated_at": 1790776812.0,
  "claims": [],
  "contradictions": [],
  "confidence": 0.6,
  "csl": {
    "id": "enwiki-001377688233",
    "type": "webpage",
    "title": "Variational autoencoder",
    "author": [
      {
        "family": "Anticloud FZ LLE",
        "literal": "Anticloud FZ LLE"
      }
    ],
    "container-title": "encyclopegia.org",
    "publisher": "Anticloud FZ LLE",
    "publisher-place": "Dubai, United Arab Emirates",
    "URL": "https://encyclopegia.org/a/variational-autoencoder",
    "issued": {
      "date-parts": [
        [
          2026,
          11,
          21
        ]
      ]
    },
    "accessed": {
      "date-parts": [
        [
          2026,
          11,
          21
        ]
      ]
    },
    "license": "CC BY-SA 4.0",
    "DOI": "eo1:enwiki-001377688233",
    "isBasedOn": "https://en.wikipedia.org/w/index.php?oldid=1377688233",
    "note": "Immutable snapshot enwiki-001377688233"
  }
}