Autoencoder

Uses your system voice; voice and rate in settings.

Snapshot

≈ 13 min read

Names in other languages

No names in other languages are held for this entry yet.

Article bodies are English and are not machine-translated. A name in another language is not a translation of the article.

An autoencoder is a type of artificial neural network used to learn efficient codings of unlabeled data (unsupervised learning). An autoencoder learns two functions: an encoding function that transforms the input data, and a decoding function that recreates the input data from the encoded representation. The autoencoder learns an efficient representation (encoding) for a set of data, typically for dimensionality reduction, to generate lower-dimensional embeddings for subsequent use by other machine learning algorithms. Variants exist which aim to make the learned representations assume useful properties. Examples are regularized autoencoders (sparse, denoising and contractive autoencoders), which are effective in learning representations for subsequent classification tasks, and variational autoencoders, which can be used as generative models. Autoencoders are applied to many problems, including facial recognition, feature detection, anomaly detection, and learning the meaning of words. In terms of data synthesis, autoencoders can also be used to randomly generate new data that is similar to the input (training) data.

An autoencoder is a type of artificial neural network used to learn efficient codings of unlabeled data (unsupervised learning). An autoencoder learns two functions: an encoding function that transforms the input data, and a decoding function that recreates the input data from the encoded representation. The autoencoder learns an efficient representation (encoding) for a set of data, typically for dimensionality reduction, to generate lower-dimensional embeddings for subsequent use by other machine learning algorithms. Variants exist which aim to make the learned representations assume useful properties. Examples are regularized autoencoders (sparse, denoising and contractive autoencoders), which are effective in learning representations for subsequent classification tasks, and variational autoencoders, which can be used as generative models. Autoencoders are applied to many problems, including facial recognition, feature detection, anomaly detection, and learning the meaning of words. In terms of data synthesis, autoencoders can also be used to randomly generate new data that is similar to the input (training) data. An autoencoder is defined by the following components: Two sets: the space of encoded messages \mathcal Z ; the space of decoded messages \mathcal X . Typically \mathcal X and \mathcal Z are Euclidean spaces, that is, \mathcal X = \R^m, \mathcal Z = \R^n with m > n. Two parametrized families of functions: the encoder family E_\phi:\mathcal{X} \rightarrow \mathcal{Z} , parametrized by \phi ; the decoder family D_\theta:\mathcal{Z} \rightarrow \mathcal{X} , parametrized by \theta . For any x\in \mathcal X , we usually write z = E_\phi(x) , and refer to it as the code, the latent variable, latent representation, latent vector, etc. Conversely, for any z\in \mathcal Z , we usually write x' = D_\theta(z) , and refer to it as the (decoded) message. Usually, both the encoder and the decoder are defined as multilayer perceptrons (MLPs). For example, a one-layer-MLP encoder E_\phi is: E_\phi(\mathbf x) = \sigma(Wx+b) where \sigma is an element-wise activation function, W is a "weight" matrix, and b is a "bias" vector. With those, we can define the loss function for the autoencoder as L(\theta, \phi) := \mathbb \mathbb E_{x\sim \mu_{ref}}[d(x, D_\theta(E_\phi(x)))] The optimal autoencoder for the given task (\mu_{ref}, d) is then \arg\min_{\theta, \phi}L(\theta, \phi) . The search for the optimal autoencoder can be accomplished by any mathematical optimization technique, but usually by gradient descent. This search process is referred to as "training the autoencoder". In most situations, the reference distribution is just the empirical distribution given by a dataset \{x_1, ..., x_N\} \subset \mathcal X , so that \mu_{ref} = \frac{1}{N}\sum_{i=1}^N \delta_{x_i} where \delta_{x_i} is the Dirac measure, the quality function is just L^2 loss: d(x, x') = \|x - x'\|_2^2 , and \|\cdot\|_2 is the Euclidean norm. Then the problem of searching for the optimal autoencoder is just a least-squares optimization: \min_{\theta, \phi} L(\theta, \phi),\qquad \text{where } L(\theta, \phi) = \frac{1}{N}\sum_{i=1}^N \|x_i - D_\theta(E_\phi(x_i))\|_2^2 At the limit of an ideal undercomplete autoencoder, every possible code z in the code space is used to encode a message x that really appears in the distribution \mu_{ref} , and the decoder is also perfect: D_\theta(E_\phi(x)) = x . This ideal autoencoder can then be used to generate messages indistinguishable from real messages, by feeding its decoder arbitrary code z and obtaining D_\theta(z) , which is a message that really appears in the distribution \mu_{ref} . If the code space \mathcal Z has dimension larger than (overcomplete), or equal to, the message space \mathcal{X} , or the hidden units are given enough capacity, an autoencoder can learn the identity function and become useless. However, experimental results found that overcomplete autoencoders might still learn useful features. In the ideal setting, the code dimension and the model capacity could be set on the basis of the complexity of the data distribution to be modeled. A standard way to do so is to add modifications to the basic autoencoder, to be detailed below.

Variations

Variational autoencoder (VAE)

Variational autoencoders (VAEs) belong to the families of variational Bayesian methods. Despite the architectural similarities with basic autoencoders, VAEs are architected with different goals and have a different mathematical formulation. The latent space is, in this case, composed of a mixture of distributions instead of fixed vectors. Given an input dataset x characterized by an unknown probability function P(x) and a multivariate latent encoding vector z , the objective is to model the data as a distribution p_\theta(x) , with \theta defined as the set of the network parameters so that p_\theta(x) = \int_{z}p_\theta(x,z)dz .

Sparse autoencoder (SAE)

Inspired by the sparse coding hypothesis in neuroscience, sparse autoencoders (SAE) are variants of autoencoders, such that the codes E_\phi(x) for messages tend to be sparse codes, that is, E_\phi(x) is close to zero in most entries. Sparse autoencoders may include more (rather than fewer) hidden units than inputs, but only a small number of the hidden units are allowed to be active at the same time. Encouraging sparsity improves performance on classification tasks. There are two main ways to enforce sparsity. One way is to simply clamp all but the highest-k activations of the latent code to zero. This is the k-sparse autoencoder. The k-sparse autoencoder inserts the following "k-sparse function" in the latent layer of a standard autoencoder: f_k(x_1, ..., x_n) = (x_1 b_1, ..., x_n b_n) where b_i = 1 if |x_i| ranks in the top k, and 0 otherwise. Backpropagating through f_k is simple: set gradient to 0 for b_i = 0 entries, and keep gradient for b_i=1 entries. This is essentially a generalized ReLU function. The other way is a relaxed version of the k-sparse autoencoder. Instead of forcing sparsity, we add a sparsity regularization loss, then optimize for \min_{\theta, \phi}L(\theta, \phi) + \lambda L_{\text{sparse}} (\theta, \phi) where \lambda > 0 measures how much sparsity we want to enforce. Let the autoencoder architecture have K layers. To define a sparsity regularization loss, we need a "desired" sparsity \hat \rho_k for each layer, a weight w_k for how much to enforce each sparsity, and a function s: [0, 1]\times [0, 1] \to [0, \infty] to measure how much two sparsities differ. For each input x , let the actual sparsity of activation in each layer k be \rho_k(x) = \frac 1n \sum_{i=1}^n a_{k, i}(x) where a_{k, i}(x) is the activation in the i -th neuron of the k -th layer upon input x . The sparsity loss upon input x for one layer is s(\hat\rho_k, \rho_k(x)) , and the sparsity regularization loss for the entire autoencoder is the expected weighted sum of sparsity losses: L_{\text{sparse}}(\theta, \phi) = \mathbb \mathbb E_{x\sim\mu_X}\left[\sum_{k\in 1:K} w_k s(\hat\rho_k, \rho_k(x)) \right] Typically, the function s is either the Kullback–Leibler (KL) divergence, as s(\rho, \hat\rho) = KL(\rho || \hat{\rho}) = \rho \log \frac{\rho}{\hat{\rho}}+(1- \rho)\log \frac{1-\rho}{1-\hat{\rho}} or the L1 loss, as s(\rho, \hat\rho) = |\rho- \hat\rho| , or the L2 loss, as s(\rho, \hat\rho) = |\rho- \hat\rho|^2 . Alternatively, the sparsity regularization loss may be defined without reference to any "desired sparsity", but simply force as much sparsity as possible. In this case, one can define the sparsity regularization loss as L_{\text{sparse}}(\theta, \phi) = \mathbb \mathbb E_{x\sim\mu_X}\left[ \sum_{k\in 1:K} w_k \|h_k\| \right] where h_k is the activation vector in the k -th layer of the autoencoder. The norm \|\cdot\| is usually the L1 norm (giving the L1 sparse autoencoder) or the L2 norm (giving the L2 sparse autoencoder).

Denoising autoencoder (DAE)

Denoising autoencoders (DAE) try to achieve a good representation by changing the reconstruction criterion. A DAE, originally called a "robust autoassociative network" by Mark A. Kramer, is trained by intentionally corrupting the inputs of a standard autoencoder during training. A noise process is defined by a probability distribution \mu_T over functions T:\mathcal X \to \mathcal X . That is, the function T takes a message x\in \mathcal X , and corrupts it to a noisy version T(x) . The function T is selected randomly, with a probability distribution \mu_T . Given a task (\mu_{\text{ref}}, d) , the problem of training a DAE is the optimization problem: \min_{\theta, \phi}L(\theta, \phi) = \mathbb \mathbb E_{x\sim \mu_X, T\sim\mu_T}[d(x, (D_\theta\circ E_\phi \circ T)(x))] That is, the optimal DAE should take any noisy message and attempt to recover the original message without noise, thus the name "denoising". Usually, the noise process T is applied only during training and testing, not during downstream use. The use of DAE depends on two assumptions: There exist representations to the messages that are relatively stable and robust to the type of noise we are likely to encounter; The said representations capture structures in the input distribution that are useful for our purposes. Example noise processes include: additive isotropic Gaussian noise, masking noise (a fraction of the input is randomly chosen and set to 0) salt-and-pepper noise (a fraction of the input is randomly chosen and randomly set to its minimum or maximum value). A contractive autoencoder (CAE) adds the contractive regularization loss to the standard autoencoder loss: \min_{\theta, \phi}L(\theta, \phi) + \lambda L_{\text{cont}} (\theta, \phi) where \lambda > 0 measures how much contractive-ness we want to enforce. The contractive regularization loss itself is defined as the expected square of Frobenius norm of the Jacobian matrix of the encoder activations with respect to the input: L_{\text{cont}}(\theta, \phi) = \mathbb E_{x\sim \mu_{ref}} \|\nabla_x E_\phi(x) \|_F^2 To understand what L_{\text{cont}} measures, note the fact \|E_\phi(x + \delta x) - E_\phi(x)\|_2 \leq \|\nabla_x E_\phi(x) \|_F \|\delta x\|_2 for any message x\in \mathcal X , and small variation \delta x in it. Thus, if \|\nabla_x E_\phi(x) \|_F^2 is small, it means that a small neighborhood of the message maps to a small neighborhood of its code. This is a desired property, as it means small variation in the message leads to small, perhaps even zero, variation in its code, like how two pictures may look the same even if they are not exactly the same. The CAE can be understood as an infinitesimal limit of DAE: in the limit of small Gaussian input noise, DAEs make the reconstruction function resist small but finite-sized input perturbations, while CAEs make the extracted features resist infinitesimal input perturbations. However, while DAEs encourage robustness of reconstruction, CAEs encourage robustness of representation. Additionally, DAEs' robustness is obtained stochastically, by having various corrupted versions of a training point aim for an identical reconstruction. In contrast, CAEs' robustness to small perturbations is obtained analytically by penalizing the Frobenius norm of the encoder Jacobian, \|\nabla_x E_\phi(x) \|_F^2 , evaluated at the training samples.

Advantages of depth

Autoencoders are often trained with a single-layer encoder and a single-layer decoder, but using many-layered (deep) encoders and decoders offers many advantages. Depth can exponentially reduce the computational cost of representing some functions. Depth can exponentially decrease the amount of training data needed to learn some functions. Experimentally, deep autoencoders yield better compression compared to shallow or linear autoencoders. Depth allows for advantages over traditional methods as one can show that after training single layer linear autoencoders have a latent space whose vectors span the same subspace as the eigenvectors found in Principal component analysis. Typically, this means that on a validation set the empirical distribution of reconstruction errors is recorded and then (e.g.) the empirical 95-percentile x_p is taken as threshold t:=x_p to flag anomalous data points: \text{loss}(x, \text{reconstruction}(x))>t \implies \text{anomaly} . Since the threshold is an empirical quantile estimate, there is an inherent difficulty with "correctly" setting this threshold: In many cases the distribution of the empirical quantile is asymptotically a normal distribution \text{empirical p-quantile} \sim \mathcal{N}\left(\mu=p, \sigma^2=\frac{p( 1 - p )}{n f(x_p)^2}\right), with f(x_p) the probability density at the quantile. This means that the variance grows if an extreme quantile is considered (because f(x_p) is small there). This means that there is a, potentially, a big uncertainty in what is the right choice for the threshold since it is estimated from a validation set. Recent literature has however shown that certain autoencoding models can, counterintuitively, be very good at reconstructing anomalous examples and consequently not able to reliably perform anomaly detection. Intuitively, this can be understood by considering those one layer auto encoders which are related to PCA - also in this case there can be perfect rein reconstructions for points which are far away from the data region but which lie on a principal component axis. It is best to analyze if the anomalies which are flagged by the auto encoder are true anomalies. In this sense all the metrics in Evaluation of binary classifiers can be considered. The fundamental challenge which comes with the unsupervised (self-supervised) learning setting is, that labels for rare events do not exist (in which case the labels first have to be gathered and the data set will be imbalanced) or anomaly indicating labels are very rare, introducing larger confidence intervals for these performance estimates.

See also

Representation learning Singular value decomposition Sparse dictionary learning Deep learning

References

Text derived from https://en.wikipedia.org/w/index.php?oldid=1375973936 under CC BY-SA 4.0; modifications have been made.

Sources cited in this revision

27 named source(s) extracted from the cited revision. A source with no author is shown by its publication; nothing is invented.