Principal component analysis

Uses your system voice; voice and rate in settings.

Snapshot

≈ 25 min read

Names in other languages

No names in other languages are held for this entry yet.

Article bodies are English and are not machine-translated. A name in another language is not a translation of the article.

centered at (1, 3) with a standard deviation of 3 in roughly the (0.866, 0.5) direction and of 1 in the orthogonal direction. The vectors shown are the eigenvectors of the covariance matrix scaled by the square root of the corresponding eigenvalue, and shifted so their tails are at the mean.

Principal component analysis (PCA) is a linear dimensionality reduction technique with applications in exploratory data analysis, visualization and data preprocessing. The data are linearly transformed onto a new coordinate system such that the directions (principal components) capturing the largest variation in the data can be easily identified.

The principal components of a collection of points in a real coordinate space are a sequence of p unit vectors, where the i -th vector is the direction of a line that best fits the data while being orthogonal to the first i-1 vectors. Here, a best-fitting line is defined as one that minimizes the average squared perpendicular distance from the points to the line.

These directions (i.e., principal components) constitute an orthonormal basis in which different individual dimensions of the data are linearly uncorrelated. Many studies use the first two principal components in order to plot the data in two dimensions and to visually identify clusters of closely related data points.…

centered at (1, 3) with a standard deviation of 3 in roughly the (0.866, 0.5) direction and of 1 in the orthogonal direction. The vectors shown are the eigenvectors of the covariance matrix scaled by the square root of the corresponding eigenvalue, and shifted so their tails are at the mean.

Principal component analysis (PCA) is a linear dimensionality reduction technique with applications in exploratory data analysis, visualization and data preprocessing. The data are linearly transformed onto a new coordinate system such that the directions (principal components) capturing the largest variation in the data can be easily identified.

The principal components of a collection of points in a real coordinate space are a sequence of p unit vectors, where the i -th vector is the direction of a line that best fits the data while being orthogonal to the first i-1 vectors. Here, a best-fitting line is defined as one that minimizes the average squared perpendicular distance from the points to the line.

These directions (i.e., principal components) constitute an orthonormal basis in which different individual dimensions of the data are linearly uncorrelated. Many studies use the first two principal components in order to plot the data in two dimensions and to visually identify clusters of closely related data points.

Principal component analysis has applications in many fields such as population genetics, microbiome studies, and atmospheric science. Mathematically, the transformation is defined by a set of size l (where l is usually selected to be strictly less than p to reduce dimensionality) of p -dimensional vectors of weights or coefficients \mathbf{w}_{(k)} = (w_1, \dots, w_p)_{(k)} that map each row vector \mathbf{x}_{(i)} = (x_1, \dots, x_p)_{(i)} of X to a new vector of principal component scores \mathbf{t}_{(i)} = (t_1, \dots, t_l)_{(i)} , given by {t_{k}}_{(i)} = \mathbf{x}_{(i)} \cdot \mathbf{w}_{(k)} \qquad \mathrm{for} \qquad i = 1,\dots,n \qquad k = 1,\dots,l in such a way that the individual variables t_1, \dots, t_l of t considered over the data set successively inherit the maximum possible variance from X, with each coefficient vector w constrained to be a unit vector.

The above may equivalently be written in matrix form as \mathbf{T} = \mathbf{X} \mathbf{W} where {\mathbf{T}}_{ik} = {t_{k}}_{(i)} , {\mathbf{X}}_{ij} = {x_{j}}_{(i)} , and {\mathbf{W}}_{jk} = {w_{j}}_{(k)} . \mathbf{w}_{(1)} = \arg\max \left\{ \frac{\mathbf{w}^\mathsf{T} \mathbf{X}^\mathsf{T} \mathbf{X w}}{\mathbf{w}^\mathsf{T} \mathbf{w}} \right\} The quantity to be maximised can be recognised as a Rayleigh quotient. A standard result for a positive semidefinite matrix such as X T X is that the quotient's maximum possible value is the largest eigenvalue of the matrix, which occurs when w is the corresponding eigenvector.

With w (1) found, the first principal component of a data vector x (i) can then be given as a score t 1(i) = x (i) ⋅ w (1) in the transformed co-ordinates, or as the corresponding vector in the original variables, {x (i) ⋅ w (1) } w (1) . \mathbf{\hat{X}}_k = \mathbf{X} - \sum_{s = 1}^{k - 1} \mathbf{X} \mathbf{w}_{(s)} \mathbf{w}_{(s)}^{\mathsf{T}} and then finding the weight vector which extracts the maximum variance from this new data matrix \mathbf{w}_{(k)} \mathbf{T} = \mathbf{X} \mathbf{W} where W is a p-by-p matrix of weights whose columns are the eigenvectors of X T X. The transpose of W is sometimes called the whitening or sphering transformation. Columns of W multiplied by the square root of corresponding eigenvalues, that is, eigenvectors scaled up by the variances, are called loadings in PCA or in Factor analysis. & = \mathbf{w}_{(j)}^\mathsf{T} \mathbf{X}^\mathsf{T} \mathbf{X} \mathbf{w}_{(k)} \\ & = \mathbf{w}_{(j)}^\mathsf{T} \lambda_{(k)} \mathbf{w}_{(k)} \\ & = \lambda_{(k)} \mathbf{w}_{(j)}^\mathsf{T} \mathbf{w}_{(k)} \end{align} where the eigenvalue property of w (k) has been used to move from line 2 to line 3.

However eigenvectors w (j) and w (k) corresponding to eigenvalues of a symmetric matrix are orthogonal (if the eigenvalues are different), or can be orthogonalised (if the vectors happen to share an equal repeated value). The product in the final line is therefore zero; there is no sample covariance between different principal components over the dataset.

Another way to characterise the principal components transformation is therefore as the transformation to coordinates which diagonalise the empirical sample covariance matrix. In matrix form, the empirical covariance matrix for the original variables can be written \mathbf{Q} \propto \mathbf{X}^\mathsf{T} \mathbf{X} = \mathbf{W} \mathbf{\Lambda} \mathbf{W}^\mathsf{T} The empirical covariance matrix between the principal components becomes \mathbf{W}^\mathsf{T} \mathbf{Q} \mathbf{W} \propto \mathbf{W}^\mathsf{T} \mathbf{W} \, \mathbf{\Lambda} \, \mathbf{W}^\mathsf{T} \mathbf{W} = \mathbf{\Lambda} where Λ is the diagonal matrix of eigenvalues λ (k) of X T X. λ (k) is equal to the sum of the squares over the dataset associated with each component k, that is, λ (k) = Σ i t k 2 (i) = Σ i (x (i) ⋅ w (k) ) 2 .

The transformation P = X W maps a data vector x (i) from an original space of x variables to a new space of p variables which are uncorrelated over the dataset. To non-dimensionalize the centered data, let X c represent the characteristic values of data vectors X i , given by: \|X\|_{\infty} (maximum norm), \frac{1}{n} \|X\|_1 (mean absolute value), or \frac{1}{\sqrt{n}} \|X\|_2 (normalized Euclidean norm), for a dataset of size n. These norms are used to transform the original space of variables x, y to a new space of uncorrelated variables p, q (given Y c with same meaning), such that p_i = \frac{X_i}{X_c}, \quad q_i = \frac{Y_i}{Y_c} ; and the new variables are linearly related as: q = \alpha p . To find the optimal linear relationship, we minimize the total squared reconstruction error: E(\alpha) = \frac{1}{1 - \alpha^2} \sum_{i=1}^{n} (\alpha p_i - q_i)^2 ; such that setting the derivative of the error function to zero (E'(\alpha) = 0) yields: \alpha = \frac{1}{2} \left( -\lambda \pm \sqrt{\lambda^2 + 4} \right) where \lambda = \frac{p \cdot p - q \cdot q}{p \cdot q} . haplotypes calculated from repeat-count values for 37 Y-chromosomal STR markers from 354 individuals.

PCA has successfully found linear combinations of the markers that separate out different clusters corresponding to different lines of individuals' Y-chromosomal genetic descent. Such dimensionality reduction can be a very useful step for visualising and processing high-dimensional datasets, while still retaining as much of the variance in the dataset as possible.

For example, selecting L = 2 and keeping only the first two principal components finds the two-dimensional plane through the high-dimensional dataset in which the data are most spread out, so if the data contains clusters these too may be most spread out, and therefore most visible to be plotted out in a two-dimensional diagram; whereas if two directions through the data (or two of the original variables) are chosen at random, the clusters may be much less spread apart from each other, and may in fact be much more likely to substantially overlay each other, making them indistinguishable. Similarly, in regression analysis, the larger the number of explanatory variables allowed, the greater is the chance of overfitting the model, producing conclusions that fail to generalise to other datasets.

One approach, especially when there are strong correlations between different possible explanatory variables, is to reduce them to a few principal components and then run the regression against them, a method called principal component regression. Dimensionality reduction may also be appropriate when the variables in a dataset are noisy.

If each column of the dataset contains independent identically distributed Gaussian noise, then the columns of T will also contain similarly identically distributed Gaussian noise (such a distribution is invariant under the effects of the matrix W, which can be thought of as a high-dimensional rotation of the co-ordinate axes). However, with more of the total variance concentrated in the first few principal components compared to the same noise variance, the proportionate effect of the noise is less—the first few components achieve a higher signal-to-noise ratio.

PCA thus can have the effect of concentrating much of the signal into the first few principal components, which can usefully be captured by dimensionality reduction; while the later principal components may be dominated by noise, and so disposed of without great loss. If the dataset is not too large, the significance of the principal components can be tested using parametric bootstrap, as an aid in determining how many principal components to retain. \mathbf{X} = \mathbf{U}\mathbf{\Sigma}\mathbf{W}^T Here Σ is an n-by-p rectangular diagonal matrix of positive numbers σ (k) , called the singular values of X; U is an n-by-n matrix, the columns of which are orthogonal unit vectors of length n called the left singular vectors of X; and W is a p-by-p matrix whose columns are orthogonal unit vectors of length p and called the right singular vectors of X. In terms of this factorization, the matrix X T X can be written \begin{align} \mathbf{X}^T\mathbf{X} & = \mathbf{W}\mathbf{\Sigma}^\mathsf{T} \mathbf{U}^\mathsf{T} \mathbf{U}\mathbf{\Sigma}\mathbf{W}^\mathsf{T} \\ & = \mathbf{W}\mathbf{\Sigma}^\mathsf{T} \mathbf{\Sigma} \mathbf{W}^\mathsf{T} \\ & = \mathbf{W}\mathbf{\hat{\Sigma}}^2 \mathbf{W}^\mathsf{T} \end{align} where \mathbf{\hat{\Sigma}} is the square diagonal matrix with the singular values of X and the excess zeros chopped off that satisfies \mathbf{\hat{\Sigma}^2}=\mathbf{\Sigma}^\mathsf{T} \mathbf{\Sigma} .

Comparison with the eigenvector factorization of X T X establishes that the right singular vectors W of X are equivalent to the eigenvectors of X T X, while the singular values σ (k) of \mathbf are equal to the square-root of the eigenvalues λ (k) of X T X. Using the singular value decomposition the score matrix T can be written \begin{align} \mathbf{T} & = \mathbf{X} \mathbf{W} \\ & = \mathbf{U}\mathbf{\Sigma}\mathbf{W}^\mathsf{T} \mathbf{W} \\ & = \mathbf{U}\mathbf{\Sigma} \end{align} so each column of T is given by one of the left singular vectors of X multiplied by the corresponding singular value. This form is also the polar decomposition of T. Efficient algorithms exist to calculate the SVD of X without having to form the matrix X T X, so computing the SVD is now the standard way to calculate a principal components analysis from a data matrix, unless only a handful of components are required.

As with the eigen-decomposition, a truncated score matrix T L can be obtained by considering only the first L largest singular values and their singular vectors: \mathbf{T}_L = \mathbf{U}_L\mathbf{\Sigma}_L = \mathbf{X} \mathbf{W}_L The truncation of a matrix M or T using a truncated singular value decomposition in this way produces a truncated matrix that is the nearest possible matrix of rank L to the original matrix, in the sense of the difference between the two having the smallest possible Frobenius norm, a result known as the Eckart–Young theorem [1936]. Theorem (Optimal k‑dimensional fit).

Let P be an n×m data matrix whose columns have been mean‑centered and scaled, and let P = U \,\Sigma\, V^{T} be its singular value decomposition. Then the best rank‑k approximation to P in the least‑squares (Frobenius‑norm) sense is P_{k} = U_{k}\,\Sigma_{k}\,V_{k}^{T} , where V k consists of the first k columns of V. Moreover, the relative residual variance is R(k)=\frac{\sum_{j=k+1}^{m}\sigma_{j}^{2}}{\sum_{j=1}^{m}\sigma_{j}^{2}} . \Sigma = \frac{1}{n}\,\mathbf X_\text{c}^{\mathsf T}\mathbf X_\text{c}.

If the j ‑th variable is multiplied by a factor \alpha_j we obtain \mathbf X_\text{c}^{(\alpha)} = \mathbf X_\text{c}D,\qquad D = \operatorname{diag}(\alpha_1,\ldots,\alpha_p). Hence the new covariance is \Sigma^{(\alpha)} = D^{\mathsf T}\,\Sigma\,D. Because the eigenvalues and eigenvectors of \Sigma^{(\alpha)} are those of \Sigma scaled by D , the principal axes rotate toward any column whose variance has been inflated, exactly as the 2‑D example below illustrates. If we have just two variables and they have the same sample variance and are completely correlated, then the PCA will entail a rotation by 45° and the "weights" (they are the cosines of rotation) for the two variables with respect to the principal component will be equal.

But if we multiply all values of the first variable by 100, then the first principal component will be almost the same as that variable, with a small contribution from the other variable, whereas the second component will be almost aligned with the second original variable. This means that whenever the different variables have different units (like temperature and mass), PCA is a somewhat arbitrary method of analysis.

(Different results would be obtained if one used Fahrenheit rather than Celsius for example.) Pearson's original paper was entitled "On Lines and Planes of Closest Fit to Systems of Points in Space" – "in space" implies physical Euclidean space where such concerns do not arise.

One way of making the PCA less arbitrary is to use variables scaled so as to have unit variance, by standardizing the data and hence use the autocorrelation matrix instead of the autocovariance matrix as a basis for PCA. However, this compresses (or expands) the fluctuations in all dimensions of the signal space to unit variance.

Classical PCA assumes the cloud of points has already been translated so its centroid is at the origin. Write each observation as \mathbf q_i = \boldsymbol\mu + \mathbf z_i,\qquad \boldsymbol\mu = \tfrac{1}{n}\sum_{i=1}^{n}\mathbf q_i. Without subtracting \boldsymbol\mu we are in effect diagonalising \Sigma_{\text{unc}} \;=\; n\,\boldsymbol\mu\boldsymbol\mu^{\mathsf T} \;+\;\tfrac{1}{n}\,\mathbf Z^{\mathsf T}\mathbf Z, where \mathbf Z is the centered matrix.

The rank‑one term n\,\boldsymbol\mu\boldsymbol\mu^{\mathsf T} often dominates, forcing the leading eigenvector to point almost exactly toward the mean and obliterating any structure in the centred part \mathbf Z . After mean subtraction that term vanishes and the principal axes align with the true directions of maximal variance. Mean-centering is unnecessary if performing a principal components analysis on a correlation matrix, as the data are already centered after calculating correlations.

Correlations are derived from the cross-product of two standard scores (Z-scores) or statistical moments (hence the name: Pearson Product-Moment Correlation). Also see the article by Kromrey & Foster-Johnson (1998) on "Mean-centering in Moderated Regression: Much Ado About Nothing".

Since covariances are correlations of normalized variables (Z- or standard-scores) a PCA based on the correlation matrix of X is equal to a PCA based on the covariance matrix of Z, the standardized version of X. PCA is a popular primary technique in pattern recognition. It is not, however, optimized for class separability.

However, it has been used to quantify the distance between two or more classes by calculating center of mass for each class in principal component space and reporting Euclidean distance between center of mass of two or more classes. The linear discriminant analysis is an alternative which is optimized for class separability. y =\mathbf{B'}x where y is a q-element vector and \mathbf{B'} is a (q × p) matrix, and let \mathbf _y = \mathbf{B'}\mathbf{\Sigma}\mathbf{B} be the variance-covariance matrix for y . Then the trace of \mathbf{\Sigma}_y , denoted \operatorname{tr} (\mathbf{\Sigma}_y) , is maximized by taking \mathbf{B} = \mathbf{A}_q , where \mathbf{A}_q consists of the first q columns of \mathbf{A} (\mathbf{B'} is the transpose of \mathbf{B}) .

( \mathbf{A} is not defined here) Property 2: Consider again the orthonormal transformation y = \mathbf{B'}x with x, \mathbf{B}, \mathbf{A} and \mathbf{\Sigma}_y defined as before. Then \operatorname{tr}(\mathbf{\Sigma}_y) is minimized by taking \mathbf{B} = \mathbf{A}_q^*, where \mathbf{A}_q^* consists of the last q columns of \mathbf{A} .

The statistical implication of this property is that the last few PCs are not simply unstructured left-overs after removing the important PCs. Because these last PCs have variances as small as possible they are useful in their own right.

They can help to detect unsuspected near-constant linear relationships between the elements of , and they may also be useful in regression, in selecting a subset of variables from , and in outlier detection. Property 3: (Spectral decomposition of ) \mathbf = \lambda_1\alpha_1\alpha_1' + \cdots + \lambda_p\alpha_p\alpha_p' Before we look at its usage, we first look at diagonal elements, \operatorname{Var}(x_j) = \sum_{k=1}^P \lambda_k\alpha_{kj}^2 Then, perhaps the main statistical implication of the result is that not only can we decompose the combined variances of all the elements of into decreasing contributions due to each PC, but we can also decompose the whole covariance matrix into contributions \lambda_k\alpha_k\alpha_k' from each PC.

Although not strictly decreasing, the elements of \lambda_k\alpha_k\alpha_k' will tend to become smaller as k increases, as \lambda_k\alpha_k\alpha_k' is nonincreasing for increasing k , whereas the elements of \alpha_k tend to stay about the same size because of the normalization constraints: \alpha_{k}'\alpha_{k}=1, k=1, \dots, p . \mathbf{x}=\mathbf{s}+\mathbf{n}, that is, that the data vector \mathbf{x} is the sum of the desired information-bearing signal \mathbf{s} and a noise signal \mathbf{n} one can show that PCA can be optimal for dimensionality reduction, from an information-theoretic point-of-view. In particular, Linsker showed that if \mathbf{s} is Gaussian and \mathbf{n} is Gaussian noise with a covariance matrix proportional to the identity matrix, the PCA maximizes the mutual information I(\mathbf{y};\mathbf{s}) between the desired information \mathbf{s} and the dimensionality-reduced output \mathbf{y}=\mathbf{W}_L^T\mathbf{x} .

If the noise is still Gaussian and has a covariance matrix proportional to the identity matrix (that is, the components of the vector \mathbf{n} are iid), but the information-bearing signal \mathbf{s} is non-Gaussian (which is a common scenario), PCA at least minimizes an upper bound on the information loss, which is defined as I(\mathbf{x};\mathbf{s}) - I(\mathbf{y};\mathbf{s}). The optimality of PCA is also preserved if the noise \mathbf{n} is iid and at least more Gaussian (in terms of the Kullback–Leibler divergence) than the information-bearing signal \mathbf{s} .

In general, even if the above signal model holds, PCA loses its information-theoretic optimality as soon as the noise \mathbf{n} becomes dependent. \mathbf{Y} = \mathbb{KLT} \{ \mathbf{X} \} Organize the data set Suppose you have data comprising a set of observations of p variables, and you want to reduce the data so that each observation can be described with only L variables, L \mathbf{x}_1 \ldots \mathbf{x}_n with each \mathbf{x}_i representing a single grouped observation of the p variables. Write \mathbf{x}_1 \ldots \mathbf{x}_n as row vectors, each with p elements.

Place the row vectors into a single matrix X of dimensions n × p. Calculate the empirical mean Find the empirical mean along each column j = 1, ..., p. Place the calculated mean values into an empirical mean vector u of dimensions p × 1. u_j = \frac{1}{n} \sum_{i=1}^n X_{ij} Calculate the deviations from the mean Mean subtraction is an integral part of the solution towards finding a principal component basis that minimizes the mean square error of approximating the data. Hence we proceed by centering the data as follows: Subtract the empirical mean vector \mathbf{u}^{T} from each row of the data matrix X. Store mean-subtracted data in the n × p matrix B. \mathbf{B} = \mathbf{X} - \mathbf{h}\mathbf{u}^T where h is an column vector of all 1s: h_i = 1 \, \qquad \qquad \text{for } i = 1, \ldots, n In some applications, each variable (column of B) may also be scaled to have a variance equal to 1 (see Z-score).

This step affects the calculated principal components, but makes them independent of the units used to measure the different variables. Find the covariance matrix Find the p × p empirical covariance matrix C from matrix B: \mathbf{C} = { 1 \over {n-1} } \mathbf{B}^{*} \mathbf{B} where * is the conjugate transpose operator.

If B consists entirely of real numbers, which is the case in many applications, the "conjugate transpose" is the same as the regular transpose. The reasoning behind using instead of n to calculate the covariance is Bessel's correction.

Find the eigenvectors and eigenvalues of the covariance matrix Compute the matrix V of eigenvectors which diagonalizes the covariance matrix C: \mathbf{V}^{-1} \mathbf{C} \mathbf{V} = \mathbf{D} where D is the diagonal matrix of eigenvalues of C. This step will typically involve the use of a computer-based algorithm for computing eigenvectors and eigenvalues. These algorithms are readily available as sub-components of most matrix algebra systems, such as SAS, R, MATLAB, Mathematica, SciPy, IDL (Interactive Data Language), or GNU Octave as well as OpenCV.

Matrix D will take the form of an p × p diagonal matrix, where D_{k\ell} = \lambda_k \qquad \text{for } k = \ell is the jth eigenvalue of the covariance matrix C, and D_{k\ell} = 0 \qquad \text{for } k \ne \ell. Matrix V, also of dimension p × p, contains p column vectors, each of length p, which represent the p eigenvectors of the covariance matrix C. The eigenvalues and eigenvectors are ordered and paired.

The jth eigenvalue corresponds to the jth eigenvector. Matrix V denotes the matrix of right eigenvectors (as opposed to left eigenvectors). In general, the matrix of right eigenvectors need not be the (conjugate) transpose of the matrix of left eigenvectors.

Rearrange the eigenvectors and eigenvalues Sort the columns of the eigenvector matrix V and eigenvalue matrix D in order of decreasing eigenvalue. Make sure to maintain the correct pairings between the columns in each matrix.

Compute the cumulative energy content for each eigenvector The eigenvalues represent the distribution of the source data's energy among each of the eigenvectors, where the eigenvectors form a basis for the data. The cumulative energy content g for the jth eigenvector is the sum of the energy content across all of the eigenvalues from 1 through j divided by the sum of energy content across all eigenvalues (shown in step 8): g_j = \sum_{k=1}^j D_{kk} \qquad \text{for } j = 1,\dots,p Select a subset of the eigenvectors as basis vectors Save the first L columns of V as the p × L matrix W: W_{kl} = V_{k\ell} \qquad \text{for } k = 1,\dots,p \qquad \ell = 1,\dots,L where 1 \leq L \leq p. Use the vector g as a guide in choosing an appropriate value for L. The goal is to choose a value of L as small as possible while achieving a reasonably high value of g on a percentage basis.

For example, you may want to choose L so that the cumulative energy g is above a certain threshold, like 90 percent. In this case, choose the smallest value of L such that \frac{g_L}{g_p} \ge 0.9 Project the data onto the new basis The projected data points are the rows of the matrix \mathbf{T} = \mathbf{B} \cdot \mathbf{W} That is, the first column of \mathbf{T} is the projection of the data points onto the first principal component, the second column is the projection onto the second principal component, etc. \operatorname{cov}(PX) &= \operatorname{E}[PX~(PX)^{*}]\\ &= \operatorname{E}[PX~X^{*}P^{*}]\\ &= P\operatorname{E}[XX^{*}]P^{*}\\ &= P\operatorname{cov}(X)P^{-1}\\ \end{align} Hence (\ast) holds if and only if \operatorname{cov}(X) were diagonalisable by P . This is very constructive, as cov(X) is guaranteed to be a non-negative definite matrix and thus is guaranteed to be diagonalisable by some unitary matrix.

In PCA, the contribution of each component is ranked based on the magnitude of its corresponding eigenvalue, which is equivalent to the fractional residual variance (FRV) in analyzing empirical data. For NMF, its components are ranked based only on the empirical FRV curves.

The residual fractional eigenvalue plots, that is, 1-\sum_{i=1}^k \lambda_i\Big/\sum_{j=1}^n \lambda_j as a function of component number k given a total of n components, for PCA have a flat plateau, where no data are captured to remove the quasi-static noise, then the curves drop quickly as an indication of over-fitting (random noise). The FRV curves for NMF is decreasing continuously when the NMF components are constructed sequentially, indicating the continuous capturing of quasi-static noise; then converge to higher levels than PCA, indicating the less over-fitting property of NMF.

Given a matrix E , it tries to decompose it into two matrices such that E=AP . A key difference from techniques such as PCA and ICA is that some of the entries of A are constrained to be 0. Here P is termed the regulatory layer.

While in general such a decomposition can have multiple solutions, they prove that if the following conditions are satisfied : A has full column rank Each column of A must have at least L-1 zeroes where L is the number of columns of A (or alternatively the number of rows of P ). The justification for this criterion is that if a node is removed from the regulatory layer along with all the output nodes connected to it, the result must still be characterized by a connectivity matrix with full column rank. P must have full row rank. then the decomposition is unique up to multiplication by a scalar.

Pagès Jérôme (2014). Multiple Factor Analysis by Example Using R. Chapman & Hall/CRC The R Series London 272 p

Text derived from https://en.wikipedia.org/w/index.php?oldid=1369268948 under CC BY-SA 4.0; modifications have been made.

Sources cited in this revision

39 named source(s) extracted from the cited revision. A source with no author is shown by its publication; nothing is invented.