Wishart distribution
{{Probability distribution | name =Wishart | type =density | pdf_image = | cdf_image = | notation = | parameters = degrees of freedom (real) scale matrix ( pos. def) | support = positive definite matrix | pdf = f_{\mathbf X}(\mathbf X) =…
≈ 14 min read
Wishart distribution. encyclopegia.org, snapshot enwiki-001372849384. /a/wishart-distribution
Names in other languages
No names in other languages are held for this entry yet.
Article bodies are English and are not machine-translated. A name in another language is not a translation of the article.
{{Probability distribution | name =Wishart | type =density | pdf_image = | cdf_image = | notation = | parameters = degrees of freedom (real) scale matrix ( pos. def) | support = positive definite matrix | pdf = f_{\mathbf X}(\mathbf X) = \frac ^{(n-p-1)/2} e^{-\frac{1}{2}\operatorname{tr}({\mathbf V}^{-1}\mathbf{X})} where \left|{\mathbf X}\right| is the determinant of \mathbf X and is the multivariate gamma function defined as \Gamma_p \left (\frac n 2 \right )= \pi^{p(p-1)/4}\prod_{j=1}^p \Gamma\left( \frac{n}{2} - \frac{j-1}{2} \right ). The density above is not the joint density of all the p^2 elements of the random matrix (such density does not exist because of the symmetry constrains X_{ij}=X_{ji} ), it is rather the joint density of p(p+1)/2 elements X_{ij} for i\le j (, page 38).
Also, the density formula above applies only to positive definite matrices \mathbf x; for other matrices the density is equal to zero. In fact the above definition can be extended to any real . If , then the Wishart no longer has a density—instead it represents a singular distribution that takes values in a lower-dimension subspace of the space of matrices.
Use in Bayesian statistics
In Bayesian statistics, in the context of the multivariate normal distribution, the Wishart distribution is the conjugate prior to the precision matrix , where is the covariance matrix. The use of Normal-Wishart conjugate priors (for mean and precision) is particularly common for vector autoregression models.
Choice of parameters
The least informative, proper Wishart prior is obtained by setting . A common choice for V leverages the fact that the mean of X ~W p (V, n) is nV. Then V is chosen so that nV equals an initial guess for X. For instance, when estimating a precision matrix Σ −1 ~ W p (V, n) a reasonable choice for V would be n −1 Σ 0 −1 , where Σ 0 is some prior estimate for the covariance matrix Σ.
Log-expectation
The following formula plays a role in variational Bayes derivations for Bayes networks involving the Wishart distribution. From equation (2.63), \operatorname{E}[\, \ln\left|\mathbf{X}\right|\, ] = \psi_p\left(\frac n 2\right) + p \, \ln(2) + \ln|\mathbf{V}| where \psi_p is the multivariate digamma function (the derivative of the log of the multivariate gamma function).
Log-variance
The following variance computation could be of help in Bayesian statistics: \operatorname{Var}\left[\, \ln\left|\mathbf{X}\right| \,\right]=\sum_{i=1}^p \psi_1\left(\frac{n+1-i} 2\right) where \psi_1 is the trigamma function. This comes up when computing the Fisher information of the Wishart random variable.
Entropy
The information entropy of the distribution has the following formula: \operatorname{H}\left[\, \mathbf{X} \,\right] = -\ln \left( B(\mathbf{V},n) \right) -\frac{n-p-1}{2} \operatorname{E}\left[\, \ln\left|\mathbf{X}\right|\,\right] + \frac{np}{2} where is the normalizing constant of the distribution: B(\mathbf{V},n) = \frac{1}{\left|\mathbf{V}\right|^{n/2} 2^{np/2}\Gamma_p\left(\frac n 2 \right)}. This can be expanded as follows: \begin{align} \operatorname{H}\left[\, \mathbf{X}\, \right] & = \frac{n}{2} \ln \left|\mathbf{V}\right| +\frac{n p}{2} \ln 2 + \ln \Gamma_p \left(\frac{n}{2} \right) - \frac{n-p-1}{2} \operatorname{E}\left[\, \ln\left|\mathbf{X}\right|\, \right] + \frac{n p}{2} \\[8pt] &= \frac{n}{2} \ln\left|\mathbf{V}\right| + \frac{n p}{2} \ln 2 + \ln\Gamma_p\left(\frac{n}{2} \right) - \frac{n-p-1} 2 \left( \psi_p \left(\frac{n}{2}\right) + p\ln 2 + \ln\left|\mathbf{V}\right|\right) + \frac{n p}{2} \\[8pt] &= \frac{n}{2} \ln\left|\mathbf{V}\right| + \frac{n p}{2} \ln 2 + \ln\Gamma_p\left(\frac n 2\right) - \frac{n-p-1}{2} \psi_p\left(\frac n 2 \right) - \frac{n-p-1} 2 \left(p\ln 2 +\ln\left|\mathbf{V}\right| \right) + \frac{n p}{2} \\[8pt] &= \frac{p+1}{2} \ln\left|\mathbf{V}\right| + \frac1 2 p(p+1) \ln 2 + \ln\Gamma_p\left(\frac n 2\right) - \frac{n-p-1}{2} \psi_p\left(\frac n 2 \right) + \frac{n p}{2} \end{align}
Cross-entropy
The cross-entropy of two Wishart distributions p_0 with parameters n_0, V_0 and p_1 with parameters n_1, V_1 is \begin{align} H(p_0, p_1) &= \operatorname{E}_{p_0}[\, -\log p_1\, ]\\[8pt] &= \operatorname{E}_{p_0} \left[\, -\log \frac{\left|\mathbf{X}\right|^{(n_1 - p_1 - 1)/2} e^{-\operatorname{tr}(\mathbf{V}_1^{-1} \mathbf{X})/2}}{2^{n_1 p_1/2} \left|\mathbf{V}_1\right|^{n_1/2} \Gamma_{p_1}\left(\tfrac{n_1}{2}\right)} \right]\\[8pt] &= \tfrac{n_1 p_1} 2 \log 2 + \tfrac{n_1} 2 \log \left|\mathbf{V}_1\right| + \log \Gamma_{p_1}(\tfrac{n_1} 2) - \tfrac{n_1 - p_1 - 1} 2 \operatorname{E}_{p_0}\left[\, \log\left|\mathbf{X}\right|\, \right] + \tfrac{1}{2}\operatorname{E}_{p_0}\left[\, \operatorname{tr}\left(\,\mathbf{V}_1^{-1}\mathbf{X}\,\right) \, \right] \\[8pt] &= \tfrac{n_1 p_1}{2} \log 2 + \tfrac{n_1} 2 \log \left|\mathbf{V}_1\right| + \log \Gamma_{p_1}(\tfrac{n_1}{2}) - \tfrac{n_1 - p_1 - 1}{2} \left( \psi_{p_0}(\tfrac{n_0} 2) + p_0 \log 2 + \log \left|\mathbf{V}_0\right|\right)+ \tfrac{1}{2} \operatorname{tr}\left(\, \mathbf{V}_1^{-1} n_0 \mathbf{V}_0\, \right) \\[8pt] &=-\tfrac{n_1}{2} \log \left|\, \mathbf{V}_1^{-1} \mathbf{V}_0\, \right| + \tfrac{p_1+1} 2 \log \left|\mathbf{V}_0\right| + \tfrac{n_0} 2 \operatorname{tr}\left(\, \mathbf{V}_1^{-1} \mathbf{V}_0\right)+ \log \Gamma_{p_1}\left(\tfrac{n_1}{2}\right) - \tfrac{n_1 - p_1 - 1}{2} \psi_{p_0}(\tfrac{n_0}{2}) + \tfrac{n_1(p_1 - p_0)+p_0(p_1+1)}{2} \log 2 \end{align} Note that when p_0=p_1 and n_0=n_1 we recover the entropy.
KL-divergence
The Kullback–Leibler divergence of p_1 from p_0 is \begin{align} D_{KL}(p_0 \| p_1) & = H(p_0, p_1) - H(p_0) \\[6pt] & =-\frac{n_1} 2 \log |\mathbf{V}_1^{-1} \mathbf{V}_0| + \frac{n_0}{2}(\operatorname{tr}(\mathbf{V}_1^{-1} \mathbf{V}_0) - p)+ \log \frac{\Gamma_p\left(\frac{n_1} 2 \right)}{\Gamma_p\left(\frac{n_0} 2 \right)} + \tfrac{n_0 - n_1 } 2 \psi_p\left(\frac{n_0} 2\right) \end{align}
Characteristic function
The characteristic function of the Wishart distribution is \Theta \mapsto \operatorname{E}\left[ \, \exp\left( \,i \operatorname{tr}\left(\,\mathbf{X}{\mathbf\Theta}\,\right)\,\right)\, \right] = \left|\, 1 - 2i\, {\mathbf\Theta}\,{\mathbf V}\, \right|^{-n/2} where denotes expectation. (Here is any matrix with the same dimensions as , indicates the identity matrix, and is a square root of ).
Properly interpreting this formula requires a little care, because noninteger complex powers are multivalued; when is noninteger, the correct branch must be determined via analytic continuation.
Theorem
If a random matrix has a Wishart distribution with degrees of freedom and variance matrix — write \mathbf{X}\sim\mathcal{W}_p({\mathbf V},m) — and is a matrix of rank , then \mathbf{C}\mathbf{X}{\mathbf C}^T \sim \mathcal{W}_q\left({\mathbf C}{\mathbf V}{\mathbf C}^T,m\right).
Corollary 1
If is a nonzero constant vector, then: \sigma_z^{-2} \, {\mathbf z}^T\mathbf{X}{\mathbf z} \sim \chi_m^2. In this case, \chi_m^2 is the chi-squared distribution and \sigma_z^2={\mathbf z}^T{\mathbf V}{\mathbf z} (note that \sigma_z^2 is a constant; it is positive because is positive definite).
Corollary 2
Consider the case where (that is, the -th element is one and all others zero). Then corollary 1 above shows that \sigma_{jj}^{-1} \, w_{jj}\sim \chi^2_m gives the marginal distribution of each of the elements on the matrix's diagonal.
George Seber points out that the Wishart distribution is not called the “multivariate chi-squared distribution” because the marginal distribution of the off-diagonal elements is not chi-squared. Seber prefers to reserve the term multivariate for the case when all univariate marginals belong to the same family.
Estimator of the multivariate normal distribution
The Wishart distribution is the sampling distribution of the maximum-likelihood estimator (MLE) of the covariance matrix of a multivariate normal distribution. A derivation of the MLE uses the spectral theorem.
Bartlett decomposition
The Bartlett decomposition of a matrix from a -variate Wishart distribution with scale matrix and degrees of freedom is the factorization: \mathbf{X} = {\textbf L}{\textbf A}{\textbf A}^T{\textbf L}^T, where is the Cholesky factor of , and: \mathbf A = \begin{pmatrix} c_1 & 0 & 0 & \cdots & 0\\ n_{21} & c_2 &0 & \cdots& 0 \\ n_{31} & n_{32} & c_3 & \cdots & 0\\ \vdots & \vdots & \vdots &\ddots & \vdots \\ n_{p1} & n_{p2} & n_{p3} &\cdots & c_p \end{pmatrix} where c_i^2 \sim \chi^2_{n-i+1} and independently. This provides a useful method for obtaining random samples from a Wishart distribution.
Marginal distribution of matrix elements
Let be a variance matrix characterized by correlation coefficient and its lower Cholesky factor: \mathbf{V} = \begin{pmatrix} \sigma_1^2 & \rho \sigma_1 \sigma_2 \\ \rho \sigma_1 \sigma_2 & \sigma_2^2 \end{pmatrix}, \qquad \mathbf{L} = \begin{pmatrix} \sigma_1 & 0 \\ \rho \sigma_2 & \sqrt{1-\rho^2} \sigma_2 \end{pmatrix} Multiplying through the Bartlett decomposition above, we find that a random sample from the Wishart distribution is \mathbf{X} = \begin{pmatrix} \sigma_1^2 c_1^2 & \sigma_1 \sigma_2 \left (\rho c_1^2 + \sqrt{1-\rho^2} c_1 n_{21} \right ) \\ \sigma_1 \sigma_2 \left (\rho c_1^2 + \sqrt{1-\rho^2} c_1 n_{21} \right ) & \sigma_2^2 \left(\left (1-\rho^2 \right ) c_2^2 + \left (\sqrt{1-\rho^2} n_{21} + \rho c_1 \right )^2 \right) \end{pmatrix} The diagonal elements, most evidently in the first element, follow the distribution with degrees of freedom (scaled by ) as expected. The off-diagonal element is less familiar but can be identified as a normal variance-mean mixture where the mixing density is a distribution.
The corresponding marginal probability density for the off-diagonal element is therefore the variance-gamma distribution f(x_{12}) = \frac{\left | x_{12} \right |^{\frac{n-1}{2}}}{\Gamma\left(\frac{n}{2}\right) \sqrt{2^{n-1} \pi \left (1-\rho^2 \right ) \left (\sigma_1 \sigma_2 \right )^{n+1}}} \cdot K_{\frac{n-1}{2}} \left(\frac{\left |x_{12} \right |}{\sigma_1 \sigma_2 \left (1-\rho^2 \right )}\right) \exp{\left(\frac{\rho x_{12}}{\sigma_1 \sigma_2 (1-\rho^2)}\right)} where is the modified Bessel function of the second kind. Similar results may be found for higher dimensions.
In general, if X follows a Wishart distribution with parameters, \Sigma, n , then for i \neq j , the off-diagonal elements X_{ij} \sim \text{VG}(n, \Sigma_{ij}, (\Sigma_{ii} \Sigma_{jj} - \Sigma_{ij}^2)^{1/2}, 0) . It is also possible to write down the moment-generating function even in the noncentral case (essentially the nth power of Craig (1936) equation 10) although the probability density becomes an infinite sum of Bessel functions.
The range of the shape parameter
It can be shown that the Wishart distribution can be defined if and only if the shape parameter belongs to the set \Lambda_p:=\{0,\ldots,p-1\}\cup \left(p-1,\infty\right). This set is named after Simon Gindikin, who introduced it in the 1970s in the context of gamma distributions on homogeneous cones.
However, for the new parameters in the discrete spectrum of the Gindikin ensemble, namely, \Lambda_p^*:=\{0, \ldots, p-1\}, the corresponding Wishart distribution has no Lebesgue density. In random matrix theory, the Wishart family is often studied through its Laguerre ensembles.
For the real case (orthogonal symmetry, =1), the joint density of the eigenvalues \lambda_1,\dots,\lambda_p \ge 0 of \mathbf{X}\sim W_p(\mathbf{I},n) is c_{n,p}\,\exp\!\Big(-\tfrac{1}{2}\sum_{i=1}^p \lambda_i\Big)\; \prod_{i=1}^p \lambda_i^{\tfrac{n-p-1}{2}}\; \prod_{1\le i which is the Laguerre orthogonal ensemble (LOE). The complex and quaternion analogues are the Laguerre unitary (LUE, \beta=2 ) and Laguerre symplectic (LSE, \beta=4 ) ensembles, respectively. \Big(\prod_{i=1}^p \lambda_i^{\alpha}\,e^{-\tfrac{\beta}{2}\lambda_i}\Big)\; \Big(\prod_{i -1, which reduces to LOE/LUE/LSE for \beta=1,2,4 .
For the classical Gaussian Wishart case (scale \mathbf{I} ), one has the identification \alpha \;=\; \tfrac{\beta}{2}\,(n-p+1)\;-\;1. A concrete probabilistic construction for any \beta>0 is provided by the Dumitriu–Edelman bidiagonal model.
One samples a random bidiagonal matrix \mathbf{B} with independent chi variables and sets \mathbf{L}=\mathbf{B}\mathbf{B}^{T} ; the eigenvalues of \mathbf{L} follow the β-Laguerre law with parameter \alpha . Sampling Classical \beta=1,2,4 .
Bartlett's decomposition gives \mathbf{X} = \mathbf{L}\mathbf{A}\mathbf{A}^{T}\mathbf{L}^{T} with \mathbf{A} lower-triangular having chi/normal entries, yielding LOE/LUE/LSE exactly. General \beta>0 .
In the Dumitriu–Edelman model, for matrix size p and parameter \alpha>-1 , one samples independently B_{ii}\sim \tfrac{1}{\sqrt{\beta}}\;\chi_{\beta(\alpha+p-i+1)},\qquad B_{i,i-1}\sim \tfrac{1}{\sqrt{\beta}}\;\chi_{\beta(p-i+1)} \quad (i=2,\dots,p), and sets \mathbf{L}=\mathbf{B}\mathbf{B}^{T} . Then the eigenvalues of \mathbf{L} have the β-Laguerre joint density above.
Rectangular data matrices If one wishes to realize these spectra as the singular values of a rectangular M\times N matrix (with M\ge N=p ), draw independent Haar-distributed \mathbf{U}\in O(M) (or U(M) ) and \mathbf{V}\in O(N) , and set X \;=\; U\begin{bmatrix}\operatorname{diag}(\sqrt{\lambda_1},\dots,\sqrt{\lambda_p})\\ 0\end{bmatrix} V^{*}, so that X^{*}X has eigenvalues \{\lambda_i\} . For i.i.d. Gaussian entries (true Wishart), \beta is fixed by the field (real, complex, quaternion) and \alpha reduces to \tfrac{\beta}{2}(n-p+1)-1 .
At the “hard edge” (near \lambda=0 ), β-Laguerre ensembles exhibit Bessel-kernel correlations and level repulsion of order \beta . In particular, with p\to\infty and fixed \alpha , the distribution of the smallest eigenvalue converges to a universal hard-edge (Bessel) law that depends only on \alpha and \beta .
Under proportional growth p/n\to\gamma\in(0,\infty) , the empirical spectral distribution of n^{-1}\mathbf{G}\mathbf{G}^{T} (with i.i.d. entries of variance \sigma^2 ) converges almost surely to the Marchenko–Pastur law, supported on \sigma^2(1\pm\sqrt{\gamma})^2 . When \gamma=1 , the density diverges as x^{-1/2} at the origin (a hard-edge singularity).
Relationships to other distributions
The Wishart distribution is related to the inverse-Wishart distribution, denoted by W_p^{-1} , as follows: If and if we do the change of variables , then \mathbf{C}\sim W_p^{-1}(\mathbf{V}^{-1},n) . This relationship may be derived by noting that the absolute value of the Jacobian determinant of this change of variables is , see for example equation (15.15) in. In Bayesian statistics, the Wishart distribution is a conjugate prior for the precision parameter of the multivariate normal distribution, when the mean parameter is known.
A generalization is the multivariate gamma distribution. A different type of generalization is the normal-Wishart distribution, essentially the product of a multivariate normal distribution with a Wishart distribution.
See also
Chi-squared distribution Complex Wishart distribution F-distribution Gamma distribution Hotelling's T-squared distribution Inverse-Wishart distribution Multivariate gamma distribution Student's t-distribution Wilks' lambda distribution
External links
A C++ library for random matrix generator
Text derived from https://en.wikipedia.org/w/index.php?oldid=1372849384 under CC BY-SA 4.0; modifications have been made.