Stein's example

In decision theory and estimation theory, Stein's example (also known as Stein's phenomenon or Stein's paradox) is the observation that when three or more parameters are estimated simultaneously, there exist combined estimators more accurate on average (that is, having lower expected mean squared error) than any method that handles the parameters separately. It is named after Charles Stein of Stanford University, who discovered the phenomenon in 1955. An intuitive explanation is that optimizing for the mean-squared error of a combined estimator is not the same as optimizing for the errors of separate estimators of the individual parameters. In practical terms, if the combined error is in fact of interest, then a combined estimator should be used, even if the underlying parameters are independent. If one is instead interested in estimating an individual parameter, then using a combined estimator does not help and is in fact worse.

In decision theory and estimation theory, Stein's example (also known as Stein's phenomenon or Stein's paradox) is the observation that when three or more parameters are estimated simultaneously, there exist combined estimators more accurate on average (that is, having lower expected mean squared error) than any method that handles the parameters separately. It is named after Charles Stein of Stanford University, who discovered the phenomenon in 1955. An intuitive explanation is that optimizing for the mean-squared error of a combined estimator is not the same as optimizing for the errors of separate estimators of the individual parameters. In practical terms, if the combined error is in fact of interest, then a combined estimator should be used, even if the underlying parameters are independent. If one is instead interested in estimating an individual parameter, then using a combined estimator does not help and is in fact worse. Under these conditions, it is intuitive and common to use each measurement as an estimate of its corresponding parameter. This so-called "ordinary" decision rule can be written as \hat{\boldsymbol{\theta}} = \mathbf{X} , which is the maximum likelihood estimator (MLE). The quality of such an estimator is measured by its risk function. A commonly used risk function is the mean squared error, defined as \mathbb{E}[\|\boldsymbol{\theta} - \hat\boldsymbol{\theta}\|^2] . Surprisingly, it turns out that the "ordinary" decision rule is suboptimal (inadmissible) in terms of mean squared error when n\geq 3 . In other words, in the setting discussed here, there exist alternative estimators which always achieve lower mean squared error, no matter what the value of \boldsymbol{\theta} is. For a given \boldsymbol{\theta} one could obviously define a perfect "estimator" which is always just \boldsymbol{\theta} , but this estimator would be bad for other values of \boldsymbol{\theta} . The estimators of Stein's paradox are, for a given \boldsymbol{\theta} , better than the "ordinary" decision rule \mathbf{X} for some \mathbf{X} but necessarily worse for others. It is only on average that they are better. More accurately, an estimator \hat{\boldsymbol{\theta}}_1 is said to dominate another estimator \hat{\boldsymbol{\theta}}_2 if, for all values of \boldsymbol{\theta} , the risk of \hat{\boldsymbol{\theta}}_1 is lower than, or equal to, the risk of \hat{\boldsymbol{\theta}}_2 , and if the inequality is strict for some \boldsymbol{\theta} . An estimator is said to be admissible if no other estimator dominates it, otherwise it is inadmissible. Thus, Stein's example can be simply stated as follows: The "ordinary" decision rule of the mean of a multivariate Gaussian distribution is inadmissible under mean squared error risk. Many simple, practical estimators achieve better performance than the "ordinary" decision rule. The best-known example is the James–Stein estimator, which shrinks \mathbf{X} towards a particular point (such as the origin) by an amount inversely proportional to the distance of \mathbf{X} from that point. For a sketch of the proof of this result, see Proof of Stein's example. An alternative proof is due to Larry Brown: he proved that the ordinary estimator for an n -dimensional multivariate normal mean vector is admissible if and only if the n -dimensional Brownian motion is recurrent. Since the Brownian motion is not recurrent for n\geq 3 , the MLE is not admissible for n\geq 3 . For any particular value of \boldsymbol{\theta} the new estimator will improve at least one of the individual mean square errors \mathbb{E}[(\theta_i - \hat{\theta}_i)^2]. This is not hard − for instance, if \boldsymbol{\theta} is between −1 and 1, and \sigma=1 , then an estimator that linearly shrinks \mathbf{X} towards 0 by 0.5 (i.e., \operatorname{sign}(X_i)\max(|X_i|-0.5,0) , soft thresholding with threshold 0.5 ) will have a lower mean square error than \mathbf{X} itself. But there are other values of \boldsymbol{\theta} for which this estimator is worse than \mathbf{X} itself. The trick of the Stein estimator, and others that yield the Stein paradox, is that they adjust the shift in such a way that there is always (for any \boldsymbol{\theta} vector) at least one X_i whose mean square error is improved, and its improvement more than compensates for any degradation in mean square error that might occur for another \hat{\theta}_i . The trouble is that, without knowing \boldsymbol{\theta} , you don't know which of the n mean square errors are improved, so you can't use the Stein estimator only for those parameters. An example of the above setting occurs in channel estimation in telecommunications, for instance, because different factors affect overall channel performance. The risk function of a decision rule d(\mathbf{x}) = \mathbf{x} is R(\theta,d) = \operatorname{E}_\theta[|\boldsymbol{\theta} - \mathbf{X}|^2] =\int(\boldsymbol{\theta}-\mathbf{x})^T(\boldsymbol{\theta}-\mathbf{x})\left(\frac{1}{2\pi}\right)^{n/2} e^{(-1/2)(\boldsymbol{\theta}-\mathbf{x})^T(\boldsymbol{\theta}-\mathbf{x})}dx =n. Now consider the decision rule d'(\mathbf{x}) = \mathbf{x} - \frac{\alpha}{|\mathbf{x}|^2}\mathbf{x}, where \alpha = n-2 . We will show that d' is a better decision rule than d . The risk function is R(\theta,d') = \operatorname{E}_\theta\left[ \left|\mathbf{\theta - X} + \frac{\alpha}{|\mathbf{X}|^2}\mathbf{X}\right|^2\right] = \operatorname{E}_\theta\left[ |\mathbf{\theta - X}|^2 + 2(\mathbf{\theta - X})^T\frac{\alpha}{|\mathbf{X}|^2}\mathbf{X} + \frac{\alpha^2}{|\mathbf{X}|^4}|\mathbf{X}|^2 \right] =\operatorname{E}_\theta\left[ |\mathbf{\theta - X}|^2 \right] + 2\alpha\operatorname{E}_\theta\left[\frac{\mathbf{(\theta-X)}^T \mathbf{X}}{|\mathbf{X}|^2}\right] + \alpha^2\operatorname{E}_\theta\left[\frac{1}{|\mathbf{X}|^2}\right], which is quadratic in \alpha . We may simplify the middle term by considering a general "well-behaved" function h:\mathbf{x}\mapsto h(\mathbf{x})\in\mathbb{R} and using integration by parts. For 1\leq i\leq n , for any continuously differentiable h growing sufficiently slowly for large x_i we have: \operatorname{E}_\theta[(\theta_i - X_i)h(\mathbf{X})\mid X_j=x_j (j\neq i) ]= \int(\theta_i - x_i)h(\mathbf{x})\left(\frac{1}{2\pi}\right)^{n/2} e^{-(1/2)(\boldsymbol{\theta}-\mathbf{x})^T(\boldsymbol{\theta}-\mathbf{x})}dx_i =\left[h(\mathbf{x})\left(\frac{1}{2\pi} \right)^{n/2} e^{-(1/2)(\boldsymbol{\theta}-\mathbf{x})^T(\boldsymbol{\theta}-\mathbf{x})}\right]^\infty_{x_i=-\infty} - \int\frac{\partial h}{\partial x_i}(\mathbf{x})\left(\frac{1}{2\pi}\right)^{n/2} e^{-(1/2)(\boldsymbol{\theta}-\mathbf{x})^T(\boldsymbol{\theta}-\mathbf{x})}dx_i = - \operatorname{E}_\theta\left[\frac{\partial h}{\partial x_i}(\mathbf{X})\mid X_j=x_j (j\neq i) \right]. Therefore, \operatorname{E}_\theta[(\theta_i - X_i)h(\mathbf{X})] = -\operatorname{E}_\theta\left[\frac{\partial h}{\partial x_i}(\mathbf{X}) \right]. (This result is known as Stein's lemma.) Now, we choose h(\mathbf{x}) = \frac{x_i}{|\mathbf{x}|^2}. If h met the "well-behaved" condition (it doesn't, but this can be remedied—see below), we would have \frac{\partial h}{\partial x_i} = \frac{1}{|\mathbf{x}|^2} - \frac{2 x_i^2}{|\mathbf{x}|^4} and so \operatorname{E}_\theta\left[\frac{(\boldsymbol{\theta}-\mathbf{X})^T\mathbf{X}}{|\mathbf{X}|^2}\right] = \sum_{i=1}^n \operatorname{E}_\theta \left[ (\theta_i - X_i) \frac{X_i}{|\mathbf{X}|^2} \right] = - \sum_{i=1}^n \operatorname{E}_\theta \left[ \frac{1}{|\mathbf{X}|^2} - \frac{2 X_i^2}{|\mathbf{X}|^4} \right] = -(n-2)\operatorname{E}_\theta \left[\frac{1}{|\mathbf{X}|^2}\right]. Then returning to the risk function of d' : R(\theta,d') = n - 2\alpha(n-2)\operatorname{E}_\theta\left[\frac{1}{|\mathbf{X}|^2}\right] + \alpha^2\operatorname{E}_\theta\left[\frac{1}{|\mathbf{X}|^2}\right]. This quadratic in \alpha is minimized at \alpha = n-2 , giving R(\theta,d') = R(\theta,d) - (n-2)^2\operatorname{E}_\theta\left[\frac{1}{|\mathbf{X}|^2}\right] which of course satisfies R(\theta,d') making d an inadmissible decision rule. It remains to justify the use of h(\mathbf{X})= \frac{\mathbf{X}}{|\mathbf{X}|^2}. This function is not continuously differentiable, since it is singular at \mathbf{x}=0 . However, the function h(\mathbf{X}) = \frac{\mathbf{X}}{\varepsilon + |\mathbf{X}|^2} is continuously differentiable, and after following the algebra through and letting \varepsilon \to 0 , one obtains the same result.

Text derived from https://en.wikipedia.org/w/index.php?oldid=1349556276 under CC BY-SA 4.0; modifications have been made.