Lehmann–Scheffé theorem
In statistics, the Lehmann–Scheffé theorem provides sufficient conditions for the existence of a best unbiased estimator in a statistical model.
≈ 12 min read
Names in other languages
No names in other languages are held for this entry yet.
Article bodies are English and are not machine-translated. A name in another language is not a translation of the article.
In statistics, the Lehmann–Scheffé theorem provides sufficient conditions for the existence of a best unbiased estimator in a statistical model. The theorem states that any unbiased estimator for a quantity that depends on the data only through a complete, sufficient statistic is the unique uniformly minimum-variance unbiased estimator (UMVUE) of that quantity.
The Lehmann–Scheffé theorem is named after Erich Leo Lehmann and Henry Scheffé, given their two early papers. Given a vector X= (X_1, X_2, \dots, X_n) of random samples from a distribution \mathbb{P}_\theta for some parameter \theta\in\Theta , the goal is to establish sufficient conditions for the existence of an UMVU estimator T^\ast(X) for some quantity g(\theta) , that is, \mathbb{E}_\theta[T^\ast(X)]=g(\theta) and for any unbiased estimator S(X) it holds \mathrm{Var}_\theta(T^\ast(X))\leq\mathrm{Var}_\theta(S(X)),\quad \forall\theta\in\Theta.
The Rao–Blackwell theorem already shows that, given a sufficient statistic T , the estimator T^\ast(X)=\mathbb E[S(X)\mid T(X)] has a uniformly smaller variance than S(X) , but it does not guarantee that T^\ast is already UMVU. This is where the Lehmann–Scheffé theorem comes in, if T is also complete, that is, for any real-valued measurable function \varphi it holds \mathbb E_\theta[\varphi(T(X))]=0\implies \varphi(T(X))=0 \quad\mathbb P_\theta\text{-a.s.}
Statement
As above, let X= (X_1, X_2, \dots, X_n) be a vector of random samples from a distribution \mathbb{P}_\theta for some parameter \theta\in\Theta and \Theta an arbitrary set. Assume that there exists a complete, sufficient statistic T for the family of distributions (\mathbb{P}_\theta)_{\theta\in\Theta} .
Then, the following two equivalent statements hold: There exists at most one measurable function h such that T^\ast=h(T(X)) is unbiased for g(\theta) and \mathbb{E}_\theta[T^\ast(X)^2] for all \theta\in\Theta , in which case T^\ast(X) is the unique UMVUE for g(\theta) . For any unbiased estimator S(X) , if it exists, with \mathbb{E}_\theta[S(X)^2] for all \theta\in\Theta , the estimator T^\ast(X)=\mathbb{E}_\theta[S(X)\mid T(X)] is the unique UMVUE for g(\theta) .
In fact, the theorem does not state that unbiased estimators exist in the first place. However, if they do, then there exists a unique square-integrable UMVUE. Moreover, the estimator T^\ast(X)=\mathbb{E}_\theta[S(X)\mid T(X)] does neither depend on \theta , since T is sufficient, nor on S , since T is also complete.
Proof
In the following, the dependence of an estimator on the data X will not be written out explicitly, i.e., we write S instead of S(X) . First of all, if there is no unbiased estimator for g(\theta) , then there is obviously no UMVUE, and if all unbiased estimator are not square-integrable, then their variances are infinity and the statement is trivial. Thus, we focus on the case where a square-integrable unbiased estimator exists.
Uniqueness of h : Let S=h_1(T) and R=h_2(T) be unbiased estimators of g(\theta) for some measurable functions h_1 and h_2 . For the expectation of the difference it holds \mathbb{E}_\theta[h_1(T)-h_2(T)]=g(\theta)-g(\theta)=0. Since T is complete, this implies h_1-h_2=0 .
Thus, T^\ast=h_1(T)=:h(T) is the unique unbiased estimator that is a function of T . h(T) is the UMVUE: Let S be any square-integrable unbiased estimator and S^\ast=\mathbb{E}_\theta[S\mid T] . By the factorization lemma there exists measurable function \tilde h such that S^\ast=\tilde h(T) and since S^\ast is unbiased as well, by the above, it must hold S^\ast=\tilde h(T)=h(T)=T^\ast . Thus, by the Rao–Blackwell theorem, it follows \mathrm{Var}_\theta(T^\ast)=\mathrm{Var}_\theta(S^\ast)=\mathrm{Var}_\theta(\mathbb{E}_\theta[S\mid T])\leq \mathrm{Var}_\theta(S), \quad \forall \theta\in\Theta.
In other words, T^\ast=h(T) is an UMVUE and according to the first part it is unique. The UMVUE, if it exists, is the (unique) solution of the equation \mathbb{E}_\theta[h(T)]=g(\theta) for all \theta\in\Theta . If h is, for example, a linear function, then solving this equation is fairly easy.
Method 2: Conditioning on an unbiased estimator First, it suffices to find any unbiased estimator S of g(\theta) , which is often easily feasible. The UMVUE can then be determined by evaluating the condition expectation \mathbb{E}_\theta[S\mid T] . Since the choice of S is arbitrary, it is preferable to choose it such that the conditional expectation is as simple as possible.
Let X_1,\dots,X_n be Bernoulli-distributed with probability p\in(0,1) . The joint probability mass function f_p is of the form f_p(x_1,\dots,x_n)=\prod_{i=1}^n p^{x_i}(1-p)^{1-x_i} = \left(\frac{p}{1-p}\right)^{\sum_{i=1}^n x_i} (1-p)^n, \quad x_i\in\{0,1\}, thus, by the Fisher–Neyman factorization theorem, T(x)=\sum_{i=1}^n x_i is a sufficient statistic.
It is also complete: Let \varphi be any measurable function such that \mathbb{E}_\theta[\varphi(T(X))]=0 . Since T is \mathrm{Ber}(n,p) -distributed, this means 0=\mathbb{E}_\theta[\varphi(T(X))]=\sum_{k=0}^n \varphi(k)\binom{n}{k}p^k(1-p)^{n-k}=(1-p)^n\sum_{k=0}^n \varphi(k)\binom{n}{k} r^k, with r=p/(1-p) . Since the right hand side is a polynomial in r>0 that is equal to zero, each coefficient must be zero as well, implying \varphi(k)=0 for all k . For the estimation of the parameter p , it is now easy to see that the UMVUE is the sample mean \overline{X}=\frac{1}{n}\sum_{i=1}^n X_i = \frac{1}{n}T(X), since it is unbiased and a function of T(X) . Finding the UMVUE for the parameter p(1-p) (that is, the variance of the distribution) is less obvious.
According to method 1, we seek a function h such that p(1-p)=\mathbb{E}_p[h(T(X))]=\sum_{k=0}^n h(k)\binom{n}{k}p^k(1-p)^{n-k}. By defining \rho=p/(1-p) , the above rewrites as \sum_{k=0}^n h(k)\binom{n}{k}\rho^k = \rho(1+\rho)^{n-2} = \sum_{k=1}^{n-1}\binom{n-1}{k-1}\rho^k. Comparing the coefficients shows that h(k)=k(n-k)/(n(n-1)) , thus, the UMVUE is given by h(T(X))=\frac{n}{n-1}\overline{X}(1-\overline{X}).
Noting that in the Bernoulli model X_i=X_i^2 and that \sum_{i=1}^n(X_i-\overline{X})^2 = \sum_{i=1}^nX_i^2 - n\overline{X}^2 , the UMVUE is, in fact, the unbiased sample variance S_n^2 . Let X_1,\dots,X_n be uniformly distributed on the interval (0,\theta) for some \theta to be determined.
The joint probability mass function f_p is of the form f_p(x_1,\dots,x_n)=\prod_{i=1}^n \frac{1}{\theta}1_{(0,\theta)}(x_i) = \frac{1}{\theta^n} 1_{[0,\infty)}\left(\min_{i=1,\dots,n}x_i\right)1_{(-\infty,\theta]}\left(\max_{i=1,\dots,n}x_i\right), \quad x\in\R^n, thus, by the Fisher–Neyman factorization theorem, T(x)=\max_{i=1,\dots,n} x_i is a sufficient statistic. To see the completeness of T , the distribution of T(X) is \mathbb P_\theta(T(X)\leq t)=\mathbb P_\theta(X_i\leq t, \forall i=1,\dots,n)=\prod_{i=1}^n\mathbb P_\theta(X_i\leq t) = \prod_{i=1}^n\frac{t}{\theta}=\frac{t^n}{\theta^n}.
Thus, for any measurable function \varphi such that \mathbb{E}_\theta[\varphi(T(X))]=0 it holds for any \theta>0 0=\mathbb{E}_\theta[\varphi(T(X))]=\int_0^\theta \varphi(x)\frac{n}{\theta^n}x^{n-1}\mathrm{d}x \implies \int_0^\theta \varphi(x)x^{n-1}\mathrm{d}x=0, and consequently for any b>a>0 \int_a^b\varphi(x)x^{n-1}\mathrm{d}x = \int_0^b\varphi(x)x^{n-1}\mathrm{d}x - \int_0^a\varphi(x)x^{n-1}\mathrm{d}x=0. This implies that \varphi=0 (almost everywhere) on [0,\infty) .
So T is also complete. Its expectation is \mathbb E_\theta[T(X)]=\int_0^\theta x\frac{n}{\theta^n}x^{n-1}\mathrm{d}x = \frac{n}{\theta^n}\frac{\theta^{n+1}}{n+1}=\frac{n}{n+1}\theta. Thus, the rescaled estimator T^\ast(X)=\frac{n+1}{n}T(X)=\frac{n+1}{n}\max_{i=1,\dots,n}X_i is unbiased and since it is a function of T , it is the UMVUE for \theta .
Let X_1,\dots,X_n be exponentially distributed with parameter \lambda and suppose the parameter g(\lambda)=\mathbb P_\lambda(X_1>t) should be estimated for some fixed t>0 . A straightforward unbiased estimator for g(\lambda) would be \widehat{g(\lambda)}=\frac{1}{n}\sum_{i=1}^n 1_{\{X_i>t\}} , however, it turns out to be sub-optimal.
Since the exponential distribution is an exponential family, it is known that its sufficient statistic \overline X=\frac{1}{n}\sum_{i=1}^nX_i is also complete. Moreover, the estimator T=1_{\{X_1>t \}} is unbiased. Thus, according to method 2, the estimator T^\ast = \mathbb E[T\mid \overline X]=\mathbb P(X_1>t\mid\overline X) is the UMVUE.
It remains to evaluate the conditional expectation. First of all, X_1/(n\overline X)\sim\mathrm{Beta}(1,n-1) , which is independent of \lambda (also called ancillary).
In fact, since X_1 and S=\sum_{i=2}^n X_i are independent and S\sim \Gamma(n-1,\lambda) , the cumulative distribution function at y\in(0,1) reads as \begin{align} \mathbb P\left(\frac{X_1}{n\overline X}\leq y\right) &= \mathbb P\left(\frac{X_1}{X_1+S}\leq y\right)\\ &= \int_0^\infty\int_0^\infty 1_{\{x/(x+s)\leq y\}}\lambda e^{-\lambda x} \frac{\lambda^{n-1}}{(n-2)!}s^{n-2}e^{-\lambda s}\mathrm{d}x\mathrm{d}s \\ &= \int_0^1\int_0^\infty 1_{\{u\leq y\}} \frac{\lambda^n}{(n-2)!}r^{n-2}(1-u)^{n-2}e^{-\lambda r}r\mathrm{d}r\mathrm{d}u \\ &= \int_0^1 1_{\{u\leq y\}}(1-u)^{n-2}\frac{1}{(n-2)!}\underbrace{\int_0^\infty \lambda^n r^{n-1}e^{-\lambda r}\mathrm{d}r}_{=(n-1)!}\mathrm{d}u \\ &= \int_0^y (n-1)(1-u)^{n-2}\mathrm{d}u, \end{align} with the substitution x=ru and s=r(1-u) . Thus, by Basu's theorem, X_1/(n\overline X) and \overline X are independent and the conditional expectation simplifies to \mathbb P(X_1>t\mid \overline X=x) = \mathbb P\left(\frac{X_1}{n\overline X}>\frac{t}{n\overline X}\mid \overline X=x\right) = \mathbb P\left(\frac{X_1}{n\overline X}>\frac{t}{nx}\right) = \left(1-\frac{t}{nx}\right)^{n-1}. due to the above derivation. The constructed estimator T^\ast=\left(1-\frac{t}{n\overline{X}} \right)^{n-1} is unbiased and has a smaller variance than the naive estimator \widehat{g(\lambda)} for all possible \lambda>0 .
An example of an improvable Rao–Blackwell improvement, when using a minimal sufficient statistic that is not complete, was provided by Galili and Meilijson in 2016. Let X_1, \ldots, X_n be a random sample from a scale-uniform distribution X \sim U ( (1-k) \theta, (1+k) \theta), with unknown mean \operatorname{E}[X]=\theta and known design parameter k \in (0,1) .
In the search for "best" possible unbiased estimators for \theta , it is natural to consider X_1 as an initial (crude) unbiased estimator for \theta and then try to improve it. Since X_1 is not a function of T = \left( X_{(1)}, X_{(n)} \right) , the minimal sufficient statistic for \theta (where X_{(1)} = \min_i X_i and X_{(n)} = \max_i X_i ), it may be improved using the Rao–Blackwell theorem as follows: \hat{\theta}_{RB} =\operatorname{E}_\theta[X_1\mid X_{(1)}, X_{( n)}] = \frac{X_{(1)}+X_{(n)}} 2.
However, the following unbiased estimator can be shown to have lower variance: \hat{\theta}_{LV} = \frac 1 {k^2\frac{n-1}{n+1}+1} \cdot \frac{(1-k)X_{(1)} + (1+k) X_{(n)}} 2. And in fact, it could be even further improved when using the following estimator: \hat{\theta}_\text{BAYES}=\frac{n+1} n \left[1- \frac{\frac{X_{(1)} (1+k)}{X_{(n)} (1-k)}-1}{ \left (\frac{X_{(1)} (1+k)}{X_{(n)} (1-k)}\right )^{n+1} -1} \right] \frac{X_{(n)}}{1+k} The model is a scale model.
Optimal equivariant estimators can then be derived for loss functions that are invariant.
See also
Completeness (statistics) Sufficient statistic Minimum-variance unbiased estimator Rao–Blackwell theorem
Text derived from https://en.wikipedia.org/w/index.php?oldid=1362933525 under CC BY-SA 4.0; modifications have been made.