Point estimation

In statistics, point estimation involves the use of sample data to calculate a single value (known as a point estimate, since it identifies a point rather than an interval), which serves as a "best guess" or "best estimate" of an unknown…

Uses your system voice; voice and rate in settings.

Snapshot

≈ 13 min read

Names in other languages

No names in other languages are held for this entry yet.

Article bodies are English and are not machine-translated. A name in another language is not a translation of the article.

In statistics, point estimation involves the use of sample data to calculate a single value (known as a point estimate, since it identifies a point rather than an interval), which serves as a "best guess" or "best estimate" of an unknown quantity, for example, the population mean, the variance of a distribution, or a model parameter (in a parametric model). Point estimation can be contrasted with interval estimation: interval estimates are typically either confidence intervals, in the case of frequentist inference, or credible intervals, in the case of Bayesian inference.

More generally, a point estimator can be contrasted with a set estimator. Examples are given by confidence sets or credible sets. A point estimator can also be contrasted with a distribution estimator.

Examples are given by confidence distributions, randomized estimators, and Bayesian posteriors. Mathematically speaking, if T=h(X_1,\dots,X_n) is an estimator based on a random sample X_1,\dots,X_n drawn from a distribution \mathbb{P}_\theta , the difference \mathbb{E}_\theta[T]-\theta is called the bias of T . The estimator T is called unbiased for the parameter \theta if the bias is zero, irrespective of the value of \theta .

Otherwise, T is called biased. Examples of unbiased estimators are the sample mean and the unbiased sample variance. The concept of (un-)biasedness can be generalized to other metrics than the mean.

An unbiased estimator T fulfills \mathbb{E}_\theta[T]=\mathrm{arg}\min_{\theta'}\mathbb{E}_\theta[(T-\theta')^2]=\theta. Thus, a more general condition for unbiasedness can be defined by \mathrm{arg}\min_{\theta'}\mathbb{E}_\theta\big[W\big(T,\theta'\big)\big]=\theta, for some function W . For example, if W(T,\theta')=|T-\theta'| , then the estimator is called median-unbiased , since the median is a minimiser of the mean absolute error.

More precisely, for a random sample X_1,\dots,X_n drawn from a distribution \mathbb{P}_\theta with parameter \theta\in\R (efficiency can also be defined in a similar manner for parameter vectors) fulfilling the Cramér–Rao regularity conditions, any regular unbiased estimator T=h(X_1,\dots,X_n) of the parameter \theta fulfills the Cramér–Rao bound \mathrm{Var}_\theta(T)\geq \frac{1}{I(\theta)}\qquad\text{for any }\theta, where I(\theta) is the Fisher-information of \mathbb{P}_\theta . If \mathrm{Var}_\theta(T)= 1/I(\theta) for any value of \theta , then T is called efficient.

In fact, it can be shown that T is efficient, if and only if, \mathbb{P}_\theta is an exponential family and T (actually h ) is the natural sufficient statistic. Sometimes, the variance of an estimator is compared to the variance of some other estimator (rather than the Cramér–Rao bound): For two unbiased estimators T_1 and T_2 of the same parameter \theta , the estimator T_2 would be called more efficient than T_1 if \mathrm{Var}_\theta(T_2) , irrespective of the value of \theta .

One can also say that the most efficient estimators are the ones with the least variability of outcomes. The notion of (relative) efficiency can be extended to biased estimators by saying that an estimator T_2 is more efficient than an estimator T_1 (for the same parameter of interest), if the MSE of T_2 is smaller than the MSE of T_1 . \mathrm{Var}_\theta(\hat\theta)\leq\mathrm{Var}_\theta(\hat T), for any \theta\in\Theta .

We define sufficient statistics as follows: Let X=(X_1,\dots,X_n) be a random sample. A statistic T(X) is said to be sufficient for \theta (or for the family of distribution) if the conditional distribution of X given T does not depend on \theta .

Y_i = f_\beta(X_i) + \varepsilon_i for some function f_\beta depending on a parameter vector \beta\in\R^{p+1} and a random variable \varepsilon_i accounting for random noise in the data. The ordinary least squares estimator (OLS) \hat\beta_{\mathrm{OLS}} assumes that f_\beta is a linear function of \beta and estimates the parameter by minimizing the sum of squared differences, called residuals: \hat\beta_{\mathrm{OLS}}\in\mathrm{arg}\min_{\beta\in\R^p} \sum_{i=1}^n \left(Y_i - \beta_0 - X_{i,1}\beta_1-\dots - X_{i,p}\beta_p \right)^2.

In fact, if the noise variables \varepsilon_i are independent and identically normally distributed with mean zero, then the OLS coincides with the maximum likelihood estimator. If the rank of the matrix of covariates X=(X_1,\dots,X_n)^\top\in\R^{n\times p} is equal to p\leq n , then the OLS is uniquely given by \hat\beta_{\mathrm{OLS}} = (X^\top X)^{-1}X^\top Y, and it is the best linear unbiased estimator (BLUE) by the Gauß–Markov theorem.

If, as above, the noise variables are independent and identically normally distributed, then the OLS is even UMVU for the entire class of unbiased estimators. There are multiple variants of least squares estimation: weighted least squares (WLS) estimation: The residuals are weighted with different weighting factors w_i . generalized least squares (GLS) estimation: The noise variables are allowed to be correlated. nonlinear least squares estimation: The function f_\beta can depend non-linearly on the parameter \beta . regularized least squares estimation: The parameter is chosen such that the sum of squared residuals plus a regularization term is minimized.

Typical choices for the regularization are the euclidian norm \lVert\beta\rVert^2_2 , leading to Ridge regression, or the 1-norm \lVert\beta\rVert_1 , leading to Lasso regression. In mathematical terms, the methods works as follows: Let X = (X_1,\dots,X_n) denote a random data sample with joint probability density function or probability mass function f_\theta depending on the vector \theta=(\theta_1,\dots,\theta_p) of model parameters.

The function \theta\mapsto f_\theta(X) is called the likelihood function, often denoted by L . Assuming that true model parameter lies in a set \Theta\subset \R^p , a maximum likelihood estimator \hat\theta_{\mathrm{MLE}} fulfills L(\hat\theta_{\mathrm{MLE}})=\max_{\theta\in\Theta} L(\theta) = \max_{\theta\in\Theta}f_\theta(X). In practice, the likelihood function is often differentiable, in which case \hat\theta_{\mathrm{MLE}} is a solution of the equations \frac{\partial L(\theta)}{\partial \theta_i}=0, \quad i=1,\dots,k. Under certain assumptions on the likelihood, the MLE is strongly consistent, asymptotically efficient and asymptotically normal. Mathematically speaking, for a distribution \mathbb P_\theta depending on some vector of parameters \theta=(\theta_1,\dots,\theta_k) , the equations of the first k moments of X\sim\mathbb P_\theta are established: \begin{align} &m_1=\mathbb E[X] = f_1(\theta_1,\dots,\theta_k), \\ &\dots \\ &m_k=\mathbb E[X^k] = f_k(\theta_1,\dots,\theta_k). \end{align} If these equations have a solution, then there exists functions g_1,\dots,g_k such that the parameters can be expressed by the moments via \begin{align} &\theta_1= g_1(m_1,\dots,m_k), \\ &\dots \\ &\theta_k= g_k(m_1,\dots,m_k). \end{align} Given a random sample X_1,\dots,X_n from \mathbb P_\theta , the moments are then replaced by the sample moments \hat m_j=\frac{1}{n}\sum_{i=1}^nX_i^j , j=1,\dots,k , yielding the moment estimators: \begin{align} &\hat\theta_1= g_1(\hat m_1,\dots,\hat m_k), \\ &\dots \\ &\hat \theta_k= g_k(\hat m_1,\dots,\hat m_k). \end{align} Under relatively low assumptions on the solution functions g_1,\dots,g_k and the moments of \mathbb P_\theta , the moment estimators are strongly consistent and asymptotically normal.

Let X=(X_1,\dots,X_n) be a vector of random samples from some distribution \mathbb P_\theta depending on a parameter \theta . Method 1: For the estimation of \theta , there exists at most one function g that solves the equation \mathbb E_\theta[g(T(X))]=\theta, in which case the estimator \hat\theta=g(T) is an UMVU estimator for \theta .

Method 2: Given any unbiased estimator \tilde\theta of \theta , the estimator \hat\theta(X)=\mathbb E_\theta[\tilde\theta(X)\mid T(X)], that is, the conditional expectation of \tilde\theta given T , is the unique UMVU estimator. In some cases, computing the conditional expectation of a suitably chosen unbiased estimator given T might be easier than solving the equation from method 1.

The data is assumed to be drawn from a distribution with probability density f(\mathbf{x},\boldsymbol{\theta}) , \mathbf{x}\in\R^d , parameterized by \boldsymbol{\theta} . The key quantity, that is needed for the score matching estimation, is the score \nabla_{\mathbf{x}}\log f(\mathbf{x},\boldsymbol{\theta}) .

The true parameter (vector) \boldsymbol{\theta} is characterized by \begin{align} \boldsymbol{\theta}&=\mathrm{arg}\min_{\boldsymbol{\theta}'} \frac{1}{2}\int_{\R^d}\lVert \nabla_{\mathbf{y}}\log f(\mathbf{y},\boldsymbol{\theta}')-\nabla_{\mathbf{y}}\log f(\mathbf{y},\boldsymbol{\theta})\rVert^2 f(\mathbf{y},\boldsymbol{\theta})\mathrm{d}\mathbf{y} \\ &=\mathrm{arg}\min_{\boldsymbol{\theta}'}\sum_{i=1}^d \int_{\R^d} \left(\partial_{y_i}^2\log f(\mathbf{y},\boldsymbol{\theta}')+\frac{1}{2}(\partial_{y_i}\log f(\mathbf{y},\boldsymbol{\theta}')^2\right) f(\mathbf{y},\boldsymbol{\theta})\mathrm{d}\mathbf{y}. \end{align} In line with the second equality, for observations \mathbf{x}_1,\dots,\mathbf{x}_n , the score matching estimate \hat\boldsymbol{\theta} is defined by \hat\boldsymbol{\theta}=\mathrm{arg}\min_{\boldsymbol{\theta}'} \sum_{j=1}^n\sum_{i=1}^d \left(\partial_{x_i}^2\log f(\mathbf{x}_j,\boldsymbol{\theta}')+\frac{1}{2}(\partial_{x_i}\log f(\mathbf{x}_j,\boldsymbol{\theta}')^2\right). The main advantage of score matching estimation compared to the maximum likelihood estimation is the following.

Typically, the probability density is of the form f(\mathbf{x},\boldsymbol{\theta})=\frac{1}{Z(\boldsymbol{\theta})}q(\mathbf{x},\boldsymbol{\theta}), with an unnormalized density q(\mathbf{x},\boldsymbol{\theta}) and a normalization constant Z(\boldsymbol{\theta}) . The computation of latter is often the bottleneck of maximum likelihood estimation, especially if the dependence on \boldsymbol{\theta} is non-analytic and the parameter vector is high-dimensional.

By taking the derivatives of the logarithm of f , Z(\boldsymbol{\theta}) does not occur in the score matching estimation procedure, thus eliminating this issue. Additionally, under mild restrictions on f(\mathbf{x},\boldsymbol\theta) , the score matching estimate can be shown to be weakly consistent. p(\theta\mid X_1,\dots,X_n)=\frac{p_\theta(X_1,\dots,X_n)\pi(\theta)}{\int_\Theta p_\theta(X_1,\dots,X_n)\pi(\theta)\mathrm{d}\theta}.

The updated distribution of the parameters given the data is called the posterior distribution. Many Bayesian point estimators are the posterior distribution's statistics of central tendency, e.g., its mean, median, or mode.

Given a vector of random observations X=(X_1,\dots,X_n) and a loss function L , a Bayes estimator is any estimator \hat\theta(X) that minimizes the Bayes risk: R_\pi(\hat\theta) = \int_\Theta \mathbb E[L(\theta,\hat\theta(X))] \pi(\theta)\mathrm{d}\theta = \mathbb E\left[ \int_\Theta L\big(\theta,\hat\theta(X)\big)p(\theta\mid X)\mathrm{d}\theta\right]. Obviously, the risk is minimized when the integral under the expectation is minimized as a function of the observations.

Thus, a Bayes estimator is equivalently given as a minimizer of the posterior loss: \hat\theta(X)\in\mathrm{arg}\min_{y\in\R^n} \int_\Theta L\big(\theta,y\big)p(\theta\mid X) \mathrm{d}\theta. The most common choices for L are the following: L(\theta,y)=(\theta-y)^2 : In this case, the posterior loss is minimized by the posterior mean \hat\theta(X)=\int_\Theta \theta\, p(\theta\mid X)\mathrm{d}\theta , as observed by Gauss.

L(\theta,y)=|\theta-y| : Here, the posterior loss is minimized by the median of the posterior distribution, as observed by Laplace. By Wald's theorem, unique Bayesian estimators are admissible.

Moreover, under certain assumptions on the prior and p_\theta , the posterior mean estimator can be shown to be consistent, asymptotically normal and asymptotically efficient. \hat\theta_{\mathrm{MAP}}(X)\in\mathrm{arg}\max_{\theta\in\Theta} p(\theta\mid X). In a sense, the MAP estimator is a Bayesian analog to the maximum likelihood estimator. In fact, for a uniform prior distribution, the MAP estimator coincides with the MLE.

The MAP estimator has good asymptotic properties, even for many difficult problems, on which the maximum-likelihood estimator has difficulties. For regular problems, where the maximum-likelihood estimator is consistent, the maximum-likelihood estimator ultimately agrees with the MAP estimator. Confidence interval describes how reliable an estimate is.

We can calculate the upper and lower confidence limits of the intervals from the observed data. Suppose a dataset x 1 , . . . , x n is given, modeled as realization of random variables X 1 , . . . , X n . Let θ be the parameter of interest, and γ a number between 0 and 1.

If there exist sample statistics L n = g(X 1 , . . . , X n ) and U n = h(X 1 , . . . , X n ) such that P(L n n ) = γ for every value of θ, then (l n , u n ), where l n = g(x 1 , . . . , x n ) and u n = h(x 1 , . . . , x n ), is called a 100γ% confidence interval for θ. The number γ is called the confidence level.

In general, with a normally-distributed sample mean, Ẋ, and with a known value for the standard deviation, σ, a 100(1-α)% confidence interval for the true μ is formed by taking Ẋ ± e, with e = z 1-α/2 (σ/n 1/2 ), where z 1-α/2 is the 100(1-α/2)% cumulative value of the standard normal curve, and n is the number of data values in that column. For example, z 1-α/2 equals 1.96 for 95% confidence.

Here two limits are computed from the set of observations, say l n and u n and it is claimed with a certain degree of confidence (measured in probabilistic terms) that the true value of γ lies between l n and u n . Thus we get an interval (l n and u n ) which we expect would include the true value of γ(θ). So this type of estimation is called confidence interval estimation.

This estimation provides a range of values which the parameter is expected to lie. It generally gives more information than point estimates and are preferred when making inferences. In some way, we can say that point estimation is the opposite of interval estimation.

Text derived from https://en.wikipedia.org/w/index.php?oldid=1378403493 under CC BY-SA 4.0; modifications have been made.

Provenance

snapshot enwiki-001378403493 · captured 2026-10-04 · not yet sealed in the ledger · verify the chain · version history

Cite this page as: encyclopegia.org, “Point estimation”, snapshot enwiki-001378403493. https://encyclopegia.org/a/point-estimation

Sources cited in this revision

4 named source(s) extracted from the cited revision. A source with no author is shown by its publication; nothing is invented.