Robust statistics

Robust statistics are statistics that maintain their properties even if the underlying distributional assumptions are incorrect. Robust statistical methods have been developed for many common problems, such as estimating location, scale, and regression parameters. One motivation is to produce statistical methods that are not unduly affected by outliers. Another motivation is to provide methods with good performance when there are small departures from a parametric distribution. For example, robust methods work well for mixtures of two normal distributions with different standard deviations; under this model, non-robust methods like a t-test work poorly.

Robust statistics are statistics that maintain their properties even if the underlying distributional assumptions are incorrect. Robust statistical methods have been developed for many common problems, such as estimating location, scale, and regression parameters. One motivation is to produce statistical methods that are not unduly affected by outliers. Another motivation is to provide methods with good performance when there are small departures from a parametric distribution. For example, robust methods work well for mixtures of two normal distributions with different standard deviations; under this model, non-robust methods like a t-test work poorly. Intuitively, the breakdown point of an estimator is the proportion of incorrect observations (e.g. arbitrarily large observations) an estimator can handle before giving an incorrect (e.g., arbitrarily large) result. Usually, the asymptotic (infinite sample) limit is quoted as the breakdown point, although the finite-sample breakdown point may be more useful. For example, given n independent random variables (X_1,\dots,X_n) and the corresponding realizations x_1,\dots,x_n , we can use \overline{X_n}:=\frac{X_1+\cdots+X_n}{n} to estimate the mean. Such an estimator has a breakdown point of 0 (or finite-sample breakdown point of 1/n ) because we can make \overline{x} arbitrarily large just by changing any of x_1,\dots,x_n . The higher the breakdown point of an estimator, the more robust it is. Intuitively, we can understand that a breakdown point cannot exceed 50% because if more than half of the observations are contaminated, it is not possible to distinguish between the underlying distribution and the contaminating distribution . Therefore, the maximum breakdown point is 0.5 and there are estimators which achieve such a breakdown point. For example, the median has a breakdown point of 0.5. The X% trimmed mean has a breakdown point of X%, for the chosen level of X. and contain more details. The level and the power breakdown points of tests are investigated in . Statistics with high breakdown points are sometimes called resistant statistics. (\mathcal{X},\Sigma) = (\mathbb{R},\mathcal{B}) , \Theta = \mathbb{R}\times\mathbb{R}^+ (\Gamma,S) = (\mathbb{R},\mathcal{B}) , The empirical influence function is defined as follows. Let n\in\mathbb{N}^* and X_1,\dots,X_n:(\Omega, \mathcal{A}) \to (\mathcal{X},\Sigma) are i.i.d. and (x_1,\dots,x_n) is a sample from these variables. T_n: (\mathcal{X}^n,\Sigma^n) \to (\Gamma,S) is an estimator. Let i\in\{1,\dots,n\} . The empirical influence function EIF_i at observation i is defined by: EIF_i: x\in\mathcal{X} \mapsto n\cdot(T_n(x_1,\dots,x_{i-1},x,x_{i+1},\dots,x_n)-T_n(x_1,\dots,x_{i-1},x_i,x_{i+1},\dots,x_n)) What this means is that we are replacing the i-th value in the sample by an arbitrary value and looking at the output of the estimator. Alternatively, the EIF is defined as the effect, scaled by n+1 instead of n, on the estimator of adding the point x to the sample. Let A be a convex subset of the set of all finite signed measures on \Sigma . We want to estimate the parameter \theta\in\Theta of a distribution F in A . Let the functional T:A\to\Gamma be the asymptotic value of some estimator sequence (T_n)_{n\in\mathbb{N}} . We will suppose that this functional is Fisher consistent, i.e. \forall \theta\in\Theta, T(F_\theta)=\theta . This means that at the model F , the estimator sequence asymptotically measures the correct quantity. Let G be some distribution in A . What happens when the data doesn't follow the model F exactly but another, slightly different, "going towards" G ? We're looking at: dT_{G-F}(F) = \lim_{t\to 0^+}\frac{T(tG + (1 - t)F) - T(F)}{t}, which is the one-sided Gateaux derivative of T at F , in the direction of G-F . Let x\in\mathcal{X} . \Delta_x is the probability measure which gives mass 1 to \{x\} . We choose G=\Delta_x . The influence function is then defined by: IF(x; T; F) := \lim_{t\to 0^+}\frac{T(t\Delta_x+(1-t)F) - T(F)}{t}. It describes the effect of an infinitesimal contamination at the point x on the estimate we are seeking, standardized by the mass t of the contamination (the asymptotic bias caused by contamination in the observations). For a robust estimator, we want a bounded influence function, that is, one which does not go to infinity as x becomes arbitrarily large. The empirical influence function uses the empirical distribution function \hat{F} instead of the distribution function F , making use of the drop-in principle. \rho^* := \inf_{r>0}\{r:IF(x;T;F)=0, |x|>r\} \gamma^*(T;F) := \sup_{x\in\mathcal{X}}|IF(x; T ; F)| \lambda^*(T;F) := \sup_{(x,y)\in\mathcal{X}^2\atop x\neq y}\left\|\frac{IF(y ; T; F) - IF(x; T ; F)}{y-x}\right\| This value, which looks a lot like a Lipschitz constant, represents the effect of shifting an observation slightly from x to a neighbouring point y , i.e., add an observation at y and remove one at x . M-estimators are not inherently robust. However, they can be designed to achieve favourable properties, including robustness. M-estimator are a generalization of maximum likelihood estimators (MLEs) which is determined by maximizing \prod_{i=1}^n f(x_i) or, equivalently, minimizing \sum_{i=1}^n-\log f(x_i) . In 1964, Huber proposed to generalize this to the minimization of \sum_{i=1}^n \rho(x_i) , where \rho is some function. MLE are therefore a special case of M-estimators (hence the name: "Maximum likelihood type" estimators). Minimizing \sum_{i=1}^n \rho(x_i) can often be done by differentiating \rho and solving \sum_{i=1}^n \psi(x_i) = 0 , where \psi(x) = \frac{d\rho(x)}{dx} (if \rho has a derivative). Several choices of \rho and \psi have been proposed. The two figures below show four \rho functions and their corresponding \psi functions. For squared errors, \rho(x) increases at an accelerating rate, whilst for absolute errors, it increases at a constant rate. When Winsorizing is used, a mixture of these two effects is introduced: for small values of x, \rho increases at the squared rate, but once the chosen threshold is reached (1.5 in this example), the rate of increase becomes constant. This Winsorised estimator is also known as the Huber loss function. Tukey's biweight (also known as bisquare) function behaves in a similar way to the squared error function at first, but for larger errors, the function tapers off. IF(x;T,F) = M^{-1}\psi(x,T(F)) with the p\times p given by: M = -\int_{\mathcal{X}} \left(\frac{\partial \psi(x,\theta)}{\partial \theta} \right)_{T(F)} \, dF(x). \psi(x) = \frac{x}{x^2 + \nu}. For \nu=1 , the t-distribution is equivalent to the Cauchy distribution. The degrees of freedom is sometimes known as the kurtosis parameter. It is the parameter that controls how heavy the tails are. In principle, \nu can be estimated from the data in the same way as any other parameter. In practice, it is common for there to be multiple local maxima when \nu is allowed to vary. As such, it is common to fix \nu at a value around 4 or 6. The figure below displays the \psi -function for 4 different values of \nu . \hat\mu = 27.40,\quad \hat\sigma = 3.81,\quad \hat\nu = 2.13. Fixing \nu = 4 and maximizing the likelihood gives \hat\mu = 27.49,\quad \hat\sigma = 4.51.

See also

Robust confidence intervals Robust regression Unit-weighted regression

Notes

References

. . Republished in paperback, 2005. . . . 2nd ed., CRC Press, 2011. . Republished in paperback, 2004. 2nd ed., Wiley, 2009. . . . . . . . Republished in paperback, 2003. . Preprint . . . . . Basu, Ayanendranath, et al. "Robust and efficient estimation by minimising a density power divergence." Biometrika 85.3 (1998): 549-559. https://academic.oup.com/biomet/article-abstract/85/3/549/228993

Text derived from https://en.wikipedia.org/w/index.php?oldid=1372192899 under CC BY-SA 4.0; modifications have been made.