Errors-in-variables model
≈ 17 min read
Names in other languages
No names in other languages are held for this entry yet.
Article bodies are English and are not machine-translated. A name in another language is not a translation of the article.
(or attenuation bias) by a range of regression estimates in errors-in-variables models. Two regression lines (red) bound the range of linear regression possibilities. The shallow slope is obtained when the independent variable (or predictor) is on the x-axis. The steeper slope is obtained when the independent variable is on the y-axis. By convention, with the independent variable on the x-axis, the shallower slope is obtained. Green reference lines are averages within arbitrary bins along each axis. Note that the steeper green and red regression estimates are more consistent with smaller errors in the y-axis variable. In statistics, an errors-in-variables model or a measurement error model is a regression model that accounts for measurement errors in the independent variables. In contrast, standard regression models assume that those regressors have been measured exactly, or observed without error; as such, those models account only for errors in the dependent variables, or responses. In the case when some regressors have been measured with errors, estimation based on the standard assumption leads to inconsistent estimates, meaning that the parameter estimates do not tend to the true values even in very large samples. For simple linear regression the effect is an underestimate of the coefficient, known as the attenuation bias.…
(or attenuation bias) by a range of regression estimates in errors-in-variables models. Two regression lines (red) bound the range of linear regression possibilities. The shallow slope is obtained when the independent variable (or predictor) is on the x-axis. The steeper slope is obtained when the independent variable is on the y-axis. By convention, with the independent variable on the x-axis, the shallower slope is obtained. Green reference lines are averages within arbitrary bins along each axis. Note that the steeper green and red regression estimates are more consistent with smaller errors in the y-axis variable.]] In statistics, an errors-in-variables model or a measurement error model is a regression model that accounts for measurement errors in the independent variables. In contrast, standard regression models assume that those regressors have been measured exactly, or observed without error; as such, those models account only for errors in the dependent variables, or responses. In the case when some regressors have been measured with errors, estimation based on the standard assumption leads to inconsistent estimates, meaning that the parameter estimates do not tend to the true values even in very large samples. For simple linear regression the effect is an underestimate of the coefficient, known as the attenuation bias. In non-linear models the direction of the bias is likely to be more complicated. y_{t} = \alpha + \beta x_{t}^{*} + \varepsilon_t\,, \quad t=1,\ldots,T, where x_{t}^{*} denotes the true but unobserved regressor. Instead, we observe this value with an error: x_{t} = x_{t}^{*} + \eta_{t}\, where the measurement error \eta_{t} is assumed to be independent of the true value x_{t}^{*} . A practical application is the standard school science experiment for Hooke's law, in which one estimates the relationship between the weight added to a spring and the amount by which the spring stretches. If the y_{t} ′s are simply regressed on the x_{t} ′s (see simple linear regression), then the estimator for the slope coefficient is \hat{\beta}_x = \frac{\tfrac{1}{T}\sum_{t = 1}^T(x_t-\bar{x})(y_t-\bar{y})} {\tfrac{1}{T}\sum_{t=1}^T(x_t-\bar{x})^2}\,, which converges as the sample size T increases without bound: \hat{\beta}_x \xrightarrow{p} \frac{\operatorname{Cov}[\,x_t,y_t\,]}{\operatorname{Var}[\,x_t\,]} \hat{\beta} = \frac{\tfrac{1}{T}\sum_{t=1}^T(x^*_t-\bar{x})(y_t-\bar{y})} {\tfrac{1}{T}\sum_{t=1}^T(x^*_t-\bar{x})^2}\,, Variances are non-negative, so that in the limit the estimated \hat{\beta}_x is smaller than \hat{\beta} , an effect which statisticians call attenuation or regression dilution. Thus the ‘naïve’ least squares estimator \hat{\beta}_x is an inconsistent estimator for \beta . However, \hat{\beta}_x is a consistent estimator of the parameter required for a best linear predictor of y given the observed x_t : in some applications this may be what is required, rather than an estimate of the 'true' regression coefficient \beta , although that would assume that the variance of the errors in the estimation and prediction is identical. This follows directly from the result quoted immediately above, and the fact that the regression coefficient relating the y_{t} ′s to the actually observed x_{t} ′s, in a simple linear regression, is given by \beta_x = \frac{\operatorname{Cov}[\,x_t,y_t\,]}{\operatorname{Var}[\,x_t\,]} . It is this coefficient, rather than \beta , that would be required for constructing a predictor of y based on an observed x which is subject to noise. It can be argued that almost all existing data sets contain errors of different nature and magnitude, so that attenuation bias is extremely frequent (although in multivariate regression the direction of bias is ambiguous ). Jerry Hausman sees this as an iron law of econometrics: "The magnitude of the estimate is usually smaller than expected." y^* = g(x^*\!,w\,|\,\theta),\\ y = y^{*} + \varepsilon, \\ x = x^{*} + \eta, \end{cases} where \theta is the model's parameter and w are those regressors which are assumed to be error-free (for example, when linear regression contains an intercept, the regressor which corresponds to the constant certainly has no "measurement errors"). Depending on the specification these error-free regressors may or may not be treated separately; in the latter case it is simply assumed that corresponding entries in the variance matrix of \eta 's are zero. The variables y , x , w are all observed, meaning that the statistician possesses a data set of n statistical units \left\{ y_{i}, x_{i}, w_{i} \right\}_{i = 1, \dots, n} which follow the data generating process described above; the latent variables x^* , y^* , \varepsilon , and \eta are not observed, however. This specification does not encompass all the existing errors-in-variables models. For example, in some of them, function g(\cdot) may be non-parametric or semi-parametric. Other approaches model the relationship between y^* and x^* as distributional instead of functional; that is, they assume that y^* conditionally on x^* follows a certain (usually parametric) distribution. Mean-independence: \operatorname{E}[\eta|x^*]\,=\,0, the errors are mean-zero for every value of the latent regressor. This is a less restrictive assumption than the classical one, as it allows for the presence of heteroscedasticity or other effects in the measurement errors. Berkson's errors: \eta\,\perp\,x, the errors are independent of the observed regressor x. This assumption has very limited applicability. One example is round-off errors: for example, if a person's age* is a continuous random variable, whereas the observed age is truncated to the next smallest integer, then the truncation error is approximately independent of the observed age . Another possibility is with the fixed design experiment: for example, if a scientist decides to make a measurement at a certain predetermined moment of time x , say at x = 10 s , then the real measurement may occur at some other value of x^* (for example due to her finite reaction time) and such measurement error will be generally independent of the "observed" value of the regressor. Misclassification errors: special case used for the dummy regressors. If x^* is an indicator of a certain event or condition (such as person is male/female, some medical treatment given/not, etc.), then the measurement error in such regressor will correspond to the incorrect classification similar to type I and type II errors in statistical testing. In this case the error \eta may take only 3 possible values, and its distribution conditional on x^* is modeled with two parameters: \alpha = \operatorname{Pr}[\eta = -1 | x^* = 1] , and \beta =\operatorname{Pr}[\eta = 1 | x^*=0] . The necessary condition for identification is that \alpha + \beta , that is misclassification should not happen "too often". (This idea can be generalized to discrete variables with more than two possible values.) y_t = \alpha + \beta x_t^* + \varepsilon_t, \\ x_t = x_t^* + \eta_t, \end{cases} where all variables are scalar. Here α and β are the parameters of interest, whereas σ ε and σ η —standard deviations of the error terms—are the nuisance parameters. The "true" regressor x* is treated as a random variable (structural model), independent of the measurement error η (classic assumption). This model is identifiable in two cases: (1) either the latent regressor x* is not normally distributed, (2) or x* has normal distribution, but neither ε t nor η t are divisible by a normal distribution. That is, the parameters α, β can be consistently estimated from the data set \scriptstyle(x_t,\,y_t)_{t=1}^T without any additional information, provided the latent regressor is not Gaussian. Before this identifiability result was established, statisticians attempted to apply the maximum likelihood technique by assuming that all variables are normal, and then concluded that the model is not identified. The suggested remedy was to assume that some of the parameters of the model are known or can be estimated from the outside source. Such estimation methods include Deming regression — assumes that the ratio δ = σ² ε /σ² η is known. This could be appropriate for example when errors in y and x are both caused by measurements, and the accuracy of measuring devices or procedures are known. The case when δ = 1 is also known as the orthogonal regression. The case when δ = σ² x /σ² y , is the geometric mean functional relationship (GMFR). This treats both variables as having the same reliability, i.e. (σ y /σ ε = σ x /σ η . The resulting slope is the geometric mean of the ordinary least squares slope and the reverse least squares slope, i.e. the two red lines in the diagram. This method is invariant under scaling, translation, and interchange of variables, the only method that depend on only first and second moments to have this property. OLS on x can be treated as δ = 0, and OLS on y can be treated as δ = ∞. Regression with known reliability ratio λ = σ² ∗ / ( σ² η + σ² ∗ ), where σ² ∗ is the variance of the latent regressor. Such approach may be applicable for example when repeating measurements of the same unit are available, or when the reliability ratio has been known from the independent study. In this case the consistent estimate of slope is equal to the least-squares estimate divided by λ. Regression with known σ² η may occur when the source of the errors in xs is known and their variance can be calculated. This could include rounding errors, or errors introduced by the measuring device. When σ² η is known we can compute the reliability ratio as λ = ( σ² x − σ² η ) / σ² x and reduce the problem to the previous case. Estimation methods that do not assume knowledge of some of the parameters of the model, include {{unordered list |1= Method of moments — the GMM estimator based on the third- (or higher-) order joint cumulants of observable variables. The slope coefficient can be estimated from \hat\beta = \frac{\hat{K}(n_1,n_2+1)}{\hat{K}(n_1+1,n_2)}, \quad n_1,n_2>0, where (n 1 ,n 2 ) are such that K(n 1 +1,n 2 ) — the joint cumulant of (x,y) — is not zero. In the case when the third central moment of the latent regressor x* is non-zero, the formula reduces to \hat\beta = \frac{\tfrac{1}{T}\sum_{t=1}^T (x_t-\bar x)(y_t-\bar y)^2} {\tfrac{1}{T}\sum_{t=1}^T (x_t-\bar x)^2(y_t-\bar y)}\ . |2= Instrumental variables — a regression which requires that certain additional data variables z, called instruments, were available. These variables should be uncorrelated with the errors in the equation for the dependent (outcome) variable (valid), and they should also be correlated (relevant) with the true regressors x*. If such variables can be found then the estimator takes form \hat\beta = \frac{\tfrac{1}{T}\sum_{t=1}^T (z_t-\bar z)(y_t-\bar y)} {\tfrac{1}{T}\sum_{t=1}^T (z_t-\bar z)(x_t-\bar x)}\ . }} The multivariable model looks exactly like the simple linear model, only this time β, η t , x t and x* t are k×1 vectors. \begin{cases} y_t = \alpha + \beta'x_t^* + \varepsilon_t, \\ x_t = x_t^* + \eta_t. \end{cases} In the case when (ε t ,η t ) is jointly normal, the parameter β is not identified if and only if there is a non-singular k×k block matrix [a A], where a is a k×1 vector such that a′x* is distributed normally and independently of A′x*. In the case when ε t , η t1 ,..., η tk are mutually independent, the parameter β is not identified if and only if in addition to the conditions above some of the errors can be written as the sum of two independent variables one of which is normal. Some of the estimation methods for multivariable linear models are y_t = g(x^*_t) + \varepsilon_t, \\ x_t = x^*_t + \eta_t. \end{cases} Here function g can be either parametric or non-parametric. When function g is parametric it will be written as g(x*, β). For a general vector-valued regressor x* the conditions for model identifiability are not known. However, in the case of scalar x* the model is identified unless the function g is of the "log-exponential" form g(x^*) = a + b \ln\big(e^{cx^*} + d\big) and the latent regressor x* has density f_{x^*}(x) = \begin{cases} A e^{-Be^{Cx}+CDx}(e^{Cx}+E)^{-F}, & \text{if}\ d>0 \\ A e^{-Bx^2 + Cx} & \text{if}\ d=0 \end{cases} where constants A,B,C,D,E,F may depend on a,b,c,d. Despite this optimistic result, as of now no methods exist for estimating non-linear errors-in-variables models without any extraneous information. However, there are several techniques which make use of some additional data: either the instrumental variables, or repeated observations. x_{1t} = x^*_t + \eta_{1t}, \\ x_{2t} = x^*_t + \eta_{2t}, \end{cases} where x* ⊥ η 1 ⊥ η 2 . Variables η 1 , η 2 need not be identically distributed (although if they are efficiency of the estimator can be slightly improved). With only these two observations it is possible to consistently estimate the density function of x* using Kotlarski's deconvolution technique. {{unordered list |1= Li's conditional density method for parametric models. The regression equation can be written in terms of the observable variables as \operatorname{E}[\,y_t|x_t\,] = \int g(x^*_t,\beta) f_{x^*|x}(x^*_t|x_t)dx^*_t , where it would be possible to compute the integral if we knew the conditional density function ƒ x* x . If this function could be known or estimated, then the problem turns into standard non-linear regression, which can be estimated for example using the NLLS method. Assuming for simplicity that η 1 , η 2 are identically distributed, this conditional density can be computed as \hat f_{x^*|x}(x^*|x) = \frac{\hat f_{x^*}(x^*)}{\hat f_{x}(x)} \prod_{j=1}^k \hat f_{\eta_{j}}\big( x_{j} - x^*_{j} \big), where with slight abuse of notation x j denotes the j-th component of a vector. All densities in this formula can be estimated using inversion of the empirical characteristic functions. In particular, \begin{align} & \hat \varphi_{\eta_j}(v) = \frac{\hat\varphi_{x_j}(v,0)}{\hat\varphi_{x^*_j}(v)}, \quad \text{where } \hat\varphi_{x_j}(v_1,v_2) = \frac{1}{T}\sum_{t=1}^T e^{iv_1x_{1tj}+iv_2x_{2tj}}, \\ \hat\varphi_{x^*_j}(v) = \exp \int_0^v \frac{\partial\hat\varphi_{x_j}(0,v_2)/\partial v_1}{\hat\varphi_{x_j}(0,v_2)}dv_2, \\ & \hat \varphi_x(u) = \frac{1}{2T}\sum_{t=1}^T \Big( e^{iu'x_{1t}} + e^{iu'x_{2t}} \Big), \quad \hat \varphi_{x^*}(u) = \frac{\hat\varphi_x(u)}{\prod_{j=1}^k \hat\varphi_{\eta_j}(u_j)}. \end{align} To invert these characteristic function one has to apply the inverse Fourier transform, with a trimming parameter C needed to ensure the numerical stability. For example: \hat f_x(x) = \frac{1}{(2\pi)^k} \int_{-C}^{C}\cdots\int_{-C}^C e^{-iu'x} \hat\varphi_x(u) du. |2= Schennach's estimator for a parametric linear-in-parameters nonlinear-in-variables model. This is a model of the form \begin{cases} y_t = \textstyle \sum_{j=1}^k \beta_j g_j(x^*_t) + \sum_{j=1}^\ell \beta_{k+j}w_{jt} + \varepsilon_t, \\ x_{1t} = x^*_t + \eta_{1t}, \\ x_{2t} = x^*_t + \eta_{2t}, \end{cases} where w t represents variables measured without errors. The regressor x* here is scalar (the method can be extended to the case of vector x* as well). If not for the measurement errors, this would have been a standard linear model with the estimator \hat{\beta} = \big(\hat{\operatorname{E}}[\,\xi_t\xi_t'\,]\big)^{-1} \hat{\operatorname{E}}[\,\xi_t y_t\,], where \xi_t'= (g_1(x^*_t), \cdots ,g_k(x^*_t), w_{1,t}, \cdots , w_{l,t}). It turns out that all the expected values in this formula are estimable using the same deconvolution trick. In particular, for a generic observable w t (which could be 1, w 1t , …, w ℓ t , or y t ) and some function h (which could represent any g j or g i g j ) we have \operatorname{E}[\,w_th(x^*_t)\,] = \frac{1}{2\pi} \int_{-\infty}^\infty \varphi_h(-u)\psi_w(u)du, where φ h is the Fourier transform of h(x*), but using the same convention as for the characteristic functions, \varphi_h(u)=\int e^{iux}h(x)dx , and \psi_w(u) = \operatorname{E}[\,w_te^{iux^*}\,] = \frac{\operatorname{E}[w_te^{iux_{1t}}]}{\operatorname{E}[e^{iux_{1t}}]} \exp \int_0^u i\frac{\operatorname{E}[x_{2t}e^{ivx_{1t}}]}{\operatorname{E}[e^{ivx_{1t}}]}dv The resulting estimator \scriptstyle\hat\beta is consistent and asymptotically normal. |3= Schennach's estimator for a nonparametric model. The standard Nadaraya–Watson estimator for a nonparametric model takes form \hat{g}(x) = \frac{\hat{\operatorname{E}}[\,y_tK_h(x^*_t - x)\,]}{\hat{\operatorname{E}}[\,K_h(x^*_t - x)\,]}, for a suitable choice of the kernel K and the bandwidth h. Both expectations here can be estimated using the same technique as in the previous method. }}
Text derived from https://en.wikipedia.org/w/index.php?oldid=1361249363 under CC BY-SA 4.0; modifications have been made.