Delta method

In statistics, the delta method is a method of deriving the asymptotic distribution of a random variable. It is applicable when the random variable being considered can be defined as a differentiable function of a random variable which is asymptotically Gaussian. More generally, the delta method applies to Hadamard directionally differentiable functionals of stochastic processes that converge to a limiting process.

In statistics, the delta method is a method of deriving the asymptotic distribution of a random variable. It is applicable when the random variable being considered can be defined as a differentiable function of a random variable which is asymptotically Gaussian. More generally, the delta method applies to Hadamard directionally differentiable functionals of stochastic processes that converge to a limiting process. ==Univariate delta method== While the delta method generalizes easily to a multivariate setting, careful motivation of the technique is more easily demonstrated in univariate terms. Roughly, if there is a sequence of random variables satisfying {\sqrt{n}[X_n-\theta]\,\xrightarrow{D}\,\mathcal{N}(0,\sigma^2)}, where θ and σ 2 are finite valued constants and \xrightarrow{D} denotes convergence in distribution, then {\sqrt{n}[g(X_n)-g(\theta)]\,\xrightarrow{D}\,\mathcal{N}(0,\sigma^2\cdot[g'(\theta)]^2)} for any function g satisfying the property that its first derivative, evaluated at \theta , g'(\theta) exists and is non-zero valued. The intuition of the delta method is that any such g function, in a "small enough" range of the function, can be approximated via a first order Taylor polynomial, i.e., a linear function. If the random variable is roughly normal then a linear transformation of it is also normal. A small range can be ensured when approximating the function around the concentrating point of the sequence of random variables, so long as the variance is "small enough". When g is applied to a random variable such as the mean, the delta method would tend to work better as the sample size increases, since it would help reduce the variance, and thus the Taylor approximation would be applied to a smaller range of the function g at the point of interest. ===Proof in the univariate case=== Demonstration of this result is straightforward under the assumption that g(x) is differentiable near the neighborhood of \theta and g'(x) is continuous at \theta with g'(\theta)\neq 0 . To begin, we use the mean value theorem (i.e.: the first order approximation of a Taylor series using Taylor's theorem): g(X_n)=g(\theta)+g'(\tilde{\theta}_n)(X_n-\theta), where \tilde{\theta}_n lies between and θ. Note that since X_n\,\xrightarrow{P}\,\theta and |\tilde{\theta}_n-\theta| , it must be that \tilde{\theta}_n \,\xrightarrow{P}\,\theta and since is continuous, applying the continuous mapping theorem yields g'(\tilde{\theta}_n)\,\xrightarrow{P}\,g'(\theta), where \xrightarrow{P} denotes convergence in probability as n tends to infinity. Rearranging the terms and multiplying by \sqrt{n} gives \sqrt{n}[g(X_n)-g(\theta)]=g'(\tilde{\theta}_n)\sqrt{n}[X_n-\theta]. Since {\sqrt{n}[X_n-\theta] \xrightarrow{D} \mathcal{N}(0,\sigma^2)} by assumption, it follows from Slutsky's theorem that {\sqrt{n}[g(X_n)-g(\theta)] \xrightarrow{D} \mathcal{N}(0,\sigma^2[g'(\theta)]^2)}. This concludes the proof. ====Proof with an explicit order of approximation==== Alternatively, one can add one more step at the end, to obtain the order of approximation: \begin{align} \sqrt{n}[g(X_n)-g(\theta)]&=g'(\tilde{\theta}_n)\sqrt{n}[X_n-\theta]\\[5pt] &=\sqrt{n}[X_n-\theta]\left[ g'(\tilde{\theta}_n )+g'(\theta)-g'(\theta)\right]\\[5pt] &=\sqrt{n}[X_n-\theta]\left[g'(\theta)\right]+\sqrt{n}[X_n-\theta]\left[ g'(\tilde{\theta}_n )-g'(\theta)\right]\\[5pt] &=\sqrt{n}[X_n-\theta]\left[g'(\theta)\right]+O_p(1)\cdot o_p(1)\\[5pt] &=\sqrt{n}[X_n-\theta]\left[g'(\theta)\right]+o_p(1). \end{align} This demonstrates that the error in the approximation converges to 0 in probability. ==Multivariate delta method== By definition, a consistent estimator B converges in probability to its true value β, and often a central limit theorem can be applied to obtain asymptotic normality: \sqrt{n}\left(B-\beta\right)\,\xrightarrow{D}\,N\left(0, \Sigma \right), where n is the number of observations and Σ is a (symmetric positive semi-definite) covariance matrix. Suppose we want to estimate the variance of a scalar-valued function h of the estimator B. Keeping only the first two terms of the Taylor series, and using vector notation for the gradient, we can estimate h(B) as h(B) \approx h(\beta) + \nabla h(\beta)^T \cdot (B-\beta) which implies the variance of h(B) is approximately \begin{align} \operatorname{Var}\left(h(B)\right) & \approx \operatorname{Var}\left(h(\beta) + \nabla h(\beta)^T \cdot (B-\beta)\right) \\[5pt] & = \operatorname{Var}\left(h(\beta) + \nabla h(\beta)^T \cdot B - \nabla h(\beta)^T \cdot \beta\right) \\[5pt] & = \operatorname{Var}\left(\nabla h(\beta)^T \cdot B\right) \\[5pt] & = \nabla h(\beta)^T \cdot \operatorname{Cov}(B) \cdot \nabla h(\beta) \\[5pt] & = \nabla h(\beta)^T \cdot \frac{\Sigma}{n} \cdot \nabla h(\beta) \end{align} One can use the mean value theorem (for real-valued functions of many variables) to see that this does not rely on taking first order approximation. The delta method therefore implies that \sqrt{n}\left(h(B)-h(\beta)\right)\,\xrightarrow{D}\,N\left(0, \nabla h(\beta)^T \cdot \Sigma \cdot \nabla h(\beta)\right) or in univariate terms, \sqrt{n}\left(h(B)-h(\beta)\right)\,\xrightarrow{D}\,N\left(0, \sigma^2 \cdot \left(h^\prime(\beta)\right)^2 \right). ==Alternative form== The delta method is often used in a form that is essentially identical to that above, but without the assumption that or B is asymptotically normal. Often the only context is that the variance is "small". The results then just give approximations to the means and covariances of the transformed quantities. For example, the formulae presented in Klein (1953, p. 258) are: \begin{align} \operatorname{Var} \left(h_r \right) = & \sum_i \left( \frac{\partial h_r}{\partial B_i} \right)^2 \operatorname{Var}\left( B_i \right) + \sum_i \sum_{j \neq i} \left( \frac{ \partial h_r }{ \partial B_i } \right) \left( \frac{ \partial h_r }{ \partial B_j } \right) \operatorname{Cov}\left( B_i, B_j \right) \\ \operatorname{Cov}\left( h_r, h_s \right) = & \sum_i \left( \frac{ \partial h_r }{ \partial B_i } \right) \left( \frac{\partial h_s }{ \partial B_i } \right) \operatorname{Var}\left( B_i \right) + \sum_i \sum_{j \neq i} \left( \frac{\partial h_r}{\partial B_i} \right) \left(\frac{\partial h_s}{\partial B_j} \right) \operatorname{Cov}\left( B_i, B_j \right) \end{align} where is the rth element of h(B) and B i is the ith element of B. When the delta method cannot be applied. However, if exists and is not zero, the second-order delta method can be applied. By the Taylor expansion, n[g(X_n)-g(\theta)]=\frac{1}{2}n[X_n-\theta]^2\left[g(\theta)\right]+o_p(1) , so that the variance of g\left(X_n\right) relies on up to the 4th moment of X_n . Note that when using the second-order method, the expectation of is no longer simply , but also has a correction term that depends on the curvature: E[g(X_n)] \approx g(\theta) + \frac{1}{2}g(\theta)\sigma^2 . The intuition is that if the function has significant curvature, even if the input variable is Gaussian (and has symmetric tails), the expected value of the function is no longer equal to the function of the mean, . The second-order delta method is also useful in conducting a more accurate approximation of g\left(X_n\right) 's distribution when sample size is small. \sqrt{n}[g(X_n)-g(\theta)]=\sqrt{n}[X_n-\theta] g'(\theta)+\frac{1}{2}\frac{n[X_n-\theta]^2}{\sqrt{n}} g(\theta) +o_p(1) . For example, when X_n follows the standard normal distribution, g\left(X_n\right) can be approximated as the weighted sum of a standard normal and a chi-square with 1 degree of freedom. ==Nonparametric delta method== A version of the delta method exists in nonparametric statistics. Let X_i \sim F be an independent and identically distributed random variable with a sample of size n with an empirical distribution function \hat{F}_n , and let T be a functional. If T is Hadamard differentiable with respect to the Chebyshev metric, then \frac{ T(\hat{F}_n) - T(F) }{ \widehat{\text{se}} } \xrightarrow{D} N(0, 1) where \widehat{\text{se}} = \frac{\hat{\tau}}{\sqrt{n}} and \hat{\tau}^2 = \frac{1}{n}\sum_{i=1}^n \hat{L}^2(X_i) , with \hat{L}(x) = L_{\hat{F}_n}(\delta_x) denoting the empirical influence function for T . A nonparametric (1-\alpha) pointwise asymptotic confidence interval for T(F) is therefore given by T(\hat{F}_n) \pm z_{1-\alpha/2} \widehat{\text{se}} where z_q denotes the q -quantile of the standard normal. T(F)=\int x\,dF(x)=\mu, contamination of F by a point mass at x gives T\bigl((1-\varepsilon)F+\varepsilon\delta_x\bigr) =(1-\varepsilon)\mu+\varepsilon x. Therefore, the influence function is L_F(x)=x-\mu. Since T(\widehat F_n)=\overline X_n , its empirical version is \widehat L(x)=x-\overline X_n. Consequently, \widehat\tau^2 =\frac{1}{n}\sum_{i=1}^{n}(X_i-\overline X_n)^2, and the standard error obtained from the general formula above is \widehat{\operatorname{se}}(\overline X_n) =\frac{\widehat\tau}{\sqrt n}. Let Z=(X,Y) , and let T(F) be the Pearson correlation coefficient T(F)=\rho= \frac{\operatorname{E}[XY]-\operatorname{E}[X]\operatorname{E}[Y]} {\sqrt{\operatorname{Var}(X)\operatorname{Var}(Y)}}. Writing \widetilde x=\frac{x-\mu_X}{\sigma_X}, \qquad \widetilde y=\frac{y-\mu_Y}{\sigma_Y}, the influence function of the correlation functional is L_F(x,y) = \widetilde x\widetilde y -\frac{\rho}{2} \left(\widetilde x^{\,2}+\widetilde y^{\,2}\right). Let \widehat x_i=\frac{X_i-\overline X}{\widehat\sigma_X}, \qquad \widehat y_i=\frac{Y_i-\overline Y}{\widehat\sigma_Y}, and let \widehat\rho be the sample correlation coefficient. The empirical influence values are then \widehat L(X_i,Y_i) = \widehat x_i\widehat y_i -\frac{\widehat\rho}{2} \left(\widehat x_i^{\,2}+\widehat y_i^{\,2}\right). Thus, the variance factor in the general standard-error formula may be estimated by \widehat\tau^2 = \frac{1}{n}\sum_{i=1}^{n} \widehat L^2(X_i,Y_i). q_p=F^{-1}(p) be the quantile of order p . Suppose that F has a density f that is continuous and positive at q_p . The influence function of the quantile functional is L_F(x) = \frac{p-\mathbf 1_{\{x\leq q_p\}}}{f(q_p)}. If \widehat q_p is the sample quantile and \widehat f(\widehat q_p) is an estimate of the density at that quantile, the influence function may be estimated by \widehat L(x) = \frac{p-\mathbf 1_{\{x\leq\widehat q_p\}}} {\widehat f(\widehat q_p)}. It follows that \widehat\tau^2 = \frac{1}{n}\sum_{i=1}^{n} \left( \frac{p-\mathbf 1_{\{X_i\leq\widehat q_p\}}} {\widehat f(\widehat q_p)} \right)^2. At the population level, \operatorname{E}_F[L_F^2(X)] = \frac{p(1-p)}{f(q_p)^2}, so the asymptotic variance of the sample quantile is \operatorname{Var}(\widehat q_p) \sim \frac{p(1-p)}{n f(q_p)^2}. ==See also== Taylor expansions for the moments of functions of random variables Variance-stabilizing transformation ==References== ==Further reading== ==External links==

Text derived from https://en.wikipedia.org/w/index.php?oldid=1376248387 under CC BY-SA 4.0; modifications have been made.