Log transformation (statistics)

In statistics, the log transformation is the application of the logarithmic function to each point in a data set—that is, each data point z i is replaced with the transformed value y i = log(z i ).

Uses your system voice; voice and rate in settings.

Snapshot

≈ 11 min read

Names in other languages

No names in other languages are held for this entry yet.

Article bodies are English and are not machine-translated. A name in another language is not a translation of the article.

In statistics, the log transformation is the application of the logarithmic function to each point in a data set—that is, each data point z i is replaced with the transformed value y i = log(z i ). The log transform is usually applied so that the data, after transformation, appear to more closely meet the assumptions of a statistical inference procedure that is to be applied, or to improve the interpretability or appearance of graphs. The log transform is invertible, continuous, and monotonic.

The transformation is usually applied to a collection of comparable measurements. For example, if we are working with data on peoples' incomes in some currency unit, it would be common to transform each person's income value by the logarithm function.

Motivation

Guidance for how data should be transformed, or whether a transformation should be applied at all, should come from the particular statistical analysis to be performed. For example, a simple way to construct an approximate 95% confidence interval for the population mean is to take the sample mean plus or minus two standard error units.

However, the constant factor 2 used here is particular to the normal distribution, and is only applicable if the sample mean varies approximately normally. The central limit theorem states that in many situations, the sample mean does vary normally if the sample size is reasonably large.

However, if the population is substantially skewed and the sample size is at most moderate, the approximation provided by the central limit theorem can be poor, and the resulting confidence interval will likely have the wrong coverage probability. Thus, when there is evidence of substantial skew in the data, it is common to transform the data to a symmetric distribution before constructing a confidence interval.

If desired, the confidence interval can be constructed for statistics in the original scale, such as the median or the mean, by transforming back to the original scale using exponent (with some adjustments for CI for the mean), the inverse of the log transformation that was applied to the data. it is possible to estimate a quantile using different methods, build a CI for it, and then transform these back to the original scale so to have a CI for the quantile in the original scale. For example, it's possible to estimate the location of the median, after the log transformation, using the arithmetic mean.

Then build CI for the median using a CI for the mean and transform the CI back to the original scale using exponent. That transformed CI is then a CI for the median, not the mean. Data can also be transformed to make them easier to visualize.

For example, suppose we have a scatterplot in which the points are the countries of the world, and the data values being plotted are the land area and population of each country. If the plot is made using untransformed data (e.g. square kilometers for area and the number of people for population), most of the countries would be plotted in tight cluster of points in the lower left corner of the graph.

The few countries with very large areas and/or populations would be spread thinly around most of the graph's area. Simply rescaling units (e.g., to thousand square kilometers, or to millions of people) will not change this. However, following logarithmic transformations of both area and population, the points will be spread more uniformly in the graph.

Another reason for applying the log data transformation is to improve interpretability, even if no formal statistical analysis or visualization is to be performed.

In regression

in which the areas of the sovereign states and dependent territories in the world are plotted on the vertical axis against their populations on the horizontal axis. The upper plot uses raw data. In the lower plot, both the area and population data have been transformed using the logarithm function.

Data transformation may be used as a remedial measure to make data suitable for modeling with linear regression if the original data violates one or more assumptions of linear regression. For example, the simplest linear regression models assume a linear relationship between the expected value of Y (the response variable to be predicted) and each independent variable (when the other independent variables are held fixed).

If linearity fails to hold, even approximately, it is sometimes possible to transform either the independent or dependent variables in the regression model to improve the linearity. For example, addition of quadratic functions of the original independent variables may lead to a linear relationship with expected value of Y, resulting in a polynomial regression model, a special case of linear regression.

Another assumption of linear regression is homoscedasticity, that is the variance of errors must be the same regardless of the values of predictors. If this assumption is violated (i.e. if the data is heteroscedastic), it may be possible to find a transformation of Y alone, or transformations of both X (the predictor variables) and Y, such that the homoscedasticity assumption (in addition to the linearity assumption) holds true on the transformed variables and linear regression may therefore be applied on these.

Yet another application of data transformation is to address the problem of lack of normality in error terms. Univariate normality is not needed for least squares estimates of the regression parameters to be meaningful (see Gauss–Markov theorem). However confidence intervals and hypothesis tests will have better statistical properties if the variables exhibit multivariate normality.

Transformations that stabilize the variance of error terms (i.e. those that address heteroscedaticity) often also help make the error terms approximately normal.

Examples

Equation: Y = a + bX Meaning: A unit increase in X is associated with an average of b units increase in Y. Equation: \log(Y) = a + bX (From exponentiating both sides of the equation: Y = e^a e^{bX} ) Meaning: A unit increase in X is associated with an average increase of b units in \log(Y) , or equivalently, Y increases on an average by a multiplicative factor of e^{b}\! . For illustrative purposes, if base-10 logarithm were used instead of natural logarithm in the above transformation and the same symbols (a and b) are used to denote the regression coefficients, then a unit increase in X would lead to a 10^{b} times increase in Y on an average.

If b were 1, then this implies a 10-fold increase in Y for a unit increase in X Equation: Y = a + b \log(X) Meaning: A k-fold increase in X is associated with an average of b \times \log(k) units increase in Y. For illustrative purposes, if base-10 logarithm were used instead of natural logarithm in the above transformation and the same symbols (a and b) are used to denote the regression coefficients, then a tenfold increase in X would result in an average increase of b \times \log_{10}(10) = b units in Y Equation: \log(Y) = a + b \log(X) (From exponentiating both sides of the equation: Y = e^a X^{b} ) Meaning: A k-fold increase in X is associated with a k^{b} multiplicative increase in Y on an average. Thus if X doubles, it would result in Y changing by a multiplicative factor of 2^{b}\! .

Relation to other transformations

The logarithm transformation is commonly used for positive data. The power transformation is a family of transformations that includes the logarithm, square root, and multiplicative inverse transformations as special cases. It is parameterized by a non-negative value λ and gives the log transformation when λ=0.

To approach data transformation systematically, it is possible to use statistical estimation techniques to estimate the parameter λ in the power transformation, thereby identifying the transformation that is approximately the most appropriate in a given setting. Since the power transformation family also includes the identity transformation, this approach can also indicate whether it would be best to analyze the data without a transformation.

In regression analysis, this approach is known as the Box–Cox transformation.

Occurrence and applications

A common situation where the log transformation is applied is when a value of interest ranges over several orders of magnitude. Many physical and social phenomena exhibit such behavior — incomes, species populations, galaxy sizes, and rainfall volumes, to name a few. Power transforms, and in particular the logarithm, can often be used to induce symmetry in such data.

The logarithm is often favored because it is easy to interpret its result in terms of "fold changes". The logarithm also has a useful effect on ratios. If we are comparing positive quantities X and Y using the ratio X / Y, then if X   Y, the ratio is in the half-line (1,∞), where the ratio of 1 corresponds to equality.

In an analysis where X and Y are treated symmetrically, the log-ratio log(X / Y) is zero in the case of equality, and it has the property that if X is K times greater than Y, the log-ratio is the equidistant from zero as in the situation where Y is K times greater than X (the log-ratios are log(K) and −log(K) in these two situations). The log-normal distribution is important in the description of natural phenomena.

Many natural growth processes are driven by the accumulation of many small percentage changes which become additive on a log scale. Under appropriate regularity conditions, the distribution of the resulting accumulated changes will be increasingly well approximated by a log-normal, as noted in the section above on "Multiplicative Central Limit Theorem".

This is also known as Gibrat's law, after Robert Gibrat (1904–1980) who formulated it for companies. If the rate of accumulation of these small changes does not vary over time, growth becomes independent of size. Even if this assumption is not true, the size distributions at any age of things that grow over time tends to be log-normal.

Consequently, reference ranges for measurements in healthy individuals are more accurately estimated by assuming a log-normal distribution than by assuming a symmetric distribution about the mean. A second justification is based on the observation that fundamental natural laws imply multiplications and divisions of positive variables.

Examples are the simple gravitation law connecting masses and distance with the resulting force, or the formula for equilibrium concentrations of chemicals in a solution that connects concentrations of educts and products. Assuming log-normal distributions of the variables involved leads to consistent models in these cases.

Specific examples are given in the following subsections. contains a review and table of log-normal distributions from geology, biology, medicine, food, ecology, and other areas. is a review article on log-normal distributions in neuroscience, with annotated bibliography. If an income distribution follows a log-normal distribution with standard deviation \sigma , then the Gini coefficient, commonly use to evaluate income inequality, can be computed as G = \operatorname{erf}\left(\frac{\sigma }{2 }\right) where \operatorname{erf} is the error function, since G=2 \Phi \left(\frac{\sigma }{\sqrt{2}}\right)-1 , where \Phi(x) is the cumulative distribution function of a standard normal distribution.

In finance, in particular the Black–Scholes model, changes in the logarithm of exchange rates, price indices, and stock market indices are assumed normal (these variables behave like compound interest, not like simple interest, and so are multiplicative). However, some mathematicians such as Benoit Mandelbrot have argued that log-Lévy distributions, which possesses heavy tails would be a more appropriate model, in particular for the analysis for stock market crashes.

Indeed, stock price distributions typically exhibit a fat tail. The fat tailed distribution of changes during stock market crashes invalidate the assumptions of the central limit theorem. In scientometrics, the number of citations to journal articles and patents follows a discrete log-normal distribution.

City sizes (population) satisfy Gibrat's Law. The growth process of city sizes is proportionate and invariant with respect to size. From the central limit theorem therefore, the log of city size is normally distributed.

The number of sexual partners appears to be best described by a log-normal distribution.

See also

Data transformation (statistics) Log-normal distribution Log–log plot Variance-stabilizing transformation Box–Cox distribution Power transform Logarithm Logarithmic scale Delta method (for exp of approx normal distributions) Median test Laplace's approximation Arcsin Feature engineering Logit Pearson correlation coefficient Power transform (Box–Cox) Wilson–Hilferty transformation Whitening transformation

Log Transformations for Skewed and Wide Distributions – discussing the log and the "signed logarithm" transformations (A chapter from "Practical Data Science with R").

Text derived from https://en.wikipedia.org/w/index.php?oldid=1372164680 under CC BY-SA 4.0; modifications have been made.

Sources cited in this revision

17 named source(s) extracted from the cited revision. A source with no author is shown by its publication; nothing is invented.