Look-elsewhere effect
≈ 3 min read
Names in other languages
No names in other languages are held for this entry yet.
Article bodies are English and are not machine-translated. A name in another language is not a translation of the article.
The look-elsewhere effect is a phenomenon in the statistical analysis of scientific experiments where an apparently statistically significant observation may have actually arisen by chance because of the sheer size of the parameter space to be searched. Once the possibility of look-elsewhere error in an analysis is acknowledged, it can be compensated for by careful application of standard mathematical techniques. More generally known in statistics as the problem of multiple comparisons, the term gained some media attention in 2011, in the context of the search for the Higgs boson at the Large Hadron Collider.
The look-elsewhere effect is a phenomenon in the statistical analysis of scientific experiments where an apparently statistically significant observation may have actually arisen by chance because of the sheer size of the parameter space to be searched. Once the possibility of look-elsewhere error in an analysis is acknowledged, it can be compensated for by careful application of standard mathematical techniques. More generally known in statistics as the problem of multiple comparisons, the term gained some media attention in 2011, in the context of the search for the Higgs boson at the Large Hadron Collider.
Use
Many statistical tests deliver a p-value, the probability that a given result could be obtained by chance, assuming the hypothesis one seeks to prove is in fact false. When asking "does X affect Y?", it is common to vary X and see if there is significant variation in Y as a result. If this p-value is less than some predetermined statistical significance threshold α, one considers the result "significant". However, if one is performing multiple tests ("looking elsewhere" if the first test fails) then a p value of 1/n is expected to occur once per n tests. For example, when there is no real effect, an event with p < 0.05 will still occur once, on average, for each 20 tests performed. In order to compensate for this, the threshold α can be divided by the number of tests n, so a result is significant when p < α/n. Or, equivalently, multiply the observed p value by the number of tests (significant when np < α). This is a simplified case; the number n is actually the number of degrees of freedom in the tests, or the number of effectively independent tests. If they are not fully independent, the number may be lower than the number of tests. The look-elsewhere effect is a frequent cause of "significance inflation" when the number of independent tests n is underestimated because failed tests are not published. One paper may fail to mention alternative hypotheses considered, or a paper producing no result may simply not be published at all, leading to journals dominated by statistical outliers.
See also
Bonferroni correction Data dredging Law of truly large numbers: with a sample size large enough, any outrageous thing is likely to happen Littlewood's law: any individual can expect a "miracle" to happen to them at the rate of about one per month Texas sharpshooter fallacy Multiple comparisons problem
References
External links
XKCD comic illustrating the Look-Elsewhere effect
Text derived from https://en.wikipedia.org/w/index.php?oldid=1375618893 under CC BY-SA 4.0; modifications have been made.