Model selection

Model selection is the task of selecting a model from among various candidates on the basis of performance criterion to choose the best one. In the context of machine learning and more generally statistical analysis, this may be the selection of a statistical model from a set of candidate models, given data. In the simplest cases, a pre-existing set of data is considered. However, the task can also involve the design of experiments such that the data collected is well-suited to the problem of model selection. Given candidate models of similar predictive or explanatory power, the simplest model is most likely to be the best choice (Occam's razor). state, "The majority of the problems in statistical inference can be considered to be problems related to statistical modeling". Relatedly, has said, "How [the] translation from subject-matter problem to statistical model is done is often the most critical part of an analysis". Model selection may also refer to the problem of selecting a few representative models from a large set of computational models for the purpose of decision making or optimization under uncertainty. In machine learning, algorithmic approaches to model selection include feature selection, hyperparameter optimization, and statistical learning theory.

Model selection is the task of selecting a model from among various candidates on the basis of performance criterion to choose the best one. In the context of machine learning and more generally statistical analysis, this may be the selection of a statistical model from a set of candidate models, given data. In the simplest cases, a pre-existing set of data is considered. However, the task can also involve the design of experiments such that the data collected is well-suited to the problem of model selection. Given candidate models of similar predictive or explanatory power, the simplest model is most likely to be the best choice (Occam's razor). state, "The majority of the problems in statistical inference can be considered to be problems related to statistical modeling". Relatedly, has said, "How [the] translation from subject-matter problem to statistical model is done is often the most critical part of an analysis". Model selection may also refer to the problem of selecting a few representative models from a large set of computational models for the purpose of decision making or optimization under uncertainty. In machine learning, algorithmic approaches to model selection include feature selection, hyperparameter optimization, and statistical learning theory. ==Introduction== In its most basic forms, model selection is one of the fundamental tasks of scientific inquiry. Determining the principle that explains a series of observations is often linked directly to a mathematical model predicting those observations. For example, when Galileo performed his inclined plane experiments, he demonstrated that the motion of the balls fitted the parabola predicted by his model . Of the countless number of possible mechanisms and processes that could have produced the data, how can one even begin to choose the best model? The mathematical approach commonly taken decides among a set of candidate models; this set must be chosen by the researcher. Often simple models such as polynomials are used, at least initially . emphasize throughout their book the importance of choosing models based on sound scientific principles, such as understanding of the phenomenological processes or mechanisms (e.g., chemical reactions) underlying the data. Once the set of candidate models has been chosen, the statistical analysis allows us to select the best of these models. What is meant by best is controversial. A good model selection technique will balance goodness of fit with simplicity. More complex models will be better able to adapt their shape to fit the data (for example, a fifth-order polynomial can exactly fit six points), but the additional parameters may not represent anything useful. (Perhaps those six points are really just randomly distributed about a straight line.) Goodness of fit is generally determined using a likelihood ratio approach, or an approximation of this, leading to a chi-squared test. The complexity is generally measured by counting the number of parameters in the model. Model selection techniques can be considered as estimators of some physical quantity, such as the probability of the model producing the given data. The bias and variance are both important measures of the quality of this estimator; efficiency is also often considered. A standard example of model selection is that of curve fitting, where, given a set of points and other background knowledge (e.g. points are a result of i.i.d. samples), we must select a curve that describes the function that generated the points. ==Methods to assist in choosing the set of candidate models== Data transformation (statistics) Exploratory data analysis Model specification Scientific method ==Criteria== Below is a list of criteria for model selection. The most commonly used information criteria are (i) the Akaike information criterion and (ii) the Bayes factor and/or the Bayesian information criterion (which to some extent approximates the Bayes factor), see for a review. Akaike information criterion (AIC), a measure of the goodness fit of an estimated statistical model Bayes factor Bayesian information criterion (BIC), also known as the Schwarz information criterion, a statistical criterion for model selection Bridge criterion (BC), a statistical criterion that can attain the better performance of AIC and BIC despite the appropriateness of model specification. Cross-validation Deviance information criterion (DIC), another Bayesian oriented model selection criterion False discovery rate Focused information criterion (FIC), a selection criterion sorting statistical models by their effectiveness for a given focus parameter Hannan–Quinn information criterion, an alternative to the Akaike and Bayesian criteria Kashyap information criterion (KIC) is a powerful alternative to AIC and BIC, because KIC uses Fisher information matrix Likelihood-ratio test Mallows's C p Minimum description length Minimum message length (MML) PRESS statistic, also known as the PRESS criterion Quasi information criterion (QIC), for model selection where the log-likelihood is replaced by the quasi log-likelihood (e.g., generalized estimating equations) Structural risk minimization Stepwise regression Watanabe–Akaike information criterion (WAIC), also called the widely applicable information criterion Extended Bayesian Information Criterion (EBIC) is an extension of ordinary Bayesian information criterion (BIC) for models with high parameter spaces. Extended Fisher Information Criterion (EFIC) is a model selection criterion for linear regression models. Constrained Minimum Criterion (CMC) is a frequentist criterion for regression model selection. It seems to be the first model selection criterion motivated by geometric considerations. In the parameter vector space of the full model, every vector represents a model. There exists a ball centered on the true vector in which all vectors are correct in that they represent correct models and the true vector is the smallest in L_0 norm. As the sample size goes to infinity, the likelihood ratio confidence region for the true vector is trapped inside the ball with probability tending to one, so all vectors in the confidence region are asymptotically correct. The CMC selects the smallest vector in the confidence region. When the confidence region captures the true vector and is inside the ball, the CMC selection is the true model. Among these criteria, cross-validation is typically the most accurate, and computationally the most expensive, for supervised learning problems. say the following: ==See also== All models are wrong Analysis of competing hypotheses Automated machine learning (AutoML) Bias-variance dilemma Feature selection Freedman's paradox Grid search Identifiability Analysis Log-linear analysis Model identification Occam's razor Optimal design Parameter identification problem Scientific modelling Statistical model validation Stein's paradox ==Notes== ==References== [this has over 38000 citations on Google Scholar] (reprinted 1965, Science 148: 754–759 [https://web.archive.org/web/20161117210123/http://www.d.umn.edu/~mille066/Teaching/3000/Chamberlin-MWH.pdf] )

Text derived from https://en.wikipedia.org/w/index.php?oldid=1378672627 under CC BY-SA 4.0; modifications have been made.