Generative model

Generative models are a class of computational models frequently used for classification. In machine learning, it typically models the joint distribution of inputs and outputs, such as P(X,Y), or it models how inputs are distributed within each class, such as P(X∣Y) together with a class prior P(Y). Because it describes a full data-generating process, a generative model can be used to draw new samples that resemble the observed data, a process often referred to as synthetic data generation. Generative models are used for density estimation, simulation, and learning with missing or partially labeled data. In classification, they can predict labels by combining P(X∣Y) and P(Y) and applying Bayes' rule. Generative models are often contrasted with discriminative models, which focus on predicting outputs from inputs directly. Generative model approaches which use a joint probability distribution instead, include naive Bayes classifiers, Gaussian mixture models, variational autoencoders, generative adversarial networks and others.

Generative models are a class of computational models frequently used for classification. In machine learning, it typically models the joint distribution of inputs and outputs, such as P(X,Y), or it models how inputs are distributed within each class, such as P(X∣Y) together with a class prior P(Y). Because it describes a full data-generating process, a generative model can be used to draw new samples that resemble the observed data, a process often referred to as synthetic data generation. Generative models are used for density estimation, simulation, and learning with missing or partially labeled data. In classification, they can predict labels by combining P(X∣Y) and P(Y) and applying Bayes' rule. Generative models are often contrasted with discriminative models, which focus on predicting outputs from inputs directly. Generative model approaches which use a joint probability distribution instead, include naive Bayes classifiers, Gaussian mixture models, variational autoencoders, generative adversarial networks and others. A discriminative model is a model of the conditional probability P(Y\mid X = x) of the target Y, given an observation x. It can be used to "discriminate" the value of the target variable Y, given an observation x. Classifiers computed without using a probability model are also referred to loosely as "discriminative". The distinction between these last two classes is not consistently made; refers to these three classes as generative learning, conditional learning, and discriminative learning, but only distinguish two classes, calling them generative classifiers (joint distribution) and discriminative classifiers (conditional distribution or no distribution), not distinguishing between the latter two classes. Analogously, a classifier based on a generative model is a generative classifier, while a classifier based on a discriminative model is a discriminative classifier, though this term also refers to classifiers that are not based on a model. In application to classification, one wishes to go from an observation x to a label y (or probability distribution on labels). One can compute this directly, without using a probability distribution (distribution-free classifier); one can estimate the probability of a label given an observation, P(Y|X=x) (discriminative model), and base classification on that; or one can estimate the joint distribution P(X, Y) (generative model), from that compute the conditional probability P(Y|X=x) , and then base classification on that. These are increasingly indirect, but increasingly probabilistic, allowing more domain knowledge and probability theory to be applied. In practice different approaches are used, depending on the particular problem, and hybrids can combine strengths of multiple approaches. An alternative division defines these symmetrically as: a generative model is a model of the conditional probability of the observable X, given a target y, symbolically, P(X\mid Y = y) a discriminative model is a model of the conditional probability of the target Y, given an observation x, symbolically, P(Y\mid X = x) Regardless of precise definition, the terminology is constitutional because a generative model can be used to "generate" random instances (outcomes), either of an observation and target (x, y) , or of an observation x given a target value y, while a discriminative model or discriminative classifier (without a model) can be used to "discriminate" the value of the target variable Y, given an observation x. The difference between "discriminate" (distinguish) and "classify" is subtle, and these are not consistently distinguished. (The term "discriminative classifier" becomes a pleonasm when "discrimination" is equivalent to "classification".) The term "generative model" is also used to describe models that generate instances of output variables in a way that has no clear relationship to probability distributions over potential samples of input variables. Generative adversarial networks are examples of this class of generative models, and are judged primarily by the similarity of particular outputs to potential inputs. Such models are not classifiers. Given a finite set of labels, the two definitions of "generative model" are closely related. A model of the conditional distribution P(X\mid Y = y) is a model of the distribution of each label, and a model of the joint distribution is equivalent to a model of the distribution of label values P(Y) , together with the distribution of observations given a label, P(X\mid Y) ; symbolically, P(X, Y) = P(X\mid Y)P(Y). Thus, while a model of the joint probability distribution is more informative than a model of the distribution of label (but without their relative frequencies), it is a relatively small step, hence these are not always distinguished. Given a model of the joint distribution, P(X, Y) , the distribution of the individual variables can be computed as the marginal distributions P(X) = \sum_y P(X , Y = y) and P(Y) = \int_x P(Y, X = x) (considering X as continuous, hence integrating over it, and Y as discrete, hence summing over it), and either conditional distribution can be computed from the definition of conditional probability: P(X\mid Y)=P(X, Y)/P(Y) and P(Y\mid X)=P(X, Y)/P(X) . Given a model of one conditional probability, and estimated probability distributions for the variables X and Y, denoted P(X) and P(Y) , one can estimate the opposite conditional probability using Bayes' rule: P(X\mid Y)P(Y) = P(Y\mid X)P(X). For example, given a generative model for P(X\mid Y) , one can estimate: P(Y\mid X) = P(X\mid Y)P(Y)/P(X), and given a discriminative model for P(Y\mid X) , one can estimate: P(X\mid Y) = P(Y\mid X)P(X)/P(Y). Note that Bayes' rule (computing one conditional probability in terms of the other) and the definition of conditional probability (computing conditional probability in terms of the joint distribution) are frequently conflated as well. (x,y) = \{(1,0), (1,1), (2,0), (2,1)\} For the above data, estimating the joint probability distribution p(x,y) from the empirical measure will be the following: while p(y|x) will be following: ==Notes== ==References== ==Sources== ==External links== , (mirror, mirror), published as book (above)

Text derived from https://en.wikipedia.org/w/index.php?oldid=1371270811 under CC BY-SA 4.0; modifications have been made.