Ensemble learning

In statistics and machine learning, ensemble methods use multiple learning algorithms to obtain better predictive performance than could be obtained from any of the constituent learning algorithms alone. Unlike a statistical ensemble in statistical mechanics, which is usually infinite, a machine learning ensemble consists of only a concrete finite set of alternative models, but typically allows for much more flexible structure to exist among those alternatives.

In statistics and machine learning, ensemble methods use multiple learning algorithms to obtain better predictive performance than could be obtained from any of the constituent learning algorithms alone. Unlike a statistical ensemble in statistical mechanics, which is usually infinite, a machine learning ensemble consists of only a concrete finite set of alternative models, but typically allows for much more flexible structure to exist among those alternatives. ==Ensemble size== While the number of component classifiers of an ensemble has a great impact on the accuracy of prediction, there is a limited number of studies addressing this problem. A priori determining of ensemble size and the volume and velocity of big data streams make this even more crucial for online ensemble classifiers. Mostly statistical tests were used for determining the proper number of components. More recently, a theoretical framework suggested that there is an ideal number of component classifiers for an ensemble such that having more or less than this number of classifiers would deteriorate the accuracy. It is called "the law of diminishing returns in ensemble construction." Their theoretical framework shows that using the same number of independent component classifiers as class labels gives the highest accuracy. y=\underset{c_j \in C}{\mathrm{argmax}} \sum_{h_i \in H}{P(c_j|h_i)P(T|h_i)P(h_i)} where y is the predicted class, C is the set of all possible classes, H is the hypothesis space, P refers to a probability, and T is the training data. As an ensemble, the Bayes optimal classifier represents a hypothesis that is not necessarily in H . The hypothesis represented by the Bayes optimal classifier, however, is the optimal hypothesis in ensemble space (the space of all possible ensembles consisting only of hypotheses in H ). This formula can be restated using Bayes' theorem, which says that the posterior is proportional to the likelihood times the prior: P(h_i|T) \propto P(T|h_i)P(h_i) hence, y=\underset{c_j \in C}{\mathrm{argmax}} \sum_{h_i \in H}{P(c_j|h_i)P(h_i|T)} ===Amended Cross-Entropy Cost: An Approach for Encouraging Diversity in Classification Ensemble=== The most common approach for training classifier is using Cross-entropy cost function. However, one would like to train an ensemble of models that have diversity so when we combine them it would provide best results. Assuming we use a simple ensemble of averaging K classifiers. Then the Amended Cross-Entropy Cost is e^k = H(p,q^k)-\frac{\lambda}{K}\sum_{j\neq k}H(q^j,q^k) where e^k is the cost function of the k^{th} classifier, q^k is the probability of the k^{ th} classifier, p is the true probability that we need to estimate and \lambda is a parameter between 0 and 1 that define the diversity that we would like to establish. When \lambda=0 we want each classifier to do its best regardless of the ensemble and when \lambda=1 we would like the classifier to be as diverse as possible. ====Land cover mapping==== Land cover mapping is one of the major applications of Earth observation satellite sensors, using remote sensing and geospatial data, to identify the materials and objects which are located on the surface of target areas. Generally, the classes of target materials include roads, buildings, rivers, lakes, and vegetation. Some different ensemble learning approaches based on artificial neural networks, kernel principal component analysis (KPCA), decision trees with boosting, random forest and automatic design of multiple classifier systems, are proposed to efficiently identify land cover objects. ====Change detection==== Change detection is an image analysis problem, consisting of the identification of places where the land cover has changed over time. Change detection is widely used in fields such as urban growth, forest and vegetation dynamics, land use and disaster monitoring. The earliest applications of ensemble classifiers in change detection are designed with the majority voting, Bayesian model averaging, and the maximum posterior probability. Given the growth of satellite data over time, the past decade sees more use of time series methods for continuous change detection from image stacks. One example is a Bayesian ensemble changepoint detection method called BEAST, with the software available as a package Rbeast in R, Python, and Matlab. ====Distributed denial of service==== Distributed denial of service is one of the most threatening cyber-attacks that may happen to an internet service provider. By combining the output of single classifiers, ensemble classifiers reduce the total error of detecting and discriminating such attacks from legitimate flash crowds. ====Malware Detection==== Classification of malware codes such as computer viruses, computer worms, trojans, ransomware and spywares with the usage of machine learning techniques, is inspired by the document categorization problem. Ensemble learning systems have shown a proper efficacy in this area. Model Hardening Malware detection models, like all machine learning models, are vulnerable to adversarial machine learning attacks where an attacker pushes the boundary of what is classified as malware. By rotating through an ensemble of models using moving target defense, the attacker has a diminished knowledge advantage. ====Intrusion detection==== An intrusion detection system monitors computer network or computer systems to identify intruder codes like an anomaly detection process. Ensemble learning successfully aids such monitoring systems to reduce their total error. == Further reading==

Text derived from https://en.wikipedia.org/w/index.php?oldid=1376024103 under CC BY-SA 4.0; modifications have been made.