Neural network (machine learning)
≈ 16 min read
Names in other languages
No names in other languages are held for this entry yet.
Article bodies are English and are not machine-translated. A name in another language is not a translation of the article.
s in a brain. Here, each blue/green circular node in the hidden and output layers represents an artificial neuron and each red circular node in the far left layer represents an input data value. An arrow represents a connection from the output of one neuron (or data node) to the input of another. In machine learning, a neural network (NN) or artificial neural network (ANN) is a computational model inspired by the structure and functions of biological neural networks.
A neural network consists of connected units or nodes called artificial neurons, which loosely model the neurons in the brain. These are connected by edges, which model the synapses in the brain. Each artificial neuron receives signals from connected neurons, then processes them and sends a signal to other connected neurons. The "signal" is a real number, and the output of each neuron is computed by some non-linear function of the totality of its inputs, called the activation function.
The strength of the signal at each connection is determined by a weight, which adjusts as part of the training process. Groups of neurons are aggregated into layers. Each layer performs a transformation on its inputs. Signals travel from the first layer (the input layer) to the last layer (the output layer), typically passing through multiple intermediate layers (hidden layers).…
s in a brain. Here, each blue/green circular node in the hidden and output layers represents an artificial neuron and each red circular node in the far left layer represents an input data value. An arrow represents a connection from the output of one neuron (or data node) to the input of another. In machine learning, a neural network (NN) or artificial neural network (ANN) is a computational model inspired by the structure and functions of biological neural networks.
A neural network consists of connected units or nodes called artificial neurons, which loosely model the neurons in the brain. These are connected by edges, which model the synapses in the brain. Each artificial neuron receives signals from connected neurons, then processes them and sends a signal to other connected neurons. The "signal" is a real number, and the output of each neuron is computed by some non-linear function of the totality of its inputs, called the activation function.
The strength of the signal at each connection is determined by a weight, which adjusts as part of the training process. Groups of neurons are aggregated into layers. Each layer performs a transformation on its inputs. Signals travel from the first layer (the input layer) to the last layer (the output layer), typically passing through multiple intermediate layers (hidden layers). A network is typically called a deep neural network if it has at least two hidden layers. Deep neural networks are capable of learning sophisticated hierarchical representations.
Training neural networks is a compute-intensive process, accelerated by the use of graphics processing units (GPUs), and large datasets. File:Simplified neural network training example.svg|Simplified example of training a neural network in object detection: The network is trained by multiple images known to depict starfish and sea urchins, which are correlated with "nodes" representing visual features. Starfish match with a ringed texture and star outline, whereas most sea urchins match with a striped texture and oval shape.
File:Simplified neural network model example.svg|The resulting model contains weighted associations between visual features and output categories. Because one training image depicts a ring-textured sea urchin, the model also develops a weak association between ringed texture and sea urchin. File:Simplified neural network example.svg|Subsequent run of the model on an input image (left): The network correctly detects the starfish.
However, the weak association between ringed texture and sea urchin also gives a weak signal to the latter from one of two intermediate nodes. In addition, a shell not included in the training gives a weak signal for the oval shape, also resulting in a weak signal for the sea urchin output. These weak signals may result in a false positive result for sea urchin. In reality, such textures and outlines would not be represented by single nodes, but rather by associated weight patterns of multiple nodes.
Architectural innovations such as convolutional neural networks (CNNs) significantly improved performance in computer vision tasks, while recurrent neural networks (RNNs) enabled modeling of sequential data such as speech and time-series information. Transformer architectures introduced attention mechanisms that allow neural networks to model long-range dependencies in data and have been the basis of large language models. Artificial neural networks are used for myriad tasks including chatbots, large-scale text, image, and video generation, and robotics.
History
1960s and 1970s
Fundamental research was conducted on NNs in the 1960s and 1970s. The first working deep learning algorithm was the group method of data handling, a method to train arbitrarily deep neural networks, published by Alexey Ivakhnenko and Valentin Lapa in the Soviet Union (19651). They regarded it as a form of polynomial regression, generalizing Rosenblatt's perceptron. A 1971 paper described a deep network with eight layers trained by this method, training layer by layer via regression analysis.
Superfluous hidden units were pruned using a separate validation set. The activation functions of the nodes were Kolmogorov-Gabor polynomials, the first deep networks with multiplicative units or "gates". The first deep learning multilayer perceptron (MLP) trained by stochastic gradient descent was published in 19671 by Shun'ichi Amari. In computer experiments conducted by Amari's student S. Saito, a five layer MLP with two modifiable layers learned internal representations to classify non-linearily separable pattern classes.
Subsequent developments in hardware and hyperparameter tuning made end-to-end stochastic gradient descent the dominant technique for reducing loss (error). In 1969, Kunihiko Fukushima introduced the ReLU (rectified linear unit) activation function. ReLU is the most common activation function. Nevertheless, research stagnated in the United States after Minsky and Papert (1969), who emphasized that perceptrons were incapable of processing the exclusive-or circuit.
However, this insight was irrelevant for the deep networks of Ivakhnenko (1965) and Amari (1967). In 1976, transfer learning was introduced.
Elements
NNs began as an attempt to replicate the architecture of the brain in the digital realm. It immediately showed promise for handling non-linear relationships, but encountered obstacles. Models soon reoriented to applying mathematical insights to improve empirical results, at the expense of biological fidelity.
Learning paradigms
Machine learning has involved a variety of approaches to training models, including supervised learning, unsupervised learning, reinforcement learning, and self-supervised learning.
Unsupervised learning
In unsupervised learning, input data is given along with the cost function for the data \textstyle x and the output, without an "answer sheet". The cost function is dependent on the task (the model domain) and reflects a priori assumptions (implicit model properties, its parameters and the observed variables). For example, the model \textstyle f(x) = a treats \textstyle a as a constant and the cost \textstyle C=E[(x - f(x))^2] . Minimizing this cost produces a value of \textstyle a that is equal to the mean of the data.
The cost function can be much more complicated. Its form depends on the application: for example, in compression it could be related to the mutual information between \textstyle x and \textstyle f(x) , whereas in statistical modeling, it could be related to the posterior probability (in both examples, those quantities are maximized). Unsupervised learning is typically applied to estimation problems; applications include clustering, the estimation of statistical distributions, compression, and filtering.
Reinforcement learning
In applications such as playing video games, an actor takes a string of actions, receiving a generally unpredictable response from the environment (the game) after each one. The goal is to win the game (get the highest score). The cost is the inverse of the score. In reinforcement learning, the aim is to weight the network to increase the score. After each action the game generates an observation and an instantaneous cost, according to its rules. The rules and the long-term cost can only be estimated.
At any juncture, the agent decides whether to explore new actions to uncover their costs or to exploit prior learning to proceed more quickly. Formally, the environment is modeled as a Markov decision process (MDP) with states \textstyle {s_1,...,s_n}\in S and actions \textstyle {a_1,...,a_m} \in A .
Because the state transitions (policies) are not known, probability distributions are used instead: the instantaneous cost distribution \textstyle P(c_t|s_t) , the observation distribution \textstyle P(x_t|s_t) and the transition distribution \textstyle P(s_{t+1}|s_t, a_t) , while a policy is defined as the conditional distribution over actions given the observations. Taken together, the two define a Markov chain (MC). The aim is to discover the lowest-cost MC. NNs serve as the learning component.
Dynamic programming coupled with NNs (giving neurodynamic programming) has been applied to problems such as vehicle routing, video games, natural resource management and medicine because of NNs' ability to mitigate cost even when reducing the discretization grid density for numerically approximating control tasks.
Self-learning
Self-learning was introduced in 1982 along with a crossbar adaptive array (CAA) NN that could teach itself. It is a system with only one input, situation s, and only one output, action (or behavior) a. It has neither external advice input nor external reinforcement from the environment. The CAA computes, in a crossbar fashion, both decisions about actions and emotions (feelings) about encountered situations. The system is driven by the interaction between cognition and emotion.
Given the memory matrix, W =||w(a,s)||, the crossbar self-learning algorithm in each iteration performs the following computation: In situation s perform action a; Receive consequence situation s'; Compute emotion of being in consequence situation v(s'); Update crossbar memory w'(a,s) = w(a,s) + v(s'). The backpropagated value (secondary reinforcement) is the emotion toward the consequence situation.
The CAA exists in two environments, its behavioral environment, and its genetic environment, from which it receives initial emotions (only once) about to be encountered situations in the behavioral environment. Having received the genome vector (species vector) from the genetic environment, the CAA will learn a goal-seeking behavior, in the behavioral environment that contains both desirable and undesirable situations.
Stochastic neural network
Stochastic neural networks originating from Sherrington–Kirkpatrick models are a type of neural network built by introducing random variations into the network, either by giving neurons stochastic transfer functions, or by giving them stochastic weights. This makes them useful tools for optimization problems, since the random fluctuations help the network escape from local minima. Stochastic neural networks trained using a Bayesian approach are known as Bayesian neural networks.
Topological deep learning
Topological deep learning, introduced in 2017, integrates topology with deep neural networks to address high-order data. Initially rooted in algebraic topology, TDL evolved into a versatile framework incorporating tools from mathematical disciplines such as differential topology and geometric topology.
Other
In a Bayesian framework, a distribution over the set of allowed models is chosen to minimize cost. Evolutionary methods, gene expression programming, simulated annealing, expectation–maximization, non-parametric methods and particle swarm optimization are other learning algorithms. Convergent recursion is a learning algorithm for cerebellar model articulation controller (CMAC) neural networks.
Theoretical properties
Computational power
The multilayer perceptron is a universal function approximator, as proven by the universal approximation theorem. However, the proof does not specify the number of neurons required, the network topology, the weights or the learning parameters. A recurrent architecture with rational-valued weights (as opposed to full precision real number-valued weights) has the power of a universal Turing machine, using a finite number of neurons and linear connections. Further, the use of irrational values for weights results in a machine with super-Turing power.
Capacity
A model's "capacity" property is its ability to model any given function. It is related to the amount of information that can be stored in the network and to the notion of complexity. Information capacity and the VC dimension are two metrics. The capacity of a network of standard neurons (not convolutional) can be derived by four rules that derive from considering a neuron as an electrical element. The information capacity captures the functions that the network can model given any data as input. The VC dimension uses the principles of measure theory and finds the maximum capacity under optimal circumstances, given input data in a specific form. The VC Dimension for arbitrary inputs is half the information capacity of a perceptron. The VC Dimension for arbitrary points is sometimes referred to as Memory Capacity.
Convergence
Models may not consistently converge on a single solution, because the system may get stuck in a local minima. Alternatively, the optimization method used might not guarantee to converge should it begin far from any local minima. Thirdly, for sufficiently large data or parameters, some methods are impractically slow/expensive. Training may also cross some saddle point that may then prevent access to the solution.
When the width of a network approaches infinity, it is well described by its first order Taylor expansion throughout training, and so inherits the convergence behavior of affine models. When parameter numbers are small, NNs often fit target functions from low to high frequencies. This behavior is referred to as spectral bias, or the frequency principle. This phenomenon is the opposite of the behavior of some well-studied iterative numerical schemes such as the Jacobi method. Deeper neural networks are more biased towards low frequency functions.
Generalization and statistics
Applications that must generalize well to unseen examples must avoid over-training. This arises in convoluted or over-specified systems when the network capacity is much larger than needed. Two approaches address over-training. Cross-validation and similar techniques can check for over-training and select appropriate hyperparameters to minimize generalization error.
Regularization in a probabilistic (Bayesian) framework can be performed by selecting a larger prior probability over simpler models; but also in statistical learning theory, where the goal is to minimize two quantities: the 'empirical risk' and the 'structural risk', which roughly corresponds to the error over the training set and the predicted error in unseen data due to overfitting. Supervised neural networks that use a mean squared error (MSE) cost function can use formal statistical methods to determine the confidence of the trained model.
The MSE on a validation set can be used as an estimate for variance. This value can then be used to calculate the confidence interval of network output, assuming a normal distribution. A confidence analysis made this way is statistically valid as long as the output probability distribution stays the same and the network is not modified.
By adopting a softmax activation function, a generalization of the logistic function, on the output layer of the neural network (or a softmax component in a component-based network) for categorical target variables, the outputs can be interpreted as posterior probabilities. This is useful in classification as it gives a certainty measure. The softmax activation function is: y_i=\frac{e^{x_i}}{\sum_{j=1}^c e^{x_j}}
Issues
Theory
A key hypothesis of NNs is that they use statistical long-developed methods.
Hardware
Neural networks require enormous computing resources for training. While the brain requires only 20 watts of power, training commercial transformers requires (2026) data centers with hundreds of megawatts. From 1991 to 2015, computing power, especially as delivered by GPGPUs (on GPUs), increased around a million-fold, enabling the use of the backpropagation algorithm. Neuromorphic engineering or a physical neural network constructed non-von-Neumann chips to directly implement neural networks in circuitry. For example, Alphabet introduced custom chips (Tensor Processing Unit).
Gallery
File:Single_layer_ann.svg|alt=A single-layer feedforward artificial neural network. Arrows originating from x 2 {\displaystyle x_{2}} are omitted for clarity. There are p inputs to this network and q outputs. In this system, the value of the qth output, y q {\displaystyle y_{q}} , is calculated as y q = K ⋅ ( ∑ i ( x i w i q ) − b q ) . {\displaystyle y_{q}=K\cdot \left(\sum _{i}(x_{i}w_{iq})-b_{q}\right).}|A single-layer feedforward neural network. Arrows originating from x_2 are omitted for clarity. There are p inputs to this network and q outputs.
In this system, the value of the qth output, y_q , is calculated as y_q = K\cdot\left(\sum_i(x_i w_{iq})-b_q\right). File:Two_layer_ann.svg|A two-layer feedforward neural network File:Artificial_neural_network.svg|An neural network File:Ann_dependency_(graph).svg|An NN dependency graph File:Single-layer_feedforward_artificial_neural_network.png|A single-layer feedforward neural network with 4 inputs, 6 hidden nodes and 2 outputs. Given position state and direction, it outputs wheel based control values.
File:Two-layer_feedforward_artificial_neural_network.png|A two-layer feedforward neural network with 8 inputs, 2x8 hidden nodes and 2 outputs. Given position state, direction and other environment values, it outputs thruster based control values. File:Cmac.jpg|Parallel pipeline structure of CMAC neural network. This learning algorithm can converge in one step.
Notes
References
Bibliography
PDF created for National Science Foundation, Contract Number EET-8716324, and Defense Advanced Research Projects Agency (DOD), ARPA Order No. 4976 under Contract F33615-87-C-1499.
External links
A Brief Introduction to Neural Networks (D. Kriesel) – Illustrated, bilingual manuscript about neural networks; Topics so far: Perceptrons, Backpropagation, Radial Basis Functions, Recurrent Neural Networks, Self Organizing Maps, Hopfield Networks. Review of Neural Networks in Materials Science Artificial Neural Networks Tutorial in three languages (Univ. Politécnica de Madrid) Another introduction to NN Next Generation of Neural Networks – Google Tech Talks Performance of Neural Networks Neural Networks and Information
Text derived from https://en.wikipedia.org/w/index.php?oldid=1378632368 under CC BY-SA 4.0; modifications have been made.
References
- ↩Wikipedia “Neural network (machine learning)” publication date (×2) · confidence 0.95 revision URL