Multilayer perceptron

In deep learning, a multilayer perceptron (MLP) is a kind of modern feedforward neural network consisting of fully connected neurons with nonlinear activation functions, organized in layers, notable for being able to distinguish data that…

Uses your system voice; voice and rate in settings.

Snapshot

≈ 5 min read

Names in other languages

No names in other languages are held for this entry yet.

Article bodies are English and are not machine-translated. A name in another language is not a translation of the article.

In deep learning, a multilayer perceptron (MLP) is a kind of modern feedforward neural network consisting of fully connected neurons with nonlinear activation functions, organized in layers, notable for being able to distinguish data that is not linearly separable. Modern neural networks are trained using backpropagation and are colloquially referred to as "vanilla" networks.

MLPs grew out of an effort to improve on single-layer perceptrons, which could only be applied to linearly separable data. A perceptron traditionally used a Heaviside step function as its nonlinear activation function. However, the backpropagation algorithm requires that modern MLPs use continuous activation functions such as sigmoid or ReLU.

Multilayer perceptrons form the basis of deep learning, and are applicable across a vast set of diverse domains. u = \sum_{i=1}^{n} w_i x_i + b = \mathbf{w}^\mathsf{T}\mathbf{x} + b Activation function: The weighted sum is passed through a nonlinear function g(u) to produce the neuron's output: y = g(u) = g\left(\sum_{i=1}^{n} w_i x_i + b\right) The bias term can be viewed as an adjustable threshold for neuron activation. \mathbf{h} = \sigma(\mathbf{W}\mathbf{x} + \mathbf{b}), where \mathbf{W} is the weight matrix, \mathbf{b} is the bias vector, and \sigma is the activation function applied element-wise. The number of parameters in a fully connected layer equals the product of the input and output dimensions, plus the bias terms.

A layer connecting n inputs to m outputs contains n \times m + m learnable parameters. This quadratic scaling means MLPs require substantially more parameters than architectures with sparse connectivity, making them less practical for high-dimensional inputs such as images. Sigmoid, \sigma(x) = 1/(1+e^{-x}) : Historically common, it maps inputs to the range (0, 1) .

It is often used in the output layer for binary classification but is less commonly used in hidden layers due to the vanishing gradient problem. Hyperbolic Tangent, \tanh x : Similar to the sigmoid but maps inputs to (-1, 1) . Its zero-centered output often leads to faster convergence than the sigmoid during training.

Rectified linear unit (ReLU), f(x) = \max(0, x) : A common choice for hidden layers. Because its gradient is exactly 1 for positive inputs (and 0 otherwise), ReLU avoids the saturation problem and enables training of deeper networks. The related softplus function provides a smooth approximation.

Softmax: Typically found in the output layer for multi-class classification. It generalizes the logistic function to multiple dimensions, normalizing a vector of real values into a probability distribution where all values sum to 1. y(v_i) = \tanh(v_i) ~~ \textrm{and} ~~ y(v_i) = (1+e^{-v_i})^{-1} .

The first is a hyperbolic tangent that ranges from −1 to 1, while the other is the logistic function, which is similar in shape but ranges from 0 to 1. Here y_i is the output of the i th node (neuron) and v_i is the weighted sum of the input connections. Alternative activation functions have been proposed, including the rectifier and softplus functions.

More specialized activation functions include radial basis functions (used in radial basis networks, another class of supervised neural network models). In recent developments of deep learning the rectified linear unit (ReLU) is more frequently used as one of the possible ways to overcome the numerical problems related to the sigmoids.

We can represent the degree of error in an output node j in the n th data point (training example) by e_j(n)=d_j(n)-y_j(n) , where d_j(n) is the desired target value for n th data point at node j , and y_j(n) is the value produced by the perceptron at node j when the n th data point is given as an input. The node weights can then be adjusted based on corrections that minimize the error in the entire output for the n th data point, given by \mathcal{E}(n)=\frac{1}{2}\sum_{\text{output node }j} e_j^2(n) . Using gradient descent, the change in each weight w_{ij} is \Delta w_{ji} (n) = -\eta\frac{\partial\mathcal{E}(n)}{\partial v_j(n)} y_i(n) where y_i(n) is the output of the previous neuron i , and \eta is the learning rate, which is selected to ensure that the weights quickly converge to a response, without oscillations.

In the previous expression, \frac{\partial\mathcal{E}(n)}{\partial v_j(n)} denotes the partial derivate of the error \mathcal{E}(n) according to the weighted sum v_j(n) of the input connections of neuron i . The derivative to be calculated depends on the induced local field v_j , which itself varies. It is easy to prove that for an output node this derivative can be simplified to -\frac{\partial\mathcal{E}(n)}{\partial v_j(n)} = e_j(n)\phi^\prime (v_j(n)) where \phi^\prime is the derivative of the activation function described above, which itself does not vary.

The analysis is more difficult for the change in weights to a hidden node, but it can be shown that the relevant derivative is -\frac{\partial\mathcal{E}(n)}{\partial v_j(n)} = \phi^\prime (v_j(n))\sum_k -\frac{\partial\mathcal{E}(n)}{\partial v_k(n)} w_{kj}(n) . This depends on the change in weights of the k th nodes, which represent the output layer. So to change the hidden layer weights, the output layer weights change according to the derivative of the activation function, and so this algorithm represents a backpropagation of the activation function.

Weka: Open source data mining software with multilayer perceptron implementation. Neuroph Studio documentation, implements this algorithm and a few others. de:Perzeptron#Mehrlagiges Perzeptron

Text derived from https://en.wikipedia.org/w/index.php?oldid=1378403252 under CC BY-SA 4.0; modifications have been made.

Sources cited in this revision

15 named source(s) extracted from the cited revision. A source with no author is shown by its publication; nothing is invented.