Rectified linear unit

Uses your system voice; voice and rate in settings.

Snapshot

≈ 10 min read

Names in other languages

No names in other languages are held for this entry yet.

Article bodies are English and are not machine-translated. A name in another language is not a translation of the article.

(green) functions near x = 0 In the context of artificial neural networks, the rectifier or ReLU (rectified linear unit) activation function is an activation function defined as the non-negative part of its argument, i.e., the ramp function: : \operatorname{ReLU}(x) = x^+ = \max(0, x) = \frac{x+|x|}{2} = \begin{cases} x & \text{if } x > 0, \\ 0 & x \le 0 \end{cases} where x is the input to a neuron. This is analogous to half-wave rectification in electrical engineering. ReLU is one of the most popular activation functions for artificial neural networks, and finds application in computer vision and speech recognition using deep neural nets and computational neuroscience.

(green) functions near x = 0 In the context of artificial neural networks, the rectifier or ReLU (rectified linear unit) activation function is an activation function defined as the non-negative part of its argument, i.e., the ramp function: \operatorname{ReLU}(x) = x^+ = \max(0, x) = \frac{x+|x|}{2} = \begin{cases} x & \text{if } x > 0, \\ 0 & x \le 0 \end{cases} where x is the input to a neuron. This is analogous to half-wave rectification in electrical engineering. ReLU is one of the most popular activation functions for artificial neural networks, and finds application in computer vision and speech recognition using deep neural nets and computational neuroscience. Nair and Hinton (2010) made a theoretical argument that the softplus activation function should be used, in that the softplus function numerically approximates the sum of an exponential number of linear models that share parameters. They then proposed ReLU as a good approximation to it. Specifically, they began by considering a single binary neuron in a Boltzmann machine that takes x as input, and produces 1 as output with probability \sigma(x) = \frac{1}{1 + e^{-x}} . They then considered extending its range of output by making infinitely many copies of it X_1, X_2, X_3, \dots , that all take the same input, offset by an amount 0.5, 1.5, 2.5, \dots , then their outputs are added together as \sum_{i=1}^\infty X_i . They then demonstrated that \sum_{i=1}^\infty X_i is approximately equal to \mathcal N(\log(1+e^x), \sigma(x)) , which is also approximately equal to \operatorname{ReLU}(\mathcal N( x, \sigma(x))) , where \mathcal N stands for the gaussian distribution. They also argued for another reason for using ReLU: that it allows "intensity equivariance" in image recognition. That is, multiplying input image by a constant k multiplies the output also. In contrast, this is false for other activation functions like sigmoid or tanh. They found that ReLU activation allowed good empirical performance in restricted Boltzmann machines. Glorot et al (2011) argued that ReLU has the following advantages over sigmoid or tanh: ReLU is more similar to biological neurons' responses in their main operating regime. ReLU avoids vanishing gradients. ReLU is cheaper to compute. ReLU creates sparse representation naturally, because many hidden units output exactly zero for a given input. They also found empirically that deep networks trained with ReLU can achieve strong performance without unsupervised pre-training, especially on large, purely supervised tasks. In 2017, the rectified linear function became a central component of the transformer architecture introduced in the Vaswani et al paper "Attention Is All You Need". Within every transformer layer, ReLU is utilized in the position-wise feed-forward networks (FFN), defined by Equation 2 of their paper: \operatorname{FFN}(x) = \max(0, xW_1 + b_1)W_2 + b_2 This equation is foundational to the model's capacity; while the attention mechanism determines the relationships between tokens, the ReLU-based FFN performs the majority of the numerical computation and houses the bulk of the model's parameters. The efficiency and scalability of this rectified framework triggered a global technological revolution, enabling the development of Large Language Models that have had a profound economic impact. The industrial response to this architecture—including the massive expansion of AI-specific hardware and the birth of the generative AI sector—has positioned the Transformer as a cornerstone of 21st-century infrastructure. During the post 2017 period of rapid AI advancement, the rectified linear unit function has been key to achieving increased model performance and scaling due to the fact that it zeros out responses that are immaterial for a given stimuli, preventing them from accumulating in massive scale models. It is the complete silencing of the parts of the model found to be stimuli-irrelevant during learning that allows for scaling. As the stimuli-irrelevant proportion of the model becomes more massive, these highly numerous connections within the model would inevitably accumulate during scaling no matter how small each individual response is. Therefore, the rectified linear unit function, with its absolute zeroing property, enabled the scaling to hundred billion parameter models and beyond. Early Transformer scaling giants like GPT-3 (2020) and Falcon-180B (2023) relied on the rectified linear unit function explicitly, while successors such as GPT-4 (2023) and Llama 3 (2024) utilized smoother variants like GELU or SwiGLU. These variants were used to improve training stability while fundamentally preserving the rectified principle of zeroing low responses. At the centre of modern artificial intelligence ReLU and its variants maintain absolute zero response across the bulk of the model at any one time, while maintaining approximately linear responses for stimuli-relevant connections enabling high performance on each specific cognitive task. This feature of activation sparsity has been critical for massive scaling and performance gains of AI models right up to the present day. \max(0, ax) = a \max(0, x) \text{ for } a \geq 0 . f(x) = \begin{cases} x & x > 0, \\ \alpha x & x \le 0, \end{cases} \qquad f'(x) = \begin{cases} 1 & x > 0, \\ \alpha & x \le 0. \end{cases} The same function can also be expressed without the piecewise notation as: f(x) = \frac{1+\alpha}{2} x+\frac{1-\alpha}{2} |x| Parametric ReLU (PReLU, 2016) takes this idea further by making \alpha a learnable parameter along with the other network parameters. Note that for \alpha \le 1 , this is equivalent to f(x) = \max(x, \alpha x) and thus has a relation to "maxout" networks. Concatenated ReLU (CReLU, 2016) preserves positive and negative phase information by returning two values: f(x) = [\operatorname{ReLU}(x), \operatorname{ReLU}(-x)].

Softplus

A smooth approximation to the rectifier is the analytic function f(x) = \ln(1 + e^x),\qquad f'(x) = \frac{e^{x}}{1 + e^{x}} = \frac{1}{1 + e^{-x}} which is called the softplus (2000) or SmoothReLU function. For large negative x it is roughly \ln 1 , so just above 0, while for large positive x it is roughly \ln(e^x) , so just above x . This function can be approximated as: \ln\left(1 + e^x \right) \approx \begin{cases} \ln2, & x=0,\\[6pt] \frac x {1-e^{-x/\ln2}}, & x\neq 0 \end{cases} By making the change of variables x = y\ln(2) , this is equivalent to \log_2(1 + 2^y) \approx \begin{cases} 1,& y=0,\\[6pt] \frac{y}{1-e^{-y}}, & y\neq 0\end{cases} A sharpness parameter k may be included: f(x) = \frac{\ln(1 + e^{kx})} k, \qquad f'(x) = \frac{e^{kx}}{1 + e^{kx}} = \frac{1}{1 + e^{-kx}} The derivative of softplus is the logistic function. This in turn can be viewed as a smooth approximation of the derivative of the rectifier, the Heaviside step function. The multivariable generalization of single-variable softplus is the LogSumExp with the first argument set to zero: \operatorname{LSE_0}^+(x_1, \dots, x_n) := \operatorname{LSE}(0, x_1, \dots, x_n) = \ln(1 + e^{x_1} + \cdots + e^{x_n}) The LogSumExp function is \operatorname{LSE}(x_1, \dots, x_n) = \ln(e^{x_1} + \cdots + e^{x_n}) and its gradient is the softmax; the softmax with the first argument set to zero is the multivariable generalization of the logistic function. Both LogSumExp and softmax are used in machine learning. f(x) = \begin{cases} x & x > 0, \\ \alpha \left(e^x - 1\right) & x \le 0 \end{cases} \qquad f'(x) = \begin{cases} 1 & x > 0, \\ \alpha e^x & x \le 0 \end{cases} In these formulas, \alpha is a hyperparameter to be tuned with the constraint \alpha \geq 0 . Given the same interpretation of \alpha , ELU can be viewed as a smoothed version of a shifted ReLU (SReLU), which has the form f(x) = \max(- \alpha, x) . f(x) = x \Phi(x) = x\cdot\frac{1}{2}\left[1+\operatorname{erf}\left(x/\sqrt{2}\right)\right], f'(x) = x \Phi'(x) + \Phi(x) where \Phi(x) = P(X \leqslant x) is the cumulative distribution function of the standard normal distribution and \operatorname{erf}(z) is the error function. This activation function is illustrated in the figure at the start of this article. It has a "bump" with negative derivative to the left of x \Phi(x) : f(x)\approx \frac{1}{2}x\left(1+\tanh\left[\sqrt{2/\pi}\left(x+0.044715x^3\right)\right]\right) The second, less-precise approximation uses the sigmoid (logistic) function as f(x)\approx x\cdot\operatorname{sigmoid}(1.702x) , whose formula resembles SiLU (see below). f(x) = x \cdot \operatorname{sigmoid}(x), f'(x) = x \cdot \operatorname{sigmoid}'(x) + \operatorname{sigmoid}(x) It is cheaper to calculate than GELU. It also has a "bump". f(x) = x \tanh\big(\operatorname{softplus}(x)\big), where \tanh(x) is the hyperbolic tangent, and \operatorname{softplus}(x) is the softplus function. Mish was obtained by experimenting with functions similar to Swish (SiLU, see above). It is non-monotonic (has a "bump") like Swish. The main new feature is that it exhibits a "self-regularizing" behavior attributed to a term in its first derivative. f(x) = \frac{x + \sqrt{x^2 + b}}{2} where b \geq 0 is a hyperparameter that determines the "size" of the curved region near x = 0 . (For example, letting b = 0 yields ReLU, and letting b = 4 yields the metallic mean function.) Squareplus shares many properties with softplus: It is monotonic, strictly positive, approaches 0 as x \to -\infty , approaches the identity as x \to +\infty , and is C^\infty smooth. However, squareplus can be computed using only algebraic functions, making it well-suited for settings where computational resources or instruction sets are limited. Additionally, squareplus requires no special consideration to ensure numerical stability when x is large. f(x) = \begin{cases} x & x > x_c, \\ (e^{ax} - 1)/b & x \le x_c \end{cases} \qquad f'(x) = \begin{cases} 1 & x > x_c, \\ (a / b) e^{ax} & x \le x_c \end{cases} In these formulas, a , b and x_c are hyperparameter values which could be set as default constraints a = 1 , b = 2 and x_c = 1.25643 , as done in the original work.

See also

Softmax function Sigmoid function Tobit model Layer (deep learning)

References

Text derived from https://en.wikipedia.org/w/index.php?oldid=1373687081 under CC BY-SA 4.0; modifications have been made.

Sources cited in this revision

19 named source(s) extracted from the cited revision. A source with no author is shown by its publication; nothing is invented.