Normalization (machine learning)

In machine learning, normalization is a statistical technique with various applications. There are two main forms of normalization, namely data normalization and activation normalization. Data normalization (or feature scaling) includes methods that rescale input data so that the features have the same range, mean, variance, or other statistical properties. For instance, a popular choice of feature scaling method is min-max normalization, where each feature is transformed to have the same range (typically [0,1] or [-1,1] ). This solves the problem of different features having vastly different scales, for example if one feature is measured in kilometers and another in nanometers. Activation normalization, on the other hand, is specific to deep learning, and includes methods that rescale the activation of hidden neurons inside neural networks. Normalization is often used to: * increase the speed of training convergence, * reduce sensitivity to variations and feature scales in input data, * reduce overfitting, * and produce better model generalization to unseen data. Normalization techniques are often theoretically justified as reducing covariance shift, smoothing optimization landscapes, and increasing regularization, though they are mainly justified by empirical success.

In machine learning, normalization is a statistical technique with various applications. There are two main forms of normalization, namely data normalization and activation normalization. Data normalization (or feature scaling) includes methods that rescale input data so that the features have the same range, mean, variance, or other statistical properties. For instance, a popular choice of feature scaling method is min-max normalization, where each feature is transformed to have the same range (typically [0,1] or [-1,1] ). This solves the problem of different features having vastly different scales, for example if one feature is measured in kilometers and another in nanometers. Activation normalization, on the other hand, is specific to deep learning, and includes methods that rescale the activation of hidden neurons inside neural networks. Normalization is often used to: increase the speed of training convergence, reduce sensitivity to variations and feature scales in input data, reduce overfitting, and produce better model generalization to unseen data. Normalization techniques are often theoretically justified as reducing covariance shift, smoothing optimization landscapes, and increasing regularization, though they are mainly justified by empirical success. x^{(0)} \mapsto x^{(1)} \mapsto x^{(2)} \mapsto \cdots where each network module can be a linear transform, a nonlinear activation function, a convolution, etc. x^{(0)} is the input vector, x^{(1)} is the output vector from the first module, etc. BatchNorm is a module that can be inserted at any point in the feedforward network. For example, suppose it is inserted just after x^{(l)} , then the network would operate accordingly: \cdots \mapsto x^{(l)} \mapsto \mathrm{BN}(x^{(l)}) \mapsto x^{(l+1)} \mapsto \cdots The BatchNorm module does not operate over individual inputs. Instead, it must operate over one batch of inputs at a time. Concretely, suppose we have a batch of inputs x^{(0)}_{(1)}, x^{(0)}_{(2)}, \dots, x^{(0)}_{(B)} , fed all at once into the network. We would obtain in the middle of the network some vectors: x^{(l)}_{(1)}, x^{(l)}_{(2)}, \dots, x^{(l)}_{(B)} The BatchNorm module computes the coordinate-wise mean and variance of these vectors: \begin{aligned} \mu^{(l)}_i &= \frac 1B \sum_{b=1}^B x^{(l)}_{(b), i} \\ (\sigma^{(l)}_i)^2 &= \frac{1}{B} \sum_{b=1}^B (x_{(b),i}^{(l)} - \mu_i^{(l)})^2 \end{aligned} where i indexes the coordinates of the vectors, and b indexes the elements of the batch. In other words, we are considering the i -th coordinate of each vector in the batch, and computing the mean and variance of these numbers. It then normalizes each coordinate to have zero mean and unit variance: \hat{x}^{(l)}_{(b), i} = \frac{x^{(l)}_{(b), i} - \mu^{(l)}_i}{\sqrt{(\sigma^{(l)}_i)^2 + \epsilon}} The \epsilon is a small positive constant such as 10^{-9} added to the variance for numerical stability, to avoid division by zero. Finally, it applies a linear transformation: y^{(l)}_{(b), i} = \gamma_i \hat{x}^{(l)}_{(b), i} + \beta_i Here, \gamma and \beta are parameters inside the BatchNorm module. They are learnable parameters, typically trained by gradient descent. The following is a Python implementation of BatchNorm: import numpy as np def batchnorm(x, gamma, beta, epsilon=1e-9): # Mean and variance of each feature mu = np.mean(x, axis=0) # shape (N,) var = np.var(x, axis=0) # shape (N,) # Normalize the activations x_hat = (x - mu) / np.sqrt(var + epsilon) # shape (B, N) # Apply the linear transform y = gamma * x_hat + beta # shape (B, N) return y x^{(l)}_{h, w, c} = \sum_{h', w', c'} K^{(l)}_{h'-h, w'-w, c, c'} x_{h', w', c'}^{(l-1)} + b^{(l)}_c where: x^{(l)}_{h, w, c} is the activation of the neuron at position (h, w) in the c -th channel of the l -th layer. K^{(l)}_{\Delta h, \Delta w, c, c'} is a kernel tensor. Each channel c corresponds to a kernel K^{(l)}_{h'-h, w'-w, c, c'} , with indices \Delta h, \Delta w, c' . b^{(l)}_c is the bias term for the c -th channel of the l -th layer. In order to preserve the translational invariance, BatchNorm treats all outputs from the same kernel in the same batch as more data in a batch. That is, it is applied once per kernel c (equivalently, once per channel c ), not per activation x^{(l+1)}_{h, w, c} : \begin{aligned} \mu^{(l)}_c &= \frac{1}{BHW} \sum_{b=1}^B \sum_{h=1}^H \sum_{w=1}^W x^{(l)}_{(b), h, w, c} \\ (\sigma^{(l)}_c)^2 &= \frac{1}{BHW} \sum_{b=1}^B \sum_{h=1}^H \sum_{w=1}^W (x_{(b), h, w, c}^{(l)} - \mu_c^{(l)})^2 \end{aligned} where B is the batch size, H is the height of the feature map, and W is the width of the feature map. That is, even though there are only B data points in a batch, all BHW outputs from the kernel in this batch are treated equally. Subsequently, normalization and the linear transform is also done per kernel: \begin{aligned} \hat{x}^{(l)}_{(b), h, w, c} &= \frac{x^{(l)}_{(b), h, w, c} - \mu^{(l)}_c}{\sqrt{(\sigma^{(l)}_c)^2 + \epsilon}} \\ y^{(l)}_{(b), h, w, c} &= \gamma_c \hat{x}^{(l)}_{(b), h, w, c} + \beta_c \end{aligned} Similar considerations apply for BatchNorm for n-dimensional convolutions. The following is a Python implementation of BatchNorm for 2D convolutions: import numpy as np def batchnorm_cnn(x, gamma, beta, epsilon=1e-9): # Calculate the mean and variance for each channel. mean = np.mean(x, axis=(0, 1, 2), keepdims=True) var = np.var(x, axis=(0, 1, 2), keepdims=True) # Normalize the input tensor. x_hat = (x - mean) / np.sqrt(var + epsilon) # Scale and shift the normalized tensor. y = gamma * x_hat + beta return y For multilayered recurrent neural networks (RNN), BatchNorm is usually applied only for the input-to-hidden part, not the hidden-to-hidden part. Let the hidden state of the l -th layer at time t be h_t^{(l)} . The standard RNN, without normalization, satisfies h^{(l)}_t = \phi(W^{(l)} h_t^{l-1} + U^{(l)} h_{t-1}^{l} + b^{(l)}) where W^{(l)}, U^{(l)}, b^{(l)} are weights and biases, and \phi is the activation function. Applying BatchNorm, this becomes h^{(l)}_t = \phi(\mathrm{BN}(W^{(l)} h_t^{l-1}) + U^{(l)} h_{t-1}^{l}) There are two possible ways to define what a "batch" is in BatchNorm for RNNs: frame-wise and sequence-wise. Concretely, consider applying an RNN to process a batch of sentences. Let h_{b, t}^{(l)} be the hidden state of the l -th layer for the t -th token of the b -th input sentence. Then frame-wise BatchNorm means normalizing over b : \begin{aligned} \mu_t^{(l)} &= \frac{1}{B} \sum_{b=1}^B h_{i,t}^{(l)} \\ (\sigma_t^{(l)})^2 &= \frac{1}{B} \sum_{b=1}^B (h_t^{(l)} - \mu_t^{(l)})^2 \end{aligned} and sequence-wise means normalizing over (b, t) : \begin{aligned} \mu^{(l)} &= \frac{1}{BT} \sum_{b=1}^B\sum_{t=1}^T h_{i,t}^{(l)} \\ (\sigma^{(l)})^2 &= \frac{1}{BT} \sum_{b=1}^B\sum_{t=1}^T (h_t^{(l)} - \mu^{(l)})^2 \end{aligned} Frame-wise BatchNorm is suited for causal tasks such as next-character prediction, where future frames are unavailable, forcing normalization per frame. Sequence-wise BatchNorm is suited for tasks such as speech recognition, where the entire sequences are available, but with variable lengths. In a batch, the smaller sequences are padded with zeroes to match the size of the longest sequence of the batch. In such setups, frame-wise is not recommended, because the number of unpadded frames decreases along the time axis, leading to increasingly poorer statistics estimates. It is also possible to apply BatchNorm to LSTMs. \begin{aligned} \mu &= \alpha E[x] + (1 - \alpha) \mu_{x, \text{ train}} \\ \sigma^2 &= (\alpha E[x]^2 + (1 - \alpha) \mu_{x^2, \text{ train}}) - \mu^2 \end{aligned} where \alpha is a hyperparameter to be optimized on a validation set. Other works attempt to eliminate BatchNorm, such as the Normalizer-Free ResNet. \hat{x_i} = \frac{x_i - \mu}{\sqrt{\sigma^2 + \epsilon}}, \quad y_i = \gamma_i \hat{x_i} + \beta_i where: \mu = \frac 1D \sum_{i=1}^D x_i, \quad \sigma^2 = \frac 1D \sum_{i=1}^D (x_i - \mu)^2 and the index i ranges over the neurons in that layer. \begin{aligned} \mu^{(l)} &= \frac{1}{HWC} \sum_{h=1}^H \sum_{w=1}^W\sum_{c=1}^C x^{(l)}_{h, w, c} \\ (\sigma^{(l)})^2 &= \frac{1}{HWC} \sum_{h=1}^H \sum_{w=1}^W\sum_{c=1}^C (x_{h, w, c}^{(l)} - \mu^{(l)})^2 \\ \hat{x}^{(l)}_{h,w,c} &= \frac{\hat{x}^{(l)}_{h,w,c} - \mu^{(l)}}{\sqrt{(\sigma^{(l)})^2 + \epsilon}} \\ y^{(l)}_{h,w,c} &= \gamma^{(l)} \hat{x}^{(l)}_{h,w,c} + \beta^{(l)} \end{aligned} Notice that the batch index b is removed, while the channel index c is added. In recurrent neural networks and transformers, LayerNorm is applied individually to each timestep. For example, if the hidden vector in an RNN at timestep t is x^{(t)} \in \mathbb{R}^{D} , where D is the dimension of the hidden vector, then LayerNorm will be applied with: \hat{x_{i}}^{(t)} = \frac{x_i^{(t)} - \mu^{(t)}}{\sqrt{(\sigma^{(t)})^2 + \epsilon}}, \quad y_i^{(t)} = \gamma_i \hat{x_i}^{(t)} + \beta_i where: \mu^{(t)} = \frac 1D \sum_{i=1}^D x_i^{(t)}, \quad (\sigma^{(t)})^2 = \frac 1D \sum_{i=1}^D (x_i^{(t)} - \mu^{(t)})^2 \hat{x_i} = \frac{x_i}{\sqrt{\frac 1D \sum_{j=1}^D x_j^2}}, \quad y_i = \gamma \hat{x_i} + \beta Essentially, it is LayerNorm where we enforce \mu, \epsilon = 0 . It is also called L2 normalization. It is a special case of Lp normalization, or power normalization: \hat{x_i} = \frac{x_i}{\left(\frac 1D \sum_{j=1}^D |x_j|^p \right)^{1/p}}, \quad y_i = \gamma \hat{x_i} + \beta where p > 0 is a constant. b_{x, y}^i=\frac{a_{x, y}^i}{\left(k+\alpha \sum_{j=\max (0, i-n / 2)}^{\min (N-1, i+n / 2)}\left(a_{x, y}^j\right)^2\right)^\beta} where a_{x,y}^i is the activation of the neuron at location (x,y) and channel i . I.e., each pixel in a channel is suppressed by the activations of the same pixel in its adjacent channels. k, n, \alpha, \beta are hyperparameters picked by using a validation set. It was a variant of the earlier local contrast normalization. b_{x, y}^i=\frac{a_{x, y}^i}{\left(k+\alpha \sum_{j=\max (0, i-n / 2)}^{\min (N-1, i+n / 2)}\left(a_{x, y}^j - \bar a_{x, y}^j\right)^2\right)^\beta} where \bar a_{x, y}^j is the average activation in a small window centered on location (x,y) and channel i . The hyperparameters k, n, \alpha, \beta , and the size of the small window, are picked by using a validation set. Similar methods were called divisive normalization, as they divide activations by a number depending on the activations. They were originally inspired by biology, where it was used to explain nonlinear responses of cortical neurons and nonlinear masking in visual perception. Both kinds of local normalization were obviated by batch normalization, which is a more global form of normalization. Response normalization reappeared in ConvNeXT-2 as global response normalization. \begin{aligned} \mu^{(l)}_c &= \frac{1}{HW} \sum_{h=1}^H \sum_{w=1}^Wx^{(l)}_{h, w, c} \\ (\sigma^{(l)}_c)^2 &= \frac{1}{HW} \sum_{h=1}^H \sum_{w=1}^W (x_{h, w, c}^{(l)} - \mu^{(l)}_c)^2 \\ \hat{x}^{(l)}_{h,w,c} &= \frac{\hat{x}^{(l)}_{h,w,c} - \mu^{(l)}_c}{\sqrt{(\sigma^{(l)}_c)^2 + \epsilon}} \\ y^{(l)}_{h,w,c} &= \gamma^{(l)}_c \hat{x}^{(l)}_{h,w,c} + \beta^{(l)}_c \end{aligned} \begin{aligned} y^{(l), \text{ content}}_{h,w,c} &= \sigma^{(l), \text{ style}}_c \left( \frac{x^{(l), \text{ content}}_{h,w,c} - \mu^{(l), \text{ content}}_c}{\sqrt{(\sigma^{(l), \text{ content}}_c)^2 + \epsilon}} \right) + \mu^{(l), \text{ style}}_c \end{aligned}

Text derived from https://en.wikipedia.org/w/index.php?oldid=1340395252 under CC BY-SA 4.0; modifications have been made.