Pooling layer
In neural networks, a pooling layer is a kind of network layer that downsamples and aggregates information that is dispersed among many vectors into fewer vectors.
≈ 6 min read
Names in other languages
No names in other languages are held for this entry yet.
Article bodies are English and are not machine-translated. A name in another language is not a translation of the article.
In neural networks, a pooling layer is a kind of network layer that downsamples and aggregates information that is dispersed among many vectors into fewer vectors. It has several uses.
It removes redundant information, thus reducing the amount of computation and memory required, which makes the model more robust to small variations in the input; and it increases the receptive field of neurons in later layers in the network. Define \mathrm{MaxPool}(x | f, s)_{0, 0, 0} = \max(x_{0:f-1, 0:f-1, 0}) where 0:f-1 means the range 0, 1, \dots, f-1 .
Note that we need to avoid the off-by-one error. The next input is \mathrm{MaxPool}(x | f, s)_{1, 0, 0} = \max(x_{s:s + f-1, 0:f-1, 0}) and so on.
The receptive field of y_{i, j, c} is x_{is + f-1, js + f-1, c} , so in general, \mathrm{MaxPool}(x | f, s)_{i,j,c} = \mathrm{max}(x_{is: is+f-1, js: js + f-1, c}) If the horizontal and vertical filter size and strides differ, then in general, \mathrm{MaxPool}(x | f, s)_{i,j,c} = \mathrm{max}(x_{is_H: is_H+f_H-1, js_W: js_W + f_W-1, c}) More succinctly, we can write y_k = \max(\{x_{k'} | k' \text{ in the receptive field of }k\}) . If H is not expressible as ks + f where k is an integer, then for computing the entries of the output tensor on the boundaries, max pooling would attempt to take as inputs variables off the tensor.
In this case, how those non-existent variables are handled depends on the padding conditions, illustrated on the right. Global Max Pooling (GMP) is a specific kind of max pooling where the output tensor has shape \R^{C} and the receptive field of y_c is all of x_{0:H, 0:W, c} . That is, it takes the maximum over each entire channel.
It is often used just before the final fully connected layers in a CNN classification head. Average pooling (AvgPool) is similarly defined \mathrm{AvgPool}(x | f, s)_{i,j,c} = \mathrm{average}(x_{is: is+f-1, js: js + f-1, c}) = \frac{1}{f^2} \sum_{k \in is: is+f-1}\sum_{l \in js: js + f-1} x_{k, l, c} Global Average Pooling (GAP) is defined similarly to GMP.
It was first proposed in Network-in-Network. Similarly to GMP, it is often used just before the final fully connected layers in a CNN classification head. Mixed Pooling is a linear sum of max pooling and average pooling.
That is, \mathrm{MixedPool}(x | f, s, w) = w \mathrm{MaxPool}(x | f, s) + (1-w)\mathrm{AvgPool}(x | f, s) where w \in [0, 1] is either a hyperparameter, a learnable parameter, or randomly sampled anew every time. Lp Pooling is similar to average pooling, but uses Lp norm average instead of average: y_k = \left(\frac 1N \sum_{k' \text{ in the receptive field of } k} |x_{k'}|^p\right)^{1/p} where N is the size of receptive field, and p \geq 1 is a hyperparameter.
If all activations are non-negative, then average pooling is the case of p = 1 , and max pooling is the case of p \to \infty . Square-root pooling is the case of p = 2 . Stochastic pooling samples a random activation x_{k'} from the receptive field with probability \frac{x_{k'}}{\sum_{k} x_{k}} .
It is the same as average pooling in expectation. Softmax pooling is like max pooling, but uses softmax, i.e. \frac{\sum_{k'} e^{\beta x_{k'}}x_{k'}}{\sum_{k} e^{\beta x_{k}}} where \beta > 0 .
Average pooling is the case of \beta \downarrow 0 , and max pooling is the case of \beta \uparrow \infty Local Importance-based Pooling generalizes softmax pooling by \frac{\sum_{k'} e^{ g(x_{k'})}x_{k'}}{\sum_{k} e^{ g(x_{k})}} where g is a learnable function. Covariance pooling computes the covariance matrix of the vectors \{x_{k, l, 0:C-1}\}_{k \in is:is + f-1, l \in js:js + f-1} which is then flattened to a C^2 -dimensional vector y_{i,j, 0:C^2-1} .
Global covariance pooling is used similarly to global max pooling. As average pooling computes the average, which is a first-degree statistic, and covariance is a second-degree statistic, covariance pooling is also called "second-order pooling". It can be generalized to higher-order poolings.
Blur Pooling means applying a blurring method before downsampling. For example, the Rect-2 blur pooling means taking an average pooling at f = 2, s = 1 , then taking every second pixel (identity with s = 2 ). Multihead attention pooling (MAP) applies a multi headed attention block to pooling.
Specifically, it takes as input a list of vectors x_1, x_2, \dots, x_n , which might be thought of as the output vectors of a layer of a ViT. It then applies a feedforward layer \mathrm{FFN} on each vector, resulting in a matrix V = [\mathrm{FFN}(v_1), \dots, \mathrm{FFN}(v_n)] . This is then sent to a multi-headed attention, resulting in \mathrm{MultiheadedAttention}(Q, V, V) , where Q is a matrix of trainable parameters.
This was first proposed in the Set Transformer architecture. Later papers demonstrated that GAP and MAP both perform better than BERT-like pooling. \mathbf{y} = \frac{\mathbf{X}\mathbf{p}}{\Vert\mathbf{p}\Vert} where \mathbf{p} is a learnable projection vector. The projection vector \mathbf{p} computes a scalar projection value for each graph node.
The top-k pooling layer can then be formalised as follows: \mathbf{X}' = (\mathbf{X} \odot \text{sigmoid}(\mathbf{y}))_{\mathbf{i}} \mathbf{A}' = \mathbf{A}_{\mathbf{i}, \mathbf{i}} where \mathbf{i} = \text{top}_k(\mathbf{y}) is the subset of nodes with the top-k highest projection scores, \odot denotes element-wise matrix multiplication, and \text{sigmoid}(\cdot) is the sigmoid function. In other words, the nodes with the top-k highest projection scores are retained in the new adjacency matrix \mathbf{A}' .
The \text{sigmoid}(\cdot) operation makes the projection vector \mathbf{p} trainable by backpropagation, which otherwise would produce discrete outputs. \mathbf{y} = \text{GNN}(\mathbf{X}, \mathbf{A}) where \text{GNN} is a generic permutation equivariant GNN layer (e.g., GCN, GAT, MPNN). The Self-attention pooling layer can then be formalised as follows: \mathbf{X}' = (\mathbf{X} \odot \mathbf{y})_{\mathbf{i}} \mathbf{A}' = \mathbf{A}_{\mathbf{i}, \mathbf{i}} where \mathbf{i} = \text{top}_k(\mathbf{y}) is the subset of nodes with the top-k highest projection scores, \odot denotes element-wise matrix multiplication.
The self-attention pooling layer can be seen as an extension of the top-k pooling layer. Differently from top-k pooling, the self-attention scores computed in self-attention pooling account both for the graph features and the graph topology.
Text derived from https://en.wikipedia.org/w/index.php?oldid=1361286348 under CC BY-SA 4.0; modifications have been made.