Transformer (deep learning)
≈ 21 min read
Names in other languages
No names in other languages are held for this entry yet.
Article bodies are English and are not machine-translated. A name in another language is not a translation of the article.
In deep learning, the transformer is a family of artificial neural network architectures based on the multi-head attention mechanism, in which input data such as text, images, or audio is converted to a sequence of numerical representations called tokens, and each token is converted into a vector through an embedding layer. At each layer, each token is then contextualized within the scope of the context window with other (unmasked) tokens via a parallel multi-head attention mechanism, allowing the signal for key tokens to be amplified and less important tokens to be diminished. Because self-attention alone is permutation-invariant, transformers inject positional information, typically through positional encodings or learned positional embeddings, so token order can affect the output. Because transformers do not process tokens one at a time, their computations can be parallelized across sequence positions during training more readily than those of recurrent neural networks like long short-term memory (LSTM). Later variations have been widely adopted for training large language models (LLMs) on large (language) datasets. The original version of the transformer architecture was proposed in the 2017 paper "Attention Is All You Need" by researchers at Google.…
In deep learning, the transformer is a family of artificial neural network architectures based on the multi-head attention mechanism, in which input data such as text, images, or audio is converted to a sequence of numerical representations called tokens, and each token is converted into a vector through an embedding layer. At each layer, each token is then contextualized within the scope of the context window with other (unmasked) tokens via a parallel multi-head attention mechanism, allowing the signal for key tokens to be amplified and less important tokens to be diminished. Because self-attention alone is permutation-invariant, transformers inject positional information, typically through positional encodings or learned positional embeddings, so token order can affect the output. Because transformers do not process tokens one at a time, their computations can be parallelized across sequence positions during training more readily than those of recurrent neural networks like long short-term memory (LSTM). Later variations have been widely adopted for training large language models (LLMs) on large (language) datasets. The original version of the transformer architecture was proposed in the 2017 paper "Attention Is All You Need" by researchers at Google. The predecessors of transformers were developed as an improvement over previous architectures for machine translation, but have found many applications since. They are used in large-scale natural language processing, computer vision (vision transformers), reinforcement learning, audio, multimodal learning, robotics, and playing chess. It has also led to the development of pre-trained systems, such as generative pre-trained transformers (GPTs). In a masked task, one or more of the tokens is masked out, and the model would produce a probability distribution predicting what the masked-out tokens are based on the context. The loss function for the task is typically sum of log-perplexities for the masked-out tokens: \text{Loss} = -\sum_{t\in\text{masked tokens}}\ln(\text{probability of }t\text{ conditional on its context}) and the model is trained to minimize this loss function. The BERT series of models are trained for masked token prediction and another task. ("Masked" as in "masked language modelling" is not "masked" as in "masked attention".) In an autoregressive task, the entire sequence is masked at first, and the model produces a probability distribution for the first token. Then the first token is revealed and the model predicts the second token, and so on. The loss function for the task is still typically the same. The GPT series of models are trained by autoregressive tasks. In a prefixLM task, the sequence is divided into two parts. The first part is presented as context, and the model predicts the first token of the second part. Then that would be revealed, and the model predicts the second token, and so on. The loss function for the task is still typically the same. The T5 series of models are trained by prefixLM tasks. ("PrefixLM" as in "prefix language modeling" is not "prefixLM" as in "prefix language model".) Each integer token identifier is converted into an embedding vector via a lookup table. Equivalently stated, it multiplies a one-hot representation of the token identifier by an embedding matrix M . For example, if the input token's identifier is 3 , then the one-hot representation is [0, 0, 0, 1, 0, 0, \dots] , and its embedding vector is \mathrm{Embed}(3) = [0, 0, 0, 1, 0, 0, \dots]M The token embedding vectors are added to their respective positional encoding vectors (see below), producing the sequence of input vectors. The dimension of an embedding vector is called hidden size or embedding size and written as d_{\text{emb}} . This size is written as d_{\text{model}} in the original transformer paper. The un-embedding layer is a linear-softmax layer: \mathrm{UnEmbed}(x) = \mathrm{softmax}(xW + b) The matrix has shape (d_{\text{emb}}, |V|) . Some architectures use the transpose of the embedding matrix M as the un-embedding matrix W in order to avoid needing double the amount of embedding-related parameters and to avoid divergence during training. This practice is called weight tying. The positional encoding is defined as a function of type f: \R \to \R^d , where d is a positive even integer. The full positional encoding defined in the original paper is: (f(t)_{2k}, f(t)_{2k+1}) = (\sin(\theta), \cos(\theta)) \quad \forall k \in \{0, 1, \ldots, d/2 - 1\} where \theta = \frac{t}{r^k}, r = N^{2/d} . Here, N is a free parameter that should be significantly larger than the biggest k that would be input into the positional encoding function. The original paper uses N=10000 . The function is in a simpler form when written as a complex function of type f: \R \to \mathbb C^{d/2} f(t) = \left(e^{it/r^k}\right)_{k=0, 1, \ldots, \frac d 2 - 1} where r = N^{2/d} . The main reason for using this positional encoding function is that using it, shifts are linear transformations: f(t + \Delta t) = \mathrm{diag}(f(\Delta t)) f(t) where \Delta t \in \R is the distance one wishes to shift. This allows the transformer to take any encoded position, and find the encoding of the position n-steps-ahead or n-steps-behind, by a matrix multiplication. By taking a linear sum, any convolution can also be implemented as linear transformations: \sum_j c_j f(t + \Delta t_j) = \left(\sum_j c_j \,\mathrm{diag}(f(\Delta t_j))\right) f(t) for any constants c_j . This allows the transformer to take any encoded position and find a linear sum of the encoded locations of its neighbors. This sum of encoded positions, when fed into the attention mechanism, would create attention weights on its neighbors, much like what happens in a convolutional neural network language model. In the author's words, "we hypothesized it would allow the model to easily learn to attend by relative position." In typical implementations, all operations are done over the real numbers, not the complex numbers, but since complex multiplication can be implemented as real 2-by-2 matrix multiplication, this is a mere notational difference. The feedforward network (FFN) modules in a transformer are 2-layered multilayer perceptrons: \mathrm{FFN}(x) = \phi(xW^{(1)} + b^{(1)})W^{(2)} + b^{(2)} where W^{(1)} and W^{(2)} are weight matrices and b^{(1)} and b^{(2)} are bias vectors, and \phi is its activation function. The original transformer used ReLU activation. The number of neurons in the middle layer is called intermediate size (GPT), filter size (BERT), or feedforward size (BERT). It is typically larger than the embedding size. For example, in both GPT-2 series and BERT series, the intermediate size of a model is 4 times its embedding size: d_{\text{ffn}} = 4 d_{\text{emb}} . For each vector x_{i, \text{query}} in the query sequence, it is multiplied by a matrix W^Q to produce a query vector q_i = x_{i, \text{query}} W^Q . The matrix of all query vectors is the query matrix: Q = X_{\text{query}} W^Q Similarly, we construct the key matrix K = X_{\text{key}} W^K and the value matrix V = X_{\text{value}} W^V . It is usually the case that all W^Q, W^K, W^V are square matrices, meaning d_{\text{emb, query}}= d_{\text{query}} , etc. Attention weights are calculated using the query and key vectors: the attention weight a_{ij} from token i to token j is the dot product between q_i and k_j . The attention weights are divided by the square root of the dimension of the key vectors, \sqrt{d_k} , which stabilizes gradients during training, and passed through a softmax which normalizes the weights. The fact that W^Q and W^K are different matrices allows attention to be non-symmetric: if token i attends to token j (i.e. q_i\cdot k_j is large), this does not necessarily mean that token j will attend to token i (i.e. q_j\cdot k_i could be small). The output of the attention unit for token i is the weighted sum of the value vectors of all tokens, weighted by a_{ij} , the attention from token i to each token. The attention calculation for all tokens can be expressed as one large matrix calculation using the softmax function, which is useful for training due to computational matrix operation optimizations that quickly compute matrix operations. The matrices Q , K and V are defined as the matrices where the i th rows are vectors q_i , k_i , and v_i respectively. Then we can represent the attention as \begin{align} \text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^\mathrm{T}}{\sqrt{d_k}}\right)V \end{align} where the softmax is applied over each of the rows of the matrix. The number of dimensions in a query vector is query size d_{\text{query}} and similarly for the key size d_{\text{key}} and value size d_{\text{value}} . The output dimension of an attention head is its head dimension d_{\text{head}} . The attention mechanism requires the following three equalities to hold: \ell_{\text {seq, key}}=\ell_{\text {seq, value}}, \;d_{\text {query}}=d_{\text {key}}, \; d_{\text {value}}=d_{\text {head}} but is otherwise unconstrained. If the attention head is used in a self-attention fashion, then X_{\text{query}} = X_{\text{key}} = X_{\text{value}} . If the attention head is used in a cross-attention fashion, then usually X_{\text{query}} \neq X_{\text{key}} = X_{\text{value}} . It is theoretically possible for all three to be different, but that is rarely the case in practice. Concretely, let the multiple attention heads be indexed by i , then we have \text{MultiheadAttention}(Q, K, V) = \text{Concat}_{i \in [n_{\text{heads}}]}(\text{Attention}(XW^Q_i, XW^K_i, XW^V_i)) W^O where the matrix X is the concatenation of word embeddings, and the matrices W^Q_i, W^K_i, W^V_i are "projection matrices" owned by individual attention head i , and W^O is a final projection matrix owned by the whole multihead attention head. It is theoretically possible for each attention head to have a different head dimension d_{\text{head}} , but that is rarely the case in practice. As an example, in the smallest GPT-2 model, there are only self-attention mechanisms. It has the following dimensions: d_{\text{emb}} = 768, n_{\text{head}} = 12, d_{\text{head}} = 64 Since 12 \times 64 = 768 , its output projection matrix W^O \in \R^{(12 \times 64) \times 768} is a square matrix. The transformer architecture is constructed to calculate output tokens iteratively. Assuming t = 0 refers to the calculation of the first output token i = 0 , for step t > 0 , the output token i = 0 shall remain constant. This ensures properties of the model similar to autoregressive models. Therefore, at every time step t , the calculation for all outputs i should not have access to tokens at position j for j >= i (as it naturally is the case for time step t=i , when tokens j>t are not yet calculated). This behavior may be accomplished before the softmax stage by adding a mask matrix M that is -\infty at entries where the attention link must be cut, and 0 at other places: \begin{align} \text{MaskedAttention}(Q, K, V) = \text{softmax}\left(M + \frac{QK^\mathrm{T}}{\sqrt{d_k}}\right)V \end{align} The following matrix is commonly used in decoder self-attention modules, called "causal masking": M_{\text{causal}} = \begin{bmatrix} 0 & -\infty & -\infty & \dots & -\infty \\ 0 & 0 & -\infty & \dots & -\infty \\ 0 & 0 & 0 & \dots & -\infty \\ \vdots & \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & 0 & \dots & 0 \end{bmatrix} In words, it means that each token can pay attention to itself, and every token before it, but not any after it. A non-masked attention module can be thought of as a masked attention module where the mask has all entries zero. As an example of an uncommon use of mask matrix, the XLNet considers all masks of the form P M_{\text{causal}} P^{-1} , where P is a random permutation matrix. \text{combine them into a matrix } H &= \begin{bmatrix} h_0 \\ h_1 \\ \vdots \end{bmatrix} \\ \text{EncoderLayer}(H) &= \begin{bmatrix} \text{FFN}(\text{MultiheadAttention}(H, H, H)_0) \\ \text{FFN}(\text{MultiheadAttention}(H, H, H)_1) \\ \vdots \end{bmatrix} \\ \end{aligned} where \text{FFN} stands for "feed-forward network". We can more succinctly write it as \text{EncoderLayer}(H) = \text{FFN}(\text{MultiheadAttention}(H, H, H)) with the implicit convention that the \text{FFN} is applied to each row of the matrix individually. The encoder layers are stacked. The first encoder layer takes the sequence of input vectors from the embedding layer, producing a sequence of vectors. This sequence of vectors is processed by the second encoder, and so on. The output from the final encoder layer is then used by the decoder. As the encoder processes the entire input all at once, every token can attend to every other token (all-to-all attention), so there is no need for causal masking. Schematically, we have: \begin{aligned} H' &= \text{MaskedMultiheadAttention}(H, H, H) \\ \text{DecoderLayer}(H) &=\text{FFN}(\text{MultiheadAttention}(H', H^E, H^E)) \end{aligned} where H^E is the matrix with rows being the output vectors from the encoder. The last decoder is followed by a final un-embedding layer to produce the output probabilities over the vocabulary. Then, one of the tokens is sampled according to the probability, and the decoder can be run again to produce the next token, etc., autoregressively generating output text. There are two common conventions in use: the post-LN and the pre-LN convention. In the post-LN convention, the output of each sublayer is \mathrm{LayerNorm}(x + \mathrm{Sublayer}(x)) where \mathrm{Sublayer}(x) is the function implemented by the sublayer itself. In the pre-LN convention, the output of each sublayer is x + \mathrm{Sublayer}(\mathrm{LayerNorm}(x)) The original 2017 transformer used the post-LN convention. It was difficult to train and required careful hyperparameter tuning and a "warm-up" in learning rate, where it starts small and gradually increases. The pre-LN convention, proposed several times in 2018, was found to be easier to train, requiring no warm-up, leading to faster convergence. A "prefixLM" (prefix language model) is a decoder-only architecture, but with prefix masking, which is different from causal masking. Specifically, it has mask of the form M_{\text{prefixLM}} = \begin{bmatrix} \mathbf{0} & -\infty \\ \mathbf{0} & M_{\text{causal}} \end{bmatrix} where the first columns correspond to the "prefix", and the subsequent columns correspond to the autoregressively generated text based on the prefix. They resemble encoder–decoder models, but has less "sparsity". Such models are rarely used, though they are cited as theoretical possibilities and benchmarked comparisons. There are also mixed seq2seq models. For example, in 2020, Google Translate replaced the previous RNN-encoder–RNN-decoder model with a transformer-encoder–RNN-decoder model, as transformer-based decoders did not appear to significantly increase quality unlike the encoder, while the RNN decoder was much faster. RoPE (rotary positional embedding), is best explained by considering a list of 2-dimensional vectors [(x^{(1)}_1, x^{(2)}_1), (x^{(1)}_2, x^{(2)}_2), (x^{(1)}_3, x^{(2)}_3), ...] . Now pick some angle \theta . Then RoPE encoding is \text{RoPE}\big(x^{(1)}_m, x^{(2)}_m, m\big) = \begin{pmatrix} \cos m \theta & - \sin m \theta \\ \sin m \theta & \cos m \theta \end{pmatrix} \begin{pmatrix} x^{(1)}_m \\ x^{(2)}_m \\ \end{pmatrix} = \begin{pmatrix} x^{(1)}_m \cos m\theta - x^{(2)}_m \sin m \theta \\ x^{(2)}_m \cos m\theta + x^{(1)}_m \sin m \theta \\ \end{pmatrix} Equivalently, if we write the 2-dimensional vectors as complex numbers z_m := x^{(1)}_m + i x^{(2)}_m , then RoPE encoding is just multiplication by an angle: \text{RoPE}\big(z_m, m\big) = e^{i m\theta} z_m For a list of 2n -dimensional vectors, a RoPE encoder is defined by a sequence of angles \theta^{(1)}, ..., \theta^{(n)} . Then the RoPE encoding is applied to each pair of coordinates. The benefit of RoPE is that the dot-product between two vectors depends on their relative location only: \text{RoPE}\big(x, m\big)^T\text{RoPE}\big(y, n\big) = \text{RoPE}\big(x, m+k\big)^T\text{RoPE}\big(y, n+k\big) for any integer k . ALiBi (Attention with Linear Biases) is not a replacement for the positional encoder on the original transformer. Instead, it is an additional positional encoder that is directly plugged into the attention mechanism. Specifically, the ALiBi attention mechanism is \begin{align} \text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^\mathrm{T}}{\sqrt{d_k}} + s B\right)V \end{align} Here, s is a real number ("scalar"), and B is the linear bias matrix defined by B = \begin{pmatrix} 0 & 1 & 2 & 3 & \cdots \\ -1 & 0 & 1 & 2 & \cdots \\ -2 & -1 & 0 & 1 & \cdots \\ -3 & -2 & -1 & 0 & \cdots \\ \vdots & \vdots & \vdots & \vdots & \ddots \\ \end{pmatrix} in other words, B_{i, j} = j - i . The idea being that the linear bias matrix is a softened mask. Just as 0 represent full attention paid, and -\infty represents no attention paid, the linear bias matrix increases attention paid in one direction and decreases attention paid in the other direction. ALiBi allows pretraining on short context windows, then fine-tuning on longer context windows. Since it is directly plugged into the attention mechanism, it can be combined with any positional encoder that is plugged into the "bottom" of the entire network (which is where the sinusoidal encoder on the original transformer, as well as RoPE and many others, are located).
Relative Position Encodings
Relative Position Encodings is similar to ALiBi, but more generic: \begin{align} \text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^\mathrm{T}}{\sqrt{d_k}} + B\right)V \end{align} where B is a Toeplitz matrix, that is, B_{i, j} = B_{i', j'} whenever i-j = i'-j' . This is contrasted with the original sinusoidal positional encoding, which is an "absolute positional encoding". The FlashAttention method is a communication-avoiding algorithm that fuses these operations into a single loop, increasing the arithmetic intensity. It is an online algorithm that computes the following quantities: \begin{aligned} z_i &= q^T k_i &\\ m_i &= \max(z_1, \dots, z_i) &={}& \max(m_{i-1}, z_i)\\ \ell_i &= e^{z_1 - m_i} + \dots + e^{z_i - m_i} &={}& e^{m_{i-1}-m_i} \ell_{i-1} + e^{z_i - m_i}\\ o_i &= e^{z_1 - m_i} v_1 + \dots + e^{z_i - m_i}v_i &={}& e^{m_{i-1}-m_i} o_{i-1} + e^{z_i - m_i}v_i \end{aligned} and returns o_N/\ell_N . In practice, FlashAttention operates over multiple queries and keys per loop iteration, in a similar way as blocked matrix multiplication. If backpropagation is needed, then the output vectors and the intermediate arrays [m_1, \dots, m_N], [\ell_1, \dots, \ell_N] are cached, and during the backward pass, attention matrices are rematerialized from these, making it a form of gradient checkpointing. An improved version, FlashAttention-2, was developed to handle longer context lengths. \text{MultiheadAttention}(Q, K, V) = \text{Concat}_{i \in [n_{\text{heads}}]}\left(\text{Attention}(XW^Q_i, XW^K_i, XW^V_i)\right) W^O with Multi-Query Attention, there is just one W^K, W^V , thus: \text{MultiQueryAttention}(Q, K, V) = \text{Concat}_{i \in [n_{\text{heads}}]}\left(\text{Attention}(XW^Q_i, XW^K, XW^V)\right) W^O This has a neutral effect on model quality and training speed, but increases inference speed. More generally, grouped-query attention (GQA) partitions attention heads into groups, each of which shares the key-value pair. MQA is GQA with one group, while standard Multihead Attention is GQA with the maximal number of groups. Multihead Latent Attention (MLA) is a low-rank approximation to standard MHA. Specifically, each hidden vector, before entering the attention mechanism, is first projected to two low-dimensional spaces ("latent space"), one for query and one for key-value (KV vector). This design minimizes the KV cache, as only the low-dimensional KV vector needs to be cached. Random Feature Attention (2021) uses Fourier random features: \varphi(x) = \frac{1}{\sqrt D}[\cos\langle w_1, x\rangle, \sin\langle w_1, x\rangle, \cdots \cos\langle w_D, x\rangle, \sin\langle w_D, x\rangle]^T where w_1, ..., w_D are independent samples from the normal distribution N(0, \sigma^2 I) . This choice of parameters satisfy \mathbb E[\langle \varphi(x), \varphi(y)\rangle] = e^{-\frac{\|x-y\|^2}{2\sigma^2}} , or e^{\langle x, y\rangle/\sigma^2} = \mathbb E[\langle e^{\|x\|^2/2\sigma^2} \varphi(x), e^{\|y\|^2/2\sigma^2}\varphi(y)\rangle] \approx \langle e^{\|x\|^2/2\sigma^2} \varphi(x), e^{\|y\|^2/2\sigma^2}\varphi(y)\rangle Consequently, the one-headed attention, with one query, can be written as \text{Attention}(q, K, V) = \text{softmax}\left(\frac{qK^\mathrm{T}}{\sqrt{d_k}}\right)V \approx \frac{\varphi(q)^T \sum_i e^{\|k_i\|^2/2\sigma^2}\varphi(k_i) v_i^T}{\varphi(q)^T \sum_i e^{\|k_i\|^2/2\sigma^2}\varphi(k_i)} where \sigma = d_K^{1/4} . Similarly for multiple queries, and for multihead attention. This approximation can be computed in linear time, as we can compute the matrix \varphi(k_i) v_i^T first, then multiply it with the query. In essence, we have managed to obtain a more precise version of \text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^\mathrm{T}}{\sqrt{d_k}}\right)V \approx Q(K^TV/\sqrt{d_k}) Performer (2022) uses the same Random Feature Attention, but w_1, ..., w_D are first independently sampled from the normal distribution N(0, \sigma^2 I) , then they are Gram–Schmidt processed.
Text derived from https://en.wikipedia.org/w/index.php?oldid=1378958346 under CC BY-SA 4.0; modifications have been made.