PagedAttention

PagedAttention is an attention algorithm for efficient serving of large language models (LLMs).

Uses your system voice; voice and rate in settings.

Snapshot

≈ 3 min read

Names in other languages

No names in other languages are held for this entry yet.

Article bodies are English and are not machine-translated. A name in another language is not a translation of the article.

PagedAttention is an attention algorithm for efficient serving of large language models (LLMs). It was introduced in 2023 by Woosuk Kwon and colleagues in the paper Efficient Memory Management for Large Language Model Serving with PagedAttention, alongside the vLLM serving engine.

The method stores the key–value cache used during autoregressive decoding in fixed-size blocks that can be mapped to non-contiguous physical memory, borrowing ideas from virtual memory, paging, and operating system design. a_{ij}=\frac{\exp(\mathbf{q}_i^\top \mathbf{k}_j/\sqrt{d})}{\sum_{t=1}^{i}\exp(\mathbf{q}_i^\top \mathbf{k}_t/\sqrt{d})},\; \mathbf{o}_i=\sum_{j=1}^{i} a_{ij}\mathbf{v}_j where \mathbf{q}_i , \mathbf{k}_j , and \mathbf{v}_j are the query, key, and value vectors, and d is the attention dimension. If the cache is partitioned into blocks of size B , the key and value blocks may be written as \mathbf{K}_j=(\mathbf{k}_{(j-1)B+1},\ldots,\mathbf{k}_{jB}),\; \mathbf{V}_j=(\mathbf{v}_{(j-1)B+1},\ldots,\mathbf{v}_{jB}) PagedAttention then performs the computation blockwise: \mathbf{A}_{ij}= \frac{\exp(\mathbf{q}_i^\top \mathbf{K}_j/\sqrt{d})} {\sum_{t=1}^{\lceil i/B \rceil}\exp(\mathbf{q}_i^\top \mathbf{K}_t/\sqrt{d})},\; \mathbf{o}_i=\sum_{j=1}^{\lceil i/B \rceil}\mathbf{V}_j\mathbf{A}_{ij}^{\top} where \mathbf{A}_{ij} is the vector of attention scores for the j -th KV block.

In the formulation given by Kwon et al., this preserves the causal attention calculation while allowing the key and value blocks to reside in non-contiguous physical memory. \operatorname{Attention}(q_i,K,V)=\operatorname{softmax}\left(\frac{q_iK^\top}{\mathrm{scale}}\right)V. In the notation of Prabhu et al., the key and value tensors for a request seen so far are K,V \in \mathbb{R}^{L' \times (H \times D)} , where L' is the context length seen so far, H is the number of KV heads on a worker, and D is the dimension of each KV head. In systems prior to PagedAttention, the K cache (or V cache) at each layer of a worker is typically allocated as a 4D tensor of shape [B, L, H, D], where B is batch size and L is the maximum context length supported by the model. vAttention preserves this contiguous virtual-memory view while deferring physical-memory allocation to runtime.

A serving framework maintains separate K and V tensors for each layer, so vAttention reserves 2N virtual-memory buffers on a worker, where N is the number of layers managed by that worker. The maximum size of one virtual-memory buffer is BS = B \times S, where S is the maximum size of a single request's per-layer K cache (or V cache) on a worker.

The paper defines S = L \times H \times D \times P, where P is the number of bytes needed to store one element. In this formulation, vAttention keeps the KV cache contiguous in virtual memory and relies on demand paging for physical allocation, rather than modifying the attention kernel to operate over non-contiguous KV-cache blocks.

Text derived from https://en.wikipedia.org/w/index.php?oldid=1359422630 under CC BY-SA 4.0; modifications have been made.

Sources cited in this revision

No external citation was extracted for this revision. That is a statement about the source, not a claim that none exists.