06 · PagedAttention

Let the KV cache grow one small page at a time .

A model server cannot predict how long every answer will be. Reserving one large continuous memory region per request wastes GPU space. PagedAttention splits each request’s saved keys and values into small blocks that can live anywhere, then reconnects them with a page table.

virtual-memory ideas for LLM serving · kwon et al. 2023 · vllm

Read the paper

part 1 of 5

Unknown answer lengths create empty reservations

Every generated token adds keys and values to a request’s cache. The server must leave room for future tokens, but it does not know how many will arrive. Try the mismatch between reserved and actually used space.

Reserve room for an answer you cannot predict

A server knows the prompt, but not how long the generated answer will be. Drag the actual answer length inside a block reserved for the worst case.

answer tokens actually used7

one contiguous reservation

16 token slots

9 empty

used

reserved

wasted

56%

Those slots hold the request’s KV cache, the keys and values saved for earlier tokens. Large reservations waste memory; small ones risk running out.

part 2 of 5

Allocate memory only when a page fills

PagedAttention divides the growing KV cache into equal-size blocks. A request receives its next block at the last possible moment, so only its final block can be partly empty.

Allocate one small page at a time

The request receives another four-slot page only when the current one fills.

tokens generated11

logical page 0

logical page 1

logical page 2

tokens stored

pages allocated

empty slots

Waste is limited to the final page. Earlier pages are full, and future pages have not been allocated yet.

part 3 of 5

Logical order no longer requires physical order

A request’s blocks can occupy any free locations in GPU memory. A small block table records the mapping, letting the attention kernel read them in the correct sequence.

Keep the story ordered even when memory is scattered

Choose a request. Its page table reconnects logical pages to physical GPU blocks.

page table

logical 0→ block 6

logical 1→ block 1

logical 2→ block 8

→

physical GPU memory

block 0

block 1crossed the

block 2

block 3

block 4

block 5

block 6The red fox

block 7

block 8quiet field

Attention still sees one continuous sequence. PagedAttention follows the table whenever it needs the next key/value block.

part 4 of 5

Shared prompts need only one physical copy

Parallel answers and beam searches often begin with identical tokens. Their page tables can point to the same prompt blocks until one branch writes a different continuation.

Share the prompt until answers diverge

Two candidate answers begin from the same prompt. Toggle the moment they start writing different tokens.

answer A

shared 1

shared 2

shared 3

answer B

shared 1

shared 2

shared 3

shared physical blocks

private blocks

copies avoided

This is copy-on-write. Both answers point to the same prompt blocks. A private block is created only when one answer writes something new.

part 5 of 5

Less waste means more active requests

PagedAttention does not compress the tokens that are actually stored. It avoids premature reservations, scattered unusable gaps, and duplicate shared prefixes, freeing capacity for a larger serving batch.

Put more requests on the same GPU

Add requests with unpredictable lengths. The traditional layout reserves twenty slots for each; paging allocates only the pages each request reaches.

simultaneous requests5

request 1

5 tokens

request 2

13 tokens

request 3

7 tokens

request 4

18 tokens

request 5

10 tokens

contiguous reservations100 slots

paged allocations64 slots

tokens actually stored53 slots

the saving

Paging uses 36% fewer slots in this toy batch. That free memory can hold more active requests.

the cost

Attention must follow a block table and use paging-aware kernels. PagedAttention changes memory management, not the model’s learned attention pattern.

canonical paper

Efficient Memory Management for Large Language Model Serving with PagedAttention Kwon et al. · 2023

Loading the interactive visualization…

08 · Reference notes

← All visual guides

PagedAttention Explained

PagedAttention manages the KV cache in fixed-size blocks, letting LLM serving systems allocate and share GPU memory efficiently without contiguous reservations.

01 · Definition

What is PagedAttention?

It is a serving-system technique rather than a new attention equation. The model's logical KV cache remains one sequence, while the physical GPU blocks that hold it can be allocated wherever space is available.

02 · Mechanism

Why contiguous KV caches waste memory

Serving many requests creates caches of different lengths that grow at different speeds. Reserving one large contiguous region for each sequence leads to fragmentation and over-allocation, especially when generations finish at unpredictable times.

03 · Mechanism

How pages map the cache

PagedAttention divides each KV cache into fixed-size blocks. A small block table maps a sequence's logical token positions to physical GPU blocks, much like virtual memory maps an address space to pages. Attention reads through the table as if the cache were continuous.

04 · Mechanism

Why block sharing helps

When requests share a prompt, their initial KV-cache blocks can be shared instead of copied. New blocks are allocated only as each request generates, which improves memory utilization and supports higher serving throughput.

05 · Mechanism

What PagedAttention changes

The attention math is unchanged. PagedAttention changes the memory manager around the cache so an inference server can pack, grow, release, and sometimes share cache storage more efficiently.

Source · Original paper

Efficient Memory Management for Large Language Model Serving with PagedAttention ↗Kwon et al. · 2023