05 · DeepSeek Sparse Attention

Search everything quickly. Read only the best matches .

A long document may contain tens of thousands of earlier tokens, while the answer needs only a handful. DeepSeek Sparse Attention sends a cheap scout across the whole past, keeps a short list of promising tokens, and runs full attention only on that list.

lightning indexer + token-level top-k selection · deepseek-ai 2025

Read the paper

part 1 of 5

Most questions need only a few old words

Ordinary attention checks the entire past for every new token. Start with a small document and switch questions to see how little of it actually supplies each answer.

Ask the document a question

The answer uses only a few words, but ordinary attention still checks every word.

ProjectOrionbeganinMarch.Mayaapprovedtherevisedbudgetof$2.4M.ThedesignteammeetsinLisbon.FinaldeliveryisdueonOctober18.

ordinary attention checks

all 25 words

answer came from

Maya

part 2 of 5

A fast scout ranks the whole past

DSA first runs a small learned scorer over earlier tokens. It does not produce the final answer; it only decides which token positions deserve the expensive attention calculation.

Let a fast scout make the shortlist

The scout gives every earlier token a quick relevance score. Drag the shortlist size and watch the cutoff move.

tokens to keep6

shortlist

best 6 of 32 tokens

cutoff score · 0.45

This scout is the lightning indexer. It is deliberately smaller and cheaper than the full attention calculation.

part 3 of 5

Select individual tokens, not fixed regions

The useful clues may be far apart. DSA can gather exact token positions from across the context instead of reading a whole window or every token inside a chosen block.

Keep the exact words, not their whole neighbourhood

Relevant tokens can be scattered across a long document. Compare selecting four whole blocks with selecting the four useful tokens directly.

useful tokens

tokens actually read

extra work

DSA uses fine-grained token selection. It can pick individual tokens from anywhere instead of paying for every neighbour in a selected block.

part 4 of 5

The shortlist and the answer are separate jobs

The lightning indexer chooses where to look. DeepSeek’s main attention mechanism then performs its normal, richer calculation over only those selected key/value entries.

Walk one query through DSA

The indexer chooses where to look. Main attention still decides what those chosen tokens mean.

stage1 of 3

Score everything cheaply

The small indexer scans all earlier tokens and gives each a rough relevance score.

part 5 of 5

Keep the expensive part bounded

A larger shortlist gives main attention more chances to find the right evidence, but costs more work. Set a context and top-k budget to see the central speed-versus-recall trade-off.

Set the search budget

Choose the document length and shortlist size. The indexer still scans the past cheaply; the expensive attention step reads only the shortlist.

earlier tokens

tokens kept

full attention · expensive reads65,536

DSA main attention · expensive reads2,048

context searched

100%

full attention used

reduction

32×

the advantage

The costly attention calculation no longer grows with every available token once the top-k budget is fixed.

the risk

If the indexer misses an important token, main attention never sees it. Selection quality matters as much as speed.

canonical paper

DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models DeepSeek-AI · 2025

Loading the interactive visualization…

05 · Reference notes

← All visual guides

DeepSeek Sparse Attention Explained

DeepSeek Sparse Attention uses a lightweight learned indexer to shortlist relevant tokens before applying expensive attention to the selected subset.

01 · Definition

What is DeepSeek Sparse Attention?

The design keeps a view of the full context without paying full-attention cost for every token pair. It separates finding promising evidence from doing the richer attention computation over that evidence.

02 · Mechanism

Why sparse selection is useful

At very long contexts, most earlier tokens are not equally relevant to the current prediction. Full attention still scores them all. DSA uses a cheaper mechanism to search broadly, then concentrates expensive attention on a small token-level shortlist.

03 · Mechanism

The lightning indexer

A learned lightweight indexer ranks the prior context for a query. Its job is retrieval, not the final attention output: it identifies which token positions are promising enough to receive the main model's more detailed calculation.

04 · Mechanism

Exact tokens instead of coarse blocks

Relevant evidence may be scattered across a document. Selecting individual tokens lets DSA collect a focused set from distant locations rather than reading every token in a large contiguous window.

05 · Mechanism

The quality-cost trade-off

The top-k selection budget controls the trade-off. A larger shortlist improves the chance of retaining useful evidence but increases computation and memory traffic. The central challenge is teaching the indexer to keep the tokens the main attention layer will need.

Source · Original paper

DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models ↗DeepSeek-AI · 2025