Eight ways a model pays attention.

Frontier models keep reinventing the same operation. Explore the tricks that make attention faster, smaller, longer-lived, and strange enough to be interesting.

mechanisms

bottlenecks

model families

find your route

What should attention fix?

8 guides found

1 · choose the bottleneck

2 · narrow by model familyRepresentative architectural uses, not an exhaustive model list.

01exact · io-aware

FlashAttention

Exact, IO-aware attention that tiles Q, K and V in SRAM instead of materializing the full attention matrix.

LlamaMistral

↗

02kv sharing

Grouped-Query

Groups query heads around shared key and value heads to reduce KV-cache memory during decoding.

MistralGemmaLlama

↗

03compressed kv

Multi-head Latent

Stores a compressed latent KV representation, then reconstructs the per-head view at decode time.

DeepSeekKimi

↗

04local attention

Sliding Window

Limits each token to a local context window, reducing cost while stacked or global layers carry long-range information.

MistralGemma

↗

05native sparse attention

DeepSeek Sparse

Uses a learned indexer to shortlist relevant tokens before applying expensive attention to the selected subset.

DeepSeek

↗

06linear / recurrent

Gated DeltaNet

Uses a fixed recurrent state and gates to decide what sequence information to write or forget.

Qwen

↗

07fine-grained memory

Kimi Delta Attention

Uses fine-grained per-channel gates to control how a fixed recurrent memory is written and forgotten.

Kimi

↗

08serving

Paged Attention

Stores KV-cache entries in fixed-size blocks so serving systems can allocate and share GPU memory efficiently.

vLLM

↗

“Attention takes every direction at a fork, then merges the paths back together.”

· Olah & Carter, Distill, 2016