07 · Multi-Head Latent Attention

Pack many head memories into one small latent .

Instead of caching a separate expanded key and value for every attention head, Multi-Head Latent Attention compresses each token into a shared latent vector. When attention needs a head-specific view, it reconstructs that view from the compact cache.

low-rank KV compression + decoupled position key · deepseek-ai 2024

Read the paper

part 1 of 5

Many heads create many cached copies

Each attention head wants its own key and value view of every earlier token. Ordinary attention stores all those expanded views.

Add more attention heads

Ordinary multi-head attention saves a separate key and value for every head and every earlier token.

attention heads6

ordinary cache

K1 · V1

K2 · V2

K3 · V3

K4 · V4

K5 · V5

K6 · V6

12 saved head-vectors per token

MLA cache

one latent

compressed content

One compact representation is cached, even as head count grows.

part 2 of 5

Compress once before caching

MLA squeezes each token’s content through a learned low-rank bottleneck. Only the compact latent representation is carried forward in the cache.

Pack sixteen details through a narrow doorway

The model learns which combinations of details are worth preserving in the latent vector.

latent width4

→

input details

cached numbers

compression

4.0×

This narrow doorway is a low-rank bottleneck. MLA caches what comes through, not every expanded K/V head.

part 3 of 5

Expand the latent into whichever view is needed

Compression does not remove the attention heads. Learned projections unpack different K/V views from the same cached latent.

Unpack a different view for each head

The cached latent is shared. Learned projections expand it into the head-specific keys and values needed now.

same latent

cached once

→

head looking for people

retrieves · Maya

part 4 of 5

Keep a small position tag outside the suitcase

Rotary position information does not compress cleanly with content. MLA separates a small position-bearing key from the compressed content latent.

Move the same word through the sentence

Its meaning can stay the same while its position tag changes.

token position5

compressed content · cached

Tokyo

The content latent does not change when the word moves.

position key · kept separately

RoPE rotates this small position-bearing part.

MLA separates content from position. The large content cache is compressed; a much smaller positional key remains explicit.

part 5 of 5

Trade cache traffic for reconstruction work

A narrower latent makes decoding move far less memory, but asks the model to reconstruct useful head views from a tighter bottleneck.

Choose the suitcase size

This toy head stack would normally cache 128 values per token. MLA stores a latent plus a small position key.

context

latent width24

expanded K/V cache per token128 units

MLA latent + position key32 units

tokens

64K

cache reduction

4.0×

head views

unchanged

DeepSeek-V2 reported a 93.3% KV-cache reduction for its full model configuration. The calculator above is deliberately a simpler relative example.

canonical paper

DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model DeepSeek-AI · 2024

Loading the interactive visualization…

03 · Reference notes

← All visual guides

Multi-head Latent Attention Explained

Multi-head latent attention (MLA) reduces KV-cache memory by storing a compressed latent representation and reconstructing per-head keys and values during decoding.

01 · Definition

What is Multi-head Latent Attention?

Instead of keeping a complete key and value vector for every head and every past token, MLA saves a smaller shared latent. The model projects that latent into the head-specific views only when it needs them.

02 · Mechanism

The cache problem MLA targets

Standard multi-head attention keeps separate keys and values for each head. That preserves flexibility, but the cache grows with sequence length, layers, heads, and hidden size. For long-context inference, those stored vectors become a practical bottleneck.

03 · Mechanism

Compress once, reconstruct when needed

MLA writes a compact latent vector to the cache. At the next decoding step, learned projections recover the key and value information required by each head. The cache stores the compressed source rather than all expanded per-head representations.

04 · Mechanism

Why positional information needs care

Attention needs position-aware keys as well as content. MLA separates the parts that can be compressed from the positional components that must remain available, allowing compression without losing the model's ability to distinguish where tokens occurred.

05 · Mechanism

Where MLA pays off

MLA is most useful when KV-cache capacity or bandwidth constrains long-context serving. It trades some projection work at decode time for a much smaller persistent memory footprint.

Source · Original paper

DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model ↗DeepSeek-AI · 2024