02 · Gated DeltaNet

Keep a long history in one small memory .

Ordinary attention saves information for every earlier token, so its memory keeps growing. Gated DeltaNet uses one fixed-size state instead. Each new token reads it, makes a small correction, and moves on.

schlag et al. 2021 · yang et al. 2024 · gated delta networks

Read the paper

part 1 of 5

The growing cache

A transformer adds more saved information for every token. DeltaNet keeps rewriting the same small state. Watch how one memory grows while the other stays the same size.

One stream, two ways to remember it

Each square is one more token.

the first token is arriving…

transformer

Keep every token

0.0 MB

Every token adds another row to the cache.

deltanet

Update one state

1.0 MB

Tokens change the cells, but the state never grows.

The difference: a KV cache grows with the history; the delta state stays the same size.

part 2 of 5

One shared memory

Think of a key as an address and a value as the answer at that address. A new fact changes a few cells in one shared table. The table never gets bigger.

A key points to an answer

A key is the address we may ask for later. Its value is the answer. Each fact changes the same small table instead of adding another saved row.

nextcapital · France→Paris

waitingcapital · Japan→Tokyo

waitingplanet · red→Mars

waitingocean · largest→Pacific

shared statealways 4 × 4

Bright cells show which parts of the same state changed.

part 3 of 5

Read, compare, correct

DeltaNet checks the current answer before it writes. It compares that answer with the new one, finds the difference, and stores only the missing part.

One update, slowed down

Watch one fact move through the delta rule. The state is read before anything is written.

1 · read

Ask the memory

capital · France

current answer?

→

2 · predict

Memory replies

what it knows now

Lyon

→

3 · compare

Find the error

new answer − old answer

Paris − Lyon

→

4 · correct

Update the state

write only the missing part

Paris

First, the key asks the current state for an answer.

The whole idea: check what memory already says, then store only the correction.

part 4 of 5

Control the update

Similar addresses can interfere with each other. The correction keeps the update focused, while two learned gates decide how much old memory survives and how strongly the new fact is written.

Similar addresses can overlap

Make nearby addresses more alike. A full write starts to mix their answers. A correction changes the target while disturbing its neighbours less.

address overlap25%

write the full answer every time

Answers start to mix

near

0.06

near

0.50

target

1.13

near

0.50

target share · 51%

read first, then write the difference

The target stays clearer

near

-0.01

near

0.03

target

0.88

near

0.50

target share · 62%

1. Read: what does memory say now?

→

2. Correct: write only what must change.

Two gates decide what changes

The model chooses both gates for every token. The memory gate controls how much old state survives. The update gate controls how strongly the correction is written.

memory gate · keep old state92%

update gate · write correction72%

recall of an early fact62%

recall of the newest fact88%

A lower memory gate fades older facts. A lower update gate makes the newest fact land softly. The network learns this balance from the token itself.

part 5 of 5

Why it matters

The fixed state is a compact summary, not a perfect archive. It may lose details, but its memory use stays flat as the context grows.

Choose a context length

Increase the context and compare the memory cost.

past tokens

64K

transformer cache

8.0 GB

delta state

1MB

archive every token8.0 GB

update one fixed summary1 MB

the advantage

Here, the growing cache is about 8192× larger. The delta layer keeps updating the same small state.

the trade-off

A small summary can forget or mix details. That is why practical models often combine delta layers with some full-attention layers.

canonical paper

Gated Delta Networks: Improving Mamba2 with Delta Rule Yang et al. · 2024

Loading the interactive visualization…

06 · Reference notes

← All visual guides

Gated DeltaNet Explained

Gated DeltaNet is a recurrent linear-attention mechanism that stores sequence information in a fixed-size state and uses gates to decide what to update or forget.

01 · Definition

What is Gated DeltaNet?

Unlike standard attention, it does not retain a growing KV cache and revisit every past token. It carries a compact state forward through the sequence, updating that state as each new token arrives.

02 · Mechanism

From a growing cache to a fixed state

Standard attention stores one key/value record per token, so memory grows with context. Delta-style recurrence stores a fixed-size memory state instead. That gives predictable memory use and can make long sequences cheaper to process.

03 · Mechanism

The delta rule

A delta update writes new information while correcting what the memory already predicts. Rather than blindly accumulating every token, the state moves toward the representation the current token should contribute.

04 · Mechanism

Why the gate matters

A gate controls how much of the update becomes part of the persistent state. It gives the model a way to preserve useful memory, admit a new fact, or suppress a distracting token—an explicit write decision that a purely additive recurrence lacks.

05 · Mechanism

Where it fits

Gated DeltaNet is attractive when fixed memory and linear scaling are more important than direct access to every past token. The trade-off is that the model must compress its history into the recurrent state rather than retrieve an arbitrary token from a cache.

Source · Original paper

Gated Delta Networks: Improving Mamba2 with Delta Rule ↗Yang et al. · 2024