04 · Kimi Delta Attention

Give every memory lane its own forget dial .

Kimi Delta Attention carries one small memory board through a conversation. The useful twist is control: names can stay while filler fades, or places can stay while an old action clears. Different parts of memory no longer have to forget at the same speed.

fixed state + delta correction + channel-wise forgetting · kimi team 2025

Read the paper

part 1 of 5

One board travels through the sentence

Full attention can look back through an archive of earlier words. KDA instead carries one fixed-size state, a small board that each new word reads and updates.

Move through one sentence

Each word changes the same little memory board. No new row is added.

word arrivingMaya

sentence so far

MayaflewtoTokyoyesterday.

A name enters the memory.

→

one fixed board4 lanes

people

grammar

actions

places

This fixed board is the state. KDA carries it from word to word instead of keeping a full archive of every earlier word.

part 2 of 5

One forget dial is too blunt

Earlier gated memories used one keep-or-forget number for a whole attention head. That is like putting every lane under one master dimmer: useful facts and disposable details fade together.

Try one master dimmer

The goal is simple: keep the person’s name, but clear an old filler detail. With one control, both fade together.

keep old memory70%

want to keep

person · Maya

still remembered70%

want to clear

filler · “actually”

still remembered70%

Turn the dimmer up and both survive. Turn it down and both disappear. Gated DeltaNet uses this kind of one-number forget gate.

part 3 of 5

KDA controls every lane separately

KDA replaces the master dimmer with many small ones. Each channel can keep or fade at its own speed, chosen from the current word.

Now give every lane its own dimmer

Choose what happens next, then fine-tune any lane yourself.

keep people92%

memory: Maya

keep small words18%

memory: actually

keep actions28%

memory: flew

keep places35%

memory: Tokyo

This is KDA’s key upgrade: a different forget value for each memory channel. Important lanes can stay bright while disposable lanes fade quickly.

part 4 of 5

It still edits with the delta rule

Fine-grained forgetting is KDA’s new part. The actual write remains careful: read the answer already stored, compare it with the new answer, and write only the correction.

Correct the answer already there

KDA first checks its current answer. It writes only the correction needed to move toward the new answer.

1 · ask memory

capital of France

Lyon

→

2 · find the error

wanted answer

Paris

→

3 · write correction

new memory now contains

Lyon→Paris

how strongly to learn the correction85%

old · Lyonnew · Paris

This is the delta rule: read, measure the difference, then write the missing part. KDA keeps this careful edit and improves the forgetting around it.

part 5 of 5

Spend a small memory more carefully

A fixed state cannot preserve every detail forever. KDA’s advantage is not infinite memory; it is better control over what survives inside a limited memory, while avoiding a cache that grows with every word.

Stretch the conversation

Full attention keeps a growing archive. A KDA layer carries the same fixed-size board from word to word.

conversation length

full-attention archive · relative size16×

KDA state · relative size1× fixed

what stays small

The KDA state does not gain a new saved row for every word. Per-word work and recurrent memory stay bounded.

what can be lost

The board is a compressed summary, not a perfect archive. Fine-grained gates help it spend that limited space more carefully.

In the Kimi Linear experiments, the full hybrid model used up to 75% less KV cache and reached up to 6× decoding throughput at one-million-token context. Those are model-level results, not a promise for every KDA layer.

canonical paper

Kimi Linear: An Expressive, Efficient Attention Architecture Kimi Team · 2025

Loading the interactive visualization…

07 · Reference notes

← All visual guides

Kimi Delta Attention Explained

Kimi Delta Attention uses a fixed recurrent memory with fine-grained per-channel gates to control how information is written and forgotten.

01 · Definition

What is Kimi Delta Attention?

It builds on delta-rule recurrence but makes the memory control more granular. Instead of one broad decision for an update, different dimensions of the state can keep, replace, or forget different amounts of information.

02 · Mechanism

Why fine-grained gates matter

Sequence information is not equally useful across every memory channel. A single scalar gate treats a whole update as one decision. Per-channel gates let the architecture preserve one feature while rapidly overwriting another.

03 · Mechanism

A controlled recurrent state

Like other linear recurrent mechanisms, Kimi Delta Attention carries a fixed-size state instead of a KV cache that grows token by token. The current token reads from that state and proposes an update; learned gates decide how that update enters memory.

04 · Mechanism

What it trades

The architecture reduces the memory growth associated with standard attention and can process long streams efficiently. In exchange, relevant history must remain encoded in the state, so the update and forget controls become a key part of model quality.

05 · Mechanism

When to use this pattern

Fine-grained recurrent memory is useful when the model needs both long-running state and selective forgetting. It is a different approach from sparse retrieval: the mechanism manages a compressed memory rather than selecting a subset of stored tokens.

Source · Original paper

Kimi Linear: An Expressive, Efficient Attention Architecture ↗Kimi Team · 2025